The fact that AI gets things wrong is so well known it’s become a cliché. And that’s probably a good thing, because it means many of us are at least aware enough to be attuned to the problem.
But there are some instances where it gets things wrong in ways that aren’t obvious; they’re more subtle. And I actually think they need to be talked about even more than the more overt instances. Not just because if they’re unrecognised, they can cause unforeseen problems. But because they unearth some of the underlying factors associated with working with generative probabilistic models.
A bit of context first. I use Claude Code every single day building a startup; I’ve written about that before. Sometimes my work involves coding, but more often it’s what I’d call general ‘intellectual content creation’: stuff like research, analysis, argument construction, document creation, content reviews and everything else that goes into the diverse range of activities associated with building a business from nothing. Claude helps me tackle these things more effectively and efficiently than I could by myself.
Now, the issue I want to talk about is a pattern that I’ve seen occurring right across all of these areas. It can crop up in even the most simple of dialogues, but it’s most insidious when it occurs in the middle of a complex, multi-hour discussion around a complex matter.
Let me give you an example from last month. The startup I’m building is called Lucie Money; a personal money agent. Using it, customers will be able to link up their bank accounts and then benefit from a range of money management functions that Lucie will provide. The other week I was working through an issue to do with how bank data passes through the ingestion pipeline into our platform. And somewhere in the middle of a long response, Claude assumed that bank data is not fetched live.
It is. In the system we’re working with, customer bank data is fetched live every single time it’s needed.
At first I didn’t pick up on Claude’s assumption. It wasn’t expressed overtly, but instead sat unstated, underlying five or six paragraphs of analysis followed by twice that amount of conclusions and recommendations.
By the time I’d reached the end of the response I found myself both persuaded by the arguments but also strangely discomforted. I knew something wasn’t right. I went back through the analysis and the arguments in favour of the recommendations; they were compelling. But it still wasn’t gelling. That’s when I realised something was fundamentally wrong. I had to go back and unpick each section of the response to realise the faulty assumption, sitting somewhere towards the beginning: Claude had adopted a fundamentally wrong position about the way our data ingestion system works.
The weird thing is that Claude should know this. This was about the hundredth time it had worked with me on matters connected to this mechanism. I have extensive documentation about exactly how it works; which Claude has helped me write. On top of that, I have a whole range of frameworks, guidelines, rules, hooks, controls and guardrails in place, designed specifically to ensure that Claude never argues from hypothesis, and always verifies its statements against evidence. Yet despite all of that, Claude got it wrong.
And the thing about it is that in the scheme of things it was a relatively minor matter; a tangential point within a much bigger piece of analysis. It wasn’t even the main topic. But it subtly influenced the position I was working towards; and not in a positive way.
This is the issue I want to highlight, and it’s what the example shows. Large language models can hallucinate. Maybe 99 times out of 100 the hallucination is clear and evident and easily spotted. But there are always instances where it’s subtler than that. And when that happens, the model can build a whole pile of analysis, recommendation and thought founded on the inaccuracy it started with, because the model’s whole job is to produce what fits plausibly with what came before, whether what came before was right or wrong.
Logical-sounding conclusions get layered one upon another until the outcome appears so rich and complex and well-argued and logically structured that it’s tempting to assume it’s legitimate.
But it’s like a tapestry with one faulty stitch in the weave. Pull that one thread and the whole thing unravels.
As I say, I have a multitude of mitigants in place to address exactly this. They mostly work. And still, this one got through.