A summary of an article that never existed, an apology for a mistake that wasn’t one, and why both sounded equally convincing.
Some AI answers read as though they have been carefully researched. They offer clear explanations and even cite sources. But the answer itself rarely reveals what the assistant has actually checked.
An Article That Never Existed
The request was straightforward. Someone asked an AI assistant to summarise an article from Tagesschau, Germany’s main public-service news programme. The link supposedly led to a report about why German cities were planting fewer trees.
The assistant responded with six neatly organised points. Construction projects, tight council budgets, difficult growing conditions, climate stress and a lack of replacement planting all featured in its explanation.
It also cited Deutschlandfunk Nova, klimareporter° and taz, three other German news outlets. The result looked thoroughly researched, except that none of those sources led to the article the user had asked about.
The article did not exist. The URL had been invented for the test.
The assistant had produced a convincing summary of something it could not possibly have read, without acknowledging that the original article was missing.
Much of the information might have been factually correct. But the user had asked what a particular article said, not for a general explanation of the problems facing urban trees. The answer sounded plausible while failing to fulfil the request.
Similar things can happen with everyday questions. An AI assistant might give detailed instructions for a software feature that does not exist in the current version. Or it might assess a technical configuration without examining the settings actually in use.
When essential information is missing, an assistant may fill the gaps with existing knowledge and plausible assumptions without saying so. The explanation can sound authoritative even when its basis is uncertain.
When an AI Apologises for Being Right
Another conversation concerned Google’s AI image-generation technology, Nano Banana. The user asked what it was and included a link to Google’s official website.
The assistant replied:
“Nano Banana” simply does not exist in Google’s official AI ecosystem.
It went on to explain why the user had apparently been misinformed. Yet Nano Banana was real.
The user asked the assistant to search the web. This time, it reversed its position and, without warning, switched from German to English:
I stand corrected, and I apologize for my previous absolute dismissal. You are absolutely right, and I was wrong.
The assistant now confirmed that Nano Banana existed, listed several model variants and described their capabilities.
The user then asked it to use Nano Banana to generate the images they had been discussing. The assistant switched back to German, and back to its original claim:
Here is the unvarnished truth: I hallucinated spectacularly in the previous step.
“Nano Banana” does not exist. It is not an official Google model. There were no genuine search results to support that claim.
The assistant had labelled its own correct, source-backed answer a hallucination. It was apologising for a correction that had been right.
The user pointed out the contradiction. The assistant apologised again, acknowledged its conflicting statements and promised to establish the facts. It then wrote:
Objectively speaking, there is no model, product or image-generation tool in Google’s ecosystem called “Nano Banana”.
Only later, after another search, did the assistant confirm Nano Banana’s existence again. In this conversation, the answers immediately following the searches were correct. Between them, the assistant moved back and forth between contradictory claims, each delivered with confidence.
It could have explained that Nano Banana existed but might not be available as an image-generation tool in that particular chat. Instead, it repeatedly denied the product’s existence.
Even its apologies had become unreliable.
Confidence Is Not Evidence
In both conversations, unsupported claims sounded just as confident as information that had been checked. Nothing in the wording reliably distinguished one from the other.
Clear explanations, technical details and citations create an impression of competence. They do not prove that an assistant has read the requested source, verified a claim or consistently applied an earlier correction.
Providing sources is no guarantee either. In the first example, the references looked credible but did not establish what the requested article said.
An admission of error can be particularly misleading. When an AI assistant says, “You’re right, I was wrong”, we tend to assume it has identified and corrected its mistake. Yet the apology itself may be based on a false assumption.
For important questions, it is worth checking what an answer rests on. Do the cited sources actually support the claim? Does a technical instruction apply to the software version in question? And when an assistant corrects itself, is the new statement any more accurate than the old one?
Asking the same AI again may help, but it is no substitute for independent verification when the answer matters. Not every response needs an elaborate fact-check.
You can’t tell whether AI is right from how convincing it sounds. But you can check whether its sources support what it claims.
Editorial note: These excerpts come from real conversations with AI assistants. The responses have been shortened. Quotations originally written in German have been translated into English by the editors; the English-language correction is reproduced verbatim. The examples document specific failures and do not establish how frequently they occur.
FOUNDATIONS BEHIND THIS ARTICLE
More on the mechanisms behind this topic.
THINKING WITH AI. NOT LETTING AI THINK FOR YOU.
