The Genius That Cannot Count to Fifteen

Why “hard” and “easy” lose their meaning with artificial intelligence

You give an AI a long contract. Twenty pages, complicated wording, cross-references and legal terminology. You ask it to explain what you should pay attention to.

A few seconds later, you have a surprisingly useful analysis. The AI identifies important clauses, explains relationships and points out possible issues. A task that might have taken you an hour.

Then you ask it to summarise the result in exactly 15 words.

Perhaps it works immediately. Perhaps it does not. Suddenly there are 17 words. You point that out — and the next attempt has 14.

Errors like this were far from unusual, especially with earlier language models. Similar trivial tests became well known with individual words: How many “r”s are there in “strawberry”? How many “e”s in “seventeen”? Language models could analyse complicated texts and write program code — yet miscount a handful of letters.

Why does it seem so absurd when that is what fails?

Humans make mistakes too. What is irritating is not that the AI makes one. It is where the mistake appears.

THE WRONG MAP

We bring the wrong map

We have a clear idea of what is easy and what is difficult. Shortening a sentence to 15 words seems trivial to us; analysing a complicated contract does not. In everyday human life, this ordering works surprisingly well: if someone can handle advanced mathematics, we assume they can also do basic arithmetic.

We transfer exactly that expectation to AI. If it has just impressed us with something difficult, the thought seems obvious: if it can do that, surely it must be able to handle something easier.

This is where our familiar map misleads us.

Researchers talk about the Jagged Frontier — the irregular boundary of AI capabilities. It describes an imagined boundary between tasks where an AI performs very reliably and others where its performance drops noticeably.

According to our intuition, that boundary should be orderly: simple tasks on one side, then increasingly difficult ones, until eventually the system reaches the limit of what it can do.

In reality, the boundary is irregular and jagged.

An AI can analyse a long document — and miscount the letters in a single word. It can develop a complex argument — and then fail to follow a simple formal instruction such as “Give exactly three points”.

These are not merely amusing AI mishaps. A large study involving 758 business consultants found a similar pattern in professional knowledge work. On many tasks, GPT-4 improved participants’ performance substantially. On another task, however, AI assistance made them more likely to reach the wrong answer — and many consultants adopted the AI’s incorrect assessment.

Once you have seen how well a system handles a demanding task, it is easy to transfer too much of that competence to the next one.

AI capabilities are therefore not random. There are reasons why certain tasks work better than others. But those differences do not reliably follow our familiar scale of “easy” and “hard”.

UNDER THE HOOD

Why “easy” may not be easy for AI

But why can something that seems so simple to us become difficult at all?

The 15-word example gives us a small glimpse under the hood. When we write such a text, we can formulate a sentence, count the words, remove some and then check again. A language model, by contrast, generates language step by step.

Put visually: The tracks are being laid while the train is already moving.

Modern AI can certainly plan ahead and check results. Even so, the difference remains important: a requirement such as “exactly 15 words” applies to the result as a whole, while the text itself is produced step by step. Content, language and formal constraints all have to fit together. What looks to us like one trivial task may consist of very different subproblems for the model.

This does not explain every jagged edge. But it does explain why “That is easy” is not a particularly useful measure for AI.

And the boundary moves.

Letter counting is itself a good example. What regularly tripped up earlier language models is handled better by newer ones. New generations of models can reduce previous weaknesses or make them disappear entirely.

Lists of “What AI can do” and “What AI cannot do” therefore age surprisingly quickly. The frontier shifts — but not necessarily along our idea of what should be easy or difficult.

GENERALISATION

A good answer proves less than we think

Back to the contract from the beginning. The AI analysed it extremely well. Now you ask about a piece of legislation mentioned in the document.

The reflex is obvious: if it understood this complicated contract, surely it can answer that question as well.

An excellent summary initially proves only that the model handled that particular task well. It does not yet prove that the numbers it cites are correct or that the legal sources it mentions actually exist. That is even more important when it comes to legal interpretation.

The more similar the next task is to one the system has already handled successfully, and the more often it performs well there, the more that experience tells us. The further we move away from it, the less we should simply carry that observed competence over.

The problem begins with generalisation. “The AI was excellent here” quickly becomes “The AI is excellent”.

For a dinner idea, that hardly matters. For medical information, a financial calculation or a legal decision, it does.

The question “How intelligent is this AI?” is not very helpful here.

More useful is: “How reliable is it at exactly what I want from it right now?”

PRACTICE

How to work with a frontier you cannot see

An impressive answer is not a general proof of competence. With new tasks, the only thing that helps is observing performance where you actually intend to use it. One successful attempt is not always enough. The more important the result and the greater the consequences of an error, the more confidence you need.

Even large tasks do not have to remain black boxes. You can break them into checkable parts, state important requirements explicitly and verify critical details such as numbers, sources or concrete facts. Precise instructions help. They are not a guarantee.

We treat human expertise in a similar way. We trust a friend’s restaurant recommendation differently from a medical diagnosis. We do not need the same level of certainty everywhere.

With AI, this distinction is harder because we are less able to judge where its competence actually ends.

If something goes wrong, the question was not necessarily bad. Unclear instructions and missing context matter. Often, a better question helps.

But not every AI error is a prompting error.

Sometimes we simply expected more than the model can reliably deliver on that particular task.

CONCLUSION

Was it ever a genius?

At the beginning, the AI looked like a genius. It analysed in seconds a contract that might have taken us an hour. Shortly afterwards, it appeared to fail at counting to 15.

The contradiction becomes smaller once we stop placing both performances on the same human scale.

Perhaps the word “genius” is already part of the problem. Just like “stupid”, “easy” or “difficult”, it comes from our experience with human abilities. These words can still be useful when talking about AI. They become dangerous when we infer all the other capabilities we would associate with them in a human being from one observed performance.

The better question is: “How good is it at what I want from it right now — and how do I know?”

Our confidence should depend on how reliably AI handles the task at hand, rather than how impressive it was at something else.

That is where the boundary that matters to us actually lies.

THINKING WITH AI. NOT LETTING AI THINK FOR YOU.