Understanding Is a Prediction

A one-minute video stopped me this morning. Nate Jones, who writes about AI, asked: what do AI math and anesthesia have in common?

His starting point: OpenAI’s systems recently produced solutions to hundreds of open math problems, and the mathematician Terence Tao gave a lecture reflecting on what that means. The results work, but they are a black box — the answers check out, and nobody yet understands the reasoning inside them. Medicine, Nate points out, already lives with exactly this. We don’t fully understand how anesthesia switches off consciousness. We use it every day anyway, because we test it — dose, monitor, observe, repeat, for well over a century.

Why doesn’t the black box bother us more? I think neuroscience quietly answered this a while ago.

The brain is a prediction machine

Neuroscience calls it the Bayesian brain. The brain does not record the world like a camera. When you see a situation, your brain predicts it — experience supplies the prior, what usually happens; the senses supply the fresh evidence; and what you actually perceive is the two combined: the model’s best guess. The only thing that travels upward is what doesn’t fit — the prediction error — and its job is to revise the guess for next time.

The lineage is old. Helmholtz saw the outline in the 1860s and called perception "unconscious inference." Rao and Ballard modeled it in the visual cortex in 1999. Karl Friston generalized it into a theory of the whole brain. Anil Seth calls what we experience a "controlled hallucination" — controlled, because the senses keep correcting it.

The world you experience is the prediction, not the input.

So what is "understanding"?

This is my reflection. We observe a phenomenon, we give it a reason, and we call it understanding. But on the Bayesian picture, a reason is just the model’s current story — the prediction that has survived so far. When the errors pile up, the story gets revised: a few years later a new theory arrives that explains things better, and the previous theory is overturned, or modified. That is not science failing. That is the same update loop, running at the scale of a civilization.

Even anesthesia has lived it: the leading explanation of how it works has itself been replaced over the past century — while the practice, tested daily, never stopped working. The explanation is a model. The working is a fact. We keep confusing the first for the second.

“We don’t need to know how an engine works to drive a car.”

Engineering: prediction you can build on

Engineering is the special case. There, experience has been compressed into formulas. Given the input, we have confidence in the output — the load, the span, the stress. The prediction error has been squeezed down over centuries of testing, until the model is trustworthy enough to bet lives on. That is why we can build skyscrapers from blueprints: a blueprint is a prediction so reliable that the building is almost a formality.

That is also why, in my own work, GitHub Spec Kit SDD is now our blueprint. Building software with AI agents is still mostly black box — powerful, unpredictable. The spec-driven flow — specify, clarify, plan, tasks — writes the prediction down before the build, where it can be reviewed, questioned, and corrected. It is the same move engineering made: turn private guesses into a shared blueprint, then build from the blueprint.

And the limit follows from the machinery. A prior is built from one life’s experience, so every mind’s model is partial by construction. I have to admit that understanding and reasoning have a limit. No individual can understand everything — nobody ever could. The black boxes of the AI age are not a new embarrassment. They are the normal condition of a predictive mind in a world larger than its experience.

So let’s take it as being. Accept the way it is: use what works, test what you rely on, let the errors teach you — and choose carefully the few things you will understand deeply.