Why chatbots make things up
Here's a puzzle. Ask a chatbot about a book that doesn't exist, and it might give you the plot, the author, and the year it was published. All made up, all delivered with total confidence. Why does it do that? It isn't broken, and it isn't lying. The real reason is genuinely beautiful, and once you see it, you will never be surprised by it again. This is Foundations, lesson one.
The one job
Start with what this machine really is. It has read an enormous library. More text than any person could read in a thousand lifetimes. And from all that reading, it learned exactly one job. Finish the sentence. Given some words, predict the words that usually come next. That is the whole machine. Everything impressive you have ever watched it do grows out of that one skill.
When the usual answer is true
Ask it the capital of France, and the sentence practically finishes itself. In the library, the words the capital of France is, are followed by Paris almost every single time. So the usual way to finish the sentence is also the true way. For famous facts, finishing smoothly and telling the truth are the same thing. That is why it feels so knowledgeable.
The invented answer
Now ask about Atlantis. The library has no real answer, because there is no real answer. But the machine's job has not changed. It still finishes the sentence, smoothly, the way capital questions usually get finished. Out comes a name that sounds exactly right. And here is the unsettling part. From the inside, those two answers look identical. Same shape, same confidence. One is true, one is invented, and nothing inside the machine checks which is which.
It was never taught to stop
So why doesn't it just say, I don't know? Think about what it studied. Almost nobody writes that. In the library, sentences get finished and questions get answered. Stopping to admit uncertainty was never part of the job it learned. It is not refusing to be honest. It is doing exactly what every page it ever read taught it to do. Keep going.
A bigger library, the same job
You might think the fix is obvious. Read more. But a bigger library does not change the job. There will always be questions the library cannot answer. Rare facts, brand new facts, things nobody ever wrote down. And for every one of them, the machine will still do the only thing it knows. Finish the sentence, smoothly. More reading makes it right more often. It does not teach it to stop.
Change the job, not the diet
What actually helps is changing the job. Let it look things up before it answers, so the sentence gets finished from a real page instead of from memory. And train it so that I don't know is a winning answer, so stopping finally pays off. Modern systems do both, and it is why they make things up less than they used to. Less, not never. The old job is still in there.
The 60-second recap
Sixty seconds, the whole idea. This machine has one job. Finishing sentences the way its library usually finishes them. When a fact is famous, the usual finish is the true finish, and it looks brilliant. When the answer is rare or missing, it finishes anyway, smoothly, and you get a made up answer that feels exactly like a real one. It was never taught to stop, and more reading alone will not teach it. What helps is changing the job. Let it check real sources, and reward it for admitting what it does not know. That is lesson one. Next time, what a token actually is, and why words is the wrong answer.
