02 Lesson 2 of 15
Where the words come from
This is the one to get right. Most of the other mistakes on this site grow out of getting this one wrong.
It looks the answer up. Somewhere there is a huge store of text. You ask a question, it finds the matching part, and it reads it back to you in a friendly voice. This is the most common way to picture it. It is wrong in a way that matters.
An answer is built forwards, a piece at a time, with no plan for how it ends.
See it in the lessonHere is what really happens. The model looks at the text so far and works out which pieces could come next, and how likely each one is. One gets picked and added to the text. Then the whole thing runs again for the next piece. An answer is built forwards, a piece at a time, with no plan for how it ends.
The diagram below shows that step. Ten words are already fixed. The bars are candidates for the eleventh word, and each bar's height is how likely that word is. Pick one, and a fresh set of candidates appears for the next word.
Read this diagram as text
The sentence so far is “The best thing about living by the sea is the”. The candidates for the next word, with their weights:
| Candidate | Weight |
|---|---|
| sound | 25% |
| smell | 20% |
| light | 13% |
| air | 11% |
| view | 9% |
| quiet | 7% |
| weather | 5% |
| people | 4% |
| water | 3% |
| rhythm | 2% |
Choosing one replaces the table with candidates for the word after it. Nothing is fetched from a store of sentences. Each list is just weights for what could come next.
Why the difference matters
If it looked things up, a wrong answer would mean the store held the wrong thing. Mistakes would be rare and random. But it predicts likely text, so a wrong answer is not a fault at all. It is the same process working normally, landing somewhere plausible that happens to be untrue. So be suspicious not of hard questions, but of questions where a wrong answer would still sound right.
It also explains something that can look like lying. Ask for references and you may get ones that do not exist, perfectly formatted, with believable authors and a believable journal. Nobody faked them on purpose. A reference is a very predictable shape of text, and a real one and a fake one come out of exactly the same process.
One analogy. It is closer to a well-read person finishing your sentence than to a librarian fetching your book. The sentence-finisher has read enough to know what usually comes next.
Where the analogy breaks: a person finishing your sentence knows they are guessing, and can say so. This has no separate sense of whether the guess is right. That is why the next three lessons exist.
Ask a chatbot for a fact you can check in ten seconds: a date, a population, the name of a street near you. Then check it. You are not testing how clever it is. You are finding out how much checking you need to do.
Worth flagging: Two simplifications above. Real models work in tokens (pieces of words), not whole words. And there are far more than ten candidates at each step.