04 Lesson 4 of 15
Why it invents things that sound right
Made-up facts are not a bug added to an otherwise reliable system. They follow straight from the two lessons you have just read.
It made a mistake. Something went wrong. The system glitched, or the question was too hard, or it did not know and should have said so. Next time it will probably be fine.
The true answer to your last question and the invented answer to this one came from the same process, running normally.
See it in the lessonPut the last two lessons together. The model produces likely text, and it picks among the options. Now ask it for something where likely-looking text is easy to make but the truth is narrow: a citation, a statistic, a quotation, a court case, a street address, the plot of a book it half knows.
A huge number of word strings look exactly like a correct citation. Almost all of them are wrong. The model aims for the shape of a citation, and shape is what it is good at. The one that happens to be true is one candidate among thousands that all look equally right from the inside.
That is why the usual name for this, hallucination, is a bit misleading. It suggests the system normally sees clearly and sometimes does not. But nothing different is happening. The true answer to your last question and the invented answer to this one came from the same process, running normally.
Where it shows up most
High risk of invention
Specific numbers and dates. Named sources, citations and page references. Quotations. Details about small or local things: a particular clinic, a village, a minor company. Anything recent. Anything about you.
Lower risk
Rewriting text you supplied. Summarising a document you pasted in. Explaining a common concept. Suggesting options you will judge yourself. Fixing grammar. Producing a first draft you intend to edit anyway.
Look at the right-hand column. In every case, the material came from you, or you can judge the answer without outside help. The most reliable way to use these tools is to bring the facts yourself and let the tool do the writing.
When university students were asked about this, they described just what you would expect: invented citations, false statements said with confidence, and a habit of sticking to a wrong answer when challenged. These are not rare events. This is what the process does when “plausible” and “true” stop lining up.
Here is that idea as a picture. Each pillar below is a citation the model could produce when asked for a source on coastal erosion. Height is how plausible it looks. Only one of them is a real paper. Ask for a source a few times, then reveal which one was real.
Read this diagram as text
Each press of Ask for a source picks one of 400 candidate citations, weighted by how plausible it looks. Nearly all of them are between 80% and 100% plausible. One stands for the paper that really exists, and it sits in the middle of the pack, not at the top. So the real one comes up about once in four hundred asks. The list under the buttons shows every citation picked, each one formatted just as neatly as the real one.
Which parts would you check?
Below is the kind of answer a chatbot gives to an ordinary question. Tap any part you would want to check before relying on it, then compare your picks with the lesson's.
Ask for three sources on a topic you know a little about, then try to find them. Do this once, on purpose, on a question that does not matter. It is much better to meet an invented citation when you are looking for one.