09 Lesson 9 of 15
Writing like someone who knows
The question that matters is not whether it has feelings. It is whether it knows things. Those are different questions, and only one of them changes your decisions.
It understands what I am asking. It picks up on nuance. It gets what you meant, not just what you said. It adjusts when you are frustrated. Whatever is going on in there, “understanding” seems the obvious word for it.
A real experiment is worth knowing about here, and its result surprises most people. Researchers asked several hundred people how far a language model had various mental abilities. Then they gave them a task where they could take or ignore the model's advice. Seeing it as having feelings and consciousness did not make people more likely to follow its advice; if anything, slightly less. Seeing it as having intelligence (reasoning, planning) was strongly linked to following it.
So the big philosophical question is not the one doing the damage. Whether these systems have any inner life is truly unresolved and truly interesting. But it is not what leads people to accept a wrong answer. What does that is the quiet assumption that something in there knows.
Two things worth separating
Fair to say
It handles nuance well. It responds to tone. It writes text that fits the situation. It is often truly useful. It can do things that, ten years ago, would have looked like understanding from the outside.
Not shown by any of that
That it has checked anything. That it has a view it is reporting. That it would notice if it were wrong. That it can tell you how sure it is. That it means what it seems to mean.
You may notice this lesson has no 3D diagram. That is on purpose. Drawing the inside of a mind is exactly what plants the mistaken idea. No arrangement of glowing boxes could make the point without also making it worse.
It is fine to be polite to it
This lesson is not saying you are foolish to type “please”, or to feel a flicker of guilt when you close the tab. People naturally treat anything that talks as social, and that is not a flaw. These systems now copy human conversation so well that just telling yourself to stop does not work well, and researchers in this area have said so.
Keep the manners. Drop the assumption. The sentence to resist is not “thank you”. It is “it wouldn't say that if it weren't true”.
Fair to say, or not established?
Sort each statement into one of the two columns from above. There is no score. When you are done, the lesson's own placement appears with a line on why.
Notice one moment today when the chat pulls a social response out of you: you thank it, you apologise, you feel judged, you feel it is being lazy. You do not have to stop. Just catch it happening.