13 Lesson 13 of 15
Verification that costs almost nothing
Elaborate checking does not survive a real, busy week. These four checks are cheap enough that you will actually do them.
I should check everything it tells me. After the last few lessons, the obvious conclusion is that you must check every answer. Honestly, you will do that for two days and then quietly stop.
It is to make checking so cheap that skipping it saves you nothing.
See it in the lessonThat is not cynicism. It is a research finding. Researchers built tools that made people slow down and think about an AI's suggestion. The tools worked (they really did cut over-reliance), but people rated them the worst of everything tested. Worse, they helped most the people who already thought carefully, and least the people who did not. If a step annoys you, you will find a way around it.
So the aim here is not diligence. It is to make checking so cheap that skipping it saves you nothing.
Four habits
1. Ask for the quote
When it summarises a document, ask it to quote the sentence it is relying on. It costs one line. It catches made-up content, and the case where it never had the document at all.
2. Ask twice, in a new chat
Same question, fresh chat. If the answers agree, that is a small comfort. If they disagree, that is a loud, cheap warning. It costs ten seconds. If your tool has memory on, use a temporary or private chat, so the second answer is not nudged by the first.
3. Check the one key fact
Not everything. Just the one thing that, if wrong, ruins the rest. Usually a number, a name, a date or a source. One search. If you cannot tell which fact is the key one, that is worth noticing too.
4. Bring the facts yourself
The cheapest check is the one you do not need. Paste in the document, the figures, the policy, and ask it to work from that. You have moved the task from a trough to a peak on the map in the last lesson.
Ask for the quote. Ask twice. Check the key fact. Better still, bring the facts with you.
Match the checking to the stakes
This does not mean everything needs checking. Brainstorming twenty names for a cat needs no checking at all, because you will judge the results yourself and nothing depends on it. The habits are for when a mistake would carry on into something you send, publish, decide, or spend money on.
A quick question to sort it out: if this were wrong, when would I find out? If the answer is “straight away, and it would cost me nothing”, do not check. If the answer is “later, from someone else, embarrassingly”, check now.
Cheap checks pay off because a mistake costs more the further it travels. The diagram shows that journey: an answer becomes your draft, the draft gets sent, and someone makes a decision based on it. Switch habits on, send a mistake down the line, and see where it gets caught.
Read this diagram as text
Four stages: the answer, your draft, sent to someone, and a decision made on it. Fixing a mistake costs about 1, 3, 10 and 30 units at those stages. Three kinds of mistake travel down the line: an invented source, a wrong number, and an answer that would have come out differently if asked again. Ask for the quote catches an invented source at the first stage. Ask twice catches an unstable answer at the first stage. Check the key fact catches a wrong number or an invented source at the draft stage. Bring the facts yourself stops invented sources and wrong numbers from appearing at all. With no habits on, every mistake reaches the decision.
If this were wrong, when would I find out?
Pick the answer that fits what you are about to use a chatbot for. You get the check that is worth its cost, and no more.
Choose one of the four. Just one. Use it on your next real question, and keep using it until it is automatic before you add another.