The evidence

What beginners get wrong about AI, and why it costs them confidence

Ten misconceptions organise most of what goes wrong when a newcomer meets a chatbot. They are not ten independent errors. Each one collapses a distinction that the system's behaviour actually depends on, and the same five distinctions recur underneath all ten, which is what makes a short curriculum possible at all. Lin 2026 sets out four of those distinctions for a scientific readership (pretraining process vs. deployed system; the learned distribution vs. any particular sample; parametric vs. contextual vs. external memory; task competence vs. agency) and uses them to diagnose six folk theories. The taxonomy below adapts that frame to general adults and adds a fifth distinction the expert-facing version does not need: interface cues vs. mechanism. This matters because for a beginner the chat window itself is the main source of evidence about what the thing is.

The organising finding for the whole project comes from a survey of 319 knowledge workers. Higher confidence in the AI predicts less critical thinking, while higher confidence in oneself predicts more (Lee et al. 2025). Those are separable targets. A site that raises both is doing half its job badly; the goal is to raise self-efficacy while calibrating trust downward from its default.

The five distinctions

Generation vs. retrieval. The model samples a continuation; it does not look up a record. Laypeople's folk theories cluster heavily around exactly this question of where the words come from, and focus-group participants build their accounts by decoding the phrase "machine learning", probing the bot, and reaching for analogies to familiar objects (Li et al. 2026).

The distribution vs. one sample. Outputs are draws, not readings. Treating a single response as the model's fixed position is the error behind premature abandonment and premature trust alike.

Where memory lives. Weights, context window, and external tools are three different stores with different persistence and different privacy consequences. Collapsing them produces both "it remembers me" and "it can reach anything".

Competence vs. agency. Performing a task well is not wanting, knowing, intending, or being reliable across neighbouring tasks.

Interface cues vs. mechanism. Fluency, first-person pronouns, apparent hesitation, and politeness are design choices, not evidence about internals. Metaphors the public uses for AI have shifted measurably toward warmth and human-likeness over time (Cheng et al. 2026).

The ten misconceptions

M1: "It looks things up." The retrieval illusion: the system is imagined as a search engine with better manners. Kernel of truth: training data really is the source of what it can produce, and retrieval-augmented deployments really do fetch documents. What it conflates: generation with lookup, and the training corpus with an indexed store. Cost to the user: no way to anticipate when the answer will be wrong, and no reason to doubt a citation that looks like a citation. Corrective intuition: it writes the most plausible next thing, then the next, and a reference is just more plausible text.

M2: "It sounds sure, so it's right." The highest-traffic error, and the one with the most evidence behind it. Overreliance, meaning acceptance of an AI suggestion that is wrong, is resilient, and adding explanations does not reliably fix it (Buçinca et al. 2021). Fluent prose hides misinformation, and in logged interactions the users who over-relied were identifiable behaviourally: frequent copy-paste, skipping task comprehension, coarse navigation, and accepting a claim despite visible hesitation (Zhang et al. 2026). Students describe the same texture from the inside: fabricated citations, overconfident but misleading answers, persistence in a wrong answer, sycophancy (arXiv:2602.17671). The philosophical framing is that such a system is indifferent to truth rather than lying about it (Hicks et al. 2024). Corrective intuition: confidence is a writing style, and it is generated by the same process as the content.

M3: "It understands me." Anthropomorphic over-attribution, and the place where the evidence is most useful because it is counter-intuitive. Attributing experience (feelings, consciousness) does not increase advice-taking; in a preregistered experiment (N = 410) that dimension related negatively to it, while attributions of intelligence (reasoning, planning) correlated strongly with accepting the model's advice (Colombatto et al. 2025). The design consequence is direct: the "is it conscious?" question is not the risk channel and does not need to be won. "Does it know?" is. Meanwhile the systems now genuinely do mimic human communication well enough that telling users to stop anthropomorphising is a weak intervention on its own (Peter et al. 2025). Corrective intuition: separate "writes like someone who knows" from "knows".

M4: "Same question, same answer." Beginners read one output as the system's position. A bad first sample then reads as a broken tool, and a good one licenses overgeneralisation. Corrective intuition: ask twice and watch what changes. What varies was never fixed.

M5: "Tell it what you want, like you'd tell a person." Non-experts given a prompt-design tool explored opportunistically rather than systematically, and the specific barriers were expectations carried over from instructing humans plus a tendency to overgeneralise from single results (Zamfirescu-Pereira et al. 2023). Corrective intuition: a prompt is a specification you revise, not an instruction that lands.

M6: "It remembers me, and my chats train it." Conflates the three memory stores. Practically consequential because it drives both misplaced privacy fear and misplaced expectation of continuity. Evidence here is conceptual rather than from user studies, a gap worth naming.

M7: "It can reach anything." Boundary confusion about the web, files, dates, and tools. Also conceptually grounded rather than user-tested. Corrective intuition: ask what it can see right now.

M8: "A refusal means censorship." A hedge or decline is read as a bug, a political choice, or a moral judgement about the user. It conflates behavioural shaping with installed values, and mis-locates agency. Analysis of unconstrained human-LLM conversations found toxic content was largely demanded or provoked by users, and raised open questions about which requests get refused (arXiv:2407.05977).

M9: "Good at that, so good at this." Capability is jagged across neighbouring tasks; beginners assume it is flat. Corrective intuition: competence is per-task and has to be checked per-task.

M10: "It's not for me / I'll break it." Not a factual error but the binding constraint on confidence. Surveyed adults aged 50+ (n = 103, mean 64) were motivated to learn and considered it important, but reported not understanding and not knowing how to start, and expressed a strong preference for hands-on learning over explanation (arXiv:2504.14649).

What the intervention literature says about fixing these

Three findings constrain the design more than anything else.

Friction works, and users dislike it. Cognitive forcing interventions significantly reduced overreliance relative to simple explainable-AI approaches, but attracted the least favourable subjective ratings of any condition tested, and benefited participants higher in Need for Cognition more, an intervention-generated inequality the authors audited for explicitly (Buçinca et al. 2021). Anything that makes verification effortful will be skipped by the people who most need it.

Explanations are not inert, but they are strategic. Across five studies (N = 731), whether an explanation reduced overreliance depended on the costs and benefits of engaging with it (task difficulty, explanation difficulty) rather than on explanation quality alone (Vasconcelos et al. 2022). The lever is the cost of checking, not the volume of explanation.

Help widens gaps before it closes them. Twenty-one novice programmers observed with generative tools split sharply: the students who accelerated used the tools differently from those who struggled, and the struggling students' metacognitive difficulties were amplified rather than relieved (arXiv:2405.17739). A site aimed at beginners must teach the use pattern, not just the concept.

Where the evidence is thin

Two of the ten (M6, M7) rest on conceptual analysis rather than studies of users, and self-efficacy as a theme returned only six papers in a 319-paper screened corpus against 63 on trust and reliance. The competency frameworks are well developed. Long & Magerko 2020 remains the reference synthesis, and a review of AI myths and preconceptions covers the schooling literature (Bewersdorff et al. 2023). But frameworks specify what a learner should end up able to do, not which wrong intuition to dismantle first. That ordering question is where this project has to make its own calls, and the curriculum step should say so where it does.

Design brief: an AI fluency site for adults starting from zero

Every decision below answers to the evidence in misconception_taxonomy.md. Where a choice is mine rather than the literature's, it says so.

The thesis

Two dials, moved in opposite directions. Confidence in oneself goes up; confidence in the tool comes down to something calibrated. These are separable in the data (Lee et al. 2025), and conflating them is the characteristic failure of beginner AI content, which reads as reassurance and produces overtrust. The site's success condition is a visitor who will try things without asking permission and check things without being told.

Ten principles

P1. Teach one mechanism, not ten facts. The ten misconceptions reduce to five distinctions. Lessons are organised by distinction, with the misconceptions as symptoms, because a learner who holds the distinction defuses misconceptions the site never mentions.

P2. Name the wrong intuition out loud before correcting it. Folk theories are constructed from interface probing and analogy (Li et al. 2026), so they are already in the visitor's head. Conceptual change needs the prior belief surfaced, not bypassed. Every lesson opens with the plausible wrong version, stated sympathetically enough that a holder of it feels recognised rather than caught.

P3. Make verification cheap, never mandatory. The binding constraint. Cognitive forcing reduces overreliance but is disliked and disproportionately benefits users already high in Need for Cognition (Buçinca et al. 2021); whether an explanation helps depends on the cost of engaging with it, not its quality (Vasconcelos et al. 2022). So: no forced quizzes, no gates, no interstitials. Instead, one-click "ask it again" and "check this" affordances that cost less effort than skipping them.

P4. Hands-on over exposition. Older adults surveyed on AI literacy wanted to learn, could not find a starting point, and preferred hands-on learning (arXiv:2504.14649). Every lesson contains something the visitor manipulates within the first screen. No lesson is more than two screens of prose before an interaction.

P5. Spend the anthropomorphism budget on competence, not sentience. Intelligence attributions predict advice-taking; experience attributions do not, and trend the other way (Colombatto et al. 2025). The site therefore does not argue about whether AI feels anything. It attacks "it knows" and leaves "it feels" alone, which also avoids a fight that makes users defensive. Telling people to stop anthropomorphising is in any case weak when the system genuinely mimics human communication (Peter et al. 2025).

P6. Teach the use pattern, not just the concept. Generative tools widened the gap between strong and struggling novices rather than closing it (arXiv:2405.17739); the accelerating students used the tools differently. Each lesson ends with a concrete behavioural move, phrased as something to do, not something to know.

P7. Analogy discipline: one analogy per concept, and state its limit. Folk theories form by analogy to familiar objects, so an unbounded analogy becomes the next misconception. Any analogy the site introduces is followed immediately by where it breaks. Banned outright: "digital brain", "it reads the internet", "like a very fast librarian": each seeds M1 or M3.

P8. Interactivity must carry the concept, not decorate it. A 3D diagram earns its place only when the concept is spatial or distributional and the misconception is visual in origin. Where the concept is procedural or factual, prose and a static diagram are better and cheaper. This is my judgement call, applied explicitly per concept in the curriculum map; the literature does not settle it.

P9. Accessible by construction. Mixed-age general-adult audience means WebGL cannot be load-bearing. Every diagram has a prose equivalent that carries the same teaching point, respects prefers-reduced-motion, works by keyboard, and degrades to static when WebGL is unavailable. Reading level: plain language, target around grade 9, no unexplained jargon, every technical term glossed on first use.

P10. Honesty about uncertainty, including the site's own. Two of the ten misconceptions have no direct user-study support. Where the site teaches something contested or thinly evidenced, it says so in one line rather than flattening it into confident instruction. This models the epistemic behaviour the site is trying to teach.

What the site will not do

No "AI is the future" framing; motivation research on this audience shows they are already motivated and stuck on starting points, not on why. No capability hype and no doom. No leaderboards, badges, or streaks, which reward completion over calibration. No live model calls in v1: they make the site fragile, introduce cost and privacy questions, and the concepts are better taught with pre-recorded, inspectable examples where the visitor can see the same prompt produce different outputs without waiting on a network.

Tone

Plain, specific, and unhurried. The register of a competent colleague showing you something on their screen, not a course, not a brand, not a warning label. Second person. Short sentences. No exclamation marks. Humour is fine where it is dry and does not undercut a safety point.

Success conditions, testable

A visitor finishing the site should be able to: say where the words come from without using the word "search"; predict that the same prompt can give a different answer, and check it; name one thing the model cannot reach; describe one verification habit they would actually use; and report that trying things carries no risk of breaking anything. The last is the self-efficacy target and the first four are the calibration target.