# What beginners get wrong about AI, and why it costs them confidence

Ten misconceptions organise most of what goes wrong when a newcomer meets a
chatbot. They are not ten independent errors. Each one collapses a distinction
that the system's behaviour actually depends on, and the same five distinctions
recur underneath all ten, which is what makes a short curriculum possible at
all. [Lin 2026](https://doi.org/10.1093/pnasnexus/pgag236) sets out four of
those distinctions for a scientific readership (pretraining process vs. deployed
system; the learned distribution vs. any particular sample; parametric vs.
contextual vs. external memory; task competence vs. agency) and uses them to
diagnose six folk theories. The taxonomy below adapts that frame to general
adults and adds a fifth distinction the expert-facing version does not need:
interface cues vs. mechanism. This matters because for a beginner the chat window itself is
the main source of evidence about what the thing is.

The organising finding for the whole project comes from a survey of 319
knowledge workers. Higher confidence *in the AI* predicts less critical
thinking, while higher confidence *in oneself* predicts more
([Lee et al. 2025](https://doi.org/10.1145/3706598.3713778)). Those are separable
targets. A site that raises both is doing half its job badly; the goal is to
raise self-efficacy while calibrating trust downward from its default.

## The five distinctions

**Generation vs. retrieval.** The model samples a continuation; it does not look
up a record. Laypeople's folk theories cluster heavily around exactly this
question of where the words come from, and focus-group participants build their
accounts by decoding the phrase "machine learning", probing the bot, and
reaching for analogies to familiar objects
([Li et al. 2026](https://doi.org/10.1177/20539517261447838)).

**The distribution vs. one sample.** Outputs are draws, not readings. Treating a
single response as the model's fixed position is the error behind
premature abandonment and premature trust alike.

**Where memory lives.** Weights, context window, and external tools are three
different stores with different persistence and different privacy consequences.
Collapsing them produces both "it remembers me" and "it can reach anything".

**Competence vs. agency.** Performing a task well is not wanting, knowing,
intending, or being reliable across neighbouring tasks.

**Interface cues vs. mechanism.** Fluency, first-person pronouns, apparent
hesitation, and politeness are design choices, not evidence about internals.
Metaphors the public uses for AI have shifted measurably toward warmth and
human-likeness over time
([Cheng et al. 2026](https://doi.org/10.1038/s44271-025-00376-6)).

## The ten misconceptions

**M1: "It looks things up."** The retrieval illusion: the system is imagined as
a search engine with better manners. Kernel of truth: training data really is
the source of what it can produce, and retrieval-augmented deployments really do
fetch documents. What it conflates: generation with lookup, and the training
corpus with an indexed store. Cost to the user: no way to anticipate *when* the
answer will be wrong, and no reason to doubt a citation that looks like a
citation. Corrective intuition: it writes the most plausible next thing, then the
next, and a reference is just more plausible text.

**M2: "It sounds sure, so it's right."** The highest-traffic error, and the one
with the most evidence behind it. Overreliance, meaning acceptance of an AI
suggestion that is wrong, is resilient, and adding explanations does not reliably fix it
([Buçinca et al. 2021](https://doi.org/10.1145/3449287)). Fluent prose hides
misinformation, and in logged interactions the users who over-relied were
identifiable behaviourally: frequent copy-paste, skipping task comprehension,
coarse navigation, and accepting a claim despite visible hesitation
([Zhang et al. 2026](https://doi.org/10.1145/3772318.3790332)). Students describe
the same texture from the inside: fabricated citations, overconfident but
misleading answers, persistence in a wrong answer, sycophancy
([arXiv:2602.17671](https://arxiv.org/abs/2602.17671)). The philosophical framing
is that such a system is indifferent to truth rather than lying about it
([Hicks et al. 2024](https://doi.org/10.1007/s10676-024-09775-5)). Corrective
intuition: confidence is a writing style, and it is generated by the same process
as the content.

**M3: "It understands me."** Anthropomorphic over-attribution, and the place
where the evidence is most useful because it is counter-intuitive. Attributing
*experience* (feelings, consciousness) does **not** increase advice-taking; in a
preregistered experiment (N = 410) that dimension related *negatively* to it,
while attributions of *intelligence* (reasoning, planning) correlated strongly
with accepting the model's advice
([Colombatto et al. 2025](https://doi.org/10.1038/s44271-025-00262-1)). The
design consequence is direct: the "is it conscious?" question is not the risk
channel and does not need to be won. "Does it know?" is. Meanwhile the systems
now genuinely do mimic human communication well enough that telling users to
stop anthropomorphising is a weak intervention on its own
([Peter et al. 2025](https://doi.org/10.1073/pnas.2415898122)). Corrective
intuition: separate "writes like someone who knows" from "knows".

**M4: "Same question, same answer."** Beginners read one output as the system's
position. A bad first sample then reads as a broken tool, and a good one licenses
overgeneralisation. Corrective intuition: ask twice and watch what changes. What
varies was never fixed.

**M5: "Tell it what you want, like you'd tell a person."** Non-experts given a
prompt-design tool explored opportunistically rather than systematically, and the
specific barriers were expectations carried over from instructing humans plus a
tendency to overgeneralise from single results
([Zamfirescu-Pereira et al. 2023](https://doi.org/10.1145/3544548.3581388)).
Corrective intuition: a prompt is a specification you revise, not an instruction
that lands.

**M6: "It remembers me, and my chats train it."** Conflates the three memory
stores. Practically consequential because it drives both misplaced privacy fear
and misplaced expectation of continuity. Evidence here is conceptual rather than
from user studies, a gap worth naming.

**M7: "It can reach anything."** Boundary confusion about the web, files,
dates, and tools. Also conceptually grounded rather than user-tested. Corrective
intuition: ask what it can see right now.

**M8: "A refusal means censorship."** A hedge or decline is read as a bug, a
political choice, or a moral judgement about the user. It conflates behavioural
shaping with installed values, and mis-locates agency. Analysis of unconstrained
human-LLM conversations found toxic content was largely demanded or provoked by
users, and raised open questions about which requests get refused
([arXiv:2407.05977](https://arxiv.org/abs/2407.05977)).

**M9: "Good at that, so good at this."** Capability is jagged across
neighbouring tasks; beginners assume it is flat. Corrective intuition: competence
is per-task and has to be checked per-task.

**M10: "It's not for me / I'll break it."** Not a factual error but the binding
constraint on confidence. Surveyed adults aged 50+ (n = 103, mean 64) were
motivated to learn and considered it important, but reported not understanding
and not knowing how to start, and expressed a strong preference for hands-on
learning over explanation
([arXiv:2504.14649](https://arxiv.org/abs/2504.14649)).

## What the intervention literature says about fixing these

Three findings constrain the design more than anything else.

Friction works, and users dislike it. Cognitive forcing interventions
significantly reduced overreliance relative to simple explainable-AI approaches,
but attracted the least favourable subjective ratings of any condition tested,
and benefited participants higher in Need for Cognition more, an
intervention-generated inequality the authors audited for explicitly
([Buçinca et al. 2021](https://doi.org/10.1145/3449287)). Anything that makes
verification effortful will be skipped by the people who most need it.

Explanations are not inert, but they are strategic. Across five studies
(N = 731), whether an explanation reduced overreliance depended on the costs and
benefits of engaging with it (task difficulty, explanation difficulty) rather
than on explanation quality alone
([Vasconcelos et al. 2022](https://arxiv.org/abs/2212.06823)). The lever is the
cost of checking, not the volume of explanation.

Help widens gaps before it closes them. Twenty-one novice programmers observed
with generative tools split sharply: the students who accelerated used the tools
differently from those who struggled, and the struggling students' metacognitive
difficulties were amplified rather than relieved
([arXiv:2405.17739](https://arxiv.org/abs/2405.17739)). A site aimed at
beginners must teach the *use pattern*, not just the concept.

## Where the evidence is thin

Two of the ten (M6, M7) rest on conceptual analysis rather than studies of
users, and self-efficacy as a theme returned only six papers in a 319-paper
screened corpus against 63 on trust and reliance. The competency frameworks are
well developed. [Long & Magerko 2020](https://doi.org/10.1145/3313831.3376727)
remains the reference synthesis, and a review of AI myths and preconceptions
covers the schooling literature
([Bewersdorff et al. 2023](https://doi.org/10.1016/j.caeai.2023.100143)). But
frameworks specify what a learner should end up able to do, not which wrong
intuition to dismantle first. That ordering question is where this project has
to make its own calls, and the curriculum step should say so where it does.