How AI Actually Works

The biggest mistake teachers make with AI

Most people treat AI like a very fast librarian: you ask a question, it looks up the answer, and it tells you. That mental model is wrong, and it causes real problems in classrooms.

A standalone language model does not consult a verified facts database for each answer. It generates text by predicting tokens using patterns learned during training and information supplied in the conversation. Some AI assistants can also search the web, retrieve documents, run calculations, and check claims with external tools.

Tom Hammond, Professor of Education at Lehigh University, puts it plainly: "AI doesn't actually know anything. Its outputs are the result of calculated probabilities." That distinction changes how you evaluate every response AI produces.

Tokens: language broken into pieces

AI does not read words the way you do. Before it processes anything you type, it breaks your text into small chunks called tokens. A token can be a full word, part of a word, or punctuation.

A word such as "understanding" may be one token or several, depending on the tokenizer. As a rough estimate for ordinary English text, one token is about three quarters of a word. A 500-word page might contain roughly 670 tokens, but the exact count varies.

This matters for two practical reasons. First, many AI API services charge according to token use, although some classroom tools offer free or subscription access. Second, models have context-window limits. In sufficiently long conversations, earlier material may be dropped or summarised, so the original details may no longer be available to the model.

Sophisticated autocomplete

At its core, a large language model is a next-token prediction machine. When you type a prompt, the model converts it to tokens, calculates probability scores for every possible next token, selects one, adds it to the sequence, then repeats, one token at a time, until it decides to stop.

Gary Marcus, Professor Emeritus at NYU, has compared language models to very large autocomplete systems. That analogy captures the token-prediction mechanism, but it is not the whole story: modern models can perform multi-step reasoning and use tools. Neither fluent reasoning nor a confident tone, however, guarantees that a claim is correct.

This is why AI can write a grammatically perfect, confident-sounding paragraph about a completely fictitious study. The words flow naturally. The structure is right. The citations look real. It is all generated token by token, shaped by patterns, not by knowledge of what is actually true.

Knowing this changes how you read AI output. The fluency is not evidence of accuracy. A confident response and a correct response are two different things.

Hallucination: the student who didn't study

Hallucination is when an AI generates information that sounds plausible but is false, fabricated, or misleading. It happens because the model bridges gaps in its training data by generating what statistically fits, not what is actually true.

OpenAI's research explains that uncommon, arbitrary facts are harder to predict reliably than familiar language patterns. There is no single meaningful hallucination rate for all leading models: reported error rates depend on the model, benchmark, question type, and whether it can search or check sources.

The classroom parallel: imagine a student who did not read the assignment but writes an essay that sounds authoritative. They string together plausible sentences based on general knowledge. Sometimes they invent citations that look real but do not exist. The essay has a confident tone, clear structure, and no obvious red flags. But it cannot be trusted without checking.

Hallucinations are an ongoing reliability problem, not simply a spelling mistake. Better training, appropriate uncertainty, retrieval, and verification tools can reduce them, but no tool should be assumed error-free. Independently verify important factual claims, especially medical, legal, scientific, or historical information.

Context windows: the AI's desk

A context window is the maximum amount of text an AI can process in a single interaction, covering both what you type and what it generates. Think of it as a desk. The AI can only work with what is on the desk. When the desk gets full, older material slides off the edge and disappears from view.

Context-window sizes vary widely between models and products. Long conversations, large uploads, and multi-step tasks can approach those limits; some products then truncate or summarise older content. A model may also overlook relevant information even when it remains inside the window, so check whether its answer actually reflects the material.

The practical rule: for any task where earlier details matter, restate the key information periodically. Do not assume the AI remembers what you told it twenty exchanges ago.

Training data: where the patterns come from

A large language model learns patterns from large datasets that can include books, websites, academic writing, code, and discussions. The GPT-3 paper described a training-data collection of approximately 499 billion tokens, with about 300 billion tokens sampled during training. Tokens are not the same as words.

Training material can contain stereotypes, omissions, errors, outdated information, and uneven representation. Bias may also arise from how data is selected, how models are trained and evaluated, and how tools are deployed. There is no single reliable percentage that captures bias across all educational AI systems; the risk depends on the model, task, and group affected.

A trained model generally has a knowledge cutoff, but the date differs by model and version. Without web search, retrieval, or newly supplied material, its answers about later events may be outdated or invented. Check the tool's current capabilities before relying on it for recent news, prices, laws, or research.

The implication for classroom use: familiar, well-documented topics may be easier for a model, but errors remain possible. Recent events, specialist topics, minority languages, and local knowledge may need additional scrutiny and independent sources.

What AI genuinely cannot do

Understanding the architecture helps explain its limitations. Modern AI can tackle unfamiliar problems, sometimes recognise uncertainty, and verify selected outputs using tools, but it can also reason incorrectly or overlook errors. It does not have a teacher's firsthand awareness of classroom relationships, student wellbeing, or what is happening in the room.

AI can support instruction, feedback, and practice, but teaching also involves judgment, care, relationships, and responsibility. Section 2 explores evidence on when AI assistance helps and when it may weaken learning. Knowing the strengths and limits of AI is part of AI literacy.