What a hallucination is

Hallucination is the name for a language model producing text that is fluent, confident, and not true - a date that is wrong, a quotation nobody said, a paper that was never written, a function that does not exist in the library. The word is borrowed from psychology and it is slightly misleading: the model is not perceiving something that is not there. It is generating the most plausible continuation of the text in front of it, and plausible and true are not the same property.

That last sentence is the whole article. Every correct answer a model gives is produced by exactly the same process as every hallucination. There is no separate "make things up" mode that switches on, and no internal fact-checker that switched off. Which is why a hallucination cannot be spotted by tone: a fabricated answer is written with the same confidence, the same grammar, and the same level of detail as a real one, because the same machinery wrote both.

Researchers split hallucinations into two kinds, and the split matters for how you catch them. A factuality hallucination contradicts the world: the answer is wrong about something that can be checked against a source outside the conversation. A faithfulness hallucination contradicts the input: you gave the model a document, an email thread, or a set of search results, and its summary says something they do not say. The first is a gap in what the model knows; the second is a failure to read what it was given, and it happens even when the truth is right there in the prompt.

Where answers come from

A large language model does one thing: given the text so far, it assigns a probability to every possible next token (a word or a piece of one), one is picked, and the process repeats. Everything the model "knows" is encoded in how those probabilities fall, and they were shaped by a single training signal: predict the next token of human-written text well. Nothing in that signal ever asked whether a statement was true. Text that appears often and consistently in the training data becomes a strong, stable pattern; text that appears once, or never, does not.

That is why facts have a long tail. "Who wrote Pride and Prejudice" has been answered the same way in millions of documents, so the correct continuation is overwhelmingly the most likely one. The title of one researcher's doctoral thesis appeared a handful of times, if at all, so the model has no strong pattern for the content - but it has a very strong pattern for the shape. Thesis titles look a certain way; citations have authors, a year, a venue. Asked for one, the model produces something with the right shape, filled with tokens that are plausible in context. Researchers at OpenAI put a floor under this in 2025: after pretraining, a model should be expected to hallucinate on at least the share of facts that appeared exactly once in its training data. If a fifth of the people in the corpus had their birthday mentioned a single time, expect the base model to get at least a fifth of the birthdays it is asked for wrong.

The second half of the mechanism is that there is no built-in "I do not know". Declining to answer is a behaviour the model has to learn like any other, and the training text rarely models it: the internet is mostly written by people who knew the answer they were writing down, so a question followed by "I have no idea" is a rare pattern. Later training stages do teach models to abstain, and current models do it far more than early ones - but the same paper shows why it stays partial: the way models are scored rewards a confident guess over a blank, exactly as a multiple-choice exam does, so a model that always answers outscores one that admits uncertainty.

Two consequences follow that are worth holding onto. First, fluency is not evidence. The model is never more hesitant when it is on thin ice, because hesitation was not part of what it optimised for. Second, turning the temperature down does not fix it. Temperature is the setting that controls how much randomness goes into picking each token; at zero the model always takes the single most likely one. But when the most likely token is a guess, the most likely guess is still a guess - it is now just the same guess every time.

What it looks like in practice

The mechanism is one, but it surfaces in different places, and each place needs a different check:

  • Invented specifics. Dates, figures, percentages, quotations, the name of the person who said a thing. The detail is the giveaway: a hallucinated fact tends to be more precise than the model's knowledge warrants, because a vague answer is a less likely continuation than a specific one.
  • Fabricated sources. Paper titles, authors, URLs, ISBNs, court cases. These are the most damaging kind, because a citation exists to make a claim checkable and a fake one borrows that authority. In 2023 a New York court sanctioned two lawyers who had filed a brief citing six cases that ChatGPT had made up, complete with quotes and internal citations. When one of them asked the model whether the cases were real, it assured him they were and could be found in the standard legal databases. Asking a model to confirm its own answer is not verification; it is the same process, run again.
  • Code that does not exist. A method that follows the library's naming convention but was never shipped; a configuration flag from a different version; an import of a package that is not on the registry. A 2025 study of code-generating models found that roughly one in twenty package names suggested by commercial models, and one in five from open-source models, did not exist - which turns an invented name into an attack surface: register a malicious package under it and wait for the next developer to be given the same suggestion.
  • Unfaithful summaries. Given a document, the model reports a conclusion the document does not reach, attributes a position to the wrong party, or resolves an ambiguity the source left open. Nothing was missing from its knowledge; it drifted from the text into what documents like it usually say. Long inputs make this worse, and so does a question that presupposes something the document does not contain.
  • Actions that did not happen. In an agent, the model narrates its own work, and it can narrate work it never did: "I ran the tests and they pass" with no test run in the log, a file "updated" that is unchanged. This is the hallucination that costs most, because the reader is a program that believes the report. Why agents fail covers it in depth.

When it is most likely

Hallucination is not evenly spread. It clusters where the training patterns are weakest, which makes it partly predictable:

  • The long tail. Obscure people, small companies, niche libraries, local events. The less was written about something, the more of the answer is shape rather than content.
  • Anything after the cutoff. A model's knowledge stops at its training date. Asked about something newer, it does not know that it does not know; it continues from the closest patterns it has, which describe the past.
  • Precision. Exact numbers, exact quotes, exact version numbers. The model can know roughly what a thing is and still invent its fourth decimal place.
  • Questions with a false premise. "Why did the library remove the retry option in version 3?" invites an explanation whether or not version 3 removed anything. A question shaped like it has an answer usually gets one.
  • Long conversations. As earlier turns are summarised or dropped to fit the context window, the model continues from an increasingly lossy record of what was actually said, and fills the gaps the way it fills all gaps.

Read the list as a checklist. If a question lands on two or more of these, treat the answer as a draft to be checked, not a fact.

Designing around it

Hallucination cannot be switched off, so a product that depends on a model has to be built so a hallucination is caught before it matters. The techniques that hold up all follow from the mechanism:

  • Put the facts in front of the model. Everything in section 02 is about answering from memory. Give the model the document, the search results, or the database row and the task changes from recall to reading, which it does far more reliably. That is the idea behind retrieval-augmented generation and behind giving a model tools. It does not remove faithfulness errors, but it removes the largest source of factual ones.
  • Make "not in the sources" an acceptable answer. An instruction like "answer only from the provided context; if the answer is not there, say so" turns a silent retrieval miss into a visible one. Without it the model falls back to memory, and the failure looks like success.
  • Ask for evidence you can check mechanically. Have the model quote the passage each claim rests on, then verify in code that the quote appears in the source. Have it return structured output and validate it against a schema or a lookup - a product ID that is not in the catalogue is a hallucination your code can catch without reading a word.
  • Let the tool be the judge for code. A compiler, a type checker, a test run, or a package registry does not care how confident the model sounded. Anything the model writes that can be executed should be executed before it is trusted, and for an agent the log of what actually ran is the record, not the model's account of it.
  • Measure it. Keep a set of questions with known answers, including some the sources do not answer, and track how often the model invents one. A hallucination rate you measure can be driven down; one you assume is zero will find you in production. How to evaluate an agent covers the mechanics.
  • Match the check to the stakes. A brainstorm can tolerate an invented example; a contract summary, a dosage, or a legal citation cannot. Route high-stakes output through a person, and make the model's sources visible so that person can check quickly instead of redoing the work.

What to keep: a hallucination is not a glitch to be patched but a property of how the model produces every answer. The model cannot tell you which of its sentences are true. Your sources, your validators, and your tests can - so the design job is to make sure one of them always stands between the model and the reader.

References