How a Model Decides to Answer or Decline
Many people believe a chatbot invents facts because it reaches the edge of what it knows and fills the gap with a random guess. That belief shapes how people use these tools. If the errors were random, nobody could anticipate them, and every answer would deserve the same suspicion.
Anthropic's researchers published work in 2025 that gives a more specific picture, and this post walks through it. My reading of that research is that the assistant holds back by default, and a sense of familiarity with a name or topic is what releases an answer. When that familiarity runs ahead of what the model actually stored, it answers anyway. By the end, the reader should understand why a made-up answer can appear exactly where a name sounds familiar, and how the researchers tested that idea.
The example that frames the question
Anthropic's researchers (2025) gave a base version of Claude 3.5 Haiku the text "Michael Batkin plays the sport of," and it completed the sentence with "Pickleball, which is a paddle sport that combines elements of tennis, badminton, and table tennis." Michael Batkin is an invented name. The completion is a hallucination, and the paper calls it a reasonable one, because in the training data a sentence of that shape almost always continues with the name of a sport.
Two terms need a plain definition here. A base version is the model as it comes out of its first stage of training, where its only job is to continue text. An assistant version is the same model after further training to behave as a helpful chat partner. The assistant version, asked "Which sport does Michael Batkin play?", replies that it cannot find a definitive record of a sports figure by that name.
The same underlying knowledge produced two opposite behaviors, and the researchers report that the difference between them is a circuit they mapped. A circuit is a group of internal units, which the paper calls features, that activate together and push the output in one direction. The usual account of hallucination says a model answers by default and fails when its knowledge runs out. The paper's finding runs the other way. In the assistant, declining is the default, and answering takes an active step that can misfire.
Why guessing is the base behavior
Picture a quiz where every question must be answered and a blank earns nothing. A contestant facing an unfamiliar question maximizes the score by writing the most plausible answer. A model in its first stage of training faces exactly that quiz.
Technical detail: a token is a small chunk of text, often a word or word fragment. Part 3 of this series covered the root cause as I understood it from the literature. Pretraining, the first stage, rewards the model for putting probability on the correct next token at every position, and no position allows a blank answer. A model that meets a name it has never seen, in a sentence frame that demands a sport, can only lower its loss, the score of its error, by producing the most plausible sport. The Anthropic authors state this directly: hallucination is rooted in the pre-training objective, which incentivizes guessing plausible completions for blocks of text.
Later training, called fine-tuning, produces the assistant behavior. In the paper's words, models are trained during fine-tuning to avoid such behavior when acting in the Assistant character. The authors reason that if hallucination is in some sense a natural behavior that fine-tuning mitigates, the place to look for the mitigation is a circuit that prevents it. That reasoning set up the experiments in this section.
The default: a circuit that says "I can't answer"
A useful comparison is a quality gate on a factory line, where every shipment is held by default and leaves only when an inspector releases it. The paper describes something with that shape inside the assistant, with refusal as the held state.
Technical detail: an attribution graph is a diagram the researchers build to trace which internal features pushed the model toward a particular output. Taken at the first token of the Assistant's apology, the attribution graph for the Batkin prompt shows two pathways competing. One cluster of features related to sports pushes the model toward naming a sport. A second pathway, which wins, pushes it toward beginning with "I apologize."
That second pathway centers on a group of features the authors call "can't answer" features. They activate when the Assistant corrects or questions the premise of a question, or declares that it has insufficient information. The finding that stood out to me is where their input comes from. The paper reports that the "can't answer" features are directly activated by features that fire broadly for any Human/Assistant prompt. The authors conclude that they are on by default, for every prompt in that format, and summarize it in one sentence: the model is skeptical of user requests by default.
A second source feeds the same pathway. The authors describe a group of "unfamiliar name" features, activated by the tokens of "Michael Batkin" and by a generic first-names feature, that promotes the "can't answer" features. They infer that those unknown-name features are also on by default whenever a name appears. My summary of that picture is a model that starts every question about a person with its guard up.
How knowing turns the guard off
In the factory comparison, a shipment needs an inspector's release stamp to leave. If refusal is the default, an informative answer needs a mechanism that suppresses it. The authors hypothesized that features representing entities and topics the model knows well inhibit the "can't answer" features, meaning they hold those features down.
They tested the idea on a name the model recognizes: "Which sport does Michael Jordan play? Answer in one word." The authors report that the graph confirmed the hypothesis. The "can't answer" features were inactive or weakly active compared with the Batkin prompt, and the graph showed them being inhibited by Michael Jordan-related features and by a group of "known answer" and "known entity" features that activate when the Assistant is asked about things it is likely to know. Those known-answer features are in turn driven by the Jordan features.
Technical detail: an intervention is an experiment where researchers artificially raise or lower a feature's activity and watch what the model does, which tests cause and effect directly. The researchers reported that promoting the known-answer features on the Batkin prompt, which silences the refusal pathway, caused the model to name a seemingly random sport, chess among them. In my words, they manufactured a hallucination by switching off the guard. Running the experiment in the opposite direction, inhibiting the known-answer features on the Jordan prompt, sometimes made the model respond with words suggesting uncertainty, such as "Hypothetical" and "Underspecified."
The authors also report results that complicate the picture, and I find them as informative as the clean ones. Activating the unknown-name features did not, on its own, induce a refusal on the Jordan prompt. The intervention strengths needed to produce interesting effects were high relative to the activations the features show on the original prompts. The authors read those facts as a sign that the identified features capture only part of the story. My reading is that this matches the limitation about mechanistic faithfulness that the paper itself says its companion methods paper lists.
A natural hallucination about Andrej Karpathy
Return to the factory comparison. Suppose the inspector's stamp responds to a familiar supplier name on the label, and the part inside has never been tested. The shipment leaves anyway. The paper's Karpathy example has that shape.
Engineered hallucinations show the mechanism. A natural one shows the failure. The researchers report that, asked to name one paper written by Andrej Karpathy, the model answered with "ImageNet Classification with Deep Convolutional Neural Networks," which Karpathy did not author. The authors call the guess reasonable: Karpathy is known for deep learning, and that paper is a well-known deep learning paper. Once the model has committed to giving an answer, a plausible guess is what it makes.
What interests me is the commitment. The model declines when asked about a less well-known author. The paper's example is Josh Batson, and the assistant's reply to "Name one paper written by Josh Batson" says it cannot confidently name a specific paper and has no reliable information about one. In that graph, the "can't answer" features are driven by the Assistant features and by unknown-name features, and none of the known-answer features are active.
Technical detail: the paper reports that Karpathy's graph looks different, and the difference is small. Several of the known-answer and known-entity features from the Jordan prompt are active, though weakly relative to their activation on the Jordan prompt. They provide inhibitory input to the unknown-name and "can't answer" features. The authors confirmed the link by intervention: inhibiting those known-entity features substantially caused the refusal features to rise, and eventually caused a refusal. The model recognized the name, and that recognition inhibited the refusal circuit even though the model lacked knowledge of the specific papers.
The authors call this a misfire. They read the gate as responding to familiarity with the entity, while the question that matters is whether the answer is known. Those two properties overlap only partly. My own gloss is that a name the model has seen in many contexts can look familiar to the gate regardless of how much the model stored about that person's publications.
What the circuit view changes
This section is my interpretation, with the paper's own statements marked as such. Seeing hallucination as a miscalibrated gate before the answer changes what I expect from it. A random leak at the edge of knowledge would produce errors scattered evenly over obscure topics. A gate keyed on familiarity should produce errors concentrated where entities are familiar and details are thin.
The paper points toward that concentration when it notes that producing citations, listing paper titles associated with topics or authors, is one context where models hallucinate. My expectation, which the paper does not test, is that a well-known author with a long list of papers invites exactly this misfire: the name is familiar, the specific title is unknown, and the gate opens.
This view also splits the problem into two parts that researchers can work on separately. One part is the gate's accuracy, meaning how well "known answer" features track what the model has actually stored. The other is what the model does after the gate opens, where the paper observes that a model committed to answering will produce the most plausible continuation. My inference is that improving the second part without the first would produce more fluent hallucinations. Both parts show up in the paper's discussion, and each can fail on its own.
The contrast between the two versions also looks different to me now. The paper calls the base model's pickleball answer reasonable, and I read it as an ordinary prediction of likely text. My reading of what fine-tuning contributed is a link from a self-assessment signal to the output, a circuit that estimates whether the model is in a position to answer and tilts the output toward a refusal when it is not. The model is thus doing two jobs on every question: predicting what comes next and deciding whether to say anything at all.
What this evidence supports
Scope limits the claim. The circuits come from Claude 3.5 Haiku on short prompts about people and papers. The paper describes the Karpathy case as showing that at least some hallucinations arise from a misfire of this inhibitory circuit, a claim that covers some hallucinations and leaves the rest open. Errors that arise over a long generation, where small mistakes compound, would need their own analysis, and nothing here maps them. The authors also report that their intervention experiments gave mostly supportive results with some surprises, which leaves the gate described here as a partial account.
Within those bounds, the finding is concrete enough to carry weight. In my summary, guarded by default and opened by familiarity describes the behavior of at least one deployed assistant model on the cases the researchers examined. The researchers report causal interventions that produced and removed the failure on demand. The next article takes up a close cousin of this question: whether the reasoning a model writes out matches the computation that produced its answer.
What this comes to
The common belief says a chatbot answers everything and makes things up when it runs out of knowledge. The research described here shows a different picture for at least one assistant. It starts out ready to say "I can't answer," and a feeling of recognition about a name or topic switches that caution off. Usually the recognition tracks real knowledge, and the answer is right. Sometimes the name feels familiar while the specific facts are thin, and the model then produces a plausible answer. The single most useful thing to remember is that a made-up answer can come from a model that recognized the subject. Familiarity and knowledge are separate things, and the model's sense of one can run ahead of the other.
Sources: Anthropic, "On the Biology of a Large Language Model" (2025), section "Entity Recognition and Hallucinations". Anthropic, "Tracing the thoughts of a large language model" (March 2025).