Skip to main content
Back to timeline
arXivSource publication:

A hidden activation edit rewrites what a reasoning model thinks it was asked, and it defends the fake request

Synopsis

A reasoning model's belief about what it was asked can be overwritten by an activation edit invisible in the prompt text: after injecting the forming-trace of an unrelated concept, the model reasons about, answers, and restates the injected topic as its task, with the forced quote-back returning that topic in 47 of 48 model-concept cells across six models. Even a warning that names the true topic leaves clean recovery under 5%; the model instead confabulates a story that the user changed their mind, a concrete limit on chain-of-thought monitoring and self-report.

Source-provided article image: Reasoning Models Confabulate to Defend Injected Beliefs About Their Own Requests
arXiv

Interpretation

An activation edit that never appears in the prompt text can replace a reasoning model's belief about its own request: the model then reasons about the injected topic, answers it, and reports it as the task it received when forced to quote the request verbatim. Chain-of-thought monitoring and self-report had assumed a model's account of what it is doing is anchored to what it was actually asked, while ordinary activation steering had been shown to change what a model says. Across six models spanning scale, base architecture, and training recipe, the forced quote-back returns the injected topic in 47 of 48 model-concept cells; on the four models with clean final haiku, 99% of sampled generations (95/96) end on the injected topic.

The method takes the residual-stream state a model forms as it prepares to write about a concept, strips a substrate component shared across concepts, and injects the concept-specific remainder during the receiver's prompt forward pass. The injection is applied only during that single prompt forward pass, with no hooks during generation, so the prompt text itself stays unchanged. On R1-Qwen-7B, the mean next-token probability mass on the injected concept's word set is 0.84 under a matched injection versus 0.05 under a mismatched-concept control, so the effect is specific to the injected concept.

A warning changes how the model talks about the conflict in its chain-of-thought, not what it finally writes: named the true topic, the model often notices both topics and tries to reconcile them, then writes the injected answer anyway. With no warning the chain-of-thought runs cleanly through the injected topic with no sign of conflict. Over 96 sampled draws per condition on the four clean-final-haiku models, the injected topic appears alone or alongside the true one in 90% to 97% of generations, clean recovery is 3% to 4%, and an explicit warning naming the truth is no better than a vague flag.

Perspective

The findings rest on open-weight reasoning models from 7B to 32B, and the intervention requires white-box activation-write access that prompt text alone cannot reproduce; behavioral counts come from the four models with clean final haiku, while Nemotron and QwQ are read as draft direction only because their generations truncate before a clean close.

Whether the model experiences the edit as corrupted input perception or as a failure of self-knowledge is left open, with the source arguing either reading leaves its central claim intact; whether the failure to notice the edit transfers to introspection-trained, closed-source frontier models with stronger introspection remains an open question.