Skip to main content
Back to timeline
arXivSource publication:

After learning new facts, LLMs still misjudge what they can answer: training feeling of knowing separately lifts post-update AUROC to 0.809

Synopsis

After knowledge fine-tuning, a large language model answers more questions, but its judgment of whether it can answer a given question often stays at the pre-update boundary. This work splits knowledge updating and the feeling-of-knowing judgment into two trainable channels, using on-policy supervised fine-tuning, feeling-of-knowing training across knowledge states, and feeling-of-knowing-guided policy optimization so the judgment follows current knowledge. On a TriviaQA knowledge update with Qwen2.5-7B, the full pipeline raises held-out feeling-of-knowing AUROC from 0.519 after vanilla knowledge fine-tuning to 0.809, and held-out answer accuracy from 0.546 to 0.692.

Source-provided article image: Feeling of Knowing in Large Language Models
arXiv

Interpretation

Knowledge fine-tuning changes what a model can answer without necessarily changing its judgment of whether it can answer, and this work treats that misalignment as a trainable target. Earlier feeling-of-knowing estimators were mostly calibrated and evaluated under fixed parameters, leaving open whether the judgment stays aligned after supervised fine-tuning, low-rank adaptation, knowledge editing or continual learning. On a TriviaQA knowledge update with Qwen2.5-7B, vanilla knowledge fine-tuning raises update-set accuracy from 0.680 to 0.874 but drops held-out accuracy to 0.546, below the base model's 0.672, while held-out feeling-of-knowing AUROC falls from 0.688 to 0.519.

The approach trains a knowledge channel and a feeling-of-knowing channel separately, then optimizes the judgment together with the response that follows it. On-policy supervised fine-tuning computes loss only on the corrected answer span of the model's own sampled responses to limit disruption to existing feeling of knowing, and knowledge-delta feeling-of-knowing training learns from several checkpoints along that update whether the current state can answer correctly. On the Qwen2.5-7B TriviaQA setting, held-out AUROC is 0.603 after on-policy supervised fine-tuning alone, rises to 0.770 with knowledge-delta feeling-of-knowing training, and reaches 0.809 with feeling-of-knowing-guided policy optimization, at a held-out accuracy of 0.692.

The feeling-of-knowing judgment becomes part of generation: the model writes a short reasoning trace, gives a yes-or-no judgment, then answers or explains why it cannot. Earlier approaches mostly scored confidence as a separate post-hoc quantity or trained the model only to say it does not know. On Qwen3-8B, varying the judgment threshold leaves the coverage-accuracy curves of the base model and vanilla knowledge fine-tuning nearly flat, on-policy fine-tuning plus knowledge-delta training restores a monotonic trade-off, and policy optimization makes judgments nearly binary so the threshold no longer matters.

Perspective

The demonstrated scope is gradient-based knowledge updates on short-answer factual QA, with main experiments on Qwen2.5-7B-Instruct, a coverage-accuracy evaluation on Qwen3-8B, three sequential update rounds, and cross-dataset tests on SelfAware, Natural Questions and WebQuestions.

Feeling of knowing here is a behavioral answerability judgment and does not imply subjective experience; causal-tracing layer locations depend on the prompt, token positions and intervention and do not establish a uniquely localized or human-equivalent metacognitive mechanism; the cue analyses lack a matched pretrained control, so they cannot tell whether training induces cue sensitivity or preserves it; and strong estimators remain competitive when there is no knowledge update.