Across three 7B-scale models and three datasets, the hallucination signal is dominated by a single mean-shift direction, and simple L2 logistic regression reaches 0.952 AUROC while bounding or outperforming twelve architectural alternatives
Synopsis
This work characterizes the geometry of hidden-state probe signals for LLM hallucination detection across three 7B-scale models and three datasets in a paired-example paradigm, finding the signal overwhelmingly dominated by a single mean-shift component whose removal collapses detection to chance, with shrinkage linear discriminant analysis closing about 73% of the gap between 1D and full-dimensional classifiers, a simple L2-regularized logistic regression at 0.952 AUROC bounding or outperforming twelve controlled architectural alternatives, and LayerMix multi-layer aggregation matching oracle-layer performance without oracle access.
Interpretation
The hallucination detection signal in hidden states is dominated by a single mean-shift component, and removing this direction collapses detection to chance. Prior work showed hidden-state probes are effective but left the signal geometry poorly characterized, driving increasingly complex probe architectures; this work directly identifies the dominant structure of that geometry. Observed across three 7B-scale models and three datasets in a paired-example paradigm, and verified by the collapse of detection to chance when the direction is removed.
Shrinkage linear discriminant analysis closes about 73% of the gap between 1D and full-dimensional classifiers, indicating that apparent architectural complexity largely reflects high-dimensional covariance estimation difficulty rather than exploitable non-linearity. Attributes the gains of complex probe architectures to high-dimensional covariance estimation rather than to a signal that requires non-linear modeling. Uses shrinkage linear discriminant analysis as a control and quantifies the closed gap between 1D and full-dimensional classifiers at about 73%.
A simple L2-regularized logistic regression at 0.952 AUROC bounds or outperforms twelve controlled architectural alternatives, and multi-layer aggregation exceeds CLAP cross-layer attention probing under a matched paradigm. In controlled comparison, a simple linear probe is no worse than, and can exceed, more complex architectural alternatives, and outperforms CLAP cross-layer attention probing. Reports 0.952 AUROC and compares against twelve controlled architectural alternatives and CLAP cross-layer attention probing under a matched paradigm.
The signal spans a contiguous layer band, and LayerMix aggregates it to match oracle-layer performance without oracle access. Provides a multi-layer aggregation approach that reaches oracle-layer performance without oracle layer selection. Based on the observation that the signal spans a contiguous layer band, LayerMix matches oracle-layer performance without oracle access.
Perspective
This work targets researchers and engineers studying LLM hallucination detection within the controlled paired-example paradigm, for 7B-scale models and the three datasets examined; its claims are explicitly scoped to that paradigm. For practitioners seeking to replace complex architectures with simple linear probes, or to perform multi-layer aggregation without oracle layer access, LayerMix and L2-regularized logistic regression offer directly usable directions.
Whether the signal geometry is likewise dominated by a single mean shift in larger models, other datasets, or other task paradigms remains an open question; the conclusion that shrinkage LDA closes about 73% of the gap depends on the 1D and full-dimensional classifier settings examined; the 0.952 AUROC and the comparison against twelve architectural alternatives are constrained by the controlled experimental conditions; and the LayerMix result matching oracle-layer performance rests on the observation that the signal spans a contiguous layer band.
