Skip to main content
Search

# Mechanistic Interpretability

17 related articles

Subscribe to updates
  1. arXiv

    Cross-Layer TopK lets the model split its sparsity budget: lower reconstruction error than per-layer TopK at equal activations, and 2.9× faster training steps

    Cross-layer transcoders (CLTs) decompose a large language model's layer activations into interpretable features, but layers differ widely in how hard they are to reconstruct, so giving every layer a fixed number of active latents is a poor fit. Cross-Layer TopK, from ByteDance Seed and collaborators, ranks the latents of all layers together and keeps a fixed total of K per token, letting the model decide how many each layer gets. On Qwen3-8B-Base, at the same 360 activations per token as per-layer TopK, validation FVU falls from 27.64% to 27.07%, and pairing it with sparse decoding cuts training step time to about one third.

  2. arXiv

    When can an interpretability claim be trusted: feature dimensionality jumps, while its ridge-regularized version holds one verdict across 12 toy models

    Mechanistic interpretability often rests safety claims on thresholded numbers that can flip under small changes to weights, training seed or threshold. Working on toy models and synthetic dictionaries with known ground-truth features, this work characterizes exactly where feature dimensionality is discontinuous and replaces it with a ridge-regularized version whose stability constant grows with feature scale and whose coefficient is optimal. On 12 trained toy models, the regularized sum rule gave the same verdict at all 21 pre-set values of its smoothing parameter, while the version that ignores short feature vectors gave the opposite verdict for two of the models.

  3. arXiv

    Glyph splits clinical foundation-model embeddings into readable concepts, surfacing 397 reproducible prognostic markers in abdominal CT cohorts

    Glyph decomposes clinical foundation-model embeddings into sparse, nameable concepts so that features previously entangled in high-dimensional vectors can be tested like clinical variables. Across abdominal CT, radiology reports and laboratory data from 41,628 patients, the study tested 604,185 concept–outcome pairs and retained 397 high-risk concepts with 4,463 Bonferroni-significant associations. Most associations reproduced in three held-out cohorts without recalibration, and the top ten concepts reproduced more often than markers recovered from the literature.

  4. arXiv

    A small transformer learns the language of an infinite Coxeter group, and its attention hints at two word-theory propositions proved by hand

    A small transformer with a single attention layer was trained to predict the left and right descent sets of elements of the infinite Coxeter group Ã₂, meaning the possible first and last letters of minimal-length words. Test accuracy improved as training data moved from random words to reduced words to ShortLex normal-form words: the random-word and reduced-word datasets each began deteriorating at some length, while the set of all 1,998 ShortLex words of length at most 36 never deteriorated. From the trained model's attention patterns, the researchers formulated and independently proved two word-theory propositions about right and left descent sets.

  5. arXiv

    Why the Jacobian lens favors slow features: a theory writes its choice as three coherence factors and predicts their ordering in toy models

    The Jacobian lens (J-lens), an interpretability tool that decodes a model's hidden states into words, had been used to find working-memory-like linear directions, but what makes a feature detectable to it was unclear. This work gives a mathematical theory: the lens preferentially selects directions that grow across layers, stay smooth across token positions, and remain consistent across contexts, that is, coherent Lyapunov modes. On a Bayes filter and a two-layer toy transformer, changing task parameters to modulate coherence shifts the ordering of latent states the lens selects, as predicted.

  6. arXiv

    MoE experts don't sort by domain label: an SAE splits each feature's activation by expert, separating Java from front-end code

    Researchers at ByteDance Seed and Shanghai University of Finance and Economics trained a sparse autoencoder (SAE, a dictionary that reconstructs a layer's output as sparse interpretable features) on the MoE output of layers 4, 30 and 46 of Qwen3-30B-A3B, and used the encoder's affinity in the expert outputs to decompose each feature's activation exactly over the experts. Averaged over a corpus, each expert gets the set of features it principally drives; most of those features have only one or two dominators, and the division of labor runs below coarse domains, down to Java versus front-end web code and geometry versus arithmetic and polynomials.

  7. arXiv

    Neural circuits get more than pairwise edges: directed simplicial complexes record what joint neuron suppression does

    Features inside a neural network are often carried by groups of neurons acting together, and a circuit drawn as pairwise edges misses the joint effect of suppressing such a group. The new construction treats the network as a structural causal model, uses counterfactual ablations that stay on the neural manifold to progressively suppress a set of neurons, and admits neuron groups into a directed simplicial complex by a statistical relevance test. On two small handcrafted multilayer perceptrons the complexes recover the designed feature pathways, and their 1-skeletons agree with the edges kept by ACDC, EAP and EAP-IG that a weight graph can express.

  8. arXiv

    An open post-training recipe makes fact-checking models harsher on low-status posters, while seven deployed models show no such tilt

    On fact-checking claims with randomized poster profiles, the open Tülu-3 post-training recipe makes OLMo-2 32B and Tülu-3 70B judge claims from lower-status posters false more often, an effect under one percentage point but statistically reliable, exceeding training drift and replicating on independent claim sets; Meta's, Alibaba's and Mistral's recipes do not produce it. In OLMo-2 32B the model does not notice the job more but interprets it differently, with a lower-status job pushing the base model toward true and the post-trained model toward false, a flip carried by a single residual-stream direction at supervised fine-tuning. None of seven deployed commercial models doubts lower-status posters more.

  9. aclanthology.org

    No retraining, fewer harmful compliances: scaling three internal component groups lifts attack-time safety from 43.2% to 69.7%

    The work decomposes an LLM's refusal behavior into a cross-layer pathway: harmful detection heads in early and middle layers pick out harmful tokens, safety neurons in mid-to-late layers carry and stabilize that signal in the residual stream, and refusal heads turn it into a refusal. Scaling the weights of these three component groups without changing architecture raised average safety under GCG attacks from 43.2% to 69.7% across six models, with a 1.7% average accuracy drop on four standard benchmarks.

  10. Discrete logarithms grok too: a 427k-parameter model memorizes first, then generalizes on prime-order groups

    A two-layer, 427k-parameter transformer shows grokking on the finite-field discrete logarithm map: after a long memorization phase it reaches 99.8% held-out accuracy at p = 113. A 12-model capacity sweep from 90k to 14.4M parameters finds grokking recurring at multiple scales, with a non-monotonic region whose outcome changes when training data increase; hidden states grow aligned with multiplicative-character structure, and prime-order and disjoint-generator controls also generalize.