Skip to main content
Back to timeline
arXivSource publication:

Cross-model interpretability stops talking past itself: GB-SAE unifies ViT, SigLIP and DINOv2 concepts in one shared dictionary

Synopsis

Foundation models trained on similar data learn overlapping semantics, yet their representations are incompatible, and earlier methods only aligned each model's own concepts in a shared coordinate system while the concepts themselves stayed unshareable. GB-SAE casts cross-model interpretability as representation factorization, using a shared bottleneck encoder and a learnable gate to split latents into a shared dictionary and private dictionaries; on ImageNet-1K representations from ViT, SigLIP and DINOv2, the shared dictionary alone reaches a mean R2R^2 of 0.53/0.55 (projected/raw) and total reconstruction reaches 0.60/0.60, a 60% improvement over predicting the feature mean.

Source-provided article image: Learning a Unified Cross-Model Semantic Dictionary via Gated Bottleneck Sparse Autoencoders
arXiv

Interpretation

GB-SAE lets several heterogeneous vision foundation models share one semantic dictionary, rather than merely aligning their concept indices in a common coordinate system. USAE and SPARC learn model-specific concepts and align latent coordinates through cross-reconstruction objectives, but concepts remain in each model's native representation space and a separate encoder-decoder pair must be kept per model. On ImageNet-1K representations from ViT, SigLIP and DINOv2, GB-SAE's shared dictionary alone reaches a mean reconstruction R2R^2 of 0.53/0.55 (projected/raw) and total reconstruction reaches 0.60/0.60; by comparison, USAE and SPARC reach cross-model reconstruction R2R^2 of 0.20 and 0.41.

The shared dictionary concentrates most transferable semantics in one set of concepts, while private dictionaries add only model-specific residuals. Shared-branch reconstruction alone is close to total reconstruction, whereas private dictionaries alone reach a mean R2R^2 of only 0.10/0.12, indicating that model-specific bias accounts for a smaller semantic share. In zero-shot cross-model feature retrieval, retrieval with shared representations beats random retrieval, private-representation retrieval and projected-feature retrieval by up to two orders of magnitude, with the latter two near random level.

The shared dictionary transfers to models unseen during training and to downstream tasks. CLIP, absent from the training ensemble, needs only its frozen random projector replaced by a learnable single-layer linear adapter, while the shared encoder, gate and shared decoder stay frozen. CLIP adaptation converges within 4–5 epochs and reaches shared-branch reconstruction R2R^2 of 0.50/0.50; downstream classification trained on shared-dictionary reconstructions reaches mean Acc@1/Acc@5 of 0.86/0.97, above projected features at 0.76/0.92, USAE at 0.72/0.92 and SPARC at 0.78/0.93.

Perspective

Evaluation is confined to vision foundation models on ImageNet-1K, with main experiments on ViT, SigLIP and DINOv2 backbones plus three-model and four-model configurations; downstream tasks cover only linear classification and cross-model feature retrieval.

Whether the approach extends to language models, multimodal architectures or domains with weaker visual regularity remains open; the pairwise cross-model alignment loss scales quadratically with the number of models, so larger ensembles may need approximate or hierarchical alignment; the shared-versus-private split is soft and sensitive to the gate regularization strength, with no principled selection criterion beyond empirical tuning.