Cross-model interpretability stops talking past itself: GB-SAE unifies ViT, SigLIP and DINOv2 concepts in one shared dictionary
Synopsis
Foundation models trained on similar data learn overlapping semantics, yet their representations are incompatible, and earlier methods only aligned each model's own concepts in a shared coordinate system while the concepts themselves stayed unshareable. GB-SAE casts cross-model interpretability as representation factorization, using a shared bottleneck encoder and a learnable gate to split latents into a shared dictionary and private dictionaries; on ImageNet-1K representations from ViT, SigLIP and DINOv2, the shared dictionary alone reaches a mean of 0.53/0.55 (projected/raw) and total reconstruction reaches 0.60/0.60, a 60% improvement over predicting the feature mean.
Interpretation
GB-SAE lets several heterogeneous vision foundation models share one semantic dictionary, rather than merely aligning their concept indices in a common coordinate system. USAE and SPARC learn model-specific concepts and align latent coordinates through cross-reconstruction objectives, but concepts remain in each model's native representation space and a separate encoder-decoder pair must be kept per model. On ImageNet-1K representations from ViT, SigLIP and DINOv2, GB-SAE's shared dictionary alone reaches a mean reconstruction of 0.53/0.55 (projected/raw) and total reconstruction reaches 0.60/0.60; by comparison, USAE and SPARC reach cross-model reconstruction of 0.20 and 0.41.
The shared dictionary concentrates most transferable semantics in one set of concepts, while private dictionaries add only model-specific residuals. Shared-branch reconstruction alone is close to total reconstruction, whereas private dictionaries alone reach a mean of only 0.10/0.12, indicating that model-specific bias accounts for a smaller semantic share. In zero-shot cross-model feature retrieval, retrieval with shared representations beats random retrieval, private-representation retrieval and projected-feature retrieval by up to two orders of magnitude, with the latter two near random level.
The shared dictionary transfers to models unseen during training and to downstream tasks. CLIP, absent from the training ensemble, needs only its frozen random projector replaced by a learnable single-layer linear adapter, while the shared encoder, gate and shared decoder stay frozen. CLIP adaptation converges within 4–5 epochs and reaches shared-branch reconstruction of 0.50/0.50; downstream classification trained on shared-dictionary reconstructions reaches mean Acc@1/Acc@5 of 0.86/0.97, above projected features at 0.76/0.92, USAE at 0.72/0.92 and SPARC at 0.78/0.93.
Perspective
Evaluation is confined to vision foundation models on ImageNet-1K, with main experiments on ViT, SigLIP and DINOv2 backbones plus three-model and four-model configurations; downstream tasks cover only linear classification and cross-model feature retrieval.
Whether the approach extends to language models, multimodal architectures or domains with weaker visual regularity remains open; the pairwise cross-model alignment loss scales quadratically with the number of models, so larger ensembles may need approximate or hierarchical alignment; the shared-versus-private split is soft and sensitive to the gate regularization strength, with no principled selection criterion beyond empirical tuning.
