Skip to main content
Back to timeline
arXivSource publication:

Frozen encoders get elastic embeddings: HERL uses hyperbolic geometry to span 8 to 8192 dimensions in two GPU-hours on one card

Synopsis

HERL lets a pretrained encoder's embedding dimension be compressed or expanded after the fact without retraining the encoder. Matryoshka representation learning needed costly end-to-end retraining and no post hoc expansion, while lightweight adaptor routes handled only compression or a single reduced size. HERL maps encoder outputs into Lorentzian hyperbolic space with an MLP and trains under a contrastive loss there, holding downstream performance from 32 to 8192 dimensions on image and text tasks, with 8-to-8192-dimensional representations trained in about two hours on one GPU.

Source-provided article image: Elastic Representations via Hyperbolic Geometry
arXiv

Interpretation

Embedding dimension can now be adjusted to a deployment budget after the encoder is frozen, both shrinking and expanding beyond the original size. Matryoshka representation learning gets truncatable embeddings from nested training but needs end-to-end encoder retraining; SMEC supports only a single reduced dimensionality, CSR operates at four times the encoder dimension, and Matryoshka-Adaptor degrades under aggressive compression. On ImageNet-1k 1-NN retrieval and linear probing, HERL stays stable from 8 to 8192 dimensions while random linear projection and SVD degrade sharply below 128 dimensions; on text, 32-dimensional unsupervised HERL reaches 94.39% Top-1 on MTOPIntent against 93.58% for the full 4096-dimensional NV-Embed-V2.

The adaptor pairs an MLP that reparameterizes the Euclidean embedding with a contrastive objective in hyperbolic space, so compression and geometry are co-designed. Earlier adaptors either projected in Euclidean space or split compression and mapping into two independent stages. The MLP output goes through the exponential map at the origin onto the Lorentz hyperboloid, similarities are computed with the Lorentzian metric, and gradients flow back through that map; at inference the first k dimensions are used directly with no hyperbolic operations. Image experiments use a ResNet-50 backbone and text experiments use NV-Embed-V2, all on a single H100.

The benefit of hyperbolic geometry comes mainly from the training objective rather than from the adaptor's layers. Making the adaptor itself fully hyperbolic is a natural alternative. At 32-dimensional embeddings, the fully hyperbolic adaptor gives at most marginal gains over the Euclidean adaptor while training time rises from 683 s to 2964 s and inference per batch of 256 rises from 0.58 s to 1.05 s.

Perspective

The demonstrated scope covers ResNet-50 image embeddings on ImageNet-1k and NV-Embed-V2 text embeddings on MTEB classification, clustering and retrieval tasks, with all experiments run on a single H100; the authors propose extending the method to additional modalities and generative modeling as future work.

HERL is somewhat weaker on retrieval, which the authors attribute to standard retrieval pipelines projecting embeddings onto the unit hypersphere and discarding the radial information that carries semantic separation in hyperbolic space; re-scoring with Lorentz distance improves all three retrieval datasets. At high curvature, performance drops because of a trade-off between hierarchical capacity and local fidelity.