VIBRA upgrades a cross-modal retriever without re-encoding the gallery, winning all 50 settings at under 1 ms extra latency
Synopsis
Upgrading the encoder of a cross-modal retrieval system normally means re-encoding billions of gallery vectors, costing hundreds of GPU hours. VIBRA pools the queries of the new, old and an auxiliary encoder into a consensus barycenter that serves as teacher, then trains the new encoder through a low-rank update to inherit that consensus, so deployment runs one encoder. Across five benchmarks and five upgrades among six CLIP, MetaCLIP and SigLIP2 backbones, it is the best compatible method in all 50 settings, improving Recall@1 by up to 5.6 points over the strongest baseline and up to 18.0 points over the old model, while adding less than 1 ms to the new encoder's latency.
Interpretation
A cross-modal retrieval system can adopt a new query encoder while the old gallery stays untouched, and the new encoder still queries items the old model already encoded. Earlier compatible methods either fit the new query to the old one, so accuracy stays near the old model, or run both the old and the new encoder for every query. Across five benchmarks and five upgrades, 50 settings in all, VIBRA ranks first among compatible methods, improving Recall@1 by up to 5.6 points over the strongest baseline and up to 18.0 points over the old model, and on average by 6.5 points over the old model and 2.3 points over SLERP, which runs two encoders.
The training target no longer imitates the old query; it comes from the optimality condition of the retrieval loss, under which the gallery barycenter weighted by a query should coincide with the barycenter of the relevant items. A target defined by the old query reproduces at best the old model's ranking, including its errors. Relevance is estimated by mixing the paired caption with the new model's own ranking, and both are needed: the label alone is 1.2 to 4.1 points lower and the new model's ranking alone 0.6 to 1.7 points lower.
The teacher is a barycenter of three views, the queries of the new model, the old model and an auxiliary encoder in the old space, weighted by a noise model, and it is itself a query in the old space. Any query derived from one model ranks wrongly whatever that model ranks wrongly with confidence; pooling corrects errors only when the views err on different items. On 88.4% to 93.0% of test queries the three views have different hardest irrelevant items, and there the barycenter corrects 10.0% to 13.4% of the new view's errors, 2.5 to 3 times the 4.0% to 4.5% on queries where the views share that item.
The new encoder reproduces the barycenter through a low-rank update of the attention query and value projections, merged into the weights before deployment, so each query runs one encoder. The teacher needs the old and the auxiliary encoder, which deployment forbids. Image-query latency stays within 0.5 ms of the new encoder and text queries add 0.4 to 0.9 ms, while SLERP costs 7.4 to 8.2 ms more per query than VIBRA; training takes 0.22 to 0.71 A100 hours.
Perspective
The demonstrated scope is vision-language retrieval with a fixed old gallery, five upgrades among CLIP, MetaCLIP and SigLIP2 backbones, training on CC3M only, and evaluation on Flickr30k, COCO, NoCaps, CxC and XM3600.
The barycenter gains only when the views' hardest irrelevant items differ; when all three views share that item, no weighting corrects the error. The noise model overestimates the absolute size of gains by a factor of three to four, and the student can only learn the part of the gain the barycenter shows on the training images.
