Skip to main content
Back to timeline
arXivSource publication:

Frozen vision-language models can spend a labeling budget too: ACTOR stores delayed labels as retrieval memory and gains 2.5 points across 15 datasets

Synopsis

Test-time adaptation for vision-language models usually relies entirely on pseudo-labels or entropy the model produces itself, which can reinforce its own errors on hard shifted samples. ACTOR treats a small amount of deployment-time human labeling as delayed evidence: it predicts first, stores returned labels in memory, and corrects later samples only when the two nearest labeled neighbors agree and are close enough, leaving model parameters frozen. Across 15 datasets it averages 2.50 and 2.45 percentage points above online prompt tuning given the same labels, with under 1 ms of extra inference per sample.

Source-provided article image: Training-Free Active Test-Time Adaptation for Vision-Language Models
arXiv

Interpretation

A vision-language model facing distribution shift at deployment can spend an explicit labeling budget during streaming inference, but it must emit a prediction for the current sample first, and any queried label can only serve later samples. Prior test-time adaptation for vision-language models largely avoids external labels and adapts from the model's own pseudo-labels, confidence scores or entropy; supervised online adaptation at the other extreme assumes labels can directly train or correct the model. The setting is formalized as Active VLM-TTA and evaluated on ImageNet-derived out-of-distribution benchmarks and ten cross-domain datasets, with a labeling budget of 10% of the test stream counted in the protocol.

ACTOR is a training-free plug-in that routes each high-entropy sample three ways: keep the base adapted prediction, spend budget on a label query, or correct using memory of previously returned labels. The correction rule is deliberately conservative, accepting a retrieved label only when the two nearest labeled samples agree and their average cosine similarity to the test sample exceeds a threshold. On TDA with CLIP-ViT-B/16, the full consistency-and-reliability rule averages 69.50% across Flowers102, DTD, Caltech101, IN-S and IN-V2, above base TDA at 65.31% and 5-NN majority voting at 67.22%, while the harmful-correction rate falls from 13.68% to 5.28%.

After the labeling budget is exhausted, ACTOR falls back to a relaxed Top-5 plausibility check, accepting a retrieved label only if it lies among the five classes with the highest base prediction probability. This fallback drops the full spatial reliability test and keeps only a label-space plausibility constraint, avoiding unrestricted nearest-neighbor correction. The Top-5 threshold is fixed uniformly across all datasets, following the standard criterion used in large-scale image classification.

Perspective

The demonstrated scope is image classification with frozen CLIP-family vision-language models in a single-sample streaming setting, with a labeling budget of 10% of the test stream and thresholds fixed before any benchmark performance was seen and applied to all 15 datasets and both backbones. The intended reader is an engineer working on test-time adaptation or active learning; the method targets settings where queried annotations are reliable and the labeled memory can supply local evidence for later samples.

Effectiveness may decrease when returned labels are noisy, when the stream has highly uneven class coverage, or when the budget is exhausted and correction must lean on the Top-5 plausibility fallback. The latency analysis shows larger drops when delay is a significant fraction of the stream, with DTD and Flowers-102 falling more than the 50k-sample ImageNet-S; the authors list noise-aware memory filtering, class-balanced querying and distance-aware fallback as future directions.