Skip to main content
Back to timeline
arXivSource publication:

Local prediction stops being a side branch: LoCo lets only incompatible labels compete, beating TagCLIP by 3.4% average F1

Synopsis

Open-vocabulary multi-label recognition has mostly been improved on the text side, with local prediction used only as a spatial repetition of image-level prediction. LoCo instead supplies local features through a spatially consistent visual encoder and applies a Selective Softmax that lets only locally incompatible categories compete while preserving compatible concepts, then adds multi-scale inference. On four multi-label benchmarks, standalone LoCo improves average F1 and mAP over TagCLIP by 3.4% and 3.8%; ensembled with DualPrompt's global predictions, it improves them by 4.8% and 5.2%.

Source-provided article image: LoCo: Selective Local Competition for Discriminative Open-Vocabulary Multi-Label Recognition
arXiv

Interpretation

Local prediction can serve as a discriminative standalone branch for open-vocabulary multi-label recognition rather than only a supplement to global prediction. Earlier open-vocabulary multi-label methods mainly improved global image-text matching or text-side representations, and local prediction was usually implemented as a spatial repetition of image-level prediction and kept as an auxiliary branch. On four multi-label benchmarks, standalone LoCo improves average F1 and mAP over TagCLIP by 3.4% and 3.8%; ensembled with DualPrompt's global predictions, it improves them by 4.8% and 5.2%.

Local prediction works when local visual features are spatially consistent and category competition distinguishes incompatible from compatible concepts. CLIP's default visual encoder yields fragmented, semantically mixed local features, making direct local prediction unreliable, while a uniform softmax over all categories over-suppresses concepts that can coexist. In ablations under single-scale inference, replacing CLIP with ClearCLIP and SC-CLIP raises average F1 from 46.4% to 52.5% and 54.9%, and mAP from 55.0% to 59.8% and 62.4%; across both encoders and single or multi-scale settings, Selective Softmax raises average F1 by 1.3–1.9% and mAP by 0.3–0.5%.

Local compatibility relations among categories can be generated offline by a large language model and fixed for reuse with the same category vocabulary. CLIP measures similarity between an image region and a text query but does not directly provide compatibility or exclusivity relations among category names. Under the same local model and 14-crop inference, Selective Softmax with the GPT-5.5 prior reaches 58.5% average F1 and 66.0% mAP, above raw logits without competition (53.1%, 62.8%) and full softmax (56.6%, 65.5%).

Perspective

Evaluation covers four multi-label benchmarks, Pascal VOC 2007, MS-COCO 2014, NUS-WIDE and Visual Genome 150, with a shared ViT-B/16 visual backbone and identical category sets, reporting macro-averaged F1 and mAP on test sets. The method targets open-vocabulary multi-label settings that must recognize multiple concepts under arbitrary category names, and the code is public.

Each category vocabulary uses one fixed binary compatibility graph, a prior that approximates richer context-dependent relations and requires compatibility relations for the target vocabulary; multi-scale inference trades roughly 14 times the computation for accuracy, and optional global ensembling adds a separate predictor's cost.