Skip to main content
Back to timeline
arXivSource publication:

Zero-shot detection drops propose-then-match: FOMO goes single-stage end-to-end, scoring 54.2 mAP at 0.23 s per image on BOP

Synopsis

Zero-shot model-based object detection has generally followed a propose-then-match pipeline, where target-agnostic proposals are generated and then matched to template features in loosely coupled stages, leaving them prone to fragmentation in clutter and mismatches under occlusion. FOMO injects CAD template information early in the detection pipeline, using learnable latent tokens that alternate bidirectional attention between templates and the query image before decoding boxes and confidence directly. On BOP-Classic-Core, FOMO reaches 54.2 mean AP at 0.23 s per image, 3.8 percentage points below the top method while running faster.

Source-provided article image: Fear of Memorizing Objects: End-to-End Zero-Shot Model-Based Object Detection
arXiv

Interpretation

Zero-shot model-based object detection moves from multi-stage proposal matching to single-stage end-to-end detection, with template information entering the image representation before boxes are decoded. Earlier approaches first generated CAD-agnostic mask proposals, then encoded and matched them by similarity in loosely coupled stages, so early errors propagated downstream. Across the seven BOP-Classic-Core datasets, FOMO reaches 54.2 mean AP, 3.8 percentage points from the top method 3PT-Det, at 0.23 s per query image.

Templates and images exchange information through a set of learnable latent tokens that first read multi-view templates, then the query image, after which image features query those tokens back. This Interleaved Cross-Modal Transformer uses the latent tokens as a compression bottleneck, avoiding attention from every image patch to every template patch. On the Clutter6D validation set the full model averages 61.6 AP; removing the Stage 3 image conditioning drops it to 55.4, and removing the Deformable DETR encoder and decoder drops it to 47.0.

Perspective

The demonstrated scope is 2D detection of objects unseen in training from RGB images alone on the seven BOP-Classic-Core datasets, with that dataset's CAD models supplied at inference; the Clutter6D validation set is strictly disjoint from training in objects and scenes.

Performance depends heavily on accurate CAD models, and degraded texture or geometry lowers results; in scenes with multiple similar objects such as T-LESS, the ICMT cannot differentiate object features even with inter-prompt communication, producing incorrect predictions.