Skip to main content
Back to timeline
arXivSource publication:

SpatialScout-V2 moves active mapping into real homes: six of seven frontier VLM agents stay below 20% all-target mean IoU

Synopsis

SpatialScout-V2 extends the active object-center mapping task from simulation to 100 real residential captures, covering 3,171 target objects across 29 categories, where agents must choose recorded viewpoints, acquire calibrated RGB views and submit or revise 3D centers within a 120-turn budget. Across seven frontier vision-language models, six fall below 20% all-target mean IoU, and the best, Claude Opus 5.5, reaches only 22.52%, with just 18.36% of targets above an IoU of 0.5.

Source-provided article image: SpatialScout-V2: Can VLM Agents Actively Explore Real-World 3D Scenes and Localize Objects?
arXiv

Interpretation

The evaluation subject for active 3D mapping shifts from simulated rendered images to real panoramic captures of homes, where agents must find viewpoints and estimate object centers inside scenes that keep their original occlusions and non-target objects. The earlier SpatialScout ran the same inventory-based mapping task in simulated indoor scenes and hid unselected candidate objects during rendering. The new benchmark uses the 100 test scenes of Realsee3D's real-world subset, with reference annotations produced by a multi-view pipeline and manually inspected box by box, totaling 3,171 targets across 29 categories.

An agent receives a fixed target inventory and a set of recorded camera positions, then moves, rotates, captures images and triangulates to write each target's 3D center into a map. The triangulation tool performs least-squares ray intersection on correspondences the agent marks across views, requires at least one camera pair separated by 5 cm or more, and returns ray residuals and the maximum pairwise ray angle as consistency diagnostics. Each scene allows at most 120 model turns, depth maps, segmentation masks and reference boxes stay closed to the agent, and scoring translates a reference-sized box to the predicted center and computes 3D IoU under separate all-target and submitted-target accounting.

High scores do not mean accurate localization: a model can submit broadly and still place centers poorly, while omitted targets further lower all-target scores. GPT-6 Astra reaches 21.65% mean IoU over submitted targets, close to Opus at 22.58%, yet its all-target score is 15.50%, 7.02 percentage points below Opus. Opus drops only 0.06 points from submitted to all targets while Astra drops 6.15 points; MiMo, Qwen and DeepSeek show identical values under both accountings but mean IoUs of only 8.68%, 4.87% and 5.49%.

Perspective

The evaluation is limited to the 100 test scenes and 3,171 target objects of Realsee3D's real-world residential subset, agents may only choose among recorded scan positions, and depth maps and reference annotations stay outside the interaction interface.

Results cover seven frontier vision-language models run at the xhigh reasoning setting and count only trajectories whose final call is a successful finish_mapping that passes the manifest check; a manual analysis of 70 trajectories from 10 scenes finds position estimates lacking triangulation support to be the most frequent failure pattern.