Geometry stops being just an alignment target: GeoReader makes vision-language models actually use camera pose and depth
Synopsis
Vision-language models still struggle with metric spatial questions about camera motion, viewpoint change and object distance, and aligning geometry-foundation features into a model's latent space does not guarantee the model uses them. GeoReader first converts geometry into continuous geometric tokens that a frozen vision-language model can read, then freezes that interface and fine-tunes the model for spatial question answering with question-conditioned geometry injection. On MindCube-Tiny, GeoReader with a Qwen3-VL-4B backbone reaches the reported accuracy, and with a Qwen3.5-9B backbone it reaches the reported six-benchmark average.
Interpretation
GeoReader turns per-view geometric signals such as camera pose and depth into continuous input tokens that a vision-language model can read directly, so geometry enters the answering process instead of staying in an intermediate representation. Two earlier routes each leave a gap: one fuses geometry-foundation features with visual tokens and relies on question-answering supervision alone to teach their use, while the other aligns vision-language-model latents with geometry features, yet capturing geometry does not ensure its use. For the released first-stage 3DThinker checkpoint, a feature-alignment model, zeroing the aligned latents leaves MindCube-Tiny accuracy essentially unchanged, so its answers barely depend on those latents; for GeoReader, with RGB and question fixed, removing the geometric residual lowers accuracy.
The recipe has two steps: lightweight Geometric Readers encode each view's geometry into one continuous token, supervised by geometry descriptions predicted by the frozen vision-language model, so geometry is expressed in a form the target model can already read through its language abilities. Geometry descriptions serve only as Stage 1 training targets; at inference the model answers directly from the continuous geometric tokens without generating a geometry chain of thought. Stage 2 freezes the Readers to preserve this interface and fine-tunes only the language model and a geometry injector, which sets each signal's strength from the images and question; training uses about 135K multi-source spatial QA examples, and Reader pretraining uses about 65K geometry descriptions from ScanNet and ScanNet++ training scenes.
Geometry interventions show that answers depend on both the supplied geometric content and its match to the views. In a controlled MindCube-Tiny experiment with annotation-derived camera geometry, removing geometry content lowers accuracy, and shuffling poses across views lowers it further. Against vanilla supervised fine-tuning under matched QA data and training settings, gains are concentrated on questions requiring metric 3D understanding, while other questions are unchanged.
Perspective
The demonstrated scope is seven spatial-reasoning benchmarks and three vision-language model families; main models use camera pose and depth predicted from RGB by Depth Anything 3, while ablations use annotated geometry to exclude estimation errors. The intended readers are researchers and engineers working on spatial reasoning and geometry interfaces for vision-language models.
When geometry is predicted from RGB, pose and depth estimation errors propagate into answers, and the reported failure cases include reference-frame confusion; benchmark performance does not establish reliability for safety-critical uses such as navigation or robotic control, which require application-specific validation and safeguards.
