Skip to main content
Back to timeline
arXivSource publication:

Multimodal models swap coordinate text for a 1D vector codebook: DVD reaches 40.0% AP3D on SUN-RGBD and 499 ms per object

Synopsis

When multimodal large language models handle 2D and 3D perception, coordinates are either written as text, which costs many tokens, or quantized into a fixed integer range, which limits range and precision. DVD converts 2D boxes, 2D masks and 3D boxes into 1D vector sequences, maps them to a few discrete tokens through a shared codebook, and restores the original representation at inference with a lightweight de-tokenizer. On SUN-RGBD, Hypersim and nuScenes, DVD-2B reaches 40.0%, 15.7% and 19.8% AP3D, and on RefCOCOg val it reaches 89.9% Precision and 70.4% cIoU.

Source-provided article image: DVD: Dynamic Vector Decoding for Efficient MLLM-based Perception
arXiv

Interpretation

Perception outputs from multimodal large language models no longer have to be coordinate text or fixed-range integers; 2D boxes, 2D masks and 3D boxes can share one discrete-token representation. Text-based coordinate representation has no minimum quantization unit and unlimited range, but generates many tokens per object; fixed-range quantization normalizes coordinates into an integer range such as 0 to 1000 and is constrained in range and precision in the theoretically unbounded 3D space. On the three 3D benchmarks SUN-RGBD, Hypersim and nuScenes, DVD-2B reaches 40.0%, 15.7% and 19.8% AP3D, and DVD-8B reaches 40.3%, 18.4% and 24.4%; the same model gives 89.9% Precision and 70.4% cIoU on RefCOCOg val.

The unified representation comes from a 1D vector autoencoder: an encoder compresses the sequence into a latent space, a codebook picks the nearest discrete entry, and a decoder reconstructs the sequence. The codebook holds 4096 entries with a latent channel of 16 and a downsampling factor of 2; training adds a geometric loss based on bidirectional chamfer distance to the reconstruction and VQ losses, and that loss is insensitive to point ordering and supervises 2D and 3D tasks together. In an ablation on DVD-2B run for 30% of full training iterations, removing the geometric loss drops reconstruction from 90.8% AP3D, 100.0% Precision and 84.8% cIoU to 80.7%, 99.1% and 74.8%; codebook sizes of 2048 or 8192 score below 4096 on all three metrics.

Hooking into the language model needs only a vocabulary extension and reuse of the pretrained decoder, with no change to the backbone and no per-task decoder. At inference the de-tokenizer first maps perception tokens back to codebook vectors, then the 1D decoder reconstructs the sequence, which is converted into the original 2D or 3D representation. Compared under identical training settings on SUN-RGBD and RefCOCOg, DVD uses 8 tokens per 3D box with 499 ms latency per object, text coordinates use 74 tokens with 4165 ms, and fixed special tokens use 15 tokens with 713 ms but only 17.5% AP3D.

Perspective

The results cover 2D grounding, 2D referring segmentation and 3D grounding, evaluated on RefCOCO, RefCOCO+, RefCOCOg, SUN-RGBD, Hypersim, KITTI, nuScenes and ARKitScenes, with Qwen3-VL backbones at 2B and 8B trained on 8 NVIDIA A100 GPUs.

The material states that reconstruction performance of the dynamic vector representation can still improve even with a larger downsampling factor, and that the de-tokenizer is a separate component from the multimodal model, so this decoupling may leave the learned discrete tokens suboptimally aligned with the model's semantic understanding.