Skip to main content
Back to timeline
arXivSource publication:

Discrete codes can carry vision too: dNaViT uses one codebook for understanding, reconstruction and generation

Synopsis

dNaViT encodes images into discrete codes of the same kind language models predict, and one shared vocabulary serves visual understanding, image reconstruction, generation and editing. It builds on a frozen pretrained vision encoder, adds reconstruction-relevant detail through residual modulation, and uses grouped product quantization to widen discrete capacity without lengthening the spatial sequence. On ImageNet-1K, the 64-group dNaViT-G64 reaches 32.90 dB PSNR and 0.91 SSIM at 1024 spatial positions, and dNaViT-8B paired with Qwen3-8B scores 885 on OCRBench.

Source-provided article image: dNaViT: Discrete Codes are General-Purpose Visual Representations
arXiv

Interpretation

Images no longer have to travel as continuous features: one discrete codebook can carry both semantic understanding and pixel-level reconstruction, putting vision and language under the same next-token prediction framework. Multimodal models have typically kept continuous semantic features for understanding and reconstruction-oriented continuous latents for generation, two representations with two separate objectives, while discretization has long been treated as an information bottleneck that loses detail. On ImageNet-1K reconstruction, the 64-group dNaViT-G64 reaches 32.90 dB PSNR and 0.91 SSIM at 1024 spatial positions, against 28.93 dB and 0.81 for the 16-group dNaViT-G16 in the same setting; on TokBench at 1024 resolution, dNaViT-G64 reaches 63.08% text recognition accuracy, 78.27% normalized edit similarity and 0.67 face similarity.

The construction is a light modification of a frozen pretrained vision encoder, so its semantic features pick up reconstruction detail before being compressed into discrete codes. The residual modulator ResMod fuses intermediate features from several encoder depths and learns a per-position scale and shift correction whose residual update is zero-initialized, so it starts from the original semantic features; grouped product quantization splits channels into groups with a separate codebook each, widening discrete capacity without adding spatial tokens. Training proceeds in three stages: continuous pixel pretraining, discrete semantic reconstruction with the encoder and ResMod frozen, then joint optimization of both branches with a frozen teacher copy fixing the semantic target; the tokenizer combines a 400M-parameter Qwen3-VL ViT with a 100M-parameter ResMod, and its corpus spans general images, text-dense documents and fine-grained images at resolutions rising to 2K.

Perspective

The demonstrated scope is evaluation on image reconstruction, linear-probe classification, multimodal understanding, text-to-image generation and instruction-guided editing, with the vision encoder frozen throughout tokenizer training and image resolutions up to 2K. A reader working on multimodal representations or unified generative models can use this to judge whether a discrete vocabulary can serve as a shared interface; the material states no next experimental step.

The authors state that dNaViT-G16 and dNaViT-G64 differ in training configuration and duration, so their gap cannot be attributed to quantization group count alone; small text remains challenging and a gap persists relative to stronger reconstruction models such as FLUX.1; and the discrete-versus-continuous comparison is model-level, with possibly different training data, resolutions and scoring protocols.