Skip to main content
Back to timeline
arXivSource publication:

3D latents stop paying for detail in tokens: per-voxel normalization lets one tokenizer scale at inference

Synopsis

Latent 3D representations are capped by latent capacity, so complex objects lose fine detail, while scaling capacity at training time runs into memory limits. Split-and-scale divides a scene into sub-regions, normalizes each back to the original coordinate range, and runs the same pretrained tokenizer on each, so equal capacity covers a smaller extent and detail rises at inference. On reconstruction it matches LiTo models trained with the same total latent count and beats naive inference-time token scaling; its autoregressive generator is competitive with existing baselines.

Source-provided article image: Split-and-scale Latent 3D Representations
arXiv

Interpretation

A 3D latent representation can gain effective capacity at inference without adding tokens during training. Capacity was previously raised mainly by adding latent tokens, but compute grows quadratically and memory linearly with token count, and GPU memory ultimately caps the trainable latent size; adding tokens only at inference saturates quickly and can even reverse when pushed too far. In reconstruction experiments, split-and-scale is competitive with a LiTo model trained at the same total latent count and outperforms the comparable LiTo model when scaled to more tokens; ablations show it consistently beats naive inference-time scaling of the underlying tokenizer.

The method runs the same tokenizer separately on sub-regions of the scene, each still using the training-time number of queries. Each voxel is shifted and rescaled back to the original scene extent before encoding and inverted at the decoder, using the space-frequency duality of the Fourier transform so every voxel presents the encoder with the frequency range it saw in training. In ablations, LPIPS degrades sharply with subdivision when normalization is omitted and recovers with voxel normalization; training at coarse subdivisions generalizes to finer inference subdivisions, and training fits within a single 80GB GPU.

Perspective

The demonstrated scope is object-level reconstruction and image-conditioned generation on Objaverse-XL with LiTo as the base, with training fitting within a single 80GB GPU; the reconstruction test set is 200 objects with the highest aesthetic score from TRELLIS's test set, and generation metrics are computed over 2000 test objects.

Independent encoding and decoding of sub-regions can produce subtle seam artifacts in strongly view-dependent areas, appearing as highlights cut off at voxel boundaries; equal-size voxels ignore content complexity; the number of output Gaussians grows with subdivision level; autoregressive generation occasionally shows global structural flaws, with slight asymmetries or floating geometry in occluded regions.