Skip to main content
Back to timeline
ACM Transactions on GraphicsSource publication:

3D generation skips the latent space: Direct3D-V3 diffuses directly on SDF voxels for finer mesh detail

Synopsis

Direct3D-V3 moves image-to-3D diffusion out of a compressed latent space and onto raw SDF voxel grids, so neither training nor sampling passes through a VAE. The dominant recipe first compresses geometry into vecset, triplane or sparse-voxel latents and then diffuses there, and detail lost in encoding cannot be recovered downstream. To make direct voxel modeling workable, the model splits the work between a patch-level and a voxel-level DiT and adds a neighbor-SDF smoothness constraint, differentiable Marching Cubes normal supervision and camera prediction; on 102 real, web-collected and AI-generated images its Uni3D and ULIP-2 consistency scores exceed the listed open-source methods, and the authors report finer local mesh detail.

Source-provided article image: Direct3D-V3: Latent-Free Diffusion Transformers in the Native Voxel Domain
arXiv

Interpretation

Image-to-3D generation can skip VAE compression and train and sample a diffusion model directly on raw SDF voxel grids, yielding meshes that keep finer local geometry. The latent route first compresses geometry into vecset, triplane or structured sparse-voxel latents and then generates in that space, so detail lost during encoding cannot be recovered downstream. On 102 input images from real photographs, web collections and AI generation, Direct3D-V3 scores 0.4198 on Uni3D and 0.4535 on ULIP-2, above TRELLIS2 at 0.4172 and 0.4405 in the table and above Hunyuan3D 2.1, Direct3D-S2 and UltraShape; adding about 200K private meshes raises these to 0.4224 and 0.4547.

Predicting SDF voxels directly produces non-watertight surfaces, and a smoothness constraint between neighboring voxels pulls the zero level set back toward a coherent manifold. In latent methods the VAE decoder implicitly learns the local distance-field structure near mesh surfaces during reconstruction training, so even noisy features often decode into closed surfaces; removing the VAE removes that prior. On the held-out Objaverse-XL subset, adding the constraint cuts boundary-edge count from 2462 to 196, and at the higher resolution from 25353 to 898.

The predicted SDF grid converts directly into a mesh via Marching Cubes and can be rendered for normal comparison against ground truth, giving geometric supervision without a VAE decode. Hi3DGen also used rendered normal supervision, but had to decode through a VAE before extracting and rendering a mesh, and that memory cost limited the usable resolution. In the ablation, the patch-level DiT alone gives Normal-FID 92.96, adding the voxel-level DiT lowers it to 75.80, and adding the constraint and normal supervision reaches 68.97.

The same native SDF output supports generative reconstruction: normals estimated from the input image correct the visible surface early in the sampling trajectory while the generative prior completes unseen regions. In latent pipelines each test-time optimization step must decode an intermediate latent into geometry before mesh extraction and rendering, making high-resolution iteration expensive. On GSO, Direct3D-V3 reaches Chamfer Distance 0.0131 and F1@0.99 0.6043, better than TRELLIS2 at 0.0147 and 0.5934 in the table, while its Volume IoU of 0.3815 is slightly below TRELLIS2's 0.3884.

Perspective

The demonstrated scope is single-object image-to-3D generation and single-view generative reconstruction: training uses 320K meshes from Objaverse-XL, evaluation uses 102 input images and GSO scanned objects, and multi-view input is shown only qualitatively on Tanks and Temples. The intended readers are researchers and engineers working on 3D generation and diffusion models.

Generative reconstruction depends on the surface patch indices produced by the first stage; if that support misses image-observed structures or places patches inconsistently, normal guidance can over-update the SDF and distort the mesh, especially around thin structures and ambiguous silhouettes. The current model handles geometry only, and texture, material and other appearance attributes are not yet part of the same native voxel space.