3D generation trains and infers at mismatched resolutions: RDT-3D anchors guidance at 512 while supervising at 1536, lifting F-Score to 0.6886
Synopsis
In two-stage high-fidelity 3D geometry generation, training aligns coordinate guidance and geometric supervision at one resolution, while inference decodes high-resolution geometry from coarse coordinates, and this mismatch over-smooths thin structures. RDT-3D fixes rotary position embedding (RoPE) spatial guidance at a coarse 512 resolution and keeps geometric supervision at 1536, forcing the refinement network to extrapolate detail from imprecise positional cues. On Toys4K, the RDT model reaches a Chamfer distance of 0.0164 and an F-Score of 0.6886, against 0.0184 and 0.6651 for the symmetric 1536 baseline.
Interpretation
In high-fidelity 3D geometry generation, the refinement stage no longer aligns training and inference spatial resolutions: positional guidance is quantized to a coarse grid during training while geometry targets stay supervised at high resolution. Earlier coarse-to-fine pipelines trained coordinate guidance and geometric supervision at the same high resolution, so the model only saw precise positional priors; at inference the positional cues come from a stage-one coarse mesh carrying localization error and missing detail. On Toys4K, the RDT model records a Chamfer distance of 0.0164 and an F-Score of 0.6886, versus 0.0184 and 0.6651 for the symmetric 1536 baseline; adding Gaussian noise to the baseline's coordinates yields only 0.0187 and 0.6659.
The approach separates spatial localization from geometric reconstruction during training so the network learns to recover fine surfaces from imprecise positional cues. Rotary position embedding (RoPE, a way of encoding position into attention) guidance is built from coarse-resolution point clouds, while target latents are encoded from high-resolution watertight surfaces at a different resolution. Coarse guidance is taken from low-resolution ground-truth meshes rather than upstream-generated coarse meshes to keep global structure aligned with the supervision target; at inference a coarse mesh from Hunyuan3D-2.1 or TRELLIS is voxelized and sampled to build RoPE guidance before decoding the high-resolution mesh.
Perspective
The demonstrated scope is a refinement pipeline taking a single image, producing a coarse mesh via Hunyuan3D-2.1 or TRELLIS, and evaluated on Toys4K with Chamfer distance and F-Score; the Dual3D-300K dataset is stated to be open-sourced.
The coarse guidance resolution of 512 is described as an empirical trade-off between quality and computational efficiency; training directly on upstream-generated coarse meshes introduces excessive noise and structural divergence, so low-resolution ground-truth meshes are used instead, and the gap between that substitute and true inference conditions is not quantified.
