Skip to main content
Back to timeline
arXivSource publication:

Separately prepared parts snap into one object: VA-Assembly estimates translation, rotation and scale with same-view diffusion scores, no training

Synopsis

VA-Assembly treats a single-image 3D reconstruction as the reference and directly optimizes the translation, rotation and scale of independently prepared part geometries, without part annotations, category priors or bounding-box initialization. It conditions a frozen view-conditioned diffusion model on reference renderings taken from the same camera view as the current part composition, so the score gradients align geometrically with that composition. Under translation-rotation, translation-scale and full-transformation protocols, its part-level transformation errors are lower than the corresponding registration and layout baselines, and ablations show same-view conditioning, not diffusion guidance alone, is the key.

Source-provided article image: VA-Assembly: View-Aligned Score Guidance for Training-Free 3D Part Assembly
arXiv

Interpretation

One reference image plus a set of independently prepared part geometries is enough to recover each part's translation, rotation and scale, while the geometries stay unchanged and separately replaceable. Earlier part-aware generation and learning-based assembly rely on part-level supervision, category priors or paired part-layout training data, and registration methods need reliable 3D correspondences between parts and reference. Under the translation-rotation protocol, VA-Assembly reaches a Chamfer distance of 0.019 and a rotation error of 14.061 degrees, against 0.181 and 44.61 degrees for Go-ICP and 0.037 and 61.038 degrees for Assembler.

The guidance comes from a frozen view-conditioned diffusion model asked to predict the noise residual of the current part composition while conditioned on the reference monolithic model rendered from the same camera view. Original Zero-1-to-3 predicts a target view from a fixed source image plus a relative camera pose, so its score is not tied to the view-specific discrepancy of the current rendering. With diffusion guidance alone and no late refinement, same-view conditioning gives a Chamfer distance of 0.038 and a rotation error of 12.795 degrees, versus 0.102 and 45.426 degrees for single-view Zero-1-to-3 and 0.111 and 57.881 degrees for conventional multi-view Zero-1-to-3.

Gradients flow back through a differentiable Gaussian-splatting renderer to the part transformation parameters and are averaged over randomly sampled viewpoints, offsetting the single-view projection's preference for screen-space motion. Under single-view image-space supervision, projected points near the image center respond less to depth-direction motion than to motion parallel to the image plane. Optimization runs 50 iterations: the first 40 use only same-view score guidance for coarse arrangement, and the last 10 add visibility-filtered Chamfer distance and RGB consistency for local refinement.

Perspective

The demonstrated setting is a single reference image with fixed, independently prepared part geometries, evaluated on synthetic benchmarks built from GSO and Breaking Bad with 80 target compositions; the comparison with part-aware generation methods uses 20 GPT-4o-generated reference images. The intended readers are researchers and engineers working on 3D generation, assembly and scene editing.

Rotation-error standard deviations are large for all methods, which the authors attribute to local minima arising from ambiguous geometry, near-symmetric structures or weak view-dependent cues. TEASER++ gives inaccurate scale estimates on Breaking Bad, which the authors link to unreliable correspondences near fracture surfaces. The authors list stronger contact reasoning, collision handling and physically constrained multi-part assembly as future work.