Single-image multi-view generation stops trusting geometry everywhere: RGMV gates injection by region, reaching 21.94 dB PSNR on GSO
Synopsis
When multi-view images are generated from one reference image, predicted geometry is unreliable in occluded and thin-structure regions, yet existing geometry-guided pipelines inject it at one global strength. RGMV builds a parameter-free reliability map from the spatial agreement among normal predictions sampled with different random seeds for the same reference image and target views, then uses it to modulate geometry-residual injection pixel by pixel. On the GSO test set, PSNR reaches 21.94 dB with Era3D as the geometry proposal source, above Era3D's standalone 20.77 dB; with MV-Adapter proposals, downstream reconstruction reaches a Chamfer Distance of 0.026 and Volume IoU of 0.62.
Interpretation
Multi-view RGB synthesis now treats predicted geometry differently by location: regions of high agreement keep stronger geometric guidance, and low-agreement regions are attenuated before those cues reach synthesis. Earlier geometry-guided pipelines treated predicted normal or depth maps as fixed conditions injected at a globally uniform strength, implicitly assuming all geometric cues are equally trustworthy. With the RGB synthesis model frozen, uniform injection raised PSNR from 24.22 dB to 24.43 dB, while the SCG reliability map reached 24.63 dB, with SSIM rising from 0.8498 to 0.8545 and LPIPS falling from 0.2073 to 0.1935.
Reliability comes from repeated sampling by the geometry proposal generator itself, requiring no reliability labels and leaving the normal maps unmodified. SCG is a parameter-free gate that converts angular deviation among normal predictions from different random seeds for the same reference image and target poses into a per-pixel reliability response. Over roughly 30 million pixel locations, angular disagreement among predicted normals correlates positively with the anchor normal's angular error against ground truth, with a pooled Spearman coefficient of 0.299 and an AUROC of 0.729 for discriminating high-error pixels.
The reliability map modulates residuals produced by a normal-conditioned ControlNet through the RGI module rather than replacing the geometry condition. Conventional synthesis multiplies geometry residuals by a fixed injection scale, whereas RGI affinely calibrates the reliability map, resizes it to each denoising block's resolution, and multiplies it element-wise into the residual. On the fixed Objaverse validation subset, SCG injection beats uniform injection with a frozen model; after the same RGB model adaptation, RGMV raises PSNR from 25.14 dB to 25.31 dB, SSIM from 0.8569 to 0.8621, and lowers LPIPS from 0.1948 to 0.1823.
Perspective
The demonstrated scope is six-view synthesis and downstream 3D reconstruction on an Objaverse training subset of roughly K objects, a fixed validation subset, and the fixed GSO test set; real captured images are covered only by qualitative comparison and a failure audit on a fixed random sample. SCG assumes no specific camera angles, but evaluation on arbitrary or continuous poses is outside the scope of this work.
The reliability map is only a proxy for prediction stability, is not supervised by ground-truth normal error, and does not assume agreement equals geometric accuracy. In the real-image audit, RGMV fixes or alleviates some of the baseline MV-Adapter's geometry-related failures, leaves some unresolved, and introduces a small number of new geometry artifacts. Raising the number of geometry proposals from 1 to 4 yields only negligible quality change while clearly increasing runtime.
