NIR-guided denoising drops the alignment requirement: UGD-FM handles 2D large-range misalignment in one or two inference steps
Synopsis
UGD-FM recasts cross-modal guided denoising as conditional flow matching: starting from the noisy RGB image, it transports the image distribution toward the clean one along a progressive path while gradually aggregating unaligned near-infrared guidance. Most prior guided denoising methods assume spatially aligned inputs, and a few extend to small-baseline rectified stereo pairs; UGD-FM targets severe noise together with large-range two-dimensional misalignment, outperforming the compared single-image and guided denoising baselines on FANVID, RGB-NIR Scene and MS2 with only one or two inference steps.
Interpretation
The input condition for guided denoising is relaxed: the noisy RGB image and the near-infrared guidance may come from different cameras with large-range two-dimensional spatial misalignment, without prior alignment or restriction to horizontally rectified small-baseline stereo pairs. Most prior guided denoising methods assume the noisy RGB and guidance images are spatially aligned, and recent work that extends to cross-modal stereo pairs still handles only horizontally rectified pairs with small baselines. Across FANVID, RGB-NIR Scene and MS2 under three noise settings, UGD-FM outperforms five single-image denoisers and five guided denoising baselines on NAFNet, Uformer and Restormer backbones; in the ablation that increases misalignment from aligned guidance upward, NAID performs well only with aligned or mild misalignment while UGD-FM keeps strong performance as misalignment grows.
Denoising is organized as a progressive trajectory: the cleaner the image becomes, the easier cross-modal correspondence is to establish, and the aggregated guidance in turn improves the next velocity prediction. Direct-regression methods must estimate reliable cross-modal correspondence in one pass from a highly corrupted target state, which is less effective under severe noise and misalignment. On FANVID under Poisson-Gaussian noise with the Restormer backbone, standard flow matching reaches 30.28 PSNR at one step, comparable to or slightly better than direct regression at 30.27, and standard flow matching's two-step inference raises PSNR to 30.34.
Few-step inference comes from aligning training with inference states: training timesteps concentrate on two critical stages, and the interpolation state is replaced by the state the solver actually reaches. Standard flow matching adds computation as solver steps increase and may accumulate prediction errors along iterative updates. In the FANVID ablation, cumulatively adding stage-focused sampling and inference-consistent state construction raises two-step PSNR to 30.42 and 30.58 respectively; standard flow matching at 8 steps instead drops in PSNR and SSIM.
Large-range matching keeps only a small set of high-confidence candidates and refines them across scales, avoiding dense search at every decoding scale. Dense large-range matching is computationally expensive at high resolution, and global attention runs out of memory in this setting. In the FANVID ablation, one-step PSNR is 28.97 without attention, 29.64 with window attention and 30.29 with sparse propagation; global attention is reported to run out of memory.
Perspective
Evaluation covers FANVID, RGB-NIR Scene and MS2 plus the real dual-camera RNS dataset; noise settings include Gaussian noise and two Poisson-Gaussian settings, and the degraded observations used in training are synthesized by randomly adding Poisson-Gaussian noise to clean RGB images; misalignment on FANVID and RGB-NIR Scene is constructed from virtual viewpoints rendered by a 3D Gaussian representation, while MS2 and RNS are real multi-camera captures; inference time and peak memory are measured on a single NVIDIA RTX 4090.
Under severe 3D parallax, occlusions and incomplete cross-view observations can make local guidance ambiguous or unreliable, and joint restoration and cross-modal alignment add computational overhead. The authors also note that restored images may contain plausible but inaccurate details, so safety-critical deployment requires careful validation and uncertainty assessment.
