Skip to main content
Back to timeline
arXivSource publication:

Iris-Z folds depth into image generation: one set of weights reaches mean AbsRel 2.92 on Marigold's benchmarks, below MoGe-3's 3.12

Synopsis

Iris-Z is a general image generator that encodes image and depth into one token sequence and denoises them together, so a single set of weights generates images, estimates depth from photographs and paints images onto a given depth. Monocular geometry accuracy had belonged to task-specific ViTs such as MoGe-3, and image generators repurposed as depth estimators, the Marigold family, stayed behind it. Iris-Z-big reaches a mean AbsRel of 2.92 on Marigold's benchmark sets against 3.12 for MoGe-3 while keeping its generative abilities, at about 15 seconds per image versus about 0.41 seconds for MoGe-3.

Source-provided article image: Iris-Z: Scaling One Generative Model Past Specialist Geometry Models
arXiv

Interpretation

A general image generator can now beat a specialist geometry model on monocular depth benchmarks without a depth head or a second network. The leader on this path had been the task-specific ViT MoGe-3, and the Marigold family, which turns image generators into depth estimators, still performed worse than MoGe-3 and MoGe-2 on MoGe-3's held-out test sets. On Marigold's benchmark sets Iris-Z-big averages 2.92 AbsRel against 3.12 for MoGe-3 and 4.10 for Marigold V2; on MoGe-3's own sets Iris-Z-big averages 3.56 against 3.70 for MoGe-3.

Image and depth are denoised together as two groups of tokens in one sequence, so each conditions the other at every denoising step. The recipe full fine-tunes a pretrained text-to-image diffusion transformer, and the only new parameters are a learned modality embedding, with the autoencoder and text encoder frozen. Depth is stored as per-image normalized log ray distance, encoded as an image by the same autoencoder; training mixes joint, image-given, depth-given and image-only tasks plus a loss comparing decoded depth in pixel space, on about 3.6 to 3.8 million RGB-D frames per model.

Perspective

Evaluation covers Marigold's NYUv2, KITTI, ETH3D, ScanNet and DIODE and MoGe-3's NYUv2, KITTI, ETH3D, iBims-1, Sintel, DIODE and HAMMER, with the high-resolution Spring and Synth4K sets left out because they exceed the training resolution. Depth is relative: the prediction is defined up to a scale and shift in log space, giving neither metric distances nor the camera.

Depth stays relative, giving neither metric distances nor the camera, while MoGe-3 predicts both. Generated images drift toward the look of the training data: Iris-Z-big's human preference score falls from 26.71 to 24.30 and CLIP from 27.85 to 27.42, while photorealism rises. Evaluation handles one image at a time, and consistency across video frames or views is not measured.