Cross-view attention stops growing with views: TTB bakes it into a fixed-size MLP and holds 23 FPS under dense views
Synopsis
Decoder-only view synthesis models cache context views in a KV cache that grows linearly with view count, making rendering impractical under dense views; TTB instead bakes cross-view attention outputs into a fixed-size fast-weight MLP during reconstruction and queries only that state at render time. On DL3DV at 16, 32 and 64 views and 960p, TTB renders at a constant 23 FPS with quality close to DVSM and clearly above LaCT at the same constant complexity.
Interpretation
Rendering a novel view no longer requires carrying a KV cache that grows with the number of context views; it carries a fixed-size set of MLP weights instead. DVSM prefills context views into a KV cache that each novel view queries with linear-complexity attention, which is impractical under dense views; LaCT replaces cross-view attention with test-time training layers for constant complexity but loses noticeable quality. On DL3DV at 16, 32 and 64 input views and 960p, TTB holds a constant 23 FPS with quality close to DVSM and above LaCT, while DVSM's rendering frame rate falls as views increase.
The inner loop of test-time training is retargeted to fit the output of cross-view attention rather than to replace attention. LaCT's fast-weight MLP learns to map queries directly to raw value tokens; TTB changes the learning target to the tokens produced by cross-view attention, so the MLP collaborates with attention instead of substituting for it. An ablation splits the gap from LaCT-style TTT to full TTB into context pairing, switching to an MSE inner-loop loss, and replacing the linear-projection target with cross-view attention output; stacking all three narrows the quality gap to DVSM while keeping TTT-level rendering frame rate.
Each context view is split into a token pair so that the baking input and the rendering input share one modality. In the conventional pipeline, context tokens used for prefilling carry image content while novel-view tokens at rendering carry camera information only, a modality misalignment that adds an unnecessary generalization burden to the baked MLP. Each context view yields a camera-only rendering token as TTB input and a ground-truth token with image content and cross-view attended information as the mapping target; in the ablation this pairing alone improves quality.
Perspective
The demonstrated scope is novel-view reconstruction on DL3DV dense views, Re10K two-view, and the out-of-domain MipNeRF360, Free and Hike datasets; the DL3DV and out-of-domain models are initialized from a DVSM checkpoint and trained on DL3DV, while the Re10K two-view model is finetuned on Re10K from DVSM's Re10K checkpoint. The method interpolates among context views and does not extrapolate to unobserved viewpoints.
The fast-weight MLP has fixed capacity, so quality saturates as the number of context views grows; the local grouping route that mitigates this is an inference-time heuristic using k-means over camera poses, and learning the grouping during training is left for future work. Cross-view attention inside the inner loop makes reconstruction time quadratic in the number of context views, a drawback for single-shot scenarios.
