Gaussian splatting learns to spend its budget: BudgetSplat sets the keep ratio at test time and holds rendering quality at 5%
Synopsis
Feed-forward 3D Gaussian splatting predicts Gaussians per pixel from input images, so the representation grows with image resolution and view count and repeats across overlapping views. BudgetSplat treats compaction as a joint problem of importance, spatial coverage and cross-view redundancy, and one trained model adjusts the Gaussian count to a user-specified preservation ratio at test time without per-budget retraining or per-scene post-processing. On ScanNet and Replica it gives a stable quality-size trade-off across view counts and ratios, holding rendering quality even at a 5% budget.
Interpretation
A single feed-forward model can now decide at inference how many Gaussians to keep according to a user-specified preservation ratio, without retraining for each budget or compressing each scene separately. Fixed-capacity methods tie representation size to a particular scene-token or decoder configuration, so larger or more complex scenes may require retraining a higher-capacity model; post-hoc compression methods apply an extra per-scene processing or optimization step to an already reconstructed Gaussian set. On ScanNet the same trained model is evaluated with 2, 3 and 10 context views and supports arbitrary preservation ratios; at the 25% budget it matches EcoSplat's Gaussian count while improving all rendering metrics.
Compaction is no longer just an opacity ranking truncation; it weighs which Gaussians matter, whether the kept set spreads over space, and whether different views observe the same 3D region repeatedly. Applying a budget-unaware Top-K rule to a conventionally trained pixel-aligned model over-allocates Gaussians to salient or high-opacity regions and leaves weakly textured areas under-covered. In a ScanNet ablation at a fixed 25% budget keeping 147k Gaussians, a plain view-wise opacity Top-K baseline reaches PSNR 13.02, intra-frame structure-and-coverage supervision raises it to 24.80, and the full combination reaches 26.77.
Budget information is injected into the features used for Gaussian prediction, so opacity and Gaussian attributes change with the requested preservation ratio instead of the same predictions being truncated differently. Changing only the selection budget applies different truncation levels to budget-independent Gaussian predictions, so the predictions themselves do not adapt to the budget. The ratio is spatially broadcast and embedded by a lightweight convolutional network into a budget feature added to the base feature before opacity, covariance and color are predicted; the second training stage samples the ratio randomly and shifts toward lower ratios, optimizing with top-k rendering.
Selection first pre-selects within patches of each view, then ranks the whole view's candidate pool by opacity, and finally removes geometric redundancy with 3D non-maximum suppression. Ranking only within individual patches concentrates retained Gaussians in salient regions and leaves holes in smooth or weakly textured areas. In the ablation, adding coverage-aware budget selection on top of intra-frame supervision and alpha completion raises PSNR from 26.14 to 26.75, an improvement the source describes as larger than that of adding cross-frame redundancy weighting.
Perspective
The demonstrated scope is indoor reconstruction on ScanNet and zero-shot cross-dataset generalization on Replica, covering 2, 3 and 10 context views and several test-time preservation ratios; the 10-view comparison is run at a 25% budget because several pixel-aligned baselines run out of memory at 100% on an RTX 4090.
The method relies on image gradients and estimated geometry for pseudo supervision, so degraded image quality or depth errors may affect selection under very compact budgets; it reduces the final representation and rendering cost while retaining the dense pixel-aligned prediction backbone.
