Skip to main content
Back to timeline
arXivSource publication:

Each Gaussian primitive now decides how much to change: GaussLearner beats the strongest baseline by 3.41 dB average PSNR across 14 scenes

Synopsis

When a 3D scene is reconstructed continually, fine-tuning an existing Gaussian field on new observations forgets previously reconstructed content. GaussLearner treats each 3D Gaussian primitive itself as a continual learner: it attributes pixel-wise inconsistencies in current observations to specific Gaussians, then uses a learning weight that trades current observations against cross-state reliability to govern both parameter adaptation and optimizer-memory refresh. Across 14 scenes from CL-Splats, WAT and the self-captured CL-3D, it improves average PSNR by 3.41 dB over the strongest baseline, O-SCD, and on CL-3D it gives the best reconstruction quality on both changed and unchanged views.

Source-provided article image: GaussLearner: 3D Gaussian Primitives as Self-Regulating Continual Learners
arXiv

Interpretation

In continual reconstruction of 3D Gaussian scenes, each Gaussian primitive can decide for itself how much to change at the current state, instead of a single update policy acting on the whole field. Earlier approaches either detect changed regions in 2D images and restrict optimization to selected Gaussians, or rely on replay and parameter regularization to retain old content. Across 14 scenes from the CL-Splats, WAT and CL-3D datasets, GaussLearner improves average PSNR by 3.41 dB over the strongest baseline, O-SCD.

Each Gaussian first receives the observation pressure attributed to it, then uses its own historical reliability to decide how much parameter change to accept and how much optimizer memory to refresh. The learning weight combines current observation coverage with cross-state reliability, in the same form as a scalar Kalman gain written in precisions. On the six real-world CL-3D scenes, test views are split into changed and unchanged subsets, and GaussLearner achieves the best reconstruction quality on both.

Pixel-wise inconsistencies are distributed to Gaussians through the renderer's own alpha-composition weights rather than compressed into a binary change mask. The selective local optimization route reduces continuous observations to hard change decisions, so errors in 2D change detection propagate directly into the Gaussian field. In ablations, replacing direction-aware pooling with a view-wise sum lowers PSNR on both changed and unchanged views.

Perspective

Validation covers four scenes each from CL-Splats and WAT plus six self-captured real-world CL-3D scenes, 14 scenes in total, with novel-view PSNR and SSIM as metrics; CL-3D test views are explicitly split into changed and unchanged subsets, and all experiments run on a single RTX 4090 GPU.

The method remains sensitive to challenging illumination such as specular reflections and refraction, and it maintains only the latest scene state, so it does not support arbitrary replay of previous states.