DeepBA learns bundle adjustment as a graph network: 2.04M parameters solve camera poses from a cold start, and outpace COLMAP as views grow
Synopsis
DeepBA replaces the per-scene bundle-adjustment solve in multi-view reconstruction with a learned graph-network refiner that consumes only the 2D matches of an off-the-shelf matcher and recovers absolute camera poses, intrinsics and scene structure from a cold start. Trained with scale-invariant supervision on datasets with incompatible metric units, its 2.04M parameters lead prior learned SfM from matches on CO3Dv2 and RealEstate10K, are competitive with classical SfM at moderate accuracy thresholds, and overtake it in wall time as the view count grows.
Interpretation
The solver stage of multi-view reconstruction gains a learned substitute: instead of solving a non-linear least-squares problem per scene, DeepBA repeatedly refines its estimate with a weight-shared graph neural network over a view graph of cameras and matches. Classical bundle adjustment is near-optimal given a good initialization, but incremental SfM must obtain that initialization through heuristics and per-scene tuning, and those heuristics break down on weak baselines and planar structure. On CO3Dv2 and RealEstate10K, DeepBA reaches RRA@15° 97.4, RTA@15° 89.2, AUC@30° 78.6 and 84.0, ahead of GASfM on the same eLoFTR matches (AUC@30° 71.2 and 82.8), and beats COLMAP and GLOMAP on every metric of both benchmarks.
Each iteration recomputes geometric features from the current state, and prediction heads emit a state delta that is composed back onto the state, so the number of test-time iterations is a free choice independent of the training schedule. Earlier learned SfM mostly regresses a reconstruction with a network and then still runs a classical bundle adjustment, and some methods fine-tune the network on each test scene. The network reads only 2D matches, so one trained set of weights runs on five front-ends, SIFT, SuperPointLightGlue, ALIKEDLightGlue, eLoFTR and DKM, without retraining, evaluated on four multi-view benchmarks.
The supervision is built to be metric-scale-free: rotation uses a chordal distance and translation compares directions only, and reprojection and 3D self-consistency losses complete the objective, so datasets with incompatible metric units can be mixed. Absolute poses receive no direct ground-truth supervision; they are pulled through consensus with the pairwise relative poses, a learned analogue of pose-graph optimization, so they follow the gradual refinement of the relative poses. Training mixes MegaDepth, CO3Dv2, ScanNet, BlendedMVS, DL3DV and Calle without per-scene normalization; on the DTU structure benchmark, DeepBA is the only learned SfM from matches to produce a non-empty reconstruction on all scans, succeeding on 15/15.
Perspective
Evaluation covers six benchmarks, CO3Dv2, RealEstate10K, ETH3D, DTU, Mip-NeRF 360 and IMC2021, five of which are absent from the training mixture and therefore measure cross-dataset generalization; the camera model is limited to pinhole, and image collections are treated as unordered with unknown intrinsics.
The authors list four remaining limits: the focal–depth ambiguity on planar scenes, drift of the learned update on some scenes, accuracy at tight thresholds, and the pinhole camera model; on the bonsai scene, DeepBA keeps tilting the cameras even when started from ground truth.
