Camera pose without pairing points: MomentPnP matches fourteen moments to localize shuffled point sets
Synopsis
Without known correspondences between 2D points and 3D model points, MomentPnP estimates the camera pose first by aligning moment-feature averages of the observations with those of the projected model, using fourteen moments to identify a perspective pose on complete noiseless inputs and expectation–maximization to refine pose and correspondence weights. Across three blind PnP benchmark test sets its median rotation error is 5 to 170 times smaller than the best learned baseline, and 82.3% to 97.0% of frames fall within 5 degrees when outliers equal inliers.
Interpretation
The camera pose can be estimated before any point is paired: MomentPnP compares statistical features of the whole 2D observation set with those of the whole projected 3D model, obtains a pose, and only then assigns each observation to its nearest projection. Local methods need an initial pose or pose prior, global methods search pose space by branch and bound, and learned methods predict correspondences from training data; all of these routes rely on correspondences, a prior or a search. On the MegaDepth, ModelNet40 and Data61 benchmark test sets, MomentPnP's median rotation error is 5 to 170 times smaller than the best learned baseline and its median translation error 9 to 320 times smaller; with as many outliers as inliers its accuracy is 82.3% to 97.0%.
The method turns pose estimation into aligning two sets of feature averages, with no point pairs involved. Moments are statistical averages of products of point coordinates, and fourteen moments through order four determine the pose on complete noiseless inputs; with outliers the method switches to bounded Fourier features, so one distant point has bounded influence. On 100 frames from each benchmark, starting from the same initial poses, moments through order three or four bring 53 to 78 poses within 5 degrees and 1% translation error before expectation–maximization, whereas the mean and covariance bring at most two.
After moment matching supplies starting poses, expectation–maximization refines the pose with each observation individually while estimating noise and the outlier fraction. The refinement adapts EM-ICP and coherent point drift to a perspective mixture in which each observation is either a uniform outlier on a rectangle or a noisy projection of a model point. On rendered ETH3D laser scans the median rotation error falls as points grow from 1,000 to 256,000, and is 0.066 degrees for the Facade view at one million points; at that size 21% to 25% of 2D points lie nearer another projection, so nearest-projection labeling can no longer resolve them individually.
Perspective
The demonstrated scope covers the test sets of three blind PnP benchmarks, tracked-point inputs from four further datasets, rendered ETH3D laser scans up to one million points, and single-image localization against complete ETH3D scene reconstructions, where all 39 tuning queries and 35 of 39 test queries are recovered within 5 degrees.
When 2D points have no corresponding model point, the feature averages need not match even at the true pose, so independently selected inputs remain harder than tracked ones, and adversarial outliers can prevent recovery. For nearly symmetric or shallow models, the seven alternative poses that the mean and covariance cannot separate are compared by expectation–maximization, and which of them is preferred is not proved.
