AnyVIO fuses visual and inertial foundation models in one forward pass: 36% lower position error than its visual-only model, poses 70 ms after each image
Synopsis
AnyVIO combines a streaming 3D geometric foundation model with an inertial foundation model into a visual-inertial odometry system that runs in a single forward pass, without a filter or factor-graph optimization at inference. A lightweight learned fusion head estimates the reliability of the two motion estimates and blends them, cutting the metric position error of its visual model by 36% across nine datasets recorded from cars, legged and ground robots, and handheld and head-mounted devices, while publishing each pose 70 ms after an image arrives. The model was also deployed on two real vehicles at up to 140 km/h, where it stayed accurate in poor lighting and heavy dust.
Interpretation
Monocular visual-inertial odometry can use images and inertial data together in one forward pass and output poses with metric scale, with no filter, factor graph or other optimization at inference. Geometric foundation models either lacked metric scale or inferred it from learned priors on appearance, while earlier ways of adding inertial data used a filter or factor graph that reintroduced expert tuning and costs growing with the trajectory. Across nine datasets recorded from cars, legged and ground robots, and handheld and head-mounted devices, one model reduces the metric position error of its visual-only version by 36% and publishes each pose 70 ms after an image is received.
The visual and inertial pathways each predict the metric motion between consecutive frames, and a learned fusion head judges the reliability of both estimates and blends them. On the visual side, a non-metric streaming geometric foundation model is adapted to incremental inference with bounded memory and given a scale head that predicts metric scale; on the inertial side, an inertial foundation model with a single shared velocity head outputs body velocity, which is integrated with gyroscope readings into a relative pose. The fusion head is trained with both models frozen, sampling the same number of sequences from each dataset so training is even across platforms; on a KITTI-360 drive, throughput stays constant while the factor-graph FoundationVIO falls from 8.1 to 0.9 frames per second.
Real-vehicle deployments covered scenes outside the training distribution, and inertial data helped in most conditions. On the tarmac track in nominal light, adding the IMU lowers translational relative error from 6.57% to 4.35% and SE(3) absolute position error from 29.98 m to 17.03 m; in heavy dust offroad, AnyVIO reaches an SE(3) absolute position error of 6.72 m against 10.53 m without the IMU. Two vehicles ran inference on onboard 4070 Ti S and L4 GPUs on the tarmac and offroad tracks of RaceDataset, with numbers computed in real time; driving the track in reverse, the IMU helps less, and AnyVIO's SE(3) absolute position error rises to 26.71 m, close to the visual-only 27.52 m.
Perspective
The demonstrated setting is streaming inference from a monocular camera plus IMU with no loop closure or global refinement, covering nine driving, indoor and walking datasets and real-time runs on two vehicles on tarmac and offroad tracks; the material states the model will be released open-source upon acceptance.
The material notes that on KITTI Odometry, which is not used for training, a large gap remains between SE(3) and Sim(3) absolute position error, so metric scale transfers only partly to unseen data; on OpenLORIS, also unseen, HorizonStream has a lower SE(3) absolute position error. It also states that the inertial model's cost depends on the IMU stream and can lower the frame rate, and that the system requires camera-IMU extrinsic calibration.
