Skip to main content
Back to timeline
arXivSource publication:

No calibration needed for ceiling cameras: ROMUX matches a robot-mounted camera while its viewpoint is randomized every episode

Synopsis

Indoor service robots that navigate with wall-mounted monitoring cameras have had to calibrate each camera's pose and intrinsics first, and redo it whenever a camera moves or the robot enters a new room. ROMUX replaces that step with supervised learning: a cross-view transformer trained on large-scale simulation predicts a metric, robot-centric obstacle grid directly from exocentric images, and a small PPO policy drives toward a goal coordinate on that grid. With camera pose and intrinsics randomized every simulated episode, one-camera success matches a robot-mounted-camera baseline, two cameras do better, and the agent transfers zero-shot to a real robot.

Source-provided article image: Navigating Robots from Unknown and Uncalibrated Exocentric Cameras via Map Estimation
arXiv

Interpretation

A robot can navigate to a point goal and avoid obstacles using only exocentric camera images whose pose and intrinsics are unknown, with no prior knowledge of where each camera sits, where it looks, or its focal length. Earlier methods of this kind either recovered camera pose and intrinsics through classical calibration or fitted perception models to each specific camera view before deployment; moving a camera or entering a new room meant repeating that work. With camera pose and intrinsics re-randomized every simulated episode, one randomized exocentric camera gives ROMUX a success rate matching a PPO policy navigating from a robot-mounted camera, while an end-to-end PPO policy on a fixed exocentric camera stays within the interval of the no-image baseline.

The approach separates the perspective transformation from the robot frame out of the reinforcement-learning objective and treats it as a supervised learning task. End-to-end PPO did not implicitly learn that transformation in this setup, so it is learned explicitly instead. A fine-tuned DINOv2 encodes patch tokens from each view; the camera embeddings the original cross-view transformer conditions on are removed, and learnable map queries attend across views through a six-layer decoder to output a binary obstacle grid centered on the robot and aligned with its heading, on which the policy is trained together with a five-dimensional state vector.

The map-estimation module and the policy are trained separately without shared gradients, so a new domain can retrain the map module while reusing the policy. When cardboard obstacles were replaced by photorealistic furniture, the map module was retrained on 180,000 snapshots while the policy stayed the one trained in cardboard rooms. On furniture layouts seen during map training, success is close to the cardboard result in the same two-camera configuration; on 13 held-out furniture assets success drops and collisions rise, yet roughly three of four episodes still succeed.

Perspective

The demonstrated scope is simulated cardboard rooms, photorealistic furniture rooms and a four-camera large room, plus a zero-shot transfer trial in one real room with black walls; the real trial used one exocentric camera at five positions and two heights across a set of trials. The authors state the architecture is not a deployment-ready system and that the perception module was trained entirely in simulation.

The system assumes the robot is visible in at least one exocentric image; when it is fully occluded the model cannot locate it and the predicted map becomes useless, and it also assumes the cameras cover the whole navigable area. Real-trial failures related more to obstacle layout than camera placement, including stopping before narrow gaps and turning in place until the step cap, occlusion of the robot by a tall box, and maps that misplace obstacle distances and sizes and lead to collisions.