Skip to main content
Back to timeline
arXivSource publication:

TOMAS guesses the operation before the path: grasp, push or pull on one object, and the lowest error on all six 3D hand-forecasting metrics

Synopsis

In egocentric video, one object can be grasped, pushed or pulled, so past hand motion alone does not pin down where the hand goes next. TOMAS combines finger-joint topology, object semantics and spatial relations to infer candidate interaction targets and latent operations, then splits the future into ordered stages with learnable temporal boundaries and refines the trajectory with stage-specific geometric subgoals. On EgoPAT3D-DT, H2O-PT and EGTEA Gaze+, TOMAS reaches the lowest error on all six 3D metrics, tying Uni-Hand on H2O-PT FDE, and on five of six 2D metrics.

Source-provided article image: TOMAS: Topology-Aware Operation Modeling and Adaptive Staging for Egocentric Hand Forecasting
arXiv

Interpretation

Hand-trajectory forecasting no longer rests on a single global motion representation; it first forms a hypothesis about which object and which operation, then generates the future hand path from it. Earlier work advanced object-centered interaction modeling, motion-aware temporal dynamics and general hand-motion representations, but the actor's intended manipulation stayed largely implicit in the forecasting process. On the seen and unseen EgoPAT3D-DT splits and H2O-PT, TOMAS has the lowest error on all six 3D metrics, with H2O-PT FDE at 0.050, tied with Uni-Hand's 0.050; it is lowest on five of six 2D metrics, while on H2O-PT its 2D FDE of 0.069 is above Uni-Hand's 0.065.

The model encodes skeletal connections and articulation changes of the finger joints together with candidate-object appearance and position and hand-object relative geometry, and each candidate object retrieves its own relevant slice of history. Candidates no longer share one scene summary; each queries the history relevant to its own interaction context, and a learnable null option represents the absence of a suitable observed candidate. On the H2O-PT validation set, the full 2D and 3D models reach 72.82% and 77.61% Top-1 target-ranking accuracy; removing articulated-joint input lowers these by 9.14 and 2.98 percentage points.

Future motion is organized into ordered latent stages whose durations the model learns, and each stage supplies a geometric subgoal relative to the candidate object that refines the trajectory. Stage order comes from a cumulative-duration parameterization rather than independent temporal attention weights, so the sequence of transitions is explicitly preserved. Across the seven evaluation settings on EgoPAT3D-DT and H2O-PT, removing global interaction conditioning, removing stage-level operation conditioning, replacing ordered boundaries with unconstrained softmax temporal attention, or removing geometric subgoals each raises FDE above the full model.

Perspective

Evaluation covers the seen and unseen EgoPAT3D-DT splits, H2O-PT and EGTEA Gaze+, with observation and future taking 60% and 40% of each forecasting window and main trajectory windows up to 40 frames; EGTEA Gaze+ uses a short-horizon protocol of 1.5 s observation and 0.5 s prediction. The target-ranking diagnostic runs only on the H2O-PT validation set with seed-0 checkpoints selected by validation mADE.

Ablations A1–A9 each use a single seed while the full model averages three seeds, so those comparisons indicate performance trends associated with individual components rather than seed-matched estimates. Targets on the H2O-PT validation set are derived by geometric association from validation action annotations; the authors state these scene-action targets are not independently verified hand-specific interaction annotations, and 100% association coverage does not establish manually verified detector recall.