Assembly no longer slips past trackers: Atlas drops the first-frame prompt and finds more than twice as many state changes on OmniST
Synopsis
Existing object tracking and state-change detection need a target mask in the first frame, so they only catch an object breaking apart; when several objects merge, the track silently ends or runs through the merged body. Atlas treats assembly as decomposition played backward, builds forward and reverse spatiotemporal partitions, and jointly judges with spatial relationships whether a newly emerging track is a decomposition product or an assembly product, after which a vision-language model confirms the event, corrects its timing and writes a description. On OmniST, a new benchmark annotating both decompositions and assemblies, Atlas recovers more than twice as many state changes as prior methods, with overall recall 36.1 against 23.2 for TubeletGraph-Bidir plus the same verification stage.
Interpretation
The tracking task moves from tracking one object given its first-frame mask to recovering all object tracks and marking every decomposition and assembly from a video alone. Earlier methods such as TubeletGraph take a first-frame mask and detect only decompositions of the prompted object; assembly events are neither annotated nor detected, and later decompositions of earlier fragments are missed as well. On OmniST, Atlas reaches overall recall 36.1 and overall precision 48.2, against 16.2 and 12.2 for TubeletGraph-Bidir; giving TubeletGraph-Bidir the same vision-language verification and realignment stage raises its overall recall only to 23.2.
The approach treats assembly as decomposition run backward, builds one spatiotemporal partition in each direction, and jointly decides which kind of event a newly emerging track came from. A spatiotemporal partition chains per-frame segmented entities into track fragments over time; forward tracking produces no new fragment for an assembly, while reverse tracking makes an assembly surface a new fragment just as a decomposition does. Candidates go to GPT-4.1 with frames sampled across the whole video and crops around the event, sampled asymmetrically because decompositions are predicted late and assemblies early; across the 97 OmniST videos, a paired bootstrap with 10,000 resamples confirms the gains over each baseline are statistically significant.
The new OmniST benchmark annotates decompositions and assemblies together, letting these systems be compared on both event types in one dataset for the first time. The earlier VOST-TAS contains only decomposition annotations, and assembly events are not annotated in the data at all. OmniST holds 97 videos, 352 objects and 302 transformations, of which 144 are assemblies and 158 decompositions; on the decomposition-only VOST, VSCOS and M3-VOS benchmarks, Atlas keeps tracking performance comparable to prompt-based methods.
Perspective
The demonstrated scope is transformations that split or merge objects: Atlas declares an event only when a new track fragment emerges in the forward or reverse direction, and evaluation is confined to the authors' OmniST plus the decomposition-only VOST, VSCOS, M3-VOS and VOST-TAS benchmarks.
Gradual in-place state changes that neither split nor merge objects, such as melting or deformation, produce no new fragment and are not detected; the method also relies on the tracker reliably failing on a transformation product so that a new fragment appears. Building both spatiotemporal partitions is the main cost, and the two partitions are independent and can be computed in parallel.
