End-to-end tracking changes how detection queries are born: TAQ-MOTR reaches 72.0 HOTA on the DanceTrack test set
Synopsis
In end-to-end multi-object tracking, detection queries have been formed before the decoder with little knowledge of existing track states, so occluded tracked objects lack complementary current-frame evidence. TAQ-MOTR forms them before decoding with a Hybrid Query Generator that fuses current-frame image features, track memory and learnable memory, plus a training-only auxiliary detection branch that adds detection supervision. On the DanceTrack test set it reaches 72.0 HOTA, 63.1 AssA and 76.6 IDF1, 3.5, 4.7 and 5.4 points above MeMOTR.
Interpretation
Detection queries now carry the state of existing tracks before they enter the decoder, a change at the query-formation stage of end-to-end tracking. Earlier practice inherited single-frame detection query initialization: MOTR used learnable content and positional embeddings, MOTRv2 used boxes and confidence scores from an external detector, and neither conditioned on trajectory states, so separating tracked from currently untracked objects waited for interaction with track queries inside the decoder. On the DanceTrack test set TAQ-MOTR reaches 72.0 HOTA, 82.4 DetA, 63.1 AssA, 91.7 MOTA and 76.6 IDF1, gains of 3.5, 1.9, 4.7, 1.8 and 5.4 points over MeMOTR; against MOTRv2 it gains 2.1 HOTA, 4.1 AssA and 4.9 IDF1 with slightly lower DetA and MOTA.
The mechanism builds a set of detection queries that already know which objects are being tracked, before decoding. The Hybrid Query Generator uses image-based detection proposals projected from the lowest-resolution feature map as queries and track memory plus learnable memory as attention keys and values; track memory is converted from updated propagated track queries and carries content, reference-box and disappearance-time priors, while learnable memory is a shared set of trainable tokens holding trajectory-independent detection priors. Scored and Top-K selected outputs are concatenated with updated track queries and decoded by a six-layer DAB-Deformable DETR decoder; on the DanceTrack validation set the full configuration with detection proposals, track memory and learnable memory reaches 65.3 HOTA, 76.5 DetA, 56.0 AssA and 70.3 IDF1, versus 64.4 HOTA, 77.1 DetA, 53.9 AssA and 69.1 IDF1 with proposals and learnable memory alone.
A training-only detection branch gives detection queries enough supervision even though track queries claim most labels. Under identity-preserving one-to-one assignment, ground-truth objects held by existing track queries are reserved for those queries, so main detection queries match only currently untracked objects and detection-side supervision is sparse. The auxiliary branch reuses the same detection proposals and generator and decoder weights but drops track memory and is isolated from main queries by attention masks, matching its outputs independently to all current-frame ground-truth objects and being removed at inference; on the DanceTrack validation set the default staged auxiliary schedule improves HOTA, DetA, AssA and IDF1 by 1.1, 1.7, 0.8 and 1.0 points over training without the branch.
Perspective
The demonstrated scope is online tracking on three multi-object tracking benchmarks, DanceTrack, SportsMOT and BFT, without offline trajectory linking, smoothing or interpolation; main results use only each benchmark's official training split and no additional detector, and the SportsMOT variant trained with extra CrowdHuman data is excluded from the main ranking.
The Hybrid Query Generator does not explicitly estimate or gate prior reliability, so under dense overlap and similar appearance image proposals may be ambiguous and propagated tracks may drift; ablations show that including recently lost tracks in memory raises DetA but lowers AssA and IDF1. Compact stride-32 proposals may limit sensitivity to small targets, local overlaps and abrupt shape changes, and the auxiliary supervision schedule is manually calibrated, with its transition points shifting the detection-association balance.
