Skip to main content
Back to timeline
arXivSource publication:

Video analytics stops decoding every frame: CoDecide decodes only where new content appears, reaching 15 to 24 times the throughput of a per-frame pipeline

Synopsis

In surveillance archive analytics, once the detector runs on only a subset of frames, decoding replaces detection as the main cost. CoDecide turns selective decoding into novelty prediction in the compressed domain: a frozen DINOv2 teacher defines semantic novelty without labels, and a lightweight student reads motion vectors, non-zero coefficient counts and macroblock types to decide when to decode and which macroblocks to reconstruct, with detections propagated to skipped frames along motion vectors. Under a leave-one-dataset-out protocol on VIRAT, MEVA and ShanghaiTech, it recovers more moving-object detections than prior compressed-domain selectors at an equal or smaller decoding budget, and runs 15 to 24 times faster end to end than a pipeline that decodes and detects every frame.

Source-provided article image: CoDecide: Content-Adaptive Video Decoding for High-Throughput Video Analytics
Figure 2 · arXiv · Page 3

Interpretation

Video archive analytics can decode only the frames where new content appears, propagating detections to the rest along motion vectors, so decoding becomes something actively cut rather than a minor cost. Most prior compressed-domain selectors still fully decode RGB frames before filtering, and uniform sampling inside a long GOP still has to walk the dependency chain almost to its end, saving little decoding. Under a leave-one-dataset-out protocol on VIRAT, MEVA and ShanghaiTech, against CoVA with the same YOLO11x detector and the same box propagation, CoDecide's moving-object recall is 0.78 versus 0.75 on VIRAT, 0.60 versus 0.54 on ShanghaiTech and 0.78 versus 0.76 on MEVA, at an equal or smaller decoding budget.

Deciding when to decode rests on semantic novelty in the compressed domain rather than on the amount of motion. The encoder emits motion vectors for any displacement, so a known object that keeps moving produces continuous vectors and residuals, and motion alone cannot separate an arrival from something that was always there. The student is supervised by novelty maps from a frozen DINOv2 teacher built without labels, taking motion vectors, non-zero coefficient counts, macroblock types and a surprise prior derived from them as input, and under leave-one-dataset-out it needs no retraining for a new camera.

Where to decode is set by a presence map, so only macroblocks whose content changed are reconstructed and the rest are filled from co-located blocks of the reference frame. Full per-frame reconstruction also decodes large static background areas whose pixels did not change between adjacent frames. Spatial selection reconstructs 13% of macroblocks on MEVA, 38% on VIRAT and 81% on ShanghaiTech, the composited frames stay close to full reconstruction at PSNR 41.96 and SSIM 0.985, and each novel GOP saves a further 6% to 10% of decode time.

Perspective

The demonstrated scope is H.264/AVC-compressed surveillance archive video with a GOP length of 250 frames, evaluated on VIRAT, MEVA and ShanghaiTech for moving-object detection, a binary predicate and counting; all methods share the YOLO11x detector and the same motion-vector box propagation, so the comparison is between frame selection policies.

The gap to dense detection is dominated by propagation drift: on VIRAT roughly two-thirds of the missed moving boxes were properly seeded initially but drifted below IoU 0.5 as quantized macroblock motion vectors accumulated error over a GOP, and one-third were never seeded. Two failure modes remain: an arriving object too small to cover enough patches for the teacher to mark it, and a novel object too small for the downstream detector.