Spiking nets can track human pose in video: SpikePose cuts estimated arithmetic energy 91.4% for 1.12 points of accuracy
Synopsis
Spiking neural networks save energy but struggle with ordinary video that arrives frame by frame. SpikePose gives per-frame spatial encoding to a spiking encoder that resets each frame and runs a single inference step, and gives cross-frame reasoning to a lightweight heatmap memory module outside the backbone that estimates joint displacement, aligns historical heatmaps and fuses them adaptively. On a fixed NTU60 test subset, against an architecture-matched artificial neural network with the same memory module, SpikePose lowers estimated arithmetic energy by 91.4% at a cost of 1.12 points in head-bone-normalised PCK at 0.5 threshold.
Interpretation
SpikePose splits video human pose estimation into two jobs: a spiking encoder supplies spatial evidence for each frame, and a lightweight heatmap memory module outside the backbone handles motion between frames. Earlier spiking networks were mostly applied to static images, unrolling membrane dynamics over several steps to refine the representation of one image; repeating the same frame cannot reveal inter-frame displacement, and directly retaining an unaligned membrane state misplaces a moving joint. On a frozen test subset of the official NTU60 Cross-Subject split, containing 4,563 videos and 419,968 physical frames, SpikePose lowers estimated arithmetic energy by 91.4% and head-bone-normalised PCK at 0.5 threshold by 1.12 points relative to the architecture-matched artificial neural network with the same memory module.
Cross-frame reasoning is compressed into a compact state kept in the heatmap domain and never enters the high-dimensional backbone. The module first computes a coarse inter-frame joint displacement from the raw spatial heatmaps of two consecutive frames, then a two-layer offset head predicts a bounded residual correction of at most two heatmap pixels per axis, equivalent to eight pixels at input resolution. Bilinear sampling then aligns the previous memory, a gating head supplies a joint-wise history-retention gate shared across spatial locations, and the result is fused residually with the current prediction; adding the module inside SpikePose reduces velocity and acceleration errors by 28.13% and 31.46% while slightly improving localisation, at a cost of 0.003M parameters and about 0.0023 analytical GFLOPs per frame.
The motion-quality gain holds on overall trajectory metrics but does not lead in every speed band. Against a One Euro filter run on the same DARK-decoded coordinates, with parameters selected on validation predictions and then fixed across test videos, SpikePose improves overall MPJVE and MPJAccE by 3.0% and 2.5%. On high-speed videos SpikePose lowers MPJAccE from 9.90 to 9.33 with PCK moving from 73.13 to 72.88, while One Euro has lower MPJAccE at low and mid speed.
Perspective
The demonstrated scope is frame-wise spatial encoding on the MPII image benchmark and a frozen test subset of 4,563 videos and 419,968 physical frames from the official NTU60 Cross-Subject split; all methods use one fixed skeleton-derived tube crop per complete video, videos are processed sequentially, and memory state is carried across inference chunks and reset only at source-video boundaries.
Energy is an analytical proxy computed from 45-nm FP32 reference costs of 4.6 pJ/MAC and 0.9 pJ/AC; it does not cover softmax, warping, membrane updates, element-wise fusion or data movement, and is not measured device energy or execution latency. Code is planned for release on GitHub upon acceptance.
