PixelSpline gives every pixel one shared trajectory: 406.41 ms per frame, less than half of V3's latency at a 0.90 dB PSNR cost
Synopsis
Continuous space-time video super-resolution must rebuild frames at arbitrary spatial scales and arbitrary times, and earlier methods inferred motion from only two endpoint frames while modeling the two directions separately, so occlusion or weak texture could produce mismatched correspondence. PixelSpline strengthens endpoint features with long-context encoding, lifts features continuously onto the requested high-resolution grid, and builds one continuous trajectory per pixel shared by both endpoints to transport color and features together. On Adobe240 average frames it trails the quality-leading V3 by 0.90 dB PSNR while taking 406.41 ms per frame against 912.21 ms for V3.
Interpretation
Continuous space-time video super-resolution can now transport content along a single pixel trajectory shared by both endpoints instead of predicting two separate motions launched from the two endpoints. Earlier continuous methods inferred motion to the target time from each endpoint separately; even with shared network parameters, the two predictions depended on different endpoint features, with no constraint that corresponding pixels follow the same physical curve. On 3,858 frames randomly drawn from the Adobe240 test set, replacing the shared bidirectional trajectory controls with independently predicted forward and backward controls while keeping the long-context architecture lowered PSNR from 32.49 dB to 32.25 dB and SSIM from 0.930 to 0.924.
The reconstruction pipeline first aggregates observations beyond the queried frame pair into the two endpoint representations with a long-context encoder, then builds appearance features directly on the requested high-resolution grid. Inferring correspondence from only two endpoint frames lets occlusion, motion blur, repetitive texture or large displacement corrupt the match, and a wrong match affects every intermediate time queried between that pair. Removing the additional context frames and using only the two anchor frames lowered PSNR from 32.49 dB to 31.80 dB and SSIM from 0.930 to 0.918 on the same subset.
Each source pixel's motion is written as one cubic curve, and the two endpoints walk the same curve from opposite ends to the same intersection at the target time. The colors and features transported from the two endpoints therefore share one geometry instead of drifting to inconsistent target locations. Endpoint correspondence starts from SEA-RAFT matches and confidence, uses frozen GMFlow for global matching where reconstruction error is large, and is merged and refined with the context-enhanced anchor features.
The trajectory position is used twice: once to forward-warp bicubically upsampled endpoint colors into a fused coarse intermediate frame, and once to carry lifted endpoint features to the same locations as an aligned feature. A lightweight pointwise decoder predicts only a bounded residual from the aligned feature and adds it to the coarse frame, so explicit transport handles motion and color while the decoder handles disocclusions, splatting artifacts and fine detail. On REDS the method reaches a temporal optical-flow consistency error of 8.171, below V3's 8.923, where a lower value means reconstructed motion is closer to the reference flow.
Perspective
Evaluation covers in-distribution results on the Adobe240 test split and cross-dataset results on GoPro and REDS without dataset-specific fine-tuning, plus spatial-only and temporal-only boundary cases on Vid4; training uses four NVIDIA H100 GPUs with a total batch size of 8 and eight temporally subsampled context frames per sample.
PSNR ranks second across all six dataset-protocol combinations, trailing V3 by 0.77/0.80 dB on GoPro Center/Average, 0.92/0.90 dB on Adobe240 Center/Average and 0.22/0.32 dB on REDS Center/Average; for temporal-only interpolation it has the highest SSIM (0.890) but a PSNR of 29.01 dB, below V3 and BF-STVSR. The authors state that reconstruction accuracy can still improve and that training remains computationally demanding and time-consuming.
