Skip to main content
Back to timeline
arXivSource publication:

A frozen fly visual circuit scores frame-interpolated video: FlyVQA reaches mean SRCC 0.613 on the held-out fifth interpolation method

Synopsis

Frame interpolation introduces blur, ghosting and broken motion continuity, and existing quality metrics rely on optical-flow differences or learned spatiotemporal features. FlyVQA sends the reference and interpolated videos through a frozen fly visual circuit pretrained on a motion task, aligns them with compound-eye sampling, and compares cell-type activity, excitatory-inhibitory synaptic drive and their lagged coupling before a lightweight regression head produces a quality score. On BVI-VFI, training on four interpolation methods and testing on the fifth gives mean SRCC/PLCC of 0.613/0.638, above retinal-input and RGB-input baselines with comparable parameter budgets.

Source-provided article image: FlyVQA: Seeing Video Frame Interpolation Quality through Fly Eyes
arXiv

Interpretation

Quality assessment of interpolated video now runs through a visual circuit whose anatomy is constrained by the fly connectome, instead of a convolutional, recurrent or Transformer network trained from scratch. Earlier interpolation-specific metrics relied on optical-flow discrepancies or learned spatiotemporal features, while general video quality models learned their spatiotemporal representations from data. Under the BVI-VFI protocol that trains on four interpolation methods and tests on the fifth, FlyVQA reaches mean SRCC/PLCC of 0.613/0.638, above the retinal-input RNN (0.513/0.523) and GRU (0.513/0.547) with comparable parameter budgets, and above the RGB-input ResNet-18 (0.603/0.602) and spatial ViT (0.610/0.645).

Quality differences are compared point by point at three levels inside the circuit before being pooled, keeping where and when an error appears. Global feature pooling averages away local response differences, yet interpolation artifacts are local and evolve over time. The comparison covers activity across 34 output cell types, excitatory and inhibitory drive at the eight T4/T5 cell types, and excitatory-inhibitory coupling at seven lags; on 90 held-out videos the three feature groups are each correlated with centered DMOS by SRCC to test how response sensitivity relates to subjective impairment.

Only the lightweight regression head is trained, while the fly circuit stays frozen. The usual practice updates the whole spatiotemporal backbone end to end on quality labels. The quality-supervised parameter count is 0.0181M (0.0218M total), matching MLP, CNN, RNN, GRU and Transformer baselines at 0.018-0.019M; permuting trained connection strengths within excitatory and inhibitory groups while fixing wiring and signs lowers agreement with subjective ratings.

Perspective

The evaluation centers on interpolation artifacts in the BVI-VFI database, covering 36 contents, five interpolation methods and frame rates of 30, 60 and 120 Hz, with correlations reported under content-separated splits and held-out interpolation methods. The intended readers are researchers in video quality assessment and biologically inspired visual modeling.

Without adaptation on CGVQD, SRCC is 0.634, below LPIPS and BoxEye CNN/GRU, while the interpolation subset reaches SRCC 0.833, indicating that the frozen motion circuit and fixed response features have limited ranking ability across heterogeneous distortions. On the Average and DVF interpolation methods, the RGB-input ResNet-18 and VGG-11 reach higher SRCC than FlyVQA, and the circuit receives only spatially integrated luminance on a 721-site hexagonal lattice, discarding color and fine texture cues.