Skip to main content
Search

Qinghang Huang

1 related articles

Subscribe to updates
  1. arXiv

    GDN recurrent states drop to 4 bits: loss-guided bit allocation plus a recent-write window keeps long reasoning from stalling

    Linear-attention models compress history into a fixed-size recurrent state, and once that state is quantized to low bits, writeback errors enter every later recurrence, so long-reasoning scores collapse and outputs turn repetitive and overlong. GGA allocates head bit widths from the gradient of language-model loss with respect to the writeback state, while PW keeps recent pure writes as FP8 vector pairs and rolls them into the low-bit historical state by write age. Across seven evaluations on Qwen3.5-4B/9B, GGA@ raises the macro-average from STEPQuant@'s 31.44/37.42 to 50.92/67.26, and adding PW-S8 reaches 69.83/78.63.

You’re all caught up