Skip to main content
Search

# PaperTrail

1 related articles

Subscribe to updates
  1. arXiv

    Sparse attention's real bottleneck is finding the keys: LSA clusters them in a learned latent space, and one attention layer prefills 10M tokens 22.7x faster

    At ultra-long context, most of a sparse attention layer's prefill time goes to the indexer scoring every key, not to the attention itself. Latent Sparse Attention (LSA) clusters keys hierarchically in a learned latent space so each query scores cluster centroids and then expands the keys of the selected clusters. On GLM-5.3-Flash at 10M tokens, the distilled variant D-LSA prefills one attention layer 22.7x and decodes 4.6x faster than that model's original sparse attention, staying within 4 points on RULER; the natively trained N-LSA matches full-attention validation loss at 128k and trains 1.6x faster per layer.

You’re all caught up