Skip to main content
Search

Yongding Tao

1 related articles

Subscribe to updates
  1. arXiv

    QK-MAP drops query-key normalization: lower training and validation loss than QK-Norm from 190M to 7B parameters

    Modern large language models commonly use QK-Norm to keep attention logits from growing, but normalization also erases the token-specific information carried by query and key magnitudes. QK-MAP instead encodes input magnitude as fixed-norm phase patterns using multiple learnable frequencies, so magnitude changes show up in the direction and phase of the representation rather than in vector length. In pretraining comparisons from 190M to 7B parameters with shared backbones and data budgets, QK-MAP had lower training and validation cross-entropy than both standard attention and QK-Norm at every evaluated scale, and the best eight-task average accuracy at 7B.

You’re all caught up