Skip to main content
Search

Yao Tao

1 related articles

Subscribe to updates
  1. arXiv

    Huawei Noah's Ark Lab prunes LLM input tokens: cutting 65% of tokens gives Mistral-7B a 1.75x end-to-end speedup

    Huawei Noah's Ark Lab's SDTP framework removes redundant tokens layer by layer from the input context before the first token is generated, shortening the whole inference path rather than only the decoding stage. A lightweight predictor is trained to imitate gradient-based saliency scores, with a ranking loss keeping the predicted token order close to the true one. On Mistral-7B, pruning 65% of input tokens gives a LongBench average of 39.82 versus 35.16 for LLMLingua-2 at the same 1.61x compression, and at 128K context the end-to-end speedup reaches 1.75x with 34.26% memory savings.

You’re all caught up