Huawei Noah's Ark Lab prunes LLM input tokens: cutting 65% of tokens gives Mistral-7B a 1.75x end-to-end speedup
Synopsis
Huawei Noah's Ark Lab's SDTP framework removes redundant tokens layer by layer from the input context before the first token is generated, shortening the whole inference path rather than only the decoding stage. A lightweight predictor is trained to imitate gradient-based saliency scores, with a ranking loss keeping the predicted token order close to the true one. On Mistral-7B, pruning 65% of input tokens gives a LongBench average of 39.82 versus 35.16 for LLMLingua-2 at the same 1.61x compression, and at 128K context the end-to-end speedup reaches 1.75x with 34.26% memory savings.
Interpretation
The compute bottleneck in long-input LLM inference sits in the prefill stage, and SDTP prunes before that stage so fewer tokens enter every later layer. Earlier token pruning mostly targeted KV cache compression during autoregressive decoding and was not designed for prefill; its token-selection criteria were often limited to local attention statistics or sigmoid functions. On Mistral-7B with LongBench English datasets at the same 1.61x compression, SDTP averages 39.82 against 35.16 for LLMLingua-2, with the gap concentrated in Code and FewShot tasks; prefill speedup is 1.71x versus 0.83x for LLMLingua-2.
Pruning decisions come from a lightweight module inserted between Transformer layers that scores each token from its hidden state and drops the low-scoring ones. The module is a two-layer MLP whose Gumbel-Softmax output yields a keep mask, making end-to-end training possible; the original model weights stay frozen and only the pruning module is updated. Training runs two epochs on databricks-dolly-15K with S=10 pruning stages and a keep ratio following a geometric sequence with r=0.9; inference always keeps the first four initial tokens and the local 10% of tokens.
Supervision comes from gradient saliency, computed as the derivative of the network output with respect to the input multiplied by the input, and the module is trained with both an MSE loss and a ranking loss to match it. Matching values alone is not enough because the kept set is the higher-scoring tokens, so a pairwise ranking loss constrains the module's ordering to agree with the saliency ordering. An ablation on Llama2-7B over four 5-shot tasks (COPA, PIQA, Winogrande, MathQA) gives 66.82 average for full SDTP, 66.27 without the ranking loss, and 65.93 without both ranking and MSE losses.
Perspective
The demonstrated scope is 5-shot downstream tasks and LongBench long-context evaluation on three 7B-parameter Transformer models; pruning ratio, starting layer and layer step were tuned separately on Llama2-7B, and the combination with KV cache compression was tested only on Llama2-7B with H2O.
The authors report comparable rather than lossless accuracy: on LongBench, SDTP's 39.82 average still trails the 42.34 of the original input, and SDTP-low at 1.32x compression reduces the per-task drop to 1 to 2.37. The pruning module requires alignment training for each target model, using databricks-dolly-15K.
