Skip to main content
Search

# Parallel Prompting

1 related articles

Subscribe to updates
  1. arXiv

    One long document, hundreds of questions: Parallel Prompting decodes them as parallel streams for a 3.7x speedup on SGLang

    When tens to hundreds of questions are asked about the same long document, prefix caching removes only the repeated prefill; every decode step still re-reads the document's key-value cache once per question, so decoding is bound by memory bandwidth. Parallel Prompting packs those questions into a single request and decodes them as independent streams, reading the shared document once per step, with a stream-id mask evaluated inside the attention kernel so each answer sees only the document and its own question. On CodeLlama-7B with 8k–16k-token documents and 64–256 questions per document, it is 3.7x faster than stock SGLang with the FlashInfer backend on an A100 and 5.9–6.0x faster on an RTX A6000, winning all 144 paired runs.

You’re all caught up