Skip to main content
Back to timeline
vLLM BlogSource publication:

vLLM splits multimodal encoding into its own instances: 6.6–9.5% more throughput per GPU on DeepSeek V4.1 Flash

Synopsis

vLLM now lets multimodal serving deploy encoding, prefill and decode as separately scaled instances (EPD), so encoder capacity for images, video and audio can grow independently of the language model. The release adds metadata-only request forwarding, a dedicated encoder execution path, remote embedding transfer and cross-encoder cache reuse. On DeepSeek V4.1 Flash document question answering at concurrency 64–128, input-plus-output throughput per GPU across all 12 EPD GPUs was 6.6–9.5% above an eight-GPU PD baseline and P90 time-to-first-token was 44–47% lower, while at concurrency 16 EPD was slower.

Source-provided article image: Encoder Disaggregation in vLLM: Scaling Multimodal Serving with EPD
vLLM

Interpretation

The encoding stage of a multimodal request can now run as its own vLLM instance holding only a vision or audio encoder, scaled separately from prefill and decode instances. Encoding was previously bound to language-model execution, so adding encoder capacity for image-heavy traffic also meant adding decode capacity. On GB200 with DeepSeek V4.1 Flash answering 512 MP-DocVQA validation questions, a baseline with four GPUs for prefill and four for decode was compared with EPD adding four single-GPU encoder instances; at concurrency 64–128 EPD gave 6.6–9.5% higher throughput per GPU and 44–47% lower P90 time-to-first-token.

Downstream instances no longer reprocess the original media; they rebuild placeholders from layout metadata published by the encoder while embeddings travel separately over a connector. When the original request is forwarded, the consumer still has to fetch, decode, resize, normalize or patchify the media to recover its input layout, work the encoder has already done. On 1,000 MMAU test-mini audio samples with Qwen3-Omni-30B-A3B-Instruct, 1E+1PD and concurrency 64, metadata-only forwarding raised throughput from 62.21 to 73.13 requests/s and cut mean latency from 980 to 853 ms, with accuracy moving from 75.90% to 75.65%.

Encoder instances follow a dedicated execution path that keeps no language-model KV cache, samples no tokens, and loads only the encoder weight shards it needs. Encoder instances previously reused an execution lifecycle designed for language models and read pure language-model shards when loading checkpoints. On a pruned Kimi-K3 checkpoint the encoder read 2 of 96 shards and weight loading dropped from 683 s to 0.31 s.

Cross-encoder cache reuse lets an image reassigned to a different encoder hit an embedding already in shared storage instead of recomputing it. Instance-local caching left a reuse gap: after load balancing or scaling routed an image to another encoder, that instance had no cached output and repeated the encoder forward. In a reassignment experiment on six H20 GPUs with InternVL3_5-38B-HF, a 2E+4PD topology and same-node Mooncake TCP, 128 requests carrying 512 images saw repeated encodings fall from 512 to zero, with time-to-first-token p50 dropping from 1.209 to 0.819 s and p90 from 1.656 to 0.891 s.

Perspective

The demonstrated scope is vLLM EPD deployments: the audio evaluation covers pure-audio Chat Completions requests on the Qwen2-Audio, AudioFlamingo3, Ultravox, Qwen2.5-Omni and Qwen3-Omni paths, with realtime and combined audiovisual inputs outside it, and cross-encoder reuse initially supports images, Model Runner V2, encoder TP=1 and no dynamic LoRA. Core capabilities ship in vLLM v0.32.0, while DeepSeek V4.1 Flash metadata-only support is on main.

On the audio path accuracy moved from 75.90% to 75.65%, a difference that remains to be investigated, and that comparison measures two audio input paths within EPD rather than EPD against colocated serving. The cross-encoder reassignment experiment is a single run in which every image was reassigned, so it shows the best case, and production savings depend on reuse frequency against the Store's memory and I/O costs. At concurrency 16 EPD had lower throughput per GPU and at 32 it was roughly break-even, so the benefit depends on load and resource allocation rather than separation alone. Asynchronous publication means concurrent cold requests can still duplicate computation.