DeepSeek open-sources DeepInfer, its production inference engine, with offline serving on NVIDIA SM100 and Ascend 950
Synopsis
DeepSeek has released DeepInfer, the high-performance inference framework behind its large-scale production services, under Apache 2.0, offering offline inference for DeepSeek-V4.1-Flash. The release bundles optimized kernels, attention DP with balanced CP, MoE expert parallelism plus EPLB, DSpark with confidence-scheduled verification, SWA bounded replay, asynchronous scheduling, batch-invariant inference, deterministic sampling and structured outputs, and supports both NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. A complete inference service still needs a KV cache system, VE/Prefill/Decode disaggregation and load balancing across data-parallel replicas, none of which are included.
Interpretation
DeepSeek has open-sourced DeepInfer, the high-performance inference framework that powers its large-scale production services; this release provides offline inference and states support for one model, DeepSeek-V4.1-Flash. Previously the framework served only DeepSeek's own production environment, so outside users could not obtain the same inference implementation directly. The release describes DeepInfer as the framework powering DeepSeek's large-scale production services, ships the source under Apache 2.0, and lists an offline inference example plus a script for reproducing the benchmarks.
The framework splits large-model inference into kernels, parallelism, decoding acceleration and scheduling, each with switchable configuration. These capabilities are exposed as environment variables and sampling parameters; for example, DSpark with confidence-scheduled verification is enabled by DEEPINFER_ENABLE_MTP_VARLEN, and the maximum draft length is controlled by DEEPINFER_USE_NEXTN_DECODE, defaulting to 5. The kernel layer lists FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect and custom TileLang kernels; the parallelism layer is attention DP with balanced CP and MoE expert parallelism plus EPLB; the decoding layer uses draft tokens and confidence estimates to schedule verification.
The release also exposes two reproducibility switches: batch-invariant inference keeps model results identical across different batch compositions, and deterministic sampling produces identical outputs for identical input tokens and sampling parameters. Both are marked NVIDIA-only, and deterministic sampling additionally requires disabling DSpark variable-length drafts and passing a fixed seq_seed. Batch-invariant inference is enabled with DEEPINFER_BATCH_INVARIANT=1; deterministic sampling requires setting DEEPINFER_BATCH_INVARIANT=1 and DEEPINFER_ENABLE_MTP_VARLEN=0 together and passing a fixed seq_seed to SamplingParams.
Perspective
The demonstrated scope is offline inference for DeepSeek-V4.1-Flash, limited to NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, with a build environment requiring Linux, GCC 13.0 or newer and Python 3.12, CUDA 13 on the NVIDIA side and CANN 9.2 on the Ascend side. The release explicitly lists the KV cache system, VE/Prefill/Decode disaggregation, and load balancing across data-parallel replicas for both prefill and decode as required for a complete inference service but not included in this release.
The release says most required engine functionality for VE/Prefill/Decode disaggregation is already implemented, yet that capability is still listed as needed for a complete service and outside this release. Batch-invariant inference and deterministic sampling are both marked NVIDIA-only, and deterministic sampling additionally requires disabling DSpark variable-length drafts and fixing seq_seed.
