Skip to main content
Back to timeline
arXivSource publication:

DeepSeek-V4 preview supports million-token context: single-token inference FLOPs at 27% of V3.2 and KV cache at 10%

Synopsis

DeepSeek-AI released a preview of the DeepSeek-V4 series, with DeepSeek-V4-Pro at 1.6T parameters (49B activated) and DeepSeek-V4-Flash at 284B parameters (13B activated), both supporting a one-million-token context length. Standard attention grows quadratically with sequence length, making ultra-long contexts a bottleneck for inference and test-time scaling; the series instead interleaves compressed sparse attention with heavily compressed attention and pairs it with low-precision storage. At the one-million-token setting, DeepSeek-V4-Pro needs 27% of the single-token inference FLOPs and 10% of the KV cache of DeepSeek-V3.2, while DeepSeek-V4-Flash needs 10% and 7%.

Interpretation

The DeepSeek-V4 series pushes routinely supported context length to one million tokens while cutting inference compute and cache footprint in long-context settings. Earlier open models were limited on ultra-long sequences by the quadratic cost of standard attention, a bottleneck that constrained test-time scaling and long-horizon tasks. In the one-million-token context setting, DeepSeek-V4-Pro requires 27% of the single-token inference FLOPs and 10% of the KV cache of DeepSeek-V3.2; DeepSeek-V4-Flash, with fewer activated parameters, requires 10% and 7%.

The efficiency comes from attention layers that alternate two compression schemes rather than relying on a single mechanism. Compressed Sparse Attention (CSA) first compresses the KV cache of every m tokens into one entry, then keeps only the top-k entries by index score for core attention; Heavily Compressed Attention (HCA) merges KV entries at a larger compression rate but keeps dense attention. DeepSeek-V4-Flash uses a CSA compression rate of 4 and an HCA compression rate of 128, and DeepSeek-V4-Pro uses the same rates of 4 and 128, with both adding a sliding-window branch of size 128 for local dependencies.

On the training side, the optimizer and the residual connection form were both changed to support this scale. DeepSeek-V4 keeps DeepSeekMoE and multi-token prediction but switches to the Muon optimizer for faster convergence and more stable training, and strengthens residual connections between adjacent Transformer blocks with Manifold-Constrained Hyper-Connections (mHC). DeepSeek-V4-Flash is pre-trained on 32T tokens and DeepSeek-V4-Pro on 33T tokens, with sequence length extended from 4K through 16K and 64K to 1M.

Perspective

The demonstrated scope is the inference FLOPs and KV cache comparison for the two preview models, DeepSeek-V4-Pro and DeepSeek-V4-Flash, in the one-million-token context setting, plus a description of the pre-training and post-training pipeline; model checkpoints are public on Hugging Face.

The material states that DeepSeek-V4-Pro-Max still trails Gemini-3.1-Pro on knowledge evaluations and falls marginally short of GPT-5.4 and Gemini-3.1-Pro on reasoning benchmarks, and describes its trajectory as roughly 3 to 6 months behind frontier models; these are conclusions under its internal evaluation.