Skip to main content
Search

# Manifold-Constrained Hyper-Connections

1 related articles

Subscribe to updates
  1. arXiv

    DeepSeek-V4 preview supports million-token context: single-token inference FLOPs at 27% of V3.2 and KV cache at 10%

    DeepSeek-AI released a preview of the DeepSeek-V4 series, with DeepSeek-V4-Pro at 1.6T parameters (49B activated) and DeepSeek-V4-Flash at 284B parameters (13B activated), both supporting a one-million-token context length. Standard attention grows quadratically with sequence length, making ultra-long contexts a bottleneck for inference and test-time scaling; the series instead interleaves compressed sparse attention with heavily compressed attention and pairs it with low-precision storage. At the one-million-token setting, DeepSeek-V4-Pro needs 27% of the single-token inference FLOPs and 10% of the KV cache of DeepSeek-V3.2, while DeepSeek-V4-Flash needs 10% and 7%.

You’re all caught up