Sarvam Token Factory opens in India: one inference stack returns GLM 5.3 Flash's first token up to 2.1× sooner
Synopsis
Sarvam launched Token Factory, an inference stack running on GPU clusters in India that serves its own Sarvam 105B alongside open-weight models such as GLM 5.3, DeepSeek V4.1 Flash, Kimi K3.1 Pro and Gemma 4 31B through one OpenAI-compatible API. Long agent prompts previously survived in GPU memory for only about 4 seconds, so returning turns were almost always re-prefilled; Token Factory splits prefill from decode and adds a three-tier cache with cache-aware routing. On GLM 5.3 Flash with AgentX coding-agent traces, first token arrives up to 2.1× sooner than stock vLLM and Dynamo on the same two B200 nodes, with a 667 ms median against 1,428 ms at 64 concurrent sessions.
Interpretation
Sarvam puts its own model and several open-weight models behind one inference stack and one OpenAI-compatible API, and says it matches or beats leading global inference providers on speed and cost for most models it serves. Sarvam compares against public providers including Together, Fireworks and DeepInfra, and against InferenceX's published Kimi K3.1 Pro results; each of those providers offers some of these models. On GLM 5.3 Flash, using SemiAnalysis's AgentX coding-agent traces on the same two B200 nodes with both stacks disaggregated, Sarvam's stack returns the first token up to 2.1× sooner at the same throughput per GPU: at 64 concurrent sessions the median is 667 ms against 1,428 ms, and the slowest 1% waits 3.2 s against 14.1 s.
Long agent requests are split into a prefill pool and a decode pool, with a shared cache holding already-computed prompts so a returning turn does not read the whole context again. Prefill reads the entire prompt in one pass while decode generates one token at a time; when both share a GPU, each new 100K-token coding prompt stalls every session already decoding there, and GPU memory held a prompt for only about 4 seconds, or about 2 under load. The cache has three tiers: GPU memory, a Mooncake G3 shared store readable by every prefill server, and cache-aware routing with admission control; on DeepSeek V4.1 Flash, the combined hit rate over a week of production traffic rose from 91.4% to 95.7%, halving the tokens that needed prefill.
Perspective
The report covers Sarvam's own measurements of its stack: latency and throughput come from an AWS Mumbai client with 1K and 10K-token prompts and 72-hour medians, or from its own production traffic and replayed load; the figures describe behaviour on Sarvam's clusters, not an independent third-party rerun.
Sarvam says it matches or beats leading global inference providers on speed and cost for most models it serves but does not list a comparison for every model; the 1.9× tokens per GPU for Kimi K3.1 Pro holds over 14 to 48 concurrent sessions against InferenceX's published results, and at 96 sessions its median first token is slightly higher than the comparison stack's.
