Together Compiler turns inference tuning into a continual loop: up to 2.6x throughput on frontier models, compiled in under 24 hours
Synopsis
Together AI released Together Compiler (TKCC), which turns optimization of an AI inference engine from a one-off compile into a continual loop driven by agents. Existing agent systems either tune individual kernels on offline benchmarks with low measurement fidelity, or restart the whole engine for every measurement and iterate slowly. TKCC decouples engine startup from forward-pass computation with a tiered persistent engine and adds a cost model for parallelism configurations; it beats hand-written Megakernels on Llama 1B and the best open-source engines on Kimi K3, GLM 5.3, MiniMax M3.1 Flash and DeepSeek v4.1 Flash on the InferenceX benchmark, within 24 hours and at most 2 nodes per model.
Interpretation
Inference optimization becomes a continual loop: agents keep submitting new forward-pass implementations, and the engine measures and keeps the faster ones without restarting. Earlier agent optimization systems traded measurement throughput against fidelity, using random data in offline kernel benchmarks with low fidelity, or restarting the engine for every end-to-end measurement. The tiered persistent engine lets TKCC process 252 times more agent experiments than a system that restarts the engine per measurement, with 11 times less idle time per agent.
The engine loads the model once, then hot-reloads agent-submitted forward passes as candidates and routes them through a fast and a slow measurement tier. Hot reloading replaces the model's forward function pointer and re-captures CUDA/HIP graphs; the fast tier weeds out candidates that crash or hang, and the slow tier gives higher-fidelity measurements. With two measurement tiers the system evaluates 8.1 times more candidates than a single slow tier (1,354 versus 168) and finds a solution 7.2% faster.
A cost model ranks parallelism configurations before agents spend expensive optimization effort on them. The cost model compares theoretical hardware specifications, measured compute, memory and communication performance, and end-to-end forward-pass performance, separating hardware limits from immature software implementations. For Kimi K3 production deployment the cost model found a configuration that raised TPS by up to 24% and TPM by up to 18%.
Perspective
The demonstrated scope is Together AI's own inference deployment: batch-size-1 Llama 1B inference, four frontier models on several hardware platforms under the InferenceX benchmark, and non-language workloads spanning weather, biology, time-series forecasting and PDE simulation. The intended readers are engineers and researchers working on inference systems and compilation.
Agents still spend much of their time waiting on measurement feedback and queuing, and measurement throughput is named the primary remaining bottleneck; some optimizations are hardware-specific, such as fusing a whole layer into one persistent kernel, which gave no lasting gain on GPUs but worked on Trainium2.
