Skip to main content
Search

# Inference Optimization

3 related articles

Subscribe to updates
  1. arXiv

    Together Compiler turns inference tuning into a continual loop: up to 2.6x throughput on frontier models, compiled in under 24 hours

    Together AI released Together Compiler (TKCC), which turns optimization of an AI inference engine from a one-off compile into a continual loop driven by agents. Existing agent systems either tune individual kernels on offline benchmarks with low measurement fidelity, or restart the whole engine for every measurement and iterate slowly. TKCC decouples engine startup from forward-pass computation with a tiered persistent engine and adds a cost model for parallelism configurations; it beats hand-written Megakernels on Llama 1B and the best open-source engines on Kimi K3, GLM 5.3, MiniMax M3.1 Flash and DeepSeek v4.1 Flash on the InferenceX benchmark, within 24 hours and at most 2 nodes per model.

  2. arXiv

    No GPUs needed to plan LLM serving: BLIS simulates the full pipeline on CPU and holds a 300 ms TTFT SLO on four H100s

    BLIS is a deterministic, CPU-only discrete-event simulator that models the whole LLM inference serving pipeline, including admission, routing, scheduling, prefill/decode execution, KV-cache management, and preemption, reaching 6.7% median end-to-end error across 36 experiments while running about 200 times faster than real execution.

  3. NVIDIA Blog

    OpenAI launches GPT-6 Astra Ultrafast on NVIDIA Blackwell, up to 8x faster token generation than Astra Standard

    OpenAI released GPT-6 Astra Ultrafast, running on NVIDIA Blackwell GPUs, which delivers up to 8x faster token generation than Astra Standard mode through inference optimizations that tap into the NVIDIA Blackwell architecture, and is available now in the OpenAI API and to eligible ChatGPT Work and Codex users.

You’re all caught up