Skip to main content
Search

# TensorRT Edge-LLM

1 related articles

Subscribe to updates
  1. NVIDIA cuts edge LLM deployment to a few commands: on DGX Spark, DSpark lifts generation from 73.3 to 108.5 tokens/s

    NVIDIA released TensorRT Edge-LLM, a lightweight C++ inference runtime, with a trtedge command-line flow for downloading a model, building TensorRT engines and starting a local OpenAI-compatible server on a 64GB DGX Spark. Previously, getting a model ready to run at the edge could involve substantial setup. On Nemotron-3.5 Lightning, enabling DSpark speculative decoding with three draft tokens raised mean generation speed on SpeedBench 1K from 73.3 to 108.5 tokens/s while median time to first token stayed essentially unchanged.

You’re all caught up