Skip to main content
Back to timeline
NVIDIA Technical BlogSource publication:

NVIDIA cuts edge LLM deployment to a few commands: on DGX Spark, DSpark lifts generation from 73.3 to 108.5 tokens/s

Synopsis

NVIDIA released TensorRT Edge-LLM, a lightweight C++ inference runtime, with a trtedge command-line flow for downloading a model, building TensorRT engines and starting a local OpenAI-compatible server on a 64GB DGX Spark. Previously, getting a model ready to run at the edge could involve substantial setup. On Nemotron-3.5 Lightning, enabling DSpark speculative decoding with three draft tokens raised mean generation speed on SpeedBench 1K from 73.3 to 108.5 tokens/s while median time to first token stayed essentially unchanged.

Source-provided article image: Run AI Models for Edge Application Development with Just a Few Commands
NVIDIA 开发者技术博客

Interpretation

Edge developers can now use the trtedge command line on DGX Spark to download a model, build TensorRT engines and start a local server, carrying the same inference runtime from the development machine to deployment platforms such as Jetson Thor, DRIVE AGX Thor and IGX Thor. Previously, getting a model ready to run within tight memory, power and latency budgets could involve substantial setup. The example flow was tested with TensorRT Edge-LLM 0.11.1 and TensorRT 10.16.1.11; the first launch waits for an approximately 21GB model download and engine build, and later launches reuse cached engines.

The runtime accelerates generation through DSpark speculative decoding: a draft model proposes several tokens, and the main model verifies them in a single pass. DSpark replaces token-by-token generation with a draft model plus main-model verification, set to three draft tokens in the example. On DGX Spark 64GB with NVFP4, batch size one, one concurrent streaming request and a 128-token output limit, mean generation speed rose from 73.3 to 108.5 tokens/s on SpeedBench 1K, from 72.8 to 105.3 on SpeedBench 2K, and from 73.4 to 126.9 on HumanEval prompts.

Time to first token barely changes with DSpark on, indicating the speedup lands in token-by-token generation after the first token. Median time to first token moved from 201.0 to 203.0 ms on SpeedBench 1K, from 329.5 to 331.6 ms on SpeedBench 2K, and from 98.1 to 98.7 ms on HumanEval prompts. Generation speed is the average token generation rate per request excluding time to first token; peak total system memory during inference with DSpark was 27.9GB, including the OS.

Perspective

The demonstrated setting is a 64GB DGX Spark running DGX OS 7.6.0 (Ubuntu 24.04.4 LTS), with NVFP4, thinking enabled, batch size one, one concurrent streaming request and a 128-token output limit; average input lengths were 1,153 tokens for SpeedBench 1K, 2,247 for SpeedBench 2K and 139 for HumanEval.

The DSpark speedup varies by workload, with a larger gain on HumanEval prompts than on either SpeedBench set; parameters such as the draft-token count need tuning for a given workload, and the example shows only the setting of three.