When GPU memory runs out, stop paging: Ember keeps cold data on the host and speeds LLM generation several-fold
Synopsis
NVIDIA's Unified Virtual Memory collapses once data exceeds GPU memory, and Ember inverts the default: allocations start mapped into host memory, GPU hardware access counters decide which regions turn hot enough to promote, and cold regions are reclaimed in batches before memory fills. Across six applications on three PCIe GPUs under oversubscription, Ember reaches geometric-mean speedups over stock UVM and sustains LLM generation at several times UVM's throughput, with no profiling run or application changes.
Interpretation
Ember lets workloads larger than GPU memory stop relying on page faults to move data: an allocation starts mapped in host memory and is reachable by the GPU over the interconnect, and only regions judged hot are promoted into device memory. Stock UVM migrates on faults by default, so once device memory fills, every fault must first reclaim a resident page, interleaving reclamation with execution on the critical path; mapping the same data entirely on the host avoids reclamation but bounds every access by interconnect bandwidth. On the RTX 5080, A40 and H200 under oversubscription, Ember-static reaches geometric-mean speedups over stock UVM of 1.9x, 2.4x and 2.6x across six data-intensive applications, with Ember-adaptive at 1.6x, 1.9x and 2.1x; UVM-mapped trails Ember-static by 1.3x to 2.4x.
Ember turns host-side accesses into placement signals using the GPU's hardware access counters, and although each counter notification covers one VA block, a promotion spans a run of consecutive blocks around it, so a few hundred counters can drive a mapped set far larger than the counter count. Stock UVM does not use these counters and migrates on faults; the counters themselves observe only host memory and cannot report the access frequency of device-resident regions, so coldness must be inferred separately. Ember-adaptive is a patch to NVIDIA's open-source UVM kernel module, adding 1194 and removing 31 lines across 11 files on CUDA 12.4 and Linux 6.17; the promotion span defaults to 64 MiB and the counter threshold to 256 accesses.
Reclamation moves off the critical path: Ember evicts cold regions in batches by decayed demand before device memory fills, and migration-aware protection keeps a region that returns soon after eviction from bouncing back and forth. Stock UVM's age-based, approximately LRU reclamation keeps running once memory is full, interleaving with execution and dragging down warp occupancy; Ember leaves a reclaimed region mapped and reachable over the interconnect and returns the freed capacity to the driver's allocator. A background worker in the driver's GPU memory allocator runs every 25 ms, evicts at most 16 physical chunks per pass, and uses a 90% high watermark and 85% low watermark; migration-aware protection grants a 100 ms base grace plus 250 ms per score point, capped at 2000 ms.
Runtime decisions replace offline profiling: Ember-adaptive beats HELM, which profiles an application and then picks a policy per allocation, on all five benchmarks without any profiling run. HELM profiles the application first and assigns each allocation a policy such as migrate, pin to host or pin to device; profiling, metric processing and selection take 32 s to 1448 s, account for 2% to 50% of its end-to-end runtime, and can tag the same allocation differently on a repeat run. Ember-adaptive outperforms HELM by a mean of 1.4x across five benchmarks; comparing kernel time alone, it stays faster on three of five, by up to 1.4x with a mean of 1.2x, and is slower on bfs and gramschm.
Perspective
Ember targets discrete GPUs whose device memory is oversubscribed, preserving the UVM memory and programming model. Ember-static runs on the stock driver with placement hints, needs no elevated privileges or bare-metal access, and suits applications whose hot set is known up front and need not be revised at runtime; Ember-adaptive runs inside the driver, requires hardware access counters and a machine where an administrator can load a module, and transparently supports unmodified applications. The evaluation spans four GPUs, three deployment settings and three workload classes, including six data-intensive applications, LLM generation and SSB analytical queries. Ember places only memory allocated via cudaMallocManaged, so buffers allocated inside libraries remain outside its control.
The hardware counters observe only host memory and cannot report whether a device-resident region has gone cold, so Ember infers this from the demand recorded before promotion, decayed over time, and a still-hot region can therefore be reclaimed by mistake and moved twice. Ember-adaptive needs bare-metal access to load its kernel module, which builds against the internal interfaces of an open-source driver release and must be rebased for each new one. On a multi-GPU node, peer memory forms a third tier, and Ember currently places data only between one GPU and host memory.
