Skip to main content
Back to timeline
arXivSource publication:

A 30B model won't fit on one consumer GPU: run some layers on the CPU and hide the weight transfers behind that work

Synopsis

When model weights do not fit in a single consumer GPU, on-demand weight transfer takes about 98% of decode time. A layer-wise schedule runs some layers on the CPU while asynchronously moving the next layer's weights to the GPU, hiding the transfer behind CPU computation; on an RTX 5090 desktop, OPT-30B inference runs 3.5 times faster than HuggingFace Accelerate and 6.3 times faster than DeepSpeed Zero-Inference.

AI-generated editorial illustration: Layer-wise CPU-GPU Scheduling for LLM Inference on Memory-Limited Consumer GPUs

Interpretation

For models whose weights exceed single-GPU memory, decode latency on a consumer desktop can be cut by overlapping CPU computation with weight transfer instead of relying on quantization. The usual on-demand loading serializes computation and transfer, so about 98% of OPT-30B decode execution time goes to weight movement; prior offloaded-inference work also reports exposed PCIe transfer at roughly 96–98% of total execution time. On a desktop with a 32 GiB RTX 5090, an Intel Core Ultra 7 265F and PCIe Gen5 x16, OPT-30B with a 256-token prompt runs 3.5, 6.3, 2.8 and 1.5 times faster than Accelerate, DeepSpeed, CPU-only and SplitGen respectively.

Each layer is assigned to one of three modes: resident on the GPU, computed on the CPU, or held in host memory and transferred to the GPU at execution time, which is called stream mode. The scheduling problem is written as minimizing one decode step's latency and simplified by an approximation that relaxes transfer timing constraints into choosing the ratio of CPU layers to stream layers. Per-layer CPU computation and transfer times come from offline profiling, and stream-layer weights are moved asynchronously while CPU layers compute; 99.6% of transfers for OPT-30B and 99.8% for OPT-66B are hidden by computation.

Perspective

The demonstrated setting is single-GPU, batch-size-1 decoding on a 32 GiB RTX 5090 with an Intel Core Ultra 7 265F, 128 GiB host memory and PCIe Gen5 x16, using 8 CPU threads, with OPT-30B, OPT-66B and Llama2-70B in FP16, prompt lengths of 256 and 1024 and a decode length of 32. The method uses no compression or approximation, so only performance and energy are evaluated.

The execution plan uses average computation and transfer times, but host-memory pressure makes those times unstable, so Llama2-70B did not follow the plan; the authors suggest worst-case profiling measurements as a countermeasure. Plan construction takes 3.2 seconds and the plan is reusable. Results come from a single hardware configuration and have not been validated across a wider range of devices.