Skip to main content
Back to timeline
arXivSource publication:

CHARMING splits attention heads between tensor array and DRAM compute, lifting prefill 23.6% and decode 44%

Synopsis

Transformer inference on edge devices is limited less by arithmetic than by moving weights and intermediate results between memory and compute. CHARMING splits attention heads across an AI Engine array and an in-DRAM compute unit that run concurrently, choosing which heads to offload under accuracy and quantization constraints; on a validation platform built from a VCK5000 and an Alveo U200, it improves prefill performance by up to 23.6% and decode by up to 44% over AIE-only CHARM, cuts per-layer energy by up to 7.6% and improves energy-delay product by up to 24.7% with fixed-point in-memory computation, while keeping accuracy similar to a 32-bit floating-point baseline.

Source-provided article image: CHARMING: Parallel Edge AI Acceleration Across Tensor Processing and In-Memory Computation
arXiv

Interpretation

CHARMING runs different heads of the same attention layer on two compute substrates at once: some stay on the AI Engine array for tensor processing, while others go to a compute unit inside DRAM that works in place. The earlier CHARM did one thing, overlapping operand movement with matrix computation to reduce how much data movement lengthened execution, with all computation still on the AIE side. On a validation platform of AMD Versal VCK5000 and Alveo U200 boards, prefill performance rises by up to 23.6% and decode by up to 44% over AIE-only CHARM.

The split follows each unit's cost curve: operations with shallow reduction depth and few output rows run faster in memory, while deep-reduction, compute-heavy operations stay on the AIE array. Splitting by head is the standard intra-layer parallelism for Transformers, but existing schemes run the same operation in the same number format on every device, forgoing the cost differences between substrates at different precisions. Offline accuracy profiling compares quantization scopes on GSM8K: quantizing attention only leaves Llama-3.1-8B at 85.8%, matching its FP32 baseline, while Llama-2-13B and Mixtral-8x7B lose only 0.9 and 1.5 percentage points.

At run time the AIE side produces both output formats from one GEMM pass, quantizing the offloaded heads to N-bit integers into a separate memory region while the rest stay FP32, and the two units then proceed concurrently. Quantization needs neither an extra data pass nor a bitstream reconfiguration, and the in-memory unit multiplies and accumulates in place using row-level DRAM commands. On the VCK5000, three projection shapes show latency changes below 0.6% when heads move to N-bit precision; profiling-based head selection limits accuracy loss to 1.5 and 1.9 points on Llama-3.1-8B and Llama-2-13B, against 5.0 and 5.3 points under stochastic selection.

Perspective

The demonstrated setting is a prototype of two FPGA boards attached over PCIe to one x86 host: the AIE compute unit runs on the VC1902 in a VCK5000, and the in-memory compute unit runs on an Alveo U200 with a soft DDR4 controller, evaluated on GSM8K with Llama-3.1-8B, Llama-2-13B and Mixtral-8x7B, plus the long-sequence WAN 2.2 and Qwen3-32B.

In-memory compute performance and energy come from gem5-CIMD simulation on HBM2, with hardware used to validate functionality; the simulation assumes native Ambit-style inversion, whereas inversion on commodity DRAM goes through a neighboring subarray and a host-mediated write-back whose overhead is excluded from the reported numbers. Reported performance also excludes the PCIe and host-memory staging overhead of the two separate boards.