MIAO narrows accumulators row by row: MX systolic arrays save up to 38.7% area with 1.06% extra perplexity
Synopsis
MIAO replaces uniform floating-point accumulation across an MX-quantized systolic array with per-row accumulator precision and jointly selects product precision, searching for the lowest area and energy configuration under an end-to-end perplexity constraint. Relative to an all-BF16 intermediate-arithmetic baseline, it cuts processing-element array area by up to 38.7% and energy per operation by up to 36.1% across MXFP8, MXFP6 and MXFP4, with at most 1.06% additional perplexity degradation over operand-only MX quantization. Configurations selected on TinyLlama transfer to five further models without retuning, adding at most 1.6% perplexity.
Interpretation
MX quantization shrinks only the operand widths, while products and partial sums are still accumulated in wide floating point such as BF16, so this cost was untouched by operand quantization. In the evaluated baseline designs, accumulation accounts for more than half of processing-element area and up to 54% of processing-element energy; in the baseline with MX operands and BF16 intermediate arithmetic, BF16 accumulation is about 65% of processing-element array area. These shares come from logic synthesis and gate-level power analysis of parameterized processing elements in a 16 nm technology, which also supplies the area and energy model.
Each row of a weight-stationary systolic array is one accumulation stage, so early rows can take narrow accumulators and later rows wider ones instead of one precision for the whole array. Conventional designs use a single accumulator format throughout the array even though partial-sum magnitudes grow with accumulation depth: under MXFP8 the maximum observed magnitude rises from 8 after the first addition to 23.93 after the fifteenth. That magnitude distribution comes from simulating quantized operands captured from 67 linear layers across 16 evaluation samples, with accumulator formats configured independently per row.
The search pairs a synthesis-derived hardware cost model with a calibrated perplexity predictor to shortlist candidates, then confirms each with bit-exact end-to-end LLM evaluation. Exhaustive end-to-end evaluation of every configuration is expensive; on MXFP8, a loss model fitted on 11 ladders predicted 319 unseen ladders with an RMS perplexity error of 0.0066, below the paired-run standard error of 0.0085. The bit-exact arithmetic simulator matched hardware arithmetic outputs on 65,536 elements for MXFP8, and final configurations are certified by end-to-end evaluation.
Perspective
The demonstrated scope is weight-stationary systolic arrays of 16×16, 32×32 and 64×64, with hardware cost estimated from a synthesis model at 16 nm and a 2 ns clock, accuracy constrained by perplexity on WikiText-2, and evaluation on TinyLlama plus five further models. The authors propose cross-model calibration and extension to other accelerator architectures and workloads as future work.
The authors state explicitly that the TinyLlama accuracy constraints are not guaranteed to hold when transferred to other models, and that MXFP4's total perplexity degradation relative to BF16 comes mainly from operand quantization. The exhaustive comparison covers only the restricted precision space, and optimality is certified only within that space.
