Skip to main content
Back to timeline
arXivSource publication:

Brain-to-text decoding moves onto an iPhone: FETTUCCINE hits 370x real-time on a phone, with word error rate as low as 4.39%

Synopsis

Speech neuroprostheses decode neural activity during attempted speech into text, but the standard cascaded pipeline of an n-gram language model plus LLM rescoring needs tens to hundreds of gigabytes of memory and a dedicated workstation, and its latency can jump from milliseconds to more than ten seconds. FETTUCCINE adapts open-source automatic speech recognition models into end-to-end brain-to-text decoders, replacing the acoustic front end with a neural adapter and applying low-rank updates only to the encoder; on the public brain-to-text 2025 data it reaches word error rates as low as 4.39% and decodes at up to 370x real-time on an iPhone 15 Pro Max.

Source-provided article image: FETTUCCINE: Fast and efficient brain-to-text decoding on mobile devices
arXiv

Interpretation

Brain-to-text decoding now runs on a commercial phone instead of requiring a workstation in a research lab. The earlier cascaded pipeline chained phoneme decoding, an n-gram language model and LLM rescoring; the 5-gram version needed about 300 GB of memory, with a median latency of 94 ms but a worst case of 15.9 s. On an iPhone 15 Pro Max, the FETTUCCINE variants keep usable word error rates: Whisper Tiny reaches 372.7x real-time throughput at 209 MB peak unified memory, and Whisper Small reaches 5.6% word error rate at 55.1x real-time.

The method swaps the speech recognition model's front end for an adapter that accepts neural signals, while the rest of the backbone stays largely frozen. Speech recognition models are trained on audio spectrograms, a different data domain from cortical microelectrode recordings. The neural adapter maps 512 neural channels (256 electrodes times two features) into the backbone's expected embedding space, using a 1D U-Net for Whisper variants and a convolutional subsampling block for Parakeet variants; training runs in two stages, first the adapter alone, then the adapter jointly with LoRA updates on the encoder.

The end-to-end structure lets gradients flow from output tokens back to raw neural features, which locates the electrodes and time windows that drive each prediction. In a cascaded pipeline the language model is separate from the neural decoder, so an output word is hard to trace to specific electrodes. The Whisper Base and Parakeet EOU variants produce highly similar spatial saliency maps (Spearman rho = 0.941), sharing 54 of the top 64 most salient electrodes; ablating the top 64 electrodes raised per-sentence word error rate to 35.9% for Whisper Base versus 11.6% for a random set of the same size.

Perspective

The demonstrated scope is the public brain-to-text 2025 data (participant T15, 1450 test sentences) and brain-to-text 2024 data (participant T12, 1200 test sentences), with phone results from Core ML conversion and Swift inference on an iPhone 15 Pro Max; the authors propose deploying FETTUCCINE in closed-loop clinical research sessions as the next step, to test whether latency and reliability gains translate to live use.

The authors note that the word error rates, while competitive among end-to-end methods, are not the lowest on the benchmark (the top five submissions reach 2% or below); the models depend on previously recorded sessions, with roughly 5 to 10 sessions needed to reach usable word error rates; and all current variants are non-causal, so realizing the latency gains will require exploring causal or small-context-window backbones.