Skip to main content
Search

# Streaming Neural Audio Tokenizer

1 related articles

Subscribe to updates
  1. After compressing a speech encoder 2.8x, the distilled student stays within 1.9% relative WER of its teacher on five of six teacher-student pairs

    This work studies how to compress, by distillation, the streaming neural audio encoder (tokenizer) used by on-device System-wide Dictation on Apple devices, taking as the supervision target neither the discrete token nor the output distribution but the pre-quantizer latent the model actually consumes; only the student encoder is trained to regress the teacher's per-frame latent under a squared-error objective, with a single affine layer absorbing the teacher-student width mismatch, and because the target precedes both the quantizer and the language-model bridge, one recipe covers both token interfaces and applies both to a tokenizer pretrained alone and to one jointly trained with a language model; at 2.8x compression the distilled student stays within 1.

You’re all caught up