Skip to main content
Back to timeline
arXivSource publication:

An offline speech translator learns to interpret as it listens: SimPOTrans drops synthesized read/write trajectories and moves the quality-latency operating point by chunk size

Synopsis

SimPOTrans, from Nara Institute of Science and Technology, lets an offline speech translation model adapt itself to dialogue-style simultaneous translation through two-stage reinforcement-learning post-training, without synthesized read/write trajectories or explicit latency optimization. The first stage strengthens translation on complete utterances with a COMET reward; the second exposes the model to chunked partial speech and lets it learn from its own multi-turn trajectories what to translate and how at each turn. On the ACL 60/60 and MCIF long-form benchmarks across EnDe, EnZh and EnJa, it reaches a better quality-latency trade-off than several strong baselines on COMET and MetricX-24, and at inference the operating point moves by changing the speech chunk size.

Source-provided article image: SimPOTrans: Self-Adaptation for Offline-to-Simultaneous Speech Translation via Reinforcement Learning
arXiv

Interpretation

An offline speech translation model can acquire dialogue-style simultaneous behavior directly through reinforcement-learning post-training, without a purpose-built simultaneous training scheme. Earlier dialogue-style simultaneous translation methods relied either on synthesized read/write trajectories or on jointly optimizing translation quality and latency, two objectives that compete. On ACL 60/60, SimPOTrans beats baselines including SimulSense and InfiniSST on both COMET and MetricX-24 across latency regimes; those baselines were reproduced with the same training data and the same LLM backbone.

Training runs in two steps: first make translation on complete speech solid, then let the model generate multi-turn translations from chunked partial speech and learn from them. The second stage updates the model with Group Relative Policy Optimization (GRPO, a reinforcement-learning method that assigns reward by the relative quality of several sampled outputs for the same input), using a reward built from whole-utterance translation quality, word-order agreement with the source, and a penalty on repeated content between adjacent turns. Training uses CoVoST 2 in EnDe, EnZh and EnJa, about 430 hours of speech per direction; the backbone is a Whisper-large-v3 encoder, a two-layer MLP adaptor and Qwen3-8B, with the speech encoder frozen, the adaptor fully fine-tuned and the LLM adapted with LoRA during supervised fine-tuning, and only LoRA parameters updated in the second reinforcement-learning stage.

At inference no extra decision module is needed: speech is cut into fixed-duration chunks and translated chunk by chunk, and changing the chunk size moves the quality-latency operating point. Earlier inference-time policies depended on handcrafted rules or a separately trained decision module, while the model itself was not trained on partial speech input. On the MCIF development set used by the IWSLT 2026 simultaneous track, in the low-latency regime of at most 2 seconds LongYAAL, EnDe reaches COMET 0.87 at 1.6 seconds LongYAAL and EnZh reaches COMET 0.76 at 1.7 seconds.

Perspective

The demonstrated scope is speech-to-text simultaneous translation: quality-latency trade-offs on the ACL 60/60 and MCIF long-form benchmarks for EnDe, EnZh and EnJa, compared with LLM baselines reproduced on the same data and backbone. Inference cuts speech into fixed-duration chunks, and the dialogue context is reset at sentence-ending punctuation once retained speech exceeds a set duration.

Fixed-duration chunking may split semantically or acoustically coherent units, and the authors list boundary-aware chunking as future work. Evaluation covers only speech-to-text translation, with speech-to-speech extension also left as future work. BLEU is clearly lower than for baselines, which the authors explain as translations following source order more closely and departing from the reference wording rather than as a quality drop.