S eq PO - S i MT : Sequential Policy Optimization for Simultaneous Machine Translation (2025)

Abstract: We present Sequential Policy Optimization for Simultaneous Machine Translation (SeqPO-SiMT), a new policy optimization framework that defines the simultaneous machine translation (SiMT) task as a sequential decision making problem, incorporating a tailored reward to enhance translation quality while reducing latency. In contrast to popular Reinforcement Learning from Human Feedback (RLHF) methods, such as PPO and DPO, which are typically applied in single-step tasks, SeqPO-SiMT effectively tackles the multi-step SiMT task. This intuitive framework allows the SiMT LLMs to simulate and refine the SiMT process using a tailored reward. We conduct experiments on six datasets from diverse domains for En → Zh and Zh → En SiMT tasks, demonstrating that SeqPO-SiMT consistently achieves significantly higher translation quality with lower latency. In particular, SeqPO-SiMT outperforms the supervised fine-tuning (SFT) model by 1.13 points in COMET, while reducing the Average Lagging by 6.17 in the NEWSTEST2021 En → Zh dataset. While SiMT operates with far less context than offline translation, the SiMT results of SeqPO-SiMT on 7B LLM surprisingly rival the offline translation of high-performing LLMs, including Qwen-2.5-7B-Instruct and LLaMA-3-8B-Instruct.

## introduction simultaneous machine translation (simt) has made huge progress by leveraging large language models (llms) @xcite . these approaches generally use partial translation data to finetune llms, enabling llms to translate partial source texts to target texts. however, the partial translation data are often generated using simple alignment tools, like heuristic methods @xcite or attention mechanisms @xcite , which may introduce noise and degrade performance. in parallel, reinforcement learning from human feedback (rlhf; @xcite has gained huge success in improving the performance of fine-tuned llms @xcite . rlhf is reward-driven and does not rely on partial translation data. applying these techniques to simt appears to be a promising approach for improving its performance. however, we find that traditional rlhf methods like ppo @xcite and dpo @xcite commonly work for a single-step process, while simt translates streaming inputs in a multistep manner. the multi-step dependency between the source and target in simt is complex. first, the source texts of simt are provided step by step, and each step's source may have ambiguous meanings that require subsequent context for clarification. for example, the first step's source (bark) in figure 1 is ambiguous, which requires subsequent source texts (tree) to clarify. second, previous translation results can influence the overall translation quality. for example, in the second example of figure 1 , misinterpreting "the bark of" leads to errors in the overall translations. we claim that traditional rlhf methods commonly used in single-step reveal deficiencies in modeling the complex dependence relations in the multi-step simt setting. to this end, we propose a new policy optimization method, sequential policy optimization (se-qpo), and apply it to the simt task. as shown in figure 2 , seqpo-simt defines simt as a sequential decision-making process. to simulate the simt process, we segment a full sentence into multiple chunks and feed these chunks to the llm sequentially. at each step, the model evaluates the new chunk and translation history to decide whether to translate or wait for subsequent context. after completing all the steps, we construct a tailed reward to assess the entire simt process according to quality and latency. finally, we optimize the simt process using policy gradient @xcite . seqpo-simt enables us to simulate the complex simt process as a multi-step decision problem and refine it based on quality and latency. to validate the effectiveness of seqpo-simt, we conduct experiments on extensive datasets for @xmath0, including formal spoken language, informal spoken language, specialized knowledge domains, and news articles. performance is measured using a range of comprehensive metrics. extensive experimental results demonstrate that seqpo-simt not only attains superior translation quality but also reduces latency. in low latency and high latency scenarios, the average comet scores of seqpo-simt are 1.3 and 1.25 points higher than those of the supervised fine-tuning (sft) method, respectively. in particular, seqpo-simt outperforms the sft model by 1.13 points in comet, while reducing the average lagging (al) by 6.17 in newstest2021 @xmath1. because the simt task has only a limited amount of contexts while offline translation utilizes the full context, simt is more challenging than offline translation. remarkably, the simt performance of seqpo-simt is comparable to the offline translation performance of highperforming open-source models, such as qwen-2.5-7b-instruct @xcite and llama-3-8b-instruct. these findings underscore the effectiveness of seqpo-simt. we summarize our contribution as follows: 1. we define the rlhf process of simt llms as a sequential decision-making process to model the complex dependencies among steps in simt. ## sequential policy optimization seqpo-simt is a policy optimization framework that defines the simt task as a sequential decision making problem. it incorporates a tailored reward to enhance translation quality while reducing latency. this section first defines the basic components of seqpo-simt: environment and policy. then we describe the multi-step simt data sampling process. finally, we describe how we optimize the policy model. the model architecture is illustrated in figure 2 , and the algorithm is described in algorithm 1. for details on notations, please refer to ## experimental setup models and training data. we focus on chinese and english, a language pair with significant structural differences. for the @xmath2, we utilize wenet @xcite and wmt19 @xcite as our training data. for the @xmath3, we employ gigast @xcite as the training data. we use qwen-2.5-7b @xcite as our backbone. because the base llm lacks simt capabilities, we initially construct a dataset for warm-up, the process is similar to the previous works @xcite . specifically, we randomly select 40,000 data samples from the training datasets and construct partial translation data by prompting llms. details about the sft data construction process is shown in appendix a.2. ## conclusion in this work, we introduce sequential policy optimization for simultaneous machine translation (seqpo-simt), a novel framework that defines the simt task as a sequential decision-making problem. specifically, we conduct multi-step simt data sampling processes and optimize according to quality and latency. this intuitive framework allows the simt model to effectively refine the translation quality and reduce latency. we conduct extensive experiments on six datasets from the diverse domains for @xmath4. experimental results demonstrate that seqpo-simt consistently achieves significantly higher translation quality with lower latency. moreover, seqpo-simt achieves comparable translation quality as high-performing offline translation models, such as qwen-2.5-7b-instruct and llama-3-8b-instruct.