Anticipating Future with Large Language Model for Simultaneous Machine Translation (2025)

Abstract: Simultaneous machine translation (SMT) takes streaming input utterances and incrementally produces target text. Existing SMT methods only use the partial utterance that has already arrived at the input and the generated hypothesis. Motivated by human interpreters’ technique to forecast future words before hearing them, we propose Translation by Anticipating Future (TAF), a method to improve translation quality while retaining low latency. Its core idea is to use a large language model (LLM) to predict future source words and opportunistically translate without introducing too much risk. We evaluate our TAF and multiple baselines of SMT on four language directions. Experiments show that TAF achieves the best translation quality-latency trade-off and outperforms the baselines by up to 5 BLEU points at the same latency (three words).

## introduction simultaneous machine translation (smt) aims to produce translations based on partial input from the source language, enabling real-time communication across language barriers @xcite . despite its potential, current smt methods often struggle to maintain high translation quality while achieving low latency. as shown in figure 1 , the translation quality of the previous state-of-the-art method sm2 @xcite drops quickly as the latency decreases. the major source of such quality drop is insufficient source information. to deal with insufficient information, human interpreters develop techniques to anticipate future source input to reduce interpretation lag @xcite . human interpreters often predict upcoming words, such as nouns and verbs, based on context, language structure, prior knowledge, familiarity with the topic, etc. most existing smt models ignore this technique or only implicitly use it @xcite @xcite @xcite . a large language model (llm) can predict the continuation of a source sentence given its prefix, mimicking this human-like anticipation @xcite . however, perfectly predicting the continuation of a sentence is challenging due to the versatility of human language. for instance, given the prefix "the cat is chasing", various completions like "a mouse", "a bird", or "a dog" are all plausible. this variability makes it essential to develop a mechanism that ensures the translation remains consistent with the actual source when relying on the predictions. in this work, we design a novel smt policy, translation by anticipating future (taf), that achieves high-quality translation at an extremely low latency. taf utilizes an llm to predict multiple possible continuations of the source input and translates each one with an mt model. it then employs a majority voting mechanism to select the prefix agreed upon by most candidates, ensuring consistency with source input. taf works on any combinations of pretrained mt models and llms without the need for further finetuning and can be further generalized to existing smt policies. experimental results demonstrate that taf consistently improves quality-latency trade-off over existing methods by up to 5 bleu score at a latency of 3 words on four language directions. we conduct an in-depth analysis of the behavior of taf to figure out the impact of llm predictions. finally, we find that providing the llm with longer context further reduces the latency without sacrificing the translation quality. ## related works recent advances in simultaneous machine translation majorly focus on its policy, either rule-based or learned adaptive. rule-based policies include wait-k and its variants @xcite @xcite , local agreement (la) @xcite , hold-n @xcite , ralcp @xcite etc. wait-k waits for k tokens at the beginning and then alternates between read and write actions. la generates a full hypothesis at each step and writes the longest common prefix (lcp) of recent hypotheses. hold-n removes the last n tokens of the full hypothesis and writes the rest. ralcp generates multiple hypotheses with beam search and outputs a common prefix that most hypotheses agree upon. though rule-based policies are easy to implement, learned adaptive policies demonstrate better quality-latency trade-offs. @xcite proposed monotonic multihead attention to model the policy. @xcite developed a generative framework with a latent variable to make decisions. @xcite quantified the information weight transported from source to target and made decisions based on the amount of information received. @xcite modeled the simultaneous translation process as a hidden markov model and optimized the likelihood of target sequence over multiple steps. @xcite further bridged the gap between smt models and offline mt models. @xcite is the state-ofthe-art adaptive policy that resolves the insufficient exploration issue of prior works by individually optimizing each source-target state. these learned adaptive policies implicitly model the future source input through prefix-to-prefix training. @xcite proposes to use a language model to explicitly predict one more source token and rewrite the translation if the prediction is wrong. however, rewriting makes it non-monotonic which limits its application to speech-tospeech scenarios. taf explicitly models the future source with an llm so that it is not restricted by the limited mt training data, and maintains the best translation quality at a low latency without rewriting using majority vote. also, taf requires no model training and works on any combination of mt models and llms, thus easy to implement. ## how taf impacts the translation we manually examine 100 instances in the en-zh direction with the llama2 as the llm and madlad-400-3b-mt as the mt model. we compare the trajectory between ralcp with and without taf using τ = 0.6. we find four major patterns where taf improves the quality-latency trade-off or hurts it. we show the frequency of these patterns in ## better prediction with longer context in real-world smt applications, such as multilingual conferences, speeches can last minutes or hours, allowing models to leverage the full context of prior speech. however, wmt datasets are pre-segmented into sentences. when evaluating smt on isolated sentences, llms cannot access the prior context of the document, making it challenging to anticipate future content accurately. for instance, given the partial source "regular physical activity can significantly reduce the risk of", the next phrase could refer to various diseases related to physical activity. however, if the prior context focuses on heart health, the continuation will likely be "heart disease". to better simulate real-world scenarios, we prepend the current partial source with previous sentences from the same document, allowing the llm to predict based on this extended context. before passing these predictions to the mt model, we remove the past sentences to ensure that translation remains sentence-level. this setup guarantees that the llm's context is the only variable being tested. results are shown in figure 8 . taf, when using a longer context, consistently reduces latency by 0.5 to 1 word while maintaining comparable translation quality on both wmt20 and wmt23 en-zh. since wmt23 test data is less likely to have appeared in llama2's training set, the improvement from taf may not be solely due to triggering the llm's memory. instead, the longer context could enable the llm to make more informed predictions about future content. ## conclusion we propose taf , a novel smt method that translates by anticipating future source input. experiments on four language directions show that taf achieves state-of-the-art quality-latency tradeoff and is universally effective on different combinations of pretrained mt models and llms. our manual analysis reveals how taf impacts the translation output. finally, we show that taf can be further improved with a longer context.