Conversational S imul MT : Efficient Simultaneous Translation with Large Language Models (2025)

Abstract: Simultaneous machine translation (SimulMT) presents a challenging trade-off between translation quality and latency. Recent studies have shown that LLMs can achieve good performance in SimulMT tasks. However, this often comes at the expense of high inference costs and latency. In this paper, we propose a conversational SimulMT framework to enhance the inference efficiency of LLM-based SimulMT through multi-turn-dialogue-based decoding where source and target chunks interleave in translation history, enabling the reuse of Key-Value cache. To adapt LLMs to the proposed conversational decoding, we create supervised fine-tuning training data by segmenting parallel sentences using an alignment tool and a novel augmentation technique to enhance generalization. Our experiments with Llama2-7b-chat on three SimulMT benchmarks demonstrate that the proposed method empowers the superiority of LLM in translation quality, meanwhile achieving comparable computational latency with specialized SimulMT models.

## introduction simultaneous machine translation (simulmt) systems provide real-time translation of text input stream @xcite . this task plays an important role in real-world applications, such as facilitating communication in online conferences and generating live subtitles with strict latency requirements. although large language models (llms) have shown the potentials in machine translation @xcite , their applications to simulmt is non-trivial, as they are not inherently designed for simultaneous decoding. recent works have attempted to adapt llms for simulmt with prefix fine-tuning, incremental decoding @xcite and learning to wait for more source 1 code, weights, and data will be released with publication. offline prompt conversational @xmath0. <@xmath1. <@xmath2. these works show llms, with careful promptengineering, could approach the performance of specialized simulmt models. however, high computational cost, slow inference, and high latency render these approaches impractical for realworld applications @xcite . this is primarily due to the use of offline prompting, where arriving source tokens are inserted at the end of the source sequence, disrupting the continuity of the translation history (figure 1 left). this prevents reusing cached target history states and requires recomputation of source and target representations. to mitigate this issue, we propose conversational prompt that resemble the multi-turn dialogue nature of llms. specifically, user inputs are treated as the source tokens to be read, while the llm's responses are considered the predicted target tokens to be written. in our conversational simulmt, newly arrived source form the current instruction, while previous source tokens and their translations are treated as conversation history (figure 1 right). this conversational prompt enables the reuse of key-value cache @xcite , as all content is appended incrementally without modifying the translation history. however, conversational simulmt poses new challenges for llms to comprehend the segmented source content and produce a coherent translation via multi-turn conversation. to adapt the llm to the conversational decoding format, we opt to perform supervised fine-tuning (sft) on the pretrained llm. but the challenge is the lack of the conversational simulmt data for sft. interleaving incomplete source and target segments in the dialogue history is unnatural (see figure 1 ). this code-switching style is exhibited in some languages @xcite ; however, it is the continuation rather than the translation of the previous content, making it challenging to leverage existing code-switched datasets for training. therefore, we propose to curate the training data by segmenting parallel sentence pairs into smaller chunks based on a transformation of the word alignments. the segmented chunks are further augmented to handle different latency requirements. experiments on three simulmt benchmarks demonstrate the effectiveness of our proposed conversational simulmt in balancing the trade-offs between accuracy, speed and flexibility to different latency requirements. compared to offline prompting, our method not only maintains strong performance, but also benefits from reduced latency. notably, our method attains similar decoding speed to the llm-based offlinemt. incremental decoding studies have explored adapting offlinemt models for simultaneous decoding by performing offline decoding on incrementally updated histories @xcite @xcite . this involves a chunk-wise read policy that reads n tokens per round and a write policy that commits stable partial translations using the longest common prefix (lcp) @xcite algorithm. lcp often causes high latency when candidates lack common initial tokens. relaxed agreement lcp (ralcp) @xcite was proposed to vote for accepting prefixes with candidate agreement above threshold γ. simulmt with llms since incremental decoding essentially repeats offline decoding, using offline-style translation prompts with llms is straightforward and aligns with their instructionfollowing capabilities @xcite . during each round, a source chunk is read and appended to source history x. llms generate translations using offline prompts as shown in figure 1 , which are then written to target history y. ## conversational simulmt while incremental decoding with offline prompt enables llms to perform simultaneous decoding, it faces high computational latency due to the insertion of newly arrived source tokens in the middle of the prompt, disrupting the reuse of cached target history states. in this section, we propose conversational prompts to improve the decoding efficiency and balance quality-latency trade-off. the flight data showed that the plane was flying at an altitude of 35,000 feet. chunk 2 input: flugzeug auch bei einem zweiten chunk 2 response: the plane was also flying during the second flight. ## conclusion this paper focuses on the feasibility of utilizing llm for simulmt. we found that leveraging the incremental-decoding framework with offline prompting leads to high computational latency, hindering the reuse of the key-value cache. to address this, we propose the conversational prompting which allows llms to conduct simulmt in a multi-turn dialogue manner. the approach significantly speeds up the inference and also preserves the quality superiority, enabling practical llmbased simulmt systems.