Simultaneous Machine Translation with Large Language Models (2024)

Abstract: Real-world simultaneous machine translation (SimulMT) systems face more challenges than just the quality-latency trade-off. They also need to address issues related to robustness with noisy input, processing long contexts, and flexibility for knowledge injection. These challenges demand models with strong language understanding and generation capabilities which may not often equipped by dedicated MT models. In this paper, we investigate the possibility of applying Large Language Models (LLM) to SimulMT tasks by using existing incremental-decoding methods with a newly proposed RALCP algorithm for latency reduction. We conducted experiments using the Llama2-7b-chat model on nine different languages from the MUST-C dataset. The results show that LLM outperforms dedicated MT models in terms of BLEU and LAAL metrics. Further analysis indicates that LLM has advantages in terms of tuning efficiency and robustness. However, it is important to note that the computational cost of LLM remains a significant obstacle to its application in SimulMT.

## introduction simultaneous machine translation (simulmt) is a highly challenging task, demanding both high quality and low latency @xcite , while also confronting various real-world challenges. since simulmt systems are typically part of a simultaneous speech translation (simulst) system cascaded with an automatic speech recognition (asr) module, these challenges include, but are not limited to: (i) asr outputs often contain errors, necessitating a degree of fault tolerance in the simulmt model @xcite ; (ii) simulmt is typically applied to nearly endless input streams, requiring translation content to maintain good contextual consistency (radford most existing work primarily focuses on building dedicated simulmt models and policies to find the optimal balance between quality and latency @xcite @xcite @xcite @xcite . some efforts have successfully transformed offline neural machine translation (nmt) models into simulmt models to avoid the high cost of training from scratch @xcite @xcite @xcite ), but they have not sufficiently explored the challenges mentioned above. recently, the rapid development of large language models (llms) has demonstrated their multitasking and multilingual capabilities, offering new solutions for many complex nlp tasks @xcite @xcite . research indicates that they also have certain advantages in offline translation tasks, specifically for highresource languages @xcite @xcite . therefore, it is natural to consider whether the powerful understanding and generation capabilities of llms can be leveraged to address the challenges in simulmt. however, applying llms to simulmt itself presents challenges, such as designing suitable read-write policies for llms and effectively handling incremental source and target states, along with their benefits or costs. therefore, in this paper, we pose two research questions: (1) whether we could effectively transform off-the-shelf opensource llms with light adjustments into simulmt models? and (2) whether llms' application in simulmt address some of the aforementioned challenges, and in doing so, are there any limitations? to address these questions, we first select the llama2-7b-chat @xcite as the backbone llm. then, considering the expensive training cost of llm, we choose to find an approach that could endue llm the ability of simultaneous decoding without training. thus, we design the "read-n & incremental decoding" policy based on the approach proposed in @xcite , namely the incrementaldecoding with local agreement (la), which could turn a sequence-to-sequence model that is trained specifically for offline decoding into a model supporting simultaneous decoding. furthermore, to address the high latency issue caused by the longest common prefix (lcp) algorithm used in the incremental decoding, we propose the relaxed agreement longest common prefix (ralcp) algorithm to improve the selection of candidates to write during incremental decoding, resulting in a significant reduction of latency. we then conduct experiments on nine language pairs from the must-c (gangi et al., 2019) dataset, comparing our llm with dedicated nmt models such as transformer @xcite . our findings indicate that llms can outperform dedicated mt models using exactly the same decoding policy. finally, we conduct a detailed analysis of different factors affecting the use of llm for simulmt, including its potential advantages (e.g. the improvement of data utilization efficiency, the robustness of noisy input) and limitations (e.g. the efficiency issue). our contributions can be summarized as follows: ## background simultaneous machine translation (simulmt) is a task requiring the mt model to return translation content with the incremental source context in a real-time manner. it can be formalized as a markov decision process (mdp), where the model can be considered as a policy function π. it receives the current state s t at a specific time step t, and returns an action: a @xmath0, where a @xmath1, w}. here, r represents continuing to read the source context, and w signifies the action to write the most recent translation segment. the state s t generally encompasses the history of the already read source text and the translated target text s @xmath2, t t j ⟩, where i and j are the length of the source and target history. therefore, we can use r(i + 1) to represent an action of reading one additional source token and use w(w, j + 1) to represent the writing of a token w. the update of state s t according to the action a t can be denoted as: large language model (llm) leverage autoregressive decoding to conduct unsupervised language modeling on extensive text corpora, which equips them with language understanding and generation capabilities. most llms nowadays are using the decoder-only transformer architecture @xcite composed of layers of self-attention and feed-forward blocks. in addition to unsupervised training, recent llms undergo supervised fine-tuning (sft) and reinforcement learning from human feedback (rlhf) to align their behavior with human preferences @xcite . this allows these models to perform various nlp tasks through conversational interactions. more specifically, users construct prompts that include instructions and context and prompt the model to generate responses containing the desired results. in our paper, we mainly use beam search instead of top-p sampling to acquire more stabilized translations. thus, we consider the calling of llms as a generative function g θ with the prompt x sequence and the beam size b as input and the response sequences y (for all beam candidates) as well as their probabilities pr as the return values: y, @xmath3, b). ## conclusion in this paper, we focus on exploring the feasibility of applying llm to simulmt. we initially transformed the llama2-7b-chat into a model that supports simultaneous translation using the existing incremental-decoding approach. we then introduced the ralcp algorithm to reduce inference latency. in our experiments, we found that the llm after sft could outperform the dedicated nmt model using the same decoding policy, showcasing the potential of llm in this task. additionally, we observed that llm exhibited a degree of robustness against noisy input and could offer effective improvements through supervised fine-tuning with limited data. however, we also identified that the computational overhead of llm is a significant challenge. in future work, we intend to propose policies more suitable for llm and further explore the possible applications of various llm capabilities in simulmt tasks.