LLM s Are Zero-Shot Context-Aware Simultaneous Translators (2024)

Abstract: The advent of transformers has fueled progress in machine translation. More recently large language models (LLMs) have come to the spotlight thanks to their generality and strong performance in a wide range of language tasks, including translation. Here we show that open-source LLMs perform on par with or better than some state-of-the-art baselines in simultaneous machine translation (SiMT) tasks, zero-shot. We also demonstrate that injection of minimal background information, which is easy with an LLM, brings further performance gains, especially on challenging technical subject-matter. This highlights LLMs’ potential for building next generation of massively multilingual, context-aware and terminologically accurate SiMT systems that require no resource-intensive training or fine-tuning.

## introduction in simultaneous translation, the translator -either a machine or human -is expected to start the translation before the source sentence is finished, often making strong assumptions about the meaning of certain words, phrases, or the intent of the entire message. to produce a coherent -although not necessarily accurate -translation, human simultaneous translators routinely use a range of techniques, one of which is delaying the translation of an initially ambiguous word or phrase in the hope that its meaning will become resolved by later context @xcite @xcite . perhaps more importantly, human translators reduce this inherent uncertainty by relying on information from other sources, such as presentation slides and glossaries of standard terms. this, and the fact that some people insist on using the term "interpreter", rather than "transla-tor" foot_0 , highlights a very different nature of this kind of translation. despite significant progress in the field of offline machine translation, recently enabled by the wide adoption of the transformer architecture @xcite , the practical use of simt systems is still limited due to a range of unsolved problems. one of these problems is that existing simt systems -in stark contrast to human simultaneous translators -operate on a sentence level, completely disregarding the context established by previous sentences, or the broader (extralinguistic) context that is implied, but not contained in the text itself. needless to say, such context-unaware translation is often logically incoherent and is prone to terminological inconsistencies, especially across long discourse. the very fact that human interpreters -even the most experienced professionalsroutinely prepare for upcoming translation jobs by studying relevant subject-matter, reviewing or compiling topic-specific glossaries of terms, names, and job titles @xcite @xcite , suggests that simt systems should have access to additional information needed to make terminologically appropriate and accurate translation. motivated by llms' strong reasoning @xcite @xcite , translation @xcite and in-context learning @xcite @xcite capabilities, we attempt to address one of the weaknesses of existing simt systems, namely that their translation takes no account of the wider context and generally cannot respect specific terminological constraints. different from previous studies which have attempted fine-tuning llms for simt tasks @xcite @xcite , our focus here is on translation in zero-shot mode. in the method we propose, the llm receives a prompt that contains both the partial input, partial translation and minimal background information, and generates the next word of the translation. at the next step, the prompt is updated with the new source and the newly translated word (see section 3 for details). we show empirically that such an approach outperforms some of the strongest bilingual simt baselines and shows competitive results to a state-of-the-art multilingual simt system. importantly, our approach makes it easy to insert background information (see fig. 1 and section 4), which helps the llm to make contextually appropriate word choices. our key contributions are as follows: the rest of the paper is structured as follows. in section 2 we provide an overview of recent simt literature. in section 3 we describe our method and the datasets used for evaluating our method. in section 4 we demonstrate the performance of our approach on the different datasets and language pairs. we conclude with a discussion of limitations and future directions and in section 5. ## related work simultaneous machine translation (simt) systems strive to balance translation quality -commonly evaluated using the bleu metric @xcite -with acceptable latency levels. this balance is managed through a "policy" that determines the timing of translation actions (i.e., a write action) versus the reception of additional input (i.e., a read action). the literature classifies these policies into two main types: fixed and adaptive @xcite . fixed policies, such as wait-k @xcite , apply predefined rules for executing read and write actions, regardless of the textual context. initially, simt models employed chunk-based strategies @xcite @xcite , where the text is divided into sub-sentence segments for translation without considering the context from preceding chunks, leading to reduced translation accuracy. in response to these drawbacks, @xcite introduced an incremental decoding method. this technique enhances chunk translations by integrating preceding contexts via the hidden states of an rnn. paired with straightforward segmentation tactics, their method surpassed the performance of prior state-of-the-art systems. meanwhile, adaptive policies, such as "wait-if" rules @xcite , allow for more flexible write/read actions by considering parts of the source and/or target text. adaptive policies can be developed using separately trained agents, often employing reinforcement learning techniques @xcite @xcite . these policies may initiate read/write actions based on model attention mechanisms @xcite @xcite or the stability of output predictions across n steps, a concept referred to as "local agreement" @xcite @xcite . recent research has also investigated policy training using binary search strategies @xcite to optimize the translation quality improvement per token processed, and has conceptualized the translation actions as a hidden markov transformer @xcite , where hidden events indicate optimal translation output times. a promising area of research, related to this study, focuses on adapting encoder-decoder transformers like mbart @xcite , initially developed for sentence-level translation, to the simt task. significant advances have been made in multilingual translation models @xcite , with some work focusing on creating more efficient versions of large models @xcite . for instance, @xcite ; @xcite have applied fine-tuning techniques using prefix-alignment data, while @xcite have employed fine-

@xmath0: {"topic": "medicine", "named_entities": [{"entity": "pvc", "definition": "premature ventricular contraction", "translation": "vorzeitige ventrikuläre kontraktion"}]}. based on the original english text, complete its translation into german.

@xmath1: vorzeitige llm (llama-3) 

@xmath2: {"topic": "medicine", "named_entities": [{"entity": "pvc", "definition": "premature ventricular contraction", "translation": "vorzeitige ventrikuläre kontraktion"}]}. based on the original english text, complete its translation into german.

@xmath3: given the prompt, the llm's generates tokens until either a new full word @xmath4. if a new full word is generated, a write action is performed: a new source word from the word buffer and the newly generated word ("vorzeitige" in this example) are added to the prompt. @xmath5, a read action is performed: the prompt is updated only with a new source word from the buffer. distinct from previous literature, we show that an off-the-shelf instruction-tuned llm can perform simt zero-shot, eliminating the need for resource-intensive model training and the complexities of making special datasets and fine-tuning. importantly, our approach enables context-aware simt which, as we empirically demonstrate, substantially improves translation quality. ## limitations and future directions prior work has demonstrated that fine-tuning on a small dataset is sufficient to enable an llm to perform the challenging task of simultaneous translation. however, these existing approaches are potentially limited to one language pair, involve constructing a specialized dataset and a nontrivial search for optimal fine-tuning hyperparameters. here we demonstrate the an off in the future, as stronger and more lightweight models become available, the llm can analyze its own translations and/or summarize source sentences or paragraphs. these summaries could be added to a vector store or a graph database and retrieved in real time to augment the translation of future sentences. the big performance gap between the 8b and 70b version of llama-3 suggests that even better translation quality could be achieved with larger closed-source models (such as gpt-4 or claude) if their apis allowed response priming. one practical limitation of our approach is that currently, to the best of our knowledge, it cannot be used with strong closed-source models that are available through api. perhaps as a countermeasure against model jailbreaking, the apis through which these instruction-tuned models (e.g. gpt-4, claude and gemini) can be accessed enforce a rigid prompt structure that is incompatible with response priming -specifying a user-specified prefix for the (assistant) model's response -which is at the core of our approach. another significant bottleneck in our llmbased simultaneous translation system is that it relies on a separate asr system that was not designed for online operation. although in general this cascaded setup works well, hallucinations sometimes occur, especially in low-latency regimes when in response to initial silence whisper outputs words that were never said in the audio. we believe this limitation can be addressed by implementing an end-to-end simt system, in which the output embeddings of an asr system or speech encoder would be directly projected into the llm's input embedding space, bypassing a text representation and improving the system's latency overall. in fact, there is already some work in this direction, e.g. by @xcite and @xcite . it is interesting to explore other ways to improve the performance and efficiency of our method, such as local agreement @xcite , efficient weight quantization (e.g. awq @xcite ), and more sophisticated prompting strategies.