T rans LL a M a: LLM -based Simultaneous Translation System (2024)

Abstract: Decoder-only large language models (LLMs) have recently demonstrated impressive capabilities in text generation and reasoning. Nonetheless, they have limited applications in simultaneous machine translation (SiMT), currently dominated by encoder-decoder transformers. This study demonstrates that, after fine-tuning on a small dataset comprising causally aligned source and target sentence pairs, a pre-trained open-source LLM can control input segmentation directly by generating a special “wait” token. This obviates the need for a separate policy and enables the LLM to perform English-German and English-Russian SiMT tasks with BLEU scores that are comparable to those of specific state-of-the-art baselines. We also evaluated closed-source models such as GPT-4, which displayed encouraging results in performing the SiMT task without prior training (zero-shot), indicating a promising avenue for enhancing future SiMT systems.

## introduction unlike conventional sequential translation, in which the target text is produced after the end of the corresponding source sentence (or long phrase), in simultaneous machine translation (simt) the target text is produced with minimal delay, aiming for the best listener experience expected from professional conference interpreters. while recent years have seen tremendous progress in sentence-based machine translation, mainstream adoption of simt systems requires solving a range of technical problems. perhaps the most important of them is that, much like human conference interpreters, simt systems must make optimal decisions about when (rather than how) to translate. in particular, naively translating each source word immediately results in compromised target quality, given that the meaning of a source word often makes sense only in the context of later words. and while waiting until the end of a sentence might seem a viable solution, in practice it would introduce unacceptable delays between the source and target message. consequently, the development of an effective simt system necessitates striking a balance between these two opposite scenarios. existing approaches to maintaining an optimal quality-latency tradeoff in simt, conventionally called policies, fall into two broad categories: fixed and adaptive. the policy's role is to signal to a separately trained translation model when to produce a partial translation (aka write action @xcite ) based of the partial input; at other times the input, which represents either text chunks from an upstream asr system (in cascade simt systems) or speech embeddings (in end-to-end systems), is just read in (read action). while with a fixed policy @xcite @xcite , the decision to output translation is based on a simple heuristic, an adaptive policy @xcite @xcite can be implemented as a separately trained model, for example an agent trained with reinforcement learning (rl) @xcite . to the best of our knowledge, state-of-the-art simt systems use encoder-decoder transformer architectures in a sequence-to-sequence paradigm. however, as of writing this paper the largestand generally most expressive -language models are causal decoder-only architectures. we wanted to explore the utility of such models for simt tasks, focusing on the english-german and english-russian language pairs, and specifically if they can be harnessed with minimal engineering effort. inspired by the recent success of llms in translation @xcite , as well as by their agential capabilities @xcite @xcite ) -here we propose transllama, a policy-free simt system, in which an off-the-shelf pre-trained decoder-only llm is fine-tuned on a dataset of causally aligned source and target sentences. the causality of the source is guaranteed by inserting one or @xmath0. we call our model policy-free, because as a result of fine-tuning on a causally aligned dataset the llm becomes capable of deciding when to output translation and when to read in more of the source, without requiring a separate policy. at inference, the fine-tuned llm is prompted with part of a source sentence concatenated with its corresponding (partial) translation and outputs one or more target tokens until either a full new word or @xmath1, which signals for more words to be read in. when extended with an offthe-shelf asr model, in addition to text-to-text translation (t2tt), our system handles speech-tospeech translation (s2tt) tasks with quality (as measured by bleu score @xcite ) approaching that of some of the recently published baselines at comparable latencies. our main contributions are as follows: the rest of the paper is structured as follows. section 2 offers a brief overview of most recent simt literature. in section 3 we detail our system's architecture, fine-tuning data preparation and training procedure. in section 4 we showcase its performance on en-de and en-ru language directions. we discuss directions for future work in section 5 and limitations in section 6. ## related work simt systems aim to deliver the best translation quality, usually measured with bleu score @xcite , while keeping its latency at an acceptable level. this quality-latency trade-off is controlled by the "policy", which decides when to translate (i.e. perform a write action) and when to receive more input (i.e. perform a read action). the various policies described in the literature can be broadly categorized into fixed and adaptive @xcite . fixed policies (e.g, wait-k @xcite ) are simple rules that determine the timing and order of write and read actions irrespective of the context. early simt systems used chunk-based approaches @xcite @xcite , in which the input is split into sub-sentence phrases and translated independently of the previous chunk's context, which compromised translation quality. attempting to overcome this limitation, @xcite proposed an incremental decoding approach, in which chunk translations incorporate previous context encapsulated by an rnn's hidden states. they showed that coupled with a simple segmentation strategy, their approach outperformed existing state of the art. on the other hand, adaptive policies (e.g. wait-if rules @xcite ) make read/write actions more flexibly by taking account of the partial source and/or target. adaptive policies can be implemented as separately trained agents (e.g. with reinforcement learning) @xcite @xcite . in such policies, read/write actions can be taken based on attention @xcite @xcite , or stability of the model's outputs over n steps (so-called local agreement @xcite @xcite ). more recent studies have also explored training the policy with binary search @xcite aiming to maximize the gain in translation quality per each token read, or cast the problem of deciding when to translate as a hidden markov transformer @xcite , in which hidden events correspond to the times at which to output translation. another promising line of work, related to the present study, aims to fine-tune encoder-decoder transformers, such as mbart @xcite , originally pre-trained for sentence-level translation, for the simt task. for example, @xcite ; @xcite utilized fine-tuning on prefix-alignment data and @xcite on meaningful units, achieving compelling performance on some language pairs. finally, in the course of writing this paper, we became aware of two concurrent projects which explored the use of fine-tuned llms for simt in llm (llama2) <@xmath2: i like to [/inst] <@xmath3: read action generates tokens until new full word @xmath4: write action update the prompt with a new source word and the newly generated target @xmath5: i like to have [/inst] ich update the prompt with a new source word from the buffer 2 ) is built with k source words (@xmath6. when the buffer has 3 words, the initial prompt is fed into the llm, which generates output tokens until either @xmath7. then the prompt is updated with a new input ("have") and target ("ich") word (write action). finally, the updated prompt (4) is fed back into the llm. @xmath8, the prompt is only updated with a new source word from the buffer (read action). ## method although the llms we consider in this paper are designed to process only text input, we add an asr stage to enable it to also perform s2tt. thus, we follow a cascaded approach shown in fig. 1 . causal alignment. training simt models, including optimal segmentation policies, with direct supervision has remained a challenge @xcite ) due to at least three reasons: (1) word order inconsistencies between the source and target, (2) omissions of words from the target that were present in the source, and/or (3) additions of words to the target not explicitly present in the source, making it difficult to establish unambiguous correspondences between each source and target words. this is less of a problem for offline translation models, because they are trained with direct supervision on pairs of complete source and target sentences, and both during training and inference the entire source context is revealed. however, it is not immediately clear how to use direct supervision for the simt task, in which the model must begin translation based on partial context. nevertheless, we believe that direct supervision for the simt task is possible and propose a way to accomplish that with a causally aligned dataset. in such a dataset, a target word never appears before its corresponding (when such correspondence can be established) source word in time, which is defined as the number of words from the sentence start. in other words, in a causally aligned source-target sentence pair, source words are guaranteed to be causal relative to their corresponding target words. we illustrate this in fig. 2 . note that the causal alignment is not always perfect: due to the word length mismatch between the source and target, not all source words will have a corresponding target word, and vice versa, not every target word will have a corresponding word in the source. however, as we demonstrate below, fine-tuning an llm on such a causally aligned dataset enabled us to achieve results comparable to some state-of-the-art baselines. in order to causally align the source and target, we split each sentence using the word_tokenize function from the nltk package @xcite , treating punctuation marks as "words", then find the best correspondences between the source and target words with @xcite , and finally insert as @xmath9. if after alignment the ## results data. for supervised fine-tuning (sft), validation and testing, we used must-c v2.0 @xcite for english-to-german (en-de) and english-to-russian (en-ru) translation direction. we randomly selected 4000 sentences for training and 100 sentences for validation. however, since it is possible that the dataset that llama2 was pre-trained on and must-c v2.0 (including its validation and test set) might have overlapping content, we also compiled another test set, which we call ted-tst-2023. this test set has a similar content type (ted talks) and follows the same format as the original must-c v2.0, but only includes talks posted after february 2023. the dataset has two parts: 102 source-target pairs for en-de and 102 for en-ru language pair. unless indicated otherwise, we report the results obtained on this test set. t2tt. we first analyzed the t2tt performance ## conclusions and future directions we have shown that with minimal fine-tuning and without resorting to sophisticated training techniques (e.g. checkpoint averaging @xcite ), an off-the-shelf pre-trained llm can perform simultaneous translation and achieve encouraging results that rival some of the recent simt models. this opens interesting directions to be explored in future work, such as multilingual finetuning, self-instruct @xcite and human preference tuning @xcite . there are several reasons to believe that we are far from unlocking the full potential of llms for simt. first, we followed the practice -standard in the simt literature -of evaluating the model on individual sentences randomly sampled from congpt-3.5-turbo-0613 llama-13b-hf-chat llama-70b-hf-chat gpt-4-0613 5 10 15 20 25 bleu fukuda et al. (2023) figure 7: zero-shot s2tt performance or our approach compared with gpt-3.5 and gpt-4 on the en-de language pair on ted-tst-2023.