Simul- LLM : A Framework for Exploring High-Quality Simultaneous Translation with Large Language Models (2024)

Abstract: Large language models (LLMs) with billions of parameters and pretrained on massive amounts of data are now capable of near or better than state-of-the-art performance in a variety of downstream natural language processing tasks. Neural machine translation (NMT) is one such task that LLMs have been applied to with great success. However, little research has focused on applying LLMs to the more difficult subset of NMT called simultaneous translation (SimulMT), where translation begins before the entire source context is available to the model. In this paper, we address key challenges facing LLMs fine-tuned for SimulMT, validate classical SimulMT concepts and practices in the context of LLMs, explore adapting LLMs that are fine-tuned for NMT to the task of SimulMT, and introduce Simul-LLM, the first open-source fine-tuning and evaluation pipeline development framework for LLMs focused on SimulMT.

## introduction modern large language models (llms) contain at least several billion and up to trillions of parameters and are remarkably capable across a wide range of tasks. pretrained on humongous amounts of unlabeled data, they have demonstrated incredible emergent capabilities. with minor prompt adjustments, such as including instructions and examples, llms are often capable of near state-of-the-art performance set by highly customized solutions. the performance of these models is further enhanced when fine-tuned for specialized downstream tasks, sometimes exceeding the performance of previously cutting-edge solutions. given their rapidly evolving capabilities, llms and their application have become a focused topic of research within nlp academia @xcite . one popular downstream task for llms is textto-text neural machine translation (nmt), which focuses on taking an input sequence in a given language and outputting a translation in another language. typically, the entire source context is available at the start of translation for nmt. a particularly challenging subset of nmt is known as simultaneous translation (simulmt), where the model begins translation without having access to the entire source sequence, and the translation progresses as the remaining source sequence is incrementally provided. for languages that are syntactically and structurally similar, near-nmt performance is fairly achievable, but for language pairs that differ significantly in structure, traditional models struggle to balance high-quality translations with delay for additional source context. this balance is typically achieved via a fixed or adaptive read-write schedule, with one of the most popular and longstanding fixed schedules being the wait-k policy @xcite , where the target translation hypothesis lags behind the incrementally available source sequence by k words or subwords. while llms have been applied to and studied actively in nmt, their application to simultaneous translation has been lagging. this is in part due to a few challenges llms face when applied to simulmt that are non-trivial to address. first and foremost, it is unclear how well llms, which are pretrained and usually fine-tuned under the assumption that the prompt is completely provided and static before generation, will adapt to an application space where the prompt dynamically changes as the simultaneous scheduler elects to read from the source sequence. second, multiple approaches exist to enable llms for simulmt and it is challenging to intuit which approach will perform best. for example, one could adapt llms fine-tuned for nmt (hereafter referred to as nmt llms) to simulmt during inference, although how well such models will deal with the source context availabil-ity mismatch between fine-tuning (full sentence) and inference (partial sentence) is nebulous. alternatively, one could fine-tune llms directly for simulmt (hereafter referred to as simulmt llms), but new prompt structuring is likely needed to match inference simulmt behavior during finetuning exactly. finally, it is also unclear how well previously understood concepts in existing simulmt work, such as higher fine-tuning waitk values increasing generalizability, will apply to simulmt llms. this paper seeks to address the above problems and contributes to the process of applying llms to simulmt in the following major ways: ## background and motivation while a range of work is relevant to this paper, we will only provide a focused and high level review of large language models applied towards machine translation and simultaneous translation as an application space. readers interested in additional details should engage further with cited works in these areas. ## simul-llm: an open-source simulmt llm fine-tuning framework simulmt is an underexplored application space for llms, at the moment, and there is plenty of room to improve performance. to facilitate the rapid development of solutions for simulmt llms (fine-tuned for simulmt) or nmt llms (finetuned for nmt) adapted for simulmt, we choose to develop and provide an open-source framework written in pytorch for researchers to actively employ for future experiments. we call this framework simul-llm, and it bridges the gap between the development of fine-tuning agents via popular libraries and proper simulmt evaluation. simul-llm is poised to support both selective classical simulmt systems as well as fine-tuning and evaluation for a variety of llm systems such as falcon @xcite , llama @xcite and mistral-based @xcite models. the high-level components of simul-llm include a fine-tuning wrapper and a simulmt evaluation agent for every supported llm; in the case of classical models, only the evaluation agent is supported in-framework. ## adapting nmt llms to simulmt existing llms that have been fine-tuned for classical nmt may have the potential to be employed directly for simulmt inference. this can be desirable under circumstances where a single deployed model is preferable to multiple, specialized models (e.g. avoiding fine-tuning costs for multiple models). however, exactly how to adapt such models is unclear in practice given the differences between prompts during nmt fine-tuning (full-sentence availability) and simulmt inference (incremental source availability). this is especially problematic when engaging with short wait-k schemes and similar low-lagging schedules, where minimal source context is available during early translation steps. to intuit why this is an issue, suppose that an nmt llm is accustomed to receiving the entire source sequence x before outputting a word or token y 1 . if engaging in a low-lagging wait-k where |x| >> k, then the output y 1 can now only be based on source context up to x k . assuming the nmt llm would typically rely on some source context x i where @xmath0, then critical information is missing from its translation decision. we have conducted preliminary quantitative studies on this front by employing the proposed simul-llm framework, and these results are presented in section 7. ## prompt structure for simulmt llms alternative to adapting nmt llms to simulmt tasks, the simulmt llm approach aims to finetune llms directly for simulmt, which requires new prompt structuring. unlike classical encoderdecoder models for translation, source context for an llm must be packaged within a prompt and structured appropriately. some existing work has studied prompt structures for typical nmt @xcite , but it is unclear whether such prompts are still optimal for simulmt. this is because significant differences exist in source context availability between finetuning and inference, as discussed in section 4. to address this need, we propose a new prompting approach to construct the dataset for fine-tuning and evaluation of simulmt llms. instead of structuring each source-target sentence pair as a single example as typically done for nmt, we propose decomposing and expanding every sentence pair into a number of examples, where each example incrementally provides additional source and target tokens. when employed, the expanded examples directly mimic the behavior of simultaneous translation. we investigate two specific structures of materializing this prompt structuring approach, as analyzed and compared below. ## evaluation methodology to validate our proposed solutions to the aforementioned challenges and to test the capabilities of simul-llm as the first simulmt llm opensource framework, we engaged in several experiments allowing for comparisons among classical non-llm-based nmt @xcite / simulmt architectures @xcite , nmt llms adapted for simulmt, and simulmt llms. all mentioned llms are fine-tuned falcon-7b models, but simul-llm features an easy to extend framework and we support both llama and mistral-based models as well (results in appendix). nmt llms were fine-tuned for one epoch, as overfitting was observed beyond that point, which we intuit to be possibly due to the well-documented ability of llms to quickly memorize training sets @xcite . in contrast, simulmt llms were fine-tuned for either 2m random examples out of 5m examples (for wait-3 fine-tuning) or up to 5m examples (for wait-7 fine-tuning) on the expanded dataset due to computational constraints. ## conclusion in this work, we introduce simul-llm, the first open-source framework that enables rapid development of llm fine-tuning and evaluation pipelines for simultaneous machine translation (simulmt). simul-llm seamlessly integrates with the fine-tuning and generation tools of the popular transformers library as well as with simuleval, the preeminent simulmt evaluation framework. in addition to introducing simul-llm, we employ this framework to explore a wide range of topics in the llm simulmt space. our proposed simul-llm framework enables multiple lines of future work that can be carried out to understand and optimize llms for simultaneous translation, and it will likely be a useful tool for the research community.