Adaptive Machine Translation with Large Language Models (2023)

Abstract: Consistency is a key requirement of high-quality translation. It is especially important to adhere to pre-approved terminology and adapt to corrected translations in domain-specific projects. Machine translation (MT) has achieved significant progress in the area of domain adaptation. However, real-time adaptation remains challenging. Large-scale language models (LLMs) have recently shown interesting capabilities of in-context learning, where they learn to replicate certain input-output text generation patterns, without further fine-tuning. By feeding an LLM at inference time with a prompt that consists of a list of translation pairs, it can then simulate the domain and style characteristics. This work aims to investigate how we can utilize in-context learning to improve real-time adaptive MT. Our extensive experiments show promising results at translation time. For example, GPT-3.5 can adapt to a set of in-domain sentence pairs and/or terminology while translating a new sentence. We observe that the translation quality with few-shot in-context learning can surpass that of strong encoder-decoder MT systems, especially for high-resource languages. Moreover, we investigate whether we can combine MT from strong encoder-decoder models with fuzzy matches, which can further improve translation quality, especially for less supported languages. We conduct our experiments across five diverse language pairs, namely English-to-Arabic (EN-AR), English-to-Chinese (EN-ZH), English-to-French (EN-FR), English-to-Kinyarwanda (EN-RW), and English-to-Spanish (EN-ES).

## introduction adaptive mt is a type of machine translation that utilizes feedback from users to improve the quality of the translations over time. feedback usually includes corrections to previous translations, terminology and style guides, as well as ratings of the quality of the translations. this can be particularly useful for domain-specific scenarios, where baseline mt systems may have insufficient relevant data to accurately translate certain terms or phrases. there are still several challenges to effectively incorporate user feedback into the translation process, especially at inference time. in this work, we use a relatively wide definition of adaptive mt to refer to learning from similar translations (fuzzy matches) found in approved translation memories (tms) on the fly (farajian et al., 2017; wuebker et al., 2018; peris and casacuberta, 2019; etchegoyhen et al., 2021), as well as realtime terminology-constrained mt (hokamp and liu, 2017; post and vilar, 2018; dinu et al., 2019; michon et al., 2020). autoregressive decoder-only llms, such as gpt-3 (brown et al., 2020; ouyang et al., 2022), bloom (bigscience workshop et al., 2022), palm (chowdhery et al., 2022), and llama (touvron et al., 2023) are trained to predict the next word given the previous context. during unsupervised pre-training, a language model develops a broad set of pattern recognition abilities. it then uses these abilities at inference time to rapidly recognize and adapt to the desired task. in their experiments, brown et al. (2020) use the term "in-context learning" to describe a scenario where a pre-trained language model at inference time learns to replicate certain input-output text generation patterns without further fine-tuning. they show that autoregressive llms such as gpt-3 can perform well on diverse tasks, through zero-shot, one-shot, and few-shot in-context learning without weight updates. instead of asking the model to directly perform a given task, the input can be augmented with relevant examples, which help the model adapt its output. the key idea of in-context learning is to learn from analogy. the model is expected to learn the pattern hidden in the demonstration and accordingly make better predictions (dong et al., 2022). previous researchers investigated using neural language models for mt through few-shot incontext learning (vilar et al., 2022) and even in zero-shot settings (wang et al., 2021). other researchers proposed using llms for generating synthetic domain-specific data for mt domain adaptation (moslem et al., 2022). recently, researchers (agrawal et al., 2022; zhang et al., 2023) confirmed the importance of in-context example selection for the quality of mt with llms. the main contribution of this paper is investigating the capabilities of llms such as gpt-3.5, gpt-4 (including chatgpt), and bloom for real-time adaptive mt through in-context learning. as illustrated by figure 1 , such llms can achieve better translation quality through adapting its output to adhere to the terminology and style used in previously approved translation pairs. in particular, we would like to understand the quality with which such models can perform the following tasks, without any further training: ## experimental setup in all our experiments, we use gpt-3.5 textdavinci-003 model via its official api. @xcite for parameters, we use top-p 1, with temperature 0.3 for the three translation tasks, and 0 for the terminology extraction task. @xcite for the maximum length of tokens, we observe that french and spanish tokens can be 3-4 times the number of english source words, while other languages can be longer. hence, we roughly choose a length multiplier value, which we set to 8 for arabic, 5 for chinese and kinyarwanda, and 4 for french and spanish. we used batch requests with a batch size of 20 segments. @xcite our scripts are publicly available. @xcite as we aim to simulate a document-level scenario where translators are required to adhere to a project's or client's tm, we use the domainspecific dataset, tico-19 (anastasopoulos et al., 2020), which includes 3070 unique segments. from now on, we will refer to it as the "context dataset". we focus on a range of languages with diverse scripts and amounts of resources, namely english as the source language, and arabic, chinese, french, kinyarwanda, and spanish as the target languages. ## adaptive mt with fuzzy matches in translation environments, similar approved translated segments are usually referred to as "fuzzy matches", and are stored in parallel datasets, known as translation memories (tms). @xcite researchers have investigated the possibilities of improving mt quality and consistency with fuzzy matches (knowles et al., 2018; bulte and tezcan, 2019; xu et al., 2020). incorporating fuzzy matches into the mt process can help the system generate more accurate translations, and try to ensure adherence to pre-approved terminology and preferred style requirements. in this set of experiments, we investigate the possibility of forcing the translation of a new sentence pair to adapt to fuzzy matches in the context dataset. to extract fuzzy matches, we use embedding similarity-based retrieval. previous researchers have shown that approaches that depend on embeddings to retrieve fuzzy matches can outperform those that use edit distance (hosseini et al., 2020; pham et al., 2020). to this end, we employ the paraphrase mining module from the sentence-transformers library (reimers and gurevych, 2019). we use the all-minilm-l6-v2 model because of its high accuracy and efficiency. @xcite for each sentence, we retrieve up to top k other sentences. we experiment with diverse values of 1 to 10 sentence(s) from the context dataset. @xcite ## gpt-3 vs encoder-decoder mt models in this section, we aim to compare evaluation results we obtained from various mt encoderdecoder transformer-based systems (vaswani et al., 2017) with those from gpt-3.5. to this end, we translate our context dataset with a range of open-source and commercial mt models, including deepl translate api, @xcite google cloud translation api, opus (tiedemann, 2020), foot_6 and nllb-200 (nllb team et al., 2022). we converted opus and nllb models to the ctranslate2 (klein et al., 2020) format with int8 quantization for efficiency. inference parameters include beam size 4 and max batch size 2024, on a gpu a100-sxm4-40gb (google colab pro). for tokenization, we used sentencepiece (kudo and richardson, 2018) with the source and target subword models provided for each opus model, and the multilingual model provided by nllb for tokenization. @xcite we observe that for high-resource languages, adaptive mt with fuzzy matches using gpt-3.5 few-shot in-context learning (cf. section 3) can outperform strong encoder-decoder mt systems. for the english-to-french and english-to-spanish language pairs, few-shot translation with gpt-3.5 incorporating only 5 fuzzy matches outperforms strong encoder-decoder mt models, as demonstrated by figure 2 . for english-to-chinese translation, only when we used 10 fuzzy matches could we achieve better results. however, for english-to-arabic and english-to-kinyarwanda translations, results were not on par with the other three language pairs. the results are detailed in among the popular adaptive encoder-decoder mt systems is modernmt. @xcite originally, the system adopted the instance-based adaptation approach proposed by farajian et al. (2017). to control our experiments with modernmt to match those with gpt-3.5 few-shot translation, we created a new tm for each segment to include only the top-10 fuzzy matches for this segment. ## incorporating encoder-decoder mt as we demonstrated in the previous section, encoder-decoder mt models have achieved high translation quality for several language pairs. nevertheless, adaptive mt with llm few-shot incontext learning can surpass such quality, especially for high-resource languages. in this section, we investigate whether we can utilize encoderdecoder mt models to further improve adaptive translation with gpt-3.5. in the next subsections, we study two scenarios: ## bilingual terminology extraction terminology extraction is the task of automatically defining domain-specific terms in a dataset. extracted terms are naturally used for building glossaries to help translators. furthermore, it is possible to improve mt performance through finding sentences that include these terms and fine-tuning the system with them (hu et al., 2019; haque et al., 2020). in this set of experiments, we ask gpt-3.5 to extract 5 bilingual terms from each sentence pair in the context dataset. for parameters, we use temperature 0 and top p 1. human evaluation was performed for arabic, french, 13 and spanish. we provided the evaluators with a random sample of 500 sentences and their extracted terms. they were asked to use a 0-1 scale @xcite we observe that the original english-to-french tico-19 dataset includes several misaligned translation pairs. this can negatively affect the quality of tasks using such sentences. that is why it is important to filter parallel datasets to remove possible misalignments. the evaluation sample has been manually refined to include only well-aligned translation pairs. automatic semantic filtering approaches can be applied to large datasets. to determine whether each source and target term were equivalent, and whether the extracted terms were actually in the sentence pair (relevant inflexions are acceptable). in several cases where the evaluators marked the extracted term pair with 0, the model had made up either the source, target, or both; although it might be correct, it was not in the provided sentence pair. in other cases, the extracted term was partial, sometimes due to reaching the maximum length of tokens. nevertheless, as ## terminology-constrained mt as observed in section 3, adding more fuzzy matches enhances in-context learning and hence improves translation quality. however, early in a real-world translation project, we might not have so many fuzzy matches. by incorporating domain-specific terminology, the system can produce translations that are more accurate and consistent with the terminology used in that field. in this section, we investigate integrating terms in the process when there are n fuzzy matches. for example, if we have only two fuzzy matches, we either extract terms from these similar sentences or from a glossary, and use those that match up to 5-gram phrases in the source sentence to be translated. in this work, we use the terminology extraction process elaborated in section 6. obviously, if a pre-approved glossary is available, it can be used instead. we investigate three scenarios: hence, it might be better to exclude shorter terms if they overlap with longer ones. @xcite in production workflows, linguists can be provided with translation alternatives with and without fuzzy matches and/or terminology to be able to use the best translation. alternatively, automatic quality estimation can be conducted to select the best translation. among interesting observations that human evaluation reveals is that in few-shot translation with fuzzy matches (even without terms), the number of successfully used terms is more than those in zero-shot translation. this can help enhance consistency with approved translations. moreover, incorporating glossary terms in a zero-shot prompt can result in quality gains comparable to those of few-shot translation with fuzzy matches. ## chatgpt at the time of writing this paper, openai has released new conversational models, publicly referred to as chatgpt. this range of models includes: gpt-3.5 turbo and gpt-4. in this section, we briefly investigate the translation capabilities of these models compared to gpt-3.5 davinci. generally, we observe that both of the new models solve some tokenization issues, especially for non-latin languages such as arabic. while gpt-3.5-turbo is more efficient than text-davinci-003, it shows comparable quality for both zero-shot and few-shot translation (with fuzzy matches). in this section, we compare gpt-3.5 to opensource multilingual models, namely bloom (bigscience workshop et al., 2022) and bloomz (muennighoff et al., 2022). while bloom is a general-purpose llm, bloomz belongs to a family of models capable of following human instructions in a zero-shot manner. we use bloom and bloomz via the hugging face's inference api. @xcite as mentioned in section 2, recommended (sampling) parameters for translation with gpt-3.5 are top-p 1 and temperature up to 0.3. for bloom, the same parameters are not good for translation. @xcite we found that "greedy search" achieves better results for bloom, which are reported in ## conclusion in this work, we conducted several experiments to assess the performance of gpt-3.5 across multiple translation tasks, namely adaptive mt using fuzzy matches (cf. section 3), mt post-editing (cf. section 5), terminology extraction (cf. section 6), and terminology-constrained mt (cf. section 7). moreover, we compared its translation quality with strong encoder-decoder mt systems. generally speaking, results obtained from these experiments are very promising. while some high-resource languages such as english-to-french, english-to-spanish and even english-to-chinese show excellent results, other languages have lower support either because they are low-resource languages such as english-to-kinyarwanda or because of issues in the gpt-3.5 tokenizer such as englishto-arabic. nevertheless, when we used gpt-3.5 for mt post-editing of the english-to-arabic translation obtained from opus, the quality significantly surpassed that obtained from both opus and google translation api. this means that different pipelines can be adopted in production for different language pairs, based on the level of support of these languages by an llm. furthermore, we briefly compared gpt-3.5 translation quality with open-source llms such as bloom and bloomz. in the future, we would like to expand our experiments with open-source llms to cover more aspects. for adaptive mt with fuzzy matches, it would be interesting to investigate dynamic few-shot example selection. for instance, instead of selecting 5 fuzzy matches for all sentences, only highquality fuzzy matches up to a certain similarity score are used. similarly, when incorporating glossary terms or mt outputs from other systems, only those with certain quality characteristics are utilized. this can potentially enhance performance gains. for terminology extraction, we would like to try "phrases" instead of "terms". this would generate longer strings. we would like to see the effect of using such longer phrases, especially for lowresource languages. this work mainly aims at understanding the quality and level of support that llms can achieve (out of the box) for a range of translation tasks across diverse language pairs. in the future, we might consider starting with fine-tuning the model, and then conducting similar experiments. this can be especially beneficial for low-resource languages and rare domains, and can help enhance quality and efficiency.