Domain Terminology Integration into Machine Translation: Leveraging Large Language Models (2023)

Abstract: This paper discusses the methods that we used for our submissions to the WMT 2023 Terminology Shared Task for German-to-English (DE-EN), English-to-Czech (EN-CS), and Chinese-to-English (ZH-EN) language pairs. The task aims to advance machine translation (MT) by challenging participants to develop systems that accurately translate technical terms, ultimately enhancing communication and understanding in specialised domains. To this end, we conduct experiments that utilise large language models (LLMs) for two purposes: generating synthetic bilingual terminology-based data, and post-editing translations generated by an MT model through incorporating pre-approved terms. Our system employs a four-step process: (i) using an LLM to generate bilingual synthetic data based on the provided terminology, (ii) fine-tuning a generic encoder-decoder MT model, with a mix of the terminology-based synthetic data generated in the first step and a randomly sampled portion of the original generic training data, (iii) generating translations with the fine-tuned MT model, and (iv) finally, leveraging an LLM for terminology-constrained automatic post-editing of the translations that do not include the required terms. The results demonstrate the effectiveness of our proposed approach in improving the integration of pre-approved terms into translations. The number of terms incorporated into the translations of the blind dataset increases from an average of 36.67% with the generic model to an average of 72.88% by the end of the process. In other words, successful utilisation of terms nearly doubles across the three language pairs.

## introduction the primary goal of the wmt 2023 terminology shared task is to evaluate the ability of mt systems to accurately translate technical terminology. the task aims to assess the extent to which mt models can utilise additional information regarding the translation of terminology. the shared task requires the participants to provide three translations, one without terms and the others with two individual sets of terms. there have been several advancements in the area of mt domain adaptation, where an mt model is expected to follow the style and terminology of a certain domain or client @xcite . moreover, some researchers give special focus to terminology while training and fine-tuning mt systems @xcite @xcite @xcite . however, forcing an mt model to adhere to certain terminology at inference time is among the most challenging aspects of mt. hence, several researchers have investigated approaches to terminology-constrained decoding at translation time @xcite @xcite @xcite . the goal is to ensure that the mt system can accommodate unseen terminology while retaining translation accuracy and fluency. recently, since the emergence of advanced llms such as @xcite , bloom @xcite , palm @xcite , falcon @xcite , llama 2 @xcite , and jais @xcite to mention just a few, researchers have been exploring the capabilities of these models for a number of tasks including @xcite @xcite @xcite . some work investigates whether it is possible to utilise llms for terminology-constrained mt using a pre-defined glossary @xcite or even a dictionary @xcite . they found the approach is generally effective in increasing the number of terms used in the translation, even for low-resource languages. we highlight our key contributions with the systems that we submitted for the wmt 2023 terminology shared task as follows: ## method in our submissions to the wmt 2023 terminology shared task, we followed these steps: ## evaluation to assess the effectiveness of our process, we conducted two types of evaluation: (i) term-level evaluation in order to measure the level of adherence to the required terminology, and (ii) sentence-level evaluation in order to see whether the process affected the quality of the overall translation. ## conclusion and future work in this work, we showed that applying a multistep process of mixed fine-tuning on terminology-based synthetic bilingual data and then terminologyconstrained automatic post-editing with an llm can increase the adherence to the pre-approved terms in the generated translations. by the end of the process, the use of the required terms has increased in the translations of the blind dataset across the three language pairs from an average of 36.67% with the baseline generic model to an average of 72.88% after instructing an llm to integrate the required terms into the translations. due to the task restrictions, we had to fine-tune opus models only. we would like to experiment with fine-tuning nllb models, and probably the new @xcite , and madlad-400 models @xcite , on the same data and compare the output quality. in our experiments, we employed chatgpt "gpt-3.5-turbo" for both terminology-based synthetic data generation and terminology-constrained automatic post-editing, as it is a relatively efficient and cost-effective option. in the future, we would like to repeat the same experiments with gpt-4 in order to assess the benefit of using a stronger language model on overall performance. we observe that bloom can be used as an alternative llm for data generation; however, one-shot generation might work better than zero-shot generation. in this case, the prompt can consist of a term, a bilingual sentence pair, and then another term. interestingly, the model will predict a new translation pair including the second term. while certain open-source models such as llama 2 and falcon might be employed for the terminology-constrained automatic post-editing step for certain languages, we suspect that they will need fine-tuning before being reliably usable for most languages. in future work, we will carry out a deeper analysis of the generated synthetic data together with the outputs of the fine-tuned models in order to understand how the properties of the synthetic data affect the fine-tuning results. it is important also to test the same approach for other languages, especially low-resource language pairs. moreover, it would be interesting to exclude the fine-tuning step and assess the overall translation quality after llm-based post-editing. it is possible that domain adaptation through fine-tuning the baseline mt model either on authentic or synthetic data would still be beneficial. it can lead to domain-specific improvements in the overall translation quality that may not be achievable by the baseline model or the terminology-constrained post-editing step. again, deploying a model finetuned on in-domain data into production can enhance terminology adherence in initial translations. as there is no need to send the translations that already include the pre-approved terms to the llm for terminology-constrained post-editing, this can reduce the number of translations that require postediting. such an efficient workflow can allow us to save resources, and minimise latency at inference time. similarly, there are potential advantages of employing an llm for post-editing rather than for direct translation. instead of solely relying on the translation quality of the llm, quality estimation can be performed to select the best mt model in general or for the current source text segment. ultimately, only segments that do not meet quality criteria are then passed to the llm for post-editing.