Instance-Based Domain Adaptation for Improving Terminology Translation (2023)

Abstract: Terms are essential indicators of a domain, and domain term translation is dealt with priority in any translation workflow. Translation service providers who use machine translation (MT) expect term translation to be unambiguous and consistent with the context and domain in question. Although current state-of-the-art neural MT (NMT) models are able to produce high-quality translations for many languages, they are still not at the level required when it comes to translating domain-specific terms. This study presents a terminology-aware instance- based adaptation method for improving terminology translation in NMT. We conducted our experiments for French-to-English and found that our proposed approach achieves a statistically significant improvement over the baseline NMT system in translating domain-specific terms. Specifically, the translation of multi-word terms is improved by 6.7% compared to the strong baseline.

## introduction nmt @xcite has been the state-of-the-art in mt research and development for some time. fine-tuning nmt models usually requires specialised domain data for translating domain text @xcite . in recent times, large-scale pre-trained models (lptms) @xcite @xcite have gained significant attention due to their remarkable performance in various natural language processing (nlp) tasks. these models have proven effective in diverse applications, from information extraction to text generation. as a result, the nlp community is increasingly focused on harnessing their potential. one of the key advantages of lptms is that they often require smaller amounts of data for domain adaptation compared to traditional machine learning models @xcite . by leveraging pre-trained knowledge, lptms can be fine-tuned on specific domains with relatively limited data, making them a valuable resource for addressing domain-specific challenges in nlp. however, despite significant improvements in translation quality, nmt systems still struggle with translating terminology. even domain-adapted models are found to have difficulty with accurately translating domain-specific terminology @xcite . this paper proposes a simple yet effective instance-based fine-tuning approach based on terminology-aware mining. we tested our approach on the french-to-english terminology translation task foot_0 for covid-19 domain data. our findings show that the proposed approach helps improve terminology translation in covid-19 domain data. our in-depth analysis showed that adapting a single instance for a larger number of epochs helps improve the translation of domain-specific terms. the rest of this paper is organised as follows. section 2 discusses work related to our study. section 3 gives details about the data we used in our experiments. we describe our methodology in section 4. our nmt model is explained in section 5. experiments and results are covered in section 6. finally, section 7 summarises our work and discusses possible future research ideas. ## related work although nmt models have shown significant improvement in many translation tasks, translating terms of specific domains, such as medical or technical @xcite , still remains challenging for nmt. numerous methods have been proposed to improve term translation in nmt. these include (i) fine-tuning with domain-specific data: these help nmt models understand and translate domain-specific terms more effectively @xcite , (ii) data augmentation approaches, including generating synthetic data through back-translation or selftraining: these methods expose the nmt model to a variety of examples, ultimately enhancing term recognition and translation @xcite , (iii) incorporating external resources like glossaries, dictionaries, or terminology databases can assist nmt models in understanding and translating specialised terms more effectively @xcite , (iv) terminology injection during inference, using techniques like inline tags @xcite , source-target alignments @xcite , or fixed source positions @xcite for reference terms, helps produce translations with accurate domain-specific terminology, and (v) introducing auxiliary objectives during training such as predicting masked source terms or generating domain-specific inflections @xcite ) can handle domain-specific terms better during inference. standard nmt domain adaptation involves fine-tuning a generic nmt model using domainspecific data. accordingly, it is essential to consider factors such as similarity or distinct domain features that characterise the specialised field to effectively select the appropriate data. in their study, @xcite showed that fine-tuning a generic model using a sentence highly similar to the source-test sentence can improve the usage of domain-specific terminology after adaptation. likewise, @xcite conducted an experiment in which they fine-tuned a generic model on a small subset of bilingual training data acquired through a similarity search with the source test sentence. their findings also indicated an improvement in translation performance. in their experiments, both @xcite and @xcite showed how only a small set of sentences based on similarity to that of the test sentence is sufficient to improve the quality of translation. however, it is crucial that the sentences used for fine-tuning exhibit considerable similarity to the sentence being translated; otherwise, this can lead to a deterioration in translation quality. unlike @xcite and @xcite , who fine-tuned their models on fewer sentences for each test instance, @xcite took a different approach by employing n-gram matching for the entire test set. their study focused on matching and selecting n-grams from the training data which are most relevant to the entire test set rather than just individual sentences. by doing so, they were able to create a more comprehensive fine-tuning dataset, which in turn led to improved terminology translation. numerous studies have investigated ways to better incorporate technical terms into mt systems during inference. for example, @xcite added special tags to the source text sentence by identifying domain-specific terms. after translating, they found that these tags were correctly replaced with the appropriate terms in the target language. a similar approach was tried by @xcite , where they replaced specific phrases in the source text with pre-selected, domain-specific translations before translating. this made it easier for the system to use the correct domain-specific terms in the final translation. @xcite carried out a comparative analysis by experimenting with variations of inline terminology tags and discussed the optimal settings in the experiment that helped improve terminology translation. in their work, dougal and lonsdale (2020) added domain-specific terminology after the translation process as a post-processing step, replacing incorrect terms with approved ones using source-target alignments. this approach offers the benefit of not requiring the translation model to handle tags, so it could potentially be used to introduce terminology to mt system outputs. however, the effectiveness of this method relies on an effective alignment model. in their work, @xcite developed constraint-aware training data by randomly choosing phrases from the reference translation to serve as constraints and subsequently merging them into the source sentence with the help of a separation symbol. their method does not require alignments and solely depends on bilingual dictionaries during translation. they inserted the reference terminology at a fixed location in the source text, facilitating the model's learning of proper alignment. similarly, niehues (2021) also placed the reference terminology at a fixed point within the source text. however, his primary focus was on using the lemma of the term, which encouraged the model to learn the appropriate inflections for the given terminology. in their experiments, @xcite presented a technique that estimates the range of masked source terms during mt training, facilitating the integration of multi-word domain-specific terms in the translation process. they found that their models produced performance similar to that of @xcite in terms of single-word accuracy but improved performance when it came to translating multi-word terms. @xcite conducted an experiment in which they mined sentences from a large general domain corpus based on the presence of domain-specific terms in the test data. they then utilised the extracted data to fine-tune the model and observed improvements in terminology translation. similar experiments were carried out by @xcite , with their approach also demonstrating improvements in terminology translation. in our experiment, we employ an approach similar to that used by @xcite and @xcite . however, we take it further by performing extraction and adaptation for each instance in the test data as in @xcite . this means that, instead of using a predetermined set of sentences containing domain-specific terms, we adapt our model on a per-instance basis, allowing the model to better handle the domain-specific terminology in each test sentence. our proposed approach aims to provide a more tailored and flexible adaptation process, potentially resulting in more significant improvements in translation performance and domain-specific term management. ## dataset in our experiment, we used french-to-english parallel data from wmt2021, foot_1 which includes sources such as europarlv10, paracrawlv7.1, news commentary v16, un parallel corpus v1.0, commoncrawl corpus, and 10 9 french-english corpus. we combine these datasets, remove duplicates, and tokenise the text using moses @xcite 3 tokeniser scripts. the resulting dataset consists of 44m unique sentence pairs. the terminologies for french-to-english translation were obtained from the tico-19 project by anastasopoulos et al. (2020), foot_3 focusing on the covid-19 domain. there are 595 unique domain-specific terms, and the test set comprises a total of 2100 sentences. ## experiments and results we evaluated our mt systems using bleu @xcite , comet @xcite , nist @xcite and term count as our evaluation metrics. term count (tc) measures the number of occurrences of domain-specific terms accurately translated by the mt system. while the graphs for tc and bleu display a similar trend, it is crucial to understand that an increase in the bleu score does not necessarily indicate an improvement in terminology. in fact, alterations made to the adapted model might have led to improvement in the meta-language without directly translating to substantial improvements in the translation of domain-specific terms. in figure 3 , we plotted our mt systems' performance based on comet scores to analyse the relationship between the number of instances used for training and epochs. the x-axis represents the number of epochs, and the y-axis represents the comet scores. the lines with varying markers correspond to different sentence combinations. we see that the comet scores exhibit a different trend. when the number of sentences is increased, the translation quality measured by comet scores appears to improve. this contrasts with the trends observed in terms of tc and bleu. we also see that increasing the number of sentences did not consistently lead to betterment in translation quality. similarly, in figure 4 , we plotted the performance of our mt systems based on nist scores to analyse the relationship between the number of instances used for training and the number of epochs. we observe that increasing the number of epochs appears to benefit the quality of translation. furthermore, we observed that training a model with more instances and epochs yields better results. the discrepancy between the trends observed for three metrics (comet, bleu, nist and tc) could be attributed to the differences in the evaluation metrics. while tc, nist, and bleu scores focus on specific aspects of translation quality, such as the handling of domain-specific terminology and n-gram overlaps between the reference and the translation, the comet metric is designed to provide a more holistic assessment of translation quality by considering factors such as fluency, adequacy, and style. ## 117 2074 base adapted we produce a venn diagram to visually compare and better understand how terms are translated by the baseline and the best domain-adapted mt systems. we show the venn diagram in figure 5 . the diagram has two overlapping circles, showing the separate terminology counts produced by each of the mt models. the left circle, labeled "base", represents the baseline mt system and contains 101 terms. this area represents the unique terminology translation counts from the baseline model. the right circle, labelled "adapted", represents the best domainadapted model and contains 117 unique domain-specific terms. the area representing the overlap between both circles contains 2074 terms. this is shared terminology translation counts from both models. baseline mt in about 14% of cases, covid-19 causes a more severe condition requiring hospitalization, while the remaining 6% develop a serious form of the disease requiring intensive care. adapted mt in about 14% of the cases, covid-19 leads to more severe illness requiring hospitalization, while 6% of the remaining cases develop a serious form of serious illness requiring intensive care to further understand how the two models differ when it comes to the quality of terminology translation, we select an example sentence from the test set. in ## 44 258 base adapted we observed that the adapted mt system better handles the translation of multi-word terms. we show a venn diagram in figure 6 where the left circle labelled as "base" represents the baseline mt system and contains 25 multi-word terms. this indicates the unique terminology translation counts from the baseline model. the right circle labelled as "adapted" represents the best domain-adapted model and contains 44 unique multi-word domain-specific terms. the area reference mechanical ventilation becomes more complex as acute respiratory distress syndrome ( ards ) develops in covid-19 and oxygenation becomes increasingly difficult . baseline mt mechanical ventilation becomes more complex with the development of acute respiratory disorder syndrome (sdra) during covid-19 and oxygenation becomes more difficult. adapted mt mechanical ventilation becomes increasingly complex as acute respiratory distress syndrome (ards) develops in covid-19 and oxygenation becomes increasingly difficult. ## conclusion and future work this study presents a terminology-aware instance-based domain adaptation method. we tested our method for english-to-french translation. our results demonstrate that the proposed approach helps improve terminology translation. furthermore, we discover that increasing the number of sentences used for fine-tuning does not significantly impact the improvement of terminology translation performance. instead, a more efficient strategy appears to be one that considers a high number of epochs for a single sentence. this observation suggests that the model may benefit from more focused training, concentrating its learning efforts on a single sentence over an extended period (i.e., more epochs). we evaluated our mt systems using bleu, nist and comet evaluation metrics. we observe that the bleu metric correlates with the correct tc, while the comet metric shows improvements for the adapted model with an increased number of sentences. nist metric shows improvement for a higher number of instances and epochs. we also found that the adapted model outperformed the baseline when it comes to translating multi-word terms. our current proposed approach fine-tunes all instances, irrespective of whether a test instance requires fine-tuning or not, which may lead to the deterioration of translation quality for some sentences. in the future, we plan to identify those sentences that require fine-tuning and adapt only to them.