Dynamic Terminology Integration for COVID -19 and Other Emerging Domains (2021)

Abstract: The majority of language domains require prudent use of terminology to ensure clarity and adequacy of information conveyed. While the correct use of terminology for some languages and domains can be achieved by adapting general-purpose MT systems on large volumes of in-domain parallel data, such quantities of domain-specific data are seldom available for less-resourced languages and niche domains. Furthermore, as exemplified by COVID-19 recently, no domain-specific parallel data is readily available for emerging domains. However, the gravity of this recent calamity created a high demand for reliable translation of critical information regarding pandemic and infection prevention. This work is part of WMT2021 Shared Task: Machine Translation using Terminologies, where we describe Tilde MT systems that are capable of dynamic terminology integration at the time of translation. Our systems achieve up to 94% COVID-19 term use accuracy on the test set of the EN-FR language pair without having access to any form of in-domain information during system training.

## introduction this work is part of wmt2021 shared task: machine translation using terminologies, which is concerned with improving machine translation (mt) accuracy and consistency on newly developed domains by utilising word and phrase-level terms. we describe tilde mt systems that are capable of dynamic terminology integration at inference time. our submissions consist of translations by terminology-enabled general-purpose mt systems for en-ru, en-fr, and cs-de translation *both authors have contributed equally. directions. our systems are deliberately trained without consideration for the test domain to follow the spirit of the shared task-mt for emerging domains. despite term collections being noisy, our mt systems with dynamic terminology integration improve term translation accuracy proving their usefulness in dynamic adaptation for novel domains, where training-time domain adaptation methods are not feasible. the remainder of this work describes the methods used for dynamic terminology integration (section 2) by describing the tasks of terminology filtering, term recognition, and dynamic terminology integration in the translation process. the bulk of section 2 describes problems due to the low-quality term collections and terminology mismanagement and our solutions to them. we hope that the examples provided will not only illustrate the selfimposed problems by the shared task but also will motive reconsidering the purpose and the desired qualities of a term collection in the context of mt. we then briefly describe the experimental setting and results in section 3 and section 4 respectively. we conclude our work with a broader discussion considering the shared task and terminology translation in mt in section 5. ## methods this section describes the three tasks necessary for successful mt with terminology: terminology filtering, term recognition, and finally, integration of terminology constraints in the translation process. ## experiments data. we use all parallel data provided for the shared task for training, except for development data which we use to choose the best model for final submission. although back-translated monolingual data could, in theory, improve the overall translation quality, we do not use it to train our systems because typically, the monolingual target data is selected based on its similarity with the target domain data. however, the scenario proposed for the shared task assumes that the domain is novel; thus, we aim to explore the merits of terminology translation and do not look for extra synthetic target domain data. ## results we trained one nmt system per translation direction and evaluated translation quality on the development sets using the terminology translation evaluation tool provided by the shared task foot_5 (ibn @xcite . we compare the baseline translation scenario where no terms are annotated in the source text with improved scenarios where terms are annotated using term collections acquired using the different filtering and term translation equivalent selection strategies. when analysing the lemmatized exact match accuracy, we must bear in mind that the evaluation data similarly to the term collections feature term entries 1) with more than one allowed synonymous translation equivalent (not counting different inflected forms), and 2) where different terms are merged into one entry (see examples in section 2.1). this consequently means that the evaluation procedure 1) allows terminological ambiguity on the target side, 2) does not allow analysing terminology translation consistency, and 3) may depict a rough estimate of terminology translation accuracy. therefore, we believe that the lemmatized exact match accuracy results should be analysed with a grain of salt. that being said, the results in we also see that the overall terminology translation quality is already relatively high for the baseline systems ranging from 76% for en-ru till 88.5% for en-fr. this makes us wonder whether the evaluated domain can be considered emerging as it features few novel terms and the majority are well handled by the baseline systems. to investigate further, we analysed whether the bilingual terminology that can be found in the development sets is also present in the training data of the nmt systems. we found that for cs-de, 97.9% and 92.5% of such (unique) bilingual terms are featured at least once or 10 times in the training data respectively. the numbers are even higher if we analyse running terms (tokens) -99.8% and 98.7% respectively. since the terminology for en-ru was human-created and not extracted from parallel data, it shows slightly lower results when analysing unique terms -93.5% and 88.3% respectively. however, the situation is similar to cs-de when analysing running terms -99.1% and 97.8% of bilingual terms found in the en-ru development data are also found in the training data at least once and 10 times respectively. based on these findings, we believe that the validation data does not depict an emerging domain and does not help analysing terminology translation quality for emerging domains. when analysing the overall translation quality (in terms of bleu), we see that term filtering using the idf-based filter is crucial when relying on very noisy and automatically acquired term collections (as was the case with the cs-de term collection). the results show that translation quality drops by 3 bleu points when using the unfiltered term collections. this shows that too general (and ambiguous) terminology can be harmful and lower translation quality. the overall translation quality change is marginal for the translation directions that featured human-created term collections (en-fr and en-ru), however we do see an increase in terminology translation accuracy. our final submission consists of machine translations of shared task test sets provided by generalpurpose mt systems that use dynamic terminology integration using tla @xcite . to translate our final submissions, term collections are filtered by basic filters (see section 2.1) for en-fr and en-ru language pairs, while for cs-de language pair, we also use @xmath0. we use the statistical word alignment term selection strategy for term entries with multiple translation equivalents for all language pairs. the development set results for the corresponding systems are marked bold in if translation with terminology is supposed to improve mt for novel domains, the term collections, being the supposed source of the expected improvement, have to be of a higher quality than the mt systems they are intended to improve.