OPUS - CAT Terminology Systems for the WMT 23 Terminology Shared Task (2023)

Abstract: This paper describes the submission of the OPUS-CAT project to the WMT 2023 terminology shared task. We trained systems for all three language pairs included in the task. All systems were trained using the same training pipeline with identical methods. Support for terminology was implemented by using the currently popular method of annotating source language terms in the training data with the corresponding target language terms.

## introduction opus-cat @xcite is a collection of open source software consisting of a local neural machine translation (nmt) engine and plugins for computer-assisted translation (cat) tools, such as trados, memoq and omegat. opus-cat enables the use of nmt models trained in the opus-mt project @xcite in professional translation. as opus-cat is aimed at professional translators, it is designed to be integrated into normal translation workflows. multilingual term bases are one part of those workflows, so we have decided to implement a functionality for utilizing term bases in opus-cat. this paper describes the methods used in opus-cat for enforcing the use of terminology in machine translation output and the results of applying these methods to the data provided in the shared task. we trained new models for all three language pairs in the shared task. the shared task results were not available at the time of the submission of this paper. ## related work most published methods of constraining an nmt model to generate terminologically correct translations fall into three categories. ## model training the models were trained using a modified version of mozilla's firefox-translations-training foot_0 , an endto-end pipeline for building nmt models, based on the snakemake workflow management system @xcite . the pipeline loads, preprocesses, cleans and filters the training data, and trains and evaluates the nmt models. for this shared task, a terminology annotation workflow has been added to the pipeline foot_1 . ## observations on the shared task this year's terminology task differs in from realworld use of terminology in machine translation in two important aspects: in actual use cases, the nmt system would have to identify the source terms based on a lemma form provided in a term base, and only the lemma form of the target term would be available. the probability of the lemma term occurring as such in a correct translation is much lower than for the inflected term from a reference translation. the shared task is therefore much easier than the real-world task of translating with a term base. due to the use of inflected terms, the shared task also favours soft constraint models where the model is trained on surface forms of terms instead of lemma forms. because of this, the models we have submitted for the shared task all use surface forms of the terms. however, this will induce the models to learn a simple copy behaviour @xcite , instead of the more desirable copy-and-inflect behaviour @xcite . in our opus-cat production models, we intend to use lemma-based constraints, since we expect them to perform better in real-world scenarios, especially with morphologically complex target languages. ## models five different models were trained for each language pair. all of the models were trained with marian (junczys-dowmunt et al., 2018) using the transformer-big model architecture @xcite . for each language pair, a combined sentencepiece @xcite vocabulary (32,000 symbols, out of which ten symbols were reserved as potential term tags by using the user-defined symbol functionality of sentence-piece) was trained and used for both source and target languages. as transformer-big models are costly to train, a single base model was trained for each language pair using just the filtered corpus, and the base model was then fine-tuned with data that had been augmented with the terminological annotations. another motivation for using finetuning is the reuse of models: opus-cat uses the opus-mt model collection that contains thousands of pre-trained models, and fine-tuning those models to support the use of terminology instead of training terminology models from scratch saves time and resources. yet another advantage of fine-tuning is that it makes it possible to quickly test the performance of different term annotation schemes. as mentioned, we experimented with the append and replace methods. for both methods, two models were trained, one where the annotated sentences were combined with the unannotated sentences when fine-tuning (add), and one where the unannotated sentences were omitted (omit). the expectation is that the omit model will specialize better to term translation, while the add model will retain better generic translation capabilities. in production use it may be best to use a specialized term model when terms are detected in the source sentence, and revert back to a generic model when no terms are detected. the zh-en base model was trained until convergence (chrf validation metric did not improve for 20 consecutive validation steps). for the en-cs and de-en base models the training did not have time to converge before the deadline for shared task submission, but both models were trained sufficiently long to obtain competitive evaluation scores (on par with scores published for existing opus-mt models). the terminology models were trained by fine-tuning the base model with annotated data for one epoch. when translating with a terminology model and a term base, a script is used to identify terms in the source text and to annotate the terms in the source sentence before translation, using the same annotation scheme as in the training data. since the target side of the training data was not modified, the translation does not need to be post-processed. ## conclusion our submission for the shared task confirms that soft terminology constraint methods work with a variety of language pairs. we also demonstrate that soft constraint models can be created by fine-tuning base transformer models, which speeds up training and the investigation of different soft constraint methods and parameters. the results also indicate that fine-tuned soft constraint models have acceptable general translation quality, and do not require a back-off base model in production use.