Adding soft terminology constraints to pre-trained generic MT models by means of continued training (2024)

Abstract: This article describes an efficient method of adding terminology support to existing machine translation models. The training of the pre-trained models is continued with parallel data where strings identified as terms in the source language data have been annotated with the lemmas of the corresponding target terms. Evaluation using standard test sets and methods confirms that continued training from generic base models can produce term models that are competitive with models specifically trained as term models.

## introduction one of the major challenges of using machine translation (mt) to enhance the productivity of human translators in professional translation is enforcing the use of correct terminology in mt output. in general, a translator is expected to adhere either to standard domain-specific terminology, or to a client-specific terminology, which can be provided as a dedicated terminology database (usually referred to as a termbase) or implicitly in the form of a translation memory. in the professional translation setting, when the output of a mt system diverges from the specified terminology, a translator needs to correct the output manually, significantly reducing the utility of mt. it is therefore important that a translator has the capability of influencing the terminological choices that the mt system makes by providing terminology to the system. in this article, we introduce a method of adding support for enforcing user-provided terminology into existing mt models. the method is based on continued training of the model using data annotated with terminology information. ## model training we generate models with terminology support (term models) for multiple language pairs by continuing the training of generic base models with data annotated with terminology information. pre-trained models published as part of the tatoeba-challenge (tiedemann, 2020) 1 project are used as the base models for the continued training. tatoeba-challenge project includes mt models for hundreds of language pairs, and for many language pairs there are multiple models available. according to automatic evaluations performed on the tatoeba-challenge models, the models with the best performance are based on the transformerbig architecture. however, as we intend to deploy these terminology models for low-latency cpu inference on desktop computers, we use the transformer-base models, which generate translations much quicker. base model en-bg opus+bt-2021-04-13 en-da opus+bt-2021-04-13 en-de opus+bt-2021-04-13 en-es opus+bt-2021-04-10 en-et opus-2019-12-18 en-fi opustcv20210807+bt-2021-09-01 en-fr opus-2021-02-22 en-it opus+bt-2021-04-14 en-lt opus+bt-2021-04-14 en-nl opus+bt-2021-04-14 en-sv opus+bt-2021-04-14 fi-en opustcv20210807+bt-2021-08-25 for the experiments, we selected a subset of language pairs for which base models of reasonable quality (according to the published automatic metrics 2 ) were available. the selection includes 12 medium-and high-resource translation directions between different pairs of european languages. for some language pairs, such as english to estonian, transformer-base models are not available among tatoeba-challenge models, and models from the opus-mt model collection (tiedemann and thottingal, 2020) are used instead. all models have been trained on data that has been segmented with sentencepiece (kudo and richardson, 2018) (see the continued training is performed with mariannmt (junczys-dowmunt et al., 2018) using the default settings (v1.11.13). while adjusting hyperparameters, such as learning rate, might make the continued training more efficient, the initial automatic metric results indicated that the default settings were sufficient for the task, so we decided not to experiment with any hyperparameter adjustments. the duration of continued training was one epoch. ## evaluation there are three important aspects to the evaluation of nmt models with terminology support: ideally, a terminology model translates terms accurately, while maintaining an overall translation quality level comparable to the base model, both when translating sentences with terms or without them. this kind of model can be used independently, with no supplementary models. minimally, a terminology translation model has to have a reasonable level of term accuracy without causing the overall translation quality of sentences with terminology to degrade too much. a term model with this kind of minimal performance can still be useful, as long as it is used together with a generic back-off model that translates sentences without terms. ## energy use considerations training of nmt models consumes considerable amounts of energy. @xcite estimate that training a transformer-base model of the type used in our experiments consumes 27 kwh of energy. since we do not train from scratch but use continued training, the energy consumption of actual model training is considerably lower than the 27 kwh baseline. unfortunately, we could not track the exact energy consumption of the experiments due to the nature of the computing infrastructure that was used (shared dual gpu in a supercomputer, where energy measurement data of the gpu includes the data for other jobs running on the same dual gpu). based on the partial energy consumption data that we have recorded and the running times on jobs, we estimate that the continued training consumed approximately 0.35 kwh per model. while the energy consumption of the continued training is low, using stanza to annotate the training corpus with terminology information consumes significant amounts of energy. we estimate that the terminology annotation consumes around 5 kwh per language pair. the energy use could be minimized by switching to a less resourceintensive parser (such as spacy 11 ). while the energy consumption of bicleaner-ai is also significant, it is not included here, since we used publicly available pre-existing bicleaner-ai scores from the tatoeba-challenge project. 12 based on a survey by @xcite , the estimates above have been multiplied with a power usage effectiveness (pue) value of 1.58. ## conclusions the experiments described in this article demonstrate that continued training can be used to add soft terminology constraints to pre-trained generic mt models. automatic and manual evaluation of the model outputs clearly indicate that high levels of terminology accuracy can be achieved at a fraction of the energy cost of training a new model from scratch.