Implementing and Evaluating Multi-source Retrieval-Augmented Translation (2025)

Abstract: In recent years, neural machine translation (NMT) systems have been integrated with external databases with the aim of improving machine translation (MT) quality and enforcing domain-specific terminology and other conventions in the MT output. Most of the work in incorporating external knowledge with NMT has concentrated on integrating a single source of information, usually either a terminology database or a translation memory. However, in real-life translation scenarios, all relevant knowledge sources should be used in parallel. In this article, we evaluate different methods of integrating external knowledge from multiple sources in a single NMT system. In addition to training single models trained to utilize multiple kinds of information, we also ensemble models that have been trained to utilize a single type of information. We evaluate our models against state-of-the-art LLMs using an extensive purpose-built English to Finnish test suite.

## introduction most nmt systems receive as their input a source sentence on its own, without any additional context. this is problematic, as producing a correct translation often requires information that is external to the source sentence. for instance, source sentences that are translated as part of a larger document have to be consistent with other parts of the document. even if the translation of a sentence is not constrained by document context, the translation often needs to conform to terminological or phraseological conventions of a genre, domain, or a house style. beyond acting as a contextual constraint, external information may also improve translation quality by providing the nmt system with translation examples. these examples can simplify the task of translation, as the nmt system does not have to generate the translation from scratch, but can adapt the provided external information. one method of providing relevant external information to an nmt system is to query an external database for data based on the source sentence, and then either include the retrieved information as part of the nmt system input or constrain the decoding process based on the retrieved information. as similar information retrieval approaches used with large language models are called retrievalaugmented generation (rag) @xcite , we will refer to this family of methods as retrievalaugmented translation (rat), following @xcite . even though almost all published rat methods concentrate on a single kind of retrieved information (usually either terminology or translation memory matches), in actual practical translation scenarios all the kinds of retrieved information are used simultaneously. for instance, a human translator working in a computer-assisted translation (cat) tool will be provided with matches from both terminology database and translation memories, and they will need to make their translations conform with both of these information sources. in this article, we introduce several nmt systems, which can utilize different kinds of retrieved information when generating translations. we experiment with both single nmt models that are trained to utilize multiple kinds of information, and with systems that combine models that have been trained to utilize a single kind of information using a novel ensembling method called contrastive ensembling. we also compare our models with instruction-tuned llms, which have a native capability of utilizing multiple types of retrieved information. our models are trained to utilize two kinds of information: the motivation for including these two kinds of information is that they are routinely used in the cat tools that professional translators use. they represent the types of information that are readily available and have been found useful in real-life translation workflows @xcite . building the rat system around widely used types of information also ensures that it can easily be integrated into existing workflows. as mentioned, rat can be used in two ways: either as constraining the mt system to utilize the retrieved information in its output (especially in the case of terminology), or as providing contextual information to enable the generation of better translations. in this article, we are mainly concerned with constraining rat, as it has applications in professional translation. one of the problems in developing constraining rat systems is that the common mt evaluation methods, such as bleu and comet, are not well suited to evaluating them, as they provide no information on how well the retrieved information has been utilized. to help us develop and evaluate our models, we therefore compiled an extensive test suite, which contains test cases consisting of source sentences, terms, fuzzy matches, and tests that can be used to check whether the terms and fuzzy matches are used in translations. ## related work we structure our system around retrieval methods that have been used in professional translation since the 1960s. these methods have been developed gradually and organically within the translation industry, so their origins are often unclear. for background on fuzzy match and term retrieval, see @xcite . the first mt method that can be characterized as rat was example-based machine translation (ebmt) @xcite , where translations were generated based on examples retrieved from a translation database. statistical machine translation (smt) methods could also be characterized as a form of rat, as they rely on retrieving partial translations from a database of translation fragments, and retrieval was also more explicitly integrated into smt systems @xcite . in the context of nmt, one of the first methods recognizable as rat was introduced in @xcite , where fuzzy matches were retrieved from a tm and the attention component of the model was modified to cover the matches in addition to the source sentence. constraining nmt to adhere to retrieved terminology was first introduced in @xcite , where the beam search decoder is modified to always produce the specified target terms in the output. @xcite was the first to implement rat using data-based methods, by replacing sub-strings in the source sentences of the training data with equivalent target language sub-strings. @xcite introduced data-based rat for terminology and @xcite for fuzzy matches. while most work on rat has concentrated on a single kind of information, there has been some recent work on unified rat, where nmt systems can utilize multiple kinds of information. wang et al. (2023) prefix the inputs and outputs of their model with three different kinds of retrieved information: fuzzy matches, translation templates, and terms. @xcite fine-tune an nmt model with data augmented with many different kinds of instructions, some of which are similar to rat, such as an instruction to utilize a particular term in the translation. @xcite experiment with prompting an llm with both terms and fuzzy matches to generate adapted translations. what differentiates our approach from the previous implementations is that we focus exclusively on the types of information that are routinely used in cat tools, which enables easy integration with professional workflows. we also use an ensemble of specialized models in addition to a single model trained to utilize multiple types of information. the concept of generating translations using multiple different inputs originates from the field of multi-source translation @xcite , where the different inputs are equivalent source sentences in different languages. @xcite were the first to implement multi-source translation by ensembling different nmt models that are provided with different inputs, which resembles our ensembling method. for evaluation, we will use a dedicated test suite for the phenomena we want to tackle. test suites have been used for evaluating mt quality since at least the 1990s @xcite ), and they have become more popular in recent years (see for example @xcite , as the dramatically improving mt quality has created a demand for more granular evaluation methods. as far as we know, our test suite is the first suite designed specifically for rat evaluation. ## models we train a selection of models, including separate term and fuzzy match rat models, and unified rat models, which process both types of retrieved information. for term models we train models that can process a single term, and models which can process up to ten terms. for fuzzy models, we train models that can process a single fuzzy match, and models that can process up to three fuzzy matches. models are created with continued training using the tatoeba-challenge (tiedemann, 2020) models as the base models. as our test suite is english to finnish, we only train models in that language direction. for all models but one we use the standard transformer model opustcv20210807+bt-2021-09-01 as the base model. to see the effect of model size, we also train one model using a base model with the transformer-big architecture (opustcv20210807+news+bt_transformer-big_2023-04-13). continued training has many advantages compared to training the models from scratch: as continued training is much faster, it is easier to test different model variations and the carbon footprint of the training is smaller. the base models can also be used as strong baselines for evaluation, as they have been trained on all the available data from the opus corpus @xcite . our models use special symbols to separate the retrieved information from normal source text (see the left column in figure 1 for an example of how the symbols are used). the vocabularies of the base models do not have any spare symbols that can be used as these special symbols, so we need to re-purpose some of the existing symbols. we pick ten of the least common symbols from the vocabulary, and assign them as our special symbols. not all symbols are used in the experiments, but we reserve extra symbols in case more are needed in future experiments with the same models. as the vocabularies remain otherwise identical, we can easily ensemble the trained models with each other and with the base model. training is continued with a high-quality subset of the tatoeba-challenge dataset that was originally used to train the base models. ten million sentences are included in the continued training subset. the subset does not include data from crawled corpora due to quality problems associated with them @xcite . the data is also scored with bicleaner-ai @xcite , and sentence pairs scoring less than 0.7 are excluded from the subset. the duration of continued training is one epoch, and the learning rate is set to 0.00001 to prevent catastrophic forgetting @xcite . for both fuzzy and term models, the training set is annotated with the appropriate rat data for that type (see figure 1 for examples of the annotations). the models are trained using the marian nmt framework @xcite . ## multisource ensembling in addition to using a unified model capable of processing both terms and fuzzies, we ensemble models trained to utilize a single kind of information to produce a system that can utilize multiple kinds of information. each model in the ensemble has its own input, which is prefixed (full matches) or interleaved (terms) with appropriate retrieved information. see figure 1 for a schematic of the inference pipeline. the ensemble decoder is implemented by modifying the generation functionality in the huggingface transformers library foot_0 . we experiment with the following ensembling methods (see figure 2 for a visual example): open @xmath0: avaa huoltoluukku painamalla painiketta shared output tokens if next token @xmath1. first it allows us to utilize multiple kinds of retrieved information during inference, but it also acts as conventional ensembling, the purpose of which is to improve generic output quality. we also hypothesize that ensembling models trained for different rat methods and provided with different inputs during inference will enhance the quality improving effect of ensembling, as it has been shown @xcite ) that ensembling diverse models produces better results than ensembling similar models, such as different checkpoints of a single training run. ## evaluation rat systems can be used both for improving generic translation quality and for domain adaptation. when used to improve generic translation quality, it is not relevant whether the translations actually adhere to the terminology and phraseology used in the retrieved examples. for domain adaptation, however, adherence to the retrieved examples is important. we evaluate model performance from two points of view: correct utilization of the retrieved information in the translations, and general translation quality. for evaluating the correct utilization, we use our test suite, which is covered in detail below. the main purpose of general translation quality evaluation is to see whether continued training with rat-augmented data degrades generic transla-tion quality. measuring general translation quality for rat systems is complicated by the fact that rat systems are meant to be used with retrieved information: their performance when not provided with any such information is almost irrelevant, as any rat system can be paired in production with a back-off system that processes source sentences with no retrieved information. therefore evaluating rat systems with standard evaluation sets, which are not paired with retrieved information, does not provide much useful information about the primary use case for rat systems. because of these concerns, we use our test suite also as the generic quality test set, so that we can provide retrieved information to the systems. as the test suite does not contain any reference translations, we use the reference-free wmt23-cometkiwida-xl metric @xcite for evaluation. this metric has been shown @xcite to correlate reasonable well with human judgments and to perform better than many standard referencebased metrics such as bleu and chrf. ## conclusion our experiments demonstrate that combining multiple types of information in a rat system remains an open problem, even though llms show much promise also in this field. the main contributions of this paper are the introduction of contrastive ensembling and the dedicated, extensive test suite for evaluating rat. using the test suite we confirm that contrastive ensembling with separate term and fuzzy models provides better results than naive ensembling or single models that can process both terms and fuzzies. while contrastive ensembling does not perform as well as gemma-12b-it llm, its computational requirements are much lower, which makes it suitable to more use cases, such as local low-latency rat. we have made the training and evaluation pipeline foot_2 and the test suite foot_3 available under a permissive license.