Incorporating Target Fuzzy Matches into Neural Fuzzy Repair (2025)

Abstract: Neural fuzzy repair (NFR) is a simple implementation of retrieval-augmented translation (RAT), based on data augmentation. In NFR, a translation database is searched for translation examples where the source sentence is similar to the sentence being translated, and the target side of the example is concatenated with the source sentences. We experiment with introducing retrieval that is based on target similarity to NFR during training. The results of our experiments confirm that including target similarity matches during training supplements source similarity matches and leads to better translations at translation time.

## introduction retrieval-augmented translation (rat) is a family of machine translation (mt) approaches where an mt system has access to translation examples when generating a translation for a source sentence. the translation examples are usually retrieved from a translation database based on similarity with the current translation context, which can be either the source sentence alone or a combination of the source sentence and the translation that has been generated so far. the similarity between the translation context and the translation examples from the database can be measured using lexical methods, such as edit distance and longest matching n-gram, or based on the distance between the vector representations of the example and the translation context. the intuition behind rat is that the mt system can, given an unseen source sentence, use the retrieved matches as additional information when constructing a translation. this supports the translation task, as the mt system no longer has to rely solely on the informa-tion embodied in the neural network, and different rat methods have been shown conclusively to improve mt quality (see for example @xcite , @xcite ). this article focuses on a variant of rat based on augmenting data with lexical matches, first discussed in @xcite , called neural fuzzy repair (nfr). our work further develops nfr by incorporating translation examples that have been retrieved based on target instead of source similarity. we also test how annotating source sentences with the similarity levels of the translation examples affects quality. ## related work many rat approaches draw inspiration from retrieval methods that have been used in professional translation from the 1960s onward. the three main traditional forms of retrieval in professional translation @xcite are terminology lookup from a terminology database, full segment fuzzy match retrieval from a translation memory (usually based on edit distance), and concordance search from a translation memory (retrieving translation pairs based on the occurrence of a particular substring on the source side). in recent decades, various subsegmental retrieval methods have also been introduced @xcite . in mt research prior to the adoption of neural machine translation (nmt), the concept of retrieving translation examples based on source similarity and the construction of new translations from the retrieved examples was first proposed in the 1980s in the form of example-based mt @xcite . in statistical mt, retrieving parts of existing translations from translation tables in order to generate new translations was a core component of mt systems, and there were also attempts to integrate translation memory retrieval more directly in a manner resembling @xcite . within nmt, various rat methods have been proposed. they can be roughly divided into three categories, depending on whether they are based on purpose-built neural network architectures, data augmentation, or changes in the decoder component of the mt system. @xcite introduces the first nmt architecture designed for rat: translation examples are retrieved from a translation database based on sentence similarity, and the attention component of the mt system is extended to cover the retrieved examples. @xcite uses a similar architecture-based approach, but uses n-gramand vector-based retrieval to increase the amount of matches. @xcite attempts to control the source-match interactions by encoding each retrieved match separately with the source sentence. rat based on data augmentation was introduced in @xcite , where source sentences are concatenated with target translations from translation examples that are retrieved from the translation database with lexical matching. @xcite extends the lexical matching to separate relevant and irrelevant target tokens by using word alignment data, and also utilizes matches based on vector similarity. concatenation-based data augmentation methods are also used to constrain mt output to contain terms from a terminology database @xcite , which can be considered a form of rat. decoder-based rat has the advantage of being usable with any nmt model, since the model parameters and architecture are not changed. one early implementation utilized phrase tables from smt systems @xcite . currently, the most prominent form of decoder-based rat is knn-mt @xcite , which generates a datastore consisting of pairs of translation contexts and output tokens. when generating the next token of a translation, the decoder searches for similar translation contexts based on vector similarity, and utilizes the output tokens corresponding to the most similar translation contexts in generating the next token. neural rat has also been implemented with large language models (llm) @xcite using in-context learning (icl), where the llm is prompted with the retrieved examples. @xcite compare llm-based rat with nfr, and nfr seems to have a clear quality advantage, although more advanced llms may have better results. ## nfr with lexical matches in nfr, the source language sentences in the training data are concatenated with target language sentences. the concatenated target language sentences originate from translation examples, where the source sentence is similar to the source sentence in the training data by some similarity measure. the concatenated target language sentences are separated from each other and the source sentence with a special symbol, and maximum amount of examples per sentence is usually limited to 3 (see nfr has been implemented using both lexical and vector-based retrieval methods @xcite . it is easier to conceptualize with lexical retrieval methods, since there is a clear mechanism for utilizing the retrieved matches: find parts of the retrieved translation that match the parts of the new source sentence, and copy them to the new translation. note that this copy behaviour has to be selective in two ways: with vector-based retrieval methods, the mechanism for utilizing the matches is more murky, as there is often no lexical similarity with the retrieved translations and any acceptable translation for the new source sentence. @xcite found that using vector-based matches improves translation quality (although not by as much as lexical matches), and they hypothesize that vectorbased matches improve quality by providing context during translation. one issue, which is not explored in the existing research literature, is how a rat system actually learns to utilize the retrieved matches. ## data models are trained using the english to finnish data from the tatoeba-challenge data set (release v2023-09-26) @xcite . this data set consists of most of the data included in the opus corpus collection foot_0 at the date of the release. the data in opus includes many crawled data sets. due to quality issues in crawled data @xcite , the data is filtered with bicleaner ai v2.0 (zaragoza-bernabeu et al., 2022): 5 million best sentence pairs according to bicleaner ai are included in the training set (referred to as train-5m from here on). during the initial experiments, we noticed that even after bicleaner ai cleaning, much of the crawled data was of very low quality (containing for instance machine translations and lists of seo terms). the crawled data also contains many repetitive text templates, which occur hundreds of times with small changes, such as you can fly from [x] to [y] indirect via [z] or [word] pronunciation in @xcite . we suspected that the presence of these repetitive similar sentences in the training data (often with substandard translations) would affect the rat training adversely. because of these concerns, we decided to create another training set, which consists of 5 million best scoring non-crawled sentence pairs in the tatoeba-challenge data set (referred to as nc-train from here on). rat can be used for domain adaptation by using a domain-specific translation database for retrieval. in order to test the domain adaptation performance of our rat models, we exclude a portion of the tatoeba-challenge data set as domain test data. as there are no domain annotations included in the data, we treat each individual corpus in the dataset as a separate pseudo-domain and extract at most 1,000 sentence pairs from each of them as domain test sets. the corpora in the dataset mostly map to actual domains, e.g. the emea corpus contains data that is mostly from the pharmaceutical/medical domain. the crawled corpora are an exception, as they contain data from many domains, and they are therefore excluded from the domain test data. the domain test data is excluded from the training sets. each 5 million sentence pair training set is used as a database from which the translation examples are retrieved during the training phase for its respective training set. the training set database is also used as a translation database during testing. we also use a larger all-filtered database consisting of all of the tatoeba-challenge data with a bicleaner-ai score of at least 0.7 for testing. the all-filtered database is used to determine whether the rat system is capable of utilizing matches that it has not seen during training. for the domain-specific test sets, we also use domain-specific translation databases, which consist of all the domain-specific data in the all-filtered database. for the nc-train, the crawled data is excluded from the all-filtered database. ## models we trained several models in the english to finnish translation direction with both the train and nc-train datasets. all the models are standard transformer-base models and were trained with the marian nmt toolkit (junczys-dowmunt et al., 2018) v1.11.13 using default settings. we use sentencepiece @xcite to create a vocabulary of 50,000 symbols, which includes marker symbols for indicating different similarity classes. a shared vocabulary is used for both source and target to facilitate the copying of tokens from the examples to the translation. all the models were trained to convergence. the validation sets were selected from the development set included in the tatoeba-challenge data set by picking the longest sentences for which retrieved examples were available (the development set skews towards short sentences, which are problematic from the point of view of example retrieval). the validation sets were augmented using the same schemes that were used with the training data. this differs from test time, where only source similarity augmentation is used, but initial experiments indicated that using a different augmentation scheme for validation than the one used in the training data leads to unstable validation scores. ## evaluation the test sets which are commonly used for mt evaluation are a bad fit for rat evaluation, as they generally have very few fuzzy matches available even in large translation databases. for instance, for the flores-devtest, matches were found for only 72 out of 1,012 sentences in the all-filtered set. more matches are found for the wmt news test sets, but the news domain is otherwise not well suited for rat, as it is more varied and less repetitive than other domains. because of these concerns, we compiled our own test set. we extracted a maximum of 1,000 sentence pairs from each of the corpora that compose the tatoeba-challenge data set. we compiled separate test sets for the train (75,249 sentence pairs) and nc-train (65,549 sentence pairs) models. these test sets are mainly designed for domain translation performance evaluation, so we designate them as the domeval and domeval-nc. as the data has not been annotated with domain information, we use the sub-corpora as pseudodomains. for each sub-corpus, we build a fuzzy match index using all the sentence pairs from that subcorpus included in the respective all-filtered set. we generate augmented versions of the source sentences of the domeval sets using the subcorpus indexes, as well as the train and all-filtered indexes, and then translate the augmented domeval source sentences using a model. during test time, only examples retrieved based on source similarity are used, also with the models that were trained with target similarity, since target-side data would not be available in actual translation scenarios. ## discussion of the results all results are in accordance with earlier evaluations of nfr in @xcite and @xcite : nfr improves translation quality very significantly (up to 10 bleu points) compared to a nmt baseline. the domain translation results for the five domains with most retrieved translation examples (see the trg-sim scheme underperforms all other schemes on its own, probably due to excessive copying from the retrieved matches. however, models combining source and target similarity matches perform better than pure src-sim models. in domain-specific evaluation, the best results are ## conclusion and future work our experiments demonstrate that both adding target similarity matches to the training data, and ensembling trg-sim models with src-sim models improve the quality of translation output compared to normal nfr. in the future, we plan to extend the 2x-combosim approach by replicating source sentences with different source and target similarity matches in the training data at a larger scale. we also plan to experiment further on ensembling nfr models, including ensembles of models trained with different numbers of translation examples. ensembling may also offer an alternative way of handling multiple translation examples: a 1-example model can be provided with multiple translation examples as separate inputs, the outputs of which can then be ensembled to produce a translation that is influenced by all the examples. ensembling could also be used to combine terminology models @xcite and nfr models, by preparing separate inputs annotated with terminology and translation examples respectively, and ensembling the outputs.