Cross-lingual neural fuzzy matching for exploiting target-language monolingual corpora in computer-aided translation (2022)

Abstract: Computer-aided translation (CAT) tools based on translation memories (MT) play a prominent role in the translation workflow of professional translators. However, the reduced availability of in-domain TMs, as compared to in-domain monolingual corpora, limits its adoption for a number of translation tasks. In this paper, we introduce a novel neural approach aimed at overcoming this limitation by exploiting not only TMs, but also in-domain target-language (TL) monolingual corpora, and still enabling a similar functionality to that offered by conventional TM-based CAT tools. Our approach relies on cross-lingual sentence embeddings to retrieve translation proposals from TL monolingual corpora, and on a neural model to estimate their post-editing effort. The paper presents an automatic evaluation of these techniques on four language pairs that shows that our approach can successfully exploit monolingual texts in a TM-based CAT environment, increasing the amount of useful translation proposals, and that our neural model for estimating the post-editing effort enables the combination of translation proposals obtained from monolingual corpora and from TMs in the usual way. A human evaluation performed on a single language pair confirms the results of the automatic evaluation and seems to indicate that the translation proposals retrieved with our approach are more useful than what the automatic evaluation shows.

## introduction despite the recent advances in machine translation (mt), translation memories (tm) play a prominent role in the translation workflows of language service providers, translation departments, and freelance translators; the 2020 edition of the annual european language industry survey @xcite reported tms to be the most popular resource among freelance translators and a relevant target of investment for companies and translation departments. even when the perceived quality of the translation proposals from tm and mt is quite similar, professional translators need more time to post-edit mt outputs than tm proposals @xcite . in addition, the use of tms foster terminology consistency and is especially suited to repetitive translation tasks as they allow for convenient recycling of previous translations. formally, a tm @xcite may be defined as a collection of translation units (tu), that is, source-target sentence pairs (s, t) which are mutual translations. they are conveniently exploited by computer-aided translation (cat) tools which, for each new sentence to be translated s , retrieve from the tm the subset of tus {(s, t)} whose source sentences s are most similar to s . translators then choose the tu that better fits their needs and make the appropriate changes to t, the target sentence in the selected tu, to convert it into t , an adequate translation of s . the similarity between s and the source sentence s in a tu is estimated by means of the fuzzy match score (fms) function, which is usually based on the word-based edit distance @xcite , and can be interpreted as the proportion of words common to s and s . the complement of the fms is also used as an estimation of the post-editing effort, i.e. the proportion of word-level edit operations to perform on t to get t . even though deep learning methods have been recently applied to tms, they have only been used to bias neural mt systems to produce translations similar to those in a tm @xcite , or to devise alternative ways of computing the fms between two source sentences @xcite . to the best of our knowledge, no approach has tried to overcome the main limitation of tms: the reduced availability, as compared to targetlanguage (tl) monolingual corpora, of in-domain tms for a given translation task. in this paper, we present an approach aimed at overcoming the aforementioned limitation by enabling the exploitation of in-domain tl monolingual corpora in cat tools and increase the chance of getting useful translation proposals. in such scenario, fms cannot be computed because the tl sentences in the monolingual corpus have no source-language counterpart. we circumvent this issue by using cross-lingual sentence embeddings @xcite @xcite to project t and s to the same space and comparing them by means of the cosine similarity. the resulting score, hereinafter neuromatch, allows us to obtain, for each s to be translated, a set of translation proposals from either the tm or the tl monolingual corpus. we then compute a form of fms, which we term as neurofms, with a comet-based @xcite model, so that it can be used to estimate the post-editing effort required to transform each translation proposal into t , the translation of s . this allows professional translators to set a threshold for discarding proposals that would require too much post-editing effort, as they would do with fms. we have conducted a thorough experimentation with four language pairs, in which we study the utility of translation proposals retrieved with the conventional fms function and with our neuromatch alternative. the results show that neuromatch successfully exploits tl monolingual corpora, similarly to the way fms exploits tms, and that it increases the amount of useful translation proposals in around 10 percentage points. our evaluation also confirms that our neurofms alternative has a performance comparable to the conventional fms, paving the way for an improved user experience, and enabling the combination and ranking of translation proposals obtained with both conventional fms and neuromatch. finally, a human evaluation on one of the language pairs, in which professional translators were asked to assess the usefulness of the translation proposals obtained with fms and neuromatch, confirms the results of the automatic evaluation. this human evaluation also seems to indicate that the translation proposals retrieved with our approach are more useful than what the automatic evaluation shows. the rest of the paper is organized as follows. next section introduces the neuromatch method for retrieving translation proposals from tl monolingual corpora, and the neurofms method for estimating a form of fms. section 3 then describes the experimental settings, whereas sections 4 and 5 present and discuss, respectively, the results of the automatic evaluation and of the human evaluation. section 6 briefly describes the most relevant works in the literature, and section 7 summarizes the main conclusions. finally, two additional sections discuss the limitations and ethical aspects of this work. ## methodology our approach relies on two components: a method to efficiently retrieve translation proposals from a tl monolingual corpus (neuromatch; sect. 2.1), and a method, more expensive from the computational point of view, to estimate a form of fms that can be used to estimate the post-editing effort of translation proposals (neurofms; sect. 2.2). ## experimental setting we evaluated our approach on four different language pairs, namely, english-spanish (en-es), english-german (en-de), english-czech (en-cs) and english-finnish (en-fi), by simulating the translation of the source sentences in a test set when using a tm and, when applicable, a tl monolingual corpus. ## results and discussion first, we discuss the results obtained when comparing the conventional, edit-distance based fms and the labse-based neuromatch (section 4.1). after that, we evaluate the fms estimation based on comet (neurofms; section 4.2). ## human evaluation to complement the results obtained with the automatic evaluation, an experiment was conducted with professional translators and a random sample of the en-es data used for the automatic evaluation. six professional translators were provided en-es en-de en-cs en-fi fms .44 .44 .41 .42 neuromatch .18 .19 .17 .19 neurofms out .16 .17 .15 .17 neurofms in .16 .16 .15 .15 oracle .13 .13 .11 .11 with a form containing a small collection of english sentences and two translation proposals in spanish: one retrieved from the tm using fms, and another retrieved from the tm plus the tl monolingual corpus using neuromatch. translators did not know the method used to obtain each translation proposal. they were asked to indicate which translation proposals they would use as a draft translation to be post-edited, or if they would rather translate the source sentence from scratch. in those cases in which they considered both translation proposals acceptable for post-editing, they could indicate if one of them would require less post-editing effort (they could also indicate that both were of similar quality). the data sets given to each translator contained about 350 instances, and were distributed in such a way that half of the instances provided to each an. 1 an. 2 an. 3 an. 4 an. 5 an. 0 .53 .71 .77 .73 .58 an. 1 -.47 .44 .69 .60 an. 2 --.68 .74 .60 an. 3 ---.65 .57 an. 4 ----.68 regarding the interanotator agreement, after analysing the assessments provided by the translators, neuromatch was considered the best choice for 856 instances ( 40% of the instances in the data set), while fms was the best choice for 388 instances ( 18% of the instances). both were considered equally useful for 132 instances ( 6% of the instances). these figures confirm the results of the automatic evaluation and seem to indicate that neuromatch could be more useful than what the automatic evaluation showed. these human annotations also allowed us to evaluate the scores obtained with neurofms when combining fms and neuromatch. we took the subset of instances for which one of the systems was considered better than the other (1,244 instances in total) and checked if neurofms and the professional translators agreed. in about 80% of the cases, neurofms and the human annotators agreed; this result is also in line with the automatic evaluation reported in previous section. ## related work in this section, we review the related work as regards the computation of sentence embeddings (section 6.1), and the exploitation of these embeddings in the context of tms (section 6.2). ## conclusions we have presented a novel neural approach to overcome the main limitation of tm-based cat tools: the reduced availability, as compared to tl monolingual corpora, of in-domain tms. our approach consists of two different neural components: neuromatch and neurofms. neuromatch uses cross-lingual sentence embeddings to search for translation proposals in tl monolingual corpora. neurofsm uses an ad-hoc comet model to estimate a form of fms between the source sentence to be translated and the translation proposals retrieved with neuromatch. we have extensively evaluated our approach on four different language pairs using an automatic evaluation metric, and found out that our approach is able to retrieve 10 percentage points more useful translation proposals than the conventional fms. in line with common practice by professional translators, we considered useful those translation proposals with a translation edit rate (ter) below 0.4. we also studied the performance of our approach with more restrictive definitions of usefulness, i.e. with ters of 0.3, 0.2 and 0.1, and in all cases neuromatch retrieves more useful translation proposals than fms. a human evaluation on one language pair confirms these results. as regards neurofms, it has shown a high correlation with fms and that when it is used to combine translation proposal obtained with fms and neuromatch, the ter of the set of sentences for which a translation proposal is found with either method is reduced. this last result opens the door to further improvements in professional translation productivity. both conventional fms and neuromatch can be seamlessly integrated: both types of translation proposals can be ranked together with neurofms, and professional translators can then use neurofms to estimate the postediting effort of the different translation proposals they are offered, as they do with fms.