Intelligent Translation Memory Matching and Retrieval with Sentence Encoders (2020)

Abstract: Matching and retrieving previously translated segments from the Translation Memory is a key functionality in Translation Memories systems. However this matching and retrieving process is still limited to algorithms based on edit distance which we have identified as a major drawback in Translation Memories systems. In this paper, we introduce sentence encoders to improve matching and retrieving process in Translation Memories systems - an effective and efficient solution to replace edit distance-based algorithms.

## introduction translation memories (tms) are "structured archives of past translations" which store pairs of corresponding text segments 1 in source and target languages known as "translation units" @xcite . tms are used during the translation process in order to reuse previously translated segments. the original idea of tms was proposed more than forty years ago when @xcite noticed that the translators working for the european commission were wasting valuable time by retranslating (parts of) texts that had already been translated before. he proposed the creation of a computerised storage of source and target texts which could easily improve the performance of translators and that could be part of a computerbased terminology system. based on this idea, c 2020 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. 1 segments are typically sentences, but there are implementations which consider longer or shorter units. many commercial tm systems appeared on the market in the early 1990s. since then the use of this particular technology has kept growing and recent studies show that it is used on regular basis by a large proportion of translators @xcite . translation memories systems help translators by continuously trying to provide them with socalled matches, which are translation proposals retrieved from its database. these matches are identified by comparing automatically the segment that has to be translated with all the segments stored in the database. there are three kinds of matches: exact, fuzzy and no matches. exact matches are found if the segment to be translated is identical to one stored in the tm. fuzzy matches are used in cases where it is possible to identify a segment which is similar enough to the one to be translated, and therefore, it is assumed that the translator will spend less time editing the translation retrieved from the database than translating the segment from scratch. no matches occur in cases where it is not possible to identify a fuzzy match (i.e. there is no segment similar enough to the one to be translated to be worth using its translation). tms distinguish between fuzzy matches and no matches by calculating the similarity between segments using a similarity measure and comparing it to a threshold. most of the existing tm systems rely on a variant of the edit distance as the similarity measure and consider a fuzzy match when the edit distance score is between 70% and 95%. 2 the main justification for using this measure is the fact that edit distance can be easily calculated, is fast, and is largely language independent. however, edit distance is unable to capture correctly the similarity between segments when different wording and syntactic structures are used to express the same idea. as a result, even if the tm contains a semantically similar segment, the retrieval algorithm will not be able to identify it in most of the cases. researchers tried to address this shortcoming of the edit distance metric by employing similarity metrics that can identify semantically similar segments even when they are different at token level. section 2 discusses some of the approaches proposed so far. recent research on the topic of text similarity employed methods that rely on deep learning and various vector based representations used in this field @xcite @xcite . one of the reasons for this is that calculating the similarity between vectors is more straightforward than calculating the similarity between texts. it is easy to calculate how close or distant two vectors are by using well understood mathematical distance metrics. in addition, deep learning based methods proved more robust in numerous nlp applications. in this paper we propose a novel tm matching and retrieval method based on the universal sentence encoder @xcite which has the capability to capture semantically similar segments in tms better than methods based on edit distance. we selected the universal sentence encoder as our sentence encoder since it outperforms other sentence encoders like infersent @xcite in many natural language processing tasks including semantic retrieval @xcite . also the recently release of multilingual universal sentence encoder 3 is available on 16 different languages @xcite . since we are planning to expand our research to other language pairs than the english -spanish pair investigated in this paper, the multilingual aspect of the universal sentence encoder can prove very useful. the rest of the paper is organised as follows. section 2 briefly describes several approaches used to improve the matching and retrieval in tms. section 3 contains information about the threshold is appropriate for a given text. 3 https://tfhub.dev/google/universalsentence-encoder-multilingual-large/3 settings of the experiments carried out in this paper. it includes the experiments that were done for semantic textual similarity tasks comparing the universal sentence encoder and edit distance. the same section also presents the results of the experiments on real world tms. section 4 discusses the results and describes future research directions. the implementation of the methods presented in this paper is available on github. 4 2 related work despite being the most used tools by professional translators, translation memories have rarely been criticised because of the quality of the segments they retrieve. instead, quite often the requests from translators focus on the quality of the user interface, the need to handle different file formats, their speed and possibility of working in the cloud @xcite . most of the current work on tms is focused on the development of addons like terminology managers and plugins which integrate machine translation engines, as well as project management features @xcite . even though retrieval of previously translated segments is a key feature in a tm system, this process is still very much limited to edit-distance based measures. researchers working on natural language processing have proposed a number of methods which try to improve the existing matching and retrieval approaches used by translation memories. however, the majority of these approaches are not suitable for large tms, like the ones normally employed by professional translators or were evaluated on very small number of segments. planas and furuse (1999) extend the edit distance metric to incorporate lemmas and part-of-speech information when calculating the similarity between two segments, but they test their approach on less than 150 segments from two domains using two translation memories with less than 40,000 segments in total. lemmas and part-of-speech information is also used in @xcite in order to improve matching, especially for morphologically rich languages like hungarian. they also experiment with sentence skeletons in which nps are automatically aligned between source and target. unfortunately, the paper presents only preliminary results. @xcite show how it is possible to improve the quality of matching by taking into consideration the syntactic structure of sentences. unfortunately, the evaluation is carried out on only a handful of carefully selected segments. another method which performs matching at level of syntactic trees is proposed in @xcite . the results presented in their paper are preliminary and the authors notice that tree matching method is "prohibitively slow". more recent work has focused on incorporating paraphrases into the matching and retrieving algorithm @xcite @xcite . @xcite proposed a finite transducer which considers paraphrases during the matching. the evaluation shows that the method improves both precision and recall of matching, but it was carried out with only one translator and focused only on segments with exactly the same meaning. gupta and orasan (2014) proposed a variant of the edit distance metric which incorporates paraphrases from ppdb foot_2 using greedy approximation and dynamic programming. both automatic evaluation and evaluation with translators show the advantages of using this approach @xcite . @xcite follows a similar approach. they use nooj foot_3 to create paraphrases for the verb constructions in all source translation units to expand the fuzzy matching capabilities when searching in the tm. evaluation with professional translators showed that the proposed method helps and speeds up the translation process. to best of our knowledge, deep learning methods have not been used successfully in translation memories. @xcite presents an attempt to use reval, an evaluation metric that was successfully applied in the wmt15 metrics task @xcite . unfortunately, none of the neural based methods used are able to lead to better results than the standard edit distance. ## experiments and results as mentioned above, the purpose of this research is to find out whether it is possible to improve the quality of the retrieved segments by using the universal sentence encoder @xcite released by google as the sentence encoder for this experiment. it comes with two versions: one trained with a transformer encoder and the other trained with a deep averaging network (dan) @xcite . the transformer encoder architecture uses an attention mechanism @xcite to compute context aware representations of words in a sentence and average those representations to calculate the embedding for the sentence. the dan encoder begins by averaging together word and bi-gram level embeddings. sentence embeddings are then obtained by passing the averaged representation through a feedforward deep neural network (dnn). the architecture of the dan encoder is similar to the one proposed in @xcite . the two architectures have a trade-off of accuracy and computational resource requirement. the one that relies on a transformer encoder has higher accuracy, but is computationally more expensive. in contrast the one with dan encoding is computationally less expensive, but has a slightly lower accuracy. for the experiments presented in this paper we used both architectures. the trained universal sentence encoder model for english is available on tensorflow hub foot_4 . ## conclusion and future work in this paper we have proposed a new tm matching and retrieval method based on the universal sentence encoder. our assumption was that by using this representation we will be able to retrieve better segments from a tm than when using a standard edit distance. as shown in 3.2.3 section, the universal sentence encoder performs better than okapi for fuzzy match scores ranged below 0.8. therefore, we believe that the sentence encoders can improve the matching and retrieval in tms and should be explored more. usually tm matches with lower fuzzy match scores (¡ 0.8) are not used by professional translators, or when used, they lead to a decrease in translation productivity. but our method can provide better matches to sentences below fuzzy match score 0.8, hence will be able to improve the translation productivity. according to the annotation guidelines of @xcite a semantic textual similarity score of 0.8 means "the two sentences are mostly source segment human translated segment universal sentence encoder suggestion okapi suggestion if applicable en su caso si procede no procede date of granting fecha de concesión de la subvención fecha de autorización fecha de la garantía otorgada this decision shall be kept under constant review and shall be renewed or amended, as appropriate, if the council deems that its objectives have not been met.' la presente decisión estará sujeta a revisión continua y se prorrogará o modificará, según proceda, si el consejo estima que no se han cumplido sus objetivos. será prorrogada o modificada, según proceda, si el consejo considera que no se han cumplido sus objetivos. se prorrogará o modificará, si procede, en caso de que el consejo estime que no se han cumplido los objetivos de la misma. the information shall include: esta información incluirá: esa información podrá versar sobre lo siguiente: los indicadores clave de rendimiento incluirán: general characteristics of the finished product características generales del producto terminado descripción del producto final, características generales del componente de servicios de copernicus such reports shall be made publicly available. dichos informes se harán públicos. sus informes se harán públicos. se pondrá a disposición del público un resumen de las evaluaciones. the commission decision to initiate the procedure ('the opening decision') was published in the official journal of the european union. la decisión de la comisión de incoar el procedimiento (en lo sucesivo, decisión de incoación) se publicó en el diario oficial de la unión europea. la decisión de la comisión de incoar el procedimiento (en lo sucesivo, decisión de incoación) fue publicada en el diario oficial de la unión europea. la decisión de la comisión de incoar el procedimiento se publicó en el diario oficial de la unión europea. chapter 2 is amended as follows: el capítulo 2 se modifica como sigue: la parte 2 se modifica como sigue: la sección 2 queda modificada como sigue: in the future, we also plan to experiment with other sentence encoders such as infersent @xcite and sbert @xcite and with alternative algorithms which are capable to capture semantic textual similarity between two sentences. we will try unsupervised methods like word vector averaging fuzzy score mean sts score 0.8 -1.0 0.952 0.6 -0.8 0.768 0.4 -0.6 0.642 0.2 -0.4 0.315 0 -0.2 0.121 and word moving distance @xcite as well as supervised algorithms such siamese neural networks @xcite and transformers @xcite .