Translation Memory Retrieval Using Lucene (2021)

Abstract: Translation Memory (TM) system, a major component of computer-assisted translation (CAT), is widely used to improve human translators’ productivity by making effective use of previously translated resource. We propose a method to achieve high-speed retrieval from a large translation memory by means of similarity evaluation based on vector model, and present the experimental result. Through our experiment using Lucene, an open source information retrieval search engine, we conclude that it is possible to achieve real-time retrieval speed of about tens of microseconds even for a large translation memory with 5 million segment pairs.

## introduction translation memory technique is a key functionality being widely used in the field of cat. translation memories (tms) are "structured archives of past translations" which store pairs of corresponding text segments in source and target languages known as "translation units" @xcite . the size of translation memories and the quality of their contents are major impact factors crucial to the effectivity of the translation memory system which uses them. due to the importance of translation memories, there has been done lots of research work for building large tms on worldwide scale, not just in individual countries @xcite . what plays an important role for tm system is also the similarity evaluation between the input sentence to be translated and the source segment in the tm. the main task of tm system is to get a translation unit whose source segment is the most similar to the input sentence out of tm. there are two possible solutions in performing the task: one solution is to adopt an intelligent tm matching mechanism which is able to correctly calculate the similarity between the input sentence and the source segment in the translation memory. the other solution is to increase the size of tm by collecting translated resources as much as possible. no matter how intelligent the tm matching mechanism is, small-size tm cannot afford rich performance. of course, the choice of tm matching method is important for improving the effectivity of tm system. but what is no less important than any tm matching method is to use a reasonable size tm. the larger the tm, the higher the possibility to get a translation unit whose source segment is very similar to the input sentence out of the tm. in general, the main value of a tm consists in the number of segments -its size. however, large tms automatically lead to slow response times. a slow tm might actually slow down a translator, so that fast response time is an essential characteristic of any tm. many research works have been reported to improve tm matching and retrieval, but the majority of those approaches were just evaluated on relatively small tms. to our best knowledge, the largest tm tested so far in previous research works is the first five parts of the 2013 dgt-tm (which consisted of 305,324 segment pairs) used in (weitz 2017) and the 2018 volume 1 of the dgt-tm (which had 230,000 segment pairs) used in @xcite . the main problem we are going to solve in this paper is to provide a tm retrieval mechanism to ensure real-time performance on very large tms, e.g. with millions of segment pairs. we propose a tm retrieval method based on vector model (vm), which is widely used in information retrieval (ir), and implement our proposal using lucene, an open source ir search engine. the rest of the paper is organized as follows: section 2 briefly reviews previous research works related to tm matching and retrieval. section 3 describes tm retrieval translation memory retrieval using lucene kwang-hyok kim foot_0 , myong-ho cho 1 , chol-ho ryang 1 , ju-song im 1 , song-yong cho 1 , yong-jun han foot_1 ## experimental result we use lucene, an open source ir engine in java, to implement the tm retrieval system using the vector model based similarity evaluation we proposed. as the data structure of a translation unit of tm, the document class of lucene is used which has two fields corresponding to the source and target segment of tm, respectively. the version number of lucene used is 8.5.1. levenshtein distance based similarity score is applied for the secondary retrieval of tm. the tm used for the evaluation of the proposed tm retrieval system is an english-to-korean tm which is made of about 5,000,000 english segments and their automatic korean translation by english-to-korean machine translator "ryongnamsan" 2.0. in the experiment, we use parameter settings for the primary search of tm such that 𝐾 1 = 100 , and 𝛼 = 0.5 . all measurement was carried out on a desktop pc with 3.0.tar.gz) intel® core™ i3-3240 cpu @ 3.40ghz and 2gb of ram. the operating system installed is windows 7, 64bit. ## conclusion and future work through a series of experiments on the primary retrieval of english-to-korean tm using vector model based similarity evaluation, we conclude that:  the time and space complexity of indexing the tm increases linear to the size of the tm. the indexing time consumed for a large-scale english-to-korean tm with about 5,000,000 segments is about 5 minutes, and the indexed data size is 1.84 gb with an increase of about 29 % compared to the text file size of the tm. the effect of the vector model based similarity evaluation for the primary retrieval of tm wholly depends on the correctness of the morphological analysis and the richness of the synonym knowledge. since the difficulty of the morphological analysis and the availability of the synonym knowledge like wordnet is all different for each language, we plan to do more research work on these aspects. furthermore, we also plan to evaluate more comprehensively the relevance of the primary retrieval of tm.