Target Language Monolingual Translation Memory based NMT by Cross-lingual Retrieval of Similar Translations and Reranking (2023)

Abstract: Retrieve-edit-rerank is a text generation framework composed of three steps: retrieving for sentences using the input sentence as a query, generating multiple output sentence candidates, and selecting the final output sentence from these candidates. This simple approach has outperformed other existing and more complex methods. This paper focuses on the retrieving and the reranking steps. In the retrieving step, we propose retrieving similar target language sentences from a target language monolingual translation memory using language-independent sentence embeddings generated by mSBERT or LaBSE. We demonstrate that this approach significantly outperforms existing methods that use monolingual inter-sentence similarity measures such as edit distance, which is only applicable to a parallel translation memory. In the reranking step, we propose a new reranking score for selecting the best sentences, which considers both the log-likelihood of each candidate and the sentence embeddings based similarity between the input and the candidate. We evaluated the proposed method for English-to-Japanese translation on the ASPEC and English-to-French translation on the EU Bookshop Corpus (EUBC). The proposed method significantly exceeded the baseline in BLEU score, especially observing a 1.4-point improvement in the EUBC dataset over the original Retrieve-Edit-Rerank method.

## introduction many studies have incorporated translation memories (tm), a set of high-quality bilingual sentences, into the nmt model in recent years. @xcite and @xcite proposed a nfr (neural fuzzy repair) model that improves translation accuracy by incorporating tm into nmt. the model retrieves a similar source sentence from the set of source language sentences in the tm based on edit distance and sent2vec @xcite , and concatenates the translation of a similar source sentence with the input source sentence to the nmt model. since this model only requires preprocessing of the input to the nmt model, tm can be incorporated without modifying the model's architecture. therefore, it is highly compatible with existing nmt models and portable in terms of implementation. on the other hand, due to the limitation of input sentence length, the number of similar sentences available figure 1 : retrieval of similar sentences from translation memory for these methods is limited to one or two at most, and the retrieved similar sentences are not fully utilized. also, if many informative similar sentences are obtained during inference, it is difficult to use all of them. @xcite proposed the retrieve-edit-rerank framework to overcome this limitation. they proposed a method that (1) retrieves multiple sentences from the training data using the input sentence as a query, (2) inputs the concatenation of the source and retrieved sentences into the model to generate multiple candidate sentences, and (3) extract the best sentence from the multiple candidates by choosing the sentence that maximizes the log-likelihood. in this paper, we focus on the (1) retrieval step and the (3) reranking step. as for the retrieval step, we compared monolingual inter-sentence similarity measures such as edit distance to cosine similarity based on language-independent sentence embedding with multilingual sentence-bert (msbert) @xcite and labse @xcite . here, as shown in figure 1 , the edit distance requires the parallel corpus as the retrieval target, while the methods based on multilingual sentence embedding only requires a monolingual corpus of target language sentences. in the reranking step, we proposed a new reranking score for selecting the best sentences. this reranking score takes into account both the log-likelihood of each candidate with normalization by sentence length and the sentence embedding based similarity between the input and the candidate. we used the english-japanese corpus of asian scientific paper excerpt corpus (aspec) @xcite and the english-french corpus of eu bookshop corpus (eubookshop) @xcite to evaluate our method and found that the proposed method achieved significantly higher translation accuracy in all settings. in summary, our contributions are as follows 1. in the framework of nfr (figure 2 ), the use of similar sentences retrieved by language-independent sentence embedding generation models such as msbert and labse significantly improved translation accuracy compared to conventional edit distance based retrieval methods ( ## related work as an nmt using the retrieve-edit framework, @xcite and @xcite proposed nfr (neural fuzzy repair), a method to incorporate translation memory (tm) into figure 3: the inference framework of retrieve-edit-rerank model nmt. they proposed a method that first retrieves similar source sentences based on edit distance using the input source sentence as the query, then concatenates the translation of the similar source sentence and the input source sentence and enters them into an lstm-based nmt model. in this method, they achieved state-of-the-art in english-german and english-hungarian translations. in addition, xu et al. (2020) introduced word-by-word fuzzy matching to improve the accuracy of english-to-french translation using the transformer model. they used cosine similarity of sentence embeddings as a similarity measure between an input sentence and the source language sentence. among bulte and tezcan (2019), tezcan et al. (2021), and xu et al. (2020), both tezcan et al. (2021) and xu et al. ( 2020 ) introduced sentence embedding based similarity measure such as sent2vec for matching of the input sentence and similar source language sentences. however, they limit the search to the source language side of the training data. this information source is the same as the data used for training the baseline models, and the available quantity of data has not significantly increased. our approach, on the other hand, is based on multilingual sentence embedding methods such as msbert and labse and its information source for retrieving similar target sentences is the monolingual translation memory of target language sentences that is much larger than the training parallel data. @xcite also proposed a method that uses a monolingual corpus of the target language, rather than a bilingual corpus, as a target for retrieving similar sentences. they proposed a learnable retrieval model which is jointly optimized with the nmt model and performed similar sentence retrieval by mips (maximum inner product search). although this approach achieves high performance, the model has to be built upon an architecture consisting of a retrieval model and a translation model. as a result, it eliminates the advantage of nfr where one can leverage the existing transformer architecture and simply expand the input. our approach, on the other hand, can be formalized as introducing the reranking phase of retrieve-edit-rerank into the architecture of the nfr framework, where it can be seen as leveraging the existing transformer architecture in the edit phase of retrieve-edit-rerank framework. for translation and summarization tasks, @xcite proposed a method to gen-erate multiple candidate output sentences and select the best output sentence by reranking them according to log-likelihood. they achieved a significant improvement in accuracy by combining nfr and retrieve-edit-rank frameworks. despite the simplicity and versatility of this method, however, the improvement in translation accuracy due to the reranking is small. in recent years, dense retrieval methods enabled us to retrieve semantically similar sentences with high accuracy and speed due to the development of transformer-based language models. @xcite proposed a sentence embedding generation model, sentence-bert, to embed semantically similar sentences close to each other in vector space based on the pre-trained bert @xcite . more recently, @xcite proposed a multilingual sentence embedding generation model, labse. these multilingual sentence embeddings can be retrieved quickly using approximate nearest neighbor search methods such as faiss @xcite . ## conclusion in this study, within the retrieve-edit-rerank framework, we introduced a method for crosslingual retrieval of similar translations through multilingual sentence embedding, along with an enhanced reranking method. we demonstrated that utilizing vector neighborhood search, based on language-agnostic sentence embedding generation models like msbert and labse, contributed to a significant improvement in translation accuracy within this framework. this proved more effective than the retrieval technique based on edit distance employed in the previous research. moreover, we applied multiple similar sentences to generate various candidate translations, subsequently selecting the optimal translation through an automatic reranking process. the reranking score considered both the output log-likelihood normalized for the length of the reconstituted subword sentences, and the cosine similarity between the input and output candidate sentences through sentence embeddings. this methodology has led to a significant enhancement in translation accuracy.