Robust Fragment-Based Framework for Cross-lingual Sentence Retrieval (2021)

Abstract: Cross-lingual Sentence Retrieval (CLSR) aims at retrieving parallel sentence pairs that are translations of each other from a multilingual set of comparable documents. The retrieved parallel sentence pairs can be used in other downstream NLP tasks such as machine translation and cross-lingual word sense disambiguation. We propose a CLSR framework called Robust Fragment-level Representation (RFR) CLSR framework to address Out-of-Domain (OOD) CLSR problems. In particular, we improve the sentence retrieval robustness by representing each sentence as a collection of fragments. In this way, we change the retrieval granularity from the sentence to the fragment level. We performed CLSR experiments based on three OOD datasets, four language pairs, and three base well-known sentence encoders: m-USE, LASER, and LaBSE. Experimental results show that RFR significantly improves the base encoders’ performance for more than 85% of the cases.

## introduction parallel corpora are essential for many nlp tasks in terms of both quality and quantity @xcite . tasks like machine translation @xcite , cross-lingual word sense disambiguation @xcite , and annotation projection @xcite require a substantial amount of high-quality parallel sentences to construct accurate models. traditionally, creating large-high-quality parallel corpora requires enormous manual effort from human annotators or translators. there are two approaches to reduce such human effort: (i) using an unsupervised learning method to reduce the reliance on parallel corpora @xcite @xcite . (ii) using a cross-lingual sentence retrieval (clsr) method to automate finding parallel sentences. while the first approach may completely avoid using parallel corpora altogether through unsupervised learning, experimental results show that incorporating parallel sentences into the training process improves the model's performance. that is, parallel corpora still play a critical role even when employing unsupervised learning. consequently, we focus our research attention on the latter approach. given a collection q of query sentences q in one language l1 and another collection t of target sentences t in a different language l2, clsr aims to find actual parallel pairs (@xmath0, @xmath1. in real-world scenarios, parallel sentences are mined from comparable corpora. consequently, not every q has a corresponding t and vice versa; we consider such sentences non-pairing. an effective clsr method has to identify parallel pairs (q, t) from many non-pairing sentences. as the number of non-pairing sentences increases, there are more distractors to actual parallel pairs, and the robustness of the method becomes critical. a popular clsr approach constructs an embedding space using a multilingual sentence encoder (encoder for short) to organize sentences from different languages according to the meanings. wellknown methods utilizing this approach include m-use @xcite , laser @xcite , and labse @xcite . for robustness, clsr methods generally include a filtering mechanism to avoid including non-pairing sentences into the results. using raw scoring from the encoder and hard threshold to filter out non-pairing sentences suffers from globally similarity score inconsistency. to improve the filtering robustness, more sophisticated re-scoring mechanisms have been studied. @xcite proposed a filtering mechanism based on variations of margin-based scorers. their method considers the margin between a query sentence and its k-nearest neighbor based on a for-ward and backward search using the cosine similarity function. @xcite found that only a forward search also obtained a comparable performance to that of @xcite . they also proposed a bert-based re-scoring function, which substantially improved the accuracy. the methods mentioned above can robustly filter out non-pairing sentences and can accurately identify sentence pairs in in-domain data. however, their performance significantly drops when applied to out-of-domain (ood) test samples. we compares results of the base multilingual sentence encoders for in-domain and out-of-domain (ood) scenarios. bucc @xcite is a standard corpus for clsr task. in contrast, jw300 @xcite is constructed from religious-society magazines that are less formal. laser was trained on formal documents such as europarl, and united nation parallel data, while labse used wikipedia data for training. thus, we consider jw300 as an ood dataset for laser and labse. note that m-use did not provide results on bucc. results from in this paper, we propose a robust fragmentlevel representation (rfr) framework to improve the clsr robustness when applied to ood scenarios. the crux of our solution lies in the n-grams sliding window mechanism, which breaks up each sentence into multiple vectors (called fragments) to allow for phrase matching at the subsentence level. to avoid accidental matching, i.e., pairing similar fragments from sentences with different meanings, we also equip each fragment with a traditional sentence encoding. since different fragments from the same sentence can now be associated with fragments from various sentences, we also propose a process to combine results from multiple fragment matchings to form one single final output for each sentence. to assess the effectiveness of our solution, we conducted experimental studies on three datasets, which were all ood with respect to the base and proposed methods. for each dataset, we used two rich-resource language pairs, french-english and german-english, as well as two limited resource language pairs, arabic-english and thai-english. we used three well-known encoders as our base encoders, namely m-use, laser, and labse. we also implemented a proposed solution on top of each base, namely rfr-m-use, rfr-laser, and rfr-labse. the combination of three datasets, four language pairs, and three bases methods formed 36 comparisons in total. experimental results show that our proposed solution could significantly enhance the performance of the base encoders in 32 out of 36 comparisons. in addition, we also applied our framework to a crosslingual qa dataset. our method consistently improves the accuracy of m-use qa , a well-known encoder for cross-lingual answer retrieval. the summary of our contributions is as follows: (i) we propose a novel sentence representation model representing each sentence as a collection of fragments. (ii) we propose a novel fragment-level clsr framework that enhances robustness to base encoders. (iii) we demonstrate significant improvement of our framework on all base encoders via extensive experimental studies. ## proposed framework we first provide an overview of our rfr framework in figure 1 . it consists of three main components: (i) preprocessing, (ii) similarity search, and (iii) prediction aggregation. preprocessing. the preprocessing step transforms each sentence into multiple fragments, where each fragment is represented as a vector. for each sentence s, we first remove all punctuations foot_0 and represent each word as a token (w s j ) foot_1 where j is the word index. then, a sliding window is applied to generate a collection of n-grams. we call these n-grams sentence fragments (f s i ) where i indicates the token index. we then encode each fragment using an encoding function g(•) in to a vector (e s i ). the encoding function can be from any multilingual encoder mentioned earlier. we also append a sentence-level encoding vector (e s s ) to form a final representation ([e s i : e s s ]). we shall refer to this collection of preprocessed fragments as the database. sentence encoder query sentence: target sentences: sentence representation embedding space probabilistic -nearest neighbor 0.8 0.4 0.6 0.0 0.3 0.4 0.2 0.3 0.0 concatenate candidate probability entropy filtering 0.8 0.6 0.0 0.4 0.2 0.0 + re-normalize 0.7 0.2 0.1 confidence classifier top score: pairing threshold: pairs with the top score is non-pairing sentence prediction aggregation similarity search preprocessing figure 1: overview of our method. query sentence has 4 words, and there are 3 sentences in target corpus which have 5, 3, and 4 words. the sentences are split into fragments and encoded into embeddings for the retrieval process. note that in our ablation studies, the addition of sentence-level information significantly improved the performance ( ## conclusion and future work we propose a novel sentence representation model representing each sentence as a collection of fragments for query-related tasks. our clsr framework can enhance the robustness of any pretrained multilingual sentence encoder. extensive experiments on four pairs of rich-and low-resource languages show that our method significantly improves over the base encoders. we also demonstrated the usefulness of our framework on document retrieval for question-answering in three languages and obtained improvements in all cases. for future work, we would like to explore the possibility of returning sub-sentence matching in order to improve the recall of our framework.