Effective Bitext Extraction From Comparable Corpora Using a Combination of Three Different Approaches (2021)

Abstract: Parallel sentences extracted from comparable corpora can be useful to supplement parallel corpora when training machine translation (MT) systems. This is even more prominent in low-resource scenarios, where parallel corpora are scarce. In this paper, we present a system which uses three very different measures to identify and score parallel sentences from comparable corpora. We measure the accuracy of our methods in low-resource settings by comparing the results against manually curated test data for English–Icelandic, and by evaluating an MT system trained on the concatenation of the parallel data extracted by our approach and an existing data set. We show that the system is capable of extracting useful parallel sentences with high accuracy, and that the extracted pairs substantially increase translation quality of an MT system trained on the data, as measured by automatic evaluation metrics.

## introduction high quality mt systems rely on the availability of parallel data. in low-resource settings, where parallel data is scarce, unsupervised methods have been proposed, where only monolingual corpora are used for training @xcite . @xcite show that supervised and semi-supervised approaches with only a small parallel corpus of 50k bilingual sentences consistently outperform the best unsupervised systems for a range of languages. however, there is a scarcity of parallel data, especially for languages with a low number of speakers. when parallel corpora are scarce, comparable corpora, which are far more common, can be used to supplement it. we will be working with the english-icelandic language pair, for which no statistical or neural mt work had been published until last year @xcite . when parallel sentences are extracted from comparable corpora, potential parallel sentence candidates can usually come from anywhere in two comparable documents. this means that a potential parallel counterpart of one sentence in the source-language document can be any sentence in the target-language document. if the average number of sentences in a comparable document is n, the number of potential sentence pairs that have to be evaluated are n 2 . this quickly becomes overwhelming (as n increases) and so it is imperative to reduce the search space. reducing the search space should ideally result in a list of a maximum of kxn candidates, where k is a constant number of allowed candidates for each sentence in the comparable documents. to retrieve useful sentence pairs from this list, the pairs have to be scored and filtered. our approach divides the problem into two main steps. we start by extracting parallel sentence candidates using an inverted index-based crosslingual information retrieval (clir) tool called fada @xcite , that requires a collection of documents in two languages and only a bilingual lexicon without the need of any mt system. in the second step, we score the sentence candidates using two different scores, one based on contextualized embeddings and the other on high-precision word alignments. a binary classifier selects sentence pairs based on these scores. we test our approach in three different ways. we use two different test sets to measure precision, recall and f1-scores, and we also use our approach to extract parallel sentences from wikipedia and use the resulting data as supplemental data for training nmt systems. the systems are then evaluated in terms of bleu scores @xcite and compared to a baseline in order to give an indication of the usefulness of the supplemental data for nmt training. our main contributions are fourfold. ## related work comparable corpora have been shown to be a useful source for mining parallel segments that can help improve mt quality @xcite . afli et al. (2015) extract parallel data from a multimodal comparable corpus from the euronews foot_0 and ted foot_1 web sites. chu et al. (2015) extract parallel texts from the chinese and japanese wikipedia and ling et al. @xcite employ a crowdsourcing approach to extract parallel text from twitter data in order to find the translations in tweets. the work of @xcite describes the approach of extracting parallel sentences from english-persian document-aligned wikipedia entries. they use two mt systems to translate from persian to english and the reverse and then use an ir system to measure the similarity of the translated sentences. multilingual sentence embeddings have also been applied to the problem, obtaining state-of-the-art performance @xcite . recently, @xcite describe the collection of parallel corpora for 11 indic languages from diverse comparable corpora using labse embeddings @xcite , a language-agnostic bert sentence embedding model trained and optimized to produce similar representations for bilingual sentence pairs that are translations of each other. word alignments have previously been used for parallel sentence extraction. zarin , a et al. (2015) identify parallel sentences using word alignments, experimenting with five different alignment based scores. they presume that if a pair of sentences are equivalent in two languages, there should be many word alignments between the sentences, and non-parallel sentences should have few or no word alignments. @xcite use alignment based heuristics to filter out sentence pairs. lu et al. (2020) use a word alignment based translation score as a part of their scoring ensemble for filtering a noisy parallel corpus. their translation score is a simplified version of the translation score introduced by @xcite . @xcite and andoni @xcite describe a method using clir and lexical translations obtained using word alignments, with a simple overlap metric. they obtained the highest results for the bucc 2017 and bucc 2018 shared tasks. our method uses an ir system to create a list of alignment candidates, thus reducing the search space. it then takes advantage of both labse embeddings and word alignments. our word alignment score is calculated by a simpler formula than most of the previous work, but relies on high precision alignments. it has been shown that they can be achieved by an ensemble method using comb-align @xcite . a binary classifier is finally used to select acceptable sentence pairs. ## data for the language pair we are working with, english-icelandic, no test sets have previously been made available for parallel sentence extraction from comparable corpora. therefore, we have to build test sets in order to be able to evaluate our approach. we prepare the following data sets for our experiments: ## evaluation we evaluate our system by calculating precision, recall and f1-scores using our (i) compnews test set and (ii) compwiki test set; and (iii) by training, testing and calculating bleu scores for nmt systems, both with and without parallel sentences extracted from all wikipedia articles that are available in both english and icelandic. ## conclusions and future work we have shown that our method, combining crosslingual information extraction, contextualized embeddings and word alignments, is efficient at finding parallel segments in comparable corpora. furthermore we introduce wascore, a metric of translational equivalence based on high-precision word alignments, and show that as well as being a useful part of a binary classifier, it can be used effectively to filter out detrimental segments from parallel corpora. finally, we publish two new test sets for extracting parallel sentences from comparable corpora, an automatically generated english-icelandic lexicon with probability scores and a set of automatically extracted parallel segments that we show are useful for training mt systems. when testing on the compwiki test set we saw that while our method is efficient in finding parallel segments in comparable corpora, it also selects partially parallel segments. although these segments seem to have information useful for training mt systems, it is difficult to know to what extent they are useful and when they may become detrimental. for this reason, we plan to study these kinds of data further and investigate how they affect translation quality of an nmt system trained on it. based on that, we want to explore more sophisticated ways to segment or concatenate alignment candidates in order to be able to build a data set that only contains segment pairs that are useful for training mt systems. there is previous work on parallel fragment extraction using word alignments @xcite , and we will use their approach as a baseline to proceed further. while the combination of the two scores used to measure the quality of the sentence pairs resulted in a list of sentence pairs that we show are useful for mt training, it still contains pairs that are detrimental, as shown by the simple filtering based on wascore threshold. other parallel sentence pairs may also remain to be found in the wikipedia data. in order to improve our approach, more scores could be added to our classifier. while we opted to use raw labse cosine similarity scores, shown by @xcite to be more accurate than cosine similarity scores from other models, the margin-based ratio score proposed by @xcite has also been shown to be very effective for this task. other scores to consider could include bleu or chrf @xcite , although they need reasonably good mt systems to be useful, margin-based cosine distance @xcite , or mahalanobis distance @xcite as described in @xcite . doing an ablation study on the scores could help determine which are the most useful. working with these scores, a comparison of applying different classifiers while using the same scoring mechanisms may be helpful. it is also to be noted that we extracted only 10 target-language candidate pairs in the first step, which is the default value used in fada as it gave optimal performance in their work. it also has the benefit of reducing the computational complexity in the next steps. however, we also plan to explore other higher values of candidate extraction in future and to investigate how it affects the overall system performance. finally, we plan to conduct our experiments on other language pairs.