Filtering and rescoring the CCM atrix corpus for Neural Machine Translation training (2023)

Abstract: There are several parallel corpora available for many language pairs, such as CCMatrix, built from mass downloads of web content and automatic detection of segments in one language and the translation equivalent in another. These techniques can produce large parallel corpora, but of questionable quality. In many cases, the segments are not in the required languages, or if they are, they are not translation equivalents. In this article, we present an algorithm for filtering out the segments in languages other than the required ones and re-scoring the segments using SBERT. A use case on the Spanish-Asturian and Spanish-Catalan CCMatrix corpus is presented.

## introduction 1.parallel corpora crawled from the web there are several web-derived very large parallel corpora available for a high number of language pairs. paracrawl 1 @xcite ) is a parallel corpus created crawling the web searching for multilingual pages. at the moment it offers parallel corpora from english to 38 languages and 6 additional language pairs not including english. wikimatrix 2 @xcite ) is created using wikipedia to automatically find translated sentences. it includes 96 languages, totalling 16,720 language pairs. ccaligned 3 @xcite is a corpus formed by parallel or comparable web-document pairs in 137 languages aligned with english. from this document corpus, parallel segments are extracted using similarity scores of laser 4 @xcite embeddings from the document pairs. oscar 5 @xcite is also a parallel corpus crawled from the web covering 166 languages. the ccmatrix foot_2 @xcite corpus has the particularity that no document information has been used. instead, all the segments in a given language are compared with all the segments in another language in order to detect parallel segments. to do so, they also use laser and calculate a margin score, defined as the ratio between the cosine distance between the two sentence embeddings, and the average cosine similarity of its nearest neighbours in both directions. this results in very large parallel corpora for 90 languages, totalling 1,197 language pairs. some of these corpora, and ccmatrix in particular, suffer from low quality, especially for language pairs with fewer resources. two main problems are easily detected by a simple visual inspection: segments are not in the correct language, and source and target segments are not translation equivalents. in this paper we present a program that verifies the languages and assesses the translation equivalence of the source and target segments. we evaluate the performance of the program on the ccmatrix corpus for spanish-asturian and spanish-catalan. ## previous works the idea of using multilingual sentence embeddings for parallel corpus cleaning is not new. in @xcite , laser is used to create representations of the segments and to score them and filter the noisy parallel segments. they used this technique in a low-resource scenario, but the authors state that it is promising even in noresource scenarios. in @xcite , the degree of parallelism of the segments is measured using bert and a domain filter is used to avoid the adverse effect of the domain of the training data. a recent study @xcite designs a filtering strategy based on a trained classifier. to train the classifier, they use a labelled dataset of parallel segments annotated as valid or invalid. they apply the filtering algorithm to english-catalan and catalan-english and achieve improvements between 1.3 and 2.9 bleu points when training nmt systems on the clean corpus. the resources and algorithms are freely available, but their use is not simple and straightforward. few of these works end in a ready-to-use algorithm. among these, we can mention the following. zipporah foot_10 @xcite uses a bagof-words translation feature, and needs to train a logistic regression models to filter the parallel corpus. the user has to train the system providing a bad corpus (containing noisy data that should be filtered), a good or training corpus and development data, that should be a clean corpus. bifixier foot_11 @xcite performs a restorative cleaning consisting on the following steps: removing of the parallel segments having an empty segment in any of the parts; character fixing; orthography fixing; respliting of the segments and duplicate identification. bicleaner foot_12 @xcite is a parallel sentence noise filter and classifier tool. the process is done in three steps: (1) pre-filtering based on a set of rules; (2) language model fluency scoring, a language-dependent step using character-based language models; and (3) classification based on a random-forest machine learning model. ## description of the resorting and filtering tool the tool is implemented in two python programs that can be freely downloaded from github foot_13 : the rescorer and the selector. the rescorer algorithm performs two actions: the algorithm creates a sqlite database with the following structure: while reading the input corpus, the sqlite database is filled with the required information. as the calculation of the sentencetransformer and the cosine similarity are slow, they are only calculated for those source and target segments with the expected detected languages. please note that along with the detected language, the confidence scores are stored in the database. once the sqlite database is created, a selection program is used (mtuoc-pcorpus-selector.py) to select the parallel segments satisfying a minimum source and target language detection confidence and a minimum sbert score (the cosine similarity). ## conclusion and future work in this paper, we have presented a simple strategy to select the higher quality segments from a large parallel corpus. this strategy is based on verifying the languages of the segments and on scoring the parallel segments with sbert. the methodology has been implemented in a python script holding a free licence that can be downloaded from github. 19 . filtered versions of the ccmatrix corpus for several language pairs are available for download.