Separating Grains from the Chaff: Using Data Filtering to Improve Multilingual Translation for Low-Resourced A frican Languages (2022)

Abstract: We participated in the WMT 2022 Large-Scale Machine Translation Evaluation for the African Languages Shared Task. This work describes our approach, which is based on filtering the given noisy data using a sentence-pair classifier that was built by fine-tuning a pre-trained language model. To train the classifier, we obtain positive samples (i.e. high-quality parallel sentences) from a gold-standard curated dataset and extract negative samples (i.e. low-quality parallel sentences) from automatically aligned parallel data by choosing sentences with low alignment scores. Our final machine translation model was then trained on filtered data, instead of the entire noisy dataset. We empirically validate our approach by evaluating on two common datasets and show that data filtering generally improves overall translation quality, in some cases even significantly.

## introduction this paper presents masakhane nlp's submission to the wmt 2022 large-scale machine translation evaluation for african languages. we participated in the constrained translation task and chose to focus on a subset of all the language pairs considered for this task due to resource constraints. we specifically explore the language directions {hau, ibo, lug, swa, tsn, yor, zul}↔eng and wol↔fra, and submitted our primary and secondary systems which were competitive with other submissions for this task. machine translation has received much attention recently, especially for low-resourced languages @xcite @xcite @xcite . a promising approach for such setups is to fine-tune large pre-trained language models on the available small amount of translation data @xcite @xcite . while most of these language models are trained on predominantly high-resourced language datasets @xcite @xcite , there have been a few models that were pre-trained @xcite or adaptively fine-tuned @xcite only on low-resourced languages. recent works have tried, successfully, to supplement the existing small amounts of natural data in low-resource languages with artificially generated parallel data. for instance, in machine translation, @xcite and @xcite padded the true parallel data with automatic translations of monolingual sentences through backtranslation and self-learning respectively. others, such as @xcite ; @xcite ; and @xcite , have used different approaches for detecting potentially aligned sentences within web datasets. while significant improvements have been achieved with these synthetic datasets, an in-depth investigation by @xcite has found them to be fraught with many issues, such as misalignment, wrongful language codes, etc. similarly, research has shown that data quality plays an important role in the performance of natural language processing (nlp) models, in machine translation specifically @xcite @xcite , but also more generally in other nlp tasks @xcite . it was found that often times, models that were trained on smaller amounts of high-quality data outperform their counterparts that are trained on larger amounts of noisy datasets @xcite @xcite . this has led to many studies @xcite and prior wmt tasks @xcite @xcite ) that attempt to find ways to improve the quality of existing data, which, as mentioned before, is often rife with errors. therefore, in our submission to the shared task, we experimented with filtering web-mined data for african languages using pre-trained language models and evaluated the effect of using this filtered data on machine translation performance. we defined our filtering approach as a sentence-pair binary classification task and fine-tuned a pre-trained language model using positive and negative samples. we used sentences from the high-quality mafand-mt @xcite dataset (which was included in the training data for the constrained task) as positive examples and created negative examples by extracting sentences with low language-agnostic sentence representations (laser) @xcite alignment scores from the wmt22-african @xcite corpus that was provided for this task. our results highlight the importance of filtering on the quality of the final machine translation system. we also detail how to create a high-quality filter for african languages using a few gold-standard parallel sentences. we release our codes on github. 1 the rest of the paper is organized as follows: in section 2, we review related work, and in section 3, we present the dataset we used. section 4 provides an overview of the bitext filtering approach, while section 5 details experimental settings and the translation model architecture. in section 6, we evaluate the model's performance, and lastly in section 7, we conclude and highlight some future research directions. ## related work one of the difficulties when dealing with lowresourced settings, as we do here, is that highquality parallel texts are particularly scarce @xcite . to curate data for such language pairs, methods for automatically mining parallel text from the web using heuristics @xcite or latent space and similarity-based filters @xcite have been proposed. these have led to the curation of publicly available web-mined datasets such as ccaligned @xcite , cc-matrix @xcite , paracrawl @xcite , and wikimatrix @xcite to mention just a few. however, the recent research work by @xcite shows that the automatically aligned and mined parallel bitexts, especially for lowresource language pairs, contain various degrees of errors and less than half of the data are of good quality. additionally, many approaches generate large amounts of synthetic data, often through backtranslation, where synthetic parallel data is generated by automatically translating monolingual data @xcite @xcite . while additional data has the potential to improve the trained models, these synthetic datasets are often of low quality @xcite . these observations have led to an increased interest in the automatic filtering of noisy bitexts as a key research topic in machine translation (mt). one approach to improve data quality is to filter out the noisy or invalid parts of a large corpus, keeping only a high-quality subset thereof @xcite . in this vein, numerous filtering methods have been developed @xcite @xcite . for instance, @xcite use the cosine similarity between sentence embeddings as a measure of how closely aligned two sentences are. @xcite perform back-translation and then filter based on the sentence-level bleu score, keeping only those sentences with a high bleu. similarly, @xcite perform a round-trip translation and only use the sentence pair if it is sufficiently close to the original sentence, according to a chosen similarity measure. there has also been work on alignment between two parallel corpora, and @xcite uses the laser score foot_1 to evaluate alignment, and filter out all sentences below a specific threshold. ## datasets we participated in the constrained translation track and used only the provided dataset. we present the various dataset used, their sizes and corresponding sources in in the following subsections, we describe these two categories of data. ## parallel data filtering to attempt to deal with the highly noisy data, we opted to use filtering techniques to remove many invalid or incorrectly aligned sentences, similar to prior work @xcite @xcite . we first used some simple heuristic approaches, described in section 4.1, and then progress to an automatic filtering method, detailed in section 4.2. ## mt experiments to evaluate the effect of our filtering techniques, we trained some multilingual nmt models for the 8 language pairs that we have selected for this task. in the following subsections, we highlight the model architectures, training setups, and different multilingual models that were trained. ## results and discussion in tables 6 and 7 , we report the bleu and chrf scores obtained by the different models that we trained, as evaluated on the flores-101 devtest and mafand-mt test datasets, respectively. ## conclusion and future work in this work, we used a sentence-pair classifier to classify parallel data as being aligned, or not. using this approach, we filtered out a large portion of the original, noisy, data and fine-tuned existing large language models on this new data. our results show that training on the filtered data significantly increases the performance of the models, resulting in improved translations. in particular, our approach outperforms (i) training only on clean data, (ii) training only on filtered data, and (iii) training on the original dataset, consisting of clean and noisy data. this provides additional evidence in favor of prioritizing data quality over quantity, as well as the need for more advanced noise detection guanghao xu, youngjoong ko, and jungyun seo. 2019. improving neural machine translation by filtering synthetic parallel data. entropy, 21(12):1213.