Score Combination for Improved Parallel Corpus Filtering for Low Resource Conditions (2020)

Abstract: This paper presents the description of our submission to WMT20 sentence filtering task. We combine scores from custom LASER built for each source language, a classifier built to distinguish positive and negative pairs and the original scores provided with the task. For the mBART setup, provided by the organizers, our method shows 7% and 5% relative improvement, over the baseline, in sacreBLEU score on the test set for Pashto and Khmer respectively.

## introduction neural machine translation (nmt) brings significant gains to the field of machine translation. however, it is known to be very sensitive to the quality of parallel data @xcite . this becomes a serious problem when using large but very noisy corpora for training. there is a lot of work on filtering noisy parallel data. for example, the work in (junczys-dowmunt, 2018) provides excellent results for large corpora. low resource languages are even more challenging and the results in @xcite using multilingual embeddings are very encouraging. this paper describes the system submitted to the wmt20 shared task on parallel corpus filtering for low resource conditions. due to time limitation our submission covers only sentence pair filtering. however, some of the proposed techniques could be used for sentence alignment and filtering. the task focuses on the pashto-english and khmer-english language pairs. it is required that the participants calculate scores to sort very noisy parallel sentence pairs provided for each language. the top scoring pairs leading to 5m tokens on the english side are then used to train machine translation for each language pair. the organizers also provide laser-based scores as a baseline according to the method in @xcite , with possibly some modifications. we describe our general system architecture followed by development experiments to evaluate the merit of different methods and finally report the performance of our best setup per language, using the fairseq recipe provided for the task for both full training (from scratch), and finetuning (mbart) settings. mbart @xcite is a recently proposed pretraining method. models initialized using mbart for both language pairs are provided with the task. our proposed approach combine the scores provided by the organizers with the following two scores: more details on the approach are provided in section 2. for the mbart setup our method shows 7% and 5% relative improvement in sacrebleu score on the test set for pashto and khmer respectively. ## system architecture in this section we describe the overall system architecture. we start by presenting language-based preprocessing in section 2.1.the scores provided by the organizers use pretrained laser @xcite and margin distance @xcite . actually their method obtained state-of-the-art results in wmt19 low resource filtering task for sinhala, nepali and hindi and hence is a very strong baseline. therefore, we opt to construct sentence embeddings using two different methods, namely custom laser and classifier-based embeddings, to complement the baseline pretrained laser. each embedding method is then used to generate a score for each sentence pair and all the scores are combined to form the final score. in the rest of this section, we will describe custom laser in section 2.2 followed by classifier-based embeddings in section 2.3 and finally we outline score combination in section 2.4. ## experimental results in this section we first present the results of various experiments to arrive at the final system architecture for both languages in section 3.1. an internal system with a small architecture is used for fast turn-around. this is followed by running experiments with the final architecture using the official scripts provided by the organizers for both the from scratch and mbart settings. ## summary this paper present the description of our submission to wmt20 sentence filtering task. by building custom laser and a classifier to distinguish positive and negative pairs. for the mbart setup our method shows 7% and 5% relative improvement in sacrebleu score on the test set for pashto and khmer respectively.there are a lot of extensions along the proposed directions to improve sentence filtering for the low resource setting.