Dual Conditional Cross Entropy Scores and LASER Similarity Scores for the WMT 20 Parallel Corpus Filtering Shared Task (2020)

Abstract: This paper describes our submission to the WMT20 Parallel Corpus Filtering and Alignment for Low-Resource Conditions Shared Task. This year’s corpora are noisy Khmer-English and Pashto-English, with 58.3 million and 11.6 million words respectively (English token count). Our submission focuses on filtering Pashto-English, building on previously successful methods to produce two sets of scores: LASER_LM, a combination of the LASER similarity scores provided in the shared task and perplexity scores from language models, and DCCEF_DUP, dual conditional cross entropy scores combined with a duplication penalty. We improve slightly on the LASER similarity score and find that the provided clean data can successfully be supplemented with a subsampled set of the noisy data, effectively increasing the training data for the models used for dual conditional cross entropy scoring.

## introduction machine translation systems require large amounts of high quality parallel corpora for training. neural machine translation models in particular have been found to both require more data @xcite , and be more sensitive to noise in training data @xcite than statistical machine translation models. while these data can be acquired from online sources, the resulting crawled texts are often noisy and require filtering to produce large amounts of sufficiently clean training data. ## related work we refer readers to @xcite for a more detailed overview of methods for parallel corpus filtering, here we describe the most relevant methods to this work. ## shared task for this year's shared task on parallel corpus filtering and alignment for low-resource conditions, participants are asked to produce scores for each of the sentence pairs in the provided noisy 58.3 million-word (english token count) khmer-english corpus and 11.6 million-word pashto-english corpus. these scores are used to subsample sentence pairs amounting to 5 million english words. the resulting subset is evaluated by the quality of an nmt system (fairseq @xcite ) trained on this data. participants were given the scripts to either train the evaluation system from scratch, or use the data to fine-tune a provided pretrained mbart model. the mbart model was trained on monolingual data, the details of which are described in @xcite . the performance of the nmt system is measured by bleu score on a held-out test set of wikipedia translations. participants may also provide re-alignments of the source and target sentences. the organizers provide clean parallel and monolingual data for both of the language pairs, as well as laser similarity scores, a previously successful method in low-resource conditions @xcite , @xcite . we participated in the pashto-english track only, after finding that the model-based methods we explored did not produce meaningful scores for khmer-english. we did not submit sentence realignments, focusing instead on sentence filtering. our submission builds on previously successful methods from past wmt shared tasks on parallel corpus filtering to produce two scores: laser lm, a combination of the laser similarity scores and perplexity scores from language models, and dccef dup, dual conditional cross entropy scores combined with a a duplication penalty. all bleu scores listed in this paper come from systems trained from scratch and run on the provided development data. ## laser lm a shortcoming of laser similarity scores is that they may produce a false positive in the event that the source and target embeddings are similar to each other, but not good translations of each other. consider, for example, a source and target pair in which the target is simply a copy of the source. this is clearly not a good translation; nothing has been translated. however, the embeddings would be exactly the same, and thus appear to be a very good match. this exact scenario is easily remedied by the use of a language identification filter, but other instances may be more difficult to root out. for example, a source and target sentence in which the target sentence is a string of literal word-for-word translations of the source sentence. to complement the laser similarity scores and introduce some measure of fluency we train a language model for both english and pashto. ## dccef dup the dual conditional cross entropy scores produced state-of-the-art performance on the wmt18 shared task on filtering corpora for high-resource languages. however, this method requires two translation models trained in both the forward and backward direction. this presents a challenge in lowresource conditions due to the limited training data available. we find that the model quality can be improved by supplementing the provided clean data with a subsampled set consisting of 1m english tokens of the noisy data, subsampled based on the laser similarity scores. ## results various other combinations of the aforementioned scores were explored, and the results are listed in none of the filtering methods significantly outperformed the laser-based method, but the improved dccef filtering method can at least match the laser-based method when the training data is augmented, and the preprocessing steps and duplication penalty are applied. ## conclusion this paper describes the our submission to the wmt20 parallel corpus filtering shared task for low-resource conditions. we find that filtering based on dccef scores can compete with filtering based on laser similarity scores when the models trained for the dccef scores are augmented with a subsample of the noisy data. this suggests that