Findings of the WMT 2020 Shared Task on Parallel Corpus Filtering and Alignment (2020)

Abstract: Following two preceding WMT Shared Task on Parallel Corpus Filtering (Koehn et al., 2018, 2019), we posed again the challenge of assigning sentence-level quality scores for very noisy corpora of sentence pairs crawled from the web, with the goal of sub-selecting the highest-quality data to be used to train ma-chine translation systems. This year, the task tackled the low resource condition of Pashto–English and Khmer–English and also included the challenge of sentence alignment from document pairs.

## introduction the field of machine translation has experienced significant advances in recent years thanks to improvements in neural modeling @xcite @xcite , as well as the availability of large parallel corpora for training @xcite @xcite . unfortunately, today's neural machine translation models, perform poorly on low-resource language pairs, for which clean, high-quality training data is lacking @xcite . improving performance on low resource language pairs has high impact considering that these languages are spoken by a large fraction of the world population. this is a particular challenge for industrial machine translation systems that need to support hundreds of languages in order to provide adequate services to their multilingual user base. while there have been advances in using monolingual corpora @xcite and parallel corpora in multiple language pairs @xcite , the best training data for machine translation are still parallel corpora in the targeted language pair and domain. parallel corpora are typically gathered from any available source without much guarantees about quality. this is especially the case for parallel corpora that are extracted from the web without much control over which web sites are mined. since noisy training data has been recognized as a challenge for neural machine translation training @xcite , an essential step in using such data is filtering or discounting noisy sentence pairs. recently, there is increased interest in the filtering of noisy parallel corpora to improve the data that can be used to train translation systems. the shared task on parallel corpus filtering and alignment at the conference for machine translation (wmt 2020) was organized to promote research to make learning from noisy data more viable for low-resource languages. it is similar to the previous year's task but tackles different languages (pashto and khmer instead of nepali and sinhala) and also included the challenge to extract sentence pairs from document pairs. the shared task is organized similarly to previous years @xcite . we provide about 11.6 million word noisy parallel data for pashto-english and 58.3 million word noisy parallel data for khmer-english. we also provide small amounts of clean parallel data of varying quality and monolingual data from wikipedia and com-moncrawl. participants developed methods to assign a quality score for each sentence pair. these scores are used to filter the web crawled corpora down to a fixed size (5 million english words), train neural machine translation systems on these subsets, and measure their quality with the bleu score on a test set of multi-domain wikipedia content @xcite . this paper gives an overview of the task, presents the results for the participating systems and provides analysis on additional subset sizes, the average sentence length of sub-selected data, and overlap between the submissions. ## related work although the idea of crawling the web indiscriminately for parallel data goes back to the 20th century @xcite , work in the academic community on extraction of parallel corpora from the web has so far mostly focused on large stashes of multilingual content in homogeneous form, such as the canadian hansards, europarl @xcite , the united nations @xcite , or european patents @xcite . a nice collection of the products of these efforts is the opus web site foot_0 @xcite . ## shared task definition the shared task tackled the problem of filtering parallel corpora. given a noisy parallel corpus (crawled from the web), participants developed methods to align sentences in document pairs and to filter it to a smaller size of high quality sentence pairs. ## data we provided three types of data for this shared task: (1) clean parallel and monolingual data, including related language data in hindi, to train models that aid with the filtering task, (2) the noisy parallel data crawled from the web which participants have to score for filtering, and (3) development and test sets that are used to evaluate translation systems trained on filtered data. ## evaluation protocol the testing setup mirrors the development environment that we provided to the participants. ## results in this section we present the results of the shared task evaluation. we added additional unofficial condition at 2, 3, and 7 million english words, to better observe tendencies. ## conclusion we report on the findings of the wmt 2020 shared task on parallel corpus filtering and alignment. ten participants used a variety of methods that gave quite different results, as measured by translation quality, optimal subset sizes, sentence length, etc. we hope that this task provides a benchmark for future research and improvements on this task.