O pus F ilter: A Configurable Parallel Corpus Filtering Toolbox (2020)

Abstract: This paper introduces OpusFilter, a flexible and modular toolbox for filtering parallel corpora. It implements a number of components based on heuristic filters, language identification libraries, character-based language models, and word alignment tools, and it can easily be extended with custom filters. Bitext segments can be ranked according to their quality or domain match using single features or a logistic regression model that can be trained without manually labeled training data. We demonstrate the effectiveness of OpusFilter on the example of a Finnish-English news translation task based on noisy web-crawled training data. Applying our tool leads to improved translation quality while significantly reducing the size of the training data, also clearly outperforming an alternative ranking given in the crawled data set. Furthermore, we show the ability of OpusFilter to perform data selection for domain adaptation.

## introduction data filtering tools are important to reduce the noise fed into machine learning algorithms such as the ones used in neural machine translation. this is especially true for data sets with suspicious sources like unrestricted web crawls or data sets that are automatically extracted from complex data formats such as pdf or html with all their different flavors and implementations. cleaning parallel corpora is a special case in which not only the raw data but also the quality of alignment between source and target language needs to checked. the aligned translations drive the mapping from input to the output language as a strong supervision during the training steps, and the amount of noise will have a decisive impact on the adequacy of the translations. the effect is especially severe for low resource settings, in which little data is available, and each mistake might directly influence the end result. the interest in automatic bitext (i.e. bilingual parallel corpora) filtering is constantly growing pushed by the advances in neural machine translation. @xcite show that noisy training data is often more harmful for neural translation models than statistical translation models. as a consequence, international evaluation campaigns like the ones organised by wmt now feature shared tasks on data cleaning and ranking @xcite . various approaches have been proposed based on such challenges and directly benefit the development of mt engines in low-resource settings. this paper presents a framework for bitext cleaning, opusfilter, focusing on processing data collected in opus @xcite , the world's largest resource of openly available parallel corpora. in contrast to tools such as bicleaner @xcite and zipporah @xcite , that implement a single method for parallel corpus filtering, opusfilter is designed as a toolbox that is useful for testing and using many different approaches. below we describe the design of opusfilter and present its application in the test case of filtering finnish-english parallel data included in paracrawl. ## opusfilter toolbox the opusfilter toolbox is implemented in python 3 and is available at https://github.com/ helsinki-nlp/opusfilter under the permissive mit open-source license. the main script provided by the package is opusfilter, which takes a configuration file as an input. the configuration files are written in yaml syntax. 1 a configuration contains common global options (currently only the output directory) and a list of steps that are run one by one. there are different step types (functions) for downloading parallel corpora from the opus database, combining and taking subsets of the corpora, filtering and scoring segment pairs with a combination of different filters, and training and using a classifier based on the scores. the configuration files for all our experiments are included in the github repository. the input and output files for the functions are defined in the configuration, and it is simple to also use external data files. in contrast to common text file processing tools, the opusfilter functions support the processing of parallel files that have corresponding data on the same lines. special attention has been paid to make the processing of large files memory-efficient: full corpora are never loaded into memory, but the segment pairs and scores are processed one at a time if possible, and in fixed-size chunks otherwise. in this section, we describe the current functionality of the opusfilter toolbox. in the future, we plan to include more functions for common monolingual and parallel data processing operations. the ultimate goal is that all pre-processing steps could be defined in a single configuration file making it easy to share them for reproducing mt experiments. ## experiments to demonstrate the usefulness of the opusfilter toolbox, we show results from two main experiments on the finnish-english news translation task (in both directions): (i) filtering noisy data, and (ii) applying domain adaptation. for training, we use data from version 4 of the paracrawl corpus @xcite . the data is taken from a general internet crawl and contains segments that are noisy and potentially harmful for machine translation models. we use the subset of the corpus that is already filtered by the bicleaner tool foot_3 @xcite . this data set contains 2,156,069 segment pairs and is ordered by the score from bicleaner, which enables us to directly compare it to our tool. we create five versions of the training data by removing 10%, 20%, 30%, 40% and 50% of the pairs from the noisy end of the collection and train translation models with the full data and with the five reduced training sets. next, we reorder the data with our toolkit and again create new data sets by removing data with the same proportions as previously. we then apply data provided for the wmt news translation task foot_4 for validation and testing. in particular, we use newstest2018 as the development set and newstest2019 as our test set for both language directions. the translation models are trained with the opennmt toolkit @xcite using rnn encoders and decoders with lstm gates. all training sets are tokenized with the tokenizer from the mosesdecoder toolkit @xcite and segmented with bpe @xcite using subword-nmt foot_5 before feeding them to opennmt. ## conclusions and future work this paper introduces opusfilter, a modular tool for parallel data selection and ranking. opusfilter can easily be configured to work with opus data and various filters to train effective classifiers in order to rank bitext segments. we demonstrate its use in a finnish-english translation task based on the noisy paracrawl data used for training. the classifiers can be trained without human annotation, and the automatic model selection methods implemented in the toolbox lead to a similar performance compared to classifiers based on small manually labeled validation data. opusfilter is open source and distributed with a permissive license to make it widely applicable. in future work, we would like to extend the toolbox with additional filters and classification options. one option could be the inclusion of sentence embedding based filtering @xcite . additionally, we would like to explore opusfilter's use in different scenarios and for other language pairs. especially interesting