Unsupervised Feature Selection for Effective Parallel Corpus Filtering (2023)

Abstract: This work presents an unsupervised method of selecting filters and threshold values for the OpusFilter parallel corpus cleaning toolbox. The method clusters sentence pairs into noisy and clean categories and uses the features of the noisy cluster center as filtering parameters. Our approach utilizes feature importance analysis to disregard filters that do not differentiate between clean and noisy data. A randomly sampled subset of a given corpus is used for filter selection and ineffective filters are not run for the full corpus. We use a set of automatic evaluation metrics to assess the quality of translation models trained with data filtered by our method and data filtered with OpusFilter’s default parameters. The trained models cover English-German and English-Ukrainian in both directions. The proposed method outperforms the default parameters in all translation directions for almost all evaluation metrics.

## introduction neural machine translation (nmt) is dependent on large parallel text corpora. available training data can often be noisy, especially if the data is retrieved by the common method of extracting bitexts from web crawls @xcite @xcite . training nmt on noisy data can be detrimental to the translation models. ensuring that the train-ing examples are clean sentence pairs leads to better translation quality and more efficient training @xcite . if clean parallel corpora are not readily available, a common practice is to refine a noisy corpus by filtering out low quality training examples. the amount and type of noise varies between different corpora. selecting the kind of filters that are optimal for cleaning a specific parallel corpus can take a lot of trial and error. several methods and tools for corpus cleaning have been proposed and developed @xcite @xcite . opusfilter @xcite ) is one such toolkit. it provides a selection of configurable filters, but suffers from the same issue of having to manually choose the filters and their parameters. in this work, we propose an unsupervised method of selecting effective filters and filtering thresholds based on the properties of a given corpus. our method automatically generates a filtering configuration file which serves as a solid starting point for finding the optimal settings for an opusfilter corpus cleaning pipeline. we assess the proposed method by comparing the translation quality of models trained with data filtered with default parameters from opusfilter and data filtered with autogenerated parameters. our implementation of the filter selection method is available at https://github . com/helsinki-nlp/opusfilter. ## related work corpus cleaning has been a part of training pipelines since the statistical machine translation (smt) era. some of the most common and most straightforward methods include sentence length based methods, for example removing too short and too long sentences and sentence pairs where the ratio of source and target lengths is above a given threshold. the moses toolkit @xcite offers commonly used scripts for this purpose. @xcite map sentence pairs into an n-dimensional space and filter out the outliers. @xcite propose a graph-based random walk filtering method which is based on the idea that better sentence pairs lead to better phrase extraction and that good sentence pairs contain more frequent phrase pairs. the zipporah data cleaning system @xcite maps sentence pairs into a feature space and uses logistic regression to classify good and bad data. as the features, they use bag-of-word translation scores and n-gram language model scores. training data quality has a strong effect on nmt performance. @xcite study several types of noise and their impact on translation quality. they report that nmt is less robust against noisy data than smt. @xcite points out common problems in parallel corpora that can result in low quality nmt and provides filters to overcome these issues. these problems include mismatch of non-alphabetic characters between source and target segments, wrong language and repeating tokens. ramírez-sánchez et al. (2020) present two tools for more careful corpus cleaning with nmt in mind: bifixer and bicleaner. bifixer is a restorative cleaner; it only removes sentence pairs with either side being empty but otherwise it fixes textrelated issues in place. bifixer corrects character encoding and orthography issues, conducts re-splitting of the sentences and identifies duplicates. bicleaner consists of filtering rules, language model scoring and a classification part. the filtering rules are predefined, but other steps of bicleaner require training a language model and a classifier. however, pretrained models are provided for many language pairs. opusfilter @xcite ) is a configurable parallel corpus cleaning toolbox. opusfilter provides a variety of data selection, text processing, filtering and classification features that can be combined into a reproducible corpus cleaning pipeline. an important step in constructing this pipeline is to choose which filters to use and with what parameters. the filters work by producing a score for a sentence pair and checking whether the score exceeds a threshold value. opusfilter defines default threshold values for each filter, but there is no guarantee that these values are optimal for a given corpus and language pair. we propose an unsupervised method to choose filters that are useful in differentiating between clean and noisy sentence pairs and to initialize threshold values based on features extracted from a parallel corpus. the approach consists of clustering sentence pairs into noisy and clean categories and using the features of the noisy cluster center as the threshold values. this method is especially useful in setting initial opusfilter parameters that are adapted to the characteristics of a given corpus. ## method our proposed method of selecting relevant filters and useful threshold values for opusfilter is based on clustering sentence pairs into clean and noisy categories and using the features of the noisy cluster center as our filtering parameters. to select the filters that are actually useful in detecting noisy sentence pairs, we convert the clustering task into a classification task and find the features that affect classification accuracy the most. for clustering, classification and feature importance inspection, we use the scikit-learn python package @xcite . ## translation experiments in order to assess the impact of our data filtering method, we train translation models for english-german (en-de) and english-ukrainian (en-uk) in ## conclusion we propose an unsupervised method for selecting filters and filtering thresholds for opusfilter. we evaluate our method in translation tasks where we train models on data filtered with the default parameters of opusfilter and another set of models trained on data filtered with generated filtering configuration files. the autogeneration method outperforms the default parameters in almost all cases. additionally, our method makes corpus filtering more efficient as we only run useful filters with appropriate parameters on the full training set. in future work, we will evaluate our method in a larger variety of corpus cleaning scenarios to confirm our findings. one point of interest is to test the method for corpora with different proportions of noisy data. we will also conduct tests in lowresource language settings. additionally, we will evaluate the effects of expanding our approach by integrating a larger range of different filters. in order to improve the autogeneration method, more careful analysis of the feature selection process will be performed, for example manual evaluation of sentence pairs in noisy and clean categories in order to assess the clustering accuracy. we will also explore using statistical inference (e.g. welch's t-test) for finding effective filters as an alternative for the feature importance analysis. relying on statistical significance could be a more robust approach for discarding filters than the current rejection coefficient method.