Bifixer and Bicleaner: two open-source tools to clean your parallel data (2020)

Abstract: This paper shows the utility of two open-source tools designed for parallel data cleaning: Bifixer and Bicleaner. Already used to clean highly noisy parallel content from crawled multilingual websites, we evaluate their performance in a different scenario: cleaning publicly available corpora commonly used to train machine translation systems. We choose four English–Portuguese corpora which we plan to use internally to compute paraphrases at a later stage. We clean the four corpora using both tools, which are described in detail, and analyse the effect of some of the cleaning steps on them. We then compare machine translation training times and quality before and after cleaning these corpora, showing a positive impact particularly for the noisiest ones.

## introduction parallel corpora are usually the main source of information used to learn machine translation models. the availability of corpora has encouraged the advance of machine translation in both academy and industry settings. publicly available parallel corpora (europarl, news commentary, united nations, etc.) have been used for decades now, not only to produce machine translation but also other by-products such as dictionaries, concordances, synonyms, paraphrases, etc. in machine translation, due to the ability of statistical models to hide imperfections without statistical significance, filtering out noise from these corpora was not very c 2020 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. inspired by recent work on filtering parallel corpora to maximize the quality of machine translation from the shared tasks organised at wmt18 1 and wmt19 2 , we review how noisy some of the most popular or recent publicly available corpora are and how this impacts the quality of the output of state-of-the-art neural machine translation. our motivation is twofold: getting high-quality monolingual and bilingual data and getting high-quality machine translation for english-portuguese. we will further use this resources to compute paraphrases in the framework of a research project. in order to inspect and filter out noise, we use bifixer and bicleaner, 3 a couple of publicly available cleaning tools released as part of the paracrawl european project. 4 these tools have been mainly used to filter out noise from the raw version of automatically crawled parallel corpora in more than 30 language combinations. here we use them in a very different scenario: we take already released publicly available corpora, either widely used in the past or recent. we analyse the main problems of the corpora and review the cleaning steps and their impact on the final size of the corpora. to evaluate the effect of cleaning, we train neural machine translation systems before and after filtering them and report both performance results and evaluation through automatic metrics. we do so for english and portuguese in both translation directions as portuguese is one of the target languages in our research project related to paraphrasing. our focus for this paper is, though, the evaluation of the cleaning tools intrinsically and extrinsically through machine translation. the rest of the paper is organised as follows: in section 2 we discuss the cleaning steps applied to the corpora and analyse the type of noise found in them; section 3 describes the mt experiments and reports on the results; finally, section 4 depicts the conclusions and some ideas for future work. ## cleaning parallel corpora although parallel corpora cleaning has been explored in previous works, the most recent stateof-the-art can be found as part of the findings of the shared tasks on parallel corpora filtering in wmt18 @xcite and wmt19 @xcite . participants in these shared tasks applied a bunch of techniques looking for highquality data inside noisy corpora. most of these techniques are a mixture of pre-filtering rules for obvious noise, scoring functions of all sorts (language models, neural translation models, etc.) and classification to discriminate between high-quality and low-quality sentence pairs. diverse techniques have been applied to both high-resource and lowresource languages. the results encourage filtering, especially for high-resource scenarios involving neural machine translation. on the other hand, no clear trend was devised for low-resource scenarios nor for statistical machine translation. some of these techniques have been already implemented and evaluated in bicleaner @xcite . bifixer adds a different way of exploring corpora cleaning: restorative cleaning. with this step, we aim at fixing content and getting unique parallel sentences before filtering out noise. ## evaluation through machine translation in order to evaluate the impact of cleaning, we train neural machine translation systems before and after cleaning for each of the four corpora inspected. this allows us to see if better and reduced versions of the corpora produce a better machine translation output. we measure the impact of cleaning in the output by using automatic metrics. we also measure training times to see if size reduction and a more consistent content leads to a more efficient training process. machine translation systems are trained on each corpus before and after cleaning, for both translation directions and for both 0.7 and 0.5 bicleaner thresholds. we train transformer-base models with 32,000 vocabulary using marian @xcite and sentencepiece. we use development and test sets from ted talks proposed by @xcite and report bleu scores computed with sacrebleu 18 . results for bleu scores for all the 24 systems are reported in from the results, we can see that cleaning has a positive impact on all the corpora, both in speeding up training times and in slightly improving bleu scores for almost all corpora and translation directions: only europarl, english-portuguese, just stays the same. thus, no degradation is introduced with corpora size reduction, but rather the opposite: the most aggressive cleaning (0.7) scenario, leading to the smallest corpora sizes, gets consistently better bleu scores for all the experiments. this scenario leads also to the best training times in most cases. indeed, the highest improvements in bleu scores (from +1 to +2.2 absolute bleu points) are obtained when 22m sentences (twothirds of the corpus) are filtered out from open-subtitles. ## conclusions we have applied bifixer and bicleaner, two opensource tools built inside the paracrawl project, to 18 signature: bleu+case.mixed+lang.pt-en+numrefs.1 +smooth.exp+tok.13a+version.1.4.2 clean four publicly available parallel corpora for english-portuguese. after a review of the tools and the cleaning steps performed to the four corpora, we evaluate the output of neural machine translation before and after cleaning them to see their impact. cleaning reduces the size of the corpora. for some of them (europarl, jw300), the reduction is low but for others, cleaning removes half of the corpus (wikimatrix) or up to two thirds (opensubtitles). cleaned corpora, in the most aggressive cleaning scenario (bicleaner scores above 0.7), lead consistently to equal or slightly better results for bleu scores in machine translation, not degrading the results in any case and speeding up machine translation training times. at bigger scale (more languages, bigger sizes for all corpora together) all this could result in remarkable savings of disk space and training times without compromising machine translation quality and producing higher-quality corpora. both tools can be currently used without any further effort for more than 30 language combinations and prove to be a cheap and effective step before using parallel corpora for machine translation or other natural language processing tasks. for non-supported languages, bifixer will only require a list of monolingual safe replacements for typos. bicleaner, though, will require training resources and time, although much less than other methods. from a closer look, we observe that, for less noisy corpora as europarl, some of the bicleaner pre-filtering rules are too severe and could probably be relaxed. in particular, the removal of too short sentences should be further inspected for already high-quality data. as further work, although bifixer and bicleaner have been used for many other languages inside the paracrawl project, it would be interesting to validate the results obtained in this paper for other language combinations and corpora. outside machine translation, we believe that cleaning is also good for other tasks such as improving sentence alignment or paraphrase extraction. both, and specially paraphrase extraction, will be explored as further work as part of a research project that will use the results of this paper as best practices to pre-process corpora.