Evaluating Corpus Cleanup Methods in the WMT ’22 News Translation Task (2022)

Abstract: This submission to the WMT22: General MT Task, consists of translations produced from a series of NMT models of the following two language pairs: german-to-english and german-to-french. All the models are trained using only the parallel training data specified by WMT22, and no monolingual training data was used. The models follow the transformer architecture employing 8 attention heads and 6 layers in both the encoder and decoder. It is also worth mentioning that, in order to limit the computational resources that we would use during the training process, we decided to train the majority of models by limiting the training to 21 epochs. Moreover, the translations submitted at WMT22 have been produced using the test data released by the WMT22.The aim of our experiments has been to evaluate methods for cleaning-up a parallel corpus to determine if this will lead to a translation model producing more accurate translations. For each language pair, the base NMT models has been trained from raw parallel training corpora, while the additional NMT models have been trained with corpora subjected to a special cleaning process with the following tools: Bifixer and Bicleaner. It should be mentioned that the Bicleaner repository doesn’t provide pre-trained classifiers for the above language pairs, consequently we trained probabilistic dictionaries in order to produce new models. The fundamental differences between these NMT models produced are mainly related to the quality and the quantity of the training data, while there are very few differences in the training parameters. To complete this work, we used the following three software packages: (i) MARIAN NMT (Version: v1.11.5), which was used for the training of the neural machine translation models and (ii) Bifixer and (iii) Bicleaner, which were used in order to correct and clean the parallel training data. Concerning the Bifixer and Bicleaner tools, we followed all the steps as described meticulously in the following article: “Ramírez-Sánchez, G., Zaragoza-Bernabeu, J., Bañón, M., & Rojas, S.O. (2020). Bifixer and Bicleaner: two open-source tools to clean your parallel data. EAMT. ” and also in the official github pages: https://github.com/bitextor/bifixer , https://github.com/bitextor/bicleaner .

## introduction this submission presents the contribution of the lt'22 team to the wmt22: general mt task. it focuses on studying the effectiveness of cleaning tools when these are applied to real-world parallel corpora, to eliminate noisy sections and improve the resulting nmt systems. traditionally, parallel corpora are used as the primary data source for machine translation (mt) models. the development of mt has been aided by the availability of extensive parallel corpora. the majority of these data have several areas of reduced parallelism and are usually characterized as imperfect or noisy. the use of noisy data may result in a neural machine translation model being inadequately prepared. researchers (e.g. @xcite ) have reported that neural machine translation models are much more affected by noisy data than statistical machine translation models. a number of software packages to implement noise-removal from parallel corpora have been implemented and released to the community. these include publicly available tools such as qe-clean (denkowski) 1 , as well as zipporah (xu and koehn, 1 https://github.com/mjdenkowski/qe-clean 2017). @xcite have used a combination of alignment-indicating features to clean corpora. for cleaning large-scale corpora in multilingual setups, a cosine-distance metric has been proposed @xcite . finally, the suite of the paired bifixer and bicleaner software tools @xcite has been proposed for parallel corpora cleaning purposes, with bifixer implementing restorative cleaning and bicleaner providing the ability to remove sentences with very low parallelism in the parallel corpus. for the experiments reported here, two language pairs have been chosen, namely german-to-english (denoted as de-to-en) and german-to-french (denoted as de-to-fr). compared with other systems reported in wmt, our nmts have a couple of identifying features: (1) the use of a fully-constrained setup with respect to wmt'22 rules and (2) the setting of a relatively low threshold to the allowed training epochs, in an effort to comply to a setup with limited computational resources. whilst our translation systems are not as accurate as they could be if more epochs were allowed, it was decided to adopt an approach that is more realistic when training resources are not unlimited. to implement the lt'22 participation to the wmt'22 shared task work, we used the following three software packages: (i) the marian nmt toolkit (version: v1.11.5), which was used for the training of the neural machine translation models and (ii) bifixer and (iii) bicleaner, which were used in order to correct and clean our data. regarding the structure of the paper, in the second section the selection of data on which to train the translation systems is reported. in the third section, the method used to carry out all essential experiments is detailed. in the fourth section, the corpus-cleaning tools are analyzed. in the fifth section the translation systems and their parameters are reported. the sixth section is devoted to details related to experiments. finally, we review the findings of this series of experiments and examine potential future research directions. ## training data our experiments involve comparing the translation outputs for a series of nmt models for two language pairs: german-to-english (denoted as de-to-en) and german-to-french (denoted as de-to-fr). it should be noted that for these two language pairs no pretrained models for either bifixer or bicleaner are available at the respective repository. all the nmt models reported here are trained using only the parallel training data specified by wmt'22, and no monolingual training data are used. in-training validation has been performed using the development data recommended in wmt'22, whilst for evaluating the trained nmt systems (developed prior to the release of wmt'22 test data), the relevant test data from wmt'20 were used. moreover, the translations submitted at the wmt22 shared task have been produced using the test data released by wmt'22. ## methodology the aim of our experiments has been to evaluate methods for cleaning-up a parallel corpus and to determine if their use leads to mt systems that generate more accurate translations. for each language pair, baseline nmt models have been trained from raw (i.e. unfiltered) parallel training corpora as specified by wmt'22, while the additional nmt models have been trained with corpora subjected to a special cleaning process via the bifixer and bicleaner suite @xcite . it should be mentioned that the bicleaner repository foot_0 doesn't include pre-trained classifiers for the above language pairs; consequently we trained probabilistic dictionaries in order to produce new models. an added benefit of this choice is that no pre-trained model was used to develop our nmt systems, and thus the submitted systems reviewed here are constrained. the fundamental differences between the nmt models produced are mainly related to the quality and quantity of the training data, while there are no differences in the training parameters or in the setup of the deep neural network architectures (unless otherwise noted in the experimental section). by doing so, it is possible to safely draw conclusions about the amount of computational resources required while also examining and comparing the translation outputs using automatic assessment methods. the following were the driving factors behind the experiments reported here: ## training the nmt systems for training neural machine translation models, we chose the marian nmt toolkit @xcite . marian was developed to allow rapid training and translation speed, to facilitate the standardization of research work. all the models we trained adopted the architecture of a sequence-tosequence transformer with 8 attention heads and 6 layers in both the encoder and decoder, thus largely adhering to the standard transformer configuration from @xcite . we also decided to set a specific limit to the number of training epochs to avoid lengthy training sessions, aiming to economize as far as possible on valuable computing resources, as per the recent acl recommendation for efficient computing @xcite . the transformer is characterized as innovative and uncomplicated @xcite . in our experiments, we activated the dropout mechanism, which is a widely adopted regularisation technique in nmt. when training our nmt systems, we opted to use the sentencepiece tokenizer, which has the ability to train subword models straight from unprocessed data @xcite . the vocabulary size was set to 32000 and the range for the batch size was from 64 to 100. for the workspace size we used a variable value across our experiments, as the size of the training corpora varied due to the bicleaner filtering. as suggested by the marian developers, the workspace was adapted via a number of trial runs at the start of the training process, to maximise the throughput of training sentences per time unit. the other main parameter choices for the transformer models are shown in ## conclusions and future work in this paper, we have presented our submission to the wmt22: general mt task. in order to rectify and filter noisy sentences from the corpora recommended by wmt'22, we have applied two cleaning approaches for the parallel corpus. after experimenting with various categorization criteria, we created seven distinct parallel corpora. we discovered that as expected, thoroughly cleaned corpora require fewer computer resources, as a large number of sentences are removed. additionally, we noticed that differences in the bleu score across cleaned corpora are relatively small. our main submissions to the shared task were two, one for each language pair. regarding the language pair german to english, the highest quality translation result was obtained by training a transformer model using the raw baseline corpus, and thus the use of bifixer/bicleaner did not lead to an improvement. the best result was obtained for the language pair german to french by training a transformer model using the bifixed and bicleaned parallel corpus with a threshold of 0.5. in upcoming research, the back translation technique is planned to be utilized in order to expand the size of the training data, since the size of the sentence pairs is reduced after the cleaning procedure. the translations that emerged from the aforementioned experimental process could be filtered and reused so as to train the nmt system with bigger and cleaner parallel corpora. in the future, it would be highly interesting to develop probabilistic dictionaries with more than 10 m parallel sentences as well as to train the bi-cleaner in more than 100 k parallel sentences. additionally, we want to use these methods on even more information about the language pairs we previously stated. in order to achieve an even cleaner corpus, it would also be quite fascinating to investigate comparatively other cleaning techniques such as those reported in the introduction. a final direction for future work would be to use larger models such as the big transformer @xcite to see if for this architecture the effect of pre-filtering with bifixer/bicleaner will be more marked, and what the trade-off between the improvement in translation quality and the increased training time would be. limitations one potential limitation of the present work is the relatively limited range and number of bicleaner thresholds tested, though the values include both the recommended and default values. another limitation concerns the use of a single architecture, whilst ideally a second architecture (such as the transformer-big configuration of @xcite ) could be used. finally, a comparison with other corpus-cleaning methods would be desirable, though such work is beyond the scope of the present work. ethics statement the present work is not expected to have any effect on ethical issues and to the authors' best knowledge complies with the acl ethics policy.