Filtering Noisy Parallel Corpus using Transformers with Proxy Task Learning (2020)

Abstract: This paper illustrates Huawei’s submission to the WMT20 low-resource parallel corpus filtering shared task. Our approach focuses on developing a proxy task learner on top of a transformer-based multilingual pre-trained language model to boost the filtering capability for noisy parallel corpora. Such a supervised task also helps us to iterate much more quickly than using an existing neural machine translation system to perform the same task. After performing empirical analyses of the finetuning task, we benchmark our approach by comparing the results with past years’ state-of-theart records. This paper wraps up with a discussion of limitations and future work. The scripts for this study will be made publicly available.

## introduction crawling web has been regarded as a de facto approach to produce bitexts, yet the crawled texts are under-qualified often in some aspects to train a proper machine translation system. underqualified bitexts present misalignments, no alignments, wrong language pairs, sentences mostly composed of numbers and mathematical formulas, etc. parallel corpus filtering in this manner holds a critical research area to improve the performance of machine translation systems. wmt organizes a shared task for parallel corpus filtering since 2018 intending to filter our noisy bitexts to this end. the challenge targets low-resource language pairs since 2019. many existing filtering methods require multiple layers of elimination by implementing manually engineered features such as length filtering, language identification, normalizing, etc. these hand-picked features work well for a language pair but don't generalize well to another language pair or domain and often bring algorithmic complexity to the overall system. the laser @xcite ) model achieved state-of-the-art (sota) records at the wmt19 shared task on low-resource parallel corpus filtering @xcite . the sentence representation model implemented in laser provides a means for measuring the similarity between a source and a target sentence. as stated in the future work at @xcite , there is still space to improve. utilizing a self-attention mechanism remains future work as the laser was not built upon the latest transformer architecture @xcite . we are also interested in designing a filtering tool that can be efficiently applied to a wide range of language pairs. pre-trained multilingual language models, such as bert @xcite and roberta @xcite , are exploited to this end. we make two contributions to the field in this manner. the first contribution is a proposal of approaching the filtering problem as a discrimination task that can be trained with a proxy task and synthetic training data generation (see in section 3.1). the other contribution is the empirical knowledge learned from an analysis of the finetuning pre-trained multilingual language models on cross-lingual discrimination tasks. ## related work in the wmt18 shared task, participants mostly used similar techniques in components as prefiltering, scoring the sentence pairs, and using a classifier for feature functions. teams applied prefiltering rules to eliminate noisy data, including: scoring functions were mostly used to correlate qualified texts. participants also used sentence embeddings @xcite @xcite altogether with a similarity function to detect the similarity of pairs. the wmt19 shared task focused on lowresource languages, namely nepali-english and sinhala-english. participants mostly applied basic filtering techniques similar to those used in 2018. @xcite used sentence embeddings that were trained on parallel sentence pairs. another approach was to train a machine translation system on the clean data and then used it to translate the non-english side to make a comparison. several metrics were used to match sentence pairs such as meteor, levenshtein distance, and bleu. we found that our work relates to the submission from @xcite . however, their submission was unable to show the effectiveness of the proposed method due to potential issues in the pretraining process. besides the parallel corpus filtering task, we come across several works utilizing a similar approach. in @xcite , bert rescoring method is more effective at bitext mining than heuristic scoring methods, i.e., marginal cosine distance. in grégoire and langlais (2018), a similar negative random sampling technique has been used for generating synthetic bad pairs. also, attempts to create harder negative pairs were proven effective in bitext mining @xcite . ## methodology transformer models are currently state-of-the-art systems on most nlp classification and regression tasks. with the emergence of multilingual pretrained models, their cross-lingual capabilities can be exploited with little effort. ## results there is a relationship between f1 scores of the proxy task and the final nmt system performance (figures 1 2 ). improvements of the final nmt in the proxy task peaks along with the negative sampling rate and decreases potentially due to overfitting. by looking at the same ratio presented in figures 1 2 , we can conclude a correlation between the performance of the proxy task and that for the filtering task, showcasing the proposed approach's effectiveness. wmt20 here we have presented our nmt performances of the submitted filtering systems in ## conclusions and future work we illustrate our submission to the wmt20 lowresource parallel corpus filtering task. by developing a proxy task learner on top of a transform-based pre-trained language model xlm-r, we are able to improve the filtering capability for noisy data, achieving sota results. the parallel corpus filtering task is recalloriented. therefore our model may not be suitable for high-precision jobs. the model has limitations in dealing with short sentences. it can be improved by finetuning on dictionaries or phase-based bitexts. the model performs better in low-resource and high-recall settings. in our experiments depicted in the subsection 3.1.6, we observe low performances several times. it may appear the model is suffering from the random seeds caused fragility mentioned in @xcite . a close look ascribes these abnormal results to the randomness in the sampling strategy. we leave this issue to future work. different kinds of synthetic noise generation techniques can be adapted to increase the robustness and accuracy of the model. for example in the filtered data we observed several false-positive cases which contains mis-translated numbers: en reference: training an nmt model on this type of data hurts the translation performance. but this kind of noise can be fixed by altering the numerical values in the clean training data to sample negative pairs for our proxy task. moreover, all the other synthetically generatable errors like a typo error, one to many alignment errors, etc. can be incorporated into the training data. but its not viable to model those kinds of errors independent from the language or domain with the naive assumptions and inventing heuristic rules. we believe further researches should focus on domain invariant noise generation techniques.