Parallel Corpus Filtering via Pre-trained Language Models (2020)

Abstract: Web-crawled data provides a good source of parallel corpora for training machine translation models. It is automatically obtained, but extremely noisy, and recent work shows that neural machine translation systems are more sensitive to noise than traditional statistical machine translation methods. In this paper, we propose a novel approach to filter out noisy sentence pairs from web-crawled corpora via pre-trained language models. We measure sentence parallelism by leveraging the multilingual capability of BERT and use the Generative Pre-training (GPT) language model as a domain filter to balance data domains. We evaluate the proposed method on the WMT 2018 Parallel Corpus Filtering shared task, and on our own web-crawled Japanese-Chinese parallel corpus. Our method significantly outperforms baselines and achieves a new state-of-the-art. In an unsupervised setting, our method achieves comparable performance to the top-1 supervised method. We also evaluate on a web-crawled Japanese-Chinese parallel corpus that we make publicly available.

## introduction training modern neural machine translation (nmt) systems requires large parallel-text resources. publicly-available parallel corpora are mostly paired with english, such as german-english, french-english, chinese-english, etc., and their domains are limited. for building machine translation systems between non-english language pairs, such as chinese and japanese, existing parallel corpora are insufficient and often low quality. to address this problem, system builders have trained nmt systems on web-crawled data and achieved promising results @xcite @xcite . however, data automatically crawled from the web is extremely noisy. @xcite and @xcite show that neural translation models are far more sensitive to noisy parallel training data than statistical machine translation. data selection methods that can filter noisy parallel sentences from large-scale web crawled resources are in demand. in this paper, we study the problem in a realworld scenario where we crawl a large japanese-chinese parallel corpus from various websites and build open-domain machine translation systems between japanese and chinese, by filtering the web crawled parallel corpus. in addition, a small amount of clean parallel data is available, in the software domain. in order to confirm our results on a public data, we also apply our filter to the wmt 2018 german-english parallel corpus filtering shared task. previous work on parallel corpus filtering performs poorly in our scenario as it either requires large clean parallel corpora or dictionaries @xcite @xcite , or relies on multilingual word embeddings and neglects context when measuring translation parallelism @xcite . in this paper, we propose a simple but effective parallel corpus filtering method. multilingual bert @xcite projects multilingual sentences into a shared space and has shown a great potential for cross-lingual model transfer @xcite . we use pre-trained multilingual bert as prior knowledge and fine-tune it on a synthetic dataset. this multilingual bert-based classifier forms an acceptability filter that determines whether or not a sentence pair consists of a bona-fide translation. as the domain of training data largely affects machine translation model performance, we also introduce a domain filter. it uses the pre-trained generative pre-training (gpt) as in-domain language model and is an extension of the existing crossentropy difference based domain filter @xcite . we evaluate our proposed method on the wmt 2018 german-english parallel corpus filtering shared task and achieve a new state-of-the-art. our unsupervised method achieves comparable performance to the top system that is trained on millions of clean parallel sentence pairs. our proposed methods also significantly outperform baselines in our own japanese-chinese parallel corpus filtering task. we make the following contributions: ## related work several recent works address parallel corpus filtering. @xcite , @xcite and @xcite use language models and word alignments to determine how likely sentences are to be a good translation of another. @xcite introduce a noise filtering tool, zipporah, that discriminates parallel and non-parallel sentences based on word-frequency vectors and a dictionary. junczys-dowmunt (2018) proposes a dual conditional cross-entropy filtering method, which achieved first place in the wmt 2018 german-english parallel corpus filtering shared task. they train two translation models in inverse directions on millions of parallel sentences and score sentence pairs based on the word-normalized conditional cross-entropy from the translation models. @xcite and @xcite propose a margin-based scoring method that compares the 1 http://iwslt.org/doku.php?@xmath0. the sentence representations are produced by a sentence encoder trained on clean parallel data via a neural encoder-decoder architecture. other works based on sentence embeddings include @xcite and @xcite , as well as @xcite , which mines millions of parallel sentences in 1620 language pairs from wikipedia. these encoder-decoder based methods require large amounts of clean parallel training data and are not applicable in our scenario where available data is noisy. ondrej bojar (2020) organize an open domain translation challenge where participants are provided a large, noisy set of japanese-chinese segment pairs built from web data, and the task is to clean the noisy data and build an end-to-end machine translation system. work on data selection is also related. @xcite ; junczys-dowmunt (2018) select domain-related data by computing the crossentropy difference between in-domain and outdomain language models. @xcite use neural language models for data selection. @xcite and @xcite expand cross-entropy difference filtering to both sides of the parallel corpus. since we aim to build a general machine translation system, instead of selecting data that are relevant to a specific domain, we select data whose domains are as general as possible, by using generative pre-training (gpt) models trained on large and diverse corpora. ## method in this section we introduce a language detection filter, a translation-acceptability filter, and a domain filter. each filter produces a score for every candidate source/target sentence pair. the partial score produced by each filter ranges from 0 to 1. values beyond this range are normalized by minmax normalization: ŷ = (y -min)/(maxmin). the final score is the product of the partial scores. ## conclusions in this paper, we address the parallel corpus filtering problem in machine translation. we propose a novel filtering method using pre-trained language models. our method outperforms strong baselines and achieves a new state-of-the-art. we release a large japanese-chinese web crawled parallel corpus for the research purposes. because it is artificial to use synthetic data for training a filter classifier, future work can focus on a better objective that models parallelism more smoothly. future work also includes extending the method to low-resource languages not covered by multilingual bert.