Parallel sentences mining with transfer learning in an unsupervised setting (2021)

Abstract: The quality and quantity of parallel sentences are known as very important training data for constructing neural machine translation (NMT) systems. However, these resources are not available for many low-resource language pairs. Many existing methods need strong supervision are not suitable. Although several attempts at developing unsupervised models, they ignore the language-invariant between languages. In this paper, we propose an approach based on transfer learning to mine parallel sentences in the unsupervised setting. With the help of bilingual corpora of rich-resource language pairs, we can mine parallel sentences without bilingual supervision of low-resource language pairs. Experiments show that our approach improves the performance of mined parallel sentences compared with previous methods. In particular, we achieve excellent results at two real-world low-resource language pairs.

## introduction parallel sentences are known as very important training data for constructing machine translation (mt) systems @xcite . the volumes of quality parallel sentences heavily affect the performance of trained machine translation systems. however, these resources are only available for a handful of language pairs and domains while the others suffer from the scarcity problem @xcite . in this situation, parallel sentences are very crucial for training machine translation systems. transfer learning is an effective approach to mine parallel data in low-resource scenarios. @xcite brought the evidence of cross-lingual transfer to mine parallel data for low-resource language pairs. however, their method is not unsupervised and relies on bilingual supervision (e.g, bilingual lexicon or sentences), which is not available for low-resource language pairs. although @xcite solved the supervised limitation by employing an unsupervised mt, the performance heavily depended on mt's quality. in this paper, we propose a parallel sentences mining model based on transfer learning in an unsupervised setting foot_0 . as illustrated in figure 1 , we obtain sentence embeddings by mean-pooling the outputs of multilingual bert @xcite , which is trained on monolingual corpora. in particular, we use a language discriminator to learn shared and refined language-invariant representations for transfer learning. @xcite pointed out the language-invariant is helpful for transfer learning. then, we treat detecting parallel sentences as a classification task and generate multi-view semantic representations for the classifier. generally, data from different views contain complementary information and multi-view learning exploits the consistency from multiple views @xcite . in our model, we use two views for the classifier: (i) word representations; (ii) sentence representations. in addition to achieving good results on bucc 2018 foot_1 shared task, we demonstrate the effectiveness of our model using an example of two low-resource language pairs where parallel corpora are almost not available. in summary, our contributions in this paper are as follows: ## related work many works mine parallel corpora from monolingual data which contain potential mutual translations. previous methods depended on engineering features. @xcite used metadata information from web crawls to mine parallel data. recent methods used crosslingual word embeddings to obtain parallel corpora @xcite @xcite . @xcite encoded the universal language embeddings that are agnostic to languages. they used transfer learning to mine parallel sentences of low-resource language pairs. this transfer learning method inspired our work and the main difference is that they required bilingual supervision (e.g, bilingual lexicon, parallel sentences), which is not available for many low-resource language pairs. recently, several works developed unsupervised method to mine parallel data @xcite @xcite . these approaches mainly rely on unsupervised cross-lingual embeddings (artetxe et al., 2018; lample and conneau, 2019) that be trained on monolingual corpora. however, several researchers question that these methods may not well capture the semantic similarity among languages @xcite . some researchers proposed to use transfer learning to solve cross-lingual applications for low-resource language pairs @xcite . @xcite ) used a multilingual neural machine translation system to learn the word representations of rich-resource language pairs. then, they used transfer learning to identify parallel sentences for low-resource language pairs. however, it has an implicit dependency on multilingual nmt that requires pre-training on large parallel sentences. our transfer learning is inspired by @xcite . the difference is that they mainly solve cross-lingual unsupervised sentiment classification. ## proposed method the overview of the model architecture is as shown in figure 1 . our proposed approach based on transfer learning to mine parallel data is composed of three components: an unsupervised multilingual bert, a language discriminator, and a multiview classifier. motivated by the success of unsupervised cross-lingual word embeddings @xcite and its application in mining parallel data @xcite , we use multilingual bert to initialize word and sentence embeddings. although previous methods are effective, they may ignore sentential context on using multilingual word embeddings, which could harm the performance of mining parallel corpora. in our work, we use multi-view representations to mine parallel data. we can get good performance on rich-resource language pairs. however, our aim is to obtain parallel data for low-resource language pairs. for this purpose, we use transfer learning to mine parallel data of the low-resource scenarios using rich-resource language pairs. note that our method doesn't rely on any bilingual data of lowresource language pairs. therefore, we can call that our method is unsupervised for low-resource language pairs. ## experimental setting in this section, we mainly present our experimental settings and describe the datasets used. dataset: we test our proposed method on four language pairs of bucc sample data (english-french, english-german, english-russian, english-chinese). the workshop on building and using comparable corpora (bucc) is a well-established evaluation framework for mining parallel corpora @xcite . the shared task provides a gold standard to assess retrieval systems for precision, recall, and f 1 -score. we applied our approach to all language pairs of the bucc18 shared task. moreover, we carry out an experiment on realworld low-resource scenarios (english-esperanto, chinese-kazakh). for the monolingual data, we extract corpora from wikipedia using wikiextractor 3 . as there is no gold standard to evaluate mining parallel sentences, we use mined parallel sentences to train a machine translation system that can reflect the quality of mined parallel sentences. baselines: in our experiments, we consider supervised baselines @xcite @xcite . we also compare several unsupervised baselines which contains @xcite @xcite . ## results and discussions in this section, we present the results of mining parallel sentences and our comparison to previous work. we also present results on real-world low-resource language pairs and demonstrate our obtained parallel corpora can improve the performance of machine translation. ## conclusion in this paper, we propose an unsupervised method that uses multi-view transfer learning to mine parallel sentences. our method can effectively use the bilingual data of rich-resource language pairs. we transfer the model of rich-resource language pairs into a low-resource situation without any supervision of low-resource language pairs. in particular, we employ a language discriminator to capture language-invariant for benefiting transfer learning. in the experiments, the results show that our method significantly and consistently outperforms the baselines. for the future, we would like to apply our model on other low-resource language pairs to test universal applicability in different language pairs.