Unsupervised Bitext Mining and Translation via Self-Trained Contextual Embeddings (2020)

Abstract: We describe an unsupervised method to create pseudo-parallel corpora for machine translation (MT) from unaligned text. We use multilingual BERT to create source and target sentence embeddings for nearest-neighbor search and adapt the model via self-training. We validate our technique by extracting parallel sentence pairs on the BUCC 2017 bitext mining task and observe up to a 24.5 point increase (absolute) in F1 scores over previous unsupervised methods. We then improve an XLM-based unsupervised neural MT system pre-trained on Wikipedia by supplementing it with pseudo-parallel text mined from the same corpus, boosting unsupervised translation performance by up to 3.5 BLEU on the WMT’14 French-English and WMT’16 German-English tasks and outperforming the previous state-of-the-art. Finally, we enrich the IWSLT’15 English-Vietnamese corpus with pseudo-parallel Wikipedia sentence pairs, yielding a 1.2 BLEU improvement on the low-resource MT task. We demonstrate that unsupervised bitext mining is an effective way of augmenting MT datasets and complements existing techniques like initializing with pre-trained contextual embeddings.

## introduction large corpora of parallel sentences are prerequisites for training models across a diverse set of applications, such as neural machine translation (nmt; @xcite , paraphrase generation @xcite , and aligned multilingual sentence embeddings @xcite . systems that extract parallel corpora typically rely on various cross-lingual resources (e.g., bilingual lexicons, parallel cor-pora), but recent work has shown that unsupervised parallel sentence mining @xcite and unsupervised nmt @xcite produce surprisingly good results. 1 existing approaches to unsupervised parallel sentence (or bitext) mining start from bilingual word embeddings (bwes) learned via an unsupervised, adversarial approach @xcite . @xcite created sentence representations by mean-pooling bwes over content words. to disambiguate semantically similar but non-parallel sentences, @xcite additionally proposed parallel segment detection by searching for paired substrings with high similarity scores per word. however, using word embeddings to generate sentence embeddings ignores sentential context, which may degrade bitext retrieval performance. we describe a new unsupervised bitext mining approach based on contextual embeddings. we create sentence embeddings by mean-pooling the outputs of multilingual bert (mbert; @xcite , which is pre-trained on unaligned wikipedia sentences across 104 languages. for a pair of source and target languages, we find candidate translations by using nearest-neighbor search with margin-based similarity scores between pairs of mbert-embedded source and target sentences. we bootstrap a dataset of positive and negative sentence pairs from these initial neighborhoods of candidates, then self-train mbert on its own outputs. a final retrieval step gives a corpus of pseudo-parallel sentence pairs, which we expect to be a mix of actual translations and semantically related non-translations. we apply our technique on the bucc 2017 parallel sentence mining task @xcite . we achieve state-of-the-art f 1 scores on unsupervised bitext mining, with an improvement of up to 24.5 points (absolute) on published results @xcite . other work (e.g., @xcite has shown that retrieval performance varies substantially with the layer of mbert used to generate sentence representations; using the optimal mbert layer yields an improvement as large as 44.9 points. furthermore, our pseudo-parallel text improves unsupervised nmt (unmt) performance. we build upon the unmt framework of @xcite and xlm @xcite by incorporating our pseudo-parallel text (also derived from wikipedia) at training time. this boosts performance on wmt'14 en-fr and wmt'16 en-de by up to 3.5 bleu over the xlm baseline, outperforming the state-of-the-art on unsupervised nmt @xcite . finally, we demonstrate the practical value of unsupervised bitext mining in the low-resource setting. we augment the english-vietnamese corpus (133k pairs) from the iwslt'15 translation task @xcite with our pseudobitext from wikipedia (400k pairs), and observe a 1.2 bleu increase over the best published model (nguyen and salazar, 2019). when we reduced the amount of parallel and monolingual vietnamese data by a factor of ten (13.3k pairs), the model trained with pseudo-bitext performed 7 bleu points better than a model trained on the reduced parallel text alone. ## our approach our aim is to create a bilingual sentence embedding space where, for each source sentence embedding, a sufficiently close nearest neighbor among the target sentence embeddings is its translation. by aligning source and target sentence embeddings in this way, we can extract sentence pairs to create new parallel corpora. @xcite construct this space by training a joint encoder-decoder mt model over multiple language pairs and using the resulting encoder to generate sentence embeddings. a marginbased similarity score is then computed between embeddings for retrieval (section 2.2). however, this approach requires large parallel corpora to train the encoder-decoder model in the first place. we investigate whether contextualized sentence embeddings created with unaligned text are useful for unsupervised bitext retrieval. previous work explored the use of multilingual sentence encoders taken from machine translation models (e.g., @xcite for zero-shot cross-lingual transfer. our work is motivated by recent success in tasks like zero-shot text classification and named entity recognition (e.g., @xcite with multilingual contextual embeddings, which exhibit cross-lingual properties despite being trained without parallel sentences. we illustrate our method in figure 1 . we first retrieve the candidate translation pairs: the remaining candidate pairs are used to bootstrap a dataset for self-training mbert as follows: after self-training, the finetuned mbert model is used to generate new sentence embeddings. parallel sentences should be closer to each other in this new embedding space, which improves retrieval performance. figure 1: our self-training scheme. left: we index sentences using our two encoders. for each source sentence, we retrieve k nearest-neighbor target sentences per the margin criterion (eq. 1), depicted here for @xmath0. if the nearest neighbor is within a threshold, it is treated with the source sentence as a positive pair, and the remaining k -1 are treated with the source sentence as negative pairs. right: we refine one of the encoders such that the cosine similarity of the two embeddings is maximized on positive pairs and minimized on negative pairs. ## unsupervised bitext mining we apply our method to the bucc 2017 shared task, ''spotting parallel sentences in comparable corpora'' @xcite . the task involves retrieving parallel sentences from monolingual corpora derived from wikipedia. parallel sentences were inserted into the corpora in a contextually appropriate manner by the task organizers. the shared task assessed retrieval systems for precision, recall, and f 1 -score on four language pairs: de-en, fr-en, ru-en, and zh-en. prior work on unsupervised bitext mining has generally studied the european language pairs to avoid dealing with chinese word segmentation @xcite . ## bitext for neural machine translation a major application of bitext mining is to create new corpora for machine translation. we conduct an extrinsic evaluation of our unsupervised bitext mining approach on unsupervised (wmt'14 french-english, wmt'16 german-english) and low-resource (iwslt'15 english-vietnamese) translation tasks. we perform large-scale unsupervised bitext extraction on the october 2019 wikipedia dumps in various languages. we use wikifil.pl 5 to extract paragraphs from wikipedia and remove markup. we then use the syntok foot_6 package for sentence segmentation. finally, we reduce the size of the corpus by removing sentences that aren't part of the body of wikipedia pages. sentences that contain *, =, //, ::, #, www, (talk), or the pattern [0-9]{2}:[0-9]{2} are filtered out. we index, retrieve, and filter candidate sentence pairs with the procedure in sec. 3. unlike bucc, the wikipedia dataset does not fit in gpu memory. the processed corpus is quite large, with 133 million, 67 million, 36 million, and 6 million sentences in english, german, french, and vietnamese respectively. we therefore shard the dataset into chunks of 32,768 sentences and perform nearest-neighbor comparisons in chunks for each language pair. we use a simple mapreduce algorithm to merge the intermediate results back together. we follow the approach outlined in sec. 2 for wikipedia bitext mining. for each source sentence, we retrieve the four nearest target neighbors across the millions of sentences that we extracted from wikipedia and compute the margin-based scores for each pair. ## conclusion in this work, we describe a novel approach for state-of-the-art unsupervised bitext mining using multilingual contextual representations. we extract pseudo-parallel sentences from unaligned corpora to create models that achieve state-of-theart performance on unsupervised and low-resource translation tasks. our approach is complementary to the improvements derived from initializing mt models with pre-trained encoders and decoders, and helps narrow the gap between unsupervised and supervised mt. we focused on mbertbased embeddings in our experiments, but we expect unsupervised self-training to improve the unsupervised bitext mining and downstream unmt performance of other forms of multilingual contextual embeddings as well. our findings are in line with recent work showing that multilingual embeddings are very useful for cross-lingual zero-shot and zero-resource tasks. even without using aligned corpora, mbert can embed sentences across different languages in a consistent fashion according to their semantic content. more work will be needed to understand how contextual embeddings discover these crosslingual correspondences.