Unsupervised Multilingual Sentence Embeddings for Parallel Corpus Mining (2020)

Abstract: Existing models of multilingual sentence embeddings require large parallel data resources which are not available for low-resource languages. We propose a novel unsupervised method to derive multilingual sentence embeddings relying only on monolingual data. We first produce a synthetic parallel corpus using unsupervised machine translation, and use it to fine-tune a pretrained cross-lingual masked language model (XLM) to derive the multilingual sentence representations. The quality of the representations is evaluated on two parallel corpus mining tasks with improvements of up to 22 F1 points over vanilla XLM. In addition, we observe that a single synthetic bilingual corpus is able to improve results for other language pairs.

## introduction parallel corpora constitute an essential training data resource for machine translation as well as other cross-lingual nlp tasks. however, large parallel corpora are only available for a handful of language pairs while the rest relies on semi-supervised or unsupervised methods for training. since monolingual data are generally more abundant, parallel sentence mining from non-parallel corpora provides another opportunity for low-resource language pairs. an effective approach to parallel data mining is based on multilingual sentence embeddings @xcite . however, existing methods to generate crosslingual representations are either heavily supervised or only apply to static word embeddings. an alternative approach to unsupervised multilingual training is that of @xcite or lample and @xcite , who train a masked language model (m-bert, xlm) on a concatenation of monolingual corpora in different languages to learn a joint structure of these languages together. while several authors @xcite @xcite bring evidence of cross-lingual transfer within the model, its internal representations are not entirely language agnostic. we propose a method to further align representations from such models into the cross-lingual space and use them to derive sentence embeddings. our approach is completely unsupervised and is applicable even for very distant language pairs. the proposed method outperforms previous unsupervised approaches on the bucc 2018 foot_0 shared task, and is even competitive with several supervised baselines. the paper is organized as follows. section 2 gives an overview of related work; section 3 introduces the proposed method; section 4 describes the experiments and reports the results. section 5 concludes. ## related work related research comprises supervised methods to model multilingual sentence embeddings and unsupervised methods to model multilingual word embeddings which can be aggregated into sentences. furthermore, our approach is closely related to the recent research in cross-lingual language model (lm) pretraining. supervised multilingual sentence embeddings. the state-of-the-art performance in parallel data mining is achieved by laser @xcite ) -a multilingual bilstm model sharing a single encoder for 93 languages trained on parallel corpora to produce language agnostic sentence representations. similarly, @xcite ; @xcite ; @xcite derive sentence embeddings from internal representations of a neural machine translation system with a shared encoder. the universal sentence encoder (use) @xcite family covers sentence embedding models with a multi-task dual-encoder training framework including the tasks of question-answer prediction or natural language inference. @xcite directly optimize the cosine similarity between the source and target sentences using a bidirectional dual-encoder. these approaches rely on heavy supervision by parallel corpora which is not available for low-resource languages. unsupervised multilingual word embeddings. cross-lingual embeddings of words can be obtained by post-hoc alignment of monolingual word embeddings @xcite and mean-pooled with idf weights to represent sentences @xcite . unsupervised techniques to find a linear mapping between embedding spaces were proposed by @xcite and @xcite , using iterative self-learning or adversarial training. several recent studies @xcite criticize this simplified approach, showing that even the embedding spaces of closely related languages are not isometric. @xcite question the robustness of unsupervised mapping methods in challenging circumstances. cross-lingual lm pretraining. @xcite ; @xcite derive monolingual sentence embeddings by mean-pooling contextualized word embeddings from bert. @xcite ; @xcite propose mapping such contextualized embeddings into the multilingual space and report favorable results on the task of dependency parsing. @xcite extract contextualized embeddings directly from unsupervised multilingual lms and use them for parallel sentence retrieval. other authors improve the alignment of representations in a multilingual lm using a parallel corpus as an anchor @xcite or using iterative self-learning @xcite . none of these works apply multilingual embeddings to mine parallel sentences. our work is the first in improving unsupervised cross-lingual models using additional unsupervised information. ## proposed method we propose a method to enhance the cross-lingual ability of a pretrained multilingual model by fine-tuning it on a small synthetic parallel corpus. the parallel corpus is obtained via unsupervised machine translation (mt) so the method remains unsupervised. in this section, we describe the pretrained model (section 3.1), the fine-tuning objective (section 3.2) and the extraction of sentence embeddings (section 3.3). we provide details on the unsupervised mt system in section 3.4. ## experiments & results in this section, we empirically evaluate the quality of our cross-lingual sentence embeddings and compare it with state-of-the-art supervised methods and unsupervised baselines. we evaluate the proposed method on the task of parallel corpus mining and parallel sentence matching. we fine-tune two different models using english-german and czech-german synthetic parallel data. ## conclusion we proposed a completely unsupervised method to train multilingual sentence embeddings which can be used for building a parallel corpus with no previous translation knowledge. we show that fine-tuning an unsupervised multilingual model with a translation objective using as little as 20k synthetic translation pairs can significantly enhance the cross-lingual alignment of its representations. since the synthetic translations were obtained from an unsupervised mt system, the entire procedure requires no authentic parallel sentences for training. our sentence embeddings yield significantly better results on the tasks of parallel data mining and parallel sentence matching than our unsupervised baselines. interestingly, targeting only one language pair during the fine-tuning phase suffices to propagate the alignment improvement to unrelated languages. it is therefore not necessary to build a working mt system for every language pair we wish to mine. the average f1 margin across four language pairs on the bucc task is ∼17 points over the original xlm model and ∼7 on the news dataset where only one of the evaluated language pairs was seen during fine-tuning. the gain in accuracy in parallel sentence matching across 8 language pairs is 7.2% absolute, lagging only 7.1% absolute behind supervised methods. for the future we would like to apply our model on other cross-lingual nlp tasks such as xnli or cross-lingual semantic textual similarity.