Learning Contextualised Cross-lingual Word Embeddings and Alignments for Extremely Low-Resource Languages Using Parallel Corpora (2021)

Abstract: We propose a new approach for learning contextualised cross-lingual word embeddings based on a small parallel corpus (e.g. a few hundred sentence pairs). Our method obtains word embeddings via an LSTM encoder-decoder model that simultaneously translates and reconstructs an input sentence. Through sharing model parameters among different languages, our model jointly trains the word embeddings in a common cross-lingual space. We also propose to combine word and subword embeddings to make use of orthographic similarities across different languages. We base our experiments on real-world data from endangered languages, namely Yongning Na, Shipibo-Konibo, and Griko. Our experiments on bilingual lexicon induction and word alignment tasks show that our model outperforms existing methods by a large margin for most language pairs. These results demonstrate that, contrary to common belief, an encoder-decoder translation model is beneficial for learning cross-lingual representations even in extremely low-resource conditions. Furthermore, our model also works well on high-resource conditions, achieving state-of-the-art performance on a German-English word-alignment task.

## introduction cross-lingual word embedding learning has the goal of learning representations for words of different languages in a common space @xcite @xcite . cross-lingual representations are beneficial for finding correspondences between languages, and are utilised in many downstream tasks such as machine translation @xcite and cross-lingual named entity recognition @xcite . the recent trend in cross-lingual embedding models is to leverage an enormous amount of monolingual data for each of the target languages, e.g. by training word embeddings monolingually and mapping them into a common space. another approach is to jointly train cross-lingual word embeddings in the same space. recently, crosslingual masked language models such as mbert @xcite have succeeded in learning cross-lingual representations using large-scale monolingual data for multiple languages. however, when dealing with endangered languages, there is generally no such large-scale corpus of monolingual data, and when trained under low-resource settings, modern pretraining methods do not perform well @xcite . in this paper, we propose a joint-training method that learns contextualised cross-lingual word embeddings using a small parallel corpus, of the scale and form constructed by field linguists for language documentation purposes. compared to previous models based on parallel corpora, our model has two strengths: (1) while previous models extend bag-of-words models such as skip-gram @xcite and capture only rudimentary word order information, our model encodes sentences with lstms @xcite and generates contextualised word embeddings; and (2) our model trains subword-aware word embeddings and captures orthographic similarities among the languages. we perform evaluation over bilingual lexicon induction and word alignment. both tasks are extremely important to facilitate language documentation, revitalisation, and education of endangered languages. we run experiments targeting three endangered languages -yongning na, shipibo-konibo, and griko (section 3.1) -as well as four high-resource language pairs, and show that our model substantially outperforms strong baselines for most language pairs. 2 methodology ## related work there are two main approaches to learning crosslingual word embeddings. one is to learn a matrix that aligns pretrained monolingual embeddings. most such methods exploit bilingual dictionaries to learn the mapping matrix @xcite @xcite , but recently a number of methods have succeeded in learning the matrix without supervision @xcite @xcite . however, @xcite show that this approach does not work well on low-resource conditions. the second approach is to jointly train crosslingual embeddings in a common space. most existing methods extend bag-of-words models (e.g. skip-gram) to incorporate cross-lingual information provided by parallel or comparable corpora @xcite @xcite @xcite @xcite , or bilingual dictionaries @xcite . recently, masked language models such as xlm @xcite and mbert @xcite have been shown to generate cross-lingual representations without parallel data, but require an enormous amount of monolingual data, which is not available for endangered languages. similar to our work, some papers use multilingual neural machine translation models to obtain cross-lingual representations @xcite @xcite . however, they employ extremely large and/or multilingual data aligned among more than two languages (e.g. europarl, united nations). another important difference is that their methods focus on learning cross-lingual sentence representations only (i.e. not at the word level), and are evaluated on cross-lingual sentence retrieval or sentence classification tasks. ## conclusion we propose a new approach for learning contextualised cross-lingual word embeddings that can be trained with a tiny parallel corpus. we evaluate models on real-world data for three endangered languages, and also on benchmark data sets for four high-resource languages, and show that our model outperforms existing methods at bilingual lexicon induction and word alignment.