Joint Training for Learning Cross-lingual Embeddings with Sub-word Information without Parallel Corpora (2020)

Abstract: In this paper, we propose a novel method for learning cross-lingual word embeddings, that incorporates sub-word information during training, and is able to learn high-quality embeddings from modest amounts of monolingual data and a bilingual lexicon. This method could be particularly well-suited to learning cross-lingual embeddings for lower-resource, morphologically-rich languages, enabling knowledge to be transferred from rich- to lower-resource languages. We evaluate our proposed approach simulating lower-resource languages for bilingual lexicon induction, monolingual word similarity, and document classification. Our results indicate that incorporating sub-word information indeed leads to improvements, and in the case of document classification, performance better than, or on par with, strong benchmark approaches.

## introduction state-of-the-art approaches in natural language processing (nlp) typically require a substantial amount of human-annotated data (i.e, for supervised approaches to tasks such as part-of-speech tagging or dependency parsing) or they need a very large amount of unannotated text for training (e.g., methods for learning word embeddings). this poses a particular problem for building nlp systems for low-resource languages. there are thousands of human languages, and creating annotated datasets for all of them would be very expensive. furthermore, many languages have a relatively small number of speakers, and in many cases large amounts of text are not readily-available for building corpora for these languages. a further related challenge is posed by morphologically-rich this work is licensed under a creative commons attribution 4.0 international license. license details: http: //creativecommons.org/licenses/by/4.0/ . word embeddings are a key feature in approaches for a wide range of nlp tasks, such as part-of-speech tagging @xcite , dependency parsing @xcite , and named entity recognition @xcite . if we are able to transfer the knowledge captured in word embeddings for a rich-resource language to another low-resource language, then developing nlp tools could become more feasible for the low-resource language. there has therefore been a wealth of research on cross-lingual word embeddings (e.g., @xcite @xcite , in which embeddings for multiple languages are learned in a shared space, and which can be used to transfer knowledge between languages, such as from a richresource language to a low-resource one @xcite . despite the wide range of research on learning cross-lingual embeddings, there are some limitations of these methods that have not been addressed. in the case of a low-resource language, due to the relatively small size of available corpora, a relatively small number of embeddings would be learned. moreover, in the case of a morphologically-rich language, many wordforms would not be observed in the corpus on which embeddings are trained. as a result, given a subsequent text to process, many words would be expected to be out-of-vocabulary (oov) with respect to the embedding model. this is a very important issue, because in the case of oovs, we do not have an embedding for these words, and models for downstream nlp tasks that use embeddings would therefore lack information for these words. where the number of oovs is relatively high, such as for low-resource and morphologically-rich languages, this could lead to particularly poor performance in down-stream tasks. this problem has been addressed in monolingual settings by learning embeddings for sub-word units, and then composing representations for oovs based on their sub-words @xcite . recently, with advances in language modelling @xcite and contextualized language models @xcite , transfer learning has become feasible between languages by using a byte pair encoding @xcite shared vocabulary, and fine-tuning the models for specific tasks @xcite . nevertheless, these models require a substantial amount of training data @xcite , and in some cases parallel corpora @xcite , and are very computationally expensive to train. there is therefore a need for methods that can be trained from a morelimited amount of data and require less computational resources for training, but that nevertheless show comparable performance. in this paper, we propose a model that can learn cross-lingual word embeddings from a modest amount of monolingual data and a bilingual dictionary. we rely on bilingual dictionaries because they are relatively-widely available. for example, panlex @xcite is a translation resource that combines many bilingual dictionaries and provides translations for 5700 languages. our proposed model is an extension of the method proposed by @xcite . in their work, they only considered word embeddings, and so their method is unable to form representations for oovs, and therefore is not expected to perform well for low-resource or morphologically-rich target languages. we extend the method of @xcite by incorporating sub-word information in the process of training cross-lingual word embeddings. in this way, we form a shared embedding space that not only contains embeddings for both source and target language words, but that has also been enriched with sub-word embeddings enabling representations to be formed for oovs. to evaluate our proposed model, we use modest amounts of data for relatively well-resourced languages. we first consider two intrinsic evaluations: ## related work a variety of methods have been proposed for learning cross-lingual word embeddings. these methods vary with respect to the level of supervision, and the cross-lingual signals used, such as parallel corpora and bilingual dictionaries. @xcite propose a method to learn cross-lingual representations by training a language model on the source and target language and optimizing their objective function jointly. this method, however, requires a parallel corpus, which is not available for many languages, especially low-resource ones. more recently, @xcite propose a bi-directional lstm language model that is trained on a very large parallel corpus, containing 223 million parallel sentences, and jointly learns representations for 93 languages. aside from requiring a parallel corpus, it is also computationally expensive to train. @xcite argue that the geometric arrangement of word embeddings in two different languages is the same. they therefore propose a method to learn a linear transformation to align the vector spaces of two languages by using a seed lexicon of known translation pairs. @xcite show that normalizing all word vectors to be unit length, and applying an orthogonality constraint on the transformation matrix, improves the approach of @xcite introduce an alignment-based method which relaxes the requirement of having a bilingual seed lexicon. their approach begins with a very small seed lexicon -as few as 25 pairs -and in a process of self-learning and through several rounds of bootstrapping, increases the size of the bilingual dictionary. @xcite further relax the need for a bilingual dictionary, and propose a fully unsupervised approach. their method solves the same mapping problem as @xcite , but creates the initial seed lexicon in an unsupervised manner by exploiting the similarity distribution of words in the source and target language. all of these mapping-based methods require pretrained monolingual word embeddings, the quality of which the final cross-lingual word embeddings are greatly dependent upon. this is problematic in the case that we do not have access to enough training data to learn high quality monolingual embeddings, as would be the case for many low-resource languages. moreover, it has been shown that fully unsupervised methods do not perform well across all languages, especially in the case of morphologically rich languages, and when the monolingual embeddings do not come from the same domain @xcite . furthermore, @xcite show that the isomorphism assumption -i.e., that embeddings for different languages have a similar geometric arrangement, which is key to the success of mappingbased models -does not always hold. they show that methods which jointly learn the embedding space for the source and target language from a parallel corpus are superior to mapping-based methods. however, parallel corpora are a very expensive cross-lingual signal. in an alternative approach to learning crosslingual word embeddings, a pesudo-bilingual corpus is first constructed using a bilingual dictionary, and embeddings for the source and target language are then learned from this corpus. @xcite concatenate and shuffle the source and target language corpora, and then randomly replace words in this corpus using a bilingual dictionary. they then run cbow on the constructed corpus to learn word embeddings for both the source and target language. similarly, @xcite also propose a method that replaces words in a pseudo-bilingual corpus with their translation during training. however, they further propose a way to handle polysemy by choosing the best translation for a word by considering its context using the expectation-maximization algorithm. compared to mapping-based methods, this approach does not require as large of a corpus for training, because for each word, the context in not only the source language, but also the target language, is used. however, these pseudo-bilingual corpus methods are more expensive to train than mapping-based methods, because the embeddings are learned from scratch, in contrast to mappingbased methods which use pre-trained embeddings and only need to learn the mapping function. recent approaches to learning cross-lingual embeddings have been trained on concatenated monolingual corpora. multilingual bert (mbert) is a bert model @xcite trained on concatenated wikipedia corpora for 105 languages. @xcite show that since mbert uses a shared vocabulary for all languages, it can represent embeddings for all languages in a shared space, rather than representing each language in a separate space. this model is therefore able to learn deep contextualized cross-lingual word embeddings without any cross-lingual signal, but is computationally expensive to train. @xcite present a method that uses sub-word information, such as lemmas, morpheme tags, and phoneme n-grams, to transfer knowledge from richresource languages to low-resource ones. they train skip-gram on concatenated monolingual corpora of two related languages and learn representations in a shared space by relying on similar subwords to map related words close to each other in the shared space. they also consider an approach which first trains a model on the rich-resource language and then uses the trained sub-word embeddings to initialize the model for the low-resource language. the approach for learning cross-lingual embeddings proposed in this paper incorporates sub-word information -similar to @xcite and mbert -but in contrast to chaudhary et al. does not require language-specific tools such as morphological analyzers which might not be available for low-resource languages, and in contrast to mbert is less computationally-expensive to train. the proposed approach is an extension of @xcite that incorporates sub-word information, and requires only modest size monolingual corpora and a bilingual lexicon for training. ## methodology in this section we first describe the approach of @xcite to learning cross-lingual word embeddings, and then present our proposed model, which is an extension of this approach. ## resources for evaluation, we simulate lower-resource languages using 9 well-resourced languages: chinese, dutch, english, french, german, italian, japanese, russian, and spanish. these languages include those considered by @xcite , as well as those in the mldoc dataset (schwenk and li, 2018, which we use for evaluation in section 5.3). following previous work (e.g., @xcite , we only consider pairs of languages with english as either the source or target language, and one of the remaining 8 languages as the other language. to train word embeddings for each language, we use pre-processed wikipedia dumps @xcite , which are already tokenized and cleaned. to simulate the case of lower-resource languages, following @xcite , we randomly select 5 million sentences for each language from their wikipedia dump. we use a bilingual dictionary as the cross-lingual signal in our proposed approach. our study builds on the work of @xcite , and so for languages that they consider -dutch, german, italian, japanese, and spanish -we use the same dictionaries that they did, which were extracted from panlex. foot_1 for chinese, french, and russian we extract dictionaries from panlex using a similar approach to duong et al. ## experimental results we present experimental results for two intrinsic evaluations, bilingual lexicon induction and monolingual word similarity, and an extrinsic evaluation on cross-lingual document classification. ## conclusions in this paper we proposed an approach to learning cross-lingual word embeddings that incorporates sub-word information during training, and relies on only monolingual corpora and a bilingual dictionary. this approach could be particularly wellsuited to lower-resource, morphologically-rich languages, for which large parallel corpora are not available. we evaluated our proposed approach, on a variety of simulated lower-resource languages, for the tasks of bli, monolingual word similarity, and document classification. our results on bli and monolingual word similarity indicated that incorporating sub-word information during training enhances the quality of the resulting cross-lingual, as well as monolingual, representations. for zero-shot crosslingual document classification, incorporating subword information again led to improvements, and our proposed model outperformed benchmark approaches that have substantially higher resource requirements for training. code and data to reproduce these results has been made available. 8 in future work, we plan to evaluate our proposed approach on truly lower-resource languages to determine the impact of smaller training corpora and bilingual dictionaries on the performance of crosslingual word embeddings. it would also be interesting to consider the morphological richness of languages in this analysis. we further intend to investigate using alternative approaches to forming sub-word representations, such as byte-pair encoding, as well as incorporating positional embeddings into our model (e.g., @xcite , to determine their impact on the quality of the resulting cross-lingual embeddings. finally we plan to evaluate our proposed approach on further extrinsic tasks, such as pos tagging and named entity recognition, focusing on lower-resource languages.