Evaluating a Joint Training Approach for Learning Cross-lingual Embeddings with Sub-word Information without Parallel Corpora on Lower-resource Languages (2021)

Abstract: Cross-lingual word embeddings provide a way for information to be transferred between languages. In this paper we evaluate an extension of a joint training approach to learning cross-lingual embeddings that incorporates sub-word information during training. This method could be particularly well-suited to lower-resource and morphologically-rich languages because it can be trained on modest size monolingual corpora, and is able to represent out-of-vocabulary words (OOVs). We consider bilingual lexicon induction, including an evaluation focused on OOVs. We find that this method achieves improvements over previous approaches, particularly for OOVs.

## introduction word embeddings are an essential component in systems for many natural language processing tasks such as part-of-speech tagging @xcite , dependency parsing @xcite and named entity recognition @xcite . cross-lingual word representations provide a shared space for word embeddings of two languages, and make it possible to transfer information between languages @xcite . a common approach to learn cross-lingual embeddings is to learn a matrix to map the embeddings of one language to another using supervised (e.g., @xcite , semi-supervised @xcite , or unsupervised (e.g., @xcite methods. these methods rely on the assumption that the geometric arrangement of embeddings in different languages is the same. however, it has been shown that this assumption does not always hold, and that methods which instead jointly train embeddings for two languages produce embeddings that are more isomorphic and achieve stronger results for bilingual lexicon induction @xcite , a well-known in-trinsic evaluation for cross-lingual word representations @xcite . the approach of ormazabal et al. uses a parallel corpus as a cross-lingual signal. parallel corpora are, however, unavailable for many language pairs, particularly low-resource languages. @xcite introduce a joint training approach that extends cbow @xcite to learn cross-lingual word embeddings from modest size monolingual corpora, using a bilingual dictionary as the cross-lingual signal. bilingual dictionaries are available for many language pairs, e.g., @xcite provides translations for roughly 5700 languages. these training resource requirements suggest this method could be well-suited to lower-resource languages. however, this word-level approach is unable to form representations for out-of-vocabulary (oov) words, which could be particularly common in the case of lowresource, and morphologically-rich, languages. hakimi @xcite propose an extension of @xcite that incorporates subword information during training and therefore can generate representations for oovs in the shared cross-lingual space. this method also does not require parallel corpora for training, and could therefore be particularly well-suited to lower-resource, and morphologically-rich, languages. however, hakimi parizi and cook only evaluate on synthetic low-resource languages. we refer to the methods of @xcite cook as duong2016 and hakimi2020, respectively. most prior work on bli focuses on invocabulary (iv) words and well-resourced languages (e.g., @xcite @xcite , although there has been some work on oovs @xcite and low-resource languages (anastasopoulos and neubig, 2020). in this paper, we evaluate hakimi2020 on bli for twelve lower-resource languages, and also consider an evaluation focused on oovs. our results indicate that hakimi2020 gives improvements over duong2016 and several strong baselines, particularly for oovs. ## joint training incorporating sub-word information equation 1 shows the cost function for duong2016, which jointly learns embeddings for a word w i and its translation wi , where h i is a vector encoding the context, α is a weight parameter, and d s and d t are the source and target language vocabularies, respectively. following @xcite , hakimi2020 modifies equation 1 by including sub-word information during the joint training process as follows: ## experimental setup we consider bli from twelve lower-resource source languages to english. the languages (shown in ## experimental results in this section, we present results for bli for iv words, and then oov source language words. ## conclusions we evaluated an extension of a joint training approach to learning cross-lingual embeddings that incorporates sub-word information during training, which could be well-suited to lower-resource and morphologically-rich languages because it can be trained on modest amounts of monolingual data and can represent oovs. in two bli tasks for twelve lower-resource languages focused on iv words and oovs, we found that this method improved over previous approaches, particularly for oovs. evaluation data and code for learning the cross-lingual embeddings is available. 6 in future work we plan to explore the impact of the target language on the quality of the crosslingual embeddings, and in particular consider source and target languages from the same family. we further intend to evaluate these cross-lingual embeddings in down-stream tasks for low-resource languages, such as language modelling @xcite and part-of-speech tagging @xcite , and to compare against approaches based on contextualized language models.