Do not neglect related languages: The case of low-resource O ccitan cross-lingual word embeddings (2021)

Abstract: Cross-lingual word embeddings (CLWEs) have proven indispensable for various natural language processing tasks, e.g., bilingual lexicon induction (BLI). However, the lack of data often impairs the quality of representations. Various approaches requiring only weak cross-lingual supervision were proposed, but current methods still fail to learn good CLWEs for languages with only a small monolingual corpus. We therefore claim that it is necessary to explore further datasets to improve CLWEs in low-resource setups. In this paper we propose to incorporate data of related high-resource languages. In contrast to previous approaches which leverage independently pre-trained embeddings of languages, we (i) train CLWEs for the low-resource and a related language jointly and (ii) map them to the target language to build the final multilingual space. In our experiments we focus on Occitan, a low-resource Romance language which is often neglected due to lack of resources. We leverage data from French, Spanish and Catalan for training and evaluate on the Occitan-English BLI task. By incorporating supporting languages our method outperforms previous approaches by a large margin. Furthermore, our analysis shows that the degree of relatedness between an incorporated language and the low-resource language is critically important.

## introduction cross-lingual word embeddings (clwes) are important for a wide range of nlp tasks including bilingual lexicon induction (bli) @xcite , machine translation @xcite , and cross-lingual transfer learning @xcite . two main types of approaches to learn clwes are mapping methods, where a set of pretrained monolingual embeddings is projected into another monolingual space @xcite , and joint methods, where the monolingual and cross-lingual objectives are optimized jointly (e.g., @xcite . since recent research is more and more interested in dealing with low-resource languages, learning multilingual representations for low-resource languages is important as well @xcite @xcite . however, a lack of parallel data impairs the performance of existing strongly supervised models, which is why a lot of recent research focuses on reducing the need for parallel data @xcite @xcite . mapping methods are sensitive to the approximate isomorphism of embedding spaces, which is not the case for many languages @xcite . the low isomorphism of distant language pairs was tackled by learning clwes jointly @xcite @xcite . however, they rely on large monolingual corpora which are not available for many languages. furthermore, the lack of large data leads to low isomorphism as well, since it results in low-quality monolingual embedding spaces @xcite . hence, mapping methods, which rely on the assumption of approximate isomorphism cannot be fruitfully applied in many cases. however, as there are still only poor clwes for many low-resource language pairs @xcite , we argue that in addition to reducing requirements for training data, methods which offer opportunities precisely for low-resource setups, like leveraging data from linguistically related high-resource languages, should be considered as well. while there exist nlp systems that make use of related languages, e.g., in machine translation @xcite , only few work focuses on including them directly into clwes. an approach considering a related language in order to improve clwes for low-resource language pairs, including english-occitan, has been proposed by @xcite . however, using pre-trained monolingual embedding spaces, they do not take into account that monolingual representations of lowresource languages might be of poor quality, which can impede mapping performance. in this paper, we propose a method where, in contrast to previous work, we consider both addressing the issue of monolingual embedding quality and leveraging information from a supporting language. to this end, we learn multilingual representations for a low-resource source language, a related language, and a target language in two steps: first, we train clwes for the low-resource language and the related higher-resource language using the joint-align approach by @xcite . in that manner, the internal structure of the low-resource embeddings becomes more similar to the structure of the higher-quality related language embeddings. in the second step, we map the resulting clwe space to the target space using the supervised muse model @xcite . since the first step results in a higher-quality embedding space for the source language, a better mapping to the target space can be found due to their higher isomorphism. in our experiments, we learn representations for occitan together with a related language and english. occitan is a low-resource romance language, which is related to high-resource languages like french and spanish, and especially closely related to catalan. since particularly good clwes exist for each of the three related languages paired with english, we make use of monolingual data from these languages in order to obtain better representations for occitan and english. by evaluating our final multilingual embedding space on the occitan-english bli task, we show that our method improves clwes for these languages compared to all the baseline settings. furthermore, we find that there are significant differences in how much of an improvement is achieved with each of the supporting languages. investigating the impact of multiple factors, such as the pairwise linguistic relatedness of the source, target and the related languages, their bli performance and the dataset sizes of the individual languages, we found the relatedness of the low-resource and the related language to be most influential. ## related work the occitan language occitan is a romance language which is spoken in the south of france, in the aran valley (a part of catalonia, spain), in a small region in italy at the french border and in monaco (see figure 1 foot_0 , where the ensemble of all colored areas represents the occitan-speaking territory). however, it is not used as a primary language in any of these countries and it only has an official status in catalonia. the language the closest related to occitan is catalan and they both belong to the occitano-romance languages @xcite . it is also closely related to other romance languages, e.g., french and spanish. occitan is (like all romance languages) an inflectional language which is morphologically richer than english: there is no case inflection, but it has a rather complex inflectional system for verbs. occitan word order follows the subject-verb-object regularities and it is therefore syntactically very similar to english. however, like spanish and catalan, but unlike french and english, occitan is a so-called pro-drop language, i.e., a conjugated verb can be used without a personal pronoun and hence the subject position does not necessarily have to be filled in an occitan sentence. the exact number of speakers of occitan is not known for certain. most sources report numbers between 1 and 10 millions, and there are significantly more people with passive knowledge of occitan than active speakers @xcite ). furthermore, rather than one occitan language, there are many different dialects (see figure 1 ). however, the languedocian variant is mostly used in written occitan and thus in the occitan wikipedia, which we use for our experiments. due to these factors the amount of available written digital resources is low. clwes for low-resource setups a lot of research on clwes for low-resource languages focuses on reducing the need for cross-lingual data. @xcite use adversarial training for aligning monolingual vector spaces without any bilingual signal. @xcite propose an unsupervised mapping method where they combine adversarial training with a procrustes analysis refinement step in every iteration. @xcite learn clwes jointly for their unsupervised neural machine translation model by concatenating corpora of source and target languages and training fasttext skipgram embeddings @xcite on this corpus. in order to combine the benefits of joint and mapping methods, @xcite propose an approach where they combine both methods. first, clwes are trained jointly on a concatenated corpus containing monolingual source and target language data. oversharing among source and target language vocabularies is then reduced by a vocabulary reallocation step, and finally, source embeddings are mapped to the target embeddings. however, despite the progress of unsupervised clwe models, multiple surveys argue against focusing on fully unsupervised approaches. firstly, giving up on every supervision signal is not necessary, since there is always a small amount of parallel data available if monolingual data is abundant @xcite . secondly, @xcite show that even the most robust unsupervised approach @xcite cannot deal properly with multiple distant and low-resource languages. nevertheless, there are still a lot of languages for which even monolingual data is extremely scarce. for these languages, monolingual embeddings are usually of poor quality @xcite . consequently, mapping methods are not fruitfully applicable, since they rely on high-quality monolingual embedding spaces. @xcite show that monolingual embedding quality of extremely lowresource languages can be improved if clwes for a low-and a high-resource language are trained jointly. @xcite propose a method for better clwes by using a small bilingual seed dictionary together with pre-trained monolingual embeddings of the higher-resource language for initialization. on the other hand, these approaches rely only on the source and target languages, while we show the benefits of incorporating further related languages into a multilingual space. leveraging related languages besides reducing data requirements, it is also helpful to explore information from linguistically related highresource languages in low-resource setups. this idea has, for example, been considered in machine translation (mt). @xcite propose a statistical mt model which requires only a small parallel corpus of the low-resource source and the high-resource target languages, and additionally a larger parallel corpus of a related high-resource language and the target language. nguyen and chiang (2017) introduce a transfer learning model for neural mt (nmt) where embeddings of shared words are kept when transferring the model from the original to a related low-resource language. @xcite train a nmt model where embeddings learned during training are computed from a universal embedding space which embed multiple languages. thus, high-resource languages can provide support for related low-resource languages. leveraging information from related highresource languages to build clwes for lowresource setups has only been considered in a few works until now. multiple approaches were proposed to build representations involving more than two languages, but they either rely on pretrained monolingual embeddings @xcite @xcite or large training corpora @xcite , and are thus not well suited for low-resource setups. @xcite proposed multi-support generalized procrustes analysis (mgpa) to directly incorporate related languages into clwes by learning a threeway alignment among english, a low-resource language, and a supporting language. they improve clwe quality for multiple low-resource language pairs, including occitan-english. however, unlike our method, mgpa does not consider the internal structure of the monolingual low-resource language space (since it relies on pre-trained monolingual embeddings). ## approach to improve clwes for low-resource setups, we incorporate a related language by learning representations in two steps: first, we train clwes for the low-resource and a related language jointly. subsequently, we use the resulting joint space to-gether with a set of monolingual target language embeddings to learn the final multilingual space including the low-resource, the supporting, and the target languages. we detail the two steps below. ## experimental setup corpora and vocabulary we pursue our experiments for the low-resource occitan language and we choose french, spanish, and catalan as supporting languages. like occitan, they are all romance languages and hence they all have a partly shared vocabulary with occitan as well as some similarities in morphology and syntax. french and spanish have been chosen because they are very high-resource. catalan has been chosen because it is the language the closest related to occitan. furthermore, it has been shown that for all three languages, very good clwes together with english can be obtained @xcite . we extract occitan, french, spanish, and catalan corpora from respective wikipedia dumps. 2 corpora and vocabulary sizes are listed in embeddings in all our experiments, we used the pre-trained english fasttext wiki word vectors released by @xcite . 3 for occitan, french, spanish, and catalan, we train our own monolingual embeddings using the gensim version of fasttext skipgram @xcite with the same parameters used for the pre-trained english embeddings. this is to ensure that they are learned on the same corpora than the embeddings in our proposed model. the monolingual occitan embedding space used for our baselines contains 111,353 word vectors. all the other monolingual spaces are restricted to the most frequent 200,000 words for training. the smaller number of occitan embeddings is due to the small corpus and the threshold of at least five occurrences for a word to be considered when training fasttext embeddings. the number of embeddings resulting from joint-align training with occitan and each of the supporting languages is shown in figure 2 . here, the proportion of occitan, related language, and shared word vectors is illustrated. parameters we compare the performance of our model against multiple baselines. we use supervised muse @xcite and generalized procrustes analysis, an extension of muse (gpa; @xcite , as baseline models where a mapping between monolingual occitan and monolingual english embeddings is performed. in addition, we train three baselines using multi-support gpa (mgpa; @xcite where pre-trained monolingual embeddings from either french, spanish or catalan are incorporated. we use all baseline models with default parameters except the threshold for ranking candidate translation pairs, which we set to 15,000 instead of default 10,000 in all models, since it results in a better alignment. in the first step of our proposed model, we use the joint-align model @xcite for occitan and a related language with default parameters. the only exception is that we use supervised muse @xcite for mapping instead of default rcsls @xcite in order to stay consistent with the second mapping step in our model. we tested using rcsls in both steps instead, but it did not yield a good mapping for occitan and english. we use supervised muse @xcite with the same parameters as in our baseline, both within joint-align training and in the second step of our proposed model. evaluation task our evaluation task is bilingual lexicon induction (bli). we use it to evaluate the quality of our final multilingual embedding spaces, translating from occitan to english. we also use it for evaluating the shared occitan and related language spaces resulting from the first step of our model. for this purpose, we run the muse evaluation script @xcite and we report scores achieved with csls retrieval. ## results we show the results for @xmath0, settings a-f and 1-6, respectively. note that as the mapping direction in case of muse and gpa, occitan was taken as the source and english as the target language. mgpa, however, can only be trained with the low-resource and the related language on the target language side. we evaluated the resulting clwes for @xmath1. for mgpa and our model, results for incorporating either french, spanish, or catalan are listed separately in different columns. furthermore, settings 1-6 of our model vary in two more dimensions. firstly, we employ two different subsets of the shared occitan-related language space as source embeddings: in settings 1-4, we use the 'full space' containing vectors of words contained in the shared and language specific (occitan and the given related language) vocabularies. in settings 5-6, we use a 'reduced space' containing only the vectors of shared and occitan vocabularies. secondly, we experiment with various bilingual supervision signals: the occitan-english training dictionary (oc-en), the dictionary of the respective incorporated related language and english (rel-en), both training dictionaries concatenated (full), or identical character string supervision (id char). in settings where the reduced source embedding space is used, we omit training with the 'rel-en' and 'full' dictionaries, since the related language words are excluded from the embedding space. it can be seen from support from related language words having a closer look at the numbers in and english words. the reason for this is that the related language does not only help to build better representations for the low-resource language in step 1 (joint-alignment) of our model, but it also helps to build a better mapping in step 2. this is due to the iterative refinement of muse which can update the initial training dictionary with goodquality related language-english word pairs as well in addition to the occitan-english pairs. ## conclusion in this paper, we presented a model for improving clwe quality in low-resource setups by learning multilingual embedding spaces with a related language. to this end, a multilingual embedding space containing the low-resource source language, a related language, and the target language words is learned in two steps: first joint training of lowresource and related language embeddings; and second mapping the resulting clwes to a target language space. we pursued our experiments for the low-resource language occitan with support from french, spanish, or catalan in different settings. we showed that our method improves the quality of clwes for these languages compared to both bilingual and multilingual baselines, especially when catalan, the closest related language to occitan, is incorporated (up to 15.5% p@1 improvement). investigating multiple factors, we found that the degree of linguistic relatedness of the low-resource and the incorporated language is the most decisive for how much support a language provides. our work indicates that novel approaches should not only focus on learning better representations using small corpora but also on incorporating data from related languages.