Anchor-based Bilingual Word Embeddings for Low-Resource Languages (2021)

Abstract: Good quality monolingual word embeddings (MWEs) can be built for languages which have large amounts of unlabeled text. MWEs can be aligned to bilingual spaces using only a few thousand word translation pairs. For low resource languages training MWEs monolingually results in MWEs of poor quality, and thus poor bilingual word embeddings (BWEs) as well. This paper proposes a new approach for building BWEs in which the vector space of the high resource source language is used as a starting point for training an embedding space for the low resource target language. By using the source vectors as anchors the vector spaces are automatically aligned during training. We experiment on English-German, English-Hiligaynon and English-Macedonian. We show that our approach results not only in improved BWEs and bilingual lexicon induction performance, but also in improved target language MWE quality as measured using monolingual word similarity.

## introduction bilingual word embeddings are useful for crosslingual tasks such as cross-lingual transfer learning or machine translation. mapping based bwe approaches rely only on a cheap bilingual signal, in the form of a seed lexicon, and monolingual data to train monolingual word embeddings (mwes) for each language, which makes them easily applicable in low-resource scenarios @xcite @xcite . it was shown that bwes can be built using a small seed lexicon @xcite or without any word pairs @xcite relying on the assumption of isomorphic mwe spaces. recent approaches showed that bwes can be built without the mapping step. @xcite built fasttext embeddings @xcite on the concatenated source and target language corpora exploiting the shared character n-grams in them. similarly, the shared source and target language subword tokens are used as a cheap cross-lingual signal in @xcite ; conneau and @xcite . furthermore, the advantages of mapping and jointly training the mwes and bwes were combined in @xcite for even better bwes. while these approaches already try to minimize the amount of bilingual signal needed for cross-lingual applications, they still require a larger amount of monolingual data to train semantically rich word embeddings @xcite . this becomes a problem when one of the two languages does not have sufficient monolingual data available @xcite . in this case, training a good embedding space can be infeasible which means mapping based approaches are not able to build useful bwes @xcite . in this paper we introduce a new approach to building bwes when one of the languages only has limited available monolingual data. instead of using mapping or joint approaches, this paper takes the middle ground by making use of the mwes of a resource rich language and training the low resource language embeddings on top of it. for this, a bilingual seed lexicon is used to initialize the representation of target language words by taking the pre-trained vectors of their source pairs prior to target side training, which acts as an informed starting point to shape the vector space during the process. we randomly initialize the representations of all non-lexicon target words and run continuous bag-of-words (cbow) and skipgram (sg) training procedures to generate target embeddings with both word2vec @xcite and fasttext @xcite . our approach ensures that the source language mwe space is intact, so that the data deficit on the target side does not result in lowered source embedding quality. the improved monolingual word embeddings for the target language outperform embeddings trained solely on monolingual data for semantic tasks such as word-similarity prediction. we study low-resource settings for english-german and english-hiligaynon, where previous approaches have failed @xcite , as well as english-macedonian. ## method previous mapping approaches rely on the alignment of two pre-trained monolingual word embedding spaces. in case one of the two languages has significantly fewer resources available, this will strongly affect the resulting mapping negatively. this is also an issue for joint approaches because the shared token representations are biased towards the language with more training samples. our approach instead leverages the high resource language to improve performance on the low-resource language. we pre-train mwes for the source language and use the source mwes to initialize the space of the low resource target language. using a set of initial seed pairs, the representation of a seed word in the target space is replaced with the representation of its translation (anchor points). then, training is performed on the initialized space using only monolingual data from the low resource language by only updating the representation of non-seed words which are initialized randomly. through this method a bwe representation is directly induced from the anchor points of the fixed vectors. in some cases there are multiple valid translations for a single target language word. we experiment with either initializing with the average over these possible translations or randomly selecting only one of them. the averaging helps by finding a common anchor for the different semantic nuances the token might represent in different target language contexts. additionally, we experiment with enabling or disabling the updates of anchor vectors during training. we implemented the anchor point based initialization in both word2vec and fast-text with only complete token representations serving as potential anchors. in the case of fast-text these initializations have no influence on the subword (character n-gram) embeddings which are still initialized randomly, which makes intuitive sense in the common case of morphologically different language pairs. training is performed using standard hyperparameters included in the gensim word2vec and fasttext packages @xcite . unless stated otherwise, vectors are of dimensionality 300 with a context window of 5 words used during training. all models are trained for 5 epochs without further hyperparameter tuning utilizing a single desktop machine on a intel core i7-7700k cpu with 4.20ghz, a nvidia geforce gtx 1080 ti graphics card and 32 gb of ddr4 sdram. the parameters of each trained model are equal to the standard implementation of the packages as listed above. training time is largely dependent on input size, but corresponds to a few seconds up to roughly 5 minutes in the low resource setting. ## results the following section evaluates different models quantitatively using acc@5 and acc@1 as a metric. the baseline runs muse tool in supervised mode using iterative procrustes refinement to obtain the mapping using default parameters as reported in @xcite . for the english embedding the full corpus size was used, while in the case of the (low-resource) languages the corpora sizes were varied to observe changes in performance. ## conclusion we proposed a novel approach to build bwes to improve performance on language pairs with limited monolingual data on the target side. by utilizing pre-trained mwes of resource rich languages and a seed lexicon to fix anchor points before training, a structurally similar embedding space can be learned for the low resource language which is aligned with the source representations. we evaluated our approach on the bdi task using english-german to test varying training parameters and corpora sizes, on englishmacedonian and the extremely low resource language pair english-hiligaynon on which previous approaches failed. we showed that the performance of existing mapping approaches degrades drastically with lower monolingual data sizes, even when there are large seed lexicons available. in contrast, our proposed system outperformed previous mapping based approaches on these setups including english-hiligaynon. on top of improved bwes, we showed improved mwe quality as well for the target language by outperforming standard mwes on the monolingual word similarity task showing that it is beneficial for monolingual tasks as well. we implemented our approach for both word2vec and fasttext which we publicly release to promote reproducibility and further research. 2