E nglish- M alay Cross-Lingual Embedding Alignment using Bilingual Lexicon Augmentation (2022)

Abstract: As high-quality Malay language resources are still a scarcity, cross lingual word embeddings make it possible for richer English resources to be leveraged for downstream Malay text classification tasks. This paper focuses on creating an English-Malay cross-lingual word embeddings using embedding alignment by exploiting existing language resources. We augmented the training bilingual lexicons using machine translation with the goal to improve the alignment precision of our cross-lingual word embeddings. We investigated the quality of the current state-of-the-art English-Malay bilingual lexicon and worked on improving its quality using Google Translate. We also examined the effect of Malay word coverage on the quality of cross-lingual word embeddings. Experimental results with a precision up till 28.17% show that the alignment precision of the cross-lingual word embeddings would inevitably degrade after 1-NN but a better seed lexicon and cleaner nearest neighbours can reduce the number of word pairs required to achieve satisfactory performance. As the English and Malay monolingual embeddings are pre-trained on informal language corpora, our proposed English-Malay embeddings alignment approach is also able to map non-standard Malay translations in the English nearest neighbours.

## introduction distributional semantic models produce word embeddings that allow us to compare the relationship between words. in (static) word embeddings, each word is associated to a continuous real-valued vector such that words that are semantically similar to each other will be in close proximity when we visualize them. word embeddings that have been adopted extensively include but are not limited to cbow @xcite , @xcite and @xcite . word embeddings that are pre-trained monolingually are limited to tasks solely in its own language. for this reason, we are unable to compare the meaning of words between languages or transfer models trained on one language to another language @xcite . crosslingual word embeddings could overcome the language constraint and make it possible for the more abundant english resources to be leveraged for emotion or other text classification in malay (i.e., the resource poor language of interest). in cross-lingual word embeddings, words that are semantically similar regardless of the languages, will appear to be close to each other in the vector space. the current state-of-the-art multilingual language models like mbert @xcite and xlm-roberta @xcite for cross-lingual transfer are computationally expensive. cross-lingual word embeddings offers an alternative that is cost-effective for cross-lingual transfer requiring a model to be trained only using the source language, which can then subsequently be applied to perform zero-shot or few-shot learning @xcite on the target language. in this study, we attempt to align english and malay monolingual static word embeddings pretrained on informal text (i.e., tweets or instagram posts) using the transformation method proposed by @xcite . corpora used to pre-train the embeddings are neither parallel nor aligned, and the only bilingual signal comes from the training bilingual lexicon. instead of constructing the ## related work word embeddings alignment is one of the approaches used to bridge the language gap between the source (resource rich) and target (resource poor) languages. prior studies can be categorized into those that require and do not require a set of bilingual seed lexicons. @xcite were one of the earliest and influential studies using bilingual seed lexicons for word embeddings alignment. a transformation matrix was learnt from the seed word pairs to linearly map the source word embeddings to the target embeddings space. @xcite enhanced this approach by introducing an l2regularization least-squares error in the objective function. xing et al. (2015) improved the method proposed by @xcite by restricting the word vectors to a unit length and constraining the transformation matrix to be orthogonal. they also redefined the objective function by using cosine similarity between the transformed source and target embeddings. these additional steps solved the inconsistency uncovered in @xcite and achieved better performance. on top of these constraints, artetxe et al. (2016) enforced dimension-wise mean centering on the word embeddings so that randomly chosen words would not be semantically similar. their study also discovered that the improvement attained by @xcite was solely from the orthogonal constraint instead of solving the inconsistency problem. @xcite proved that an orthogonal transformation matrix must also be selfconsistent. faruqui and dyer (2014) used canonical correlation analysis (cca) to learn the transformation matrices from the seed lexicon. unlike @xcite , a transformation matric was learnt for both source language and target language, respectively. the source and target word embeddings were then mapped to a new shared vector space where the seed word pairs from a lexicon would be maximally correlated. lu et al. (2015) adopted a non-linear extension of cca to train the transformation matrices. two neural networks were trained to obtain the transformation matrices by maximizing the correlation of the transformed source and target word embeddings in the new vector space. barone (2016) aligned word embeddings by eliminating the need for seed lexicons. they adopted an adversarial autoencoder in mapping the source embeddings to target embeddings. the source embeddings were transformed using an encoder, and the discriminator then tried to match the latent representations to the distribution of the target embeddings. zhang et al. (2017) matched the distribution of the transformed source embeddings to target embeddings using adversarial training. they learnt an orthogonal transformation matrix using the generator, and the discriminator would then try to distinguish the transformed source embeddings from target embeddings. additionally, they also attempted to relax the orthogonality constraint by using an adversarial autoencoder. artetxe et al. (2018) induced the initial seed word pairs by exploiting the similarity distribution between words using an unsupervised method. these initial seed word pairs were then refined through iterative self-learning. they also enforced the transformation matrices used to map the source and target embeddings to the new vector space to be orthogonal. feng and wan (2019) proposed an approach to nonlinearly map the source word embeddings and target word embeddings to a new vector space. they induced the seed word pairs using nearest neighbour retrieval, and the seed word pairs were then refined iteratively using self-learning. instead of orthogonality, they adopted the euclidean norm to guide the learning of the transformation matrices. recent studies proposed to align contextual embeddings. @xcite generalized the approach by @xcite and @xcite to align embeddings from mbert. since one word can have different embeddings based on the context, schuster et al. represented each word using the embedding anchor that was obtained by averaging a subset of its contextual embeddings. @xcite adopted the approach by @xcite to map contextual embeddings from the elmo @xcite . instead of using the embedding anchor, they constructed a dynamic bilingual lexicon from a parallel corpus with word alignment. additionally, they also proposed sentence-level mapping in which the transformation matrix was learnt on aligned sentences. @xcite also extended the method by @xcite to contextual embeddings. similar to @xcite , they formed the alignment matrix based on aligned word pairs extracted from parallel corpora. the representations extracted from mbert were then aligned by multiplying with the alignment matrix. existing studies have not explored static or contextual word embeddings alignment between english and malay languages. as word embeddings alignment has shown promising performance in cross-lingual transfer tasks in other language pairs, it provides strong motivation for us to explore how word embeddings alignment can also benefit the english-malay language pair, and subsequently any future study that may require english-malay cross-lingual word embeddings. ## data sources the method requires a monolingual english embedding, a monolingual malay embedding and a bilingual english-malay lexicon to map the two monolingual embeddings into a bilingual malay-english embedding. the quality of the bilingual english-malay lexicon plays an important role because it serves as the only bridge to map two separate english and malay monolingual lexicons into a single shared space. malay is written in the latin alphabet and shares lexical similarities with indonesian as they are from the same language family. ## conclusion in this study, we attempted to create english-malay cross-lingual word embeddings using an english-malay bilingual lexicon to map the english and malay monolingual word embeddings into a single representation that was empirically and intrinsically evaluated based on word pair coverage and alignment precision. despite the fact that the bilingual lexicon from malaya documentation being the current state-of-the-art, we demonstrated that its quality has room for improvement. our bilingual lexicons (n-bl) obtained using a naïve approach easily outperformed malaya documentation in terms of the english-malay alignment precision. we also investigated the effect of malay word coverage on bilingual lexicon induction and discovered that a higher coverage would not necessarily improve the alignment precision. also, we did not select our training or test lexicons based on word frequency in any corpora, thus our evaluation is more unbiased and generalizable. we are aware that there are semi-supervised and unsupervised approaches in creating cross-lingual word embeddings that require limited or almost no bilingual signals. however, we did not adopt such an approach because both our embeddings were pre-trained on informal corpora, especially our malay monolingual embeddings still containing significant noise even after applying the filter. hence, legitimate words would easily be paired up with noise, or vice versa, without bilingual supervision. we adopted word2vec in this study as we were not aware of any existing malay fasttext embeddings pre-trained on the social media domain, and pre-training it ourselves is not within the scope of this study. in the future, we wish to pre-train malay fasttext embeddings that may work better on informal corpora and subsequently explore the feasibility of creating embeddings using semi-supervised and unsupervised methods. we also plan to evaluate the performance of our english-malay cross-lingual word embeddings on downstream tasks such as emotion classification.