Towards Multi-Sense Cross-Lingual Alignment of Contextual Embeddings (2022)

Abstract: Cross-lingual word embeddings (CLWE) have been proven useful in many cross-lingual tasks. However, most existing approaches to learn CLWE including the ones with contextual embeddings are sense agnostic. In this work, we propose a novel framework to align contextual embeddings at the sense level by leveraging cross-lingual signal from bilingual dictionaries only. We operationalize our framework by first proposing a novel sense-aware cross entropy loss to model word senses explicitly. The monolingual ELMo and BERT models pretrained with our sense-aware cross entropy loss demonstrate significant performance improvement for word sense disambiguation tasks. We then propose a sense alignment objective on top of the sense-aware cross entropy loss for cross-lingual model pretraining, and pretrain cross-lingual models for several language pairs (English to German/Spanish/Japanese/Chinese). Compared with the best baseline results, our cross-lingual models achieve 0.52%, 2.09% and 1.29% average performance improvements on zero-shot cross-lingual NER, sentiment classification and XNLI tasks, respectively.

## introduction cross-lingual word embeddings (clwe) provide a shared representation space for knowledge transfer between languages, yielding state-of-the-art performance in many cross-lingual natural language processing (nlp) tasks. most of the previous works have focused on aligning static embeddings. to utilize the richer information captured by the pretrained language model, more recent approaches attempt to extend previous methods to align contextual representations. aligning the dynamic and complex contextual spaces poses significant challenges, so most of the existing approaches only perform coarse-grained alignment. @xcite compute the average of contextual embeddings for each word as an anchor, and then learn to align the static anchors using a bilingual dictionary. in another work, @xcite use parallel sentences in their approach, where they compute sentence representations by taking the average of contextual word embeddings, and then they learn a projection matrix to align sentence representations. they find that the learned projection matrix also works well for word-level nlp tasks. besides, unsupervised multilingual language models @xcite @xcite pretrained on multilingual corpora have also demonstrated strong cross-lingual transfer performance. however, studies @xcite @xcite have shown that adjusting the unsupervised multilingual language model with parallel sentences can help further improve cross-lingual performance. though contextual word embeddings are intended to provide different representations of the same word in distinct contexts, @xcite find that the contextual embeddings of different senses of one word are much closer compared with that of different words. this contributes to the anisomorphic embedding distribution of different languages and causes problems for crosslingual alignment. for example, it will be difficult to align the english word bank and its japanese translations 銀行 and 岸 that correspond to its two different senses, since the contextual embeddings of different senses of bank are close to each other while those of 銀行 and 岸 are far. @xcite propose two solutions to handle multi-sense words: 1) remove multi-sense words and then align anchors in the same way as @xcite ; 2) generate cluster level average anchor for contextual embeddings of multi-sense words and then learn a projection matrix in an unsupervised way with muse @xcite . they do not make good use of the bilingual dictionaries, which are usually easy to obtain, even in low-resource scenarios. moreover, their projection-based approach still cannot handle the anisomorphic embedding distribution problem. in this work, we propose a novel sense-aware cross entropy loss to model multiple word senses explicitly, and then leverage a sense level translation task on top of it for cross-lingual model pretraining. the proposed sense level translation task enables our models to provide more isomorphic and better aligned cross-lingual embeddings. we only use the cross-lingual signal from bilingual dictionaries for supervision. our pretrained models demonstrate consistent performance improvements on zero-shot cross-lingual ner, sentiment classification and xnli tasks. though pretrained on less data, our model achieves the state-of-the-art result on zero-shot cross-lingual german ner task. to the best of our knowledge, we are the first to perform sense-level contextual embedding alignment with only bilingual dictionaries. ## background: prediction tasks of language models next token prediction and masked token prediction are two common tasks in neural language model pretraining. we take two well-known language models, elmo @xcite and bert @xcite , as examples to illustrate these two tasks (architectures are shown in §a). next token prediction elmo uses next token prediction tasks in a bidirectional language model. given a sequence of n tokens (t 1 , t 2 , . . . , t n ), it first prepares a context independent representation for each token by using a convolutional neural network over the characters or by word embedding lookup (a.k.a. input embeddings). these representations are then fed into l layers of lstms to generate the contextual representations: h i,j for token t i at layer j. the model assigns a learnable output embedding w for each token in the vocabulary, which has the same dimension as h i,l . then, the forward language model predicts the token at position k with: ## proposed framework we first describe our proposed sense-aware cross entropy loss to model multiple word senses explicitly in language model pretraining. then, we present our joint training approach with sense alignment objective for cross-lingual mapping of contextual word embeddings. the proposed framework can be applied to most of the recent neural language models, such as elmo, bert and their variants. see ## related work cross-lingual word embedding demonstrates strong performance in many cross-lingual transfer tasks @xcite @xcite . the projection-based approach has a long line of research on aligning static embeddings @xcite @xcite @xcite . it assumes that the embedding spaces of different languages have an isomorphic structure, and fit an orthogonal matrix to project multiple monolingual embedding spaces to a shared space. many studies @xcite have extended the projection-based approach to contextual representation alignment. besides, there are many discussions on the limitations of the projection-based approach, arguing that the isomorphic assumption is not true in general @xcite @xcite , so non-linear mapping methods are also explored in recent work @xcite . joint training is another line of research and early methods @xcite @xcite learn static word embeddings of multiple languages simultaneously. extending joint training to cross-or multi-lingual language model pretraining has gained more attention recently. as discussed above, unsupervised multilingual language models @xcite @xcite @xcite also demonstrate strong cross-lingual transfer performance. there has been some work on sense-aware language models/embeddings @xcite @xcite , and most of them require wordnet @xcite or other additional resource for supervision. @xcite utilize both monolingual and bilingual information from parallel corpora to learn multi-sense word embeddings. @xcite embed wordnet knowledge into bert with attention mechanism. @xcite pretrain sensebert to predict both the masked words and their wordnet supersenses. similar to our framework, there are also some unsupervised approaches, but most of them are used to learn static embeddings. @xcite learn word representations with both local and global context, and then apply a clustering algorithm to learn multiprototype vectors. @xcite propose an extension to the skip-gram model that leverage k-means clustering algorithm learns multiple embeddings per word type. @xcite leverage reinforcement learning for modularized unsupervised sense level embedding learning. boyd-graber et al. (2020) use gumbel softmax for sense disambiguation when learning sense embeddings. ## conclusions in this paper, we have introduced a novel senseaware cross entropy loss to model word senses explicitly, then we have further proposed a senselevel alignment objective for cross-lingual model pretraining using only bilingual dictionaries. the results of the experiments show the effectiveness of our monolingual and bilingual models on wsd, zero-shot cross-lingual ner, sentiment classification and xnli tasks. in future work, we will study how to extend our method to multilingual models.