Explicit Alignment Objectives for Multilingual Bidirectional Encoders (2021)

Abstract: Pre-trained cross-lingual encoders such as mBERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2020) have proven impressively effective at enabling transfer-learning of NLP systems from high-resource languages to low-resource languages. This success comes despite the fact that there is no explicit objective to align the contextual embeddings of words/sentences with similar meanings across languages together in the same space. In this paper, we present a new method for learning multilingual encoders, AMBER (Aligned Multilingual Bidirectional EncodeR). AMBER is trained on additional parallel data using two explicit alignment objectives that align the multilingual representations at different granularities. We conduct experiments on zero-shot cross-lingual transfer learning for different tasks including sequence tagging, sentence retrieval and sentence classification. Experimental results on the tasks in the XTREME benchmark (Hu et al., 2020) show that AMBER obtains gains of up to 1.1 average F1 score on sequence tagging and up to 27.3 average accuracy on retrieval over the XLM-R-large model which has 3.2x the parameters of AMBER. Our code and models are available at http://github.com/junjiehu/amber .

## introduction cross-lingual embeddings, both traditional noncontextualized word embeddings @xcite and the more recent contextualized word embeddings @xcite , are an essential tool for cross-lingual transfer in downstream applications. in particular, multilingual contextualized word representations have proven effective in reducing the amount of supervision needed in a variety of cross-lingual nlp tasks such as sequence labeling @xcite , question answering @xcite , parsing @xcite ., * *work partially done at google research. some attempts at training multilingual representations @xcite simply train a (masked) language model on monolingual data from many languages. these methods can only implicitly learn which words and structures correspond to each-other across languages in an entirely unsupervised fashion, but are nonetheless quite effective empirically @xcite . on the other hand, some methods directly leverage multilingual parallel corpora @xcite @xcite @xcite , which gives some degree of supervision implicitly aligning the words in the two languages. however, the pressure on the model to learn clear correspondences between the contextualized representations in the two languages is still implicit and somewhat weak. because of this, several follow-up works @xcite @xcite have proposed methods that use word alignments from parallel corpora as the supervision signals to align multilingual contextualized representations, albeit in a post-hoc fashion. in this work, we propose a training regimen for learning contextualized word representations that encourages symmetry at both the word and sentence levels at training time. our word-level alignment objective is inspired by work in machine translation that defines objectives encouraging consistency between the source-to-target and targetto-source attention matrices @xcite . our sentence-level alignment objective encourages prediction of the correct translations within a minibatch for a given source sentence, which is inspired by work on learning multilingual sentence representations @xcite . in experiments, we evaluate the zero-shot crosslingual transfer performance of amber on four different nlp tasks in the xtreme benchmark @xcite including part-of-speech (pos) tagging, paraphrase classification, and sentence retrieval. we show that amber obtains gains of up to 1.1 average f1 score on cross-lingual pos tagging, up to 27.3 average accuracy score on sentence retrieval, and achieves competitive accuracy in paraphrase classification when compared with the xlm-r-large model. this is despite the fact that xlm-r-large is trained on data 23.8x as large 1 and has 3.2x parameters of amber. this shows that compared to large amounts of monolingual data, even a small amount of parallel data leads to significantly better cross-lingual transfer learning. ## cross-lingual alignment this section describes three objectives for training contextualized embeddings. we denote the monolingual and parallel data as m and p respectively. ## related work while cross-lingual alignment is a long-standing challenge dating back to the early stage of research in word alignment @xcite , cross-lingual embeddings @xcite @xcite are highly promising in their easy integration into neural network models for a variety of cross-lingual applications. analysis studies on recent cross-lingual contextualized representations @xcite @xcite further demonstrates this advantage for zero-shot cross-lingual transfer in a representative set of languages and tasks. in particular to improve cross-lingual transfer, some attempts directly leverage multilingual parallel corpus to train contextualized representations @xcite @xcite with the hope of aligning words implicitly. the other line of work uses word alignments from parallel corpora as the alignment supervision in a posthoc fashion @xcite . notably, amber does not rely on any word alignment tools, and explicitly encourage the correspondence both on the word and sentence level. ## discussion and future work in this paper, we demonstrate the effectiveness of our proposed explicit alignment objectives in learning better cross-lingual representations for downstream tasks. nonetheless, several challenging and promising directions can be considered in the future. first, most existing multilingual models tokenize words into subword units, which makes the alignment less interpretable. how to align a span of subword units with meaningful semantics at the phrase level deserves further investigation. second, several studies @xcite have shown that attention may fail to capture word alignment for some language pairs, and a few works @xcite proposed neural word alignment to improve the word alignment quality. incorporating such recent advances into the alignment objective is one future direction. third, how to fine-tune a well-aligned multilingual model on english annotations without catastrophic forgetting of the alignment information is a potential way to improve cross-lingual generalization on the downstream applications.