Do Explicit Alignments Robustly Improve Multilingual Encoders? (2020)

Abstract: Multilingual BERT (mBERT), XLM-RoBERTa (XLMR) and other unsupervised multilingual encoders can effectively learn cross-lingual representation. Explicit alignment objectives based on bitexts like Europarl or MultiUN have been shown to further improve these representations. However, word-level alignments are often suboptimal and such bitexts are unavailable for many languages. In this paper, we propose a new contrastive alignment objective that can better utilize such signal, and examine whether these previous alignment methods can be adapted to noisier sources of aligned data: a randomly sampled 1 million pair subset of the OPUS collection. Additionally, rather than report results on a single dataset with a single model run, we report the mean and standard derivation of multiple runs with different seeds, on four datasets and tasks. Our more extensive analysis finds that, while our new objective outperforms previous work, overall these methods do not improve performance with a more robust evaluation framework. Furthermore, the gains from using a better underlying model eclipse any benefits from alignment training. These negative results dictate more care in evaluating these methods and suggest limitations in applying explicit alignment objectives.

## introduction unsupervised massively multilingual encoders including multilingual bert @xcite and xlm-roberta @xcite are now standard tools for zeroshot cross-lingual transfer for nlp tasks @xcite . while almost all encoders are pretrained without explicit crosslingual objective, i.e. enforcing similar words from different languages have similar representation, improvements can be attained through the use of explicit cross-lingually linked data during pretraining, such as bitexts @xcite @xcite and dictionaries @xcite . as with cross-lingual embeddings @xcite , these data can be used to support explicit alignment objectives with either linear mappings @xcite @xcite or fine-tuning @xcite . however, as word-level alignments from an unsupervised aligner are often suboptimal, we develop a new cross-lingual alignment objective for training our model. we base on our objective on contrastive learning, in which two similar inputs -such as from a bitext -are directly optimized to be similar, relative to a negative set. these methods have been effective in computer vision tasks @xcite . additionally, most previous work on contextual alignments consider high-quality bitext like europarl @xcite or multiun @xcite . while helpful, these resources are unavailable for most languages for which we seek a zero-shot transfer. to better reflect the quality of bitext available for most languages, we additionally use opus-100 @xcite , a randomly sampled 1 million subset (per language pair) of the opus collection @xcite . we show that our new contrastive learning alignment objectives outperform previous work @xcite when applied to bitext from previous works or the opus-100 bitext. however, our experiments also produce a negative result. while previous work showed improvements from alignmentbased objectives on zero-shot cross-lingual transfer for a single task (xnli) with a single random seed, our more extensive analysis tells a different story. we report the mean and standard derivation of multiple runs with the same hyperparam-eters and different random seeds. we find that previously reported improvements disappear, even while our new method shows a small improvement. furthermore, we extend the evaluation to multiple languages on 4 tasks, further supporting our conclusions. finally, we evaluate xlmr large on these tasks, which dominate the results obtained from the alignment objectives. we conclude that explicit alignments do not improve cross-lingual representations under a more extensive evaluation with noisier bitexts, and improvements are lost when compared to larger models. this negative result shows the limitation of explicit alignment objective with larger-scale bitext and encoders. ## explicit alignment objectives we begin with a presentation of objective functions that use parallel data across languages for training multilingual encoders. these objectives assume multilingual data in the form of word pairs in parallel sentences. since gold word alignments are scarce, we use an unsupervised word aligner. let s and t be the contextual hidden state matrix of corresponding words from a pretrained multilingual encoder. we assume s is english while t is a combination of different target languages. as both mbert and xlmr operate at the subword level, we use the representation of the first subword, which is consistent with the evaluation stage. each s i and t i are a corresponding row of s and t, respectively. s and t come from the final layer of the encoder while s l and t l come from the l th -layer. linear mapping if s and t are static feature (such as from elmo @xcite )) then t can be aligned so that it is close to s via a linear mapping @xcite @xcite , similar to aligning monolingual embeddings to produce cross-lingual embeddings. for feature s l and t l from layer l, we can learn a mapping w l . 1 ) is known as procrustes problem @xcite and can be solved by svd. alternatively, eq. ( 1 ) can also be solved by gradient descent, without the need to store in memory huge matrices s and t. we adopt the latter more memory efficient approach. following @xcite , we enforce the orthogonality by alternating the gradient update and the following update rule with β = 0.01. note we learn different w l for each target language. this approach has yielded improvements in several studies. wang et al. (2019 ) used mbert and 10k parallel sentences from europarl to improve dependency parsing. wang et al. (2020) used mbert and 30k parallel sentences from europarl to improve named entity recognition (ner) on spanish, dutch, and german. wu et al. (2019) used bilingual bert and 10k parallel sentences from xnli (conneau et al., 2018) to improve dependency parsing (but not ner) on french, russian, and chinese. liu et al. (2019) did not evaluate on cross-lingual transfer tasks. l2 alignment instead of using s and t as static features, cao et al. (2020) proposed fine-tuning the entire encoder while being more efficient than eq. ( 4 ). as a result, we adopt the following objective (with λ = 1): ## discussion our proposed contrastive alignment objective outperforms l2 alignment @xcite and consistently performs as well as or better than no alignment using various quality bitext on 4 nlp tasks under a comprehensive evaluation with multiple seeds. however, to our surprise, previously proposed methods do not show consistent improvement over no alignment in this setting. therefore, we make the following recommendations for future work on cross-lingual alignment or multilingual representations: 1) evaluations should consider average quality data, not exclusively high-quality bitext. 2) evaluation must consider multiple nlp tasks or datasets. 3) evaluation should report mean and variance over multiple seeds, not a single run. more broadly, the community must estab-lish a robust evaluation scheme for zero-shot crosslingual transfer as a single run with one random seed does not reflect the variance of the method (especially in a zero-shot or few-shot setting). 5 while @xcite advocate using oracle for model selection, we instead argue reporting the variance of test performance, following the few-shot learning literature. additionally, no alignment methods improve xlmr and larger xlmr large performs much better, and raw text is easier to obtain than bitext. therefore, scaling models to more raw text and larger capacity models may be more beneficial for producing better cross-lingual models.