Cross-lingual Alignment Methods for Multilingual BERT : A Comparative Study (2020)

Abstract: Multilingual BERT (mBERT) has shown reasonable capability for zero-shot cross-lingual transfer when fine-tuned on downstream tasks. Since mBERT is not pre-trained with explicit cross-lingual supervision, transfer performance can further be improved by aligning mBERT with cross-lingual signal. Prior work propose several approaches to align contextualised embeddings. In this paper we analyse how different forms of cross-lingual supervision and various alignment methods influence the transfer capability of mBERT in zero-shot setting. Specifically, we compare parallel corpora vs dictionary-based supervision and rotational vs fine-tuning based alignment methods. We evaluate the performance of different alignment methodologies across eight languages on two tasks: Name Entity Recognition and Semantic Slot Filling. In addition, we propose a novel normalisation method which consistently improves the performance of rotation-based alignment including a notable 3% F1 improvement for distant and typologically dissimilar languages. Importantly we identify the biases of the alignment methods to the type of task and proximity to the transfer language. We also find that supervision from parallel corpus is generally superior to dictionary alignments.

## introduction multilingual bert (mbert) @xcite is the bert architecture trained on data from 104 languages where all languages are embedded in the same vector space. due to the multilingual and contextual representation properties of mbert, it has gained popularity in various multilingual and cross-lingual tasks @xcite . in particular, it has demonstrated good zero-shot cross-lingual transfer perfor- * work done during an internship at amazon. mance on many downstream tasks, such as document classification, nli, ner, pos tagging, and dependency parsing (wu and dredze, 2019), when the source and the target languages are similar. many experiments @xcite suggest that to achieve reasonable performance in the zeroshot setup, the source and the target languages need to share similar grammatical structure or lie in the same language family. in addition, since mbert is not trained with explicit language signal, mbert's multilingual representations are less effective for languages with little lexical overlap @xcite . one branch of work is therefore dedicated to improve the multilingual properties of mbert by aligning the embeddings of different languages with cross-lingual supervision. broadly, two methods have been proposed in prior work to induce cross-lingual signals in contextual embeddings: 1) rotation alignment as described in section 2 aims at learning a linear rotation transformation to project source language embeddings into their respective locations in the target language space @xcite @xcite ; 2) fine-tuning alignment as explained in section 3 internally aligns language sub-spaces in mbert through tuning its weights such that distances between embeddings of word translations decrease while not losing the informativity of the embeddings @xcite . additionally, two sources of crosslingual signal have been considered in literature to align languages: parallel corpora and bilingual dictionaries. while the choice of each alignment method and source of supervision have a variety of advantages and disadvantages, it is unclear how these affect the performance of the aligned spaces across languages and various tasks. in this paper, we empirically investigate the effect of these cross-lingual alignment methodologies and applicable sources of cross-lingual super-vision by evaluating their performance on zero-shot named entity recognition (ner), a structured prediction task, and semantic slot-filling (sf), a semantic labelling task, across eight language pairs. the motivation for choice of these tasks to evaluate are two-fold: 1. prior work has already studied alignment methods on sentence level tasks. @xcite show the effectiveness of mbert alignment methods on xnli (2018). 2. wordlevel tasks do not benefit from more pre-training unlike other language tasks that improve by simply supplementing with more pre-training data. in experiments over the xtreme benchmark, @xcite find that transfer performance improves across all tasks when multilingual language models are pre-trained with more data, with the sole exception of word-level tasks. they note that this indicates current deep pre-trained models do not fully exploit the pre-training data to transfer to word-level tasks. we believe that ner and slotfilling tasks are strong candidate tasks to assess alignment methods due to limited cross-lingual transfer capacity of current models to these tasks. to the authors' knowledge, this is the first paper exploring the comparison of alignment methods for contextual embedding spaces: rotation vs. finetuning alignment and two sources of cross-lingual supervision: dictionary vs. parallel corpus supervision on a set of tasks of structural and semantic nature over a wide range of languages. from the results, we find that parallel corpora are better suited for aligning contextual embeddings. in addition, we find that rotation alignment is more robust for primarily structural ner downstream tasks while the fine-tuning alignment is found to improve performance across semantic sf tasks. in addition, we propose a novel normalisation procedure which consistently improves rotation alignment, motivated by the structure of mbert space and how languages are distributed across it. we also find the effect of language proximity on transfer improvement for these alignment methods. x s and x t are matrices of size d×k where d is the dimensionality of embeddings and k is the number of parallel words from word-aligned corpora, or word pairs from a bilingual dictionary between the source and target languages. further work @xcite demonstrated that restricting w to a purely rotational transform improves cross-lingual transfer across similar languages. the orthogonality assumption reduces eq.( 1 ) into the so-called procrustes problem with the closed form solution: ## fine-tuning alignment with parallel corpora rotation alignment has a strong assumption that the two language spaces (or sub-spaces in case of mbert) are approximately isometric @xcite . @xcite reported that the geometry of language embeddings becomes dissimilar for distant languages, and the isometry assumption degrades the alignment performance in such cases. in addition, as explained in section 2.1 rotation alignment alone cannot achieve effective mapping when two languages spaces have separate centroids. therefore, next we consider existing work to non-linearly align two language spaces. @xcite proposed to directly align languages within mbert model through fine-tuning. english sentences willows lined the bank of the stream. they walked along the bank making conversation. a bank is a institution that accepts deposits from public. paychecks automatically deposited into the bank. went to the bank to make a withdrawal. open a bank account. german sentences die stadt liegt am ufer der elbe. wir gingen am ufer spazieren. jeden frühling tritt der fluss hier über die ufer. objective w.e i bank ≃ e i ufer "bank" word cloud "ufer" word cloud e i bank e i ufer english sentences willows lined the bank of the stream. they walked along the bank making conversation. a bank is a financial institution that accepts deposits. open a bank account. parallel translated sentences weiden säumten das ufer des baches. sie gingen am ufer entlang und unterhielten sich. eine bank ist ein finanzinstitut, das einlagen akzeptiert. ein bankkonto eröffnen. english mbert features german mbert features the objective of the fine-tuning is to minimise the distance between the two contextual representations of an aligned word pair in parallel corpora: however, fine-tuning with only the above objective would led to lose the semantic information in mbert learnt during pre-training, since a trivial solution to the eq.( 7 ) can be simply to make all the embeddings equal. to deal with this, cao et al. ( 2020 ) also proposed a regularisation loss that does not allow the embedding of a source language word to stray too far away from its original location e i sm in the pre-trained mbert model, namely: note that e i sm is generated from a copy of the original pre-trained mbert model where the parameters are kept frozen. both of the alignment and the regularization losses are combined and jointly optimised in order to align the two language subspaces while maintaining informativity of embeddings: here n s to n e is the range of mbert layers aligned. we experimented with two variants of the finetuning approach: 1) moving target language towards source language while keeping the source embeddings approximately fixed through the regularization term in eq.( 8 ); 2) moving the source language embeddings towards the target space while keeping the target language space relatively fixed, then the regularisation loss changes to: in this section, we firstly describe the resources and implementation details of the alignment methods followed by the zero-shot ner and sf tasks used to evaluate the alignments. in addition, we briefly explain the datasets used in the experiments. ## results and analysis the evaluation results of each alignment method on the downstream ner and sf tasks are reported in ## conclusion in this paper, we investigate cross-lingual alignment methods for multilingual bert. we empirically evaluate their effect on zero-shot transfer for downstream tasks of two types: structural ner and semantic slot-filling, across a set of diverse languages. specifically, we compare rotation alignment and fine-tuning cross-lingual alignment. we compare the effect of dictionary and parallel corpora supervision across all tasks. we also propose a novel normalisation technique that improves state-of-the-art performance on zero-shot ner and semantic slot-filling downstream tasks, motivated by how languages are distributed across the mbert space. our experimental settings cover four datasets families (one for ner and three for sf) across eight language pairs. key findings of this paper are as follows: (1) rotation-based alignments show large performance improvements (up to +19% absolute for turkish atis-sf) on moderately close languages, only a small improvement for very close target languages and no improvement for very distant languages; this work aims to pave the way for optimising language transfer capability in contextual multilingual models. in the future, we would like to further investigate patterns in the embedding space and apply alignment methods into specific regions of the multilingual hyperspace to obtain more tailorsuited alignments between language pairs. we would also like to evaluate zero-shot capabilities of alignments when applied to other language tasks.