First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT (2021)

Abstract: Multilingual pretrained language models have demonstrated remarkable zero-shot cross-lingual transfer capabilities. Such transfer emerges by fine-tuning on a task of interest in one language and evaluating on a distinct language, not seen during the fine-tuning. Despite promising results, we still lack a proper understanding of the source of this transfer. Using a novel layer ablation technique and analyses of the model’s internal representations, we show that multilingual BERT, a popular multilingual language model, can be viewed as the stacking of two sub-networks: a multilingual encoder followed by a task-specific language-agnostic predictor. While the encoder is crucial for cross-lingual transfer and remains mostly unchanged during fine-tuning, the task predictor has little importance on the transfer and can be reinitialized during fine-tuning. We present extensive experiments with three distinct tasks, seventeen typologically diverse languages and multiple domains to support our hypothesis.

## introduction zero-shot cross-lingual transfer aims at building models for a target language by reusing knowledge acquired from a source language. historically, it has been tackled with a two-step standard crosslingual pipeline @xcite : (1) building a shared multilingual representation of text, typically by aligning textual representations across languages. this step can be done using feature extraction @xcite as with the delexicalized approach @xcite or using word embedding techniques @xcite by projecting monolingual embeddings onto a shared multilingual embedding space, this step requiring explicit supervision signal in the target language in the form of features or parallel data. (2) training a task-specific model using supervision on a source language on top of the shared representation. recently, the rise of multilingual language models entailed a paradigm shift in this field. multilingual pretrained language models @xcite have been shown to perform efficient zero-shot cross-lingual transfer for many tasks and languages @xcite . such transfer relies on three-steps: (i) pretraining a mask-language model (e.g. @xcite ) on the concatenation of monolingual corpora across multiple languages, (ii) fine-tuning the model on a specific task in the source language, and (iii) using the finetuned model on a target language. the success of this approach is remarkable, and in contrast to the standard cross-lingual pipeline, the model sees neither aligned data nor task-specific annotated data in the target language at any training stage. the source of such a successful transfer is still largely unexplained. @xcite hypothesize that these models learn shared multilingual representations during pretraining. focusing on syntax, @xcite recently showed that the multilingual version of bert (mbert) @xcite , encodes linguistic properties in shared multilingual sub-spaces. recently, @xcite suggest that mbert learns a language encoding component and an abstract cross-lingual component. in this work, we are interested in understanding the mechanism that leads mbert to perform zero-shot cross-lingual transfer. more specifically, we ask what parts of the model and what mechanisms support cross-lingual transfer? by combining behavioral and structural analyses @xcite , we show that mbert operates as the stacking of two modules: (1) a multilingual encoder, located in the lower part of the model, critical for cross-lingual transfer, is in charge of aligning multilingual representations; and (2) a task-specific, language-agnostic predictor which has little importance for cross-lingual transfer and is dedicated to performing the downstream task. this mechanism that emerges out-of-the-box, without any explicit supervision, suggests that mbert behaves like the standard cross-lingual pipeline. our contributions advance the understanding of multilingual language models and as such have the potential to support the development of better pretraining processes. ## analysis techniques we study mbert with a novel behavioral test that disentangles the task fine-tuning influence from the pretraining step ( §2.1), and a structural analysis on the intermediate representations ( §2.2). combining the results from these analyses allows us to locate the cross-lingual transfer and gain insights into the mechanisms that enable it. ## experimental setup tasks, datasets and evaluation we consider three tasks covering both syntactic and semantic aspects of language: part-of-speech tagging (pos), dependency parsing, and named-entity recognition (ner). for pos tagging and parsing we use the universal dependency (nivre et al., 2018) treebanks, and for ner, we use the wikiann dataset @xcite . we evaluate our systems with the standard metrics per task; word-level accuracy for pos tagging, f1 for ner and labeled attachment score (las) for parsing. all the reported scores are computed on the test set of each dataset. we experiment with english, russian and arabic as source languages, and fourteen typologically diverse target languages, including chinese, czech, german and hindi. the complete list can be found in the appendix a.1.2. the results of a model that is fine-tuned and evaluated on the same language are referred to as same-language and those evaluated on distinct languages are referred to as cross-language. multilingual model we focus on mbert @xcite , a 12-layer model trained on the concatenation of 104 monolingual wikipedia corpora, including our languages of study. fine-tuning we fine-tune the model for each task following the standard methodology of @xcite . the exact details for reproducing our results can be found in the appendix. all reported scores are averaged on 5 runs with different seeds. ## discussion understanding the behavior of pretrained language models is currently a fundamental challenge in nlp. a popular approach consists of probing the intermediate representations with external classifiers @xcite @xcite to measure if a specific layer captures a given property. using this technique, @xcite showed that bert encodes linguistic properties in the same order as the "classical nlp pipeline". however, probing techniques only indirectly explain the behavior of a model and do not explain the relationship between the information captured in the representations and its effect on the task @xcite . moreover, recent works have questioned the usage of probing as an interpretation tool (hewitt and liang, 2019; @xcite . this motivates our approach to combine a structural analysis based on representation similarity with behavioral analysis. in this regard, our findings extend recent work from merchant et al. (2020) in the multilingual setting, who show that fine-tuning impacts mainly the up-per layers of the model and preserves the linguistic features learned during pretraining. in our case, we show that the lower layers are in charge of aligning representations across languages and that this cross-lingual alignment learned during pretraining is preserved after fine-tuning. ## conclusion the remarkable performance of multilingual languages models in zero-shot cross-lingual transfer is still not well understood. in this work, we combine a structural analysis of the similarities between hidden representation across languages with a novel behavioral analysis that randomlyinitialize the models' parameters to understand it. by combining those experiments on 17 languages and 3 tasks, we show that mbert is constructed from: (1) a multilingual encoder in the lower layers, which aligns hidden representations across languages and is critical for cross-language transfer, and (2) a task-specific, language-agnostic predictor that has little effect to cross-language transfer, in the upper layers. additionally, we demonstrate that hidden cross-lingual similarity strongly correlates with downstream cross-lingual performance suggesting that this alignment is at the root of these cross-lingual transfer abilities. this shows that mbert reproduces the standard cross-lingual pipeline described by @xcite without any explicit supervision signal for it. practically speaking, our findings provide a concrete tool to measure cross-lingual representation similarity that could be used to design better multilingual pretraining processes. roee aharoni and yoav goldberg. 2020. unsupervised domain clusters in pretrained language models. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 7747-7763, online. association for computational linguistics. guillaume alain and yoshua bengio. 2017. understanding intermediate layers using linear classifier probes. in 5th international conference on learning representations, iclr 2017, toulon, france, april 24-26, 2017, workshop track proceedings. openreview.net. chinatsu aone and douglas mckee. 1993. a language-independent anaphora resolution system for understanding multilingual texts. in 31st annual meeting of the association for computational linguistics, pages 156-163. yonatan belinkov, sebastian gehrmann, and ellie pavlick. 2020. interpretability and analysis in neural nlp. in proceedings of the 58th annual meeting of the association for computational linguistics: tutorial abstracts, pages 1-5. samuel r. bowman, gabor angeli, christopher potts, and christopher d. manning. 2015. a large annotated corpus for learning natural language inference. in proceedings of the 2015 conference on empirical methods in natural language processing, pages 632-642, lisbon, portugal. association for computational linguistics. ethan a. chi, john hewitt, and christopher d. manning. 2020. finding universal grammatical relations in multilingual bert. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 5564-5577, online. association for computational linguistics. alexis conneau, german kruszewski, guillaume lample, loïc barrault, and marco baroni. 2018. what you can cram into a single $&!#* vector: probing sentence embeddings for linguistic properties. in proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: long papers), pages 2126-2136, melbourne, australia. association for computational linguistics. alexis conneau and guillaume lample. 2019. crosslingual language model pretraining. in advances in neural information processing systems, pages 7057-7067. alexis conneau, shijie wu, haoran li, luke zettlemoyer, and veselin stoyanov. 2020. emerging cross-lingual structure in pretrained language models. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 6022-6034, online. association for computational linguistics. ido dagan, oren glickman, and bernardo magnini. 2005. the pascal recognising textual entailment challenge. in machine learning challenges workshop, pages 177-190. springer. jacob devlin, ming-wei chang, kenton lee, and kristina toutanova. 2019. bert: pre-training of deep bidirectional transformers for language understanding. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171-4186, minneapolis, minnesota. association for computational linguistics. yanai elazar, shauli ravfogel, alon jacovi, and y. goldberg. 2020. amnesic probing: behavioral explanation with amnesic counterfactuals. arxiv: computation and language. hila gonen, shauli ravfogel, yanai elazar, and yoav goldberg. 2020. it's not greek to mbert: inducing word-level translations from multilingual bert. in proceedings of the third blackboxnlp workshop on analyzing and interpreting neural networks for nlp, pages 45-56. rob van der goot and gertjan van noord. 2018. modeling input uncertainty in neural network dependency parsing. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 4984-4991. john hewitt and percy liang. 2019. designing and interpreting probes with control tasks. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 2733-2743. junjie hu, sebastian ruder, aditya siddhant, graham neubig, orhan firat, and melvin johnson. 2020. xtreme: a massively multilingual multitask benchmark for evaluating cross-lingual generalisation. in proceedings of the 37th international conference on machine learning, volume 119 of proceedings of machine learning research, pages 4411-4421. pmlr. diederik p. kingma and jimmy ba. 2015. adam: a method for stochastic optimization. corr, abs/1412.6980. simon kornblith, mohammad norouzi, honglak lee, and geoffrey hinton. 2019. similarity of neural network representations revisited. in international conference on machine learning, pages 3519-3529. xiaoman pan, boliang zhang, jonathan may, joel nothman, kevin knight, and heng ji. 2017. crosslingual name tagging and linking for 282 languages. in proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1946-1958, vancouver, a appendices a.1 reproducibility a.1.1 optimization we fine-tune our models using the standard adam optimizer (kingma and ba, 2015). we warmup the learning rate on the first 10% steps and use linear decay in the rest of the training. using the validation set of the source language, we find the best combination of hyper-parameters with a grid search on batch size among {16, 32} and learning rate initialization among {1e-5, 2.5e-5, 5e-5} we select the model with the highest validation performance out of 15 epochs for parsing and out of 6 epochs for pos tagging and ner. hyperparameters in languages for all our experiments, we use english, russian and arabic as source languages in addition to chinese, czech, finish, french, indonesian, italian, japanese, german, hindi, polish, portuguese, slovenian, spanish, and turkish as target languages. fine-tuning data for all the cross-lingual experiments, we use english, russian and arabic as source languages on which we fine-tune mbert. for english, we take the english-ewt treebank for fine-tuning, for russian the russian-gsd treebank and for arabic the arabic-padt treebank. evaluation data for all our experiments, we perform the evaluation on all the 17 languages. for parsing and pos tagging we use the test set from the pud treebanks released for the conll shared task 2017 @xcite . for ner, we use the corresponding annotated datasets in the wikiner dataset.