On the Language-specificity of Multilingual BERT and the Impact of Fine-tuning (2021)

Abstract: Recent work has shown evidence that the knowledge acquired by multilingual BERT (mBERT) has two components: a language-specific and a language-neutral one. This paper analyses the relationship between them, in the context of fine-tuning on two tasks – POS tagging and natural language inference – which require the model to bring to bear different degrees of language-specific knowledge. Visualisations reveal that mBERT loses the ability to cluster representations by language after fine-tuning, a result that is supported by evidence from language identification experiments. However, further experiments on ‘unlearning’ language-specific representations using gradient reversal and iterative adversarial learning are shown not to add further improvement to the language-independent component over and above the effect of fine-tuning. The results presented here suggest that the process of fine-tuning causes a reorganisation of the model’s limited representational capacity, enhancing language-independent representations at the expense of language-specific ones.

## introduction since the introduction of transformer architectures and the demonstration that they improve the state of the art on tasks such as machine translation and parsing @xcite , there has been a decisive turn in nlp towards the development of large, pre-trained transformer models, such as bert @xcite . such models are pretrained on tasks such as masked language modelling (mlm) and next-sentence prediction (nsp) and are intended to be task-agnostic, facilitating their transfer to new tasks following fine-tuning with limited amounts of data. extending such models to multiple languages is a natural next step, as evidenced by the recent proliferation of multilingual transformers, including multilingual bert (mbert), xlm (conneau and lample, 2019), and xlm-r @xcite . these follow from a line of earlier work which sought to achieve transferable multilingual representations using recurrent networkbased methods (e.g. @xcite , as well as work on developing multilingual embedding representations @xcite . the considerable capacity of these multilingual models and their success in cross-lingual tasks has motivated a lot of research into the nature of the representations learned during pretraining. on the one hand, there is a significant amount of research suggesting that models such as mbert acquire robust language-specific representations @xcite @xcite . on the other hand, it has been suggested that in addition to language-specific information, models like mbert also have language-neutral representations, which cut across linguistic distinctions and enable the model to handle aspects of meaning language-independently. this also allows the model to be fine-tuned on a monolingual labelled data set and achieve good results in other languages, a process known as cross-lingual zero-shot learning @xcite @xcite . these results have motivated researchers to try and disentangle the language-specific and languageneutral components of mbert (e.g. @xcite . this background provides the motivation for the work presented in this paper. we focus on the relationship between language-specific and languageneutral representations in mbert. however, our main goal is to study the impact of fine-tuning on the balance between these two types of representations. more specifically, we measure the effect of fine-tuning on mbert's representations in the context of two different tasks -part-ofspeech (pos) tagging and natural language inference (nli) -which lay different demands on the model's semantic and language-specific knowledge. while nli involves reasoning about deep semantic relations between texts, pos tagging requires a model to bring to bear knowledge of a language's morphosyntactic features. though many languages share such features as a result of typological relations (which mbert is known to exploit; see, e.g. @xcite @xcite , there are also language-specific features to which, we hypothesise, mbert needs to dedicate a greater share of its representational capacity, compared to the nli task. we show that the model accommodates language-specific and language-neutral representations to different degrees as a function of the task it is fine-tuned on. this is supported by results from language identification (lid) experiments, conducted both on task-specific data and on a new data set extracted from wikipedia. we then consider two alternative strategies that force the model to 'unlearn' language-specific representations, via gradient reversal or iterative adversarial learning. these are shown not to further improve the language-independent component for cross-lingual transfer, over and above the effect of fine-tuning. thus, we conclude that the reorganisation of mbert's representations that happens with fine-tuning is already taking on this role. note that our goal is not to improve mbert's multilinguality but to acquire a better understanding of it, extending previous work along these lines. our main contributions are @xcite to provide further support for the distinction between languagespecific and language-neutral representation in mbert; (b) to show that fine-tuning results in a reorganisation of mbert's representations in a way that destroys existing language clusters; @xcite to study two methods to enhance language-neutrality in mbert, both of which are shown not to improve performance on fine-tuned tasks; (d) a new wikipedia-based language identification data set. ## tasks and data to perform cross-lingual zero-shot learning, we fine-tune mbert on english only and evalu-ate it on multiple languages at once. we focus on two tasks: the low-level structured prediction task of cross-lingual pos tagging and the high-level semantic task of cross-lingual nli. we chose these two tasks, because each task requires a different type of linguistic knowledge. crosslingual pos tagging requires a model to bring to bear language-specific knowledge related to morphosyntactic properties, word order, etc., in determining the correct part-of-speech to assign to a token in context. previous work, for example by @xcite , showed good results on this task for mbert in a cross-lingual zero-shot setting. in contrast, nli requires a model to determine the semantic relationship between two texts, determining whether it is one of entailment, contradiction or neutrality @xcite . we expected this task to require more language-neutral, semantic knowledge, compared to pos tagging. 1 in addition to results on these two tasks, we report results on language identification (lid) experiments. these are reported both on the test data for the tasks themselves, as well as on an independent data set consisting of wikipedia texts, described below. in all the experiments reported, we reserve a development set for hyperparameter tuning and a validation set to monitor progress during training. data set statistics are provided in the appendix. following the practice of @xcite for the xtreme benchmark, all texts were truncated to their first 128 tokens (with xnli having a combined premise-hypothesis length of 128). all the data was tokenised using the bert-base-multilingual-cased tokeniser. foot_1 data containing unknown tokens (according to the tokeniser) was omitted. all our experiments are conducted on data for 33 languages, the same set of languages included in the ud-pos task of the xtreme benchmark @xcite . 3 the exception is xnli, for which crosslingual test data exists for only 15 of these lan-guages @xcite . udpos for pos tagging, we use data from the universal dependencies treebank (udpos; marneffe et al., 2020) v2.7, using the train/dev/test splits provided. a validation set is randomly sampled from the training set. since we are interested in cross-lingual zero-shot learning, we removed all non-english data from the train/val splits. xnli for nli, we use the monolingual english multinli data set @xcite as a training set, and the cross-lingual natural language inference data (xnli; @xcite for the development set and test set. again, a validation set is randomly sampled from the training set. wikipedia an independent data set for lid was extracted from wikipedia. for each language, we randomly selected 5 000 paragraphs which are at least 100 characters in length. further details are provided in the appendix. ## model architecture the general architecture used in our experiments is shown in figure 1 . we use a pre-trained mbert model to encode the input using its final hidden layer, because we are interested in the linguistic capabilities of mbert's typical use case, which is also the practice in the xtreme benchmark @xcite . the same mbert model is shared between two single-layer softmax classifiers, both trained using a categorical cross-entropy loss. one of these assigns a task-specific label (a pos tag or an nli class). this is trained on udpos or xnli data. we will sometimes refer to this as the target task. the other is a language classifier that predicts which of the 33 languages (see above) the input text is written in. the language classifier is trained on the wikipedia data. we conduct separate experiments for each target task (udpos and xnli). in the case of ud-pos, the classification is for individual tokens. for xnli, the classification is for sentence pairs, represented as a single text consisting of the concatenation of the premise and hypothesis, separated by a [sep] token. the language classifier is also trained to either predict the language of each token or of an entire text according to the target task. in this way, we are also able to test this classifier for predictions both on the independent wikipedia data and on the test data from the target task (which is also labelled by language). unless otherwise specified, gradients from the language classifier are not propagated to the pretrained mbert model. thus, the mbert model parameters are only fine-tuned on the target task data set whilst the language classifier is finetuned in isolation. this allows us to monitor how much language-specific information exists in the mbert encoding, without directly influencing it. more information about how the model was trained together with the hyperparameters used can be found in the appendix. foot_3 ## experiments in what follows, we first discuss the language clustering and lid capabilities of mbert both in its unmodified form, and after fine-tuning on ud-pos and xnli. then, we consider the impact of gradient reversal and entropy maximisation, both of which seek to shift mbert representations towards greater language neutrality. ## related work several studies have shown that mbert performs well in zero-shot transfer settings in low-level structured prediction tasks @xcite and in comparison to models that use explicit multilingual information for a variety of tasks and languages @xcite . another focus of research has been the correlation between cross-lingual performance and shared vocabulary (e.g. as measured by wordpiece overlap). k et al. some previous work has also shed light on the impact of training strategies on cross-lingual zeroshot learning. tan and joty (2021) show that training on code-switched data improves cross-lingual transfer, while @xcite show that intermediate task training, prior to fine-tuning proper, also results in better transfer with xlm-r @xcite . this is related to the present finding that mbert representations become less language-specific after fine-tuning. ## conclusions this paper explored the interplay between language-specific and language-neutral representations in the multilingual transformer model mbert. our results show that fine-tuning on specific tasks has a differential impact on how much language-specific information is retained. on the other hand, gradient reversal and iterative entropy maximisation interfere with fine-tuning and do not improve task performance. fine-tuning for tasks requiring identification of morphosyntactic properties, such as pos tagging, can result in greater loss of language-specific information, compared to deeper semanticallyoriented tasks such as nli 9 . however, there are also differences in the input data of these two tasks which could have impacted the results reported here. the data sets differ in their homogeneity (unlike xnli, udpos is constructed from multiple sources) and size. the tasks also differ in their granularity, in that nli is text-level, while pos is token-level. furthermore, nli is known to be susceptible to 'shortcut learning', whereby models rely on recurrent biases in training to solve the task (d'amour et al.) whereas shortcuts in part of speech tagging are less known. these considerations are worthy of further investigation. we believe these results contribute towards greater understanding of fine-tuning on multilingual model representations. this results in a reorganisation of the representation space, suggesting that language-specific and language-independent subspaces are dependent on task.