Multilingualism Encourages Recursion: a Transfer Study with m BERT (2022)

Abstract: The present work constitutes an attempt to investigate the relational structures learnt by mBERT, a multilingual transformer-based network, with respect to different cross-linguistic regularities proposed in the fields of theoretical and quantitative linguistics. We pursued this objective by relying on a zero-shot transfer experiment, evaluating the model’s ability to generalize its native task to artificial languages that could either respect or violate some proposed language universal, and comparing its performance to the output of BERT, a monolingual model with an identical configuration. We created four artificial corpora through a Probabilistic Context-Free Grammar by manipulating the distribution of tokens and the structure of their dependency relations. We showed that while both models were favoured by a Zipfian distribution of the tokens and by the presence of head-dependency type structures, the multilingual transformer network exhibited a stronger reliance on hierarchical cues compared to its monolingual counterpart.

## introduction massively multilingual models (mmms) are neural networks that can perform a nlp task in multiple languages, relying on a shared set of parameters. at the time of writing, the state-of-the-art performance of mmms is achieved by transformerbased models such as multilingual bert (mbert, @xcite , xlm @xcite , and xlm-r @xcite . they are usually derived from monolingual language models, trained simultaneously on multilingual text in up 104 languages without major architectural changes nor any reliance on explicit cross-lingual signal. the practical need for mmms in nlp is undisputed: they drastically reduce resource and maintenance requirements with respect to multiple monolingual models, and benefit in particular lowand mid-resource languages @xcite . mmms reach impressive performance levels in zero-shot cross-lingual transfer, enabling the fine-tuning of a model on supervised data in a set of n languages {l i } @xmath0. . . n and its application to a different language l n+1 , with no additional training foot_0 . zero-shot cross-lingual transfer has been shown to be effective across a variety of tasks and languages @xcite @xcite ; see @xcite for a review), and, although performance levels tend to be higher for typologically similar languages, it yields surprising results in languages written in different scripts @xcite and with little @xcite or no @xcite vocabulary overlap. the distribution of resources available for nlp researchers in the world's languages is extremely skewed, with only a small subset of them being represented in the evolving language technologies @xcite . mmms constitute an attempt to mitigate the effects of this uneven allocation of resources by leveraging the knowledge that can be shared across languages. besides the obvious practical advantages that mmms can bring to the nlp community, the nature of the cross-linguistic information extracted by these models is of high theoretical interest from a linguistic standpoint, and can contribute to the domain of artificial intelligence research in relevant subfields such as representation learning and interpretability. a modest but growing body of findings suggests that the structure of the representation space that mmms exploit is multilingual in nature @xcite @xcite ; although see @xcite for opposite conclusions). for instance, syntactic trees can be retrieved from mbert's intermediate representational subspaces, with these subspaces being approximately shared across languages @xcite . if mmms learn universal patterns which generalize across languages, the structure of the representations they induce could inform us of the presence of latent regularities in different language spaces. furthermore, the benefits of the study of the mmms' behaviour extend to the domain of representation learning, a subfield of ai research focusing on the development of computational representations and the analysis of their properties @xcite . while the present study will not analyze the internal states of the networks, it will be possible to draw conclusions on the generality of their learned representations through non-parametric probing, by directly examining their behaviour in response to non-linguistic input. more precisely, we will compare the suitability of the representational formats induced by mono-and multilingual models with respect to different properties that are desirable from a linguistic perspective. the present work aims to analyze the generalizations that bert and mbert induced from natural language data in a set of transfer learning experiments. the use of transfer learning methods to shed light on the relational structures learned by neural networks has been recently adopted for monolingual models @xcite . here, we extended the transfer approach to a multilingual setting, and compare the performance of mbert and its monolingual counterpart in generalizing their native task (i.e. masked language modelling) to artificial languages that display different degrees of structural similarity with natural languages. we wish to highlight three main differences between our paradigm and the methodologies papadimitriou and jurafsky proposed. our experiment focused on the zipf's law and hierarchy, which have been considered as universal linguistic features @xcite . we evaluated the transfer performances of the two transformers in four corpora, characterized by increasing statistical and structural consistency with natural languages. the models were tested on (a) a random corpus, composed by sequences of tokens sampled from a uniform distribution, (b) a zipfian corpus, where the tokens were extracted from a zipfian distribution, (c) a flat brack-ets corpus, composed of sequences of matching parentheses with crossed dependencies, and (d) a nested brackets corpus, consisting of paired symbols nested hierarchically. to anticipate the results, we found that both models showed higher performance scores in the zipfian compared to the random condition, and in the flat brack-ets as opposed to the zipfian corpus, while only the multilingual model showed a significant performance advantage in the comparison between the flat brackets and the nested brackets corpora. we conclude that while mathematical regularities and pairwise head-dependent relationships are detected across model types, the multilingual input favours the reliance on structural cues, and specifically on balanced constituent structures, a hallmark of theoretical linguistic formalisms. ## methods as a testing procedure, we froze all mbert's weights setting it in evaluation mode, and assessed its structural knowledge by studying its predictive ability. to do so, we employed a non-parametric evaluation procedure, which we named iterative token-level cloze task (itct). the itct consists of an adaptation of mbert's native functionality, i.e. masked language modelling (mlm). the main difference consists in the fact that while in mlm the model has to predict the tokens corresponding to the masks applied to a randomly selected subpart of the input (15% of the tokens in the sentence), in the itct all the tokens are masked iteratively. this mitigates the aleatory dimension in the selection of the tokens that are masked, and provides an index of the predictability of a sequence, where each token has to be predicted by the model given the whole remaining context. after freezing its weights foot_1 , at the first timestep t 0 the model is presented with the input sequence where the first token is masked, i.e. substituted with a mask token, and two special characters -[cls] and [sep] -are appended to the beginning and the end of the sequence, to mark the sequence boundaries. the model then predicts the original token relying upon the right context, and the hidden vector corresponding to the masked token is passed through a softmax over the vocabulary, in order to assign it a probability. at t 1 , the mask is moved from the first to the second token, and now mbert's prediction is conditioned by both the right and the left contexts. the process is repeated until the end of the sequence, with the number of timesteps n being equal to the length of the tokenized input (see figure 1 ). the mean probability assigned to the masked tokens across all the timesteps is taken as an index of the overall predictability of the sequence. note that a proper sequence probability metric would require a multiplicative chain rule of the kind that is applicable for auto-regressive models but not for masked language models. the notion of average probability does not correspond to any well-defined notion in probability theory, but it serves the purpose of comparing different structural configurations in the context of our study foot_2 . the predictions of mbert were compared with the ones produced by its monolingual counterpart; this comparison provides a crucial element for distinguishing which generalizations are driven by the multilingual input, and which can be extracted by the same model from monolingual data. ## data the corpora on which our analyses were performed were created in a way such that the sequence length varied within each condition, but was identical across all conditions, both in terms of the number of sequences and of the number of tokens within each sequence; they all shared a 50,000 three-letter tokens vocabulary. these design choices were made in order to license pairwise comparisons at the sequence level, so that the difference in probability assigned by the models to a given item of a corpus could be compared with the probability assigned to the corresponding item in the other datasets. this approach allowed us to rule out the effects of intervening variables such as vocabulary and length, so that the differences in the models' predictions could be driven only by structural differences between the corpora. the models were tested on 1,000 sequences in each corpus; within each condition, the mean sequence length was 9.90, with a standard deviation equal to 14.74. ## models and experimental setup all our experiments were performed employing bert's native masked language modelling component. the configuration of the model was left unaltered with respect to @xcite release. in particular, we relied on the monolingual and multilingual models derived from bert base , which is composed of 12 layers, 12 self-attention heads, and a hidden size of 768; the overall network comprises 110m parameters. the networks did not undergo any fine-tuning nor adaptation process, as they were employed as out-of-the-box masked language models. as mentioned above, bert and mbert only differ in their vocabulary and the weights learned during training, sharing an identical configuration both in terms of architectural choices and learning objectives. while bert was pre-trained on 800m words of the monolingual bookscorpus @xcite and 2,500 words of the english wikipedia, mbert was trained on the entire wikipedia dump of 104 languages. the two models rely on different tokenizers, each comprising a separate wordpiece vocabulary @xcite . the different tokenizations of the input sequences do not allow us to directly compare the raw probabilities assigned by the two models to a given sequence; for this reason, we will not consider the absolute item-wise difference in the models' predictive performance, but rather the pattern of results between the experimental conditions. this comparative approach is also needed in light of the models' vocabulary. if we simply compared probabilities across models and conditions, our results might be biased by the fact that some of the three-letter tokens might be assigned different probabilities depending on whether they form english meaning-bearing vocabulary items (e.g. "for") or not (e.g. "zyi"). however, since we only compare conditions within models, and the two conditions of main interest (i.e. flat and nested brack-ets) are composed by the same tokens in different arrangements, our contrasts are robust with respect to this possible confound. ## results we tested the statistical significance of this dissociation through a set of paired samples t-tests between the mean probability assigned by the models to the sequences of two corpora. we performed the tests only on the pairs of corpora that were adjacent in the structural hierarchy; this choice led us to three comparisons which we summarized in ## discussion in discussing the present findings, we begin by focusing on the commonalities in the networks' output, and conclude by commenting on the dissociation in their results on the two corpora with token pairing. first, both models showed a preference for sequences where the mathematical distribution of the tokens resembled their empirical distribution in natural languages. we believe that the higher average predictability that characterized the zipf's corpus when compared with the ran-dom corpus reflects a tendency of the networks to expect a non-uniform distribution of the tokens in input that is coherent with the data on which the pre-training had been performed. then, we maintain that the substantial gain in performance on the flat bracket corpus is to be attributed to the paired correspondences between tokens, which in turn might mimic head-dependency type structures in natural language corpora. arguably, the most surprising result that we obtained is the dissociation between bert and mbert's results in the transfer to the two corpora characterized by token pairing. while the multilingual model was facilitated in its native task by the presence of nested structuresalthough with a minimal effect size -, the same improvement in performance is not found in its monolingual counterpart. on the contrary, the strongest transfer performance is achieved by bert in the flat brackets corpus. these results suggest that the presence of multiple languages in the input during pre-training leads the models to rely on more structured grammatical abstractions. obviously, this finding does not imply that mbert does not capture the paired relationships with crossing arcs; the biggest progress is undoubtedly obtained when these simpler one-to-one correspondences are included in the input. nonetheless, it seems that when more complex structures are instantiated between the tokens in the sequences, the multilingual model is able to capture these configurational regularities, and exploit them in order to make stronger predictions regarding the masked input. this difference should be attributed to the nature of the input that the networks had been presented with during pre-training, since under every other aspect except vocabulary and training set size (e.g. architectures, training regimes, objective functions) the models were identical. ## conclusion the mathematical regularities of the pre-training input seemed to have been absorbed to the same extent by the monolingual and the multilingual transformer models, since the zipf's corpus exerted a similar facilitation effect with respect to the ran-dom corpus across model types. furthermore, token pairing appears to have elicited a much stronger advantage in the predictive task we employed in our study, suggesting that regardless of the monoor multilingual nature of the input the pre-training procedure had induced a strong structural transfer towards non-hierarchical and non-projective structures. we agree with papadimitriou and jurafsky's length.