Cross-Linguistic Syntactic Difference in Multilingual BERT : How Good is It and How Does It Affect Transfer? (2022)

Abstract: Multilingual BERT (mBERT) has demonstrated considerable cross-lingual syntactic ability, whereby it enables effective zero-shot cross-lingual transfer of syntactic knowledge. The transfer is more successful between some languages, but it is not well understood what leads to this variation and whether it fairly reflects difference between languages. In this work, we investigate the distributions of grammatical relations induced from mBERT in the context of 24 typologically different languages. We demonstrate that the distance between the distributions of different languages is highly consistent with the syntactic difference in terms of linguistic formalisms. Such difference learnt via self-supervision plays a crucial role in the zero-shot transfer performance and can be predicted by variation in morphosyntactic properties between languages. These results suggest that mBERT properly encodes languages in a way consistent with linguistic diversity and provide insights into the mechanism of cross-lingual transfer.

## introduction cross-lingual transfer aims to address the huge linguistic disparity in nlp by transferring the knowledge acquired in high-resource languages to low-resource ones, where pretrained multilingual encoders, such as multilingual bert (mbert) @xcite , have proven a powerful facilitator. compared to other approaches learning certain cross-lingual alignment in a supervised @xcite @xcite or unsupervised @xcite @xcite manner, mbert directly learns to encode different languages in a shared representation space through self-supervised joint training, dispensing with explicit alignment. it has exhibited notable crosslingual ability and can perform effective zero-shot cross-lingual transfer across a variety of downstream tasks, albeit the performance varies @xcite . the simplicity and efficacy of mbert are crucial for cross-lingual transfer and have sparked interest in investigating the reason for its success. previous work has looked into its representation space and found that mbert automatically performs certain alignment across languages @xcite @xcite . the extent of alignment is shown correlated with the transfer performance @xcite . despite these insights into the source of the transfer, it is also intriguing why different languages are aligned to varying degrees and what implication such variation bears. another line of work has demonstrated that the zero-shot transfer performance is affected by certain linguistic features such as word order @xcite , whereas the underlying mechanism is left unexplored. taken together, it remains unclear how different aspects of cross-linguistic differences impact the representations and further affect the cross-lingual transfer of different tasks. in this paper, we focus on the syntactic level and investigate the cross-lingual transfer of mbert based on 24 typologically distinct languages, with the purpose of figuring out the following questions: q1: does mbert properly induce cross-linguistic syntactic difference via selfsupervision? the distance between distributions over mbert representations of grammatical relations in different languages can be used to evaluate the syntactic difference between languages encoded in mbert (section 2). we compare it with the cross-linguistic syntactic difference in terms of linguistic formalisms for validation and rely on it to investigate the cross-lingual ability of mbert. q2: how does the syntactic difference learnt by mbert impact its cross-lingual transfer? the zero-shot cross-lingual transfer is typically realized through fine-tuning the pretrained multilingual model on a certain source language. we analyze the change pretraining and fine-tuning brought to the distance between distributions (i.e., the syntactic difference between languages in mbert) to understand the mechanism behind the transfer (section 3). q3: if and to what extent do various morphosyntactic properties impact the transfer performance? we then investigate the reason for the variation in the transfer performance based on syntactic-related linguistic properties. we exploit all the morphosyntactic properties available in the world atlas of language structures (wals) @xcite and examine the extent to which variation in them impacts the distance between distributions and further affects the transfer performance through regression analysis (section 4). our quantitative results and qualitative analysis demonstrate that: at a language level and the word-level contextual representations, we look into the distributions over mbert representations of different languages. we first derive representations of syntactic information (i.e., grammatical relations) from mbert and then use the divergence between distributions over the representations to measure the language-wide difference encoded in it. finally, we compare the measured difference with the cross-linguistic syntactic difference in terms of formal syntax to examine whether mbert properly induces syntactic difference via self-supervision. ## mechanism behind cross-lingual transfer the training procedure of zero-shot transfer typically involves two steps: pretraining on a multilingual corpus and fine-tuning on a specific source language. to understand the mechanism behind the zero-shot transfer and why the transfer performance varies across languages, we look into the change they bring to the syntactic difference in mbert. specifically, we first compare the syntactic difference learnt during pretraining with the transfer performance to evaluate the impact of pretraining on the transfer and then examine how fine-tuning on a specific language changes the syntactic difference. ## impact of linguistic diversity to better understand the cross-linguistic syntactic difference learnt by mbert, we employ the structural and functional features in linguistic typology which allows for description of linguistic diversity and analyze to what extent variation in these features affects the syntactic difference in mbert. we further examine whether these features can be exploited to select better source languages and thus benefit cross-lingual transfer. ## conclusion languages vary profoundly at almost every level including lexicon, grammar and meaning. pretrained multilingual encoders learn to encode them in a shared representation space simply via self-supervision, but it is unclear how they address the linguistic variation at different levels. this paper investigates the cross-lingual syntactic ability of mbert. through a measure of distance between distributions over its representations, we demonstrate that mbert encodes universal grammatical relations in a way highly consistent with the cross-linguistic syntactic difference in terms of formal syntax. such cross-linguistic syntactic knowledge plays a decisive role in the zero-shot cross-lingual transfer performance of dependency parsing. this evidence suggests that linguistic knowledge such as typological resources can be incorporated in improvement of cross-lingual transfer and thus help to better accommodate the rich linguistic diversity.