Cross-linguistic Comparison of Linguistic Feature Encoding in BERT Models for Typologically Different Languages (2022)

Abstract: Though recently there have been an increased interest in how pre-trained language models encode different linguistic features, there is still a lack of systematic comparison between languages with different morphology and syntax. In this paper, using BERT as an example of a pre-trained model, we compare how three typologically different languages (English, Korean, and Russian) encode morphology and syntax features across different layers. In particular, we contrast languages which differ in a particular aspect, such as flexibility of word order, head directionality, morphological type, presence of grammatical gender, and morphological richness, across four different tasks.

## introduction transformers @xcite and especially pre-trained language models based on them, such as bert @xcite , had a revolutionary impact on the field of natural language processing (nlp), allowing to achieve new heights in classification, retrieval, text understanding, and generation tasks. however, though major progress was made in adapting transformers to different downstream tasks, they largely remain black-box models, especially from the linguistic point of view. in particular, though we know that they roughly follow the same pipeline as human-made natural language processing (nlp) systems when encoding the features (the lower layers of a transformer encode part-of-speech information, the middle layers perform syntax parsing, while the top layers enable such tasks as coreference resolution) @xcite , it is still unclear if there is a systematic relation between the type of morphological, syntactical and discourse features encoded and particular layers of transformer-based models. more importantly, though there have been some attempts to examine this for languages other than english (see, for example, a study by @xcite for dutch), we still do not know if there is a consistency between models for different languages in encoding such features, especially if the languages in question are typologically different. thus it is important to analyse how transformers encode different linguistic features for dissimilar languages, as it would help us to better understand how they work and ultimately allow to improve their performance in such tasks as translation or multi-lingual information retrieval and summarisation. in this study, we compare how transformers encode particular linguistic features for the following languages: english, russian, and korean. the choice of languages are motivated linguistically: english, russian and korean are morphologically very distant languages (analytical, fusional and agglutinative respectively) which are also different in term of syntax such as word order. to examine transformers encodings more systematically, we conduct a series of pairwise comparisons of languages which contrast in a particular linguistic feature, similarly to how it is performed in theoretical linguistics. in particular, using targeted manipulation of inputs to produce a binary correctness classification or a masked token prediction task, we examine the following research questions: in the following sections we describe our experiments for these tasks and present our findings. in this section we compare the ability of a bertbased classification model to detect the corruption of word order in languages with fixed vs free word order. we hypothesise that as languages with free word order allow for more permutations in terms of token positions in the sentences, it would be more difficult for the model to detect word order corruption. ## long-distance agreement in morphologically rich and poor languages in this task we test the ability of attention-based models to encode long-distance agreement, in particular the agreement in number (plural vs singular). we choose this task as it tests the model's ability to encode hierarchical syntactic structure, for example to determine the number of a verb based on the form of the noun related to it in the syntactic tree, rather than on the form of the closest noun. ## conclusions and future work in this study we used linguistic probes and masked language models to explore several aspects of morphology and syntax representation in transformerbased models. in particular, we examined the ability of the model and its particular layers to encode the correct word order in languages with contrasting morphology and syntax, their ability to capture hierarchical structure represented by long-distance agreement, and the degree of bias encoding in languages with and without gender morphology. in doing so, we once again showed that the number of layers in the model roughly corresponds to the complexity of the encoded features, but also discovered that languages differ in layers where such encoding happens. one of the most important takeaways of analysing the pre-trained models' performance layer by layer is that the best accuracy is not necessarily achieved at the last layer, which leads us to question the practice of using the complete model for all downstream tasks. therefore, a potential extension of this work would be to explore the performance of such tasks as classification or generation when using only some layers of the pre-trained model. another observation is that in general the russian model needed more layers to achieve its optimal performance, while both korean and english ones showed their best results at much earlier layers. it can be explained by more complicated morphology and syntax of russian language which potentially can require more layers to be properly encoded. thus it leads to a question whether adding more layers to pre-trained models for inflected languages with rich morphology and syntax (for example, spanish or german) can help to improve performance of downstream tasks. that said, one of the limitation of the present study is that we focused only on one type of transformer-based models and compared only two languages at a time; to ensure the general applicability of our experiments, they should be expanded to more languages with similar typological characteristics to those analysed above, and to attention-based models with different training approaches.