Exploring Linguistic Properties of Monolingual BERT s with Typological Classification among Languages (2023)

Abstract: The impressive achievements of transformers force NLP researchers to delve into how these models represent the underlying structure of natural language. In this paper, we propose a novel standpoint to investigate the above issue: using typological similarities among languages to observe how their respective monolingual models encode structural information. We aim to layer-wise compare transformers for typologically similar languages to observe whether these similarities emerge for particular layers. For this investigation, we propose to use Centered Kernel Alignment to measure similarity among weight matrices. We found that syntactic typological similarity is consistent with the similarity between the weights in the middle layers, which are the pretrained BERT layers to which syntax encoding is generally attributed. Moreover, we observe that a domain adaptation on semantically equivalent texts enhances this similarity among weight matrices.

## introduction natural language processing (nlp) is dominated by powerful but opaque deep neural networks. the rationale behind this trend is that deep architecture training allows rules and structural information about language to emerge directly from sentences in the target language, sacrificing the interpretable and transparent definition of language regularities. some exceptions exist where structural syntactic information is explicitly encoded in multilayer perceptrons @xcite with relevant results on unseen sentences @xcite . yet, pre-trained transformers @xcite are offered as versatile universal sentence/text encoders that contain whatever is needed to solve any downstream task. these models outperform all other models nearly consistently after fine-tuning or domain-adaptation (jin et al., * these authors contributed equally to this work 2022). however, there is no guarantee that when considering languages that share similar structures, the models for those languages will represent those structures in the same way. conversely, decades of studies in nlp have created symbol-empowered architectures where everything is explicitly represented. these architectures implement different levels of linguistic analysis: morphology, syntax, semantics, and pragmatics are a few subdisciplines of linguistics that shaped how symbolic-based nlp has been conceived. in this case, a linguistic model of a language -a set of rules and regularities defining its behavior -directly influences the system processing that language. understanding whether linguistic models emerge in opaque pre-trained transformer architecture is a compelling issue. probing transformers have mainly been used to investigate whether these model classical linguistic properties of languages. probing consists of preparing precise sets of examples -probe tasks -and, eventually, observing how these examples activate transformers. in this way, bert @xcite contextual representations have been tested to assess their ability to model syntactic information and morphology @xcite @xcite @xcite , also comparing different monolingual bert models @xcite . in this study, we take a different standpoint to investigate if traces of linguistic models are encoded in monolingual bert models: using linguisticallymotivated typological similarities among languages @xcite , we aim to layerwise compare transformers for different languages to observe whether these similarities emerge between weight matrices for particular layers. for this investigation, we propose to use centered kernel alignment to measure similarity among weight latin: hostes (s -'enemies') castra (o -'the camp') oppugnant (v -'attack') japanese: maria-san (s -'mary') wa (particle) sarada (o -'salad') wo (particle) tabemasu (v -'eats') turkish: maria (s -'mary') elmayi (o -'the apple') yiyor (v -'eats') italian: although different, natural languages show more or less evident similarities at all levels of analysis -phonetics, morphology, syntax, and semantics -due to the fact that "human language capacity is a species-specific biological property, essentially unique to humans, invariant among human groups, and dissociated from other cognitive systems" @xcite . hence, some properties, called "language universals", are shared by all natural languages: for example, all spoken languages use vowels, all languages have nouns and verbs. some properties, on the other hand, are common among groups of languages, and are called "universal tendencies": for instance, many languages have nasal consonants. @xcite . a classification of languages enables us to organize the shared characteristics of languages. languages are classified in two main ways: typological classification and genealogical classification. language typology is the branch of linguistics that, studying universal tendencies of languages @xcite , maps out "the variation space filled by the languages of the world" and finds "regular patterns and limits of variation" @xcite . these typological patterns can be used to classify all languages at different levels of linguistic analysis. therefore, a typological classification enhances the shared structures among languages. typological classification differs from the genealogical (or genetic) one, which aims to categorize languages originating from a common ancestor @xcite . indeed, languages belonging to a genealogical family can share specific typological properties or not. likewise, languages that appear similar, according to a specific typological category, can be genealogically linked or not. for example, latin, japanese and turkish are final-verb languages, whereas italian is not, despite being directly derived from latin (see examples in in this work, we focus on typological classification because it directly describes languages as a set of rules of composition of words from the syntactic point of view and morphemes in the morphological analysis. in fact, in the following analysis, our aim is to understand whether rules shared across similar languages are encoded into the corresponding models. typological similarities between languages have been previously investigated with probes in two ways: (1) by using syntactic probes in comparing the behavior of different monolingual bert models @xcite ; and, (2) by searching shared syntax representation in multilingual bert (mbert) @xcite @xcite . @xcite suggest that mbert has a behavior similar to monolingual bert since its shared syntax representation can be found in middle layers. however, mbert seems to separate the representations for each language into different subspaces @xcite . hence, mbert is not exploiting typological similarities among languages to build better language models: this model does not exploit similarity among languages to reuse part of the representation it builds across different languages. another perspective to exploit similarity among languages is to compare activation matrices of transformers produced over probing examples. in this case, the underlying assumption is that parallel sentences should have similar activation matrices if languages are similar. to compare these activation matrices, canonical correlation analysis (cca) @xcite , singular vector cca (svcca) @xcite , and projection weighted cca (pwcca) @xcite have been widely used. more recently, centered kernel alignment (cka) @xcite measure has been proposed to quantify representational similarity. @xcite also claims that cca and svcca are less adequate for assessing similarity between representations because they are invariant to linear transformations. using these metrics, @xcite compared activation matrices derived from monolingual models for pairs of translated words: using cka, it measured the similarity between the representation of words in different languages. they noticed that the similarity between the representations of words is higher for more similar languages. similarities between mbert activation matrices have also been used to derive a phylogenetic tree of languages @xcite . in our study, building on typological similarities among languages and on the measures to compare matrices, we aim to compare monolingual berts by directly evaluating layer-wise the similarities among parameters of berts for different languages. to the best of our knowledge, this is the first study using this idea to investigate whether transformers replicate linguistic models without probing tasks. ## method and data if languages are typologically similar for syntactic or for morphological properties, their bert models should have some similar weight matrices in some layers. this is our main intuition. then, in this section, we introduce a method to investigate the above intuition. section 3.1 describes the selected typological classification of languages and how we use it to compute the similarity between languages. section 3.2 briefly introduces the subject of our study, bert. finally, section 3.3 introduces a novel way, bicka, to compare weight matrices. ## results and discussion this section reports the results of the three sets of experiments: full -all pairs of languages (sec. 5.1); focused -only pairs from different clusters (sec. 5.2); domain adapted -analysis is performed after domain adaptation on a parallel corpus (sec. 5.3). ## conclusions understanding whether large language models encode linguistic models is a challenging and important research question. despite their flexibility and their sota performances on multiple nlp tasks, transformer-based models do not explicitly express what they learn about language constructions and, thus, it is hard to control their failures directly. our paper proposes a novel way to observe bert capability to encode linguistic models and achieves two critical contributions: confirming syntactic probing results from another point of view and opening a new way to investigate specific linguistic features in layers. differently from all pre-vious approaches, our methodology is based on directly comparing layer-by-layer weight matrices of berts and evaluating whether typological similar languages have similar matrices in specific layers. from a different standpoint, our experimental results confirm that layers 4, 5, and 6 encode syntactic linguistic information. moreover, they also suggest that the attention's value matrix v and the attention's output layer are more important than other matrices in encoding linguistic models. this latter is an important and novel result. in fact, these findings can be important because they can help interpret the inner workings of these models to go toward the so-called actionable explainability indicating which matrix weights could be changed to obtain a desired behavior. moreover, this methodology could help to build a more informed architecture that takes advantage of the most promising layers in multilingual models. hence, in future work, our findings could be helpful: (1) for defining cross-language training procedures that consider similarities between languages and between models, and (2) for fostering ways to act in specific weight matrices of specific layers of bert to change the undesired behavior of final bert downstream systems. moreover, this methodology could be used on other transformer architectures.