Is G erman secretly a S lavic language? What BERT probing can tell us about language groups (2023)

Abstract: In the light of recent developments in NLP, the problem of understanding and interpreting large language models has gained a lot of urgency. Methods developed to study this area are subject to considerable scrutiny. In this work, we take a closer look at one such method, the structural probe introduced by Hewitt and Manning (2019). We run a series of experiments involving multiple languages, focusing principally on the group of Slavic languages. We show that probing results can be seen as a reflection of linguistic classification, and conclude that multilingual BERT learns facts about languages and their groups.

## introduction transformers @xcite have revolutionised the area of natural language processing. state-of-the-art solutions for virtually all nlp problems -including machine translation, text summarization and generation -are nowadays transformerbased. in recent years models such as bert @xcite and generative pre-trained transformers @xcite have shifted the public view of artificial intelligence. this is also true for slavic languages -for example, the polish language understanding benchmark klej @xcite is dominated by models such as herbert @xcite or polish roberta @xcite . this success has led to a significant interest in studying the interpretability of such models. multiple probing techniques have been developed to assess the extent of linguistic knowledge learned in masked language modelling, especially by models based on bert. those methods typically feature a set of secondary tasks that are learned by a smaller model (the probe), using bert's embeddings as inputs. using probing with multiple tasks, @xcite and @xcite have found a sur-prisingly regular structure encoded in bert's layers. their results are supported by @xcite , where the authors use the task of dependency tree prediction in a method they call the structural probe. they use it to find evidence of syntax learning, especially exhibited by bert's middle layers. going a step further, authors of @xcite apply structural probing to a multilingual version of @xcite , and find a degree of universality in how the syntactic relations are encoded in a single embedding space for multiple languages. on the other hand, the interpretability of probing results is the subject of much discussion. although authors typically use a baseline to quantify what the probe actually learned, those results are still called into question. a parameter-free method of probing is introduced by @xcite , although the results prove to be much more conservative. the problem of whether probes extract knowledge from embeddings or learn new tasks is discussed in depth by @xcite , where they are shown to be able to learn randomly generated control tasks. in @xcite , the authors find a strong argument against interpreting accuracy as a measure of information contained. they show that performance drops when more layers become accessible to the probe, which theoretically should provide it with more information. in this work, we aim to investigate the usability of probing techniques -specifically the structural probe of @xcite -by relating them to real-life ideas developed by theoretical linguists, such as the classification of languages into families and word order types. we take a closer look at the group of slavic languages and the claim that they constitute a separate word order class, as proposed by @xcite . ## methodology our methodology is based on the structural probing method introduced in @xcite and applied to a multilingual setting in @xcite . in this method, the most important data form is the dependency tree, which is a formal way of representing a sentence's syntax. each word in a sentence is represented by a node, with (directed and labeled) edges indicating syntactical relations between words they connect. the authors' idea is to find the structure of dependency trees in bert's embedding space. to recover the structure of a tree, they aim to find a metric in the embedding space that approximates the distance between words in dependency trees (expressed as the number of edges). they search for an appropriate geometry in the family of linear transformations of the embeddings. our loss function (l) thus becomes we can thus see that the real probe here is the matrix b, which is found by minimizing the loss using gradient descent. evaluation we assess the probes based on their ability to predict the structures of unseen dependency trees. for that, we utilise two metrics defined in @xcite . the first metric is spearman's rank correlation coefficient between predicted and gold standard distances (originally named distance spearman, or "dspr."). the coefficient is designed to measure monotonicity of a relation between two variables. here, it is calculated separately for each sentence, averaged across all sentences of a given length, and then over lengths between 5 and 50. the coefficient is expressed as to give a sense of scale here, in hewitt and manning (2019) a non-contextualised baseline reaches a score of 26.8, and a randomly contextualised one -59.8, while the highest value reached on bert is 82.5, indicating over 82% of correctly predicted edges. we collected values of both uuas and dspr. since we found that both metrics are highly correlated (pearson's @xmath0.97) and lead to identical qualitative conclusions, our reporting focuses on the uuas, which is easily interpretable as a percentage of successses. datasets in our work, we selected two groups of languages: train and test languages, listed in ## discussion the results for pre-trained mbert described in the previous section and shown in there is no such interpretation to be found for baseline results. as noted in the previous section, those results are not correlated with non-baseline results for polish. in figure 2 and additionally, we can note that for main results, the scores achieved using the same train and test language differ between languages, ranging from 78.82 (belarusian) to 83.19 (polish). although in ## conclusions using slavic languages as an example, we have shown that the method of structural probing can be used to achieve results that are clearly related to pre-existing linguistic knowledge. in spite of interpretability problems, we conclude that probing can be used to extract linguistic knowledge from transformer models. this can be used both to enhance our knowledge about language models, and about languages themselves. in this case, we show that mbert implicitly learns facts about language groups during its simple pre-training tasks. we also conclude that the implication of @xcite that german has a similar word order heritage as slavic languages can be related to empirical data.