R u S ent E val: Linguistic Source, Encoder Force! (2021)

Abstract: The success of pre-trained transformer language models has brought a great deal of interest on how these models work, and what they learn about language. However, prior research in the field is mainly devoted to English, and little is known regarding other languages. To this end, we introduce RuSentEval, an enhanced set of 14 probing tasks for Russian, including ones that have not been explored yet. We apply a combination of complementary probing methods to explore the distribution of various linguistic properties in five multilingual transformers for two typologically contrasting languages – Russian and English. Our results provide intriguing findings that contradict the common understanding of how linguistic knowledge is represented, and demonstrate that some properties are learned in a similar manner despite the language differences.

## introduction transformer language models @xcite have achieved state-of-the-art results on a wide range of nlp tasks in multiple languages, demonstrated strong performance in zero-shot cross-lingual transfer @xcite , and even surpassed human solvers in nlu benchmarks such as superglue @xcite . the success has stimulated research in how these models work, and what they acquire about language. the majority of the introspection techniques are based on the concept of probing tasks @xcite @xcite which allow analyzing what linguistic properties are encoded in the intermediate representations. a rich variety of tasks has been introduced so far, ranging from token-level and sub-sentence probing @xcite to sentence-level probing @xcite . a prominent method to explore the inner workings of the models involves training a lightweight classifier to solve a probing task over features produced by them, and assess their knowledge by the classifier's performance. recently, the methods have been greatly extended to latent subclass learning @xcite , correlation similarity measures @xcite , information-theoretic probing @xcite , investigation of individual neurons @xcite , and many more. despite growing interest in the field, english remains the focal point of prior research @xcite leaving other languages understudied. to this end, several monolingual and cross-lingual probing suites have been assembled (see section 2), with a few of them following senteval toolkit @xcite . however, most of them directly apply an english-oriented method which is not guaranteed to be universal across languages, or use the universal dependencies (ud) treebanks @xcite which tend to be inconsistent @xcite @xcite . this work proposes rusenteval, a probing suite for evaluation of sentence embeddings for the russian language. we adapted the method for english @xcite to complement the peculiarities of russian. in contrast to closely related datasets @xcite , rusenteval is fully guided by linguistic expertise, relies on annotations obtained with the current state-of-the-art model for russian morphosyntactic analysis @xcite , and includes tasks that have not been explored yet. the contributions are summarized as three-fold. first, we present an enhanced set of 14 probing tasks for russian, organized by the type of linguistic properties. second, we carry out a series of probing experiments on two typologically different languages: english, which is an analytic germanic language, and russian, which is a fusional slavic one. keeping in mind that the english @xcite and russian datasets are combined from different annotation schemas, we introspect five multilingual transformer-based encoders, including their distilled versions. we apply several probing methods to conduct the analysis from different perspectives, and support the findings with statistical significance. besides, we establish countbased and neural-based baselines for the tasks. finally, rusenteval is publicly available foot_0 , and we hope it will be used for evaluation and interpretation of language models and sentence embeddings for russian. ## related work introspection of pre-trained language models for languages other than english, specifically russian, is usually conducted in the cross-lingual setting. the primary goal of such experiments is to explore how particular linguistic properties are distributed in a given collection of multilingual embedding and language models. linspector (s ¸ahin et al., 2020) is one of the first probing suites that covers a wide range of linguistic phenomena in 24 languages. the benchmark uses unimorph 2.0 @xcite to design the tasks since ud do not provide a sufficient amount of data for the considered languages. despite this, ud treebanks have become the main source for collection of multilingual probing tasks, limiting the scope to morphology and syntax @xcite . other sources for the assembly include multilingual datasets created by means of machine translation, such as xnli @xcite , or datasets that labelled with similar annotation schemes for named entity recognition (ner) and semantic role labelling (srl) tasks @xcite . a few prior works @xcite ) that follow senteval toolkit (conneau and kiela, 2018) directly apply the method designed for english to multiple typologically diverse languages, which raises doubts if such strategy is universal across languages that exhibit unique peculiarities, e.g. a free word order and rich inflectional morphology. this can lead to low quality of the datasets and unreliable experimental results, particularly for russian. consider a few examples for (bshift) task in russian: fedra zatem povesilas', a tesey uznal pravdu. "phaedra later hanged herself, and theseus unraveled the truth." @xcite , and shestoe -zanimatsya nado svoim obrazovaniem "the sixth point is that you to take care need of your education" @xcite . the sentences are labelled as positive, meaning that they exhibit incorrect word order. while the word order changes can lead to the syntax perturbations for english, both sentences are still acceptable in terms of syntax for russian. moreover, the dataset sizes tend to be inconsistent across languages due to using ud treebanks which makes it difficult to compare the results @xcite . another line of research includes probing machine translation models @xcite over multiple languages, and probing for crosslingual similarity by utilizing paired sentences in mutually intelligible languages @xcite . last but not least, such benchmarks as xglue @xcite and xtreme @xcite ) allow evaluating the current state of language transferring methods. ## probing tasks data the sentences for our probing tasks were extracted from the following publicly available resources: russian wikipedia articles foot_1 and news corpora such as lenta.ru foot_2 and the news segment of taiga corpus @xcite . we used rusenttokenize library foot_3 , a rule-based sentence segmenter for russian, to split texts into sentences. each sentence was tokenized with spacy russian tokenizer foot_4 . the sentences were filtered by the 5-to-25 token range and annotated with the current state-of-the-art model for joint morphosyntactic analysis in russian @xcite . besides, we performed two additional preprocessing steps. (1) we computed the ipm frequency of each sentence using the new frequency vocabulary of russian words @xcite to control the word frequency distribution. the ipm values of each token lemma in the sentence (if present in the vocabulary) were averaged over a total number of token lemmas. the sentences with the ipm frequency of lower than 0.9 were filtered out. this allows discarding the majority of sentences that contain rare words, acronyms, abbreviations, or loanwords. (2) syncretism is peculiar to fusional languages, that is, when a word can belong to multiple part-of-speech tags, or express multiple ambiguous morphosyntactic features @xcite . following @xcite , we removed sentences in the semantic tasks (described below) where the target word has multiple part-of-speech tags. this step allows simplifying the probe interpretation and ensuring a fairer experimental setup in terms of the language comparison. the total number of the annotated sentences after filtering is 3.6 mln, and they are publicly available. each task consists of a 100k-sentence training set and 10k-sentence validation and test sets. there is no sentence overlap across the splits, and all sets are balanced by the number of instances per target class. surface properties tasks test if it is possible to recover information about surface properties from the contextualized representations. (sentlen) is a 6-way classification task aimed to predict a number of tokens given a sentence representation. similar to @xcite , we grouped sentences into 6 equal-width bins by length. the word content (wc) task tests if the information on the original words in a sentence can be inferred from its representation. we selected 1k lemmas from the source corpus vocabulary within the 1.5k-3k rank range when sorted by frequency, and sampled equal numbers of sentences that contain only one of these lemmas. the task is treated as a 1k-way classification that requires knowledge about lexical items and their inflectional paradigms. syntactic properties is a group of tasks that probe the encoder representations for syntactic properties. in the (conjtype) task, the sentences must be classified in terms of the type of connection between complex clauses. the objective of the classifier is to tell whether a sentence involves coordination or subordination. the n-gram shift task (nshift) is analogous to senteval's (bshift) that tests the encoder's sensitivity to incorrect word order. as opposed to english, only specific cases of word inversion in russian lead to syntax perturbation. we, therefore, perturbed n-grams that correspond to a set of predefined morphosyntactic patterns. we used tf-idf method from scikit-learn library @xcite to build an n-gram feature matrix that was further applied for the word order perturbation. for instance, we reversed adjacent words in prepositional phrases, numeral phrases, compound noun phrases, etc. below is an example where the head of the prepositional phrase v schkolu 'to school' is inverted with the dependent noun: segodnya on ne poshel shkolu v. the semantic tasks test if the contextualized representations not only capture the morphosyntactic features but also encode higher-level, structural and syntactic-semantic information (namely, the syntax tree hierarchy and the actant structure of a predicate). note that the boundary between the surface, syntactic and semantic tasks is relatively blurred. ## results how is the linguistic knowledge of two contrasting languages distributed in pre-trained multilingual encoders? this section describes the results for shared english and russian probing tasks, covering surface properties (sentlen, wc), syntax (treedepth, nshift), and semantics (objnumber, subjnumber, and tense). we also report the results for the remaining tasks and the baselines in appendix b. ## discussion three encoders exhibit similar behavior, but two other differ from them in capturing linguistic properties the most striking distinction based upon the results is that m-bart and labse exhibit different behavior for both languages, as opposed to m-bert, xlm-r, and minilm (section 5.1, 5.2). specifically, m-bart generally tends to distribute the surface properties across all the layers, unlike other encoders where the information is specialized by the lower ones. the syntactic properties tend to be localized at the higher layers @xcite which is as well demonstrated in other russian tasks (gapping, impersonalsent). in contrast to other models that capture the semantic properties at the middle-to-higher layers, labse typically displays the knowledge at the lower-to-middle layers. the analysis of the understudied encoders contradicts the common findings on transformer models that syntactic information is stored at the middle layers, while semantic knowledge is most prominent at the higher layers @xcite . therefore, there is still room for exploring transformer-based models that differ by architecture type, or by a set of pre-training objectives. minilm inherits linguistic properties from m-bert teacher models are usually compared with their distilled versions on a range of downstream tasks @xcite , and little is explored on what language properties they bequeath to their students. we find that minilm is likely to mimic the behavior of xlm-r rather than of m-bert, most probably due to using the same tokenization method. the similarity is most demonstrated by the individual neuron analysis (see appendix c) and lincka (see figure 3b ). along with that, the model receives comparative performance under layer-wise supervised probing (see appendix b). surface and syntactic information is learned in a similar manner the probing curves demonstrate that the surface and semantic properties of the two languages are similarly distributed in the encoders. the surface properties are generally captured at the lower layers, and the pattern of the curves is decaying towards the output layer. the syntactic properties are predominantly inferred at the middle or higher layers, and the semantic tasks reveal a number of differences described in section 5. notably, the results obtained under layer-wise supervised probing (section 5.1) are supported by the individual neuron analysis (section 5.2), and the representation-level analysis (section 5.3). ## conclusion this paper introduces rusenteval, an enhanced probing suite of 14 probing tasks that cover various linguistic phenomena of the russian language. we explored five multilingual transformer encoders over the probing tasks on two typologically contrasting languages -russian and english. the experiments are conducted using a combination of complementary probing methods, including layer-wise supervised probing, individual neuronanalysis, neuron-and representation-level similarity measures. particularly, the behavior of the encoders under probing classifiers is reflected in distributions of top neurons with respect to a task, and the similarity is supported by linear centered kernel alignment method. we found that despite the language distinctions, the surface and syntactic properties are learned in a fairly similar manner, and the semantic knowledge is captured differently in a majority of tasks. we believe that the findings make the ongoing studies on cross-lingual transfer even more promising, specifically from english to russian or vice versa. in contrast to prior works on how linguistic knowledge is represented in transformers, the analysis of the understudied models reveals that syntax and semantics can be differently represented across the layers. besides, we found that different encoders often have similar distributions of neurons that contribute most to a probing task, and yet differ under neuron-level similarities. we also observed that distilled models inherit linguistic properties from their teachers, and receive comparative performance on a number of probing tasks. an exciting direction for future work is to investigate the correlation between probing and high-level downstream tasks, in order to identify which linguistic properties anticipate the behavior of a model in action.