Morph Call: Probing Morphosyntactic Content of Multilingual Transformers (2021)

Abstract: The outstanding performance of transformer-based language models on a great variety of NLP and NLU tasks has stimulated interest in exploration of their inner workings. Recent research has been primarily focused on higher-level and complex linguistic phenomena such as syntax, semantics, world knowledge and common-sense. The majority of the studies is anglocentric, and little remains known regarding other languages, specifically their morphosyntactic properties. To this end, our work presents Morph Call, a suite of 46 probing tasks for four Indo-European languages of different morphology: Russian, French, English and German. We propose a new type of probing tasks based on detection of guided sentence perturbations. We use a combination of neuron-, layer- and representation-level introspection techniques to analyze the morphosyntactic content of four multilingual transformers, including their understudied distilled versions. Besides, we examine how fine-tuning on POS-tagging task affects the probing performance.

## introduction in the last few years, transformer language models @xcite have accelerated the growth in the field of nlp. the models have established new state-of-the-art results in multiple languages and even demonstrated superiority in nlu benchmarks compared to human solvers @xcite @xcite . their distilled versions, or so-called student models, have shown competitive performance on many nlp tasks while having fewer parameters @xcite . however, many questions remain on how these models work and what they know about language. the previous research focuses on what knowledge has been learned during and after pre-training phases @xcite , and how it is affected by fine-tuning @xcite @xcite . besides, a wide variety of language phenomena has been investigated including syntax @xcite , world knowledge @xcite , reasoning @xcite , common sense understanding @xcite , and semantics @xcite . most of these studies involve probing which measures how well linguistic knowledge can be inferred from the intermediate representations of the model. the methods range from individual neuron analysis @xcite , examination of attention mechanisms @xcite , correlationbased similarity measures @xcite , to probing tasks accompanied by linguistic supervision @xcite . despite growing interest in interpreting the models, morphology has remained understudied, specifically for languages other than english. the majority of prior works on this subject are devoted to the introspection of machine translation models, word-level embedding models, or transformers, fine-tuned for pos-tagging (see section 2). to this end, we introduce morph call, a probing suite for the exploration of morphosyntactic content in transformer language models. the contributions of this paper are summarized as follows. first, we propose 46 probing tasks in four indo-european languages of different morphology: russian, french, english, and german. inspired by techniques for model acceptability judgments @xcite and adversarial training @xcite , we present a new type of probing tasks based on the detection of guided sentence perturbations. since the latter is automatically generated, the tasks can be adapted to other languages. second, we use complementary probing methods to analyze four multilingual transformer encoders, including their distilled versions. we examine how fine-tuning for pos-tagging affects the probing performance and establish count-based and non-contextualized baselines for the tasks. finally, we publicly release the tasks and code 1 , hoping to fill the gaps in the less studied aspect of transformers. ## related work a large body of recent research is devoted to analyzing and interpreting the linguistic capacities of pre-trained contextualized encoders. the most common approach is to train a simple classifier for solving a probing task over the word-or sentencelevel features produced by the models @xcite . the classifier's performance is used as a proxy to assess the model knowledge about a particular linguistic property. however, lately, the method has been critiqued: is the property truly learned by the model, or does the model encode the property for the classifier to easily extract it given the supervision? besides, a new set of additional classifier parameters can make it challenging to interpret the results @xcite @xcite . nevertheless, the probing classifiers are widely applied in the field of model interpretation, including morphology. one of the first works on morphological content is carried out on machine translation models where the classifier is learned to predict pos-tags in multiple languages @xcite . the latest studies involving pos properties in transformers show that they are predominantly captured at the lower layers @xcite @xcite , and can be evenly distributed across all layers @xcite . amnesic probing explores how removing information at a particular layer affects the probe performance at the final layer @xcite . this allows measuring the layer importance with respect to a linguistic 1 https://github.com/ morphology-probing/morph-call property. the results claim that removing pos information may affect the performance more at the higher layers as compared to the lower ones. another line of research is devoted to various linguistic phenomena at the juxtaposition of morphology, syntax, and semantics. lstm-based models and transformers are probed to capture subjectverb agreement in different languages @xcite @xcite . recently, the agreement has been at the core of inflectional perturbations for adversarial training @xcite , and linguistic acceptability judgments along with morphological, syntactic, and semantic violations @xcite . our work is closely related to @xcite who explore morphological properties and subjectverb agreement in the hidden representations and self-attention heads of transformer models. however, there are several differences. first, we investigate the knowledge in multilingual transformers and their distilled versions instead of monolingual ones. second, we carry out the experiments on an extended set of tasks, such as detecting syntactic and inflectional perturbations (see section 3.2). third, we apply several probing methods to analyze from different perspectives. finally, we study the impact of fine-tuning for pos-tagging on the probe performance. despite the similarities and differences, we find the studies complementary. finally, such benchmarks as linspector @xcite and xtreme @xcite provide means for evaluation of multilingual embedding models and cross-lingual transferring methods with regards to multiple linguistic properties, specifically morphology. ## conclusion this paper proposes morph call, a suite of 46 probing tasks in four indo-european languages that differ significantly in their richness of morphology: russian, french, english, and german. the suite includes a new type of probing task based on the detection of syntactic and inflectional sentence perturbations. we apply a combination of three introspection methods based on neuron-, layer-and representation-level analysis to probe five multilingual transformer models, including their less explored distilled versions. the analysis of transformers' understudied aspect contradicts the common findings on how morphology is represented in the models. we find that the knowledge for english is predominantly distributed across all layers of the models in contrast to more morphologically rich languages (german, russian, french), which house the properties at the middle-to-higher lay-ers. the models demonstrate their sensitivity to the perturbations, and xlm-r tends to be the most robust among the others. we observe that distilled models inherit their teachers' knowledge, showing a comparative performance and exhibiting similar property distribution on several probing tasks. another finding is that fine-tuning for pos-tagging can affect the model knowledge in various manners, ranging from improving and decreasing the probing classifier performance to changing the information's localization. we believe there is still room for exploring the models' morphosyntactic content and the effect of fine-tuning, specifically across a more diverse set of languages and types of model architectures.