Behavior of Modern Pre-trained Language Models Using the Example of Probing Tasks (2021)

Abstract: Modern transformer-based language models are revolutionizing NLP. However, existing studies into language modelling with BERT have been mostly limited to English-language material and do not pay enough attention to the implicit knowledge of language, such as semantic roles, presupposition and negations, that can be acquired by the model during training. Thus, the aim of this study is to examine behavior of the model BERT in the task of masked language modelling and to provide linguistic interpretation to the unexpected effects and errors produced by the model. For this purpose, we used a new Russian-language dataset based on educational texts for learners of Russian and annotated with the help of the National Corpus of the Russian language. In terms of quality metrics (the proportion of words, semantically related to the target word), the multilingual BERT is recognized as the best model. Generally, each model has distinct strengths in relation to a certain linguistic phenomenon. These observations have meaningful implications for research into applied linguistics and pedagogy, contribute to dialogue system development, automatic exercise making, text generation and potentially could improve the quality of existing linguistic technologies

## introduction as is well-known, 2018 saw a breakthrough in natural language processing (nlp) with the advent of several novel pre-trained language models, including bert @xcite . these models are capable of fine-tuning, that is, additional training on a smaller dataset for a specific task. such models develop a certain degree of understanding natural language and therefore require further studies @xcite . to this end, the socalled probing tasks are commonly used, such as testing the model's ability to identify semantically coherent sentences in running text. a considerable amount of recent research @xcite @xcite was devoted to the general language modelling problem and also to masked language modelling. other scholars @xcite @xcite ) explored bert specifically. however, there is a lack of studies concerning bert's behavior on russian language material. in addition, existing papers discuss the problem without taking into consideration systematic linguistic features. this paper aims to explore the behavior of the pre-trained bert language model using the example of the diagnostic task of masked language modeling for the russian language and give a linguistic interpretation of the cases when the model shows unsatisfactory results. this study will focus on new language data with experiment design based on linguistic theories 1 . the problem will be discussed in terms of the language modelling concept, cognitive science, theory of language, in particular the language frames theory, semantic roles concept, and also negations processing depending on a context. model: probability distribution over word sequences. such models can be the basis for solving a large number of nlp tasks, with the help of fine-tuning the general language models. the pre-trained language model weights already encode a lot of information about natural language. specifically, a pre-trained model provides features of semantics, syntax, and "verbal knowledge" which may be transferred from general to specific tasks @xcite . in the past several years, the bert model and its modifications have become widely used in the natural language processing community @xcite . the model can be applied to all general types of tasks: single sentence prediction, sentence pair classification, question answering and sentence tagging. as bert is inherently an encoder for language information, it often plays the role of a text feature extractor @xcite . some authors @xcite focus on different types of tasks regarding language models, namely probing tasks. these authors suggest options for surface information (e.g., recovering a word from its embedding), syntactic information (e.g., the model sensitivity to word order change), semantic information (e.g., identification of main-clause verb tense). their basic application is testing the pre-training effect: how pre-trained models encode various language phenomena. currently, there are lists of linguistic capabilities available that allow to explore the of behavior of language models @xcite . thereby, the main benefit of probing tasks is the analysis of linguistic knowledge that can be extracted from sentence representations @xcite . other authors @xcite suggested a new probing task for language models that is nowadays known as masked language modelling. the key difference is that the model masks some random words from a sequence and then predicts them again based on the contextual information, both left and right. ultimately, it was suggested to train with masked language modeling, and then fine-tune for specific tasks @xcite . differences in human and machine understanding of language were investigated in another paper @xcite , which inspired our experimental setup. the author uses psycholinguistic tests to find out the sensitivity of the model to such phenomena as hypernymy, semantic roles and negation. the tests revealed weaknesses in the language model and proved that computational models have great potential for natural language understanding. as was pointed out in the previous section, despite this scholarly interest for bert, there is a lack of studies with a strong linguistic basis. additionally, previous studies have tended to focus primarily on english language material. the present study is based on a russian-language dataset and uses theories from cognitive linguistics and the theory of language. ## linguistic basis from the linguistic perspective, the work of language models is based on the concept of semantic roles @xcite . speaking about the subject of action, it can have both an agentive (or active) position, and a non-agentive (or inactive) position (uskova, 2012). ch. fillmore also developed the theory of presupposition, which is important for this study. it is the preliminary knowledge that is responsible for the semantic correctness of the utterance. for example, the sentence snow is expected in moscow on february 30th, is incorrect due to the fact that it includes a false presupposition on february 30th. meanwhile v.z. demyankov identifies several types of presuppositions. in our experiment, we will use pragmatic and logical types. pragmatic presupposition is the conditions and contexts that must be present in order for the speaker's intention to be realized correctly. the semantic presupposition characterizes the relationship between a sentence and the proposition it expresses. another linguistic aspect that requires our consideration is negation. it is considered a semantic primitive integrated into the grammatical and lexical systems of all languages of the world @xcite . we will explore predicate negation. speaking about studies of language models, it is necessary to mention the type of lexical paradigmatic relations, such as hypohyperonymic. this type of relations reflects the direction of human thinking to systematize lexical units and non-linguistic structures behind them and bring them to a hierarchical form @xcite . this fact can be used to evaluate the performance of the language model in such a way that the resulting metric value is interpretable. so, after considering some linguistic theories and concepts, we came to the term "behavior of the language model". according to the philosophical encyclopedia @xcite , behavior is a way of reacting to any influence. in this paper, the behavior of a language model is understood as text data that the model provides as output under certain conditions of its use, namely, when testing or applying to a problem. ## datasets and methods the final dataset consists of three parts, each containing 50 sentences, which were collected using a linguistic observation method. each part was designed to test and evaluate a certain aspect of the functioning of the language model, namely: common sense inference, the interpretation of semantic roles, and processing negatives depending on the context. in the case of semantic roles, complex or simple common sentences were selected from educational texts, the predicate was replaced with a token mask, and the linguistic unit, which had the role of an agent, took on the role of a patient (or experiencer). to check common sense inference, we selected two consecutive sentences from the educational texts where one word in the second sentence was manually replaced with a mask. then, using semantic search in the rnc, we selected a word of a related topic and a word of a more general topic with respect to the expected topic. next, the subcorpus with negations was compiled as follows: in the rnc, using semantic search, we selected sentences with an adjective at the end, and masked the adjective. finally, the sentence was copied and transformed into a negative one, and the adjective was replaced by its antonym. the dataset was examined using a jupyter notebook in the python programming language with the help of the libraries tensorflow, pytorch and transformers. to perform a comparison, the bert drmatters 3 and bert deeppavlov foot_1 models for the russian language, and the multilingual bert foot_2 were selected. ## discussion crucially, the models are unable to understand most of the background information (mostly pragmatic) out of context. additionaly, the models actually handle the subject better in a more typical, i.e., agentive, position. there are mistakes in processing negatives: all models often choose options for negative contexts that are relevant for affirmative contexts. comparing the results of the linguistic analysis of predictions, it is impossible to conclude unambiguously which of the models works better with russian-language material. generally speaking, the multilingual model performs better in working with background knowledge and implicit information hidden in the models and for tasks in which strong restrictions are not imposed on the predicted words. the russian-language model is more suitable for modeling logical relationships and analyzing the subject in its typical position. however, both models do not seem suitable for the problem of negation processing since they consider only one kind of context. ## conclusion and future work this paper has investigated the pre-trained bert language models with probing tasks. as far as we are aware, this is the first time that bert was explored by professional linguists based on russian-language material. as a result of this research, we have listed the frequent mistakes made by the bert model in the masked language modelling task and conducted their analysis. in contrast to previous studies @xcite , where the focus was on psycholinguistic diagnostics and comparison between machine and human performance, we made our study more linguistically grounded by adding knowledge from theory of language. while conducting the research, we created a unique dataset that represents several cognitive phenomena with special annotations. the dataset is made available to the community. in the future, we plan to expand the dataset and annotations to cover more cognitive and linguistic aspects.