Probing of pretrained multilingual models on the knowledge of discourse (2024)

Abstract: With the raise of large language models (LLMs), different evaluation methods, including probing methods, are gaining more attention. Probing methods are meant to evaluate LLMs on their linguistic abilities. However, most of the studies are focused on morphology and syntax, leaving discourse research out of the scope. At the same time, understanding discourse and pragmatics is crucial to building up the conversational abilities of models. In this paper, we address the problem of probing several models of discourse knowledge in 10 languages. We present an algorithm to automatically adapt existing discourse tasks to other languages based on the Universal Dependencies (UD) annotation. We find that models perform similarly on high- and low-resourced languages. However, the overall low performance of the models’ quality shows that they do not acquire discourse well enough.

## introduction various methods of evaluating language models, including probing methods @xcite , have recently been popular. the probing methods help to shed light on the linguistic abilities of large language models (llms), which could be later used to improve models' qualities @xcite . however, probing studies were mainly conducted at such language levels as morphology and syntax @xcite . while pre-trained language models have shown remarkable performance on various language tasks, there is still much to be explored regarding their ability to capture broader discourse in documents. by discourse, we understand a language level that operates linguistic units bigger than sentences. it involves organizing and connecting ideas to create coherent and cohesive communication. in this paper, we are testing models' ability to capture different aspects of discourse knowledge. discourse probing can involve tasks such as identifying the relations between sentences within a document or the role of one sentence in the document structure, investigating main topics, discovering a suitable ending, and finding out whether one sentence belongs to a particular paragraph or not @xcite . such tasks shed light on the strengths and limitations of pre-trained language models in capturing the nuances of discourse structure. our main contribution is a new suite of probing tasks on multilingual data from ten languages. moreover, our method can be used for other languages with data available in universal dependencies format @xcite . overall, we state our contributions as follows: ## related work probing tasks were first introduced in @xcite and described as simple classification tasks that would reveal if a model contains any linguistic knowledge. probing involves different methods, for instance, probing classifiers @xcite . after training a model on a specific task, we create representations using the model and then train a separate classifier to predict a particular attribute based on these representations. if the classifier demonstrates strong performance, we conclude that the model has acquired relevant information for the attribute. however, upon further examination, it becomes clear that additional complexities are at play. also, probing methods involve prompting: transforming a set of probing tasks into question-answer pairs and directing the model to respond to the questions with a specific prefix @xcite . this approach essentially serves as a probe that is independent of the model. by using prompting instead of a diagnostic probe, researchers can circumvent the challenge of distinguishing between the content of the representations and what the probe learns. after all, one of the most developing fields in probing llms is task relevance which is aimed at investigating whether the information encoded in sentence representations, as discovered through a probe, is used by the model to perform its task. task relevance is also our method of research, which will be discussed later. most probing studies focus on evaluating semantic knowledge, which focuses on the meaning of individual words and sentences. the probing methodology combining various annotated data is commonly used as the benchmark for language model comparison and evaluation of their generalizing ability @xcite . on the other hand, probing of discourse examines how linguistic units are organized and connected to form coherent texts, a crucial ability to generate long sequences. however, only some works investigate the ability of llms to understand discourse. @xcite shows that bert produces pragmatically incorrect outputs because it does not consider an extended context. among other works, @xcite evaluate models on discourse relations expressed with conjunctions. @xcite propose a benchmark for model evaluation on different discourse tasks such as prediction of implicit discourse relations based on the penn discourse treebank annotation @xcite , discourse coherence, and others. ## discussion the influence of the discourse structure in english the so-called 'complicated simple sentences' @xcite in bulgarian generate heavy complementation, and that is the main difference between bulgarian and english rhetorical structure. it can be that the model borrows discourse patterns from the language that prevails in the training sample. thus, presumably, the fewer languages in the model and the greater the presence of english, the greater the accuracy in those languages whose discursive patterns are similar to patterns in english. the results obtained for catalan, which is also structurally significantly different from english, display the same trend and can be explained by rightbranching (right-dislocation constructions), which is not often found in english. mgpt's sparse attention mechanism due to mgpt's performance for french and russian we can hardly consider that the sparse attention mechanism applied for mgpt helps to cope best with long sequences found in russian, rather it turns out to be the best in the case when the main topic of the utterance is concentrated at the beginning and end of the text (as in french). at the same time, for russian kaplan @xcite establishes a structure characterized by situationality, instability of discourse patterns and a constant change of focus of text, which, although in some sense similar to the ornate rhetorical structure in french (both are nonlinear with respect to discourse in english), differs in the lack of integrity according to kaplan. it can be assumed that this difference is the reason for the strong decrease in the accuracy of the mgpt for russian compared to french. models performing similarly with languages belonging to the same group the hypothesis that the mod-els act equally (in relation to each other) for languages belonging to the same language group and therefore having common discourse patterns is confirmed by the example of french and latin. at the same time, this is still a hypothesis, since such a distribution seems to be universal in most cases and has also been recorded for most languages of the slavic group. this assumption is contradicted by the distribution of model accuracy obtained for serbian, but in this case it seems appropriate to refer to the lack of resources of this language. advantages of dynamic masking procedure in the case of turkish, we were talking about shared arguments that occur when two distinct discourse connectives use the same text span as their argument. this can create ambiguity or confusion for the reader or listener, as it may not be immediately clear which connective governs the argument. properly contained arguments occur when a larger text span that is the argument of one connective contains a smaller text span that is the argument of another connective. for xlm-roberta, the complexity of text may be potentially overcome via dynamic masking, as in this case the number of potentially different masked versions of each sentence is not bounded like in mbert, therefore the probability of understanding complicated structures gets bigger. at the same time, we can see that dynamic masking procedure benefits only in cases where the complicated structure of the text does not change drastically. for instance, in so task this change could lead to a deterioration in the quality of the model's performance in this case, since the so task assumes that for incorrect examples all sentences in a sequence are being shuffled. accordingly, in this case, masking the fixed part of the input can serve as an advantage of mbert. how context and focus sentence position affects models' performance in tasks in which the highest accuracy of the models' performance was recorded, the focus sentence for prediction is fixed (always the last, only the size of the sequence varied). nevertheless, context definitely affects the model's performance on the task. for example, models perform worse on a task in which it is required to determine the correctness of the order of sentences within a binary sequence (0.61) than on a task containing multiple sequences (0.77). also, quite unexpected and contrary to hypotheses results were obtained for the task sentence position. in the original paper, the bert-large accuracy for sp was 0.538, while in our case we got an 0.8 accuracy. such a difference in the results may indicate the importance of the first position in the sequence, the weight of which in the context of the multi-head attention method is the largest. ## conclusion our work is devoted to the study of the degree of discourse acquisition by various multilingual models. despite the fact that many tasks and hypotheses were built on the materials of their predecessors, our research differs from them in that it involves several languages in discourse probing at once and combines completely different tasks that ultimately somehow test the understanding of the model of the whole text. also, some of our results do not correspond to the conclusions of other researchers which analyzed english and other few languages (chinese in most cases) and add new information about the understanding of the language by individual models. moreover, we have come to a conclusion that models, on average, perform equally in low-resource and conventional (popular) languages with binary-classification tasks. this result may indicate the presence of certain trends associated with the assimilation of the document structure by models, which apply to all idiolects. we also identified some characteristics of tasks and training samples that affect the performance of the model, such as the size of the sequence, the number of sentences involved in shuffle, the focus of prediction (the last sentence is often easier to predict than the first) -and this factor is stronger than the significance of the size of the context. the more randomness there is in choosing proposals that will change the position in the document, the better the performance of some models, for example, xlm-roberta, since its main principle is masking an unfixed set of tokens. consequently, we have identified certain aspects of tasks that models generally do worse with, such as predicting the connective marker when there is a limited amount of resources, as well as those factors of individual model's architecture that worsen the results. we also compared the results obtained with the accuracy of the predictions of monolingual models and did not reveal a significant deterioration in the quality of transformers.