Investigating the Effect of Discourse Connectives on Transformer Surprisal: Language Models Understand Connectives, E ven So They Are Surprised (2023)

Abstract: As neural language models (NLMs) based on Transformers are becoming increasingly dominant in natural language processing, several studies have proposed analyzing the semantic and pragmatic abilities of such models. In our study, we aimed at investigating the effect of discourse connectives on NLMs with regard to Transformer Surprisal scores by focusing on the English stimuli of an experimental dataset, in which the expectations about an event in a discourse fragment could be reversed by a concessive or a contrastive connective. By comparing the Surprisal scores of several NLMs, we found that bigger NLMs show patterns similar to humans’ behavioral data when a concessive connective is used, while connective-related effects tend to disappear with a contrastive one. We have additionally validated our findings with GPT-Neo using an extended dataset, and results mostly show a consistent pattern.

## introduction psychologists and cognitive scientists have claimed that understanding a discourse involves constructing a situation model; that is, a dynamic mental representation of the state of affairs denoted by the text @xcite . extensive evidence has shown that humans use a general knowledge of events and their connections to anticipate upcoming input in the process of language comprehension @xcite . in this sense, discourse connectives might play an important role in updating human situation models and in modulating our expectations about "what is coming next", because concessive (such as even so) and contrastive connectives (such as however) signal to the comprehender that the upcoming proposition is going to contradict what was previously said, or negate the previous expectations @xcite . experimental studies have shown that such connectives have a facilitating effect on human sentence processing (asr and demberg, 2020), especially when humans are processing incoherent words and scenarios @xcite . consider the following example taken from @xcite , in which the concessive connective even so causes an effect of expectation reversal: given the scenario described in the first sentence (failing a test), the underlined verb in the second sentence is surprising and unexpected. however, after including a connective reversing the readers' expectations, examples like (1) are considered as coherent by human speakers. the recent literature on natural language processing (nlp) has shown an increasing interest in the use of surprisal scores @xcite computed by neural language models (nlms) to account for sentence processing phenomena @xcite @xcite , including facilitation @xcite @xcite and interference effects @xcite in online sentence processing. however, to the best of our knowledge, no studies have attempted to model the facilitation effects of concessive and contrastive connectives at different levels of discourse coherence thus far. in our study, we aim to fill this research gap by investigating the effect of discourse connectives on nlms' surprisal scores. first, we focus on the concessive connective even so, and on the contrastive connective however as an alternative. based on the whole discourse, we first computed the surprisal scores for target words using nlms to observe the extent to which they were affected by the coherence of the stories. we found that nlms, and particularly the larger models, show patterns that are quite similar to human behavioral data. moreover, we noticed that the connective-related effects do not show up with contrastive connective, suggesting that the nlms are sensitive to the difference between connective types: the semantics of concessive connectives entails a reversal of previous expectations about an upcoming event that is not conveyed by contrastive connectives. using our biggest model, gpt-neo, we ran additional analysis adding more connectives of the two types and computing the surprisal scores either in an inter-sentential and an intra-sentential setting. the results were mostly consistent with our first experiment, corroborating the previous findings. ## results for the original even so data ( in gpt-2 and gpt-neo, and again, only gpt-neo showed sensitivity to length. the pairwise comparisons examining the effects of even so at each level of coherence ( fewer significant effects were found after replacing even so with however ( comparing the outcome with the study by @xcite , one can observe that the model scores tend to align with human typicality judgements (cf. our results suggest that nlms are sensitive to the expectation reversal determined by connectives. besides the human-like pattern in the distribution of the surprisal scores for the even so dataset, it is also noticeable that replacing the connective with however makes the connective-related effects disappear. this is coherent with the intuition and the claims made in formal semantics literature, for which however simply introduces a semantic opposition, while even so additionally presupposes an expectation being denied @xcite . ## conclusion in our paper, we proposed an analysis of the surprisal scores of nlms on the target verbs of a psycholinguistic dataset where the items differed by the coherence of the discourse, and by the inclusion of a connective reversing the expectations on the verb. we found that our nlms show patterns that are quite similar to human behavioral data, in particular the biggest model, gpt-neo. more interestingly, in all models the effects related to the connective disappear when a contrastive connective is used to replace the concessive one. this suggests that the concessive connective leads to an expectation reversal for the upcoming verb, while the contrastive one does not, coherently with previous descriptions of connectives from the formal semantics literature @xcite . given that the psycholinguistic dataset we used in our modeling is relatively small for nlp settings and limited to two connectives, we additionally constructed an extended datasets to validate our findings. we extended our investigation in two ways: expanding the dataset by varying the setting in which the connective was found (inter-or intrasentential) and connectives themselves, including six more contrastive/concessive connectives. our results indicate that the findings even so are mostly consitent, as they generalize across settings and concessive connectives. we acknowledge that there are some significant limitations in our current investigations. first, we have human data only for even so in the intersentential setting, for which we could establish an interpretation baseline. there is no behavioral or neural data for all the other connectives. this implies that we interpreted some of our findings based on intuitions about discourse connectives, assuming that the human behavioral pattern will be similar for other concessive types. we recognize that this is an important limitation to the cognitive plausibility of our evaluation, and for future research we plan to collect more human judgements for discourse connectives, possibly including also languages other english. second, our analysis was mainly focused on comparing surprisal scores to human behavioral patterns (or, by extension, to the patterns found in the original experiment with concessive connectives), but we did not apply any advanced interpretability method to identify which specific input tokens influence the predictions for the target verb. more direct evidence for the causal role of discourse connectives in reversing the predictions could be obtained, for example, by analyzing the changes of the probability rank of the verb in the target position; or by applying contrastive explanations to sentences differing only for the presence of connectives @xcite .