Do large language models and humans have similar behaviours in causal inference with script knowledge? (2024)

Abstract: Recently, large pre-trained language models (LLMs) have demonstrated superior language understanding abilities, including zero-shot causal reasoning. However, it is unclear to what extent their capabilities are similar to human ones. We here study the processing of an event B in a script-based story, which causally depends on a previous event A . In our manipulation, event A is stated, negated, or omitted in an earlier section of the text. We first conducted a self-paced reading experiment, which showed that humans exhibit significantly longer reading times when causal conflicts exist ( \neg A \rightarrow B ) than under logical conditions ( A \rightarrow B ). However, reading times remain similar when cause A is not explicitly mentioned, indicating that humans can easily infer event B from their script knowledge. We then tested a variety of LLMs on the same data to check to what extent the models replicate human behavior. Our experiments show that 1) only recent LLMs, like GPT-3 or Vicuna, correlate with human behavior in the \neg A \rightarrow B condition. 2) Despite this correlation, all models still fail to predict that nil \rightarrow B is less surprising than \neg A \rightarrow B , indicating that LLMs still have difficulties integrating script knowledge.

## introduction causal reasoning is fundamental for both human and machine intelligence @xcite and plays an important role in language comprehension @xcite @xcite . large pre-trained language models (llms) such as gpt-3.5 @xcite have demonstrated excellent zero-shot capabilities in causal reasoning tasks and humanlike behaviors @xcite . the capabil-ity of causal reasoning is essential to new prompting techniques like the chain-of-thought prompting @xcite . on the other hand, some early pieces of evidence show that llms lack global planning of different events in stories @xcite . so it is unclear to what extent llms can conduct causal reasoning about events. in turn, humans have been shown to be extremely good at building causal connections in long discourse comprehension @xcite . in doing so, they rely not only on explicit causal links (signaled in the text -see @xcite but also on implicit ones that are inferable based on commonsense knowledge @xcite . in particular, subjects were found to be sensitive to causal conflicts arising from contradictions to earlier text segments or conflicts with subjects' commonsense knowledge @xcite . an example of a causal conflict is presented in figure 1 , part ii, condition ¬@xmath0, where decorating a cake with star-shaped sprinkles is inconsistent with the previously mentioned information that cake decorations are not available. in this paper, we investigate language processing in humans and compare it to a large variety of llms, following the "psycholinguistic assessment of language models paradigm" @xcite . in our analyses, we compare human reading times to llm surprisal estimates. surprisal is the negative log probability of a word in context and has been previously related to human reading times @xcite @xcite as well as neuropsychological effects such as the n400 @xcite , which represent human processing difficulty. we collect a new dataset, causality in script knowledge (csk), how sure are you that anne grabbed some cake decorations? not sure at all vs. very sure (0-7) question about event a: question about event b: how sure are you that anne added star sprinkles to the cake? not sure at all vs. very sure (0-7) ## can llms detect causal conflicts (rq1)? in this section, we measure the ability of different llms to track event contingency. we feed the script stories into the language models and record the lm's surprisal scores on a word-by-word basis. we then test whether the mean surprisal scores for the critical region (event b) differ between conditions. as the script stories corpus is relatively small, we additionally test the models on the trip dataset @xcite to assess their recognition of causal incongruencies on a wider set of materials (see section 4.5). ## conclusions in this paper, we inspect the behaviors of both large language models and humans in zero-shot causal inference. we conducted a self-paced reading experiment on common sense stories to inspect human processing difficulty when reading the stories. reading time results indicate that humans stumble across causally incoherent text segments, exhibiting longer reading times in these cases. on the other hand, they easily integrate script-predictable information, even if the explicit causal component (event a) is missing from the story. when we apply the same study to llms, only the newest llms show similar behavior to humans on encountering casual conflicts. all models fail to replicate human behaviors when the cause is omitted. even models trained with programming code and instructions fail to make use of script knowledge, which indicates that script knowledge may not be represented sufficiently well in the llms tested in this study.