What do Large Language Models Learn about Scripts? (2022)

Abstract: Script Knowledge (Schank and Abelson, 1975) has long been recognized as crucial for language understanding as it can help in filling in unstated information in a narrative. However, such knowledge is expensive to produce manually and difficult to induce from text due to reporting bias (Gordon and Van Durme, 2013). In this work, we are interested in the scientific question of whether explicit script knowledge is present and accessible through pre-trained generative language models (LMs). To this end, we introduce the task of generating full event sequence descriptions (ESDs) given a scenario as a natural language prompt. Through zero-shot probing, we find that generative LMs produce poor ESDs with mostly omitted, irrelevant, repeated or misordered events. To address this, we propose a pipeline-based script induction framework (SIF) which can generate good quality ESDs for unseen scenarios (e.g., bake a cake). SIF is a two-staged framework that fine-tunes LM on a small set of ESD examples in the first stage. In the second stage, ESD generated for an unseen scenario is post-processed using RoBERTa-based models to filter irrelevant events, remove repetitions, and reorder the temporally misordered events. Through automatic and manual evaluations, we demonstrate that SIF yields substantial improvements (1-3 BLEU points) over a fine-tuned LM. However, manual analysis shows that there is great room for improvement, offering a new research direction for inducing script knowledge.

## introduction scripts are structured commonsense knowledge in the form of event sequences that characterize commonplace scenarios, such as, eating at a restaurant @xcite . scripts are fundamental pieces of commonsense knowledge that humans share and assume to be tacitly understood 2 ) to generate completely ordered esds for evaluating extent of script knowledge accessible through lms. by each other. when someone says "i went to a restaurant for lunch", our script knowledge allows us to infer that a waiter would have taken the order, the speaker would have eaten the lunch, payed for it, and tipped the waiter, even if these events are not explicitly mentioned. knowledge of scripts, whether implicit or explicit, has been recognized as important for language understanding tasks @xcite . earlier efforts to automatically induce scripts from text on a large scale include @xcite who treat the problem of script induction as one of learning narrative chains using textual co-occurrence statistics. however, reporting bias @xcite remains an obstacle for script induction as many events are not mentioned explicitly in text, relying on the reader's ability to infer missing script-related events. moreover, manual creation of such knowledge resources is challenging due to the wide coverage and complexity of relevant scenario knowledge. although crowdsourced efforts @xcite @xcite @xcite address these issues and acquire script knowledge in the form of esds, the collected datasets are small, domain-specific, and crowdsourcing is not scalable. with the success of pre-trained language mod-els (henceforth, plms) @xcite @xcite in various natural language understanding tasks, we are interested in investigating the extent and accessibility of explicit script knowledge present in plms. in this work, unlike cloze-based script evaluations @xcite which lms are uniquely optimized for @xcite , we evaluate plms on the ability to fully generate event sequence descriptions (esds) @xcite in free-form natural language (figure 1 ). this is a challenging task as scripts are complex structures with varied granularity of describing a scenario (e.g., starting from going to grocery store to buy ingredients or starting with finding a recipe for baking a cake scenario), and the requirement to produce all the scenariorelevant events in the correct temporal order. to this end, we first probe lms via carefully crafted prompts to analyze the quality of esds generated in a zero-shot setting ( §3) and find that the generated esds are of poor quality with many scenario-irrelevant, repeated, temporally misordered, and missing events. to address this we propose a, lm-agnostic, pipeline-based script induction framework ( §4), sif, which can generate good quality esds for novel scenarios that lm has not seen during the training phase of the framework. sif is a two-staged framework with finetuning lm on a small set of esds as the first stage followed by a three-stepped post-processing stage which corrects the esds generated from a finetuned lm for irrelevant, repeated, and temporally misordered events. this work makes the following contributions: ## related work narrative chain induction there has been a growing body of research into statistical script learning systems which can automatically infer implicit events from text. seminal work by @xcite describe a number of simple event co-occurrence based systems that infer (verb, dependency) pairs (known as narrative events) with partial-ordering related to one or multiple participants @xcite in discourse (known as narrative chains). as statistical co-occurrences cannot capture long-range dependencies between events, pichotta and mooney (2016a) represent events using lstm leading to improved narrative cloze task performance. however, much of the information about events is usually left implicit in text. moreover, narrative events are highly abstracted @xcite and cloze task is insufficient to evaluate script knowledge @xcite . therefore efforts have been made to acquire crowdsourced esds @xcite @xcite @xcite and to learn similar events in a scenario using unsupervised @xcite and semisupervised @xcite approaches. temporal ordering and relevance previous works @xcite @xcite have investigated induction or prediction of temporal ordering of prototypical events. others have predicted next @xcite or related @xcite events in natural language form. @xcite acquire commonsense procedural knowledge directly from natural language source, like wikihow, by learning representations for scenarios and events which are predictive of both relevance of event to the scenario and temporal ordering. @xcite propose a non-learning based approach to predict fixed-length events given an unseen scenario and related scenarios with their events. a recent work @xcite generates partially-ordered scripts using plms by predicting events and edges for partial-order while we are interested in completely ordered event descriptions. @xcite propose the task of goal oriented script construction for multilingual wikihow dataset and propose generation and retrieval-based approaches. however, their generation-based approach using lm only involves fine-tuning. we focus on different lms to evaluate them on the task ## probing for script knowledge we design a zero-shot probing experiment to evaluate plms' ability to generate esds by carefully selecting natural language prompts, which lms are known to be sensitive to @xcite . we experiment with 16 manually crafted prompts 2 ( bart and t5 were not able to output anything except the input prompt or start, end, and pad tokens and hence we only present qualitative outputs from gpt2, when probed with various prompts for baking a cake scenario, in ## sif: script induction framework in this section, we provide details on our pipelinebased script induction framework, sif (figure 3 ), which addresses the limitations of zero-shot esd generation. sif is a two-staged framework which fine-tunes lm on a small set of esds in the first 1 these are the things that happen when you bake a cake: 3 tablespoons (30ml) of unsalted butter, at room temperature 2 tablespoons (30ml) of granulated sugar 2 tablespoons (30ml) of brown sugar, sifted 1 large egg, at room temperature 1 teaspoon vanilla extract 3 large egg yolks 1 cup (350ml) buttermilk to the bowl of an electric mixer fitted with a paddle attachment, add the butter, sugar, brown sugar, egg yolks, and vanilla extract and beat until light and fluffy. add the buttermilk, then continue beating until fully incorporated. sift the flour, baking powder, and salt into the bowl of the mixer and beat 2 is it done? 7. is it hot? 8. what is that crust? 9. what does the cake look like ## evaluation we use sif to induce script knowledge in gpt2, bart, and t5, and evaluate full esds generated for a given unseen scenario using bleu metric @xcite , following pichotta and mooney (2016b) who use bleu to score individual lmgenerated events. as bleu is a precision-based metric, we measure n-gram overlap of the sampled esds against multiple gold-reference esds foot_2 for each scenario in the test fold. additionally, for deeper analysis of the generated esds, two of the authors evaluate a subset of the generated esds (blinded to the identity of the models and prompt variants) on three levels - ## limitations de-duplication of events. as mentioned previously, sif cannot de-deuplicate paraphrased version of an event. therefore, more sophisticated paraphrase identification systems could be used to de-duplicate such events. there could be scenarios where multiple occurrence of same event is required. for instance, washing dishes wherein faucet needs to be opened and closed once at the starting before applying soap and secondly after applying soap (when washed by hands). hence, it is required to differentiate between desirable and undesirable repetition of events. full vs partial temporal ordering. while we consider the task of generating full event sequence descriptions for a scenario, we acknowledge that many scenarios may not have strict ordering of events (e.g., either wet ingredients can be mixed first or dry ones in a baking a cake scenario) or there can be overlapping events (e.g., while oven is pre-heating, batter can be prepared). instead of considering partial ordering of events @xcite , we focus on generating multiple possible full sequence of events for a scenario and report the averaged scores. ## conclusion and future work we investigate whether pre-trained language models are capable of generating full event sequence descriptions with minimal prompting and find that pre-trained gpt2 has an incomplete understanding of scripts, while bart and t5 did not even produce anything useful through zero-shot probing experiments. we propose sif, an lm-agnostic script induction framework, that is shown to produce meaningful esds for unseen scenarios and mitigate errors (such as scenario-irrelevant, repeated, and misordered events) that were observed during probing experiments, as measured by automatic and manual evaluation. we also provide evidence for the generalization capability of our framework to novel scenarios. however, there is great room for improvement which is evident from manual error analysis and qualitative outputs. future work may focus on developing more sophisticated automatic metrics as well as an end-to-end system for script induction which might help in mitigating cascading of errors, due to each component, common to any pipeline-based approaches.