pro S cript: Partially Ordered Scripts Generation (2021)

Abstract: Scripts – prototypical event sequences describing everyday activities – have been shown to help understand narratives by providing expectations, resolving ambiguity, and filling in unstated information. However, to date they have proved hard to author or extract from text. In this work, we demonstrate for the first time that pre-trained neural language models can be finetuned to generate high-quality scripts, at varying levels of granularity, for a wide range of everyday scenarios (e.g., bake a cake). To do this, we collect a large (6.4k) crowdsourced partially ordered scripts (named proScript), that is substantially larger than prior datasets, and develop models that generate scripts by combining language generation and graph structure prediction. We define two complementary tasks: (i) edge prediction: given a scenario and unordered events, organize the events into a valid (possibly partial-order) script, and (ii) script generation: given only a scenario, generate events and organize them into a (possibly partial-order) script. Our experiments show that our models perform well (e.g., F1=75.7 on task (i)), illustrating a new approach to overcoming previous barriers to script collection. We also show that there is still significant room for improvement toward human level performance. Together, our tasks, dataset, and models offer a new research direction for learning script knowledge.

## introduction scripts @xcite represent structured commonsense knowledge about prototypical events in everyday situations/scenarios such as bake a cake (figure 1 ). however, while scripts have been shown to help understand narratives by providing expectations, resolving ambiguity, and filling in unstated information @xcite , they have proved hard to author or extract from text, with only small script databases available (regneri find the cake recipe gather the ingredients turn on the oven mix the ingredients put the cake batter in the oven bake for the right amount of time take the cake out of the oven scenario: bake a cake figure 1 : we collected 6.4k of partially ordered scripts (proscript) and developed models that take a scenario (e.g., bake a cake) as the input and generate a (possibly partial-order) script. in proscript, an event (node) requires that all the precedent events and paths are happened/executed in advance. @xcite @xcite . in this work, we show for the first time that pre-trained neural language models (lms) can be adapted to generate high-quality scripts, including appropriately partial ordering events where a specific temporal ordering is required only when it is necessary. lms have previously been shown to successfully generate stories @xcite , summaries @xcite , and commonsense facts @xcite . here we investigate their application to script generation. first, we collect large amount (6.4k) of partially ordered script from crowdsourcing with a similar but simplified collection method @xcite . we call the dataset as proscript (partial order script), and this is substantially larger and more diverse than prior (crowdsourced) dataset such as descript @xcite that has 40 scripts. in proscript, all the events/paths need to be happened/executed (cf. and arcs in and/or graphs), whereas prior work on scripts do not distinct core and optional/alternative events explicitly. additionally, temporal duration of each event is annotated (e.g., take the cake out of the oven typically takes one minute in the bake a cake script), which will potentially link script knowledge with temporal reasoning in future work. 1 second, with the collected data, we introduce two complementary tasks: script edge prediction and entire script generation. in the edge prediction task, given a scenario and unordered intermediate events, models must organize the events as a valid partial-order script. on the other hand, the script generation task is to generate intermediate events and the partial-order of those events for a given scenario. this task requires both natural language generation (for nodes) and graph structure prediction (for edges). finally, based on our proposed dataset, we develop models for both edge prediction and entire script generation tasks. as @xcite has revealed that models trained and evaluated on missing events prediction (i.e., narrative cloze) are insufficient to assess script knowledge, our evaluation scheme evaluate the entire script. we compare the models against baselines, and show that our models outperform the baselines for both the edge prediction and the script generation tasks. nonetheless, there is a significant room for improvement toward human-level performance -e.g., for edge prediction, the best model achieves 75.71 of f1 score while human achieves 89.28, and for script generation, the best model obtains a graph edit distance of 4.97 (i.e., number of human edits), while human-created scripts achieve 2.98 on average. our contributions are thus: script as paraphrase sets script as paraphrase sets @xcite @xcite is more recent approach to gather script knowledge, where crowd workers are asked to write down a sequence of events for a given everyday scenario (e.g., bake a cake) and the collected sequences (called event sequence description) are aligned with paraphrased events being clustered. the collected (partially ordered) scripts cover wide variety of everyday situations compared to narrative chains (news domain), but one shortcoming of this approach is the scalability; it is not easy to scale because of the cost for manual data collection @xcite . in fact, @xcite crowdsourced 1000 stories that cover only 10 scripts, and similarly @xcite end up with collecting 40 scripts. the limited amount of data hinders learning script knowledge by models. furthermore, they provide no evaluation metric on the dataset for assessing model's script knowledge. ## conclusions we show for the first time that pre-trained neural language models can be adapted to generate partial order scripts. we collect 6,400 partially ordered script from crowdsourcing (proscript), which is substantially larger than prior manually crafted datasets. with the proscript dataset, we introduced two complementary task and models, which combine language generation and graph structure prediction, providing the first demonstration that generative models can be successfully applied to script generation, although it is still below human performance. we believe that proscript dataset and models would advance future work on various nlp tasks such as story generation, machine comprehension, temporal reasoning, and high-level planning.