Open-Domain Hierarchical Event Schema Induction by Incremental Prompting and Verification (2023)

Abstract: Event schemas are a form of world knowledge about the typical progression of events. Recent methods for event schema induction use information extraction systems to construct a large number of event graph instances from documents, and then learn to generalize the schema from such instances. In contrast, we propose to treat event schemas as a form of commonsense knowledge that can be derived from large language models (LLMs). This new paradigm greatly simplifies the schema induction process and allows us to handle both hierarchical relations and temporal relations between events in a straightforward way. Since event schemas have complex graph structures, we design an incremental prompting and verification method IncPrompt to break down the construction of a complex event graph into three stages: event skeleton construction, event expansion, and event-event relation verification. Compared to directly using LLMs to generate a linearized graph, IncSchema can generate large and complex schemas with 7.2% F1 improvement in temporal relations and 31.0% F1 improvement in hierarchical relations. In addition, compared to the previous state-of-the-art closed-domain schema induction model, human assessors were able to cover ~10% more events when translating the schemas into coherent stories and rated our schemas 1.3 points higher (on a 5-point scale) in terms of readability.

## introduction schemas, defined by @xcite as "a predetermined, stereotyped sequence of actions that defines a well-known situation", are a manifestation of world knowledge. with the help of schemas, a model can then infer missing events such as a person must have "been within contact with a pathogen" before the event "the person was 1 code and odin dataset available at https:// github.com/raspberryice/inc-schema . instance graphs entity extraction event extraction coreference relation extraction ie system documents schema graph large language model incremental prompting & veri cation graph learning model sent to the hospital for treatment" and also predict that if a large-scale incident happened, this might trigger an "investigation of the source of the pathogen". to automate schema creation, two mainstream approaches are to learn from manually created reference schemas or learn from large amounts of event instances automatically extracted from documents. manual creation of complex hierarchical schemas requires expert annotation, which is not scalable foot_0 . on the other hand, instance-based schema induction methods @xcite @xcite rely on complicated preprocessing foot_1 to transform documents into instance graphs for learning. moreover, supervised information extraction systems @xcite are domain-specific and suffer from error propagation through multiple components, which makes the downstream schema induction model closed-domain and low in quality. tracing back to the original definition of schemas, we observe that "stereotyped sequences of events" or "the typical progression of events" can be viewed as commonsense knowledge that can be implicitly learned by training on large corpora. through the language modeling objective, models can pick up which events statistically frequently co-occur and how their relationship is typically described. more recently, large language models (llms) such as gpt3 @xcite and palm @xcite have shown impressive zero-shot performance on closely-related commonsense reasoning tasks such as goal-step reasoning @xcite and temporal reasoning 4 . by utilizing llms to directly prompt for schematic knowledge, our approach is opendomain, extensible and more interpretable for humans. given a new scenario name, our model only requires lightweight human guidance in providing some top-level chapter structure (as shown in the left of figure 2 ) and can produce the entire schema in under an hour whereas instance-based methods require months to collect the data and retrain the ie system for new domains. our model is extensible and can support new types of event-event relations by adding new prompt templates. to showcase this, in addition to the temporal relation between events which is the focus of prior work, we also account for the different event granularities by supporting hierarchical relations between events (for example, a physical conflict could happen as a part of a protest). finally, by representing events with free-form text instead of types and organizing them into a hierarchy, our generated schemas are considered more interpretable. we find that directly asking llms to generate linearized strings of schemas leads to suboptimal results due to the size and complexity of the graph structure. to solve this problem, we design an incremental prompting and verification scheme to break down the construction of a complex event graph schema into three major stages: event skeleton construction, event expansion, and event-event relation verification. as shown in figure 2 , each stage utilizes templated prompts (what happens before cases increase?) which can be instantiated either with the scenario name or the name of a previously generated event. the key contributions of this paper are: given a scenario name, a schema depicts the general progression of events within that scenario. following @xcite , we consider the schema to be a graph structure of events. we output a schema graph of event nodes and event-event relation edges, including temporal relations and hierarchical relations. since our algorithm is designed to be opendomain, we represent each event e with a description string such as "a person shows early symptoms of the disease" instead of a type from a restricted ontology (e.g., illness). description strings are more flexible in representing different granularities of events and are more informative. it is noteworthy that event descriptions in a schema should be general, instead of a specific instance, such as "john had a mild fever due to covid". outbreak preconditions outbreak authority response society response input chapter structure disease outbreak stage 1: event skeleton construction outbreak pathogen emergence cases increase diagnosis list the major events that happen in an {outbreak}: stage 2: event expansion and validation what happens before {cases increase}? what happens after {diagnosis}? outbreak pathogen emergence cases increase diagnosis disease transmit monitor stage 3: event-event relation veri cation does {pathogen emergence} start before {disease transmit}? outbreak pathogen emergence cases increase diagnosis disease transmit monitor in addition, we support the representation of chapters, which are "a collection of events that share the same theme and are connected in spacetime". when a high-level chapter structure g c (as shown in the left side of figure 2 ) is available, we condition on the given chapters to guide the schema generation process. chapters are also treated as events and can potentially have temporal relations between them. every other event must be a descendant of a chapter event. if no chapter structure is available, we create a single chapter from the scenario name. ## our approach leveraging llms to directly generate the full schema graph is challenging due to the size and complexity of schemas. thus, we divide our schema induction algorithm incschema into three stages as depicted in figure 2 . starting from the scenario node or one of the chapter nodes, the skeleton construction stage first produces a list of major events that are subevents of the scenario (chapter) following sequential order. for each generated event, we expand the schema graph to include its temporally-related neighbors and potential children in the event expansion stage. for each pair of events, we further rescore their temporal and hierarchical relation probability in the relation verification stage to enrich the relations between events. ## experiments we design our experiments based on the following three research questions: q1: hierarchical schema quality can our model produce high-quality event graph schemas with both temporal and hierarchical relations? q2: interpretability is our model's output more interpretable than prior instance-based schema induction methods? q3: model generalization can our model also be applied to everyday scenarios as in @xcite ? ## related work event schema induction event schema induction, or script induction, is the task of inducing typical event-event relation structures for given scenarios/situations foot_7 . a large fraction of work considers event schemas as narrative chains @xcite @xcite @xcite @xcite @xcite , limiting the structure to include only sequential temporal relations. more recently, non-sequential, partially ordered temporal relations have been taken into consideration @xcite @xcite @xcite but they do not consider the different scales of events and the potential hierarchical relations. in terms of schema expressiveness, @xcite is the most similar to ours as they also consider both partial temporal order and hierarchical relations. our work also resembles a line of recent research on inducing schema knowledge from pre-trained language models. our schema induction process can be seen as a super-set of the post-processing in @xcite , which comprises irrelevant event removal, de-duplication, and temporal relation correction. we compare our incremental prompting approach with the end-to-end approach proposed in @xcite in section 4. the work of @xcite is orthogonal to ours as they use llms for data generation instead of probing for schema knowledge. language model prompting prompting has been the major method of interaction with billionscale language models @xcite @xcite . prompting can either be used to inform the model of the task instructions @xcite , provide the model with task input/output examples @xcite , or guide the model with explanations @xcite and reasoning paths @xcite . in this work, we explore how a complex knowledge structure such as an event graph schema can be induced using llms by decomposing the task through incremental prompting. ## conclusions and future work prior work on schema induction has either relied on existing information extraction pipelines to convert unstructured documents into event graphs, or require massive human effort in annotating event schemas. we propose to view schema induction as a type of event-oriented commonsense that can be implicitly learned with large language models. however, since schemas are complex graph structures, instead of directly querying for schemas, we design an incremental prompting and verification framework incschema to decompose the schema induction task into a series of simple questions. as a result, our model is applicable to the open-domain and can jointly induce temporal and hierarchical relations between events. for future work, we plan to cover more aspects of schemas, including accounting for entity coreference, entity relations and entity attributes. while this work is focused on the task of schema induction, we hope to show the possibility of using llms for constructing complex knowledge structures.