Zero-Shot On-the-Fly Event Schema Induction (2023)

Abstract: What are the events involved in a pandemic outbreak? What steps should be taken when planning a wedding? The answers to these questions can be found by collecting many documents on the complex event of interest, extracting relevant information, and analyzing it. We present a new approach in which large language models are utilized to generate source documents that allow predicting, given a high-level event definition, the specific events, arguments, and relations between them to construct a schema that describes the complex event in its entirety. Using our model, complete schemas on any topic can be generated on-the-fly without any manual data collection, i.e., in a zero-shot manner. Moreover, we develop efficient methods to extract pertinent information from texts and demonstrate in a series of experiments that these schemas are considered to be more complete than human-curated ones in the majority of examined scenarios. Finally, we show that this framework is comparable in performance with previous supervised schema induction methods that rely on collecting real texts and even reaching the best score in the prediction task.

## introduction event processing refers to tracking, analyzing, and drawing conclusions from streams of information about events. this event analysis aims at identifying meaningful events (such as opportunities or threats) in real-time situations and responding appropriately. event processing can also be utilized to gain a deep understanding of the specific steps, arguments, and relations between them that are involved in a complex event. the information above can be consolidated into a graphical representation called an event schema @xcite . for instance in fig. 1 , the graph representation of events and participants assists in gaining an understanding of the complex event of kidnapping and could help composing a reaction plan if needed. the nlp community has devoted much effort to understanding events that are described in a document or in a collection of documents for this purpose. these efforts include identifying event triggers @xcite @xcite , extracting event arguments @xcite @xcite , and predicting the relations between events, e.g., temporal, coreferential, causal or hierarchical relations @xcite @xcite @xcite @xcite . previous works on event schema induction relied on the information extracted from manually collected documents to build the schema graph. for instance, @xcite learn an auto-regressive language model (lm) over paths in the instance graphs depicting events, arguments and relations of instances of the complex events, and then construct a schema graph by merging the top k ranked paths. their approach, however, requires access to many documents on each topic of interest, which can be extremely laborious and time consuming to obtain. in this paper, our goal is to allow creating schemas on-the-fly by taking as input only the name of the complex event of interest (like a "pandemic outbreak" or an "armed robbery"). to avoid manually collecting many documents on the topic of the schema, we utilize pre-trained text generators, e.g., @xcite , to obtain documents of diverse genres on the desired topic (examples presented in fig. 2 ). these documents are then processed to extract pertinent information from which a schema is constructed. the fact that we do not collect any data makes our learning framework zero-shot since we do not rely on any human-collected articles or example schemas. preparation kidnapping kidnapper plans and collects information kidnapper looks for victim kidnapper chooses victim or location kidnapper kidnaps victim kidnapper transports victim kidnapper hides victim kidnapper asks for ransom or makes demands in addition to eliminating the need to collect data, we also made the information extraction process faster by implementing new and efficient methods for identifying temporal and hierarchical relations between events mentioned in the text. these two steps are the most time consuming in the process of schema induction and could take up to 2 hours each using state-of-the-art models proposed by @xcite ; @xcite . sending the whole text as input instead of two sentences at each time, our proposed model shortens the inference time significantly to several minutes without enduring a major loss in performance. the process of generating texts is explained in section §3, and the process of extracting relevant and salient information is described in section §4, then we introduce the construction of schema graphs in section §5. to evaluate our zero-shot schema generator we conduct experiments on a benchmark dataset for schema induction, ldc2020e25, and provide a new dataset for further evaluation called schema-11. additionally, we design a subject-matter expert turing test, a.k.a. feigenbaum test @xcite , to determine whether our algorithm could mimic experts' response. we also demonstrate that documents generated by gpt-3 are informative and useful for the task of schema induction. the experiments and results are presented in section §6. the contributions of our work include: ## data generation the schema induction process begins with generating texts using large lms as text generation models. these texts are joined to form a knowledge base for the schema, including all of the potential information that the schema may present. one could, of course, create this knowledge base by crawling the web for real news articles or wikipedia entries related to a certain topic. we argue, however, that in addition to the obvious advantages of not having to rely on the availability of data online and not having to crawl the entire web for relevant documents on each topic, the generated data from these large generative models is more efficient in reporting salient events than random events described in the news, i.e., generated texts are more likely to mention important information than real documents do. our analysis shows that the generated stories contain a higher percentage of relevant tokens than generated text real text # events / # tokens 12.52% 6.31% # arguments / # tokens 5.45% 3.01% we calculate the number of relevant event triggers and arguments identified in the text, where a relevant mention is one whose type appears in the ontology. the results shown in given a topic for which we want to create a schema, we generate multiple texts that discuss the topic event using the openai gpt-3 api foot_2 with the davinci-instruct-beta-v3 model and we also experiment with the hugging face gpt-2 api 4 . we use three prompting methods to generate documents of diverse genres as follows: news articles: we begin by generating a headline using the prompt: "write a news headline about topic." the output from this prompt is then used in the following prompt: "write a news story titled headline." the output from the second prompt is added to the pool of generated texts. the process is repeated 30 times. see example in fig. 2b . how-to articles: we use the prompt: "describe how to topic." to generate wikihow-like instruction prompt: what are the steps involved in holding a summit meeting? 1. output: ## information extraction for each document, we extract event triggers, arguments and relations between the events that are important and relevant to the schema topic. we do not work with a predefined ontology that defines what events and arguments are salient in advance because we allow generating a schema on any topic. instead, we employ a statistical approach by extracting all the information and later filter it down to include just frequent items. here are the steps involved in our information extraction pipeline: semantic role labeling (srl): we use the sota srl system foot_5 trained on conll12 @xcite and nombank dataset @xcite to extract both verb and nominal event triggers and arguments. ## schema induction to consolidate the information extracted from the previous step, we build a schema as follows: make a list of events and relations: to compare similar event mentions in different texts, we compare the event trigger itself (whether they are the same verb or coreferential verbs foot_9 ) and the ner types of its arguments. for example, the trigger "(take) precautions" appeared in 5 documents generated for the topic of pandemic outbreak. in two documents the subject of the verb phrase "take precautions" was "residents", in another two it was "people" and in the last one, it was "public". nevertheless, the ner type is identical in all cases (per), and thus we set the frequency of "(take) precautions" to 5. similarly, we calculate the frequency of the temporal and hierarchical relations. we only consider relations and events that appeared in more than one document. ## experiments 6.1 data we conduct experiments on a dataset for general schema learning released by ldc (ldc2020e25). the corpus includes 84 types of complex events, such as cyber attack, farming and recycling. this dataset includes ground-truth schemas created by ldc annotators. in addition, we also collected human generated schemas for 11 newsworthy sce-narios 11 . the schemas were generated by four human experts who were instructed to write a schema on each topic based on their commonsense knowledge that includes a list of event triggers, event arguments and their ner types 12 , and relations 13 . ## conclusion we propose a method to generate schemas given the sole input of a topic. we use gpt-3 to generate texts of diverse genres and a pipeline of information extraction tools to obtain relevant information before inducing logical relations and integrating the events and relations into a schema graph. to improve the efficiency of the pipeline, we implement one-pass models for identifying temporal and hierarchical relations that achieve comparable performances with sota models but require far less inference time and memory space. to evaluate our framework, we conduct experiments on a benchmark ldc dataset to show that our schemas cover a decent amount of pertinent information and display comparable ability for event prediction with supervised approaches. we observe a high percentage of valid events and relations generated for the schema-11 dataset and the testers endorsed the completeness of our machine-generated schemas.