SAGEV iz: S chem A GE neration and Visualization (2023)

Abstract: Schema induction involves creating a graph representation depicting how events unfold in a scenario. We present SAGEViz, an intuitive and modular tool that utilizes human-AI collaboration to create and update complex schema graphs efficiently, where multiple annotators (humans and models) can work simultaneously on a schema graph from any domain. The tool consists of two components: (1) a curation component powered by plug-and-play event language models to create and expand event sequences while human annotators validate and enrich the sequences to build complex hierarchical schemas, and (2) an easy-to-use visualization component to visualize schemas at varying levels of hierarchy. Using supervised and few-shot approaches, our event language models can continually predict relevant events starting from a seed event. We conduct a user study and show that users need less effort in terms of interaction steps with SAGEViz to generate schemas of better quality. We also include a video demonstrating the system.

## introduction event schemas are central to understanding and reasoning about events. they provide a way to organize and represent how complex events unfold @xcite . schema-based reasoning enables reliable and explainable prediction of next events, inference of missing events or entities @xcite @xcite @xcite @xcite @xcite , and drawing connections between events that have already occurred @xcite . for example, when a disease outbreak happens, it is likely that an investigation into the outbreak and mitigation steps will follow. one main challenge in schema-based reasoning is in acquiring the schematic knowledge at scale. one approach to automate schema curation is to learn from manually created reference schemas. manual creation of complex hierarchical schemas require expert annotation, which is time-consuming and not scalable. further, supervised systems @xcite are domainspecific, of poor quality and unable to handle unseen world events foot_0 . we propose sageviz, a human-in-the-loop schema curation and visualization pipeline, a combined approach to producing human-verified schemas using automated acquisition strategies. we leverage large language models (llms) for acquiring various types of events and entity knowledge automatically. we use human inputs to ensure the contextual validity of model produced outputs at various stages in the curation pipeline. sageviz's visualization allows a human curator to easily navigate through the complex event schemas at various levels of granularity starting with the higher level big picture and drilling down to the lower levels or vice versa. sageviz allows users to build and curate a schema from scratch or edit existing ones. sageviz begins from a set of domain-expertidentified seed events. for each seed event, an event language model produces a suggested set of next events in a systematic fashion. for each participant in the seed event, we generate next events by asking the model to predict the next events in which the participant plays agent-based or patient-based roles (event expansion). human curator selects a subset of valid events from the model generated lists (event filtering) and enhances it with explanations, event-entity details such as roles, coreference and wikidata links (event enhancement) before linking event pairs based on logical, temporal and hierarchical event relationships (relation linking). human curation in the enhancement stage ensures proper identification of entities with the goal of linking an event to multiple valid related documents. in the linking stage, we identify and validate the relationship between an event pair with the goal of building schemas with a hierarchical structure to support both specific and generic events. sageviz's visualizer displays the schema as a graph at different levels of granularity, showing various event relation types with different widgets and color schemes. sageviz has multiple advantages. first, multiple human annotators can provide input and curate a single complex schema in parallel. second, it allows for iterative updating, to add, edit and delete non-conforming events at any stage. third, with sageviz allows for modeling complex schemas through quick and intuitive means for generating and expanding sub-events and visualizing relations between them. last, the back end and front end architecture of the tool allows curators to ensure global coherence of the generated schemas, across iterative updates and multiple simultaneous annotators, leading to more reliable schemas especially suited for safety-critical downstream applications. in summary, our contributions are as follows: ## system description in this section we provide a high-level overview and a detailed description of our human-in-the-loop schema curation framework. ## user study we aimed to study the effectiveness and efficiency of sageviz at generating schemas using the process outlined in @xcite and compared it with the schema generation interactive system (cli) used in this study. for a fair comparison, we selected only the event generator component of sageviz. we followed the same guidelines and used the same test data consisting of 35 seeds from 8 domains. four different users interacted with the system spending 4 minutes on each seed, as was done in the previous study, selecting and adding events to the schema that are sensible, relevant, unique and typical for the schema context. for each 4-minute interaction, a user started with a seed event as the schema context and a participating entity to use in the question to the event generator, and generated events using the question guided event language model (qgelm) @xcite . from the generated events, the user selected and added each event that satisfied all 4 of the selection criteria (each is a binary judgment). if any of the events was selected and added, the generation step is counted as an accepted metric cli sageviz # events ĳ 8.8 12.1 % accepted steps ĳ 73.3 80.0 % rejected steps i̇ 26.7 20.0 total steps 12.0 7.48 ## conclusion prior work on schema induction either relied on existing information extraction pipelines to convert unstructured documents into event graphs, or required massive human effort to annotate event schemas. the former lacks the guarantee of being high-quality, while the latter is hard to scale due to its demands of time, effort and cost. we alleviated these problems with a web application that leverages the power of llms and the reliability provided by human intervention to create, expand, and visualize event schemas. our tool enables users to collaboratively and seamlessly generate and update event sequences. we believe that our human-inthe-loop tool reduces expert effort for creating new schemas and analyzing complex event sequences.