Machine-Assisted Script Curation (2021)

Abstract: We describe Machine-Aided Script Curator (MASC), a system for human-machine collaborative script authoring. Scripts produced with MASC include (1) English descriptions of sub-events that comprise a larger, complex event; (2) event types for each of those events; (3) a record of entities expected to participate in multiple sub-events; and (4) temporal sequencing between the sub-events. MASC automates portions of the script creation process with suggestions for event types, links to Wikidata, and sub-events that may have been forgotten. We illustrate how these automations are useful to the script writer with a few case-study scripts.

## introduction scripts have been of interest for encoding procedural knowledge and understanding stories for over 40 years @xcite . in the form of checklists, recording procedural knowledge has revolutionized fields like medicine and aviation by encoding expert knowledge and best practices @xcite . in the last few years, researchers have turned their attention to automatic script discovery from text @xcite @xcite . however, exclusively data-driven sub-event discovery methods face the challenge that narrative descriptions often omit common knowledge. 1 we aim for a process for building a library of scripts through human-machine collaboration leveraging nlp techniques to augment human background knowledge. the resulting demonstration system serves two related purposes. first, it is a knowledge acquisition tool that supports the development of a repository of scripts for use by downstream applications. second, it is an annotation tool that supports the creation of a library to 1 common knowledge might be missing from narrative descriptions due to the quantity and relevance maxims @xcite . while approaches to script discovery suffer from the incompleteness of text, human attempts to write machine-interpretable scripts suffer from the writer's own tendency to omit steps and, where required, the challenge of mapping to a formal ontology. to assist the script creators, masc makes three types of suggestions: (1) the ontological type for each event; in the following sections, we describe the process of creating a script in masc and the nlp components that support suggestions. 2 while a large-scale script repository is beyond this paper's scope, we have created five sample scripts, which we use as case studies for understanding the script creation process and the suggestion capabilities. in section 4, we use these scripts to measure the utility of masc's suggestion capabilities. in section 5, we describe the scripts' characteristics. both symbolic and neural approaches suffer from the lack of generic knowledge to "fill-in-theblanks" or reject impossible events. training systems to incorporate common-sense knowledge @xcite has not yet addressed script creation. another source of information for script discovery could be extraction from multiple languages and modalities. while some extraction systems have incorporated these other sources @xcite , such extractions have not yet fed into script discovery. resolving the cooccurrence of events or entities between languages and modalities often relies on a common mapping, e.g., a structured ontology, such as ace @xcite or ere @xcite . while our machine-aided script curator (masc) does employ a structured ontology, it does not currently incorporate multi-modal or non-english sources. however, the limited ontology allows the eventsequencing background knowledge we encode to be used as a supplement to state-of-the-art information extraction systems, like oneie @xcite and dygie++ @xcite , providing connections between otherwise disconnected extractions. ## overview of script creation the curator initiates script creation by providing a name and description for the script and then enters, as text, the events in the script (figure 1 ). step entry is free-form, but we have noticed a tendency for curators to enter short, imperative sentences around a central agent's actions (e.g., go to a car dealership, take a test drive). currently, script creation, unlike traditional annotation, is decoupled from any particular document. in cases where the curator is not familiar with a topic, we have used external resources to provide context (e.g., a wikihow page open in a different window). in this setting, curation is akin to annotation that encourages the annotator to use both the material they read and prior knowledge. the curators assign an ontology type to the main event in each step (e.g., movement for both go to a car dealership and take a test drive). the ontology is configurable and can be replaced. we include a project-specific ontology with masc's source code. when saved, scripts include both the curators' description and the selected ontology type (described in section 4.1). this choice allows type decisions to be revisited if the ontology changes and limits the degree to which the small number of event types constrains the script's expressiveness. downstream applications can choose whether to use the linguistic representation of the events or the normalized ontology types. after the curators finish entering events, they encode connections between the events (figure 2 ). there are two ways to connect events: the first, traditionally the focus of scripts, is temporal order; and the second is shared arguments (e.g., the same person is the agent of both movement events go to a car dealership and take a test drive). to add sequential order, the curators enter pairwise before relations. alternatively, they select multiple events and anchor them as coming before or after a single event. the latter method is convenient when the complete order is under-defined. 3 the curators add shared arguments to the script by selecting multiple events with the same argument, naming the argument (e.g., buyer, seller in figure 2 ), and assigning an entity type (e.g., per in figure 2 ) and ontological role to each argument (e.g., identifier, researcher in figure 2 ). while this process is mostly manual, masc uses the ontology's constraints to limit the available label options. in addition to project-specific entity types, masc suggests links to the much larger set of types available using wikidata entities (e.g., suggesting q786803 for car dealership). these links provide a connection to an extensive knowledge graph and can provide additional information when the scripts are applied. finally, the curators review events that are automatically generated based on the manually entered description and initial script (described in section 4.3). the suggestions can add intermediate steps that the curators may have missed, complete a script that was intentionally unfinished by the curator, or suggest alternative related paths (e.g., leasing instead of purchasing a car). ## suggestion capabilities to aid script creation, masc incorporates three suggestion capabilities: suggestions for the ontological event type, suggestions for links to wikidata, and suggestions for additional events to incorporate in the script. below, we describe the models behind these capabilities and, for each model, report the accuracy using the five sample scripts created for this paper. given the small sample size, the five sample scripts are best thought of as case studies, not a benchmark. ## discussion and future work with this demonstration system, we provide an approach to human-machine collaboration for building a repository of scripts. having such a repository, for a diverse set of events, will allow us to investigate how procedural knowledge introduced to the ai community 40 years ago @xcite can be broadly applied. by facilitating the human creation of scripts, we can better understand what is required to develop automatic script discovery approaches. while we have not yet created a large repository of scripts, we have created five scripts with which we start this analysis. the scripts cover topics with varying degree of "common knowledge": planning and managing an evacuation (evac), ordering food at a restaurant (food), finding and starting a new job (job), obtaining medical treatment (med), and corporate merger or acquisition (merger). a single curator created these scripts, which we use to illustrate future directions for masc and interesting properties of the scripts themselves. having multiple curators for even a small number of scripts would provide insights into the diversity, prior knowledge, and level of detail a script author uses. in our analysis, we have seen that the scripts created with masc encode knowledge that is uncommon in news-like data sets. for example, our curator included sign confidentiality agreement as an event in the script for a merger. while news frequently reports the final step of a merger, the full process is rarely described. all scripts contain entities that play a role in multiple events (row 3, first and second numbers). for example, in evac, the evacuation manager plays some role in all events, while the evacuee plays a role in most but not all. while some arguments cannot be linked to wikidata, all five scripts contain at least one argument that can be linked (row 3, last number). future work could both improve linking accuracy and use wikidata as a source of knowledge to provide additional context (and suggestions) to the curator. while the prototypical script is a timeline with complete order between all pairs of events, we see sub-graphs with unordered steps in our data. three of the five sample scripts display this behavior; for example, in job, searching for open positions and notifying network that they are looking for a job are unordered. the visualization of the schema in figure 2 illustrates this pattern with no order between e2 and e3. masc incorporates machine suggestions of unrecorded events. in four of the five scripts, the curator accepted at least one suggestion. interestingly, the curator incorporated more suggestions for two events that one thinks of as everyday experiences (food and med) than they did for the script they were unfamiliar with (merger). this suggests that the recommendation functionality can be useful even in a familiar domain; by capturing what the curator omits through forgetfulness or because they assume common knowledge. further exploration of how a machine can aid a person whose knowledge may be incomplete or may forget to be explicit seems promising. examples of possible research directions include incorporating suggestions from approaches that discover scripts (e.g., @xcite ; @xcite ) and leveraging background knowledge (e.g., wikidata). ## ethical considerations many technological innovations require ethical considerations, even more so for those involving machine learning while also being a demonstration paper that provides working technology. below we address the review questions raised in the naacl ethics review questions. 10 bias. the bias in generative language models has been well documented. in general, using a human-in-the-loop process means that rather than treating an automatically generated label or event as correct, we treat it as a suggestion that the curator can ignore. still, the suggestions can influence the curator. thus it is vital that the metrics reported in this paper be interpreted with an understanding of the potential for bias and any use of masc account for bias. masc incorporates both a predefined ontology and the ability to link to an extensive external resource (wikidata). given the size of the predefined ontology is small, to apply masc to a new domain, users would likely need to update the ontology. masc's approach to aligning english descriptions to the ontology makes adding new event classes easy. wikidata, while much larger and growing, is also subject to the bias of wikidata's editors, their knowledge, and their choices about what to include. wikidata over-represents some issues, while some socially important ones are under-represented or missing. wikidata linking is optional; thus in a domain that is not well covered, a curator can skip the linking step or replace wikidata with a domainrelevant resource. the suggestion capabilities described in section 4 use pretrained language models (gpt-2 and roberta). the bias of these algorithms, measuring that bias, and mitigating it is an active area of work. recent work has provided data sets for measuring bias @xcite and metastudies of the approaches taken to study and address bias @xcite . much work has focused on bias as it impacts demographic groups. masc focuses on events, not individuals. the publicly available gpt-2 models have learned from data that might not cover current events (e.g., gpt-2 was trained before the covid-19 epidemic), represents only english dialects from the inner-circle (dunn and adams, 2020), and contains toxic language @xcite . in our immediate context, we mitigate against the challenge presented by language model bias by requiring manual review of all automatically suggested output. if the ideas in this paper were extended to a fully automatic approach, language model and domain-specific studies of the impact of bias on lm-based suggestions would be necessary.