A Graphical Interface for Curating Schemas (2021)

Abstract: Much past work has focused on extracting information like events, entities, and relations from documents. Very little work has focused on analyzing these results for better model understanding. In this paper, we introduce a curation interface that takes an Information Extraction (IE) system’s output in a pre-defined format and generates a graphical representation of its elements. The interface supports editing while curating schemas for complex events like Improvised Explosive Device (IED) based scenarios. We identify various schemas that either have linear event chains or contain parallel events with complicated temporal ordering. We iteratively update an induced schema to uniquely identify events specific to it, add optional events around them, and prune unnecessary events. The resulting schemas are improved and enriched versions of the machine-induced versions.

## introduction understanding events, how they progress, and who is involved in them is fundamental to our knowledge of the world and our ability to anticipate future events. human beings have mental representations of typical scenarios at various levels of granularity. defining such scenarios or templates for use in information extraction, knowledge base construction, and narrative prediction has a long history. as these fields have progressed, the complexity of the events and sequences being represented has increased. any machine-readable format capable of representing multiple events, tracking their participants across the events, and delineating the temporal and causal relations between the events will be extremely difficult for a person to read and review. because the extraction and construction of such complex schemas has accelerated in recent years @xcite , the need for a way for people to easily review them has increased. in this paper we will describe a tool designed to take complex event schemas in a json format and render them graphically for human review and revision. the complex schemas handled by our tool include multiple levels of intersecting information. for example, imagine a typical emergency medical intervention. we know this includes a victim who is injured or ill and usually begins with communication by the victim or a bystander to an emergency dispatcher, then progresses to communication from the dispatcher to medical personnel, travel by medical personnel to the victim, immediate medical assessment of the victim, and, finally, possible transportation of the victim to a medical facility. a more complete schema would include some alternative or parallel events, such as the possibility of medical personnel already being on site or the death of the victim at any point in the sequence of subevents. as we will describe in more detail, the tool uses distinctive nodes and edges to represent these subevents and participants and their relationships to each other. left to right progression of subevents across the visual field represents temporal progression, and types of edges further indicate specific temporal and causal relations. users can zoom in to specific subevents to see participants and their relations. in addition, our tool allows for simultaneous visualization of a complex schema and direct revision of the underlying json file. our paper begins by describing the background and motivation for our schema editing tool in section 2. section 3 gives detailed description of the tool implementation, while sections 4 and 5 provide examples of its use, and discuss general issues with respect to editing schemas. we conclude in section 6. ## 2 background the idea of typical event scenarios being stored by people as abstract mental frames or schemas with slots for particular participants has a long history in psychology @xcite , linguistics @xcite and artificial intelligence @xcite . prototypical sequences of events, such as the medical intervention sequence described above, can be made explicit in scripts that detail the usual subevents, their typical sequence, the people and objects generally involved in those events, and the progression of those participants through the subevents. proving useful for both linguistic analysis and natural language processing, repositories of defined schemas and event representations were manually developed, such as framenet @xcite , @xcite and verbnet @xcite . most longstanding repositories of such frames or scripts were built at the level of single events and their participants, although some have made sparse connections between these simple event descriptions (e.g., framenet's uses and precedes relations). the desire to extend these event representations to include more complex relations led to the development of systems of temporal, causal and other semantic relations, such as timeml @xcite , richer event descriptions (o' @xcite , reference event ontology @xcite , and abstract meaning representations @xcite . because of the substantial effort involved in manually creating schemas, automatically inducing schemas from textual and visual data has become a priority. automatically generated schemas have advanced from schemas for single events @xcite @xcite @xcite to complex, multistep schemas @xcite however, for optimal usefulness, these generated schemas still benefit from human revision. to our knowledge, no open-source interface for complex schema visualization and editing has previously been developed. one large-scale effort to create a repository of complex event schemas is the darpa knowlegedirected artificial intelligence reasoning over schemas (kairos) program. kairos relies on the assumption discussed above, that humans make sense of events by organizing them into frequently occurring narrative structures that in this context are called schemas. the goal of the program is to develop schema-based ai systems that can identify, link and temporally sequence complex events and their subsidiary elements and participants. the program is set up with separate tasks. task 1 involves inducing schemas from large amounts of text, followed by careful hand curation, with the goal of creating a schema library. task 2 is aimed at finding schema instances that match schemas from the schema library in streaming news feeds. in order to evaluate system performance for both of these tasks, a common, agreed upon kairos schema format is needed. this also allows one darpa team to try to instantiate schemas from another team's schema library and vice versa. the kairos schema format (ksf) stores representations of real-world complex events in a systematic json-ld format containing primitive events, their participants, possible entities acting as the participants, and the relations between these events and entities. our tool takes these json files as input and assumes that all the schemas are validated and tested for format consistency before use. an additional complexity in this beginning phase of the program is a restriction against hierarchical schemas, in which subschemas could be collapsed into a single parent node. the schemas are orderings of all individual events, that can therefore get quite lengthy. ## schema curation interface the schema curation interface 12 is a web application designed for interpreting induced schemas. it provides a visual representation of the schema to understand the underlying structure, reflects relations between events and entities, and allows correction of potential flaws. the interface accepts ksf-validated schemas as input, extracts the events and participants as nodes and relations as edges, and visualizes them as a graph on a canvas. these graphs, in turn, can be corrected at the discretion of the curators. the interface is an open-source project that is accessible from 'cu-clear' github repository. we use react.js and flask for designing the web application. react.js is a javascript library to build interactive user interfaces (ui). it allows encapsulated component building and easy debugging, which makes new features easy to integrate. flask is a micro web framework written in python that acts as the web server for receiving requests from the user and sending a response. react passes the schema, uploaded by a user, to the flask web server, which in turn extracts the possible nodes and edges between them and returns them to the ui for rendering. cytoscape.js @xcite uses these nodes and links to generate the graph on a canvas. cytoscape.js is a fully featured graph library written in javascript that allows users to display and manipulate rich, interactive graphs. in the curation interface, it controls the positioning and layout. explicit configuration constrains cytoscape.js to orient the representation from left to right within the canvas, which preserves any possible temporal ordering and parallel events. it also has standard gestures like dragging and zooming on desktops as well as touch devices. besides the canvas, a json-viewer provides a json view of the uploaded schema. it allows the user to edit the schema and dynamically update the graph structure. add, edit, and delete are the three operations the "react-json-view" library allows for manipulating schemas. "react-json-view" is a react component for displaying and editing javascript arrays and json objects. the interface is currently accessible from the web using google cloud platform (gcp). we use docker and kubernetes in this process. docker enables the packing, shipping, and running of our application as a portable and self-sufficient container, which can run virtually anywhere. kubernetes runs and coordinates these containerized applications across a cluster of machines, automating the deployment, scaling, and management process. kubernetes' load balancing configuration keeps at least one instance always available to a user. since the user count is small, six replicas serve the purpose. but as the users increase, we can scale it up. a separate log server keeps track of any issue or error that occurs while parsing the schemas using rabbitmq. it helps in debugging and analyzing the usage of the interface. the representation consists of nodes and edges. the nodes signify an event (referred to in the schema as "step"), entity (as "participant" or "slot"), or filler (as "value" for the mentioned "participant"). shapes like ellipse, round-rectangle, and round-pentagon distinguish one node type from another. the edges signify temporal relations between any two events, an entity's participation in an event, or co-reference between two entities. the current implementation allows a dual-layer view. the first (default) view (figure 2 ) shows only the representation of events and their temporal relations. selecting an event opens the second view (figure 3 ), consisting of entities and values for that event. as a result, all the entities and fillers remain hidden in the default view and reduce the clutter in the visualization, increasing readability. while the dual-layer view allows a cleaner visualization of schemas, having many events in a schema (currently constrained to not make use of hierarchical structure) can still clutter the representation. since the layout is from left to right, a complex event with a long sequential chain of events becomes partially hidden on reaching the screen width. cytoscape.js reacts by zooming out the canvas to keep the graph in view as much as possible. however, in doing so, the schema graph becomes illegible. the only solution is manually arranging the nodes so they are in the scope and are using an appropriate zoom setting. ## interface use approximately 10 people collaborating across three institutions used the interface heavily to induce schemas and manually curate them. the schemas were induced from shared data using information extraction systems. after evaluating a system's output, the inducers shared the schema draft with the human curators for editing. the interface facilitated the interpretation of the schemas and simplified the identification of needed changes to events, entities, values, or relations between them. we then created a new set of visualizations for comparison with the pre-curated version. the schema library inducers, after discussion, used these curated sets of schemas for improved inductions, which were passed back for further curation. we repeated this process until we reached a version where the schema had the highest coverage over the data for each complex event. the human curators worked with two sets of automatically induced schemas, from two different institutions. one set of schemas consisted of events in a linear chain, lacking parallel events. the other set contained parallel events with more complex temporal ordering. by comparing the visualizations of these schemas, curators were able to identify which schemas covered similar events and could select the best parts of each to create a single, comprehensive schema. in addition, we aimed to include all possible events in a scenario from planning stages to results. every induced schema was missing events or had extraneous events, which were easy to identify and fix with the curation interface. however, the greatest improvement to curation came from the ease of visualizing parallel and optional events and of tracking entities across multiple subevents, all of which are obscured in the necessarily linear presentation of a json file. kairos event primitives are the backbone of the schemas. kairos has defined event primitives to be elemental single events that are unlikely to be decomposed into subevents, but could themselves form crucial elements of more complex events. the event primitives are comprised of the event definition, the roles associated with the event and their corresponding constraints, as well as temporal information about the event. figure 1 provides an example of a life.die event primitive. events in the schema are automatically generated in temporal order from left to right in the curation interface. parallel events originate from the same source and can occur simultaneously or independently of the sibling events. a gray rectangle labeled start indicates the beginning of the schema. individual events are green ellipses. the general attack schema (figure 2 ) can be used to represent any attack. we include a demonstration event as an instigating event that could motivate an attack. alternative instigating events will be added in future work. after the instigating event, the attack event happens, leading to three simultaneously occurring parallel events -death, injury, and damages due to the attack, and one independently occurring parallel event -an investigation. events related to medical intervention, an arrest, a trial, and sentencing follow. by clicking on individual events, the event's arguments are revealed, along with various relationships between them. the entity nodes are peachcolored pentagons. coreference relationships are indicated using the relation "same as." in figure 3 , these coreference relations indicate that the attacker is the same as the agent of the death event and of the damage event. on the right side of the schema, the interface shows the schema in the json format (see figure 2 ), which is directly editable. therefore, if one sees that the order of events is wrong, or deletion or addition of events is necessary, these changes can be made within the interface. we can also see details of the events and participant entities more legibly on the screen's left side by right-clicking on any specific event or entity, as shown in figure 3 . the drone ied (figure 4 ) is an expansion of the general attack schema. we added more events specific to drone ied's, such as buying drones, buying parts to make an ied, moving both to a common place, and assembling them. we added options like a drone crash, and a detonate event in place of the attack event from the general attack schema. we also added more event primitives related to damage and destruction. all other events are similar to the general attack schema. ## iterative schema updates as mentioned above, there were several rounds of schema curation. one of our goals was identifying distinctive schema events. for example, in a drone-based ied schema, acquiring a drone is an essential step; and in a vehicle-based ied schema, acquiring a vehicle is an essential step. the final step was checking the proper temporal ordering between the events. we went through several wiki articles about various ied attacks to determine generic event types and the correct temporal ordering for all the steps. the visualization in the interface facilitated these tasks by allowing us to see generalities across schemas and to easily spot gaps in the temporal ordering. in the subsequent rounds, we focused on robustness. we introduced new optional events to schemas, such as an acquittal event, which better encompassed the possible outcomes of a trial. we also introduced additional phases to the schemas, such as instigating events like conflict between the terrorist organization and a government. we also introduced retaliatory events to describe what actions were taken after the attack happened, such as demonstrations against the ied attacks or retaliatory actions taken by the army. to demonstrate a typical editing task, the following example walks through the steps needed to make a new temporal connection between events. this example is also illustrated in our demo video. in figure 5a , there is a missing link between the events injury and medical intervention. to add the link, we get the ids of the injury and medical intervention events by right-clicking on these events, and then create a new entry in the order key provided in the json editor on the right side of the screen by clicking on the plus sign next to the or-der key. this creates a new order step with null value. as shown in figure 5b , within this box, we write the id of the injury event as the value to the "before" key, and the id of the medical intervention event as the value to the "after" key. this signifies that an injury event needs to happen before medical intervention. we can also add flags like optional or precondition. the flag name is displayed on the arrow connecting the events. if no flags are given, "before" is displayed on the arrow. we also need to provide a unique id to this order step. after saving the json, the changes are automatically reflected in the visualization. figure 5c shows the curated schema, with a link present between the (a) no temporal link between injury and medical intervention (b) editing json (c) temporal link added figure 5: schema curation demo injury and medical intervention event. similarly, we can delete the events or link them. an event can be deleted by removing it from the step key in the json, and links can be deleted by deleting the order id containing the mentioned events. ## conclusion and future work in this paper, we introduced an interface that assists human curators in refining induced schemas. these schemas contain events, entities, and the relations between them. the curation interface extracts these elements in the form of nodes and edges and represents them in a graphical structure. the visualization enables the curators to better understand the ordering of events and the relations between entities, resulting in an improved and enriched schema. we also discussed leveraging the attributes of two structurally different induced schemas to design a single unified schema. this schema captures the salient events from its parents while reducing the schema size. we explored various ied-based schemas, general attack, medical intervention, and disease outbreak schemas using the curation interface. the schemas currently include a small set of primitive events, limiting the scope of an induced complex event. in the future, a schema can have a hierarchical composition, meaning a combination of complex and primitive events within a single schema. future work will focus on improving the handling of hierarchical schemas, provide dynamicity to the visualization, and comparison of two schemas. it will also allow changes, like editing or deleting nodes and edges, on the graph beside the json editor.