RESIN - EDITOR : A Schema-guided Hierarchical Event Graph Visualizer and Editor (2023)

Abstract: In this paper, we present RESIN-EDITOR, an interactive event graph visualizer and editor designed for analyzing complex events. Our RESIN-EDITOR system allows users to render and freely edit hierarchical event graphs extracted from multimedia and multi-document news clusters with guidance from human-curated event schemas. RESIN-EDITOR’s unique features include hierarchical graph visualization, comprehensive source tracing, and interactive user editing, which significantly outperforms existing Information Extraction (IE) visualization tools in both IE result analysis and general model improvements. In our evaluation of RESIN-EDITOR, we demonstrate ways in which our tool is effective in understanding complex events and enhancing system performances. The source code, a video demonstration, and a live website for RESIN-EDITOR have been made publicly available.

## introduction complex events, such as an international negotiation or a disease outbreak, can take place over a prolonged period of time spanning from weeks to even months. typically, such complex events can be further deconstructed into many atomic sub-events, where each occurs at a specific time and place. fully modeling such complex events requires an understanding of the temporal, logical, and hierarchical connections among many sub-events, making the task of event modeling difficult for existing information extraction (ie) visualization and analysis tools. recent research efforts @xcite @xcite extend the event understanding paradigm to model eventevent relations and propose to use event schemas to guide the organization of complex event structures and predict possible future events. however, convenient visualization and editing tools specifically designed to handle such types of complex events are still largely undeveloped. a visualization and editing tool is essential for developing and improving an ie system. by utilizing such a tool, system developers can conduct a qualitative error analysis, which involves human analysts carefully investigating any missing or spurious errors, tracing these inaccuracies back to the original documents, and record any proposed modifications or suggestions. existing tools designed for atomic events @xcite are limited when directly applied to complex events, which are much more intricate with complicated event-event relations and multi-document multi-media source inputs. furthermore, prior analysis tools lack support for interactive user editing, which poses a challenge for model developers to comprehend any real feedback from actual users. to address the previously discussed challenges and accelerate research of complex events, we present the resin-editor, an analysis interface that allows users to visualize, analyze, and freely edit the multi-granularity graphical event narratives extracted from multi-document multimedia news clusters. compared with existing analysis interfaces, our tool has the following three significant advantages: 1) hierarchical graph visualization. our tool is able to generate a comprehensive visualization of the extracted hierarchical event graph (as shown in figures 1 and 2 ). this graph not only includes atomic events and entities, but also includes all event-event hierarchical, temporal, and logical relations of the entire complex event. such an informative visualization can offer human analysts a rapid and comprehensive understanding of any complex event, significantly improving the efficiency of our error analysis process. moreover, our graphical visualization is initialized with carefully-curated schemas, and the extracted events are grounded to the schema whenever possible, which improves the reliability of our error analysis. 2) comprehensive source tracing. our tool is able to show multimedia provenances for all extracted events, entities, and relations, including source documents, text spans, images, and bounding boxes. these trace-back provenances are presented in a clear-structured and tabular format, which greatly simplifies the users' task of understanding and evaluating the performance of the extraction system. 3) interactive user editing. most importantly, with our analysis tool, users are allowed to edit and manipulate almost every extracted element depicted in the visualized event graph and save their edited event graph for downstream usage. this includes not only the node attributes of events and entities, such as names, types, and description sentences but also the graphical structures, like the entity-entity relations and event-event temporal links. through open and direct interactions with intended users, our tool equips model developers with a comprehensive and precise understanding of actual user requirements. our main contributions can be summarized as: ## backend system overview the resin-editor system provides an interface for event graph visualization with the added capability of graph editing. the visualized graph is an instantiated schema graph, which results from matching an extracted event graph with an event schema. an event schema is a prototypical description of the usual progression of a particular type of complex event, which can be induced from historical news documents. in this section, we present an overview of how this event graph is generated, with the main components being multimedia event extraction, cross-document event organization, and event prediction in hierarchical schemas. ## graphical interface our graphical interface visualizes the instantiated schema graph after matching and prediction along with the source documents that provided provenance for the extracted events. the interface (figure 2 ) is organized in the panel hierarchy illustrated in figure 3 . the graph panel, which is the canvas for showing the event graph, is the primary access point. the overlay panels, such as the entity table, provenance, and mini-map, enhance the exploration experience with contextually relevant information or provide navigation support. ## experiments we present our empirical experiments with multiple document clusters to evaluate the effectiveness of our editor. evaluation setup we use two high-impact news scenarios disease outbreak and terrorist attacks with four newsworthy complex events, where the detailed statistics are shown in ## related work in the ie visualization domain, @xcite offer event-focused indexed documents, while others @xcite use temporal-relation entity graphs. however, both methods lack hierarchical event organization which leads to clutter in the graph display. additionally, despite presenting entity provenance triggers, they are not able to assist in amending extraction errors. in terms of schema visualization, @xcite introduce a graph-based interface tailored to depict schemas, emphasizing event order and entity relations. while the interface allows users to edit schemas, the user must come up with editing suggestions based on his/her own knowledge without any help from the interface. more recently, @xcite introduce a human-in-the-loop interface that employs pre-trained llms to help create schema graphs detailing event sequences in scenarios. whereas the use of llms can alleviate the manual effort of the user, the llm-generated schema is not grounded in real news documents and may be prone to hallucination. existing tools mainly focus on either ie extraction or schema visualization. inspired by the resin-11 system @xcite , which generates schema-guided event graphs from multi-modal news documents, we introduce a unified tool that integrates both the schema and the extracted event graph, providing a comprehensive visualization for complex event analysis. by matching the extracted event graph with the human-curated schema, we are able to effectively organize the complex event and guide users in providing informed feedback on both the ie results and the schema. ## conclusion and future work to the best of our knowledge, resin-editor is the first graphical user interface for editing and visualizing schema-guided event graphs. our tool assists users by highlighting the matching results, allowing them to fix discrepancies between event attributes and their argument entities and resolve disparities between extracted entities and their multimedia provenance. our results show that our solution outperforms the traditional method of error straight from their sources in primary ie tasks. while we recognize the importance of comparing with other tools, our current experimentation is limited due to time constraints. we plan to conduct extensive experiments in the future to validate our results further. in future work, we aim to develop a ranking system that considers the relevance of multimedia provenance so that users can quickly view the most related sentence/image. we also plan to integrate the event grounding tool, so that users can easily add new events by providing natural language descriptions, and the system will provide event type suggestions and fill in the argument template.