MUMOSA , Interactive Dashboard for MU lti- MO dal Situation Awareness (2024)

Abstract: enter abstract here

## introduction after a significant incident or crisis, how do investigators determine and assess what happened when in order to produce a report with clear evidence from the sequence of events in detailing lessons learned? how do communities prepare their responders to handle similarly complex, critical situations that may come their way in the future? some crisis response procedures are well-established for specific situations, e.g., an initial fire suppression response to a wildfire foot_0 , so responders can be consistently trained in advance and investigators know what to look for afterwards. but other times, the crisis is so sudden and unexpected that established lines of communication struggle to convey up-todate information. following these unforeseen circumstances, both groups, investigators and responders, have a shared need to understand the various types of information about events in the evidence collected and analyzed for post-crisis reports. the field of information extraction (ie) within computational linguistics has led the way since the late 1980's applying symbolic, then statistical, and most recently neural methods to natural language texts to identify the types of essential elements of information needed for such reports, including entities, relations, and events @xcite . most recently, with neural models such as llms and vlms that can bring together multiple modalities to provide additional sources of information about events, there is now the opportunity to leverage various combinations of multimodal event information to support investigators in combing through text and photographic evidence for report writing and to train responders in preparing to handle such information in the future. furthermore, the recent development of schema graphs with access to over 3k event types @xcite and 3d reconstruction methods for scenario simulation from as little as 24 images, e.g., @xcite , users can now have hands-on access to interfaces to visualize and annotate complex events online, as they learn from available evidence and documentation what has happened over the course of those events. in this paper, we introduce our approach to bringing together these various resources in an interactive, multi-modal situation awareness (mu-mosa) dashboard for complex event understanding, ultimately in support of users' real-time event situational awareness (sa) and decision-making during a crisis. for a specific role, such as a first responder onsite or incident coordinator at an emergency operations center, the specifics of their sa will be determined by tasks and decisions for their job. however in all cases, their sa will entail "the perception of the elements in the environment within a volume of time and space, the comprehension of their meaning and the projection of their status in the near future." @xcite 2 figure 1 : users engage with the mumosa dashboard through (a) interactive multi-modal q/a, selecting the incident with source date and entering questions or directed requests in natural language. they may then choose to examine the source data for the system response via panels in (b) textual and visual evidence, or continue engaging the dashboard via q/a. the selected incident can also be explored in its entirety across events via panels in (c) schemas and simulation evidence. the mumosa dashboard's panels are intended to provide for user perception of essential elements of information (sa level 1) about the complex event they select in the form of multi-modal evidence. the workflow design enables users to explore and compare information across evidence panels, as well as edit and annotate the content of the complex schema graphs and scenario simulation. this aims to support them in interpreting and retaining the panel content with multiple elements of information, and in building their own narrative of the complex event (level 2). users engage in interactive multi-modal question-answering (q/a) and exploration of events and simulated environments. the user initializes the dashboard for the complex event of interest and time frame. after inputting a question, they receive a text answer and access to supporting evidence from text documents and photographic visuals. they may simultaneously designated as progressive levels 1, 2, and 3 of sa. in this paper, we select one unexpected crisis to focus on-the ohio 2023 train derailment-as we describe the design and capabilities of the dashboard for the following applications: ## background we briefly overview here existing resources that we build on in four panels of our dashboard. textual evidence. to show textual evidence, we leverage text-based q/a and frequently asked questions (faqs), where a set of common user questions pertaining to a particular topic are compiled into an accessible list where the user may look up answers if their query is common @xcite . prior work has shown how to find these matches in a dialogue q/a. by using statistical text classifiers, @xcite compared a user question in real-time against a distributions of common user questions paired with responses. after successfully matching the input question to the pre-processed question set, the paired answer was returned. the same statistical classifier was leveraged in human-robot dialogue extending beyond the constraint of a 'question,' allowing for different types of frequently issued robot-directed commands @xcite . the matching was re-implemented in @xcite using sentence embeddings and cosine similarity to find close matches between vectorized input and pre-stored questions, and showed significant improvement in accuracy over the statistical approach in the same domain of human-robot dialogue. section 3.1.1 covers our work incorporating a modified version of this approach with llms in responding to user input. ## mumosa dashboard evidence panels the user starts their interactions with the mu-mosa dashboard by initializing it for a particular incident and date of interest. they can then proceed by posing questions about events in the incident and exploring the source data evidence provided with the system responses along with full incident visualizations in the dashboard panels. we focus here and in section 4 on interactions specifically for forensic use cases of the dashboard with pre-processed data for investigators and first responders. we postpone till section 5 discussion of future dashboard research for real-time conditions with dynamic changes to both data availability and user information needs. the intent for the dashboard panels is to provide users with essential elements of information (sa level 1). user workflow across panels for building their understanding of the sequences of events within the incident (sa level 2) will be addressed in the section that follows. ## user workflow the mumosa dashboard represents a powerful tool for users to query events by enabling q/a over multi-modal data sources where the modalities offer complementary supporting evidence. while the evidence in the individual panels provides users with essential elements of information (sa level 1), the mumosa dashboard itself provides for an easy-access workflow to detect and compare events across panels and modalities. by using all the panels together, the dashboard provides unique opportunities for users to iterate in their information foraging and annotate the underlying data to enhance their understanding of the sequence of events (sa level 2). in our forensic use of this case study, a user may want to understand the initial response of the derailment by asking the question, "what time did the derailment in east palestine happen?" to which the answer was "about 9 p.m. est friday" with the textual evidence showing an article published on february 4, 2023. this answer may prompt the user towards several lines of inquiry, one of which may be to ask "was it hard to see at night during the initial incident response?" this might return a photograph of the nighttime scene in the visual evidence panel, which may in turn inspire another question that could be answered by the textual evidence panel, e.g., "what challenges did the first responders face in the dark?" we also readily imagine that the open-ended workflow with schema and simulation panels will elicit follow-up questions. when exploring the schema view, users can navigate the hierarchical structure of events related to the train derailment. for example, they might explore sub-events under investigation, as seen figure 6 in appendix a. this exploration may prompt a user to follow-up with "what criminal charges are being reported?" the response in the schema evidence would highlight the node of interest within the schema, and search stored documents for supportive textual evidence. giving users access to the simulation affords them greater situational awareness to ask questions pertaining to accessibility and route planning, obstacle and target identification, and scene overview. examination of the simulation might prompt not only new questions, but the ability to display answers within the simulation using annotations. the user may ask the question "where is the immediate danger?" and the simulation would highlight the clusters of train cars in orange and red, as seen in figure 3 . this may be followed up by "what buildings are in immediate danger?" and would highlight the buildings in green (also figure 3 ). in this way, the simulation view displays portions of visual evidence to the user without constraining their viewpoint to the original camera pose. 5 discussion: toward real-time event tracking the mumosa dashboard currently aims to serve as an interactive forensic resource, providing support to post-crisis incident investigations and training exercises for first responders at sa levels 1 and 2. the questions we have included above showcase how users may search for information looking back at events within an incident across modalities. the evidence supplied in one modality may inspire new questions or may lead to further insights in conjunction with evidence from another modality. with this groundwork in place, we now shift our discussion to how we envision the dashboard will support real-time crisis responses. ## related work text-based search methods, begun in the 1950's, have been central to modern information retrieval systems @xcite . most recently, generative ai has been leading a new human-written caption "this centrifuge separates solid waste from liquid wastewater in holding bins to determine whether the waste is hazardous and then disposed properly." alt-text "huge machine called a centrifuge separates the solid waste." text in image "strad" and "sb162" generated description from idefics3-8b-llama3 (prompt: "describe this image") "the object is most likely a type of pump station designed for heavy-duty use, probably utilized in industries like construction, mining, or wastewater management due to its robust build and extensive piping setup. pumps labeled with identifiers like sb168 often denote standardized models within these sectors, known for their reliability and durability. given the context clues provided by the surrounding environment (industrial equipment, trees), we infer that this equipment plays a critical role in managing fluid flow efficiently across different sites where liquid transfer is necessary." (see appendix c for the full generated output) we know of no other research that combines information for user access across modalities as we have within an interactive dashboard. mu-mosa provides an interactive, multi-modal interface where users can iteratively forage for answers about complex events to meet their information requirements. our approach, by retaining a dialogue history with text and visual evidence for documenting user searches, paves the way to building an ai agent-based system @xcite . event detection has recently been expanded by novel methods of embedding and extracting events across modalities from multimedia sources. for example, by constructing shared semantic vector spaces for texts and images @xcite , systems can generate text descriptions of events detected in images where only objects have been identified @xcite . image retrieval has seen advances by using global features @xcite , augmenting query or image vectors @xcite , and general purpose vlms @xcite . our faq approach on documents and images is intended to achieve high accuracy to support our investigator users looking forensically at data, and serve as a strong feasibility test in bringing together the evidence panels. we also give users access to the document and image collections using the semantic search ranking to 10 https://blogs.microsoft.com/blog/2023/02/07/reinventingsearch-with-a-new-ai-powered-microsoft-bing-and-edgeyour-copilot-for-the-web enable users to forage in a less constrained way by exposing the evidence directly to the user. the automated construction of regular patterns of events from news reports, where the task of event schema induction applies, continues to challenge researchers @xcite . the innovation of building path language models by connecting shared arguments across events within instance graphs has provided for more complete schema induction @xcite . recent advances in 3dgs for 3d reconstruction have begun to support natural language querying of a scene @xcite , however these approaches presently only highlight segmentation masks based on keywords, e.g., 'train cars.' there is no framework in place to support interactive querying of the 3d reconstruction from natural language questions. in order to understand that the question we pose "where is the immediate danger?" refers to the train cars requires greater understanding of 'danger' in the context of the simulation. ## conclusion our mumosa dashboard aims to provide a user with level 1 and 2 situational awareness for understanding essential elements of information and complex events by uniting complimentary modalities and interactions. we further envision how the dashboard will support real-time crisis response (sa level 3). by integrating document-based q/a, visual evidence retrieval, event schema visualization, and 3d scene simulation, our dashboard offers a comprehensive solution for complex event understanding. this multifaceted approach not only supports various levels of situational awareness, from initial perception to comprehensive understanding, but also provides a flexible, future-ready framework that can evolve with advancements in ai and data processing technologies.