Grounding Partially-Defined Events in Multimodal Data (2024)

Abstract: How are we able to learn about complex current events just from short snippets of video? While natural language enables straightforward ways to represent under-specified, partially observable events, visual data does not facilitate analogous methods and, consequently, introduces unique challenges in event understanding. With the growing prevalence of vision-capable AI agents, these systems must be able to model events from collections of unstructured video data. To tackle robust event modeling in multimodal settings, we introduce a multimodal formulation for partially-defined events and cast the extraction of these events as a three-stage span retrieval task. We propose a corresponding benchmark for this task, MultiVENT-G, that consists of 14.5 hours of densely annotated current event videos and 1,168 text documents, containing 22.8K labeled event-centric entities. We propose a collection of LLM-driven approaches to the task of multimodal event analysis, and evaluate them on MultiVENT-G. Results illustrate the challenges that abstract event understanding poses and demonstrates promise in event-centric video-language systems.

## introduction event recognition is critical to how we understand the world. evidence suggests that even prelinguistic infants demonstrate a capacity for recognizing discrete events in real-world stimuli @xcite . natural language reflects this cognitive inclination towards event modeling through the various linguistic structures tied to event representations @xcite ; bohnemeyer et al., 2007), and significant work considers how to model events conveyed through these structures formally @xcite and through datadriven approaches @xcite . in comparison, there is less work considering the relationship between these event models and the what emergency-response where when who-affected 0:00 1:00 role: emergency-response entity: "fire" ocr: true confidence: ★★★★☆ (a) (b) (c) (d) (e) downstream homes in the coastal city of townsville are flooded and more than a thousand people have moved to safety and been evacuated from their homes. in some parts of townsville, salt-water crocodiles displaced by the floodwaters have been spotted, adding extra risk and danger. ## experiments the three-stage task introduced in section 3 is depicted in figure 5 . the task is difficult, as it requires extracting complex information from multiple modalities across highly varied domains. therefore, we propose a collection of approaches that ## conclusion in this paper we explore the problem of partiallydefined event extraction in noisy, multimodal content, and propose a multimedia event formulation for tackling it. we subsequently introduce multivent-g, a corresponding three-stage event extraction benchmark consisting of densely annotated news content building on the multivent dataset, and a set of potential methods for peforming the task stages. we report method performance on this benchmark, comparing zero-shot/few-shot llm based methods to more traditional fine-tuned approaches. this paper marks an important step towards robust ai agents that can synthesize complex amalgamations of natural stimuli and processed information with human-level ability. in future work, we hope to develop comprehensive systems that can address each stage of the event extraction task in tandem, allowing the outputs for each stage to be conditioned on the intermediate states and outputs of the other two stages. we believe that incorporating multilingual ocr systems into our pipeline would also improve performance, especially for videos not taken in predominantly english-speaking countries. finally, while we introduce a preliminary exploration of partiallydefined events in visual data, there is a wide range of ideas left to explore further on this topic, with the goals of better performance on event-centric vision benchmarks and a better understanding of human event understanding.