Benchmarking and Improving LVLM s on Event Extraction from Multimedia Documents (2025)

Abstract: The proliferation of multimedia content necessitates the development of effective Multimedia Event Extraction (M²E²) systems. Though Large Vision-Language Models (LVLMs) have shown strong cross-modal capabilities, their utility in the M²E² task remains underexplored. In this paper, we present the first systematic evaluation of representative LVLMs, including DeepSeek-VL2 and the Qwen-VL series, on the M²E² dataset. Our evaluations cover text-only, image-only, and cross-media subtasks, assessed under both few-shot prompting and fine-tuning settings. Our key findings highlight the following valuable insights: (1) Few-shot LVLMs perform notably better on visual tasks but struggle significantly with textual tasks; (2) Fine-tuning LVLMs with LoRA substantially enhances model performance; and (3) LVLMs exhibit strong synergy when combining modalities, achieving superior performance in cross-modal settings. We further provide a detailed error analysis to reveal persistent challenges in areas such as semantic precision, localization, and cross-modal grounding, which remain critical obstacles for advancing M²E² capabilities.

## introduction event extraction (ee) is a fundamental task in nlp, aiming to identify structured events from unstructured data @xcite . it typically involves two subtasks: event detection (ed), which detects the event trigger words and classifies their event type, and event argument extraction (eae), which extracts entities and labels their argument roles. traditional ee methods have primarily focused on a single modality, either text @xcite @xcite , image @xcite @xcite , or video @xcite @xcite . in the era of digital media, content like news articles often contains both textual and visual modalities, and some arguments presented in images may not appear in the accompanying text. this has prompted the rise of multimedia event extraction (m²e², @xcite , targeting to extract information from combined texts and images (see figure 1 ). prior works either rely on limited hand-crafted features or require expensive annotation efforts. meanwhile, large vision-language models (lvlms) have demonstrated strong cross-modal reasoning abilities @xcite , but their utility for m²e² remains underexplored. motivated by this gap, we present the first systematic evaluation of lvlms on the m²e² dataset. evaluations are conducted across six subtasks for ed and eae in text-only, image-only, and crossmodal settings, under both few-shot and fine-tuning paradigms. our key findings reveal that: (1) fewshot lvlms demonstrate strong performance compared to specialized baselines in visual-only ee but lag behind established pre-trained language models (plm) baselines textually. (2) fine-tuning lvlms dramatically improves their capabilities, enabling smaller models to surpass strong textual baselines and achieve state-of-the-art results. (3) utilizing lvlms on multimedia ee consistently yields the best results, showcasing the effective cross-modal synergy of lvlms. (4) despite performance improvements, persistent challenges exist across all modalities, including difficulties with semantic discrimination, localization, effective crossmodal grounding, and safety filter behaviors. multimedia event extraction combines textual and visual modalities to improve robustness, employing fusion strategies, such as feature-level concatenation @xcite , shared latent spaces @xcite , and decision-level aggregation. recent frameworks such as wase @xcite , clip-event @xcite , and camel @xcite incorporate weak supervision or inject structured event schemas to improve multimodal grounding. instruction-tuned models like umie @xcite and video-text paired learning @xcite enhance event representation across modalities. however, despite the success of generative models in event understanding and reasoning tasks @xcite , their potential in the critical m²e² remains underexplored. we pioneer this field by utilizing lvlms to m²e², offering valuable analysis and insights for future research. ## problem formulation we focus on the textual, visual, and multimedia ee defined by @xcite . each event includes a trigger and one or more arguments. formally: (1) event trigger (τ ) is a verb or noun in the text indicating the occurrence of the event. (2) event argument (α) plays a role in the event, either a text entity, a temporal expression, a numerical value, or a detected object in the image. (3) argument role (ρ) is the semantic relationship linking an argument α to its event mention. our two main tasks over a multimodal document are as follows: event detection seeks to detect all event mentions {em 1 , . . . , em n } within the input context. each em i is assigned a category c @xmath0: ## conclusion we present the first comprehensive evaluation of lvlms for m²e², assessing their performance in both few-shot and fine-tuning settings. our results reveal a critical interplay between modality and training strategy: few-shot lvlms excel on visual tasks but falter in the texts, a gap that is dramatically closed by fine-tuning. ultimately, our work confirms that lvlms excel in combining text and vision modalities, creating powerful cross-modal synergy that yields the best performance, highlighting the significant potential of adapted lvlms for complex multimedia understanding. we further highlight the persistent challenges in our error analysis, including semantic precision, localization, and cross-modal grounding, which we will design more robust architectures to resolve in future research.