MMUTF : Multimodal Multimedia Event Argument Extraction with Unified Template Filling (2024)

Abstract: With the advancement of multimedia technologies, news documents and user-generated content are often represented as multiple modalities, making Multimedia Event Extraction (MEE) an increasingly important challenge. However, recent MEE methods employ weak alignment strategies and data augmentation with simple classification models, which ignore the capabilities of natural language-formulated event templates for the challenging Event Argument Extraction (EAE) task. In this work, we focus on EAE and address this issue by introducing a unified template filling model that connects the textual and visual modalities via textual prompts. This approach enables the exploitation of cross-ontology transfer and the incorporation of event-specific semantics. Experiments on the M2E2 benchmark demonstrate the effectiveness of our approach. Our system surpasses the current SOTA on textual EAE by +7% F1, and performs generally better than the second-best systems for multimedia EAE.

## introduction traditional event extraction (ee) consists of two tasks: event detection (ed) and event argument extraction (eae), which address event type identification and argument role extraction, respectively. traditional methods focus on individual modalities, such as text, images, or videos @xcite @xcite . however, focusing on single modalities results in incomplete event understanding, as up to 33% of visual arguments are absent in the accompanying text @xcite (cf. figure 1 ). as the creation of multimodal ee datasets is time-consuming and costly, recent advanced mee methods focus on (weak) image-text alignment based on available unimodal task-dependent and task-independent vision-language datasets @xcite @xcite . however, current multimedia eae models the international organization for migration [org] estimated that more than a quarter-of-a million migrants [per] crossed the libya-niger border [loc] last year. the eu [org] already has five migrant centers located ... are still based on simple classification techniques while ignoring cross-ontology transfer capabilities and event template semantics. to address this gap, we reformulate existing query-based (template filling) methods @xcite @xcite into a unified template filling framework, which enables us to utilize event templates as natural language prompts for different input modalities. first, our multimodal multimedia event argument extraction with unified template filling (mmutf) model exploits candidate structures, such as textual entities and visual objects, and connects them via query representations (i.e., argument roles) in a unified latent space. finally, these representations are used to match event argument roles with the corresponding candidates via queries extracted from event templates. ## method for event argument roles, the extraction task can be framed as a candidate-query matching problem, where argument roles serve as the queries. we can utilize the matching between candidates and queries to address the multimedia eae task with a unified template filling model. specifically, we aim to create candidate representations for either textual entities or image objects and compute a matching score for each query (i.e., argument role) in a given event template. foot_0 ## experiments datasets we conduct our experiments on the widely used m2e2 multmedia event extraction benchmark @xcite with 8 event types and 15 argument roles. the dataset contains 245 multimedia documents with 6,167 sentences and 1,014 images. it encompasses 1,297 textual events and 391 visual events, with 192 textual events and 203 visual events aligning to 309 multimedia events (cf. figure 1 ). since the m2e2 dataset does not provide a training split, we follow previous work @xcite @xcite and employ ace2005 and swig for the training phase. ace2005 @xcite consists of 33 textual event types and 36 argument roles. swig @xcite annotates visual events with 504 activity verbs and 1,788 semantic roles. following @xcite , we align both datasets with the m2e2 event ontology. metrics for evaluation, we follow previous studies on eae @xcite and use precision (p), recall (r), and f1-score (f1) as our evaluation metrics. in this work, we focus on the more challenging eae task but also report event classification results for completeness. baselines we compare our proposed approach with a wide range of state-of-the-art models. these models include flat @xcite , wase @xcite , clip event @xcite , unicl @xcite , camel @xcite , mgim @xcite , and umie @xcite . the majority of baselines set the focus on weak image-text alignment but neglect more sophisticated classification approaches w.r.t. eae. we provide a more comprehensive overview in a.1. experimental setup to ensure a fair comparison with the baselines, we utilize the clip-base model @xcite as visual encoder with a 16 × 16 patch size, and the t5-base @xcite model as textual encoder-decoder. we generate object candidates with yolov8 @xcite trained on the coco @xcite dataset and use entities as textual span candidates. our system is trained jointly for 5 epochs on the visual task and 25 epochs on the textual task with a batch size of 16 and 32, respectively. the varying number of epochs reduces the imbalance of modalities in the training datasets. for optimization, we ## conclusion in this work, we investigate the multimedia eae task and propose mmutf, a simple yet effective unified template filling approach that assigns candidates to corresponding argument roles. we encode entity candidates and object candidates from textual and visual inputs, respectively. then, we feed a natural language event template into the query decoder, which facilitates interactions with document contexts via cross-attention mechanisms. finally, candidate and query representations serve as the basis for computing candidate-query matching scores. our results demonstrate the effectiveness of the model and highlight the benefits of cross-ontology transfer and event semantic incorporation. this assumption is supported by srl transfer learning experiments, which we adapt to the multimedia eae task. future work will focus on incorporating additional textual and visual cues from input documents, integrating event detection, and exploring bi-encoder architectures to reduce inference time.