Cross-media Structured Common Space for Multimedia Event Extraction (2020)

Abstract: We introduce a new task, MultiMedia Event Extraction, which aims to extract events and their arguments from multimedia documents. We develop the first benchmark and collect a dataset of 245 multimedia news articles with extensively annotated events and arguments. We propose a novel method, Weakly Aligned Structured Embedding (WASE), that encodes structured representations of semantic information from textual and visual data into a common embedding space. The structures are aligned across modalities by employing a weakly supervised training strategy, which enables exploiting available resources without explicit cross-media annotation. Compared to uni-modal state-of-the-art methods, our approach achieves 4.0% and 9.8% absolute F-score gains on text event argument role labeling and visual event extraction. Compared to state-of-the-art multimedia unstructured representations, we achieve 8.3% and 5.0% absolute F-score gains on multimedia event extraction and argument role labeling, respectively. By utilizing images, we extract 21.4% more event mentions than traditional text-only methods.

## introduction traditional event extraction methods target a single modality, such as text @xcite , images @xcite or videos @xcite @xcite . however, the practice of contemporary journalism @xcite distributes news via multimedia. by randomly sampling 100 multimedia news articles from the voice of america (voa), we find that 33% of images in the articles contain visual objects that serve as event arguments and are not mentioned in the text. take event from text, but can extract the vehicle argument only from the image. nevertheless, event extraction is independently studied in computer vision (cv) and natural language processing (nlp), with major differences in task definition, data domain, methodology, and terminology. motivated by the complementary and holistic nature of multimedia data, we propose multimedia event extraction (m 2 e 2 ), a new task that aims to jointly extract events and arguments from multiple modalities. we construct the first benchmark and evaluation dataset for this task, which consists of 245 fully annotated news articles. we propose the first method, weakly aligned structured embedding (wase), for extracting events and arguments from multiple modalities. complex event structures have not been covered by existing multimedia representation methods @xcite @xcite , so we propose to learn a structured multimedia embedding space. more specifically, given a multimedia document, we represent each image or sentence as a graph, where each node represents an event or entity and each edge represents an argument role. the node and edge embeddings are represented in a multimedia common semantic space, as they are trained to resolve event co-reference across modalities and to match images with relevant sentences. this enables us to jointly classify events and argument roles from both modalities. a major challenge is the lack of multimedia event argument annotations, which are costly to obtain due to the annotation complexity. therefore, we propose a weakly supervised framework, which takes advantage of annotated uni-modal corpora to separately learn visual and textual event extraction, and uses an image-caption dataset to align the modalities. we evaluate wase on the new task of m foot_0 e 2 . compared to the state-of-the-art uni-modal methods and multimedia flat representations, our method significantly outperforms on both event extraction and argument role labeling tasks in all settings. moreover, it extracts 21.4% more event mentions than text-only baselines. the training and evaluation are done on heterogeneous data sets from multiple sources, domains and data modalities, demonstrating the scalability and transferability of the proposed model. in summary, this paper makes the following contributions: ## conclusions and future work in this paper we propose a new task of multimedia event extraction and setup a new benchmark. we also develop a novel multimedia structured common space construction method to take advantage of the existing image-caption pairs and singlemodal annotated data for weakly supervised training. experiments demonstrate its effectiveness as a new step towards semantic understanding of events in multimedia data. in the future, we aim to extend our framework to extract events from videos, and make it scalable to new event types. we plan to expand our annotations by including event types from other text event ontologies, as well as new event types not in existing text ontologies. we will also apply our extraction results to downstream applications including cross-media event inference, timeline generation, etc.