Joint Multimedia Event Extraction from Video and Article (2021)

Abstract: Visual and textual modalities contribute complementary information about events described in multimedia documents. Videos contain rich dynamics and detailed unfoldings of events, while text describes more high-level and abstract concepts. However, existing event extraction methods either do not handle video or solely target video while ignoring other modalities. In contrast, we propose the first approach to jointly extract events from both video and text articles. We introduce the new task of Video MultiMedia Event Extraction and propose two novel components to build the first system towards this task. First, we propose the first self-supervised cross-modal event coreference model that can determine coreference between video events and text events without any manually annotated pairs. Second, we introduce the first cross-modal transformer architecture, which extracts structured event information from both videos and text documents. We also construct and will publicly release a new benchmark of video-article pairs, consisting of 860 video-article pairs with extensive annotations for evaluating methods on this task. Our experimental results demonstrate the effectiveness of our proposed method on our new benchmark dataset. We achieve 6.0% and 5.8% absolute F-score gain on multimodal event coreference resolution and multimedia event extraction.

## introduction traditional event extraction methods target a single modality, such as text, images, or videos. however, real-world multimedia (e.g. online news) features content in multiple modalities which collectively convey a cross-modal narrative. as a consequence, components of events described by the document may lie jointly or solely in either the textual or visual modalities. by randomly watching 100 videos and associated articles from bbc official youtube * equal contribution. ## minute video article sentences video segments stage 2: joint video and text event extraction multimedia event frame conflict.attack event type officers(text only) attacker vans instrument protestors . target (street) (videoonly) place officers drove two vans into protesters who stood in front. ## conclusions we have introduced a novel task vm 2 e 2 -given a video with a paired article, our first goal is to find coreferenced events across modalities. also, our task requires extracting the event type and argument roles from both modalities. 860 video-article pairs were labeled to support this task. we developed a novel self-supervised multimodal network that learns a common embedding space by processing local (object region) and global (video level) se-mantic relationships to perform multimodal event coreference resolution. in addition, we present a new architecture jmmt that jointly extracts events and arguments from both modalities using an encoder-decoder-based multimodal transformer. our extensive experiments on multiple settings show that considering region information and a joint transformer for both modalities is essential for good performance on the two subtasks in vm 2 e 2 . our dataset collection pipeline and approach can be extended to more scenarios such as instructional videos and other videos that contain video-article pairs for extracting multimodal events across both modalities.