Three Stream Based Multi-level Event Contrastive Learning for Text-Video Event Extraction (2023)

Abstract: Text-video based multimodal event extraction refers to identifying event information from the given text-video pairs. Existing methods predominantly utilize video appearance features (VAF) and text sequence features (TSF) as input information. Some of them employ contrastive learning to align VAF with the event types extracted from TSF. However, they disregard the motion representations in videos and the optimization of contrastive objective could be misguided by the background noise from RGB frames. We observe that the same event triggers correspond to similar motion trajectories, which are hardly affected by the background noise. Moviated by this, we propose a Three Stream Multimodal Event Extraction framework (TSEE) that simultaneously utilizes the features of text sequence and video appearance, as well as the motion representations to enhance the event extraction capacity. Firstly, we extract the optical flow features (OFF) as motion representations from videos to incorporate with VAF and TSF. Then we introduce a Multi-level Event Contrastive Learning module to align the embedding space between OFF and event triggers, as well as between event triggers and types. Finally, a Dual Querying Text module is proposed to enhance the interaction between modalities. Experimental results show that TSEE outperforms the state-of-the-art methods, which demonstrates its superiority.

## introduction event extraction (ee) is a fundamental task which aims to recognize the event structure from texts @xcite @xcite . recent years have witnessed the booming of the multimodal event extraction (mee). mee @xcite @xcite extends ee similar trump held his own eleventhhour rally at an airfield. after trump used the song at his recent rally in tulsa. ## conclusion in this paper, we propose a three stream multimodal event extraction framework that explores the utilization of motion representations in textvideo based multimodal event extraction (tvmee) tasks. optical flow features are extracted from videos as motion representations to incorporate with other modalities. to improve alignment among feature representations, we propose a multilevel event contrastive learning module. a dual querying text module is also designed to help en-hance the interaction between different modalities. tsee achieves the state-of-the-art results on two datasets, demonstrating the effectiveness of our framework. in future work, we will explore the utilization of large language model (llm) in fusing modality features to boost timee performance.