Coreference by Appearance: Visually Grounded Event Coreference Resolution (2021)

Abstract: Event coreference resolution is critical to understand events in the growing number of online news with multiple modalities including text, video, speech, etc. However, the events and entities depicting in different modalities may not be perfectly aligned and can be difficult to annotate, which makes the task especially challenging with little supervision available. To address the above issues, we propose a supervised model based on attention mechanism and an unsupervised model based on statistical machine translation, capable of learning the relative importance of modalities for event coreference resolution. Experiments on a video multimedia event dataset show that our multimodal models outperform text-only systems in event coreference resolution tasks. A careful analysis reveals that the performance gain of the multimodal model especially under unsupervised settings comes from better learning of visually salient events.

## introduction with the advance of internet, news articles nowadays are becoming increasingly multimodal. for example, news about a recent launch of a spacex rocket may contain the traditional text article, a drone video of the launch and a narration from the talking head or reporter. such a diverse source of information provides us a multi-faceted understanding of the newsworthy event, and helps us better follow the content of the article. event coreference resolution, the task of resolving whether two words or phrases called mentions in the article refer to the same real-world event or not, is the first step toward such an understanding. while human can efficiently and accurately identify the coreference relations when reading an article, this task has shown to be quite challenging to machines. the state-of-the-art coreference resolution system not only accepts textual input, but typically requires over 4000 annotated articles with more than 400 words fully annotated with mention span and coreference relation to train @xcite . further, since event coreference resolution requires careful reading and deep understanding of the text, it becomes prohibitively expensive to annotate for low-resource languages. another issue with the current system is that the model lacks a way to incorporate real-world knowledge typically required for coreference. one potential way to incorporate such knowledge is to consider an alternative modality -videos. a fascinating aspect of multimodal event coreference resolution is that the correlation between the visual modality and the text can be quite weak, and the text tends to contain most of the information for resolving coreference. nevertheless, the meanings of the event triggers often require additional information sources to resolve. for example, we never know an "attack" is a physical one or verbal one without seeing or imagining a particular scene based on additional contextual constraints, or it is hard to know whether "chant" and "protest" co-refer without knowing the event in the text is a demonstration. but if a video of the relevant events is played, we can immediately understand the relations between the trigger words. in this paper, we answer the following research question: how to leverage weakly aligned multimodal data for event coreference resolution system, especially in an unsupervised setting? ## related work motivated by event coreference resolution for low-resource languages, unsupervised event coreference resolution algorithms have been proposed @xcite ; bejan video m2e2 pairwise f1 muc b 3 ceafe full model 57.7±0.7 62.8±0.4 60.8±0.5 62.5±0.4 -textual mode 50.7±0.2 53.0±0.2 54.0±0.1 55.2±0.1 ∆ -7.0 -9.8 -6.8 -7.3 -trigger mode 47.7±1.5 53.4±0.8 51.5±0.7 53.0±0.8 ∆ -10.0 -9.4 -9.3 -9.5 -attribute mode 55.1±0.4 61.3±0.3 59.5±0.4 61.7±0.3 ∆ -2.6 -1.5 -1.3 -0.8 ## conclusions in this work, we proposed one supervised and one unsupervised model for multimodal event coreference resolution, which outperform the text-only baselines on a realistic multimedia event dataset.