Multilingual Synopses of Movie Narratives: A Dataset for Vision-Language Story Understanding (2024)

Abstract: Story video-text alignment, a core task in computational story understanding, aims to align video clips with corresponding sentences in their descriptions. However, progress on the task has been held back by the scarcity of manually annotated video-text correspondence and the heavy concentration on English narrations of Hollywood movies. To address these issues, in this paper, we construct a large-scale multilingual video story dataset named Multilingual Synopses of Movie Narratives (M-SyMoN), containing 13,166 movie summary videos from 7 languages, as well as manual annotation of fine-grained video-text correspondences for 101.5 hours of video. Training on the human annotated data from SyMoN outperforms the SOTA methods by 15.7 and 16.2 percentage points on Clip Accuracy and Sentence IoU scores, respectively, demonstrating the effectiveness of the annotations. As benchmarks for future research, we create 6 baseline approaches with different multilingual training strategies, compare their performance in both intra-lingual and cross-lingual setups, exemplifying the challenges of multilingual video-text alignment. The dataset is released at:https://github.com/insundaycathy/M-SyMoN

## introduction computational story understanding aims to empower ai systems with the ability to delve into the intricacies of diverse stories, unlocking their deep semantics such as character motivations and intentions @xcite , event structures @xcite , and social relationships among story characters @xcite @xcite . in recent years, computation story understanding has garnered significant research interest @xcite @xcite @xcite and many story understanding tasks @xcite @xcite have emerged. story video-text alignment is a fundamental task of computational story understanding, which aims to find the best correspondence between a sequence of video clips and a sequence of sentences (see an alignment example in figure 1 ). different from traditional video-text retrieval that relies on keyword or temporal cue matching @xcite , story video-text alignment requires various story understanding abilities such as causal chain reasoning, mental state description, and long-range context understanding @xcite . establishing such video-text correspondence will facilitate many applications such as text-to-video generation @xcite , and visual story generation @xcite . thus, the story video-text alignment task has recently attracted increasing attention @xcite @xcite . however, a major obstacle of this task stems from the scarcity of annotated data, as it is costly and time-consuming to manually annotate sentencelevel alignments between text and clip sequences. the story understanding datasets used in previous studies are limited for several reasons. first, although the large scale movie description challenge (lsmdc) dataset @xcite manually aligns 158 hours of movies and audio descriptions, it only considers one-to-one matching between movie clips and audio descriptions and its audio descriptions are designed for visually impaired audiences, containing excessive details. second, the youtube movie summary (yms) dataset @xcite is small and only contains 6.7 hours of video and texts, and thus is typically employed for video-text alignment evaluation. lastly, hollywood movies and english narrations are dominant in existing story understanding datasets @xcite @xcite , neglecting the the example is from https://youtu.be/n5v9hzsyxpq , a summary of the movie 407 dark flight @xcite . to address the aforementioned limitations, we construct a multilingual video story dataset named multilingual synopses of movie narratives (m-symon). m-symon is sourced from movie recap videos on youtube and contains 13,166 videos spanning 7 languages and totalling 2,136 hours. furthermore, we manually annotated a portion of this dataset, providing exact video-text correspondence for 480 videos or 101.5 video hours. compared to lsmdc and yms, our annotated videotext alignment subset is large-scale, contains oneto-many matching and unmatched text or video clips, and covers 7 different languages. we further investigate the multi-lingual characteristics of the dataset, and make the following observations. first, in the intra-lingual setup, compared to translating all languages to english for training and inference, additional language-specific finetuning on weakly supervised data brings an average improvement of 5.4 percentage points across 7 languages. adding a small portion of the manual annotations further boosts performance. second, in the cross-lingual setup, for source and target languages that are linguistically similar (e.g., spanish to portuguese or french), the transfer performance is generally good; for source and target languages that are different (e.g., chinese to english or hindi), the transfer performance is quite limited. third, an out-of-domain evaluation on the yms dataset shows that training on the weakly supervised data from m-symon outperforms the state of the art methods by 12.3 and 13.2 percentage points on clip accuracy and sentence iou scores. moreover, adding manually annotated video-text alignment data further improves the performance by 2.4 and 3.0 percentage points, indicating the utility of our annotated alignment data. the main contributions of this work are summarized as follows: ## task formulation given a consecutive sequence of video clips @xmath0, v 2 , ..., v m ) and a consecutive sequence of sentences @xmath1, t 2 , ..t n ), the video-text alignment task aims to find the alignment between these two sequences by learning a function f that maps each input sentence t i to its corresponding video clips: in the task formulation, we make the simplifying assumption that multiple video clips can be aligned with one sentence, but a clip cannot be aligned with more than one sentence. the reason is that a video clip is typically very short (around 2.4 seconds) and clips outnumber sentences, so it is rare that one clip aligns with more than one sentence. ## dataset construction in this section, we describe the details of constructing the m-symon dataset. ## methodology this section first details our base model for videotext alignment and then introduces its several variants with different multilingual training strategies. ## experiments in this section, we conduct extensive experiments in intra-lingual, cross-lingual, and out-of-domain setups to show the usefulness of m-symon. ## conclusion in this paper, we introduced m-symon, a largescale multilingual video story dataset. it contains 13,166 movie summary videos from 7 languages, featuring 480 videos with human-annotated videotext correspondence. we established several multilingual multimodal baselines to benchmark m-symon. experimental results show the value of m-symon for video story understanding. its size, multilingual nature, and rich alignment annotations make m-symon a valuable contribution.