Movie101: A New Movie Understanding Benchmark (2023)

Abstract: To help the visually impaired enjoy movies, automatic movie narrating systems are expected to narrate accurate, coherent, and role-aware plots when there are no speaking lines of actors. Existing works benchmark this challenge as a normal video captioning task via some simplifications, such as removing role names and evaluating narrations with ngram-based metrics, which makes it difficult for automatic systems to meet the needs of real application scenarios. To narrow this gap, we construct a large-scale Chinese movie benchmark, named Movie101. Closer to real scenarios, the Movie Clip Narrating (MCN) task in our benchmark asks models to generate role-aware narration paragraphs for complete movie clips where no actors are speaking. External knowledge, such as role information and movie genres, is also provided for better movie understanding. Besides, we propose a new metric called Movie Narration Score (MNScore) for movie narrating evaluation, which achieves the best correlation with human evaluation. Our benchmark also supports the Temporal Narration Grounding (TNG) task to investigate clip localization given text descriptions. For both two tasks, our proposed methods well leverage external knowledge and outperform carefully designed baselines. The dataset and codes are released at https://github.com/yuezih/Movie101 .

## introduction the estimated number of visually impaired people worldwide was about 285 million by 2020, according to reports @xcite . while regulations are in place to ensure increased access for these audiences to experience the culturally dominant movies and tv shows on popular media platforms, technologies that provide them with genuine experience are becoming increasingly important. audio description (ad, also known as video description) * *corresponding author. to address this issue, attempts have been carried out to automate ad production. datasets of movies with ads are constructed to support the research on automatic ad generation, including the mpii-md dataset @xcite and m-vad dataset @xcite , with shotlevel ads or scripts aligned to the visual contents of movie. consequently, different solutions for automatic movie narrating have been proposed based on these datasets @xcite . however, existing benchmarks suffer from several limitations. firstly, there is a gap between the designed tasks and the actual movie narration scenario. these tasks mainly focus on generating single-sentence narrations for shots of a few seconds. they can not support the generation of coherent narrations for longer plots, which is critical for the visually impaired to better understand the movie, and the timestamps of these shots are carefully annotated, which are difficult to obtain for new movies in real application. meanwhile, these tasks treat the very distinctive movie narrating task as a normal video captioning task through some simplifications such as replacing role names with someone, resulting in the inability to connect roles to plots. secondly, these benchmarks evaluate the generated narrations with ngram-based metrics, which can over-penalize a semantically correct but textually inconsistent narration, especially when there is only one reference available. in addition, these existing datasets are all in english. however, about one-fifth of the world's population speaks chinese as their mother tongue, of whom more than 17 million are visually impaired @xcite . therefore, building a chinese movie narration benchmark is necessary. intending to address the limitations of the existing narrating benchmarks, in this work, we propose a new benchmark with 101 chinese movies for movie understanding, named movie101. we collect the movies from the barrier-free channel on xigua video platform foot_0 , where normal movies are remastered with ads. through automatic process and manual correction, we obtain the ads and actor lines from the raw videos. we crawl rich metainformation relevant to the movies as well. finally, movie101 contains 30,174 narration clips totaling 92 hours, with data samples as shown in fig. 1 . as our investigation shows that narrations mostly occur at those times when no actors are speaking (see appendix a), to achieve realistic movie narrating, we propose the movie clip narrating (mcn) task that requires a model to narrate where there are no lines. it brings a potential benefit for identifying where to narrate in an unlabeled new movie, since the timestamps of the actor lines are easily accessible foot_1 . meanwhile, in order for the audience to accurately comprehend the role-related plots, concrete role names should be contained in the generated narration. for the mcn task, we reorganize the movie101 dataset, merging the narration clips between two actor dialogues into a longer clip, to simulate real-scenario movie narrating. we thus obtain 14,109 long clips of variable length for narration generation. moreover, to better evaluate the quality of model-generated narrations, we conduct human evaluations and design a new metric specific to movie narrating, namely movie narration score (mnscore), which well aligns with human evaluation. in addition to the mcn task, our dataset also supports the temporal narration grounding (tng) task, which asks a model to locate target clips in the movie according to some text descriptions. for both tasks, we benchmark the performance of existing methods, and further propose our improved models by incorporating auxiliary external knowledge. in addition to mcn and tng tasks, movie101 can also potentially support other movie understanding tasks such as visual question answering and action recognition, etc. the main contributions of this paper are as follows: 1) we propose a new benchmark for movie understanding, movie101, with a large number of video-aligned text descriptions in chinese. 2) we propose two primary tasks, mcn and tng, and a new narrating evaluation metric mnscore, where mcn is more in line with the needs of actual movie narrating, while mnscore is more consistent with human evaluation. 3) we benchmark state-of-theart models and propose improved models enhanced by external knowledge for mcn and tng, respectively. we expect our proposed movie101 benchmark can inspire more explorations on narrating and understanding a whole movie. video captioning. as a classic vision and language task, the video captioning task requires a model to generate natural language descriptions for given videos. solutions for normal video captioning go through stages from pre-designed templates @xcite to sequence-to-sequence generation with deep neural networks @xcite . a challenging variant for this task is dense video captioning @xcite , which requires the generation of multi-sentence descriptions for long multievent videos. the two-stage generation approach, which firstly performs proposal detection on the video and then generates descriptions for each proposal separately, has been the dominant approach @xcite @xcite . recently, some works avoid event detection and generate paragraph descriptions directly based on the video, such as the one-stage paragraphing model (ovp) @xcite , obtaining competitive performance compared to previous works, inspired by which we propose our knowledge-enhanced movie narrating model. identity-aware video description that distinguishes different persons is more practical in real applications. @xcite attempt to achieve role-aware movie narrating by distinguishing different people using labels such as person1, person2, etc. however, it fails to generate concrete role names and falls short in terms of practicality. temporal sentence grounding. the temporal sentence grounding (tsg) task aims to localize the moment in a video based on a natural language query @xcite . a two-step pipeline has been the mainstream approach, which first produces a large number of moment candidates via sliding windows, then ranks them with their similarity to the query sentence. the following works try to improve the grounding performance by enhancing interaction between video and query modalities @xcite or introducing novel detection heads @xcite . specifically, for interaction methods, @xcite adopt an iterative alignment network (ia-net) to iteratively interact interand intra-modal features within multiple steps. @xcite explicitly decompose video and query into multiple structured hierarchies and learn finegrained semantic alignment among them. in this work, we propose to incorporate external knowledge based on the ia-net model structure. ## temporal narration grounding 5.1 task description to help people locate clips of interest during movie entertainment, an ai agent should be able to understand users' intentions and locate the target clips. to achieve this goal, we propose the temporal narration grounding (tng) task. given a clip narration as the query, tng aims to predict the starting and ending time of the clip in the whole movie. ## conclusion in this work, we propose movie101, a chinese large-scale video benchmark for movie understanding. to assist visually impaired people in enjoying movies, we propose a more realistic movie clip narrating task to address the automatic movie description issue and design a human-preferencecompatible metric mnscore for narrating evaluation. movie101 also supports the temporal narration grounding task, which is more challenging than the previous tsg benchmarks. furthermore, our experiments validate the importance of external knowledge including genres and roles for movie understanding. however, there is still a significant gap between our models and expert annotations. this reveals that further research endeavors are still needed to help visually impaired people enjoy movies by ai.