M ovie UN : A Dataset for Movie Understanding and Narrating (2022)

Abstract: Automatic movie narration generation and narration grounding are very important to provide a true movie experience for the blind and visually impaired. To tell the movie story well, it is necessary to mention plot-related details (such as character names) and keep the narrations in a plot coherent. Taking these two points into consideration, we construct a Chinese large-scale video benchmark from 101 movies for Movie Understanding and Narrating (MovieUN) to support the Movie Clip Narrating (MCN) task and Temporal Narration Grounding (TNG) task. We split movies in MovieUN into movie clips according to plots, and pair them with corresponding narrations provided by the movie narrators. Ultimately, the TNG task involves 3,253 long video clips totaling 179 hours. The MCN task contains 33,060 video clips totaling 105 hours. We benchmark state-of-the-art video captioning models and temporal grounding models in MCN and TNG tasks, respectively. Furthermore, to accurately comprehend plots of different characters, we propose methods to incorporate portraits of actors as external knowledge in both tasks. The experiment results demonstrate the effectiveness of our proposed methods. The dataset and codes are released at https://github.com/yuezih/MovieUN.

## introduction the estimated number of visually impaired people worldwide is about 285 million by 2020, according to reports @xcite . while regulations are in place to ensure increased access for these audiences to experience the culturally dominant movies and tv shows on popular media platforms, technologies that provide them with genuine experience are becoming increasingly important. audio description (ad, also known as video description) is a form of such technology intended for visually impaired audiences to experience the movie or tv show by hearing what is happening on-screen. however, producing movie narration scripts is not trivial, often requiring a professional writer to oversee the original movie. the high cost of narration generation @xcite greatly hinders the production of movies with ad and thus limits the opportunities for visually impaired users to experience movies. to address this issue, attempts have been carried out to automate ad production. datasets of movies with ad are constructed to support the research on automatic ad generation, including the mpii-md dataset @xcite and m-vad dataset @xcite , with shot-level ads or scripts aligned to the movie visual contents. consequently, different solutions for automatic movie narrating have been proposed based on these datasets @xcite . however, these efforts mainly focus on generating a single sentence narration for a shot of a few seconds, which gaps with the actual movie narration scenario. shot-level single-sentence narration generation has some obvious deficiencies. first, it does not consider whether the shot contains something worth narrating and just describes all shots indiscriminately. second, it does not consider the context of the shot, and thus if simply splicing all the shots together to form the narration of the plot segment, there will be apparent redundancy and incoherence problems. we consider that a highquality narration should be generated from a plot segment (which we define as a movie clip) rather than the smallest unit of shot in a movie. therefore, in this paper, we focus on the movie clip narrating task. moreover, these existing datasets are all in english. however, about one-fifth of the world's population speaks chinese as their mother tongue, of whom more than 17 million are visually impaired @xcite . contrary to the huge demand, the accessibility of chinese media content @xcite to visually impaired audiences is minimal. for chinese movie narrating, the english datasets are of little help due to cultural differences and poor translation quality, given the complexity of discourse in movie narrations. therefore, building a chinese movie narration dataset is urgent and meaningful. intending to support research on clip-level movie narrating and to fill in the gap in the chinese movie narration datasets, in this work, we propose a new chinese movie narration dataset for movie understanding and narrating, named movieun, where movies are collected from the barrier-free channel on xigua video platform foot_0 . we crawl 101 movies with ads in total. as the narrations are in audio format, we leverage the speech recognition tools to automatically transcribe the narration audio. we then design a manual correction procedure to filter the errors based on the automatic transcriptions. we crawl rich meta information relevant to the movie as well. finally, movieun contains 101 movies with 33,060 clip-level narrations totaling 105 hours, with data samples as shown in 2 related works datasets. existing datasets to support the automatic narration generation task include m-vad @xcite and mpii-md @xcite . m-vad, which is collected based on an automatic ad segmentation and alignment method, contains 47k videos from 92 dvds, with an average length of 6.2s, each with an aligned narration. mpii-md contains 68k videos from 94 movies with an average length of 3.9s, about half of which come with paired scripts and the other half with paired ads. in addition to movies, tv shows are also good data sources for automatic narration generation. @xcite , a variant of tv show retrieval (tvr) @xcite . it contains 11k short videos averaging 9.1s in length, and 26k captions describing the visual content, dialogues and subtitles. all the existing datasets are in english. video captioning. video captioning (vc), as a classic vision and language task, has received much research attention in recent years. early works identify visual objects from videos and generated descriptions using pre-designed templates @xcite . with the advancement of deep learning, efficient video feature aggregation and high-quality description generation have been made possible @xcite . compared to the traditional video captioning task, dense video captioning is more challenging as it requires generating descriptions of multiple sentences for multiple events in a long video. the two-stage generation approach, which firstly performs proposal detection on the video and then generates descriptions for each proposal separately, has been the dominant approach @xcite @xcite . recently, some works avoid event detection and generate paragraph descriptions directly based on the video @xcite , obtaining competitive performance compared to previous works. inspired by the one-stage paragraph generation work, we propose our knowledge-enhanced movie narration generation model. identity-aware video description that distinguishes different persons is more practical in real applications. park et al. @xcite attempts to achieve roleaware movie narrating by distinguishing different people using labels such as person1, person2, etc. however, it fails to generate concrete role names and falls short in terms of practicality. temporal sentence grounding. the temporal sentence grounding task aims to localize the moment in a video based on a natural language sentence @xcite . a two-step pipeline has been the mainstream approach, which first produces a large number of moment candidates via sliding windows, then performs final localization based on the similarity between each candidate and the query sentence. the following works try to improve the grounding performance by enhancing interaction between video and query modalities @xcite or introducing novel detection heads @xcite . specifically, for interaction methods, @xcite adopt an iterative alignment network (ia-net) to iteratively interact interand intra-modal features within multiple steps. li et al. (2022) explicitly decompose video and query into multiple structured hierarchies and learns finegrained semantic alignment among them. for detection heads, zhang et al. (2020) proposes a twodimensional map, one dimension indicates the starting time of a moment, and the other indicates the end time, to model the temporal relations. lei et al. (2021) adopt a transformer encoder-decoder model to directly generate moment proposals, which uses a detr-based (carion et al., 2020) detection head. in this work, we propose incorporating external knowledge based on the ia-net model structure. ## conclusion in this work, we propose movieun, a chinese large-scale video benchmark for movie understanding and narrating. to assist visually impaired people in enjoying movies, movieun supports two challenging tasks, namely movie clip narrating (mcn) and temporal narration grounding (tng). in both tasks, we design role-aware models that outperform strong baselines. furthermore, our experiments validate the importance of movie genres and external actor portraits for movie understanding and narrating. however, there is still a significant gap between our models and expert annotations. this reveals that further research endeavors are still needed to help visually impaired people enjoy movies by ai.