Synchronized Video Storytelling: Generating Video Narrations with Structured Storyline (2024)

Abstract: Video storytelling is engaging multimedia content that utilizes video and its accompanying narration to share a story and attract the audience, where a key challenge is creating narrations for recorded visual scenes. Previous studies on dense video captioning and video story generation have made some progress. However, in practical applications, we typically require synchronized narrations for ongoing visual scenes. In this work, we introduce a new task of Synchronized Video Storytelling, which aims to generate synchronous and informative narrations for videos. These narrations, associated with each video clip, should relate to the visual content, integrate relevant knowledge, and have an appropriate word count corresponding to the clip’s duration. Specifically, a structured storyline is beneficial to guide the generation process, ensuring coherence and integrity. To support the exploration of this task, we introduce a new benchmark dataset E-SyncVidStory with rich annotations. Since existing Multimodal LLMs are not effective in addressing this task in one-shot or few-shot settings, we propose a framework named VideoNarrator that can generate a storyline for input videos and simultaneously generate narrations with the guidance of the generated or predefined storyline. We further introduce a set of evaluation metrics to thoroughly assess the generation. Both automatic and human evaluations validate the effectiveness of our approach. Our dataset, codes, and evaluations will be released.

## introduction video storytelling @xcite is a tactic that utilizes the naturally engaging video format to share stories and has been widely used as a powerful tool for marketing, educational, or entertainment purposes. the video and its accompanying narration are both critical to successful video storytelling. therefore, two aspects ## method as shown in figure 5 , our proposed videonarrator consists of the visual feature extractor, the memory consolidation component, the video projection layer, and the large language model (llm). inspired by @xcite , we utilize the powerful clip-l/14 visual encoder @xcite and the q-former module from blip-2 @xcite to obtain frame-level visual features. these features already achieve a good alignment with the textual modality. to capture temporal information, we further feed them into a linear video projector. however, inputting all video frames would result in significant computational complexity and memory usage @xcite . moreover, it would make it challenging to extract prior knowledge or maintain coherence with the previously generated story. to address these difficulties, we consolidate the visual information from previous clips into fixedlength memory tokens. additionally, we compress the visual frames of the current clip, retaining only the key information. ## conclusion this paper introduces the new task of synchronized video storytelling, which aims to generate synchronous narrations for sequential video scenes. these narrations are guided by a structured storyline and can incorporate relevant knowledge, resulting in a coherent and informative story. the new task is more practical than existing tasks of dense video captioning and visual storytelling because the generated narrations can be directly combined with the visual scenes, creating an engaging videoformat story. we collect a benchmark dataset called e-syncvidstory and introduce an effective framework named videonarrator. both automatic and human evaluations verify the effectiveness of our proposed framework.