D e VA n: Dense Video Annotation for Video-Language Models (2024)

Abstract: We present a novel human annotated dataset for evaluating the ability for visual-language models to generate both short and long descriptions for real-world video clips, termed {\bf DeVAn} (Dense Video Annotation). The dataset contains 8.5K YouTube video clips of 20-60 seconds in duration and covers a wide range of topics and interests. Each video clip is independently annotated by 5 human annotators, producing both captions (1 sentence) and summaries (3-10 sentences). Given any video selected from the dataset and its corresponding ASR information, we evaluate visual-language models on either caption or summary generation that is grounded in both the visual and auditory content of the video. Additionally, models are also evaluated on caption- and summary-based retrieval tasks, where the summary-based retrieval task requires the identification of a target video given \textit{excerpts} of a given summary. Given the novel nature of the paragraph-length video summarization task, we compared different existing evaluation metrics and their alignment with human preferences and found that model-based evaluation metrics provide more semantically-oriented and human-aligned evaluation. Finally, we benchmarked a wide range of current video-language models on DeVAn, and we aim for DeVAn to serve as a useful evaluation set in the age of large language models and complex multi-modal tasks. Code is available at https://github.com/TK-21st/DeVAn.

## introduction with billions of active users on video content platforms such as youtube and tiktok, there has been an unprecedented need for automated complex video understanding. classically, video understanding has focused on captioning and/or retrieval tasks on short videos with brief (sentence-long) captions. the concise nature of both the videos selected and captions labeled has partly been the result of model limitations, where detailed and nuanced multi-sentence video descriptions have not been possible with lightweight text decoders. with the recent leaps in large language models (llms), however, vision-language models (vlms) now have the opportunity to tap into the immense natural language capabilities of models such as llama @xcite and chatgpt @xcite . with tens to hundreds of billions of parameters, these llms are able to write entire essays with details and poise that mimic human to an unprecedented extent. with video conversational models such as imagebind-llm @xcite , video-llama @xcite , video-chatgpt @xcite and videochat @xcite claiming to be able to generate detailed and fine-grained descriptions of video inputs, we believe the time is ripe for an evaluation benchmark that matches the capabilities of modern vlms powered by llms. in the current work, we focus on videos with multi-shot compositions containing diverse information streams such as dialogues, background music, and complex visual sequences. we developed devan, a novel task and dataset for dense longform video descriptions. this new multi-modal dataset contains 8.5k video clips carefully selected from previously published youtube-based video datasets (youtube-8m @xcite and yt-temporal-1b @xcite ) that integrate visual and auditory information. over the span of 10 months, a team of 24 human annotators (college and graduate level students) created 5 short captions (1 sentence each) and 5 long summaries (3-10 sentences) for each video clip, resulting in a rich and comprehensive human-annotated dataset that serves as a robust ground truth for subsequent model training and evaluation (see figure 1 for example). as opposed to short video captions where n-kids' disappointment turns to joy as they receive rideable toys from their parents on christmas day this video records 2 kids and their christmas presents. their parents made a series of puzzles for them to find out the presents. first, they got 2 printed materials, and they needed to flip them over to get more information. then the man with the camera gave them a hint to "eat the cookies", and following that they found their presents: 2 toy cars, one for the boy and one for the girl. "what are those things? cherry bones! what are those pieces of paper there? stop sign? ... oh! that scared me" captions summaries this video records 2 kids and their christmas presents. excerpt retrieval retrieval figure 1: example of devan dataset. for each video, 5 captions and 5 summaries are independently annotated based on both visual and auditory information of the selected videos. text-to-video retrievals from video summaries are performed by randomly sampling single-sentence excerpts. our contributions are summarized as follows: ## related work the endeavor to understand and provide textual descriptions of video content has been the subject of numerous research initiatives. we briefly review recent models and datasets relevant to devan. ## devan dataset in this section, we describe the procedure with which devan was constructed and how generation and retrieval task performances are evaluated on devan. ## experiments in this section, we describe both human performance and model performance on the devan evaluation set. to comprehensively evaluate the capabilities and limitations of various architectures for video captioning and retrieval, we consider two distinct types of models, each with its own set of advantages and drawbacks. ## conclusion in this paper, we introduce a new multi-modal dataset curated from a diverse range of youtube videos, designed to gauge the capabilities of visuallanguage models in long-form video summarization tasks. through a carefully orchestrated annotation process involving multiple human annotators, multiple rounds of video selection and quality control, we ensure that the dataset is comprised of high-quality and diverse captions and summaries. we show that while qualities of one-sentence captions can be accurately evaluated by n-gram, multi-sentence summaries require a more semantically aligned metric such as bleurt. using such metrics, we provide an extensive benchmarking of the current state-of-the-art video-to-text generation models. however, current visual-language models with frozen llms often do not support retrieval tasks due to lack of language encoders. to provide a baseline for text-to-video retrieval tasks, we finetuned videococa using our automatically generated training dataset, which achieved reasonable retrieval, but comparatively weaker generation performances. as the field of video summarization and captioning continues to evolve, it is imperative that datasets and evaluation metrics keep pace. our work aims to serve as a stepping stone in this direction, providing a balanced approach to video understanding.