Video- C hat GPT : Towards Detailed Video Understanding via Large Vision and Language Models (2024)

Abstract: Conversation agents fueled by Large Language Models (LLMs) are providing a new way to interact with visual data. While there have been initial attempts for image-based conversation models, this work addresses the under-explored field of video-based conversation by introducing Video-ChatGPT. It is a multimodal model that merges a video-adapted visual encoder with an LLM. The resulting model is capable of understanding and generating detailed conversations about videos. We introduce a new dataset of 100,000 video-instruction pairs used to train Video-ChatGPT acquired via manual and semi-automated pipeline that is easily scalable and robust to label noise. We also develop a quantitative evaluation framework for video-based dialogue models to objectively analyze the strengths and weaknesses of video-based dialogue models. Code: https://github.com/mbzuai-oryx/Video-ChatGPT.

## introduction the surge of deep learning applications for video understanding has lead to major advancements in video-related tasks. however, the current video understanding models are still unable to hold an open-ended conversation about the video content in a coherent manner. a video-based dialogue model can revolutionize video search, surveillance operations and help summarize key events and abnormal event detection. above all, it can provide a unified human-understandable interface to video-related tasks such as action recognition, localization, detection, segmentation, retrieval, and tracking. further, such a capability is of great interest as it will demonstrate the model's ability to encode temporal and spatial cues, contextual relationships and long-term dependencies. recent advancements in multimodal understanding are largely based on the combination of pre-1 equal contribution. video-chatgpt leverages an adapted llm @xcite that integrates the visual encoder of clip @xcite with vicuna @xcite as a language decoder, fine-tuned on generated instructional image-text pairs. our approach further adapts the design for spatiotemporal video modeling and fine-tunes the model on video-instruction data to capture temporal dynamics and frame-to-frame consistency relationships available in video data. in contrast to other concurrent works for video-based conversation @xcite @xcite , video-chatgpt excels at temporal understanding, spatial consistency and contextual comprehension as demonstrated by our extensive evaluations. a fundamental contribution of this work is the creation of a dataset of 100,000 video-instruction pairs using a combination of human-assisted and semi-automatic annotation methods. each pair consists of a video and its associated instruction in the form of a question-answer. this provides video-chatgpt with a large and diverse dataset to learn from, increasing its video-specific understanding, attention to temporal relationships and conversation capabilities. moreover, we introduce the first quantitative video conversation evaluation framework for bench-marking, allowing for a more accurate evaluation of the performance of video conversation models. this framework evaluates models on a variety of capabilities, such as correctness of information, detail orientation, contextual understanding, temporal understanding, and consistency. ## related works vision language models: significant advancements in the field of computer vision have recently been observed due to the development of many foundational vision-language models. these models represent a significant leap towards creating general-purpose vision models capable of tackling various tasks simultaneously @xcite @xcite . a prime example is clip @xcite , which is trained on 400m image-text pairs and has demonstrated impressive zero-shot performance on numerous benchmarks. it has been employed in various downstream applications, from imagebased object detection and segmentation @xcite to 3d applications @xcite . numerous attempts have also been made to adapt clip for video applications @xcite . similar to our design, vifi-clip @xcite suggests employing temporal pooling across video frames to adapt the image-based clip model for video-based tasks. ## video-chatgpt video-chatgpt is a large vision-language model that aligns video representations with a large language model (llm), thus enhancing its ability to generate meaningful conversation about videos. our approach draws from the approach employed in designing vision-language (vl) models for the video domain. given the limited availability of video-caption pairs and the substantial resources required for training on such data from scratch, these models commonly adapt pretrained image-based vl models for video tasks @xcite @xcite . we adopt a similar approach, starting with the language-aligned large vision assistant (llava) @xcite as our foundation. llava is a lmm that integrates the visual encoder of clip @xcite with the vicuna language decoder @xcite and is fine-tuned end-to-end on generated instructional vision-language data. we fine-tune this model using our video-instruction data, adapting it for video conversation task. the video-instruction data is obtained as a combination of manual and automated pipelines in our proposed instruction generation setup. this adaptation on video-specific instructions allows for accommodating additional temporal dynamics, frame-to-frame consistency, and long-range relationships present in video data. as a result, our video-chatgpt excels in video reasoning, creativity, and understanding of spatial, temporal, and action-oriented components within videos. ## video instruction data generation in this section, we discuss our data-focused approach, which uses both human-assisted and semiautomatic annotation methods to generate highquality video instruction data. this data is cru-cial for training video-chatgpt, ensuring accurate and meaningful responses. our data collection involves two key methods. the human-assisted annotation, involves expert annotators analysing video content and providing detailed descriptions. this process generates data rich in context and detail, which helps our model understand complex aspects of video content. on the other hand, the semi-automatic annotation framework is more cost-effective and scalable. leveraging state-of-theart vision-language models, this method generates broad, high-volume annotations, thus increasing the quantity of data without compromising the quality substantially. through these combined methods, we have successfully accumulated a robust set of 100,000 video-instruction pairs. this extensive dataset is crucial in fine-tuning our model to comprehend video content effectively, integrating both spatial and temporal cues into its understanding. our instructional data is both diverse and comprehensive, incorporating a wide range of data types. these include detailed descriptions, summarizations, question-answer pairs, tasks that stimulate creativity or generation of new ideas, and conversational tasks. the data spans a broad spectrum of concepts, ranging from visual appearance and temporal relations to complex reasoning tasks and beyond, providing a diverse training ground for our model to learn from. ## conclusion in this work, we presented video-chatgpt, a multimodal model that merges a pretrained visual encoder with a large language model (llm) to enable video understanding and conversations based on videos. video-chatgpt leverages an adapter on top of pretrained llm and vision backbones and is fine-tuned on video-instruction data to capture temporal dynamics and spatial consistency relationships in spatiotemporal sequences. a dataset of 100,000 video-instruction pairs is created to en-hance video-chatgpt's video-specific understanding and conversation capabilities. the work also introduced a quantitative video conversation evaluation framework for benchmarking, evaluating models on a diverse set of capabilities including conventional video question answering as well as open-ended descriptions.