V ideo C o T : A Video Chain-of-Thought Dataset with Active Annotation Tool (2024)

Abstract: Multimodal large language models (MLLMs) are flourishing, but mainly focus on images with less attention than videos, especially in sub-fields such as prompt engineering, video chain-of-though (CoT), and instruction tuning on videos. Therefore, we try to explore the collection of CoT datasets in videos to lead to video OpenQA and improve the reasoning ability of MLLMs. Unfortunately, making such video CoT datasets is not an easy task. Given that human annotation is too cumbersome and expensive, while machine-generated is not reliable due to the hallucination issue, we develop an automatic annotation tool that combines machine and human experts, under the active learning paradigm. Active learning is an interactive strategy between the model and human experts, in this way, the workload of human labeling can be reduced and the quality of the dataset can be guaranteed. With the help of the automatic annotation tool, we strive to contribute three datasets, namely VideoCoT, TopicQA, TopicCoT. Furthermore, we propose a simple but effective benchmark based on the collected datasets, which exploits CoT to maximize the complex reasoning capabilities of MLLMs. Extensive experiments demonstrate the effectiveness our solution, and we will release our source codes and datasets to facilitate the research community.

## introduction with the emergence of chatgpt 1 , large language models (llms) have experienced unprecedented growth and have gradually expanded into the multimodal domain. pioneers have explored multiple feasible paths around multimodal large models (mllms), such as training mllms from scratch (e.g. kosmos-1 @xcite ), or bridging llms and vision modules (e.g. blip-2 @xcite ). moreover, prompt engineering, chainof-thought (cot), and instruction tuning for multimodal llms are also flourishing. however, the majority of current research focuses on images, with video research @xcite remaining underdeveloped. for instance, @xcite employs a video understanding model to extract features, which are then inputted, while @xcite utilizes multiple frames of the video as input. similarly, few researchers have devoted attention to sub-fields such as video prompt engineering @xcite , and video instruction fine-tuning @xcite . we attribute this phenomenon to the fact that mllms are less mature than llms that solely rely on natural language input, and there are still numerous issues to be explored. to advance the development of mllms for videos, our primary interest lies in cot in videos. video cot has multiple benefits as follows: 1) towards openqa in video. currently, the videoqa dataset widely adopts the form of multiple-choice questions, but there are significant differences between the answer options @xcite . as illustrated in fig. 1 (a), the options between a-e are significantly different, especially the descriptions of eating and being a hurdler are completely irrelevant to the video. this fact lead to models dataset rationale language #videos #q video source annotation qa task msvd-qa @xcite english 1.9k 50k web videos auto oe movieqa @xcite english 6.7k 6.4k movies manual mc msrvtt-qa @xcite english 10k 243k web videos auto oe tvqa @xcite english 21k 152k tv manual mc activitynet-qa @xcite english 5.8k 58k web videos manual oe next-qa @xcite english 5.4k 52k yfcc-100m manual mc,oe causal-vidqa @xcite english 26k 107k kinetics-700 manual mc fiber @xcite finding shortcuts to the dataset pattern. 2) enhance understanding. videos contain more temporal and spatial changes than images, and cot can help capture the complex semantics of these changes @xcite . as shown in fig. 1(b) , the key to solving the question, that is, the girl changes from moving to stopping (temporal) and the appearance of the bitumen road (spatial), is to develop with the video. 3) improving the reasoning ability of mllms. a more logical cot can enhance the reasoning ability of mllms when used for training. although video cot shows great potential, creating a video cot dataset is a non-trivial task. the process of fully annotating cots by humans is both tedious and expensive, which is why we aim to develop an automatic pipeline for generating cots. intuitively, one widely adopted strategy is to use off-the-shelf mllms or llms as assistants for reasoning. however, there are several challenges that need to be addressed. firstly, mllms do not possess strong reasoning abilities and cannot directly generate reliable cots. secondly, while llms have reasoning capabilities, they cannot use images as input for cot generation. lastly, machinegenerated data is often unreliable due to ethical doubts and hallucination issues @xcite , which require human correction for quality control. therefore, in this paper, we develop an automatic annotation tool that combines machine and human experts, under the active learning paradigm @xcite . as shown in fig. 2 , active learning is a strategy that involves interaction between the model and human experts, where the model actively seeks the opinions and standards of experts when encountering difficult samples @xcite . in this way, the workload of human la-beling can be reduced and the quality of the dataset can be guaranteed in the process @xcite . specifically, we will train a prompt generator to guide llms to generate complex cot based on video information. meanwhile, we will formulate a quality score to evaluate the generated cot sentences from multiple aspects. among them, low-quality sentences will be modified by human experts, and the modified cot will be used to train the prompt generator to guide llms to generate more reasonable cot @xcite . with the help of the aforementioned automatic annotation tools under the active learning paradigm, we strive to contribute three videocot datasets, namely videocot, topicqa, topiccot. among them, videocot is designed to supplement cot between question and answer from existing datasets. furthermore, we leverage the topic items in the dataset to construct topicqa, which enables mllms to learn the relevant relationship between videos and topics, and topiccot, which facilitates reasoning about the topic relevance. furthermore, we apply these datasets to propose a simple benchmark. extensive experiments demonstrate the effectiveness of our datasets and solution. the main contributions are summarized as follows: unlabeled pool llms background spatial objects temporal changes relations summary manual refining update dataset training model prompt generation quality score description: the [person_1] is standing in front of a mirror in... question:why does [person_1] stand in front of the mirror? answer: [person_1] wants to look at the mirror to comb her hair. prompt 5: please explain why the answer to [q] is [a]. prompt n: first describe the scene and its actions, then describe the essential function of the key objects in the question or scene...... @xmath0.9 scene descriptions questions answers perplexity (a) the process of active annotation tool. prompt 1: the description is [d], and the answer to [q] is [a]. based on the collected datasets, which exploits cot to achieve better reasoning ability. ## dataset collection following causal-vidqa @xcite , we built three datasets around videos based on kinetics-700, namely videocot, topicqa, and topiccot. in this section, we will introduce the process of active annotation tool, on which both videocot and topiccot are collected. ## proposed method the overall training framework is depicted with an illustration in fig. 7 . for the task of video question answering @xcite , multiple choice (mc) is more popular, but the differences between the options are too significant, and it is easy for the model to find shortcuts. therefore, we are committed to achieving a free-form open-ended (oe) with logic rationale @xcite . ## conclusions in this work, we strive to explore the collection of cot datasets on videos to bootstrap openqa on videos and improve the inference ability of mllms. to reduce the cost of manual annotation, we develop an automatic annotation tool that com-