UNICORN : A Unified Causal Video-Oriented Language-Modeling Framework for Temporal Video-Language Tasks (2024)

Abstract: The great success of large language models has encouraged the development of large multimodal models, with a focus on image-language interaction. Despite promising results in various image-language downstream tasks, it is still challenging and unclear how to extend the capabilities of these models to the more complex video domain, especially when dealing with explicit temporal signals. To address the problem in existing large multimodal models, in this paper we adopt visual instruction tuning to build a unified causal video-oriented language modeling framework, named UNICORN. Specifically, we collect a comprehensive dataset under the instruction-following format, and instruction-tune the model accordingly. Experimental results demonstrate that without customized training objectives and intensive pre-training, UNICORN can achieve comparable or better performance on established temporal video-language tasks including moment retrieval, video paragraph captioning and dense video captioning. Moreover, the instruction-tuned model can be used to automatically annotate internet videos with temporally-aligned captions. Compared to commonly used ASR captions, we show that training on our generated captions improves the performance of video-language models on both zero-shot and fine-tuning settings. Source code can be found at https://github.com/xyh97/UNICORN.

## introduction recent breakthroughs in large language models (llms) @xcite @xcite @xcite have reignited the enthusiasm about the achievement of artificial general intelligence where a single foundation model can accomplish a large variety of downstream tasks based on human instructions. towards this ultimate goal, the community has witnessed promising advances in large multimodal models (lmms) for vision and language @xcite visual input example, playing tennis (34s in total): despite impressive results in the image domain, videos, another important data format in the vision modality, are under-explored. in contrast to images, videos have an extra temporal dimension and are much more difficult to process due to increased complexity. existing approaches either directly apply lmms trained on image-text pairs @xcite to the video domain without fine-tuning or develop video-oriented lmms @xcite @xcite ) on short trimmed videos. however, such models focus more on problems which are less dependent on temporal information like action recognition and video question answering. it still remains unclear how to solve video-language tasks that requires explicitly modeling temporal information, including moment retrieval @xcite , video paragraph captioning @xcite , and dense video captioning @xcite in one single lmm. in fact, the inherent disparities among these task formats pose a challenge to the development of such models: moment retrieval requires predicting the temporal location of a moment described by language, paragraph captioning entails to write a coherent story from an untrimmed video, while the goal of dense video captioning is to generate captions and temporal locations for a series of moments simultaneously. these tasks are typically solved individually by specifically-designed models @xcite @xcite . while attempts have been made to unify these temporal video-language tasks @xcite , separate modules and training objectives tailored for each task are involved in these methods, making them complicated in both training and inference. to address the above challenge, we propose a unified causal video-oriented language modeling framework (unicorn) that unifies the tasks as a simple yet generic language modeling problem. for moment retrieval and video paragraph captioning, we convert original training datasets into corresponding instruction-following formats, as shown in on the other hand, the development of large video-language models is hindered by the lack of semantically-and temporally-aligned videotext pairs, an issue unique to the video domain. as pointed out in @xcite , the models pre-trained on commonly-used noisy datasets such as howto100m @xcite and yt-temporal-1b @xcite suffer from the misalignment between videos and asr captions severely. thanks to the generalization ability of lmms, our unicorn can be leveraged to automatically generate captions for internet videos. we demonstrate that qualitatively the generated captions are better semantically-and temporallyaligned with the videos than the original asr captions, and quantitatively incorporating our generated captions in either instruction-tuning for moment retrieval or end-to-end video representation learning leads to significant performance gains. our contributions are threefold: (1) we propose unicorn, a simple and generic framework that unifies various temporal video-language tasks via language modeling; (2) our approach achieves comparable or better performance to state-of-theart methods on multiple downstream tasks, including moment retrieval, video paragraph captioning, and dense video captioning; (3) compared to existing captions, those automatically generated by our method have shown to be better aligned with the videos, both semantically and temporally. empirically, the generated captions have demonstrated to improve performance of models trained on them. our automatic annotation pipeline is useful for empowering the development of future lmms. ## method in this section, we introduce our unified framework unicorn in detail. we start by discussing how to transform the original datasets for different downstream tasks into the general instruction-following format in section 3.1. then in section 3.2, we describe the model architecture designed for videolanguage interaction. in section 3.3, we present the training pipeline of unicorn including datasets and training objective. finally in section 3.3, we demonstrate how to conduct inference with the obtained model on downstream tasks together with the process to generate captions for noisy datasets. ## experiments in this section, we evaluate unicorn comprehensively against state-of-the-art methods to show its effectiveness. we first introduce experimental setup in section 4.1. then we present results on downstream tasks including moment retrieval, video paragraph captioning and dense video captioning in section 4.2. ablation studies are conducted in section 4.3 for better understanding of our designs. finally, in section 4.4 we investigate the quality of the automatic annotation generated by unicorn on howto100m. ## conclusion in this paper, we propose a unified causal videooriented language modeling framework unicorn to address temporal video-language tasks. by finetuning on instruction-following data constructed from existing datasets, our model achieves outstanding performance on various downstream tasks including moment retrieval, video paragraph captioning and dense video captioning. we further show that unicorn can be leveraged in automatic annotation on internet videos such as howto100m for semantically-and temporally-aligned captions. these captions can be used to improve videolanguage model performance against asr ones. in conclusion, unicorn paves the way towards a general-purpose foundation model that explicitly considers temporal information.