Video- LL a MA : An Instruction-tuned Audio-Visual Language Model for Video Understanding (2023)

Abstract: We present Video-LLaMA, a multi-modal framework that empowers Large Language Models (LLMs) with the capability of understanding both visual and auditory content in the video. Video-LLaMA bootstraps cross-modal training from the frozen pre-trained visual & audio encoders and the frozen LLMs. Unlike previous works that complement LLMs to process the visual or audio signals only, Video-LLaMA enables video comprehension by tackling two challenges: (1) capturing the temporal changes in visual scenes, (2) integrating audio-visual signals. To counter the first challenge, we propose a Video Q-former to assemble a pre-trained image encoder into our video encoder and introduce a video-to-text generation task to learn video-language correspondence. For the second challenge, we leverage ImageBind, a universal embedding model aligning multiple modalities, as the pre-trained audio encoder and introduce an Audio Q-former on top of ImageBind to learn reasonable auditory query embeddings for the LLM module. To align the output of both visual & audio encoders with LLM’s embedding space, we first train Video-LLaMA on massive video/image-caption pairs and then tune our model with visual-instruction datasets of moderate amount but higher quality. We found Video-LLaMA shows the ability to perceive and comprehend video content and generate meaningful responses grounded in the visual and auditory information presented in the videos.

## introduction large language models (llms) @xcite @xcite have demonstrated remarkable capability of understanding and following user intentions and instructions 234 . typically, the user requests and the corresponding responses from llms are all in texts, however, textonly human-computer interaction is not sufficient for many application scenarios because real-world information is usually multi-modal. in order to further explore the potential of llms, many researchers attempt to endow llms with the capability of understanding multi-modal content @xcite @xcite . among these efforts, @xcite ; @xcite ; @xcite ; @xcite ; @xcite ; @xcite pretrain multi-modal llms with massive interleaved image-text data or speech-text data to accommodate multi-modal input. meanwhile, another group of works adopts a more parameter-efficient way by complementing llms with off-the-shelf vision or speech foundation models to achieve multi-modal understanding @xcite @xcite @xcite @xcite @xcite . despite their effectiveness, these approaches are dedicated to aligning the input from exactly one additional modality with text (i.e., image or audio), which is unsatisfactory for video understanding. concretely, empowering llms to understand video requires comprehensive processing for different modalities including visual input, auditory input, and textual output, which is more challenging than image-only understanding and audio-only understanding tasks. although there are several recent works attempt to unleash the video understanding capability of llms @xcite @xcite , their primary objective is to comprehend only the visual content of the video, with the auditory content remaining unused. ## method video-llama aims to empower frozen llms with the capability of understanding both visual and auditory content in videos. as shown in figure 1 , we design two branches, namely vision-language branch and audio-language branch, to respectively transform the video frames and audio signals into query representations that are compatible with the textual inputs of llms. in this section, we first introduce the overall architecture and the building blocks of each branch. then, we delineate the procedures of the proposed multi-branch cross-modal pre-training and audio-visual instruction tuning. describe this video: # human: this video is an animation of a rocket launching from a launch pad at night... ## related works large language models: large language models (llms) @xcite @xcite have demonstrated remarkable language understanding and reasoning abilities, enabling the generation of high-quality natural language text across various domains, including articles, conversations, stories, and poetry. llms have already sparked a technological revolution and have been widely applied in different applications. moreover, a series of open source large models, such as llama @xcite , bloom @xcite and opt @xcite , have greatly promoted technological advancement and made outstanding contributions to the nlp community. building upon these llms, researchers have further extended their capabilities and developed excellent models for various nlp tasks. examples include vicuna @xcite and baize @xcite . our work is based on these llms and provides plug-and-play plugins that empower them with the capability of comprehending both visual and auditory content in videos. multi-modal large language models: researchers have been actively exploring the use of llms for processing multi-modal inputs @xcite . existing approaches can be categorized into two main groups. the first category involves employing llms as controllers and utilizing existing multi-modal models as tools. in this approach, when receiving the user's text instruction, the llm recognizes the user's intention and makes decisions about which tools to call. it then generates comprehensive responses by incorporating the results obtained from these offthe-shelf multi-modal models. examples include chatgpt @xcite , hugginggpt @xcite , and audiogpt @xcite . the second category focuses on training fundamental large-scale multi-modal models. the key idea of this line of work is to align the pre-trained foundation models for other modalities to textual llms. for instance, flamingo @xcite utilizes a perceiver resampler and a gated cross-attention layer to connect a frozen image encoder and llm. blip2 @xcite introduces a q-former to map learned image queries to the textual embedding space of llms. @xcite , mplug-owl @xcite and minigpt4 @xcite develop instructionfollowing image-llms using image-instructionfollowing dataset. video-chat @xcite and video-chatgpt @xcite extend image encoders to video encoders and connect them with llms to understand visual content in videos. pandagpt @xcite utilizes multi-modal encoders from imagebind, trained exclusively on image-instruction pairs, to enable large models to understand six modalities. our work falls into the second category, where we train fundamental models to comprehend both the visual and auditory content in videos. ## examples in this section, we show some cases to demonstrate video-llama's multi-modal instruction-following capability in video/audio/image-grounded conversations. figure 2 shows some of these examples, while additional cases can be found in appendix a. ## conclusion in this paper, we present video-llama, a cuttingedge multi-modal framework that empowers large language models with both audio & video understanding capabilities. our experiments demonstrated the impressive abilities of video-llama in audio and video-grounded conversations, highlighting its potential as a promising prototype for audio-visual ai assistants. we have open-sourced the entire training code and various model weights, along with detailed instructions to assist developers in utilizing our code for further development. in addition, we have provided online demo websites and offline demo deployment guides for users to experi-ence video-llama's capabilities firsthand. we are committed to constantly maintaining and improving video-llama, and will continue to contribute to the open-source community.