Video- LL a VA : Learning United Visual Representation by Alignment Before Projection (2024)

Abstract: Large Vision-Language Model (LVLM) has enhanced the performance of various downstream tasks in visual-language understanding. Most existing approaches encode images and videos into separate feature spaces, which are then fed as inputs to large language models. However, due to the lack of unified tokenization for images and videos, namely misalignment before projection, it becomes challenging for a Large Language Model (LLM) to learn multi-modal interactions from several poor projection layers.In this work, we unify visual representation into the language feature space to advance the foundational LLM towards a unified LVLM. As a result, we establish a simple but robust LVLM baseline, Video-LLaVA, which learns from a mixed dataset of images and videos, mutually enhancing each other.As a result, Video-LLaVA outperforms Video-ChatGPT by 5.8%, 9.9%, 18.6%, and 10.1% on MSRVTT, MSVD, TGIF, and ActivityNet, respectively. Additionally, our Video-LLaVA also achieves superior performances on a broad range of 9 image benchmarks.Notably, extensive experiments demonstrate that Video-LLaVA mutually benefits images and videos within a unified visual representation, outperforming models designed specifically for images or videos. We aim for this work to provide modest insights into the multi-modal inputs for the LLM.

## introduction recently, llms have gained rapid popularity in the ai community, such as gpt-3.5, @xcite , palm @xcite , and bloom @xcite . they rely on their powerful language comprehension abilities to follow human-provided instructions and provide corresponding responses. typically, llms can only respond within the text input provided by the user, which is insufficient because human interaction with the world involves multiple channels, such as visual and textual. to this end, recent video-llava aligns images and videos before projection, allowing llm to learn from a unified visual representation and endowing llm with the ability to comprehend both images and videos simultaneously. however, most current lvlms @xcite @xcite @xcite can primarily handle a single visual modality, either imagelanguage or video-language. we compare different lvlm paradigms as shown in figure 1 , where videochat @xcite and video-llama @xcite utilize a share visual encoder to handle both images and videos. however, due to the inherent differences in the media types of images and videos, it is challenging to learn a unified representation, and the performance falls significantly behind that of the specialized video expert model, video-chatgpt. therefore, x-llm @xcite and macaw-llm @xcite allocate a modality-specific encoder for each modality, attempting to enable a llm to comprehend images or videos through several projection layers. but their performances are inferior to dedicated video expert models such as video-chatgpt @xcite . we attribute this phenomenon to the lack of alignment before projection. because image features and video features reside in their own spaces, this poses a challenge for a llm to learn their interactions from several poor projection layers. some similar phenomenon such as alignment before fusion has been discussed by albef @xcite and vilt @xcite in multi-model models. more recently, imagebind-llm @xcite focuses on enabling the llm to simultaneously process multiple modal inputs by pre-aligning each modality to a common feature space @xcite . based on a large image-language model, imagebind-llm converts other modalities into the most similar image features by retrieving from a training-free image cached database. however, the indirect alignment approach of imagebind-llm may lead to performance degradation, and the llm has no knowledge of actual video data. in this work, we introduce video-llava, a simple but powerful baseline for the lvlm simultaneously handling both images and videos. specifically, as shown in figure 1 , video-llava initially aligns the representations of images and videos to a unified visual feature space. since the visual representations are already aligned prior to projection, we employ a shared projection layer to map the unified visual representation for the llm. to enhance computational efficiency, video-llava undergoes joint training of images and videos, achieving remarkable results with 1 training epoch. as a result, the proposed video-llava greatly enhances the ability of the llm to simultaneously understand both images and videos. for image understanding, video-llava surpasses advanced lvlms such as mplug-owl-7b and instructblip-7b in 5 image benchmarks. additionally, utilizing 4 benchmark toolkits for a more comprehensive evaluation, video-llava-7b even outperforms idefics-80b by 6.4% in mmbench. moreover, similar trends can be observed in video understanding, where video-llava surpasses video-chatgpt by 5.8%, 9.9%, 18.6%, and 10.1% respectively on the msvd, msrvtt, tgif, and activitynet video question-answering datasets. ex-tensive ablation experiments demonstrate that alignment before projection yields greater benefits. additionally, joint training of images and videos can facilitate a unified visual representation in llm comprehension. we summarize our primary contributions as follows: ## conclusion in this work, we introduce video-llava, a simple but powerful large visual-language baseline model. we propose a novel framework to address the issue of misalignment before projection, utilizing a languagebind encoder to pre-bind visual signals into the language feature space. to enable a llm to comprehend both images and videos simultaneously, we conduct joint training on images and videos, allowing the llm to learn multi-modal interactions from a unified visual representation. extensive experiments demonstrate that joint training on images and videos mutually benefits performance. furthermore, we validate that aligning visual representations before projection aids llm learning. remarkably, llm, after learning from a unified visual representation, exhibits the remarkable ability to simultaneously engage with both images and videos, showcasing a powerful comprehension of unified visual concepts. these results collectively demonstrate the effectiveness of the video-llava training framework. as a unified visual training framework, the performance of video-llava even surpasses that of expert models designed specifically for images or videos. the video shows a man playing the dru his hands and sticks, and he continues t