Audio-visual training for improved grounding in video-text LLM s (2024)

Abstract: Recent advances in multimodal LLMs, have led to several video-text models being proposed for critical video-related tasks. However, most of the previous works support visual input only, essentially muting the audio signal in the video. Few models that support both audio and visual input, are not explicitly trained on audio data. Hence, the effect of audio towards video understanding is largely unexplored. To this end, we propose a model architecture that handles audio-visual inputs explicitly. We train our model with both audio and visual data from a video instruction-tuning dataset. Comparison with vision-only baselines, and other audio-visual models showcase that training on audio data indeed leads to better grounding of responses. For better evaluation of audio-visual models, we also release a human-annotated benchmark dataset, with audio-aware question-answer pairs.

## introduction conversational agents fueled by llms have made it possible for us to interact in a new way with data from multiple modalities @xcite . image-text multimodal llms(mllms) like llava @xcite have demonstrated the effectiveness of visual instruction-tuning(it) data. several works like videochatgpt @xcite , videochat @xcite , pllava @xcite have extended the image-text model architecture for video related tasks. however, most of the above works rely only on the visual input, and do not consider audio signal for video understanding. in real world, listening to audio while playing the video, adds immensely to our perception of the video. we propose a videotext mllm, with phi-2 @xcite as the llm backbone. it supports both audio and visual inputs, using whisper @xcite figure 1 : an example of improved grounding in the video-text llm outputs, due to the additional audio signal as input. and siglip @xcite encoders respectively. unlike previous works, we train the model using audio data explicitly, in addition to the visual data. we aim to explore the role of audio in video understanding and if audio input can be utilized for better grounding of video-text llms. we also explore the creation of better benchmarks that encompass variety of question-answer pairs. evaluation on several benchmarks demonstrates the effectiveness of audio as an additional signal in better understanding of the video content. overall we make the following key contributions: 1.we propose an efficient video-text mllm architecture consisting of separate encoders to process the audio and visual inputs. 2.we train our video-text model using both audio and visual signals simultaneously, aiming to explore the effect of audio input on model outputs. 3.we release a human-annotated benchmark dataset containing video instruction-tuning samples, which are audio-aware. ## model architecture following the idea of fusing the modality inputs into llm @xcite ) @xcite , we build a video-text mllm architecture consisting of two separate branches for audio and visual inputs. each branch consists of modality encoder, projector layers to transform the encoder representations into llm embedding space, followed by the backbone llm. we use whisper @xcite as an audio encoder, and use its last hidden state as audio representations @xcite . to encode the video, we use siglip image encoder @xcite . following @xcite , we treat video as a sequence of images, and compute frame representations using siglip. we then compute spatial and temporal average of representations across 100 uniformly sampled frames, and use it as a video representation. inspired from bunny @xcite , we rely on low-cost, efficient, lightweight llm backbone with 2.7 billion parameters, phi-2 @xcite . projector layer for both vision and audio branch is mlp2x-gelu @xcite . the exact flow of input data through both the audio and visual branches is shown in the form of tensor dimensions, in figure 2. audio and visual input is converted into 64 and 829 token embeddings respectively. audio, visual, and text token embeddings are then concatenated before passing to the backbone llm. ## training setup and datasets training different components of our model with appropriate data is a key focus of our research. typically, these mllms go through a pretraining stage, followed by the finetuning stage. pretraining: pretraining aims to align different modalities to text llm space, by training on some generic modality-to-text task. only projector layer weights are trained during this phase, while encoders, and llm weights are frozen. we pretrain our audio projector layers using a combination of speech-to-text(stt) dataset(commonvoice @xcite ) and audio captioning dataset(audiocaps @xcite ) with 50k samples each. we convert these datasets into our instruction-tuning prompt template by creating 10 instructions each for transcription and captioning. since our visual branch relies on image encoder, we employ already trained checkpoint by bunny (he et al., 2024) to initialize vision projector layers. it has been trained on 2m subset of an image-text dataset laion (schuhmann et al., 2022). we freeze the vision branch while pretraining audio projector layers, and vice versa. finetuning: finetuning or instruction tuning is aimed to train the llm model to follow the exact requests or questions in the user prompt @xcite . unlike previous works, we explicitly train both the audio and visual branches of the model simultaneously, using video instructiontuning dataset containing both the audio and visual data. we rely on videoinstruct100k @xcite dataset with 100k samples containing video and question answer pair. although the dataset authors had used the dataset only for visual instruction tuning, we extract the audios(wav format) from the videos(mp4 format) for our use-case. we aim to explore if including audio features during training helps the model to better understand the video. to measure this effect, we also train a baseline vision-only model, without the audio branch. we train the vision branch of the model, using the visual data from same dataset. experiment details we implement the audio and video functionality by extending the codebases of bunny and llasm. we use whisper-small, siglip-so400m-patch14-384, and phi-2 models from hug-gingface. pretraining for audio projector layer was done using a100, with global batch size of 128. finetuning was implemented using lora for training llm weights, on a40 machine. ## benchmark dataset several evaluation criteria and datasets have been introduced to benchmark the vision-text mllms @xcite @xcite . videochatgpt has released a human verified benchmark dataset consisting of 500 videos and corresponding question-answer pairs for video-text tasks. however, these benchmarks do not consider audio information while creating the question-answer pairs based on videos. thus, it is challenging to evaluate the capability of model to attend to both the audio and visual signals while generating the output. therefore, we annotate such an audio-visual instruction-tuning dataset that contains questionanswer pairs based both on audio and visual information in the video. we include both generic questions, like 'what is happening in the video?', as well as more specific questions related to the video. answer of each question is around 2 sentences, with most of the videos available on youtube. we release a set of 120 such samples, as we intend to scale the size and quality of the data in future. example samples from our benchmark dataset are shown below. ## evaluation we extensively evaluate our model using videochatgpt evaluation framework across 5 key metrics. it relies on llm-based evaluation(using gpt-3.5) which rates the output on the scale of 1-5. we compare our audio-visual model with the visual-only baseline that we have trained, as well as other audio-visual model, video-llama. the evaluation results are summarized in the the audio-visual model clearly performs better than the vision-only baseline by a margin. interestingly, video-llama which is also an audio-visual model performs poorly on both the benchmarks. video-llama does not utilize the audio inputs explicitly, and instead rely on visual signals only during training. we could not compare against another audio-visual model, next-gpt, as it relies on llama-v0 weights which couldn't be available to us due to licensing. qualitative analysis of audio-visual model outputs demonstrate better overall quality compared to vision-only model. we also analyze the model outputs at intermediate stages, i.e. after pre-training. our model could very well generate the captions of audio data, which showed the efficacy of pretraining step. there is scope for better encoding strategies and training regimes for utilizing audio information even more. ## conclusion and future work we performed several experiments and evaluations to specifically study how audio signal can be utilized for better video understanding. training the mllm simultaneously on audio-visual signals of the video indeed results in a better performance, as seen in quantitative evaluation using several metrics. we also contributed a benchmark dataset curated to evaluate the video-understanding capability using both visual and audio information. based on these results, we are motivated to experiment with sophisticated ways of incorporating audio and visual signals together for video related tasks. future work also consists of the extensive analysis of the type of question-answer pairs in video it datasets, and work on creating better evaluation benchmarks catering to wide range of videorelated use-cases.