Empowering Large Language Model for Continual Video Question Answering with Collaborative Prompting (2024)

Abstract: In recent years, the rapid increase in online video content has underscored the limitations of static Video Question Answering (VideoQA) models trained on fixed datasets, as they struggle to adapt to new questions or tasks posed by newly available content. In this paper, we explore the novel challenge of VideoQA within a continual learning framework, and empirically identify a critical issue: fine-tuning a large language model (LLM) for a sequence of tasks often results in catastrophic forgetting. To address this, we propose Collaborative Prompting (ColPro), which integrates specific question constraint prompting, knowledge acquisition prompting, and visual temporal awareness prompting. These prompts aim to capture textual question context, visual content, and video temporal dynamics in VideoQA, a perspective underexplored in prior research. Experimental results on the NExT-QA and DramaQA datasets show that ColPro achieves superior performance compared to existing approaches, achieving 55.14% accuracy on NExT-QA and 71.24% accuracy on DramaQA, highlighting its practical relevance and effectiveness.

## introduction video question answering (videoqa) is critical for video understanding, involving training machine learning models to accurately respond to questions across various tasks (e.g., finding specific information @xcite @xcite based on given video content. however, existing videoqa models are typically trained on fixed datasets in static environments. with a continual increase in the number of videos on the internet every day, these static models may face challenges in answering new questions posed … cl task 1 videos questions cl task n videos questions … inference task 1 𝑃 1 𝑃 𝑁 𝑃 1 𝑃 𝑁 colpro … task-specific prompts … task n feature space task 1 feature space pre-trained llm pre-trained llm task n feature space why did the boy in yellow spread out his hands when he ski? ans: to balance himself √ (a) the issue of catastrophic forgetting in videoqa (b) the proposed collaborative prompting with continual learning figure 1: (a) existing fine-tuning techniques train for different videoqa tasks, which could lead to catastrophic forgetting, and generate inferior results. (b) we introduce the collaborative prompting (colpro) within a continual learning framework, which retains taskspecific knowledge to generate accurate answers, where p n denotes a projection layer. by the newly available content. one straightforward solution to overcome this challenge is to finetune the models when new data is introduced. however, this approach can lead to higher computational costs when retraining on all the data. alternatively, fine-tuning only on newly available video question-and-answer pairs may lead to the catastrophic forgetting issue @xcite , as shown in figure 1 @xcite . this motivates us to explore continual learning techniques @xcite @xcite for videoqa, facilitating ongoing fine-tuning of models across a sequence of data while mitigating catastrophic forgetting of previous tasks (e.g., finding information, or counting objects mentioned earlier), thereby addressing the needs of real-world dynamic environments. recent continual learning techniques @xcite have achieved good performance by employing rehearsal-free methods, such as learnable prompting @xcite and prefix-tuning @xcite . these approaches eliminate the need for memory-intensive stored experiences from previous tasks @xcite , reduce computation costs, and minimize forgetting. specifically, l2p @xcite , dualprompt @xcite , @xcite , and progprompt @xcite employ task-aware prompting techniques to finetune pre-trained models for downstream tasks using fewer learnable parameters. while these methods have improved performance in vision and language continual learning tasks, they often transfer either single-modal (text only) or multimodal (text and images) information from task to task. in terms of videoqa, it is crucial to incorporate textual question context (a1), visual content (a2), and video temporal dynamics (a3) in the continual training setting. in this paper, we introduce collaborative prompting (colpro), which explores these aspects for the videoqa problem, and represents an area that has not been fully explored in prior research, as shown in figure 1(b) . the core idea of colpro is to empower a base model to achieve enhanced performance when transferring across a sequence of tasks. inspired by the robust reasoning abilities of recent large language models (llms), we instantiate the base model as a llm (e.g., llama @xcite to generate accurate answers from textual questions and video inputs. specifically, colpro integrates three types of prompting techniques: taskspecific question constraint prompting (tqcp), knowledge acquisition prompting (kap), and visual temporal awareness prompting (vtap), aimed at enhancing accuracy in answer prediction while minimizing forgetting. tqcp enables the model to gain awareness of the task type and select the correct prompt representation using a negative guiding approach @xcite . this method allows the prompt representation to positively correlate with the current task-specific question and negatively correlate with negative question samples. additionally, kap acquires taskspecific question and video information to enhance accurate answer prediction (for a1). furthermore, vtap integrates visual information with the llm by continuously incorporating video dynamics into prompts through autoregressive temporal dynamics and video distillation loss (for a2 and a3). with these prompting strategies, colpro encapsulates multimodal information to enhance task-specific question answering and mitigate catastrophic forgetting during inference. our main contributions to this paper are as follows: (1) we explore the novel problem of video question answering (videoqa) in a continual learning context, and demonstrate a critical issue: efficiently fine-tuning a llm for a sequence of tasks leads to catastrophic forgetting. this motivates us to conduct empirical studies to mitigate this issue. (2) we propose collaborative prompting (col-pro), which utilizes three distinct aspects: textual question context, visual content, and video temporal dynamics, in videoqa to facilitate knowledge transfer to future tasks. (3) we conduct extensive experiments on the split videoqa dataset (next-qa @xcite and dramaqa @xcite ) for continual task-specific answer prediction. our findings show that colpro achieves state-of-the-art results, with 55.14% accuracy on next-qa and 71.24% accuracy on dramaqa. ## conclusion in this work, we explore the novel problem of videoqa, which efficiently fine-tunes the llm to answer new questions with video in a continual learning context. we propose collaborative prompting (colpro) to integrate textual question context, visual content, and video temporal dynamics in each learning phase, facilitating knowledge transfer to future tasks while minimizing catastrophic forgetting. we achieves state-of-the-art results on split next-qa and dramaqa datasets.