Training-free Deep Concept Injection Enables Language Models for Video Question Answering (2024)

Abstract: Recently, enabling pretrained language models (PLMs) to perform zero-shot crossmodal tasks such as video question answering has been extensively studied. A popular approach is to learn a projection network that projects visual features into the input text embedding space of a PLM, as well as feed-forward adaptation layers, with the weights of the PLM frozen. However, is it really necessary to learn such additional layers? In this paper, we make the first attempt to demonstrate that the PLM is able to perform zero-shot crossmodal tasks without any crossmodal pretraining, when the observed visual concepts are injected as both additional input text tokens and augmentation in the intermediate features within each feed-forward network for the PLM. Specifically, inputting observed visual concepts as text tokens helps to inject them through the self-attention layers in the PLM; to augment the intermediate features in a way that is compatible with the PLM, we propose to construct adaptation layers based on the intermediate representation of concepts (obtained by solely inputting them to the PLM). These two complementary injection mechanisms form the proposed Deep Concept Injection, which comprehensively enables the PLM to perceive instantly without crossmodal pretraining. Extensive empirical analysis on zero-shot video question answering, as well as visual question answering, shows Deep Concept Injection achieves competitive or even better results in both zero-shot and fine-tuning settings, compared to state-of-the-art methods that require crossmodal pretraining.

## introduction pretrained language models have been shown to be a powerful base model to deal with tasks beyond natural language processing, such as visual question answering @xcite and video question answering @xcite @xcite , 2022b). these tasks require reasoning over information from multiple modalities. thus, the key challenge is to find a common representation so that the information from different modalities can be fused and processed by the plm. conventional methods @xcite usually rely on a two-stage training process to obtain satisfying results on downstream datasets. assuming pretrained language models and feature extractors like vision-text contrastive models (e.g., clip @xcite ) are available, the first stage aims at crossmodal pretraining on webcollected vision-text dataset with techniques like masked token modeling @xcite or contrastive learning @xcite @xcite to learn the alignment and fusion of visual and textual inputs. in the second stage, the model is further fine-tuned with human annotation on specific downstream datasets @xcite @xcite @xcite @xcite to obtain better models for specific tasks. however, such a two-stage training process has been criticized to be lack of efficiency, flexibility and generalization @xcite @xcite . therefore, researchers @xcite have been actively exploring the possibility of relying solely on the first crossmodal pretraining stage and aims at learning a general vision-language model that can perform well without any additional downstream fine-tuning. successful representative methods in this line of work like frozenbilm @xcite ) freeze the language model and only train a few projection layers and a few adaptation layers during the training process to improve the efficiency. this line of research, while notable for its effectiveness, raises a pertinent question: is the training of the projection networks truly necessary? in this paper, we challenge the prevailing methodology and propose a novel method that eliminates the need for training projection networks while enabling the plms to perform zero-shot crossmodal tasks. as in figure 1 , our approach, deep concept injection (dci), injects the observed visual concepts as both additional input text tokens and augmentation in intermediate features within each feed-forwards network to enable plms to perceive and reason over multimodal inputs. our key insights are two-fold. first, towards zero-shot crossmodal tasks, it is necessary to represent the observed visual information in a way that the plm directly understands, and our solution is to represent the observation using concepts. inspired by @xcite and @xcite , these visual concepts can be extracted through retrieval over a predefined vocabulary given the visual input, with the help of pretrained vision-text contrasting models like clip @xcite . second and more importantly, in modern plms based on transformers @xcite , there are two complementary ways of fusing multimodal information. one commonly used way is to provide visual information as additional elements in the input, where the interaction between visual input and textual input is modeled in the self-attention layers. however, self-attention layers were trained on natural sentences but not between concept words and a natural sentence. moreover, the other possibility within feed-forward networks has been ignored. we propose to leverage the intermediate representations of concept words (when they are solely input to the plm) to construct adaptation layers and to achieve crossmodal fusion by estimating conditional distribution of the concept given the visual observation and the current word being processed in the plm. with the above two key insights, there remains one design choice to complete deep concept injection: how do we choose the set of concepts? one intuitive solution is to leverage existing ontology in computer vision datasets @xcite @xcite . however, such generic datasets might not be aligned with the specific downstream tasks we are interested in. to obtain task-relevant prior, we explore two orthogonal solutions. we first exploit the setting where the access to all the possible answer words of the dataset is allowed, which is actually true for open-ended question answering datasets @xcite @xcite . second, to further eliminate the assumption over prior information about the task and dataset, we propose to obtain the set of relevant concepts by querying the language model. with extensive empirical analysis on fourteen datasets, the proposed deep concept injection achieves competitive or even better performance than state-of-the-art methods, without any crossmodal pretraining. we believe this paper will stimulate further research and exploration in the field, potentially opening new paths towards more efficient and versatile utilization of plms for crossmodal tasks. the contribution of this paper could be summarized as follows: ## technical approach in this section, we first present some preliminaries (more detailed related work is discussed in the supplementary material) and then introduce the deep concept injection in detail. we propose dci based on two key ideas: speak the "language" that plms understand and comprehensively leverage both ways in transformer block for crossmodal fusion. the first idea motivates us to leverage concepts (e.g., action, objects, attributes and etc.) as the bridge to transform visual information into text representations. the second idea motivates us to also utilize feed-forward networks for crossmodal fusion. last we discuss possible ways of acquiring prior information for vocabulary construction. ## experimental results in this section, we will first introduce the implementation and evaluation settings. then we organize the following subsections by answering a set of important questions. more ablations, further analysis and other details are in the supplementary material. ## related work pre-trained vision-text contrastive models. recently, a family of contrastively pre-trained models are introduced, which are learned from large-scale vision-text data @xcite @xcite . these models typically contain a visual encoder and a text encoder, and learn to map visual and text embeddings into a common space. they sample positive/ negative pairs from aligned/unaligned image/video and texts, and train the visual and text encoders with a contrastive objective in a self-supervised manner. with access to large-scale multimodal data (e.g., 400 million web image-text pairs), they are shown superior on zero-shot recognition tasks. the resulting visual encoders have also been shown to be great feature extractors for downstream tasks @xcite @xcite @xcite . crossmodal tasks with pretrained language models. conventional methods @xcite @xcite usually rely on a two-stage training process to obtain satisfying results on downstream datasets. assuming pretrained language models and feature extractors like vision-text contrastive models (e.g., s3d @xcite and clip @xcite ) are available, the first stage aims at training on webcollected vision-text dataset with techniques like masked token modeling @xcite or contrastive learning @xcite @xcite to learn to align and fuse visual and textual inputs. in the second stage, the model is further fine-tuned with human annotation on downstream datasets @xcite @xcite @xcite for better downstream performance. such a two-stage training process has been criticized for a lack of efficiency and flexibility because of the huge cost of the first training stage @xcite , and they are also not general enough @xcite . there are two lines of following research trying to address the limitation of the two-stage training process. one line of work @xcite focuses on obtaining competitive models with only the second training stage on downstream datasets and one successful idea is to transform every modality into concept text @xcite so that the plm can immediately understand and leverage the information from other modalities without the expensive first training stage. however, such methods still rely on human annotation and specific training towards each downstream dataset. the other line of work @xcite @xcite relies solely on the first training stage and aims at learning a general vision-language model that can perform well in the zero-shot setting without any additional downstream fine-tuning. during the training process, successful methods in this line of work such as frozenbilm @xcite freeze the language model and only train a few projection layers and a few feed-forward adaptation layers to project the visual features extracted by a frozen feature extractor like clip, to improve the efficiency. the typical training target is, with the video/image as input, generating the associated text. it is noteworthy that, although the pretrained model exhibits the ability to perform zero-shot crossmodal tasks such as video questions answering, to obtain higher performance on downstream tasks, fine-tuning is still crucial @xcite to achieve superior performance. unlike existing research, we explore a more challenging new problem where there is no additional training or labeled training samples for downstream tasks. ## conclusion in this paper, we present a novel approach to enabling pretrained language models to perform video question answering without any training. the proposed deep concept injection effectively circumvents the necessity of training projection networks, a widely accepted practice in this field, and instead makes insightful use of observed visual concepts as additional input text tokens and as a means for augmenting intermediate features. extensive results show that they function synergistically to realize strong zero-shot crossmodal capabilities of the plm and to bypass the costly crossmodal pretraining stage in versatile tasks and settings.