Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectives (2024)

Abstract: Humans use multiple senses to comprehend the environment. Vision and language are two of the most vital senses since they allow us to easily communicate our thoughts and perceive the world around us. There has been a lot of interest in creating video-language understanding systems with human-like senses since a video-language pair can mimic both our linguistic medium and visual environment with temporal dynamics. In this survey, we review the key tasks of these systems and highlight the associated challenges. Based on the challenges, we summarize their methods from model architecture, model training, and data perspectives. We also conduct performance comparison among the methods, and discuss promising directions for future research.

## introduction vision and language constitute fundamental components of our perception: vision allows us to perceive the physical world, while language enables us to describe and converse about it. however, the world is not merely a static image but exhibits dynamics in which objects move and interact across time. with the temporal dimension, videos are able to capture such temporal dynamics that characterize the physical world. consequently, in pursuit of endowing artificial intelligence with human-like perceptual abilities, researchers have been developing video-language understanding models that are capable of interpreting the spatio-temporal dynamics of videos and the semantics of language, dating back to the 1970s @xcite . these models are distinctive from image-language understanding models, since they exhibit an additional ability to interpret the temporal dynamics @xcite . they have demonstrated impressive performance in various video-language understanding * corresponding author tasks. these tasks evaluate video-language models from coarse-grained to fine-grained understanding capacity. for example, for coarse-grained understanding, text-video retrieval task assesses the model's ability to holistically associate a language query with a whole video @xcite . for more fine-grained understanding capacity, a video captioning model is required to understand the overall and detailed video content, then describe the content in concise language @xcite . fine-grained understanding in video questioning answering remains a difficult task, where a model needs to recognize minute visual objects or actions, and infers their semantic, spatial, temporal, and causal relationships @xcite . in order to effectively perform such videolanguage understanding tasks, there are three challenges that video-language understanding works have to explore. the first challenge lies in devising an appropriate neural architecture to model the interaction between video and language modalities. the second challenge is to design an effective strategy to train video-language understanding models in order to effectively adapt to multiple target tasks and domains. the third challenge is preparing highquality video-language data that fuel the training of these models. although a handful of recent works have tried to review video-language understanding, they mostly focus on one challenge, for example, transformerbased @xcite and llm-augmented architecture @xcite (the 1st challenge), self-supervised learning @xcite and pre-training @xcite (the 2nd challenge), and data augmentation @xcite (the 3rd challenge). moreover, others also focus merely on one video-language understanding task, e.g. video question answering @xcite ), text-video retrieval @xcite , and video captioning @xcite . such a narrow focus contradicts the growing consensus advocat-video-language understanding video-language understanding tasks text-video retrieval e.g. @xcite @xcite @xcite video captioning e.g. @xcite @xcite @xcite videoqa e.g. @xcite @xcite @xcite @xcite other tasks e.g. @xcite @xcite @xcite video-language understanding methods ## challenges of video-language understanding the discussed video-language understanding tasks present unique challenges compared with imagelanguage understanding, since a video incorporates an additional temporal channel. we summarize their important challenges as follows: intra-modal and cross-modal interaction. while intra-modal interaction modeling within language can be directly taken from image-language understanding, intra-modal interaction modeling within video is different since it jointly consists of spa-figure 3 : timeline of the established video-language understanding methods (tvr: text-video retrieval, vc: video captioning, vqa: video question answering, tf: transformer, llm: large language model). from left to right, our legend table follows the order: pre-transformer (pre-tf), task-specific transformer, multi-task transformer, and llm-augmented architectures. given the larger semantic gap for videolanguage compared to image-language, crossmodal interaction plays a crucial role in videolanguage understanding. the interaction between visual and language features is pivotal for aligning the semantics of video and text query to associate them for text-video retrieval, or identifying relevant parts to answer the question and writing the caption in videoqa and video captioning, respectively. in addition, incorporating the interaction of motion and language features can mitigate the extraction of noisy information from videos @xcite . @xcite also discover that the interaction between audio and language features can compactly capture information related to objects, actions, and complex events, compensating for sparsely extracted video frames. cross-domain adaptation. given the infinitude of online videos, that our video-language understanding model will encounter testing scenarios which are identically distributed to our training data is an impractical assumption. moreover, with the advent of llm-augmented models that can tackle a variety video-language understanding tasks @xcite , it is currently more advisable to train a model that can effectively adapt to multiple tasks and domains than to obtain a model which specializes in a specific understanding task. furthermore, since a video can be considered as a sequence of images, training a model on video-text data is more computationally expensive than imagetext data. combined with the large-scale of recent video-language understanding models @xcite , there is also a need to devise an efficient fine-tuning strategy to save the computational cost of fine-tuning these models. ## model architecture for video-language understanding addressing the challenge of intra-modal and crossmodal interaction is the key aim in designing video-language understanding model architectures, which can be divided into pre-transformer and transformer-based architectures. the advent of llms with remarkable zero-shot capability in addressing multiple tasks led to the design of llm-augmented architectures that exhibit cross-domain adaptation ability to various videolanguage understanding tasks. ## model training for video-language understanding model training seeks to address the cross-domain adaptation ability of video-language understanding models. to achieve this goal, pre-training strategies have been devised to gain world knowledge that generalizes across multiple scenarios, then taskspecific fine-tuning is conducted to specifically improve downstream task performance. ## future directions fine-grained understanding. existing methods excel at video-language understanding at a coarsegrained level, enabling effective responses to questions like "what is" or the recognition of global events without significant difficulty @xcite . nevertheless, limiting comprehension to this coarse level could hinder practical utility of existing systems. in real-world scenarios, a user might require a precise timestamp and location of an object within a video @xcite , or request the ai agent to forecast potential alternative events, which is a common need in predictive analytics @xcite . these tasks necessitate an advanced understanding and inference capability regarding the causal and temporal relationships present in a video. at present, models exhibit a constrained visio-linguistic capacity to engage in temporal reasoning, categorizing them as image-sequence-and-language models rather than video-language models @xcite . therefore, future research in this direction deserves more attention and exploration. long-form video-language understanding. current understanding systems have demonstrated remarkable performance on short video clips lasting several seconds. however, they tend to struggle when switching to long-form videos which last several minutes or hours. to enhance the applicability of these systems, it is essential to enhance their capability of understanding long-form videos. current approaches mainly feature reducing computational cost through architectures more efficient than transformer-based ones such as state space models @xcite , which can be considered as linear rnn with specifically designed fixed weights, or compensating sparsely extracted video frames with additional information @xcite . in general, how to effectively model long-form videos and adapt them to the joint context with language deserves more attention. trustworthiness of video-language understanding models. although modern video-language understanding systems have demonstrated remarkable performance, their black-box nature undermines our trust to deploy them. in particular, we still do not precisely understand what part of the video a videoqa model looks at to answer the question @xcite , or how video and language semantic information flows into the common representation space of the video retrieval model @xcite . furthermore, adversarial noise sensitivity or hallucination of video-language understanding models are also open problems. future trustworthiness benchmarks such as @xcite for video-language understanding are of great significance towards practical systems. ## conclusion in this paper, we survey the broad research field of video-language understanding. particularly, we categorize related video-language understanding tasks and discuss meaningful insights from model architecture, model training, and data perspectives. we thoroughly analyze each perspective, and finally conclude with promising future directions. we hope our survey can foster more research towards constructing effective ai systems that can comprehensively understand dynamic visual world and meaningfully interact with humans.