Diving Deep into the Motion Representation of Video-Text Models (2024)

Abstract: Videos are more informative than images becausethey capture the dynamics of the scene.By representing motion in videos, we can capturedynamic activities. In this work, we introduceGPT-4 generated motion descriptions thatcapture fine-grained motion descriptions of activitiesand apply them to three action datasets.We evaluated several video-text models on thetask of retrieval of motion descriptions. Wefound that they fall far behind human expertperformance on two action datasets, raisingthe question of whether video-text models understandmotion in videos. To address it, weintroduce a method of improving motion understandingin video-text models by utilizingmotion descriptions. This method proves tobe effective on two action datasets for the motiondescription retrieval task. The results drawattention to the need for quality captions involvingfine-grained motion information in existingdatasets and demonstrate the effectiveness ofthe proposed pipeline in understanding finegrainedmotion during video-text retrieval.

## introduction since the introduction of large-scale use of contrastive learning for image and text representation @xcite , various efforts have been made to build video-text models @xcite @xcite to relate video to text. videos provide a way to access the dynamics or motion in the scene that a single image cannot capture @xcite @xcite . motion in videos could be due to the action depicted, the effect of camera movement (for example, in egocentric action videos), or a combination of camera motion and action @xcite . we investigate how existing video-text models perceive motion due to the action. for this work, we define motion in action videos as the movement of actors or the movement of actors and objects. the challenge is the lack of datasets that explicitly describe video motion. figure 1 shows some examples of captions from activitynet @xcite and the msr-vtt @xcite dataset. although verbs are included in captions of multimodal datasets like activitynet, msr-vtt, howto100m @xcite , spoken moments in time @xcite , a detailed description of motion is not available. this calls for the need to have an exclusive benchmark to evaluate how video-text models interpret and respond to motion descriptions. we use the human action datasets kinetics-400 @xcite , ucf-101 @xcite , and hmdb-51 @xcite to circumvent the lack of quality annotations of motion descriptions. the advantage of action datasets is that for every action label, we can obtain the corresponding characteristic motion of the action by using large language models like gpt-3 @xcite and @xcite , which, to a large degree, produce accurate descriptions of the all the actions in the datasets. figure 2 shows some examples of the descriptions generated by gpt-4 and corresponding videos and original captions. we evaluate several video-text models on the motion description retrieval task using the hmdb-51 and the ucf-101 datasets. we compare against human performance and show that all models fall far behind, raising questions about the design of video-text models and the role of quality captions in training better models to capture human motion. to address this question, we propose a method described in section 4 to investigate if providing better motion description captions helps video-text models understand fine-grained motion descriptions. to validate this fairly, we compare our method with video-text models that similarly initialize their video encoder and text encoder with pre-trained clip @xcite weights so that undue performance gain is not obtained by pre-training on video-text data. our results show that our proposed pipeline is very effective in learning fine-grained motion descriptions on both the ucf-101 dataset and the hmdb-51 dataset. in summary, our contributions are: video-text models: various efforts have been made to build video-text models @xcite mainly for video-to-text retrieval tasks. these models have developed mechanisms based on clip and extended them to video frames. video-text models @xcite @xcite @xcite have also been used for general video recognition both in the supervised and the zero-shot action recognition setting. @xcite introduced verb-focused contrastive training to improve better verb reasoning by learning with hard negative verb examples. @xcite introduced contrast sets to identify pitfalls in video-text models and recommended the need for fine-grained action understanding to tackle hard negatives in contrast sets. we differ from them as we utilize motion descriptions, which has its unique challenge. we compare our video-text model with vanilla clip @xcite , xclip @xcite , text4vis @xcite and vi-ficlip @xcite , primarily because they utilize the pre-trained clip for fair evaluation purposes. ## benchmark designing a motion description benchmark: we perform experiments on kinetics-400, ucf-101, and hmdb-51 datasets. for each class in these action datasets, we prompt gpt-4 to produce the characteristic motions of the action (given by the caption annotation). figure 2 shows the overall process of obtaining the motion descriptions. more details about the dataset statistics and generation are described in appendix d. the dataset can be accessed at https://github.com/ chinmayad/motiondescriptions.git we conducted a user study to evaluate the quality of generated motion descriptions. following @xcite , which questions the use of amazon mechanical turk for such studies, we conducted this study with expert graduate student volunteers who have taken courses in computer vision and natural language processing. the evaluators were shown different questions in a training session. all the evaluators were trained with different questions and explained the project's overall goal and how they contributed to it. we asked two volunteer graduate students of different ages and ethnicities to participate in this study. the following definitions were given. each of the above attributes is evaluated on a 5point likert scale. we report the mean 5-point likert score and iaa%, the inter-annotator agreement that measures the percentage of descriptions where annotators gave the same rating. we asked the volunteers to rate the generated motion description with the above attributes for each motion description in each dataset. the study results are given in table1. we noticed that many of the actions in the ucf-101 and hmdb-51 datasets involve objects, and the retrieval task is easier when an object is present in the generated motion description. for this reason, we also created another set of motion descriptions in which we replaced the names of objects with the generic word "object," which made the task slightly more challenging. ## proposed method this section describes our proposed video-text model that incorporates the textual motion description. a typical vision-language model like clip is usually trained using contrastive learning loss or its variations @xcite . the quality of representations learned from the video-text model @xcite will depend on the captions used during pre-training. since no large video-caption dataset containing motion descriptions exists, we can't directly train a video-text model using contrastive learning. we, therefore, propose a method that utilizes the rich linguistical understanding of motion in actions from gpt-4 to give us the captions required for training. our approach has two parts: generating the required motion descriptions of actions using gpt-4 described in section 3 and training videotext models to utilize these motion descriptions described in section 4.2. ## results task: for the target datasets ucf-101 and hmdb-51, the input is a video and the list of generated motion descriptions for all the classes in the dataset. the video-text model predicts the closest motion description that describes the video. the metric used is the percentage accuracy of correctly predicted motion descriptions. the models we evaluate are vanilla clip (radford et al., 2021), xclip @xcite , text4vis @xcite and vificlip @xcite 2023). details about the models are given in appendix b. human estimated performance: we sampled five videos randomly from each class in ucf-101 and hmdb-51. a human expert was asked to select the motion description that correctly describes the video from the list of descriptions generated by gpt4. human experts are graduate students who have taken computer vision and nlp graduate courses and volunteered for this study. ## conclusion we introduced a benchmark to understand how motion is understood in video-text models. we highlighted the limitations in obtaining quality annotations describing motion in video. we also showed that the performance of video-text models for retrieving motion descriptions is poor compared to human expert performance. our proposed method circumvents some of these issues and improves over other video-text models. while designing video-text models, we leave it to future work to build better models to capture motion and ignore the biases due to the object or the scene.