Text-Derived Knowledge Helps Vision: A Simple Cross-modal Distillation for Video-based Action Anticipation (2023)

Abstract: Anticipating future actions in a video is useful for many autonomous and assistive technologies. Prior action anticipation work mostly treat this as a vision modality problem, where the models learn the task information primarily from the video features in the action anticipation datasets. However, knowledge about action sequences can also be obtained from external textual data. In this work, we show how knowledge in pretrained language models can be adapted and distilled into vision based action anticipation models. We show that a simple distillation technique can achieve effective knowledge transfer and provide consistent gains on a strong vision model (Anticipative Vision Transformer) for two action anticipation datasets (3.5% relative gain on EGTEA-GAZE+ and 7.2% relative gain on EPIC-KITCHEN 55), giving a new state-of-the-art result.

## introduction anticipating future actions in the video of an unfolding scenario is an important capability for many applications in augmented reality @xcite , robotics @xcite , and autonomous driving @xcite . anticipating what actions will likely happen in a scenario, requires one to both recognize what has happened so far, and use anticipative general knowledge about how action sequences tend to play out. most models for this task use a pre-trained video encoder to extract information about what has happened so far in the scenario, and use a text-based decoder to predict what action is likely to happen in the future @xcite @xcite . however, when trained on the target video datasets, the generalization of the models depends figure 1 : a model learning the action anticipation from only the vision modality (video frames) is essentially exposed to a very limited set of action sequences. language models, which are pre-trained on large-scale text, can learn this distribution from the task, and a much larger domain-relevant text. we propose distilling this knowledge from text modality models to vision modality model for video action anticipation task. on how well these video datasets cover the space of action sequence distributions. in other words, the knowledge that is learnt for predicting future actions is, in effect, limited to the information in the target video datasets, where obtaining large scale coverage of action sequences is difficult. knowledge about action sequences can also be obtained from text resources at scale. language models, (e.g. bert @xcite , roberta @xcite ), are typically pretrained on large collections of unlabeled texts with billions of tokens, where they acquire a widevariety of knowledge including large scale knowledge about common action sequences. for example, ## related work there has been a wide range of solutions for action anticipation ranging from hierarchical representaions @xcite , unsupervised representation learning @xcite , to encoder-decoder frameworks that decode future actions at different time scales @xcite , and transformers trained on multiple auxiliary tasks @xcite . however, these only use the vision modality features of the observed video to train the model for the anticipation task. our work aims to distill text-derived knowledge to improve action anticipation. here we relate our work to others that have made use of (i) textual knowledge for related tasks, (ii) general knowledge distillation, and (iii) multimodal models which also allow for integration of information from different modalities. ## language-to-vision knowledge distillation for action anticipation the action anticipation task asks to predict the class label of a future action based on information from an observed video sequence. in this task setting, the model has access to both, video and annotated action segments (action text) during the train time, but needs to make the inference only using the video sequence. the input to the prediction model is a sequence of video frames up until time step t: @xmath0, x 2 , . . . , x t ), and the desired output of the model is the class label y of the action at time t + τ , where τ is the anticipation time. to learn an anticipation model, we assume there is training data of the following form: our task is to learn a model g that can predict the future action label based on the vison modality of the video sequence x i only. a common approach is to optimize cross entropy loss l between the model's predicted label g(x i ) and the ground truth label y i of each training instances, i.e., to minimize: although the sequence of action labels l i is available in the training data, the semantics associated with these labels is not properly used by the existing methods for training the anticipation model. here we propose to learn a text-based anticipation model g text and use it to supervise the training of the vision-based anticipation model g. this training approach utilizes the knowledge from the text domain, which is easier to learn than the visionbased knowledge, given the abundance of event sequences described in text corpora. hereafter, we will refer to the language-based model as the teacher, and the vision-based model as the student. ## results and analysis we present the results of text to video knowledge distillation on the avt @xcite model as the student. avt is the state-of-the-art model for action anticipation on the egtea-gaze+ and epic-kitchen 55 datasets on all performance metrics. for each of these datasets, we consider the avt variants with the best performance as our baseline and student model. for the egtea-gaze+ dataset, we consider avt-h + avt-b in @xcite as our baseline model. similarly for the epic-kitchen 55 dataset, we consider, avt-h + ircsn152 in @xcite as our baseline model. 3 throughout this section, we refer to avt-h + avt-b and avt-h + ircsn152 as avt-1 and avt-2 respectively. the baseline models distilled with lm based teacher is denoted as avt-1(or 2) + lm distillation and in case teacher lm is pre-trained on the recipe domain text, the resulting model is referred to as avt-1(or 2) + rcplm distillation. we tried to reproduce the avt model to use as our student and obtain stronger results than the published version (see ## conclusions action anticipation is a challenging problem that requires training large capacity video models. in for the egtea-gaze+ dataset, the teacher performance is much lower than the video only avt model, where as for the epic-kitchen 55 dataset, the teacher performance is much better than the video-only avt model. ♣ this work, we showed how the textual modality of the input videos, which is often ignored in training, can be leveraged to improve the performance of the video models. in particular, we can exploit the large scale knowledge acquired by pre-trained language models to build a text-modality teacher that can provide useful complementary information about the action sequences to a vision modality student. this cross-modal distillation strategy yields consistent gains achieving new state-of-the-art results on multiple datasets. last, the gap between the performance of the teacher and the student models for domains with large label space suggests that there is still room for improvement with better distillation techniques.