V ideo CLIP : Contrastive Pre-training for Zero-shot Video-Text Understanding (2021)

Abstract: We present VideoCLIP, a contrastive approach to pre-train a unified model for zero-shot video and text understanding, without using any labels on downstream tasks. VideoCLIP trains a transformer for video and text by contrasting temporally overlapping positive video-text pairs with hard negatives from nearest neighbor retrieval. Our experiments on a diverse series of downstream tasks, including sequence-level text-video retrieval, VideoQA, token-level action localization, and action segmentation reveal state-of-the-art performance, surpassing prior work, and in some cases even outperforming supervised approaches. Code is made available at https://github.com/pytorch/fairseq/examples/MMPT .

## introduction the popular "pre-training + fine-tuning" paradigm has revolutionized nlp @xcite @xcite and cv @xcite over the last few years. although models trained this way can achieve impressive performance, they still require task-specific annotated data and fine-tuning for each end task. recent work adopt pre-training for zero-shot transfer to end tasks without finetuning, including gpt @xcite @xcite for nlp tasks and clip @xcite for image classification. this paper focuses on pre-training for zero-shot transfer to video-text understanding tasks. our approach pre-trains a transformer model @xcite with a contrastive objective @xcite using pairs of video-text clips. different from clip that scales pre-training data for zero-shot transfer to image classification on an explicitly assembled dataset using a simple contrastive objective @xcite , this paper uses a publicly established video 1 first, you need.. next, pour sauce.. 1 overlapping video-text clips 2 retrieve harder videos video k videoclip: contrastive learning with hard-retrieved negatives and overlapping positives for video-text pre-training. … time how to season a wok video 2 now, wipe with a paper towel ## videoclip pre-training in the paradigm of multi-modal video-text pretraining for zero-shot transfer, the key challenge is to learn fine-grained association in-between video and text to cover the diverse needs of end tasks. we cover videoclip pre-training in this section, and discuss the needs of zero-shot transfer to different end tasks in the next section. we first describe video and text model backbone and contrastive loss; then we propose overlapped video and text clips to improve the association of positive pairs; lastly, we describe retrieval augmented pre-training to improve the mining of negative examples. ## end our overall training process can be described as a two-stage method that alternatively performs retrieval and training in each epoch, and is summarized in algorithm 1. for each epoch, line 2-4 corresponds to the retrieval stage and line 5 corresponds to the training stage. specifics are as folows. line 2 computes the global features z v for each video by averaging the embeddings of all of its video-text clips. an ablation (in appendix) shows that this is better than using the starting clip of a video to infer the representative video embedding. line 3 constructs the dense index foot_1 for all videos to be used in our retrieval-based training. line 4 first finds |c| (corresponds to the number of overall batches in the training set) random videos, where each video v yields a video cluster c as follows. we sample |c| videos from k neighboring videos of v . instead of searching k nearest videos directly (see ablation in ## zero-shot transfer to end tasks we present methods for zero-shot transfer of videoclip to a variety of end tasks (without using any labels). for each task, we specify requirements that highlight the aspect of pre-training. @xmath0. @xmath1. nce(z t , z v ) in equation 4 contributes to this task as it discriminates different video clips in the numerator and denominator for a given text clip. it also tests the distribution of hard negative examples in the denominator given it reports multiple recall metrics. multiple-choice videoqa. in multiple-choice videoqa @xcite , the model aligns each video with one out of several text candidate answers. it tests @xmath2. we formulate this task as ranking candidate textual answers for a given video question query. this corresponds to the nce(z v , z t ) term in equation 4, where the subtle differences in texts are discriminated against each other. action segmentation. action segmentation assigns each token (or frame) of a video with one of the pre-defined labels to separate meaningful segments of videos from the rest tokens (or frames). this is similar to sequence labeling (e.g. named entity recognition (ner)) in nlp. inspired by the setup of clip @xcite , the text encoder of videoclip can serve as self-supervision for videos during pre-training and as a hyper network to provide hidden states of segment textual labels for a video token. as such, the hidden state of each video token can have a distribution of similarity over segment labels. this task tests video token to text similarities. one challenge in action segmentation is that it contains an outside label that does not exist in transcription during pre-training. this outside label is task-dependent because it means a token does not belong to any of the pre-defined labels. this is similar to open set recognition @xcite or out-of-domain intent detection @xcite , where the rejection label is not presented during training but all new classes during inference (not shown in training) should be covered by the rejection label. let @xmath3. we apply the following conditions to each video token u to curate the prediction with the outside label ŷu : ## conclusion we have presented videoclip, an approach to pretrain a video-text model for zero-shot transfer to end tasks that require fine-grained association between video and language. videoclip uses an objective that contrasts temporally overlapping positives with hard negatives stemming from nearest neighbor retrieval. in evaluation this approach outperforms prior work on a variety of tasks, without any supervision on downstream datasets, and in some cases videoclip is competitive or better than prior work that uses full supervision; nevertheless, we still observe gains for fine-tuning our model. we hope that our code and model will foster future research in multi-modal video understanding. dimitri zhukov, jean-baptiste alayrac, ramazan gokberk cinbis, david fouhey, ivan laptev, and josef sivic. 2019. cross-task weakly supervised learning from instructional videos. in proceedings of the ieee/cvf conference on computer vision and pattern recognition, pages 3537-3545.