Revealing Single Frame Bias for Video-and-Language Learning (2023)

Abstract: Training an effective video-and-language model intuitively requires multiple frames as model inputs. However, it is unclear whether using multiple frames is beneficial to downstream tasks, and if yes, whether the performance gain is worth the drastically-increased computation and memory costs resulting from using more frames. In this work, we explore single-frame models for video-and-language learning. On a diverse set of video-and-language tasks (including text-to-video retrieval and video question answering), we show the surprising result that, with large-scale pre-training and a proper frame ensemble strategy at inference time, a single-frame trained model that does not consider temporal information can achieve better performance than existing methods that use multiple frames for training. This result reveals the existence of a strong “static appearance bias” in popular video-and-language datasets. Therefore, to allow for a more comprehensive evaluation of video-and-language models, we propose two new retrieval tasks based on existing fine-grained action recognition datasets that encourage temporal modeling. Our code is available at https://github.com/jayleicn/singularity .

## introduction video and language are the two primary signals that constitute much of the world we perceive every day -we observe our surrounding environment with our eyes in the form of continuous visual input (video), and communicate with others via language. intuitively, this leads one to assume that training an effective video-and-language model should require multiple video frames as input. standard methods @xcite @xcite in this area typically use multiple densely sampled frames for training. recent work @xcite proposes sparse sampling for video-and-language understanding, where it claims that a few sparsely sampled clips are sufficient for learning due to the high redundancy in videos. this technique has shown @xcite to be successful in various video-language benchmarks @xcite @xcite @xcite @xcite . however, as demonstrated by @xcite ; @xcite ; @xcite , training with fewer frames (e.g., a single frame) leads to significantly worse performance compared to their multi-frame counterparts. in contrast, in this work, we show that with proper modeling, single-frame models could achieve competitive performance, hence revealing "static appearance bias" in popular video-and-language datasets. we start by building a standard image-language model, with a vision encoder and a language encoder for image and text encoding, followed by a multi-modal encoder with cross-attention for crossmodal fusion. we pre-train the model on largescale image-text and video-text datasets @xcite @xcite @xcite . for fine-tuning, we randomly sample a single frame for training, and ensemble multiple uniformly sampled frames per video for making a video-level prediction at inference. single-frame predictions are often noisy and inaccurate, as they are made from incomplete information from single-frames without any context (see examples in figure 3 ). due to this issue, single-frame training typically performs significantly worse than multi-frame training @xcite @xcite . previous work @xcite suggests that pretraining improves model robustness in the face of label corruption for image recognition. inspired by this, we hypothesize that large-scale pre-training helps mitigate noise from single-frame training. our analyses in section 5 agree with our hypothesis, showing that as we increase pre-training data size, the performance of our single-frame model improves drastically and its gap with a similarly trained multi-frame model is largely eliminated. besides training, these noisy single-frame predictions also render simple late fusion (e.g., meanpooling in clipbert @xcite ) less effective at inference time. to deal with this issue, we propose an early fusion strategy, which takes all frames as model inputs for directly making a videolevel prediction. our analyses show that this early fusion ensemble method outperforms late fusion strategies and also delivers consistently improved performance when more frames are used. we compare our approach with existing methods on six datasets across two video-language tasks, including text-to-video retrieval (msrvtt @xcite , didemo @xcite , and activitynet captions @xcite ) and video question answering (msrvtt-qa @xcite , activitynet-qa @xcite , and msrvtt-mc @xcite ). results show that our approach achieves competitive (mostly better) performance than existing methods that use more training frames and pre-training data, setting new state-of-the-art for multiple tasks. this conclusion holds for both short 15-second msrvtt videos and 180-second activitynet videos, showing the effectiveness of our approach in various scenarios. more importantly, this strong single-frame performance reveals that the current evaluation is biased towards still objects, scenes, etc., while the temporal dynamics seem negligible, which in fact should be important for "true" video-language understanding. to address this issue, we next propose two new tasks that are designed to test models' true temporal modeling ability. based on the find-grained action recognition dataset something-something v2 (ssv2) @xcite , we create two text-to-video retrieval tasks, one that use ssv2's action template as text queries, e.g., "throwing [something] in the air and catching it", and another that uses its annotated label as text queries, e.g., "throwing keys in the air and catching it". see examples in figure 2 . this template task removes the objects and only keeps the actions, enabling an evaluation that focuses almost solely on temporal modeling. the label task, on the other hand, contains both actions and objects, requiring an understanding of both still objects and their motion. lastly, we present several baselines on these new tasks and show that temporal modeling is es-sential in achieving high scores. in summary, our contributions are three-fold: (i) we explore single-frame training for videolanguage tasks. while simple, our approach can achieve state-of-the-art performance on a range of datasets, including both text-to-video retrieval and video question answering. importantly, this result reveals the surprising static appearance bias in existing datasets. (ii) we conduct careful analyses, which show that large-scale pre-training and a proper multi-frame ensemble strategy are the core for single-frame trained models to be successful. (iii) we propose two new tasks specifically designed for testing models' temporal modeling ability. these two new tasks complement existing benchmarks for a more comprehensive evaluation. we denote a video v contains t frames as @xmath0, f 2 , ..., f t ], its paired text as s. during training, we randomly sample a single frame f t from v as model input , where @xmath1, ..., t }. its encoded representation can be written as f v (f t ) ∈ r lv×d . for text, the encoded representation is f l (s) ∈ r l l ×d . l v and l l are encoded sequence lengths, d is hidden size. we next make a prediction p as: q, k, v for self-att; q for cross-att k, v for cross-att where q, k, v denote the query, key, and value matrices of self-and cross-attention @xcite . we calculate loss based on this prediction. during inference, we uniformly sample t test frames {f τ i } ttest @xmath2. each frame is encoded separately, and their encoded representations are concatenated as inputs to the multi-modal encoder to get a video-level prediction score: @xmath3, [f v (f τ 1 ); ...; f v (f τ t test )] ), (2) where [; ] denotes concatenation, and [f v (f τ 1 ); ...; f v (f τ t test )] ∈ r (ttest×lv)×d . this early fusion design allows our model to make an informed prediction given full context. in clipbert @xcite , an alternative late fusion design is used: scores are computed for each frame separately, and video-level score is obtained via a manually designed aggregation function g (e.g., mean-pooling): since the predictions in late fusion are made with incomplete information from individual frames, they can be quite noisy. in section 5, we provide a detailed comparison w.r.t. these different frame ensemble methods and show that early fusion consistently outperforms late fusion. pre-training objectives. the model is trained with 3 losses: (i) vision-text contrastive: a contrastive loss that aligns the pooled vision and text representations from the vision and language encoders. (ii) masked language modeling (mlm) @xcite : predicting masked tokens from their text and visual context, with multimodal encoder. (iii) vision-text matching: predicting the matching score of a vision-text pair with multi-modal encoder. these losses have shown to be effective in learning multi-modal representations @xcite @xcite @xcite . more details are in appendix. implementation details. we use both image-text and video-text data for pre-training. for imagetext data, we use a combination of coco @xcite , visual genome (vg) @xcite , sbu captions @xcite , cc3m @xcite , and cc12m @xcite . for video-text data, we use webvid @xcite . note that, for videotext data, we only sample a single frame from the video for training. we pre-train the model on two different subsets of the datasets: (i) 5m corpus that contains 5.44m images and videos from cc3m+webvid, and (ii) 17m corpus that contains 17.28m images and videos from all the datasets. our model is implemented in pytorch @xcite . the vision encoder is initialized using beit base @xcite ) model trained on imagenet-21k @xcite . the text encoder is initialized from the first 9 layers of bert base @xcite . the multi-modal encoder is initialized from the last 3 layers of the same bert base model, and its cross-attention layers are randomly initialized. we optimize the model for 10 epochs using adamw @xcite with an initial learning rate of 1e-4. we warm up the learning rate in the first epoch followed by cosine decay @xcite to 1e-6. mixed precision is used for faster training. the batch size is set to 128/gpu, we train the model on 3 nvidia a100 gpus with image size 224×224. we perform basic augmentations: random resize, crop, and flip to the images during training. this pre-training takes around 1 day on the 5m corpus, and 4 days on the 17m corpus. our pre-training is quite efficient compared to other similar work, e.g., 10 epochs' pre-training in align-prompt @xcite takes 3 days on the same 5m corpus using 16 a100 gpus, this amounts to 16× computation cost of our pre-training. ## analysis frames ensemble strategy. our model is trained with a single-frame regime, while using multiple frames covering the full video at inference. as shown in figure 5a (concat), encoded video frames are concatenated as input to the multi-modal encoder's cross-attention for making a video-level prediction. a naive alternative is to compute the the highest score for each is indicated by ✓, the correct answer is highlighted in green. single-frame predictions are often inaccurate, unstable and they fluctuate across the frames. late fusion can be biased by inaccurate but high confidence frame predictions, e.g., the late fusion prediction is biased towards the 4th frame prediction. in figure 4 , we compare these different frame ensemble strategies, with varying number of frames at inference. from the comparison, we can draw the following conclusions: (i) our early fusion strategy (concat) shows a significant gain over the three late fusion strategies (lse, max, mean) for both msrvtt retrieval and activitynet-qa, demonstrating the importance of considering the whole video when making the predictions. (ii) in general, for all ensemble strategies, using more frames at inference improves model performance. however, a group of people play kites… (a) early fusion: concat (b) late fusion: lse/max/mean lang. enc. vision enc. multi-modal enc. concat is our early fusion strategy, lse, max, mean are the late fusion strategies in clipbert @xcite . for the late fusion strategies, sometimes using more frames hurts performance, e.g., for activitynet-qa, inference with over 4 frames underperforms that with 4 frames for max-pooling. this observation agrees with the msrvtt-qa results in clip-bert @xcite . in contrast, early fusion delivers consistently improved performance when more frames are used. overall, we hypothesize that the low and unstable performance of late fusion is because its video-level prediction is obtained via aggregating frame-level predictions, while these frame-level predictions can be inaccurate and unstable (see example in figure 3 ) -as they are separately predicted using incomplete information within each frame, ignoring their context. besides better accuracy, in appendix, we show early fusion also runs faster. pre-training data size. in figure 6 , we study the effect of cross-modal pre-training data size for both the single-frame and the multi-frame model. we show downstream fine-tuning performance under 4 different pre-training data setups: no cross-modal pre-training (0m), pre-train on webvid (2.49m videos), on 5m corpus (5.44m images+videos), or on 17m corpus (17.28m images+videos). we obsereve that both 1-and 4-frame model greatly benefit from large-scale pre-training. when comparing the two models, an interesting observation is that, as the pre-training data size increases, the performance gap between the 1-frame and the 4frame model decreases almost monotonically. this phenomenon suggests that, when pre-trained on a sufficient amount of data, the performance of models trained with single frames might be very close to models trained with multiple frames. though there can be exceptions for tasks that require fine-grained temporal modeling, such as ssv2-label retrieval, where multi-frame modeling is necessary. one possible explanation is that single-frame training is noisier than multi-frame trainingdue to incomplete context and random sampling, single-frame predictions are often inaccurate and less stable than multi-frame predictions, and pretraining is helpful @xcite sources from multiple frames. therefore, for downstream tasks, it is essential for the single-frame model to initialize from a strong pre-trained model. ## conclusion in this work, we explore single-frame training for video-and-language learning. we find that, with sufficient pre-training data and a proper frame ensemble strategy at inference, our model trained with a single frame achieves surprisingly good performance on various video-text tasks, including text-to-video retrieval and video question answering. while these results show the potential of using single-frame training for various video-text tasks, it also reveals that current benchmarks are biased towards static objects and scenes, etc. to address this issue, we propose two new tasks designed to test models' true temporal modeling ability and build several baseline methods for these new tasks. we hope these new tasks can complement existing benchmarks for a more comprehensive video-andlanguage understanding.