Contrastive Video-Language Learning with Fine-grained Frame Sampling (2022)

Abstract: Despite recent progress in video and language representation learning, the weak or sparse correspondence between the two modalities remains a bottleneck in the area. Most video-language models are trained via pair-level loss to predict whether a pair of video and text is aligned. However, even in paired video-text segments, only a subset of the frames are semantically relevant to the corresponding text, with the remainder representing noise; where the ratio of noisy frames is higher for longer videos. We propose FineCo (Fine-grained Contrastive Loss for Frame Sampling), an approach to better learn video and language representations with a fine-grained contrastive objective operating on video frames. It helps distil a video by selecting the frames that are semantically equivalent to the text, improving cross-modal correspondence. Building on the well established VideoCLIP model as a starting point, FineCo achieves state-of-the-art performance on YouCookII, a text-video retrieval benchmark with long videos. FineCo also achieves competitive results on text-video retrieval (MSR-VTT), and video question answering datasets (MSR-VTT QA and MSR-VTT MC) with shorter videos.

## introduction human perception is multimodal, including visual, textual, and audial information. to achieve humanlevel perceptional ability, intelligent systems need to understand and interpret these multimodal signals and summarise the relevant information in them. learning from video and language data has received significant attention in recent multimodal machine learning work for downstream tasks that require joint understanding of video and textual information, including text-video retrieval @xcite @xcite @xcite , video question answering @xcite @xcite @xcite ; lei fli p the pan cak es whe n the edg e tur ns bro wn min ce the tun a and add it to a bow l figure 1 : illustration of the weak correspondence problem in video-language learning. given a pair of video and its text (e.g. caption, instruction, or transcription), only a subset of the frames (here indicated by coloured bounding boxes) is semantically aligned to the textual content. the remaining frames represent irrelevant visual information and will not contribute to language grounding on videos. @xcite , and video captioning @xcite @xcite . in most of this work, contrastive learning @xcite is used as training objective. the aim of a cross-modal contrastive loss is to maximise the similarity between an aligned videotext pair while minimising the similarity for all other pairs. one issue with standard cross-modal contrastive loss is that it focuses on pair-level alignment but ignores the negative effects of irrelevant frames that are present in a single video clip, even in a pair of aligned video and text. we define irrelevant frames as those with no or little shared semantics with the text. these irrelevant frames may negatively affect the contribution of frames that are semantically similar to the text, which further results in less informative video representation. therefore, we posit that frame-level learning is a better strategy for video-language tasks. in this paper, we propose fineco, an approach that has a frame selector to sample relevant frames in a video and is trained with a fine-grained con-trastive loss on frame-text pairs, in order to mitigate the problem of weak correspondence in videolanguage representation learning. existing videolanguage learning approaches @xcite only optimise pair-level alignment but do not explicitly learn which part of a video contributes to its alignment with the text. fineco focuses on aligning relevant frames with the text. it is inspired by the text-based temporal localisation task @xcite , however, the motivation of fineco is different: to learn better video-level representation by adding a frame-level contrastive learning signal to the pair-level objective, with no need for temporal annotation within a video-text pair. we hypothesise that fineco is particularly beneficial for long videos, where each video provides more information and only a small proportion of frames will be relevant to its text counterpart, as shown in figure 1 . fineco is able to model frame-text similarity through fine-grained contrastive learning, where the most informative frames are paired with the text as positive pairs and the remaining frames, as negatives. it then explicitly contrasts the selected informative frames against the noisy frames, without the need for frame-text annotations. this frame-level distillation provides a strong learning signal, which encourages the alignment of semantically equivalent video-text pairs. the fine-grained contrastive loss abstracts the learning signal from pair-level annotations and is trained in an end-to-end manner. this combination of pair-level learning signal and framelevel contrastive loss is novel and effective, and boosts the performance on two important videolanguage benchmark tasks, especially in text-video retrieval with longer videos. we devised fineco by building on the recently proposed and well performing videoclip @xcite , in which a video clip is represented as sequence of frame features. our contributions are summarised as follows: video-language learning @xcite @xcite @xcite @xcite have shown promising results for video-language learning with pre-training followed by fine-tuning. this strategy has become very prominent since the release of bert @xcite and many image-text pre-training frameworks @xcite @xcite @xcite @xcite @xcite . the release of datasets such as howto100m @xcite and webvid-2m @xcite has enabled large-scale pre-training on unlabelled video-text pairs to improve representation learning of video and language. many approaches @xcite @xcite use howto100m as their pre-training dataset. fit @xcite uses webvid-2m and google conceptual captions (cc3m) to take advantage of the large collection of video-text and image-text pairs for pre-training. however, large pre-training datasets rely on loosely aligned videotext pairs, without any fine-grained supervision on alignment. this makes it difficult to learn crossmodal cues present in the given video-text pairs. it is also computationally expensive to improve video-language representation learning, given that videos can contain a large number of frames, especially longer videos. clipbert @xcite randomly samples a few frames from a video for video-language representation learning. their motivation is to minimise memory and computation costs from processing the full sequence of frames. this sampling strategy is over simplistic and can thus be improved by better approaches to select frames based on their relevance to the paired text. ## experiments in this section, we describe the tasks and datasets used in our experiments with fineco. ## results in this section, we describe the experimental results and compare fineco with state-of-the-art approaches (section 5.1). we further explore different sampling strategies to select positive frames (section 5.2), and fine-grained word sampling (section 5.3). we also provide examples of the frames selected by fineco (section 5.4). ## conclusions we propose fineco, an approach with a finegrained contrastive loss to mitigate the weak correspondence problem in video-language representation learning. experiments conducted on text-video retrieval and video question answering datasets suggest that fineco can distil video frames that are relevant to its corresponding text and contribute to significant gains in performance, especially on the text-video retrieval dataset youcookii with long videos. fineco achieves state-of-the-art on youcookii and msr-vtt mc, and for text-video retrieval datasets with shorter videos, it substantially improves over the base model. ablation studies analyse the key factors in fineco including number of positive frames and word sampling. our strategy for frame selection is simple and can generalise to different video-language frameworks, as long as they are based on contrastive learning, which is standard in this area. in addition, we posit that fineco can be useful for video-language pretraining on large loosely or misaligned video-text datasets. we hope that our work will draw attention to the need for frame-level alignment to improve video-language representation learning.