R a P : Redundancy-aware Video-language Pre-training for Text-Video Retrieval (2022)

Abstract: Video language pre-training methods have mainly adopted sparse sampling techniques to alleviate the temporal redundancy of videos. Though effective, sparse sampling still suffers inter-modal redundancy: visual redundancy and textual redundancy. Compared with highly generalized text, sparsely sampled frames usually contain text-independent portions, called visual redundancy. Sparse sampling is also likely to miss important frames corresponding to some text portions, resulting in textual redundancy. Inter-modal redundancy leads to a mismatch of video and text information, hindering the model from better learning the shared semantics across modalities. To alleviate it, we propose Redundancy-aware Video-language Pre-training. We design a redundancy measurement of video patches and text tokens by calculating the cross-modal minimum dis-similarity. Then, we penalize the high-redundant video patches and text tokens through a proposed redundancy-aware contrastive learning. We evaluate our method on four benchmark datasets, MSRVTT, MSVD, DiDeMo, and LSMDC, achieving a significant improvement over the previous state-of-the-art results.

## related work video-language pre-training aims to learn joint representations between video and language. videos consist of consecutive frames and often contain visually similar redundant information. redundant information will bring two problems to videolanguage pre-training. one is the extra computational overhead, and the other is that the semantics of video and text cannot be well aligned. to alleviate the problem above, prior approaches @xcite @xcite @xcite use offline tools to extract video features, but cannot achieve end-to-end pre-training. clip-bert @xcite efficiently trains the video encoder end-to-end using only a few sparsely sampled frames. later, some well-performing methods @xcite @xcite adopt the sparse sampling strategy to alleviate temporal redundancy to reduce computational overhead. since we focus on mitigating the misleading caused by redundant information, rather than temporal redundancy, we follow @xcite to use sparsely sampled frames as input to the video encoder. to better align video and text, some recent works introduce fine-grained alignment in video-language pre-training @xcite @xcite . those works identify regions with objects in video via offline trained object detectors or prompters. then they align regions containing the objects with the text description, which alleviates inter-modal redundancy implicitly. unlike them, we explicitly propose an efficient redundancy measure that quantifies the impact of different redundancy. the most related work is oa-trans @xcite , which extracts objects and tags from an anchor frame with an offline detector, and uses the objectrelated regions or tags as additional input to reduce redundancy. unlike oa-trans @xcite , our learnable redundancy measurement matrix can be optimized end-to-end in pre-training. we do not rely on an offline detector, so we do not suffer the drawbacks of offline detectors. ## backgroud this section introduces some background knowledge of video-language pre-training, including video-text input, encoders, and contrastive learning for training. ## experiments we conduct experiments on the four most commonly used text-video retrieval benchmark datasets, which will introduced in the 5.3 section. following existing literature @xcite , we report recall@1 (r1), recall@5 (r5), recall@10 (r10) and median rank (mdr). ## conclusion in this paper, we summarize the inter-modal redundancy in video language pre-training and propose a redundancy measurement. then, we propose redundancy-aware contrastive learning to alleviate redundancy. significant improvements on sev-figure 4 : the pre-training tasks. in addition to our designed redundancy-aware contrastive learning task, there are auxiliary tasks: a video-text matching task, a video-grounded language model task and two intra-modal contrastive learning tasks.