Video-Text Retrieval by Supervised Sparse Multi-Grained Learning (2023)

Abstract: While recent progress in video-text retrieval has been advanced by the exploration of better representation learning, in this paper, we present a novel multi-grained sparse learning framework, S3MA, to learn an aligned sparse space shared between the video and the text for video-text retrieval. The shared sparse space is initialized with a finite number of sparse concepts, each of which refers to a number of words. With the text data at hand, we learn and update the shared sparse space in a supervised manner using the proposed similarity and alignment losses. Moreover, to enable multi-grained alignment, we incorporate frame representations for better modeling the video modality and calculating fine-grained and coarse-grained similarities. Benefiting from the learned shared sparse space and multi-grained similarities, extensive experiments on several video-text retrieval benchmarks demonstrate the superiority of S3MA over existing methods.

## introduction as a fundamental task in visual-language understanding @xcite @xcite @xcite @xcite , video-text retrieval (vtr) @xcite @xcite @xcite @xcite has attracted interest from academia and industry. although recent years have witnessed the rapid development of vtr with the support from powerful pretraining models @xcite @xcite , improved retrieval methods @xcite @xcite , and videolanguage datasets construction @xcite , it remains challenging to precisely match video and language due to the raw data being in heterogeneous spaces with significant differences. sparse concepts original dense space sparse sentence rep sparse video rep sparse frame rep sentence rep video rep frame rep shared sparse space fine-grained similarity fine-grained similarity coarse-grained similarity coarse-grained similarity ## related works video-text retrieval (vtr), which involves crossmodal alignment and abstract understanding of temporal images (videos), has been a popular and fundamental task of language-grounding problems @xcite @xcite . most existing conventional video-text retrieval frameworks @xcite @xcite @xcite @xcite focus on learning powerful representations for video and text and extracting separated representations. inspired by the success of self-supervised pretraining methods @xcite @xcite and visionlanguage pretraining @xcite @xcite on large-scale unlabeled cross-modal data, recent works @xcite @xcite , ## methods in this section, we introduce our proposed framework for video-text retrieval, which aligns language and video in a shared sparse space. typically, in video-text retrieval, we have a set of examples {(v i , t i )} @xmath0, where n is the number of examples that are of video and language. ## conclusion in this paper, to better align video and text modalities, we proposed a multi-space, multi-grained video-text retrieval framework, s3ma. specifically, s3ma aligned different modalities in a finegrained shared sparse space, which is initialized with a finite number of concept clusters consisting of a number of basic concepts (words) and updated in a supervised fashion with the guide of text. besides, s3ma employed frame (finegrained) and video (coarse-grained) features to encourage models to perform multi-grained similarity alignment. finally, we conducted extensive experiments on three representative video-text retrieval benchmarks, showing the superiority of s3ma.