GHAN : Graph-Based Hierarchical Aggregation Network for Text-Video Retrieval (2022)

Abstract: Text-video retrieval focuses on two aspects: cross-modality interaction and video-language encoding. Currently, the mainstream approach is to train a joint embedding space for multimodal interactions. However, there are structural and semantic differences between text and video, making this approach challenging for fine-grained understanding. In order to solve this, we propose an end-to-end graph-based hierarchical aggregation network for text-video retrieval according to the hierarchy possessed by text and video. We design a token-level weighted network to refine intra-modality representations and construct a graph-based message passing attention network for global-local alignment across modality. We conduct experiments on the public datasets MSR-VTT-9K, MSR-VTT-7K and MSVD, and achieve Recall@1 of 73.0%, 65.6%, and 64.0% , which is 25.7%, 16.5%, and 14.2% better than the current state-of-the-art model.

## introduction text-video retrieval (tvr) is a fundamental research task in multimodal video and language understanding, which aims to retrieve the most relevant video for a given text query, or to retrieve relevant text for a given video query. the mainstream modeling approaches for tvr are to jointly learn cross-modal interactive information between video and text in the same representation space @xcite @xcite @xcite . however, the mismatched problem of information capacity and information density between video and text is not fully studied in these approaches. as shown in figure 1 , there are structural and semantic differences between the modalities, making this approach challenging for tvr. the video itself expresses a much wider range of global representations than the text, so the textual semantic usually cannot be fully mapped to every detailed information of the video. if the content mentioned by the text appears in a small number of frames in video, the text may be misled by unrelated semantics when interacting across modalities. in addition, the local representations in text are usually more specific than that in video. due to different annotators, text may have different description habits and contain different semantic ranges, such as using person/man/actor to describe a character appearing in the video, which makes it difficult with semantic understanding and alignment across modalities. if text and video are divided into multiple parts and then aligned, the problem in figure 1 will be reduced. therefore, we can divide video into frames and clips, and text into words and phrases according to the structural hierarchy of them. from a semantic point of view, it also realizes the semantic hierarchy segmentation from abstract to specifics. moreover, different parts have different extents of importance for retrieval. in figure 2 , we call the parts related to query information "effective semantics". tvr aims to refine and align the effective semantics of paired text-video through structural hierarchy. through the step-by-step aggregation of structural and semantic hierarchy, intra-modality feature refining and cross-modality interaction are realized, so that the effective semantics of text and video can be aligned. to this end, we design a graph-based hierarchical aggregation network (ghan) to refine effective semantics within modality and align effective semantics cross modality. to achieve the former, a token-level weighted network is constructed to aggregate words and frames. as for the latter, we aggregate clips and phrases through graph-based message passing attention network for global-local alignment. our contributions are summarized below: ## related work vision-language research. visual-language research is currently a popular research field, including image-text research @xcite and video-text research @xcite . in the early days, visual-language models @xcite were usually designed independently. images were usually encoded using handcrafted descriptors @xcite . videos were mostly encoded using 2d/3d spatial-temporal convolution @xcite . texts were encoded using pre-trained word vectors @xcite or tf-idf features @xcite . recently, language pretraining models @xcite have achieved great success on nlp tasks. vision-language research has been similarly inspired @xcite @xcite . for image-text pre-training, @xcite , @xcite used two independent transformers @xcite to encode image and text respectively. as for @xcite ; @xcite , a shared bert model was then used. clip @xcite learns images directly from text, which leverages a wider range of source of supervision. @xcite proposed an end-to-end method through sparse sampling, which extracted visual and linguistic features with higher efficiency and lower memory usage. @xcite directly used clip and trained the model in an end-to-end manner. text-video retrieval. although text-image research @xcite ) has been extensively studied, text-video retrieval is still quite challenging. the earlier works, such as @xcite al. (2019) and gabeur et al. (2020) @xcite or netvlad @xcite , and fused multimodal features together. @xcite encoded features through transformer @xcite , calculating the similarity with the text and weighting different modalities according to the text. based on the idea of netvlad @xcite , @xcite adaptively aggregated sound, action, scene, speech, ocr, face, etc. in video and a series of shared semantics in text. with the rise of pre-training models, @xcite used the image-text pre-training model clip @xcite to encode the original video, which improved the retrieval accuracy. @xcite transfered clip @xcite to the video domain to solve text-video retrieval task. @xcite used moe and clip to extract multi-view video representations, including actions, entities, scenes, etc., and then aligned them with corresponding text parts. our model ghan also benefits from existing image-text pretraining models, but we are more concerned about the design of interaction between modalities, and extensive ablation studies are conducted to demonstrate the effectiveness of our method. ## method we propose an end-to-end graph-based hierarchical aggregation network, aiming to obtain features with the simplest structure and the most abundant semantics for tvr. figure 3 describes the architecture of our model named ghan. details are described in the rest of this section. ## conclusion in this paper, we propose a graph-based hierarchical aggregation network named ghan for textvideo retrieval. we noticed that text and video have structural and semantic hierarchies. in this regard, we propose the concept of effective semantics, which maps retrieval tasks to refine and align effective semantics between modalities. we design a token-level weighted network to refine intramodality features, and build a message passing attention network for global-local alignment across modality. our model significantly improves on multiple datasets. more experiments indicate that our model can well achieve semantic refining and alignment.