An Empirical Study of Frame Selection for Text-to-Video Retrieval (2023)

Abstract: Text-to-video retrieval (TVR) aims to find the most relevant video in a large video gallery given a query text. The intricate and abundant context of the video challenges the performance and efficiency of TVR. To handle the serialized video contexts, existing methods typically select a subset of frames within a video to represent the video content for TVR. How to select the most representative frames is a crucial issue, whereby the selected frames are required to not only retain the semantic information of the video but also promote retrieval efficiency by excluding temporally redundant frames. In this paper, we make the first empirical study of frame selection for TVR. We systemically classify existing frame selection methods into text-free and text-guided ones, under which we detailedly analyze six different frame selections in terms of effectiveness and efficiency. Among them, two frame selections are first developed in this paper. According to the comprehensive analysis on multiple TVR benchmarks, we empirically conclude that the TVR with proper frame selections can significantly improve the retrieval efficiency without sacrificing the retrieval performance.

## introduction text-to-video retrieval (tvr) aims to search for the most relevant video given a query text, involving intra-modal understanding and inter-modal alignment. videos are composed of a sequence of frames, each of which can be considered as the primary processing unit for video understanding and alignment with the text. however, due to the constraints of computing resources, it is impractical to process all frames of one video for tvr. therefore, existing methods @xcite @xcite select a subset of frames from the original videos to perform tvr. generally speaking, selecting fewer frames * corresponding author can benefit higher computational efficiency, while leading to the loss of visual details and thus impairing retrieval performance. considering both effectiveness and efficiency, it is imperative to adopt an optimal frame selection method for tvr. current frame selection methods can be divided into text-free frame selection and text-guided frame selection. the text-free frame selection only explores the video contents themselves while ignoring their relevancy to the texts in selecting frames. specifically, the conventional techniques @xcite @xcite for frame selection primarily rely on the color features of frames. however, these approaches fall short of comprehending the semantic content of videos and are not readily applicable to end-to-end training in various video-related tasks. for simplicity, existing works @xcite @xcite typically employ uniform frame selection (uni) or random frame selection (rand) to eliminate the temporally redundant frames that contribute little to the retrieval. in contrast, the text-guided frame selection is aimed at utilizing textual information to filter the inessential frames. we partition the text-guided frame selection into two categories: non-interactive frame selection (n-int) @xcite and interactive frame selection (int) @xcite @xcite . the former directly calculates the similarities between the representations of frames and the text from two unimodal encoders without interaction, and then selects the frames with high similarities as the representatives of the video. the latter constructs a cross-modal interactive module with extra parameters to assess the relevancy between frames and the text. although persistent efforts have been made in frame selection, there still exist three main problems. 1) the critical clues to the retrieval in a video typically exhibit a non-uniform distribution. the text-free frame selection with the uni/rand does not follow this distribution, which may filter out the key frames that carry the critical clues and eventually hamper the performance of tvr. and the textguided frame selection circumvents this problem with the aid of text information, but at the expense of simpleness. nevertheless, both of them have not considered the distribution of the video content itself. 2) the videos usually contain low-quality frames that are out-of-interest and less meaningful to tvr, leading to increased noise and decreased efficiency. these frames could caused by transitions, out-of-focus shots, inappropriate angles, etc. yet, such frames are overlooked by the current textfree frame selection methods. 3) currently, there is a noticeable absence of a comprehensive comparison and analysis of frame selection methods that take into account the aforementioned two factors. as a result, the optimal frame selection method for tvr still remains elusive, with no definitive answer yet. to this end, we extend the current frame selection methods and conduct a thorough empirical study of the frame selection methods to explore the optimal one in this paper. specifically, beyond the existing frame selections, we first develop a redundancy-aware frame selection (redun-a) method for the first problem and a low-qualityaware frame selection method (lq-a) for the second problem. redun-a selects the most representative and diverse frames by clustering the video frames, and lq-a chooses the frames of high quality by designing a scorer network and retaining frames with high scores, both of which belong to the text-free frame selection. then, based on the existing frame selections and the proposed ones, we investigate the effect of each frame selection method on tvr, as well as the potential combinations of them. through extensive study, we conclude that: (i) giving priority to the text-free frame selection, combining the proposed redun-a and lq-a is an optimal policy; (ii) by further considering the text-guided frame selection, the better trade-off between effectiveness and efficiency is achieved by the combination of redun-a and n-int. our main contributions can be summarized as follows: ## methods in this section, we first introduce the model architecture used for the empirical study of frame selection. then, we elaborate on the core content of our empirical study, including four text-free frame selection methods (i.e., uniform, random, redundancy-aware, and low-quality-aware frame selections) and two text-guided frame selection methods (i.e., non-interactive and interactive frame selections). finally, we present the training and inference processes. ## conclusion in this paper, we make an empirical study to explore the optimal frame selection methods for balancing the retrieval performance and efficiency in tvr. beyond the existing four frame selection methods, we additionally develop two text-free paradigms, i.e., redundancy-aware and low-qualityaware frame selection. we conduct abundant experiments to analyze the impact on tvr of the above six frame selections under the condition of various frame number, and make a comprehensive comparison from the perspective of retrieval efficiency and resource consumption. extended experimental results on multiple tvr benchmarks effectively demonstrate that with proper frame selection, we can not only get a competitive performance but also realize a higher retrieval efficiency under lower resource consumption. hopefully, this study could spur further study on deeper interaction between text and video in tvr.