Memory-efficient Temporal Moment Localization in Long Videos (2023)

Abstract: Temporal Moment Localization is a challenging multi-modal task which aims to identify the start and end timestamps of a moment of interest in an input untrimmed video, given a query in natural language. Solving this task correctly requires understanding the temporal relationships in the entire input video, but processing such long inputs and reasoning about them is memory and computationally expensive. In light of this issue, we propose Stochastic Bucket-wise Feature Sampling (SBFS), a stochastic sampling module that allows methods to process long videos at a constant memory footprint. We further combine SBFS with a new consistency loss to propose Locformer, a Transformer-based model that can process videos as long as 18 minutes. We test our proposals on relevant benchmark datasets, showing that not only can Locformer achieve excellent results, but also that our sampling is more effective than competing counterparts. Concretely, SBFS consistently improves the performance of prior work, by up to 3.13% in the mean temporal IoU, leading to a new state-of-the-art performance on Charades-STA and YouCookII, while also obtaining up to 12.8x speed-up at testing time and reducing memory requirements by up to 5x.

## introduction processing long untrimmed videos for understanding and reasoning is a computationally expensive task that not only demands a smart approach that can capture global and local interaction, but also requires allocating thousands of frames in memory @xcite . previous work so far has mostly focused on parsing the visual content in independent snapshots of a video, which limits the approaches in terms of modeling long-term dependency between events in the input. out-of-memory video duration [s] gpu memory @xcite locformer transformer-base transformer-base + sampling dori figure 1 : empirical upper-bound memory consumption of locformer, transformer-base, with and without enabling our proposed sampling technique (sbfs), and dori @xcite . memory usage (y-axis) is computed on an nvidia rtx-8000 gpu, using a batch size of 32 and assuming all sequences have maximum length for the given video duration. in this paper, we particularly look at the task of temporal moment localization, which aims to identify the start and end timestamps of a moment of interest in an input untrimmed video given the query in natural language @xcite @xcite @xcite . recent approaches have aimed at directly predicting the starting and ending temporal locations, or regressing them from the input video, moving away from the proposeand-rank based approaches @xcite @xcite . although these models are more efficient and can potentially capture the influence between different events in a video, they still require a considerable amount of computation and/or memory since they need to process the whole input video at once. recent improvements in both natural language processing (nlp) and computer vision (cv) tasks can be attributed to the transformer @xcite model. despite their success, one main draw-back of these models is their computational cost, with memory usage ballooning as model sizes increase to attain better performance. this issue is exemplified by many existing developments in transformer models for video understanding only being capable of processing short inputs at a time due to memory constraints. for example, timesformerlarge @xcite can only process inputs that are 24 seconds long foot_0 , vivit @xcite and x-vit @xcite can receive inputs that are up to 32 frames long, while other approaches like merlot @xcite and clipbert @xcite specifically sample a single or a few frames from the whole video. in light of this issue, we present stochastic bucket-wise feature sampling (sbfs). during training, sbfs first splits the input feature sequence into a fixed number of sections (buckets) and then selects a single feature per bucket using a stochastic approach. at testing time, our module selects the sufficient statistic of each bucket, which allows us to obtain a feature sample-set that is representative of the video content for the task at hand while keeping the memory footprint constant to show the effectiveness of our sampling technique, we combine sbfs with a broad selection of models from previous work, and also use it to propose locformer, a transformer-based model that operates at a constant memory footprint regardless of the input length, as shown in figure 1 . we conduct experiments on three challenging datasets, charades-sta @xcite , activ-itynet captions @xcite and youcookii @xcite . we show that by applying sbfs in previous work we improve their performance in all considered datasets, leading to a new state-ofthe-art on charades-sta and youcookii, while reducing the memory requirements by up to 5x. furthermore, locformer is able to obtain stateof-the-art performance in the latter, and competitive results elsewhere. we believe our results highlight the importance of sampling techniques as a valid mechanism to obtain better coverage of long input videos while keeping memory usage under budget. this ultimately provides a concrete direction for further research on tasks where it is necessary to cover long untrimmed videos, which include but are not limited to video grounding. we release our code and data foot_1 to encourage future research in this area. ## related work sampling to the best of our knowledge, the earliest example of a sampling technique that is similar to ours is the work of @xcite , who proposed a stochastic version of dynamic time warping for speech recognition of the japanese language in the late 80s. this idea was further extended by @xcite in the context of motion recognition, where a randomized version of dynamic time warping for this task was introduced. we also find several models that utilize sampling techniques for action recognition in videos, in this case to specifically select salient clips, such as scsampler @xcite and mgsampler @xcite , which were later adopted by models like mvfnet @xcite . also, adaframe @xcite recently proposed a framework that adaptively selects relevant frames on a per-input basis for fast video recognition. @xcite introduced differentiable parameter-free adaptive token sampling (ats), which uses a scoring to adaptively sample significant tokens and can be plugged into any existing vision transformer architecture. finally, @xcite , recently proposed to process short video segments at a time also augmenting the models with a long-term bank from where global features can be sampled. this ultimately enables them to process longer overall inputs without losing global information. while their motivation is similar to ours, we note that their proposed sampling technique is not stochastic and requires learning. moreover, our results show that performance improvements are possible without access to global context, making our contributions fundamentally different in this sense. ## stochastic bucket-wise feature sampling let g be a sequence of input video features extracted by a video encoding function f v (v ). we are interested in developing a module to limit the memory budget of a given model when dealing with long video inputs. in order to do this, we limit the overall memory budget of the model by shortening the sequence of video features fed into the localization module. we do this in practice by proposing a technique that we call stochastic bucket-wise feature sampling (sbfs), which returns a sequence of length at most b derived from g, as follows. in equation 1 , m(n, b) characterizes the number of buckets to be allocated to host video features, and is defined as m(n, b) = ⌊ n ⌈n/b⌉ ⌋ ≤ b, where ⌊⌋, ⌈⌉ are the floor and ceiling operators, respectively. the index f (k) is sampled according to a uniform distribution over the indices of the features in the bucket, as equation 2 shows, below. more intuitively, we create a fixed number of buckets and allocate features to each by equally distributing them into the buckets. during training, we randomly sample a single feature for each bucket, following a uniform distribution, effectively reducing the input sequence length to at most b, the number of buckets. when doing this, we also accordingly convert the original set of labels τ s , τ @xmath0, . . . , n] into τ s , τ @xmath1, . . . , b]. for simplicity, without loss of generality, for the rest of the paper we will assume the sampling module always returns total of b video features. it is possible to prove that any sampled video feature sequence g obtained using sbfs contains sufficient statistics of g, which allows us to train our models on very long videos with adequate guarantees. from this, it also follows that although sbfs can also be applied during inference, it is better to decouple the model from this stochastic component and instead utilize the max pooling operator over the features of the bucket at inference time. this gives models increased stability when predicting, without sacrificing performance, as we will show in section 5.3. we refer the reader to section a of our supplementary material for the full details on our theoretical analysis of sbfs. ## locformer in what follows, we assume that a given video @xmath2, . . . , l. each video in v is annotated with a natural language passage @xmath3: our method uses a sampling technique that divides the video into a fixed number of buckets. then, we uniformly sample a single i3d feature from each bucket, which is then fed into the localization module, a multi-modal transformer model that receives video features and language features obtained from bert. our model is trained to predict the most likely temporal localization of the contents of a given input query s in terms of its start and end positions t s⋆ and t e⋆ in the video. we apply the mapping τ = (t • n • fps)/l to transform frame/feature index to time, converting t s and t e into τ s and τ e , which correspond to specific integer feature positions such that τ s , τ @xmath4, . . . , n]. locformer follows the transformer architecture @xcite , which has been recently extended to multi-modal scenarios as in uniter @xcite in the context of vision-andlanguage, and recurrent vln-bert @xcite for vision-and-language navigation. our model operates on sequences of tokens {s j } and a video {v t } characterized as a sequence of frames, as specified earlier. the overall architecture is composed by three main modules, as shown in figure 2 . video encoding module with sbfs our video encoding module is in charge of mapping the l input video frames into a sequence of video features @xmath5, @xmath6, . . . , n. at the core of this module lies sbfs, which we use to select a subset of the features that are representative of the contents of the video. thus, the sampled video feature sequence g will later be fed into the localization module. text encoding module sentences are processed using the bert tokenizer, which also prepends the special cls token, and adds the sep marker at the end. each token is mapped to learned embeddings of dimension d m and summed with learned positional encodings of the same size. these vectors are passed through l encoder transformer blocks with m attention heads, to produce final text representations [ h0 , . . . , hm ]. localization module a transformer model that receives both textual and video features, previously obtained by the respective modules. for the former, we directly input h0 , . . . , hm , while for the latter we first project @xmath7, . . . , gb ] into the hidden dimension using a trainable linear layer and further combine this with a set of learned positional encodings. these two encoded vector sequences are concatenated lengthwise and passed through l encoder blocks with m attention heads, to produce from these vectors, we select [h m+1 , . . . , h m+b ] and utilize the same localization function proposed by @xcite as the main training signal, namely, feature-level soft classification task on the time dimension. concretely, two different mlp layers produce scores of each position being the start/end of the location, which are passed through a softmax activation to obtain τ s , τ @xmath8, which are compared to soft-labels using the kullback-leibler divergence (l kl ). in order to guide the model to utilize the information in the relevant section of the video, we encourage the attention heads of this module to put more weight into the target video portions during training, adapting the approach proposed by @xcite as shown below. in equation 3 , a (l,m) is the attention matrix of the l-th layer and m-th attention head of the localization module and @xmath9, and is defined as @xmath10; δ τ @xmath11, where ; denotes concatenation, δ is the kronecker delta returning 1 when i is inside the range of τ , and 1 k denotes a vector of ones of size k. since our proposed localization loss is not sensitive to the order of the predictions of the starting and ending locations because there is no conditioning on the time in the model portions that generate them, in this paper we additionally propose to induce the model to respect the start-end order, taking a probabilistic approach. concretely, we push the expected location of the start of the segment (s) to be before the expected location of the ending (e) location, which is equivalent to requiring e(e) -e(s) > 0. replacing the values of the expectations, we obtain the following. in equation 4 above, τ s i and τ e i are integers that denote the predicted probability value of the starting and ending localizations at position i. based on this derivation, we formally implement our loss by minimizing the negative difference of the expected values as shown in equation 5, below. finally, our model is trained with the direct summation of the three losses introduced earlier, such that @xmath12. ## conclusion in this paper we have presented locformer, a transformer-based model for the task of temporal moment localization which operates at a constant maximum memory footprint regardless of the input length. the success of our model fundamentally relies on our modular design, which allows us to separate functionality, and sbfs, where we split the sequence of input video features into a fixed number of buckets and select a single feature per bucket using a stochastic approach. experiments conducted on three challenging datasets show that locformer obtains competitive results, and show that our sampling technique can improve the performance of prior work on all considered datasets, leading to a new state-ofthe-art on youcookii and charades-sta. we think these results highlight the importance of sampling techniques as a valid mechanism to obtain better coverage of long input videos while keeping memory usage low. for future work, we are interested in testing our sampling approach in other relevant tasks in the context of video-and-language, for example video retrieval. we are also interested in extending our approach to address its limitations, for example, using adaptive or iterative sampling to treat different areas of the video with different granularity.