Towards Parameter-Efficient Integration of Pre-Trained Language Models In Temporal Video Grounding (2023)

Abstract: This paper explores the task of Temporal Video Grounding (TVG) where, given an untrimmed video and a query sentence, the goal is to recognize and determine temporal boundaries of action instances in the video described by natural language queries. Recent works tackled this task by improving query inputs with large pre-trained language models (PLM), at the cost of more expensive training. However, the effects of this integration are unclear, as these works also propose improvements in the visual inputs. Therefore, this paper studies the role of query sentence representation with PLMs in TVG and assesses the applicability of parameter-efficient training with NLP adapters. We couple popular PLMs with a selection of existing approaches and test different adapters to reduce the impact of the additional parameters. Our results on three challenging datasets show that, with the same visual inputs, TVG models greatly benefited from the PLM integration and fine-tuning, stressing the importance of the text query representation in this task. Furthermore, adapters were an effective alternative to full fine-tuning, even though they are not tailored to our task, allowing PLM integration in larger TVG models and delivering results comparable to SOTA models. Finally, our results shed light on which adapters work best in different scenarios.

## introduction temporal video grounding (tvg) is a fundamental task in computer vision (cv), where the goal is to have models recognize and determine temporal boundaries of action instances in videos @xcite @xcite using queries provided in natural language @xcite . over the past few years, interest in this task has grown substantially due to its complexity and potential applications, which has led to the release of several models that either use propose-and-rank techniques or directly predict the starting and ending temporal locations @xcite . following trends in other vision-and-language (v&l) tasks, some of the latest models combine vision encoders and pre-trained language models (plm). we find works that directly encode the query using plms @xcite , which are later fine-tuned with the rest of the architecture, or that try to project both the query and video into the same embedding space using a transformer @xcite . despite the performance improvements, it is difficult to isolate the effects of the improved language representations, as these works also propose new video-language matching approaches or use different video encoders. another drawback is their computational cost for training, as parameter counts grow substantially once plms are incorporated. to address this problem, several parameterefficient training methods have been recently proposed for both natural language processing (nlp) @xcite and cv models @xcite . among these approaches, adapters @xcite and their variations have been particularly effective foot_0 , as they lead to performance as high as fine-tuning while training only a small set of parameters. adapters have been successfully combined with vision models for several tasks @xcite , showing that using a few parameters to learn to fuse vision and language representations without losing performance is possible. however, we note that efforts so far have focused only on image @xcite and video @xcite classification tasks or on leveraging pre-trained generative models, e.g., by re-casting existing vision-and-language tasks as language generation @xcite . in contrast, as the training signal in tvg comes from the visual modality, we cannot cast it as language generation. therefore, this paper studies the effects of large plms in the tvg task and investigates the applicability of nlp adapters for a parameter-efficient integration. we couple popular plm models with a selection of previous works, allowing us to isolate and understand their effects on performance. concretely, we analyze excl @xcite , tmlga @xcite and dori @xcite , three proposal-free tvg models with different levels of complexity. moreover, we also benchmark several parameterefficient training alternatives based on adapters. we conduct thorough experiments on three challenging datasets, charades-sta @xcite , activitynet captions @xcite and youcookii @xcite , covering videos and queries with varying lengths of different activities. concretely, we seek to answer the following research questions: rq1: does incorporating a plm improve the performance of existing tvg models?; rq2: are adapters an alternative to full fine-tuning of the plm parameters within tvg?; rq3: is there an adapter that works best for tvg?; rq4: what is the impact of different plms?; rq5: how does the combination of existing tvg models with plms trained with adapters perform against state-of-the-art models? our results offer concrete answers to these questions, helping us clarify the role of plms in the tvg task and quantify how much they can improve the existing model's grounding capabilities. they suggest that, by only changing the query sentence representation using plms, tvg models can greatly improve performance, especially when plms are fine-tuned, stressing the importance of the text query representation in this task. our contributions can be summarized as follows: ## related work temporal video grounding: work on this task can be divided into two main approaches. on the one hand, we find techniques based on proposal generation, where the idea is to, given a query, output a set of candidate clips which could later be ranked @xcite . further research has mostly focused on reducing the number of proposals by producing query-guided or querydependent approaches @xcite @xcite , or on creating maps that can cover diverse video moments with different lengths @xcite . @xcite adopted a transformer-based multi-modal model (msat) which is pre-trained for this setting. more recently, models have incorporated contrastive losses to improve performance further. this is the case of both cpl @xcite and mnm @xcite , which also incorporate transformer-based components in their pipelines. the second line of approaches has instead proposed to directly predict the start and end locations through the video span (excl; @xcite . work on this line of research has focused on improving the performance by modelling label uncertainty (tmlga; @xcite , adding spatial features (dori; @xcite or improving the text-to-video matching strategies @xcite . for example, cpn @xcite adopted an ad-hoc graph-based technique. other approaches have focused on exploiting local and global features for better performance like @xcite . cp-net @xcite recently proposed a pyramidlike approach where the model progressively replenishes the temporal contexts and refines the location of the queried activity by enlarging the temporal receptive fields. finally, two recent approaches, vslnet @xcite and bcpn, @xcite have proposed to cast the task as visual question answering. on top of these lines, recently, an effort has been made to solve a variant of the task named spatiotemporal video grounding. in this case, besides predicting when the moment starts and ends, the model should also identify where the action described by the textual query occurs in the frames. works on this task heavily rely on the transformers architecture @xcite , with significant effort in eliminating the need for any pre-trained object detectors @xcite . while closely related to our task, due to the addition of the spatial dimension, this task naturally emphasizes the role of the visual modality in the grounding, further deviating from our languagedriven approach. therefore, our study does not consider models tailored for this task. parameter-efficient model training: as machine learning models continue to grow, updating their parameters efficiently is becoming increasingly important. one key idea has been to only update newly-added parameters @xcite . with the advent of large pre-trained transformer-based models in nlp, this idea has led to the development of adapters @xcite : sub-networks with few parameters that are inserted after every attention and feed-forward layer in a given model, as illustrated in figure 1 . we also find a variety of prompt-based approaches, which add trainable parameters into the model inputs @xcite @xcite . alternative techniques, such as sparsely updating a small number of parameters of the model (ben zaken et al., 2022; guo et al., 2021; sung et al., 2021), or low-rank factorization for the weights to be updated (mahabadi et al., 2021; karimi mahabadi et al., 2021a; hu et al., 2022) have also been proposed recently. finally, he et al. (2022a); mao et al. (2022) combined some of these techniques to propose a unified parameterefficient training framework. though we focus on nlp adapters, we deviate from previous work as our approach incorporates the visual modality. only updating newly-added parameters has also been proposed in cv, with some work predating the advent of transformers @xcite . more recently, we find works combining pre-trained language models with multi-modal in- text encoder woman sitting on a couch video encoder localization layer text query input video glove localization layer p(start) p(end) localization layer updated frozen text encoder woman sitting on a couch video encoder text query input video plm text encoder woman sitting on a couch video encoder text query input video plm a) original models b) proposed modification c) proposed training method p(start) p(end) p(start) p(end) adapters figure 2: illustration of our proposed approach to incorporate pre-trained language encoders and adapters into existing temporal video grounding pipelines. orange color refers to trainable parameters, and blue color refers to frozen ones. puts. for example, tsimpoukelli et al. (2021) trained a vision encoder to represent each image as a sequence of continuous embeddings, such that a frozen language model prompted with this prefix generates the appropriate image caption. yang et al. (2022b) showed that it is possible to perform zero-shot video question answering by leveraging frozen bidirectional language models. more recently, sung et al. (2022) cast multiple v&l tasks as text generation, combining nlp adapters with pre-trained encoder-decoders, such as bart @xcite , with existing image encoders, such as clip @xcite . the latter has also lately been the target of several studies that extend parameter-efficient techniques for cv @xcite @xcite . though our approach follows a similar trend, our interest lies in a grounding task where the training signal comes from the visual modality, which keeps us from casting our task as language generation. ## proposed approach consider a video @xmath0, represented as a sequence of frames such that @xmath1, . . . , l. each video in v is annotated with a natural language passage @xmath2, where s is a sequence of words @xmath3, . . . , m, which describes what is happening at a certain period of time in the video. this interval is formally defined by t s and t e , the starting and ending points of the annotations in time, respectively. the goal of the temporal video grounding task is to predict t s and t e given the video v and the text query s. ## conclusions this paper studied the effects of plms in the tvg task and assessed the applicability of nlp parameter-efficient training alternatives based on adapters. we coupled bert, roberta, and de-berta, with a selection of previous tvg works, i.e., excl, tmlga, and dori, and tested different adapters to reduce the impact of the additional parameters. our results showed that, by only changing the query representation using plms, tvg models can greatly benefit from such integration, especially when plms are fine-tuned, highlighting the importance of the query representation in this task. moreover, we verified that adapters are an effective alternative to full fine-tuning, even though they were not tailored for our task. they saved on computational cost, allowing improvements for larger tvg models, such as dori, and also delivered results comparable to sota models. finally, we observed that while parallel adapters struggled to learn in this task, bottleneck adapters such as houlsby and pfeiffer performed across all tested tvg models and datasets.