D ema F ormer: Damped Exponential Moving Average Transformer with Energy-Based Modeling for Temporal Language Grounding (2023)

Abstract: Temporal Language Grounding seeks to localize video moments that semantically correspond to a natural language query. Recent advances employ the attention mechanism to learn the relations between video moments and the text query. However, naive attention might not be able to appropriately capture such relations, resulting in ineffective distributions where target video moments are difficult to separate from the remaining ones. To resolve the issue, we propose an energy-based model framework to explicitly learn moment-query distributions. Moreover, we propose DemaFormer, a novel Transformer-based architecture that utilizes exponential moving average with a learnable damping factor to effectively encode moment-query inputs. Comprehensive experiments on four public temporal language grounding datasets showcase the superiority of our methods over the state-of-the-art baselines.

## introduction temporal language grounding (tlg) is a task to determine temporal boundaries of video moments that semantically correspond (relevant) to a language query @xcite . tlg is a complex and challenging task, since video processing demands understanding across multiple modalities, including image, text, and even audio. however, tlg has received increasing attention in cv and nlp communities because it provides myriad usage for further downstream tasks, e.g. vqa @xcite @xcite , relation extraction @xcite , and information retrieval @xcite . early methods for tlg deploy concatenation with linear projection @xcite al., * corresponding author -0.1 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 -0.10 -0.05 0.00 0.05 0.10 remaining moment target localization (a) umt -0.10 -0.05 0.00 0.05 0.10 0.15 0.20 0.25 0.30 -0.05 0.00 0.05 0.10 remaining moment target localization (b) demaformer 2019b) or similarity functions @xcite to fuse textual and visual features. to further enhance the localization performance, recent works divide the video into equal-length moments and employ the attention mechanism of the transformer model to learn the relations between video moments and the language query. for example, moment-detr model @xcite concatenates the visual moments and the textual tokens, and then passes the concatenated sequence to the transformer encoder to capture the alignment. the umt model @xcite includes an additional audio channel into the transformer encoder to construct a unified architecture. however, for the tlg task, previous works have shown that such attention-based approach is still insufficient to capture the rich semantic interaction between the text query and video moments @xcite . as in figure 2 , the localized video moments hardly align with the query statement. moreover, the attention mechanism in the transformer encoder does not assume any prior knowledge towards the input elements @xcite . for language localization, this design choice does not leverage the fact that video moments in temporal neighborhoods tend to exhibit closely related features. therefore, this approach could lead to ineffective modeling of joint moment-query inputs. as evidence, figure 1 demonstrates the distribution of joint moment-query representations. particularly, the representations of target moment-query localizations and the remaining ones mingle together, making the grounding task more challenging. to address these limitations, we dive deeper into polishing the distribution of moment-query representations. in addition to supervised training to correctly localize the language in the video, we perform unsupervised training to explicitly maximize the likelihood of moment-query localizations. this could help the multimodal model focus on capturing the distribution of target moment-query pairs and distinguishing them from others. as such, we propose to model the distribution of moment-query representations under the framework of the energy-based model (ebm). in contrast to other probabilistic models such as normalizing flow @xcite or autoencoder (kingma and welling, 2013), the ebm framework allows us to directly integrate the video moment's salience score into the density function, which results in accurate modeling of momentquery representations. our implementation develops into a contrastive divergence objective which aims to minimize the energy of the relevant localizations while maximizing the energy of the deviating ones. accordingly, the framework needs negative samples to represent high-energy regions. therefore, we adapt the langevin dynamics equation to directly sample negative inputs from the ebm distribution. such approach is appropriate because in the beginning the distribution will not match the true distribution, hence the generated samples are assured to be negative. as the training progresses, the distribution will approximate the true distribution, consequently the langevin equation is able to produce hard negative samples. in addition, we incorporate the inductive bias that captures local dependencies among the moment-query inputs. we propose demaformer in which we equip the damped exponential moving average (dema) computation for the transformer architecture. technically, the computation applies exponentially decaying factors that consider the information from adjacent inputs. we further introduce learnable damping coefficients to enable the model to absorb adjacent information in a sufficient manner that ensures distinction among inputs. eventually, we combine the dema computation with the attention mechanism to construct demaformer encoder and decoder modules. to sum up, the contributions of our paper can be summarized as follows: ## methodology our task is to localize moments in videos from natural language queries. formally, given a language query q of l q tokens and a video v composed of l v equal-length input moments, where each moment is represented by a visual frame sampled from the moment, we aim to localize l m time spans from v that is aligned with the query q, noted as {(l i , r i )} lm @xmath0, where each moment spans from l i to r i scaled by the video timelength and l @xmath1. thus, we first describe our proposed damping exponential moving average attention for modeling video-language inputs in section 3.1, the overall architecture in section 3.2, and the training strategy empowered with energy-based modeling in section 3.3 and 3.4. ## conclusion in this paper, we propose demaformer, a novel neural architecture for the temporal language grounding (tlg) task. by leveraging the exponential moving average approach with a damping factor, de-maformer is capable of incorporating local dependencies among moment-query localizations. additionally, we propose an energy-based strategy to explicitly model localization distribution. on four public benchmarks for the tlg task, our method is effective and outperforms state-of-the-art approaches with a significant margin.