On Pursuit of Designing Multi-modal Transformer for Video Grounding (2021)

Abstract: Video grounding aims to localize the temporal segment corresponding to a sentence query from an untrimmed video. Almost all existing video grounding methods fall into two frameworks: 1) Top-down model: It predefines a set of segment candidates and then conducts segment classification and regression. 2) Bottom-up model: It directly predicts frame-wise probabilities of the referential segment boundaries. However, all these methods are not end-to-end, i.e., they always rely on some time-consuming post-processing steps to refine predictions. To this end, we reformulate video grounding as a set prediction task and propose a novel end-to-end multi-modal Transformer model, dubbed as GTR. Specifically, GTR has two encoders for video and language encoding, and a cross-modal decoder for grounding prediction. To facilitate the end-to-end training, we use a Cubic Embedding layer to transform the raw videos into a set of visual tokens. To better fuse these two modalities in the decoder, we design a new Multi-head Cross-Modal Attention. The whole GTR is optimized via a Many-to-One matching loss. Furthermore, we conduct comprehensive studies to investigate different model design choices. Extensive results on three benchmarks have validated the superiority of GTR. All three typical GTR variants achieve record-breaking performance on all datasets and metrics, with several times faster inference speed.

## introduction video grounding is a fundamental while challenging task for video understanding and has recently attracted unprecedented research attention @xcite @xcite @xcite . formally, it aims to identify the two temporal boundaries of the moment of interest based on an input untrimmed video and a figure 1 : performance comparisons on tacos in terms of r@1, iou@0.5 and query per second (the number of queries that are retrieved each second during inference). marker sizes are proportional to the model size. our gtr-h is 4.9% better than 2d-tan @xcite with 5 times faster speed. the overwhelming majority of the state-of-theart video grounding methods fall into two frameworks: 1) top-down models @xcite @xcite @xcite @xcite : they always use a propose-and-rank pipeline, where they first generate a set of moment proposals and then select the best matching one. to avoid the proposal bottlenecks and achieve high recall, a vast number of proposals are needed. correspondingly, some time-consuming post-processing steps (e.g., non-maximum suppression, nms) are introduced to eliminate redundancy, which makes the matching process inefficient (cf. figure 1 ). 2) bottom-up models @xcite @xcite @xcite : they directly regress the two temporal boundaries of the referential segment from each frame or predict boundary probabilities framewisely. similarly, they need post-processing steps to group or aggregate all frame-wise predictions. although these two types of methods have realized impressive progress in video grounding, it is worth noting that they still suffer several notorious limitations: 1) for top-down methods, the heuristic proposal generation process introduces a series of hyper-parameters. meanwhile, the whole inference stage is computation-intensive for densely placed candidates. 2) for bottom-up methods, the framewise prediction manner overlooks fruitful temporal context relationships, which strictly limits their performance. 3) all these methods are not end-to-end, which need complex post-processing steps to refine predictions, and easily fall into the local optimum. in this paper, we reformulate the video grounding as a set prediction problem and propose a novel end-to-end multi-modal transformer model gtr (video grounding with transformer). gtr has two different encoders for video and language feature encoding, and a cross-modal decoder for final grounding result prediction. specifically, we use a cubic embedding layer to transform the raw video data into a set of visual tokens, and regard all word embeddings of the language query as textual tokens. both these visual and textual tokens are then fed into two individual transformer encoders for respective single-modal context modeling. afterwards, these contextualized visual and textual tokens serve as an input to the cross-modal decoder. other input for the decoder is a set of learnable segment queries, and each query try to regress a video moment by interacting with these contextualized tokens. to better fuse these two modalities in the decoder, we design a new multi-head cross-modal attention module (mcma). the whole gtr model is trained end-to-end by optimizing a many-to-one matching loss which produces an optimal bipartite matching between predictions and ground-truth. thanks to this simple pipeline and the effective relationship modeling capabilities in transformer, our gtr is both effective and computationally efficient with extremely fast inference speed (cf. figure 1 ). since our community has few empirical experiences on determining the best design choice for multi-modal transformer-family models, we conduct extensive exploratory studies on gtr to investigate the influence of different model designs and training strategies, including: (a) visual tokens acquisition. we use a cubic embedding layer to transform raw videos to visual tokens, and discuss the design of the cubic embedding layer from three dimensions. (b) multi-modal fusion mecha-nism. we propose six types of multi-modal fusion mechanisms, and compare their performance and computation cost thoroughly. (c) decoder design principles. we explore some key design principles of a stronger multi-modal transformer decoder, such as the tradeoff between depth and width, or the attention head number in different layers. (d) training recipes. we discuss the influence of several training tricks. we hope our exploration results and summarized take-away guidelines can help to open the door for designing more effective and efficient transformer models in multi-modal tasks. in summary, we make three contributions in this paper: video grounding. the overwhelming majority of state-of-the-art video grounding methods are topdown models @xcite @xcite @xcite @xcite @xcite @xcite @xcite . although these top-down models have dominated the performance, they suffer from two inherent limitations: ## video grounding with transformer as shown in figure 2 , gtr yields temporal segment predictions semantically corresponding to the given query by four consecutive steps: (1) input embedding. given a raw video and a query, this step aims to encode them into the feature space (i.e., visual and textual tokens). 2) encoder. visual and textual token embeddings are enhanced with a standard transformer encoder by modeling the intra-modality correlations. (3) cross-modal decoder. contextualized visual and textual token embeddings are fused by a multi-head cross-modal attention module (mcma), and a set of learnable segment queries are fed into the decoder to interact with these two modal features. (4) prediction heads. a simple feed-forward network (ffn) is applied to predict final temporal segments. ## experiments we first introduce experimental settings in sec. 4.1. then, we present detailed exploratory studies on the design of gtr in sec. 4.2. the comparisons with sota methods are discussed in sec. 4.3, and we show visualization results in sec. 4.4. more results are left in supplementary materials. ## conclusions and future works in this paper, we propose the first end-to-end multimodal transformer-family model gtr for video grounding. by carefully designing several novel components (e.g., cubic embedding layer, multihead cross-modal attention module, and many-toone matching loss), our gtr achieves new stateof-the-art performance on three challenging benchmarks. as a pioneering multi-modal transformer model, we also conducted comprehensive explorations to summarize several empirical guidelines for model designs, which can help to open doors for the future research on multi-modal transformers. moving forward, we aim to extend gtr to more general settings, such as weakly-supervised video grounding or spatial-temporal video grounding. we also compare our cubic embedding layer with the framewise partition in vision transformer. part of the results have been listed in the main paper and the full results are shown in table . 9. it demonstrates the superiority of our cubic embedding layer.