End-to-End Modeling via Information Tree for One-Shot Natural Language Spatial Video Grounding (2022)

Abstract: Natural language spatial video grounding aims to detect the relevant objects in video frames with descriptive sentences as the query. In spite of the great advances, most existing methods rely on dense video frame annotations, which require a tremendous amount of human effort. To achieve effective grounding under a limited annotation budget, we investigate one-shot video grounding and learn to ground natural language in all video frames with solely one frame labeled, in an end-to-end manner. One major challenge of end-to-end one-shot video grounding is the existence of videos frames that are either irrelevant to the language query or the labeled frame. Another challenge relates to the limited supervision, which might result in ineffective representation learning. To address these challenges, we designed an end-to-end model via Information Tree for One-Shot video grounding (IT-OS). Its key module, the information tree, can eliminate the interference of irrelevant frames based on branch search and branch cropping techniques. In addition, several self-supervised tasks are proposed based on the information tree to improve the representation learning under insufficient labeling. Experiments on the benchmark dataset demonstrate the effectiveness of our model.

## introduction natural language spatial video grounding is a vital task for video-text understanding @xcite @xcite @xcite , which aims to detect the objects described by the natural first author. mengzeli@zju.edu.cn figure 1 : an example of spatially grounding natural language in video frames. existing methods that are devised for supervised video grounding are not directly applicable to this novel setting. we summarize several critical challenges: to address these challenges, we devise an endto-end model via the information tree for the one shot natural language spatial video grounding (it-os). different from previous works, we design a novel tree structure to shield off the one-shot learning from frames that are irrelevant to either the language query or the labeled frame. we devise several self-supervised tasks based on the tree structure to strengthen the representation learning under limited supervision signals. specifically, the calculation processes of the key module, information tree, contains four steps: (1) to construct the information tree, we view video frame features as nodes, and then compress the adjacent nodes to non-leaf nodes based on the visual similarity of themselves and the semantic similarity with the language query; (2) we search the information tree and select branch paths that are consistently relevant to the language query both in the abstractive non-leaf node level and in the fine-grained leaf node level; (3) we drop i) the leaf nodes that do not belong the same semantic unit with the labeled node; and ii) the non-leaf nodes on the low relevance branch paths. we also down-weight the importance of the leaf nodes that belong to the same semantic unit with the labeled node but are on the low relevance paths; (4) finally, we input the extracted and weighted information to the transformer, and conduct training with the one-shot label and self-supervised tasks, including masked feature prediction and video-text matching. we note that both the information tree and the transformer are jointly trained in an end-to-end manner. we conduct experiments on two benchmark datasets, which demonstrate the effectiveness of it-os over state-of-the-arts. extensive analysis including ablation studies and case studies jointly demonstrate the merits of it-os on one-shot video grounding. our contributions can be summarized as follows: deep neural network has convincingly demonstrated high capability in many domains @xcite @xcite , especially for video related tasks @xcite @xcite , like video grounding. for example, @xcite use the neural network to detect language query related objects in the first frame and track the detected object in the whole video. compared to it, @xcite and @xcite go further. they extract all the object proposals through the pretrained detector, and choose the right proposal described in the text. supervised training for the natural language video object detection needs high labeling costs. to reduce it, some researchers pay attention to weakly-supervised learning fashion using multiple instances learning(mil) method @xcite @xcite @xcite transfers contextualized knowledge in crossmodal alignment to release the unstable training problem in mil. based on contrastive learning @xcite , @xcite proposes an asynce loss to disentangle false-positive frames in mil, which allows for mitigating the uncertainty of from negative instance-sentence pairs. weakly supervised false-positive identification based on contrastive learning has witnessed success in some other domains @xcite one-shot learning for videos. one-shot learning has been applied in some other video tasks. @xcite proposes a meta-learning-based approach to perform one-shot action localization by capturing task-specific prior knowledge. @xcite investigates the one-shot video person re-identification task by progressively improving the discriminative capability of cnn via stepwise learning. different from these works, @xcite and @xcite define the one-shot learning as only one frame being labeled per video. specifically, @xcite use a fully convolutional neural network architecture to solve the one-shot video segmentation task. @xcite ) decouple the detection task, and uses the modified mask-rcnn to predict local segmentation masks. following this setting, we investigate one-shot natural language spatial video grounding, and devise a novel information-tree based end-to-end framework for the task. ## conclusion in this paper, we introduce the one-shot learning into the natural language spatial video grounding task to reduce the labeling cost. to achieve the goal, the main point is to make full use of only one frame label for each video. the invalid frames unrelated to the input text and target objects bring confounding to the one-shot training process. we design an end-to-end model (it-os) via the information tree to avoid it. specifically, the information tree module merges frames with similar semantics into one node. then, by searching the tree and cropping the invalid nodes, we can get the complete and valid semantic unit of the video. finally, two self-supervised tasks are used to make up the insufficient supervision.