Weakly-Supervised Spoken Video Grounding via Semantic Interaction Learning (2023)

Abstract: The task of spoken video grounding aims to localize moments in videos that are relevant to descriptive spoken queries. However, extracting semantic information from speech and modeling the cross-modal correlation pose two critical challenges. Previous studies solve them by representing spoken queries based on the matched video frames, which require tremendous effort for frame-level labeling. In this work, we investigate weakly-supervised spoken video grounding, i.e., learning to localize moments without expensive temporal annotations. To effectively represent the cross-modal semantics, we propose Semantic Interaction Learning (SIL), a novel framework consisting of the acoustic-semantic pre-training (ASP) and acoustic-visual contrastive learning (AVCL). In ASP, we pre-train an effective encoder for the grounding task with three comprehensive tasks, where the robustness task enhances stability by explicitly capturing the invariance between time- and frequency-domain features, the conciseness task avoids over-smooth attention by compressing long sequence into segments, and the semantic task improves spoken language understanding by modeling the precise semantics. In AVCL, we mine pseudo labels with discriminative sampling strategies and directly strengthen the interaction between speech and video by maximizing their mutual information. Extensive experiments demonstrate the effectiveness and superiority of our method.

## introduction temporal video grounding @xcite is an important task in the crossmodal understanding field @xcite @xcite @xcite , aiming to retrieve a target moment within a video based on a given query. with the progress in deep learning, there has been significant achievements in this area. while most previous studies focus on textual queries, recent work @xcite introduce the spoken video grounding task by incorporating the spoken query into the video grounding, as shown in figure 1 . however, such video grounding task with spoken queries presents unique challenges compared to its text-based counterpart. first, the encoding of speech is inherently difficult due to its weak and volatile semantic information, making it arduous to extract useful features for video grounding. second, even with the acquisition of valuable semantic features, modeling speech-video interaction and extracting cross-modal content still poses an inevitable obstacle. prior work @xcite address these two problems simultaneously through the proposed video-guided contrastive predictive coding, which utilizes aligned video clips to learn semantic representations from spoken queries. however, a critical drawback is its heavy reliance on precise temporal matching between video and spoken queries. the acquisition of such fine-grained annotations requires substantial manual labor, hindering the practicality and applicability of this approach. in this work, we address the issue of intensive labor by investigating a novel task called weakly-supervised spoken video grounding (wsvg), aiming to localize speech-aligned moments in videos under a weakly-supervised scheme. in this setting, we only have access to aligned video-speech pairs during training without any temporal annotations, which poses a greater challenge. to tackle the aforementioned problem, we propose a novel framework semantic interaction learning (sil), following a progressive pipeline to first pre-train a speech encoder for semantic encoding and then learn speech-video alignment during grounding training. it consists of two key components: acoustic-semantic pre-training (asp) and acoustic-visual contrastive learning (avcl). in the pre-training stage, the asp module utilizes external speech-text data to train a speech encoder capable of extracting rich semantic information from speech. to adapt the encoded features for the downstream weakly-supervised video grounding, asp includes three specialized tasks targeting three specific characteristics. (1) the robustness task focuses on the encoder's ability to handle complex and noisy speech, which is a practical problem in the real world. considering time series data can be split into the time and frequency domains to provide invariance regardless of varying distributions @xcite , we utilize both time-and frequency-based speech feature for pre-training and forces their encoded semantic information to be consistent. (2) the conciseness task addresses the issue of long sequence features with redundant information, which results in scattered distribution of the video-speech attention and impedes effective interaction. hence, we compress the encoded features into discrete segments via i&f algorithm @xcite , refining the feature sequence for effective interaction. (3) the semantic task emphasizes the extraction of key semantics for grounding, which is a crucial requirement in this fine-grained cross-modal understanding task. unlike the trivial self-supervised method @xcite or knowledge distillation method @xcite , we draw inspiration from the human understanding system that encompasses auditory perception and cognitive processing @xcite . concretely, we introduce a connectionist temporal classification (ctc) @xcite loss to facilitate training, and further consider both sequence-level and word-level semantic to ensure the comprehensive semantic transfer. in the grounding stage, the avcl module directly enhances the correlation between video and speech. despite the effective semantic encoding of spoken queries, the discrepancy between video and speech still hinders the cross-modal interaction. as video and speech are from two distinct feature spaces, avcl leverages contrastive learning to maximize their agreement. first, we perform location-based selection and score-based mining to select pseudo labels with high confidence. with the located boundary of the predicted pseudo proposal, we can coarsely select the negative samples from regions outside and further calculate the clip-level score inside the boundary to mine positive/negative samples. then, based on these discriminative samples, we contrastively maximize the mutual information between speech and positive clips. our main contributions are listed as follows: ## conclusion in this paper, we propose a new task named weakly-supervised spoken video grounding and present a novel framework sil. concretely, we conduct an acoustic-semantic pre-training to achieve effective and robust semantic encoding. besides, we develop an acoustic-visual contrastive learning to optimize representations for cross-modal interaction. the extensive experiments demonstrate the superiority of our proposed method.