Annotations Are Not All You Need: A Cross-modal Knowledge Transfer Network for Unsupervised Temporal Sentence Grounding (2023)

Abstract: This paper addresses the task of temporal sentence grounding (TSG). Although many respectable works have made decent achievements in this important topic, they severely rely on massive expensive video-query paired annotations, which require a tremendous amount of human effort to collect in real-world applications. To this end, in this paper, we target a more practical but challenging TSG setting: unsupervised temporal sentence grounding, where both paired video-query and segment boundary annotations are unavailable during the network training. Considering that some other cross-modal tasks provide many easily available yet cheap labels, we tend to collect and transfer their simple cross-modal alignment knowledge into our complex scenarios: 1) We first explore the entity-aware object-guided appearance knowledge from the paired Image-Noun task, and adapt them into each independent video frame; 2) Then, we extract the event-aware action representation from the paired Video-Verb task, and further refine the action representation into more practical but complicated real-world cases by a newly proposed copy-paste approach; 3) By modulating and transferring both appearance and action knowledge into our challenging unsupervised task, our model can directly utilize this general knowledge to correlate videos and queries, and accurately retrieve the relevant segment without training. Extensive experiments on two challenging datasets (ActivityNet Captions and Charades-STA) show our effectiveness, outperforming existing unsupervised methods and even competitively beating supervised works.

## introduction as a popular yet challenging natural language processing task, temporal sentence grounding (tsg) (b3) (b2) (b1) search app match(query noun, noun) search app match(query noun, noun) search action match(query verb, verb) search action match(query verb, verb) match(video action, action) match(video action, action) retrieve {video action, query verb} match(video action, action) retrieve {video action, query verb} retrieve {video app, query noun} match(video app, app) retrieve {video app, query noun} match(video app, app) figure 1: (a) example of the temporal sentence grounding (tsg) task. (b) illustration of our proposed cmkt @xcite is the appearance knowledge learning module, (b2) is the action knowledge learning module, and (b3) is the knowledge transfer module, where "app" denotes "appearance". note that there are no ground-truth annotations in (b3). best viewed in color. @xcite @xcite has drawn increasing attention in recent years. tsg aims to locate a temporal video segment with an activity that semantically corresponds to a given sentence query. as shown in figure 1 (a), only a short video segment semantically matches the query, while most of the video contents are queryirrelevant. clearly, tsg tries to break through the barrier between computer vision and natural language processing techniques for more challenging cross-modal grounding @xcite @xcite @xcite . most previous tsg works @xcite @xcite @xcite @xcite @xcite @xcite ) are under a fully-supervised setting, where each frame is manually annotated as query-relevant or query-irrelevant. despite the decent progress, these datahungry methods severely rely on numerous framequery annotations, which are significantly laborintensive and time-consuming to collect. to alleviate annotation reliance to a certain extent, some recent works @xcite @xcite @xcite @xcite explore a weakly-supervised setting to only leverage the coarse-grained video-query annotations instead of the fine-grained frame-query annotations. unfortunately, this weakly-supervised setting still requires expensive video-query annotations and its performance validates unsatisfactory. to this end, in this paper, we try to tackle a more practical but challenging setting for the tsg task, i.e., unsupervised tsg, which excludes both coarse-grained video-query annotations and fine-grained frame-query annotations. therefore, how to guide the unsupervised model to extract the vision-language correlations becomes a crucial problem. luckily, unlike the expensively annotated tsg task requiring long sentences and complicated videos, some cross-modal tasks (e.g., image retrieval and video retrieval) always provide a massive number of cheap and fully-annotated datasets, i.e., matched image-noun and paired video-verb. hence, we pose a brand-new idea: can we transfer the annotation knowledge from other cheap crossmodal annotations to the complicated and challenging unsupervised tsg task? our main motivation is that we can first learn simple image-noun and video-verb correlations to extract the visual appearance and action knowledge, respectively. then, by generalizing such cross-modal knowledge and transferring it into unsupervised tsg, we can not only well correlate the queries with corresponding videos without large-scale video-query pairs, but also match them with image-aware frames without frame-query pairs. therefore, the current problem becomes how to separately collect the appearance and action knowledge and adaptively transfer the knowledge into our unsupervised tsg task. to tackle these issues, we propose a novel cross-modal knowledge transferring (cmkt) network, which fine-tunes and transfers the extracted appearance and action knowledge from other cross-modal tasks to search the best-matching video activity related to given queries in the unsupervised tsg setting. as shown in figure 1 (b), our cmkt contains three modules: (i) in the appearance knowledge module, we try to learn the object-guided matched region-noun information from the image-noun task to model the appearance information. (ii) in the action knowledge module, single-action and multiaction branches are designed to simulate complex video scenes. in the single-action branch, we extract the pure action information based on a set of labeled single-action videos. in the multi-action branch, since real-world videos often contain multiple actions, we introduce a clip-level copy-paste strategy to construct the action-hybrid videos to adapt to more complex scenes. (iii) finally, we refine both the appearance and action knowledge, and aggregate and transfer them into our complicated unsupervised tsg for inference without further training. our main contributions are summarized as follows: ## related works most existing tsg methods are under the fullysupervised setting @xcite @xcite @xcite @xcite @xcite , where all video-query pairs and precise segment boundaries are manually annotated. based on the top-down approach, some methods @xcite @xcite first pre-define multiple segment proposals and then align these proposals with the query for cross-modal semantic matching based on the similarity. finally, the video 2 video n knowledge transfer textual feature visual feature "climb" "throw", ... action words (verbs) glove "climb" video database any query ... ... ... ... ... ... "throw" "climb" "climb" ... ... copy-paste single-action feature video-level feature refined knowledge "climb" "throw" "person" "box" "person" "box" nouns "close" verbs glove noun feature verb feature knowledge searching appearance knowledge action knowledge video appearance c3d video action roialign "climb" knowledge "throw" knowledge "person" knowledge "box" knowledge l app consensus feature consensus feature ... ... ... ... replace ... ... replace single -action copy-paste "climb" "climb" "climb" "climb" "throw" "throw" "throw" appearance action consensus feature match match similarity similarity multiaction cross-modal interaction appearance action video feature cosine similarity grounding multi-action feature figure 2: overview of our cmkt framework for unsupervised tsg. we design two knowledge modules to extract both appearance and action knowledge from other simple yet cheap cross-modal tasks. (i) in appearance branch, we learn the region-word pair information and the appearance information can be viewed as the visual semantic of a given noun. (ii) the action knowledge module contains two branch: single-action branch and multi-action branch. for the single-action branch, we extract the pure action information based a set of labeled action video. specifically, based on the branch, we can obtain the visual represent of the verbs. as for multi-action branch, since many video often contain multiple action, we introduce the clip-level copy-paste method to construct the action-hybrid videos. the global information in these videos contain the action features. (iii) finally, we introduce the appearance and action knowledge into our tsg task by knowledge searching. best viewed in color. ## conclusion in this paper, we address the challenging yet practical unsupervised tsg task from a brand-new perspective of knowledge transfer. without further grounding training, we can directly utilize the general knowledge from other cross-modal tasks to guide to match the video and query for retrieving the target segment. to our best knowledge, we make the first attempt to transfer the knowledge from other multi-modal topics to our tsg task. experimental results validate the effectiveness of cmkt, outperforming the existing unsupervised method by a large margin.