Weakly-Supervised Temporal Article Grounding (2022)

Abstract: Given a long untrimmed video and natural language queries, video grounding (VG) aims to temporally localize the semantically-aligned video segments. Almost all existing VG work holds two simple but unrealistic assumptions: 1) All query sentences can be grounded in the corresponding video. 2) All query sentences for the same video are always at the same semantic scale. Unfortunately, both assumptions make today’s VG models fail to work in practice. For example, in real-world multimodal assets (eg, news articles), most of the sentences in the article can not be grounded in their affiliated videos, and they typically have rich hierarchical relations (ie, at different semantic scales). To this end, we propose a new challenging grounding task: Weakly-Supervised temporal Article Grounding (WSAG). Specifically, given an article and a relevant video, WSAG aims to localize all “groundable” sentences to the video, and these sentences are possibly at different semantic scales. Accordingly, we collect the first WSAG dataset to facilitate this task: YouwikiHow, which borrows the inherent multi-scale descriptions in wikiHow articles and plentiful YouTube videos. In addition, we propose a simple but effective method DualMIL for WSAG, which consists of a two-level MIL loss and a single-/cross- sentence constraint loss. These training objectives are carefully designed for these relaxed assumptions. Extensive ablations have verified the effectiveness of DualMIL.

## introduction video grounding (vg), i.e., localizing video segments that semantically correspond to (coreference relation) query sentences, is one of the fundamental tasks in multimodal understanding. further, video grounding can serve as an indispensable technique for many downstream applications, such as the textoriented highlight detection @xcite , video 1 mil: multiple instance learning. 2 codes: https://github.com/zjuchenlong/wsag . sentence: the man cracks an egg and whisks the ingredients in a bowl. sentence1: a man in a chef's coat is standing in a kitchen talking to the camera. sentence2: the man cracks an egg and whisks the ingredients in a bowl. sentence3: the man flips the pancakes. sentence4: the man is eating a piece of the pancake. article (how to make pancakes) 1. crack the eggs into a bowl and beat until creamy. 2. melt the butter in a bowl. make sure that it's completely melted. a minute is sufficient. ## dataset: youwikihow we built youwikihow dataset from wikihow articles and youtube videos. as shown in figure 2 , we group a wikihow article and any video about the same task as a pair. thanks to the inherent hierarchical structure of wikihow articles, we can easily obtain sentences from different scales: high-level summaries and low-level details. as in figure 2 , "pour the lemon juice into the pitcher." is a highlevel sentence summary and "you may add the pulp if .... along with the seeds." is a low-level sentence detail of this summary. in this section, we first introduce the details of dataset construction, and then compare youwikihow to existing vg benchmarks. ## conclusions in this paper, we discussed the weaknesses of default assumptions in existing video grounding work, and proposed a more challenging task: weaklysupervised article grounding (wsag). to facilitate the research in this direction, we collected the first wsag dataset youwikihow. further, we proposed dualmil for wsag, including a two-level mil loss and a single-/cross-sentence constraint loss. this work paves the way for a number of exciting future works: 1) designing more reasonable backbones for multiple sentence inputs by considering their semantic relations; 2) extending to more general domains beyond instructional articles.