Exploiting Intrinsic Multilateral Logical Rules for Weakly Supervised Natural Language Video Localization (2024)

Abstract: Weakly supervised natural language video localization (WS-NLVL) aims to retrieve the moment corresponding to a language query in a video with only video-language pairs utilized during training. Despite great success, existing WS-NLVL methods seldomly consider the complex temporal relations enclosing the language query (e.g., between the language query and sub-queries decomposed from it or its synonymous query), yielding illogical predictions. In this paper, we propose a novel plug-and-play method, Intrinsic Multilateral Logical Rules, namely IMLR, to exploit intrinsic temporal relations and logical rules for WS-NLVL. Specifically, we formalize queries derived from the original language query as the nodes of a directed graph, i.e., intrinsic temporal relation graph (ITRG), and the temporal relations between them as the edges. Instead of directly prompting a pre-trained language model, a relation-guided prompting method is introduced to generate ITRG in a hierarchical manner. We customize four types of multilateral temporal logical rules (i.e., identity, inclusion, synchronization, and succession) from ITRG and utilize them to train our model. Experiments demonstrate the effectiveness and superiority of our method on the Charades-STA and ActivityNet Captions datasets.

## introduction natural language video localization (nlvl) @xcite @xcite @xcite aims to locate the temporal interval that semantically corresponds to a language query in an untrimmed video @xcite @xcite . due to its promising applications in various fields such as video editing and intelligent surveillance, nlvl has attracted increasing research interest in the last @xcite and cnm @xcite . ground-truths and predictions are represented by filled rectangles and grid lines, respectively. best viewed in color. early methods of ws-nlvl are based on multiple instance learning (mil) @xcite , where each segment of the input video is treated as an instance and the input video as a bag of instances. the predictions of instances are aggregated to form the bag-level prediction. recently, reconstruction-based methods @xcite @xcite @xcite are proposed to solve the task through joint learning with the reconstruction loss, assuming that the proposal corresponding to the language query should best reconstruct it. despite great success, existing ws-nlvl methods directly fuse the video features with word or sentence-level language features, which seldomly consider the complex temporal relations enclosing the language query and thus yield illogical predictions. as the first example shown in fig. 1 , given an untrimmed video and a language query "person sits in a chair", the localization result is supposed to be identical to that of its synonymous query "person sits down on a chair" and be included by that of its sub-query "person". however, as represented by grad lines in fig. 1 , a start-of-the-art ws-nlvl method cpl @xcite ) yields predictions far from logical. fig. 1 also presents another example of a language query "a long curly haired woman inspecting the blow dryer then drying her hair looking irritated and bored". the temporal intervals of its sub-queries, i.e., "a long curly haired woman inspecting the blow dryer" and "a long curly haired woman drying her hair looking irritated and bored", should be temporally successive and integrated to that of the original language query. unfortunately, the existing methods fail to capture such complex temporal relations. to tackle the issue above, we propose a novel framework, dubbed imlr, to exploit intrinsic temporal relations and logical rules for ws-nlvl. concretely, we first propose to understand the complex language query by generating its intrinsic temporal relation graph (itrg). in this graph, the original language query and queries derived from it are formulated as the nodes while the temporal relations between them are represented as the edges. we systematically consider all possible temporal relations for ws-nlvl and customize four types of relations with corresponding multilateral temporal logical rules (mtlrs), i.e., identity, inclusion, synchronization, and succession. since it is challenging to generate itrg by manually designing rules or directly prompting @xcite ) a pre-trained language model (lm), we then introduce a relation-guided prompting method to generate itrg in a hierarchical manner. finally, mtlrs from the graph are utilized to train our model of boundary-aware transformer. we conduct experiments to integrate our method with two recent ws-nlvl approaches, i.e., cnm @xcite and cpl @xcite , verifying the effectiveness and superiority of our method on the charades-sta and activitynet captions datasets. our contributions can be summarized as follows: ## methodology given an untrimmed video v and a natural language query q, nlvl aims to learn a model f that can predict the time interval @xmath0, t e ) in the video corresponding to the language query, i.e., f : (v, q) → i, where t s and t e represent the start and end time, respectively. different from fs-nlvl that utilizes accurate temporal boundaries {v m , q m , i m } m @xmath1, only videolanguage pairs {v m , q m } m @xmath2. here m denotes the number of samples in the training set. fig. 2 shows the overall framework of our method for ws-nlvl, which consists of relationguided prompting, video encoder, language en-a group of young people play field hockey on a field while a man watches from a fence bordering the playing field. a group of young people play field hockey on a field. a man watches from a fence bordering the playing field. a group of young people a field a man a fence bordering the playing field a bunch of teenagers engage in a game of field hockey on a grassy area while a man observes from a fence that runs alongside the playing field. ## conclusion and discussion in this paper, we propose to exploit intrinsic temporal relations and multilateral logical rules for ws-nlvl. language queries derived from the given one are formalized as the nodes of a directed graph and the temporal relations between them as the edges. we introduce relation-guided prompting to hierarchically generate the graph by leveraging a pre-trained lm. four types of multilateral temporal logical rules (i.e., identity, synchronization, succession, and inclusion) are customized to guide the training of our models. extensive experiments on the charades-sta and activitynet captions datasets demonstrate the effectiveness and superiority of our method. limitations. since only the video-language pairs are available during training, the performance of our method is still lag from several state-of-theart fully supervised approaches trained with accurate annotations.