W MAN: W EAKLY - SUPERVISED M OMENT A LIGNMENT N ETWORK FOR T EXT - BASED V IDEO S EGMENT R ETRIEVAL
Anonymous authors
Paper under double-blind review
A BSTRACT
Given a video and a sentence, the goal of weakly-supervised video moment retrieval is to locate the video segment which is described by the sentence without having access to temporal annotations during training.
Instead, a model must learn how to identify the correct segment ( i.e . moment) when only being provided with video-sentence pairs.
Thus, an inherent challenge is automatically inferring the latent correspondence between visual and language representations.
To facilitate this alignment, we propose our Weakly-supervised Moment Alignment Network (wMAN) which exploits a multi-level co-attention mechanism to learn richer multimodal representations.
The aforementioned mechanism is comprised of a Frame-By-Word interaction module as well as a novel Word-Conditioned
Visual Graph (WCVG).
Our approach also incorporates a novel application of positional encodings, commonly used in Transformers, to learn visual-semantic representations that contain contextual information of their relative positions in the temporal sequence through iterative message-passing.
Comprehensive experiments on the DiDeMo and Charades-STA datasets demonstrate the effectiveness of our learned representations: our combined wMAN model not only outperforms the state-of-the-art weakly-supervised method by a signiﬁcant margin but also obtains an improvement of 10% for the Recall@1 accuracy metric over stronglysupervised state-of-the-art methods on the DiDeMo dataset.
I NTRODUCTION
Video understanding has been a mainstay of artiﬁcial intelligence research.
Recent work has sought to better reason about videos by learning more effective spatio-temporal representations (Tran et al., 2015; Carreira & Zisserman, 2017).
The video moment retrieval task, also known as text-to-clip retrieval , combines language and video understanding to ﬁnd activities described by a natural language sentence.
The main objective of the task is to identify the video segment within a longer video that is most relevant to a sentence.
This requires a model to learn the mapping of correspondences (alignment) between the visual and natural language modalities.
In the strongly-supervised setting, existing methods (Hendricks et al., 2017; Chen et al., 2018;
Ghosh et al., 2019) generally learn joint visual-semantic representations by projecting video and language representations into a common embedding space and leverage provided temporal annotations to learn regressive functions (Gao et al., 2017) for localization.
However, such temporal annotations are often ambiguous and expensive to collect.
Mithun et al.
(2019) seeks to circumvent these problems by proposing to address this task in the weakly-supervised setting where only full video-sentence pairs are provided as weak labels.
However, the lack of temporal annotations renders the aforementioned approaches infeasible.
In their approach (Figure 1a), Mithun et al.
(2019) proposes a Text-Guided Attention (TGA) mechanism to attend on segment-level features w.r.t.
the sentence-level representations.
However, such an approach treats the segment-level visual representations as independent inputs and ignores the contextual information derived from other segments in the video.
More importantly, it does not exploit the ﬁne-grained semantics of each word in the sentence.
Consequently, existing methods are not able to reason about the latent alignment between the visual and language representations comprehensively.
[Figure]
In this paper, we take another step towards addressing the limitations of current weakly-supervised video moment retrieval methods by exploiting the ﬁne-grained temporal and visual relevance of each video frame to each word (Figure 1b).
Our approach is built on two core insights: 1) The temporal occurrence of frames or segments in a video provides vital visual information required to reason about the presence of an event; 2)
The semantics of the query are integral to reasoning about the relationships between entities in the video.
With this in mind, we propose our Weakly-Supervised
Moment Alignment Network (wMAN).
An illustrative overview of our model is shown in Figure 2.
The key component of wMAN is a multi-level co-attention mechanism that is encapsulated by a
Frame-by-Word (FBW) interaction module as well as a Word-Conditioned Visual Graph (WCVG).
To begin, we exploit the similarity scores of all possible pairs of visual frame and word features to create frame-speciﬁc sentence representations and word-speciﬁc video representations.
The intuition is that frames relevant to a word should have a higher measure of similarity as compared to the rest.
The word representations are updated by their word-speciﬁc video representations to create visual-semantic representations.
Then a graph (WCVG) is built upon the frame and visualsemantic representations as nodes and introduces another level of attention between them.
During the message-passing process, the frame nodes are iteratively updated with relational information from the visual-semantic nodes to create the ﬁnal temporally-aware multimodal representations.
The contribution of each visual-semantic node to a frame node is dynamically weighted based on their similarity.
To learn such representations, wMAN also incorporates positional encodings (Vaswani et al., 2017) into the visual representations to integrate contextual information about their relative positions.
Such contextual information encourages the learning of temporally-aware multimodal representations.
To learn these representations, we use a Multiple Instance Learning (MIL) framework that is similar in nature to the Stacked Cross Attention Network (SCAN) model (Lee et al., 2018).
The SCAN model leverages image region-by-word interactions to learn better representations for image-text matching.
In addition, the WCVG module draws inspiration from the Language-Conditioned Graph
Network (LCGN) by Hu et al.
(2019) which seeks to create context-aware object features in an image.
However, the LCGN model works with sentence-level representations, which does not account for the semantics of each word to each visual node comprehensively.
wMAN also distinguishes itself from the above-mentioned models by extracting temporally-aware multimodal representations from videos and their corresponding descriptions, whereas SCAN and LCGN only work on images.
[Figure]
A man is swinging a ...
GRU
Frame
[Figure]
By
[Figure]
Word
Interactions
Message
[Figure]
passingt
[Figure]
Message
[Figure]
Message
[Figure]
Visual
[Figure]
Semantic
[Figure]
contributions of our paper are summarized below:
• We propose a simple yet intuitive MIL approach for weakly-supervised video moment retrieval from language queries by exploiting ﬁne-grained frame-by-word alignment.
• Our novel Word-Conditioned Visual Graph learns richer visual-semantic context through a multi-level co-attention mechanism.
• We introduce a novel application of positional embeddings in video representations to learn temporally-aware multimodal representations.
To demonstrate the effectiveness of our learned temporally-aware multimodal representations, we perform extensive experiments over two datasets, Didemo (Hendricks et al., 2017) and CharadesSTA (Gao et al., 2017), where we outperform the state-of-the-art weakly supervised model by a signiﬁcant margin and strongly-supervised state-of-the-art models on some metrics.
R ELATED W ORK
Most of the recent works in video moment retrieval based on natural language queries (Hendricks et al., 2017; Ghosh et al., 2019; Xu et al., 2019; Zhang et al., 2019; Chen et al., 2018; Yuan et al., 2019; Chen & Jiang, 2019; Chen et al., 2019; Ge et al., 2019) are in the strongly-supervised setting, where the provided temporal annotations can be used to improve the alignment between the visual and language modalities.
Among them, the Moment Alignment Network (MAN) introduced by Zhang et al.
(2019) utilizes a structured graph network to model temporal relationships between candidate moments, but one of the distinguishing factors with our wMAN is that our iterative message-passing process is conditioned on the multimodal interactions between frame and word representations.
The TGN (Chen et al., 2018) model bears some resemblance to ours by leveraging frame-by-word interactions to improve performance.
However, it only uses a single level of attention which is not able to infer the correspondence between the visual and language modalities comprehensively.
In addition, we reiterate that all these methods train their models using strong supervision, whereas we address the weakly supervised setting of this task.
There are also a number of closely-related tasks to video moment retrieval such as temporal activity detection in videos. A general pipeline of proposal and classiﬁcation is adopted by various temporal activity detection models (Xu et al., 2017; Zhao et al., 2017; Shou et al., 2016) with the temporal proposals learnt by temporal coordinate regression.
However, these approaches assume you are provided with a predeﬁned list of activities, rather than an open-ended list provided via natural language queries at test time.
Methods for visual phrase grounding also tend to be provided with natural language queries as input (Chen et al., 2017; Liu et al., 2017; Faghri et al., 2018;
Nam et al., 2017; Karpathy & Fei-Fei, 2015; Plummer et al., 2018), but the task is performed over image regions to locate a related bounding box rather than video segments to locate the correct moment.
W EAKLY -S UPERVISED M OMENT A LIGNMENT N ETWORK
In the video moment retrieval task, given a ground truth video-sentence pair, the goal is to retrieve the most relevant video moment related to the description.
The weakly-supervised version of this task we address can be formulated under the multiple instance learning (MIL) paradigm.
When training using MIL, one receives a bag of items, where the bag is labeled as a positive if at least one item in the bag is a positive, and is labeled as a negative otherwise.
In weakly-supervised moment retrieval, we are provided with a video-sentence pair ( i.e ., a bag) and the video segments are the items that we must learn to correctly label as relevant to the sentence ( i.e ., positive) or not.
Following Mithun et al.
(2019), we assume sentences are only associated with their ground truth video, and any other videos are negative examples.
To build a good video-sentence representation, we introduce our Weakly-Supervised Moment Alignment Network (wMAN), which learns context-aware visualsemantic representations from ﬁne-grained frame-by-word interactions.
As seen in Figure 2, our network has two major components - (1) representation learning constructed from the Frame-ByWord attention and Positional Embeddings (Vaswani et al., 2017), described in Section 3.1, and (2) a Word-Conditioned Visual Graph where we update video segment representations based on context from the rest of the video, described in Section 3.2.
These learned video segment representations are used to determine their relevance to their corresponding attended sentence representations using a LogSumExp (LSE) pooling similarity metric, described in Section 3.3.
3. L EARNING T IGHTLY C OUPLED M ULTIMODAL R EPRESENTATIONS
In this section we discuss our initial video and sentence representations which are updated with contextual information in Section 3.2.
Each word in an input sentence is encoded using GloVe embeddings (Pennington et al., 2014) and then fed into a Gated Recurrent Unit (GRU) (Cho et al., 2014).
The output of this GRU is denoted as W = { w 1 , w 2 , ··· , w Q } where Q is the number of words in the sentence.
Each frame in the input video is encoded using a pretrained Convolutional Neural
Network (CNN).
In the case of a 3D CNN this actually corresponds to a small chunk of sequential frames, but we shall refer to this as a frame representation throughout this paper for simplicity.
To capture long-range dependencies, we feed the frame features into a Long Short-Term Memory (LSTM) (Hochreiter & Schmidhuber, 1997).
The latent hidden state output from the LSTM are concatenated with positional encodings (described below) to form the initial video representations, denoted as V = { v 1 , v 2 , ..., v N } where N is the number of frame features for video V .
Positional Encodings (PE).
To provide some notion of the relative position of each frame we include the PE features which have been used in language tasks like learning language representations using BERT (Devlin et al., 2018; Vaswani et al., 2017).
These PE features can be thought of similar to the temporal endpoint features (TEF) used in prior work for strongly supervised moment retrieval task ( e.g ., Hendricks et al.
, but the PE features provide information about the temporal position of each frame rather than the rough position at the segment level.
For the desired PE features of dimension d , let pos indicates the temporal position of each frame, i is the index of the feature being encoded, and M is a scalar constant, then the PE features are deﬁned as:
[Equation]
Through experiments, we found the hyper-parameter M = 10 ,000 works well for all videos.
These PE features are concatenated with the LSTM encoded frame features at corresponding frame position before going to the cross-modal interaction layers.
3.1. F RAME -B Y -W ORD I NTERACTION
Rather than relating a sentence-level representation with each frame as done in prior work (Mithun et al., 2019), we aggregate similarity scores between all frame and word combinations from the input video and sentence.
These Frame-By-Word (FBW) similarity scores are used to compute attention weights to identify which frame and word combinations are important for retrieving the correct video segment.
More formally, for N video frames and Q words in the input, we compute:
[Equation]
where i ∈ [1 , N ] and j ∈ [1 , Q ] .
[Equation]
Note that v now represents the concatenation of the video frame features and the PE features.
Frame-Speciﬁc Sentence Representations.
We obtain the normalized relevance of each word w.r.t.
to each frame from the FBW similarity matrix, and use it to compute attention for each word:
[Equation]
Using the above-mentioned attention weights, a weighted combination of all the words are created, with correlated words to the frame gaining high attention.
Intuitively, a word-frame pair should have a high similarity score if the frame contains a reference to the word.
Then the frame-speciﬁc sentence representation emphasizes words relevant to the frame and is deﬁned as:
[Equation]
Note that these frame-speciﬁc sentence representations don’t participate in the iterative messagepassing process (Section 3.2).
Instead, they are used to infer the ﬁnal similarity score between a video segment and the query (Section 3.3).
Word-Speciﬁc Video Representations.
To determine the normalized relevance of each frame w.r.t.
to each word, we compute the attention weights of each frame:
[Equation]
[Equation]
Similarly, we attend to the visual frame features with respect to each word by creating a weighted combination of visual frame features determined by the relevance of each frame to the word.
The formulation of each word-speciﬁc video-representation is deﬁned as:
[Equation]
These word-speciﬁc video representations are used in our Word-Conditioned Visual Graph, which we will discuss in the next section.
3. W ORD -C ONDITIONED V ISUAL G RAPH N ETWORK
Given the sets of visual representations, word representations and their corresponding word-speciﬁc video representations, the WCVG aims to learn temporally-aware multimodal representations by integrating visual-semantic and contextual information into the visual features.
To begin, the word representations are updated with their corresponding video representations to create a new visualsemantic representation w visj by concatenating each word w j and the word-speciﬁc video representation f j .
Intuitively, the visual-semantic representations not only contain the semantic context of each word but also a summary of the video with respect to each word.
A fully connected graph is then constructed with the visual features v i and the embedded attention of visual-semantic representations w vis j as nodes.
Iterative Word-Conditioned Message-Passing The iterative message-passing process introduces a second round of FBW interaction similar to that in Section 3.1.1 to infer the latent temporal correspondence between each frame v i and visual-semantic representation w visj .
The goal is to update the representation of each frame v i with the video context information from each word-speciﬁc video representation w vis j .
To realize this, we ﬁrst learn a projection W 1 followed by a ReLU of w vis j to obtain a new word representation to compute a new similarity matrix s (cid:48) ij on every message-passing iteration, namely, we obtain a replacement for w j in Eq.
(2) via w (cid:48)
j = ReLU ( W 1 ( w visj )) .
Updates of Visual Representations During the update process, each visual-semantic node sends its message (represented by its representation) to each visual node weighted by their edge weights.
The representations of the visual nodes at the t-th iteration are updated by summing up the incoming messages as follows:
[Equation]
where a ij is obtained by applying Eq.
(3) to the newly computed FBW similarity matrix s (cid:48) ij , and Wis a learned projection to make v ti the same dimensions as the frame-speciﬁc sentence representation l i (refer to Eq.
(4) ) which are ﬁnally used to compute a sentence-segment similarity score.
3. M ULTIMODAL S IMILARITY I NFERENCE
The ﬁnal updated visual representations V T = { v T 1 , v T 2 , ··· , v TV } ) are used to compute the relevance of each frame to its attended sentence-representations.
A segment is deﬁned as any arbitrary continuous sequence of visual features.
We denote a segment as S
= { v T 1 , · · · , v TK } where K is the number of frame features contained within the segment S .
We adopt the LogSumExp (LSE) pooling similarity metric used in SCAN (Lee et al., 2018), to determine the relevance each proposal segment has to the query: [Equation] R ( v k , l [Equation]
=
[Equation]
λ is a hyperparameter that weighs the relevance of the most salient parts of the video segment to the corresponding frame-speciﬁc sentence representations.
Finally, following Mithun et al.
(2019), given a triplet ( X + , Y + , Y − ) , where ( X + , Y + ) is a positive pair and ( X + , Y − ) a negative pair, we use a margin-based ranking loss L T to train our model which ensures the positive pair’s similarity score is better than the negative pair’s by at least a margin.
Our model’s loss is then deﬁned as:
[Equation]
Sim LSE is used as the similarity metric between all pairs.
At test time, Sim LSE is also used to rank the candidate temporal segments generated by sliding windows, and the top scoring segments will the localized segments corresponding to the input query sentence.
E XPERIMENTS
We evaluate the capability of wMAN to accurately localize video moments based on natural language queries without temporal annotations on two datasets - DiDeMo and Charades-STA.
On the
DiDeMo dataset, we adopt the mean Intersection-Over-Union (IOU) and Recall@N at IOU threshold = θ .
Recall@N represents the percentage of the test sliding window samples which have a overlap of at least θ with the ground-truth segments.
mIOU is the average IOU with the groundtruth segments for the highest ranking segment to each query input.
On the Charades-STA dataset, only the Recall@N metric is used for evaluation.
4. D ATASETS
Charades-STA
The Charades-STA dataset is built upon the original Charades [Sigurdsson et al.
(2016)] dataset which contains video-level paragraph descriptions and temporal annotations for activities.
Charades-STA is created by breaking down the paragraphs to generate sentence-level annotations and aligning the sentences with corresponding video segments.
In total, it contains 12, and 3,720 query-moment pairs in the training and test sets respectively.
For fair comparison with the weakly model TGA (Mithun et al., 2019), we use the same non-overlapping sliding windows of sizes 128 and 256 frames to generate candidate temporal segments. [Table]
DiDeMo The videos in the Distinct Describable Moments (DiDeMo) dataset are collected from
Flickr.
The training, validation and test sets contain 8395, 1065 and 1004 videos respectively.
Each query contains the temporal annotations from at least 4 different annotators.
Each video is limited to a maximum duration of 30 seconds and equally divided into six segments with ﬁve seconds each.
With the ﬁve-second segment as basic temporal unit, there are 21 possible candidate temporal segments for each video.
These 21 segments will used to compute the similarities with the input query and the top scored segment will be returned as the localization result [Table]
4. I MPLEMENTATION D ETAILS
For fair comparison, we utilize the same input features as the state-of-the-art method (Mithun et al., 2019).
Speciﬁcally, the word representations are initialized with GloVe embeddings and ﬁne-tuned during the training process.
For the experiments on DiDeMo, we use the provided mean-pooled visual frame and optical ﬂow features.
The visual frame features are extracted from the fc7 layer of
VGG-16 [Simonyan & Zisserman (2014)] pretrained on ImageNet [Deng et al.
The input visual features for our experiments on Charades-STA are C3D
[Tran et al.
(2015)] features.
We adopt an initial learning rate of 1 e − 5 and a margin = 0 .
5 used in our model’s triplet loss (Eq. 9).
In addition, we use three iterations for the message-passing process.
Our model is trained end-to-end using the ADAM optimizer.
4. R ESULTS 4.3. C HARADES -STA
The results in Table 1 show that our full model outperforms the TGA model by a signiﬁcant margin on all metrics.
In particular, the Recall@1 accuracy when IOU = 0.7 obtained by our model is almost doubled that of TGA.
It is notable that we observe a consistent trend of the Recall@1 accuracies improving the most across all IOU values.
This not only demonstrates the importance of richer joint visual-semantic representations for accurate localization but also the superior capability of our model to learn them.
Our model also performs comparably to the strongly-supervised MAN model on several metrics.
To better understand the contributions of each component of our model, we present a comprehensive set of ablation experiments in Table 2.
Note that our combined wMAN model is comprised of the
FBW and WCVG components as well as the incorporation of PEs.
The results obtained by our FBW variant demonstrate that capturing ﬁne-grained frame-by-word interactions is essential to inferring the latent temporal alignment between these two modalities.
More importantly, the results in the second row (FBW-WCVG) show that the second stage of multimodal attention, introduced by the
WCVG module, encourages the augmented learning of intermodal relationships.
Finally, we also observe that incorporating positional encodings into the visual representations (FBW-WCVG + PE) are especially helpful in improving Recall@1 accuracies for all IOU values.
We provide results for a model variant that include TEFs which encode the location of each video segment.
In Table 2, our experiments show that TEFs actually hurt performance slightly.
Our model variant with PEs (FBW-
[Table]
[Table] WCVG + [Table] ) outperforms the model variant with TEFs (FBW-WCVG + TEF) on all of the metrics.
We theorize that the positional encodings aid in integrating temporal context and relative positions into the learned visual-semantic representations.
This makes it particularly useful for Charades-STA since its videos are generally much longer.
To gain insights into the ﬁne-grained interactions between frames and words, we provide visualizations in Figure 3.
Our model is able to determine the most salient frames with respect to each word relatively well.
In both examples, we observe that the top three salient frames with respect to each word are generally distributed over the same subset of frames.
This seems to be indicative of the fact that our model leverages contextual information from all video frames as well as words in determining the salience of each frame to a speciﬁc word.
4.3. D I D E M O
Table 3 reports the results on the DiDeMo dataset.
In addition to reporting the state-of-the-art weakly supervised results, we also include the results obtained by strongly-supervised methods.
It can be observed that our model outperforms the TGA model by a signiﬁcant margin, even tripling the
Recall@1 accuracy achieved by them.
This demonstrates the effect of learning richer joint visualsemantic representations on the accurate localization of video moments.
In fact, our full model outperforms the strongly-supervised TGN and MCN models on the Recall@1 metric by approximately 10%.
We observe a consistent trend in the ablation studies (Table 4) as with those of Charades-STA.
In particular, through comparing the ablation models FBW and FBW-WCVG, we demonstrate the effectiveness of our multi-level co-attention mechanism in WCVG where it improves the Recall@ accuracy by a signiﬁcant margin.
Similar to our observations in Table 2, PEs help to encourage accurate latent alignment between the visual and language modalities, while TEFs fail in this aspect.
C ONCLUSION
In this work, we propose our weakly-supervised Moment Alignment Network with WordConditioned Visual Graph which exploits a multi-level co-attention mechanism to infer the latent alignment between visual and language representations at ﬁne-grained word and frame level.
Learning context-aware visual-semantic representations helps our model to reason about the temporal occurrence of an
[Table]
A person is holding a glass cup
[Figure]
Finally, our experimental results empirically demonstrate the effectiveness of such representations on the accurate localization of video moments.
A E XPERIMENTS FOR N UMBER OF M ODEL P ARAMETERS ON C HARADES -S TA
[Table]
In Table
[Table]
we show the comparisons of the different methods with different number of model parameters.
While wMAN has 18M parameters as compared to 3M parameters in TGA, the performance gains are not simply attributed to the number of model parameters.
We increase the dimensions of visual and semantic representations as well as corresponding fully-connected layers in the TGA model which leads to a total of 19M parameters. Despite having more parameters than wMAN, it still does signiﬁcantly worse on all metrics. We also provide results obtained by a direct adaptation of the Language-Conditioned Graph Network (LCGN), which is designed to work on the image level for VQA as well. While LCGN leverages attention over the words in the natural language query, the computed attention is only conditioned on the entire sentence without contextual information derived from the objects’ visual representations.
In contrast, the co-attention mechanism in our combined wMAN model is conditioned on both semantic and contextual visual information derived from words and video frames respectively.
LCGN is also a lot more complicated and requires signiﬁcantly more computing resources than wMAN. Despite possessing much more parameters than wMAN, it is still not able to achieve comparable results to ours. B A BLATION ON N UMBER OF M ESSAGE P ASSING R OUNDS
[Table] [Table]
In this section, we include ablation results on the number of message-passing rounds required to learn effective visual-semantic representations.
In our experiments, we have found that three rounds work best on both Charades-Sta and DiDeMo.
C A BLATION ON T EST S ETS
[Table]
[Table] The results in Tables 8 and 9 emphasize the importance of both PEs and WCVG in our approach. We observe the best performances when PEs are integrated into the visual-semantic representations learned by the WCVG. The consistency of these performance gains across both datasets seems indicative of the increased capability of our model to learn temporally-aware visual-semantic representations.