LGDN: Language-Guided Denoising Network for Video-Language Modeling
Abstract
Video-language modeling has attracted much attention with the rapid growth of web videos.
Most existing methods assume that the video frames and text description are semantically correlated, and focus on video-language modeling at video level.
However, this hypothesis often fails for two reasons: (1) With the rich semantics of video contents, it is difficult to cover all frames with a single videolevel description; (2) A raw video typically has noisy/meaningless information (e.g., scenery shot, transition or teaser).
Although a number of recent works deploy attention mechanism to alleviate this problem, the irrelevant/noisy information still makes it very difficult to address.
To overcome such challenge, we thus propose an efficient and effective model, termed Language-Guided Denoising Network (LGDN), for video-language modeling.
Different from most existing methods that utilize all extracted video frames, LGDN dynamically filters out the misaligned or redundant frames under the language supervision and obtains only 2–4 salient frames per video for cross-modal token-level alignment.
Extensive experiments on five public datasets show that our LGDN outperforms the state-of-the-arts by large margins.
We also provide detailed ablation study to reveal the critical importance of solving the noise issue, in hope of inspiring future video-language work.
Introduction
Humans are exposed to the world through a variety of sensory organs, such as eyes, ears, and the sense of touch.
In the past few years, multi-modal data (e.g., text or video) has grown and accumulated rapidly on the Internet, which brings the increasing demands for video-language understanding.
As one of the fundamental topics, video-language modeling is still challenging due to the heterogeneity of the video-text data.
More notably, the video-text data is typically noisy (e.g., misaligned or semi-relevant, as shown in Figure 1), leading to intractable video-language modeling.
The dominant paradigm [8, 30, 13, 14, 45] for video-language modeling is to first extract language features and dense video features via off-the-shelf language and vision models (e.g., BERT [7], 3D
CNN [48]), and then model the cross-modal representation by defining the objective function (e.g., triplet loss [17]) within a joint semantic space.
Although achieving great success, these methods typically densely sample frames from a full sequence of raw video to obtain richer representation and thus cost excessive computation.
Since the heavy computation makes it challenging to train the whole network end-to-end, they often achieve sub-optimal performance in video-language modeling.
Recently, ClipBERT [25] proposes a sparse sampling strategy to tackle this drawback.
Concretely,
[Figure] cartoon woman [Figure] a bench while a woman in blue appears
[Figure]
ClipBERT first samples video frames sparsely (8–16 frames per video), and then models the crossmodal alignment at frame-level.
This sparse sampling paradigm enables end-to-end training, leading to much better performance.
Nevertheless, token-level cross-modal interaction, which has achieved great success in image-text modeling [21, 26], is still not well explored for video-language modeling due to the heavy resource computation (even with 8–16 frames per video).
Moreover, both the dominant paradigm and ClipBERT’s sparse sampling paradigm assume that video frames and the text description (w.r.t.
a video-text pair) are semantically correlated, which is often invalid in practice.
The correlation hypothesis often fails for two reasons: (1) With the rich semantics of video contents, it is hard to cover all frames with a single video-level description; (2) A raw video often has noisy or meaningless information (e.g., scenery shot, transition or teaser).
For the dominant paradigm which utilizes densely-sampled frames, though often with self-attention mechanism [43], the irrelevant/noisy information makes it hard to learn high-quality video-language representation.
For the sparse sampling paradigm used in ClipBERT that models the cross-modal alignment at frame-level, the misaligned frame-text pairs are wrongly forced to become closer, which inevitably leads to inaccurate crossmodal alignment.
Overall, due to this noise issue (see Figure 1), video-language modeling is still challenging.
Note that humans also encounter such problem in reality, but seem to be born with the ability to resist noise.
That is, everyone can quickly scan through the entire video, easily ignore the noisy frames and focus on the salient the
Motivated by this human ability, we propose a L anguage- G uided D enoising N etwork termed LGDN to dynamically filter out irrelevant or redundant information under the language supervision for better video-language modeling.
Concretely, we devise a S alient F rame P roposal (SFP) mechanism which adopts four strategies to estimate frame-level relevance scores under the language supervision and proposes/selects only salient frames (per video) for precisely video-language modeling.
Although the frame embeddings and text embeddings can be (roughly) aligned by introducing a M omentum
V ideo-Level C ontrastive L earning (MVCL) module, it is vital to precisely establish frame-text alignment for proposing salient frames.
Therefore, based on multiple instance learning (MIL), we propose a M omentum F rame-Level Multiple Salient-instance learning (MSL) - C ontrastive L earning (MFCL) module for video-language modeling at frame-level.
Finally, with our SFP mechanism, we propose a L anguage-Guided S alient F rame M atching (LSFM) module for fine-grained alignment, which adopts a token-aware cross-attention Transformer for cross-modal token-level alignment.
Our main contributions are as follows: (1) We devise a salient frame proposal mechanism that can dynamically filter out irrelevant information under the language supervision, meanwhile maintaining salient information.
(2) We propose an end-to-end framework termed LGDN for video-language modeling with cross-modal interaction at three levels: language-guided salient frame matching at token-level, momentum frame-level MSL-contrastive learning, and momentum video-level contrastive learning.
(3) We evaluate our LGDN on five public datasets and find that our LGDN outperforms the latest competitors by large margins.
We also provide detailed ablation study to reveal the critical importance of solving the noise issue, in hope of inspiring future video-language work.
Related Work
Video-Language Modeling.
Video-language modeling, a fundamental research topic that is beneficial for search engine and video recommendation, has attracted a lot of attention in recent years with the rapid growth of web videos.
Previous works have made great efforts to model richer rep- resentations for video and text modalities and then align the features of the two modalities by the objective function (e.g., triplet loss).
One common representative approach [5, 20] is to adopt a
Graph Convolution Network (GCN) to extract richer information for video-text retrieval.
Another representative approach [30, 13, 14, 52, 29] is to exploit extra experts (e.g., object, motion, speech) for video-language modeling.
Recently, ClipBERT [25] proposes a sparse sampling strategy that enables end-to-end training, thus achieving higher performance.
Moreover, Frozen in Time [2] also follows a sparse sampling paradigm, and proposes an end-to-end trainable model that is designed to take advantage of both large-scale image and video captioning datasets.
However, as illustrated in Figure 1, a raw video typically has noisy/meaningless information, and thus the presence of misaligned frames is inevitable during video-language modeling.
Note that most existing methods assume that the video frames and paired text are semantically correlated, without considering the noise phenomenon.
Although a self-attention mechanism has been widely applied, the misaligned frames still harm the cross-modal alignment.
In this work, we thus propose a salient frame proposal mechanism to effectively (and directly) address this problem.
Functions
Most previous methods adopt triplet loss as a major objective function for video-language modeling.
CGMSCD [14] points out that the triplet loss sometimes leads to a wrong learning direction and thus devises an adaptive margin triplet loss for representation learning.
More recent works [41, 18, 19] propose to apply the InfoNCE contrastive loss [47, 38, 6] to enhance representation learning.
Particularly, COTS [32] and BriVL [12] introduce a momentum mechanism [15] to maintain more negative samples for image-text contrastive learning.
Following these two state-of-the-art models, we propose momentum video-level contrastive learning for video-text global alignment in this paper.
Note that MIL-NCE [35] enhances the InfoNCE loss with multiple-instance learning (MIL) to cope with the misaligned narration descriptions in HowTo100M [36].
In this work, we thus propose momentum frame-level MSL-contrastive learning to assist in addressing the misaligned frame problem.
Methodology
Figure 2 gives a brief overview of our LGDN framework for video-language modeling, which is composed of four main components: 1) language and vision representation extractors; 2) momentum video-level contrastive learning; 3) momentum frame-level MSL-contrastive learning, and 4) language-guided salient frame matching.
In the following, we will describe each component in detail.
3.1 Feature Representation
Vision Representation.
Given an input video V as a sequence of frames { E i } Ni =1 , where N is the length of the video, we utilize a 2-D vision Transformer (e.g., ViT) as our vision backbone to extract frame-level features E = { E 1 , E 2 , ..., E N } .
Each frame E i of video V can be represented as E i =
[ e cls ; e 1 ; ... ;
e k v − 1 ] ∈ R k v × D v , where e cls denotes the [CLS] token, k v denotes the patch sequence length, and D v denotes the dimension of the patch embeddings.
We utilize a fully-connected layer to project the [CLS] token into the frame embedding f ei .
We then deploy a temporal module T (e.g., a Transformer layer) to aggregate the frame embeddings to obtain the final video embedding:
[Equation]
where f v denotes the entire vision (video) encoder.
Language Representation.
Given an input text L , we utilize BERT-Base as our language backbone to extract text feature L , which can be represented as L =
[ l cls ; l 1 ; ... ;
l k l − 1 ] ∈ R k l × D l , where l cls is the [CLS] token, k l is the token sequence length, and D l is the dimension of the token embeddings.
We deploy a fully-connected layer to project the [CLS] token into the text embedding f l = f l ( L ) , where f l is the language encoder.
3.2 Momentum Video-Level Contrastive Learning (MVCL) Module
Note that our LGDN is designed to filter out the unmatched/redundant frames for better token-level alignment, without leveraging the temporal information of the videos explicitly.
Therefore, we firstly introduce a M omentum V ideo-Level C ontrastive L earning (MVCL) module to address this problem.
[Figure]
The MVCL module utilizes a temporal module (e.g., Transformer block) to aggregate the frame embeddings to obtain the video embedding.
Contrastive learning is then applied for holistic video-text alignment.
However, video data takes up large GPU memory and the mini-batch size tends to be small with strict resource, which brings harm to contrastive learning.
Inspired by MoCo [15], we introduce the momentum mechanism to maintain massive negative samples in memory bank for contrastive learning.
Concretely, We firstly maintain video memory bank M v = { ˆq vj } N m j =1 and text memory bank M l = { ˆq lj } N m j =1 to store video/text features, where N m denotes the memory bank size and ˆq vj / ˆq lj denotes the j -th stored video/text feature vector.
Let f v (with parameters θ v ) and ˆ f v (with parameters ˆ θ v ) denote vision encoder and vision momentum encoder, respectively.
Similarly, let f l (with parameters θ l ) and ˆ f l (with parameters ˆ θ l ) denote language encoder and language momentum encoder, respectively.
The parameters of momentum encoders are updated by:
[Equation]
where m is the momentum coefficient hyper-parameter.
The loss function is thus constructed as follow: for each video V i in mini-batch B , we define the video-to-text contrastive loss between its paired text L i and all negative samples in the text memory bank M l , resulting in an InfoNCE loss (with τ being the temperature hyper-parameter):
[Equation]
where ˆf li = ˆ f v ( l i ) , and the similarity of two features is measured by the cosine similarity.
Similarly, given each text description L i in mini-batch B , we define the text-to-video contrastive loss as:
[Equation]
where ˆf vi = ˆ f v ( V i ) .
Finally, the objective function for MVCL is defined as follows:
[Equation]
[Table]
R ( j | i ) =
f ei,j · f li
R ( j | i ) =
f ei,j · f li + ˆf ei,j · ˆf li
R ( j | i ) = ˆf ei,j · f li + f ei,j · ˆf li
R ( j | i ) =
( f ei,j + ˆf ei,j ) · ( f li + ˆf li )
3.3 Salient Frame Proposal (SFP) Mechanism
As shown in Figure 1, video-text data inevitably contains misaligned frame-text pairs.
Although an attention mechanism has been applied in Eq.
(1), the irrelevant and noisy information would still mislead the cross-modal alignment in our model.
To alleviate this problem, we thus propose a S alient
F rame P roposal (SFP) mechanism for video-language modeling.
The core idea of our SFP mechanism is to dynamically filter out misaligned or redundant frames and maintain only a few important frames to represent the video well, which are called as salient frames.
Formally, for each video-text pair, we first identify the relevance score R ( j | i ) between the text L i and the j -th frame E i,j of the video V i .
Further, we perform language-guided denoising to retain only top- N salient salient frames by filtering out the unmatched/redundant frames from each video.
Since only video-level annotations are provided, we need to estimate the relevance scores R automatically.
As shown in Table 1, we introduce four strategies for estimating relevance scores.
(1) SimDot prediction relies on the output of two separate encoders (i.e., frame encoder f e and language encoder f l ) to model the relevance score R ( j | i ) by computing the dot product of the frame embedding f ei,
j = f e ( E i,j ) and the text embedding f li = f l ( L i ) .
However, since video-text data is noisy, only utilizing single-modality encoders may result in incorrect salient frames.
(2) Momentum prediction improves SimDot prediction by introducing the supervision of momentum encoders (i.e., momentum frame encoder ˆ f e and momentum language encoder ˆ f l ), where the momentum frame embedding ˆf ei,j = ˆ f e ( E i,j ) and the momentum text embedding ˆf li =
ˆ f l ( L i ) .
(3) CrossMom prediction considers the frame-text alignment that is directly built on the interaction between one modality encoder and another modality’s momentum encoder.
(4) Collaborative prediction combines Momentum prediction and CrossMom prediction for better performance.
Although the frame embeddings and text embeddings can be (roughly) aligned through applying video-text contrastive learning in Sec.
3.2, it is vital to precisely establish frame-text alignment for proposing/selecting salient frames.
To this end, we introduce the MFCL module below.
3.4 Momentum Frame-Level MSL-Contrastive Learning (MFCL) Module
To dynamically filter out the unmatched/redundant frames, we propose to adopt frame-level contrastive learning to directly measure the relevance scores R between video frames and paired text.
However, video data often contains misaligned frame-text pairs.
Simply applying standard NCE-based contrastive learning would force the misaligned frame-text pairs to be pulled closer, which inevitably has negative effect on learning high-quality frame-text representation.
Inspired by MIL-NCE [35], we thus propose a M omentum F rame-Level Multiple Salient-instance Learning (MSL) C ontrastive
L earning (MFCL) module to assist in alleviating the noise problem.
The core idea is to use the salient frames filtered by the SFP Mechanism in each video to form a set of positive candidate pairs, instead of considering each positive pair independently.
In this work, we suppose that MFCL and SFP have mutual interdependence so that they can bring boost to each other during training.
Similar to MVCL, we additionally maintain a frame-level memory bank M e = { ˆq ej ′ } N m ∗ N j ′ = to store frame features, where N m is the memory bank size, N is the number of sampled frames per video, and ˆq ej ′ is a stored frame feature vector.
Given each text description L i in mini-batch B , we select salient frames filtered by the SFP Mechanism in the paired video V i to form a set of positive candidate (frame-text) pairs S i and all frame samples in M e to form the negative ones.
We then define the text-to-frame contrastive loss as:
[Equation]
where S i =
{ E i,j } Nj =1 is the positive frame set of the video V i , N is the frame sequence length of the video, f li = f l
L i ) , and ˆf si = { ˆf eij } Nj =
= { ˆ f e ( E i,j ) } Nj =1 .
Similarly, given each positive frame set S i , we define the frame-to-text contrastive loss as:
[Equation]
where ˆf li
= ˆ f l ( L i ) and f si = { f eij } Nj =
= { f e ( E i,j ) } Nj =1 (text memory bank M l = { ˆq lj ′ } N m j ′
=1 is defined in Sec. 3.2).
As a result, by combining the text-to-frame and frame-to-text contrastive losses, the objective function for MFCL is given by:
[Equation]
3.5 Language-Guided Salient Frame Matching (LSFM) Module
After obtaining language-guided salient frames, we utilize a multi-modal cross-attention fusion
Transformer (see Figure 2) to capture token-level semantic alignment between visual patches and words for better performance (see the design details of this Transformer in the supp.
material).
Further, we take the [CLS] token embedding outputted by the multi-modal fusion Transformer as the joint representation of a frame-text pair ( V i , L i ), and deploy a fully-connected layer to predict the matched probability, which is similar to the sentence pair classification task in BERT’s pre-training phase.
The matching loss is defined as:
[Equation]
where E i,j denotes j -th frame feature of video V
i , L i denotes text feature, D salient is the set of salient frame-text pairs obtained by applying the SFP mechanism to the mini-batch, and y i,j is the ground-truth matching label (0 or 1) of the frame-text pair ( E i,j , L i ) .
During inference, we use a mean pooling layer to aggregate all salient frame scores as the video-level prediction score.
Finally, by combining all the proposed modules for video-language modeling at three levels, we train our LGDN model via minimizing the total objective function:
[Equation]
Experiments 4.1 Datasets and Settings
Pre-Training Datasets.
Due to the restricted computing resources, we follow COTS [25] to pre-train our LGDN on the pure image-text datasets.
Our pre-training datasets consists of Conceptual
Captions [42], SBU [39], VG [23] and MSCOCO [28], which contains 5.2 million image-text pairs.
We additionally apply CC12M [3] (about 2 million URLs are now invalid) for better performance, which accumulates 15.2 million image-text pairs in total.
Downstream Datasets.
We evaluate our proposed LGDN on four public video-text retrieval datasets:
MSR-VTT [50], MSVD [4], DiDeMo [16], and VATEX [46].
To further demonstrate the general applicability of our LGDN, we also carry out experiments on a public video-question answering dataset: MSRVTT-QA [49].
We present the details of these downstream datasets as well as the evaluation metrics for downstream tasks in the supp.
material.
Implementation Details.
Following previous work [25], we sample N = 16 frames per video: each video is equally split into 16 segments and one frame is randomly sampled from each segment.
We empirically set the initial learning rate to 1e-5 and adopt AdamW [31] with a weight decay of 0.02 for 5 epochs.
In the warm-up stage (first epoch), the model is trained to optimize Eq.
(10) without applying SFP mechanism.
We also set the other hyper-parameters uniformly as: salient frame numbers N salient = 2 , mini-batch size |B| = 24 , momentum hyper-parameter m = 0 . 99 , temperature τ = 0 .
07 , and queue size N m = 9 , 600 .
We adopt pre-trained BERT-Base as language encoder and ViT-Base [9] as vision encoder.
More details are given in the supp.
material.
Evaluation Metrics.
We adopt two widely-used metrics in cross-modal retrieval: Recall at K (R@K,
K = 1 , 5 , 10 ), and Median Rank (MdR) / Mean Rank (MnR).
R@K means the percentage of correct matching in the K nearest points, and MdR / MnR measures the median / mean rank of target items in the retrieved ranking list.
We also report two additional metrics named ‘R@Sum’ and ‘R@Mean’ in our ablation study, which sums/averages all recall metrics for overall evaluation. Following ClipBERT [25], we also report accuracy (Acc) in video-question answering task. Didemo Recall@Mean
SpeedUp 60.
MSRVTT Recall@Mean
SpeedUp 𝑁 𝑠𝑎𝑙𝑖𝑒𝑛𝑡 𝑁 𝑠𝑎𝑙𝑖𝑒𝑛𝑡 (a)
（ b ） 52.
asalinet2= asalinet2= 𝑁 𝑁 𝑠𝑎𝑙𝑖𝑒𝑛𝑡 =
𝑁 𝑠𝑎𝑙𝑖𝑒𝑛𝑡 =(c)
[Table]
4.2 Ablation Study
In this subsection
[Table]
we conduct comprehensive ablation study to investigate the contributions of different components of our full model.
If not specifically indicated, we set N = 16 for global alignment and N salient = 2 for token-level alignment as the default setting.
Effect of Value Change of N salient and N .
A common perspective for video/video-language understanding is that more frames per video bring better performance.
We thus conduct experiments on the frame number used for token-level alignment in Figure 3(a-b).
We sample N = 16 frames from each video and evaluate different variants that use N salient ∈ { 1 , 2 , 3 , 4 , 8 , 16 } frames.
Note that when N salient = 16 , sampling by our SFP degrades to w/o SFP.
It can be observed that utilizing only N salient = { 2 , 3 , 4 } salient frames filtered by our SFP significantly outperforms utilizing all 16 extracted frames meanwhile enjoying the faster speed (see the green lines).
This suggests that our SFP mechanism not only selects correct salient frames but also alleviates the noise problem.
To investigate the influence of value change of N on our LGDN, we evenly sample N ∈ { 2 , 3 , 4 , 8 } frames per video and freeze N salient =
{ 1 , 2 } salient frames.
The results in Figure 3(c) indicate that more extracted frames per video are beneficial to the token-level alignment in our LGDN model, as it provides larger candidate set for selecting salient frames.
Meanwhile, when N becomes larger (> 4), the performance tends to converge, further demonstrating the redundancy in the videos.
Contributions of Each Components.
We further demonstrate the contributions of the three objective functions as well as the salient frame proposal (SFP) mechanism used in our full LGDN model in Table 2.
We start with the objective function L LSFM (w/o SFP), which means only applying matching loss in token-level alignment without using the SFP mechanism.
It can be observed that: (1)
L MFCL (and L MVCL ) combined with L LSFM (w/o SFP) can bring improvements, suggesting that global alignment is beneficial to token-level alignment (during the training stage).
(2) Simply applying the frame-level alignment may cause negative effect while combing with our MSL design brings better results.
This demonstrates that our design of L MFCL does help alleviate the noise problem.
(3) When the SFP mechanism is added (see L LSFM (w/o SFP)
+ L MFCL + L MVCL vs. L LSFM + L MFCL + L MVCL ), the performance is significantly improved, which clearly shows the effectiveness of our proposed
SFP mechanism.
(4) For the same trained full LGDN model, combining the global and token-level alignment during inference can bring further improvements.
Note that our full LGDN still achieves the state-of-the-art on MSR-VTT even without considering global alignment during inference.
[Table]
[Table] [Table]
[Table] [Table]
4.3 Comparison to the State-of-the-Arts
We first report the text-video retrieval results on MSR-VTT with three data partitions in Table 3.
It can be observed that: (1) our LGDN outperforms all previous works by large margins.
Particularly, as compared with the most recent model Frozen in Time [2], our LGDN achieves an improvement of 7.9% (38.9% vs. 31.0%) for Text-to-Video R@1 on the MSR-VTT 1k-A test set.
(2) Our LGDN also outperforms methods utilizing extra modalities (e.g., motion and audio) or those pre-trained on extremely-large video data (e.g., HowTo100M).
(3) When leveraging a much larger pre-training (image-text) dataset, our LGDN (marked with † ) achieves significant improvements.
To demonstrate the robustness of our model, we also evaluate it on VATEX, MSVD, and Didemo in Tables 4–6, respectively.
Due to limited space, only text-to-video retrieval is considered here.
For VATEX (Table 4), our LGDN significantly outperforms the state-of-the-art method Support
Set which is trained on an order of magnitude more data.
Our LGDN still performs the best on
MSVD (Table 5) and Didemo (Table 6).
Particularly, in the Didemo dataset, each description is
[Table] [Table]
[Table]
annotated with localization information, in other words, annotations may only be aligned with the localized moments, thus causing the noise problem as many methods utilize all frames as the input.
Recent works exploit temporal labels of captions to alleviate the noise problem and achieve higher performance.
However, even without considering this, our LGDN still largely outperforms the most recent method Frozen [2] [Table] further demonstrating the effectiveness of our LGDN [Table]
To show the general applicability of our LGDN [Table] we evaluate our LGDN on the VideoQA task in
Table 7.
Even without utilizing large-scale video datasets devoted to the VideoQA task, our LGDN outperforms all competitors, validating the effectiveness of our LGDN in VideoQA.
In addition, to reveal the critical importance of solving the noise issue for video-language modeling, we directly apply the SFP mechanism to the latest model CLIP4Clip [34] in Table 8.
We find that applying the
SFP mechanism brings boost to Clip4CLIP.
The ensemble mechanism further improves the results, indicating that the proposed SFP mechanism is complementary to the baseline.
4.4 Additional Results
Applying SFP to Different Frame Sampling Techniques.
Note that our SFP mechanism must be combined with a frame sampling technique since we adopt a two-stage sampling strategy in this paper.
Thus, we apply our SFP mechanism to three frame sampling techniques: Sparse Sampling, Random
Sampling, and Dense Uniform (equally interval sampling).
The obtained results on the MSR-VTT 1kA test set are provided in Table 9.
It can be observed that our SFP significantly boosts different sampling strategies, further demonstrating the general applicability of our SFP mechanism.
Expansion of Relevance Score Estimator.
In Sec. 3.3, we have proposed four relevance score estimators for the LSFM module.
To find out which is the best, we present the ablation study results for different relevance score estimators in Table 10.
We can see a large gap between SFP and random sampling (w/o SFP), directly demonstrating the effectiveness of the proposed SFP mechanism.
Meanwhile, both Momentum and CrossMom outperform SimDot, suggesting that introducing momentum encoder is beneficial to relevance score estimation.
Collaborative that combines Momentum and CrossMom generally leads to further improvements.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
Model Capacity.
We also provide the detailed comparison to other methods in terms of model capacity and R@SUM (on the MSR-VTT 1kA test set) in Table 11.
It can be clearly seen that:
(i) When fusion layers
are not
used (i
[Table]
e., only global alignment is adopted)
our LGDN (global) outperforms the state-of-the-art method Frozen in Time
2], but with
much
less model parameters.
(ii)
Our full LGDN performs much better than all the competitors, but its parameter number (215M)
is still comparable to that of Frozen in Time (180M) and even significantly smaller than those of the other competitors.
These observations suggest that the performance gains obtained by our LGDN is not due to utilizing more model parameters.
4.5 Visualization Results
We provide visualization of our LGDN in Figure 4.
We uniformly sample 5 frames from each video and provide relevance scores of 5 frames on the left, and the red ones denote salient frames selected by the SFP mechanism.
It can be seen that: (1) Although the holistic video is semantically related to the paired text, there still exist noisy frames (e.g., the transition in Frame 1 and Frame 3 of Query7500) and unrelated frames (e.g., in Frame 4 and Frame 5 of Query7544, a man is rolling while the paired text is ‘a car goes racing down the road’).
The relevance scores obtained from the SFP mechanism correctly measure the consistency between each frame and the paired text, which indeed helps our
LGDN to precisely filter out noisy information for better video-language modeling.
Conclusion
In this work, we propose a novel Language-Guided Denoising Network (LGDN) for video-language modeling, which can dynamically filter out the unmatched or redundant frames under the language supervision and thus maintain only 2–4 salient frames per video for cross-modal token-level alignment.
Extensive experiments on five public datasets show that our LGDN outperforms the state-of-the-arts by large margins.
In the future, we will consider aggregating temporal information on salient frames and apply our approach to more challenging video-language tasks (e.g., video grounding). Acknowledgments and Disclosure of Funding
This work was supported in part by National Natural Science Foundation of China (61976220 and 61832017), Beijing Outstanding Young Scientist Program (BJJWZYJH012019100020098), and the
Research Seed Funds of School of Interdisciplinary Studies, Renmin University of China.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] We specified all the training details in Sec.
4.1 and Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] We included the details of training time and resources used in Appendix. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cited the creators of all used datasets in Sec. 4.1.
(b) Did you mention the license of the assets?
[No] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[No] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[No] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]