LGDN: Language-Guided Denoising Network for Video-Language Modeling
Anonymous Author(s)
AffiliationAddress email
Abstract
Video-language modeling has attracted much attention with the rapid growthof web videos.
Most existing methods assume that the video frames and textdescription are semantically correlated, and focus on video-language modeling atvideo level.
However, this hypothesis often fails for two reasons: (1) With the richsemantics of video contents, it is difficult to cover all frames with a single videolevel description; (2) A raw video typically has noisy/meaningless information(e.g., scenery shot, transition or teaser).
Although a number of recent works deployattention mechanism to alleviate this problem, the irrelevant/noisy information stillmakes it very difficult to address.
To overcome such challenge, we thus proposean efficient and effective model, termed Language-Guided Denoising Network(LGDN), for video-language modeling.
Different from most existing methods thatutilize all extracted video frames, LGDN dynamically filters out the misalignedor redundant frames under the language supervision and obtains only 2–4 salientframes per video for cross-modal token-level alignment.
Extensive experiments onfive public datasets show that our LGDN outperforms the state-of-the-arts by largemargins.
We also provide detailed ablation study to reveal the critical importanceof solving the noise issue, in hope of inspiring future video-language work.
Introduction
Humans are exposed to the world through a variety of sensory organs, such as eyes, ears, and the senseof touch.
In the past few years, multi-modal data (e g., text or video) has grown and accumulatedrapidly on the Internet, which brings the increasing demands for video-language understanding.
Asone of the fundamental topics, video-language modeling is still challenging due to the heterogeneityof the video-text data.
More notably, the video-text data is typically noisy (e.g., misaligned orsemi-relevant, as shown in Figure 1), leading to intractable video-language modeling.
The dominant paradigm [8, 29, 12, 13, 44] for video-language modeling is to first extract languagefeatures and dense video features via off-the-shelf language and vision models (e.g., BERT [7], 3D
CNN [47]), and then model the cross-modal representation by defining the objective function (e.g.,triplet loss [16]) within a joint semantic space.
Although achieving great success, these methodstypically densely sample frames from a full sequence of raw video to obtain richer representationand thus cost excessive computation.
Since the heavy computation makes it challenging to train thewhole network end-to-end, they often achieve sub-optimal performance in video-language modeling.
Recently, ClipBERT [24] proposes a sparse sampling strategy to tackle this drawback.
Concretely,
ClipBERT first samples video frames sparsely (8–16 frames per video), and then models the cross-
[Figure]
modal alignment at frame-level.
This sparse sampling paradigm enables end-to-end training, leadingto much better performance.
Nevertheless, token-level cross-modal interaction, which has achievedgreat success in image-text modeling [20, 25], is still not well explored for video-language modelingdue to the heavy resource computation (even with 8–16 frames per video).
Moreover, both thedominant paradigm and ClipBERT’s sparse sampling paradigm assume that video frames and the textdescription (w.r.t.
a video-text pair) are semantically correlated, which is often invalid in practice.
The correlation hypothesis often fails for two reasons: (1) With the rich semantics of video contents,it is hard to cover all frames with a single video-level description; (2) A raw video often has noisy ormeaningless information (e.g., scenery shot, transition or teaser).
For the dominant paradigm whichutilizes densely-sampled frames, though often with self-attention mechanism [42], the irrelevant/noisyinformation makes it hard to learn high-quality video-language representation.
For the sparse samplingparadigm used in ClipBERT that models the cross-modal alignment at frame-level, the misalignedframe-text pairs are wrongly forced to become closer, which inevitably leads to inaccurate crossmodal alignment.
Overall, due to this noise issue (see Figure 1), video-language modeling is stillchallenging.
Note that humans also encounter such problem in reality, but seem to be born with theability to resist noise.
That is, everyone can quickly scan through the entire video, easily ignore thenoisy frames and focus on the salient ones given the text.
Motivated by this human ability, we propose a L anguage- G uided D enoising N etwork termed LGDNto dynamically filter out irrelevant or redundant information under the language supervision for bettervideo-language modeling.
Concretely, we devise a S alient F rame P roposal (SFP) mechanism whichadopts four strategies to estimate frame-level relevance scores under the language supervision andproposes/selects only salient frames (per video) for precisely video-language modeling.
Althoughthe frame embeddings and text embeddings can be (roughly) aligned by introducing a M omentum
V ideo-Level C ontrastive L earning (MVCL) module, it is vital to precisely establish frame-textalignment for proposing salient frames.
Therefore, based on multiple instance learning (MIL), wepropose a M omentum F rame-Level Multiple Salient-instance learning (MSL) - C ontrastive L earning(MFCL) module for video-language modeling at frame-level.
Finally, with our SFP mechanism, wepropose a L anguage-Guided S alient F rame M atching (LSFM) module for fine-grained alignment,which adopts a token-aware cross-attention Transformer for cross-modal token-level alignment.
Our main contributions are as follows: (1) We devise a salient frame proposal mechanism that candynamically filter out irrelevant information under the language supervision, meanwhile maintainingsalient information.
(2) We propose an end-to-end framework termed LGDN for video-languagemodeling with cross-modal interaction at three levels: language-guided salient frame matching attoken-level, momentum frame-level MSL-contrastive learning, and momentum video-level contrastivelearning.
(3) We evaluate our LGDN on five public datasets and find that our LGDN outperformsthe latest competitors by large margins.
We also provide detailed ablation study to reveal the criticalimportance of solving the noise issue, in hope of inspiring future video-language work.
Related Work
Video-language Modeling.
Video-language modeling, a fundamental research topic that is beneficial for search engine and video recommendation, has attracted a lot of attention in recent yearswith the rapid growth of web videos.
Previous works have made great efforts to model richer rep- resentations for video and text modalities and then align the features of the two modalities by theobjective function (e.g., triplet loss).
One common representative approach [5, 19] is to adopt a
Graph Convolution Network (GCN) to extract richer information for video-text retrieval.
Anotherrepresentative approach [29, 12, 13, 51, 28] is to exploit extra experts (e.g., object, motion, speech)for video-language modeling.
Recently, ClipBERT [24] proposes a sparse sampling strategy thatenables end-to-end training, thus achieving higher performance.
Moreover, Frozen in Time [2] alsofollows a sparse sampling paradigm, and proposes an end-to-end trainable model that is designedto take advantage of both large-scale image and video captioning datasets.
However, as illustratedin Figure 1, a raw video typically has noisy/meaningless information, and thus the presence ofmisaligned frames is inevitable during video-language modeling.
Note that most existing methodsassume that the video frames and paired text are semantically correlated, without considering thenoise phenomenon.
Although a self-attention mechanism has been widely applied, the misalignedframes still harm the cross-modal alignment.
In this work, we thus propose a salient frame proposalmechanism to effectively (and directly) address this problem.
Cross-Modal Alignment Objective Functions
Most previous methods adopt triplet loss as amajor objective function for video-language modeling.
CGMSCD [13] points out that the triplet losssometimes leads to a wrong learning direction and thus devises an adaptive margin triplet loss forrepresentation learning.
More recent works [40, 17, 18] propose to apply the InfoNCE contrastiveloss [46, 37, 6] to enhance representation learning.
Particularly, COTS [31] introduces a momentummechanism [14] to maintain more negative samples for image-text contrastive learning.
Following
COTS, we propose momentum video-level contrastive learning for video-text global alignment.
Notethat MIL-NCE [34] enhances the InfoNCE loss with multiple-instance learning (MIL) to cope withthe misaligned narration descriptions in HowTo100M [35].
In this paper, we thus propose momentumframe-level MSL-contrastive learning to assist in addressing the misaligned frame problem.
Methodology
Figure 2 gives a brief overview of our LGDN framework for video-language modeling, which iscomposed of four main components: 1) language and vision representation extractors; 2) momentum video-level contrastive learning; 3) momentum frame-level MSL-contrastive learning, and 4)language-guided salient frame matching.
In the following, we will describe each component in detail.
3.1 Feature Representation
Vision Representation.
Given an input video V as a sequence of frames { E i } Ni =1 , where N isthe length of the video, we utilize a 2-D vision Transformer (e.g., ViT) as our vision backbone toextract frame-level features E = { E 1 , E 2 , ..., E N } .
Each frame E i of video V can be representedas E i =
[ e cls ; e 1 ; ... ;
e k v − 1 ] ∈ R k v × D v , where e cls denotes the [CLS] token, k v denotes the patchsequence length, and D v denotes the dimension of the patch embeddings.
We utilize a fully-connectedlayer to project the [CLS] token into the frame embedding f ei .
We then deploy a temporal module T(e.g.
, a Transformer layer) to aggregate the frame embeddings to obtain the final video embedding:
[Equation]
where f v denotes the entire vision (video) encoder.
Language Representation.
Given an input text L , we utilize BERT-Base as our language backboneto extract text feature L , which can be represented as L =
[ l cls ; l 1 ; ... ;
l k l − 1 ] ∈ R k l × D l , where l cls isthe [CLS] token, k l is the token sequence length, and D l is the dimension of the token embeddings.
We deploy a fully-connected layer to project the [CLS] token into the text embedding f l = f l ( L ) ,where f l is the language encoder.
3.2 Momentum Video-Level Contrastive Learning (MVCL) Module
Note that our LGDN is designed to filter out the unmatched/redundant frames for better token-levelalignment, without leveraging the temporal information of the videos explicitly.
Therefore, we firstlyintroduce a M omentum V ideo-Level C ontrastive L earning (MVCL) module to address this problem.
[Figure]
The MVCL module utilizes a temporal module (e.g., Transformer block) to aggregate the frameembeddings to obtain the video embedding.
Contrastive learning is then applied for holistic video-textalignment.
However, video data takes up large GPU memory and the mini-batch size tends to besmall with strict resource, which brings harm to contrastive learning.
Inspired by MoCo [14], weintroduce the momentum mechanism to maintain massive negative samples in memory bank forcontrastive learning.
Concretely, We firstly maintain video memory bank M v = { ˆq vj } N m j =1 and textmemory bank M l = { ˆq lj } N m j =1 to store video/text features, where N m denotes the memory bank sizeand ˆq vj / ˆq lj denotes the j -th stored video/text feature vector.
Let f v (with parameters θ v ) and ˆ f v (withparameters ˆ θ v ) denote vision encoder and vision momentum encoder, respectively.
Similarly, let f l(with parameters θ l ) and ˆ f l (with parameters ˆ θ l ) denote language encoder and language momentumencoder, respectively.
The parameters of momentum encoders are updated by:
[Equation]
where m is the momentum coefficient hyper-parameter.
The loss function is thus constructed as follow: for each video V i in mini-batch B , we define thevideo-to-text contrastive loss between its paired text L i and all negative samples in the text memorybank M l , resulting in an InfoNCE loss (with τ being the temperature hyper-parameter):
[Equation]
where ˆf li = ˆ f v ( l i ) , and the similarity of two features is measured by the cosine similarity.
Similarly,given each text description L i in mini-batch B , we define the text-to-video contrastive loss as:
[Equation]
where ˆf vi = ˆ f v ( V i ) .
Finally, the objective function for MVCL is defined as follows:
[Equation]
[Table]
R ( j | i ) =
f ei,j · f li
R ( j | i ) =
f ei,j · f li + ˆf ei,j · ˆf li
R ( j | i ) = ˆf ei,j · f li + f ei,j · ˆf li
R ( j | i ) =
( f ei,j + ˆf ei,j ) · ( f li + ˆf li )
3.3 Salient Frame Proposal (SFP) Mechanism
As shown in Figure 1, video-text data inevitably contains misaligned frame-text pairs.
Although anattention mechanism has been applied in Eq.
(1), the irrelevant and noisy information would stillmislead the cross-modal alignment in our model.
To alleviate this problem, we thus propose a S alient
F rame P roposal (SFP) mechanism for video-language modeling.
The core idea of our SFP mechanism is to dynamically filter out misaligned or redundant frames andmaintain only a few important frames to represent the video well, which are called as salient frames.
Formally, for each video-text pair, we first identify the relevance score R ( j | i ) between the text L iand the j -th frame E i,j of the video V i .
Further, we perform language-guided denoising to retainonly top- N salient salient frames by filtering out the unmatched/redundant frames from each video.
Since only video-level annotations are provided, we need to estimate the relevance scores R automatically.
As shown in Table 1, we introduce four strategies for estimating relevance scores.
(1) SimDotprediction relies on the output of two separate encoders (i.e., frame encoder f e and language encoderf l ) to model the relevance score R ( j | i ) by computing the dot product of the frame embeddingf ei,j = f
e ( E i,j ) and the text embedding f li = f l ( L i ) .
However, since video-text data is noisy, onlyutilizing single-modality encoders may result in incorrect salient frames.
(2) Momentum predictionimproves SimDot prediction by introducing the supervision of momentum encoders (i.e., momentumframe encoder ˆ f e and momentum language encoder ˆ f l ), where the momentum frame embeddingˆf ei,j = ˆ f e
( E i,j ) and the momentum text embedding ˆf li =
ˆ f l ( L i ) .
(3) CrossMom prediction considers the frame-text alignment that is directly built on the interaction between one modality encoderand
another modality’s momentum encoder.
(4) Collaborative prediction combines Momentumprediction and CrossMom prediction for better performance.
Although the frame embeddings and text embeddings can be (roughly) aligned through applyingvideo-text contrastive learning in Sec.
3.2, it is vital to precisely establish frame-text alignment forproposing/selecting salient frames.
To this end, we introduce the MFCL module below.
3.4 Momentum Frame-Level MSL-Contrastive Learning (MFCL) Module
To dynamically filter out the unmatched/redundant frames, we propose to adopt frame-level contrastive learning to directly measure the relevance scores R between video frames and paired text.
However, video data often contains misaligned frame-text pairs.
Simply applying standard NCE-basedcontrastive learning would force the misaligned frame-text pairs to be pulled closer, which inevitablyhas negative effect on learning high-quality frame-text representation.
Inspired by MIL-NCE [34],we thus propose a M omentum F rame-Level Multiple Salient-instance Learning (MSL) C ontrastive
L earning (MFCL) module to assist in alleviating the noise problem.
The core idea is to use the salientframes filtered by the SFP Mechanism in each video to form a set of positive candidate pairs, insteadof considering each positive pair independently.
In this work, we suppose that MFCL and SFP havemutual interdependence so that they can bring boost to each other during training.
Similar to MVCL, we additionally maintain a frame-level memory bank M e = { ˆq ej ′ } N m ∗ N j ′ = to storeframe features, where N m is the memory bank size, N is the number of sampled frames per video,and ˆq ej ′ is a stored frame feature vector.
Given each text description L i in mini-batch B , we selectsalient frames filtered by the SFP Mechanism in the paired video V i to form a set of positive candidate(frame-text) pairs S i and all frame samples in M e to form the negative ones.
We then define thetext-to-frame contrastive loss as:
[Equation]
where S i =
{ E i,j } Nj =1 is the positive frame set of the video V i , N is the frame sequence length ofthe video, f li = f l
L i ) , and ˆf si = { ˆf eij } Nj =
= { ˆ f e ( E i,j ) } Nj =1 .
Similarly, given each positive frame set S i , we define the frame-to-text contrastive loss as:
[Equation]
where ˆf li
= ˆ f l ( L i ) and f si = { f eij } Nj =
= { f e ( E i,j ) } Nj =1 (text memory bank M l = { ˆq lj ′ } N m j ′
=1 isdefined in Sec. 3.2).
As a result, by combining the text-to-frame and frame-to-text contrastive losses,the objective function for MFCL is given by:
[Equation]
3.5 Language-Guided Salient Frame Matching (LSFM) Module
After obtaining language-guided salient frames, we utilize a multi-modal cross-attention fusion
Transformer (see Figure 2) to capture token-level semantic alignment between visual patches andwords for better performance (see the design details of this Transformer in the supp.
material).
Further, we take the [CLS] token embedding outputted by the multi-modal fusion Transformer as thejoint representation of a frame-text pair ( V i , L i ), and deploy a fully-connected layer to predict thematched probability, which is similar to the sentence pair classification task in BERT’s pre-trainingphase.
The matching loss is defined as:
[Equation]
where E i,j denotes j -th frame feature of video V
i , L i denotes text feature, D salient is the set ofsalient frame-text pairs obtained by applying the SFP mechanism to the mini-batch, and y i,j is theground-truth matching label (0 or 1) of the frame-text pair ( E i,j , L i ) .
During inference, we use amean pooling layer to aggregate all salient frame scores as the video-level prediction score.
Finally, by combining all the proposed modules for video-language modeling at three levels, we trainour LGDN model via minimizing the total objective function:
[Equation]
Experiments4.1 Datasets and SettingsPre-Training Datasets.
Due to the restricted computing resources, we follow COTS [24] topre-train our LGDN on the pure image-text datasets.
Our pre-training datasets consists of Conceptual
Captions [41], SBU [38], VG [22] and MSCOCO [27], which contains 5.2 million image-text pairs.
We additionally apply CC12M [3] (about 2 million URLs are now invalid) for better performance,which accumulates 15.2 million image-text pairs in total.
Downstream Datasets.
We evaluate our proposed LGDN on four public video-text retrieval datasets:
MSR-VTT [49], MSVD [4], DiDeMo [15], and VATEX [45].
To further demonstrate the generalapplicability of our LGDN, we also carry out experiments on a public video-question answeringdataset: MSRVTT-QA [48].
We present the details of these downstream datasets as well as theevaluation metrics for downstream tasks in the supp.
material.
Implementation Details.
Following previous work [24], we sample N = 16 frames per video:each video is equally split into 16 segments and one frame is randomly sampled from each segment.
We empirically set the initial learning rate to 1e-5 and adopt AdamW [30] with a weight decay of0.02 for 5 epochs.
In the warm-up stage (first epoch), the model is trained to optimize Eq.
(10)without applying SFP mechanism.
We also set the other hyper-parameters uniformly as: salientframe numbers N salient
= 2 , mini-batch size |B| =
24 , momentum hyper-parameter m = 0 . 99 ,temperature τ = 0 .
07 , and queue size N m = 9 , 600 .
We adopt pre-trained BERT-Base as languageencoder and ViT-Base [9] as vision encoder.
More details are given in the supp.
material.
Didemo Recall@Mean
SpeedUp 60.
MSRVTT Recall@Mean
SpeedUp 𝑁 𝑠𝑎𝑙𝑖𝑒𝑛𝑡 𝑁 𝑠𝑎𝑙𝑖𝑒𝑛𝑡 (a)
（ b ） 52.
asalinet2= asalinet2= 𝑁 𝑁 𝑠𝑎𝑙𝑖𝑒𝑛𝑡 =
𝑁 𝑠𝑎𝑙𝑖𝑒𝑛𝑡 =(c)
[Table]
Local
[Table] [Table]
4.2 Ablation Study
In this subsection
[Table]
we conduct comprehensive ablation study to investigate the contributions ofdifferent components of our full model.
If not specifically indicated, we set N = 16 for globalalignment and N salient = 2 for token-level alignment as the default setting.
Effect of Value Change of N salient and N .
A common perspective for video/video-languageunderstanding is that more frames per video bring better performance.
We thus conduct experimentson the frame number used for token-level alignment in Figure 3(a-b).
We sample N = 16 framesfrom each video and evaluate different variants that use N salient ∈ { 1 , 2 , 3 , 4 , 8 , 16 } frames.
Notethat when N salient = 16 , sampling by our SFP degrades to w/o SFP.
It can be observed that utilizingonly N salient = { 2 , 3 , 4 } salient frames filtered by our SFP significantly outperforms utilizing all16 extracted frames meanwhile enjoying the faster speed (see the green lines).
This suggests thatour SFP mechanism not only selects correct salient frames but also alleviates the noise problem.
Toinvestigate the influence of value change of N on our LGDN, we evenly sample N ∈ { 2 , 3 , 4 , 8 }frames per video and freeze N salient =
{ 1 , 2 } salient frames.
The results in Figure 3(c) indicate thatmore extracted frames per video are beneficial to the token-level alignment in our LGDN model, as itprovides larger candidate set for selecting salient frames.
Meanwhile, when N becomes larger (> 4),the performance tends to converge, further demonstrating the redundancy in the videos.
Contributions of Each Components.
We further demonstrate the contributions of the three objectivefunctions as well as the salient frame proposal (SFP) mechanism used in our full LGDN modelin Table 2.
We start with the objective function L LSFM (w/o SFP), which means only applyingmatching loss in token-level alignment without using the SFP mechanism.
It can be observed that: (1)
L MFCL (and L MVCL ) combined with L LSFM (w/o SFP) can bring improvements, suggesting that globalalignment is beneficial to token-level alignment (during the training stage).
(2) Simply applying theframe-level alignment may cause negative effect while combing with our MSL design brings betterresults.
This demonstrates that our design of L MFCL does help alleviate the noise problem.
(3) Whenthe SFP mechanism is added (see L LSFM (w/o SFP)
+ L MFCL + L MVCL vs. L LSFM + L MFCL + L MVCL ),the performance is significantly improved, which clearly shows the effectiveness of our proposed
SFP mechanism.
(4) For the same trained full LGDN model, combining the global and token-levelalignment during inference can bring further improvements.
Note that our full LGDN still achievesthe state-of-the-art on MSR-VTT even without considering global alignment during inference.
[Table]
[Table] [Table]
[Table] [Table]
4.3 Comparison to the State-of-the-Arts
We first report the text-video retrieval results on MSR-VTT with three data partitions in Table 3.
Itcan be observed that: (1) our LGDN outperforms all previous works by large margins.
Particularly,as compared with the most recent model Frozen in Time [2], our LGDN achieves an improvement of7.9% (38.9% vs. 31.0%) for Text-to-Video R@1 on the MSR-VTT 1k-A test set.
(2) Our LGDNalso outperforms methods utilizing extra modalities (e.g., motion and audio) or those pre-trainedon extremely-large video data (e.g., HowTo100M).
(3) When leveraging a much larger pre-training(image-text) dataset, our LGDN (marked with † ) achieves significant improvements.
To demonstrate the robustness of our model, we also evaluate it on VATEX, MSVD, and Didemoin Tables 4–6, respectively.
Due to limited space, only text-to-video retrieval is considered here.
For VATEX (Table 4), our LGDN significantly outperforms the state-of-the-art method Support
Set which is trained on an order of magnitude more data.
Our LGDN still performs the best on
MSVD (Table 5) and Didemo (Table 6).
Particularly, in the Didemo dataset, each description is
[Table] [Table]
[Figure]
annotated with localization information, in other words, annotations may only be aligned with thelocalized moments, thus causing the noise problem as many methods utilize all frames as the input.
Recent works exploit temporal labels of captions to alleviate the noise problem and achieve higherperformance.
However, even without considering this, our LGDN still largely outperforms the mostrecent method Frozen [2], further demonstrating the effectiveness of our LGDN.
To show the general applicability of our LGDN, we evaluate our LGDN on the VideoQA task in
Table 7.
Even without utilizing large-scale video datasets devoted to the VideoQA task, our LGDNoutperforms all competitors, validating the effectiveness of our LGDN in VideoQA.
In addition, toreveal the critical importance of solving the noise issue for video-language modeling, we directlyapply the SFP mechanism to the latest model CLIP4Clip [33] in Table 8.
We find that applying the
SFP mechanism brings boost to Clip4CLIP.
The ensemble mechanism further improves the results,indicating that the proposed SFP mechanism is complementary to the baseline.
4.4 Visualization Results
We provide visualization of our LGDN in Figure 4.
We uniformly sample 5 frames from each videoand provide relevance scores of 5 frames on the left, and the red ones denote salient frames selected bythe SFP mechanism.
It can be seen that: (1) Although the holistic video is semantically related to thepaired text, there still exist noisy frames (e.g., the transition in Frame 1 and Frame 3 of Query7500)and unrelated frames (e.g., in Frame 4 and Frame 5 of Query7544, a man is rolling while the pairedtext is ‘a car goes racing down the road’).
The relevance scores obtained from the SFP mechanismcorrectly measure the consistency between each frame and the paired text, which indeed helps our
LGDN to precisely filter out noisy information for better video-language modeling.
Conclusion
In this work, we propose a novel Language-Guided Denoising Network (LGDN) for video-languagemodeling, which can dynamically filter out the unmatched or redundant frames under the languagesupervision and thus maintain only 2–4 salient frames per video for cross-modal token-level alignment.
Extensive experiments on five public datasets show that our LGDN outperforms the state-of-the-artsby large margins.
In the future, we will consider aggregating temporal information on salient framesand apply our approach to more challenging video-language tasks (e.g., video grounding).
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes](c)
Did you discuss any potential negative societal impacts of your work?
[No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] We specified all the training details in Sec.
4.1 and Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] We included the details of training
time and resources used in Appendix.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cited the creatorsof all used datasets in Sec. 4.1.
(b) Did you mention the license of the assets?
[No]
) Did you include any new assets either in the supplemental material or as a URL?
[No]
) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No]
Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No]
you used crowdsourcing or conducted research with human subjects..
you include the full text of instructions given to participants and screenshots, if
applicable
[N
you describe any potential participant risks, with links to Institutional Review
Board
applicable
you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N A]