Original: Enhancing and ScalingCross-Modality Alignment for Multimodal Pre-Training via  Gradient Harmonization
Revised: Scaling Multimodal Pre-Training via Cross-Modality Gradient Harmonization

Original: Anonymous Author(s)

Original: AfﬁliationAddress

Original: Abstract
Revised: Abstract

Original: Self-supervised pre-training recently demonstrates success on large-scale multimodal data, and state-of-the-art contrastive  methods often enforce the featureconsistency from cross-modality inputs, such as video/audio or video/text pairs.
Revised: Self-supervised pre-training recently demonstrates success on large-scale multimodal data, and state-of-the-art contrastive learning methods often enforce the feature consistency from cross-modality inputs, such as video/audio or video/text pairs.

Original: Despite its convenience to formulate and leverage in practice, such cross-modalityalignment (CMA) is only a weak and noisy supervision, since two modalities can besemantically misaligned even they are temporally aligned.
Revised: Despite its convenience to formulate and leverage in practice, such cross-modality alignment (CMA) is only a weak and noisy supervision, since two modalities can be semantically misaligned even they are temporally aligned.

Original: For example, even in the(often adopted) instructional videos, a speaker can sometimes refer to somethingthat is not visually present in the current frame; and the semantic misalignmentwould only be more unpredictable for the raw videos collected from unconstrainedinternet sources.
Revised: For example, even in the (often adopted) instructional videos, a speaker can sometimes refer to something that is not visually present in the current frame; and the semantic misalignment would only be more unpredictable for the raw videos collected from unconstrained internet sources.

Original: We conjecture that might cause conﬂicts and biases among modalities, and may hence prohibit CMA from scaling up to training with larger andmore heterogeneous data.
Revised: We conjecture that might cause conﬂicts and biases among modalities, and may hence prohibit CMA from scaling up to training with larger and more heterogeneous data.

Original: This paper ﬁrst veriﬁes our conjecture by observing that,even in the latest VATT pre-training using only narrated videos, there exist stronggradient conﬂicts between different CMA losses within the same sample triplet(video, audio, text), indicating them as the noisy source of supervision.
Revised: This paper ﬁrst veriﬁes our conjecture by observing that, even in the latest VATT pre-training using only narrated videos, there exist strong gradient conﬂicts between different CMA losses within the same sample triplet (video, audio, text), indicating them as the noisy source of supervision.

Original: We thenpropose to harmonize such gradients during pre-training, via two techniques: (i)cross-modality gradient realignment: modifying different CMA loss gradients forone sample triplet, so that their gradient directions are in more agreement; and (ii)gradient-based curriculum learning: leveraging the gradient conﬂict informationon an indicator of sample noisiness, to develop a curriculum learning strategyto prioritize training with less noisy sample triplets.
Revised: We then propose to harmonize such gradients during pre-training, via two techniques: (i) cross-modality gradient realignment: modifying different CMA loss gradients for one sample triplet, so that their gradient directions are in more agreement; and (ii) gradient-based curriculum learning: leveraging the gradient conﬂict information on an indicator of sample noisiness, to develop a curriculum learning strategy to prioritize training with less noisy sample triplets.

Original: Applying those gradientharmonization techniques to pre-training VATT on the HowTo100M dataset, weconsistently improve its performance on different downstream tasks.
Revised: Applying those gradient harmonization techniques to pre-training VATT on the HowTo100M dataset, we consistently improve its performance on different downstream tasks.

Original: Moreover, weare able to scale VATT pre-training to more complicated non-narrative Youtube8Mdataset{+Youtube
 M dataset+} to further improve the state-of- the-arts.
Revised: Moreover, we are able to scale VATT pre-training to more complicated non-narrative {+Youtube
 M dataset+} to further improve the state-of-the-arts.

Original: We will open-source our codes andmodels.

Original:  Introduction
Revised:  Introduction

Original: Self-supervised pre-training scales up deep learning to leverage massive unlabeled data.
Revised: Self-supervised pre-training scales up deep learning to leverage massive unlabeled data.

Original: Beyondthe maturity of pre-training over single-modality data such as language [2] or images [3], therecent success on multimodal pre-training [4, 5] demonstrates the versatility to synergize the richmultimodal information and to beneﬁt a variety of downstream tasks.
Revised: Beyond the maturity of pre-training over single-modality data such as language [2] or images [3], the recent success on multimodal pre-training [4, 5] demonstrates the versatility to synergize the rich multimodal information and to beneﬁt a variety of downstream tasks.

Original: Many methods of this category[6, 7, 8, 4], especially the latest contrastive pre-training methods [5, 9, 10, 11], consider the mostorganic supervision as the cross-modality alignment ( CMA ), e.g., in the same video sequence, thevideo frames, audio stream, and text scripts are temporally aligned and hence should naturallyhave correspondence.
Revised: Many methods of this category [6, 7, 8, 4], especially the latest contrastive pre-training methods [5, 9, 10, 11], consider the most organic supervision as the cross-modality alignment ( CMA ), e.g., in the same video sequence, the

Original: Such assumed cross-modal correspondence could be exploited as “mutualsupervision" for each modality to learn consistent semantic features with each other. For example, Aligned “On
Revised: [Figure]

Original: “World Cup Semi Final” “Raising fund for UNICEF” “Whether it's in a kitchen”
Revised: “World Cup Semi Final” “Raising fund for UNICEF” “Whether it's in a kitchen”

Original: [Figure]
Revised: [Figure]

Revised: video frames, audio stream, and text scripts are temporally aligned and hence should naturally have correspondence.

Revised: Such assumed cross-modal correspondence could be exploited as “mutual supervision" for each modality to learn consistent semantic features with others.

Original:  the latest Video-Audio-Text Transformer (VATT) [11] hinges on this multimodal CMA a priori totrain a transformer-based architecture (even a single backbone by sharing weights among the threemodalities), using two contrastive losses to align video-text and video-audio pairs, respectively.
Revised: For example, the latest Video-Audio-Text Transformer (VATT) [11] hinges on this multimodal CMA a priori to train a transformer-based architecture (even a single backbone by sharing weights among three modalities), using two contrastive losses to align video-text and video-audio pairs, respectively.

Original: Theirresults set many new records in downstream tasks with neither supervised pre-training nor predeﬁnedinductive biases, showing strong “universal" generalizability despite the modality and the domaingaps.
Revised: Their results set many new records in downstream tasks with neither supervised pre-training nor predeﬁned inductive biases, showing strong “universal" generalizability despite the modality and domain gaps.

Original: In this paper, we focus on studying gradients in a modality-agnostic single-backbone setting of(VATT) [11].
Revised: In this paper, we focus on studying gradients in a modality-agnostic single-backbone setting of VATT [11].

Original: Unfortunately, the CMA assumption rests on a very shaky foundation, especially for uncurated videoscollected in the wild.
Revised: Unfortunately, the CMA assumption rests on a very shaky foundation, especially for uncurated videos collected in the wild.

Original: The CMA can be weak even for the instructional videos such as HowTo100M[12] that are commonly adopted as the current multimodal pre-training data source.
Revised: The CMA can be weak even for the instructional videos such as HowTo100M [12] that are commonly adopted as the current multimodal pre-training data source.

Original: For instance,a speaker can refer to something before or after actually demonstrating it visually; she or he couldalso skip verbally explaining something that is visually happening, because that may be trivial or bealready clear enough in the visual context [13].
Revised: For instance, a speaker can refer to something before or after actually demonstrating it visually; she or he could also skip verbally explaining something that is visually happening, because that may be trivial or be already clear enough in the visual context [13].

Original: Such discrepancy would be ampliﬁed if consideringmany other outliers in video (e.g. background objects) or language (e.g. off-topic small talks).
Revised: Such discrepancy would be ampliﬁed if considering many other outliers in video (e.g. background objects) or language (e.g. off-topic small talks).

Original: Thesemantic misalignment will be even more severe, and in unpredictable forms, when we go beyondinstructional videos and explore training with non-narrative, free-from internet videos.
Revised: The semantic misalignment will be even more severe, and in unpredictable forms, when we go beyond instructional videos and explore training with non-narrative, free-from internet videos.

Original: Figure 1shows a few examples of such cross-modal misalignment in the non-narrative
Revised: Figureshows a few examples of such cross-modal misalignment in the non-narrative

Original: Youtube

Revised: Youtube

Original: M dataset [1],which would put CMA-guided pre-training in jeopardy.
Revised: M dataset [1], which would put CMA-guided pre-training in jeopardy.

Original: This important problem is yet under-studiedin the self-supervised multimodal pre-training literature.
Revised: This important problem is yet under-studied in the self-supervised multimodal pre-training literature.

Original: That said, there has been a few recent effortssuch as Multiple-Instance Learning (MIL) NCE [13] and noisy pair pruning as in [14].
Revised: That said, there has been a few recent efforts such as Multiple-Instance Learning (MIL) NCE [13] and noisy pair pruning as in [14].

Original: We conjecture that the vanilla CMA might provide only weak, noisy and even misleading supervisionfor multimodal pre-training; hence it might constitute a hidden bottleneck when scaled up with morerealistic data (beyond instructional videos) whose modalities are poorly aligned.
Revised: We conjecture that the vanilla CMA might provide only weak, noisy and even misleading supervision for multimodal pre-training; hence it might constitute a hidden bottleneck when scaled up with more realistic data (beyond instructional videos) whose modalities are poorly aligned.

Original: We verify thisconjecture by inspecting two large-scale multimodal datasets, HowTo100M [15] and Youtube8M[1].
Revised: We verify our conjecture by inspecting two large-scale multimodal datasets, HowTo100M [15] and Youtube8M [1].

Original: Speciﬁcally, when we adopt pairwise CMA losses (e.g. video-audio, video-text) within thecross-modal sample triplets (video, audio, text), a majority of the gradients strongly conﬂict with eachother in terms of their directions, indicating the noisy and misaligned supervision.
Revised: Speciﬁcally, when we adopt pairwise CMA losses (e.g. video-audio, video-text) within the cross-modal sample triplets (video, audio, text), a majority of the gradients strongly conﬂict with each other in terms of their directions.

Revised: We further identify (in Figure 3) that such strong conﬂict in gradients are correlated to noisiness of samples, indicating the noisy and misaligned supervision could be an important cause of gradient conﬂicts.

Original: The cross-modalconﬂict not onlycauses unstable training convergence, it alsocauses modality-speciﬁc overﬁtting;e.g. semantically strong representations for one modality and meaningless representations for anothermodality.
Revised: Moreover, the cross-modal conﬂict leads to not only unstable training convergence, but also modality-speciﬁc overﬁtting; e.g. semantically strong representations for one modality while collapsed representations for another [16], yielding weak overall performance in cross-modality tasks such as Text-Video Retrieval as shown in Table 1.

Original: Having identiﬁed this gap, we propose to harmonize those gradients so that they become mutuallycompatible in learning the uniﬁed representations.
Revised: Build upon the above conjectures and observations, we propose to harmonize those gradients so that they become mutually compatible in learning the uniﬁed representations.

Original: Speciﬁcally, we introduce two techniques: (i)cross-modal gradient realignment, where we modify different pairwise CMA loss gradients to aligntheir directions for the same sample triplet, using gradient surgery [16] developed for multi-task learning; and (ii) gradient-based curriculum learning, where we leverage the gradient conﬂict informationto indicate sample noisiness (e.g. a triplet whose CMA gradients are in more agreement is consideredless “noisy" and would be prioritized for cross-modal pre-training) and a curriculum learning strategyis developed accordingly.
Revised: Speciﬁcally, we introduce two techniques: (i) cross-modal gradient realignment, where we modify different pairwise CMA loss gradients to align their directions for the same sample triplet, using gradient surgery [17] developed for multi-task learning; and (ii) gradient-based curriculum learning, where we leverage the gradient conﬂict information to indicate sample noisiness (e.g. a triplet whose CMA gradients are in more agreement is considered less “noisy" and would be prioritized for cross-modal pre-training) and a curriculum learning strategy is developed accordingly.

Original: Both techniques are found to boost VATT performance on downstreamtasks, not only on instructional videos, but even more when the more complicated non-narrative datais involved.
Revised: Both techniques are found to boost VATT performance on downstream tasks, not only on instructional videos, but even more when the more complicated non-narrative data is involved.

Original: Our contributions can be summarized as the following:
Revised: Our contributions can be summarized as the following:

Original: • We suggest that the commonly assumed CMA might be an important yet overlooked perfor- mance hurdle for scaling up multimodal pre-training, and we observe severe misalignmenteven from the state-of-the-art (SOTA) model [11] on the relatively aligned data [12].• We propose to consistently improve the pre-training pipeline in the only baseline in thisdirection, VATT (modality-agnostic setting), resulting in better downstream performance.•
Revised: • We suggest that the commonly assumed CMA might be an important yet overlooked performance hurdle for scaling up multimodal pre-training, and we observe severe misalignment even from the state-of-the-art (SOTA) model [11] on the relatively aligned data [12]. • We propose to consistently improve the pre-training pipeline in the only baseline in this direction, VATT (modality-agnostic setting), resulting in better downstream performance.

Original:  With the help of our proposed techniques, we consistently improve the pre-training ofVATT, resulting in better downstream performance, e g. up to 58% gain in the rank metricof video-text retrieval task.
Revised: • With the help of our proposed techniques, we consistently improve the pre-training of VATT, resulting in better downstream performance, e.g. up to 58% gain in the rank metric of video-text retrieval task.

Original: Moreover, we are able to scale VATT pre-training to more complicated non-narrative Youtube8M dataset, which further yields new state-of-the-artperformance in a modality-agnostic setting.
Revised: Moreover, we are able to scale VATT pre-training to more complicated non-narrative Youtube8M dataset, which further yields new state-of-the-art performance in a modality-agnostic setting.

Original:  Related Work2.1 Self-Supervised Multimodal Pre-training
Revised:  Related Work 2.1 Self-Supervised Multimodal Pre-training

Original: The most organic self-supervision can arguably be found in the multimodal data that are abundantlyavailable in the digital world: their temporal alignment naturally lead to cross-modality mutualsupervision, hence meaningful features can be learned without requiring human annotation.
Revised: The most organic self-supervision can arguably be found in the multimodal data that are abundantly available in the digital world: their temporal alignment naturally lead to cross-modality mutual supervision, hence meaningful features can be learned without requiring human annotation.

Original: Multimodal pre-training can thus leverage richer semantic cues than single-modality counterparts, andoutperform in various downstream tasks such as image classiﬁcation [5, 11], video action recognition[5, 11, 13, 10], audio event classiﬁcation [5, 11, 10], and video-text retrieval [5, 11, 13].
Revised: Multimodal pre-training can thus leverage richer semantic cues than single-modality counterparts, and outperform in various downstream tasks such as image classiﬁcation [5, 11], video action recognition [5, 11, 13, 10], audio event classiﬁcation [5, 11, 10], and video-text retrieval [5, 11, 13].

Original: Mostof those methods formulate pre-text tasks, including cross-modality correspondence [6, 7, 8, 10],cross-modality clustering [17], cross-modality longer context prediction [18], or a combination ofmultiple pre-text tasks [19].
Revised: Most of those methods formulate pre-text tasks, including cross-modality correspondence [6, 7, 8, 10], cross-modality clustering [18], cross-modality longer context prediction [19], or a combination of multiple pre-text tasks [20].

Original: Contrastive learning [3, 20] has recently emerged as the dominant approach in self-supervisedlearning.
Revised: Contrastive learning [3, 21] has recently emerged as one dominant approach in self-supervised learning.

Original: In multimodal pre-training, the temporal co-occurrence naturally supplies the positivesample to contrast with.
Revised: In multimodal pre-training, the temporal co-occurrence naturally supplies the positive sample to contrast with.

Original: [5] adopts a multimodal versatile network to embed each modality intothe same vector space, that is trained from end to end with multimodal pairwise contrastive losses(video-audio, and video text).
Revised: [5] adopts a multimodal versatile network to embed each modality into the same vector space, that is trained from end to end with multimodal pairwise contrastive losses (video-audio, and video text).

Original: CrossCLR [9] further takes the intra-modality similarities into account.[10] generalizes the instance discrimination idea to design a cross-modal discrimination task, i.e.,predicting which audio matches a video.
Revised: CrossCLR [9] further takes the intra-modality similarities into account. [10] generalizes the instance discrimination idea to design a cross-modal discrimination task, i.e., predicting which audio matches a video.

Original: Most recently, VATT [11] combines the strength ofconvolution-free Transformer and multimodal contrastive learning.
Revised: Most recently, VATT [11] combines the strength of convolution-free Transformer and multimodal contrastive learning.

Original: Speciﬁcally, it ﬁrst validate theidea of using a modality-agnostic single-backbone transformer, to work on video, audio and textmodalities, it follows the exact BERT [21] and ViT [22] architecture, except injecting modalityspeciﬁc tokenization layers and linear projections.
Revised: Speciﬁcally, it ﬁrst validate the idea of using a modality-agnostic single-backbone transformer, to work on video, audio and text modalities, it follows the exact BERT [22] and ViT [23] architecture, except injecting modalityspeciﬁc tokenization layers and linear projections.

Original: Despite their success, existing multimodal contrastive learning methods [5, 9, 11] hinge on the“free" cross-modalities correspondence mined through the multimodal contrastive loss, i.e., the CMAassumption.
Revised: Despite their success, existing multimodal contrastive learning methods [5, 9, 11] hinge on the “free" cross-modalities correspondence mined through the multimodal contrastive loss, i.e., the CMA assumption.

Original: As a result, they are often trained withrelatively well curated or narrated video datasets,such as AudioSet [23], YouCook2 [15], and HowTo100M [12].
Revised: As a result, they are often trained with well curated or narrated video datasets, such as AudioSet [24], YouCook2 [15], and HowTo100M [12].

Original: It remains questionable whether such
Revised: It remains questionable whether such

Original: CMA-guided contrastive learning can generalize well to larger-scale multimodal data in the wild.
Revised: CMA-guided contrastive learning can generalize well to larger-scale multimodal data in the wild.

Original: Infact, even in the HowTo100M dataset, the authors of [12] estimate that around 50% of clip-narrationpairs are not well aligned.
Revised: In fact, even in the HowTo100M dataset, the authors of [12] estimate that around 50% of clip-narration pairs are not well aligned.

Original: The validity of CMA recently arises some attention.
Revised: The validity of CMA has arisen concerns.

Original: Most related to us is [13] which explicitlymodels the misalignment noise using the MIL-NCE loss, that is inherited by VATT [11].
Revised: Most related to us is [13] which explicitly models the misalignment noise using the MIL-NCE loss, that is inherited by VATT [11].

Original: 2.2 Multi-task versus Multimodal Learning
Revised: 2.2 Multi-task versus Multimodal Learning

Original: While earlier self-supervised works used different modality-speciﬁc encoders, the latest multimodalpre-training sees a tendency to explore a versatile and “universal" model shared across modalities[5, 11].
Revised: While earlier self-supervised works used different modality-speciﬁc encoders, the latest multimodal pre-training sees a tendency to explore a versatile and “universal" model shared across modalities [5, 11].

Original: It is easy to notice the resemblance between multimodal learning (especially with a uniﬁedbackbone) and multi-task learning [24, 25].
Revised: It is easy to notice the resemblance between multimodal learning (especially with a uniﬁed backbone) and multi-task learning [25, 26].

Original: The later often assumes multiple tasks to share transferable knowledge and can help each other learn.
Revised: The later often assumes multiple tasks to share transferable knowledge and can help each other learn.

Original: Yet in practice, different tasks can be heterogeneous innature too, often leading to heavily misaligned gradient scales [26] or directions [16].
Revised: Yet in practice, different tasks can be heterogeneous in nature too, often leading to heavily misaligned gradient scales [27] or directions [17].

Original: Such conﬂictinggradients from multiple losses would lead to several undesirable effects, including (i) strong biastowards learning some tasks with largest gradient scales; and (2) slow and unstable convergence.
Revised: Such conﬂicting gradients from multiple losses would lead to several undesirable effects, including (i) strong bias [Figure] towards learning some tasks with largest gradient scales; and (2) slow and unstable convergence.

Original: Similar problems are often faced by multimodal pre-training too.
Revised: Similar problems are often faced by multimodal pre-training too.

Original: As a result, several works have been developed to mitigate the gradient dilemma in multi-task learning.
Revised: As a result, several works have been developed to mitigate the gradient dilemma in multi-task learning.

Original: Methods such as gradient normalization [26] and adaptive loss weight [27, 28] address the gradient[Figure] scale/learning speed imbalance issues.
Revised: Methods such as gradient normalization [27] and adaptive loss weight [28, 29] address the gradient scale/learning speed imbalance issues.

Original: Gradient surgery [16] tries to project gradients with conﬂictingdirections onto the normal plane of each other, to prevent each gradient’s interfering component frombeing applied to the network.
Revised: Gradient surgery [17] tries to project gradients with conﬂicting directions onto the normal plane of each other, to prevent each gradient’s interfering component from being applied to the network.

Original: Later, Gradient vaccine [29] further generalizes this idea extending thegradient projection to arbitrary angle, and uses the exponential moving average of cosine similarityto measure the alignment between different tasks.
Revised: Later, Gradient vaccine [30] further generalizes this idea extending the gradient projection to arbitrary angle, and uses the exponential moving average of cosine similarity to measure the alignment between different tasks.

Original: When it comes to the multimodal learning, [30]investigates optimally blending modality-speciﬁc gradients by linearly re-scaling them.
Revised: When it comes to the multimodal learning, [31] investigates optimally blending modality-speciﬁc gradients by linearly re-scaling them.

Original: Meanwhile,to our best knowledge, there has been no prior study on modality-speciﬁc gradient directions.
Revised: Meanwhile, to our best knowledge, there has been no prior study on modality-speciﬁc gradient directions.

Original:  Methodology
Revised:  Methodology

Original: Pre-requisites: Without loss of generality, we focus on the SOTA multimodal contrastive learningmodel, VATT [11], as our default subject of study; and we follow its modality-agnostic singlebackbone setting due to its compelling performance-efﬁciency trade-off, and due to the fact that itis the most intuitive setting to modify gradients (shared between paired modalities) w.r.t differentend-point objectives.
Revised: Pre-requisites: Without loss of generality, we focus on the SOTA multimodal contrastive learning model, VATT [11], as our default subject of study; and we follow its modality-agnostic singlebackbone setting due to its compelling performance-efﬁciency trade-off, and due to the fact that it is the most intuitive setting to modify gradients (shared between paired modalities) w.r.t different end-point objectives.

Original: In our modality-agnostic single-backbone setting of VATT, the goal of pre-training is to ﬁnd theparameter θ of the model f θ to learn meaningful features for all target modalities (video, audio, text),usually measured by a set of pairwise similarity metrics.
Revised: In our modality-agnostic single-backbone setting of VATT, the goal of pre-training is to ﬁnd the parameter θ of the model f θ to learn meaningful features for all target modalities (video, audio, text), usually measured by a set of pairwise similarity metrics.

Original: VATT uses two pairwise contrastive lossesfor video-audio and video-text respectively to solve cross-modal alignment:
Revised: VATT uses two pairwise contrastive losses for video-audio and video-text respectively to solve cross-modal alignment:

Original: [Equation]
Revised: [Equation]

Original: where CMA va and CMA vt penalize the cross-modal alignment for video-audio and video-text pairs,using the Noise-Contrastive Estimation objective [31], respectively.
Revised: where CMA va and CMA vt penalize the cross-modal alignment for video-audio and video-text pairs, using the Noise-Contrastive Estimation objective [32], respectively.

Original: Figure 2 illustrates the VATT pipeline.
Revised: Figure 2 illustrates the VATT pipeline.

Original: First, video-audio-text triplets are sampled from randomtemporal locations, and then video-text and video-audio pairs are formed accordingly.
Revised: First, video-audio-text triplets are sampled from random temporal locations, and then video-text and video-audio pairs are formed accordingly.

Original: Positive pairsare formed by selecting the two modalities at the same temporal locations from the same video clip;while the negative pairs are obtained by randomly sampling any video, audio, or text from other videoclips.
Revised: Positive pairs are formed by selecting the two modalities at the same temporal locations from the same video clip; while the negative pairs are obtained by randomly sampling any video, audio, or text from other video clips.

Original: We follow the convention in [5, 11] to use the vanilla NCE loss for video-audio pairs, and touse the MIL-NCE loss proposed in [13] to align video-text pairs.
Revised: We follow the convention in [5, 11] to use the vanilla NCE loss for video-audio pairs, and to use the MIL-NCE loss proposed in [13] to align video-text pairs.

Original: Hence, CMA va and CMA vt aredeﬁned as follows:
Revised: Hence, CMA va and CMA vt are deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  where N is the pool of negative pairs, P k ( z t ) denotes the k -neighbouring narrations surrounding theoriginal text z t .
Revised: (3) where N is the pool of negative pairs, P k ( z t ) denotes the k -neighbouring narrations surrounding the original text z t .

Original: The gradients of two pairwise losses CMA va and CMA vt , denoted as g va and g vt ,will be together applied (i.e. simple adding) to updating the model f θ .
Revised: The gradients of two pairwise losses CMA va and CMA vt , denoted as g va and g vt , will be together applied (i.e. simple adding) to updating the model f θ .

Original: 3.1 Observation of Conﬂicting Gradients
Revised: 3.1 Observation of Conﬂicting Gradients

Original: 162 While qualitatively recognizing the cross-modal misalignment is easy, it is a highly challengingendeavor to quantify the misalignment and to automatically ﬁnd misaligned triplets at scale, due tothe vaguely deﬁned criterion as well as the absence of labels [13, 14].
Revised: While qualitatively recognizing the cross-modal misalignment is easy, it is a highly challenging endeavor to quantify the misalignment and to automatically ﬁnd misaligned triplets at scale, due to the vaguely deﬁned criterion as well as the absence of labels [13, 14].

Original: Therefore, the foremost needis to identify a surrogate indicator on how noisy the alignment of a sample triplet is, without makingunrealistic assumptions nor incurring much computational overhead.
Revised: Therefore, the foremost need is to identify a surrogate indicator on how noisy the alignment of a sample triplet is, without making unrealistic assumptions nor incurring much computational overhead.

Original: Our empirical choice of surrogate indicator is the directional alignability (i.e., vector angle) ofgradients generated by different pairwise cross-modal losses, i.e., the cosine similarity between g vaand g vt in the particular example of VATT, after ﬂattening them into two vectors.
Revised: Our empirical choice of surrogate indicator is the directional alignability (i.e., vector angle) of gradients generated by different pairwise cross-modal losses, i.e., the cosine similarity between g va and g vt in the particular example of VATT, after ﬂattening them into two vectors.

Original: The choice is ﬁrstlyrationalized by the coherent gradient hypothesis [32], that healthy model training should witness theoverall gradient stronger in certain directions along which the samples’ gradients reinforce each other.
Revised: The choice is ﬁrstly rationalized by the coherent gradient hypothesis [33], that healthy model training should witness the overall gradient stronger in certain directions along which the samples’ gradients reinforce each other.

Original: Secondly, comparing the gradient directions between multiple pairwise losses could be considered asan ensemble or “cross-check": intuitively, if both video-audio and video-text pairwise consistencylead to the same update direction, then there is a good chance that those modalities are well alignedand the update direction is reliable; otherwise, at least one pair (video-audio, or video-text) mightsuffer from misalignment andCMA provides noisy  guidance.
Revised: Secondly, comparing the gradient directions between multiple pairwise losses could be considered as an ensemble or “cross-check": intuitively, if both video-audio and video-text pairwise consistency lead to the same update direction, then there is a good chance that those modalities are well aligned and the update direction is reliable; otherwise, at least one pair (video-audio, or video-text) might suffer from misalignment and provides noisy cross-modality guidance.

Original: Overall, we conjecture that: aligned video-text-audio triplets should have higher cosine similarity forg va and g vt , and vice versa .
Revised: Overall, we conjecture that: aligned video-text-audio triplets should have higher cosine similarity for g va and g vt , and vice versa .

Original: We compute the cosine similarity between g va and g vt using the samesetting as [32].
Revised: In Figure 3, we display a simple sanity check for our conjecture by depicting a few qualitative examples. On the left side of Figure 3, we randomly sample four videoaudio-text triplets, manually inspected to be aligned (on the topics of football), and then calculate their cos ( g va , g va ) . Similarly on the right side of Figure 3, we choose misaligned video-audio-text triplets to calculate their corresponding gradients’ alignment by cos ( g (cid:48) va , g (cid:48) va ) . cos ( g va , g va ) is clearly positive while cos ( g (cid:48) va , g (cid:48) va ) is negative, that empirically demonstrate the correlation between the alignment of video-text-audio triplets and that of gradients ( g va , g va ) .

Original: In the supplementary, We perform a sanity check for our conjecture by examining itqualitatively.
Revised: To further validate the conjecture, we conduct another experiment, started from a pre-trained VATT network, we further optimizer it with gradient descent for 100 iterations with a batch size of 1024, we then randomly sample 20 video-text-audio triplets out of the samples with top 1% and bottom 1% most aligned gradient, measured by cos ( g (cid:48) va , g (cid:48) va ) . We visually observed that the top 1% group triplets are more semantically aligned while the bottom 1% group triplets are much more noisy in their alignments, the visualization of the 20 random triplets from the top 1% group and bottom 1% group can be found in the supplementary.

Original: Using this surrogate indicator, we examine the cosine similarity between gradient g va and g vt (over500k iterations) of VATT on the HowTo100M dataset [12].
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Equation]

Revised: With the assumption that misalignment in gradients correlates to the noisiness of the sample’s cross-modality alignment, we use the former as the computable surrogate indicator of the later, and examine the cosine similarity between gradient g va and g vt (over 500k iterations) of VATT on the

Revised: HowTo100M dataset [12].

Original: In Figure 3a, we plot the distribution ofcosine similarities, cos ( g va , g vt ) , across 500k iterations of pre-training.
Revised: In Figure 4a, we plot the distribution of cosine similarities, cos ( g va , g vt ) , across 500k iterations of pre-training.

Original: We observe that cos ( g va , g vt )at any iteration resembles a normal distribution, and about half of the gradients g va and g va havemisaligned directions (negative cosine similarities).
Revised: We observe that cos ( g va , g vt ) at any iteration resembles a normal distribution, and about half of the gradients g va and g va have misaligned directions (negative cosine similarities).

Original: The plot echos the empirical ﬁnding in [15] thataround 50% of video-text pairs are not well-aligned on the HowTo100M dataset.
Revised: The plot echos the empirical ﬁnding in [15] that around 50% of video-text pairs are not well-aligned on the HowTo100M dataset.

Original: In the supplementary,we have included similar plots for the YouTube8M dataset [1].
Revised: In the supplementary, we have included similar plots for the YouTube8M dataset [1].

Original: Comparing those plots verify thatnon-narrative video sets have more severe misalignment thanthe narrative ones, hence challengingthe CMA assumption even more.
Revised: Comparing those plots verify that non-narrative datasets have much worse misalignment than narrative ones, hence challenging the CMA assumption even more.

Original: [Figure]
Revised: [Figure]

Original: observations, see supplementary.

Original: The existence of conﬂicting gradients not only makes the convergence harder, but also makes thelearned representation heavily overﬁt some modalities.
Revised: The existence of conﬂicting gradients not only makes the convergence harder, but also makes the learned representation heavily overﬁt some modalities.

Original: In our case, we observe the training to bebiased towards video/audio modalities, which favors the downstream performance of video/audioclassiﬁcation task, yet sets back the performance of video-to-text retrieval task (veriﬁed in Table 1).
Revised: In our case, we observe the training to be biased towards video/audio modalities, which favors the downstream performance of video/audio classiﬁcation task, yet sets back the performance of video-to-text retrieval task (veriﬁed in Table 1).

Original: 3.2 Harmonizing Gradients during Pre-training
Revised: 3.2 Harmonizing Gradients during Pre-training

Original: We present two strategies to mitigate the conﬂicting gradients: a “hard" way to project gradientdirections, and a “soft" way to progressively select sample triplets in curriculum.
Revised: We present two strategies to mitigate the conﬂicting gradients: a “hard" way to project gradient directions, and a “soft" way to progressively select sample triplets in curriculum.

Original: 3.2.1 Cross-Modality Gradient Realignment
Revised: 3.2.1 Cross-Modality Gradient Realignment

Original: Since g va and g vt are misaligned in terms of their directions, one natural idea is to realign the cross-modality gradients, by enforcing or re-projecting their directions to be aligned.

Original: An intuitive choice is an effective remedy developed for multi-task learning named Gradient Surgery [16]. Here, instead of applying it to g i and g j where i, j are the two heterogeneoustasks, we apply it to g va and g vt , which refer to the gradients of the two pairwise losses CMA va and CMA vt w.r.t the weights in the model, following our rationale in Section 3.1.
Revised: Algorithm 1 Cross-Modality Gradient Realignment

Original: Require: Model parameter θ , minibatchs B va , minibatchs
Revised: Require: Model parameter θ , minibatchs B va , minibatchs

Original: B vt for ( b va , b vt ) ∈ ( B va , B vt ) do g va ← ∇ θ CMA va ( θ ) , g vt ← ∇ θ CMA vt ( θ ) g va ← ﬂatten ( g va ) , g vt ← ﬂatten ( g vt )
Revised: B vt for ( b va , b vt ) ∈ ( B va , B vt ) do g va ← ∇ θ CMA va ( θ ) , g vt ← ∇ θ CMA vt ( θ ) g va ← ﬂatten ( g va ) , g vt ← ﬂatten ( g vt )

Original: ˆ g va ← g va , ˆ g vt ← g vt if g va · g vt < 

Revised: ˆ g va ← g va , ˆ g vt ← g vt if g va · g vt <

Original: then ˆ g va ← ˆ g va − ˆ g va · g vt || g vt ||(cid:46) projection g vt ← g va ˆ g vt ← ˆ g vt − ˆ g vt · g va || g va ||(cid:46) projection g va ← g vt end if ˆ g va ← reshape (ˆ g va ) , ˆ g vt ← reshape (ˆ g vt ) ∆ θ ← ˆ g vt + ˆ g va (cid:46) sum up gradients update θ with ∆ θ (cid:46) update parameter end for
Revised: then ˆ g va ← ˆ g va − ˆ g va · g vt || g vt ||(cid:46) projection g vt ← g va ˆ g vt ← ˆ g vt − ˆ g vt · g va || g va ||(cid:46) projection g va ← g vt end if ˆ g va ← reshape (ˆ g va ) , ˆ g vt ← reshape (ˆ g vt ) ∆ θ ← ˆ g vt + ˆ g va (cid:46) sum up gradients update θ with [Table] end for

Original: If the gradients g va and g vt have negative cosine similarity, we alter them byprojecting each onto the normal planeof the other.
Revised: Since g va and g vt are misaligned in terms of their directions, one natural idea is to re-align the crossmodality gradients, by enforcing or re-projecting their directions to be aligned. An intuitive choice is an effective remedy developed for multitask learning named Gradient Surgery [17]. Here, instead of applying it to g i and g j where i, j are the two heterogeneous tasks, we apply it to g va and g vt , which refer to the gradients of the two pairwise losses CMA va and CMA vt w.r.t the weights in the model, following our rationale in Section 3.1.

Revised: If the gradients g va and g vt have negative cosine similarity, we alter them by projecting each onto the normal plane of the other.

Original: Otherwise, if g va and g vthave zero or positive similarity, weretain the original gradients.
Revised: Otherwise, if g va and g vt have zero or positive similarity, we retain the original gradients.

Original: We refer to this particularly adapted formof gradient surgery as Cross-Modality 
Revised: We refer to this particularly adapted form of gradient surgery as Cross-Modality Gradient

Original: Gradient Realignment .
Revised: Realignment .

Original: Its update procedure is outlined in Algorithm 1, andthe overhead very cheap, mainly justcomputing inner products of two ﬂattened gradients.
Revised: Its update procedure is outlined in Algorithm 1, and the overhead very cheap, mainly just computing inner products of two ﬂattened gradients.

Original: Additionally, we also tried anothersimilar algorithm named Gradient Vaccine [29], and the results are almost fully consistent and comparableto using Gradient Surgery [16].
Revised: We also tried another similar algorithm named Gradient Vaccine [30], and the results are almost identical to using Gradient Surgery [17].

Original: Wehence only report one for the sake of conciseness.
Revised: We hence only report one for the sake of conciseness.

Original: 3.2.2 Gradient-based Curriculum Learning
Revised: 3.2.2 Gradient-based Curriculum Learning

Original: We next propose a gradient-based curriculum learning strategy that gradually identiﬁes and removes more misaligned samples based on the similarity indicator as the training goes on.
Revised: We next propose a gradient-based curriculum learning strategy that gradually identiﬁes and removes more misaligned samples based on the similarity indicator as the training goes on.

Original: Formally, given a video-audio pair b va and video-text pair b vt , we only update the parameter ifcos ( g va , g vt ) > γ and drop this sample triplet otherwise.
Revised: Formally, given a video-audio pair b va and video-text pair b vt , we only update the parameter if cos ( g va , g vt ) > γ and drop this sample triplet otherwise.

Original: This ensures us to gradually prioritize training with well-aligned samples while suppressing the noisy supervision from potentiallymisaligned samples.
Revised: This ensures us to gradually prioritize training with well-aligned samples while suppressing the noisy supervision from potentially misaligned samples.

Original: The drop rate here is controlled by a hyperparameter γ .
Revised: The drop rate here is controlled by a cut-off threshold γ .

Original: In general, γ isa negative value monotonically increasing during training, and there is an underlying rationale.
Revised: In general, γ is a negative value monotonically increasing during training, and there is an underlying rationale.

Original: Algorithm 2 Gradient-based Curriculum Learning
Revised: Algorithm 2 Gradient-based Curriculum Learning

Original: Require: Model parameter θ , minibatchs B va , minibatchs
Revised: Require: Model parameter θ , minibatchs B va , minibatchs

Original: B vt , initial γfor ( b va , b vt ) ∈
Revised: B vt , initial γfor ( b va , b vt ) ∈

Original: ( B va , B vt ) do update γ (cid:46) curriculumly update γ g va ← ∇ θ CMA va ( θ ) , g vt ← ∇ θ CMA vt ( θ ) g va ← ﬂatten ( g va ) , g vt ← ﬂatten ( g vt ) if g va · g vt > γ then g va ← reshape ( g va ) , g vt ← reshape ( g vt ) ∆ θ ← g va + g vt (cid:46) sum up gradients update θ with ∆ θ (cid:46) update parameter end if end for
Revised: ( B va , B vt ) do update γ (cid:46) curriculumly update γ g va ← ∇ θ CMA va ( θ ) , g vt ← ∇ θ CMA vt ( θ ) g va ← ﬂatten ( g va ) , g vt ← ﬂatten ( g vt ) if g va · g vt > γ then g va ← reshape ( g va ) , g vt ← reshape ( g vt ) ∆ θ ← g va + g vt (cid:46) sum up gradients update θ with ∆ θ (cid:46) update parameter end if end for

Original: Initially, the network is under-trainedand the similarity between its learnedfeatures is not reliable to indicate astrong CMA; hence we are more conservative in excluding samples.
Revised: Initially, the network is under-trained and the similarity between its learned features is not reliable to indicate a strong CMA; hence we are more conservative in excluding samples.

Original: As thenetwork becomes more sufﬁcientlytrained, the features become more semantically aligned, allowing us to remove more misaligned samples in thelater stage of training.
Revised: As the network becomes more sufﬁciently trained, the features become more semantically aligned, allowing us to remove more misaligned samples in the later stage of training.

Original: The full procedure of Gradient-based Curriculum
Revised: The full procedure of Gradient-based Curriculum

Original: Learning is outlined in Algorithm 2.
Revised: Learning is outlined in Algorithm 2.

Original: We empirically choice γ 0 = − 0 .
Revised: We empirically choice γ 0 = − 0 .

Original: 3 atthe beginning and linearly increase itto γ 500 K = 0 at the end, we includean ablation on the adaptive choice of γ in the Table 2.
Revised: 3 at the beginning and linearly increase it to γ 500 K = 0 at the end, we include an ablation on the adaptive choice of γ in the Table 2.

Original: Similar classes of ideas, i.e, “gradually focusing a small training subset", have been explored by aprominent class of algorithms which rely on sample selection to robustify [33, 34, 35] or to accelerate[36] training.
Revised: Similar classes of ideas, i.e, “gradually focusing a small training subset", have been explored by a prominent class of algorithms which rely on sample selection to robustify [34, 35, 36] or to accelerate [37] training.

Original: Most of them use the “small loss trick" wherein a fraction of samples with smallest lossvalues below a certain threshold are considered as reliable.
Revised: Most of them use the “small loss trick" wherein a fraction of samples with smallest loss values below a certain threshold are considered as reliable.

Original: However, they did not explore gradientconﬂict indicators, which targets solving our CMA problem here.
Revised: However, they did not explore gradient conﬂict indicators, which targets solving our CMA problem here.

Original: Also note that our two steps canfully reuse the cosine similarity computed, so applying them together incurs no extra overhead.
Revised: Also note that our two steps can fully reuse the cosine similarity computed, so applying them together incurs no extra overhead.

Original:  Experiments 
Revised:  Experiments 4.1 Pre-Training Datasets and Settings

Original: 4.1 Pre-Training Datasets and Settings

Original: In this paper, we focus on studying gradients in a modality-agnostic single-backbone setting, thuswe do not consider multimodal pre-training baseline with modality-speciﬁc models, e.g. HERO[37],
Revised: In this paper, we focus on studying gradients in a modality-agnostic single-backbone setting, thus we do not consider multimodal pre-training baseline with modality-speciﬁc models, e.g. HERO[38],

Original: MMV[5], PerceiverIO[38], AVLnet[39].
Revised: MMV[5], PerceiverIO[39], AVLnet[40].

Original: Apart from VATT[11], there exist other concurrent workthat use modality-agnostic setting[19, 40, 41], yet they all have their respective limitations.
Revised: Apart from VATT[11], there exist other concurrent work that use modality-agnostic setting[20, 41, 42], yet they all have their respective limitations.

Original: Forexample, UniVL[19] lacks audio modalities thus cannot constitute video-audio CMA loss, VLM[40]applies separate attention mask for each modality while PolyViT[41] use alternate training betweenmodalities and tasks, thus eliminate the possibilities of any gradients conﬂicts.
Revised: For example, UniVL[20] lacks audio modalities thus cannot constitute video-audio CMA loss, VLM[41] applies separate attention mask for each modality while PolyViT[42] use alternate training between modalities and tasks, thus eliminate the possibilities of any gradients conﬂicts.

Original: Therefore, this leave
Revised: Therefore, this leave

Original: VATT[11] to be the most suitable baseline besides it SoTA performance.
Revised: VATT[11] to be the most suitable baseline besides it SoTA performance.

Original: Our pre-training adopts three settings for comprehensive comparison: (1) using HowTo100M asour training set, following [19, 12]; (2) combining HowTo100M and AudioSet as our training setfollowing [5] and VATT [11]; and (3) additionally merging a noisy
Revised: Our pre-training adopts three settings for comprehensive comparison: (1) using HowTo100M as our training set, following [20, 12]; (2) combining HowTo100M and AudioSet as our training set following [5] and VATT [11]; and (3) additionally merging a noisy

Original: Youtube8M dataset alongsidewith HowTo100M and AudioSet, to create a much more compound and complicated training set thanconsidered before, to stretch-test our proposal’s beneﬁts to scale up pre-training in uncurated andpoorly aligned data.
Revised: Youtube8M dataset alongside with HowTo100M and AudioSet, to create a much more compound and complicated training set than considered before, to stretch-test our proposal’s beneﬁts to scale up pre-training in uncurated and poorly aligned data.

Original: For each setting, we i.i.d sample video-audio-text clips from the mixture of allcandidate sets in every batch.
Revised: For each setting, we i.i.d sample video-audio-text clips from the mixture of all candidate sets in every batch.

Original: (HowTo100M, Audioset or Youtube8M, if applicable).
Revised: (HowTo100M, Audioset or Youtube8M, if applicable).

Original: For all setting,we only use a subset of HowTo100M, AudioSet, Youtube8M, Kinetics400 and Youcook2 dataset incompliance with Youtube’s wipe-out policies.
Revised: For all setting, we only use a subset of HowTo100M, AudioSet, Youtube8M, Kinetics400 and Youcook2 dataset in compliance with Youtube’s wipe-out policies.

Original: HowTo100M 1 [12] is a large-scale dataset of narrated videos with an emphasis on instructional videosthat have well synchronized modalities.
Revised: HowTo100M 1 [12] is a large-scale dataset of narrated videos with an emphasis on instructional videos that have well synchronized modalities.

Original: The training set consists of 127 Millions video clips with thecorresponding close-captioned text from automatic speech recognition (ASR).
Revised: The training set consists of 127 Millions video clips with the corresponding close-captioned text from automatic speech recognition (ASR).

Original: AudioSet 1 [23] is a large-scale audio-visual dataset originally intended for audio event classiﬁcation.
Revised: AudioSet 1 [24] is a large-scale audio-visual dataset originally intended for audio event classiﬁcation.

Original: It consists of 1.78 Million 10-second clips sampled from Youtube Videos that contains a variety ofaudio tracks, such as musical instruments, animals or mechanical sounds.
Revised: It consists of 1.78 Million 10-second clips sampled from Youtube Videos that contains a variety of audio tracks, such as musical instruments, animals or mechanical sounds.

Original: Youtube8M 1 [1] is a large-scale video classiﬁcation dataset based on Youtube videos.
Revised: Youtube8M 1 [1] is a large-scale video classiﬁcation dataset based on Youtube videos.

Original: It consists ofa diverse vocabulary of 3862 visual entities and a total number of 8 Million videos.
Revised: It consists of a diverse vocabulary of 3862 visual entities and a total number of 8 Million videos.

Original: To the best ofour knowledge, we are the ﬁrst to scale contrastive multimodal pre-training beyond instructionalvideos, onto this noisy Youtube8M videos that more faithfully represent videos the wild.
Revised: To the best of our knowledge, we are the ﬁrst to scale contrastive multimodal pre-training beyond instructional videos, onto this noisy Youtube8M videos that more faithfully represent videos the wild.

Original: For data preprocessing, we split each video into 5s clips and use ASR close-captions to generate the correspondingtext, resulting in 165 Millions video-audio-text clips.
Revised: For data preprocessing, we split each video into 5s clips and use ASR close-captions to generate the corresponding text, resulting in 165 Millions video-audio-text clips.

Original: 4.2 Pre-Training Settings
Revised: 4.2 Pre-Training Settings

Original: Network Architecture: For all of our experiments, we use modality-agnostic Transformer variant in[11], VATT-MA-Medium.
Revised: Network Architecture: For all of our experiments, we use modality-agnostic Transformer variant in [11], VATT-MA-Medium.

Original: Speciﬁcally, we use a 12-layer Transformer, in which each Transformermodule has a feed-forward MLP with hidden size of 4096, and 16 attention heads with a hidden sizeof 1024.
Revised: Speciﬁcally, we use a 12-layer Transformer, in which each layer has a feed-forward MLP with hidden size of 4096, and 16 attention heads with a hidden size of 1024.

Original: Pre-training Hyperparameter: We strictly follow the setting in [11], pre-training VATT fromscratch with Adam optimizer with an initial learning rate of 1e-4, 10k warmup steps, 500k steps intotal, a batch size of 2048 and using a cosine learning rate scheduler to anneal the learning rate from1e-4 to 5e-5.
Revised: Pre-training Hyperparameter: We strictly follow the setting in [11], pre-training VATT from scratch with Adam optimizer with an initial learning rate of 1e-4, 10k warmup steps, 500k steps in total, a batch size of 2048 and using a cosine learning rate scheduler to anneal the learning rate from 1e-4 to 5e-5.

Original: Our framework is implemented in Tensorﬂow 2.8, and train with 256 TPUV3s, it took atotal of 3 days to train our models.
Revised: Our framework is implemented in Tensorﬂow 2.8, and train with 256 TPUV3s, it took a total of 3 days to train our models.

Original: 4.3 Downstream Tasks for Evaluation
Revised: 4.3 Downstream Tasks for Evaluation

Original: We evaluate the performance of our pre-trained representations when tuned towards various downstream tasks, including a variety of video/audio and cross modalities tasks.
Revised: We evaluate the performance of our pre-trained representations when tuned towards various downstream tasks, including a variety of video/audio and cross modalities tasks.

Original: Video Action Recognition: Following [5], we evaluate on UCF101[42] (101 action classes, 13,320videos) and the HMDB51[43] (51 classes, 6,766 videos) benchmarks.
Revised: Video Action Recognition: Following [5], we evaluate on UCF101[43] (101 action classes, 13, videos) and the HMDB51[44] (51 classes, 6,766 videos) benchmarks.

Original: For both settings, we directlyattach a linear classiﬁer on the learned representations while freezing the transformer backbone.
Revised: For both settings, we directly attach a linear classiﬁer on the learned representations while freezing the transformer backbone.

Original: We also evaluate on Kinetics400 1 [44] dataset (400 classes, 234,584 video), where we ﬁne-tune thetransformer backbone.
Revised: We also evaluate on Kinetics400 1 [45] dataset (400 classes, 234,584 video), where we ﬁne-tune the transformer backbone.

Original: Audio Event Classiﬁcation: Following [5]
Revised: Audio Event Classiﬁcation: Following [5]

Original: , we used ESC50[45] (50 classes, 2000 audio clips) asthe benchmark for audio modalities.
Revised: , we used ESC50[46] (50 classes, 2000 audio clips) as the benchmark for audio modalities.

Original: Similarly, we directly attach and tune a linear classiﬁer on top ofthe frozen transformer backbone.
Revised: Similarly, we directly attach and tune a linear classiﬁer on top of the frozen transformer backbone.

Original: Zero-shot Video-Text Retrieval: To evaluate our performance on cross-modality task, we follow[5] using YouCook2 1 [15] (3.1K video-text clips) and MSR-VTT 1 [46] (1K video-text clips) as ourevaluation datasets.
Revised: Zero-shot Video-Text Retrieval: To evaluate our performance on cross-modality task, we follow [5] using YouCook2 1 [15] (3.1K video-text clips) and MSR-VTT 1 [47] (1K video-text clips) as our evaluation datasets.

Original: We directly use the embedding from the pre-trained model without any ﬁnetuning, and report the recall at 10 (R@10) and Median Rank as described in [5].[Table]
Revised: We directly use the embedding from the pre-trained model without any ﬁnetuning, and report the recall at 10 (R@10) and Median Rank as described in [5].

Original: 4.4 Result Analysis on HowTo100M and AudioSet Pre-training
Revised: 4.4 Result Analysis on HowTo100M and AudioSet Pre-training

Original: Gradient Re-weighting We set up a simple gradient re-weighting baseline similar to [30], where wesimply scale up the gradient magnitude of g va by 2.5 times and scale down the gradient magnitudeof its counterpart g vt by 0.5 times, we denote this baseline as Gradient Re-weighting (VA), we alsodid it vice versa and denoted it as Gradient Re-weighting (VT).
Revised: Gradient Re-weighting We set up a simple gradient re-weighting baseline similar to [31], where we simply scale up the gradient magnitude of g va by 2.5 times and scale down the gradient magnitude of its counterpart g vt by 0.5 times, we denote this baseline as Gradient Re-weighting (VA), we also did it vice versa and denoted it as Gradient Re-weighting (VT).

Original: As shown in Table 1, we did notobserve much performance boost with gradient re-weighting, on the contrary, there are signiﬁcantperformance drop in Text-Video Retrieval task on both YouCook2 and MSRVTT datasets.
Revised: As shown in Table 1, we did not observe much performance boost with gradient re-weighting, on the contrary, there are signiﬁcant performance drop in Text-Video Retrieval task on both YouCook2 and MSRVTT datasets.

Original: Cross-Modality Gradient Realignment Compared to Gradient Re-weighting, our Cross-Modality
Revised: Cross-Modality Gradient Realignment Compared to Gradient Re-weighting, our Cross-Modality

Original: Gradient Realignment mitigates the gradient conﬂicts by modifying gradients direction.
Revised: Gradient Realignment mitigates the gradient conﬂicts by modifying gradients direction.

Original: In Table 1,we can see signiﬁcant performance gains in video-text retrieval task (e.g. 58% gain in rank metric on
Revised: In Table 1, we can see signiﬁcant performance gains in video-text retrieval task (e.g. 58% gain in rank metric on

Original: YouCook2, 46% gain in R@10 metric on MSR-VTT), while maintaining a comparable performancein video/audio tasks.
Revised: YouCook2, 46% gain in R@10 metric on MSR-VTT), while maintaining a comparable performance in video/audio tasks.

Original: Since video-text retrieval task heavily hinge on cross-modality alignment(CMA), this improvement verify the beneﬁts of re-aligning the cross-modality gradients. 
Revised: Since video-text retrieval task heavily hinge on cross-modality alignment (CMA), this improvement verify the beneﬁts of re-aligning the cross-modality gradients. [Table] [Table]

Original: Gradient-based Curriculum Learning Compared to Cross-Modality Gradient Realignment,
Revised: Gradient- Curriculum Learning Compared to Cross-Modality Gradient Realignment,

Original: Gradient-based Curriculum Learning inherents its performance gains in video-text retrieval task, yetenjoy an additional performance boost in audio classiﬁcation (e.g. 1.40% gain ESC50 dataset in
Revised: Gradient-based Curriculum Learning inherents its performance gains in video-text retrieval task, yet enjoy an additional performance boost in audio classiﬁcation (e.g. 1.40% gain ESC50 dataset in

Original: Table 1) and video classiﬁcation (e.g. 1.94% gains UCF101 dataset in Table 1).
Revised: Table 1) and video classiﬁcation (e.g. 1.94% gains UCF101 dataset in Table 1).

Original: We also perform anablation on the adaptive choice of γ on Howto100M dataset in Table 2.
Revised: We also perform an ablation on the adaptive choice of γ on Howto100M dataset in Table 2.

Original: We can see that curriculumlyevolving γ (i.e. ( γ 0 , γ 500 K ) =
Revised: We can see that curriculumly evolving γ (i.e. ( γ 0 , γ 500 K ) =

Original: ( − 0 .
Revised: ( − 0 .

Original: 3 , 0 . 0) ) yields the best overall results, while maintaining a ﬁxed γ(i.e. ( γ 0 , γ 500 K ) =
Revised: 3 , 0 . 0) ) yields the best overall results, while maintaining a ﬁxed γ (i.e. ( γ 0 , γ 500 K ) =

Original: (0 . 0 , 0 . 0) ) yields the worst results, this verify the effectiveness of curriculumlyremoving mis-aligned samples during training.
Revised: (0 . 0 , 0 . 0) ) yields the worst results, this verify the effectiveness of curriculumly removing mis-aligned samples during training.

Original: Cross-Modality Gradient Realignment + Gradient-based Curriculum Learning
Revised: Cross-Modality Gradient Realignment + Gradient-based Curriculum Learning

Original: We further verify the effectiveness with the combination of both techniques.
Revised: We further verify the effectiveness with the combination of both techniques.

Original: Speciﬁcally, given a video-audio pair and video-text pair, (a) if cos ( g va , g vt ) ≤ γ , we drop this sample triplet; (b) if γ < cos ( g va , g vt ) < 0 ,we apply gradient projection in Algorithm 1; (c) if cos ( g va , g vt ) ≥ 0 , we retain the original gradients.
Revised: Speciﬁcally, given a video-audio pair and video-text pair, (a) if cos ( g va , g vt ) ≤ γ , we drop this sample triplet; (b) if γ < cos ( g va , g vt ) < 0 , we apply gradient projection in Algorithm 1; (c) if cos ( g va , g vt ) ≥ 0 , we retain the original gradients.

Original: We found they are complimentary to each other and the combination of both lead to betterperformance in the majority of downstream tasks.
Revised: We found they are complimentary to each other and the combination of both lead to better performance in the majority of downstream tasks.

Original: [Table] decay
Revised: 4.5 Scaling Pre-training to YouTube8M

Original: As shown in Table 1, adding YouTube8M dataset brings the video classiﬁcation and video-textretrieval performance to a new level (e.g. 88.28% Top1 on UCF101, 56.0 rank on MSRVTT)), yet itset back the performance in audio classiﬁcation task, we conjecture this is largely due to the noisymisaligned samples introduced by Youtube8M dataset.
Revised: As shown in Table 1, adding YouTube8M dataset brings the video classiﬁcation and video-text retrieval performance to a new level (e.g. 88.28% Top1 on UCF101, 56.0 rank on MSRVTT)), yet it set back the performance in audio classiﬁcation task, we conjecture this is largely due to the noisy misaligned samples introduced by Youtube8M dataset.

Original: However, by leveraging our proposed techniques, we further yields a new state-of-the-art performancein the modality-agnostic setting, without any modiﬁcation in architecture, striking 89.70% Top1 on
Revised: However, by leveraging our proposed techniques, we further yields a new state-of-the-art performance in the modality-agnostic setting, without any modiﬁcation in architecture, striking 89.70% Top1 on

Original: UCF101, 92.15% Top5 on HMDB51, 80.01% Top1 on Kinetics400 and 85.00% Top1 on ESC50dataset, demonstrating the effectiveness of our method when scaling to noisy Youtub

Revised: UCF101, 92.15% Top5 on HMDB51, 80.01% Top1 on Kinetics400 and 85.00% Top1 on ESC dataset, demonstrating the effectiveness of our method when scaling to noisy Youtub

Original: M dataset.
Revised: M dataset.

Revised: [Figure]

Original: 4.6 Feature Visualization
Revised: 4.6 Feature Visualization

Original: To better illustrate why gradient harmonization techniques works , in Figure 4, we follow the visualization used in VATT[11], which take YouCook2 Video-Text Retrieval dataset, showing theoutput space visualization of video and text, respectively.
Revised: To better illustrate why gradient harmonization techniques works , in Figure 5, we follow the visualization used in VATT[11], which take YouCook2 Video-Text Retrieval dataset, showing the output space visualization of video and text, respectively.

Original: We can see the initial video and textfeature representation of VATT baseline are mixed together (which aligned with the ﬁndings in [11]),however, after using gradient realignment, curriculum learning or applying both, the video and textrepresentations become disentangled, this indicates our gradient harmonization techniques adds tocross-modality diversity, which makes video/text more discriminative from each other, making themodel more discriminative and better in generalization. 
Revised: We can see the initial video and text feature representation of VATT baseline are mixed together (which aligned with the ﬁndings in [11]), however, after using gradient realignment, curriculum learning or applying both, the video and text representations become disentangled, this indicates our gradient harmonization techniques adds to cross-modality diversity, which makes video/text more discriminative from each other, making the model more discriminative and better in generalization. 4.7 Extending Gradient Realignment to Modality-Speciﬁc VATT So far, our techniques were built upon the modality-agnostic setting of VATT with a shared backbone. We also try to apply the same technique to the modality-speciﬁc setting, given there is still a shared video head where we can harmonization the gradients.

Original: [Figure]
Revised: In Table 3, we observe that applying Gradient Realignment (GR) to modality-speciﬁc setting has some gain over uni-modality tasks (e.g. Video/Audio Classﬁcation), yet set back on cross-modality tasks (e.g. Text-Video Retrieval). We hypothesize this is due the lack of cross-modality signals: since there is no longer shared backbone across modalities, the only shared video head would provide bias signala towards the video modality, for cross-modality alignment.

Revised: We leave the improvement of modality-speciﬁc VATT for future work.

Revised: [Figure]

Revised: [Table]

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we take a deep dive into the common CMA assumption used in multimodal pretraining.
Revised: In this paper, we take a deep dive into the common CMA assumption used in multimodal pretraining.

Original: Using the gradient directions between pairwise losses as the surrogate indicator, we observeubiquitous gradient conﬂicts even in relatively well-aligned narrative datasets.
Revised: Using the gradient directions between pairwise losses as the surrogate indicator, we observe ubiquitous gradient conﬂicts even in relatively well-aligned narrative datasets.

Original: We then proposetwo plug-and-play techniques to harmonize the gradients during training.
Revised: We then propose two plug-and-play techniques to harmonize the gradients during training.

Original: They are demonstrated tosubstantially enhance VATT pre-training across a number of downstream tasks, and further scale it upto pre-training with even more complicated and poorly aligned data.
Revised: They are demonstrated to substantially enhance VATT pre-training across a number of downstream tasks, and further scale it up to pre-training with even more complicated and poorly aligned data.

Original: Our ﬁndings underline the (oftenoverlooked) importance to carefully revisit the basic assumption and training dynamics of multimodallearning, besides advancing the model architectures.
Revised: Our ﬁndings underline the (often overlooked) importance to carefully revisit the basic assumption and training dynamics of multimodal learning, besides advancing the model architectures.

Revised: However, our cross-modality gradient harmonization technique still has room to generalize better to the modality-speciﬁc setting.

Original: However, our work did not fully explore otherregularization techniques in multi-tasking learning literature, such as [26, 27, 28].
Revised: Additionally, we did not fully explore other regularization techniques in multi-tasking learning literature, such as [27, 28, 29].

Original: The potentialnegative societal impacts include the malicious use of multimodal models, such as unintended use ofits downstream visual/audio recognition capability.Checklist [Table]
Revised: The potential negative societal impacts include the malicious use of multimodal models, such as unintended use of its downstream visual/audio recognition capability.

Original: For all authors...

Original: [Table]

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s

Original: [Table] contributions and scope? [Yes] [Table]
Revised: Acknowledgments and Disclosure of Funding

Original: (b) Did you describe the limitations of your work?

Original: [Yes] See Section 5, e.g. our work did

Original: not fully explore other regularization techniques in multi-tasking learning literature,such as [26, 27, 28]

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] SeeSection 5, e.g. the malicious use of multimodal models.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?

Original: [Yes]

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main exper-

Original: imental results (either in the supplemental material or as a URL)?

Original: [No] The code isproprietary, However, we include detailed discussions on experiment settings and datain Section 4.1 to replicate the experiment results.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they

Original: were chosen)?

Original: [Yes] See Section 4.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [No] Since pre-training are very computationally expensive,and it is infeasible do to multiple runs.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Section 4.2:

Original: Pre-training

Original: Hyperparameter

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Section 4. (b) Did you mention the license of the assets? [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: We give special thanks to Tianqi Liu and Jialu Liu for the discussion.

Original: [No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating? [N/A]
Revised: Z. Wang is supported in part by

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: information or offensive content?

Original: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?

Original: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
Revised: US Army Research Ofﬁce Young Investigator Award W911NF2010240.