Original: Enhancing and Scaling Cross-Modality Alignment for Multimodal Pre-Training via Gradient Harmonization
Revised: Enhancing and Scaling Cross-Modality Alignment for Multimodal Pre-Training via Gradient Harmonization

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress
Revised: AfﬁliationAddress

Original: Abstract
Revised: Abstract

Original: Self-supervised pre-training recently demonstrates success on large-scale multimodal data, and state-of-the-art contrastive methods often enforce the featureconsistency from cross-modality inputs, such as video/audio or video/text pairs.
Revised: Self-supervised pre-training recently demonstrates success on large-scale multimodal data, and state-of-the-art contrastive methods often enforce the featureconsistency from cross-modality inputs, such as video/audio or video/text pairs.

Original: Despite its convenience to formulate and leverage in practice, such cross-modalityalignment (CMA) is only a weak and noisy supervision, since two modalities can besemantically misaligned even they are temporally aligned.
Revised: Despite its convenience to formulate and leverage in practice, such cross-modalityalignment (CMA) is only a weak and noisy supervision, since two modalities can besemantically misaligned even they are temporally aligned.

Original: For example, even in the(often adopted) instructional videos, a speaker can sometimes refer to somethingthat is not visually present in the current frame; and the semantic misalignmentwould only be more unpredictable for the raw videos collected from unconstrainedinternet sources.
Revised: For example, even in the(often adopted) instructional videos, a speaker can sometimes refer to somethingthat is not visually present in the current frame; and the semantic misalignmentwould only be more unpredictable for the raw videos collected from unconstrainedinternet sources.

Original: We conjecture that might cause conﬂicts and biases among modalities, and may hence prohibit CMA from scaling up to training with larger andmore heterogeneous data.
Revised: We conjecture that might cause conﬂicts and biases among modalities, and may hence prohibit CMA from scaling up to training with larger andmore heterogeneous data.

Original: This paper ﬁrst veriﬁes our conjecture by observing that,even in the latest VATT pre-training using only narrated videos, there exist stronggradient conﬂicts between different CMA losses within the same sample triplet(video, audio, text), indicating them as the noisy source of supervision.
Revised: This paper ﬁrst veriﬁes our conjecture by observing that,even in the latest VATT pre-training using only narrated videos, there exist stronggradient conﬂicts between different CMA losses within the same sample triplet(video, audio, text), indicating them as the noisy source of supervision.

Original: We thenpropose to harmonize such gradients during pre-training, via two techniques: (i)cross-modality gradient realignment: modifying different CMA loss gradients forone sample triplet, so that their gradient directions are in more agreement; and (ii)gradient-based curriculum learning: leveraging the gradient conﬂict informationon an indicator of sample noisiness, to develop a curriculum learning strategyto prioritize training with less noisy sample triplets.
Revised: We thenpropose to harmonize such gradients during pre-training, via two techniques: (i)cross-modality gradient realignment: modifying different CMA loss gradients forone sample triplet, so that their gradient directions are in more agreement; and (ii)gradient-based curriculum learning: leveraging the gradient conﬂict informationon an indicator of sample noisiness, to develop a curriculum learning strategyto prioritize training with less noisy sample triplets.

Original: Applying those gradientharmonization techniques to pre-training VATT on the HowTo100M dataset, weconsistently improve its performance on different downstream tasks.
Revised: Applying those gradientharmonization techniques to pre-training VATT on the HowTo100M dataset, weconsistently improve its performance on different downstream tasks.

Original: Moreover, weare able to scale VATT pre-training to more complicated non-narrative Youtube8Mdataset to further improve the state-of- the-arts.
Revised: Moreover, weare able to scale VATT pre-training to more complicated non-narrative Youtube8Mdataset to further improve the state-of- the-arts.

Original: We will open-source our codes andmodels.
Revised: We will open-source our codes andmodels.

Original:  Introduction
Revised:  Introduction

Original: Self-supervised pre-training scales up deep learning to leverage massive unlabeled data.
Revised: Self-supervised pre-training scales up deep learning to leverage massive unlabeled data.

Original: Beyondthe maturity of pre-training over single-modality data such as language [2] or images [3], therecent success on multimodal pre-training [4, 5] demonstrates the versatility to synergize the richmultimodal information and to beneﬁt a variety of downstream tasks.
Revised: Beyondthe maturity of pre-training over single-modality data such as language [2] or images [3], therecent success on multimodal pre-training [4, 5] demonstrates the versatility to synergize the richmultimodal information and to beneﬁt a variety of downstream tasks.

Original: Many methods of this category[6, 7, 8, 4], especially the latest contrastive pre-training methods [5, 9, 10, 11], consider the mostorganic supervision as the cross-modality alignment ( CMA ), e.g., in the same video sequence, thevideo frames, audio stream, and text scripts are temporally aligned and hence should naturallyhave correspondence.
Revised: Many methods of this category[6, 7, 8, 4], especially the latest contrastive pre-training methods [5, 9, 10, 11], consider the mostorganic supervision as the cross-modality alignment ( CMA ), e.g., in the same video sequence, thevideo frames, audio stream, and text scripts are temporally aligned and hence should naturallyhave correspondence.

Original: Such assumed cross-modal correspondence could be exploited as “mutualsupervision" for each modality to learn consistent semantic features with each other.
Revised: Such assumed cross-modal correspondence could be exploited as “mutualsupervision" for each modality to learn consistent semantic features with each other.

Original: For example,
Revised: For example,

Original: Aligned “On
Revised: [Figure]

Original:  “World Cup Semi Final” “Raising fund for UNICEF” “Whetherit's in a kitchen”
Revised: “On the football “World Cup Semi Final” “Raising [Figure] for UNICEF” “Whether

Original: [Figure]
Revised: [Figure]

Original:  the latest Video-Audio-Text Transformer (VATT) [11] hinges on this multimodal CMA a priori totrain a transformer-based architecture (even a single backbone by sharing weights among the threemodalities), using two contrastive losses to align video-text and video-audio pairs, respectively.
Revised: in a kitchen” [Figure] the latest Video-Audio-Text Transformer (VATT) [11] hinges on this multimodal CMA a priori totrain a transformer-based architecture (even a single backbone by sharing weights among the threemodalities), using two contrastive losses to align video-text and video-audio pairs, respectively.

Original: Theirresults set many new records in downstream tasks with neither supervised pre-training nor predeﬁnedinductive biases, showing strong “universal" generalizability despite the modality and the domaingaps.
Revised: Theirresults set many new records in downstream tasks with neither supervised pre-training nor predeﬁnedinductive biases, showing strong “universal" generalizability despite the modality and the domaingaps.

Original: In this paper, we focus on studying gradients in a modality-agnostic single-backbone setting of(VATT) [11].

Original: Unfortunately, the CMA assumption rests on a very shaky foundation, especially for uncurated videoscollected in the wild.
Revised: Unfortunately, the CMA assumption rests on a very shaky foundation, especially for uncurated videoscollected in the wild.

Original: The CMA can be weak even for the instructional videos such as HowTo100M[12] that are commonly adopted as the current multimodal pre-training data source.
Revised: The CMA can be weak even for the instructional videos such as HowTo100M[12] that are commonly adopted as the current multimodal pre-training data source.

Original: For instance,a speaker can refer to something before or after actually demonstrating it visually; she or he couldalso skip verbally explaining something that is visually happening, because that may be trivial or bealready clear enough in the visual context [13].
Revised: For instance,a speaker can refer to something before or after actually demonstrating it visually; she or he couldalso skip verbally explaining something that is visually happening, because that may be trivial or bealready clear enough in the visual context [13].

Original: Such discrepancy would be ampliﬁed if consideringmany other outliers in video (e.g. background objects) or language (e.g. off-topic small talks).
Revised: Such discrepancy would be ampliﬁed if consideringmany other outliers in video (e.g. background objects) or language (e.g. off-topic small talks).

Original: Thesemantic misalignment will be even more severe, and in unpredictable forms, when we go beyondinstructional videos and explore training with non-narrative, free-from internet videos.
Revised: Thesemantic misalignment will be even more severe, and in unpredictable forms, when we go beyondinstructional videos and explore training with non-narrative, free-from internet videos.

Original: Figure 1shows a few examples of such cross-modal misalignment in the non-narrative
Revised: Figure ?? shows a few examples of such cross-modal misalignment in the non-narrative

Original: [-Youtube
 M-] dataset [1],which would put CMA-guided pre-training in jeopardy.
Revised: [-Youtube
 M-]Youtube8M dataset [1],which would put CMA-guided pre-training in jeopardy.

Original: This important problem is yet under-studiedin the self-supervised multimodal pre-training literature.
Revised: This important problem is yet under-studiedin the self-supervised multimodal pre-training literature.

Original: That said, there has been a few recent effortssuch as Multiple-Instance Learning (MIL) NCE [13] and noisy pair pruning as in [14].
Revised: That said, there has been a few recent effortssuch as Multiple-Instance Learning (MIL) NCE [13] and noisy pair pruning as in [14].

Original: We conjecture that the vanilla CMA might provide only weak, noisy and even misleading supervisionfor multimodal pre-training; hence it might constitute a hidden bottleneck when scaled up with morerealistic data (beyond instructional videos) whose modalities are poorly aligned.
Revised: We conjecture that the vanilla CMA might provide only weak, noisy and even misleading supervisionfor multimodal pre-training; hence it might constitute a hidden bottleneck when scaled up with morerealistic data (beyond instructional videos) whose modalities are poorly aligned.

Original: We verify thisconjecture by inspecting two large-scale multimodal datasets, HowTo100M [15] and Youtube8M[1].
Revised: We verify thisconjecture by inspecting two large-scale multimodal datasets, HowTo100M [15] and Youtube8M[1].

Original: Speciﬁcally, when we adopt pairwise CMA losses (e.g. video-audio, video-text) within thecross-modal sample triplets (video, audio, text), a majority of the gradients strongly conﬂict with eachother in terms of their directions, indicating the noisy and misaligned supervision.
Revised: Speciﬁcally, when we adopt pairwise CMA losses (e.g. video-audio, video-text) within thecross-modal sample triplets (video, audio, text), a majority of the gradients strongly conﬂict with eachother in terms of their directions, indicating the noisy and misaligned supervision.

Original: The cross-modalconﬂict not only causes unstable training convergence, it also causes modality-speciﬁc overﬁtting;e.g.
Revised: The cross-modalconﬂict not only causes unstable training convergence, it also causes modality-speciﬁc overﬁtting;e.g.

Original: semantically strong representations for one modality and meaningless representations for anothermodality.
Revised: semantically strong representations for one modality and meaningless representations for anothermodality.

Original: Having identiﬁed this gap, we propose to harmonize those gradients so that they become mutuallycompatible in learning the uniﬁed representations.
Revised: Having identiﬁed this gap, we propose to harmonize those gradients so that they become mutuallycompatible in learning the uniﬁed representations.

Original: Speciﬁcally, we introduce two techniques: (i)cross-modal gradient realignment, where we modify different pairwise CMA loss gradients to aligntheir directions for the same sample triplet, using gradient surgery [16] developed for multi-task learning; and (ii) gradient-based curriculum learning, where we leverage the gradient conﬂict informationto indicate sample noisiness (e.g. a triplet whose CMA gradients are in more agreement is consideredless “noisy" and would be prioritized for cross-modal pre-training) and a curriculum learning strategyis developed accordingly.
Revised: Speciﬁcally, we introduce two techniques: (i)cross-modal gradient realignment, where we modify different pairwise CMA loss gradients to aligntheir directions for the same sample triplet, using gradient surgery [16] developed for multi-task learning; and (ii) gradient-based curriculum learning, where we leverage the gradient conﬂict informationto indicate sample noisiness (e.g. a triplet whose CMA gradients are in more agreement is consideredless “noisy" and would be prioritized for cross-modal pre-training) and a curriculum learning strategyis developed accordingly.

Original: Both techniques are found to boost VATT performance on downstreamtasks, not only on instructional videos, but even more when the more complicatednon-narrative datais involved.
Revised: Both techniques are found to boost VATT performance on downstreamtasks, not only on instructional videos, but even more when the more complicated

Original: Our contributions can be summarized as the following:
Revised: Our contributions can be summarized as the following:

Original: • We suggest that the commonly assumed CMA might be an important yet overlooked perfor- mance hurdle for scaling up multimodal pre-training, and we observe severe misalignmenteven from the state-of-the-art (SOTA) model [11] on the relatively aligned data [12].• We propose to consistently improve the pre-training pipeline in the only baseline in thisdirection, VATT (modality-agnostic setting), resulting in better downstream performance.•
Revised: • We suggest that the commonly assumed CMA might be an important yet overlooked performance hurdle for scaling up multimodal pre-training, and we observe severe misalignmenteven from the state-of-the-art (SOTA) model [11] on the relatively aligned data [12].• We propose to consistently improve the pre-training pipeline in the only baseline in thisdirection, VATT (modality-agnostic setting), resulting in better downstream performance.•

Original: With the help of our proposed techniques, we consistently improve the pre-training ofVATT, resulting in better downstream performance, e
Revised: With the help of our proposed techniques, we consistently improve the pre-training ofVATT, resulting in better downstream performance, e

Original: g. up to 58% gain in the rank metricof video-text retrieval task.
Revised: g. up to 58% gain in the rank metricof video-text retrieval task.

Original: Moreover, we are able to scale VATT pre-training to more complicated non-narrative Youtube8M dataset, which further yields new state-of-the-artperformance in a modality-agnostic setting.
Revised: Moreover, we are able to scale VATT pre-training to morecomplicated non-narrative Youtube8M dataset, which further yields new state-of-the-artperformance in a modality-agnostic setting.

Original:  Related Work2.1 Self-Supervised Multimodal Pre-training
Revised:  Related Work2.1 Self-Supervised Multimodal Pre-training

Original: The most organic self-supervision can arguably be found in the multimodal data that are abundantlyavailable in the digital world: their temporal alignment naturally lead to cross-modality mutualsupervision, hence meaningful features can be learned without requiring human annotation.
Revised: The most organic self-supervision can arguably be found in the multimodal data that are abundantlyavailable in the digital world: their temporal alignment naturally lead to cross-modality mutualsupervision, hence meaningful features can be learned without requiring human annotation.

Original: Multimodal pre-training can thus leverage richer semantic cues than single-modality counterparts, andoutperform in various downstream tasks such as image classiﬁcation [5, 11], video action recognition[5, 11, 13, 10], audio event classiﬁcation [5, 11, 10], and video-text retrieval [5, 11, 13].
Revised: Multimodal pre-training can thus leverage richer semantic cues than single-modality counterparts, andoutperform in various downstream tasks such as image classiﬁcation [5, 11], video action recognition[5, 11, 13, 10], audio event classiﬁcation [5, 11, 10], and video-text retrieval [5, 11, 13].

Original: Mostof those methods formulate pre-text tasks, including cross-modality correspondence [6, 7, 8, 10],cross-modality clustering [17], cross-modality longer context prediction [18], or a combination ofmultiple pre-text tasks [19].
Revised: Mostof those methods formulate pre-text tasks, including cross-modality correspondence [6, 7, 8, 10],cross-modality clustering [17], cross-modality longer context prediction [18], or a combination ofmultiple pre-text tasks [19].

Original: Contrastive learning [3, 20] has recently emerged as the dominant approach in self-supervisedlearning.
Revised: Contrastive learning [3, 20] has recently emerged as the dominant approach in self-supervisedlearning.

Original: In multimodal pre-training, the temporal co-occurrence naturally supplies the positivesample to contrast with.
Revised: In multimodal pre-training, the temporal co-occurrence naturally supplies the positivesample to contrast with.

Original: [5] adopts a multimodal versatile network to embed each modality intothe same vector space, that is trained from end to end with multimodal pairwise contrastive losses(video-audio, and video text).
Revised: [5] adopts a multimodal versatile network to embed each modality intothe same vector space, that is trained from end to end with multimodal pairwise contrastive losses(video-audio, and video text).

Original: CrossCLR [9] further takes the intra-modality similarities into account.[10] generalizes the instance discrimination idea to design a cross-modal discrimination task, i.e.,predicting which audio matches a video.
Revised: CrossCLR [9] further takes the intra-modality similarities into account.[10] generalizes the instance discrimination idea to design a cross-modal discrimination task, i.e.,predicting which audio matches a video.

Original: Most recently, VATT [11] combines the strength ofconvolution-free Transformer and multimodal contrastive learning.
Revised: Most recently, VATT [11] combines the strength ofconvolution-free Transformer and multimodal contrastive learning.

Original: Speciﬁcally, it ﬁrst validate theidea of using a modality-agnostic single-backbone transformer, to work on video, audio and textmodalities, it follows the exact BERT [21] and ViT [22] architecture, except injecting modalityspeciﬁc tokenization layers and linear projections.
Revised: Speciﬁcally, it ﬁrst validate theidea of using a modality-agnostic single-backbone transformer, to work on video, audio and textmodalities, it follows the exact BERT [21] and ViT [22] architecture, except injecting modalityspeciﬁc tokenization layers and linear projections.

Original: Despite their success, existing multimodal contrastive learning methods [5, 9, 11] hinge on the“free" cross-modalities correspondence mined through the multimodal contrastive loss, i.e., the CMAassumption.
Revised: Despite their success, existing multimodal contrastive learning methods [5, 9, 11] hinge on the“free" cross-modalities correspondence mined through the multimodal contrastive loss, i.e., the CMAassumption.

Original: As a result, they are often trained with relatively well curated or narrated video datasets,such as AudioSet [23], YouCook2 [15], and HowTo100M [12].
Revised: As a result, they are often trained with relatively well curated or narrated video datasets,such as AudioSet [23], YouCook2 [15], and HowTo100M [12].

Original: It remains questionable whether such
Revised: It remains questionable whether such

Original: CMA-guided contrastive learning can generalize well to larger-scale multimodal data in the wild.
Revised: CMA-guided contrastive learning can generalize well to larger-scale multimodal data in the wild.

Original: Infact, even in the HowTo100M dataset, the authors of [12] estimate that around 50% of clip-narrationpairs are not well aligned.
Revised: Infact, even in the HowTo100M dataset, the authors of [12] estimate that around 50% of clip-narrationpairs are not well aligned.

Original: The validity of CMA recently arises some attention.
Revised: The validity of CMA recently arises some attention.

Original: Most related to us is [13] which explicitlymodels the misalignment noise using the MIL-NCE loss, that is inherited by VATT [11].
Revised: Most related to us is [13] which explicitlymodels the misalignment noise using the MIL-NCE loss, that is inherited by VATT [11].

Original: 2.2 Multi-task versus Multimodal Learning
Revised: 2.2 Multi-task versus Multimodal Learning

Original: While earlier self-supervised works used different modality-speciﬁc encoders, the latest multimodalpre-training sees a tendency to explore a versatile and “universal" model shared across modalities[5, 11].
Revised: While earlier self-supervised works used different modality-speciﬁc encoders, the latest multimodalpre-training sees a tendency to explore a versatile and “universal" model shared across modalities[5, 11].

Original: It is easy to notice the resemblance between multimodal learning (especially with a uniﬁedbackbone) and multi-task learning [24, 25].
Revised: It is easy to notice the resemblance between multimodal learning (especially with a uniﬁedbackbone) and multi-task learning [24, 25].

Original: The later often assumes multiple tasks to share transferable knowledge and can help each other learn.
Revised: The later often assumes multiple tasks to share transferable knowledge and can help each other learn.

Original: Yet in practice, different tasks can be heterogeneous innature too, often leading to heavily misaligned gradient scales [26] or directions [16].
Revised: Yet in practice, different tasks can be heterogeneous innature too, often leading to heavily misaligned gradient scales [26] or directions [16].

Original: Such conﬂictinggradients from multiple losses would lead to several undesirable effects, including (i) strong biastowards learning some tasks with largest gradient scales; and (2) slow and unstable convergence.
Revised: Such conﬂictinggradients from multiple losses would lead to several undesirable effects, including (i) strong biastowards learning some tasks with largest gradient scales; and (2) slow and unstable convergence.

Original: Similar problems are often faced by multimodal pre-training too.
Revised: Similar problems are often faced by multimodal pre-training too.

Original: As a result, several works have been developed to mitigate the gradient dilemma in multi-task learning.
Revised: As a result, several works have been developed to mitigate the gradient dilemma in multi-task learning.

Original: Methods such as gradient normalization [26] and adaptive loss weight [27, 28] address the gradient
Revised: Methods such as gradient normalization [26] and adaptive loss weight [27, 28] address the gradient

Original: [Figure]
Revised: [Figure]

Original: scale/learning speed imbalance issues.
Revised: scale/learning speed imbalance issues.

Original: Gradient surgery [16] tries to project gradients with conﬂictingdirections onto the normal plane of each other, to prevent each gradient’s interfering component frombeing applied to the network.
Revised: Gradient surgery [16] tries to project gradients with conﬂictingdirections onto the normal plane of each other, to prevent each gradient’s interfering component frombeing applied to the network.

Original: Later, Gradient vaccine [29] further generalizes this idea extending thegradient projection to arbitrary angle, and uses the exponential moving average of cosine similarityto measure the alignment between different tasks.
Revised: Later, Gradient vaccine [29] further generalizes this idea extending thegradient projection to arbitrary angle, and uses the exponential moving average of cosine similarityto measure the alignment between different tasks.

Original: When it comes to the multimodal learning, [30]investigates optimally blending modality-speciﬁc gradients by linearly re-scaling them.
Revised: When it comes to the multimodal learning, [30]investigates optimally blending modality-speciﬁc gradients by linearly re-scaling them.

Original: Meanwhile,to our best knowledge, there has been no prior study on modality-speciﬁc gradient directions.
Revised: Meanwhile,to our best knowledge, there has been no prior study on modality-speciﬁc gradient directions.

Original:  Methodology
Revised:  Methodology

Original: Pre-requisites: Without loss of generality, we focus on the SOTA multimodal contrastive learningmodel, VATT [11], as our default subject of study; and we follow its modality-agnostic singlebackbone setting due to its compelling performance-efﬁciency trade-off, and due to the fact that itis the most intuitive setting to modify gradients (shared between paired modalities) w.r.t differentend-point objectives.
Revised: Pre-requisites: Without loss of generality, we focus on the SOTA multimodal contrastive learningmodel, VATT [11], as our default subject of study; and we follow its modality-agnostic singlebackbone setting due to its compelling performance-efﬁciency trade-off, and due to the fact that itis the most intuitive setting to modify gradients (shared between paired modalities) w.r.t differentend-point objectives.

Original: In our modality-agnostic single-backbone setting of VATT, the goal of pre-training is to ﬁnd theparameter θ of the model f
Revised: In our modality-agnostic single-backbone setting of VATT, the goal of pre-training is to ﬁnd theparameter θ of the model f

Original: θ to learn meaningful features for all target modalities (video, audio, text),usually measured by a set of pairwise similarity metrics.
Revised: θ to learn meaningful features for all target modalities (video, audio, text),usually measured by a set of pairwise similarity metrics.

Original: VATT uses two pairwise contrastive lossesfor video-audio and video-text respectively to solve cross-modal alignment:
Revised: VATT uses two pairwise contrastive lossesfor video-audio and video-text respectively to solve cross-modal alignment:

Original: [Equation]
Revised: [Equation]

Original: where CMA va and CMA vt penalize the cross-modal alignment for video-audio and video-text pairs,using the Noise-Contrastive Estimation objective [31], respectively.
Revised: where CMA va and CMA vt penalize the cross-modal alignment for video-audio and video-text pairs,using the Noise-Contrastive Estimation objective [31], respectively.

Original: Figure 2 illustrates the VATT pipeline.
Revised: Figure 2 illustrates the VATT pipeline.

Original: First, video-audio-text triplets are sampled from randomtemporal locations, and then video-text and video-audio pairs are formed accordingly.
Revised: First, video-audio-text triplets are sampled from randomtemporal locations, and then video-text and video-audio pairs are formed accordingly.

Original: Positive pairsare formed by selecting the two modalities at the same temporal locations from the same video clip;while the negative pairs are obtained by randomly sampling any video, audio, or text from other videoclips.
Revised: Positive pairsare formed by selecting the two modalities at the same temporal locations from the same video clip;while the negative pairs are obtained by randomly sampling any video, audio, or text from other videoclips.

Original: We follow the convention in [5, 11] to use the vanilla NCE loss for video-audio pairs, and touse the MIL-NCE loss proposed in [13] to align video-text pairs.
Revised: We follow the convention in [5, 11] to use the vanilla NCE loss for video-audio pairs, and touse the MIL-NCE loss proposed in [13] to align video-text pairs.

Original: Hence, CMA va and CMA vt aredeﬁned as follows:
Revised: Hence, CMA va and CMA vt aredeﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where N is the pool of negative pairs, P k ( z t ) denotes the k -neighbouring narrations surrounding theoriginal text z t .
Revised: where N is the pool of negative pairs, P k ( z t ) denotes the k -neighbouring narrations surrounding theoriginal text z t .

Original: The gradients of two pairwise losses CMA va and CMA vt , denoted as g va and g vt ,will be together applied (i.e. simple adding) to updating the model f θ .
Revised: The gradients of two pairwise losses CMA va and CMA vt , denoted as g va and g vt ,will be together applied (i.e. simple adding) to updating the model f θ .

Original: 3.1 Observation of Conﬂicting Gradients
Revised: 3.1 Observation of Conﬂicting Gradients

Original: 162 While qualitatively recognizing the cross-modal misalignment is easy, it is a highly challengingendeavor to quantify the misalignment and to automatically ﬁnd misaligned triplets at scale, due tothe vaguely deﬁned criterion as well as the absence of labels [13, 14].
Revised: While qualitatively recognizing the cross-modal misalignment is easy, it is a highly challengingendeavor to quantify the misalignment and to automatically ﬁnd misaligned triplets at scale, due tothe vaguely deﬁned criterion as well as the absence of labels [13, 14].

Original: Therefore, the foremost needis to identify a surrogate indicator on how noisy the alignment of a sample triplet is, without makingunrealistic assumptions nor incurring much computational overhead.
Revised: Therefore, the foremost needis to identify a surrogate indicator on how noisy the alignment of a sample triplet is, without makingunrealistic assumptions nor incurring much computational overhead.

Original: Our empirical choice of surrogate indicator is the directional alignability (i.e., vector angle) ofgradients generated by different pairwise cross-modal losses, i.e., the cosine similarity between g vaand g vt in the particular example of VATT, after ﬂattening them into two vectors.
Revised: Our empirical choice of surrogate indicator is the directional alignability (i.e., vector angle) ofgradients generated by different pairwise cross-modal losses, i.e., the cosine similarity between g vaand g vt in the particular example of VATT, after ﬂattening them into two vectors.

Original: The choice is ﬁrstlyrationalized by the coherent gradient hypothesis [32], that healthy model training should witness theoverall gradient stronger in certain directions along which the samples’ gradients reinforce each other.
Revised: The choice is ﬁrstlyrationalized by the coherent gradient hypothesis [32], that healthy model training should witness theoverall gradient stronger in certain directions along which the samples’ gradients reinforce each other.

Original: Secondly, comparing the gradient directions between multiple pairwise losses could be considered asan ensemble or “cross-check": intuitively, if both video-audio and video-text pairwise consistencylead to the same update direction, then there is a good chance that those modalities are well alignedand the update direction is reliable; otherwise, at least one pair (video-audio, or video-text) mightsuffer from misalignment and CMA provides noisy guidance.
Revised: Secondly, comparing the gradient directions between multiple pairwise losses could be considered asan ensemble or “cross-check": intuitively, if both video-audio and video-text pairwise consistencylead to the same update direction, then there is a good chance that those modalities are well alignedand the update direction is reliable; otherwise, at least one pair (video-audio, or video-text) mightsuffer from misalignment and CMA provides noisy guidance.

Original: Overall, we conjecture that: aligned video-text-audio triplets should have higher cosine similarity forg va and g vt , and vice versa .
Revised: Overall, we conjecture that: aligned video-text-audio triplets should have higher cosine similarity forg va and g vt , and vice versa .

Original: We compute the cosine similarity between g va and g vt using the samesetting as [32].
Revised: We compute the cosine similarity between g va and g vt using the samesetting as [32].

Original: In the supplementary, We perform a sanity check for our conjecture by examining itqualitatively.
Revised: In the supplementary, We perform a sanity check for our conjecture by examining itqualitatively.

Original: Using this surrogate indicator, we examine the cosine similarity between gradient g va and g vt (over500k iterations) of VATT on the HowTo100M dataset [12].
Revised: Using this surrogate indicator, we examine the cosine similarity between gradient g va and g vt (over500k iterations) of VATT on the HowTo100M dataset [12].

Original: In Figure 3a, we plot the distribution ofcosine similarities, cos ( g va , g vt ) , across 500k iterations of pre-training.
Revised: In Figure 3a, we plot the distribution ofcosine similarities, cos ( g va , g va ) , across 500k iterations of pre-training.

Original: We observe that cos ( g va , g vt )at any iteration resembles a normal distribution, and about half of the gradients g va and g va havemisaligned directions (negative cosine similarities).
Revised: We observe that cos ( g va , g va )at any iteration resembles a normal distribution, and about half of the gradients g va and g va havemisaligned directions (negative cosine similarities).

Original: The plot echos the empirical ﬁnding in [15] thataround 50% of video-text pairs are not well-aligned on the HowTo100M dataset.
Revised: The plot echos the empirical ﬁnding in [15] thataround 50% of video-text pairs are not well-aligned on the HowTo100M dataset.

Original: In the supplementary,we have included similar plots for the YouTube8M dataset [1].
Revised: In the supplementary,we have included similar plots for the YouTube8M dataset [1].

Original: Comparing those plots verify thatnon-narrative video sets have more severe misalignment than the narrative ones, hence challengingthe CMA assumption even more.
Revised: Comparing those plots verify thatnon-narrative video sets have more severe misalignment than the narrative ones, hence challengingthe CMA assumption even more.

Original: [Figure]
Revised: [Figure]

Original:  observations, see supplementary.
Revised: similar observations, see supplementary.

Original: The existence of conﬂicting gradients not only makes the convergence harder, but also makes thelearned representation heavily overﬁt some modalities.
Revised: The existence of conﬂicting gradients not only makes the convergence harder, but also makes thelearned representation heavily overﬁt some modalities.

Original: In our case, we observe the training to bebiased towards video/audio modalities, which favors the downstream performance of video/audioclassiﬁcation task, yet sets back the performance of video-to-text retrieval task (veriﬁed in Table 1).
Revised: In our case, we observe the training to bebiased towards video/audio modalities, which favors the downstream performance of video/audioclassiﬁcation task, yet sets back the performance of video-to-text retrieval task (veriﬁed in Table 1).

Original: 3.2 Harmonizing Gradients during Pre-training
Revised: 3.2 Harmonizing Gradients during Pre-training

Original: We present two strategies to mitigate the conﬂicting gradients: a “hard" way to project gradientdirections, and a “soft" way to progressively select sample triplets in curriculum.
Revised: We present two strategies to mitigate the conﬂicting gradients: a “hard" way to project gradientdirections, and a “soft" way to progressively select sample triplets in curriculum.

Original: 3.2.1 Cross-Modality Gradient Realignment
Revised: 3.2.1 Cross-Modality Gradient Realignment

Original: Since g va and g vt are misaligned in terms of their directions, one natural idea is to realign the cross-modality gradients, by enforcing or re-projecting their directions to be aligned.
Revised: Since g va and g vt are misaligned in terms of their directions, one natural idea is to realign the cross-modality gradients, by enforcing or re-projecting their directions to be aligned.

Original: An intuitive choice is an effective remedy developed for multi-task learning named Gradient
Revised: An intuitive choice is an effective remedy developed for multi-task learning named Gradient

Original: Surgery [16].
Revised: Surgery [16].

Original: Here, instead of applying it to g i and g j where i, j are the two heterogeneoustasks, we apply it to g va and g vt , which refer to the gradients of the two pairwise losses
Revised: Here, instead of applying it to g i and g j where i, j are the two heterogeneoustasks, we apply it to g va and g vt , which refer to the gradients of the two pairwise losses

Original: CMA va and CMA vt w.r.t the weights in the model, following our rationale in Section 3.1.
Revised: CMA va and CMA vt w.r.t the weights in the model, following our rationale in Section 3.1.

Revised: Algorithm

Original: Require: Model parameter θ , minibatchs B va , minibatchs
Revised: Require: Model parameter θ , minibatchs B va , minibatchs

Original: B vt for ( b va , b vt ) ∈ ( B va , B vt ) do g va ← ∇ θ CMA va ( θ ) , g vt ← ∇ θ CMA vt ( θ ) g va ← ﬂatten ( g va ) , g vt ← ﬂatten ( g vt )
Revised: B vt for ( b va , b vt ) ∈ ( B va , B vt ) do g va ← ∇ θ CMA va ( θ ) , g vt ← ∇ θ CMA vt ( θ ) g va ← ﬂatten ( g va ) , g vt ← ﬂatten ( g vt )

Original: ˆ g va ← g va , ˆ g vt ← g vt if g va · g vt < 

Revised: ˆ g va ← g va , ˆ g vt ← g vt if g va · g vt <

Original: then ˆ g va ← ˆ g va − ˆ g va · g vt || g vt ||(cid:46) projection g vt ← g va ˆ g vt ← ˆ g vt − ˆ g vt · g va || g va ||(cid:46) projection g va ← g vt end if ˆ g va ← reshape (ˆ g va ) , ˆ g vt ← reshape (ˆ g vt ) ∆ θ ← ˆ g vt + ˆ g va (cid:46) sum up gradients update θ with ∆ θ (cid:46) update parameter end for
Revised: then ˆ g va ← ˆ g va − ˆ g va · g vt || g vt ||(cid:46) projection g vt ← g va ˆ g vt ← ˆ g vt − ˆ g vt · g va || g va ||(cid:46) projection g va ← g vt end if ˆ g va ← reshape (ˆ g va ) , ˆ g vt ← reshape (ˆ g vt ) ∆ θ ← ˆ g vt + ˆ g va (cid:46) sum up gradients update θ with ∆ θ (cid:46) update parameter end for

Original: If the gradients g va and g vt have negative cosine similarity, we alter them byprojecting each onto the normal planeof the other.
Revised: If the gradients g va and g vt have negative cosine similarity, we alter them byprojecting each onto the normal planeof the other.

Original: Otherwise, if g va and g vthave zero or positive similarity, weretain the original gradients.
Revised: Otherwise, if g va and g vthave zero or positive similarity, weretain the original gradients.

Original: We refer to this particularly adapted formof gradient surgery as Cross-Modality
Revised: We refer to this particularly adapted formof gradient surgery as Cross-Modality

Original: Gradient Realignment .
Revised: Gradient Realignment .

Original: Its update procedure is outlined in Algorithm 1, andthe overhead very cheap, mainly justcomputing inner products of two ﬂattened gradients.
Revised: Its update procedure is outlined in Algorithm 1, andthe overhead very cheap, mainly justcomputing inner products of two ﬂattened gradients.

Original: Additionally, we also tried anothersimilar algorithm named Gradient
Revised: Additionally, we also tried anothersimilar algorithm named Gradient

Original: Vaccine [29], and the results are almost fully consistent and comparableto using Gradient Surgery [16].
Revised: Vaccine [29], and the results are almost fully consistent and comparableto using Gradient Surgery [16].

Original: Wehence only report one for the sake of conciseness.
Revised: Wehence only report one for the sake of conciseness.

Original: 3.2.2 Gradient-based Curriculum Learning
Revised: 3.2.2 Gradient-based Curriculum Learning

Original: We next propose a gradient-based curriculum learning strategy that gradually identiﬁes and removes more misaligned samples based on the similarity indicator as the training goes on.
Revised: We next propose a gradient-based curriculum learning strategy that gradually identiﬁes and removes more misaligned samples based on the similarity indicator as the training goes on.

Original: Formally, given a video-audio pair b va and video-text pair b vt , we only update the parameter ifcos ( g va , g vt ) > γ and drop this sample triplet otherwise.
Revised: Formally, given a video-audio pair b va and video-text pair b vt , we only update the parameter ifcos ( g va , g va ) > γ and drop this sample triplet otherwise.

Original: This ensures us to gradually prioritize training with well-aligned samples while suppressing the noisy supervision from potentiallymisaligned samples.
Revised: This ensures us to gradually prioritize training with well-aligned samples while suppressing the noisy supervision from potentiallymisaligned samples.

Original: The drop rate here is controlled by a hyperparameter γ .
Revised: The drop rate here is controlled by a hyperparameter γ .

Original: In general, γ isa negative value monotonically increasing during training, and there is an underlying rationale.
Revised: In general, γ isa negative value monotonically increasing during training, and there is an underlying rationale.

Original: Algorithm 2 Gradient-based Curriculum Learning
Revised: Algorithm 2 Gradient-based Curriculum Learning

Original: Require: Model parameter θ , minibatchs B va , minibatchs
Revised: Require: Model parameter θ , minibatchs B va , minibatchs

Original: B vt , initial γfor ( b va , b vt ) ∈
Revised: B vt , initial γfor ( b va , b vt ) ∈

Original: ( B va , B vt ) do update γ (cid:46) curriculumly update γ g va ← ∇ θ CMA va ( θ ) , g vt ← ∇ θ CMA vt ( θ ) g va ← ﬂatten ( g va ) , g vt ← ﬂatten ( g vt ) if g va · g vt > γ then g va ← reshape ( g va ) , g vt ← reshape ( g vt ) ∆ θ ← g va + g vt (cid:46) sum up gradients update θ with ∆ θ (cid:46) update parameter end if end for
Revised: ( B va , B vt ) do update γ (cid:46) curriculumly update γ g va ← ∇ θ CMA va ( θ ) , g vt ← ∇ θ CMA vt ( θ ) g va ← ﬂatten ( g va ) , g vt ← ﬂatten ( g vt ) if g va · g vt > γ then g va ← reshape ( g va ) , g vt ← reshape ( g vt ) ∆ θ ← g va + g vt (cid:46) sum up gradients update θ with ∆ θ (cid:46) update parameter end if end for

Original: Initially, the network is under-trainedand the similarity between its learnedfeatures is not reliable to indicate astrong CMA; hence we are more conservative in excluding samples.
Revised: Initially, the network is under-trainedand the similarity between its learnedfeatures is not reliable to indicate astrong CMA; hence we are more conservative in excluding samples.

Original: As thenetwork becomes more sufﬁcientlytrained, the features become more semantically aligned, allowing us to remove more misaligned samples in thelater stage of training.
Revised: As thenetwork becomes more sufﬁcientlytrained, the features become more semantically aligned, allowing us to remove more misaligned samples in thelater stage of training.

Original: The full procedure of Gradient-based Curriculum
Revised: The full procedure of Gradient-based Curriculum

Original: Learning is outlined in Algorithm 2.
Revised: Learning is outlined in Algorithm 2.

Original: We empirically choice γ 0 = − 0 .
Revised: We empirically choice γ 0 = − 0 .

Original: 3 atthe beginning and linearly increase itto γ 500 K = 0 at the end, we includean ablation on the adaptive choice of γ in the Table 2.
Revised: 3 atthe beginning and linearly increase itto γ 500 K = 0 at the end, we includean ablation on the adaptive choice of γ in the Table 2.

Original: Similar classes of ideas, i.e, “gradually focusing a small training subset", have been explored by aprominent class of algorithms which rely on sample selection to robustify [33, 34, 35] or to accelerate[36] training.
Revised: Similar classes of ideas, i.e, “gradually focusing a small training subset", have been explored by aprominent class of algorithms which rely on sample selection to robustify [33, 34, 35] or to accelerate[36] training.

Original: Most of them use the “small loss trick" wherein a fraction of samples with smallest lossvalues below a certain threshold are considered as reliable.
Revised: Most of them use the “small loss trick" wherein a fraction of samples with smallest lossvalues below a certain threshold are considered as reliable.

Original: However, they did not explore gradientconﬂict indicators, which targets solving our CMA problem here.
Revised: However, they did not explore gradientconﬂict indicators, which targets solving our CMA problem here.

Original: Also note that our two steps canfully reuse the cosine similarity computed, so applying them together incurs no extra overhead.
Revised: Also note that our two steps canfully reuse the cosine similarity computed, so applying them together incurs no extra overhead.

Original:  Experiments
Revised:  Experiments

Original: 4.1 Pre-Training Datasets and Settings
Revised: 4.1 Pre-Training Datasets and Settings

Original: In this paper, we focus on studying gradients in a modality-agnostic single-backbone setting, thuswe do not consider multimodal pre-training baseline with modality-speciﬁc models, e.g. HERO[37],
Revised: In this paper, we focus on studying gradients in a modality-agnostic single-backbone setting, thuswe do not consider multimodal pre-training baseline with modality-speciﬁc models, e.g. HERO[37],

Original: MMV[5], PerceiverIO[38], AVLnet[39].
Revised: MMV[5], PerceiverIO[38], AVLnet[39].

Original: Apart from VATT[11], there exist other concurrent workthat use modality-agnostic setting[19, 40, 41], yet they all have their respective limitations.
Revised: Apart from VATT[11], there exist other concurrent workthat use modality-agnostic setting[19, 40, 41], yet they all have their respective limitations.

Original: Forexample, UniVL[19] lacks audio modalities thus cannot constitute video-audio CMA loss, VLM[40]applies separate attention mask for each modality while PolyViT[41] use alternate training betweenmodalities and tasks, thus eliminate the possibilities of any gradients conﬂicts.
Revised: Forexample, UniVL[19] lacks audio modalities thus cannot constitute video-audio CMA loss, VLM[40]applies separate attention mask for each modality while PolyViT[41] use alternate training betweenmodalities and tasks, thus eliminate the possibilities of any gradients conﬂicts.

Original: Therefore, this leave
Revised: Therefore, this leave

Original: VATT[11] to be the most suitable baseline besides it SoTA performance.
Revised: VATT[11] to be the most suitable baseline besides it SoTA performance.

Original: Our pre-training adopts three settings for comprehensive comparison: (1) using HowTo100M asour training set, following [19, 12]; (2) combining HowTo100M and AudioSet as our training setfollowing [5] and VATT [11]; and (3) additionally merging a noisy
Revised: Our pre-training adopts three settings for comprehensive comparison: (1) using HowTo100M asour training set, following [19, 12]; (2) combining HowTo100M and AudioSet as our training setfollowing [5] and VATT [11]; and (3) additionally merging a noisy

Original: Youtube8M dataset alongsidewith HowTo100M and AudioSet, to create a much more compound and complicated training set thanconsidered before, to stretch-test our proposal’s beneﬁts to scale up pre-training in uncurated andpoorly aligned data.
Revised: Youtube8M dataset alongsidewith HowTo100M and AudioSet, to create a much more compound and complicated training set thanconsidered before, to stretch-test our proposal’s beneﬁts to scale up pre-training in uncurated andpoorly aligned data.

Original: For each setting, we i.i.d sample video-audio-text clips from the mixture of allcandidate sets in every batch.
Revised: For each setting, we i.i.d sample video-audio-text clips from the mixture of allcandidate sets in every batch.

Original: (HowTo100M, Audioset or Youtube8M, if applicable).
Revised: (HowTo100M, Audioset or Youtube8M, if applicable).

Original: For all setting,we only use a subset of HowTo100M, AudioSet, Youtube8M, Kinetics400 and Youcook2 dataset incompliance with Youtube’s wipe-out policies.
Revised: For all setting,we only use a subset of HowTo100M, AudioSet, Youtube8M, Kinetics400 and Youcook2 dataset incompliance with Youtube’s wipe-out policies.

Original: HowTo100M 1 [12] is a large-scale dataset of narrated videos with an emphasis on instructional videosthat have well synchronized modalities.
Revised: HowTo100M 1 [12] is a large-scale dataset of narrated videos with an emphasis on instructional videosthat have well synchronized modalities.

Original: The training set consists of 127 Millions video clips with thecorresponding close-captioned text from automatic speech recognition (ASR).
Revised: The training set consists of 127 Millions video clips with thecorresponding close-captioned text from automatic speech recognition (ASR).

Original: AudioSet 1 [23] is a large-scale audio-visual dataset originally intended for audio event classiﬁcation.
Revised: AudioSet 1 [23] is a large-scale audio-visual dataset originally intended for audio event classiﬁcation.

Original: It consists of 1.78 Million 10-second clips sampled from Youtube Videos that contains a variety ofaudio tracks, such as musical instruments, animals or mechanical sounds.
Revised: It consists of 1.78 Million 10-second clips sampled from Youtube Videos that contains a variety ofaudio tracks, such as musical instruments, animals or mechanical sounds.

Original: Youtube8M 1 [1] is a large-scale video classiﬁcation dataset based on Youtube videos.
Revised: Youtube8M 1 [1] is a large-scale video classiﬁcation dataset based on Youtube videos.

Original: It consists ofa diverse vocabulary of 3862 visual entities and a total number of 8 Million videos.
Revised: It consists ofa diverse vocabulary of 3862 visual entities and a total number of 8 Million videos.

Original: To the best ofour knowledge, we are the ﬁrst to scale contrastive multimodal pre-training beyond instructionalvideos, onto this noisy Youtube8M videos that more faithfully represent videos the wild.
Revised: To the best ofour knowledge, we are the ﬁrst to scale contrastive multimodal pre-training beyond instructionalvideos, onto this noisy Youtube8M videos that more faithfully represent videos the wild.

Original: For data preprocessing, we split each video into 5s clips and use ASR close-captions to generate the correspondingtext, resulting in 165 Millions video-audio-text clips.
Revised: For data preprocessing, we split each video into 5s clips and use ASR close-captions to generate the correspondingtext, resulting in 165 Millions video-audio-text clips.

Original: 4.2 Pre-Training Settings
Revised: 4.2 Pre-Training Settings

Original: Network Architecture: For all of our experiments, we use modality-agnostic Transformer variant in[11], VATT-MA-Medium.
Revised: Network Architecture: For all of our experiments, we use modality-agnostic Transformer variant in[11], VATT-MA-Medium.

Original: Speciﬁcally, we use a 12-layer Transformer, in which each Transformermodule has a feed-forward MLP with hidden size of 4096, and 16 attention heads with a hidden sizeof 1024.
Revised: Speciﬁcally, we use a 12-layer Transformer, in which each Transformermodule has a feed-forward MLP with hidden size of 4096, and 16 attention heads with a hidden sizeof 1024.

Original: Pre-training Hyperparameter: We strictly follow the setting in [11], pre-training VATT fromscratch with Adam optimizer with an initial learning rate of 1e-4, 10k warmup steps, 500k steps intotal, a batch size of 2048 and using a cosine learning rate scheduler to anneal the learning rate from1e-4 to 5e-5.
Revised: Pre-training Hyperparameter: We strictly follow the setting in [11], pre-training VATT fromscratch with Adam optimizer with an initial learning rate of 1e-4, 10k warmup steps, 500k steps intotal, a batch size of 2048 and using a cosine learning rate scheduler to anneal the learning rate from1e-4 to 5e-5.

Original: Our framework is implemented in Tensorﬂow 2.8, and train with 256 TPUV3s, it took atotal of 3 days to train our models.
Revised: Our framework is implemented in Tensorﬂow 2.8, and train with 256 TPUV3s, it took atotal of 3 days to train our models.

Original: 4.3 Downstream Tasks for Evaluation
Revised: 4.3 Downstream Tasks for Evaluation

Original: We evaluate the performance of our pre-trained representations when tuned towards various downstream tasks, including a variety of video/audio and cross modalities tasks.
Revised: We evaluate the performance of our pre-trained representations when tuned towards various downstream tasks, including a variety of video/audio and cross modalities tasks.

Original: Video Action Recognition: Following [5], we evaluate on UCF101[42] (101 action classes, 13,320videos) and the HMDB51[43] (51 classes, 6,766 videos) benchmarks.
Revised: Video Action Recognition: Following [5], we evaluate on UCF101[42] (101 action classes, 13,320videos) and the HMDB51[43] (51 classes, 6,766 videos) benchmarks.

Original: For both settings, we directlyattach a linear classiﬁer on the learned representations while freezing the transformer backbone.
Revised: For both settings, we directlyattach a linear classiﬁer on the learned representations while freezing the transformer backbone.

Original: We also evaluate on Kinetics400 1 [44] dataset (400 classes, 234,584 video), where we ﬁne-tune thetransformer backbone.
Revised: We also evaluate on Kinetics400 1 [44] dataset (400 classes, 234,584 video), where we ﬁne-tune thetransformer backbone.

Original: Audio Event Classiﬁcation: Following [5]
Revised: Audio Event Classiﬁcation: Following [5]

Original: , we used ESC50[45] (50 classes, 2000 audio clips) asthe benchmark for audio modalities.
Revised: , we used ESC50[45] (50 classes, 2000 audio clips) asthe benchmark for audio modalities.

Original: Similarly, we directly attach and tune a linear classiﬁer on top ofthe frozen transformer backbone.
Revised: Similarly, we directly attach and tune a linear classiﬁer on top ofthe frozen transformer backbone.

Original: Zero-shot Video-Text Retrieval: To evaluate our performance on cross-modality task, we follow[5] using YouCook2 1 [15] (3.1K video-text clips) and MSR-VTT 1 [46] (1K video-text clips) as ourevaluation datasets.
Revised: Zero-shot Video-Text Retrieval: To evaluate our performance on cross-modality task, we follow[5] using YouCook2 1 [15] (3.1K video-text clips) and MSR-VTT 1 [46] (1K video-text clips) as ourevaluation datasets.

Original: We directly use the embedding from the pre-trained model without any ﬁnetuning, and report the recall at 10 (R@10) and Median Rank as described in [5].
Revised: We directly use the embedding from the pre-trained model without any ﬁnetuning, and report the recall at 10 (R@10) and Median Rank as described in [5].

Original: [Table]
Revised: [Table]

Original: 4.4 Result Analysis on HowTo100M and AudioSet Pre-training
Revised: 4.4 Result Analysis on HowTo100M and AudioSet Pre-training

Original: Gradient Re-weighting We set up a simple gradient re-weighting baseline similar to [30], where wesimply scale up the gradient magnitude of g va by 2.5 times and scale down the gradient magnitudeof its counterpart g vt by 0.5 times, we denote this baseline as Gradient Re-weighting (VA), we alsodid it vice versa and denoted it as Gradient Re-weighting (VT).
Revised: Gradient Re-weighting We set up a simple gradient re-weighting baseline similar to [30], where wesimply scale up the gradient magnitude of g va by 2.5 times and scale down the gradient magnitudeof its counterpart g vt by 0.5 times, we denote this baseline as Gradient Re-weighting (VA), we alsodid it vice versa and denoted it as Gradient Re-weighting (VT).

Original: As shown in Table 1, we did notobserve much performance boost with gradient re-weighting, on the contrary, there are signiﬁcantperformance drop in Text-Video Retrieval task on both YouCook2 and MSRVTT datasets.
Revised: As shown in Table 1, we did notobserve much performance boost with gradient re-weighting, on the contrary, there are signiﬁcantperformance drop in Text-Video Retrieval task on both YouCook2 and MSRVTT datasets.

Original: Cross-Modality Gradient Realignment Compared to Gradient Re-weighting, our Cross-Modality
Revised: Cross-Modality Gradient Realignment Compared to Gradient Re-weighting, our Cross-Modality

Original: Gradient Realignment mitigates the gradient conﬂicts by modifying gradients direction.
Revised: Gradient Realignment mitigates the gradient conﬂicts by modifying gradients direction.

Original: In Table 1,we can see signiﬁcant performance gains in video-text retrieval task (e.g. 58% gain in rank metric on
Revised: In Table 1,we can see signiﬁcant performance gains in video-text retrieval task (e.g. 58% gain in rank metric on

Original: YouCook2, 46% gain in R@10 metric on MSR-VTT), while maintaining a comparable performancein video/audio tasks.
Revised: YouCook2, 46% gain in R@10 metric on MSR-VTT), while maintaining a comparable performancein video/audio tasks.

Original: Since video-text retrieval task heavily hinge on cross-modality alignment(CMA), this improvement verify the beneﬁts of re-aligning the cross-modality gradients.
Revised: Since video-text retrieval task heavily hinge on cross-modality alignment(CMA), this improvement verify the beneﬁts of re-aligning the cross-modality gradients.

Original: Gradient-based Curriculum Learning Compared to Cross-Modality Gradient Realignment,
Revised: Gradient-based Curriculum Learning Compared to Cross-Modality Gradient Realignment,

Original: Gradient-based Curriculum Learning inherents its performance gains in video-text retrieval task, yetenjoy an additional performance boost in audio classiﬁcation (e.g. 1.40% gain ESC50 dataset in
Revised: Gradient-based Curriculum Learning inherents its performance gains in video-text retrieval task, yetenjoy an additional performance boost in audio classiﬁcation (e.g. 1.40% gain ESC50 dataset in

Original: Table 1) and video classiﬁcation (e.g. 1.94% gains UCF101 dataset in Table 1).
Revised: Table 1) and video classiﬁcation (e.g. 1.94% gains UCF101 dataset in Table 1).

Original: We also perform anablation on the adaptive choice of γ on Howto100M dataset in Table 2.
Revised: We also perform anablation on the adaptive choice of γ on Howto100M dataset in Table 2.

Original: We can see that curriculumlyevolving γ (i.e. ( γ 0 , γ 500 K ) =
Revised: We can see that curriculumlyevolving γ (i.e. ( γ 0 , γ 500 K ) =

Original: ( − 0 .
Revised: ( − 0 .

Original: 3 , 0 . 0) ) yields the best overall results, while maintaining a ﬁxed γ(i.e.
Revised: 3 , 0 . 0) ) yields the best overall results, while maintaining a ﬁxed γ(i.e.

Original: ( γ 0 , γ 500 K ) =
Revised: ( γ 0 , γ 500 K ) =

Original: (0 . 0 , 0 . 0) ) yields the worst results, this verify the effectiveness of curriculumlyremoving mis-aligned samples during training.
Revised: (0 . 0 , 0 . 0) ) yields the worst results, this verify the effectiveness of curriculumlyremoving mis-aligned samples during training.

Original: Cross-Modality Gradient Realignment + Gradient-based Curriculum Learning
Revised: Cross-Modality Gradient Realignment + Gradient-based Curriculum Learning

Original: We further verify the effectiveness with the combination of both techniques.
Revised: We further verify the effectiveness with the combination of both techniques.

Original: Speciﬁcally, given a video-audio pair and video-text pair, (a) if cos ( g va , g vt ) ≤ γ , we drop this sample triplet; (b) if γ < cos ( g va , g vt ) < 0 ,we apply gradient projection in Algorithm 1; (c) if cos ( g va , g vt ) ≥ 0 , we retain the original gradients.
Revised: Speciﬁcally, given a video-audio pair and video-text pair, (a) if cos ( g va , g va ) ≤ γ , we drop this sample triplet; (b) if γ < cos ( g va , g va ) < 0 ,we apply gradient projection in Algorithm 1; (c) if cos ( g va , g va ) ≥ 0 , we retain the original gradients.

Original: We found they are complimentary to each other and the combination of both lead to betterperformance in the majority of downstream tasks.
Revised: We found they are complimentary to each other and the combination of both lead to betterperformance in the majority of downstream tasks.

Original: [Table]
Revised: [Table]

Original: decay
Revised: [Figure]

Original: As shown in Table 1, adding YouTube8M dataset brings the video classiﬁcation and video-textretrieval performance to a new level (e.g. 88.28% Top1 on UCF101, 56.0 rank on MSRVTT)), yet itset back the performance in audio classiﬁcation task, we conjecture this is largely due to the noisymisaligned samples introduced by Youtube8M dataset.
Revised: As shown in Table 1, adding YouTube8M dataset brings the video classiﬁcation and video-textretrieval performance to a new level (e.g. 88.28% Top1 on UCF101, 56.0 rank on MSRVTT)), yet itset back the performance in audio classiﬁcation task, we conjecture this is largely due to the noisymisaligned samples introduced by Youtube8M dataset.

Original: However, by leveraging our proposed techniques, we further yields a new state-of-the-art performancein the modality-agnostic setting, without any modiﬁcation in architecture, striking 89.70% Top1 on
Revised: However, by leveraging our proposed techniques, we further yields a new state-of-the-art performancein the modality-agnostic setting, without any modiﬁcation in architecture, striking 89.70% Top1 on

Original: UCF101, 92.15% Top5 on HMDB51, 80.01% Top1 on Kinetics400 and 85.00% Top1 on ESC50dataset, demonstrating the effectiveness of our method when scaling to noisy Youtub

Revised: UCF101, 92.15% Top5 on HMDB51, 80.01% Top1 on Kinetics400 and 85.00% Top1 on ESC50dataset, demonstrating the effectiveness of our method when scaling to noisy Youtub

Original: M dataset.
Revised: M dataset.

Original: 4.6 Feature Visualization
Revised: 4.6 Feature Visualization

Original: To better illustrate why gradient harmonization techniques works , in Figure 4, we follow the visualization used in VATT[11], which take YouCook2 Video-Text Retrieval dataset, showing theoutput space visualization of video and text, respectively.
Revised: To better illustrate why gradient harmonization techniques works , in Figure 4, we follow the visualization used in VATT[11], which take YouCook2 Video-Text Retrieval dataset, showing theoutput space visualization of video and text, respectively.

Original: We can see the initial video and textfeature representation of VATT baseline are mixed together (which aligned with the ﬁndings in [11]),however, after using gradient realignment, curriculum learning or applying both, the video and textrepresentations become disentangled, this indicates our gradient harmonization techniques adds tocross-modality diversity, which makes video/text more discriminative from each other, making themodel more discriminative and better in generalization.
Revised: We can see the initial video and textfeature representation of VATT baseline are mixed together (which aligned with the ﬁndings in [11]),however, after using gradient realignment, curriculum learning or applying both, the video and textrepresentations become disentangled, this indicates our gradient harmonization techniques adds tocross-modality diversity, which makes video/text more discriminative from each other, making themodel more discriminative and better in generalization.

Original: [Figure]
Revised: [Figure]

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we take a deep dive into the common CMA assumption used in multimodal pretraining.
Revised: In this paper, we take a deep dive into the common CMA assumption used in multimodal pretraining.

Original: Using the gradient directions between pairwise losses as the surrogate indicator, we observeubiquitous gradient conﬂicts even in relatively well-aligned narrative datasets.
Revised: Using the gradient directions between pairwise losses as the surrogate indicator, we observeubiquitous gradient conﬂicts even in relatively well-aligned narrative datasets.

Original: We then proposetwo plug-and-play techniques to harmonize the gradients during training.
Revised: We then proposetwo plug-and-play techniques to harmonize the gradients during training.

Original: They are demonstrated tosubstantially enhance VATT pre-training across a number of downstream tasks, and further scale it upto pre-training with even more complicated and poorly aligned data.
Revised: They are demonstrated tosubstantially enhance VATT pre-training across a number of downstream tasks, and further scale it upto pre-training with even more complicated and poorly aligned data.

Original: Our ﬁndings underline the (oftenoverlooked) importance to carefully revisit the basic assumption and training dynamics of multimodallearning, besides advancing the model architectures.
Revised: Our ﬁndings underline the (oftenoverlooked) importance to carefully revisit the basic assumption and training dynamics of multimodallearning, besides advancing the model architectures.

Original: However, our work did not fully explore otherregularization techniques in multi-tasking learning literature, such as [26, 27, 28].
Revised: However, our work did not fully explore otherregularization techniques in multi-tasking learning literature, such as [26, 27, 28].

Original: The potentialnegative societal impacts include the malicious use of multimodal models, such as unintended use ofits downstream visual/audio recognition capability.
Revised: The potentialnegative societal impacts include the malicious use of multimodal models, such as unintended use ofits downstream visual/audio recognition capability.

Original: Checklist
Revised: Checklist

Original: [Table]
Revised: [Table]

Original: For all authors...
Revised: For all authors...

Original: [Table]
Revised: [Table]

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s

Original: [Table]
Revised: [Table]

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: [Table]
Revised: [Table]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Section 5, e.g. our work did
Revised: [Yes] See Section 5, e.g. our work did

Original:  not fully explore other regularization techniques in multi-tasking learning literature,such as [26, 27, 28]
Revised: [Table] not fully explore other regularization techniques in multi-tasking learning literature,such as [26, 27, 28](c)

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [Yes] SeeSection 5, e.g. the malicious use of multimodal models.
Revised: [Yes] SeeSection 5, e.g. the malicious use of multimodal models.

Original:  (d) Have you read the ethics review guidelines and ensured that your paper conforms to 
Revised: [Table] (d) Have you read the ethics review guidelines and ensured that your paper conforms to [Table]

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Revised: [Table]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Revised: [Table]

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Revised: [Table]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Revised: [Table]

Original: If you ran experiments...
Revised: If you ran experiments...

Revised: [Table]

Original: (a) Did you include the code, data, and instructions needed to reproduce the main exper-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main exper-

Original: imental results (either in the supplemental material or as a URL)?
Revised: imental results (either in the supplemental material or as a URL)?

Original: [No] The code isproprietary, However, we include detailed discussions on experiment settings and datain Section 4.1 to replicate the experiment results. 
Revised: [No] The code isproprietary, However, we include detailed discussions on experiment settings and datain Section 4.1 to replicate the experiment results. [Table]

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] See Section 4.
Revised: [Yes] See Section 4.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [No] Since pre-training are very computationally expensive,and it is infeasible do to multiple runs. 
Revised: [No] Since pre-training are very computationally expensive,and it is infeasible do to multiple runs. [Table]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Section 4.2:
Revised: [Yes] See Section 4.2:

Original: Pre-training
Revised: Pre-training

Original: Hyperparameter
Revised: Hyperparameter

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Section 4.
Revised: [Yes] See Section 4.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]