Original: Scaling Multimodal Pre-Training via Cross-Modality Gradient Harmonization
Revised: Scaling Multimodal Pre-Training via Cross-Modality Gradient Harmonization

Original: Abstract
Revised: Abstract

Original: Self-supervised pre-training recently demonstrates success on large-scale multimodal data, and state-of-the-art contrastive learning methods often enforce the feature consistency from cross-modality inputs, such as video/audio or video/text pairs.
Revised: Self-supervised pre-training recently demonstrates success on large-scale multimodal data, and state-of-the-art contrastive learning methods often enforce the feature consistency from cross-modality inputs, such as video/audio or video/text pairs.

Original: Despite its convenience to formulate and leverage in practice, such cross-modality alignment (CMA) is only a weak and noisy supervision, since two modalities can be semantically misaligned even they are temporally aligned.
Revised: Despite its convenience to formulate and leverage in practice, such cross-modality alignment (CMA) is only a weak and noisy supervision, since two modalities can be semantically misaligned even they are temporally aligned.

Original: For example, even in the (often adopted) instructional videos, a speaker can sometimes refer to something that is not visually present in the current frame; and the semantic misalignment would only be more unpredictable for the raw videos collected from unconstrained internet sources.
Revised: For example, even in the (often adopted) instructional videos, a speaker can sometimes refer to something that is not visually present in the current frame; and the semantic misalignment would only be more unpredictable for the raw videos collected from unconstrained internet sources.

Original: We conjecture that might cause conﬂicts and biases among modalities, and may hence prohibit CMA from scaling up to training with larger and more heterogeneous data.
Revised: We conjecture that might cause conﬂicts and biases among modalities, and may hence prohibit CMA from scaling up to training with larger and more heterogeneous data.

Original: This paper ﬁrst veriﬁes our conjecture by observing that, even in the latest VATT pre-training using only narrated videos, there exist strong gradient conﬂicts between different CMA losses within the same sample triplet (video, audio, text), indicating them as the noisy source of supervision.
Revised: This paper ﬁrst veriﬁes our conjecture by observing that, even in the latest VATT pre-training using only narrated videos, there exist strong gradient conﬂicts between different CMA losses within the same sample triplet (video, audio, text), indicating them as the noisy source of supervision.

Original: We then propose to harmonize such gradients during pre-training, via two techniques: (i) cross-modality gradient realignment: modifying different CMA loss gradients for one sample triplet, so that their gradient directions are in more agreement; and (ii) gradient-based curriculum learning: leveraging the gradient conﬂict information on an indicator of sample noisiness, to develop a curriculum learning strategy to prioritize training with less noisy sample triplets.
Revised: We then propose to harmonize such gradients during pre-training, via two techniques: (i) cross-modality gradient realignment: modifying different CMA loss gradients for one sample triplet, so that their gradient directions are in more agreement; and (ii) gradient-based curriculum learning: leveraging the gradient conﬂict information on an indicator of sample noisiness, to develop a curriculum learning strategy to prioritize training with less noisy sample triplets.

Original: Applying those
Revised: Applying those

Original: gradient harmonization techniques to pre-training VATT on the HowTo100M dataset, we consistently improve its performance on different downstream tasks.
Revised: gradient harmonization techniques to pre-training VATT on the HowTo100M dataset, we consistently improve its performance on different downstream tasks.

Original: Moreover, we are able to scale VATT pre-training to more complicated non-narrative
Revised: Moreover, we are able to scale VATT pre-training to more complicated non-narrative

Original: Youtube

Revised: Youtube

Original: M dataset to further improve the state-of-the-arts.
Revised: M dataset to further improve the state-of-the-arts.

Original:  Introduction
Revised:  Introduction

Original: Self-supervised pre-training scales up deep learning to leverage massive unlabeled data.
Revised: Self-supervised pre-training scales up deep learning to leverage massive unlabeled data.

Original: Beyond the maturity of pre-training over single-modality data such as language [2] or images [3], the recent success on multimodal pre-training [4, 5] demonstrates the versatility to synergize the rich multimodal information and to beneﬁt a variety of downstream tasks.
Revised: Beyond the maturity of pre-training over single-modality data such as language [2] or images [3], the recent success on multimodal pre-training [4, 5] demonstrates the versatility to synergize the rich multimodal information and to beneﬁt a variety of downstream tasks.

Original: Many methods of this category [6, 7, 8, 4], especially the latest contrastive pre-training methods [5, 9, 10, 11], consider the most organic supervision as the cross-modality alignment ( CMA ), e.g., in the same video sequence, the
Revised: Many methods of this category [6, 7, 8, 4], especially the latest contrastive pre-training methods [5, 9, 10, 11], consider the most organic supervision as the cross-modality alignment ( CMA ), e.g., in the same video sequence, the

Original: [Figure]
Revised: [Figure]

Original: “On the football
Revised: “On the football

Original: “World Cup Semi Final” “Raising fund for UNICEF” “Whether it's in a kitchen”
Revised: “World Cup Semi Final” “Raising fund for UNICEF” “Whether it's in a kitchen”

Original: [Figure]
Revised: [Figure]

Original: video frames, audio stream, and text scripts are temporally aligned and hence should naturally have correspondence.
Revised: video frames, audio stream, and text scripts are temporally aligned and hence should naturally have correspondence.

Original: Such assumed cross-modal correspondence could be exploited as “mutual supervision" for each modality to learn consistent semantic features with others.
Revised: Such assumed cross-modal correspondence could be exploited as “mutual supervision" for each modality to learn consistent semantic features with others.

Original: For example, the latest Video-Audio-Text Transformer (VATT)
Revised: For example, the latest Video-Audio-Text Transformer (VATT)

Original: [11] hinges on this multimodal CMA a priori to train a transformer-based architecture (even a single backbone by sharing weights among three modalities), using two contrastive losses to align video-text and video-audio pairs, respectively.
Revised: [11] hinges on this multimodal CMA a priori to train a transformer-based architecture (even a single backbone by sharing weights among three modalities), using two contrastive losses to align video-text and video-audio pairs, respectively.

Original: Their results set many new records in downstream tasks with neither supervised pre-training nor predeﬁned inductive biases, showing strong “universal" generalizability despite the modality and domain gaps.
Revised: Their results set many new records in downstream tasks with neither supervised pre-training nor predeﬁned inductive biases, showing strong “universal" generalizability despite the modality and domain gaps.

Original: In this paper, we focus on studying gradients in a modality-agnostic single-backbone setting of VATT [11].
Revised: In this paper, we focus on studying gradients in a modality-agnostic single-backbone setting of VATT [11].

Original: Unfortunately, the CMA assumption rests on a very shaky foundation, especially for uncurated videos collected in the wild.
Revised: Unfortunately, the CMA assumption rests on a very shaky foundation, especially for uncurated videos collected in the wild.

Original: The CMA can be weak even for the instructional videos such as HowTo100M [12] that are commonly adopted as the current multimodal pre-training data source.
Revised: The CMA can be weak even for the instructional videos such as HowTo100M [12] that are commonly adopted as the current multimodal pre-training data source.

Original: For instance, a speaker can refer to something before or after actually demonstrating it visually; she or he could also skip verbally explaining something that is visually happening, because that may be trivial or be already clear enough in the visual context [13].
Revised: For instance, a speaker can refer to something before or after actually demonstrating it visually; she or he could also skip verbally explaining something that is visually happening, because that may be trivial or be already clear enough in the visual context [13].

Original: Such discrepancy would be ampliﬁed if considering many other outliers in video (e.g. background objects) or language (e.g. off-topic small talks).
Revised: Such discrepancy would be ampliﬁed if considering many other outliers in video (e.g. background objects) or language (e.g. off-topic small talks).

Original: The semantic misalignment will be even more severe, and in unpredictable forms, when we go beyond instructional videos and explore training with non-narrative, free-from internet videos.
Revised: The semantic misalignment will be even more severe, and in unpredictable forms, when we go beyond instructional videos and explore training with non-narrative, free-from internet videos.

Original: Figureshows a few examples of such cross-modal misalignment in the non-narrative
Revised: Figureshows a few examples of such cross-modal misalignment in the non-narrative

Original: Youtube

Revised: Youtube

Original: M dataset [1], which would put CMA-guided pre-training in jeopardy.
Revised: M dataset [1], which would put CMA-guided pre-training in jeopardy.

Original: This important problem is yet under-studied in the self-supervised multimodal pre-training literature.
Revised: This important problem is yet under-studied in the self-supervised multimodal pre-training literature.

Original: That said, there has been a few recent efforts such as Multiple-Instance Learning (MIL) NCE [13] and noisy pair pruning as in [14].
Revised: That said, there has been a few recent efforts such as Multiple-Instance Learning (MIL) NCE [13] and noisy pair pruning as in [14].

Original: We conjecture that the vanilla CMA might provide only weak, noisy and even misleading supervision for multimodal pre-training; hence it might constitute a hidden bottleneck when scaled up with more realistic data (beyond instructional videos) whose modalities are poorly aligned.
Revised: We conjecture that the vanilla CMA might provide only weak, noisy and even misleading supervision for multimodal pre-training; hence it might constitute a hidden bottleneck when scaled up with more realistic data (beyond instructional videos) whose modalities are poorly aligned.

Original: We verify our conjecture by inspecting two large-scale multimodal datasets, HowTo100M [15] and Youtube8M [1].
Revised: We verify our conjecture by inspecting two large-scale multimodal datasets, HowTo100M [15] and Youtube8M [1].

Original: Speciﬁcally, when we adopt pairwise CMA losses (e.g. video-audio, video-text) within the cross-modal sample triplets (video, audio, text), a majority of the gradients strongly conﬂict with each other in terms of their directions.
Revised: Speciﬁcally, when we adopt pairwise CMA losses (e.g. video-audio, video-text) within the cross-modal sample triplets (video, audio, text), a majority of the gradients strongly conﬂict with each other in terms of their directions.

Original: We further identify (in Figure 3) that such strong conﬂict in gradients are correlated to noisiness of samples, indicating the noisy and misaligned supervision could be an important cause of gradient conﬂicts.
Revised: We further identify (in Figure 3) that such strong conﬂict in gradients are correlated to noisiness of samples, indicating the noisy and misaligned supervision could be an important cause of gradient conﬂicts.

Original: Moreover, the cross-modal conﬂict leads to not only unstable training convergence, but also modality-speciﬁc overﬁtting; e.g. semantically strong representations for one modality while collapsed representations for another [16], yielding weak overall performance in cross-modality tasks such as Text-Video Retrieval as shown in Table 1.
Revised: Moreover, the cross-modal conﬂict leads to not only unstable training convergence, but also modality-speciﬁc overﬁtting; e.g. semantically strong representations for one modality while collapsed representations for another [16], yielding weak overall performance in cross-modality tasks such as Text-Video Retrieval as shown in Table 1.

Original: Build upon the above conjectures and observations, we propose to harmonize those gradients so that they become mutually compatible in learning the uniﬁed representations.
Revised: Build upon the above conjectures and observations, we propose to harmonize those gradients so that they become mutually compatible in learning the uniﬁed representations.

Original: Speciﬁcally, we introduce two techniques: (i) cross-modal gradient realignment, where we modify different pairwise CMA loss gradients to align their directions for the same sample triplet, using gradient surgery [17] developed for multi-task learning; and (ii) gradient-based curriculum learning, where we leverage the gradient
Revised: Speciﬁcally, we introduce two techniques: (i) cross-modal gradient realignment, where we modify different pairwise CMA loss gradients to align their directions for the same sample triplet, using gradient surgery [17] developed for multi-task learning; and (ii) gradient-based curriculum learning, where we leverage the gradient

Original: conﬂict information to indicate sample noisiness
Revised: conﬂict information to indicate sample noisiness

Original: (e.g. a triplet whose CMA gradients are in more agreement is considered less “noisy" and would be prioritized for cross-modal pre-training) and a curriculum learning strategy is developed accordingly.
Revised: (e.g. a triplet whose CMA gradients are in more agreement is considered less “noisy" and would be prioritized for cross-modal pre-training) and a curriculum learning strategy is developed accordingly.

Original: Both techniques are found to boost VATT performance on downstream tasks, not only on instructional videos, but even more when the more complicated non-narrative data is involved.
Revised: Both techniques are found to boost VATT performance on downstream tasks, not only on instructional videos, but even more when the more complicated non-narrative data is involved.

Original: Our contributions can be summarized as the following:
Revised: Our contributions can be summarized as the following:

Original: • We suggest that the commonly assumed CMA might be an important yet overlooked performance hurdle for scaling up multimodal pre-training, and we observe severe misalignment even from the state-of-the-art (SOTA) model [11] on the relatively aligned data [12].
Revised: • We suggest that the commonly assumed CMA might be an important yet overlooked performance hurdle for scaling up multimodal pre-training, and we observe severe misalignment even from the state-of-the-art (SOTA) model [11] on the relatively aligned data [12].

Original: • We propose to consistently improve the pre-training pipeline in the only baseline in this direction, VATT (modality-agnostic setting), resulting in better downstream performance.
Revised: • We propose to consistently improve the pre-training pipeline in the only baseline in this direction, VATT (modality-agnostic setting), resulting in better downstream performance.

Original: • With the help of our proposed techniques, we consistently improve the pre-training of VATT, resulting in better downstream performance, e.g. up to 58% gain in the rank metric of video-text retrieval task.
Revised: • With the help of our proposed techniques, we consistently improve the pre-training of VATT, resulting in better downstream performance, e.g. up to 58% gain in the rank metric of video-text retrieval task.

Original: Moreover, we are able to scale VATT pre-training to more complicated non-narrative Youtube8M dataset, which further yields new state-of-the-art performance in a modality-agnostic setting.
Revised: Moreover, we are able to scale VATT pre-training to more complicated non-narrative Youtube8M dataset, which further yields new state-of-the-art performance in a modality-agnostic setting.

Original:  Related Work 2.1 Self-Supervised Multimodal Pre-training
Revised:  Related Work 2.1 Self-Supervised Multimodal Pre-training

Original: The most organic self-supervision can arguably be found in the multimodal data that are abundantly available in the digital world: their temporal alignment naturally lead to cross-modality mutual supervision, hence meaningful features can be learned without requiring human annotation.
Revised: The most organic self-supervision can arguably be found in the multimodal data that are abundantly available in the digital world: their temporal alignment naturally lead to cross-modality mutual supervision, hence meaningful features can be learned without requiring human annotation.

Original: Multimodal pre-training can thus leverage richer semantic cues than single-modality counterparts, and outperform in various downstream tasks such as image classiﬁcation [5, 11], video action recognition [5, 11, 13, 10], audio event classiﬁcation [5, 11, 10], and video-text retrieval [5, 11, 13].
Revised: Multimodal pre-training can thus leverage richer semantic cues than single-modality counterparts, and outperform in various downstream tasks such as image classiﬁcation [5, 11], video action recognition [5, 11, 13, 10], audio event classiﬁcation [5, 11, 10], and video-text retrieval [5, 11, 13].

Original: Most of those methods formulate pre-text tasks, including cross-modality correspondence [6, 7, 8, 10], cross-modality clustering [18], cross-modality longer context prediction [19], or a combination of multiple pre-text tasks [20].
Revised: Most of those methods formulate pre-text tasks, including cross-modality correspondence [6, 7, 8, 10], cross-modality clustering [18], cross-modality longer context prediction [19], or a combination of multiple pre-text tasks [20].

Original: Contrastive learning [3, 21] has recently emerged as one dominant approach in self-supervised learning.
Revised: Contrastive learning [3, 21] has recently emerged as one dominant approach in self-supervised learning.

Original: In multimodal pre-training, the temporal co-occurrence naturally supplies the positive sample to contrast with.
Revised: In multimodal pre-training, the temporal co-occurrence naturally supplies the positive sample to contrast with.

Original: [5] adopts a multimodal versatile network to embed each modality into the same vector space, that is trained from end to end with multimodal pairwise contrastive losses (video-audio, and video text).
Revised: [5] adopts a multimodal versatile network to embed each modality into the same vector space, that is trained from end to end with multimodal pairwise contrastive losses (video-audio, and video text).

Original: CrossCLR [9] further takes the intra-modality similarities into account.
Revised: CrossCLR [9] further takes the intra-modality similarities into account.

Original: [10] generalizes the instance discrimination idea to design a cross-modal discrimination task, i.e., predicting which audio matches a video.
Revised: [10] generalizes the instance discrimination idea to design a cross-modal discrimination task, i.e., predicting which audio matches a video.

Original: Most recently, VATT [11] combines the strength of convolution-free Transformer and multimodal contrastive learning.
Revised: Most recently, VATT [11] combines the strength of convolution-free Transformer and multimodal contrastive learning.

Original: Speciﬁcally, it ﬁrst validate the idea of using a modality-agnostic single-backbone transformer, to work on video, audio and text modalities, it follows the exact BERT [22] and ViT [23] architecture, except injecting modalityspeciﬁc tokenization layers and linear projections.
Revised: Speciﬁcally, it ﬁrst validate the idea of using a modality-agnostic single-backbone transformer, to work on video, audio and text modalities, it follows the exact BERT [22] and ViT [23] architecture, except injecting modalityspeciﬁc tokenization layers and linear projections.

Original: Despite their success, existing multimodal contrastive learning methods [5, 9, 11] hinge on the “free" cross-modalities correspondence mined through the multimodal contrastive loss, i.e., the CMA assumption.
Revised: Despite their success, existing multimodal contrastive learning methods [5, 9, 11] hinge on the “free" cross-modalities correspondence mined through the multimodal contrastive loss, i.e., the CMA assumption.

Original: As a result, they are often trained with well curated or narrated video datasets, such as AudioSet [24], YouCook2 [15], and HowTo100M [12].
Revised: As a result, they are often trained with well curated or narrated video datasets, such as AudioSet [24], YouCook2 [15], and HowTo100M [12].

Original: It remains questionable whether such
Revised: It remains questionable whether such

Original: CMA-guided contrastive learning can generalize well to larger-scale multimodal data in the wild.
Revised: CMA-guided contrastive learning can generalize well to larger-scale multimodal data in the wild.

Original: In fact, even in the HowTo100M dataset, the authors of [12] estimate that around 50% of clip-narration pairs are not well aligned.
Revised: In fact, even in the HowTo100M dataset, the authors of [12] estimate that around 50% of clip-narration pairs are not well aligned.

Original: The validity of CMA has arisen concerns.
Revised: The validity of CMA has arisen concerns.

Original: Most related to us is [13] which explicitly models the misalignment noise using the MIL-NCE loss, that is inherited by VATT [11].
Revised: Most related to us is [13] which explicitly models the misalignment noise using the MIL-NCE loss, that is inherited by VATT [11].

Original: 2.2 Multi-task versus Multimodal Learning
Revised: 2.2 Multi-task versus Multimodal Learning

Original: While earlier self-supervised works used different modality-speciﬁc encoders, the latest multimodal pre-training sees a tendency to explore a versatile and “universal" model shared across modalities [5, 11].
Revised: While earlier self-supervised works used different modality-speciﬁc encoders, the latest multimodal pre-training sees a tendency to explore a versatile and “universal" model shared across modalities [5, 11].

Original: It is easy to notice the resemblance between multimodal learning (especially with a uniﬁed backbone) and multi-task learning [25, 26].
Revised: It is easy to notice the resemblance between multimodal learning (especially with a uniﬁed backbone) and multi-task learning [25, 26].

Original: The later often assumes multiple tasks to share transferable knowledge and can help each other learn.
Revised: The later often assumes multiple tasks to share transferable knowledge and can help each other learn.

Original: Yet in practice, different tasks can be heterogeneous in nature too, often leading to heavily misaligned gradient scales [27] or directions [17].
Revised: Yet in practice, different tasks can be heterogeneous in nature too, often leading to heavily misaligned gradient scales [27] or directions [17].

Original: Such conﬂicting gradients from multiple losses would lead to several undesirable effects, including (i) strong bias
Revised: Such conﬂicting gradients from multiple losses would lead to several undesirable effects, including (i) strong bias

Original: [Figure]
Revised: [Figure]

Original: towards learning some tasks with largest gradient scales; and (2) slow and unstable convergence.
Revised: towards learning some tasks with largest gradient scales; and (2) slow and unstable convergence.

Original: Similar problems are often faced by multimodal pre-training too.
Revised: Similar problems are often faced by multimodal pre-training too.

Original: As a result, several works have been developed to mitigate the gradient dilemma in multi-task learning.
Revised: As a result, several works have been developed to mitigate the gradient dilemma in multi-task learning.

Original: Methods such as gradient normalization [27] and adaptive loss weight [28, 29] address the gradient scale/learning speed imbalance issues.
Revised: Methods such as gradient normalization [27] and adaptive loss weight [28, 29] address the gradient scale/learning speed imbalance issues.

Original: Gradient surgery [17] tries to project gradients with conﬂicting directions onto the normal plane of each other, to prevent each gradient’s interfering component from being applied to the network.
Revised: Gradient surgery [17] tries to project gradients with conﬂicting directions onto the normal plane of each other, to prevent each gradient’s interfering component from being applied to the network.

Original: Later, Gradient vaccine [30] further generalizes this idea extending the gradient projection to arbitrary angle, and uses the exponential moving average of cosine similarity to measure the alignment between different tasks.
Revised: Later, Gradient vaccine [30] further generalizes this idea extending the gradient projection to arbitrary angle, and uses the exponential moving average of cosine similarity to measure the alignment between different tasks.

Original: When it comes to the multimodal learning, [31] investigates optimally blending modality-speciﬁc gradients by linearly re-scaling them.
Revised: When it comes to the multimodal learning, [31] investigates optimally blending modality-speciﬁc gradients by linearly re-scaling them.

Original: Meanwhile, to our best knowledge, there has been no prior study on modality-speciﬁc gradient directions.
Revised: Meanwhile, to our best knowledge, there has been no prior study on modality-speciﬁc gradient directions.

Original:  Methodology
Revised:  Methodology

Original: Pre-requisites: Without loss of generality, we focus on the SOTA multimodal contrastive learning model, VATT [11], as our default subject of study; and we follow its modality-agnostic singlebackbone setting due to its compelling performance-efﬁciency trade-off, and due to the fact that it is the most intuitive setting to modify gradients (shared between paired modalities) w.r.t different end-point objectives.
Revised: Pre-requisites: Without loss of generality, we focus on the SOTA multimodal contrastive learning model, VATT [11], as our default subject of study; and we follow its modality-agnostic singlebackbone setting due to its compelling performance-efﬁciency trade-off, and due to the fact that it is the most intuitive setting to modify gradients (shared between paired modalities) w.r.t different end-point objectives.

Original: In our modality-agnostic single-backbone setting of VATT, the goal of pre-training is to ﬁnd the parameter θ of the model f θ to learn meaningful features for all target modalities (video, audio, text), usually measured by a set of pairwise similarity metrics.
Revised: In our modality-agnostic single-backbone setting of VATT, the goal of pre-training is to ﬁnd the parameter θ of the model f θ to learn meaningful features for all target modalities (video, audio, text), usually measured by a set of pairwise similarity metrics.

Original: VATT uses two pairwise contrastive losses for video-audio and video-text respectively to solve cross-modal alignment:
Revised: VATT uses two pairwise contrastive losses for video-audio and video-text respectively to solve cross-modal alignment:

Original: [Equation]
Revised: [Equation]

Original: where CMA va and CMA vt penalize the cross-modal alignment for video-audio and video-text pairs, using the Noise-Contrastive Estimation objective [32], respectively.
Revised: where CMA va and CMA vt penalize the cross-modal alignment for video-audio and video-text pairs, using the Noise-Contrastive Estimation objective [32], respectively.

Original: Figure 2 illustrates the VATT pipeline.
Revised: Figure 2 illustrates the VATT pipeline.

Original: First, video-audio-text triplets are sampled from random temporal locations, and then video-text and video-audio pairs are formed accordingly.
Revised: First, video-audio-text triplets are sampled from random temporal locations, and then video-text and video-audio pairs are formed accordingly.

Original: Positive pairs are formed by selecting the two modalities at the same temporal locations from the same video clip; while the negative pairs are obtained by randomly sampling any video, audio, or text from other video clips.
Revised: Positive pairs are formed by selecting the two modalities at the same temporal locations from the same video clip; while the negative pairs are obtained by randomly sampling any video, audio, or text from other video clips.

Original: We follow the convention in [5, 11] to use the vanilla NCE loss for video-audio pairs, and to use the MIL-NCE loss proposed in [13] to align video-text pairs.
Revised: We follow the convention in [5, 11] to use the vanilla NCE loss for video-audio pairs, and to use the MIL-NCE loss proposed in [13] to align video-text pairs.

Original: Hence, CMA va and CMA vt are deﬁned as follows:
Revised: Hence, CMA va and CMA vt are deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where N is the pool of negative pairs, P k ( z t ) denotes the k -neighbouring narrations surrounding the original text z t .
Revised: where N is the pool of negative pairs, P k ( z t ) denotes the k -neighbouring narrations surrounding the original text z t .

Original: The gradients of two pairwise losses CMA va and CMA vt , denoted as g va and g vt , will be together applied (i.e. simple adding) to updating the model f θ .
Revised: The gradients of two pairwise losses CMA va and CMA vt , denoted as g va and g vt , will be together applied (i.e. simple adding) to updating the model f θ .

Original: 3.1 Observation of Conﬂicting Gradients
Revised: 3.1 Observation of Conﬂicting Gradients

Original: While qualitatively recognizing the cross-modal misalignment is easy, it is a highly challenging endeavor to quantify the misalignment and to automatically ﬁnd misaligned triplets at scale, due to the vaguely deﬁned criterion as well as the absence of labels [13, 14].
Revised: While qualitatively recognizing the cross-modal misalignment is easy, it is a highly challenging endeavor to quantify the misalignment and to automatically ﬁnd misaligned triplets at scale, due to the vaguely deﬁned criterion as well as the absence of labels [13, 14].

Original: Therefore, the foremost need is to identify a surrogate indicator on how noisy the alignment of a sample triplet is, without making unrealistic assumptions nor incurring much computational overhead.
Revised: Therefore, the foremost need is to identify a surrogate indicator on how noisy the alignment of a sample triplet is, without making unrealistic assumptions nor incurring much computational overhead.

Original: Our empirical choice of surrogate indicator is the directional alignability (i.e., vector angle) of gradients generated by different pairwise cross-modal losses, i.e., the cosine similarity between g va and g vt in the particular example of VATT, after ﬂattening them into two vectors.
Revised: Our empirical choice of surrogate indicator is the directional alignability (i.e., vector angle) of gradients generated by different pairwise cross-modal losses, i.e., the cosine similarity between g va and g vt in the particular example of VATT, after ﬂattening them into two vectors.

Original: The choice is ﬁrstly rationalized by the coherent gradient hypothesis [33], that healthy model training should witness the overall gradient stronger in certain directions along which the samples’ gradients reinforce each other.
Revised: The choice is ﬁrstly rationalized by the coherent gradient hypothesis [33], that healthy model training should witness the overall gradient stronger in certain directions along which the samples’ gradients reinforce each other.

Original: Secondly, comparing the gradient directions between multiple pairwise losses could be considered as an ensemble or “cross-check": intuitively, if both video-audio and video-text pairwise consistency lead to the same update direction, then there is a good chance that those modalities are well aligned and the update direction is reliable; otherwise, at least one pair (video-audio, or video-text) might suffer from misalignment and provides noisy cross-modality guidance.
Revised: Secondly, comparing the gradient directions between multiple pairwise losses could be considered as an ensemble or “cross-check": intuitively, if both video-audio and video-text pairwise consistency lead to the same update direction, then there is a good chance that those modalities are well aligned and the update direction is reliable; otherwise, at least one pair (video-audio, or video-text) might suffer from misalignment and provides noisy cross-modality guidance.

Original: Overall, we conjecture that: aligned video-text-audio triplets should have higher cosine similarity for g va and g vt , and vice versa .
Revised: Overall, we conjecture that: aligned video-text-audio triplets should have higher cosine similarity for g va and g vt , and vice versa .

Original: To further validate the conjecture, we conduct a sanity check, where we started from a pre-trained VATT network and further optimize it for 200 iterations with a batch size of 64, on Youtube8M dataset.
Revised: To further validate the conjecture, we conduct a sanity check, where we started from a pre-trained VATT network and further optimize it for 200 iterations with a batch size of 64, on Youtube8M dataset.

Original: We then randomly sample video-text-audio triplets out of the samples with top 5% and bottom 5% most aligned gradient, measured by cos ( g (cid:48) va , g (cid:48) va ) .
Revised: We then randomly sample video-text-audio triplets out of the samples with top 5% and bottom 5% most aligned gradient, measured by cos ( g (cid:48) va , g (cid:48) va ) .

Original: In Figure 3, we visualize 10 frames within a 32-frame video clip and its corresponding text narration (includingneighbouring text narrations similar to [13]).
Revised: In Figure 3, we visualize 10 frames within a 32-frame video clip and its corresponding text narration (includingneighbouring text narrations similar to [13]).

Original: We visually observed that the top 5% group triplets has more semantically aligned words in the corresponding text narration (highlighted in green ) thus enjoy a better cross-modality alignment, while the bottom 5% group triplets has fewer semantically aligned words, therefore are much more noisy in their alignments.
Revised: We visually observed that the top 5% group triplets has more semantically aligned words in the corresponding text narration (highlighted in green ) thus enjoy a better cross-modality alignment, while the bottom 5% group triplets has fewer semantically aligned words, therefore are much more noisy in their alignments.

Original: We include more visualization samples in the supplementary.
Revised: We include more visualization samples in the Appendix.

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original: [Equation] [Figure] [Table]
Revised: [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table][Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: your partner and ask for help when you need it learn the wisdom of…before this benefit then the break remember that it meets will love of hash browns…keenan willie cindy and donnie in the…smothered and covered and ice is the basis of any lasting…this union we call marriage and they ask for your blessings on this joyous day…now this is especially true for you girls look for the best of your beloved…w co l today <unk> ready to pledge their love to one another
Revised: your partner and ask for help when you need it learn the wisdom of…before this benefit then the break remember that it meets will love of hash browns…keenan willie cindy and donnie in the…smothered and covered and ice is the basis of any lasting…this union we call marriage and they ask for your blessings on this joyous day…now this is especially true for you girls look for the best of your beloved…w co l today <unk> ready to pledge their love to one another

Original: [Figure]
Revised: [Figure]

Original: With the assumption that misalignment in gradients correlates to the noisiness of the sample’s cross-modality alignment, we use the former as the computable surrogate indicator of the later, and examine the cosine similarity between gradient g va and g vt (over 500k iterations) of VATT on the
Revised: With the assumption that misalignment in gradients correlates to the noisiness of the sample’s cross-modality alignment, we use the former as the computable surrogate indicator of the later, and examine the cosine similarity between gradient g va and g vt (over 500k iterations) of VATT on the

Original: HowTo100M dataset [12].
Revised: HowTo100M dataset [12].

Original: In Figure 4a, we plot the distribution of cosine similarities, cos ( g va , g vt ) , across 500k iterations of pre-training.
Revised: In Figure 4a, we plot the distribution of cosine similarities, cos ( g va , g vt ) , across 500k iterations of pre-training.

Original: We observe that cos ( g va , g vt ) at any iteration resembles a normal distribution, and about half of the gradients g va and g va have misaligned directions (negative cosine similarities).
Revised: We observe that cos ( g va , g vt ) at any iteration resembles a normal distribution, and about half of the gradients g va and g va have misaligned directions (negative cosine similarities).

Original: The plot echos the empirical ﬁnding in [15] that around 50% of video-text pairs are not well-aligned on the HowTo100M dataset.
Revised: The plot echos the empirical ﬁnding in [15] that around 50% of video-text pairs are not well-aligned on the HowTo100M dataset.

Original: In the supplementary, we have included similar plots for the YouTube8M dataset [1].
Revised: In the Appendix, we have included similar plots for the YouTube8M dataset [1].

Original: Comparing those plots verify that non-narrative datasets have much worse misalignment than narrative ones, hence challenging the CMA assumption even more.
Revised: Comparing those plots verify that non-narrative datasets have much worse misalignment than narrative ones, hence challenging the CMA assumption even more.

Original: [Figure]
Revised: [Figure]

Original: The existence of conﬂicting gradients not only makes the convergence harder, but also makes the learned representation heavily overﬁt some modalities.
Revised: The existence of conﬂicting gradients not only makes the convergence harder, but also makes the learned representation heavily overﬁt some modalities.

Original: In our case, we observe the training to be biased towards video/audio modalities, which favors the downstream performance of video/audio classiﬁcation task, yet sets back the performance of video-to-text retrieval task (veriﬁed in Table 1).
Revised: In our case, we observe the training to be biased towards video/audio modalities, which favors the downstream performance of video/audio classiﬁcation task, yet sets back the performance of video-to-text retrieval task (veriﬁed in Table 1).

Original: 3.2 Harmonizing Gradients during Pre-training
Revised: 3.2 Harmonizing Gradients during Pre-training

Original: We present two strategies to mitigate the conﬂicting gradients: a “hard" way to project gradient directions, and a “soft" way to progressively select sample triplets in curriculum.
Revised: We present two strategies to mitigate the conﬂicting gradients: a “hard" way to project gradient directions, and a “soft" way to progressively select sample triplets in curriculum.

Original: 3.2.1 Cross-Modality Gradient Realignment
Revised: 3.2.1 Cross-Modality Gradient Realignment

Original: Algorithm 1 Cross-Modality Gradient Realignment
Revised: Algorithm 1 Cross-Modality Gradient Realignment

Original: Require: Model parameter θ , minibatchs B va , minibatchs
Revised: Require: Model parameter θ , minibatchs B va , minibatchs

Original: B vt for ( b va , b vt ) ∈ ( B va , B vt ) do g va ← ∇ θ CMA va ( θ ) , g vt ← ∇ θ CMA vt ( θ ) g va ← ﬂatten ( g va ) , g vt ← ﬂatten ( g vt )
Revised: B vt for ( b va , b vt ) ∈ ( B va , B vt ) do g va ← ∇ θ CMA va ( θ ) , g vt ← ∇ θ CMA vt ( θ ) g va ← ﬂatten ( g va ) , g vt ← ﬂatten ( g vt )

Original: ˆ g va ← g va , ˆ g vt ← g vt if g va · g vt < 

Revised: ˆ g va ← g va , ˆ g vt ← g vt if g va · g vt <

Original: then ˆ g va ← ˆ g va − ˆ g va · g vt || g vt ||(cid:46) projection g vt ← g va ˆ g vt ← ˆ g vt − ˆ g vt · g va || g va ||(cid:46) projection g va ← g vt end if ˆ g va ← reshape (ˆ g va ) , ˆ g vt ← reshape (ˆ g vt ) ∆ θ ← ˆ g vt + ˆ g va (cid:46) sum up gradients update θ with
Revised: then ˆ g va ← ˆ g va − ˆ g va · g vt || g vt ||(cid:46) projection g vt ← g va ˆ g vt ← ˆ g vt − ˆ g vt · g va || g va ||(cid:46) projection g va ← g vt end if ˆ g va ← reshape (ˆ g va ) , ˆ g vt ← reshape (ˆ g vt ) ∆ θ ← ˆ g vt + ˆ g va (cid:46) sum up gradients update θ with

Original: [Table]
Revised: [Table]

Original: end for
Revised: end for

Original: Since g va and g vt are misaligned in terms of their directions, one natural idea is to re-align the crossmodality gradients, by enforcing or re-projecting their directions to be aligned.
Revised: Since g va and g vt are misaligned in terms of their directions, one natural idea is to re-align the crossmodality gradients, by enforcing or re-projecting their directions to be aligned.

Original: An intuitive choice is an effective remedy developed for multitask learning named Gradient Surgery [17].
Revised: An intuitive choice is an effective remedy developed for multitask learning named Gradient Surgery [17].

Original: Here, instead of applying it to g i and g j where i, j are the two heterogeneous tasks, we apply it to g va and g vt , which refer to the gradients of the two pairwise losses CMA va and
Revised: Here, instead of applying it to g i and g j where i, j are the two heterogeneous tasks, we apply it to g va and g vt , which refer to the gradients of the two pairwise losses CMA va and

Original: CMA vt w.r.t the weights in the model, following our rationale in Section 3.1.
Revised: CMA vt w.r.t the weights in the model, following our rationale in Section 3.1.

Original: If the gradients g va and g vt have negative cosine similarity, we alter them by projecting each onto the normal plane of the other.
Revised: If the gradients g va and g vt have negative cosine similarity, we alter them by projecting each onto the normal plane of the other.

Original: Otherwise, if g va and g vt have zero or positive similarity, we retain the original gradients.
Revised: Otherwise, if g va and g vt have zero or positive similarity, we retain the original gradients.

Original: We refer to this particularly adapted form of gradient surgery as Cross-Modality Gradient
Revised: We refer to this particularly adapted form of gradient surgery as Cross-Modality Gradient

Original: Realignment .
Revised: Realignment .

Original: Its update procedure is outlined in Algorithm 1, and the overhead very cheap, mainly just computing inner products of two ﬂattened gradients.
Revised: Its update procedure is outlined in Algorithm 1, and the overhead very cheap, mainly just computing inner products of two ﬂattened gradients.

Original: We also tried another similar algorithm named Gradient Vaccine [30], and the results are almost identical to using Gradient Surgery [17].
Revised: We also tried another similar algorithm named Gradient Vaccine [30], and the results are almost identical to using Gradient Surgery [17].

Original: We hence only report one for the sake of conciseness.
Revised: We hence only report one for the sake of conciseness.

Original: 3.2.2 Gradient-based Curriculum Learning
Revised: 3.2.2 Gradient-based Curriculum Learning

Original: We next propose a gradient-based curriculum learning strategy that gradually identiﬁes and removes more misaligned samples based on the similarity indicator as the training goes on.
Revised: We next propose a gradient-based curriculum learning strategy that gradually identiﬁes and removes more misaligned samples based on the similarity indicator as the training goes on.

Original: Formally, given a video-audio pair b va and video-text pair b vt , we only update the parameter if cos ( g va , g vt ) > γ and drop this sample triplet otherwise.
Revised: Formally, given a video-audio pair b va and video-text pair b vt , we only update the parameter if cos ( g va , g vt ) > γ and drop this sample triplet otherwise.

Original: This ensures us to gradually prioritize training with well-aligned samples while suppressing the noisy supervision from potentially misaligned samples.
Revised: This ensures us to gradually prioritize training with well-aligned samples while suppressing the noisy supervision from potentially misaligned samples.

Original: The drop rate here is controlled by a cut-off threshold γ .
Revised: The drop rate here is controlled by a cut-off threshold γ .

Original: In general, γ is a negative value monotonically increasing during training, and there is an underlying rationale.
Revised: In general, γ is a negative value monotonically increasing during training, and there is an underlying rationale.

Original: Algorithm 2 Gradient-based Curriculum Learning
Revised: Algorithm 2 Gradient-based Curriculum Learning

Original: Require: Model parameter θ , minibatchs B va , minibatchs
Revised: Require: Model parameter θ , minibatchs B va , minibatchs

Original: B vt , initial γfor ( b va , b vt ) ∈
Revised: B vt , initial γfor ( b va , b vt ) ∈

Original: ( B va , B vt ) do update γ (cid:46) curriculumly update γ g va ← ∇ θ CMA va ( θ ) , g vt ← ∇ θ CMA vt ( θ ) g va ← ﬂatten ( g va ) , g vt ← ﬂatten ( g vt ) if g va · g vt > γ then g va ← reshape ( g va ) , g vt ← reshape ( g vt ) ∆ θ ← g va + g vt (cid:46) sum up gradients update θ with ∆ θ (cid:46) update parameter end if end for
Revised: ( B va , B vt ) do update γ (cid:46) curriculumly update γ g va ← ∇ θ CMA va ( θ ) , g vt ← ∇ θ CMA vt ( θ ) g va ← ﬂatten ( g va ) , g vt ← ﬂatten ( g vt ) if g va · g vt > γ then g va ← reshape ( g va ) , g vt ← reshape ( g vt ) ∆ θ ← g va + g vt (cid:46) sum up gradients update θ with ∆ θ (cid:46) update parameter end if end for

Original: Initially, the network is under-trained and the similarity between its learned features is not reliable to indicate a strong CMA; hence we are more conservative in excluding samples.
Revised: Initially, the network is under-trained and the similarity between its learned features is not reliable to indicate a strong CMA; hence we are more conservative in excluding samples.

Original: As the network becomes more sufﬁciently trained, the features become more semantically aligned, allowing us to remove more misaligned samples in the later stage of training.
Revised: As the network becomes more sufﬁciently trained, the features become more semantically aligned, allowing us to remove more misaligned samples in the later stage of training.

Original: The full procedure of Gradient-based Curriculum
Revised: The full procedure of Gradient-based Curriculum

Original: Learning is outlined in Algorithm 2.
Revised: Learning is outlined in Algorithm 2.

Original: We empirically choice γ 0 = − 0 .
Revised: We empirically choice γ 0 = − 0 .

Original: 3 at the beginning and linearly increase it to γ 500 K = 0 at the end, we include an ablation on the adaptive choice of γ in the Table 2.
Revised: 3 at the beginning and linearly increase it to γ 500 K = 0 at the end, we include an ablation on the adaptive choice of γ in the Table 2.

Original: Similar classes of ideas, i.e, “gradually focusing a small training subset", have been explored by a prominent class of algorithms which rely on sample selection to robustify [34, 35, 36] or to accelerate [37] training.
Revised: Similar classes of ideas, i.e, “gradually focusing a small training subset", have been explored by a prominent class of algorithms which rely on sample selection to robustify [34, 35, 36] or to accelerate [37] training.

Original: Most of them use the “small loss trick" wherein a fraction of samples with smallest loss values below a certain threshold are considered as reliable.
Revised: Most of them use the “small loss trick" wherein a fraction of samples with smallest loss values below a certain threshold are considered as reliable.

Original: However, they did not explore gradient conﬂict indicators, which targets solving our CMA problem here.
Revised: However, they did not explore gradient conﬂict indicators, which targets solving our CMA problem here.

Original: Also note that our two steps can fully reuse the cosine similarity computed, so applying them together incurs no extra overhead.
Revised: Also note that our two steps can fully reuse the cosine similarity computed, so applying them together incurs no extra overhead.

Original:  Experiments 4.1 Pre-Training Datasets and Settings
Revised:  Experiments 4.1 Pre-Training Datasets and Settings

Original: In this paper, we focus on studying gradients in a modality-agnostic single-backbone setting, thus we do not consider multimodal pre-training baseline with modality-speciﬁc models, e.g. HERO[38],
Revised: In this paper, we focus on studying gradients in a modality-agnostic single-backbone setting, thus we do not consider multimodal pre-training baseline with modality-speciﬁc models, e.g. HERO[38],

Original: MMV[5], PerceiverIO[39], AVLnet[40].
Revised: MMV[5], PerceiverIO[39], AVLnet[40].

Original: Apart from VATT[11], there exist other concurrent work that use modality-agnostic setting[20, 41, 42], yet they all have their respective limitations.
Revised: Apart from VATT[11], there exist other concurrent work that use modality-agnostic setting[20, 41, 42], yet they all have their respective limitations.

Original: For example, UniVL[20] lacks audio modalities thus cannot constitute video-audio CMA loss, VLM[41] applies separate attention mask for each modality while PolyViT[42] use alternate training between modalities and tasks, thus eliminate the possibilities of any gradients conﬂicts.
Revised: For example, UniVL[20] lacks audio modalities thus cannot constitute video-audio CMA loss, VLM[41] applies separate attention mask for each modality while PolyViT[42] use alternate training between modalities and tasks, thus eliminate the possibilities of any gradients conﬂicts.

Original: Therefore, this leave
Revised: Therefore, this leave

Original: VATT[11] to be the most suitable baseline besides it SoTA performance.
Revised: VATT[11] to be the most suitable baseline besides it SoTA performance.

Original: Our pre-training adopts three settings for comprehensive comparison: (1) using HowTo100M as our training set, following [20, 12]; (2) combining HowTo100M and AudioSet as our training set following [5] and VATT [11]; and (3) additionally merging a noisy
Revised: Our pre-training adopts three settings for comprehensive comparison: (1) using HowTo100M as our training set, following [20, 12]; (2) combining HowTo100M and AudioSet as our training set following [5] and VATT [11]; and (3) additionally merging a noisy

Original: Youtube8M dataset alongside with HowTo100M and AudioSet, to create a much more compound and complicated training set than considered before, to stretch-test our proposal’s beneﬁts to scale up pre-training in uncurated and poorly aligned data.
Revised: Youtube8M dataset alongside with HowTo100M and AudioSet, to create a much more compound and complicated training set than considered before, to stretch-test our proposal’s beneﬁts to scale up pre-training in uncurated and poorly aligned data.

Original: For each setting, we i.i.d sample video-audio-text clips from the mixture of all candidate sets in every batch.
Revised: For each setting, we i.i.d sample video-audio-text clips from the mixture of all candidate sets in every batch.

Original: (HowTo100M, Audioset or Youtube8M, if applicable).
Revised: (HowTo100M, Audioset or Youtube8M, if applicable).

Original: For all setting, we only use a subset of HowTo100M, AudioSet, Youtube8M, Kinetics400 and Youcook2 dataset in compliance with Youtube’s wipe-out policies.
Revised: For all setting, we only use a subset of HowTo100M, AudioSet, Youtube8M, Kinetics400 and Youcook2 dataset in compliance with Youtube’s wipe-out policies.

Original: HowTo100M 1 [12] is a large-scale dataset of narrated videos with an emphasis on instructional videos that have well synchronized modalities.
Revised: HowTo100M 1 [12] is a large-scale dataset of narrated videos with an emphasis on instructional videos that have well synchronized modalities.

Original: The training set consists of 127 Millions video clips with the corresponding close-captioned text from automatic speech recognition (ASR).
Revised: The training set consists of 127 Millions video clips with the corresponding close-captioned text from automatic speech recognition (ASR).

Original: AudioSet 1 [24] is a large-scale audio-visual dataset originally intended for audio event classiﬁcation.
Revised: AudioSet 1 [24] is a large-scale audio-visual dataset originally intended for audio event classiﬁcation.

Original: It consists of 1.78 Million 10-second clips sampled from Youtube Videos that contains a variety of audio tracks, such as musical instruments, animals or mechanical sounds.
Revised: It consists of 1.78 Million 10-second clips sampled from Youtube Videos that contains a variety of audio tracks, such as musical instruments, animals or mechanical sounds.

Original: Youtube8M 1 [1] is a large-scale video classiﬁcation dataset based on Youtube videos.
Revised: Youtube8M 1 [1] is a large-scale video classiﬁcation dataset based on Youtube videos.

Original: It consists of a diverse vocabulary of 3862 visual entities and a total number of 8 Million videos.
Revised: It consists of a diverse vocabulary of 3862 visual entities and a total number of 8 Million videos.

Original: To the best of our knowledge, we are the ﬁrst to scale contrastive multimodal pre-training beyond instructional videos, onto this noisy Youtube8M videos that more faithfully represent videos the wild.
Revised: To the best of our knowledge, we are the ﬁrst to scale contrastive multimodal pre-training beyond instructional videos, onto this noisy Youtube8M videos that more faithfully represent videos the wild.

Original: For data preprocessing, we split each video into 5s clips and use ASR close-captions to generate the corresponding text, resulting in 165 Millions video-audio-text clips.
Revised: For data preprocessing, we split each video into 5s clips and use ASR close-captions to generate the corresponding text, resulting in 165 Millions video-audio-text clips.

Original: 4.2 Pre-Training Settings
Revised: 4.2 Pre-Training Settings

Original: Network Architecture: For all of our experiments, we use modality-agnostic Transformer variant in [11], VATT-MA-Medium.
Revised: Network Architecture: For all of our experiments, we use modality-agnostic Transformer variant in [11], VATT-MA-Medium.

Original: Speciﬁcally, we use a 12-layer Transformer, in which each layer has a feed-forward MLP with hidden size of 4096, and 16 attention heads with a hidden size of 1024.
Revised: Speciﬁcally, we use a 12-layer Transformer, in which each layer has a feed-forward MLP with hidden size of 4096, and 16 attention heads with a hidden size of 1024.

Original: Pre-training Hyperparameter: We strictly follow the setting in [11], pre-training VATT from scratch with Adam optimizer with an initial learning rate of 1e-4, 10k warmup steps, 500k steps in total, a batch size of 2048 and using a cosine learning rate scheduler to anneal the learning rate from 1e-4 to 5e-5.
Revised: Pre-training Hyperparameter: We strictly follow the setting in [11], pre-training VATT from scratch with Adam optimizer with an initial learning rate of 1e-4, 10k warmup steps, 500k steps in total, a batch size of 2048 and using a cosine learning rate scheduler to anneal the learning rate from 1e-4 to 5e-5.

Original: Our framework is implemented in Tensorﬂow 2.8, and train with 256 TPUV3s, it took a total of 3 days to train our models.
Revised: Our framework is implemented in Tensorﬂow 2.8, and train with 256 TPUV3s, it took a total of 3 days to train our models.

Original: 4.3 Downstream Tasks for Evaluation
Revised: 4.3 Downstream Tasks for Evaluation

Original: We evaluate the performance of our pre-trained representations when tuned towards various downstream tasks, including a variety of video/audio and cross modalities tasks.
Revised: We evaluate the performance of our pre-trained representations when tuned towards various downstream tasks, including a variety of video/audio and cross modalities tasks.

Original: Video Action Recognition: Following [5], we evaluate on UCF101[43] (101 action classes, 13, videos) and the HMDB51[44] (51 classes, 6,766 videos) benchmarks.
Revised: Video Action Recognition: Following [5], we evaluate on UCF101[43] (101 action classes, 13, videos) and the HMDB51[44] (51 classes, 6,766 videos) benchmarks.

Original: For both settings, we directly attach a linear classiﬁer on the learned representations while freezing the transformer backbone.
Revised: For both settings, we directly attach a linear classiﬁer on the learned representations while freezing the transformer backbone.

Original: We also evaluate on Kinetics400 1 [45] dataset (400 classes, 234,584 video), where we ﬁne-tune the transformer backbone.
Revised: We also evaluate on Kinetics400 1 [45] dataset (400 classes, 234,584 video), where we ﬁne-tune the transformer backbone.

Original: Audio Event Classiﬁcation: Following [5]
Revised: Audio Event Classiﬁcation: Following [5]

Original: , we used ESC50[46] (50 classes, 2000 audio clips) as the benchmark for audio modalities.
Revised: , we used ESC50[46] (50 classes, 2000 audio clips) as the benchmark for audio modalities.

Original: Similarly, we directly attach and tune a linear classiﬁer on top of the frozen transformer backbone.
Revised: Similarly, we directly attach and tune a linear classiﬁer on top of the frozen transformer backbone.

Original: Zero-shot Video-Text Retrieval: To evaluate our performance on cross-modality task, we follow [5] using YouCook2 1 [15] (3.1K video-text clips) and MSR-VTT 1 [47] (1K video-text clips) as our evaluation datasets.
Revised: Zero-shot Video-Text Retrieval: To evaluate our performance on cross-modality task, we follow [5] using YouCook2 1 [15] (3.1K video-text clips) and MSR-VTT 1 [47] (1K video-text clips) as our evaluation datasets.

Original: We directly use the embedding from the pre-trained model without any ﬁnetuning, and report the recall at 10 (R@10) and Median Rank as described in [5].
Revised: We directly use the embedding from the pre-trained model without any ﬁnetuning, and report the recall at 10 (R@10) and Median Rank as described in [5].

Original: 4.4 Result Analysis on HowTo100M and AudioSet Pre-training
Revised: 4.4 Result Analysis on HowTo100M and AudioSet Pre-training

Original: Gradient Re-weighting We set up a simple gradient re-weighting baseline similar to [31], where we simply scale up the gradient magnitude of g va by 2.5 times and scale down the gradient magnitude of its counterpart g vt by 0.5 times, we denote this baseline as Gradient Re-weighting (VA), we also did it vice versa and denoted it as Gradient Re-weighting (VT).
Revised: Gradient Re-weighting We set up a simple gradient re-weighting baseline similar to [31], where we simply scale up the gradient magnitude of g va by 2.5 times and scale down the gradient magnitude of its counterpart g vt by 0.5 times, we denote this baseline as Gradient Re-weighting (VA), we also did it vice versa and denoted it as Gradient Re-weighting (VT).

Original: As shown in Table 1, we did not observe much performance boost with gradient re-weighting, on the contrary, there are signiﬁcant performance drop in Text-Video Retrieval task on both YouCook2 and MSRVTT datasets.
Revised: As shown in Table 1, we did not observe much performance boost with gradient re-weighting, on the contrary, there are signiﬁcant performance drop in Text-Video Retrieval task on both YouCook2 and MSRVTT datasets.

Original: Cross-Modality Gradient Realignment Compared to Gradient Re-weighting, our Cross-Modality
Revised: Cross-Modality Gradient Realignment Compared to Gradient Re-weighting, our Cross-Modality

Original: Gradient Realignment mitigates the gradient conﬂicts by modifying gradients direction.
Revised: Gradient Realignment mitigates the gradient conﬂicts by modifying gradients direction.

Original: In Table 1, we can see signiﬁcant performance gains in video-text retrieval task (e.g. 58% gain in rank metric on
Revised: In Table 1, we can see signiﬁcant performance gains in video-text retrieval task (e.g. 58% gain in rank metric on

Original: YouCook2, 46% gain in R@10 metric on MSR-VTT), while maintaining a comparable performance in video/audio tasks.
Revised: YouCook2, 46% gain in R@10 metric on MSR-VTT), while maintaining a comparable performance in video/audio tasks.

Original: Since video-text retrieval task heavily hinge on cross-modality alignment (CMA), this improvement verify the beneﬁts of re-aligning the cross-modality gradients.
Revised: Since video-text retrieval task heavily hinge on cross-modality alignment (CMA), this improvement verify the beneﬁts of re-aligning the cross-modality gradients.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Gradient-
Revised: Gradient-

Original: Curriculum Learning Compared to Cross-Modality Gradient Realignment,
Revised: Curriculum Learning Compared to Cross-Modality Gradient Realignment,

Original: Gradient-based Curriculum Learning inherents its performance gains in video-text retrieval task, yet enjoy an additional performance boost in audio classiﬁcation (e.g. 1.40% gain ESC50 dataset in
Revised: Gradient-based Curriculum Learning inherents its performance gains in video-text retrieval task, yet enjoy an additional performance boost in audio classiﬁcation (e.g. 1.40% gain ESC50 dataset in

Original: Table 1) and video classiﬁcation (e.g. 1.94% gains UCF101 dataset in Table 1).
Revised: Table 1) and video classiﬁcation (e.g. 1.94% gains UCF101 dataset in Table 1).

Original: We also perform an ablation on the adaptive choice of γ on Howto100M dataset in Table 2.
Revised: We also perform an ablation on the adaptive choice of γ on Howto100M dataset in Table 2.

Original: We can see that curriculumly evolving γ (i.e. ( γ 0 , γ 500 K ) =
Revised: We can see that curriculumly evolving γ (i.e. ( γ 0 , γ 500 K ) =

Original: ( − 0 .
Revised: ( − 0 .

Original: 3 , 0 . 0) ) yields the best overall results, while maintaining a ﬁxed γ (i.e. ( γ 0 , γ 500 K ) =
Revised: 3 , 0 . 0) ) yields the best overall results, while maintaining a ﬁxed γ (i.e. ( γ 0 , γ 500 K ) =

Original: (0 . 0 , 0 . 0) ) yields the worst results, this verify the effectiveness of curriculumly removing mis-aligned samples during training.
Revised: (0 . 0 , 0 . 0) ) yields the worst results, this verify the effectiveness of curriculumly removing mis-aligned samples during training.

Original: Cross-Modality Gradient Realignment + Gradient-based Curriculum Learning
Revised: Cross-Modality Gradient Realignment + Gradient-based Curriculum Learning

Original: We further verify the effectiveness with the combination of both techniques.
Revised: We further verify the effectiveness with the combination of both techniques.

Original: Speciﬁcally, given a video-audio pair and video-text pair, (a) if cos ( g va , g vt ) ≤ γ , we drop this sample triplet; (b) if γ < cos ( g va , g vt ) < 0 , we apply gradient projection in Algorithm 1; (c) if cos ( g va , g vt ) ≥ 0 , we retain the original gradients.
Revised: Speciﬁcally, given a video-audio pair and video-text pair, (a) if cos ( g va , g vt ) ≤ γ , we drop this sample triplet; (b) if γ < cos ( g va , g vt ) < 0 , we apply gradient projection in Algorithm 1; (c) if cos ( g va , g vt ) ≥ 0 , we retain the original gradients.

Original: We found they are complimentary to each other and the combination of both lead to better performance in the majority of downstream tasks.
Revised: We found they are complimentary to each other and the combination of both lead to better performance in the majority of downstream tasks.

Original: 4.5 Scaling Pre-training to YouTube8M
Revised: 4.5 Scaling Pre-training to YouTube8M

Original: As shown in Table 1, adding YouTube8M dataset brings the video classiﬁcation and video-text retrieval performance to a new level (e.g. 88.28% Top1 on UCF101, 56.0 rank on MSRVTT)), yet it set back the performance in audio classiﬁcation task, we conjecture this is largely due to the noisy misaligned samples introduced by Youtube8M dataset.
Revised: As shown in Table 1, adding YouTube8M dataset brings the video classiﬁcation and video-text retrieval performance to a new level (e.g. 88.28% Top1 on UCF101, 56.0 rank on MSRVTT)), yet it set back the performance in audio classiﬁcation task, we conjecture this is largely due to the noisy misaligned samples introduced by Youtube8M dataset.

Original: However, by leveraging our proposed techniques, we further yields a new state-of-the-art performance in the modality-agnostic setting, without any modiﬁcation in architecture, striking 89.70% Top1 on
Revised: However, by leveraging our proposed techniques, we further yields a new state-of-the-art performance in the modality-agnostic setting, without any modiﬁcation in architecture, striking 89.70% Top1 on

Original: UCF101, 92.15% Top5 on HMDB51, 80.01% Top1 on Kinetics400 and 85.00% Top1 on ESC dataset, demonstrating the effectiveness of our method when scaling to noisy Youtub

Revised: UCF101, 92.15% Top5 on HMDB51, 80.01% Top1 on Kinetics400 and 85.00% Top1 on ESC dataset, demonstrating the effectiveness of our method when scaling to noisy Youtub

Original: M dataset.
Revised: M dataset.

Original: 4.6 Feature Visualization
Revised: 4.6 Feature Visualization

Original: To better illustrate why gradient harmonization techniques works , in Figure 5, we follow the visualization used in VATT[11], which take YouCook2 Video-Text Retrieval dataset, showing the output space visualization of video and text, respectively.
Revised: To better illustrate why gradient harmonization techniques works , in Figure 5, we follow the visualization used in VATT[11], which take YouCook2 Video-Text Retrieval dataset, showing the output space visualization of video and text, respectively.

Original: We can see the initial video and text feature representation of VATT baseline are mixed together (which aligned with the ﬁndings in [11]), however, after using gradient realignment, curriculum learning or applying both, the video and text representations become disentangled, this indicates our gradient harmonization techniques adds to cross-modality diversity, which makes video/text more discriminative from each other, making the model more discriminative and better in generalization.
Revised: We can see the initial video and text feature representation of VATT baseline are mixed together (which aligned with the ﬁndings in [11]), however, after using gradient realignment, curriculum learning or applying both, the video and text representations become disentangled, this indicates our gradient harmonization techniques adds to cross-modality diversity, which makes video/text more discriminative from each other, making the model more discriminative and better in generalization.

Original: [Figure]
Revised: [Figure]

Original: 4.7 Extending Gradient Realignment to Modality-Speciﬁc VATT
Revised: 4.7 Extending Gradient Realignment to Modality-Speciﬁc VATT

Original: So far, our techniques were built upon the modality-agnostic setting of VATT with a shared backbone.
Revised: So far, our techniques were built upon the modality-agnostic setting of VATT with a shared backbone.

Original: We also try to apply the same technique to the modality-speciﬁc setting, given there is still a shared video head where we can harmonization the gradients.
Revised: We also try to apply the same technique to the modality-speciﬁc setting, given there is still a shared video head where we can harmonization the gradients.

Original: In Table 3, we observe that applying
Revised: In Table 3, we observe that applying

Original: Gradient Realignment (GR) to modality-speciﬁc setting has some gain over uni-modality tasks (e.g.
Revised: Gradient Realignment (GR) to modality-speciﬁc setting has some gain over uni-modality tasks (e.g.

Original: Video/Audio Classﬁcation), yet set back on cross-modality tasks (e.g. Text-Video Retrieval).
Revised: Video/Audio Classﬁcation), yet set back on cross-modality tasks (e.g. Text-Video Retrieval).

Original: We hypothesize this is due the lack of cross-modality signals: since there is no longer shared backbone across modalities, the only shared video head would provide bias signala towards the video modality, for cross-modality alignment.
Revised: We hypothesize this is due the lack of cross-modality signals: since there is no longer shared backbone across modalities, the only shared video head would provide bias signala towards the video modality, for cross-modality alignment.

Original: We leave the improvement of modality-speciﬁc VATT for future work.
Revised: We leave the improvement of modality-speciﬁc VATT for future work.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we take a deep dive into the common CMA assumption used in multimodal pretraining.
Revised: In this paper, we take a deep dive into the common CMA assumption used in multimodal pretraining.

Original: Using the gradient directions between pairwise losses as the surrogate indicator, we observe ubiquitous gradient conﬂicts even in relatively well-aligned narrative datasets.
Revised: Using the gradient directions between pairwise losses as the surrogate indicator, we observe ubiquitous gradient conﬂicts even in relatively well-aligned narrative datasets.

Original: We then propose two plug-and-play techniques to harmonize the gradients during training.
Revised: We then propose two plug-and-play techniques to harmonize the gradients during training.

Original: They are demonstrated to substantially enhance VATT pre-training across a number of downstream tasks, and further scale it up to pre-training with even more complicated and poorly aligned data.
Revised: They are demonstrated to substantially enhance VATT pre-training across a number of downstream tasks, and further scale it up to pre-training with even more complicated and poorly aligned data.

Original: Our ﬁndings underline the (often overlooked) importance to carefully revisit the basic assumption and training dynamics of multimodal learning, besides advancing the model architectures.
Revised: Our ﬁndings underline the (often overlooked) importance to carefully revisit the basic assumption and training dynamics of multimodal learning, besides advancing the model architectures.

Original: However, our cross-modality gradient harmonization technique still has room to generalize better to the modality-speciﬁc setting.
Revised: However, our cross-modality gradient harmonization technique still has room to generalize better to the modality-speciﬁc setting.

Original: Additionally, we did not fully explore other regularization techniques in multi-tasking learning literature, such as [27, 28, 29].
Revised: Additionally, we did not fully explore other regularization techniques in multi-tasking learning literature, such as [27, 28, 29].

Original: The potential negative societal impacts include the malicious use of multimodal models, such as unintended use of its downstream visual/audio recognition capability.
Revised: The potential negative societal impacts include the malicious use of multimodal models, such as unintended use of its downstream visual/audio recognition capability.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: We give special thanks to Tianqi Liu and Jialu Liu for the constructive feedback and discussions.
Revised: We give special thanks to Tianqi Liu and Jialu Liu for the constructive feedback and discussions.

Original: Z.
Revised: Z.

Original: Wang is supported in part by US Army Research Ofﬁce Young Investigator Award W911NF2010240.
Revised: Wang is supported in part by US Army Research Ofﬁce Young Investigator Award W911NF2010240.

Revised: A Qualitative examples of our proposed measure based on agreement of gradients

Revised: To further validate the conjecture that aligned video-text-audio triplets should have higher cosine similarity for g va and g vt , and vice versa , we conduct a sanity check, where we started from a pre-trained VATT network and further optimize it for 200 iterations with a batch size of 64 on

Revised: Youtube8M dataset, we then randomly sample video-text-audio triplets out of the samples with top 5% and bottom 5% most aligned gradient, measured by cos ( g (cid:48) va , g (cid:48) va ) .

Revised: In Figure 6 and 7, we visually observed that the top 5% group triplets has more semantically aligned words in the corresponding text narration (highlighted in green ) thus enjoy a better cross-modality alignment, while the bottom 5% group triplets has fewer semantically aligned words, therefore are much more noisy in their alignments.

Revised: B Training dynamics of our proposed measure on Youtube8M dataset

Revised: In Figure 8, we plot the distribution of cosine similarities, cos ( g va , g va ) , across 500k iterations of pre-training, on the noisy Youtube8M dataset.

Revised: We observe that cos ( g va , g va ) at any iteration resembles a normal distribution, and about half of the gradients g va and g va have misaligned directions (negative cosine similarities).

Revised: We also calculate the mean of cos ( g va , g va ) across all 500k iterations on Youtube8M dataset, and found it to be 30% smaller than that on Howto100M dataset, indicating stronger mis-alignment in gradient directions on Youtube8M, which further verify that non-narrative video sets such as Youtube8M have more severe misalignment than the narrative ones such as Howto100M, hence challenging the CMA assumption commonly used in multimodal pre-training[11, 13].

Revised: [Table]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: Bottom

Revised: [Table]

Revised: [Figure]

Revised: [Figure]