Original: P ARAMETER E FFICIENT M ULTIMODAL T RANSFORMERS FOR V IDEO R EPRESENTATION L EARNING
Revised: P ARAMETER E FFICIENT M ULTIMODAL T RANSFORMERS FOR V IDEO R EPRESENTATION L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original:  Transformershave delivered impressive results in various language understanding tasks. This has motivated adapting pretrained language Transformers to a multimodal setting, where a new visual model is trained in tandem with the language model.
Revised: The recent success of Transformers in the language domain has motivated adapting it to a multimodal setting, where a new visual model is trained in tandem with an already pretrained language model.

Original: However,  existing work typically ﬁxes thepretrained language model and train only the vision module, which limits its ability to learn cross-modal information. This is partly because of excessive memory requirements from overparameterized multimodal Transformers.
Revised: However, due to the excessive memory requirements from Transformers, existing work typically ﬁxes the language model and train only the vision module, which limits its ability to learn cross-modal information in an end-to-end manner.

Original: In this work, we focus on reducing the parameters of multimodal Transformers in the context of audio-visual video representation learning.
Revised: In this work, we focus on reducing the parameters of multimodal Transformers in the context of audio-visual video representation learning.

Original: We alleviate the high memory requirement by sharing the weights of Transformers across layers and modalities; we decompose the Transformer into modality-speciﬁc and modality-shared parts so that the model learns the dynamics of each modality both individually and together, and propose a novel parameter sharing scheme based on low-rank approximation.
Revised: We alleviate the high memory requirement by sharing the weights of Transformers across layers and modalities; we decompose the Transformer into modality-speciﬁc and modality-shared parts so that the model learns the dynamics of each modality both individually and together, and propose a novel parameter sharing scheme based on low-rank approximation.

Original: We show that our approach reduces parameters up to 80 % , allowing us to train our model end-to-end from scratch.
Revised: We show that our approach reduces parameters up to 80 % , allowing us to train our model end-to-end from scratch.

Revised: We also propose a negative sampling approach based on an instance similarity measured on the CNN embedding space that our model learns with the

Original:  To demonstrate our approach, we pretrain our model on 30-second clips from Kinetics-700 and transfer it to audio-visual classiﬁcation tasks.
Revised: Transformers. To demonstrate our approach, we pretrain our model on 30-second clips from Kinetics-700 and transfer it to audio-visual classiﬁcation tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Learning multimodal representation from unlabeled videos has received considerable attention (Baltrušaitis et al., 2018).
Revised: Learning multimodal representation from unlabeled videos has received considerable attention (Baltrušaitis et al., 2018).

Original: Audio-visual learning is of particular interest due to the abundance of videos with natural audio-visual co-occurrence (Owens & Efros, 2018;
Revised: Audio-visual learning is of particular interest due to the abundance of videos with natural audio-visual co-occurrence (Owens & Efros, 2018;

Original: Owens et al., 2018; Arandjelovic & Zisserman, 2018; Ephrat et al., 2018; Gao & Grauman, 2019; Alwassel et al., 2019).
Revised: Owens et al., 2018; Arandjelovic & Zisserman, 2018; Ephrat et al., 2018; Gao & Grauman, 2019; Alwassel et al., 2019).

Original: However, existing approaches learn localized representations from short videos (hundreds of milliseconds to just under a few seconds), capturing only short-term dependencies in data.
Revised: However, existing approaches learn localized representations from short videos (hundreds of milliseconds to just under a few seconds), capturing only short-term dependencies in data.

Original: While this is useful for certain applications, e.g., source separation (Ephrat et al., 2018) and atomic action recognition (Gu et al., 2018), learning representation that captures long-term dependencies is equally important, e.g., for activity recognition (Kay et al., 2017; Carreira et al., 2019; Sigurdsson et al., 2016).
Revised: While this is useful for certain applications, e.g., source separation (Ephrat et al., 2018) and atomic action recognition (Gu et al., 2018), learning representation that captures long-term dependencies is equally important, e.g., for activity recognition (Kay et al., 2017; Carreira et al., 2019; Sigurdsson et al., 2016).

Original: Unfortunately, processing long videos requires large memory resource and capturing long-term dependencies is a long-standing problem (Hochreiter & Schmidhuber, 1997; Cho et al., 2014; Vaswani et al., 2017).
Revised: Unfortunately, processing long videos requires large memory resource and capturing long-term dependencies is a long-standing problem (Hochreiter & Schmidhuber, 1997; Cho et al., 2014; Vaswani et al., 2017).

Original: In language understanding, strong progress has been made in large-scale learning of contextualized language representations using Transformers (Vaswani et al., 2017; Howard & Ruder, 2018; Peters et al., 2018;
Revised: In language understanding, strong progress has been made in large-scale learning of contextualized language representations using Transformers (Vaswani et al., 2017; Howard & Ruder, 2018; Peters et al., 2018;

Original: Radford et al., 2018; 2019; Devlin et al., 2019; Liu et al., 2019;
Revised: Radford et al., 2018; 2019; Devlin et al., 2019; Liu et al., 2019;

Original: Yang et al., 2019).
Revised: Yang et al., 2019).

Original: Riding on the success of Transformers, several recent works have extended it to the multimodal setting by adding an additional vision module to the Transformer framework (Sun et al., 2019b; Lu et al., 2019).
Revised: Riding on the success of Transformers, several recent works have extended it to the multimodal setting by adding an additional vision module to the Transformer framework (Sun et al., 2019b; Lu et al., 2019).

Original: However, these models are typically not end-to-end trained; they rely on a language-pretrained
Revised: However, these models are typically not end-to-end trained; they rely on a language-pretrained

Original: BERT (Devlin et al., 2019), which is ﬁxed throughout, and train only the visual components.
Revised: BERT (Devlin et al., 2019), which is ﬁxed throughout, and train only the visual components.

Original: While the pretrained BERT helps accelerate convergence and brings reliable extra supervision signal to the vision component, this partial learning setup can be undesirable if the text data comes from different distributions (of topics, dialects, or foreign languages) or if we want to apply it to different modalities (e.g., audio-visual).
Revised: While the pretrained BERT helps accelerate convergence and brings reliable extra supervision signal to the vision component, this partial learning setup can be undesirable if the text data comes from different distributions (of topics, dialects, or foreign languages) or if we want to apply it to different modalities (e.g., audio-visual).

Original: Unfortunately, end-to-end training of such multimodal Transformer architectures is challenging for most existing compute environments due to the excessive memory requirement.
Revised: Unfortunately, end-to-end training of such multimodal Transformer architectures is challenging for most existing compute environments due to the excessive memory requirement.

Original: In this work, we make three key contributions.
Revised: In this work, we make three key contributions.

Original: First, we propose an end-to-end trainable bidirectional transformer architecture that learns contextualized audio-visual representations of long videos.
Revised: First, we propose an end-to-end trainable bidirectional transformer architecture that learns contextualized audio-visual representations of long videos.

Original: Ourmodel, shown in Figure 1, consists of audio/visual CNNs, audio/visual Transformers, and a multi-
Revised: Our

Original: [Figure]
Revised: [Figure]

Original: The quality of negative samples is crucial in contrastive learning, which is part of our learning objective.
Revised: The quality of negative samples is crucial in contrastive learning, which is part of our learning objective.

Original: As our second contribution, we propose a content-aware negative sampling strategy that favors negatives sufﬁciently similar to a positive instance.
Revised: As our second contribution, we propose a content-aware negative sampling strategy that favors negatives sufﬁciently similar to a positive instance.

Original: Our approach measures the similarity by reusing the CNN embeddings obtained during model training and thus do not introduce extra parameters to learn.
Revised: Our approach measures the similarity by reusing the CNN embeddings obtained during model training and thus do not introduce extra parameters to learn.

Original: We show that this improves performance over the standard sampling strategies.
Revised: We show that this improves performance over the standard sampling strategies.

Original: Our third contribution is a systematic evaluation of different modality fusion strategies.
Revised: Our third contribution is a systematic evaluation of different modality fusion strategies.

Original: Existing works on multimodal BERT (all using vision-and-language data) typically apply one fusion strategy without thoroughly comparing with alternatives, e.g., some works perform early fusion (Sun et al., 2019b; Su et al., 2020) while others perform mid-level fusion (Lu et al., 2019; Tan & Bansal, 2019).
Revised: Existing works on multimodal BERT (all using vision-and-language data) typically apply one fusion strategy without thoroughly comparing with alternatives, e.g., some works perform early fusion (Sun et al., 2019b; Su et al., 2020) while others perform mid-level fusion (Lu et al., 2019; Tan & Bansal, 2019).

Original: As a result, it is unclear how different fusion methods affect the ﬁnal performance.
Revised: As a result, it is unclear how different fusion methods affect the ﬁnal performance.

Original: In this work, we compare three fusion strategies (early, mid, late) and show the superiority of mid-level fusion.
Revised: In this work, we compare three fusion strategies (early, mid, late) and show the superiority of mid-level fusion.

Original: To demonstrate our approach, we pretrain our model on long (30-second) video clips from Kinetics700 (Carreira et al., 2019) and ﬁnetune it on various video classiﬁcation tasks.
Revised: To demonstrate our approach, we pretrain our model on long (30-second) video clips from Kinetics700 (Carreira et al., 2019) and ﬁnetune it on various video classiﬁcation tasks.

Original: One beneﬁt of the modular design of our architecture is ﬂexibility: once pretrained, we can use any of the subnetworks for downstream tasks depending on the modalities involved (audio-only, visual-only, audio-visual) and video lengths (short and long).
Revised: One beneﬁt of the modular design of our architecture is ﬂexibility: once pretrained, we can use any of the subnetworks for downstream tasks depending on the modalities involved (audio-only, visual-only, audio-visual) and video lengths (short and long).

Original: To show this, we evaluate our model on UCF101 (Soomro et al., 2012) and ESC50 (Gemmeke et al., 2017) for short-term visual/audio classiﬁcation, and
Revised: To show this, we evaluate our model on UCF101 (Soomro et al., 2012) and ESC50 (Gemmeke et al., 2017) for short-term visual/audio classiﬁcation, and

Original: Charades (Sigurdsson et al., 2016) and Kinetics-Sounds (Arandjelovic & Zisserman, 2017) for long-term audio-visual action recognition.
Revised: Charades (Sigurdsson et al., 2016) and Kinetics-Sounds (Arandjelovic & Zisserman, 2017) for long-term audio-visual action recognition.

Original:  A PPROACH
Revised:  A PPROACH

Original: Figure 1 shows an overview of the proposed model architecture.
Revised: Figure 1 shows an overview of the proposed model architecture.

Original: The input to our model is a sequence of visual clips v 1: T and the corresponding sequence of audio streams a 1: T .
Revised: The input to our model is a sequence of visual clips v 1: T and the corresponding sequence of audio streams a 1: T .

Original: For example, each sequence is a 30 second-long video divided into 30 non-overlapping clips (each clip is one second long).
Revised: For example, each sequence is a 30 second-long video divided into 30 non-overlapping clips (each clip is one second long).

Original: We divide our model into three parts with different characteristics, which are explained below.
Revised: We divide our model into three parts with different characteristics, which are explained below.

Original: Local Feature Embedding.
Revised: Local Feature Embedding.

Original: We feed each of T video clips to a visual CNN f V ( v t ) to obtain x v 1: T ∈ R T × D , and each audio stream to an audio CNN f A ( a t ) to obtain x a 1: T ∈ R T × D .
Revised: We feed each of T video clips to a visual CNN f V ( v t ) to obtain x v 1: T ∈ R T × D , and each audio stream to an audio CNN f A ( a t ) to obtain x a 1: T ∈ R T × D .

Original: 1 Intuitively, the CNN outputs are temporally local embeddings as they have access to only a short-range temporal window of the entire video sequence.
Revised: 1 Intuitively, the CNN outputs are temporally local embeddings as they have access to only a short-range temporal window of the entire video sequence.

Original: Thus, they are suitable for representing short-range atomic actions (e.g., sit down, raise arms) that constitute long-range events (e.g., gym workout).
Revised: Thus, they are suitable for representing short-range atomic actions (e.g., sit down, raise arms) that constitute long-range events (e.g., gym workout).

Original: We use the SlowFast network (Feichtenhofer et al., 2019) with a ResNet-50 backbone (He et al., 2016) as a visual CNN f V , and a ResNet-50 as an audio CNN f A .
Revised: We use the SlowFast network (Feichtenhofer et al., 2019) with a ResNet-50 backbone (He et al., 2016) as a visual CNN f V , and a ResNet-50 as an audio CNN f A .

Original: The weights of both CNNs are randomly initialized and trained end-to-end with the Transformer layers.
Revised: The weights of both CNNs are randomly initialized and trained end-to-end with the Transformer layers.

Original: Unimodal Contextualized Embedding.
Revised: Unimodal Contextualized Embedding.

Original: The local feature embeddings capture short-term dynamics but lack long-term contextual information.
Revised: The local feature embeddings capture short-term dynamics but lack long-term contextual information.

Original: We use Transformers (Vaswani et al., 2017) to enrich the embeddings with sequence-level context.
Revised: We use Transformers (Vaswani et al., 2017) to enrich the embeddings with sequence-level context.

Original: We start by learning unimodal contextualized representations using the visual Transformer g V and the audio Transformer g A , respectively.
Revised: We start by learning unimodal contextualized representations using the visual Transformer g V and the audio Transformer g A , respectively.

Original: The Transformer consists of L layers, each with two sub-layers: a multi-head attention layer and a feed-forward layer.
Revised: The Transformer consists of L layers, each with two sub-layers: a multi-head attention layer and a feed-forward layer.

Original: Given an input sequence of embeddings x ∈ R T × D and A attention heads, the j -th head in the attention layer computes the output embedding sequence a j ∈ R T × γ , γ = D/A as
Revised: Given an input sequence of embeddings x ∈ R T × D and A attention heads, the j -th head in the attention layer computes the output embedding sequence a j ∈ R T × γ , γ = D/A as

Original: [Equation]
Revised: [Equation]

Original: where W qj , W kj , W vj ∈ R D × γ are weight matrices for computing the ( query , key , value ) triplet given the input x .
Revised: where W qj , W kj , W vj ∈ R D × γ are weight matrices for computing the ( query , key , value ) triplet given the input x .

Original: This operation is repeated for each attention head, and the outputs are combined (with concatenation followed by one linear layer with weights W b ∈ R D × D ), producing a ∈ R T × D .
Revised: This operation is repeated for each attention head, and the outputs are combined (with concatenation followed by one linear layer with weights W b ∈ R D × D ), producing a ∈ R T × D .

Original: Next, the feed-forward layer takes this intermediate output and computes o ∈ R T × D using a twolayer fully-connected network with weights W c ∈ R D × E and W d ∈ R E × D .
Revised: Next, the feed-forward layer takes this intermediate output and computes o ∈ R T × D using a twolayer fully-connected network with weights W c ∈ R D × E and W d ∈ R E × D .

Original: The output of each sub-layer is computed using a residual function followed by layer normalization (Ba et al., 2016), i.e., LayerNorm ( x + Sublayer ( x )) .
Revised: The output of each sub-layer is computed using a residual function followed by layer normalization (Ba et al., 2016), i.e., LayerNorm ( x + Sublayer ( x )) .

Original: In this work, we set the number of layers L = 6 , the number of attention heads A
Revised: In this work, we set the number of layers L = 6 , the number of attention heads A

Original: = 12 , the feature dimension D = 768 and the intermediate dimension E = 3072 .
Revised: = 12 , the feature dimension D = 768 and the intermediate dimension E = 3072 .

Original: For simplicity, we use this design for all layers across all three Transformers in our model.
Revised: For simplicity, we use this design for all layers across all three Transformers in our model.

Original: Before feeding local embeddings x v and x a to unimodal Transformers, we augment them with “positional” embeddings.
Revised: Before feeding local embeddings x v and x a to unimodal Transformers, we augment them with “positional” embeddings.

Original: Speciﬁcally, we append to the beginning of each sequence a special vector
Revised: Speciﬁcally, we append to the beginning of each sequence a special vector

Original: BOS (beginning of sequence), i.e., x v 0 for visual and x a 0 for audio streams; their dimensions are same as x vt and x at , respectively.
Revised: BOS (beginning of sequence), i.e., x v 0 for visual and x a 0 for audio streams; their dimensions are same as x vt and x at , respectively.

Original: We also deﬁne positional embeddings p 0:
Revised: We also deﬁne positional embeddings p 0:

Original: T encoding time indices (we call this “time” embedding).
Revised: T encoding time indices (we call this “time” embedding).

Original: This is necessary to preserve information about temporal ordering of local feature embeddings, which is otherwise lost in Eqn.
Revised: This is necessary to preserve information about temporal ordering of local feature embeddings, which is otherwise lost in Eqn.

Original: We combine them via layer normalization,
Revised: We combine them via layer normalization,

Original: [Equation]
Revised: [Equation]

Original: We initialize { x v 0 , x a 0 , p v 0: T , p a 0: T } to the normal distribution and train them with the rest of the model.
Revised: We initialize { x v 0 , x a 0 , p v 0: T , p a 0: T } to the normal distribution and train them with the rest of the model.

Original: We feed the augmented visual embeddings into the visual Transformer g V and obtain y v 0: T = g V ( u v 0: T ) , and similarly obtain y a 0: T =
Revised: We feed the augmented visual embeddings into the visual Transformer g V and obtain y v 0: T = g V ( u v 0: T ) , and similarly obtain y a 0: T =

Original: g A ( u a 0: T ) .
Revised: g A ( u a 0: T ) .

Original: The embeddings at each time step has a direct access to the entire input sequence regardless of their position (it has a one-step signal path during forward and backward inference).
Revised: The embeddings at each time step has a direct access to the entire input sequence regardless of their position (it has a one-step signal path during forward and backward inference).

Original: Multiple layers of such feature transformation thus allow the resulting embedding to be deeply contextualized in the time dimension.
Revised: Multiple layers of such feature transformation thus allow the resulting embedding to be deeply contextualized in the time dimension.

Original: We denote the output embeddings corresponding to the BOS positions by BOS  v g
Revised: We denote the output embeddings corresponding to the BOS positions by BOS vg = y v 0 and BOS ag

Original: = yv 0 and BOS a g = y a 0 , and designate them as the summary embeddingsrepresenting the sequence of each modality.
Revised: = y a 0 , and designate them as the summary embeddings

Original: Multimodal Contextualized Embedding.
Revised: Multimodal Contextualized Embedding.

Original: The unimodal embeddings capture long-term temporal context but miss out on cross-modal information.
Revised: The unimodal embeddings capture long-term temporal context but miss out on cross-modal information.

Original: The ﬁnal step in forward inference is to use a multimodal Transformer h AV to obtain embeddings contextualized in the audio-visual space.
Revised: The ﬁnal step in forward inference is to use a multimodal Transformer h AV to obtain embeddings contextualized in the audio-visual space.

Original: We ﬁrst augment the embeddings y v 0: T and y a 0: T with modality and time embeddings.
Revised: We ﬁrst augment the embeddings y v 0: T and y a 0: T with modality and time embeddings.

Original: The modality embeddings m v and m a are vectors of the same dimension as y vt and y at , respectively.
Revised: The modality embeddings m v and m a are vectors of the same dimension as y vt and y at , respectively.

Original: We share m v (and m a ) across all the unimodal embeddings y v 0: T (and y a 0: T ); thus, they add modality-discriminative information to the Transformer.
Revised: We share m v (and m a ) across all the unimodal embeddings y v 0: T (and y a 0: T ); thus, they add modality-discriminative information to the Transformer.

Original: We also add time embeddings p 0: T as before; however, unlike in the previous step, we share the same p 0: T between embeddings from the two modalities to correctly indicate the time indices.
Revised: We also add time embeddings p 0: T as before; however, unlike in the previous step, we share the same p 0: T between embeddings from the two modalities to correctly indicate the time indices.

Original: We augment the modality and time embeddings via layer normalization,
Revised: We augment the modality and time embeddings via layer normalization,

Original: [Equation]
Revised: [Equation]

Original: We feed the augmented visual embeddings w v 0: T and audio embeddings w a 0: T to the multimodal
Revised: We feed the augmented visual embeddings w v 0: T and audio embeddings w a 0: T to the multimodal

Original: Transformer h AV , one after another, and obtain z 0:(2 T +1)
Revised: Transformer h AV , one after another, and obtain z 0:(2 T +1)

Original: =
Revised: =

Original: h AV ([ w v 0: T ; w a 0: T ]) .
Revised: h AV ([ w v 0: T ; w a 0: T ]) .

Original: We again denote the output embeddings corresponding to the BOS positions by BOS vh =
Revised: We again denote the output embeddings corresponding to the BOS positions by BOS vh =

Original: z v 0 (= z 0 ) and BOS ah =
Revised: z v 0 (= z 0 ) and BOS ah =

Original: z a 0 (= z T +1 ) , and use them as summary embeddings encoding multimodal context.
Revised: z a 0 (= z T +1 ) , and use them as summary embeddings encoding multimodal context.

Original: We emphasize the importance of feeding w v 0: T and w a 0: T one after another.
Revised: We emphasize the importance of feeding w v 0: T and w a 0: T one after another.

Original: An alternative would be concatenating them before feeding them to h AV and obtaining an output z 0: T (instead of z 0:(2 T +1) ).
Revised: An alternative would be concatenating them before feeding them to h AV and obtaining an output z 0: T (instead of z 0:(2 T +1) ).

Original: However, this restricts the Transformer to access audio-visual embeddings only from the same time slices, which could be problematic when there is a temporally asynchronous relationship between the two modalities (e.g., a visual clip matches with sound captured a few times steps before).
Revised: However, this restricts the Transformer to access audio-visual embeddings only from the same time slices, which could be problematic when there is a temporally asynchronous relationship between the two modalities (e.g., a visual clip matches with sound captured a few times steps before)

Revised: (Kazakos et al., 2019; Morgado et al., 2020).

Original: By arranging the two sequences one after the other, the Transformer can mix-and-match appropriate audio-visual embeddings in an asynchronous manner.
Revised: By arranging the two sequences one after the other, the Transformer can mix-and-match appropriate audio-visual embeddings in an asynchronous manner.

Original: Another practical concern with the alternative approach is that it signiﬁcantly increases the model size; the weight matrices W q , W k , W v grow quadratically with the input feature dimension D .
Revised: Another practical concern with the alternative approach is that it signiﬁcantly increases the model size; the weight matrices W q , W k , W v grow quadratically with the input feature dimension D .

Original: Serializing the input resolves both issues.
Revised: Serializing the input resolves both issues.

Original: 2. S ELF -S UPERVISED P RETRAINING O BJECTIVES
Revised: 2. S ELF -S UPERVISED P RETRAINING O BJECTIVES

Original: Task 1: Masked Embedding Prediction (MEP).
Revised: Task 1: Masked Embedding Prediction (MEP).

Original: BERT (Devlin et al., 2019) is trained using the masked language model (MLM) task, which randomly selects input tokens and replaces them with a mask token.
Revised: BERT (Devlin et al., 2019) is trained using the masked language model (MLM) task, which randomly selects input tokens and replaces them with a mask token.

Original: The model is then trained to predict the original (unmasked) tokens by solving a classiﬁcation task with a cross-entropy loss.
Revised: The model is then trained to predict the original (unmasked) tokens by solving a classiﬁcation task with a cross-entropy loss.

Original: However, inputs to our model are real-valued audiovisual signals (rather than discrete tokens),  so applying the MLM task requires input discretization, which causes information loss (Lu et al., 2019; Sun et al., 2019a).
Revised: However, inputs to our model are real-valued audiovisual signals (rather than discrete tokens), 2 so applying the MLM task requires input discretization, which causes information loss (Lu et al., 2019; Sun et al., 2019a).

Original: We instead train our model to identify the correct visual clip or audio stream compared to a set of negative samples in a contrastive manner, which does not require input discretization.
Revised: We instead train our model to identify the correct visual clip or audio stream compared to a set of negative samples in a contrastive manner, which does not require input discretization.

Original: We formulate our MEP task using InfoNCE (Oord et al., 2018), which is the softmax version of the noise contrastive estimation (NCE) (Gutmann & Hyvärinen, 2010).
Revised: We formulate our MEP task using InfoNCE (Oord et al., 2018), which is the softmax version of the noise contrastive estimation (NCE) (Gutmann & Hyvärinen, 2010).

Original: Let ˜ o t be the t -th output of any of the three Transformers obtained by masking the t -th input x t .
Revised: Let ˜ o t be the t -th output of any of the three Transformers obtained by masking the t -th input x t .

Original: Our InfoNCE loss is then deﬁned as
Revised: Our InfoNCE loss is then deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where neg ( t ) are negative sample indices and the compatibility function I ( x t , ˜ o t ) is,
Revised: where neg ( t ) are negative sample indices and the compatibility function I ( x t , ˜ o t ) is,

Original: [Equation]
Revised: [Equation]

Original: where W I ∈ R P × D ( P = 256 ) and FFN is a two-layer feed-forward network.
Revised: where W I ∈ R P × D ( P = 256 ) and FFN is a two-layer feed-forward network.

Original: The use of a non-linear prediction head has shown to improve the quality of the representations learned in a contrastive learning setup (Chen et al., 2020); following the recent work in Transformers (Devlin et al., 2019; Liu et al., 2019; Lan et al., 2020), we use a GELU non-linear activation function (Hendrycks & Gimpel, 2016) in FFN.
Revised: The use of a non-linear prediction head has shown to improve the quality of the representations learned in a contrastive learning setup (Chen et al., 2020); following the recent work in Transformers (Devlin et al., 2019; Liu et al., 2019; Lan et al., 2020), we use a GELU non-linear activation function (Hendrycks & Gimpel, 2016) in FFN.

Original: Optimizing Eqn.
Revised: Optimizing Eqn.

Original: 4 enforces I ( x t , ˜ o t ) to approximate the density ratio p ( x t | ˜ o t ) p ( x t ) ; this can be seen as maximizing the mutual information between x t and ˜ o t (Oord et al., 2018).
Revised: 4 enforces I ( x t , ˜ o t ) to approximate the density ratio p ( x t | ˜ o t ) p ( x t ) ; this can be seen as maximizing the mutual information between x t and ˜ o t (Oord et al., 2018).

Original: Intuitively, this encourages the Transformer to capture the underlying dynamics of x from each modality without explicitly learning a generative model p ( x t | ˜ o t ) .
Revised: Intuitively, this encourages the Transformer to capture the underlying dynamics of x from each modality without explicitly learning a generative model p ( x t | ˜ o t ) .

Original: Negative sampling.
Revised: Negative sampling.

Original: We ﬁnd that a good negative sampling strategy is essential for the model’s convergence.
Revised: We ﬁnd that a good negative sampling strategy is essential for the model’s convergence.

Original: Existing approaches either use all but x t (positive) within a mini-batch as negative samples or limit it to the current sequence only.
Revised: Existing approaches either use all but x t (positive) within a mini-batch as negative samples or limit it to the current sequence only.

Original: However, both these methods ignore the data content and thus can miss useful negatives.
Revised: However, both these methods ignore the data content and thus can miss useful negatives.

Original: Oord et al.
Revised: Oord et al.

Original: (2018) showed that leveraging prior knowledge about data can improve the negative sample quality (e.g., by sampling negatives from the same speaker as the positive).
Revised: (2018) showed that leveraging prior knowledge about data can improve the negative sample quality (e.g., by sampling negatives from the same speaker as the positive).

Original: Unfortunately, such prior knowledge is often not available in unlabeled videos.
Revised: Unfortunately, such prior knowledge is often not available in unlabeled videos.

Original: We propose a content-aware negative sampling strategy that favors negatives sufﬁciently similar to a positive instance in the CNN embedding space.
Revised: We propose a content-aware negative sampling strategy that favors negatives sufﬁciently similar to a positive instance in the CNN embedding space; we call our approach CANS-Similar .

Original: We compute a pairwise (cid:96) 2 distance between x t (positive) and all other instances within a mini-batch, and normalize them to the [0, 1] interval.

Original: We discard instances that fall outside the 95% conﬁdence interval in the normalized distance space, and sample the negatives from the remainder using the normalized distance as sampling probability.

Original: This makes instances similar to the positive instance have more chance to become negatives.

Original: Our approach is inspired by Ulyanov et al.
Revised: Our approach is inspired by Ulyanov et al.

Original: (2018), who showed  randomly initialized CNNs provide a strong prior over natural images.
Revised: (2018) who showed that randomly initialized CNNs provide a strong prior over natural images due to the inductive bias already built into the design of the CNNs.

Original: This suggests that our local feature embeddings x v (and x a )  capture inherent regularities in video clips (and audio streams), and thus the distance measured on them can approximate content dissimilarity (this will improve as the training progresses). {+Motivated by this, we sample the negatives based on local feature embeddings x v (and x a ). Specifically, we compute a pairwise (cid:96) 2 distance between x t (positive) and all other instances within a 
+}
Revised: This suggests that our local feature embeddings x v (and x a ) can capture the underlying statistical regularities in video clips (and audio streams) right from the beginning, which can be sufﬁcient to assess the similarity/dissimilarity between clips. Therefore, the distance measured on them can approximate content dissimilarity well (and this will improve as the training progresses). {+Motivated by this, we sample the negatives based on local feature embeddings x v (and x a ). Specifically, we compute a pairwise (cid:96) 2 distance between x t (positive) and all other instances within a 
+}

Revised: In the form of RGB images and log-mel-scaled spectrograms.

Revised: [Figure]

Original: Finally, our MEP loss is the InfoNCE loss computed on all three Transformers,
Revised: mini-batch, and normalize them to the [0, 1] interval. To remove samples that are either too similar or too different from the positive sample, we discard instances that fall outside the 95% conﬁdence interval in the normalized distance space. We then sample the negatives from the remainder using the normalized distance as sampling probability. This makes instances similar to the positive instance have more chance to become negatives.

Revised: We emphasize the importance of sampling, instead of deterministically taking top most similar samples; the stochasticity allows our model to be robust to potentially inaccurate distance estimates because samples with low probabilities will still have a chance to be selected as negatives.

Revised: Finally, our MEP loss is the InfoNCE loss computed on all three Transformers,

Original: [Equation] [Figure]
Revised: [Equation] [Equation]

Original: Task 2: Correct Pair Prediction (CPP).
Revised: Task 2: Correct Pair Prediction (CPP).

Original: The MEP task encourages our model to learn the underlying dynamics within each modality.
Revised: The MEP task encourages our model to learn the underlying dynamics within each modality.

Original: To help our model learn cross-modal dynamics, we design a task that predicts whether a pair of audio-visual embeddings is from the same video.
Revised: To help our model learn cross-modal dynamics, we design a task that predicts whether a pair of audio-visual embeddings is from the same video.

Original: Speciﬁcally, we deﬁne two binary classiﬁers, one for the two unimodal Transformers and another for the multimodal
Revised: Speciﬁcally, we deﬁne two binary classiﬁers, one for the two unimodal Transformers and another for the multimodal

Original: Transformer.
Revised: Transformer.

Original: Each classiﬁer takes as input either s g =
Revised: Each classiﬁer takes as input either s g =

Original: [ y v 0 ; y a 0 ] (and [ z v 0 ;
Revised: [ y v 0 ; y a 0 ] (and [ z v 0 ;

Original: z a 0 ] ), a pair of audio-visual “summary” embeddings corresponding to the BOS positions, or s h
Revised: z a 0 ] ), a pair of audio-visual “summary” embeddings corresponding to the BOS positions, or s h

Original: =
Revised: =

Original: [ y vt ; y
Revised: [ y vt ; y

Original: at ] (or [ z vt ; z at ] ), the output embeddings sampled at random positions (we take two random positions t ∈ [1 , T ] ).
Revised: at ] (or [ z vt ; z at ] ), the output embeddings sampled at random positions (we take two random positions t ∈ [1 , T ] ).

Original: The classiﬁer predicts p ( c | s ) indicating whether the pair is from the same video ( c = 1 ) or from different videos ( c = 0 ).
Revised: The classiﬁer predicts p ( c | s ) indicating whether the pair is from the same video ( c = 1 ) or from different videos ( c = 0 ).

Original: We train the classiﬁers with a binary cross-entropy loss,
Revised: We train the classiﬁers with a binary cross-entropy loss,

Original: [Equation]
Revised: [Equation]

Original: (7) where · is the inner product.
Revised: where · is the inner product.

Original: We generate a random derangement of the input mini-batch so that the number of positive and negative pairs are guaranteed to be the same.
Revised: We generate a random derangement of the input mini-batch so that the number of positive and negative pairs are guaranteed to be the same.

Original: Overall Pretraining Objective .
Revised: Overall Pretraining Objective .

Original: We train our model end-to-end from scratch by optimizing L MEP + α L CPP with a balancing term α .
Revised: We train our model end-to-end from scratch by optimizing L MEP + α L CPP with a balancing term α .

Original: We ﬁnd our model is insensitive to this term, so we set α = 1 . 0 .
Revised: We ﬁnd our model is insensitive to this term, so we set α = 1 . 0 .

Original: 2. P ARAMETER R EDUCTION
Revised: 2. P ARAMETER R EDUCTION

Original: Optimizing our model is challenging due to the large memory requirement.
Revised: Optimizing our model is challenging due to the large memory requirement.

Original: The most expensive part is the Transformers, which take up 76% of model parameters.
Revised: The most expensive part is the Transformers, which take up 76% of model parameters.

Original: One could reduce the model size by making the Transformers shallower, but the depth of Transformers has shown to be crucial to get good performance (Devlin et al., 2019).
Revised: One could reduce the model size by making the Transformers shallower, but the depth of Transformers has shown to be crucial to get good performance (Devlin et al., 2019).

Original: We propose to reduce the model size by aggressively sharing parts of weights across Transformers as well as layers within each Transformer (see Figure 2 (d)).
Revised: We propose to reduce the model size by aggressively sharing parts of weights across Transformers as well as layers within each Transformer (see Figure 2 (d)).

Original: Sharing across Transformers.
Revised: Sharing across Transformers.

Original: We ﬁrst consider sharing weights across Transformers.
Revised: We ﬁrst consider sharing weights across Transformers.

Original: Each Transformer encodes data coming from different distributions: g V encodes x v , g A encodes x a , and h AV encodes ( y v , y a ) .
Revised: Each Transformer encodes data coming from different distributions: g V encodes x v , g A encodes x a , and h AV encodes ( y v , y a ) .

Original: These input distributions may each exhibit different dynamics, yet together share certain regularities because they all come from the same videos.
Revised: These input distributions may each exhibit different dynamics, yet together share certain regularities because they all come from the same videos.

Original: Motivated by this, we decompose Transformer weights into shared and private parts so that different patterns can be learned in a parameter-efﬁcient manner.
Revised: Motivated by this, we decompose Transformer weights into shared and private parts so that different patterns can be learned in a parameter-efﬁcient manner.

Original: Recall that each layer of a Transformer contains weights { W q , W k , W v , W b , W c , W d } .
Revised: Recall that each layer of a Transformer contains weights { W q , W k , W v , W b , W c , W d } .

Original: We decompose each of these weights into W = U Σ V  , where
Revised: We decompose each of these weights into W = U Σ V (cid:62) , where

Original: W ∈ R M × N , U ∈ R M
Revised: W ∈ R M × N , U ∈ R M

Original: ×
Revised: ×

Original: O , Σ ∈ R
Revised: O , Σ ∈ R

Original: O × O , V ∈ R N × O .
Revised: O × O , V ∈ R N × O .

Original: We perform low-rank approximation of W by setting the rank O (cid:28) M, N , and share U across Transformers while keeping Σ and V private to each
Revised: We perform low-rank approximation of W by setting the rank O (cid:28) M, N , and share U across Transformers while keeping Σ and V private to each

Original: Transformer.
Revised: Transformer.

Original: This helps reduce parameters because MO +3( O 2 + NO ) (cid:28) 3 MN .
Revised: This helps reduce parameters because MO +3( O 2 + NO ) (cid:28) 3 MN .

Original: We experimented with different matrix ranks O
Revised: We experimented with different matrix ranks O

Original: but the differences were small; we set O = 128 ( M, N = 768 or 3072). 
Revised: but the differences were small; we set O = 128 ( M, N = 768 or 3072). [Table] -L:

Revised: Cross-layer, X.-T: Cross-Transformer sharing.

Revised: We report top-1 and top-5 accuracy (%).

Original:  The decomposition converts a linear projection of input W x into a series of (unconstrained) linear projections U Σ V  x .
Revised: † : Ours. The decomposition converts a linear projection of input W x into a series of (unconstrained) linear projections U Σ V (cid:62) x .

Original: However, this can cause numerical instability during optimization (Nocedal & Wright, 2006).
Revised: However, this can cause numerical instability during optimization (Nocedal & Wright, 2006).

Original: We could perform the Singular Value Decomposition (SVD) over W so that it performs rotation ( V  ), stretch ( Σ ), and rotation ( U ) with orthogonal  vectors in U and V .
Revised: We could perform the Singular Value Decomposition (SVD) over W so that it performs rotation ( V (cid:62) ), stretch ( Σ ), and rotation ( U ) with orthogonal basis vectors in U and V .

Original: Unfortunately, solving the full SVD has a computational complexity of O (max( M, N ) 2 ) (Golub & Van Loan, 2012).
Revised: Unfortunately, solving the full SVD has a computational complexity of O (max( M, N ) 2 ) (Golub & Van Loan, 2012).

Original: Here, we put an orthogonality constraint only on Σ and perform projection ( V  ), rotation ( Σ ), and projection ( U ) of input x .
Revised: Here, we put an orthogonality constraint only on Σ and perform projection ( V (cid:62) ), rotation ( Σ ), and projection ( U ) of input x .

Original: In addition, we put V  x in a unit sphere (via (cid:96) 2 -normalization) before rotating it with Σ .
Revised: In addition, we put V (cid:62) x in a unit sphere (via (cid:96) 2normalization) before rotating it with Σ .

Original: This not only improves numerical stability, but also removes magnitude information in V  x and keeps angular information only, which has been shown to provide sample discriminative information (Chen et al., 2019a).
Revised: This not only improves numerical stability, but also removes magnitude information in V (cid:62) x and keeps angular information only, which has been shown to provide sample discriminative information (Chen et al., 2019a).

Original: To impose the orthogonality constraint, we use the Padé approximation with a scalesquaring trick of Lezcano-Casadoand Martínez-Rubio (Lezcano-Casado & Martínez-Rubio, 2019).
Revised: To impose the orthogonality constraint on Σ , we use the Padé approximation with a scale-squaring trick of Lezcano-Casado & Martínez-Rubio (2019).

Original: Intuitively, we  project  andthen rotate the input ( Σ V  x ) in eachTransformer so that it captures the [Table]
Revised: Intuitively, we linearly project x onto a unit sphere ( V (cid:62) x ) and rotate it ( Σ V (cid:62) x ) in each

Original:  dynamics of each input distribution independently.
Revised: Transformer so that it captures the dynamics of each input distribution independently.

Original: We then project it to the shared space via U , capturing shared regularities across all three Transformers.
Revised: We then project it to the shared space via U , capturing shared regularities across all three Transformers.

Original: Sharing across Layers.
Revised: Sharing across Layers.

Original: Recently, Bai et al. (2019a) showed that sharing parameters across layers in deep neural networks does not hurt the representational power of the network.
Revised: Recently, Bai et al. (2019a) showed that sharing parameters across layers in deep neural networks does not hurt the representational power of the network.

Original: Furthermore, (Lan et al., 2020) demonstrated that cross-layer parameter sharing in the Transformer leads to a lighter and faster-to-train model without sacriﬁcing the performance on various language understanding benchmarks.
Revised: Furthermore, (Lan et al., 2020) demonstrated that cross-layer parameter sharing in the Transformer leads to a lighter and faster-to-train model without sacriﬁcing the performance on various language understanding benchmarks.

Original: Motivated by this, we let each Transformer share parameters across different layers.
Revised: Motivated by this, we let each Transformer share parameters across different layers.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We pretrain our model on Kinetics-700 (Carreira et al., 2019) or AudioSet (Gemmeke et al., 2017) and ﬁnetune it on various downstream tasks.
Revised: We pretrain our model on Kinetics-700 (Carreira et al., 2019) or AudioSet (Gemmeke et al., 2017) and ﬁnetune it on various downstream tasks.

Original: The ofﬁcial release of Kinetics-700 contains 10-second clips only, so we download 410K original videos from YouTube and take 30-second clips from each video.
Revised: The ofﬁcial release of Kinetics-700 contains 10-second clips only, so we download 410K original videos from YouTube and take 30-second clips from each video.

Original: For fair comparison with prior work, we use 10-second clips from the ofﬁcial release of AudioSet (we used 1.8M clips).
Revised: For fair comparison with prior work, we use 10-second clips from the ofﬁcial release of AudioSet (we used 1.8M clips).

Original: We pretrain our model on 64 NVIDIA Tesla V100 GPUs with a batch size of 256 for 220K iterations.
Revised: We pretrain our model on 64 NVIDIA Tesla V100 GPUs with a batch size of 256 for 220K iterations.

Original: For downstream tasks, we evaluate on short -video/audio classiﬁcation using UCF-101 (Soomro et al., 2012) (13K clips from 101 classes; 7.2 seconds on average) and ESC50 (Gemmeke et al., 2017) (2K clips from 50 classes; 5 seconds), and on long -video classiﬁcation using Kinetics-Sounds (Arandjelovic & Zisserman, 2017) (23K videos from 32 classes; 10 seconds on average) and Charades (Sigurdsson et al., 2016)
Revised: For downstream tasks, we evaluate on short -video/audio classiﬁcation using UCF-101 (Soomro et al., 2012) (13K clips from 101 classes; 7.2 seconds on average) and ESC50 (Gemmeke et al., 2017) (2K clips from 50 classes; 5 seconds), and on long -video classiﬁcation using Kinetics-Sounds (Arandjelovic & Zisserman, 2017) (23K videos from 32 classes; 10 seconds on average) and Charades (Sigurdsson et al., 2016)

Original: (10K videos from 157 classes; 30 seconds on average).
Revised: (10K videos from 157 classes; 30 seconds on average).

Original: We describe various details about experimental setup in Appendix.
Revised: We describe various details about experimental setup in Appendix.

Original: Code will be released.
Revised: Code will be released.

Original: 3. R ESULTS AND D ISCUSSION
Revised: 3. R ESULTS AND D ISCUSSION

Original: Multimodal Fusion Methods.
Revised: Multimodal Fusion Methods.

Original: To evaluate different fusion methods on the quality of learned representation, we test the following settings: (i) Early uses a single multimodal Transformer with 2 × L layers, (ii) Mid is our approach described in Figure 1, (iii) Late uses two unimodal
Revised: To evaluate different fusion methods on the quality of learned representation, we test the following settings: (i) Early uses a single multimodal Transformer with 2 × L layers, (ii) Mid is our approach described in Figure 1, (iii) Late uses two unimodal

Original: Transformers each with 2 × L layers.
Revised: Transformers each with 2 × L layers.

Original: All the methods are pretrained on audio-visual data using CPP and MEP losses, except for (iv) Late-w/o-CPP where we use only the MEP loss.
Revised: All the methods are pretrained on audio-visual data using CPP and MEP losses, except for (iv) Late-w/o-CPP where we use only the MEP loss.

Original: We ﬁnetune the pretrained models on audio-visual, audio-only, and visual-only scenarios.
Revised: We ﬁnetune the pretrained models on audio-visual, audio-only, and visual-only scenarios.

Original: For fair comparisons across different fusion methods, we do not perform parameter sharing in this ablation setting.
Revised: For fair comparisons across different fusion methods, we do not perform parameter sharing in this ablation setting.

Original: Table 1 (a) shows that Early and Mid outperform Late on the audio-visual scenario.
Revised: Table 1 (a) shows that Early and Mid outperform Late on the audio-visual scenario.

Original: This suggests the importance of encoding cross-modal information.
Revised: This suggests the importance of encoding cross-modal information.

Original: Note that Late-w/-CPP gets cross-modal self-supervision, which gives marginal performance improvement over Late-w/o-CPP ; however, both methods miss the opportunity to encode any cross-modal relationship, leading to inferior results.
Revised: Note that Late-w/-CPP gets cross-modal self-supervision, which gives marginal performance improvement over Late-w/o-CPP ; however, both methods miss the opportunity to encode any cross-modal relationship, leading to inferior results.

Original: While both Early and Late perform similarly in the audio-visual scenario, only Late can be used in unimodal downstream scenarios (c.f., Early requires the presence of both modalities).
Revised: While both Early and Late perform similarly in the audio-visual scenario, only Late can be used in unimodal downstream scenarios (c.f., Early requires the presence of both modalities).

Original: This has practical implications:
Revised: This has practical implications:

Original: Mid and Late can effectively handle missing modalities, i.e., once pretrained on audio-visual data, we can use it on any of audio-visual, audio-only, and visual-only scenarios.
Revised: Mid and Late can effectively handle missing modalities, i.e., once pretrained on audio-visual data, we can use it on any of audio-visual, audio-only, and visual-only scenarios.

Original: Our
Revised: Our

Original: Mid fusion approach enjoys both the advantages, i.e., learning cross-modal relationship and being robust to missing modalities, achieving overall the best performance.
Revised: Mid fusion approach enjoys both the advantages, i.e., learning cross-modal relationship and being robust to missing modalities, achieving overall the best performance.

Original: Negative Sampling Strategies.
Revised: Negative Sampling Strategies.

Original: We compare four strategies: (i) Current-Sequence takes all but the positive instance from the same sequence as negatives, (ii) Current-MiniBatch takes all but the positive instance in the mini-batch as negatives; this subsumes Current-Sequence , (iii) CANS-Dissimilar stochastically samples negatives using a modiﬁed version of our contentaware negative sampling (CANS) that favors dissimilar samples, and (iv) CANS-Similar is our proposed CANS approach that favors negatives that are similar to the positive instance.
Revised: We compare four strategies: (i) Current-Sequence takes all but the positive instance from the same sequence as negatives, (ii) Current-MiniBatch takes all but the positive instance in the mini-batch as negatives; this subsumes Current-Sequence , (iii) CANS-Dissimilar stochastically samples negatives using a modiﬁed version of our contentaware negative sampling (CANS) that favors dissimilar samples, and (iv) CANS-Similar is our proposed CANS approach that favors negatives that are similar to the positive instance.

Original: Table 1 (b) shows Current-Sequence is the least effective: It makes MEP too difﬁcult because negatives are (sometimes too much) similar to positives.
Revised: Table 1 (b) shows Current-Sequence is the least effective: It makes MEP too difﬁcult because negatives are (sometimes too much) similar to positives.

Original: As a result, the training dynamics is dominated by CPP, which is relatively easier, leading to inferior performance.
Revised: As a result, the training dynamics is dominated by CPP, which is relatively easier, leading to inferior performance.

Original: We make quite the contrary observations from Current-MiniBatch : the inclusion of negatives from different videos makes MEP easier and thus makes it dominate the training dynamics.
Revised: We make quite the contrary observations from Current-MiniBatch : the inclusion of negatives from different videos makes MEP easier and thus makes it dominate the training dynamics.

Original: Our CANS approach solves both these issues by eliminating negatives that are either almost identical to or trivial to distinguish from the positives, based on the 95% CI over the CNN embedding distances.
Revised: Our CANS approach solves both these issues by eliminating negatives that are either almost identical to or trivial to distinguish from the positives, based on the 95% CI over the CNN embedding distances.

Original: It also samples negatives in a stochastic manner so a wide variety of samples can be included as negatives.
Revised: It also samples negatives in a stochastic manner so a wide variety of samples can be included as negatives.

Original: Our proposed
Revised: Our proposed

Original: CANS-Similar can be considered as a “softened” version of Current-Sequence ; it samples negatives that are similar to positives with a high probability (this can be considered as online hard negative mining), but it also takes instances from different videos with a lower probability.
Revised: CANS-Similar can be considered as a “softened” version of Current-Sequence ; it samples negatives that are similar to positives with a high probability (this can be considered as online hard negative mining), but it also takes instances from different videos with a lower probability.

Original: This balances out hard and easy negatives, making the MEP task effective.
Revised: This balances out hard and easy negatives, making the MEP task effective.

Original: Parameter Sharing Schemes.
Revised: Parameter Sharing Schemes.

Original: Our parameter reduction scheme reduces the model size from 155M to 31M (by 80%)
Revised: Our parameter reduction scheme reduces the model size from 155M to 31M (by 80%)

Original: (Table 1 (a)).
Revised: (Table 1 (a)).

Original: We reduce the model size by sharing weights across Transformers and across layers.
Revised: We reduce the model size by sharing weights across Transformers and across layers.

Original: We validate these ideas in two sets of experiments.
Revised: We validate these ideas in two sets of experiments.

Original: Table 1 (c-left) compares cross-Transformer weight sharing schemes.
Revised: Table 1 (c) compares crossTransformer weight sharing schemes.

Original: We use Multi-6 that uses all three Transformers withlayers each, and compare four methods that correspond to Figure 2 (a)-(d).
Revised: We use Multi-6 that uses all three Transformers with 6 layers each, and compare four methods that correspond to Figure 2 (a)-(d).

Original: Note that No sharing is too large to ﬁt in a Tesla V100 GPU (16GB) even with 2 samples so we only report the number of parameters.
Revised: Note that No sharing is too large to ﬁt in a Tesla V100 GPU (16GB) even with 2 samples, so we deﬁne Multi-2 that uses three

Revised: Transformers with 2 layers each, and with the reduced number of attention heads A to 5, the feature dimension D to 320 and the intermediate dimension E to 1280.

Original: We see that our proposed approach, Part , achieves the best performance with the least number of parameters.
Revised: We see that our proposed approach, Part , achieves the best performance with the least number of parameters.

Original: One might ask how Part leads to a smaller model when All shares all the weights across Transformers:
Revised: One might ask how Part leads to a smaller model when All shares all the weights across Transformers:

Original: We decompose weights W = U Σ V  with low-rank approximation and share only U across Transformers, while the Σ V  part learns modality-speciﬁc dynamics.
Revised: We decompose weights W = U Σ V (cid:62) with low-rank approximation and share only U across Transformers, while the Σ V (cid:62) part learns modality-speciﬁc dynamics.

Original: Table(c-right) compares cross-layer weight sharing schemes using the visual Transformer with either( Vis-2 ) or 6 ( Vis-6 ) layers.
Revised: Table 1 (d) compares cross-layer weight sharing schemes using the visual Transformer with either 2 ( Vis-2 ) or 6 ( Vis-6 ) layers.

Original: The results show that sharing weights across layers does not hurt the performance, conﬁrming the observations by Lan et al.
Revised: The results show that sharing weights across layers does not hurt the performance, conﬁrming the observations by Lan et al.

Original: (2020) in the audio-visual setting.
Revised: (2020) in the audio-visual setting.

Original: Pretraining Objectives.
Revised: Pretraining Objectives.

Original: To evaluate the importance of MEP and CPP tasks, we test two settings: (i) Mid-w/o-CPP and (ii) Mid-w/o-MEP .
Revised: To evaluate the importance of MEP and CPP tasks, we test two settings: (i) Mid-w/o-CPP and (ii) Mid-w/o-MEP .

Original: On Kinetics-Sounds, these achieve 65.9% and 64.6%, respectively; ours achieve 67.5% (top-1 accuracy).
Revised: On Kinetics-Sounds, these achieve 65.9% and .6%, respectively; ours achieve 67.5% (top-1 accuracy).

Original: The result show that the MEP task plays an important role during pretraining, conﬁrming the ﬁndings from Sun et al. (2019a) that the InfoNCE loss, as deployed in CBT, is effective in the cross-modal setting.
Revised: The result show that the MEP task plays an important role during pretraining, conﬁrming the ﬁndings from Sun et al. (2019a) that the InfoNCE loss, as deployed in CBT, is effective in the cross-modal setting.

Original: The result also shows that augmenting
Revised: The result also shows that augmenting

Original: MEP with CPP provides further performance improvement by learning cross-modal correspondence.
Revised: MEP with CPP provides further performance improvement by learning cross-modal correspondence

Original: Downstream Evaluation.
Revised: Downstream Evaluation.

Original: We pretrain our model with Mid fusion using MEP and CPP tasks (with
Revised: We pretrain our model with Mid fusion using MEP and CPP tasks (with

Original: CANS-Similar ), and employ Part weight sharing.
Revised: CANS-Similar ), and employ Part weight sharing.

Original: We use either Kinetics-700 or AudioSet for fair comparisons with prior work.
Revised: We use either Kinetics-700 or AudioSet for fair comparisons with prior work.

Original: Table 2 (a)/(b) shows short -video/audio classiﬁcation results on
Revised: Table 2 (a)/(b) shows short -video/audio classiﬁcation results on

Original: UCF-101/ESC-50.
Revised: UCF-101/ESC-50.

Original: We use only the visual/audio CNN (no Transformers)and add a linear classiﬁer on top, and ﬁnetune the model end-to-end.
Revised: For fair comparisons to the baselines, we use only the visual/audio CNN (no Transformers)

Original:  Although our model is pretrained on long video clips with no direct supervision to the CNN layers (gradients mustﬂow through Transformers), it outperforms most of the baselines (except for AVTS on UCF-101) that received direct supervision from short video clips.
Revised: ; we ﬁnetune a linear classiﬁer on top of the visual CNN end-to-end for UCF-101, and train a multi-class one-vs-all linear SVM on top of the ﬁxed audio CNN for ESC-50. Although our model is pretrained on long video clips with no direct supervision to the CNN layers (gradients must

Original: We note that, similar to ours, CBT (Su et al., 2020) is a multimodal Transformer pretrained on long video clips and thus is the most meaningful comparison to ours; ours outperform CBT on
Revised: [Table] , AE (Aytar et al., 2016), SVM (Piczak, 2015a), RF (Piczak, 2015a),ConvNet(Piczak,2015b),SoundNet(Aytaretal.,2016), L 3 -Net(Arandjelovic&Zisserman,2017),DMC(Huetal.

Original: UCF-101 by 5.7%.
Revised: ,2019),ATF(Sigurdssonetal.

Original: For sound classiﬁcation, our approach outperform all existing published results.
Revised: ,2017) ﬂow through Transformers outperforms most of the baselines (except for on that received direct supervision from short video clips We note that, similar to ours, CBT (Su et al , 2020) is a multimodal Transformer pretrained on long video clips and thus is the most meaningful comparison to ours; ours outperform CBT

Original: Table 2 (c) shows long -video classiﬁcation results on Charades and Kinetics-Sounds (KS) when pretrained on Kinetics-700.
Revised: Table 2 (c) shows long -video classiﬁcation results on Charades and Kinetics-Sounds (KS) when pretrained on Kinetics-700.

Original: We test Visual-only (V) , Audio-only (A) , and Multimodal (M) settings to verify the beneﬁt of multimodal learning.
Revised: We test Visual-only (V) , Audio-only (A) , and Multimodal (M) settings to verify the beneﬁt of multimodal learning.

Original: Because there is no published selfsupervised learning results on these datasets, we demonstrate long-term representations by comparing
Revised: Because there is no published selfsupervised learning results on these datasets, we demonstrate long-term representations by comparing

Original: CNNs ( CNN ; short-term) to Transformers ( BERT ; long-term) on KS that contains 10-second clips.
Revised: CNNs ( CNN ; short-term) to Transformers ( BERT ; long-term) on KS that contains 10-second clips.

Original: Since CNNs process 1-second clips, we feed 10 non-overlapping clips to CNNs and average the prediction output.
Revised: Since CNNs process 1-second clips, we feed 10 non-overlapping clips to CNNs and average the prediction output.

Original: In all settings, we add a 2-layer MLP with softmax classiﬁer on top.
Revised: In all settings, we add a 2-layer MLP with softmax classiﬁer on top.

Original: The results show that Transformers outperform CNNs on Kinetics-Sounds, suggesting the superiority of long-term representations.
Revised: The results show that Transformers outperform CNNs on Kinetics-Sounds, suggesting the superiority of long-term representations.

Original: We also see that combining audio-visual information performs the best.We notice [Table] , 2015a)
Revised: We also see that combining audio-visual information performs the best.

Original: RF (Piczak, 2015a),ConvNet(Piczak,2015b)

Original: SoundNet(Aytaretal.,2016)

Original: L (Arandjelovic Zisserman Huetal that audio representations are generally stronger than visual representations; we believe that learning discriminative visual representations is generally more challenging, especially when the CNNs receive (self-)supervision signals only indirectly through Transformers.
Revised: We notice that audio representations are generally stronger than visual representations; we believe that learning discriminative visual representations is generally more challenging, especially when the CNNs receive (self-)supervision signals only indirectly through Transformers.

Original: We believe that providing (self-)supervision directly to CNNs, e.g., by ﬁrst pretraining CNNs on 3D rotation prediction (Jing et al., 2018) and then jointly training the whole model (as was done in CBT (Sun et al., 2019a)), could further improve performance.
Revised: We believe that providing (self-)supervision directly to CNNs, e.g., by ﬁrst pretraining CNNs on 3D rotation prediction (Jing et al., 2018) and then jointly training the whole model (as was done in CBT (Sun et al., 2019a)), could further improve performance.

Original: Incorporating contrastive learning (Chen et al., 2020) over the CNN embeddings and training the whole model end-to-end is another promising direction for future work.
Revised: Incorporating contrastive learning (Chen et al., 2020) over the CNN embeddings and training the whole model end-to-end is another promising direction for future work.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Multimodal BERT.
Revised: Multimodal BERT.

Original: Extending BERT (Devlin et al., 2019) to vision-and-language has been actively studied.
Revised: Extending BERT (Devlin et al., 2019) to vision-and-language has been actively studied.

Original: Existing work typically adopt early fusion (Li et al., 2019; Alberti et al., 2019;
Revised: Existing work typically adopt early fusion (Li et al., 2019; Alberti et al., 2019;

Original: Sun et al., 2019b; Li et al., 2020;
Revised: Sun et al., 2019b; Li et al., 2020;

Original: Zhou et al., 2020; Su et al., 2020; Chen et al., 2019b; Zhu & Yang, 2020) or mid fusion (Tan & Bansal, 2019; Lu et al., 2019; Sun et al., 2019a; Luo et al., 2020) without thorough validation, and they train only visual components while relying on a language-pretrained BERT.
Revised: Zhou et al., 2020; Su et al., 2020; Chen et al., 2019b; Zhu & Yang, 2020) or mid fusion (Tan & Bansal, 2019; Lu et al., 2019; Sun et al., 2019a; Luo et al., 2020) without thorough validation, and they train only visual components while relying on a language-pretrained BERT.

Original: Compared to all the previous work, our approach is the ﬁrst toextend BERT to the audio-visual setting, and the ﬁrst to demonstratea fully end-to-end trained multimodal BERT.
Revised: Although there have been some efforts to leverage the Transformer architecture (Vaswani et al., 2017) for audio and visual inputs (Boes & Van hamme, 2019; Tian et al., 2020), our approach is the ﬁrst to demonstrate multimodal audio-visual BERT trained from scratch in an end-to-end manner.

Original: This is enabled by our novel parameter reduction technique, which is one of our main technical contributions.
Revised: This is enabled by our novel parameter reduction technique, which is one of our main technical contributions.

Original: Audio-Visual Learning.
Revised: Audio-Visual Learning.

Original: Early work in audio-visual learning focused on speech signals, improving audio-visual speech recognition than unimodal approaches (Ngiam et al., 2011; Srivastava & Salakhutdinov, 2012).
Revised: Early work in audio-visual learning focused on speech signals, improving audio-visual speech recognition than unimodal approaches (Ngiam et al., 2011; Srivastava & Salakhutdinov, 2012).

Original: Recent approaches leverage unlabeled videos from speciﬁc domains (Owens et al., 2016; Gao & Grauman, 2019; Zhao et al., 2018; Ephrat et al., 2018) and often demonstrate on audio-visual source separation, localization, and co-segmentation.
Revised: Recent approaches leverage unlabeled videos from speciﬁc domains (Owens et al., 2016; Gao & Grauman, 2019; Zhao et al., 2018; Ephrat et al., 2018; Alwassel et al., 2019; Miech et al., 2020; Piergiovanni et al., 2020) and often demonstrate on audio-visual source separation, localization, and co-segmentation.

Original: However, these approaches rely on short-term audio-visual correspondence and thus may not generalize to long-term video recognition that requires global context (as was suggested in (Hjelm et al., 2019)), which this work focuses on.
Revised: However, these approaches rely on short-term audio-visual correspondence and thus may not generalize to long-term video recognition that requires global context (as was suggested in (Hjelm et al., 2019)), which this work focuses on.

Original: Parameter Reduction.
Revised: Parameter Reduction.

Original: Network pruning (Reed, 1993; Caron et al., 2020) trains a large model and then reduces its size while maintaining performance.
Revised: Network pruning (Reed, 1993; Caron et al., 2020) trains a large model and then reduces its size while maintaining performance.

Original: Reducing the size of CNNs for mobile applications is an active research area (Rastegari et al., 2016; Howard et al., 2017; 2019; Zhang et al., 2018; Iandola et al., 2016).
Revised: Reducing the size of CNNs for mobile applications is an active research area (Rastegari et al., 2016; Howard et al., 2017; 2019; Zhang et al., 2018; Iandola et al., 2016).

Original: Our work is closely related to the work that shares parameters across layers in deep neural networks.
Revised: Our work is closely related to the work that shares parameters across layers in deep neural networks.

Original: Trellis network (Bai et al., 2019b) is a temporal convolutional architecture with weight-tying across time and depth.
Revised: Trellis network (Bai et al., 2019b) is a temporal convolutional architecture with weight-tying across time and depth.

Original: Similar to ours, Universal Transformer (Dehghani et al., 2019), RSNMT (Dabre & Fujita, 2019), DEQ (Bai et al., 2019a), ALBERT (Lan et al., 2020) share weights across layers in Transformers.
Revised: Similar to ours, Universal Transformer (Dehghani et al., 2019), RSNMT (Dabre & Fujita, 2019), DEQ (Bai et al., 2019a), ALBERT (Lan et al., 2020) share weights across layers in Transformers.

Original: We combine this idea with our novel cross-Transformer weight sharing, which decomposes weight matrices with low-rank approximation. 
Revised: We combine this idea with our novel cross-Transformer weight sharing, which decomposes weight matrices with low-rank approximation. Negative Sampling.

Revised: Hard negative mining has been shown to be crucial for contrastive learning (Arandjelovic & Zisserman, 2017; Owens & Efros, 2018; Korbar et al., 2018; Schroff et al., 2015; Zhuang et al., 2019; Morgado et al., 2020; Wu et al., 2020).

Revised: Korbar et al.

Revised: (2018) use the time difference between clips to approximate clip similarity (i.e., clips that are further apart are deemed more different).

Revised: However, such an assumption may not hold for real-world videos, e.g., periodic actions such as push-ups.

Original:   C ONCLUSION
Revised: Unlike this line of approaches, we directly use the feature embeddings learned by our model. Several apparoaches adapted a similar idea (Schroff et al., 2015; Zhuang et al., 2019; Morgado et al., 2020; Wu et al., 2020). Different from prior work, we bring the stochasticity to the sampling procedure by using the content similarity as the sampling probability; this helps reduce potential errors especially during the early stage of training.  C ONCLUSION

Original: We introduced a multimodal bidirectional Transformer architecture for self-supervised learning of contextualized audio-visual representation from unlabeled videos.
Revised: We introduced a multimodal bidirectional Transformer architecture for self-supervised learning of contextualized audio-visual representation from unlabeled videos.

Original: Our main technical contributions include: (1) we propose a parameter efﬁcient multimodal Transformers based on matrix decomposi - tion with low- rank approximation 2) we propose a novel content- aware negative sampling technique for contrastive learning
Revised: Our main technical contributions include: (1) we propose a parameter efﬁcient multimodal Transformers based on matrix decomposition with low-rank approximation; (2) we propose a novel content-aware negative sampling technique for contrastive learning.

Original: We demonstrate a successful end -to end training of multimodal Transformers for audio-visual learning (which is, to the best of our knowledge, the ﬁrst time in the literature)
Revised: We demonstrate a successful end-to-end training of multimodal Transformers for audio-visual learning (which is, to the best of our knowledge, the ﬁrst time in the literature).

Original: We also report comprehensive evaluation of various design decisions in multimodal learning
Revised: We also report comprehensive evaluation of various design decisions in multimodal learning.

Original: A I MPLEMENTATION D ETAILS A. A RCHITECTURES OF V ISUAL /A UDIO CNN S
Revised: A I MPLEMENTATION D ETAILS A. A RCHITECTURES OF V ISUAL /A UDIO CNN S

Original: Table 3 shows the architectures of visual and audio CNNs we use for our model.
Revised: Table 3 shows the architectures of visual and audio CNNs we use for our model.

Original: For the visual CNN, we use the SlowFast network (Feichtenhofer et al., 2019) with a ResNet-50 backbone (He et al., 2016).
Revised: For the visual CNN, we use the SlowFast network (Feichtenhofer et al., 2019) with a ResNet-50 backbone (He et al., 2016).

Original: We use the speed ratio α = 8 and the channel ratio β
Revised: We use the speed ratio α = 8 and the channel ratio β

Original: = 1 / 8 for the SlowFast architecture, so T f = 8 × T s .
Revised: = 1 / 8 for the SlowFast architecture, so T f = 8 × T s .

Original: We use different values of T s and T f for different tasks.
Revised: We use different values of T s and T f for different tasks.

Original: During pretraining, we set T s = 4 and T f = 32 .
Revised: During pretraining, we set T s = 4 and T f = 32 .

Original: During ﬁnetuning, we use T s = 8 and T f = 64 for short-video action classiﬁcation on UCF101 (Soomro et al., 2012) while we use T s = 4 and T f = 32 for long-video action classiﬁcation on Charades (Sigurdsson et al., 2016) and Kinetics-Sounds (Arandjelovic &
Revised: During ﬁnetuning, we use T s = 8 and T f = 64 for short-video action classiﬁcation on UCF101 (Soomro et al., 2012) while we use T s = 4 and T f = 32 for long-video action classiﬁcation on Charades (Sigurdsson et al., 2016) and Kinetics-Sounds (Arandjelovic &

Original: Zisserman, 2017).
Revised: Zisserman, 2017).

Original: For the audio CNN, we use a ResNet-50 without the downsampling layer pool 1 to preserve information along both frequency and time axis in early stages.
Revised: For the audio CNN, we use a ResNet-50 without the downsampling layer pool 1 to preserve information along both frequency and time axis in early stages.

Original: We use different values of
Revised: We use different values of

Original: T a for different training phases.
Revised: T a for different training phases.

Original: We set T a = 220 for one-second clip during pretraining while we use T a = 440 for two-second clip during ﬁnetuning.
Revised: We set T a = 220 for one-second clip during pretraining while we use T a = 440 for two-second clip during ﬁnetuning.

Original: [Table]
Revised: [Table]

Original: stride

Original: [Table]
Revised: [Table]

Original: [Table]

Original: stride
Revised: pool

Original: [Table]
Revised: [Table]

Original: stride

Original: [Table]
Revised: [Table]

Original:  stride 
Revised: max stride 1 ,

Original: [Table]
Revised: [Table]

Original: max stride

Original: [Table] [Table]
Revised: [Table] [Table]

Original: – res
Revised: – res

Original: [Table]
Revised: [Table]

Original:   1 × 1 2 ,1 ×
Revised:   1 × 1 2 ,1 ×

Original: [Table]
Revised: [Table]

Revised: ,1 ×

Revised: [Table]

Revised: 2 ,

Revised: [Table]

Original: 2 ,1 × 1 2 ,  ×  3 × 1 2 ,1 × 
Revised: ×  3 × 1 2 , [Table] × [Table]

Revised: [Table]

Revised: 1 × 1 2 ,

Original: [Table]
Revised: [Table]

Revised: (cid:34)

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: ,(cid:35)

Revised: [Table]

Revised: 

Revised: [Table]

Revised: [Table]

Revised: 

Revised: 

Revised: [Table]

Revised:  

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: 2 ,

Revised: 

Revised: [Table]

Revised: (cid:34)

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: (cid:35)

Revised: [Table]

Revised:  

Original: [Table]
Revised: [Table]

Original: 1 × 1 2 , ×(cid:34) 1 × 1 ,3 × 3 ,1 × 1 ,(cid:35) ×res
Revised: [Table] [Table] [Table] 1 2 ,

Original: [Table]
Revised: [Table]

Revised:  

Revised: [Table]

Revised: [Table]

Original:   1 × 1 2 ,1 × 3 2 ,1 × 1 2 ,
Revised: [Table]   [Table]

Original:  ×  3 × 1 2 ,1 × 3 2 ,1 × 1 2 ,  ×(cid:34) 1 × 1 ,3 × 3 ,1 × 1 ,(cid:35) ×res  3 × 1 2 ,1 × 3 2 ,1 × 1 2 ,
Revised: (cid:34) [Table] [Table] (cid:35)

Original:  × 
Revised: [Table]  [Table]  

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: 

Original: [Table]
Revised: [Table]

Original: A. D ATA P REPROCESSING
Revised: A. D ATA P REPROCESSING

Original: We preprocess the data by dividing T -second clips into T non-overlapping parts ( T = 30 for Kinetics700 (Carreira et al., 2019) and T = 10 for AudioSet (Gemmeke et al., 2017)) and sampling 16 frames from each.
Revised: We preprocess the data by dividing T -second clips into T non-overlapping parts ( T = 30 for Kinetics700 (Carreira et al., 2019) and T = 10 for AudioSet (Gemmeke et al., 2017)) and sampling 16 frames from each.

Original: For audio stream, we take waveform sampled at 44.1 kHz and convert it to log-mel-scaled spectrogram.
Revised: For audio stream, we take waveform sampled at 44.1 kHz and convert it to log-mel-scaled spectrogram.

Original: We augment audio data with random frequency/time masking using SpecAugment (Park et al., 2019), and visual data with color normalization, random resizing, random horizontal ﬂip, and random cropping to obtain 112 × 112 pixel frames; for test data, we resize videos to 128 pixels on the shorter side and take three equidistant crops of 128 × 128 pixels to cover the entire region.
Revised: We augment audio data with random frequency/time masking using SpecAugment (Park et al., 2019), and visual data with color normalization, random resizing, random horizontal ﬂip, and random cropping to obtain 112 × 112 pixel frames; for test data, we resize videos to 128 pixels on the shorter side and take three equidistant crops of 128 × 128 pixels to cover the entire region.

Original: We also apply audio-visual synchronized temporal jittering (Patrick et al., 2020). 
Revised: We also apply audio-visual synchronized temporal jittering (Patrick et al., 2020). [Figure] [Figure]

Original: A. D OWNSTREAM E VALUATION
Revised: D OWNSTREAM E VALUATION

Original: For evaluation on UCF101, we follow the test protocol of (Feichtenhofer et al., 2019):
Revised: For evaluation on UCF101, we follow the test protocol of (Feichtenhofer et al., 2019):

Original: We sample 10 clips from each test video at a uniform time interval, and for each sampled clip, we take three equidistant spatial crops, resulting in a total of 30 views.
Revised: We sample 10 clips from each test video at a uniform time interval, and for each sampled clip, we take three equidistant spatial crops, resulting in a total of 30 views.

Original: We use each of the 30 views as input to our visual CNN and average the prediction scores from all 30 views to obtain the ﬁnal prediction result.
Revised: We use each of the 30 views as input to our visual CNN and average the prediction scores from all 30 views to obtain the ﬁnal prediction result.

Original: For evaluation on ESC-50 (Piczak, 2015a), we extract 10 equally spaced 2-second clips from each test audio sample.
Revised: For evaluation on ESC-50 (Piczak, 2015a), we extract 10 equally spaced 2-second clips from each test audio sample.

Original: We use each of 10 clips as input to our audio CNN and average the prediction scores to obtain the ﬁnal prediction result.
Revised: We use each of 10 clips as input to our audio CNN and average the prediction scores to obtain the ﬁnal prediction result.

Original: For evaluation on Charades and Kinetics-Sounds, we use three audio-visual sequences with different spatial crops from a test video and max-pool/average the prediction scores from each sequence, respectively.[Figure]
Revised: For evaluation on Charades and Kinetics-Sounds, we use three audio-visual sequences with different spatial crops from a test video and max-pool/average the prediction scores from each sequence, respectively.

Original: A. O PTIMIZATION
Revised: A. O PTIMIZATION

Original: In all experiments, we use the AMSGrad (Reddi et al., 2018) variant of AdamW (Loshchilov &
Revised: In all experiments, we use the AMSGrad (Reddi et al., 2018) variant of AdamW (Loshchilov &

Original: Hutter, 2019) optimizer with β 1 = 0 .
Revised: Hutter, 2019) optimizer with β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 98 , L2 weight decay of 1e-4.
Revised: 98 , L2 weight decay of 1e-4.

Original: We use a learning rate warm-up for the ﬁrst 6% of iterations followed by a linear decay of learning rate.
Revised: We use a learning rate warm-up for the ﬁrst 6% of iterations followed by a linear decay of learning rate.

Original:  We pretrain our model on Kinetics-700 (Carreira et al., 2019) with a batch size 256 for 220K iterations and AudioSet (Gemmeke et al., 2017) with a batch size 300 for 220K iterations in the main experiments; for the ablation study, we use a much smaller batch size of 4 and pretrain our model onKineticsfor 80K iterations. From the observations of Lezcano-Casado and Martínez-Rubio (Lezcano-Casado & Martínez-Rubio, 2019), we have 10 times less learning rate for the orthogonal parameters than that for the non-orthogonal parameters: we use 1e-5 for the former and 1e-4 for the latter.
Revised: From the observations of Lezcano-Casado and Martínez-Rubio (Lezcano-Casado & Martínez-Rubio, 2019), we have 10 times less learning rate for the orthogonal parameters than that for the nonorthogonal parameters: we use 1e-5 for the former and 1e-4 for the latter. We pretrain our model on Kinetics-700 (Carreira et al., 2019) with a batch size 256 for 220K iterations and AudioSet (Gemmeke et al., 2017) with a batch size 300 for 220K iterations in the main experiments; for the ablation study, we use a much smaller batch size of 4 and pretrain our model on

Revised: Kinetics-700 for 80K iterations.

Original: For ﬁnetuning on UCF101, we train our model for 40K iterations with a batch size of 64 and learning rate of 0.02.
Revised: For ﬁnetuning on UCF101, we train our model for 40K iterations with a batch size of 64 and learning rate of 0.02.

Original: For ﬁnetuning on ESC-50, we train  our model for 10K iterations with a batch size of 128 and learning rate of 0.02.
Revised: For evaluation on ESC-50, we train a multi-class one-vs-all linear SVM on top of our ﬁxed audio CNN for 38K iterations with a batch size of 128 and learning rate of 0.003.

Original: For ﬁnetuning on Charades, we train for 40K iterations with a batch size of 8, with learning rate of 0.001 for the classiﬁer and CNN parameters, 1e-5 for the orthogonal parameters and 1e-4 for the rest parameters.
Revised: For ﬁnetuning on Charades, we train for 40K iterations with a batch size of 8, with learning rate of 0.001 for the classiﬁer and CNN parameters, 1e-5 for the orthogonal parameters and 1e-4 for the rest parameters.

Original: For ﬁnetuning on Kinetics-Sounds, we train for 24K iterations with a batch size of 32, with learning rate of 0.005 for the classiﬁer and CNN parameters, 1e-4 for the orthogonal parameters and 1e-3 for the rest parameters.
Revised: For ﬁnetuning on Kinetics-Sounds, we train for 24K iterations with a batch size of 32, with learning rate of 0.005 for the classiﬁer and CNN parameters, 1e-4 for the orthogonal parameters and 1e-3 for the rest parameters.

Original: B E XTRA R ESULTS FROM THE A BLATION S TUDY B. N EGATIVE S AMPLING S TRATEGIES
Revised: B E XTRA R ESULTS FROM THE A BLATION S TUDY B. N EGATIVE S AMPLING S TRATEGIES

Original: We proposed the content-aware negative sampling strategy (CANS) using pairwise l 2 distances between CNN embeddings.
Revised: We proposed the content-aware negative sampling strategy (CANS) using pairwise l 2 distances between CNN embeddings.

Original: We introduced two variants of CANS: CANS-Dissimilar that favors negatives that are dissimilar to the positive instance and CANS-Similar that favors negatives that are similar to the positive instance.
Revised: We introduced two variants of CANS: CANS-Dissimilar that favors negatives that are dissimilar to the positive instance and CANS-Similar that favors negatives that are similar to the positive instance.

Original: We chose to use CANS-Similar based on the results from our ablation study presented in the main paper, Table 1 (b).
Revised: We chose to use CANS-Similar based on the results from our ablation study presented in the main paper, Table 1 (b).

Original: Figure 3 (a) in this appendix provides additional evidence that supports our decision.
Revised: Figure 3 (a) in this appendix provides additional evidence that supports our decision.

Original: We see that the loss of CANS-Disimilar initially drops rapidly but starts increasing around iteration 7K and continues to increase until around 15K; this is mainly caused by the visual MEP loss shown in Figure 3 (a-2).
Revised: We see that the loss of CANS-Disimilar initially drops rapidly but starts increasing around iteration 7K and continues to increase until around 15K; this is mainly caused by the visual MEP loss shown in Figure 3 (a-2).

Original: One explanation for this might that CANS-Disimilar is easier to solve than
Revised: One explanation for this might that CANS-Disimilar is easier to solve than

Original: CANS-Similar , which causes the loss landscape of CANS-Disimilar to contain too many shallow local minima compared to that of CANS-Similar .
Revised: CANS-Similar , which causes the loss landscape of CANS-Disimilar to contain too many shallow local minima compared to that of CANS-Similar .

Original: Recall that we use a learning rate warm-up for the ﬁrst 6% of iterations during pretraining; this roughly equals to the ﬁrst 13K (out of 220K) iterations.
Revised: Recall that we use a learning rate warm-up for the ﬁrst 6% of iterations during pretraining; this roughly equals to the ﬁrst 13K (out of 220K) iterations.

Original: Given this, we speculate that the model got out of a local minima around iteration 7K (most likely due to the increasing learning rate), and then eventually settled in another (bad) local minima after the warm-up period ended.
Revised: Given this, we speculate that the model got out of a local minima around iteration 7K (most likely due to the increasing learning rate), and then eventually settled in another (bad) local minima after the warm-up period ended.

Original: Compared to this, we observe much milder learning dynamics with CANS-Similar : the loss decreases relatively slowly but steadily, and eventually leaps around 35K to go below the loss of CANS-Disimilar .
Revised: Compared to this, we observe much milder learning dynamics with CANS-Similar : the loss decreases relatively slowly but steadily, and eventually leaps around 35K to go below the loss of CANS-Disimilar .

Original: We, again, believe that this is because CANS-Similar is more difﬁcult to solve than CANS-Disimilar (as shown by the slower decrease in loss values), which caused the resulting loss landscape to contain steeper local minima.
Revised: We, again, believe that this is because CANS-Similar is more difﬁcult to solve than CANS-Disimilar (as shown by the slower decrease in loss values), which caused the resulting loss landscape to contain steeper local minima.

Original: Our model eventually found one of those after round 40K of iterations, resulting in a better performing model in the downstream tasks shown in Table 1 (b) of the main paper (the loss kept slowly decreasing after iteration 50K).
Revised: Our model eventually found one of those after round 40K of iterations, resulting in a better performing model in the downstream tasks shown in Table 1 (b) of the main paper (the loss kept slowly decreasing after iteration 50K).

Original: B. P ARAMETER S HARING S CHEMES
Revised: B. P ARAMETER S HARING S CHEMES

Original: We proposed a cross-Transformer weight sharing technique, which decomposes weight matrices with low-rank approximation.
Revised: We proposed a cross-Transformer weight sharing technique, which decomposes weight matrices with low-rank approximation.

Original: Recall that each layer of a Transformer contains the multi-head attention layer weights { W q , W k , W v , W b } and the feed-forward layer weights { W c , W d } .
Revised: Recall that each layer of a Transformer contains the multi-head attention layer weights { W q , W k , W v , W b } and the feed-forward layer weights { W c , W d } .

Original: We chose to share all six weight matrices across Transformers, though we could have shared any combination of them.
Revised: We chose to share all six weight matrices across Transformers, though we could have shared any combination of them.

Original: To justify this design choice, we empirically compared three variants: (i) Multi-6 that do not share parameters across Transformers, (ii) Multi-6-Part_Att that shares only { W q , W k , W v , W b } (but not { W c , W d } ) and (iii) Multi-6-Part that shares all six weight matrices.
Revised: To justify this design choice, we empirically compared three variants: (i) Multi-6 that do not share parameters across Transformers, (ii) Multi-6-Part_Att that shares only { W q , W k , W v , W b } (but not { W c , W d } ) and (iii) Multi-6-Part that shares all six weight matrices.

Original: Figure 3 (b) shows that there is not much difference between all the variants in terms of the loss curves; we chose to use Multi-6-Part that requires the least number of parameters.
Revised: Figure 3 (b) shows that there is not much difference between all the variants in terms of the loss curves; we chose to use Multi-6-Part that requires the least number of parameters.

Original: We showed that our approach outperforms Multi-6 in the ablation study (Table 1 (c-left) in the main paper).
Revised: We showed that our approach outperforms Multi-6 in the ablation study (Table 1 (c-left) in the main paper).

Original: B. J USTIFICATION FOR THE MEP L OSS F ORMULATION
Revised: B. J USTIFICATION FOR THE MEP L OSS F ORMULATION

Original: Since the multimodal Transformer h AV has access to both visual and audio inputs, one might think that the model could “leak” information about visual input into z a and information about audio input into z v , which could make MEP trivial to solve.
Revised: Since the multimodal Transformer h AV has access to both visual and audio inputs, one might think that the model could “leak” information about visual input into z a and information about audio input into z v , which could make MEP trivial to solve.

Original: Here we show that this is not the case.
Revised: Here we show that this is not the case.

Original: By construction, we mask the same positions in audio and visual streams when designing the MEP task, so the model has no access to the masked input even in a cross-modal manner.
Revised: By construction, we mask the same positions in audio and visual streams when designing the MEP task, so the model has no access to the masked input even in a cross-modal manner.

Original: Empirically, removing the third term in Eq.
Revised: Empirically, removing the third term in Eq.

Original: 6 ( L NCE ([ x a ; x v ] , ˜ z ) ) leads to performance degradation in Kinetics-Sounds, i.e., top-1 accuracy 66.7% vs. ours 67.5% (see Table 1), which suggests that solving the MEP task in the multimodal Transformer is beneﬁcial to our model.
Revised: 6 ( L NCE ([ x a ; x v ] , ˜ z ) ) leads to performance degradation in Kinetics-Sounds, i.e., top-1 accuracy 66.7% vs. ours 67.5% (see Table 1), which suggests that solving the MEP task in the multimodal Transformer is beneﬁcial to our model.

Original: B. J USTIFICATION FOR THE CPP L OSS F ORMULATION
Revised: B. J USTIFICATION FOR THE CPP L OSS F ORMULATION

Original: Recall that our CPP loss has two terms; the ﬁrst term uses the summary embeddings s g and the second term uses output embeddings s h sampled at random positions; see Eq. 7.
Revised: Recall that our CPP loss has two terms; the ﬁrst term uses the summary embeddings s g and the second term uses output embeddings s h sampled at random positions; see Eq. 7.

Original: One could argue that the two terms are redundant as bidirectional Transformers have “one-step” access to all the input embeddings, and thus solving CPP only with the summary embeddings (the ﬁrst term) would be enough.
Revised: One could argue that the two terms are redundant as bidirectional Transformers have “one-step” access to all the input embeddings, and thus solving CPP only with the summary embeddings (the ﬁrst term) would be enough.

Original: This is not the case.
Revised: This is not the case.

Original: We encode s h with position-speciﬁc information through the time embeddings p t , which makes every s h different compared to s g .
Revised: We encode s h with position-speciﬁc information through the time embeddings p t , which makes every s h different compared to s g .

Original: Empirically, we ﬁnd that removing the second term of Eqn.
Revised: Empirically, we ﬁnd that removing the second term of Eqn.

Original: 7 ( s h ) in our CPP loss leads to an inferior accuracy 66.9% vs. ours 67.5% on
Revised: 7 ( s h ) in our CPP loss leads to an inferior accuracy 66.9% vs. ours 67.5% on

Original: Kinetics-Sounds, suggesting its importance in learning.
Revised: Kinetics-Sounds, suggesting its importance in learning.

Original: B. U SE OF M ODALITY E MBEDDINGS IN THE M ULTIMODAL T RANSFORMER
Revised: B. U SE OF M ODALITY E MBEDDINGS IN THE M ULTIMODAL T RANSFORMER

Original: We use modality embeddings m v and m a as part of input to the multimodal Transformer in order to distinguish embeddings coming from visual and audio Transformers.
Revised: We use modality embeddings m v and m a as part of input to the multimodal Transformer in order to distinguish embeddings coming from visual and audio Transformers.

Original: They are learnable weights trained end-to-end with other parameters.
Revised: They are learnable weights trained end-to-end with other parameters.

Original: Conceptually, incorporating modality-discriminative embeddings is crucial because of our aggressive weight sharing scheme.
Revised: Conceptually, incorporating modality-discriminative embeddings is crucial because of our aggressive weight sharing scheme.

Original: Without them, the multimodal
Revised: Without them, the multimodal

Original: Transformer will see the output from audio/visual Transformers ( y a and y v ) as if they are coming from the same distribution because the two Transformers share a large part of weights.
Revised: Transformer will see the output from audio/visual Transformers ( y a and y v ) as if they are coming from the same distribution because the two Transformers share a large part of weights.

Original: Using modality embeddings encourages our model to preserve modality-speciﬁc information in the ﬁnal output, and this empirically leads to performance improvements: ours 67.5% vs. without modality embeddings 67.1% on Kinetics-Sounds.
Revised: Using modality embeddings encourages our model to preserve modality-speciﬁc information in the ﬁnal output, and this empirically leads to performance improvements: ours 67.5% vs. without modality embeddings 67.1% on Kinetics-Sounds.

Revised: B. O N THE I MPORTANCE OF E ND - TO -E ND P RETRAINING

Revised: Previous work in multimodal visual-and-language tasks (Tan & Bansal, 2019; Lu et al., 2019) point out that using partially ﬁxed Transformers of different modalities is detrimental to multimodal representation learning (c.f., Sun et al.

Revised: (2019a;b)).

Revised: We make the same observation in our audiovisual learning scenario.

Revised: We compare two variants of Multi-6 Part in Table 1 (c), each of which pretrains only the audio (or visual) CNN/Transformer in the ﬁrst half of pretraining stage and then continues pretraining the remaining weights while ﬁxing the weights of the audio (or visual)

Revised: CNN/Transformer in the second half.

Revised: This leads to inferior performance (audio-ﬁxed 62.8% and visual-ﬁxed 63.1% vs. ours 67.5%), which is consistent with the results reported in Tan & Bansal (2019);

Revised: Lu et al.