Original: LAVA: Language Audio Vision Alignment for Pre-Training Transformers on Video Data
Revised: LAVA: Language Audio Vision Alignment for Data-Efficient Video Pre-Training

Original: Anonymous Authors

Original: Abstract
Revised: Abstract

Original: Generating representations of video data is of key importance in advancing the field of machine perception.
Revised: Generating representations of video data is of key importance in advancing the field of machine perception.

Original: Most current techniques rely on handannotated data, which can be difficult to work with, expensive to generate, and hard to scale.
Revised: Most current techniques rely on handannotated data, which can be difficult to work with, expensive to generate, and hard to scale.

Original: In this work, we propose a novel learning approach based on contrastive learning, LAVA, which is capable of learning joint language, audio, and video representations in a self-supervised manner.
Revised: In this work, we propose a novel learning approach based on contrastive learning, LAVA, which is capable of learning joint language, audio, and video representations in a self-supervised manner.

Original: We pre-train LAVA on the Kinetics 700 dataset using transformer encoders to learn representations for each modality.
Revised: We pre-train LAVA on the Kinetics 700 dataset using transformer encoders to learn representations for each modality.

Original: We then demonstrate that LAVA performs competitively with the current state-ofthe-art self-supervised and weakly-supervised pretraining techniques on UCF-101 and HMDBvideo action recognition while using a fraction of the unlabeled data.
Revised: We then demonstrate that LAVA performs competitively with the current state-ofthe-art self-supervised and weakly-supervised pretraining techniques on UCF-101 and HMDBvideo action recognition while using a fraction of the unlabeled data.

Original: 1. Introduction
Revised: 1. Introduction

Original: Supervised learning has generally driven the progressof video representation learning, however, labeling datasets is both time-consuming and expensive, making it especially hard to leverage large amounts of data using supervised learning.
Revised: Supervised learning has generally driven the progress video representation learning, however, labeling datasets is both time-consuming and expensive, making it especially hard to leverage large amounts of data using supervised learning.

Revised: Moreover, while attention-based architectures such as

Original: Moreover, while attention-based architectures such as (Dosovitskiy et al., 2020) (Arnab et al., 2021; Bertasius et al., 2021) have started to outperform CNNs on key benchmarks, they often require much larger amounts of training data thanCNNs.
Revised: Dosovitskiy et al. Arnab et al. ; Bertasius et al. (2021) have started to outperform CNNs on key benchmarks, they often require much larger amounts of training data than

Revised: CNNs.

Original: Self-supervised methods have emerged to answer the challenge; as powerful pre-training strategies for vision tasks, they can scale to larger training datasets without being constrained by labeling needs.
Revised: Self-supervised methods have emerged to answer the challenge; as powerful pre-training strategies for vision tasks, they can scale to larger training datasets without being constrained by labeling needs.

Original: One common approach is to use data augmentation to learn representational invariants for vision (Qian et al., 2021; Jing et al., 2018).
Revised: One common approach is to use data augmentation to learn representational invariants for vision Qian et al.

Original: Instead 1 Anonymous Institution, Anonymous City, Anonymous Region,

Original: Anonymous Country. Correspondence to: Anonymous Author < anon.email@domain.com > . Preliminary work.
Revised: ; Jing et al.

Original: Under review by the First Workshop of Pre- training: Perspectives, Pitfalls, and Paths Forward at ICML 2022.

Original: Do not distribute. of relying on hand-designed augmentations for the visual modality, another approach is to exploit the multi-modal nature of video and learn audio-visual correspondence, as seen with contrastive methods such as (Morgado et al., 2020b;
Revised: Instead of relying on hand-designed augmentations for the visual modality, another approach is to exploit the multi-modal nature of video and learn audio-visual correspondence, as seen with contrastive methods such as those in Morgado et al. (2020b);

Original: Patrick et al., 2021; 2020a; Morgado et al., 2020a; Korbar et al., 2018).
Revised: Patrick et al. (2021; 2020a); Morgado et al. (2020a) ; Korbar et al.

Original: Similarly, other methods use text metadata from videos to learn joint visual-text representations in a self/weakly-supervised manner (Stroud et al., 2020; Patrick et al., 2020b; Li & Wang, 2020; Sun et al., 2019a;
Revised: Similarly, other methods use text metadata from videos to learn joint visual-text representations in a self/weakly-supervised manner Stroud et al. (2020); Patrick et al. (2020b) ; Li & Wang (2020); Sun et al.

Revised: (2019a)

Original:  Miech et al., 2020; Sun et al., 2019b).
Revised: ; Miech et al. (2020); Sun et al. (2019b).

Original: A less common but even more label-efficient strategy is to learn audio, visual and text representations together.
Revised: A less common but even more label-efficient strategy is to learn audio, visual and text representations together.

Original: While methods such as (Akbari et al., 2021; Alayrac et al., 2020;
Revised: While methods such as Akbari et al. ; Alayrac et al.

Original: Chen et al., 2021) pre-train on these three modalities, they do so leveraging HowTo100M, which has a massiveyears of unlabeled video data.
Revised: Chen et al. pre-train on these three modalities, they do so leveraging HowTo100M, which has a massive 15 years of unlabeled video data.

Original: Moreover, excepting Akbari et al.
Revised: Moreover, excepting Akbari et al.

Original: (Akbari et al., 2021), most of the above techniques use highly modality-specific encoders (e.g. only CNNs for vision), instead of exploring more generic, attention-based backbones for all modalities.
Revised: (2021), most of the above techniques use highly modalityspecific encoders (e.g. only CNNs for vision), instead of exploring more generic, attention-based backbones for all modalities.

Original: These observations are the main motivation for LAVA, which introduces a more data and label-efficient method for pre-training transformer encoders on audio, visual, and text modalities for video data.
Revised: These observations are the main motivation for LAVA, which introduces a more data and label-efficient method for pre-training transformer encoders on audio, visual, and text modalities for video data.

Original: To achieve this, our proposed method uses novel cross-modal and centroid-based contrastive objectives seen in Figure 1.
Revised: To achieve this, our proposed method uses novel cross-modal and centroid-based contrastive objectives seen in Figure 1.

Original: We evaluate pre-trained
Revised: We evaluate pre-trained

Original: LAVA on UCF-101 and HMDB-51 downstream video action recognition.
Revised: LAVA on UCF-101 and HMDB-51.

Original: 2. Pre-Training Approach
Revised: 2. Pre-Training Approach

Original: Overview: Given a set of n unlabelled videos X , each video x i ∈
Revised: Overview Given a set of n unlabelled videos X , each video x i ∈

Original: X is decomposed into different modalities a i , v i and t i , which are audio, visual and text features, respectively.
Revised: X is decomposed into different modalities a i , v i and t i , which are audio, visual and text features, respectively.

Original: Details regarding modality-specific extraction are in Section 3.
Revised: Details regarding modality-specific extraction are in Section 3.

Original: Then, LAVA’s three encoders f a , f v , and f t produce output embeddings given their respective modality features m i as inputs.
Revised: Then, LAVA’s three encoders f a , f v , and f t produce output embeddings given their respective modality features m i as inputs.

Original: LAVA then uses projection functions to project these embeddings to various multi-modal latent spaces (e.g. audiovideo, video-text and audio-video-text spaces) in some R d .
Revised: LAVA then uses projection functions to project these embeddings to various multi-modal latent spaces (e.g. audio-video, video-text and audio-videotext spaces) in some R d .

Original: Let us denote z m as the embedding for modality m .
Revised: Let us denote z m as the embedding for modality m .

Original: We define projection function g m,m ′ to project z m and z m ′ , to a multi-modal latent space m, m ′ .
Revised: We define projection function g m,m ′ to project z m and z m ′ , to a multi-modal latent space m, m ′ .

Original: In these multi-modal latent spaces, we apply a contrastive framework to jointly compare embeddings using a similarity function s , such that ∀ i, m ′ ̸
Revised: In these multi-modal latent spaces, we apply a contrastive framework to jointly compare embeddings using a similarity function s , such that ∀ i, m ′ ̸

Original: = m we have high s ( g m,m ′ ( z m,i ) , g m,m ′ ( z m ′ ,j ) and ∀
Revised: = m we have high s ( g m,m ′ ( z m,i ) , g m,m ′ ( z m ′ ,i ) and ∀

Original: i ̸
Revised: i ̸

Original: = j, m ′ ̸
Revised: = j, m ′ ̸

Original: = m we have low s ( f m ( m i ) , f m ′ ( m ′ j )) .
Revised: = m we have low s ( f m ( m i ) , f m ′ ( m ′ j )) .

Original: Cross-Modal Contrastive Loss 
Revised: Cross-Modal Contrastive Loss We use the function s ( · , · ) =

Original: We use exp ( g m,m ′ ( z m,i ) T g m,m ′ ( z m ′ ,j ) /τ ) as the similarity function, where τ denotes temperature.
Revised: exp ( g m,m ′ ( z m,i ) T g m,m ′ ( z m ′ ,j ) /τ ) as the similarity function, where τ denotes temperature, and the noise contrastive estimation (NCE) Gutmann &

Original: We use the noise contrastive estimation (NCE) (Gutmann & Hyv ¨ arinen, 2010) as our contrastive loss; positive pairs are embeddings from the same instance and negatives are embeddings from difference instances.
Revised: Hyv¨arinen (2010) as our contrastive loss, where positive pairs are embeddings from the same instance and negatives are embeddings from difference instances.

Original: This objective is intended to make LAVA a dictionary, effectively mapping input features a i , v i , t i to unified representations
Revised: This objective is intended to make LAVA a dictionary, effectively mapping input features a i , v i , t i to unified representations

Original: z i in a multi-modal latent space.
Revised: z i in a multi-modal latent space.

Original: The NCE loss is formulated below (for simplicity the projection functions are omitted):
Revised: The NCE loss is formulated below (for simplicity the projection functions are omitted):

Original: [Equation]
Revised: [Equation]

Original: Following (Miech et al., 2020; Alayrac et al., 2020; Akbari et al., 2021), we use NCE for audio-video and video-text pairs:
Revised: [Figure] [Equation] [Figure] [Equation]

Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: Centroid Contrastive Loss LAVA also enforces audiovideo-text correspondence via a novel centroid contrastive loss.
Revised: [Figure] [Equation] Following Miech et al. (2020), we use NCE for audio-video and video-text pairs:

Revised: [Equation]

Revised: Centroid Contrastive Loss LAVA also enforces audiovideo-text correspondence via a novel centroid contrastive loss.

Original: Following Chen et al.
Revised: Following Chen et al.

Original: (Chen et al., 2021), for a given instance x i , we calculate a centroid c i by averaging projected LAVA embeddings: c i = ( g avt ( z a,i ) + g avt ( z v,i ) + g avt ( z t,i )) / 3 .
Revised: (2021), for a given instance x i , we calculate a centroid c i by averaging projected LAVA embeddings: c i = ( g avt ( z a,i ) + g avt ( z v,i ) + g avt ( z t,i )) / 3 .

Original: g avt projects modal embeddings to a joint, tri-modal latent space.
Revised: g avt projects modal embeddings to a joint, tri-modal latent space.

Original: However, unlike Chen et al.
Revised: However, unlike Chen et al.

Original: (Chen et al., 2021)

Original: we do not k-means cluster the centroids via k-means clustering and align embeddings to their centroid’s cluster assignment.
Revised: we do not kmeans cluster the centroids via k-means clustering and align embeddings to their centroid’s cluster assignment.

Original: Thus, we avoid potential detriments of grouping representations into a fixed K clusters across all batches and the additional training time needed for clustering.
Revised: Thus, we avoid potential detriments of grouping representations into a fixed K clusters across all batches and the additional training time needed for clustering.

Original: Instead, we instead directly optimize for alignment between embeddings and their centroid via the following loss:
Revised: Instead, we instead directly optimize for alignment between embeddings and their centroid via the following loss:

Original: [Equation]
Revised: [Equation]

Original: Combining all losses, we define the total loss for LAVA’s pre-training:
Revised: Combining all losses, we define the total loss for LAVA’s pre-training:

Original: [Equation]
Revised: [Equation]

Original: 3. Experiments
Revised: 3. Experiments

Original: Pre-training: We use the training set of Kinetics-700, which currently has 480k videosacross 700 action categories, of which 300k videos have usable audio and webcrawled titles.
Revised: Pre-training We use the training set of Kinetics-700: 480k videos of which 300k videos have usable audio and [Table] [Table] web-crawled titles.

Original: To the best of our knowledge, LAVA is the first method to extract this text data for Kinetics-700. Given that Kinetics is among the most commonly used video pre-training datasets and that multi-modal pre-training in computer vision is becoming more common, webelieve this is a meaningful contribution and intend to release this data for others to use.
Revised: To the best of our knowledge, LAVA is the first method to extract this text data for Kineticsand we intend to release this data for others to use.

Original: Video clips are [16 x 224 x 224 x 3] atfps, with random h-flip and random space crop augmentations.
Revised: Video clips are [16 x 224 x 224 x 3] at 10 fps, with random h-flip and space-crop augmentations.

Original: Audio features are [80, 256] log mel-spectrograms augmented with Gaussian noise.
Revised: Audio features are [80, 256] log mel-spectrograms augmented with Gaussian noise.

Original: Text sequences have a maximum length of 128 from a 48k vocab size using BPE tokenization.
Revised: Text sequences have a maximum length of 128 from a 48k vocab size using BPE tokenization.

Original: Most video transformers use ImageNet-pretrained ViT (Arnab et al., 2021; Bertasius et al., 2021), pre-train on massive video datasets like HowTo100M as in (Akbari et al., 2021) or use convolutions followed transformers as in (Patrick et al., 2021).
Revised: Most video transformers use ImageNet-pre-trained ViT Arnab et al. ; Bertasius et al. (2021), pre-train on massive video datasets like HowTo100M as in Akbari et al.

Revised: or use convolutions followed transformers as in Patrick et al.

Original: We follow the latter, except we freeze a ImageNet-pre-trained ResNet-18 backbone and use it to extract frame-wise patch features for each clip.
Revised: We follow the latter, except we freeze a ImageNet-pre-trained ResNet18 backbone and use it to extract frame-wise patch features for each clip.

Original: Audio features are encoded by an MLP.
Revised: Audio features are encoded by an MLP.

Original: Text tokens are mapped to embeddings.
Revised: Text tokens are mapped to embeddings.

Original: Besides the audio MLP and text embedding layer, all LAVA feature extraction is entirely off-line, meaning no gradients are propagated to it.

Original: As seen in Figure 2, audio, video and text features are then encoded by transformer encoders f a , f v , and f t , each with 4 layers and a hidden size of 1024.
Revised: As seen in Figure 2, audio, video and text features are then encoded by transformer encoders f a , f v , and f t , each with 4 layers and a hidden size of 1024.

Original: Missing modalities in a batch are masked out during loss calculation.
Revised: Missing modalities in a batch are masked out during loss calculation.

Original: We use a batch size of 32, 0.07 temperature, 1 e − 5 learning rate, Adam optimizer, and a cosine-learning schedule.
Revised: We use a batch size of 32, 0. temperature, 1 e − 5 learning rate, Adam optimizer, and a cosine-learning schedule.

Original: Pre-training is done forepochs on a single Titan X GPU.
Revised: Pre-training is done for 25 epochs on a single Titan X GPU.

Original: UCF-101 Downstream The UCF-101 dataset includes 13k videos across 101 action categories.
Revised: UCF-101 Downstream The UCF-101 dataset has 13k videos across 101 action categories.

Original: We train a linear classifier on top of the frozen LAVA video encoder using a 1elearning rate and a batch size of 32.
Revised: We train a linear classifier on top of the frozen LAVA video encoder using a 1elearning rate and a batch size of 32.

Original: Also, we compare the downstream performance of the classifier when using the frozen video encoder vs. frozen audio and video encoders.
Revised: Also, we compare the downstream performance of the classifier when using the frozen video encoder vs. frozen audio and video encoders.

Original: Logits are averaged across multiple clips per test set video.
Revised: Logits are averaged across multiple clips per test set video.

Original: Top-1 accuracy is averaged across all 3 splits, unless otherwise specified.
Revised: Top-1 accuracy is averaged across all 3 splits.

Original: HMDB-51 Downstream
Revised: HMDB-51 Downstream

Original: The HMDB-51 dataset includes 7k videos across 51 action categories and three splits.
Revised: The HMDB-51 dataset has 7k videos across 51 action categories.

Original: We follow the UCF-101 evaluation procedure, except we do not evaluate with audio+video mode as most HMDB-51 videos have no audio.
Revised: We follow the UCF- [Table] 101 evaluation procedure, except we do not evaluate with audio+video mode as most HMDB-51 videos have no audio.

Original: Otherwise, ourevaluation procedure followsthat of UCF-101.
Revised: Otherwise, our procedure follows UCF-101.

Original: 4. Results4.1. Downstream Tasks
Revised: 4. Results

Original:  Table 1 compares LAVA linear evaluation performance on UCF-101 and HMDB-51 to various self-supervised benchmarks.
Revised: Downstream Tasks Table 1 compares LAVA linear evaluation performance on UCF-101 and HMDB-51 to various self-supervised benchmarks.

Original: LAVA outperforms all methods trained on datasets comparable to Kinetics-700 size besides (Li & Wang, 2020;
Revised: LAVA outperforms all methods trained on datasets comparable to Kinetics-700 size besides Li & Wang (2020);

Original: Qian et al., 2021).
Revised: Qian et al.

Original: This can be attributed to LAVA’s increased data efficiency as it makes use of audio, video, and text for each sample, whereas these benchmarks rely on at most two modalities.
Revised: This can be attributed to LAVA’s increased data efficiency as it makes use of audio, video, and text for each sample, whereas these benchmarks rely on at most two modalities.

Original: Notably, LAVA significantly outperforms Sun et al.
Revised: Notably, LAVA significantly outperforms Sun et al.

Original: (Sun et al., 2019a), which omits audio and uses ASR-extracted captions as text, whereas LAVA pretrains on audio and video titles.
Revised: (2019a), which omits audio and uses ASR-extracted captions as text, whereas LAVA is pre-trained on audio and video titles.

Original: LAVA also slightly outperforms Patrick et al.
Revised: LAVA also slightly outperforms Patrick et al.

Original: (Patrick et al., 2021), which has a R(2+1)-18+transformer video encoder and ResNet audio encoder, but does not pre-train on text.
Revised: (2021), which has a R(2+1)- 18+transformer video encoder and ResNet audio encoder, but does not pre-train on text.

Original: LAVA performs competitively with Liet al. (Li & Wang, 2020), which also uses video titles as text, and is outperformed by Qian et al.
Revised: LAVA performs competitively with Li & Wang (2020), which also uses video titles as text, and is outperformed by Qian et al.

Original: (Qian et al., 2021), whichonly uses the video modality, but makes use of extensive spatiotemporal augmentations and a large batch size of 1024, both of which add significantly higher computational cost.
Revised: (2021), which uses the video modality using extensive spatio-temporal augmentations and a large batch size of 1024. However, both methods

Original: Moreover, (Li & Wang, 2020;
Revised: Li & Wang (2020); Qian et al.

Original: Qian et al., 2021) both use ResNet3D-50 as their backbone, which is more performant when trained from scratch than transformer encoders trained with the same amount of data (Bertasius et al., 2021; Arnab et al., 2021).
Revised: (2021) use ResNet3D- as their backbone, which is more performant when trained from scratch than transformer encoders trained with the same amount of data Bertasius et al.

Original: LAVA is pretrained with much lower compute (e.g. single GPU) than these benchmarks, and LAVA’s transformer encoders are pre-trained from scratch for all modalities as part of our goal to explore more general, attention-based mechanisms forprocessing video data. Given these differences in pretraining, LAVA’s performance relative to (Li & Wang, 2020; Qian et al., 2021) is reasonably competitive.
Revised: See Sectionfor plans to directly compare for to these methods.

Original: Interestingly, Miech et al.
Revised: Interestingly, Miech et al.

Original: (Miech et al., 2020), despite having been pre-trained on over 150X samples than Kinetics700, only slightly outperforms LAVA on UCF-101 and HMDB-51.
Revised: (2020), despite having been pre-trained on over 150X samples than Kinetics-700, only slightly outperforms LAVA on UCF-101 and HMDB-51.

Original: As Miech et al.
Revised: We believe this is because Miech et al.

Original: (Miech et al., 2020) uses instructional captions for text and does not pre-train on audio, this indicates pre-training on audio and titles-based text can achieve comparable downstream performance while vastly increasing data efficiency.
Revised: (2020) uses captionbased text and does not use audio, whereas LAVA is pretrained on audio and titles-based text.

Original: (Akbari et al., 2021; Alayrac et al., 2020) outperform LAVA using audio, video and caption-based text, while Stroud et al.
Revised: Akbari et al. Alayrac et al. (2020) outperform LAVA using audio, video and caption-based text, while Stroud et al.

Original: (Stroud et al., 2020) uses video and titles-based text.
Revised: (2020) uses video and titles-based text.

Original: This performance gap can  be attributed to LAVA being pre-trained on less than 1%the [Table] [Table] amount of  data as these benchmarks.
Revised: While this performance gap can likely be attributed to LAVA’s reliance on less than 1% of the data used in these benchmarks, it does seem using audio and titles-based text can increase data efficiency as LAVA’s performance rivals that of Miech et al.

Original: Given that LAVA uses Kinetics-700 video titles as text, it can be argued that this is a form of weak supervision as per (Li & Wang, 2020; Stroud et al., 2020).
Revised: Given that LAVA uses Kinetics-700 video titles as text, it can be argued that this is a form of weak supervision as per Li & Wang (2020); Stroud et al.

Original: We found that while all HMDB-51 classes are fully covered by Kinetics700 text, the following UCF-101 classes are not covered:
Revised: We found that while all HMDB-51 classes are fully covered by Kinetics700 text, the following UCF-101 classes are not covered:

Original: IceDancing, PizzaTossing, PommelHorse, SkiJet, StillRings.
Revised: IceDancing, PizzaTossing, PommelHorse, SkiJet, StillRings.

Original: The 3-split average top-1 accuracy for these classes is 79.9%, which is comparable to LAVA’s overall 81.3% for UCF101, indicating that LAVA’s representations transfer well to unseen classes downstream.
Revised: The 3-split average top-1 accuracy for these classes is 79.9%, which is comparable to LAVA’s overall 81.3% for UCF101, indicating that LAVA’s representations transfer well to unseen classes downstream.

Original: We understand that as not all benchmarks are pre-trained on audio and video, the common evaluation procedure uses video embeddings only.

Original: However since LAVA is a multimodal model, we use our novel evaluation strategy to quantify downstream performance improvements when using embeddings from multiple modalities.
Revised: Since LAVA is a multi-modal model, we use our novel evaluation strategy to quantify downstream performance improvements when using embeddings from audio and video modalities.

Original: Table 1 shows that audio-video LAVA outperforms video-only by 3%, indicating that audio embeddings can provide additional information for action recognition.
Revised: Table 1 shows that audio-video LAVA outperforms video-only by 3%, indicating that audio embeddings may provide additional information for action recognition.

Revised: See Section 5 for more plans in this direction.

Original: 4.2. Ablations
Revised: Lastly, pre-training LAVA takes 408 GPU hours, which is over 70% faster than all other benchmarks documented in Table 1. Notably, LAVA also completes all pre-training using just 1 GPU, whereas all other benchmarks use few as 4 and as many as 256 accelerators. This combination of 70% fewer accelerators and 70% fewer GPU hours means LAVA is significantly more efficient in terms of computational cost, in addition to label and data-efficiency.

Original:  As shown in  3, we ablate pre-training duration by evaluating LAVA at 1, 10, and 25 epochs of pre-training; video-only increases from 73.1% to 80.0% to 81.3%, while audio+video increases from 76.0% to 82.3% to 84.3%.
Revised: Ablations As shown in Table 3, we ablate pre-training duration by evaluating LAVA at 1, 10, and 25 epochs of pre-training; video-only increases from 73.1% to 80.0% to 81.3%, while audio+video increases from 76.0% to 82.3% to 84.3%.

Original: Lastly, we ablatethe various pre-training losses as seen in Table 2.
Revised: Additionally, we ablate pre-training losses as seen in Table 2.

Original: As expected, LAVA pre-trained without text and centroid contrastive loss performs the worst.
Revised: As expected, LAVA pre-trained without text and centroid contrastive loss performs the worst.

Original: LAVA pretrained with L AV and L V T , but no centroid loss performs worse than LAVA pre-trained with the centroid loss.
Revised: LAVA pretrained with L AV and L V T , but no centroid loss performs worse than LAVA pre-trained with the centroid loss.

Original: This indicates that the text modality and the centroid loss, both ofwhich increase data efficiency by making greater use of
Revised: This indicates that the text modality and the centroid loss, both of

Original: [Table]
Revised: [Figure]

Original: [Table]

Original: [Figure]
Revised: [Figure]

Original:  same amount of samples, significantly improve downstream video action recognition.
Revised: which increase data efficiency by making greater use of the same number of samples, significantly improve downstream video action recognition

Original: The performance boost from using audio+video embeddings during linear evaluation is also seen in Table 2.
Revised: The performance boost from using audio+video embeddings during linear evaluation is also seen in Table 2.

Original: 5. Conclusion
Revised: 5. Conclusion

Original: We present LAVA, a novel self-supervised pre-training method for learning audio, visual, and text representations from unlabeled video data using transformers.
Revised: We present LAVA, a novel self-supervised pre-training method for learning audio, visual, and text representations from unlabeled video data using transformers.

Original: By using multiple modalities and introducing novel cross-modal and centroid contrastive objectives, LAVA increases data efficiency while performing competitively with self and weakly supervised benchmarks.
Revised: By using multiple modalities and introducing novel cross-modal and centroid contrastive objectives, LAVA increases data efficiency while performing competitively with self and weakly supervised benchmarks.

Original: As LAVA’s pre-training effectively combines multi-modal data and generic transformer encoders, we believe it can scale both in terms of the amount on unlabelled data and the number of modalities in the data.
Revised: As LAVA’s pre-training effectively combines multi-modal data and generic transformer encoders, we believe it can scale both in terms of the amount on unlabelled data and the number of modalities in the data.

Original: In future work, we intend to explore pre-training on more/different modalities and new downstream tasks (e.g. retrieval, captioning, video/audio generation) using LAVA.
Revised: Future Work To more directly compare LAVA performance to Li & Wang (2020); Qian et al. (2021), we plan to pre-train the transformer vision encoder on ImageNet first or use the ResNet3D-50 encoder from scratch, as well as larger batch sizes for contrastive pre-training and more extensive video augmentation. We also plan to pre-train LAVA on larger datasets with non-title-based text (e.g. HowTo100M) to compare more directly with Akbari et al.

Revised: ; Alayrac et al.

Revised: Since contrastive pre-training with audio may dilute information in the video embeddings, we plan to compare downstream performance between video embeddings from VT-pretrained LAVA and those of current LAVA.

Revised: Additionally, we plan to ablate how different coefficients between various LAVA losses will affect downstream performance.

Revised: Lastly, we intend to explore pre-training on more/different modalities and new downstream tasks (e.g. retrieval, captioning, video/audio generation) using LAVA.