Original: VIMPAC: V IDEO P RE -T RAINING VIA M ASKED T OKEN P REDICTION AND C ONTRASTIVE L EARNING
Revised: VIMPAC: V IDEO P RE -T RAINING VIA M ASKED T OKEN P REDICTION AND C ONTRASTIVE L EARNING

Original: Anonymous
Revised: Anonymous

Original: authors
Revised: authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Video understanding relies on perceiving the overall global content and modeling its internal connections (e.g., causality, movement, and spatio-temporal correspondence).
Revised: Video understanding relies on perceiving the overall global content and modeling its internal connections (e.g., causality, movement, and spatio-temporal correspondence).

Original: To learn these interactions, we apply a mask-then-predict pre-training task on the discretized video tokens generated via VQ-VAE.
Revised: To learn these interactions, we apply a mask-then-predict pre-training task on the discretized video tokens generated via VQ-VAE.

Original: Unlike language, where the text tokens are more independent, neighboring video tokens typically have strong correlations (e.g., consecutive video frames usually look similar), and hence uniformly masking individual tokens will make the task too trivial to learn useful representations.
Revised: Unlike language, where the text tokens are more independent, neighboring video tokens typically have strong correlations (e.g., consecutive video frames usually look similar), and hence uniformly masking individual tokens will make the task too trivial to learn useful representations.

Original: To deal with this issue, we propose a block masking strategy where we mask neighboring video tokens in both spatial and temporal domains.
Revised: To deal with this issue, we propose a block masking strategy where we mask neighboring video tokens in both spatial and temporal domains.

Original: We also add a contrastive learning objective to further capture the global content by predicting whether the video clips are sampled from the same video.
Revised: We also add a contrastive learning objective to further capture the global content by predicting whether the video clips are sampled from the same video.

Original: We pre-train our model on uncurated videos and show that our pre-trained model can reach state-of-the-art results on several video understanding datasets (e.g.,
Revised: We pre-train our model on uncurated videos and show that our pre-trained model can reach state-of-the-art results on several video understanding datasets (e.g.,

Original: SSV2, Diving48).
Revised: SSV2, Diving48).

Original: Lastly, we provide detailed analyses of the model scalability and pre-training method design.
Revised: Lastly, we provide detailed analyses of the model scalability and pre-training method design.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In recent years, state-of-the-art self-supervised methods have been exploring different directions for pre-training images and text representations, with Contrastive Learning (CL) providing strong results for vision representation learning (Oord et al., 2018;
Revised: In recent years, state-of-the-art self-supervised methods have been exploring different directions for pre-training images and text representations, with Contrastive Learning (CL) providing strong results for vision representation learning (Oord et al., 2018;

Original: Chen et al., 2020b; He et al., 2020; Chen et al., 2020c; Tian et al., 2020), and Language Modeling (LM) becoming the de-facto standard in language pre-training (Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019; Lan et al., 2019).
Revised: Chen et al., 2020b; He et al., 2020; Chen et al., 2020c; Tian et al., 2020), and Language Modeling (LM) becoming the de-facto standard in language pre-training (Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019; Lan et al., 2019).

Original: Both approaches are quite different from each other.
Revised: Both approaches are quite different from each other.

Original: A contrastive objective compares positive/negative examples at a coarse/sample level, focusing on global-content (e.g., for image classiﬁcation) while a token modeling objective predict missing tokens from context at a much ﬁner/sub-sample level to model sequential and short range interactions between tokens (e.g. in text generation tasks).
Revised: A contrastive objective compares positive/negative examples at a coarse/sample level, focusing on global-content (e.g., for image classiﬁcation) while a token modeling objective predict missing tokens from context at a much ﬁner/sub-sample level to model sequential and short range interactions between tokens (e.g. in text generation tasks).

Original: Interestingly, video understanding naturally combines both types of requirements.
Revised: Interestingly, video understanding naturally combines both types of requirements.

Original: 2D processing along the spatial dimensions of the video bears similarity to image processing, while 1D processing along the temporal dimension often involves modeling sequential events and short range coherence.
Revised: 2D processing along the spatial dimensions of the video bears similarity to image processing, while 1D processing along the temporal dimension often involves modeling sequential events and short range coherence.

Original: Hence, in this work, we propose to combine both text and image representation learning approaches for improved video pre-training, taking advantage of recent advances in self-supervised methods of both ﬁelds.
Revised: Hence, in this work, we propose to combine both text and image representation learning approaches for improved video pre-training, taking advantage of recent advances in self-supervised methods of both ﬁelds.

Original: We name our method as VIMPAC: VIdeo pre-training via Masked token Prediction And Contrastive learning.
Revised: We name our method as VIMPAC: VIdeo pre-training via Masked token Prediction And Contrastive learning.

Original: From language research, we adopt a ‘masked language model’ pretraining objective (Devlin et al., 2019) where a model is trained to reconstruct local masked regions in videos.
Revised: From language research, we adopt a ‘masked language model’ pretraining objective (Devlin et al., 2019) where a model is trained to reconstruct local masked regions in videos.

Original: From the computer vision world, we borrow a contrastive learning objective, speciﬁcally the InfoNCE (Oord et al., 2018) objective is applied on positive/negative video samples.
Revised: From the computer vision world, we borrow a contrastive learning objective, speciﬁcally the InfoNCE (Oord et al., 2018) objective is applied on positive/negative video samples.

Original: While the masked language model objective encourages models to learn low-level semantics and sequential interaction, the contrastive loss provide a supervision for the model to learn more global and separable representations that are useful for many downstream tasks (e.g., action classiﬁcation (Soomro et al., 2012b; Carreira & Zisserman, 2017)).
Revised: While the masked language model objective encourages models to learn low-level semantics and sequential interaction, the contrastive loss provide a supervision for the model to learn more global and separable representations that are useful for many downstream tasks (e.g., action classiﬁcation (Soomro et al., 2012b; Carreira & Zisserman, 2017)).

Original: The two objectives provide complementary signals for training: while short range correlations can be predominantly modeled from the training signal of the mask-and-predict task, the contrastive learning objective can provide signals on a more coarse-grained global-context and semantic level.
Revised: The two objectives provide complementary signals for training: while short range correlations can be predominantly modeled from the training signal of the mask-and-predict task, the contrastive learning objective can provide signals on a more coarse-grained global-context and semantic level.

Original: However, unlike language which is composed of discrete tokens from a compact vocabulary, videos are typically represented as RGB pixels in an almost continuous, high dimensional vector space.
Revised: However, unlike language which is composed of discrete tokens from a compact vocabulary, videos are typically represented as RGB pixels in an almost continuous, high dimensional vector space.

Original: Naively masking pixels in videos induces a prohibitive computation cost while also tending to overemphasize local details.
Revised: Naively masking pixels in videos induces a prohibitive computation cost while also tending to overemphasize local details.

Original: To overcome these issues, we ﬁrst tokenize input videos using the latent codes of a pretrained Vector Quantized-Variational Auto-Encoder (VQ-VAE) (van den Oord et al., 2017; Ramesh et al., 2021) to encode them in smaller quantized representations on which a reconstruction model can then be trained with a masked token modeling objective.
Revised: To overcome these issues, we ﬁrst tokenize input videos using the latent codes of a pretrained Vector Quantized-Variational Auto-Encoder (VQ-VAE) (van den Oord et al., 2017; Ramesh et al., 2021) to encode them in smaller quantized representations on which a reconstruction model can then be trained with a masked token modeling objective.

Original: In practice, we also discovered that models trained with a uniform random token masking strategy can fail to learn meaningful and useful visual representations as neighboring pixels may contain similar and correlated content (in particular along the temporal dimension), making the task of predicting a randomly masked token from its visible neighbors trivial.
Revised: In practice, we also discovered that models trained with a uniform random token masking strategy can fail to learn meaningful and useful visual representations as neighboring pixels may contain similar and correlated content (in particular along the temporal dimension), making the task of predicting a randomly masked token from its visible neighbors trivial.

Original: We therefore also introduce a block-masking scheme that simultaneously masking video tokens in a 3D spatio-temporal block.
Revised: We therefore also introduce a block-masking scheme that simultaneously masking video tokens in a 3D spatio-temporal block.

Original: Reconstructing such an extended spatio-temporal cube requires performing long-range predictions, forcing the models to learn a more complex set of relations between the video tokens, resulting in better visual representations.
Revised: Reconstructing such an extended spatio-temporal cube requires performing long-range predictions, forcing the models to learn a more complex set of relations between the video tokens, resulting in better visual representations.

Original: Our contrastive learning approaches also departs from previous work in several aspects.
Revised: Our contrastive learning approaches also departs from previous work in several aspects.

Original: First, since we apply the contrastive objective on token-discretized video samples and in combination with the token modeling loss, we observe strong performance without requiring the usual extensive set of data augmentations (Chen et al.
Revised: First, since we apply the contrastive objective on token-discretized video samples and in combination with the token modeling loss, we observe strong performance without requiring the usual extensive set of data augmentations (Chen et al.

Original: , 2020b;c; Qian et al., 2021; Feichtenhofer et al., 2021).
Revised: , 2020b;c; Qian et al., 2021; Feichtenhofer et al., 2021).

Original: Second, we are able to leverage positive clip pairs that are temporally distant from each other (can be as far as 400 seconds away), while previous work favors using positives within a shorter range (maximumseconds for uncurated videos in Feichtenhofer et al.
Revised: Second, we are able to leverage positive clip pairs that are temporally distant from each other (can be as far as 400 seconds away), while previous work favors using positives within a shorter range (maximumseconds for uncurated videos in Feichtenhofer et al.

Original: (2021) or 10 seconds in Qian et al.
Revised: (2021) or 10 seconds in Qian et al.

Original: We evaluate the performances of our method VIMPAC on several video understanding datasets, including two temporally-heavy tasks, SSV2 and Diving48 on which it achieves state-of-the-art results with regard to both self-supervised and supervised pre-training works, and a set of more spatially-heavy datasets, UCF101, HMDB51, and Kinetics-400, on which it also achieves competitive results.
Revised: We evaluate the performances of our method VIMPAC on several video understanding datasets, including two temporally-heavy tasks, SSV2 and Diving48 on which it achieves state-of-the-art results with regard to both self-supervised and supervised pre-training works, and a set of more spatially-heavy datasets, UCF101, HMDB51, and Kinetics-400, on which it also achieves competitive results.

Original: Overall, taking advantage of VQ-VAE discretized video tokens, we present a method for self-supervised learning of video representations that combines two general streams of research in self-supervision: masked language modeling and contrastive learning.
Revised: Overall, taking advantage of VQ-VAE discretized video tokens, we present a method for self-supervised learning of video representations that combines two general streams of research in self-supervision: masked language modeling and contrastive learning.

Original: Our contribution is 3-folds: ( i )
Revised: Our contribution is 3-folds: ( i )

Original: We apply the mask-then-predict task to video understanding and introduce the use of block masking.
Revised: We apply the mask-then-predict task to video understanding and introduce the use of block masking.

Original: ( ii )
Revised: ( ii )

Original: We propose a contrastive learning method which is able to achieve strong performance without spatial data augmentation.
Revised: We propose a contrastive learning method which is able to achieve strong performance without spatial data augmentation.

Original: ( iii ) We empirically show that this method achieves strong performance on several video classiﬁcation datasets, especially on temporally-heavy datasets, SSV and Diving48, where it sets new state-of-the-art results.
Revised: ( iii ) We empirically show that this method achieves strong performance on several video classiﬁcation datasets, especially on temporally-heavy datasets, SSV and Diving48, where it sets new state-of-the-art results.

Original: We also present comprehensive ablation studies to analyze the various aspects of our proposed approach.
Revised: We also present comprehensive ablation studies to analyze the various aspects of our proposed approach.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Unsupervised representation learning, with the promise of learning from large-scale unlabeled data, has drawn increasing attention in recent years, in both computer vision and natural language processing (NLP) communities.
Revised: Unsupervised representation learning, with the promise of learning from large-scale unlabeled data, has drawn increasing attention in recent years, in both computer vision and natural language processing (NLP) communities.

Original: Most mainstream self-supervised methods can be categorized into three general directions: generative, denoising, and discriminative (Chen et al., 2020b; Grill et al., 2020; Doersch et al., 2015).
Revised: Most mainstream self-supervised methods can be categorized into three general directions: generative, denoising, and discriminative (Chen et al., 2020b; Grill et al., 2020; Doersch et al., 2015).

Original: Generative and denoising methods seek to generate or reconstruct corrupted text/image/video tokens according to their empirical distributions.
Revised: Generative and denoising methods seek to generate or reconstruct corrupted text/image/video tokens according to their empirical distributions.

Original: In generative and auto-regressive methods, next tokens are predicted given a causal context (Chen et al., 2020a; van den Oord et al., 2016) while denoising methods seek to reconstruct corrupted or masked tokens given an extended context (Devlin et al., 2019; Raffel et al., 2019)
Revised: In generative and auto-regressive methods, next tokens are predicted given a causal context (Chen et al., 2020a; van den Oord et al., 2016) while denoising methods seek to reconstruct corrupted or masked tokens given an extended context (Devlin et al., 2019; Raffel et al., 2019).

Original: For text, since the tokens (words or sub-words (Sennrich et al., 2016;
Revised: For text, since the tokens (words or sub-words (Sennrich et al., 2016;

Original: Wu et al., 2016)) are discrete and has relatively high entropy rate, language modeling has became the de-facto approach for pre-training models for most natural language tasks (Ruder et al., 2019).
Revised: Wu et al., 2016)) are discrete and has relatively high entropy rate, language modeling has became the de-facto approach for pre-training models for most natural language tasks (Ruder et al., 2019).

Original: In the case of images, generative approaches often operate on pixel space (Bertalmio et al., 2001;
Revised: In the case of images, generative approaches often operate on pixel space (Bertalmio et al., 2001;

Original: Yu et al., 2018;
Revised: Yu et al., 2018;

Original: Kim et al., 2019; Chen et al., 2020a; van den Oord et al., 2016), which can be extremely expensive for larger input size like videos and has hence limited the widespread adoption of these methods.
Revised: Kim et al., 2019; Chen et al., 2020a; van den Oord et al., 2016), which can be extremely expensive for larger input size like videos and has hence limited the widespread adoption of these methods.

Original: Recently, discretizing images and videos with discrete variational auto-encoders (VQ-VAE), has been explored in compression and generative setups (van den Oord et al., 2017;
Revised: Recently, discretizing images and videos with discrete variational auto-encoders (VQ-VAE), has been explored in compression and generative setups (van den Oord et al., 2017;

Original: Razavi et al., 2019; Walker et al., 2021; Ramesh et al., 2021; Yan et al., 2021), and Sun et al.
Revised: Razavi et al., 2019; Walker et al., 2021; Ramesh et al., 2021; Yan et al., 2021), and Sun et al.

Original: (2019) tackles the video-language problem by frame-level quantization.
Revised: (2019) tackles the video-language problem by frame-level quantization.

Original: Such approaches avoid modeling
Revised: Such approaches avoid modeling

Original: [Figure]
Revised: [Figure]

Original: pixel-level details and have enabled the use of generative models for images and videos (Walker et al., 2021;
Revised: pixel-level details and have enabled the use of generative models for images and videos (Walker et al., 2021;

Original: Ramesh et al., 2021).
Revised: Ramesh et al., 2021).

Original: Differing from these works, our framework investigates the use of such quantized representations in a denoising/reconstruction setup rather than generative, which has been shown in the NLP community to learn better representations (Raffel et al., 2019; Devlin et al., 2019).
Revised: Differing from these works, our framework investigates the use of such quantized representations in a denoising/reconstruction setup rather than generative, which has been shown in the NLP community to learn better representations (Raffel et al., 2019; Devlin et al., 2019).

Original: Moreover, beyond simply applying MLM to the video tokens, we propose a block masking strategy to reduce the strong local correlation in neighboring video tokens.
Revised: Moreover, beyond simply applying MLM to the video tokens, we propose a block masking strategy to reduce the strong local correlation in neighboring video tokens.

Original: This 3D block masking strategy is inspired from recent span-masking schemes (Raffel et al., 2019; Joshi et al., 2020) for language modeling.
Revised: This 3D block masking strategy is inspired from recent span-masking schemes (Raffel et al., 2019; Joshi et al., 2020) for language modeling.

Original: The concurrent work (Bao et al., 2021) explores using the VQ-VAE tokens as labels for masked patches in the image domain.
Revised: The concurrent work (Bao et al., 2021) explores using the VQ-VAE tokens as labels for masked patches in the image domain.

Original: The other direction of research, which our framework combines, is discriminative methods which start from the hypothesis that learning to reconstruct local details is not necessary for learning good visual representations.
Revised: The other direction of research, which our framework combines, is discriminative methods which start from the hypothesis that learning to reconstruct local details is not necessary for learning good visual representations.

Original: In some of these approaches, an objective is constructed around hand-crafted heuristics tasks like spatial arrangement, color, playback speed or frame order predictions (Doersch et al., 2015; Zhang et al., 2016; Gidaris et al., 2018; Fernando et al., 2017; Lee et al., 2017; Wei et al., 2018; Epstein et al., 2020; Benaim et al., 2020; Sun et al., 2021).
Revised: In some of these approaches, an objective is constructed around hand-crafted heuristics tasks like spatial arrangement, color, playback speed or frame order predictions (Doersch et al., 2015; Zhang et al., 2016; Gidaris et al., 2018; Fernando et al., 2017; Lee et al., 2017; Wei et al., 2018; Epstein et al., 2020; Benaim et al., 2020; Sun et al., 2021).

Original: Another line of discriminative approaches is contrastive learning which aims at training a model to be able to recognize different views (e.g., different augmentation of images or different temporal samples of videos) of the same image or video, as a way to learn general representations (Chen et al., 2020b;
Revised: Another line of discriminative approaches is contrastive learning which aims at training a model to be able to recognize different views (e.g., different augmentation of images or different temporal samples of videos) of the same image or video, as a way to learn general representations (Chen et al., 2020b;

Original: He et al., 2020; Chen et al., 2020c; Grill et al., 2020; Caron et al., 2020; Feichtenhofer et al., 2021).
Revised: He et al., 2020; Chen et al., 2020c; Grill et al., 2020; Caron et al., 2020; Feichtenhofer et al., 2021).

Original: This direction of research is reminiscent of sentence-order prediction tasks introduced in NLP (Devlin et al., 2019;
Revised: This direction of research is reminiscent of sentence-order prediction tasks introduced in NLP (Devlin et al., 2019;

Original: Lan et al., 2019) with the goal of predicting whether two text sequences should be juxtaposed or not, an approach challenged in more recent literature (Liu et al., 2019; Lan et al., 2019).
Revised: Lan et al., 2019) with the goal of predicting whether two text sequences should be juxtaposed or not, an approach challenged in more recent literature (Liu et al., 2019; Lan et al., 2019).

Original: In the present work, inspired by visual representation rather than text representation learning literature, we adapt the contrastive learning approach to video by training a model to differentiate pairs of clips from a single video from pairs of clips from disparate videos.
Revised: In the present work, inspired by visual representation rather than text representation learning literature, we adapt the contrastive learning approach to video by training a model to differentiate pairs of clips from a single video from pairs of clips from disparate videos.

Revised: Another thread of research focuses on the way to adapt transformer models to video tasks (Bertasius et al., 2021; Arnab et al., 2021).

Revised: Recent works (Fan et al., 2021; Liu et al., 2021) extend it with hierarchical modeling and efﬁcient attention patterns.

Revised: These methods focus on the modeling and achieve good results with supervised pre-training.

Revised: Besides the difference in focus (modeling vs. pretraining) to our paper, the hierarchical design is also not directly applicable to our mask prediction tasks.

Original:  M ETHODS
Revised:  M ETHODS

Original: In this section, we present our proposed video pre-training method VIMPAC (Fig. 1) as well its detailed components.
Revised: In this section, we present our proposed video pre-training method VIMPAC (Fig. 1) as well its detailed components.

Original: We ﬁrst introduce the mask-then-predict task in Sec. 3.1, and then the contrastive learning task in Sec. 3.2.
Revised: We ﬁrst introduce the mask-then-predict task in Sec. 3.1, and then the contrastive learning task in Sec. 3.2.

Original: Lastly, we discuss how these two tasks are combined in Sec. 3.3. 
Revised: Lastly, we discuss how these two tasks are combined in Sec. 3.3. [Figure]

Original: 3. M ASK - THEN -P REDICT T ASK
Revised: 3. M ASK - THEN -P REDICT T ASK

Original: Suppose that a video clip input comprises T frames t f 1 , f 2 , . .
Revised: Suppose that a video clip input comprises T frames t f 1 , f 2 , . .

Original: , f T u , the mask-then-predict task learns video representations by predicting the masked contents from their spatio-temporal context.
Revised: , f T u , the mask-then-predict task learns video representations by predicting the masked contents from their spatio-temporal context.

Original: Denote the set of mask-token locations as M , we learn to predict the original tokens t x t,i,j u (see[Figure] details below) by optimizing the negative log-likelihood:
Revised: Denote the set of mask-token locations as M , we learn to predict the original tokens t x t,i,j u (see details below) by optimizing the negative log-likelihood:

Original: [Equation]
Revised: [Equation]

Original: where M C is the complement of M and thus indicates the unmasked context.
Revised: where M C is the complement of M and thus indicates the unmasked context.

Original: Video Quantization with VQ-VAE.
Revised: Video Quantization with VQ-VAE.

Original: Since directly applying mask-then-predict over raw pixels and masking/predicting pixels leads to prohibitive computational costs and also tends to make the model overﬁt on detailed low-level visual information, we quantize the input videos with Vector
Revised: Since directly applying mask-then-predict over raw pixels and masking/predicting pixels leads to prohibitive computational costs and also tends to make the model overﬁt on detailed low-level visual information, we quantize the input videos with Vector

Original: Quantized-Variational Auto Encoder (VQ-VAE) (van den Oord et al., 2017; Ramesh et al., 2021).
Revised: Quantized-Variational Auto Encoder (VQ-VAE) (van den Oord et al., 2017; Ramesh et al., 2021).

Original: The VQ-VAE encoder takes an image as input and produces a token map, where the tokens belong to a predeﬁned vocabulary V .
Revised: The VQ-VAE encoder takes an image as input and produces a token map, where the tokens belong to a predeﬁned vocabulary V .

Original: The VQ-VAE decoder then tries to reconstruct the original image from these latent codes.
Revised: The VQ-VAE decoder then tries to reconstruct the original image from these latent codes.

Original: In our method, we use a frozen and pretrained generic VQ-VAE encoder as a compressor that converts an input from an original input space
Revised: In our method, we use a frozen and pretrained generic VQ-VAE encoder as a compressor that converts an input from an original input space

Original: R H
Revised: R H

Original: ˆ W ˆ 3 into a discretized space r V
Revised: ˆ W ˆ 3 into a discretized space r V

Original: s H 

Revised: s H

Original: ˆ W 8 .
Revised: ˆ W 8 .

Original: We independently apply the VQ-VAE encoder to each frame f t inside a clip.
Revised: We independently apply the VQ-VAE encoder to each frame f t inside a clip.

Original: Speciﬁcally, we use the VQ-VAE trained in DALL-E (Ramesh et al., 2021).
Revised: Speciﬁcally, we use the VQ-VAE trained in DALL-E (Ramesh et al., 2021).

Original: We keep the VQ-VAE weights frozen and do not ﬁnetune or adapt this model on our corpus.
Revised: We keep the VQ-VAE weights frozen and do not ﬁnetune or adapt this model on our corpus.

Original: Block Masking For sampling tokens to mask, the original BERT methods proposes the i.i.d.
Revised: Block Masking For sampling tokens to mask, the original BERT methods proposes the i.i.d.

Original: (independent and identically distributed) random mask M iid that constitutes of masked tokens:
Revised: (independent and identically distributed) random mask M iid that constitutes of masked tokens:

Original: [Equation]
Revised: [Equation]

Original: ) where U t, [Equation] ,j r 0 , 1 s is the uniform distribution from 0 to 1 .
Revised: where U t,i,j r 0 , 1 s is the uniform distribution from 0 to 1 .

Original: Intuitively, ξ is the expectation of maskedtoken ratio and hence controls the difﬁculty of our mask-then-predict task.
Revised: Intuitively, ξ is the expectation of maskedtoken ratio and hence controls the difﬁculty of our mask-then-predict task.

Original: In our early experiments, we found it easy to infer a masked token from its direct spatio-temporal neighbours (e.g., neighboring frames in a video tend to look similar thus contain similar tokens).
Revised: In our early experiments, we found it easy to infer a masked token from its direct spatio-temporal neighbours (e.g., neighboring frames in a video tend to look similar thus contain similar tokens).

Original: To overcome this issue, we propose to use block masking (see Fig.
Revised: To overcome this issue, we propose to use block masking (see Fig.

Original: 2 (a)), which masks continuous tokens inside spatio-temporal blocks.
Revised: 2 (a)), which masks continuous tokens inside spatio-temporal blocks.

Original: For each mask block B , we randomly sample lower ( B ˚ , 0 ) and upper boundaries ( B ˚ , 1 ) for each of the temporal ( T ), height ( H ), and width ( W ) dimensions.
Revised: For each mask block B , we randomly sample lower ( B ˚ , 0 ) and upper boundaries ( B ˚ , 1 ) for each of the temporal ( T ), height ( H ), and width ( W ) dimensions.

Original: The direct product of the intervals delimited by these boundaries constructs the block mask.
Revised: The direct product of the intervals delimited by these boundaries constructs the block mask.

Original: The ﬁnal mask M block is the union of them:
Revised: The ﬁnal mask M block is the union of them:

Original: [Equation]
Revised: [Equation]

Original: 3.  C ONTRASTIVE L EARNING
Revised:  C ONTRASTIVE L EARNING

Original: Contrastive learning aims to distinguishing positive pairs from negative pairs (see Fig.
Revised: Contrastive learning aims to distinguishing positive pairs from negative pairs (see Fig.

Original: 2 (b)).
Revised: 2 (b)).

Original: For each video video i , we uniformly and independently sample two clips
Revised: For each video video i , we uniformly and independently sample two clips

Original: c i , c 1 i as a positive pair, while the clips in a batch belonging to other videos are used to construct negative pairs.
Revised: c i , c 1 i as a positive pair, while the clips in a batch belonging to other videos are used to construct negative pairs.

Original: A model (described in Sec. 3.4) processes clips
Revised: A model (described in Sec. 3.4) processes clips

Original: c i , c 1 i to build respective vector representations f i , f 

Revised: c i , c 1 i to build respective vector representations f i , f

Original: i and an
Revised: i and an

Original: InfoNCE (Oord et al., 2018) loss is used to distinguishes the positive feature pair ( f i , f 1 i ) from the [Equation]
Revised: InfoNCE (Oord et al., 2018) loss is used to distinguishes the positive feature pair ( f i , f 1 i ) from the negative pairs Ť

Original: Ť [Equation]
Revised: ttp f i , f k q , p f i , f 1 k qu | k ‰ i u for each clip c

Original: c i 
Revised: i :

Original: [Equation]
Revised: [Equation]

Original: which we combine with the symmetric loss L 1 InfoNCE p i q for paired clip sample c 1 i .
Revised: which we combine with the symmetric loss L 1 InfoNCE p i q for paired clip sample c 1 i .

Original: The ﬁnal loss for a mini batch L cl is the average loss for all n clips in the mini-batch:
Revised: The ﬁnal loss for a mini batch L cl is the average loss for all n clips in the mini-batch:

Original: [Equation]
Revised: [Equation]

Original: 3. P RE -T RAINING O BJECTIVE
Revised: 3. P RE -T RAINING O BJECTIVE

Original: We combine the two pre-training methods discussed above to deﬁne the overall objective as:
Revised: We combine the two pre-training methods discussed above to deﬁne the overall objective as:

Original: [Equation]
Revised: [Equation]

Original: where α is a hyperparameter controlling the weight of the contrastive loss and multiplying the temperature γ will smooth training (Grill et al., 2020; Chen et al., 2021).
Revised: where α is a hyperparameter controlling the weight of the contrastive loss and multiplying the temperature γ will smooth training (Grill et al., 2020; Chen et al., 2021).

Original: The inputs for both tasks are shared in mini-batches with the contrastive learning loss using the same block-masked inputs necessary for the mask-then-predict task.
Revised: The inputs for both tasks are shared in mini-batches with the contrastive learning loss using the same block-masked inputs necessary for the mask-then-predict task.

Original: We highlight that the masked tokens are the only noise introduced in the contrastive learning, and that no other data augmentation is applied to raw pixels, in contrast to previous vision contrastive learning methods in which data-augmentation was paramount to the ﬁnal performances of the model.
Revised: We highlight that the masked tokens are the only noise introduced in the contrastive learning, and that no other data augmentation is applied to raw pixels, in contrast to previous vision contrastive learning methods in which data-augmentation was paramount to the ﬁnal performances of the model.

Original: This phenomenon is empirically studied in Sec.
Revised: This phenomenon is empirically studied in Sec.

Original: 3. M ODELING
Revised: 3. M ODELING

Original: The model architecture follows the standard transformer architecture in its post-layer-norm variant (Vaswani et al., 2017; Devlin et al., 2019) with two more recent additions: divided temporalspatial attention (Bertasius et al., 2021), and sparse spatial attention (Child et al., 2019).
Revised: The model architecture follows the standard transformer architecture in its post-layer-norm variant (Vaswani et al., 2017; Devlin et al., 2019) with two more recent additions: divided temporalspatial attention (Bertasius et al., 2021), and sparse spatial attention (Child et al., 2019).

Original: We detail both additions in the Appendix.
Revised: We detail both additions in the Appendix.

Original: The model embedding layer maps the discrete tokens t x t,i,j u of a quantized input video (see Sec. 3.1) into dense vectors and sum them with positional embeddings.
Revised: The model embedding layer maps the discrete tokens t x t,i,j u of a quantized input video (see Sec. 3.1) into dense vectors and sum them with positional embeddings.

Original: The backbone transformer model then outputs corresponding features t h t,i,j u .
Revised: The backbone transformer model then outputs corresponding features t h t,i,j u .

Original: We append an additional [CLS] token to each input sequence following Devlin et al.
Revised: We append an additional [CLS] token to each input sequence following Devlin et al.

Original: and use its output feature h cls as a representation for the whole video.
Revised: and use its output feature h cls as a representation for the whole video.

Original: For pre-training, we use two heads: a 2-layer MLP after each token outputs t h t,i,j u for the mask-then-predict task following BERT (Devlin et al., 2019), and a 3-layer MLP after the CLS output h cls for the contrastive learning task following SimCLR (Chen et al., 2020b).
Revised: For pre-training, we use two heads: a 2-layer MLP after each token outputs t h t,i,j u for the mask-then-predict task following BERT (Devlin et al., 2019), and a 3-layer MLP after the CLS output h cls for the contrastive learning task following SimCLR (Chen et al., 2020b).

Original: For ﬁne-tuning on classiﬁcation tasks, we remove the pre-training heads and add a fully-connected layer to the [CLS] output h cls .
Revised: For ﬁne-tuning on classiﬁcation tasks, we remove the pre-training heads and add a fully-connected layer to the [CLS] output h cls .

Original: More implementation details are in Appendix.
Revised: More implementation details are in Appendix.

Original:  E XPERIMENTS AND R ESULTS 4. D ATASETS
Revised:  E XPERIMENTS AND R ESULTS 4. D ATASETS

Original: For pre-training , we use the HowTo100M dataset (Miech et al., 2019).
Revised: For pre-training , we use the HowTo100M dataset (Miech et al., 2019) 1 .

Original: This dataset is constructed by searching YouTube videos with a list of text queries, it is signiﬁcantly larger and more diverse than human-annotated datasets such as Kinetics 400 (Carreira et al., 2019).
Revised: This dataset is constructed by searching YouTuBe videos with a list of text queries, it is signiﬁcantly larger and more diverse than human-annotated datasets such as Kinetics 400 (Carreira et al., 2019).

Original: HowTo100M has 1.2M uncurated videos, with an average duration of 6.5 minutes.
Revised: HowTo100M has 1.2M uncurated videos, with an average duration of 6.5 minutes.

Original: We only use videos and do not use other signals such as ASR captions in this dataset.
Revised: We only use videos and do not use other signals such as ASR captions in this dataset.

Original: For downstream evaluation, we experiment with several action classiﬁcation datasets: UCF101 (Soomro et al., 2012a), HMDB51 (Kuehne et al., 2011a), Kinetics-400 (Carreira & Zisserman, 2017), SSV2 (Goyal et al., 2017), and Diving48 (Li et al., 2018).
Revised: For downstream evaluation, we experiment with several action classiﬁcation datasets: UCF101 (Soomro et al., 2012a), HMDB51 (Kuehne et al., 2011a), Kinetics-400 (Carreira & Zisserman, 2017), SSV2 (Goyal et al., 2017), and Diving48 (Li et al., 2018).

Original: It is important to note that in many cases, actions in UCF101, HMDB51, and Kinetics400 can be recognized from a single frame of the video, thus these datasets are ‘ spatially-heavy ’. 
Revised: It is important to note that in many cases, actions in UCF101, HMDB51, and Kinetics400 can be recognized from a single frame of the video, thus these datasets are ‘ spatially-heavy ’. As

Original: As a consequence, image-level methods (Bertasius et al., 2021; Radford et al., 2021) show competitive results without modeling the temporal interactions inside the videos.
Revised: [Table] a consequence, image-level methods (Bertasius et al., 2021; Radford et al., 2021) show competitive results without modeling the temporal interactions inside the videos.

Original: To test the video model’s ability beyond recognizing static images, we lay our focus on ‘ temporally-heavy ’ datasets (SSV2 and
Revised: To test the video model’s ability beyond recognizing static images, we lay our focus on ‘ temporally-heavy ’ datasets (SSV2 and

Original: Diving48), in which action recognition from a single frame is more difﬁcult.
Revised: Diving48), in which action recognition from a single frame is more difﬁcult.

Original: For example, it is almost impossible to distinguish two SSV2 classes moving something up and moving something down without reasoning across frames, and the same for different diving classes in Diving48.
Revised: For example, it is almost impossible to distinguish two SSV2 classes moving something up and moving something down without reasoning across frames, and the same for different diving classes in Diving48.

Original: Additional dataset details (e.g., statistics) are presented in Appendix.[Table] [Table]
Revised: Additional dataset details (e.g., statistics) are presented in Appendix.

Original: 4. E XPERIMENTAL S ETUP
Revised: 4. E XPERIMENTAL S ETUP

Original: Our model shapes follow BERT LARGE with 24 layers and hidden size 1024, but with halved attention head size and MLP intermediate size as in Child et al.
Revised: Our model shapes follow BERT LARGE with 24 layers and hidden size 1024, but with halved attention head size and MLP intermediate size as in Child et al.

Original: For pre-training , we train the model for 100 epochs on HowTo100M with frames sampled at 2 FPS.
Revised: For pre-training , we train the model for 100 epochs on HowTo100M with frames sampled at 2 FPS.

Original: We sample two clips from each video as model inputs as described in Sec. 3.2.
Revised: We sample two clips from each video as model inputs as described in Sec. 3.2.

Original: To reduce computation cost, we train the ﬁrstepochs with a smaller input resolution (#frames T = 5 and frame size S = 128 ) and increase the spatial resolution ( T = 5 , S = 256 ) for the last 10 epochs following Devlin et al.
Revised: To reduce computation cost, we train the ﬁrstepochs with a smaller input resolution (#frames T = 5 and frame size S = 128 ) and increase the spatial resolution ( T = 5 , S = 256 ) for the last 10 epochs following Devlin et al.

Original: Positional embeddings are interpolated as in Dosovitskiy et al.
Revised: Positional embeddings are interpolated as in Dosovitskiy et al.

Original: (2021) when input resolution changes.
Revised: (2021) when input resolution changes.

Original: Importantly, our pre-training scheme does not involve spatial augmentations: all frames are resized and centered cropped without random ﬂipping, color distortion, etc.
Revised: Importantly, our pre-training scheme does not involve spatial augmentations: all frames are resized and centered cropped without random ﬂipping, color distortion, etc.

Original: We use a batch size of 1024 in pre-training.
Revised: We use a batch size of 1024 in pre-training.

Original: The number of negative clips used for contrastive learning is 255 for the ﬁrst 90 epochs and 127 for the last 10 epochs.
Revised: The number of negative clips used for contrastive learning is 255 for the ﬁrst 90 epochs and 127 for the last 10 epochs.

Original: The number of negative pairs used in our ablation analyses is kept constant at 127.
Revised: The number of negative pairs used in our ablation analyses is kept constant at 127.

Original: More details are in Appendix.
Revised: More details are in Appendix.

Original: For ﬁne-tuning , we use more input frames ( T = 10 and S = 256 ), and batch size 128.
Revised: For ﬁne-tuning , we use more input frames ( T = 10 and S = 256 ), and batch size 128.

Original: We sample frames at 2 FPS for datasets with longer videos (i.e., UCF101 and
Revised: We sample frames at 2 FPS for datasets with longer videos (i.e., UCF101 and

Original: Kinetics-400), and sample 4 FPS for datasets with shorter videos (i.e., HMDB51, SSV2, Diving48).
Revised: Kinetics-400), and sample 4 FPS for datasets with shorter videos (i.e., HMDB51, SSV2, Diving48).

Original: During inference, we follow Feichtenhofer et al.
Revised: During inference, we follow Feichtenhofer et al.

Original: (2019; 2021) to use 3 spatial crops and 10 temporal crops (in total 30 crops), and average their prediction scores as the ﬁnal score.
Revised: (2019; 2021) to use 3 spatial crops and 10 temporal crops (in total 30 crops), and average their prediction scores as the ﬁnal score.

Original: 1 All models are trained with AdamW (Loshchilov & Hutter, 2018) optimizer with linear warm-up and linear learning rate decay.
Revised: 2 All models are trained with AdamW (Loshchilov & Hutter, 2018) optimizer with linear warm-up and linear learning rate decay.

Original: We observe similar pre-training instability as reported in Chen et al.
Revised: We observe similar pre-training instability as reported in Chen et al.

Original: (2020a; 2021) and follow their practice to sequentially choose learning rate at 1e-3, 5e-4, 3e-4, ..., until convergence. 
Revised: (2020a; 2021) and follow their practice to sequentially choose learning rate at 1e-3, 5e-4, 3e-4, ..., until convergence. [Table]

Original: 4. R ESULTS
Revised: 4. R ESULTS

Original: Table 1 shows our primary results.
Revised: Table 1 shows our primary results.

Original: We mainly compare with self-supervised pre-training methods on uncurated videos.
Revised: We mainly compare with self-supervised pre-training methods on uncurated videos.

Original: In addition, we list methods with (weakly) supervised pre-training as references, though they can not be fairly compared with our method as they use large-scale (weakly) labeled data.
Revised: In addition, we list methods with (weakly) supervised pre-training as references, though they can not be fairly compared with our method as they use large-scale (weakly) labeled data.

Original: As discussed in Sec. 4.1, recognizing actions in SSV2 and Diving48 require a strong temporal reasoning ability, while in the other datasets, spatial understanding is dominant.To bet-
Revised: As discussed in Sec. 4.1, recognizing actions in SSV2 and Diving48 require a strong temporal reasoning ability, while in the other datasets, spatial understanding is dominant.

Original: [Table] ter illustrate the differences between temporally-heavy and spatially-heavy datasets, we compare two variants of TimeSformer (Bertasius et al., 2021), one with attention on space-time, and one on space only.
Revised: To better illustrate the differences between temporally-heavy and spatially-heavy datasets, we compare two variants of TimeSformer (Bertasius et al., 2021), one with attention on space-time, and one on space only.

Original: Note the gaps between these two variants are signiﬁcantly larger for temporallyheavy datasets (SSV2) than spatially-heavy datasets (Kinetics-400), demonstrating the importance of temporal modeling for temporally-heavy datasets.
Revised: Note the gaps between these two variants are signiﬁcantly larger for temporallyheavy datasets (SSV2) than spatially-heavy datasets (Kinetics-400), demonstrating the importance of temporal modeling for temporally-heavy datasets.

Original: On the two temporally-heavy datasets SSV and Diving48, when comparing to previous best models among all self-supervised and supervised pre-training methods, our model VIMPAC sets new state of the art, where we achieve 2.7% and 4.5% absolute improvement, respectively.
Revised: On the two temporally-heavy datasets SSV and Diving48, when comparing to previous best models among all self-supervised and supervised pre-training methods, our model VIMPAC sets new state of the art, where we achieve 2.7% and 4.5% absolute improvement, respectively.

Original: This is especially surprising considering the two previous
Revised: This is especially surprising considering the two previous

Original: SotA models ViViT (Arnab et al., 2021) and TimeSformer (Bertasius et al., 2021)
Revised: SotA models ViViT (Arnab et al., 2021) and TimeSformer (Bertasius et al., 2021)

Original: both use largescale supervised pre-training, and ViViT also uses various regularization techniques (e.g., stochastic depth (Huang et al., 2016), random augment (Cubuk et al., 2020), and mixup (Zhang et al., 2018)).
Revised: both use largescale supervised pre-training, and ViViT also uses various regularization techniques (e.g., stochastic depth (Huang et al., 2016), random augment (Cubuk et al., 2020), and mixup (Zhang et al., 2018)).

Original: On spatially-heavy datasets,UCF101, HMDB51 and Kinetics-400, VIMPAC achieves competitive results to self-supervised pre-training methods, while being lower when compared to supervised methods.
Revised: On spatially-heavy datasets,UCF101, HMDB51 and Kinetics-400, VIMPAC achieves competitive results to self-supervised pre-training methods, while being lower when compared to supervised methods.

Original: These relatively low results of our VIMPAC (e.g., UCF101) are possibly due to the spatial information loss during the VQ-VQA quantization process.
Revised: These relatively low results of our VIMPAC (e.g., UCF101) are possibly due to the spatial information loss during the VQ-VQA quantization process.

Original: Concurrent work BEiT
Revised: Concurrent work BEiT

Original: (Bao et al., 2021) addresses this issue by using image patches instead of VQ-VAE tokens as inputs, where they show strong performance on image classiﬁcation tasks.
Revised: (Bao et al., 2021) addresses this issue by using image patches instead of VQ-VAE tokens as inputs, where they show strong performance on image classiﬁcation tasks.

Original: We encourage future work to study using image patches as inputs for better spatial modeling under our framework.
Revised: We encourage future work to study using image patches as inputs for better spatial modeling under our framework.

Original: Previous self-supervised pre-training methods such as BYOL (Grill et al., 2020; Feichtenhofer et al., 2021) and MoCo (He et al., 2020; Feichtenhofer et al., 2021) are good at global understanding, but the pre-training schema does not consider the internal interactions inside videos (especially for the temporal dimension).
Revised: Previous self-supervised pre-training methods such as BYOL (Grill et al., 2020; Feichtenhofer et al., 2021) and MoCo (He et al., 2020; Feichtenhofer et al., 2021) are good at global understanding, but the pre-training schema does not consider the internal interactions inside videos (especially for the temporal dimension).

Original: As a result, it could reach or even outperform the supervised alternatives on UCF101.
Revised: As a result, it could reach or even outperform the supervised alternatives on UCF101.

Original: However, it shows lower results on SSV2 compared to the transformers (Bertasius et al., 2021; Arnab et al., 2021)
Revised: However, it shows lower results on SSV2 compared to the transformers (Bertasius et al., 2021; Arnab et al., 2021)

Original: (although with different backbones) that warm up from image-pre-trained models and learn the temporal interactions directly from the downstream tasks.
Revised: (although with different backbones) that warm up from image-pre-trained models and learn the temporal interactions directly from the downstream tasks.

Original: We also show the cross-modal selfsupervised learning methods, MIL-NCE (Miech et al., 2020) and MMV (Alayrac et al., 2020) that are trained on uncurated videos but leverage other modalities (e.g., text) to help video learning.
Revised: We also show the cross-modal selfsupervised learning methods, MIL-NCE (Miech et al., 2020) and MMV (Alayrac et al., 2020) that are trained on uncurated videos but leverage other modalities (e.g., text) to help video learning.

Original:  A NALYSIS
Revised:  A NALYSIS

Original: We also analyze the model’s scalability and the effectiveness of our pre-training methods.
Revised: We also analyze the model’s scalability and the effectiveness of our pre-training methods.

Original: To save computation, for all analyses, we use a smaller model (6-layer transformer with hidden dimension 512) and smaller input resolution (5 input frames with spatial size 128, i.e., T = 5 , S = 128 ) throughout this section, unless otherwise stated.
Revised: To save computation, for all analyses, we use a smaller model (6-layer transformer with hidden dimension 512) and smaller input resolution (5 input frames with spatial size 128, i.e., T = 5 , S = 128 ) throughout this section, unless otherwise stated.

Original: We also perform pre-training with fewer epochs (i.e., 10).
Revised: We also perform pre-training with fewer epochs (i.e., 10).

Original: For downstream tasks, we use the same input resolution as pre-training (i.e., T = 5 , S = 128 ), and we use 2 temporal crops for inference.
Revised: For downstream tasks, we use the same input resolution as pre-training (i.e., T = 5 , S = 128 ), and we use 2 temporal crops for inference.

Original: All results are reported on the train-val split 1 if applicable. 
Revised: All results are reported on the train-val split 1 if applicable. [Table] [Table] [Table]

Original: 5. S CALABILITY
Revised: S CALABILITY

Revised: In Table

Original: In Table 2, we illustrate the scalability of our method with different model sizes (i.e., number of layers and hidden dimensions).
Revised: [Table] we illustrate the scalability of our method with different model sizes (i [Table] e., number of layers and hidden dimensions).

Original: Larger models have more parameters (‘Params’) and higher computational cost (measured by the normalized pre-training ‘Speed’).
Revised: Larger models have more parameters (‘Params’) and higher computational cost (measured by the normalized pre-training ‘Speed’).

Original: To evaluate the pre-training tasks performance, we provide both pre-training metrics (mask-then-predict accuracy denoted by ‘Mask- [Table] [Table] Accu.’, and contrastive learning loss denoted by ‘CL-Loss’) and UCF101 downstream ﬁne-tuning results.
Revised: To evaluate the pre-training tasks performance, we provide both pre-training metrics (mask-then-predict accuracy denoted by ‘MaskAccu.’, and contrastive learning loss denoted by ‘CL-Loss’) and UCF101 downstream ﬁne-tuning results.

Original: As the size of the model grows, the ﬁne-tuning results show consistent improvement with the pre-training metrics.
Revised: As the size of the model grows, the ﬁne-tuning results show consistent improvement with the pre-training metrics.

Original: Note that for the last row in Table 2, we halve the attention head and MLP intermediate dimensions.
Revised: Note that for the last row in Table 2, we halve the attention head and MLP intermediate dimensions.

Original: We also illustrate the scalability over input resolution in Appendix F.2.
Revised: We also illustrate the scalability over input resolution in Appendix F.2.

Original: 5. P RE -T RAINING M ETHODS 5.2. T HE I MPACT OF P RE -T RAINING
Revised: 5. P RE -T RAINING M ETHODS 5.2. T HE I MPACT OF P RE -T RAINING

Original: [Table]
Revised: [Table]

Original: We ﬁrst compare different pre-training tasks and the non-pre-training results.
Revised: We ﬁrst compare different pre-training tasks and the non-pre-training results.

Original: As shown in Table 3, maskthen-predict is good at temporally-heavy datasets (SSV2, Diving48) while contrastive learning improves the spatially-heavy datasets.
Revised: As shown in Table 3, maskthen-predict is good at temporally-heavy datasets (SSV2, Diving48) while contrastive learning improves the spatially-heavy datasets.

Original: 3 We also compare with the non-pre-training results (the ﬁrst row of Table 3) and observe that both tasks signiﬁcantly improve the results.
Revised: 4 We also compare with the non-pre-training results (the ﬁrst row of Table 3) and observe that both tasks signiﬁcantly improve the results.

Original: We notice that these non-pretraining results are lower than previous from-scratch models, which might be caused by the difﬁculty in training video transformers (Bertasius et al., 2021; Arnab et al., 2021) and the information loss in our input quantization process (Ramesh et al., 2021).
Revised: We notice that these non-pretraining results are lower than previous from-scratch models, which might be caused by the difﬁculty in training video transformers (Bertasius et al., 2021; Arnab et al., 2021) and the information loss in our input quantization process (Ramesh et al., 2021).

Original: 5.2. M ASK - THEN -P REDICT
Revised: 5.2. M ASK - THEN -P REDICT

Original: In this analysis, we exclude the contrastive learning loss (i.e., loss weight α = 0 ) to avoid side effects.
Revised: In this analysis, we exclude the contrastive learning loss (i.e., loss weight α = 0 ) to avoid side effects.

Original: Block Masking versus I.I.D. Masking.
Revised: Block Masking versus I.I.D. Masking.

Original: We ﬁrst compare our proposed block masking strategy and the uniform i.i.d.
Revised: We ﬁrst compare our proposed block masking strategy and the uniform i.i.d.

Original: masking strategy (discussed in Sec. 3.1 and illustrated in Fig. 2).
Revised: masking strategy (discussed in Sec. 3.1 and illustrated in Fig. 2).

Original: As shown in
Revised: As shown in

Original: Table 4, although the i.i.d.
Revised: Table 4, although the i.i.d.

Original: masking achieves higher pre-training mask-token-prediction accuracy (‘Mask-Accu.’), it shows lower downstream results (‘UCF101’) than block masking.
Revised: masking achieves higher pre-training mask-token-prediction accuracy (‘Mask-Accu.’), it shows lower downstream results (‘UCF101’) than block masking.

Original: The higher mask accuracy is possibly due to the easier i.i.d.
Revised: The higher mask accuracy is possibly due to the easier i.i.d.

Original: mask-then-predict task.
Revised: mask-then-predict task.

Original: We show in Appendix that simply copy-paste already yield reasonable reconstruction results.
Revised: We show in Appendix that simply copy-paste already yield reasonable reconstruction results.

Original: The existence of such a trivial solution potentially prevents the model from learning useful video representations for downstream tasks.
Revised: The existence of such a trivial solution potentially prevents the model from learning useful video representations for downstream tasks.

Original: Meanwhile, we also ﬁnd that the model with larger input frame size 256 beneﬁts more from the block masking strategy, because the adjacent tokens are closer in the original 2D image for these larger frames.
Revised: Meanwhile, we also ﬁnd that the model with larger input frame size 256 beneﬁts more from the block masking strategy, because the adjacent tokens are closer in the original 2D image for these larger frames.

Original: Hence, the spatial locality is ampliﬁed.
Revised: Hence, the spatial locality is ampliﬁed.

Original: Masking Ratio.
Revised: Masking Ratio.

Original: In Table 5, we study the impact of masking ratio, by varying the number of masked blocks for block masking.
Revised: In Table 5, we study the impact of masking ratio, by varying the number of masked blocks for block masking.

Original: Empirically, the result differences among different masking ratios are marginal and the original BERT’s 15 % masking ratio (with roughly 5 masking blocks) works slightly  better.
Revised: Empirically, the result differences among different masking ratios are marginal and the original BERT’s 15 % masking ratio (with roughly 5 masking blocks) works slightly [Table] [Table] better.

Original: Thus we always select the number of mask blocks whose induced masking ratio is closest to 15 %.
Revised: Thus we always select the number of mask blocks whose induced masking ratio is closest to 15 %.

Original: The detailed choices of number of masking blocks are listed in Appendix.
Revised: The detailed choices of number of masking blocks are listed in Appendix.

Original: 5.2. C ONTRASTIVE L EARNING
Revised: 5.2. C ONTRASTIVE L EARNING

Original: Positive Sampling Distance.
Revised: Positive Sampling Distance.

Original: As illustrated in Sec. 3.2 and Fig. 2.(b), we uniformly sample positive clip pairs across the whole video without any distance restriction.
Revised: As illustrated in Sec. 3.2 and Fig. 2.(b), we uniformly sample positive clip pairs across the whole video without any distance restriction.

Original: To analyze the effect of such[Table] [Table] a sampling strategy, We perform a set of experiments by varying the maximum sampling distance d max (in seconds) between two positive clips.
Revised: To analyze the effect of such a sampling strategy, We perform a set of experiments by varying the maximum sampling distance d max (in seconds) between two positive clips.

Original: The results are shown in Table 6.
Revised: The results are shown in Table 6.

Original: d max = 8 denotes our default setup without any distance restriction.
Revised: d max = 8 denotes our default setup without any distance restriction.

Original: d max = 

Revised: d max =

Original: samples two same clips, and d max =samples two positive clips with a maximum distance of 10 seconds.
Revised: samples two same clips, and d max =samples two positive clips with a maximum distance of 10 seconds.

Original: Although previous contrastive learning methods (Qian et al., 2021; Feichtenhofer et al., 2021) favor the sampling of temporal positives within a shorter range (e.g., maximum 36 seconds for uncurated videos in Feichtenhofer et al.
Revised: Although previous contrastive learning methods (Qian et al., 2021; Feichtenhofer et al., 2021) favor the sampling of temporal positives within a shorter range (e.g., maximum 36 seconds for uncurated videos in Feichtenhofer et al.

Original: , we observe a performance gain when using larger distance.
Revised: , we observe a performance gain when using larger distance.

Original: We also want to emphasize that the results with d max =10 and d max =0 are not better than the model pre-trained with only maskthen-predict (UCF101 accuracy 68.3), which suggests that short-range contrastive learning does not improve upon our mask-then-predict task.
Revised: We also want to emphasize that the results with d max =10 and d max =0 are not better than the model pre-trained with only maskthen-predict (UCF101 accuracy 68.3), which suggests that short-range contrastive learning does not improve upon our mask-then-predict task.

Original: This is potentially because our mask-then-predict already gives the model the ability to model local interactions, thus contrastive learning objective can only be useful when it focuses on longer-range interactions.
Revised: This is potentially because our mask-then-predict already gives the model the ability to model local interactions, thus contrastive learning objective can only be useful when it focuses on longer-range interactions.

Original: Number of Negative Samples.
Revised: Number of Negative Samples.

Original: Previous constrastive learning methods (Chen et al., 2020b; 2021;
Revised: Previous constrastive learning methods (Chen et al., 2020b; 2021;

Original: Feichtenhofer et al., 2021) beneﬁt from more negative samples.
Revised: Feichtenhofer et al., 2021) beneﬁt from more negative samples.

Original: In this section, we show that the number of negative samples has less impact on our method when mask-then-predict task is added.
Revised: In this section, we show that the number of negative samples has less impact on our method when mask-then-predict task is added.

Original: As shown in Table 7, we use different contrastive learning sample sizes (i.e., n in Sec. 3.2) and always accumulate the gradients to 1024 samples before updating the parameters.
Revised: As shown in Table 7, we use different contrastive learning sample sizes (i.e., n in Sec. 3.2) and always accumulate the gradients to 1024 samples before updating the parameters.

Original: Although increasing sample size makes the contrastive learning task harder (reﬂected by ‘CL-Loss’), it does not show clear evidence of improving UCF101 downstream performance.
Revised: Although increasing sample size makes the contrastive learning task harder (reﬂected by ‘CL-Loss’), it does not show clear evidence of improving UCF101 downstream performance.

Original: [Table]
Revised: [Table]

Original: Input Masking as Augmentation.
Revised: Input Masking as Augmentation.

Original: Most selfsupervised visual representation learning methods (Chen et al., 2020c;b; Grill et al., 2020; Feichtenhofer et al., 2021; Qian et al., 2021) based on contrastive learning suffer from a large drop when removing strong spatial augmentations.
Revised: Most selfsupervised visual representation learning methods (Chen et al., 2020c;b; Grill et al., 2020; Feichtenhofer et al., 2021; Qian et al., 2021) based on contrastive learning suffer from a large drop when removing strong spatial augmentations.

Original: In contrast, our pre-training does not use any spatial augmentations on raw frames.
Revised: In contrast, our pre-training does not use any spatial augmentations on raw frames.

Original: However, as we tie the input between mask-then-predict and contrastive learning to reduce computation cost, the random masking noise is naturally introduced.
Revised: However, as we tie the input between mask-then-predict and contrastive learning to reduce computation cost, the random masking noise is naturally introduced.

Original: We here investigate its impact in Table 8.
Revised: We here investigate its impact in Table 8.

Original: When pre-trained jointly with mask-then-predict, adding mask noise improves
Revised: When pre-trained jointly with mask-then-predict, adding mask noise improves

Original: UCF101 accuracy by +2.0; however, when pre-trained without it, adding mask noise hurts the performance (-1.6).
Revised: UCF101 accuracy by +2.0; however, when pre-trained without it, adding mask noise hurts the performance (-1.6).

Original: We hypothesize that this is due to the large input mismatches between pre-training and ﬁne-tuning when mask-then-predict objective is not applied.
Revised: We hypothesize that this is due to the large input mismatches between pre-training and ﬁne-tuning when mask-then-predict objective is not applied.

Original: Noisy masking creates ‘holes’ to the input token maps during pre-training, while for ﬁne-tuning the input token maps are intact.
Revised: Noisy masking creates ‘holes’ to the input token maps during pre-training, while for ﬁne-tuning the input token maps are intact.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We present the video pre-training framework VIMPAC that introduces mask-then-predict task to video self-supervised learning.
Revised: We present the video pre-training framework VIMPAC that introduces mask-then-predict task to video self-supervised learning.

Original: mask-then-predict task helps model spatio-temporal interactions that is important for video understanding.
Revised: mask-then-predict task helps model spatio-temporal interactions that is important for video understanding.

Original: We use the VQ-VAE quantizer and propose the block masking method that is essential to overcome the strong locality in video.
Revised: We use the VQ-VAE quantizer and propose the block mask- ing method that is essential to overcome the strong locality in video.

Original: The contrastive learning task is also added to learn separable global features.
Revised: The contrastive learning task is also added to learn separable global features.

Original: Different from previous methods, our contrastive learning does not use data augmentation over raw frames and is less sensitive to the temporal sampling distribution for positive pairs.
Revised: Different from previous methods, our contrastive learning does not use data augmentation over raw frames and is less sensitive to the temporal sampling distribution for positive pairs.

Original: We show that our frameworks could achieve state-of-the-art performance on two temporally-heavy dataset (SSV2 and Diving48) and reach competitive results on other datasets.
Revised: We show that our frameworks could achieve state-of-the-art performance on two temporally-heavy dataset (SSV2 and Diving48) and reach competitive results on other datasets.

Original: Detailed analysis is provided regarding the model scalability and task design.
Revised: Detailed analysis is provided regarding the model scalability and task design.

Original: Reproducibility Statement.
Revised: Reproducibility statement.

Original: The code to reproduce the results in this paper is submitted in the supplementary material and will be made public.
Revised: The code to reproduce the results in this paper is submitted in the supplementary material and will be made public.

Original: The code is runnable and contains the scripts with faithful hyperparameters for our experiments.
Revised: The code is runnable and contains the scripts with faithful hyperparameters for our experiments.

Original: We include comprehensive instructions about the feature extraction, pre-training, and ﬁne-tuning.
Revised: We include comprehensive instructions about the feature extraction, pre-training, and ﬁne-tuning.

Original: A detailed “readme” ﬁle is attached in the codebase, and we will publicly release the pre-trained weights as well.
Revised: A detailed “readme” ﬁle is attached in the codebase, and we will publicly release the pre-trained weights as well.

Original: Ethics Statement.
Revised: Ethical considerations and limitations.

Original: The main purpose of this work is to design a video pre-training method to capture internal interactions.
Revised: The main purpose of this work is to design a video pretraining method to capture internal interactions.

Original: In real life, a lot of tasks are temporally-heavy such that the task completion relies on understanding the past.
Revised: In real life, a lot of tasks are temporally-heavy such that the task completion relies on understanding the past.

Original: Video understanding is a proxy to these embodied studies eventually and thus could one day beneﬁt our daily life.
Revised: Video understanding is a proxy to these embodied studies eventually and thus could one day beneﬁt our daily life.

Original: Our work is also potentially useful for recovering corrupted videos (but this could also be possibly misused, hence we recommend careful and safe use of this technology, including previous works).
Revised: Our work is also potentially useful for recovering corrupted videos (but this could also be possibly misused, hence we recommend careful and safe use of this technology, including previous works).

Revised: In this paper, we mainly consider the action classiﬁcation tasks with different characteristics (i.e., spatially-heavy and temporally-heavy) and we show the recovering ability of our model.

Revised: It’s also possible to employ

Revised: VIMPAC backbone to video detection tasks where tokens’ outputs can be viewed as anchors.

Original: The DALL-E VQ-VAE code and model are published under MIT license.
Revised: The DALL-E VQ-VAE code and model are published under MIT license.

Revised: We are among the ﬁrst line of work that use masked token reconstruction, etc.

Revised: as a self-supervised learning approach for vision, and we do agree the the use of VQ-VAE tokens might limit the models’ ability in certain aspects, such as spatial info loss.

Revised: Future work could explore combining the techniques from BEiT (Bao et al., 2021) for a more capable approach.

Original: A M ODEL A RCHITECTURE
Revised: Meanwhile, token-level approach has its unique advantage of generative modeling (as shown in VQ-GAN (Esser et al., 2021), DALL-E (Ramesh et al., 2021)), more robust to the input noise, and has the possibility to transfer to other modality (e.g., text; since they share the same types of input). This cross-modality transferability is another future direction that we consider.

Original: As described inSec. 3.
Revised: In this supplementary materials we start with describing details of the model (Sec. A), pre-training (Sec. B), experiments (Sec. C) and dataset (Sec. D). We then provide additional analysis results in

Revised: Sec. F and visualization in Sec.

Original:  we use a transformer model on top of the discrete video tokens generated by VQ-VAE.
Revised: G. A M ODEL A RCHITECTURE As described in Sec. 3.4, we use a transformer model on top of the discrete video tokens generated by VQ-VAE.

Original: Since transformers have different variants we here show details of our architecture for clarity.
Revised: Since transformers have different variants, we here show details of our architecture for clarity.

Original: The design of our method largely follows the practice in BERT (Devlin et al., 2019),
Revised: The design of our method largely follows the practice in BERT (Devlin et al., 2019),

Original: TimeSformer (Bertasius et al., 2021), Sparser Transformer (Child et al., 2019), ViViT (Arnab et al., 2021) and MoCoV3 (Chen et al., 2021).
Revised: TimeSformer (Bertasius et al., 2021), Sparser Transformer (Child et al., 2019), ViViT (Arnab et al., 2021), and MoCoV3 (Chen et al., 2021).

Original: A. B ACKBONE M ODEL
Revised: A. B ACKBONE M ODEL

Original: Embedding.
Revised: Embedding.

Original: Given the video frames t f t P R H ˆ W | t P r T su , we ﬁrst use VQ-VAE (van den
Revised: Given the video frames t f t P R H ˆ W | t P r T su , we ﬁrst use VQ-VAE (van den

Original: Oord et al., 2017; Ramesh et al., 2021) to discrete them into video tokens t x t,i,j P r V s | t P r ˆ T s , i P
Revised: Oord et al., 2017; Ramesh et al., 2021) to discrete them into video tokens t x t,i,j P r V s | t P r ˆ T s , i P

Original: r ˆ H s , j P r ˆ W su .
Revised: r ˆ H s , j P r ˆ W su .

Original: We use the speciﬁc VQ-VAE in DALL-E (Ramesh et al., 2021) which is trained on trained on 250 million images from the Internet.
Revised: We use the speciﬁc VQ-VAE in DALL-E (Ramesh et al., 2021) which is trained on trained on 250 million images from the Internet.

Original: Since the VQ-VAE encoder largely compresses a 8x8x3 vector (ranging from 0-255) to an integer of 0-8191, it is considered as a compression method with image prior.
Revised: Since the VQ-VAE encoder largely compresses a 8x8x3 vector (ranging from 0-255) to an integer of 0-8191, it is considered as a compression method with image prior.

Original: We do not use the Video-VQVAE (Walker et al., 2021) method since the imagetrained VQVAE has been pretrained on a very large image corpus and as a consequence cover a much more diverse set of visual scenes and elements.
Revised: We do not use the Video-VQVAE (Walker et al., 2021) method since the imagetrained VQVAE has been pretrained on a very large image corpus and as a consequence cover a much more diverse set of visual scenes and elements.

Original: We next use an embedding layer ( embedding ) that to map these discrete tokens to continuous vectors.
Revised: We next use an embedding layer ( embedding ) that to map these discrete tokens to continuous vectors.

Original: Since transformer layers are permutationinvariant, we follow (Dosovitskiy et al., 2021; Devlin et al., 2019) to add positional information into the input.
Revised: Since transformer layers are permutationinvariant, we follow (Dosovitskiy et al., 2021; Devlin et al., 2019) to add positional information into the input.

Original: The positional embedding ( pos ) is factorized as a sum of the temporal embedding pos T , the height embedding pos H , and the width embedding pos W .
Revised: The positional embedding ( pos ) is factorized as a sum of the temporal embedding pos T , the height embedding pos H , and the width embedding pos W .

Original: This factorization reduces the number of trainable parameters to encode positional information, which empirically shows a slightly better result.
Revised: This factorization reduces the number of trainable parameters to encode positional information, which empirically shows a slightly better result.

Original: Finally, a Layer-Normalization (Ba et al., 2016) layer is added to get the initial hidden outputs h 0 t,i,j :
Revised: Finally, a Layer-Normalization (Ba et al., 2016) layer is added to get the initial hidden outputs h 0 t,i,j :

Original: [Equation]
Revised: [Equation]

Original: where we use the superscript 0 to denote that it is the hidden outputs before the ﬁrst transformer layer.
Revised: where we use the superscript 0 to denote that it is the hidden outputs before the ﬁrst transformer layer.

Original: Attention Blocks.
Revised: Attention Blocks.

Original: Before introducing the detailed model architecture, we ﬁrst describe the basic building components: the attention block.
Revised: Before introducing the detailed model architecture, we ﬁrst describe the basic building components: the attention block.

Original: An attention block is built based on the attention operator (i.e., ‘ Attn ’) with a residual connection.
Revised: An attention block is built based on the attention operator (i.e., ‘ Attn ’) with a residual connection.

Original: The attention operator takes a single query vector x and its context t
Revised: The attention operator takes a single query vector x and its context t

Original: y i u as input.
Revised: y i u as input.

Original: It ﬁrst computes the attention score between x and each context vector
Revised: It ﬁrst computes the attention score between x and each context vector

Original: y i , then the attention scores are normalized by the softmax .
Revised: y i , then the attention scores are normalized by the softmax .

Original: Lastly, the output is a weighted-sum over all the context vectors (transferred by a ‘value’ matrix W value ):
Revised: Lastly, the output is a weighted-sum over all the context vectors (transferred by a ‘value’ matrix W value ):

Original: [Equation]
Revised: [Equation]

Original: To compose the attention block from the previous attention operator, the residual connection and layer normalization (i.e., ‘ LayerNorm ’) are added.
Revised: To compose the attention block from the previous attention operator, the residual connection and layer normalization (i.e., ‘ LayerNorm ’) are added.

Original: We follow the original transformer model (Vaswani et al., 2017) that uses a post-layer-norm layout:
Revised: We follow the original transformer model (Vaswani et al., 2017) that uses a post-layer-norm layout:

Original: [Equation]
Revised: [Equation]

Original: In order to reduce computational cost and memory, we also adapt the attention block suggested in Sparse Transformer (Child et al., 2019) that takes two sets of context vectors t
Revised: In order to reduce computational cost and memory, we also adapt the attention block suggested in Sparse Transformer (Child et al., 2019) that takes two sets of context vectors t

Original: y i u and t z j u as input.
Revised: y i u and t z j u as input.

Original: This special attention block computes attention for the two context-vector sets separately and concatenates their output together.
Revised: This special attention block computes attention for the two context-vector sets separately and concatenates their output together.

Original: In our case, suppose t y i u and t z j u are the rows and columns of a square matrix, then it reduces the computation cost of calculating attention scores from Θ p n 4 q to Θ p n 2 q , where n is the number of rows/columns:
Revised: In our case, suppose t y i u and t z j u are the rows and columns of a square matrix, then it reduces the computation cost of calculating attention scores from Θ p n 4 q to Θ p n 2 q , where n is the number of rows/columns:

Original: [Equation]
Revised: [Equation]

Original: Spatio-Temporal Transformer Layer.
Revised: Spatio-Temporal Transformer Layer.

Original: The spatio-temporal transformer layer is composed with the previously-introduced attention blocks and an additional MLP block.
Revised: The spatio-temporal transformer layer is composed with the previously-introduced attention blocks and an additional MLP block.

Original: The l -th layer takes the output of the previous layer t h l ´ 1 t,i,j u as input and outputs the hidden states t h lt,i,j u .
Revised: The l -th layer takes the output of the previous layer t h l ´ 1 t,i,j u as input and outputs the hidden states t h lt,i,j u .

Original: We separate the attention into two attention blocks: the temporal attention block AttnBlock TIME and the spatial attention block AttnBlock SPACE .
Revised: We separate the attention into two attention blocks: the temporal attention block AttnBlock TIME and the spatial attention block AttnBlock SPACE .

Original: Without loss of generality, we will use g TIME and g SPACE to denote the intermediate results from temporal and spatial attention blocks, respectively.
Revised: Without loss of generality, we will use g TIME and g SPACE to denote the intermediate results from temporal and spatial attention blocks, respectively.

Original: First, the temporal attention block attends to the tokens at the same spatial location but in different frames (i.e., at different timesteps): t h l ´ 1 t,i,j | t P r T su .
Revised: First, the temporal attention block attends to the tokens at the same spatial location but in different frames (i.e., at different timesteps): t h l ´ 1 t,i,j | t P r T su .

Original: Next, the spatial attention block attends to the tokens in the same frame: t g T t,i,j | p i, j q P r ˆ H s ˆ r ˆ W su .
Revised: Next, the spatial attention block attends to the tokens in the same frame: t g T t,i,j | p i, j q P r ˆ H s ˆ r ˆ W su .

Original: To reduce the computational cost, we incorporate the sparse attention block (Child et al., 2019)
Revised: To reduce the computational cost, we incorporate the sparse attention block (Child et al., 2019)

Original: (detailed in the previous paragraph) that factorizes the attention over height and width: t g T t,i,j | i P r ˆ H su , t g T t,i,j | j P r ˆ W su .
Revised: (detailed in the previous paragraph) that factorizes the attention over height and width: t g T t,i,j | i P r ˆ H su , t g T t,i,j | j P r ˆ W su .

Original: The MLP block has two fully-connected layers with GeLU (Hendrycks & Gimpel, 2016) activation in the middle.
Revised: The MLP block has two fully-connected layers with GeLU (Hendrycks & Gimpel, 2016) activation in the middle.

Original: Overall, the formula of one spatio-temporal transformer layer is:
Revised: Overall, the formula of one spatio-temporal transformer layer is:

Original: [Equation]
Revised: [Equation]

Original: [CLS] Token.
Revised: [CLS] Token.

Original: Following the practice in BERT (Devlin et al., 2019) design, we add a special [CLS] (abbreviation of ‘classiﬁcation’) token and take its output as the representation of the whole sequence.
Revised: Following the practice in BERT (Devlin et al., 2019) design, we add a special [CLS] (abbreviation of ‘classiﬁcation’) token and take its output as the representation of the whole sequence.

Original: We follow TimeSformer (Bertasius et al., 2021) to compute the its output: the [CLS] token attends over the context separately and then the outputs are averaged.
Revised: We follow TimeSformer (Bertasius et al., 2021) to compute the its output: the [CLS] token attends over the context separately and then the outputs are averaged.

Original: We take the temporal attention layer as an example.
Revised: We take the temporal attention layer as an example.

Original: Suppose h l ´ 1 cls is the [CLS] feature vector output by layer l ´ 1 , then the temporal attention layer do the following computation:
Revised: Suppose h l ´ 1 cls is the [CLS] feature vector output by layer l ´ 1 , then the temporal attention layer do the following computation:

Original: [Equation]
Revised: [Equation]

Original: The other attention blocks process the [CLS] token similarly.
Revised: The other attention blocks process the [CLS] token similarly.

Original: A. P RE -T RAINING AND F INE -T UNING H EADS
Revised: A. P RE -T RAINING AND F INE -T UNING H EADS

Original: Pre-training or ﬁne-tuning usually requires a few additional modules (i.e., heads) on top of the transformer layers that convert the output features to the desired probabilities or vectors.
Revised: Pre-training or ﬁne-tuning usually requires a few additional modules (i.e., heads) on top of the transformer layers that convert the output features to the desired probabilities or vectors.

Original: We next describe the heads used in our pre-training and ﬁne-tuning process.
Revised: We next describe the heads used in our pre-training and ﬁne-tuning process.

Original: Token Head for Mask-then-Predict.
Revised: Token Head for Mask-then-Predict.

Original: We ﬁrst deﬁne the prediction head over the tokens following BERT(Devlin et al., 2019).
Revised: We ﬁrst deﬁne the prediction head over the tokens following BERT(Devlin et al., 2019).

Original: It ﬁrst processes the last-layer hidden outputs h Lt,i,j using a fullyconnected layer (with GELU activation (Hendrycks & Gimpel, 2016) and layer normalization (Ba et al., 2016)):
Revised: It ﬁrst processes the last-layer hidden outputs h Lt,i,j using a fullyconnected layer (with GELU activation (Hendrycks & Gimpel, 2016) and layer normalization (Ba et al., 2016)):

Original: [Equation]
Revised: [Equation]

Original: In our mask-then-predict method (Sec. 3.1), we will predict the masked tokens (i.e., the token before masking) from their context.
Revised: In our mask-then-predict method (Sec. 3.1), we will predict the masked tokens (i.e., the token before masking) from their context.

Original: We thus further convert this hidden vector into a distribution over the token vocabulary:
Revised: We thus further convert this hidden vector into a distribution over the token vocabulary:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The weight W word is shared with input word embedding layer embedding (Press & Wolf, 2017;
Revised: The weight W word is shared with input word embedding layer embedding (Press & Wolf, 2017;

Original: Devlin et al., 2019) while the bias b word is trained independently. 
Revised: Devlin et al., 2019) while the bias b word is trained independently. [Table]

Original: Contrastive Learning Head
Revised: Contrastive Learning Head

Original: Next we discuss the pre-training heads for contrastive learning.
Revised: Next we discuss the pre-training heads for contrastive learning.

Original: It is on top of the [CLS] hidden output h CLS .
Revised: It is on top of the [CLS] hidden output h CLS .

Original: We encode the hidden state with MLP.
Revised: We encode the hidden state with MLP.

Original: We use batch normalization (Ioffe & Szegedy, 2015) inside the MLP head following the practice in (Chen et al., 2021).[Equation]
Revised: We use batch normalization (Ioffe & Szegedy, 2015) inside the MLP head following the practice in (Chen et al., 2021).

Original: [Equation]
Revised: [Equation]

Original: This f CLS feature is used in computing the contrastive loss as in Sec. 3.2.
Revised: This f CLS feature is used in computing the contrastive loss as in Sec. 3.2.

Original: FC Layer for Fine-Tuning.
Revised: FC Layer for Fine-Tuning.

Original: When ﬁne-tuning for action classiﬁcation task, we add a fullyconnected (FC) layer to the [CLS] output h cls .
Revised: When ﬁne-tuning for action classiﬁcation task, we add a fullyconnected (FC) layer to the [CLS] output h cls .

Original: We initialize its weight and bias to zero.
Revised: We initialize its weight and bias to zero.

Original: Special Tokens.
Revised: Special Tokens.

Original: Besides the V token types introduced in the vocabulary of the VQ-VAE (see
Revised: Besides the V token types introduced in the vocabulary of the VQ-VAE (see

Original: Sec. 3.1), we add several special tokens into the ‘vocabulary’, namely a [CLS] token is introduced as a stub for the whole-video representation, a [PAD] token is used when the actual clip length is less than the model’s expected input length.
Revised: Sec. 3.1), we add several special tokens into the ‘vocabulary’, namely a [CLS] token is introduced as a stub for the whole-video representation, a [PAD] token is used when the actual clip length is less than the model’s expected input length.

Original: For the mask-then-predict task, we follow BERT (Devlin et al., 2019) to replace the masked tokens with a speciﬁc [MASK] token.
Revised: For the mask-then-predict task, we follow BERT (Devlin et al., 2019) to replace the masked tokens with a speciﬁc [MASK] token.

Original: B P RE -T RAINING D ETAILS B. M ASKING B LOCKS
Revised: B P RE -T RAINING D ETAILS B. M ASKING B LOCKS

Original: As described in Sec. 3.1, we mask the tokens by blocks (a cube-shape set of tokens).
Revised: As described in Sec. 3.1, we mask the tokens by blocks (a cube-shape set of tokens).

Original: To avoid masking all the tokens in the clip, we control the maximum block length for the time domain, height, and width.
Revised: To avoid masking all the tokens in the clip, we control the maximum block length for the time domain, height, and width.

Original: For spatial dimensions (i.e., height and width), the maximum length is half of the full length (e.g., the maximum block length will be 16 for a token map of length 32).For temporal [Table]
Revised: For spatial dimensions (i.e., height and width), the maximum length is half of the full length (e.g., the maximum block length will be 16 for a token map of length 32).

Original: [Table] dimension (i [Table] e., the clip length), the maximum length will be 2/3 (round up) of the full length so that it allows long-range modeling.
Revised: For temporal dimension (i.e., the clip length), the maximum length will be 2/3 (round up) of the full length so that it allows long-range modeling.

Original: Under these constraints, we uniformly sample a ﬁxed number of mask blocks and take their union to construct the ﬁnal mask.
Revised: Under these constraints, we uniformly sample a ﬁxed number of mask blocks and take their union to construct the ﬁnal mask.

Original: The number of blocks is decided by the induced masking ratio, which depends on the input resolutions.
Revised: The number of blocks is decided by the induced masking ratio, which depends on the input resolutions.

Original: In Table 9, we show the induced masking ratio w.r.t.
Revised: In Table 9, we show the induced masking ratio w.r.t.

Original: different input resolutions and #masking blocks.
Revised: different input resolutions and #masking blocks.

Original: We take the VQ-VAE (van den
Revised: We take the VQ-VAE (van den

Original: Oord et al., 2017) provided in DALL-E (Ramesh et al., 2021) that has a compression factor of 8, thus the length of the token map is always 1/8 of the frame size.
Revised: Oord et al., 2017) provided in DALL-E (Ramesh et al., 2021) that has a compression factor of 8, thus the length of the token map is always 1/8 of the frame size.

Original: For each input resolution, we select the number of blocks (shown in bold in Table 9) whose induced masking ratio is closet to 15 % following
Revised: For each input resolution, we select the number of blocks (shown in bold in Table 9) whose induced masking ratio is closet to 15 % following

Original: BERT (Devlin et al., 2019).
Revised: BERT (Devlin et al., 2019).

Original: B. C ONTRASTIVE L EARNING L OSS
Revised: B. C ONTRASTIVE L EARNING L OSS

Original: For completeness, we list the two losses used in contrastive learning here.
Revised: For completeness, we list the two losses used in contrastive learning here.

Original: The ﬁrst loss for clip c
Revised: The ﬁrst loss for clip c

Original: i from video i is:
Revised: i from video i is:

Original: [Equation]
Revised: [Equation]

Original: The symmetric loss L 1 InfoNCE p i q for feature of the other clip sample c 1 i from video i (and its feature f 1 i ) is:
Revised: The symmetric loss L 1 InfoNCE p i q for feature of the other clip sample c 1 i from video i (and its feature f 1 i ) is:

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original: C E XPERIMENT D ETAILS
Revised: C E XPERIMENT D ETAILS

Original: In this section, we show our model conﬁguration and training hyperparameters in details to support the reproducibility of our experiments.
Revised: In this section, we show our model conﬁguration and training hyperparameters in details to support the reproducibility of our experiments.

Original: Model Conﬁguration.
Revised: Model Conﬁguration.

Original: Our model conﬁguration details is shown in Table 10.
Revised: Our model conﬁguration details is shown in Table 10.

Original: Most analysis results (Sec. 5) take ‘Small’ models and our ﬁnal results (Sec. 4.3) take ‘Large-Half’ model.
Revised: Most analysis results (Sec. 5) take ‘Small’ models and our ﬁnal results (Sec. 4.3) take ‘Large-Half’ model.

Original: Other models[Table] are used in Sec. 5.1.
Revised: Other models are used in Sec. 5.1.

Original: The ﬁnal ‘Large-Half’ model halves the attention head dimension and MLP intermediate size as in (Child et al., 2019).
Revised: The ﬁnal ‘Large-Half’ model halves the attention head dimension and MLP intermediate size as in (Child et al., 2019).

Original: For the pre-training heads, we follow BERT to take the intermediate dimension of the token-head to be the same as the backbone’s hidden dimension.
Revised: For the pre-training heads, we follow BERT to take the intermediate dimension of the token-head to be the same as the backbone’s hidden dimension.

Original: For the CLS head, we take 3 layers in MLP and 4096 intermediate dimensions.
Revised: For the CLS head, we take 3 layers in MLP and 4096 intermediate dimensions.

Original: The output dimension is 256.
Revised: The output dimension is 256.

Original: We test with different number of layers and hidden dimensions of CLS head and generally ﬁnd that larger head gives better results (as in (Qian et al., 2021; Chen et al., 2021)).
Revised: We test with different number of layers and hidden dimensions of CLS head and generally ﬁnd that larger head gives better results (as in (Qian et al., 2021; Chen et al., 2021)).

Original: This CLS head contributes to about 1% pre-training computational cost overhead.
Revised: This CLS head contributes to about 1% pre-training computational cost overhead.

Original: Training Hyperparameters.
Revised: Training Hyperparameters.

Original: We list the training hyperparameters in Table 11.
Revised: We list the training hyperparameters in Table 11.

Original: Most of the hyperparameters are inherited from previous works to allow fair comparison and reduce tuning effort.
Revised: Most of the hyperparameters are inherited from previous works to allow fair comparison and reduce tuning effort.

Original: For optimizer hyperparameters, we mostly follow the implementation of DALL-E (Ramesh et al., 2021) and BERT (Devlin et al., 2019).
Revised: For optimizer hyperparameters, we mostly follow the implementation of DALL-E (Ramesh et al., 2021) and BERT (Devlin et al., 2019).

Original: SSV2 follows the epoch number in (Feichtenhofer et al., 2019) and (Feichtenhofer et al., 2021).
Revised: SSV2 follows the epoch number in (Feichtenhofer et al., 2019) and (Feichtenhofer et al., 2021).

Original: To reduce the computational cost, we pre-extract the VQ-VAE tokens thus we employ a ﬁxed set of spatial data augmentations.
Revised: To reduce the computational cost, we pre-extract the VQ-VAE tokens thus we employ a ﬁxed set of spatial data augmentations.

Original: As listed in the bottom of Table 11, we exclude any color distortion and gray scale augmentation.
Revised: As listed in the bottom of Table 11, we exclude any color distortion and gray scale augmentation.

Original: We resize the video clip to the desired frame size and center-crop it during pre-training.
Revised: We resize the video clip to the desired frame size and center-crop it during pre-training.

Original: For downstream tasks, we resize the video clip to frame size or 1.5 times of the frame size, then crop the clip (with frame-size by frame-size spatial size) from the top-left, center, and bottom-right. We apply (horizontal) ﬂip to the raw frames thus a total of 12 spatial augmentations are extracted (12 = 2 resize ˆ 3 crops ˆ 3 ﬂip/no-ﬂip). The only exception is SSV2.
Revised: For downstream tasks, we resize the video clip to frame size or 1.5 times of the frame size, then crop the clip (with frame-size by frame-size spatial size) from analyze

Original: This dataset needs to distinguish left/right motions thus we exclude the ﬂip augmentation and only use the center crop during inference following previous works (Feichtenhofer et al., 2019; 2021).

Original: During pre-training, we always accumulate the gradient to a batch size ofbefore updating the weights but use different numbers of negative examples.

Original: We analyze this effect in Sec.

Original: Following previous works (Feichtenhofer et al., 2019), we increase the training epochs for the nonpre-training models by 4 ˆ for small datasets (Diving48, UCF101, HMDB51) and 1.5 ˆ for larger datasets (SSV2, Kinetics-400).
Revised: Following previous works (Feichtenhofer et al., 2019), we increase the training epochs for the nonpre-training models by 4 ˆ for small datasets (Diving48, UCF101, HMDB51) and 1.5 ˆ for larger datasets (SSV2, Kinetics-400).

Original: When analyzing the mask-then-predict task in Sec.
Revised: When analyzing the mask-then-predict task in Sec.

Original: 5.2.2 (and all other analysis focusing on maskthen-predict), we exclude the contrastive learning loss (by setting loss weight α = 0 ) to preclude potential side effects.
Revised: 5.2.2 (and all other analysis focusing on maskthen-predict), we exclude the contrastive learning loss (by setting loss weight α = 0 ) to preclude potential side effects.

Original: However, we still use masked prediction loss when assessing the contrastive learning task in Sec.
Revised: However, we still use masked prediction loss when assessing the contrastive learning task in Sec.

Original: 5.2.3 as we observe very low performance with only contrastive learning objective.
Revised: 5.2.3 as we observe very low performance with only contrastive learning objective.

Original: [Table]
Revised: Pre-Training with Kinetics datasets. Besides pre-training on the HowTo100M (Miech et al., 2019) dataset, we have an experiment with smaller models (i.e., base model in Table 10) pre-trained on K600 dataset. We found that the pure mask-prediction task on K600 (Carreira et al., 2019) can reach 88% on UCF101, but adding the contrastive learning task does not show signiﬁcant further improvement (+0.5% „ 1% according to the hyperparameters and seeds). This is possibly due to

Original: [Table]
Revised: K600 dataset is designed speciﬁcally for action recognition on shorter video-clip where the average length of the video is 10 second. This short-range video would limit the success of the masked prediction and contrastive learning combination (as illustrated in Table 6 on HowTo100M dataset, d max = 10s). Given these observations, we did not run the full model (large model in Table 10) on

Original: [Table]
Revised: K600 because of the budget constraint.

Original: D D ATASET D ETAILS
Revised: D D ATASET D ETAILS

Original: In Table [Table] we list the key statistics of the datasets used in our paper.
Revised: In Table 12, we list the key statistics of the datasets used in our paper.

Original: HowTo100M is our pretraining datasets that has long-duration uncurated videos.
Revised: HowTo100M is our pretraining datasets that has long-duration uncurated videos.

Original: The videos are collected from YouTube by searching key phrases thus the scale could be easily increased.
Revised: The videos are collected from YouTube by searching key phrases thus the scale could be easily increased.

Original: SSV2 and Kinetics-400 are two large downstream datasets, where SSV2 focuses more on the actions and Kinetics-400 focuses more on the scenes.
Revised: SSV2 and Kinetics-400 are two large downstream datasets, where SSV2 focuses more on the actions and Kinetics-400 focuses more [Table] [Table] [Table]

Revised: on the scenes.

Original: Diving48, UCF101, HMDB51 are three small datasets.
Revised: Diving48, UCF101, HMDB51 are three small datasets.

Original: Different from previous datasets on classifying different action types (thus might be potentially inferred from single frames),
Revised: Different from previous datasets on classifying different action types (thus might be potentially inferred from single frames),

Original: Diving48 studies the three stages (takeoff, ﬂight, and entry) of competitive diving.
Revised: Diving48 studies the three stages (takeoff, ﬂight [Table] and entry) of competitive diving.

Original: Thus achieving good results on Diving48 requires an understanding of the whole video clip.
Revised: Thus achieving good results on Diving48 requires an understanding of the whole video clip.

Original: The license of each dataset allows academic use.
Revised: The license of each dataset allows academic use.

Original: SSV2 uses the ‘Free Academic License’.
Revised: SSV2 uses the ‘Free Academic License’.

Original: HMDB51 is licensed under ‘Creative Commons Attribution 4.0 International License’.
Revised: HMDB51 is licensed under ‘Creative Commons Attribution 4.0 International License’.

Original: ‘The kinetics dataset is licensed by
Revised: ‘The kinetics dataset is licensed by

Original: Google Inc. under a Creative Commons Attribution 4.0 International License.’
Revised: Google Inc. under a Creative Commons Attribution 4.0 International License.’

Original: We use the YouTube videos under the Fair Use.
Revised: We use the YouTube videos under the Fair Use.

Original: E C OMPUTATIONAL C OST
Revised: E C OMPUTATIONAL C OST

Original: The pre-training takes around 8.9K V100 GPU hours.
Revised: The pre-training takes around 8.9K V100 GPU hours.

Original: This computational cost is at the same level as ViT (Dosovitskiy et al., 2021) supervised training (5.5K hours on ImageNet-21K (Russakovsky et al., 2015) and 12.8K on JFT(Sun et al., 2017)).
Revised: This computational cost is at the same level as ViT (Dosovitskiy et al., 2021) supervised training (5.5K hours on ImageNet-21K (Russakovsky et al., 2015) and 12.8K on JFT(Sun et al., 2017)).

Original: It is also at the same level of supervised training a model on Kinetics-400 dataset (6.4K for SlowFast (Feichtenhofer et al., 2019), about 5.6K for
Revised: It is also at the same level of supervised training a model on Kinetics-400 dataset (6.4K for SlowFast (Feichtenhofer et al., 2019), about 5.6K for

Original: TimeSformer-L (Bertasius et al., 2021)).
Revised: TimeSformer-L (Bertasius et al., 2021)).

Original: For ﬁne-tuning, SSV2, Diving48, UCF101, HMDB51, and Kinetics-400 take 1K, 200, 150, 40, 2K GPU hours, respectively.
Revised: For ﬁne-tuning, SSV2, Diving48, UCF101, HMDB51, and Kinetics-400 take 1K, 200, 150, 40, 2K GPU hours, respectively.

Original: For analysis, the pre-training takes about 160 GPU hours.
Revised: For analysis, the pre-training takes about 160 GPU hours.

Original: Besides the ﬁnal model training, energy is also spent on tuning the model and ﬁnding the best conﬁguration.
Revised: Besides the ﬁnal model training, energy is also spent on tuning the model and ﬁnding the best conﬁguration.

Original: As shown in Sec.
Revised: As shown in Sec.

Original: 5.2.3, our method is more robust to the hyperparameters.
Revised: 5.2.3, our method is more robust to the hyperparameters.

Original: F A DDITIONAL A NALYSIS R ESULTS F. M ODEL A RCHITECTURE C OMPARISONS
Revised: F A DDITIONAL A NALYSIS R ESULTS F. M ODEL A RCHITECTURE C OMPARISONS

Original: Attention Layouts.
Revised: Attention Layouts.

Original: We here compare different alternative model architectures in Table 13.
Revised: We here compare different alternative model architectures in Table 13.

Original: We ﬁrst experiment with different attention layouts discussed in Sec. 3.4.
Revised: We ﬁrst experiment with different attention layouts discussed in Sec. 3.4.

Original: We consider the sparse attention as proposed in (Child et al., 2019) and the sequential attention blocks as in (Bertasius et al., 2021).
Revised: We consider the sparse attention as proposed in (Child et al., 2019) and the sequential attention blocks as in (Bertasius et al., 2021).

Original: The ‘TxHxW’ model is the basic attention module that takes the ﬂattened tokens as input (of 
Revised: The ‘TxHxW’ model is the basic attention module that takes the ﬂattened tokens as input (of shape

Original: [Table] [Table]
Revised: T

Original: shape T ˆ H ˆ W).
Revised: ˆ H ˆ W).

Original: At each layer, each token attends to all other tokens.
Revised: At each layer, each token attends to all other tokens.

Original: The ‘T,HxW’ model separates the temporal attention and spatial attention (‘Divided Space-Time’ in (Bertasius et al., 2021)).
Revised: The ‘T,HxW’ model separates the temporal attention and spatial attention (‘Divided Space-Time’ in (Bertasius et al., 2021)).

Original: The ‘T,H,W’ model processes three attention sequentially (‘Axial Attention’ in (Bertasius et al., 2021)).
Revised: The ‘T,H,W’ model processes three attention sequentially (‘Axial Attention’ in (Bertasius et al., 2021)).

Original: The ‘T, H—W’ model is our default model that sequentially conduct temporal attention and spatial attention, where the spatial attention are parallel into the height attention and width attention.
Revised: The ‘T, H—W’ model is our default model that sequentially conduct temporal attention and spatial attention, where the spatial attention are parallel into the height attention and width attention.

Original: As shown in Table 13, ‘T,H—W’ reaches a balance between speed and accuracy. 
Revised: As shown in Table 13, ‘T,H—W’ reaches a balance between speed and accuracy. [Table] [Table]

Original: Pre-Layer-Normalization vs.
Revised: Pre-Layer-Normalization vs.

Original: Post-Layer-Normalization.
Revised: Post-Layer-Normalization.

Original: Besides the architectures listed above, we also consider the pre-layer-norm (used in GPT and ViT) and post-layer-norm (used in
Revised: Besides the architectures listed above, we also consider the pre-layer-norm (used in GPT and ViT) and post-layer-norm (used in

Original: BERT) variation.
Revised: BERT) variation.

Original: We empirically ﬁnd that post-layer-norm architecture is better for our pre-training tasks as shown in Table 13 (comparing the last 2 columns).
Revised: We empirically ﬁnd that post-layer-norm architecture is better for our pre-training tasks as shown in Table 13 (comparing the last 2 columns).

Original: F. I NPUT R ESOLUTION
Revised: F. I NPUT R ESOLUTION

Original: In Table 14, we show model scalability over input resolution (i.e., #frames T and frame size S ).
Revised: In Table 14, we show model scalability over input resolution (i.e., #frames T and frame size S ).

Original: With the same frame size S , longer clips perform better than shorter clips (e.g., T =10, S =128 is better than T =5, S =128).
Revised: With the same frame size S , longer clips perform better than shorter clips (e.g., T =10, S =128 is better than T =5, S =128).

Original: With the same number of input frames T , larger frame size improves the performance (e.g., T =10, S =256 is better than T =10, S =128).
Revised: With the same number of input frames T , larger frame size improves the performance (e.g., T =10, S =256 is better than T =10, S =128).

Original: For each pre-training resolution, we also try to ﬁne-tune under a full-resolution with T =10, S =256 (denoted as ‘UCF101-Full-Reso.’).
Revised: For each pre-training resolution, we also try to ﬁne-tune under a full-resolution with T =10, S =256 (denoted as ‘UCF101-Full-Reso.’).

Original: As in pre-training, ﬁne-tuning with larger resolution generally improves the results.
Revised: As in pre-training, ﬁne-tuning with larger resolution generally improves the results.

Original: Although longer and smaller clips ( T =10, S =128) show better results than shorter and larger clips ( T =5, S =256) when using the same pre-training and ﬁne-tuning resolutions, they show different trends with the full-resolution ﬁne-tuning.
Revised: Although longer and smaller clips ( T =10, S =128) show better results than shorter and larger clips ( T =5, S =256) when using the same pre-training and ﬁne-tuning resolutions, they show different trends with the full-resolution ﬁne-tuning.

Original: Increasing frame size during ﬁne-tuning (the second block in Table 14) only improves the UCF101 result by 0 .
Revised: Increasing frame size during ﬁne-tuning (the second block in Table 14) only improves the UCF101 result by 0 .

Original: 4 , while increasing the clip length (the third block) improves the UCF101 result by 3.8.
Revised: 4 , while increasing the clip length (the third block) improves the UCF101 result by 3.8.

Original: These results call for a need of pre-training with large spatial size, and we follow this practice in our large-scale experiments as in Sec. 4.3.
Revised: These results call for a need of pre-training with large spatial size, and we follow this practice in our large-scale experiments as in Sec. 4.3.

Original: F. N OISY M ASKING FOR M ASK - THEN -P REDICT
Revised: F. N OISY M ASKING FOR M ASK - THEN -P REDICT

Original: Our default masking strategy replaces all masked tokens with a special [MASK] symbol.
Revised: Our default masking strategy replaces all masked tokens with a special [MASK] symbol.

Original: We also experiment with BERT’s masking strategy that only replaces 80 % of masked tokens to the MASK symbol.
Revised: We also experiment with BERT’s masking strategy that only replaces 80 % of masked tokens to the MASK symbol.

Original: For other tokens, 10 % are randomly sampled from the ‘vocabulary’ and 10 % are kept the same.
Revised: For other tokens, 10 % are randomly sampled from the ‘vocabulary’ and 10 % are kept the same.

Original: For smaller experiments, the two masking strategies show similar results.
Revised: For smaller experiments, the two masking strategies show similar results.

Original: However, this
Revised: However, this

Original: BERT’s noisy masking strategy has lower convergence stability on the larger model pre-training.
Revised: BERT’s noisy masking strategy has lower convergence stability on the larger model pre-training.

Original: The pre-training diverges after about 10 epochs (out of the 100 epochs).[Table]
Revised: The pre-training diverges after about 10 epochs (out of the 100 epochs).

Original: F. M ASKING R ATIO FOR B LOCK -M ASKING AND I.I.D. M ASKING
Revised: F. M ASKING R ATIO FOR B LOCK -M ASKING AND I.I.D. M ASKING

Original: We test the effect of different masking ratios.
Revised: We test the effect of different masking ratios.

Original: In the main text, we control the number of blocks for block masking.
Revised: In the main text, we control the number of blocks for block masking.

Original: In Table 15, we here also show the results of matched masking ratio for i.i.d.
Revised: In Table 15, we here also show the results of matched masking ratio for i.i.d.

Original: masking for completeness.
Revised: masking for completeness.

Original: Empirically, the result differences among various masking ratios are marginal and the original BERT’s 15 % masking ratio (with roughly 5 masking blocks) works slightly better.
Revised: Empirically, the result differences among various masking ratios are marginal and the original BERT’s 15 % masking ratio (with roughly 5 masking blocks) works slightly [Table] [Figure] [Table]

Original:  Thus we always select the number of mask blocks whose induced masking ratio is closest to 15 %.
Revised: better. Thus we always select the number of mask blocks whose induced masking ratio is closest to 15 %.

Original: For all masking ratios, block masking shows signiﬁcantly better results than the i.i.d.
Revised: For all masking ratios, block masking shows signiﬁcantly better results than the i [Table] i

Revised: [Table]

Original: masking.
Revised: d. masking

Original: F. I MPACT OF C ONTRASTIVE L EARNING L OSS W EIGHT
Revised: F. I MPACT OF C ONTRASTIVE L EARNING L OSS W EIGHT

Original: In Table 16, we show the impact of loss weight α
Revised: In Table [Table] we show the impact of loss weight α

Original: (see Sec. 3.3).
Revised: (see Sec. 3.3).

Original: Since the loss have been calibrated by multiplying the temperature, α = 1 shows stable results and α = 0 .
Revised: Since the loss have been calibrated by multiplying the temperature, α = 1 shows stable results and α = 0 .

Original: 5 is slightly better.
Revised: 5 is slightly better.

Original: Setting α = 2 .will let the model to focus mostly on contrastive learning task and its result is worse than pure maskthen-predict pre-training (i.e., α = 0 . 0 ).
Revised: Setting α = 2 .will let the model to focus mostly on contrastive learning task and its result is worse than pure maskthen-predict pre-training (i.e., α = 0 . 0 ).

Original: We also list the pure contrastive learning pre-training results here (denoted as α = 8 ˚
Revised: We also list the pure contrastive learning pre-training results here (denoted as α = 8 ˚

Original: but it excludes the mask-then-predict loss and set α = 1 . 0 ) for reference.
Revised: but it excludes the [Figure] Even under heavy masking (e.g., 45% masked), this simple reconstruction strategy still yields a reasonable results for i.i.d masking , e.g., we can easily recognize the action ‘petting cat’ from the reconstructed images, especially the one reconstructed from spatio-temporal neighbours. However, this becomes signiﬁcantly more difﬁcult when using block masking .

Original: G V ISUALIZATIONS G. T EMPORALLY -H EAVY VS . S PATIALLY -H EAVY D ATASETS
Revised: G V ISUALIZATIONS G. T EMPORALLY -H EAVY VS . S PATIALLY -H EAVY D ATASETS

Original: We illustrate the differences between temporally-heavy and spatially-heavy datasets in Fig.
Revised: We illustrate the differences between temporally-heavy and spatially-heavy datasets in Fig.

Original: We here show equally-distributed frames from the video and the label of the video clip.
Revised: We here show equally-distributed frames from the video and the label of the video clip.

Original: Note that we do not cherry-pick the data but aim for showing the nature of each dataset.
Revised: Note that we do not cherry-pick the data but aim for showing the nature of each dataset.

Original: Overall, understanding in temporally-heavy datasets needs temporal modeling, whereas the action labels of spatially-heavy datasets could be inferred from a single frame.
Revised: Overall, understanding in temporally-heavy datasets needs temporal modeling, whereas the action labels of spatially-heavy datasets could be inferred from a single frame.

Original: To understand the SSV2(Goyal et al., 2017) example in Fig. 3.(a), the model needs to understand the causality, i.e., the order of the frames decides the action label.
Revised: To understand the SSV2(Goyal et al., 2017) example in Fig. 3.(a), the model needs to understand the causality, i.e., the order of the frames decides the action label.

Original: In Fig. 3.(b), the competitive diving dataset Diving48 (Li et al., 2018) also requires considering nearly all frames to make the decision.
Revised: In Fig. 3.(b), the competitive diving dataset Diving48 (Li et al., 2018) also requires considering nearly all frames to make the decision.

Original: However, for the spatially-heavy datasets (UCF101 (Soomro et al., 2012b), HMDB51 (Kuehne et al., 2011b), Kinetics-400 (Carreira & Zisserman, 2017)), the action label could be inferred from any single sampled frame.
Revised: However, for the spatially-heavy datasets (UCF101 (Soomro et al., 2012b), HMDB51 (Kuehne et al., 2011b), Kinetics-400 (Carreira & Zisserman, 2017)), the action label could be inferred from any single sampled frame.

Original: These observations result in the pretty high frame-level accuracy (i.e., not modeling temporal interactions) in Sec. 4.3.
Revised: These observations result in the pretty high frame-level accuracy (i.e., not modeling temporal interactions) in Sec. 4.3.

Original: G. M ASKING S TRATEGY C OMPARISONS
Revised: G . M ASKING S TRATEGY C OMPARISONS

Original: We propose to use block masking (in Sec. 3.1) since i.i.d.
Revised: We propose to use block masking (in Sec. 3.1) since i.i.d.

Original: masking may lead to trivial solutions for the mask-then-predict task given the strong localities in videos.
Revised: masking may lead to trivial solutions for the mask-then-predict task given the strong localities in videos.

Original: We illustrate this point in Fig.with a simple copy-paste reconstruction method.
Revised: We illustrate this point in Fig.with a simple copy-paste reconstruction method.

Original: Speciﬁcally, after masking, we ﬁrst replace the masked tokens with their nearest visible neighbours (i.e., the unmasked token that has the shortest distance in spatial or spatio-temporal domain), and then forward the reconstructed tokens to the
Revised: Speciﬁcally, after masking, we ﬁrst replace the masked tokens with their nearest visible neighbours (i.e., the unmasked token that has the shortest distance in spatial or spatio-temporal domain), and then forward the reconstructed tokens to the

Original: VQ-VAE decoder to generate the RGB images.
Revised: VQ-VAE decoder to generate the RGB images.

Original: For the default 15% masking ratio, i.i.d.
Revised: For the default 15% masking ratio, i.i.d.

Original: masking is recoverable while block masking causes striped noisy patterns.
Revised: masking is recoverable while block masking causes striped noisy patterns.

Original: 4 We also test with the extreme case of masking 45% tokens (in Fig.
Revised: 5 We also test with the extreme case of masking 45% tokens (in Fig.

Original: 4 (d), (e)).
Revised: 4 (d), (e)).

Original: The block-masked images are hard to reconstruct, however, some objects in reconstructed images from i.i.d.
Revised: The block-masked images are hard to reconstruct, however, some objects in reconstructed images from i.i.d.

Original: masking are still recognizable.
Revised: masking are still recognizable.

Original: When comparing images under the same masking strategy, recovered images using spatio-temporal neighbours is better than using only spatial neighbours, especially when comparing the images under 45% [Figure] [Figure]
Revised: When comparing images under the same masking strategy, recovered images using spatio-temporal neighbours is better than using only spatial neighbours, especially when comparing the images under 45% i.i.d.

Original: Even under heavy masking (e. reconstruction strategy still yields a easily i.i.d.

Original: masking (i.e., (d).2 and (e).2 in Fig. 4).
Revised: masking (i.e., (d).2 and (e).2 in Fig. 4).

Original: Overall, these results indicate that using i.i.d.
Revised: Overall, these results indicate that using i.i.d.

Original: masking in mask-then-predict task has a potential trivial solution by copying the neighbourhood, while block-masking resolves this issue.
Revised: masking in mask-then-predict task has a potential trivial solution by copying the neighbourhood, while block-masking resolves this issue.

Original: [Figure]
Revised: [Figure]

Original: and 4., our could
Revised: and 4., our could

Original: G. M ODEL R ECONSTRUCTION
Revised: G. M ODEL R ECONSTRUCTION

Original: Since our model is trained with mask-then-predict task, it is able to reconstruct masked tokens.
Revised: Since our model is trained with mask-then-predict task, it is able to reconstruct masked tokens.

Original: In this section, we showcase the reconstructed video frames by our ﬁnal model (i.e., 24 layers,dimensions, 5 clip length, and 256 frame size).
Revised: In this section, we showcase the reconstructed video frames by our ﬁnal model (i.e., 24 layers,dimensions, 5 clip length, and 256 frame size).

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 5, we provide two examples from the SSV2 and Kinetics-400 dataset.
Revised: 5, we provide two examples from the SSV2 and Kinetics-400 dataset.

Original: We uniformly sample 5 consecutive frames from the video atframe per second.
Revised: We uniformly sample 5 consecutive frames from the video atframe per second.

Original: We show the original frames in the ﬁrst rows (Fig.
Revised: We show the original frames in the ﬁrst rows (Fig.

Original: 5.(a).1, (b).1).
Revised: 5.(a).1, (b).1).

Original: As illustrated in
Revised: As illustrated in

Original: Sec.
Revised: Sec.

Original: G.1, the temporally-heavy SSV2 dataset has object motions between frames while the spatiallyheavy Kinetics-400 dataset has almost static frames.
Revised: G.1, the temporally-heavy SSV2 dataset has object motions between frames while the spatiallyheavy Kinetics-400 dataset has almost static frames.

Original: In the second rows, we show the images after
Revised: In the second rows, we show the images after

Original: VQ-VAE compression.
Revised: VQ-VAE compression.

Original: To do this, we ﬁrst use VQ-VAE encoder to encode the images, and then use
Revised: To do this, we ﬁrst use VQ-VAE encoder to encode the images, and then use

Original: VQ-VAE decoder to reconstruct the images, without any corruptions in between.
Revised: VQ-VAE decoder to reconstruct the images, without any corruptions in between.

Original: We see that there is some information loss caused by the VQ-VAE compression (e.g., the text ‘comfort’ in Fig.
Revised: We see that there is some information loss caused by the VQ-VAE compression (e.g., the text ‘comfort’ in Fig.

Original: 5.(a).1).
Revised: 5.(a).1).

Original: It potentially contributes to relative lower results on spatially-heavy datasets.
Revised: It potentially contributes to relative lower results on spatially-heavy datasets.

Original: In the third and fourth rows, we illustrate the masked area and the prediction from our model.
Revised: In the third and fourth rows, we illustrate the masked area and the prediction from our model.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 5.(a).4, our model could faithfully redraw the shape and texture of the object.
Revised: 5.(a).4, our model could faithfully redraw the shape and texture of the object.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 5.(b).4, the shape of the cat’s head is pretty similar to the original frames while the shading is different (but still consistent in different frames).
Revised: 5.(b).4, the shape of the cat’s head is pretty similar to the original frames while the shading is different (but still consistent in different frames).