VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training
Anonymous Author(s)
AffiliationAddress email
Abstract
Pre-training video transformers on extra large-scale datasets is generally requiredto achieve premier performance on relatively small datasets.
In this paper, weshow that video masked autoencoders (VideoMAE) are
data-efficient learners forself-supervised video pre-training (SSVP).
We are inspired by the recent ImageMAE [22] and propose customized video tube masking with an extremely highratio.
This simple design makes video reconstruction a more challenging andmeaningful self-supervision task, thus encouraging extracting more effective videorepresentations during this pre-training process.
We obtain three important findingson SSVP:
(1) An extremely high proportion of masking ratio (i.e., 90% to 95% )still yields favorable performance of VideoMAE.
The temporally redundant videocontent enables higher masking ratio than that of images.
(2) VideoMAE achievesimpressive results on very small datasets (i.e., around 3k-4k videos) without usingany extra data.
This is partially ascribed to the challenging task of video reconstruction to enforce high-level structure learning.
(3) VideoMAE shows that data qualityis more important than data quantity for SSVP.
Domain shift between pre-trainingand target datasets is an important issue.
Notably, our VideoMAE with the vanilla
ViT backbone can achieve 84.7% on Kinects-400, 75.3% on Something-Something
V2, 90.8% on UCF101, and 61.1% on HMDB51, without using any extra data.
Introduction
Transformer [45] has brought significant progress in natural language processing [12, 5, 34].
Thevision transformer [14] also improves a series of computer vision tasks including image classification [42, 59], object detection [6, 25], semantic segmentation [54], object tracking [11] and videorecognition [
The multi-head self-attention upon linearly projected image/video tokens iscapable of modeling global dependency among visual content either spatially or temporally.
Theinductive bias is effectively reduced via this flexible attention mechanism.
Training effective vision transformers (ViTs) typically necessitates large-scale supervised datasets.
Initially, the pre-trained ViTs achieve favorable performance by using hundreds of millions of labeledimages [14].
For video transformers [1, 4], they are usually derived from image-based transformersand heavily depend on the pre-trained models from large-scale image data (e.g., ImageNet [37]).
Previous trials [1, 4] on training video transformers from scratch yield unsatisfied results (exceptfor MViT [15] with a strong inductive bias).
Therefore, the learned video transformers are naturallybiased by image-based models, and it still remains a challenge that how to effectively and efficientlytrain a vanilla vision transformer on the video dataset itself without using any pre-trained modelor extra image data.
Moreover, the existing video datasets are relatively small compared withimage datasets, which further increases the difficulty of training video transformers from scratch.
Meanwhile, self-supervised learning has shown remarkable performance by using large-scale image
[Figure]
datasets [10, 7].
The learned representations have outperformed the ones via supervised learningwhen being transferred to downstream tasks.
It is expected that this self-supervised learning paradigmcan provide a promising solution to address the challenge of training video transformers.
Following the success of masked autoencoding in NLP [12] and images [22, 2], we present a new selfsupervised video pre-training (SSVP) method, termed as Video Masked Autoencoder (VideoMAE).
Our VideoMAE inherits the simple pipeline of masking random cubes and reconstructing the missingones.
However, the extra time dimension of videos makes them different from images in this maskedmodeling.
First, video frames are often densely captured, and their semantics varies slowly intime [58].
This temporal redundancy would increase the risk of recovering missing pixels fromthe spatiotemporal neighborhood with little high-level understanding.
Furthermore, video could beviewed as the temporal evolution of static appearance, and there exists a correspondence betweenframes.
This temporal correlation could lead to information leakage (i.e., masked spatiotemporalcontent re-occurrence) during reconstruction unless a specific masking strategy is considered.
Inthis sense, for each masked cube, it is easy to find a corresponding and unmasked copy in adjacentframes.
This property would make the learned models identify some “shortcut” features that are hardto generalize to new scenarios.
To make video masked modeling more effective, in this paper, we present a customized design of tubemasking with an extremely high ratio in our VideoMAE.
First, due to temporal redundancy, we usean extremely high masking ratio to drop the cubes from the downsampled clips.
This simple strategynot only effectively increases the pre-training performance but also greatly reduces the computationalcost due to the asymmetric encoder-decoder architecture.
Second, to consider temporal correlation,we devise a simple yet effective tube masking strategy, which turns out to be helpful in relieving therisk of information leakage for cubes with no or negligible motion during reconstruction.
With thissimple yet effective design in our VideoMAE, we are able to successfully train vanilla ViT backboneson the relatively small-scale video datasets such as Something-Something [19], UCF101 [39], and
HMDB51 [24], which significantly outperform the previous state of the art under the setting withoutextra data.
In summary, the main contribution of this paper is threefold:
• We present a simple but effective video masked autoencoder that unleashes the potential of
vanilla vision transformer for video analysis and performs well for SSVP on relatively smallscale video datasets.
To relieve the information leakage issue in the video reconstruction task, we present a simple design of tube masking with an extremely high ratio, which turns
out to be effective in improving the performance of VideoMAE.
• Aligned with the results in NLP and Images on masked modeling, our VideoMAE demon-
strates that this simple masking and reconstruction strategy provides a good solution toself-supervised video pre-training.
The models pre-trained with our VideoMAE significantly outperform those trained from scratch or pre-trained with contrastive learning methods.
• We obtain extra important finds on masked modeling that might be ignored in the previous research in NLP and Images.
(1) Our results demonstrate that VideoMAE is a data-efficient
learner that could be successfully trained with only 3.5k videos.
(2) VideoMAE shows that data quality is more important than quantity for SSVP when a domain shift exists between the source and target dataset.
Related Works
Video representation learning.
Learning good video representations has been heavily investigatedin the literature.
The supervised learning methods [49, 44, 17, 16, 4] usually depend on the imagebackbones.
The video encoder backbones are first pre-trained with image data in a supervisedform.
Then, these backbones are fine-tuned on the video dataset for classifying human actions.
Besides supervised learning, semi-supervised video representation learning is also studied [38].
The representations of labeled training samples are utilized to generate supervision signals forunlabeled ones.
Supervised or semi-supervised representation learning mainly uses a top-downtraining paradigm, which is not effective in exploring the inherent video data structure itself.
For self-supervised learning, the prior knowledge of temporal information has been widely exploitedto design pretext tasks [52, 28, 56, 3] for SSVP.
Recently, contrastive learning [20, 29, 21, 33] ispopular to learn better video representation.
However these methods heavily rely on strong dataaugmentation and large batch size [18].
Predicting the video clip with autoencoders in pixel spacehas been explored for representation learning [31, 40] by using CNN or LSTM backbones or videogeneration with autoregressive GPT [57].
Our VideoMAE aims to use the simple masked autoencoderwith recent ViT backbones to perform data-efficient SSVP.
Masked visual modeling.
Masked visual autoencoder has been proposed to learn effective visualrepresentations based on the simple pipeline of masking and reconstruction.
These works mainlyfocus on the image domain.
The early work [47] treated the masking and a noise type in denoisedautoencoders [46] or inpainted missing regions with context [30] by using convolutions.
Recently,masked visual autoencoder used Transformer backbones for masking and reconstruction.
iGPT [9]followed the success of GPT [5, 35] in NLP and operated a sequence of pixels for prediction.
Theoriginal ViT [14] investigated the masked token prediction for self-supervised pre-training.
Morerecently, the success of vision transformer has led to investigation of Transformer-based architecturesfor masked visual modeling [2, 13, 22, 53, 55, 60].
BEiT [2], BEVT [50] and VIMPAC [41] followed BERT [12] and proposed to learn visual representations from images and videos by predicting thediscrete tokens [36].
ImageMAE [22] introduced an asymmetric encoder-decoder architecture formasked image modeling.
MaskFeat [53] proposed to reconstruct the features of masked tokens toperform self-supervised pre-training in videos.
VideoMAE is inspired by the ImageMAE and introduces specific design in implementation for SSVP.
Our work differs from these video masked modeling methods in several aspects.
1) Prediction target .
Our VideoMAE performs pre-training by directly predicting the normalized cube pixels. BEVT and VIMPAC predict the discrete tokens from pre-trained VQ-VAE. MaskFeat predict the HOG features.2) Computational efficiency .
Our VideoMAE only needs to feed the unmasked tokens (e.g., 10%)into the encoder in self-supervised learning.
BEVT, VIMPAC and MaskFeat put all tokens into theencoder during the pre-training phase.
3) Backbone .
Our VideoMAE uses the plain ViT backbone,while the Swin and MViT in BEVT and MaskFeat are with multi-scale and hierarchical architectures.
VIMPAC uses a pre-trained VQ-VAE to discrete the input video into discrete video tokens and adopta sparse transformer on top of the tokens.
4) Prediction head .
Our VideoMAE generates the targetby a shallow Transformer decoder, while BEVT and MaskFeat all use two-layer MLP as the targethead.
VIMPAC performs pre-training by predicting the discrete tokens from pre-trained VQ-VAEand introducing a contrastive learning head.
Overall, our VideoMAE enjoys a simpler design for thewhole pipeline, obtaining higher efficiency and slightly better accuracy.
Proposed Method
In this section, we first revisit ImageMAE [22].
Then we analyze the characteristics of video data.
Finally, we show how we explore MAE in the video data by presenting our VideoMAE.
3.1 Revisiting Masked Autoencoders
ImageMAE [22] performs the masking and reconstruction task with an asymmetric encoder-decoderarchitecture.
The input image I ∈ R 3 × H × W is first divided into regular non-overlapping patchesof size 16 × 16, and each patch is represented with token embedding .
Then a subset of tokens arerandomly masked with a high masking ratio (75%), and only the remaining ones are fed into thetransformer encoder Φ enc .
Finally, a shallow decoder Φ dec is placed on top of the visible tokens from the encoder and learnable mask tokens to reconstruct the image.
The loss function is mean squarederror (MSE) loss between the normalized masked tokens and reconstructed ones in the pixel space:
[Equation]
where p is the token index, Ω is the set of masked tokens, I is the input image, and ˆ I is thereconstructed one.
3.2 Characteristics of Video Data
Compared with static images, video data contain temporal relations.
We show the motivation of our
VideoMAE by analyzing video characteristics.
Temporal redundancy.
There are frequently captured frames in a video.
The semantics vary slowlyin the temporal dimension [58].
We observe that consecutive frames are highly redundant, as shownin Figure 2.
This property leads to two critical issues in masked video autoencoding.
First, it wouldbe less efficient to keep the original temporal frame rate for pre-training.
This would draw us to focusmore on static or slow motions in our masked modeling.
Second, temporal redundancy greatly dilutesmotion representations.
This would make the task of reconstructing missing pixels not difficult underthe normal masking ratio (e.g., 50% to 75%).
The encoder backbone is not effective in capturingmotion representations.
Temporal correlation.
Videos could be viewed as the temporal extension of static appearance, andtherefore there exists an inherent correspondence between adjacent frames.
This temporal correlationcould increase the risk of information leakage in the masking and reconstruction pipeline.
In thissense, as shown in Figure 2, we can reconstruct the masked patches by finding the spatiotemporalcorresponding unmasked patches in the adjacent frames under plain random masking or framemasking.
In this case, it might guide the VideoMAE to learn low-level temporal correspondencerather than high-level information such as spatiotemporal reasoning over the content.
To alleviate thisbehavior, we need to propose a new masking strategy to make the reconstruction more challengingand encourage effective learning of spatiotemporal structure representations.
3.3 VideoMAE
To relieve the above issues in video masked modeling, we make the customized design in our
VideoMAE, and the overall pipeline is shown in Figure 1.
Our VideoMAE takes the downsampledframes as inputs and uses the cube embedding to obtain video tokens.
Then, we propose a simpledesign of tube masking with high ratio to perform MAE pre-training with an asymmetric encoderdecoder architecture.
Our backbone uses the vanilla ViT with joint space-time attention .
Temporal downsampling.
According to the above analysis on temporal redundancy over consecutiveframes, we propose to use the strided temporal sampling strategy to perform more efficient videopre-training.
Formally, one video clip consisting of t consecutive frames is first randomly sampledfrom the original video V .
We then use temporal sampling to compress the clip to T frames, eachof which contains H × W × 3 pixels.
In experiments, the stride τ is set to 4 and 2 on Kinetics and
Something-Something, respectively.
Cube embedding.
We adopt the joint space-time cube embedding [1, 15, 26] in our VideoMAE,where we treat each cube of size 2 × 16 × 16 as one token embedding.
Thus, the cube embeddinglayer obtains T 2 × H 16 × W 16 3D tokens and maps each token to the channel dimension D .
This designcan decrease the spatial and temporal dimension of input, which helps to alleviate the spatiotemporalredundancy in videos.
Tube masking with extremely high ratio.
First, temporal redundancy is a factor affecting VideoMAEdesign.
We find that VideoMAE is in favor of extremely high masking ratios (e.g. 90% to 95%)compared with the ImageMAE.
Video information density is much lower than images, and we expecta high ratio to increase the reconstruction difficulty.
This high masking ratio is helpful to mitigate theinformation leakage during masked modeling and make masked video reconstruction a meaningfulself-supervised pre-training task.
Second, temporal correlation is another factor in our VideoMAE design.
We find even under theextremely high masking ratio, we can still improve the masking efficiency by proposing the temporal
[Figure]
tube masking mechanism.
Temporal tube masking enforces a mask to expand over the wholetemporal axis, namely, different frames sharing the same masking map.
Mathematically, the tubemask mechanism can be expressed as I [ p x,y, · ∈ Ω] ∼ Bernoulli( ρ mask ) and different time t sharesthe same value.
With this mechanism, temporal neighbors of masked cubes are always masked.
Sofor some cubes with no or small motion (e.g., finger cube in 4th row of Figure 2 (d)), we can notfind the spatiotemporal corresponding content in all frames.
In this way, it would encourage our
VideoMAE to reason over high-level semantics to recover these totally missing cubes.
This simplestrategy can alleviate the information leakage for cubes with no or negligible motion and turns out tobe effective in practice for masked video pre-training.
Backbone: joint space-time attention.
Due to the high proportion of masking ratio mentioned above,only a few tokens are left as the input for the encoder.
To better capture high-level spatio-temporalinformation in the remaining tokens, we use the vanilla ViT backbone [14] and adopt the jointspace-time attention [1, 26].
Thus, all pair tokens could interact with each other in the multi-headself-attention layer [45].
The specific architecture design for the encoder and decoder is shown insupplementary materials.
The quadratic complexity of the joint space-time attention mechanism is acomputational bottleneck, while our design of an extremely high masking ratio alleviates this issueby only putting the unmasked tokens (e.g., 10%) into the encoder during the pre-training phase.
Experiments
4.1 Datasets
We evaluate our VideoMAE on four canonical video recognition datasets: Kinetics-400 [23],
Something-Something V2 [19], UCF101 [39], and HMDB51 [24].
The Kinetics-400 contains around240k training videos and 20k validation videos of 10s from 400 classes.
The Something-Something
V2 is a relatively large-scale video dataset, having around 169k videos for training and 20k videos forvalidation.
In contrast to Kinetics-400, this dataset contains 174 motion-centric action classes.
Thesetwo large-scale video datasets focus on different visual cues for action recognition.
UCF101 and
HMDB51 are two relatively small and old video datasets, which contain around 9.5k/3.5k train/valvideos and 3.5k/1.5k train/val videos, respectively.
Compared with those large-scale video datasets,these two small datasets are more suitable for verifying the effectiveness of VideoMAE, as traininglarge ViT models is more challenging on small datasets.
In experiments, we strictly follow the officialsetting on these datasets by training VideoMAE on the training set and reporting the top-1 accuracyon the validation set.
The implementation details are described in supplementary materials.
4.2 Ablation Studies
In this subsection, we perform in-depth ablation studies of VideoMAE with the default backbone16-frame vanilla ViT-B on Something-Something V2.
The specific architecture design for the encoder
[Table]
Decoder design.
The lightweight decoder is one key component of our VideoMAE.
We conductexperiments with the different depths in Table 1a.
Unlike in ImageMAE, a deep decoder here is important for better performance, while a shallow decoder could reduce the GPU memory consumption.
We take 4 blocks for the decoder by default.
The decoder width is set to half channel of the encoder(e.g.
, 384-d for ViT-B), following the empirical design in the image domain.
Masking strategy.
We compare different masking strategies in Table 1b.
When increasing themasking ratio from 75% to 90% for tube masking, the performance boosts from 67.3% to 69.3%.
Then, with an extremely high ratio, we find tube masking also achieves better performance than plainrandom masking and frame masking.
We attribute these interesting observations to the redundancyand temporal correlation in videos.
We argue that our default designs enforce the networks to capturemore useful spatiotemporal structures and therefore make VideoMAE a more challenging task, whicha good self-supervised learner hunger for.
Reconstruction target.
First, if we only employ the center frame as the target, the results woulddecrease greatly.
The sampling stride is also sensitive.
The result of consecutive 16 frames issignificantly lower than our default downsampled clip (53.2% vs. 69.3%).
We also try to reconstructthe consecutive 32 frames from the downsampled 16 frames, but it obtains slightly worse results.
Pre-training strategy.
We compare different pre-training strategies in Table 1d.
Similar to previoustrials [1, 4], training video transformers from scratch yields unsatisfied results on this motion-sensitivedataset.
When pre-trained on the large-scale ImageNet-21K dataset, the video transformer obtainsbetter accuracy from 32.6% to 61.4%.
Using the models pre-trained on both ImageNet-21K and
Kinetics further increases accuracy to 64.8%.
Our VideoMAE can effectively train a video transformeron the video dataset itself without using any extra data and achieve the best performance of 69.3%.
Pre-training dataset.
First, we pre-train the ViT-B on ImageNet-1K for 1600 epochs, followingthe recipes in [22].
Then we inflate the 2D patch embedding layer to our cube embedding layerfollowing [8] and fine-tune the model on the Something-Something dataset.
This result surpassesthe model trained from scratch as shown in Table 1e.
We also compare this ImageMAE pre-trainedmodel with the VideoMAE model pre-trained on Kinetics.
We see that our VideoMAE model canachieve better performance than ImageMAE.
However, these two pre-trained models fail to yield
[Table] [Table]
[Table]
better performance than our VideoMAE only pre-trained on the Something-Something dataset.
Weargue that domain shift between pre-training and target datasets could be an important issue.
Loss function.
Table 1f contains an ablation study of loss function.
We find that the MSE loss couldachieve a higher result compared with the L1 loss and smooth L1 loss.
Therefore, we employ the
MSE loss by default.
4.3 Main Results and Analysis
VideoMAE: data-efficient learner.
The self-supervised video pre-training (SSVP) has been extensively studied in previous works, but they mainly use the CNN-based backbones.
Few works haveinvestigated transformer-based backbone in SSVP.
Therefore, to demonstrate the effectiveness of
VideoMAE for transformer-based SSVP, we compare two methods implemented by ourselves: (1)training from scratch and (2) pre-training with contrastive learning (MoCo v3 [10]).
For trainingfrom scratch, we carefully tune these hyper-parameters to successfully pre-train ViT-Base from thetraining set of the dataset.
For pre-training with MoCo v3, we strictly follow the training practice inits image counterpart and carefully avoid the collapse issue.
The recognition accuracy is reported in Table 2.
We see that our VideoMAE significantly outperformsthe other two training methods.
For instance, on the largest dataset of Kinetics-400, our VideoMAEoutperforms training from scratch by around 10% and MoCo v3 pre-training by around 5%.
Thissuperior performance demonstrates that masked autoencoder provides an effective pre-trainingmechanism for video transformers.
We also see that the performance gap between our VideoMAEand the other two methods becomes larger as the training set becomes smaller.
Notably, even with only3.5k training clips on HMDB51, our VideoMAE pre-training can still obtain a satisfying accuracy(around 61%).
This new result demonstrates that VideoMAE is a more data-efficient learner for SSVP.
This property is particularly important for scenarios with limited data available and different withcontrastive learning methods.
We compare the efficiency of VideoMAE pre-training and MoCo v3 pre-training in Table 3.
The taskof masked autoencoding with a high ratio is more challenging and thereby requires more trainingepochs (800 vs. 300).
Thanks to the asymmetric encoder-decoder in our VideoMAE and extremelyhigh masking ratio, our pre-training time is much shorter than MoCo v3 (19.5 vs. 61.7 hours).
High masking ratio.
In VideoMAE, one core design is the extremely high masking ratio.
Weperform an investigation of this design on the Kinetics-400 and Something-Something V2 datasets.
The results are shown in Figure 3.
We see that the best masking ratio is extremely high, and even
[Figure]
achieves
[Figure]
Our default backbone is 16-frame vanilla ViT-B. • denotes that all models are trained for the same 132k iterations, and • denotes that all models are trained for the sameepochs.
Note that it takes 132k iterations to pre-train the model for 800 epochs on the full training set of SomethingSomething V2.
[Figure]
Extra data
Ex.
labels Frames GFLOPs Param Top-1 TopSlowFast [17]
ResNet
Kinetics-
✓ 8+ 106 × 1 × 63.
MViTv1 [15]
MViTv1-B ✓ 455 × 1 × 67.
TimeSformer [4]
ViT-B
ImageNet-21K ✓ 196 × 1 × 59.
N/A
TimeSformer [4]
ViT-L ✓ 5549 × 1 × 62.
N/A
ViViT FE [1]
ViT-L
IN-21K+K ✓ 995 × 4 ×
N/A 65.
Motionformer [32]
ViT-B ✓ 370 × 1 × 66.
Motionformer [32]
ViT-L ✓ 1185 × 1 × 68.
Video Swin [26]
Swin-B ✓ 321 × 1 × 69.
VIMPAC [41]
ViT-L HowTo100M+DALLE ✗ N/A × 10 × 68. N/A
BEVT [50]
Swin-B
IN-1K+
[Table]
+DALLE ✗ 321 × 1 × 70.
N/A
MaskFeat ↑ 312 [53]
MViT-L
Kinetics- ✓ 2828 × 1 × 75.
VideoMAE ρ =90%
ViT-B Kinetics- ✗ 180 × 2 × 69.
VideoMAE ρ =90% ViT-L
Kinetics- ✗ 597 × 2 × 74. VideoMAE ρ =90% ViT-L Kinetics- ✗ 597 ×
[Table]
95% can achieve good performance for both datasets.
This result is difference from BERT [12] in
NLP and MAE [22] in images.
We analyze the temporal redundancy and correlation in videos makesit possible for our VideoMAE to learn plausible outputs with such a high masking ratio.
We also visualize the reconstructed examples in supplementary materials.
We see that even under anextremely high masking ratio, VideoMAE can produce satisfying reconstructed results.
This impliesthat our VideoMAE is able to learn useful representations that capture the holistic spatiotemporalstructure in videos.
Transfer learning: quality vs. quantity.
To further investigate the generalization ability of
VideoMAE in representation learning, we transfer the learned VideoMAE from Kinetics-400 to
Something-Something V2, UCF101, and HMDB51.
The results are shown in Table 4, and wecompare them with the MoCo v3 pre-training method.
The models pre-trained by our VideoMAE arebetter than those pre-trained by MoCo v3, demonstrating that our VideoMAE learns more transferablerepresentations.
[Table]
[Table]
Comparing Table 2 and Table 4, the transferred representation outperforms the original VideoMAEmodels trained from its own dataset on UCF101 and HMDB51.
In contrast, the transferred representation is worse on Something-Something V2.
To figure out whether this inconsistent result iscaused by the large scale of Something-Something V2, we further perform a detailed investigation bydecreasing the pre-training video numbers.
In this study, we run two experiments: (1) pre-trainingwith the same epochs and (2) pre-training with the same time budget.
The result is shown in Figure 4.
We see that more training iterations could contribute to better performance when we decrease the sizeof the pre-training set.
Surprisingly, even with only 42k pre-training videos, we can still obtain betteraccuracy than the K400 pre-trained models with 240k videos (68.5% vs. 68.2%).
This result impliesthat domain shift is another important factor, and data quality is more important than data quantity in
SSVP when there exists a difference between pre-training and target datasets.
It also demonstratesthat VideoMAE is a data-efficient learner for SSVP.
Comparison with the state of the art.
We compare with the previous state-of-the-art performanceon the Kinetics-400 and Something-Something V2 datasets.
The results are reported in Table 5 and
Table 6.
Our VideoMAE can easily scale up with more powerful backbones (ViT-Large) and moreframes (32).
Our VideoMAE achieves the top-1 accuracy of 75.3% on the Something-Something V2and 84.7% on the Kinetics-400 without using any extra data.
We see that the existing state-of-the-artmethods all depend on the external data for pre-training on the Something-Something V2.
On thecontrary, our VideoMAE without any external data significantly outperforms previous methods withthe same input resolution by around 5%.
Our ViT-L VideoMAE also achieves very competitiveperformance on the Kinetics-400 without using any extra data, which is slightly better than the ViViT
FE-L with on JFT-300M pre-training.
Conclusion
In this paper, we have presented a simple and data-efficient self-supervised learning method (VideoMAE) for video transformer pre-training.
Our VideoMAE introduces two critical designs of extremelyhigh masking ratio and tube masking strategy to make the video reconstruction task more challenging.
This harder task would encourage VideoMAE to learn more representative features and relieve theinformation leakage issue.
Empirical results demonstrate this simple algorithm works well for videodatasets of different scales.
In particular, we are able to learn effective VideoMAE only with thousandsof video clips, which has significant practical value for scenarios with limited data available.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes
(b) Did you describe the limitations of your work?
[Yes] Shown in supplementary materi-
als.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Shown in
supplementary materials.(d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] Shown insupplementary materials.(b)
Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] Shown in supplementary materials.
(c) Did you report error bars (e.g., with respect to the random seed after running exper-
iments multiple times)?
[No] Because of the computation costs, we did not run the
experiments multiple times.
(d) Did you include the total amount of compute and the type of resources used (e.g.,
type of GPUs, internal cluster, or cloud provider)?
[Yes] Please see Table 3 and
supplementary materials.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] Shown in supplementary materials.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] We used publicly available datasets whose licenses allow research
usage.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No] To the best of our knowledge, the data we used
contains no personally identifiable information or offensive content.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]