Original: D EEP D YNAMIC A UTO E NCODER FOR V ISION BERT P RETRAINING
Revised: D EEP D YNAMIC A UTO E NCODER FOR V ISION BERT P RETRAINING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recently, masked image modeling (MIM) has demonstrated promising prospects in self-supervised representation learning.
Revised: Recently, masked image modeling (MIM) has demonstrated promising prospects in self-supervised representation learning.

Original: However, existing MIM frameworks recover all masked patches equivalently, ignoring that the reconstruction difﬁculty of different patches can vary sharply due to their diverse distance from visible patches.
Revised: However, existing MIM frameworks recover all masked patches equivalently, ignoring that the reconstruction difﬁculty of different patches can vary sharply due to their diverse distance from visible patches.

Original: In this paper, we propose Deep Dynamic AutoEncoder (DDAE), a novel
Revised: In this paper, we propose Deep Dynamic AutoEncoder (DDAE), a novel

Original: MIM framework that dynamically focuses on patch reconstructions with different degrees of difﬁculty at different pretraining phases and depths of the model.
Revised: MIM framework that dynamically focuses on patch reconstructions with different degrees of difﬁculty at different pretraining phases and depths of the model.

Original: In addition to raw pixel regression, DDAE performs dynamic feature self-distillation for intermediate layers to learn semantic information.
Revised: In addition to raw pixel regression, DDAE performs dynamic feature self-distillation for intermediate layers to learn semantic information.

Original: Our methodology provides more locality inductive bias for ViTs, especially in deep layers, which inherently makes up for the absence of local prior for self-attention mechanism.
Revised: Our methodology provides more locality inductive bias for ViTs, especially in deep layers, which inherently makes up for the absence of local prior for self-attention mechanism.

Original: Moreover, our core design deep dynamic supervision can be migrated into existing MIM methods ( e.g. , MAE, BEiT-v2) seamlessly.
Revised: Moreover, our core design deep dynamic supervision can be migrated into existing MIM methods ( e.g. , MAE, BEiT-v2) seamlessly.

Original: The Experimental results demonstrate the effectiveness of our approach.
Revised: The Experimental results demonstrate the effectiveness of our approach.

Original: As a tokenizer-free framework, the base-size
Revised: As a tokenizer-free framework, the base-size

Original: DDAE can achieve 83.5% top-1 accuracy with only 100 epochs pretraining, surpassing MAE and BEiT pretrained for 800 epochs.
Revised: DDAE can achieve 83.5% top-1 accuracy with only 100 epochs pretraining, surpassing MAE and BEiT pretrained for 800 epochs.

Original: For a longer pretraining schedule, DDAE achieves 84.3% top-1 accuracy on Imagenet-1K, and 49.3% mIoU on
Revised: For a longer pretraining schedule, DDAE achieves 84.3% top-1 accuracy on Imagenet-1K, and 49.3% mIoU on

Original: ADE20K for semantic segmentation.
Revised: ADE20K for semantic segmentation.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Aided by the rapid gains in hardware, deep learning has ushered in the era of big models and big data.
Revised: Aided by the rapid gains in hardware, deep learning has ushered in the era of big models and big data.

Original: Along with the ever-growing model capacity, the demand for data can easily reach hundreds of millions (Dosovitskiy et al., 2020), which is not publicly accessible for labeled data.
Revised: Along with the ever-growing model capacity, the demand for data can easily reach hundreds of millions (Dosovitskiy et al., 2020), which is not publicly accessible for labeled data.

Original: Self-Supervised
Revised: Self-Supervised

Original: Learning(SSL) frameworks, such as DINO (Caron et al., 2021), MOCO (Chen et al., 2021), BeiT (Bao et al., 2021), etc., have grown in concern in vision model pretraining without the need for labels.
Revised: Learning(SSL) frameworks, such as DINO (Caron et al., 2021), MOCO (Chen et al., 2021), BeiT (Bao et al., 2021), etc., have grown in concern in vision model pretraining without the need for labels.

Original: In particular, the recently proposed Masked Image Modeling(MIM) methods (He et al., 2022; Xie et al., 2022b; Dong et al., 2021; Wei et al., 2022b; Chen et al., 2022b; Dong et al., 2022; Chen et al., 2022c) have shown remarkably impressive performance in a variety of vision tasks, demonstrating the promise to unify computer vision and natural language processing(NLP)
Revised: In particular, the recently proposed Masked Image Modeling(MIM) methods (He et al., 2022; Xie et al., 2022b; Dong et al., 2021; Wei et al., 2022b; Chen et al., 2022b; Dong et al., 2022; Chen et al., 2022c) have shown remarkably impressive performance in a variety of vision tasks, demonstrating the promise to unify computer vision and natural language processing(NLP)

Original: pretraining (Peng et al., 2022a).
Revised: pretraining (Peng et al., 2022a).

Original: Inspired by BERT (Devlin et al., 2018) in NLP, MIM pretrains the encoder by reconstructing the masked image patches from visible patches.
Revised: Inspired by BERT (Devlin et al., 2018) in NLP, MIM pretrains the encoder by reconstructing the masked image patches from visible patches.

Original: Existing MIM methods can be divided into two categories according to the need for an additional tokenizer.
Revised: Existing MIM methods can be divided into two categories according to the need for an additional tokenizer.

Original: Two-stage methods: Represented by the pioneer work BEiT
Revised: Two-stage methods: Represented by the pioneer work BEiT

Original: (Bao et al., 2021), two-stage methods ﬁrstly transform image patches into semantic visual tokens through a pre-trained tokenizer, then reconstruct the corresponding tokens of masked image patches to pretrain the encoder.
Revised: (Bao et al., 2021), two-stage methods ﬁrstly transform image patches into semantic visual tokens through a pre-trained tokenizer, then reconstruct the corresponding tokens of masked image patches to pretrain the encoder.

Original: Tokenizer needs to be ofﬂine pretrained with ﬁxed model architectures and extra data (Zhang et al., 2019b;
Revised: Tokenizer needs to be ofﬂine pretrained with ﬁxed model architectures and extra data (Zhang et al., 2019b;

Original: Ramesh et al., 2021; Radford et al., 2021), some methods further require an off-the-shelf DNN as teacher to distill tokenizer pre-training (Peng et al., 2022a).
Revised: Ramesh et al., 2021; Radford et al., 2021), some methods further require an off-the-shelf DNN as teacher to distill tokenizer pre-training (Peng et al., 2022a).

Original: One-stage methods: Taking the recent work MAE
Revised: One-stage methods: Taking the recent work MAE

Original: (He et al., 2022) as a representative, MAE constructs an asymmetric encoder-decoder structure, and directly performs raw pixel regression of masked image patches.
Revised: (He et al., 2022) as a representative, MAE constructs an asymmetric encoder-decoder structure, and directly performs raw pixel regression of masked image patches.

Original: One-stage methods are tokenizer-free, the reconstruction target includes raw pixels, features ( e.g. , HOG) and self-distillation targets (He et al., 2022; Xie et al., 2022b;
Revised: One-stage methods are tokenizer-free, the reconstruction target includes raw pixels, features ( e.g. , HOG) and self-distillation targets (He et al., 2022; Xie et al., 2022b;

Original: Wei et al., 2022a; Chen et al., 2022c).
Revised: Wei et al., 2022a; Chen et al., 2022c).

Original: MIM methods enable Vision Transformers(ViTs)to learn rich visual representations and show great potential in various downstream tasks.
Revised: MIM methods enable Vision Transformers(ViTs)to learn rich visual representations and show great potential in various downstream tasks.

Original: Naturally, we ask why MIM works.
Revised: Naturally, we ask why MIM works.

Original: Facilitated by self-attention mechanism, ViTs excel at modeling long-range dependencies, but different from CNN, ViTs’ lack of local prior( i.e. , a pixel is more related to its neighbors than the distant pixels) largely leads to its need for large dataset pretraining ( e.g. , JFT-300M) or special training techniques ( e.g. , a DeiT-style distillation method (Touvron et al., 2021b)).
Revised: Facilitated by self-attention mechanism, ViTs excel at modeling long-range dependencies, but different from CNN, ViTs’ lack of local prior( i.e. , a pixel is more related to its neighbors than the distant pixels) largely leads to its need for large dataset pretraining ( e.g. , JFT-300M) or special training techniques ( e.g. , a DeiT-style distillation method (Touvron et al., 2021b)).

Original: Therefore, we conjecture that the success of MIM comes from the ﬁne-grained reconstruction of masked image patches.
Revised: Therefore, we conjecture that the success of MIM comes from the ﬁne-grained reconstruction of masked image patches.

Original: The generation task built by MIM enables ViTs to pay more attention to vicinity besides using global semantic reasoning.
Revised: The generation task built by MIM enables ViTs to pay more attention to vicinity besides using global semantic reasoning.

Original: Thus, it inherently makes up for the absence of local prior in the structure of ViTs, which is essentially different from the discriminative task in supervised pretraining or contrastive learning.
Revised: Thus, it inherently makes up for the absence of local prior in the structure of ViTs, which is essentially different from the discriminative task in supervised pretraining or contrastive learning.

Original: As analyzed in Xie et al.

Original: (2022a), compared with traditional supervised pretraining and contrastive learning, MIM makes the average attention distance of ViTs closer, especially in deeper blocks.

Original: This observation partially supports our conjecture that it is beneﬁcial to inject local priors into ViT through pretraining.

Original: However, success comes with remaining obstacles.
Revised: However, success comes with remaining obstacles.

Original: At present, all MIM frameworks recover all patches equivalently, ignoring the fact that the reconstruction difﬁculty of different patches can vary sharply, and the semantic reasoning and local perception ability required for patches are not the same.
Revised: At present, all MIM frameworks recover all patches equivalently, ignoring the fact that the reconstruction difﬁculty of different patches can vary sharply, and the semantic reasoning and local perception ability required for patches are not the same.

Original: Intuitively, recovering patches with more visible patches around will be simpler, as long as the model has sufﬁcient local perception ability.
Revised: Generally, recovering patches with more visible patches around will be simpler, as long as the model has sufﬁcient local perception ability.

Original: In contrast, reconstruction with few visible patches around requires the model to have strong semantic reasoning ability, given that there is little access to neighboring information.
Revised: In contrast, reconstruction with few visible patches around requires the model to have strong semantic reasoning ability, given that there is little access to neighboring information.

Original: Treating all pixels equally will neglect this demand for different properties of the model, inevitably limiting its representation ability.
Revised: Treating all pixels equally will neglect this demand for different properties of the model, inevitably limiting its representation ability.

Original: On the other hand, layers at differentdepths and training phases naturally learn features at different levels.
Revised: On the other hand, layers at different training phases naturally learn features at different levels.

Original: Therefore, we ask if there is a way to focus on objectives with diverse characteristics as training progresses so that better representations can be learned overall.
Revised: Therefore, we ask if there is a way to focus on objectives with diverse characteristics as training progresses so that better representations can be learned overall.

Original: Motivated by this observation and answering the question above, we propose a simple yet effective
Revised: Motivated by this observation and answering the question above, we propose a simple yet effective

Original: SSL framework named D eep D ynamic A uto E ncoder (DDAE).
Revised: SSL framework named D eep D ynamic A uto E ncoder (DDAE).

Original: With deep dynamic supervision, the model dynamically focuses on different patches at different training phases.
Revised: With deep dynamic supervision, the model dynamically focuses on different patches at different training phases.

Original: First, it attaches importance to patches close to visible ones, which only demands fair local perception to recover.
Revised: First, it attaches importance to patches close to visible ones, which only demands fair local perception to recover.

Original: Then compared with the beginning of training, the model further pays attention to the distant and difﬁcult patches aided by the gradually reconstructed vicinity.
Revised: Then compared with the beginning of training, the model further pays attention to the distant and difﬁcult patches aided by the gradually reconstructed vicinity.

Original: Speciﬁcally, we ﬁrst deﬁne the reconstruction difﬁculty according to the distance between masked patches and visible ones, generating a distance map.
Revised: Speciﬁcally, we ﬁrst deﬁne the reconstruction difﬁculty according to the distance between masked patches and visible ones, generating a distance map.

Original: Then, we exert different supervision signals (which are controlled by learnable parameters β ) for intermediate layers.
Revised: Then, we exert different supervision signals (which are controlled by learnable parameters β ) for intermediate layers.

Original: As the training progresses, the model dynamically focuses on different regions with the update of β .
Revised: As the training progresses, the model dynamically focuses on different regions with the update of β .

Original: Note that β of layers are independent of each other.

Original: Since the pixel representations of perceptually similar images might be very different, only pixellevel reconstruction can be vulnerable to the perceptual difference between images (Dong et al., 2021).
Revised: Since the pixel representations of perceptually similar images might be very different, only pixellevel reconstruction can be vulnerable to the perceptual difference between images (Dong et al., 2021).

Original: Besides pixel regression, we further apply feature self-distillation to provide semantic information.
Revised: Besides pixel regression, we further apply feature self-distillation to provide semantic information.

Original: We directly align the encoder’s intermediate features with the corresponding features of the momentum encoder in deep dynamic supervision, where the momentum encoder is updated by
Revised: We directly align the encoder’s intermediate features with the corresponding features of the momentum encoder in deep dynamic supervision, where the momentum encoder is updated by

Original: Exponential Moving Average (EMA) (Grill et al., 2020;
Revised: Exponential Moving Average (EMA) (Grill et al., 2020;

Original: He et al., 2020).
Revised: He et al., 2020).

Original: It is worth noting that the feature distillation target here is the corresponding feature of the momentum encoder, while the regression targets for intermediate layers are all raw pixels.
Revised: It is worth noting that the feature distillation target here is the corresponding feature of the momentum encoder, while the regression targets for intermediate layers are all raw pixels.

Original: Dynamic Loss is applied for both feature self-distillation and pixel regression.
Revised: Dynamic Loss is applied for both feature self-distillation and pixel regression.

Original: By analyzing the characteristics of the pretrained encoder, we show that DDAE can effectively strengthen ViTs’ attention to local patterns, establishing a pretraining task that beneﬁts ViTs from structural priority.
Revised: By analyzing the characteristics of the pretrained encoder, we show that DDAE can effectively strengthen ViTs’ attention to local patterns, establishing a pretraining task that beneﬁts ViTs from structural priority.

Original: DDAE forms target perception attention effectively in the very early stage, while
Revised: DDAE forms target perception attention effectively in the very early stage, while

Original: MAE still behaves to be rudimentary (Fig 3).
Revised: MAE still behaves to be rudimentary (Fig 3).

Original: Furthermore, we directly migrate the core design deep dynamic supervision to the representative methods in one-stage and two-stage, MAE (He et al., 2022) and BEiT-v2 (Peng et al., 2022a) respectively, surpassing original methods with nontrivial margins.
Revised: Furthermore, we directly migrate the core design deep dynamic supervision to the representative methods in one-stage and two-stage, MAE (He et al., 2022) and BEiT-v2 (Peng et al., 2022a) respectively, surpassing original methods with nontrivial margins.

Original: Since deep dynamic supervision does not introduce any additional structure, it can be used as a general MIM pretraining plug-andplay module.
Revised: Since deep dynamic supervision does not introduce any additional structure, it can be used as a general MIM pretraining plug-andplay module.

Original: The contributions are summarized as follows:
Revised: The contributions are summarized as follows:

Original: • We propose DDAE, a one-stage tokenizer-free framework for self-supervised representation learning.
Revised: • We propose DDAE, a one-stage tokenizer-free framework for self-supervised representation learning.

Original: Our core design deep dynamic supervision can bemigrated to existing MIM
Revised: Our core design deep dynamic supervision can be

Original:  approaches seamlessly, providing more local priors for ViTs to make up for their inherent structural defects.
Revised: migrated to existing MIM approaches seamlessly, providing more local priors for ViTs to make up for their inherent structural defects.

Original: • Although deep supervision does not perform well in modern supervised learning, this paper makes a small step towards unleashing the potential of deep supervision in MIM.
Revised: • Although deep supervision does not perform well in modern supervised learning, this paper makes a small step towards unleashing the potential of deep supervision in MIM.

Original: • The experimental results demonstrate the effectiveness of the proposed DDAE, we achieve state-of-the-art performance among tokenizer-free methods and have competitive performance with many two-stage methods.
Revised: • The experimental results demonstrate the effectiveness of the proposed DDAE, we achieve state-of-the-art performance among tokenizer-free methods and have competitive performance with many two-stage methods.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Local Prior for ViTs.
Revised: Local Prior for ViTs.

Original: Aided by the global receptive ﬁeld of multi-head self-attentions(MSAs) mechanism, ViTs (Dosovitskiy et al., 2020) have signiﬁcantly advanced computer vision tasks such as image classiﬁcation (Wu et al., 2020; Touvron et al., 2021a; Yuan et al., 2021), object detection (Carion et al., 2020; Zhu et al., 2020;
Revised: Aided by the global receptive ﬁeld of multi-head self-attentions(MSAs) mechanism, ViTs (Dosovitskiy et al., 2020) have signiﬁcantly advanced computer vision tasks such as image classiﬁcation (Wu et al., 2020; Touvron et al., 2021a; Yuan et al., 2021), object detection (Carion et al., 2020; Zhu et al., 2020;

Original: Beal et al., 2020), and semantic segmentation (Zheng et al., 2021).However, since self-attention does not have CNN’s local prior, it often ignores local feature details, which largely leads to ViT’s demand for large dataset pretraining.
Revised: Beal et al., 2020), and semantic segmentation (Zheng et al., 2021).However, since self-attention does not have CNN’s local prior, it often ignores local feature details, which largely leads to ViT’s demand for large dataset pretraining.

Original: To solve, DeiT (Touvron et al., 2021a) transfers CNN-based knowledge to vision transformer through a distillation token.
Revised: To solve, DeiT (Touvron et al., 2021a) transfers CNN-based knowledge to vision transformer through a distillation token.

Original: T2T-ViT (Yuan et al., 2021) proposed using a tokenization module to recursively reorganize the image to consider neighboring pixels.
Revised: T2T-ViT (Yuan et al., 2021) proposed using a tokenization module to recursively reorganize the image to consider neighboring pixels.

Original: Local MSAs (Yang et al., 2019; Liu et al., 2021; Chu et al., 2021) are proposed to calculate self-attention only within small windows, achieving better performance than global MSAs not only on small datasets but also on large datasets, e.g. , ImageNet21K. Since local prior is the natural inductive bias of convolution, there are also works aimed at integrating convolution and vision transformer.
Revised: Local MSAs (Yang et al., 2019; Liu et al., 2021; Chu et al., 2021) are proposed to calculate self-attention only within small windows, achieving better performance than global MSAs not only on small datasets but also on large datasets, e.g. , ImageNet21K. Since local prior is the natural inductive bias of convolution, there are also works aimed at integrating convolution and vision transformer.

Original: Works like PVT (Wang et al.
Revised: Works like PVT (Wang et al.

Original: , 2021)and CvT (Wu et al., 2021) insert spatial reduction or convolution before global self-attention, yielding the merits of self-attention and convolution.
Revised: , 2021)and CvT (Wu et al., 2021) insert spatial reduction or convolution before global self-attention, yielding the merits of self-attention and convolution.

Original: MixFormer (Chen et al., 2022a) and Conformer (Peng et al., 2021) integrate the advantages by building parallel interactions.
Revised: MixFormer (Chen et al., 2022a) and Conformer (Peng et al., 2021) integrate the advantages by building parallel interactions.

Original: It should be noted that all above methods fundamentally change ViTs’ basic structure, bringing negative effects such as restricted receptive ﬁeld or increased computational cost of downstream tasks.
Revised: It should be noted that all above methods fundamentally change ViTs’ basic structure, bringing negative effects such as restricted receptive ﬁeld or increased computational cost of downstream tasks.

Original: Different from them, our work focuses on providing local prior for vanilla ViTs without introducing any extra structural modiﬁcation.
Revised: Different from them, our work focuses on providing local prior for vanilla ViTs without introducing any extra structural modiﬁcation.

Original: Masked Image Modeling.
Revised: Masked Image Modeling.

Original: Inspired by BERT (Devlin et al., 2018) in NLP, MIM learn representation by reconstructing masked image patches from visible patches.
Revised: Inspired by BERT (Devlin et al., 2018) in NLP, MIM learn representation by reconstructing masked image patches from visible patches.

Original: Existing MIM methods can be divided into two categories according to the need for the additional tokenizer.
Revised: Existing MIM methods can be divided into two categories according to the need for the additional tokenizer.

Original: Two-stage methods: Represented by the pioneering work BEiT
Revised: Two-stage methods: Represented by the pioneering work BEiT

Original: (Bao et al., 2021), two-stage methods (Bao et al., 2021;
Revised: (Bao et al., 2021), two-stage methods (Bao et al., 2021;

Original: Dong et al., 2021; Wei et al., 2022b; Chen et al., 2022b; Peng et al., 2022b) ﬁrstly transform image patches into semantic visual tokens through a pretrained discrete VAE (Ramesh et al., 2021)as visual tokenizer, then the corresponding tokens of masked image patches are reconstructed to pretrain the encoder.
Revised: Dong et al., 2021; Wei et al., 2022b; Chen et al., 2022b; Peng et al., 2022b) ﬁrstly transform image patches into semantic visual tokens through a pretrained discrete VAE (Ramesh et al., 2021)as visual tokenizer, then the corresponding tokens of masked image patches are reconstructed to pretrain the encoder.

Original: Nevertheless, the tokenizer needs to be ofﬂine pretrained with ﬁxed model architectures and extra data (Zhang et al., 2019b;
Revised: Nevertheless, the tokenizer needs to be ofﬂine pretrained with ﬁxed model architectures and extra data (Zhang et al., 2019b;

Original: Ramesh et al., 2021; Radford et al., 2021), some methods even further require an off-the-shelf DNN as teacher to distill tokenizer pre-training (Peng et al., 2022b), which complicates the process and increases cost.
Revised: Ramesh et al., 2021; Radford et al., 2021), some methods even further require an off-the-shelf DNN as teacher to distill tokenizer pre-training (Peng et al., 2022b), which complicates the process and increases cost.

Original: One-stage methods: MAE (He et al., 2022)constructs an asymmetric encoder-decoder structure, and directly performs raw pixel regression of masked image patches.
Revised: One-stage methods: MAE (He et al., 2022)constructs an asymmetric encoder-decoder structure, and directly performs raw pixel regression of masked image patches.

Original: SimMIM (Xie et al., 2022b) allows hierarchical transformers such as Swin (Liu et al., 2021) to be directly applied to MIM.
Revised: SimMIM (Xie et al., 2022b) allows hierarchical transformers such as Swin (Liu et al., 2021) to be directly applied to MIM.

Original: MaskFeat (Wei et al., 2022a),
Revised: MaskFeat (Wei et al., 2022a),

Original: Data2vec (Baevski et al., 2022), BootMAE (Dong et al., 2022), sDAE (Chen et al., 2022c) explored the choice of reconstruction targets.
Revised: Data2vec (Baevski et al., 2022), BootMAE (Dong et al., 2022), sDAE (Chen et al., 2022c) explored the choice of reconstruction targets.

Original: However, existing MIM methods reconstruct all patches equivalently, which is suboptimal.
Revised: However, existing MIM methods reconstruct all patches equivalently, which is suboptimal.

Original: Several inpainting works explored the dynamic loss design (Pathak et al., 2016; Yeh et al., 2016; Yu et al., 2018; Wang et al., 2018), but there is essential difference between MIM and inpainting.
Revised: Several inpainting works explored the dynamic loss design (Pathak et al., 2016; Yeh et al., 2016; Yu et al., 2018; Wang et al., 2018), but there is essential difference between MIM and inpainting.

Original: Inpainting focuses on the quality of the generated image, while MIM focuses on the representation of the encoder obtained from pretraining.
Revised: Inpainting focuses on the quality of the generated image, while MIM focuses on the representation of the encoder obtained from pretraining.

Original: Our DDAE is a tokenizerfree framework that focuses on dynamic reconstruction at different pretraining phases.
Revised: Our DDAE is a tokenizerfree framework that focuses on dynamic reconstruction at different pretraining phases.

Original: Deep Supervision.
Revised: Deep Supervision.

Original: Deep supervision methods are proposed to accelerate model’s convergence and alleviate the problem of gradient vanishment.
Revised: Deep supervision methods are proposed to accelerate model’s convergence and alleviate the problem of gradient vanishment.

Original: Through applying aux layers to transmit the supervision signal to the shallow layers, deep supervision has been used in early classiﬁcation models (Szegedy et al., 2015; Lee et al., 2015) and extended to other visual recognition tasks (Xie & Tu, 2015;
Revised: Through applying aux layers to transmit the supervision signal to the shallow layers, deep supervision has been used in early classiﬁcation models (Szegedy et al., 2015; Lee et al., 2015) and extended to other visual recognition tasks (Xie & Tu, 2015;

Original: Newell et al., 2016; Zhang et al., 2018b; Mosinska et al., 2018; Zhang et al., 2018a).
Revised: Newell et al., 2016; Zhang et al., 2018b; Mosinska et al., 2018; Zhang et al., 2018a).

Original: Despite[Figure] these advances, modern CNN classiﬁcation models rarely use auxiliary classiﬁers since directly appending simple auxiliary classiﬁers on top of the early layers of a modern network hurts its performance (Huang et al., 2017).
Revised: Despite these advances, modern CNN classiﬁcation models rarely use auxiliary classiﬁers since directly appending simple auxiliary classiﬁers on top of the early layers of a modern network hurts its performance (Huang et al., 2017).

Original: Several works utilize deep supervision to enhance the network through  knowledge distillation(Sun et al., 2019; Zhang et al., 2019a).
Revised: Several works utilize deep supervision to enhance the network through [Figure] knowledge distillation(Sun et al., 2019; Zhang et al., 2019a).

Original: Zhang et al.
Revised: Zhang et al.

Original: (2022) proposed Contrastive Deep Supervision to use contrastive learning signals for intermediate layers, but the signals of intermediate layers are still the same.
Revised: (2022) proposed Contrastive Deep Supervision to use contrastive learning signals for intermediate layers, but the signals of intermediate layers are still the same.

Original: Different from them, we introduce inconsistency in the intermediate signals for the ﬁrst time.
Revised: Different from them, we introduce inconsistency in the intermediate signals for the ﬁrst time.

Original: To the best of our knowledge, this paper is also the ﬁrst to apply deep supervision to the MIM task.
Revised: To the best of our knowledge, this paper is also the ﬁrst to apply deep supervision to the MIM task.

Original: State-of-the-art results show the great potential of deep supervision in the application of MIM.
Revised: State-of-the-art results show the great potential of deep supervision in the application of MIM.

Original:  M ETHODS
Revised:  M ETHODS

Original: In this section, we ﬁrstly elaborate on the basic framework of MIM.
Revised: In this section, we ﬁrstly elaborate on the basic framework of MIM.

Original: Then we introduce our proposed DDAE’s two core designs termed Deep Dynamic Supervision and Deep Self-Distillation in
Revised: Then we introduce our proposed DDAE’s two core designs termed Deep Dynamic Supervision and Deep Self-Distillation in

Original: Section 3.2 and Section 3.3 respectively.
Revised: Section 3.2 and Section 3.3 respectively.

Original: Our framework is illustrated in Fig 1.
Revised: Our framework is illustrated in Fig 1.

Original: 3. P RELIMINARY
Revised: P RELIMINARY

Original: Formally, MIM ﬁrstly divides the input image
Revised: Formally, MIM ﬁrstly divides the input image

Original: X ∈ R
Revised: X ∈ R

Original: H
Revised: H

Original: × W × C into non-overlapping ﬂattened patches x = [ x 1 , x 2 , ..., x N ] , where x i ∈ R P 2 C according to the patch size P , and N =
Revised: × W × C into non-overlapping ﬂattened patches x = [ x 1 , x 2 , ..., x N ] , where x i ∈ R P 2 C according to the patch size P , and N =

Original: ( H ×
Revised: ( H ×

Original: W ) /P
Revised: W ) /P

Original: 2 is the number of patches.
Revised: 2 is the number of patches.

Original: It then samples a random binary mask m =
Revised: It then samples a random binary mask m =

Original: [ m 1 , m 2 , ..., m N ] , where m i ∈ { 0 , 1 } to mask a portion of the ﬂattened patches.
Revised: [ m 1 , m 2 , ..., m N ] , where m i ∈ { 0 , 1 } to mask a portion of the ﬂattened patches.

Original: The masked patches x M (cid:44) x (cid:12) m are discarded (He et al., 2022) or substituted by a learnable token [MASK] (Bao et al., 2021), and the rest patches x 1 − M (cid:44) x (cid:12) (1 − m ) are used to reconstruct the dropped features or images to learn rich representations.
Revised: The masked patches x M (cid:44) x (cid:12) m are discarded (He et al., 2022) or substituted by a learnable token [MASK] (Bao et al., 2021), and the rest patches x 1 − M (cid:44) x (cid:12) (1 − m ) are used to reconstruct the dropped features or images to learn rich representations.

Original: The optimization target of MIM can be formulated as follow:
Revised: The optimization target of MIM can be formulated as follow:

Original: [Equation]
Revised: [Equation]

Original: where (cid:12) denotes element-wise multiplication; f θ ( · ) and d φ ( · ) are encoder and decoder respectively;
Revised: where (cid:12) denotes element-wise multiplication; f θ ( · ) and d φ ( · ) are encoder and decoder respectively;

Original: M ( · , · ) is the similarity measurement, which varies in different works, e.g. , l 2 -distance in pixel space (He et al., 2022), perceptual loss in codebook space (Dong et al., 2021) or self-distillation loss in feature space (Chen et al., 2022c).
Revised: M ( · , · ) is the similarity measurement, which varies in different works, e.g. , l 2 -distance in pixel space (He et al., 2022), perceptual loss in codebook space (Dong et al., 2021) or self-distillation loss in feature space (Chen et al., 2022c).

Original: In our work, we use the l 2 -distance as our measurement,
Revised: In our work, we use the l 2 -distance as our measurement,

Original: M ( a, b ) = ||
Revised: M ( a, b ) = ||

Original: a − b || 2 , for both of the pixel reconstruction and self-distillation.
Revised: a − b || 2 , for both of the pixel reconstruction and self-distillation.

Original: To simplify the formulation, we ignore the mask token term and use d φ ( · ) to represent d φ ([ · , [MASK] (cid:12) m ]) .
Revised: To simplify the formulation, we ignore the mask token term and use d φ ( · ) to represent d φ ([ · , [MASK] (cid:12) m ]) .

Original:      
Revised:      

Original: [Figure]
Revised: [Figure]

Original: 3. D EEP D YNAMIC S UPERVISION
Revised: 3. D EEP D YNAMIC S UPERVISION

Original: Dynamic.
Revised: Dynamic.

Original: Firstly, we deﬁne the difﬁculty of reconstruction according to the distance between each masked patch and visible patches, generating a distance map with a distance transform D ( · ) .
Revised: Firstly, we deﬁne the difﬁculty of reconstruction according to the distance between each masked patch and visible patches, generating a distance map with a distance transform D ( · ) .

Original: For each masked token ( m i = 1 ), the distance transform assigns a number that is the Euclidean distance between that token and the nearest unmasked token in 2D space.
Revised: For each masked token ( m i = 1 ), the distance transform assigns a number that is the Euclidean distance between that token and the nearest unmasked token in 2D space.

Original: Naturally, it is difﬁcult to recover a patch that is far from visible ones and demands stronger semantic reasoning ability.
Revised: Naturally, it is difﬁcult to recover a patch that is far from visible ones and demands stronger semantic reasoning ability.

Original: On the contrary, reconstruction of a patch with visible ones nearby only requires fair local perception.
Revised: On the contrary, reconstruction of a patch with visible ones nearby only requires fair local perception.

Original: As shown in Fig 2, existing MIM methods often use a high mask ratio like 75% (which is proved to be critical to MIM’s success
Revised: As shown in Fig 2, existing MIM methods often use a high mask ratio like 75% (which is proved to be critical to MIM’s success

Original: (He et al., 2022)), values of the distance map vary diversely.
Revised: (He et al., 2022)), values of the distance map vary diversely.

Original: Since the distance map is based on patches ( 14 × 14 ) rather than pixels ( 224 × 224 ), it is cost-free.
Revised: Since the distance map is based on patches ( 14 × 14 ) rather than pixels ( 224 × 224 ), it is cost-free.

Original: To guide the model to focus on different patches (corresponding to the requirement of distinctive properties) in different training phases, we propose h β ( · ) to learn a dynamic coefﬁcient β of the distance map to generate loss weight, directly applying dynamic focus to the loss weight of corresponding patches.
Revised: To guide the model to focus on different patches (corresponding to the requirement of distinctive properties) in different training phases, we propose h β ( · ) to learn a dynamic coefﬁcient β of the distance map to generate loss weight, directly applying dynamic focus to the loss weight of corresponding patches.

Original: h β is derived as follows:
Revised: h β is derived as follows:

Original: [Equation]
Revised: [Equation]

Original: We re-scale the loss weight map into [0 , 1] according to the max value of the weights.
Revised: We re-scale the loss weight map into [0 , 1] according to the max value of the weights.

Original: When β > 0 , larger distance leads to greater loss weight, so the model pays more attention to the reconstruction of difﬁcult patches.
Revised: When β > 0 , larger distance leads to greater loss weight, so the model pays more attention to the reconstruction of difﬁcult patches.

Original: Increasing the value of β can exacerbate this trend.
Revised: Increasing the value of β can exacerbate this trend.

Original: Conversely, when β < 0 , larger distance leads to smaller loss weight, and decreasing β results in more importance attached to simple patches.
Revised: Conversely, when β < 0 , larger distance leads to smaller loss weight, and decreasing β results in more importance attached to simple patches.

Original: Along with the changing of β , the model dynamically focuses on patches with diverse degrees of recovery difﬁculty.
Revised: Along with the changing of β , the model dynamically focuses on patches with diverse degrees of recovery difﬁculty.

Original: Deep supervision.
Revised: Deep supervision.

Original: As commented in the introduction, the reconstruction of patches with diverse difﬁculties requires distinct characteristics of the model, and layers at different depths naturally learn features at different levels.
Revised: As commented in the introduction, the reconstruction of patches with diverse difﬁculties requires distinct characteristics of the model, and layers at different depths naturally learn features at different levels.

Original: Therefore, we apply dynamic loss to the intermediate layers.
Revised: Therefore, we apply dynamic loss to the intermediate layers.

Original: Taking the standard ViT with B blocks as the encoder, we divide the blocks into K groups and extract the features at the end of each group.
Revised: Taking the standard ViT with B blocks as the encoder, we divide the blocks into K groups and extract the features at the end of each group.

Original: For example, we set K = 4 , then we extract output features of block 3, 6, 9 and 12 and feed them into the decoder respectively to recover masked patches.
Revised: For example, we set K = 4 , then we extract output features of block 3, 6, 9 and 12 and feed them into the decoder respectively to recover masked patches.

Original: K is also called supervisory number , since K is equal to the number of supervisory signals.
Revised: K is also called supervisory number , since K is equal to the number of supervisory signals.

Original: Then the loss function of pixel reconstruction L pixel is derived as follows:
Revised: Then the loss function of pixel reconstruction L pixel is derived as follows:

Original: [Equation]
Revised: [Equation]

Original: where f ( i ) ( · ) denotes to the output features of block- i in encoder, and g i =
Revised: where f ( i ) ( · ) denotes to the output features of block- i in encoder, and g i =

Original: BK i denotes to the group index; d φ p is the regression decoder; λ are the scale factors of each group, and set to [0 . 75 , 0 .
Revised: BK i denotes to the group index; d φ p is the regression decoder; λ are the scale factors of each group, and set to [0 . 75 , 0 .

Original: 75 , 1] by default.
Revised: 75 , 1] by default.

Original: Note that β of different layers are independent of each other.
Revised: Note that β of different layers are independent of each other.

Original: Deep dynamic supervision does not introduce any additional structure, so it can be seamlessly migrated to existing MIM structures.
Revised: Deep dynamic supervision does not introduce any additional structure, so it can be seamlessly migrated to existing MIM structures.

Original: We discussed its universality in Section 4.2 where we migrated it to the representative MIM methods of one-stage and multi-stage.
Revised: We discussed its universality in Section 4.2 where we migrated it to the representative MIM methods of one-stage and multi-stage.

Original: 3. D EEP S ELF -D ISTILLATION
Revised: 3. D EEP S ELF -D ISTILLATION

Original: In addition to raw pixel regression, the layer-by-layer correspondence between intermediate features is more suitable for the design of deep dynamic supervision.
Revised: In addition to raw pixel regression, the layer-by-layer correspondence between intermediate features is more suitable for the design of deep dynamic supervision.

Original: Therefore, we designed Deep
Revised: Therefore, we designed Deep

Original: Self-Distillation based on BootMAE (Dong et al., 2022) to further strengthen the model through high-level semantic features.
Revised: Self-Distillation based on BootMAE (Dong et al., 2022) to further strengthen the model through high-level semantic features.

Original: Speciﬁcally, the momentum encoder provides the features of masked patches of each layer as the target of self-distillation, so that the intermediate features of each layer corresponding to the encoder have their own targets of self-distillation.
Revised: Speciﬁcally, the momentum encoder provides the features of masked patches of each layer as the target of self-distillation, so that the intermediate features of each layer corresponding to the encoder have their own targets of self-distillation.

Original: The momentum encoder is updated by the encoder using the exponential moving average method (EMA).
Revised: The momentum encoder is updated by the encoder using the exponential moving average method (EMA).

Original: The feature selfdistillation also uses deep dynamic supervision as pixel regression.
Revised: The feature selfdistillation also uses deep dynamic supervision as pixel regression.

Original: Note that the regression of raw pixels is one-vs-all, the features of all layers will be reconstructed by one regression decoder.
Revised: Note that the regression of raw pixels is one-vs-all, the features of all layers will be reconstructed by one regression decoder.

Original: The feature self-distillation is all-vs-all, which means the shallow features of the encoder are selfdistilled by the corresponding shallow features of the momentum encoder, and the deep features are self-distilled by the corresponding deep features.
Revised: The feature self-distillation is all-vs-all, which means the shallow features of the encoder are selfdistilled by the corresponding shallow features of the momentum encoder, and the deep features are self-distilled by the corresponding deep features.

Original: Pixel regression and feature self-distillation use separate decoders, which both consist of two-layer transformer blocks.
Revised: Pixel regression and feature self-distillation use separate decoders, which both consist of two-layer transformer blocks.

Original: Since the decoders are light-weighted, the additional cost brought by deep supervision is acceptable.
Revised: Since the decoders are light-weighted, the additional cost brought by deep supervision is acceptable.

Original: Now the deep selfdistillation target formula is:
Revised: Now the deep selfdistillation target formula is:

Original: [Equation]
Revised: [Equation]

Original: where f θ ( · ) and f θ (cid:48) ( · ) are encoder and momentum encoder respectively
Revised: where f θ ( · ) and f θ (cid:48) ( · ) are encoder and momentum encoder respectively

Original: We add an L 2 regularization term to constrain the magnitude of β , and the overall loss function is:
Revised: We add an L 2 regularization term to constrain the magnitude of β , and the overall loss function is:

Original: [Equation]
Revised: [Equation]

Original: where λ is the scale factor to tune the L 2 regularization term and is set to 0.5 by default.
Revised: where λ is the scale factor to tune the L 2 regularization term and is set to 0.5 by default.

Original:  E XPERIMENTS 4. I MPLEMENTATION
Revised:  E XPERIMENTS 4. I MPLEMENTATION

Original: We conduct experiments on ImageNet-1K without labels as the pretraining data for self-supervised learning.
Revised: We conduct experiments on ImageNet-1K without labels as the pretraining data for self-supervised learning.

Original: The input resolution is set as 224 × 224 during pretraining and partitioned into 16 × 16 size patches.
Revised: The input resolution is set as 224 × 224 during pretraining and partitioned into 16 × 16 size patches.

Original: We pretrain the standard ViT small and base architectures, i.e., ViT-S/16 and ViT-B/16.
Revised: We pretrain the standard ViT small and base architectures, i.e., ViT-S/16 and ViT-B/16.

Original: The pixel regression decoder and feature distillation decoder both consist of 2 transformer blocks, along with an extra linear projection head for predictions.
Revised: The pixel regression decoder and feature distillation decoder both consist of 2 transformer blocks, along with an extra linear projection head for predictions.

Original: The dimension for the pixel regression decoder is 512 while for feature distillation decoder is set the same as the encoder.
Revised: The dimension for the pixel regression decoder is 512 while for feature distillation decoder is set the same as the encoder.

Original: We use blockwise masking with a ratio of 75%.
Revised: We use blockwise masking with a ratio of 75%.

Original: The data augmentation is only standard random cropping and horizontal ﬂipping.
Revised: The data augmentation is only standard random cropping and horizontal ﬂipping.

Original: All β are initialized as − 0 .
Revised: All β are initialized as − 0 .

Original: 5 by default.
Revised: 5 by default.

Original: The pretraining settings are almost the same as BootMAE (Dong et al., 2022)
Revised: The pretraining settings are almost the same as BootMAE (Dong et al., 2022)

Original: (See Appendix A for details).
Revised: (See Appendix A for details).

Original: 4. I MAGE C LASSIFICATION
Revised: 4. I MAGE C LASSIFICATION

Original: We evaluate both ﬁne-tuning accuracy and linear probing accuracy on ImageNet-1k.
Revised: We evaluate both ﬁne-tuning accuracy and linear probing accuracy on ImageNet-1k.

Original: Table 1 presents the comparison with previous state-of-the-art MIM-based methods including one-stage and twostage.
Revised: Table 1 presents the comparison with previous state-of-the-art MIM-based methods including one-stage and twostage.

Original: As a tokenizer-free framework, our DDAE can not only achieve higher performance, but also accelerate the convergence speed of pretraining.
Revised: As a tokenizer-free framework, our DDAE can not only achieve higher performance, but also accelerate the convergence speed of pretraining.

Original: In particular, with only 100 epochs pre-training,
Revised: In particular, with only 100 epochs pre-training,

Original: DDAE can achieve comparable performance with MAE using 1600 epochs pre-training and surpass 800 epochs pre-trained BEiT.
Revised: DDAE can achieve comparable performance with MAE using 1600 epochs pre-training and surpass 800 epochs pre-trained BEiT.

Original: Furthermore, with a longer pretraining schedule, DDAE achieves
Revised: Furthermore, with a longer pretraining schedule, DDAE achieves

Original: [Table]
Revised: [Table]

Original: 84.3% top-1 accuracy, developing a new state-of-the-art on ImageNet-1K among one-stage selfsupervised methods.
Revised: 84.3% top-1 accuracy, developing a new state-of-the-art on ImageNet-1K among one-stage selfsupervised methods.

Original: Note that despite SdAE reaching comparable results with DDAE when pretrained for 100 epochs, our method outperforms it with + 0.3% gains with 800 epoch schedule, demonstrating our consistent advantages both in a short schedule and a longer schedule.
Revised: Note that despite SdAE reaching comparable results with DDAE when pretrained for 100 epochs, our method outperforms it with + 0.3% gains with 800 epoch schedule, demonstrating our consistent advantages both in a short schedule and a longer schedule.

Original: In terms of linear probing, our approach surpasses the above MIM methods with the same training epochs, but is not as good as the contrastive-based methods.
Revised: In terms of linear probing, our approach surpasses the above MIM methods with the same training epochs, but is not as good as the contrastive-based methods.

Original: Contrastive-based methods compare across images while MIM-based methods exploit the whole image structure, which may care about more than 1000 classes (Chen et al., 2022b).
Revised: Contrastive-based methods compare across images while MIM-based methods exploit the whole image structure, which may care about more than 1000 classes (Chen et al., 2022b).

Original: This phenomenon is also reported in MAE (He et al., 2022) and BEiT (Bao et al., 2021), so for MIM-based methods, ﬁne-tuning measurement may be a better metric to validate their effectiveness.
Revised: This phenomenon is also reported in MAE (He et al., 2022) and BEiT (Bao et al., 2021), so for MIM-based methods, ﬁne-tuning measurement may be a better metric to validate their effectiveness.

Original: 4. A NALYSIS OF DDAE
Revised: 4. A NALYSIS OF DDAE

Original: Migrating Deep Dynamic Supervision to Existing MIM methods.
Revised: Migrating Deep Dynamic Supervision to Existing MIM methods.

Original: Although for simplicity we build our framework as a one-stage approach, our core design, Deep Dynamic Supervision(DDS), is compatible with other existing MIM methods.
Revised: Although for simplicity we build our framework as a one-stage approach, our core design, Deep Dynamic Supervision(DDS), is compatible with other existing MIM methods.

Original: To demonstrate that, we conduct experiments to migrate DDS into representative one-stage and two-stage MIM methods, MAE (He et al., 2022) and
Revised: To demonstrate that, we conduct experiments to migrate DDS into representative one-stage and two-stage MIM methods, MAE (He et al., 2022) and

Original: BEiT-v2 (Peng et al., 2022a) respectively.
Revised: BEiT-v2 (Peng et al., 2022a) respectively.

Original: We extract intermediate features of the encoder, then feed them into decoder together with the encoder’s ﬁnal output.
Revised: We extract intermediate features of the encoder, then feed them into decoder together with the encoder’s ﬁnal output.

Original: The Reconstruction targets are all set the same as the ﬁnal output.
Revised: The Reconstruction targets are all set the same as the ﬁnal output.

Original: Note reconstruction losses for intermediate layers are multiplied by their corresponding dynamic loss weight as mentioned in Section 3.2.
Revised: Note reconstruction losses for intermediate layers are multiplied by their corresponding dynamic loss weight as mentioned in Section 3.2.

Original: The plug-and-play results are reported in Table 2, which demonstrates our design’s generality for both one-stage and two-stage approaches.
Revised: The plug-and-play results are reported in Table 2, which demonstrates our design’s generality for both one-stage and two-stage approaches.

Original: More local priors for ViTs.
Revised: More local priors for ViTs.

Original: We conjecture that the ﬁne-grained task built by MIM enables ViT to pay more attention to local information besides using global semantic inference, which inherently makes up for the absence of local prior in the structure of ViTs.
Revised: We conjecture that the ﬁne-grained task built by MIM enables ViT to pay more attention to local information besides using global semantic inference, which inherently makes up for the absence of local prior in the structure of ViTs.

Original: [Table]
Revised: [Table]

Original: Peng
Revised: Peng

Original: [Table] [Table]
Revised: [Table] [Table]

Original: BEiT
Revised: BEiT

Original: [Table]
Revised: [Table]

Original: w/
Revised: w/

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: We compute averaged attention distance in each attention head of each layer (Dosovitskiy et al., 2020).
Revised: We compute averaged attention distance in each attention head of each layer (Dosovitskiy et al., 2020).

Original: We conduct experiments based on MAE since it is a simple enough MIM framework.
Revised: We conduct experiments based on MAE since it is a simple enough MIM framework.

Original: The encoder is ViT-B with 100 epoch pretraining.
Revised: The encoder is ViT-B with 100 epoch pretraining.

Original: Speciﬁcally, we draw vanilla MAE and DDS-MAE to demonstrate our design’s effectiveness.
Revised: Speciﬁcally, we draw vanilla MAE and DDS-MAE to demonstrate our design’s effectiveness.

Original: As illustrated in Fig 4, DDS remarkably reduces the average attention distance of the model, especially in deep layers.
Revised: As illustrated in Fig 4, DDS remarkably reduces the average attention distance of the model, especially in deep layers.

Original: This phenomenon indicates that through deep dynamic supervision the model focuses more on local vicinity, thus obtaining more local priors, especially in deep layers.
Revised: This phenomenon indicates that through deep dynamic supervision the model focuses more on local vicinity, thus obtaining more local priors, especially in deep layers.

Original: Moreover, for DDS part, different heads in relatively deeper layers behave more diversely, which may also contribute to DDS’s effectiveness (+ 1.22% gains over MAE on
Revised: Moreover, for DDS part, different heads in relatively deeper layers behave more diversely, which may also contribute to DDS’s effectiveness (+ 1.22% gains over MAE on

Original: ViT-S and more than + 0.3% gains on ViT-B).
Revised: ViT-S and more than + 0.3% gains on ViT-B).

Original: We further draw the curve of dynamic coefﬁcients β changes during training for different layers in
Revised: We further draw the curve of dynamic coefﬁcients β changes during training for different layers in

Original: Fig 5.
Revised: Fig 5.

Original: Firstly, for all intermediate layers, β drops sharply at the beginning, then rises slowly as the training progresses.
Revised: Firstly, for all intermediate layers, β drops sharply at the beginning, then rises slowly as the training progresses.

Original: Note that a smaller β value means that the model pays more attention to simple patches which are close to visible patches.
Revised: A smaller β value means that the model pays more attention to simple patches which are close to visible ones.

Original: Therefore, in the overall trend, each layer attaches more importance to the reconstruction of simple patches and gradually pays more attention to difﬁcult ones.
Revised: Therefore, in the overall trend, each layer attaches more importance to the reconstruction of simple patches and gradually pays more attention to difﬁcult ones.

Original: Ablation on Deep Dynamic Supervision.
Revised: Ablation on Deep Dynamic Supervision.

Original: We conduct ablation studies to analyze the contributions of each component in Deep Dynamic Supervision.
Revised: We conduct ablation studies to analyze the contributions of each component in Deep Dynamic Supervision.

Original: The models are ﬁnetuned on ImageNet-1K for 100 epochs.
Revised: The models are ﬁnetuned on ImageNet-1K for 100 epochs.

Original: We set ViT-S as encoder and pretrain for 100 epochs.
Revised: We set ViT-S as encoder and pretrain for 100 epochs.

Original: Table 3 shows that either ablation degrades the performance, suggesting that both deep supervision and dynamic loss beneﬁt the training dynamics
Revised: Table 3 shows that either ablation degrades the performance, suggesting that both designs beneﬁt the training dynamics. Ablation on Deep Self-Distillation.

Revised: Then, we study Deep Self-Distillation in DDAE, we conduct ablation experiments with ViT-B as encoder, pretrain for 100 epochs and ﬁnetune 100 epochs on

Revised: Methods Deep Supervision

Original: [Table]
Revised: [Table]

Revised: Loss

Original: [Table]
Revised: [Table]

Original: 
Revised: 

Original:      (SRFK         IRU%ORFN IRU%ORFN IRU%ORFN (a) Overall trend    
Revised:      (SRFK         IRU%ORFN IRU%ORFN IRU%ORFN (a) Overall trend    

Original:   (SRFK  
Revised:   (SRFK  

Original: 
Revised: 

Original:    IRU%ORFN IRU%ORFN IRU%ORFN (b) Detailed trend
Revised:    IRU%ORFN IRU%ORFN IRU%ORFN (b) Detailed trend

Original: [Table]
Revised: [Table]

Original: Ablation on Deep Self-Distillation.
Revised: ImageNet-1K. Results in Table 4 suggests that applying DDS on Pixel and Feature both can improve the feature representation learned.

Original: Then, we study Deep Self-Distillation in DDAE, we conduct ablation experiments with ViT-B as encoder, pretrain for 100 epochs and ﬁnetune 100 epochs on
Revised: 4. S EMANTIC S EGMENTATION

Original: ImageNet-1K. Results in Table 4 suggests that applying DDS on Pixel and Feature both can improve the feature representation learned.

Original: 4. S EMANTIC S EGMENTATION

Original: We evaluate the learned representation of DDAE on the ADE20K benchmark (Zhou et al., 2019), which consists of 25K images and 150 semantic categories.
Revised: We evaluate the learned representation of DDAE on the ADE20K benchmark (Zhou et al., 2019), which consists of 25K images and 150 semantic categories.

Original: We use the UperNet (Xiao et al., 2018) task layer for semantic segmentation.
Revised: We use the UperNet (Xiao et al., 2018) task layer for semantic segmentation.

Original: We train Upernet 160K iterations with single-scale inference.
Revised: We train Upernet 160K iterations with single-scale inference.

Original: The results are reported in Table 5 with mean Intersection of Union (mIoU) as the evaluation metric.
Revised: The results are reported in Table 5 with mean Intersection of Union (mIoU) as the evaluation metric.

Original: Compared with previous state-of-the-art self-supervised models, our proposed DDAE outperforms all the one-stage methods, further validating the effectiveness of our framework.
Revised: Compared with previous state-of-the-art self-supervised models, our proposed DDAE outperforms all the one-stage methods, further validating the effectiveness of our framework.

Original: on ADE20K. ViT-B is used as backbone.
Revised: [Table]

Original:  [Table] 
Revised: ) w/o 46. MAE [Table] w/o 48.

Original: C ONCLUSION
Revised: PeCo [Table] w/ 48.

Original: In this paper, we propose a new framework DDAE with two core designs.
Revised: CAE

Revised: [Table]

Original: (1) We present deep dynamic supervision to enable dynamic focus on different patches through pretraining, providing progressively richer representation.
Revised: w/ 48. SdAE [Table] ) w/o 49.

Revised: BootMAE

Revised: [Table]

Revised: w/o 49.

Revised: [Table]

Revised: ours)

Revised: w/o 49.

Revised:  C ONCLUSION AND L IMITIATIONS

Revised: In this paper, we propose a new framework DDAE with two core designs.

Revised: (1) We present deep dynamic supervision to enable dynamic focus on different patches through pretraining, providing progressively richer representation.

Original: (2) We propose to self-distill intermediate features of momentum encoder with dynamic supervision, which enhance the representation from a high semantic level.
Revised: (2) We propose to self-distill intermediate features of momentum encoder with dynamic supervision, which enhance the representation from a high semantic level.

Original: More importantly, we ﬁnd our approach can bring more local priors for ViTs, especially in deep layers, which we empirically ﬁnd crucial for MIM’s success.
Revised: More importantly, we ﬁnd our approach can bring more local priors for ViTs in deep layers, which we empirically ﬁnd crucial for MIM’s success.

Original: A T RAINING D ETAILS
Revised: Although our DDAE is effective, the correlation between difﬁculty and distance is less relevant for background patches.

Original: ImageNet Pretraining.
Revised: How to effectively distinguish foreground and background in an unsupervised way and apply different weights is a problem that needs to be further explored.

Revised: A T RAINING D ETAILS

Revised: ImageNet Pretraining.

Original: We use Adam (Kingma & Ba, 2014) as the optimizer and train the DDAE for 100/800 epochs to illustrate both the rapid convergence and a higher upper limit.
Revised: We use Adam (Kingma & Ba, 2014) as the optimizer and train the DDAE for 100/800 epochs to illustrate both the rapid convergence and a higher upper limit.

Original: The learning rate is set as 1 . 5 e − 4 ∗ batchsize/ 256 with cosine learning rate decay and warmed up for 20/ epochs respectively.
Revised: The learning rate is set as 1 . 5 e − 4 ∗ batchsize/ 256 with cosine learning rate decay and warmed up for 20/ epochs respectively.

Original: The total batch size is set as 4096.
Revised: The total batch size is set as 4096.

Original: The weight decay is set as 0.05.
Revised: The weight decay is set as 0.05.

Original: We increase the momentum parameter from 0.999 to 0.99999 in the ﬁrst 400 epochs.
Revised: We increase the momentum parameter from 0.999 to 0.99999 in the ﬁrst 400 epochs.

Original: The mask ratio is 75%.
Revised: The mask ratio is 75%.

Original: ImageNet Fine-tuning.
Revised: ImageNet Fine-tuning.

Original: We follow the ﬁne-tuning protocol in BEiT to use layer-wise learning rate decay as 0.75/0.65 for ViT-S/ViT-B respectively.
Revised: We follow the ﬁne-tuning protocol in BEiT to use layer-wise learning rate decay as 0.75/0.65 for ViT-S/ViT-B respectively.

Original: The batch size is 1024, the warmup epoch is 5 and the weight decay is 0.05.
Revised: The batch size is 1024, the warmup epoch is 5 and the weight decay is 0.05.

Original: For ViT-S, we train 100 epochs with a learning rate 1.6e-2.
Revised: For ViT-S, we train 100 epochs with a learning rate 1.6e-2.

Original: For ViT-B, We train 100 epochs with a learning rate 8e-3.
Revised: For ViT-B, We train 100 epochs with a learning rate 8e-3.

Original: Linear probing.
Revised: Linear probing.

Original: We use the LARS (Karpathy et al., 2014) optimizer with momentum 0.9.
Revised: We use the LARS (Karpathy et al., 2014) optimizer with momentum 0.9.

Original: We train the model for 90 epochs with a batch size of 16384.
Revised: We train the model for 90 epochs with a batch size of 16384.

Original: The warmup epoch is 10 and the base learning rate is 0.1.
Revised: The warmup epoch is 10 and the base learning rate is 0.1.

Original: We set weight decay as zero.
Revised: We set weight decay as zero.

Original: ADE20K Semantic segmentation.
Revised: ADE20K Semantic segmentation.

Original: We use UperNet (Xiao et al., 2018) based on mmsegmentation (Contributors, 2020).
Revised: We use UperNet (Xiao et al., 2018) based on mmsegmentation (Contributors, 2020).

Original: Following BEiT, we use AdamW (Loshchilov & Hutter, 2017) optimizer with initial learning rate 4 e − 4 , weight decay of 0.05 and batch size of 16 for 160 K iterations.
Revised: Following BEiT, we use AdamW (Loshchilov & Hutter, 2017) optimizer with initial learning rate 4 e − 4 , weight decay of 0.05 and batch size of 16 for 160 K iterations.

Original: We warm up the learning rate for 1500 iterations and use a linear decay strategy.
Revised: We warm up the learning rate for 1500 iterations and use a linear decay strategy.

Original: We set the layer decay as 0.65.
Revised: We set the layer decay as 0.65.

Original: Since ViTs can not output hierarchical features, we utilize four different scales FPNs to scale the feature map into different sizes.
Revised: Since ViTs can not output hierarchical features, we utilize four different scales FPNs to scale the feature map into different sizes.

Original: The input resolution is 512 × 512.
Revised: The input resolution is 512 × 512.

Original: We report single-scale test results.
Revised: We report single-scale test results.

Original: B M ORE A BLATION S TUDIES
Revised: B M ORE A BLATION S TUDIES

Original: Ablation on the supervisory number K .
Revised: Ablation on the supervisory number K .

Original: We further explored the impact of supervisory number
Revised: We further explored the impact of supervisory number

Original: K on performance.
Revised: K on performance.

Original: Speciﬁcally, we set K to be 1, 2, 4, and 6 respectively.
Revised: Speciﬁcally, we set K to be 1, 2, 4, and 6 respectively.

Original: The method to determine the block index is the same as that described in Section 3.2.
Revised: The method to determine the block index is the same as that described in Section 3.2.

Original: As shown in Table 6, the additional beneﬁts brought by more than four supervisory signals have been saturated.
Revised: As shown in Table 6, the additional beneﬁts brought by more than four supervisory signals have been saturated.

Original: So we set K = 4 as default in our framework.
Revised: So we set K = 4 as default in our framework.

Original: [Table]
Revised: [Table]

Original: Ablation on the choice of β .
Revised: Ablation on the choice of β .

Original: We also explored the choice of β .
Revised: We also explored the choice of β .

Original: Firstly, we ﬁx β with different initialization, then we set β as learnable parameters.
Revised: Firstly, we ﬁx β with different initialization, then we set β as learnable parameters.

Original: Table 7 demonstrates that learnable β beneﬁts most.
Revised: Table 7 demonstrates that learnable β beneﬁts most.

Original: We also draw the β curves with different initialization in Fig 6, the ﬁnal values of β and the relative relationship among block 3,6,9 tend to be stable and robust to initialization.
Revised: We also draw the β curves with different initialization in Fig 6, the ﬁnal values of β and the relative relationship among block 3,6,9 tend to be stable and robust to initialization.

Original: We take β = [ − 0 .
Revised: We take β = [ − 0 .

Original: 5 , − 0 .
Revised: 5 , − 0 .

Original: 5 , − 0 .
Revised: 5 , − 0 .

Original: 5 , − 0 . 5] as initialization in our experiments by default.
Revised: 5 , − 0 . 5] as initialization in our experiments by default.

Original: [Table]
Revised: [Table]

Original: 
Revised: 

Original:      (SRFK      IRU%ORFN IRU%ORFN IRU%ORFN
Revised:      (SRFK      IRU%ORFN IRU%ORFN IRU%ORFN

Original: [Figure]
Revised: [Figure]

Original:    
Revised:    

Original:   (SRFK    
Revised:   (SRFK    

Original:  IRU%ORFN IRU%ORFN IRU%ORFN
Revised:  IRU%ORFN IRU%ORFN IRU%ORFN

Original: [Figure]
Revised: [Figure]

Original: 
Revised: 

Original:      (SRFK      IRU%ORFN IRU%ORFN IRU%ORFN
Revised:      (SRFK      IRU%ORFN IRU%ORFN IRU%ORFN

Original: [Figure]
Revised: [Figure]

Original:    
Revised:    

Original:   (SRFK    
Revised:   (SRFK    

Original:  IRU%ORFN IRU%ORFN IRU%ORFN
Revised:  IRU%ORFN IRU%ORFN IRU%ORFN

Original: [Figure]
Revised: [Figure]

Original: C M ORE V ISUALIZATIONS
Revised: C M ORE V ISUALIZATIONS

Original: Our DDAE can form target-aware attention at a very early stage (Block 3), which is in sharp contrast to MAE.
Revised: Our DDAE can form target-aware attention at a very early stage (Block 3), which is in sharp contrast to MAE.

Original: Here we show more shallow attention maps of DDAE and MAE in Fig.
Revised: Here we show more shallow attention maps of DDAE and MAE in Fig.