Original: E FFICIENT S ELF - SUPERVISED V ISION T RANSFORMERS FOR R EPRESENTATION L EARNING
Revised: E FFICIENT S ELF - SUPERVISED V ISION T RANSFORMERS FOR R EPRESENTATION L EARNING

Original: Anonymous authors Paper under double-blind review A BSTRACT This paper investigates two techniques for developing efﬁcient self-supervised vision transformers (EsViT) for visual representation learning.
Revised: This paper investigates two techniques for developing efﬁcient self-supervised vision transformers (EsViT) for visual representation learning.

Original: First, we show through a comprehensive empirical study that multi-stage architectures with sparse self-attentions can signiﬁcantly reduce modeling complexity but with a cost of losing the ability to capture ﬁne-grained correspondences between image regions.
Revised: First, we show through a comprehensive empirical study that multi-stage architectures with sparse self-attentions can signiﬁcantly reduce modeling complexity but with a cost of losing the ability to capture ﬁne-grained correspondences between image regions.

Original: Second, we propose a new pre-training task of region matching which allows the model to capture ﬁne-grained region dependencies and as a result signiﬁcantly improves the quality of the learned vision representations.
Revised: Second, we propose a new pre-training task of region matching which allows the model to capture ﬁne-grained region dependencies and as a result signiﬁcantly improves the quality of the learned vision representations.

Original: Our results show that combining the two techniques, EsViT achieves 81.3% top-1 accuracy on the ImageNet linear probe evaluation, outperforming prior arts with around an order magnitude of higher throughput.
Revised: Our results show that combining the two techniques, EsViT achieves 81.3% top-1 accuracy on the ImageNet linear probe evaluation, outperforming prior arts with around an order magnitude of higher throughput.

Original: When transferring to downstream linear classiﬁcation tasks, EsViT outperforms its supervised counterpart on 17 out ofdatasets.
Revised: When transferring to downstream linear classiﬁcation tasks, EsViT outperforms its supervised counterpart on 17 out ofdatasets.

Original: The code and  models will be publicly available.
Revised: The code and pre-trained models are released at: https://github. com/microsoft/esvit

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Self-supervised learning (SSL) with Transformers (Vaswani et al., 2017) has become a de facto standard of model choice in natural language processing (NLP).
Revised: Self-supervised learning (SSL) with Transformers (Vaswani et al., 2017) has become a de facto standard of model choice in natural language processing (NLP).

Original: The dominant approaches such as
Revised: The dominant approaches such as

Original: GPT (Radford et al., 2018) and BERT (Devlin et al., 2019) are pre-training on a large text corpus and then ﬁne-tuning to various smaller task-speciﬁc datasets, showing superior performance.
Revised: GPT (Radford et al., 2018) and BERT (Devlin et al., 2019) are pre-training on a large text corpus and then ﬁne-tuning to various smaller task-speciﬁc datasets, showing superior performance.

Original: Larger
Revised: Larger

Original: Transformers pre-trained with larger-scale language datasets often lead to a stronger generalization ability, demonstrated by improved performance in downsteam tasks (with no sign of performance saturation yet), as exempliﬁed in GPT-3 (Brown et al., 2020).
Revised: Transformers pre-trained with larger-scale language datasets often lead to a stronger generalization ability, demonstrated by improved performance in downsteam tasks (with no sign of performance saturation yet), as exempliﬁed in GPT-3 (Brown et al., 2020).

Original: In computer vision (CV), however, self-supervised visual representation learning is still dominated by convolutional neural networks (CNNs).
Revised: In computer vision (CV), however, self-supervised visual representation learning is still dominated by convolutional neural networks (CNNs).

Original: Sharing a similar goal/spirit with NLP, SSL in CV aims to learn general-purpose image features from raw pixels without relying on manual supervisions, and the learned networks are expected to serve as the backbone of various downstream tasks such as classiﬁcation, detection and segmentation.
Revised: Sharing a similar goal/spirit with NLP, SSL in CV aims to learn general-purpose image features from raw pixels without relying on manual supervisions, and the learned networks are expected to serve as the backbone of various downstream tasks such as classiﬁcation, detection and segmentation.

Original: Recently, impressive performance have been achieved by
Revised: Recently, impressive performance have been achieved by

Original: CNN-based SSL, outperforming state-of-the-art (SoTA) fully-supervised pre-training methods (He et al., 2020; Caron et al., 2020) on tasks with a limited number of labels.
Revised: CNN-based SSL, outperforming state-of-the-art (SoTA) fully-supervised pre-training methods (He et al., 2020; Caron et al., 2020) on tasks with a limited number of labels.

Original: The key to success is view-level learning: maximizing agreement of learned representations between differently augmented views of the same example.
Revised: The key to success is view-level learning: maximizing agreement of learned representations between differently augmented views of the same example.

Original: Recent works, including SimCLR-v2 (Chen et al., 2020d), BYOL (Grill et al., 2020) and SwAV (Caron et al., 2020), have scaled up the CNN-based models to hundreds of millions of parameters.
Revised: Recent works, including SimCLR-v2 (Chen et al., 2020d), BYOL (Grill et al., 2020) and SwAV (Caron et al., 2020), have scaled up the CNN-based models to hundreds of millions of parameters.

Original: However, SSL has not enjoyed the same scaling success in CV as that in NLP.
Revised: However, SSL has not enjoyed the same scaling success in CV as that in NLP.

Original: Several attempts have been made to close the gap by combining SSL with Transformer and selfattention architectures.
Revised: Several attempts have been made to close the gap by combining SSL with Transformer and selfattention architectures.

Original: Early works include Selﬁe (Trinh et al., 2019), which generalizes the concept of masked language modeling of BERT for images.
Revised: Early works include Selﬁe (Trinh et al., 2019), which generalizes the concept of masked language modeling of BERT for images.

Original: The idea has been recently revisited in Vision
Revised: The idea has been recently revisited in Vision

Original: Transformer (ViT) (Dosovitskiy et al., 2021) via pre-training on a much larger scale dataset, e.g., JFT300M. ImageGPT (iGPT) (Chen et al., 2020b) generalizes the concept of auto-regressive language modeling of GPT for images, showing encouraging ImageNet recognition accuracy with a large model size.
Revised: Transformer (ViT) (Dosovitskiy et al., 2021) via pre-training on a much larger scale dataset, e.g., JFT300M. ImageGPT (iGPT) (Chen et al., 2020b) generalizes the concept of auto-regressive language modeling of GPT for images, showing encouraging ImageNet recognition accuracy with a large model size.

Original: Contrastive learning with ViT has also been studied very recently in DINO (Caron et al., 2021) and MoCo-v3 (Chen et al., 2021), where new SoTA result by linear probe evaluation on
Revised: Contrastive learning with ViT has also been studied very recently in DINO (Caron et al., 2021) and MoCo-v3 (Chen et al., 2021), where new SoTA result by linear probe evaluation on

Original:  ImageNet-1K is achieved, by exhaustively consuming computation resource on full self-attention operators with long sequences of split image patches.[Figure]
Revised: [Figure] ImageNet-1K is achieved, by exhaustively consuming computation resource on full self-attention operators with long sequences of split image patches.

Original: Aiming to improve the efﬁciency of Transformer-based SSL, this paper presents E fﬁcient s elfsuperivsed Vi sion T ransformers (EsViT), by using a multi-stage architecture and a region-based pre-training task for self-supervised representation learning.
Revised: Aiming to improve the efﬁciency of Transformer-based SSL, this paper presents E fﬁcient s elfsuperivsed Vi sion T ransformers (EsViT), by using a multi-stage architecture and a region-based pre-training task for self-supervised representation learning.

Original: Our main ﬁndings and contributions can be summarized as follows:
Revised: Our main ﬁndings and contributions can be summarized as follows:

Original: (1) An intriguing property of self-supervised monolithic Transformers is ﬁrstly reported in our paper: automatic discovery of semantic correspondence between local regions.
Revised: (1) An intriguing property of self-supervised monolithic Transformers is ﬁrstly reported in our paper: automatic discovery of semantic correspondence between local regions.

Original: (2) We present the ﬁrst comprehensive empirical study to show the pros and cons of multi-stage vision Transformer architectures for SSL.
Revised: (2) We present the ﬁrst comprehensive empirical study to show the pros and cons of multi-stage vision Transformer architectures for SSL.

Original: Though greatly reducing compute complexity, we ﬁnd that the multi-stage architecture causes the loss of the property in (1).
Revised: Though greatly reducing compute complexity, we ﬁnd that the multi-stage architecture causes the loss of the property in (1).

Original: (3) A region matching pre-train task is proposed to alleviate the issue in (2), and further improve the learned representations and attentions.
Revised: (3) A region matching pre-train task is proposed to alleviate the issue in (2), and further improve the learned representations and attentions.

Original: (4) We validate the new EsViT, which combines the two techniques, on a range of tasks.
Revised: (4) We validate the new EsViT, which combines the two techniques, on a range of tasks.

Original: It signiﬁcantly reduces the cost in building SoTA SSL vision systems, as summarized in Figure 1, and shows better scaling performance on accuracy vs. throughput and model size.
Revised: It signiﬁcantly reduces the cost in building SoTA SSL vision systems, as summarized in Figure 1, and shows better scaling performance on accuracy vs. throughput and model size.

Original: Under the linear evaluation protocol, EsViT achieves 81.3% top-1 accuracy, showing the best performance compared with all systems, and is 3 .
Revised: Under the linear evaluation protocol, EsViT achieves 81.3% top-1 accuracy, showing the best performance compared with all systems and is 3 .

Original: 5 × parameter-efﬁcient and has at least 10 × higher throughput than previous SoTA (81.0%, MoCo-v3 with ViT-BN-L/

Revised: 5 × parameter-efﬁcient and has at least 10 × higher throughput than previous SoTA (81.0%, MoCo-v3 with ViT-BN-L/

Original: (Chen et al., 2021)).
Revised: (Chen et al., 2021)).

Original: Compared with its supervised counterpart Swin Transformers (Liu et al.,2021), EsViT shows superior performance on 17 out 18 datasets, when transferring the learned representations to downstream linear classiﬁcation tasks.
Revised: Compared with its supervised counterpart Swin Transformers (Liu et al., EsViT shows superior performance on 17 out 18 datasets, when transferring the learned representations to downstream linear classiﬁcation tasks.

Original:  M ETHODS
Revised:  M ETHODS

Original: Transformer-based SSL methods emerge very recently to lead the state-of-the-art performance on the ImageNet linear probe task (Chen et al., 2021; Caron et al., 2021).
Revised: Transformer-based SSL methods emerge very recently to lead the state-of-the-art performance on the ImageNet linear probe task (Chen et al., 2021; Caron et al., 2021).

Original: It inherits the successes from (1) monolithic Transformer architectures that dominate in NLP (Devlin et al., 2019; Radford et al., 2018), and (2) instance-level contrastive learning objectives that demonstrate arguably the best SSL performance in computer vision (Chen et al., 2020c).
Revised: It inherits the successes from (1) monolithic Transformer architectures that dominate in NLP (Devlin et al., 2019; Radford et al., 2018), and (2) instance-level contrastive learning objectives that demonstrate arguably the best SSL performance in computer vision (Chen et al., 2020c).

Original: Though simple and effective, the existing Transformer-based SSL methods require a large amount of compute resources ( e.g., > 1.
Revised: Though simple and effective, the existing Transformer-based SSL methods require a large amount of compute resources ( e.g., > 1.

Original: TPU years of training) to reach SoTA performance.
Revised: TPU years of training) to reach SoTA performance.

Original: We believe that the SSL system efﬁciency is highly related to two ingredients: the network architecture and the pre-train task.
Revised: We believe that the SSL system efﬁciency is highly related to two ingredients: the network architecture and the pre-train task.

Original: To strike for a better tradeoff between accuracy and efﬁciency, we present EsViT, showing better synergy of networks (a multi-stage Transformer architecture) and pre-train tasks (a non-contrastive region-matching task).
Revised: To strike for a better tradeoff between accuracy and efﬁciency, we present EsViT, showing better synergy of networks (a multi-stage Transformer architecture) and pre-train tasks (a non-contrastive region-matching task).

Original: 2. N ETWORK A RCHITECTURES : F ROM M ONOLITHIC TO M ULTI - STAGE V I T B ACKBONE
Revised: 2. N ETWORK A RCHITECTURES : F ROM M ONOLITHIC TO M ULTI - STAGE V I T B ACKBONE

Original: Multi-stage ViT.
Revised: Multi-stage ViT.

Original: This paper presents the ﬁrst empirical study of multi-stage Transformer architectures (Vaswani et al., 2021; Wang et al., 2021; Liu et al., 2021; Zhang et al., 2021; Wu et al., 2021) for SSL.
Revised: This paper presents the ﬁrst empirical study of multi-stage Transformer architectures (Vaswani et al., 2021; Wang et al., 2021; Liu et al., 2021; Zhang et al., 2021; Wu et al., 2021) for SSL.

Original: Each stage consists of a patch merging/embedding module, and a Transformer with sparse self-attention module.
Revised: Each stage consists of a patch merging/embedding module, and a Transformer with sparse self-attention module.

Original: ( i )
Revised: ( i )

Original: The patch merging module plays a slightly different roles in different stages.
Revised: The patch merging module plays a slightly different roles in different stages.

Original: In the ﬁrst stage, it splits an input RGB image into non-overlapping patches.
Revised: In the ﬁrst stage, it splits an input RGB image into non-overlapping patches.

Original: Each patch is treated as a “token”, constructed as a concatenation of the raw pixel RGB values, which is further projected into a C -dimension feature.
Revised: Each patch is treated as a “token”, constructed as a concatenation of the raw pixel RGB values, which is further projected into a C -dimension feature.

Original: In the later stage, the patch merging module concatenates the features of each group of 2 × 2 neighboring patches, and applies a linear layer on the 4 C -dimensional concatenated features.
Revised: In the later stage, the patch merging module concatenates the features of each group of 2 × 2 neighboring patches, and applies a linear layer on the 4 C -dimensional concatenated features.

Original: This reduces the number of tokens by a multiple of 2 × 2 = 4 , and the output dimension is set to 2 C .
Revised: This reduces the number of tokens by a multiple of 2 × 2 = 4 , and the output dimension is set to 2 C .

Original: ( ii ) A Transformer with sparse self-attention module are then employed to enable interactions among the merged features.
Revised: ( ii ) A Transformer with sparse self-attention module are then employed to enable interactions among the merged features.

Original: The two modules above are repeated for multiple times, typically 4 times, resulting in a multi-stage ViT. As a result, a hierarchical representation is generated: the number of tokens is reduced and the feature dimension (and the number of heads in self-attentions) of each token is increased, as the network gets deeper.
Revised: The two modules above are repeated for multiple times, typically 4 times, resulting in a multi-stage ViT. As a result, a hierarchical representation is generated: the number of tokens is reduced and the feature dimension (and the number of heads in self-attentions) of each token is increased, as the network gets deeper.

Original: An overview comparison of the monolithic and multi-stage Transformer architectures for SSL is illustrated in Figure 7 in Appendix.
Revised: An overview comparison of the monolithic and multi-stage Transformer architectures for SSL is illustrated in Figure 7 in Appendix.

Original: An intriguing property of self-supervised monolithic ViT. Though straightforward in implementation, changing from monolithic to multi-stage architecture without careful treatments may lose some desirable properties of self-supervised Transformers In out study, we ﬁrst empirically note an intriguing property of self-supervised monolithic ViT(Caron et al., 2021): the pre-trained model exhibits a very strong ability to automatically discovers correspondences, even without a region-level matching objective speciﬁed in training .
Revised: An intriguing property of self-supervised monolithic ViT. Though straightforward in implementation, changing from monolithic to multi-stage architecture without careful treatments may lose some desirable properties of self-supervised Transformers In out study, we ﬁrst empirically note an intriguing property of self-supervised monolithic ViT(Caron et al., 2021): the pre-trained model exhibits a very strong ability to automatically discovers correspondences, even without a region-level matching objective speciﬁed in training .

Original: We quantitatively evaluate the correspondence learning to illustrate this property, as discussed in the following process.
Revised: We quantitatively evaluate the correspondence learning to illustrate this property, as discussed in the following process.

Original: ( i ) Simulated benchmark .
Revised: ( i ) Simulated benchmark .

Original: Based on 50K images in the ImageNet validation dataset, we create a simple evaluation benchmark with mild augmentations:
Revised: Based on 50K images in the ImageNet validation dataset, we create a simple evaluation benchmark with mild augmentations:

Original: For a center-crop image, we apply HorizontalFlip , then ColorJitter and RandomGrayscale to create a new augmented view.
Revised: For a center-crop image, we apply HorizontalFlip , then ColorJitter and RandomGrayscale to create a new augmented view.

Original: In this way, ground-truth correspondences are created.
Revised: In this way, ground-truth correspondences are created.

Original: ( ii ) Evaluation process .
Revised: ( ii ) Evaluation process .

Original: Given two views of the same image, we use the pre-trained backbone to extract the top-layer features, and ﬁnd the feature vector in one view that best matches the other in terms of highest cosine similarity.
Revised: Given two views of the same image, we use the pre-trained backbone to extract the top-layer features, and ﬁnd the feature vector in one view that best matches the other in terms of highest cosine similarity.

Original: The accuracy is measured as the averaged percentage of correctly identifying the region-to-region correspondences.
Revised: The accuracy is measured as the averaged percentage of correctly identifying the region-to-region correspondences.

Original: Please see details in Section C. in Appendix.
Revised: Please see details in Section C.7 in Appendix.

Original: ( iii ) Results .
Revised: ( iii ) Results .

Original: We quantitatively show that a self-supervised monolithic ViT yields 95% accuracy.
Revised: We quantitatively show that a self-supervised monolithic ViT yields 95% accuracy.

Original: However, simply replacing the network with a multi-stage Transformer yields only 66% accuracy.
Revised: However, simply replacing the network with a multi-stage Transformer yields only 66% accuracy.

Original: This signiﬁcant degradation (absolute 29% accuracy drop) reveals the loss of the correspondence learning property.
Revised: This signiﬁcant degradation (absolute 29% accuracy drop) reveals the loss of the correspondence learning property.

Original: We ﬁrst raise this critical problem, and believe that it has a large impact on the pre-trained model’s performance in various downstream tasks.
Revised: We ﬁrst raise this critical problem, and believe that it has a large impact on the pre-trained model’s performance in various downstream tasks.

Original: 2. P RE - TRAINING T ASKS : D ELVING INTO V IEWS WITH R EGIONS
Revised: 2. P RE - TRAINING T ASKS : D ELVING INTO V IEWS WITH R EGIONS

Original: We employ a non-contrastive learning framework to build our SSL method.
Revised: We employ a non-contrastive learning framework to build our SSL method.

Original: Speciﬁcally, Selfdistillation with no labels (DINO) (Caron et al., 2021) is considered.
Revised: Speciﬁcally, Selfdistillation with no labels (DINO) (Caron et al., 2021) is considered.

Original: It leverages the knowledge distillation learning paradigm where a student network g θ s is trained to match the output of a given teacher network g θ t , parameterized by θ s and θ t respectively.
Revised: It leverages the knowledge distillation learning paradigm where a student network g θ s is trained to match the output of a given teacher network g θ t , parameterized by θ s and θ t respectively.

Original: The neural network g is composed of a backbone f
Revised: The neural network g is composed of a backbone f

Original: ( e.g., Transformers or ConvNets), and of a projection head h : g
Revised: ( e.g., Transformers or ConvNets), and of a projection head h : g

Original: = h ◦ f .
Revised: = h ◦ f .

Original: The features used in downstream tasks are the output of backbone f .
Revised: The features used in downstream tasks are the output of backbone f .

Original: In SSL, different augmented views ˜ x of an image x are fed into backbone network to obtain feature maps z = f (˜ x ) .
Revised: In SSL, different augmented views ˜ x of an image x are fed into backbone network to obtain feature maps z = f (˜ x ) .

Original: Two MLP heads followed by softmax per network further convert the feature vectors z ∈ z into probability vectors p =
Revised: Two MLP heads followed by softmax per network further convert the feature vectors z ∈ z into probability vectors p =

Original: h ( z ) ; one head for view-level and the other head for region-level, respectively.
Revised: h ( z ) ; one head for view-level and the other head for region-level, respectively.

Original: More precisely, from a given image, we generate a set V of different views 1 following (Caron et al., 2021).
Revised: More precisely, from a given image, we generate a set V of different views 1 following (Caron et al., 2021).

Original: The resulting feature map at the top layer for each view is z =
Revised: The resulting feature map at the top layer for each view is z =

Original: [ z 1 , . .
Revised: [ z 1 , . .

Original: , z T ] , where T is the sequence length, and z i is a region-level representation for the local patch at position i .
Revised: , z T ] , where T is the sequence length, and z i is a region-level representation for the local patch at position i .

Original: Average pooling is applied to obtain the view-level representation ¯ z = avg-pool ( z ) .
Revised: Average pooling is applied to obtain the view-level representation ¯ z = avg-pool ( z ) .

Original: View-level task Given the augmented view set for student V and teacher V ∗ , a set of pairs P = { ( s, t ) | ˜ x s ∈ V , ˜ x t ∈ V ∗ and s (cid:54) =
Revised: View-level task Given the augmented view set for student V and teacher V ∗ , a set of pairs P = { ( s, t ) | ˜ x s ∈ V , ˜ x t ∈ V ∗ and s (cid:54) =

Original: t } is constructed to perform cross-view prediction tasks.
Revised: t } is constructed to perform cross-view prediction tasks.

Original: We consider the pre-training task at the view level proposed by (Caron et al., 2021):
Revised: We consider the pre-training task at the view level proposed by (Caron et al., 2021):

Original: [Equation]
Revised: [Equation]

Original: where p s =
Revised: where p s =

Original: h (¯ z s ) and p t
Revised: h (¯ z s ) and p t

Original: = h (¯ z t ) are the probability output of an MLP head h over the view-level representations ¯ z s and ¯ z t , learned by student and teacher, respectively.
Revised: = h (¯ z t ) are the probability output of an MLP head h over the view-level representations ¯ z s and ¯ z t , learned by student and teacher, respectively.

Original: In DINO, ViT/DeiT are considered, hence the view-level representation is the feature of the [CLS] token.
Revised: In DINO, ViT/DeiT are considered, hence the view-level representation is the feature of the [CLS] token.

Original: Region-level task In (Caron et al., 2021)
Revised: Region-level task In (Caron et al., 2021)

Original: , the L V encourages “local-to-global” correspondences only at a coarse level: the large crop and the small crop are matched in the view level, leaving region-to-region correspondence unspeciﬁed.
Revised: , the L V encourages “local-to-global” correspondences only at a coarse level: the large crop and the small crop are matched in the view level, leaving region-to-region correspondence unspeciﬁed.

Original: In monolithic Transformers, the drop paths and skip connections from low-level features to high-level features help the the latter to remain discriminative, thus maintain good region-matching performance.
Revised: In monolithic Transformers, the drop paths and skip connections from low-level features to high-level features help the the latter to remain discriminative, thus maintain good region-matching performance.

Original: However, such a property gets diluted due to the merging operators in multi-stage Transformers.
Revised: However, such a property gets diluted due to the merging operators in multi-stage Transformers.

Original: As shown in our experiments later, training a multi-stage network with L V only indeed results in sub-optimal representations.
Revised: As shown in our experiments later, training a multi-stage network with L V only indeed results in sub-optimal representations.

Original: Further, it could be a waste of computation not to leverage region-level features z that are computed in the process of extracting view-level feature.
Revised: Further, it could be a waste of computation not to leverage region-level features z that are computed in the process of extracting view-level feature.

Original: Inspired by the success of masked language modeling task in BERT, we argue that it is important to have region-level pre-training task for computer vision, so that the model can (1) amortize the computation and fully leverage the extracted region-level features, and (2) take into account the co-occurrences/structures between local features.
Revised: Inspired by the success of masked language modeling task in BERT, we argue that it is important to have region-level pre-training task for computer vision, so that the model can (1) amortize the computation and fully leverage the extracted region-level features, and (2) take into account the co-occurrences/structures between local features.

Original: Unfortunately, directly performing masked patch prediction (MPP) for the multi-stage Transformer architecture is infeasible, as the one-to-one correspondences between the input visual tokens and output features get diluted due to the merging operation.
Revised: Unfortunately, directly performing masked patch prediction (MPP) for the multi-stage Transformer architecture is infeasible, as the one-to-one correspondences between the input visual tokens and output features get diluted due to the merging operation.

Original: Even for monolithic architectures, MPP has not been proved effective in computer vision, as empirically shown in (Dosovitskiy et al., 2021).
Revised: Even for monolithic architectures, MPP has not been proved effective in computer vision, as empirically shown in (Dosovitskiy et al., 2021).

Original: To address this problem, we propose a non-contrastive, region-matching method that directly works at the level of local features by taking into account their correspondences:[Equation] , with
Revised: To address this problem, we propose a non-contrastive, region-matching method that directly works at the level of local features by taking into account their correspondences:

Original: [Equation]
Revised: [Equation]

Original: where p i
Revised: where p i

Original: =
Revised: =

Original: h (cid:48)
Revised: h (cid:48)

Original: ( z i ) and p j = h (cid:48)
Revised: ( z i ) and p j = h (cid:48)

Original: ( z j ) are the probability outputs of a new MLP head h (cid:48) over the local features of student z i ∈
Revised: ( z j ) are the probability outputs of a new MLP head h (cid:48) over the local features of student z i ∈

Original: z s and teacher z j ∈ z t , respectively.
Revised: z s and teacher z j ∈ z t , respectively.

Original: j ∗ is the index of the feature in z t that best matches the i -th feature in z s , in the sense of highest cosine similarity.
Revised: j ∗ is the index of the feature in z t that best matches the i -th feature in z s , in the sense of highest cosine similarity.

Original: Note that z i and z j ∗ are contextualized features of two best matched regions from different augmentated views, minimizing
Revised: Note that z i and z j ∗ are contextualized features of two best matched regions from different augmentated views, minimizing

Original: L R encourages different contexts ( i.e., surrounding regions) to learn invariant features, and thus captures the region-dependency.
Revised: L R encourages different contexts ( i.e., surrounding regions) to learn invariant features, and thus captures the region-dependency.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Revised: View

Revised: View

Original: The overall pre-training objective of EsViT is
Revised: The overall pre-training objective of EsViT is

Original: L = L R + L V , we learn to match the fea- ture distributions at both the view and region levels by minimizing the cross-entropy loss w.r.t.
Revised: L = L R + L V , we learn to match the fea- ture distributions at both the view and region levels by minimizing the cross-entropy loss w.r.t.

Original: the parameters of the student network g θ s .
Revised: the parameters of the student network g θ s .

Original: A visual illustration is in Figure 2, and the full algorithm is in Appendix.
Revised: A visual illustration is in Figure 2, and the full algorithm is in Appendix.

Original: We updates teacher/student network alternatively: ( i )
Revised: We updates teacher/student network alternatively: ( i )

Original: Given a ﬁxed teacher network, the student network is updated by minimizing the full cross-entropy loss: θ s ← arg min θ s L ( s, t ; θ s ) .
Revised: Given a ﬁxed teacher network, the student network is updated by minimizing the full cross-entropy loss: θ s ← arg min θ s L ( s, t ; θ s ) .

Original: ( ii )
Revised: ( ii )

Original: The teacher model is updated as an exponential moving average (EMA) of the student weights θ t ← λ θ t +(1 − λ ) θ s , with λ following a cosine schedule from 0.996 to 1 during training.
Revised: The teacher model is updated as an exponential moving average (EMA) of the student weights θ t ← λ θ t +(1 − λ ) θ s , with λ following a cosine schedule from 0.996 to 1 during training.

Original: By default, the full objective L is used from the beginning.
Revised: By default, the full objective L is used from the beginning.

Original: One can also load a checkpoint trained by L V only, and add L R for continual pre-training, which is shown effective in boosting performance in our experiments.
Revised: One can also load a checkpoint trained by L V only, and add L R for continual pre-training, which is shown effective in boosting performance in our experiments.

Original: Computational overhead Note that applying L R on the traditional monolithic Transformer architecture can be prohibitively computationally expensive, as it requires O ( T 2 ) to compute L R .
Revised: Computational overhead Note that applying L R on the traditional monolithic Transformer architecture can be prohibitively computationally expensive, as it requires O ( T 2 ) to compute L R .

Original: For a typical image of resolution 224 × 224 , the feature map length of ViT/DeiT (with patch size 16) at the top layer is T = 196 , while the multi-stage architecture yields T = 49 , which requires 3 times less compute in computing L R .
Revised: For a typical image of resolution 224 × 224 , the feature map length of ViT/DeiT (with patch size 16) at the top layer is T = 196 , while the multi-stage architecture yields T = 49 , which requires 3 times less compute in computing L R .

Original: To empirically illustrate this, we show in Appendix Section C.2 that
Revised: To empirically illustrate this, we show in Appendix Section C.2 that

Original: L R adds acceptable extra memory and computational cost (around 1.2 and 1.05 × , respectively) for multi-stage Transformers, while it will quickly go out-of-memory for monolithic Transformers when the batch size is increased.
Revised: L R adds acceptable extra memory and computational cost (around 1.2 and 1.05 × , respectively) for multi-stage Transformers, while it will quickly go out-of-memory for monolithic Transformers when the batch size is increased.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Relation to mask prediction tasks We can consider the proposed L R as a proxy to mimick masked language modeling in BERT, where the “ground-truth” local token is a soft label provided by the teacher network, while the student network makes predictions to match that target, based on the context of regions in a different augmented view.
Revised: Relation to mask prediction tasks We can consider the proposed L R as a proxy to mimick masked language modeling in BERT, where the “ground-truth” local token is a soft label provided by the teacher network, while the student network makes predictions to match that target, based on the context of regions in a different augmented view.

Original: Importantly, our L R considers softmax with cross-entropy in the objective, rather than MSE as in MPP.
Revised: Importantly, our L R considers softmax with cross-entropy in the objective, rather than MSE as in MPP.

Original: A very sharp teacher distribution is used by choosing small temperatures.
Revised: A very sharp teacher distribution is used by choosing small temperatures.

Original: This encourages the model to focus on the salient dimensions, rather than waste modeling capability on training short-range dependencies and high-frequency details (Ramesh et al., 2021).
Revised: This encourages the model to focus on the salient dimensions, rather than waste modeling capability on training short-range dependencies and high-frequency details (Ramesh et al., 2021).

Original: Relation to DenseCL The proposed L R mostly related to DenseCL (Wang et al., 2020b) in that the region correspondences in both methods are determined as the two most similar grid features.
Revised: Relation to DenseCL The proposed L R mostly related to DenseCL (Wang et al., 2020b) in that the region correspondences in both methods are determined as the two most similar grid features.

Original: One critical difference is that DenseCL is a contrastive region-matching task, while our L R is a non-contrastive region-matching task, where no negative samples/queue is needed.
Revised: One critical difference is that DenseCL is a contrastive region-matching task, while our L R is a non-contrastive region-matching task, where no negative samples/queue is needed.

Original: This technical difference has a signiﬁcant impact on the downstream task performance.
Revised: This technical difference has a signiﬁcant impact on the downstream task performance.

Original: We ﬁnd that L R is particularly effective in serving our goal to improve image classiﬁcation performance and build efﬁcient & affordable SoTA SSL system; In contrast, DenseCL degrades the classiﬁcation performance.
Revised: We ﬁnd that L R is particularly effective in serving our goal to improve image classiﬁcation performance and build efﬁcient & affordable SoTA SSL system; In contrast, DenseCL degrades the classiﬁcation performance.

Original: Relation to other region-level tasks The ideas of leveraging local region-level pre-training tasks for visual representation learning have been explored for ConvNets (Misra & Maaten, 2020; Xiong et al., 2020;
Revised: Relation to other region-level tasks The ideas of leveraging local region-level pre-training tasks for visual representation learning have been explored for ConvNets (Misra & Maaten, 2020; Xiong et al., 2020;

Original: Wang et al., 2020b; Xie et al., 2021a;
Revised: Wang et al., 2020b; Xie et al., 2021a;

Original: Yang et al., 2021; Xie et al., 2021c).
Revised: Yang et al., 2021; Xie et al., 2021c).

Original: We summarize the differences in three aspects: ( i ) Motivation.
Revised: We summarize the differences in three aspects: ( i ) Motivation.

Original: Our region-matching task L R aims to recover the lost property of automatic correspondence learning in self-supervised monolithic Transformers, while most existing region-level tasks aim to improve dense visual prediction tasks.
Revised: Our region-matching task L R aims to recover the lost property of automatic correspondence learning in self-supervised monolithic Transformers, while most existing region-level tasks aim to improve dense visual prediction tasks.

Original: ( ii ) Technical difference.
Revised: ( ii ) Technical difference.

Original: Our L R is a non-contrastive region-matching task, while others are contrastive learning.
Revised: Our L R is a non-contrastive region-matching task, while others are contrastive learning.

Original: ( iii )
Revised: ( iii )

Original: Empirical performance.
Revised: Empirical performance.

Original: Most region-level tasks improve dense visual prediction tasks but sacriﬁce their image classiﬁcation performance, while L R consistently improves classiﬁcation performance.
Revised: Most region-level tasks improve dense visual prediction tasks but sacriﬁce their image classiﬁcation performance, while L R consistently improves classiﬁcation performance.

Original: Among them, EsViT training method achieves the best ImageNet linear probe performance with minimum computational overhead.
Revised: Among them, EsViT training method achieves the best ImageNet linear probe performance with minimum computational overhead.

Original: For detailed comparisons, please refer to Table 7 in Appendix.
Revised: For detailed comparisons, please refer to Table 7 in Appendix.

Original: Self-supervised vision Transformers.
Revised: Self-supervised vision Transformers.

Original: The research on Transformer-based self-supervised representation learning just scratches the tip of the iceberg, and only a few attempts are made on this topic.
Revised: The research on Transformer-based self-supervised representation learning just scratches the tip of the iceberg, and only a few attempts are made on this topic.

Original: ImageGPT (Chen et al., 2020b) and MoCo-v3 (Chen et al., 2021) dedicate huge compute resource with large models to exploring the frontier.
Revised: ImageGPT (Chen et al., 2020b) and MoCo-v3 (Chen et al., 2021) dedicate huge compute resource with large models to exploring the frontier.

Original: DINO (Caron et al., 2021) achieves comparable performance of large self-supervised ConvNets using small/medium-size Transformers.
Revised: DINO (Caron et al., 2021) achieves comparable performance of large self-supervised ConvNets using small/medium-size Transformers.

Original: The proposed
Revised: The proposed

Original: EsViT further pursues efﬁcient and affordable solutions to self-supervised vision Transformers.
Revised: EsViT further pursues efﬁcient and affordable solutions to self-supervised vision Transformers.

Original: For more general related works on Transformers for vision tasks and self-supervised ConvNets, please refer to Section B in Appendix.
Revised: For more general related works on Transformers for vision tasks and self-supervised ConvNets, please refer to Section B in Appendix.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: We describe the experimental settings in Appendix Section C.3, and evaluate the proposed EsViT to answer three questions: Q1 :
Revised: We describe the experimental settings in Appendix Section C.3, and evaluate the proposed EsViT to answer three questions: Q1 :

Original: How does EsViT perform on standard ImageNet benchmark compared to SoTA methods?
Revised: How does EsViT perform on standard ImageNet benchmark compared to SoTA methods?

Original: Q2 :
Revised: Q2 :

Original: How effective EsViT is when transferring to downstream tasks?
Revised: How effective EsViT is when transferring to downstream tasks?

Original: Q3 :
Revised: Q3 :

Original: What are the design choices and empirical contributions of L R ?
Revised: What are the design choices and empirical contributions of L R ?

Original: Q4 : When does the intriguing property of self-supervised Transformers exist, including learned correspondence and attentions?
Revised: Q4 : When does the intriguing property of self-supervised Transformers exist, including learned correspondence and attentions?

Original: 4. C OMPARISONS WITH P RIOR A RT ON I MAGE N ET
Revised: 4. C OMPARISONS WITH P RIOR A RT ON I MAGE N ET

Original: We report top-1 linear probe and k -NN accuracy on the ImageNet validation set.
Revised: We report top-1 linear probe and k -NN accuracy on the ImageNet validation set.

Original: Table 1 presents comparisons with SoTA SSL systems across various architectures.
Revised: Table 1 presents comparisons with SoTA SSL systems across various architectures.

Original: Please refer to Figure 1 for comparisons over scaling parameter counts and throughput.
Revised: Please refer to Figure 1 for comparisons over scaling parameter counts and throughput.

Original: Our ﬁndings are summarized below.
Revised: Our ﬁndings are summarized below.

Original: Comparisons with self-supervised Transformers.
Revised: Comparisons with self-supervised Transformers.

Original: The DINO- and MoCo-based ViT has higher accuracy and smaller models than iGPT, under the same linear probing protocol and training data.
Revised: The DINO- and MoCo-based ViT has higher accuracy and smaller models than iGPT, under the same linear probing protocol and training data.

Original:  At the similar level of model size and compute complexity, the proposed EsViT improve SoTA methods DINO/MoCo-v3 by a large margin: EsViT (Swin-B) outperforms DINO (ViT-B/16) by 2.2% linear probe accuracy and 2.8% k -NN accuracy in absolute values.
Revised: [Table] At the similar level of model size and compute complexity, the proposed EsViT improve SoTA methods DINO/MoCo-v3 by a large margin: EsViT (Swin-B) outperforms DINO (ViT-B/16) by 2.2% linear probe accuracy and 2.8% k -NN accuracy in absolute values.

Original: EsViT (Swin-B) even performs slightly better than DINO (ViT-B/8) (0.3% higher linear probe accuracy and 1.5% higher k -NN accuracy), with 4 × higher throughput.
Revised: EsViT (Swin-B) even performs slightly better than DINO (ViT-B/8) (0.3% higher linear probe accuracy and 1.5% higher k -NN accuracy), with 4 × higher throughput.

Original: MoBY (Xie et al., 2021b) is a con-current work that investigates multi-stage ViT in SSL.
Revised: MoBY (Xie et al., 2021b) is a con-current work that investigates multi-stage ViT in SSL.

Original: With the same architecture Swin-T, our EsViT pre-training tasks signiﬁcantly outperform MoBY, showing 3% higher accuracy.
Revised: With the same architecture Swin-T, our EsViT pre-training tasks signiﬁcantly outperform MoBY, showing 3% higher accuracy.

Original: In EsViT, longer sequences in self-attention is implemented by increasing the window size.
Revised: In EsViT, longer sequences in self-attention is implemented by increasing the window size.

Original: We experiment this by considering a window size of W =14 .
Revised: We experiment this by considering a window size of W =14 .

Original: Overall, the proposed EsViT (Swin-B/ W =14) shows the best performance[Table] (top-1 accuracy 81.3%, top-5 accuracy 95.5%, k -NN accuracy 79.3%), compared with all systems and is 3 . 5 × parameter-efﬁcient and has at least 10 × higher throughput than previous SoTA MoCo-v
Revised: Overall, the proposed EsViT (Swin-B/ W =14) shows the best performance (top-1 accuracy 81.3%, top-5 accuracy 95.5%, k -NN accuracy 79.3%), compared with all systems, and is 3 . 5 × parameter-efﬁcient and has at least 10 × higher throughput than previous SoTA MoCo-v3.

Original: Comparisons with big ConvNets.
Revised: Comparisons with big ConvNets.

Original: We compare with the SoTA big ResNets reported by SimCLRv2 (Chen et al., 2020d), BYOL (Grill et al., 2020) and SwAV (Caron et al., 2020).
Revised: We compare with the SoTA big ResNets reported by SimCLRv2 (Chen et al., 2020d), BYOL (Grill et al., 2020) and SwAV (Caron et al., 2020).

Original: Among them, the best accuracy 79 .
Revised: Among them, the best accuracy 79 .

Original: 8% under the linear probing protocol is reported by SimCLR-v2 with SK-ResNet, where Selective Kernel (SK) (Li et al., 2019c) is a form of attention to enhance CNNs.
Revised: 8% under the linear probing protocol is reported by SimCLR-v2 with SK-ResNet, where Selective Kernel (SK) (Li et al., 2019c) is a form of attention to enhance CNNs.

Original: It is clear in Figure 1 (b) that all ConvNets-based SSL methods show an envelope in the regime of scaling up model sizes after passing 500 M. EsViT achieves better accuracy than their highest envelope, with 16 × less model parameters and 8 × higher throughput.
Revised: It is clear in Figure 1 (b) that all ConvNets-based SSL methods show an envelope in the regime of scaling up model sizes after passing 500 M. EsViT achieves better accuracy than their highest envelope, with 16 × less model parameters and 8 × higher throughput.

Original: 4. T RANSFER L EARNING
Revised: 4. T RANSFER L EARNING

Original: We also conduct transfer learning in downstream tasks to evaluate the quality of learned representations.
Revised: We also conduct transfer learning in downstream tasks to evaluate the quality of learned representations.

Original: Two sets of tasks are considered:
Revised: Two sets of tasks are considered:

Original: • Classiﬁcation on a suite of 18 small datasets .
Revised: • Classiﬁcation on a suite of 18 small datasets .

Original: As exempliﬁed in (Radford et al., 2021), it is a common and clean approach to evaluate a learned representation by ﬁtting a linear classiﬁer on the representation and measuring its performance across multiple datasets.
Revised: As exempliﬁed in (Radford et al., 2021), it is a common and clean approach to evaluate a learned representation by ﬁtting a linear classiﬁer on the representation and measuring its performance across multiple datasets.

Original: We studydatasets used in (Radford et al., 2021).
Revised: We studydatasets used in (Radford et al., 2021).

Original: Automatic hyper-parameter tuning is considered to ensure fairness of comparison.
Revised: Automatic hyper-parameter tuning is considered to ensure fairness of comparison.

Original: Besides averaged scores , we report # wins as the number of datasets on which the model outperforms its supervised counterpart.
Revised: Besides averaged scores , we report # wins as the number of datasets on which the model outperforms its supervised counterpart.

Original: Detailed dataset description and settings are in Appendix.
Revised: Detailed dataset description and settings are in Appendix.

Original: • Detection and segmentation on COCO .
Revised: [Figure]

Original: Different from previous monolithic self-supervised ViT, the multi-stage architecture in EsViT can be readily used for dense visual tasks that require hierarchical feature representations. Comparison with supervised counterparts. We compare with the supervised-learning Swin, whose checkpoints are downloaded from the ofﬁcial codebase 2 . Figure 3 shows the classiﬁcation results of Swin-S, EsViT consistently outperforms its supervised variant, often by a large margin.
Revised: F A

Original: [Figure]
Revised: [Figure]

Original: C
Revised: C

Original: I F
Revised: I F

Original: A R
Revised: A R

Original: [Figure]
Revised: [Figure]

Original: 0 C I FA R 1 0 0 D T [Figure] F E R
Revised: F

Original: [Figure]
Revised: [Figure]

Original: FG
Revised: F

Original: [Figure]
Revised: [Figure]

Original: r c r a f t F l o w e r
Revised: r c r a f t F l o w e r

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: F o o d 1 0 1 GT
Revised: F o o d [Figure] G

Original: [Figure]
Revised: [Figure]

Original: H a t e f u l M e m e s M N I S T
Revised: H a t e f u l M e m e s M N I S T

Original: P a t c h C a m e l y o n
Revised: P a t c h C a m e l y o n

Original: O x f o r d P e t[-s S t a n f o r d C a r s S T L 1 0 S U N 
-]
Revised: O x f o r d P e t[-s S t a n f o r d C a r s S T L 1 0 S U N 
-]

Original: [Figure]
Revised: [Figure]

Original: U C [-F 
-]
Revised: t a n f o r d C [-F 
-]a r

Original: [Figure]
Revised: [Figure]

Original: V O C
Revised: T L

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: +
Revised: U N

Original: [Figure]
Revised: [Figure]

Original: +
Revised: U

Original: [Figure]
Revised: [Figure]

Revised: Supervised

Original: [Figure]
Revised: [Figure]

Original: +
Revised: EsViT

Original: [Figure]
Revised: [Figure]

Original: +
Revised: V)

Original: [Figure]
Revised: [Figure]

Original: +
Revised: EsViT (

Original: [Figure]
Revised: [Figure]

Original: +
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: +2.

Original: [Figure]
Revised: [Figure]

Original: Supervised: 77. EsViT (V): 79. EsViT (V+R): 80.
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 1K
Revised: 1K

Original: [Figure]
Revised: [Figure]

Original: k
Revised: Scores #

Original: [Figure]
Revised: [Figure]

Original: NN

Original: Scores # Wins

Original: Supervised - - 77.
Revised: Supervised - - 77.

Original: - ImageNet-1K

Original: [Figure]
Revised: [Figure]

Revised: 1K

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Revised: • Detection and segmentation on COCO .

Revised: Different from previous monolithic self-supervised ViT, the multi-stage architecture in EsViT can be readily used for dense visual tasks that require hierarchical feature representations.

Revised: Comparison with supervised counterparts.

Revised: We compare with the supervised-learning Swin, whose checkpoints are downloaded from the ofﬁcial codebase 2 .

Revised: Figure 3 shows the classiﬁcation results of Swin-S, EsViT consistently outperforms its supervised variant, often by a large margin.

Original: Similar conclusions are drawn for other model sizes.
Revised: Similar conclusions are drawn for other model sizes.

Original: On COCO detection and segmentation task, however, EsViT shows comparable results with the variant with L V only (shown in parentheses) and the supervised counterpart (Swin-T trained with 3 × schedule), as shown in Table 2.
Revised: On COCO detection and segmentation task, however, EsViT shows comparable results with the variant with L V only (shown in parentheses) and the supervised counterpart (Swin-T trained with 3 × schedule), as shown in Table 2.

Original: We hypothsize this is related to the non-constrastive nature of EsViT, as explained later.
Revised: We hypothsize this is related to the non-constrastive nature of EsViT, as explained later.

Original: Effects of larger, less-curated pre-train datasets.
Revised: Effects of larger, less-curated pre-train datasets.

Original: The performance of Transformer-based SSL research has thus far been limited to highly curated pre-train data such as ImageNet-1K. To push the frontier in leveraging large amounts of unlabeled data, we explore the effects of pre-training from larger, less-curated image datasets: WebVision-v1 (Li et al., 2017), OpenImages-v4 (Kuznetsova et al., 2020) and ImageNet-22K (Deng et al., 2009), described in Appendix.
Revised: The performance of Transformer-based SSL research has thus far been limited to highly curated pre-train data such as ImageNet-1K. To push the frontier in leveraging large amounts of unlabeled data, we explore the effects of pre-training from larger, less-curated image datasets: WebVision-v1 (Li et al., 2017), OpenImages-v4 (Kuznetsova et al., 2020) and ImageNet-22K (Deng et al., 2009), described in Appendix.

Original: The pre-train epochs on different datasets are adjusted so that all models see a similar number of augmented views.
Revised: The pre-train epochs on different datasets are adjusted so that all models see a similar number of augmented views.

Original: We summarize the results in Table 3 and would like to emphasize the following ﬁndings.
Revised: We summarize the results in Table 3 and would like to emphasize the following ﬁndings.

Original: First, L R improves L V (shown in parentheses) on all datasets.
Revised: First, L R improves L V (shown in parentheses) on all datasets.

Original: Second, all EsViT pre-trained checkpoints outperform supervised checkpoint in downstream classiﬁcation tasks, but performance varies a lot, with ImageNet-22K checkpoint showing the best transfer ability.
Revised: Second, all EsViT pre-trained checkpoints outperform supervised checkpoint in downstream classiﬁcation tasks, but performance varies a lot, with ImageNet-22K checkpoint showing the best transfer ability.

Original: Third, ImageNet-1K pre-trained model shows the best ImageNet-1K linear probe performance.
Revised: Third, ImageNet-1K pre-trained model shows the best ImageNet-1K linear probe performance.

Original: We hypothesize that it is not only the size of pre-train dataset matters, but also the distribution of image classes matters: more diverse and well-balanced distribution results in a stronger generalization ability.
Revised: We hypothesize that it is not only the size of pre-train dataset matters, but also the distribution of image classes matters: more diverse and well-balanced distribution results in a stronger generalization ability.

Original: 4. D ISCUSSION ON THE N ON - CONTRASTIVE R EGION -M ATCHING T ASK
Revised: 4. D ISCUSSION ON THE N ON - CONTRASTIVE R EGION -M ATCHING T ASK

Original: Compatibility with various network architectures.
Revised: Compatibility with various network architectures.

Original: We investigate ResNet-50 and different efﬁcient sparse Transformers in Table 4.
Revised: We investigate ResNet-50 and different efﬁcient sparse Transformers in Table 4.

Original: DeiT is shown as a baseline reference.
Revised: DeiT is shown as a baseline reference.

Original: Batch size = 1024 in this experiment.
Revised: Batch size = 1024 in this experiment.

Original: To ensure fair comparison, we modify all into a 4-stage architecture with the number of Transformer layers in each stage as 2-2-6-2.
Revised: To ensure fair comparison, we modify all into a 4-stage architecture with the number of Transformer layers in each stage as 2-2-6-2.

Original: We see that L R improves all network architectures, including ResNet-50, Swin (Liu et al., 2021), ViL (Zhang et al., 2021), CvT (Wu et al., 2021) and PvT (Wang et al., 2021).
Revised: We see that L R improves all network architectures, including ResNet-50, Swin (Liu et al., 2021), ViL (Zhang et al., 2021), CvT (Wu et al., 2021) and PvT (Wang et al., 2021).

Original: Though directly adding L R to monolithic ViT is computationally infeasible, we uniformly sampled top-layer grid features of DeiT and then add L R , but did not observe performance improvement.
Revised: Though directly adding L R to monolithic ViT is computationally infeasible, we uniformly sampled top-layer grid features of DeiT and then add L R , but did not observe performance improvement.

Original: This is partly because the monolithic ViT itself already has a good corresponding ability, an extra region-matching task does not provide new learning signals.
Revised: This is partly because the monolithic ViT itself already has a good corresponding ability, an extra region-matching task does not provide new learning signals.

Original: As compared in Appendix Table 12 with the ResNet-50 backbone, EsViT learning method shows the highest accuracy, compared with existing SSL methods.

Original: Model scaling with L R . We compare the pre-training objective with and without L R in Table 1. Across different model scales and window sizes, the proposed region level L R can consistently improve the performance. The gains can be clearly seen by k -NN accuracy (around 1-2%), where no additional tuning is needed as in linear probe.
Revised: As

Original: Figure 4 demonstrates that L R helps model convergence, and can be used as a drop-in to improve models trained with the view level task.

Original: Method #Param.
Revised: Method #Param.

Original: Im./s
Revised: Im./s

Original: Pre-train tasks
Revised: Pre-train tasks

Original: Linear k -NN
Revised: Linear k -NN

Original: DeiT
Revised: DeiT

Original: L V 75.
Revised: L V 75.

Original: RL V 75.3 † 67.5 †
Revised: RL V 75.3 † 67.5 †

Original: L V 75.
Revised: L V 75.

Original: L V + L R 75.
Revised: L V + L R 75.

Original: Swin
Revised: Swin

Original: L V 77.
Revised: L V 77.

Original: L V + L R 77.
Revised: L V + L R 77.

Original: ViL
Revised: ViL

Original: L V 77.
Revised: L V 77.

Original: L V + L R 77.
Revised: L V + L R 77.

Original: CvT
Revised: CvT

Original: L V 77.
Revised: L V 77.

Revised: [Figure]

Original: L V + L R 78.
Revised: V + L R 78.

Original: PvT
Revised: PvT

Original: L V 75.
Revised: L V 75.

Revised: [Figure]

Original: L V + L R 76.
Revised: + [Figure] R 76.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Revised: COCO

Original: [Figure]
Revised: [Figure]

Original: ImageNet-1K

Original: COCO Types

Original: Methods #Epochs #Views
Revised: Methods #Epochs #Views

Original: Linear k -NN
Revised: Linear k -NN

Original: AP bb
Revised: AP bb

Original: AP mb
Revised: AP mb

Original: Supervised - - 38.
Revised: Supervised - - 38.

Original: Contrastive
Revised: [Figure]

Original: MoCo-v   67.
Revised: MoCo-v   67.

Original: 33.9 DenseCL  63.6 (-3.9) 48.6 (-7.0) 39.1 (+0.4) 34.2 (+0.3)
Revised: 33.9 DenseCL  63.6 (-3.9) 48.6 (-7.0) 39.1 (+0.4) 34.2 (+0.3)

Original: Non
Revised: Non

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Revised: compared in Appendix Table 12 with the ResNet-50 backbone, EsViT learning method shows the highest accuracy, compared with existing SSL methods.

Revised: Model scaling with L R .

Revised: We compare the pre-training objective with and without L R in Table 1.

Revised: Across different model scales and window sizes, the proposed region level L R can consistently improve the performance.

Revised: The gains can be clearly seen by k -NN accuracy (around 1-2%), where no additional tuning is needed as in linear probe.

Revised: Figure 4 demonstrates that L R helps model convergence, and can be used as a drop-in to improve models trained with the view level task.

Original: Contrastive vs Non-contrastive region-matching tasks.
Revised: Contrastive vs Non-contrastive region-matching tasks.

Original: The proposed L R adds a non-contrastive region-matching task to the non-contrastive view-level task L V ;
Revised: The proposed L R adds a non-contrastive region-matching task to the non-contrastive view-level task L V ;

Original: On the contrary, DenseCL adds a contrastive region-matching task to the contrastive view-level task MoCo-v2.
Revised: On the contrary, DenseCL adds a contrastive region-matching task to the contrastive view-level task MoCo-v2.

Original: In Table 5, we compare four methods in the same setting with ResNet-50.
Revised: In Table 5, we compare four methods in the same setting with ResNet-50.

Original: DenseCL improves dense visual prediction performance, but hurts classiﬁcation performance.
Revised: DenseCL improves dense visual prediction performance, but hurts classiﬁcation performance.

Original: L R improves both tasks, especially the classiﬁcation performance.
Revised: L R improves both tasks, especially the classiﬁcation performance.

Original: One limitation is that the non-contrastive methods show lower performance in dense prediction tasks, this is consistent with the observations for BYOL in (Wang et al., 2020b).
Revised: One limitation is that the non-contrastive methods show lower performance in dense prediction tasks, this is consistent with the observations for BYOL in (Wang et al., 2020b).

Original: The simple L R shows the best ImageNet accuracy compared with all sophisticated region-level tasks in this 200-epoch setting in Appendix Table 7, and the best overall accuracy in Table 12.
Revised: The simple L R shows the best ImageNet accuracy compared with all sophisticated region-level tasks in this 200-epoch setting in Appendix Table 7, and the best overall accuracy in Table 12.

Original: It indicates that L R well serves our goal in building efﬁcient SoTA SSL systems.
Revised: It indicates that L R well serves our goal in building efﬁcient SoTA SSL systems.

Original: Design choices of L R .
Revised: Design choices of L R .

Original: We ablate a couple of choices in constructing L R in Eq.
Revised: We ablate a couple of choices in constructing L R in Eq.

Original: ( i ) Softmax vs MSE .
Revised: ( i ) Softmax vs MSE .

Original: One alternative way to measure the distance between two projected vectors is MSE , as employed in the popular non-contrastive SSL algorithm BYOL (Grill et al., 2020).
Revised: One alternative way to measure the distance between two projected vectors is MSE , as employed in the popular non-contrastive SSL algorithm BYOL (Grill et al., 2020).

Original: When adding region-matching tasks to BYOL and pre-training 50 epochs, Softmax and MSE yield k -NN accuracy of 37.2% and 34.9%, while the baseline BYOL yields 33.1%.
Revised: When adding region-matching tasks to BYOL and pre-training 50 epochs, Softmax and MSE yield k -NN accuracy of 37.2% and 34.9%, while the baseline BYOL yields 33.1%.

Original: We also replace the region-matching metric in EsViT as MSE , yielding k -NN accuracy 72.6%, which lower than the view-level task only (74.2%).
Revised: We also replace the region-matching metric in EsViT as MSE , yielding k -NN accuracy 72.6%, which lower than the view-level task only (74.2%).

Original: These results show that Softmax is essential in L R .
Revised: These results show that Softmax is essential in L R .

Original: ( ii ) Optimal Transport (OT)
Revised: ( ii ) Optimal Transport (OT)

Original: vs
Revised: vs

Original: Simple Argmax.
Revised: Simple Argmax.

Original: To avoid heavy computational overhead, a simple feature-level argmax solution is considered in Eq.
Revised: To avoid heavy computational overhead, a simple feature-level argmax solution is considered in Eq.

Original: (2) to pair two local regions.
Revised: (2) to pair two local regions.

Original: To study the impact of high region-matching quality, we consider OT.
Revised: To study the impact of high region-matching quality, we consider OT.

Original: Empirically, we observe OT yields slightly higher k -NN accuracy at the early stage, but the gain is diminished in the end.
Revised: Empirically, we observe OT yields slightly higher k -NN accuracy at the early stage, but the gain is diminished in the end.

Original: Considering the extra computational cost of solving OT with an inner loop in sinkhorn algorithm (Cuturi, 2013), we opt for simple argmax in our experiments. 
Revised: Considering the extra computational cost of solving OT with an inner loop in sinkhorn algorithm (Cuturi, 2013), we opt for simple argmax in our experiments. [Figure] [Figure]

Original: 4. Q UALITATIVE S TUDIES
Revised: 4. Q UALITATIVE S TUDIES

Original: Visualization of correspondences.
Revised: Visualization of correspondences.

Original: Given two views of the same image, we use the pre-trained backbone to extract the top-layer features z 1 and z 2 .
Revised: Given two views of the same image, we use the pre-trained backbone to extract the top-layer features z 1 and z 2 .

Original: For each feature vector in z 1 , we ﬁnd the feature vector in z 2 that best matches it in terms of highest cosine similarity, as deﬁned in Equation (2).
Revised: For each feature vector in z 1 , we ﬁnd the feature vector in z 2 that best matches it in terms of highest cosine similarity, as deﬁned in Equation (2).

Original: In
Revised: In

Original: Figure 5, we show the top-10 correspondences between two views for three methods.
Revised: Figure 5, we show the top-10 correspondences between two views for three methods.

Original: In Figure(b), EsViT with L V tends to identify pairs in the background as the most matched ones (and in a wrong way in this example).
Revised: In Figure(b), EsViT with L V tends to identify pairs in the background as the most matched ones (and in a wrong way in this example).

Original: This could be a valid solution to L V , as the invariance in the level of aggregated global features does not necessarily induce invariances in the local region level.
Revised: This could be a valid solution to L V , as the invariance in the level of aggregated global features does not necessarily induce invariances in the local region level.

Original: This is signiﬁcantly alleviated with L R (shown in Figure 5 (c)), a task that implicitly requires local matching.[Figure] [Figure]
Revised: This is signiﬁcantly alleviated with L R (shown in Figure 5 (c)), a task that implicitly requires local matching.

Original: Surprisingly, DINO is able to learn good correspondences even without the region-level matching task.
Revised: Surprisingly, DINO is able to learn good correspondences even without the region-level matching task.

Original: To the best of our knowledge, this is a previously unreported intriguing property of self-supervised
Revised: To the best of our knowledge, this is a previously unreported intriguing property of self-supervised

Original: Transformers with monolithic architectures: good semantic correspondences are automatically learned.
Revised: Transformers with monolithic architectures: good semantic correspondences are automatically learned.

Original: We hypothesize that features at lower layers (image patch itself in the extreme case) can directly pass to higher layers, and the former regularizes the latter to remain discriminative.
Revised: We hypothesize that features at lower layers (image patch itself in the extreme case) can directly pass to higher layers, and the former regularizes the latter to remain discriminative.

Original: Nevertheless, the proposed L R can dramatically reduce the issue, and is good remedy to rescue the loss of semantic correspondence for the multi-stage architecture.
Revised: Nevertheless, the proposed L R can dramatically reduce the issue, and is good remedy to rescue the loss of semantic correspondence for the multi-stage architecture.

Original: In Appendix, we quantitatively measures the correspondence learning ability of these SSL methods on ImageNet validation dataset, the observations are consistent: L R improves the matching accuracy from 66% to 91%.
Revised: In Appendix, we quantitatively measures the correspondence learning ability of these SSL methods on ImageNet validation dataset, the observations are consistent: L R improves the matching accuracy from 66% to 91%.

Original: Visualization of attention maps.
Revised: Visualization of attention maps.

Original: We look at the self-attention in the different heads of the last layer in Figure 6.
Revised: We look at the self-attention in the different heads of the last layer in Figure 6.

Original: A local region on the edge of the main object is employed as query, and the attended regions are highlighted in red for those the query’s top 60% mass are assigned.
Revised: A local region on the edge of the main object is employed as query, and the attended regions are highlighted in red for those the query’s top 60% mass are assigned.

Original: In Appendix, we visualize more examples with different query positions.
Revised: In Appendix, we visualize more examples with different query positions.

Original: DINO tends to automatically learn class-speciﬁc attention maps leading to foreground object segmentation, regardless of its query located in foreground or background.
Revised: DINO tends to automatically learn class-speciﬁc attention maps leading to foreground object segmentation, regardless of its query located in foreground or background.

Original: This is probably because main objects remain as the major invariance factor in different augmented views.
Revised: This is probably because main objects remain as the major invariance factor in different augmented views.

Original: This property is lost when a multi-stage architecture is employed, as shown in EsViT with L V .
Revised: This property is lost when a multi-stage architecture is employed, as shown in EsViT with L V .

Original: These patterns are consistent for different heads.
Revised: These patterns are consistent for different heads.

Original: After introducing L R for EsViT, we note that the attention maps become more diverse in different heads, i.e., entropy values of attentions get more skewed, and attended regions are more different.
Revised: After introducing L R for EsViT, we note that the attention maps become more diverse in different heads, i.e., entropy values of attentions get more skewed, and attended regions are more different.

Original: This is perhaps because L R requires each region to consider many matching tasks to regions in different augmented views, each head automatically learns to distribute the tasks and complete a few of them.
Revised: This is perhaps because L R requires each region to consider many matching tasks to regions in different augmented views, each head automatically learns to distribute the tasks and complete a few of them.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper, we ﬁrst discover the automatic correspondence learning property of self-supervised monolithic Transformers.
Revised: In this paper, we ﬁrst discover the automatic correspondence learning property of self-supervised monolithic Transformers.

Original: Inspired by this, we present efﬁcient self-supervised vision Transformers (EsViT) to with two major insights: a multi-stage Transformer architecture with sparse self-attentions, and a non-contrastive region-matching pre-training task.
Revised: Inspired by this, we present efﬁcient self-supervised vision Transformers (EsViT) to with two major insights: a multi-stage Transformer architecture with sparse self-attentions, and a non-contrastive region-matching pre-training task.

Original: The synergy of both helps EsViT reach the
Revised: The synergy of both helps EsViT reach the

Original: SoTA performance of SSL vision systems with signiﬁcantly less compute and smaller model size.
Revised: SoTA performance of SSL vision systems with signiﬁcantly less compute and smaller model size.

Original: Our study also reveals that exploration of effective solutions to learn from larger and less curated pre-training data in the wild is a key but less studied factor in paving the way toward the scaling success of SSL vision systems.
Revised: Our study also reveals that exploration of effective solutions to learn from larger and less curated pre-training data in the wild is a key but less studied factor in paving the way toward the scaling success of SSL vision systems.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: Though self-supervised learning (SSL) has great potentials to learn powerful representation without human annotation, the existing techniques to build SoTA SSL vision systems tend to be Red
Revised: Though self-supervised learning (SSL) has great potentials to learn powerful representation without human annotation, the existing techniques to build SoTA SSL vision systems tend to be Red

Original: AI (Schwartz et al., 2020): it could be environmentally unfriendly and the computational cost is extensively high.
Revised: AI (Schwartz et al., 2020): it could be environmentally unfriendly and the computational cost is extensively high.

Original: The required training resource is typically not accessible for a lab environment (thus raising barriers to participation in AI research).
Revised: The required training resource is typically not accessible for a lab environment (thus raising barriers to participation in AI research).

Original: For example, the prior art MoCo-v3 has greatly pushes the performance limit of SSL system (Chen et al., 2021).
Revised: For example, the prior art MoCo-v3 has greatly pushes the performance limit of SSL system (Chen et al., 2021).

Original: The authors kindly reported that “it (MoCo-v3, ViT-H) takes 9.8 hours per 100 epochs using 512 TPUs.
Revised: The authors kindly reported that “it (MoCo-v3, ViT-H) takes 9.8 hours per 100 epochs using 512 TPUs.

Original: This is a gigantic scale of training: for the 300-epoch ViT-H, this amounts to ∼ 625 TPU days, or ∼ 1.7 TPU years of training.”
Revised: This is a gigantic scale of training: for the 300-epoch ViT-H, this amounts to ∼ 625 TPU days, or ∼ 1.7 TPU years of training.”

Original: The SoTA model MoCo-v3 with ViT-BN-L/7 should have a higher cost than this.
Revised: The SoTA model MoCo-v3 with ViT-BN-L/7 should have a higher cost than this.

Original: Even for a smaller model ViT-B, “it takes 24 hours in 128 GPUs (vs. 2.1 hours in 256 TPUs)”.
Revised: Even for a smaller model ViT-B, “it takes 24 hours in 128 GPUs (vs. 2.1 hours in 256 TPUs)”.

Original: Hence, improving the efﬁciency of building SoTA SSL systems is of high value for the community and society to achieve
Revised: Hence, improving the efﬁciency of building SoTA SSL systems is of high value for the community and society to achieve

Original: Green AI (Schwartz et al., 2020).
Revised: Green AI (Schwartz et al., 2020).

Original: To this end, we propose EsViT to provide more affordable and efﬁcient solutions for the community to experiment and explore the directions of SoTA SSL in computer vision.
Revised: To this end, we propose EsViT to provide more affordable and efﬁcient solutions for the community to experiment and explore the directions of SoTA SSL in computer vision.

Original: Our EsViT model shows the best ImageNet linear probe performance compared with all existing SSL vision systems, and is 3.5 × parameter-efﬁcient and has 10 × higher throughput than previous SoTA.
Revised: Our EsViT model shows the best ImageNet linear probe performance compared with all existing SSL vision systems, and is 3.5 × parameter-efﬁcient and has 10 × higher throughput than previous SoTA.

Original: This efﬁciency gain can signiﬁcantly decrease its carbon footprint and increase its inclusivity, encouraging more researchers to participate the study of the SSL topic.
Revised: This efﬁciency gain can signiﬁcantly decrease its carbon footprint and increase its inclusivity, encouraging more researchers to participate the study of the SSL topic.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: Our paper provides comprehensive empirical studies on the EsViT algorithm.
Revised: Our paper provides comprehensive empirical studies on the EsViT algorithm.

Original: We provide PyTorchstyle pseudo-code in Appendix.
Revised: We provide PyTorchstyle pseudo-code in Appendix.

Original: We also include an example code with instruction as supplementary material to ensure the reproducibility.
Revised: We also include an example code with instruction as supplementary material to ensure the reproducibility.

Original: For empirical results on both various network architecture and large-scale datasets, we provide detailed hyper-parameter speciﬁcations.
Revised: For empirical results on both various network architecture and large-scale datasets, we provide detailed hyper-parameter speciﬁcations.

Original: We will release the pre-trained checkpoints and codebase for the research community for reproducible research
Revised: We will release the pre-trained checkpoints and codebase for the research community for reproducible research

Original: A M ETHODS A. A LGORITHMS
Revised: A M ETHODS A. A LGORITHMS

Original: We summarize the training algorithm procedure of EsViT with L V + L R in Algorithm 1.
Revised: We summarize the training algorithm procedure of EsViT with L V + L R in Algorithm 1.

Original: To clearly outline the main idea of the algorithm, we show the algorithm for two augmented views.
Revised: To clearly outline the main idea of the algorithm, we show the algorithm for two augmented views.

Original: For the full algorithm to deal with multi-crop, please refer to our codebase.
Revised: For the full algorithm to deal with multi-crop, please refer to our codebase.

Original: In Algorithm 1, for a mini-batch of size n , the teacher/student network consists of three output variables: (1) p ∈ R n × K is the probability vector for the view-level representation, output by an MLP head.
Revised: In Algorithm 1, for a mini-batch of size n , the teacher/student network consists of three output variables: (1) p ∈ R n × K is the probability vector for the view-level representation, output by an MLP head.

Original: (2) z ∈ R n × T × P is the feature map, containing T region-level features of dimension P .
Revised: (2) z ∈ R n × T × P is the feature map, containing T region-level features of dimension P .

Original: (3) pz ∈ R n × T × K are probability vectors of z , output by a different MLP head.
Revised: (3) pz ∈ R n × T × K are probability vectors of z , output by a different MLP head.

Original: # gs, gt: student and teacher networks # Cv, Cr: view and region center (K)
Revised: # gs, gt: student and teacher networks # Cv, Cr: view and region center (K)

Original: # tmp s, tmp t: student and teacher temperatures # a, b: network and center momentum rates.
Revised: # tmp s, tmp t: student and teacher temperatures # a, b: network and center momentum rates.

Original: # n: batch size, K: MLP-head-projected probability vector length, T: last layer feature map length, P: last layer feature vector length 1 gt.params = gs.params
Revised: # n: batch size, K: MLP-head-projected probability vector length, T: last layer feature map length, P: last layer feature vector length 1 gt.params = gs.params

Original: # The main training loop 2 for x in loader: x1 , x2 = augment(x) , augment(x) # two random views # student output, p:n × K, pz:n × T × K, z:n × T × P p s1 , pz s1 , z s1 = gs(x1) p s2 , pz s2 , z s2 = gs(x2) # teacher output, p:n × K, pz:n × T × K, z:n × T × P p t1 , pz t1 , z t1 = gt(x1) p t2 , pz t2 , z t2 = gt(x2) # view-level loss loss v =
Revised: # The main training loop 2 for x in loader: x1 , x2 = augment(x) , augment(x) # two random views # student output, p:n × K, pz:n × T × K, z:n × T × P p s1 , pz s1 , z s1 = gs(x1) p s2 , pz s2 , z s2 = gs(x2) # teacher output, p:n × K, pz:n × T × K, z:n × T × P p t1 , pz t1 , z t1 = gt(x1) p t2 , pz t2 , z t2 = gt(x2) # view-level loss loss v =

Original: Hv(p s1, p t2) / 2 + Hv(p s2, p t1) / 2 # region-level loss loss r =
Revised: Hv(p s1, p t2) / 2 + Hv(p s2, p t1) / 2 # region-level loss loss r =

Original: Hr(pz s1, pz t2, z s1, z t2) / 2 + Hr(pz s2, pz t1, z s2, z t1) / loss = loss v / 2 + loss r / loss.backward() # back-propagate # update student, teacher and centers update(gs) # AdamW for student gt.params = a ∗ gt.params + (1 − a ) ∗ gs.params # EMA for teacher
Revised: Hr(pz s1, pz t2, z s1, z t2) / 2 + Hr(pz s2, pz t1, z s2, z t1) / loss = loss v / 2 + loss r / loss.backward() # back-propagate # update student, teacher and centers update(gs) # AdamW for student gt.params = a ∗ gt.params + (1 − a ) ∗ gs.params # EMA for teacher

Original: Cv = b ∗ Cv + (1 − b ) ∗ cat([p t1,p t2].mean(0))
Revised: Cv = b ∗ Cv + (1 − b ) ∗ cat([p t1,p t2].mean(0))

Original: # EMA for view center
Revised: # EMA for view center

Original: Cr =
Revised: Cr =

Original: b ∗ Cr + (1 − b ) ∗ cat([pz t1,pz t2].mean(0))
Revised: b ∗ Cr + (1 − b ) ∗ cat([pz t1,pz t2].mean(0))

Original: # EMA for region center 19 #
Revised: # EMA for region center 19 #

Original: The view-level loss function 20 def Hv(s, t): t = t.detach() # stop gradient s = softmax(s / tmp s, dim=-1) t = softmax((t - Cv) / tmp t, dim=-1) return - (t * log(s)).sum(dim=-1).mean()
Revised: The view-level loss function 20 def Hv(s, t): t = t.detach() # stop gradient s = softmax(s / tmp s, dim=-1) t = softmax((t - Cv) / tmp t, dim=-1) return - (t * log(s)).sum(dim=-1).mean()

Original: # The region-level loss function 25 def Hr(ps, pt, zs, zt): pt = pt.detach() # stop gradient ps = softmax(ps / tmp s, dim=-1)
Revised: # The region-level loss function 25 def Hr(ps, pt, zs, zt): pt = pt.detach() # stop gradient ps = softmax(ps / tmp s, dim=-1)

Original: # n × T × K pt = softmax((pt - Cr) / tmp t, dim=-1) # n × T × K sim matrix =
Revised: # n × T × K pt = softmax((pt - Cr) / tmp t, dim=-1) # n × T × K sim matrix =

Original: torch.matmul(zs , zt.permute(0, 2, 1))
Revised: torch.matmul(zs , zt.permute(0, 2, 1))

Original: # n × T × T sim idx = sim matrix.max(dim=-1)[1].unsqueeze(2) # n × T × pt idxed = [Table] [Table] [Table]
Revised: # n × T × T sim idx = sim matrix.max(dim=-1)[1].unsqueeze(2) # n × T × pt idxed = torch

Original: [Table]
Revised: [Table]

Original: A. N ETWORK ARCHITECTURE CONFIGURATIONS AND IMPLEMENTATION DETAILS
Revised: A. N ETWORK ARCHITECTURE CONFIGURATIONS AND IMPLEMENTATION DETAILS

Original: Inspired by great successes of the multi-stage ConvNet architecture such as VGG (Simonyan &
Revised: Inspired by great successes of the multi-stage ConvNet architecture such as VGG (Simonyan &

Original: Zisserman, 2014)/ResNets (He et al., 2016) for computer vision, the multi-stage Transformer-based networks have been explored very recently in the supervised learning setting (Vaswani et al., 2021;
Revised: Zisserman, 2014)/ResNets (He et al., 2016) for computer vision, the multi-stage Transformer-based networks have been explored very recently in the supervised learning setting (Vaswani et al., 2021;

Original: Wang et al., 2021; Liu et al., 2021; Zhang et al., 2021; Wu et al., 2021).
Revised: Wang et al., 2021; Liu et al., 2021; Zhang et al., 2021; Wu et al., 2021).

Original: In multi-stage vision
Revised: In multi-stage vision

Original: Transformers, since a larger number of patches is often produced at the early stages, an efﬁcient
Revised: Transformers, since a larger number of patches is often produced at the early stages, an efﬁcient

Original: Transformer with sparse self-attentions is considered to reduce the computational complexity.
Revised: Transformer with sparse self-attentions is considered to reduce the computational complexity.

Original: The basic idea is to split the feature maps into non-overlapping local windows (with size W × W ), and self-attention is performed within each local window.
Revised: The basic idea is to split the feature maps into non-overlapping local windows (with size W × W ), and self-attention is performed within each local window.

Original: This however has one drawback that features in different local windows cannot interact.
Revised: This however has one drawback that features in different local windows cannot interact.

Original: Various methods have been proposed to best approximate full-attention, with different trade-off between accuracy and efﬁciency.
Revised: Various methods have been proposed to best approximate full-attention, with different trade-off between accuracy and efﬁciency.

Original: We brieﬂy describe three schemes as follows, and benchmark them in the experiments.
Revised: We brieﬂy describe three schemes as follows, and benchmark them in the experiments.

Original: ( i )
Revised: ( i )

Original: Swin
Revised: Swin

Original: Transformer (Liu et al., 2021):
Revised: Transformer (Liu et al., 2021):

Original: A shifted window partitioning approach is proposed, which alternates between two partitioning conﬁgurations in consecutive Transformer blocks, so that each local feature is grouped into different windows in self-attentions.
Revised: A shifted window partitioning approach is proposed, which alternates between two partitioning conﬁgurations in consecutive Transformer blocks, so that each local feature is grouped into different windows in self-attentions.

Original: ( ii ) Vision Longformer (ViL) (Zhang et al., 2021):
Revised: ( ii ) Vision Longformer (ViL) (Zhang et al., 2021):

Original: Features in each local window are further allowed to attend all featuresin the
Revised: Features in each local window are further allowed to attend all features

Revised: in the

Original: -neighboring windows.
Revised: -neighboring windows.

Original: ( iii ) Convolution vision Transformer (CvT) (
Revised: ( iii ) Convolution vision Transformer (CvT) (Wu

Original: Wu et al., 2021)
Revised: et al., 2021)

Original: : Features in neighboring windows are considered in 
Revised: : Features in neighboring windows are considered in the

Original: the convolutional projection in self-attentions.
Revised: convolutional projection in self- attentions

Original: The window size is set to W = 7 by default.
Revised: The window size is set to W = 7 by default.

Original: The query dimension of each head in self-attentions is d = 32 , and the hidden layer width of each MLP is 4 × of its input’s width, for all experiments.
Revised: The query dimension of each head in self-attentions is d = 32 , and the hidden layer width of each MLP is 4 × of its input’s width, for all experiments.

Original: The architecture conﬁgurations of model variants employed in the experiments are summarized in Table 6.
Revised: The architecture conﬁgurations of model variants employed in the experiments are summarized in Table 6.

Original: Some notable implementation detailed are described as follows:
Revised: Some notable implementation detailed are described as follows:

Original: • The three conﬁgurations Swin-T, Swin-S and Swin-B indicate Tiny, Small, and Base models, respectively, which are almost identical to the original implementation (Liu et al., 2021), except that we add special treatments to deal with input augmented views of different resolutions, when the resolution (feature map size more speciﬁcally) is not divisible by the window size ( i
Revised: • The three conﬁgurations Swin-T, Swin-S and Swin-B indicate Tiny, Small, and Base models, respectively, which are almost identical to the original implementation (Liu et al., 2021), except that we add special treatments to deal with input augmented views of different resolutions, when the resolution (feature map size more speciﬁcally) is not divisible by the window size ( i

Original: e., resolution 96 and window size=7 or 14).
Revised: e., resolution 96 and window size=7 or 14).

Original: • Swin-T and Swin-S with window size W =14 are customized by us to allow full self-attention in stage 3 (where the majority of model capacity is allocated to) and stage 4, to study the impact of longer sequences in EsViT. •
Revised: • Swin-T and Swin-S with window size W =14 are customized by us to allow full self-attention in stage 3 (where the majority of model capacity is allocated to) and stage 4, to study the impact of longer sequences in EsViT. •

Original: In the original ViL (Zhang et al., 2021) and CvT (Wu et al., 2021) papers, different positional embedding strategies and multi-stage network conﬁgurations were employed.
Revised: In the original ViL (Zhang et al., 2021) and CvT (Wu et al., 2021) papers, different positional embedding strategies and multi-stage network conﬁgurations were employed.

Original: We modify them by only utilizing relative position bias and their proposed sparse self-attention mechanisms, and create a similar 4-stage architecture with Swin-T for fair comparison.
Revised: We modify them by only utilizing relative position bias and their proposed sparse self-attention mechanisms, and create a similar 4-stage architecture with Swin-T for fair comparison.

Original: Relative Position Bias.
Revised: Relative Position Bias.

Original: To facilitate SSL, we consider relative position bias (Liu et al., 2021) to characterize the spatial information between features for the three efﬁcient Transformers aforementioned, and do not use absolute position embeddings.
Revised: To facilitate SSL, we consider relative position bias (Liu et al., 2021) to characterize the spatial information between features for the three efﬁcient Transformers aforementioned, and do not use absolute position embeddings.

Original: This is because augmented views of varied resolutions can be cropped from anywhere in an image in SSL, maintaining the relative positions is easy in implementation, and is largely sufﬁcient for invariance learning among these views.
Revised: This is because augmented views of varied resolutions can be cropped from anywhere in an image in SSL, maintaining the relative positions is easy in implementation, and is largely sufﬁcient for invariance learning among these views.

Original: B R ELATED W ORK
Revised: B R ELATED W ORK

Original: Self-supervised ConvNets.
Revised: Self-supervised ConvNets.

Original: ConvNets-based SSL has been extensively studied in the literature.
Revised: ConvNets-based SSL has been extensively studied in the literature.

Original: Based on the pre-training tasks, they can be broadly categorized into three classes: Handcrafted pretext tasks (Doersch et al., 2015; Noroozi & Favaro, 2016; Pathak et al., 2016; Gidaris et al., 2018;
Revised: Based on the pre-training tasks, they can be broadly categorized into three classes: Handcrafted pretext tasks (Doersch et al., 2015; Noroozi & Favaro, 2016; Pathak et al., 2016; Gidaris et al., 2018;

Original: Zhang et al., 2016; Larsson et al., 2016;
Revised: Zhang et al., 2016; Larsson et al., 2016;

Original: Zhang et al., 2017; Pu et al., 2016; Donahue & Simonyan, 2019), contrastive learning (Dosovitskiy et al., 2015; Zhuang et al., 2019; Oord et al., 2018; Hjelm et al., 2018; Bachman et al., 2019;
Revised: Zhang et al., 2017; Pu et al., 2016; Donahue & Simonyan, 2019), contrastive learning (Dosovitskiy et al., 2015; Zhuang et al., 2019; Oord et al., 2018; Hjelm et al., 2018; Bachman et al., 2019;

Original: He et al., 2020; Chen et al., 2020c; Grill et al., 2020) and prototype learning (Caron et al., 2018; 2020;
Revised: He et al., 2020; Chen et al., 2020c; Grill et al., 2020) and prototype learning (Caron et al., 2018; 2020;

Original: Li et al., 2020b; Xie et al., 2016; Yang et al., 2016; Ji et al., 2019;
Revised: Li et al., 2020b; Xie et al., 2016; Yang et al., 2016; Ji et al., 2019;

Original: Zhan et al., 2020).
Revised: Zhan et al., 2020).

Original: It is also known that data augmentations play a crucial role in SSL pipeline (Chen et al., 2020e; Caron et al., 2020; Tian et al., 2020; Li et al., 2020a).
Revised: It is also known that data augmentations play a crucial role in SSL pipeline (Chen et al., 2020e; Caron et al., 2020; Tian et al., 2020; Li et al., 2020a).

Original: The impact of pre-training dataset size/quality is explored for ConvNets in SSL (Goyal et al., 2021; Yonglong et al., 2021).
Revised: The impact of pre-training dataset size/quality is explored for ConvNets in SSL (Goyal et al., 2021; Yonglong et al., 2021).

Original: To date, the search of best pre-taining tasks/datasets and augmentations are based on CNNs.
Revised: To date, the search of best pre-taining tasks/datasets and augmentations are based on CNNs.

Original: Among them,
Revised: Among them,

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: SimCLR-v2 (Chen et al., 2020d), BYOL (Grill et al., 2020) and SwAV (Caron et al., 2020) achieve the highest ImageNet linear probe performance with large ConvNet architectures.
Revised: SimCLR-v2 (Chen et al., 2020d), BYOL (Grill et al., 2020) and SwAV (Caron et al., 2020) achieve the highest ImageNet linear probe performance with large ConvNet architectures.

Original: The performance tends to saturate with an increasingly growing model size, raising a question if ConvNets reach a limit in SSL.
Revised: The performance tends to saturate with an increasingly growing model size, raising a question if ConvNets reach a limit in SSL.

Original: Transformers for vision.
Revised: Transformers for vision.

Original: Vision Transformers (ViT) (Dosovitskiy et al., 2021) shows the great potentials of generalizing Transformers for computer vision, by achieving compelling accuracy in supervised learning, especially with large-scale data and high capacity models.
Revised: Vision Transformers (ViT) (Dosovitskiy et al., 2021) shows the great potentials of generalizing Transformers for computer vision, by achieving compelling accuracy in supervised learning, especially with large-scale data and high capacity models.

Original: DeiT (Touvron et al., 2020) further provides an effective ViT training strategy to ease the adaption of Transformers for practitioners.
Revised: DeiT (Touvron et al., 2020) further provides an effective ViT training strategy to ease the adaption of Transformers for practitioners.

Original: Transformers have also been applied to other vision tasks, ranging from low-level tasks such as image generation (Parmar et al., 2018; Chen et al., 2020b) and enhancement (Chen et al., 2020a; Yang et al., 2020), to high-level tasks such as object detection (Carion et al., 2020;
Revised: Transformers have also been applied to other vision tasks, ranging from low-level tasks such as image generation (Parmar et al., 2018; Chen et al., 2020b) and enhancement (Chen et al., 2020a; Yang et al., 2020), to high-level tasks such as object detection (Carion et al., 2020;

Original: Zhu et al., 2020; Zheng et al., 2020; Dai et al., 2020) and segmentation (Wang et al., 2020a;c), and to vision-language tasks (Lu et al., 2019; Tan & Bansal, 2019; Chen et al., 2019; Su et al., 2019;
Revised: Zhu et al., 2020; Zheng et al., 2020; Dai et al., 2020) and segmentation (Wang et al., 2020a;c), and to vision-language tasks (Lu et al., 2019; Tan & Bansal, 2019; Chen et al., 2019; Su et al., 2019;

Original: [Table]
Revised: [Table]

Original: Li et al., 2019b; al
Revised: Li et al., 2019b; et al.,

Original: Li et al.,
Revised: Li et al.,

Original: Marrying Transformers with multi-stage architectures (Vaswani et al., 2021; Wang et al., 2021; Liu et al., 2021; Zhang et al., 2021; Wu et al., 2021) show higher classiﬁcation accuracy in supervised learning, and enables applicability of
Revised: Marrying Transformers with multi-stage architectures (Vaswani et al., 2021; Wang et al., 2021; Liu et al., 2021; Zhang et al., 2021; Wu et al., 2021) show higher classiﬁcation accuracy in supervised learning, and enables applicability of

Original: Transformers for a broader range of vision tasks.
Revised: Transformers for a broader range of vision tasks.

Original: Given these properties, we believe multi-stage ViT is a must-study baseline for SSL in computer vision.
Revised: Given these properties, we believe multi-stage ViT is a must-study baseline for SSL in computer vision.

Original: Discussion with other region-level tasks.
Revised: Discussion with other region-level tasks.

Original: In Table 7, we compare L R against the existing regionlevel tasks, including PIRL (Misra & Maaten, 2020), DenseCL (Wang et al., 2020b), DetCo (Xie et al., 2021a), InstLoc (Yang et al., 2021), PixPro (Xie et al., 2021c).
Revised: In Table 7, we compare L R against the existing regionlevel tasks, including PIRL (Misra & Maaten, 2020), DenseCL (Wang et al., 2020b), DetCo (Xie et al., 2021a), InstLoc (Yang et al., 2021), PixPro (Xie et al., 2021c).

Original: Most of these region-level tasks improve object detection tasks, but hurt the ImageNet classiﬁcation accuracy.
Revised: Most of these region-level tasks improve object detection tasks, but hurt the ImageNet classiﬁcation accuracy.

Original: DetCo achieves the best trade-off: improving the performance of both tasks, but with a sophisticated multi-level, global-local interaction algorithm.
Revised: DetCo achieves the best trade-off: improving the performance of both tasks, but with a sophisticated multi-level, global-local interaction algorithm.

Original: With the same number of pre-training epochs and augmented views, EsViT achieve the best ImageNet linear probe accuracy among all region-level tasks, with as minimum computational overhead as possible.
Revised: With the same number of pre-training epochs and augmented views, EsViT achieve the best ImageNet linear probe accuracy among all region-level tasks, with as minimum computational overhead as possible.

Original: This well serves our goal of building efﬁcient SSL
Revised: This well serves our goal of building efﬁcient SSL

Original: SoTA image classiﬁcation system.
Revised: SoTA image classiﬁcation system.

Original: [Table]
Revised: [Table]

Original: [Table][Table] [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: C E XPERIMENTS C. T HROUGHPUT ESTIMATE AND CONVERSION
Revised: C E XPERIMENTS C. T HROUGHPUT ESTIMATE AND CONVERSION

Original: Since different papers report throughput on different hardwares
Revised: Since different papers report throughput on different hardwares

Original: [Table]
Revised: [Table]

Original: it is not ready to compare the numbers directly.
Revised: it is not ready to compare the numbers directly.

Original: Noting that all papers report the throughput for ViT-B/DeiT-B, we use this number to align and convert the throughput.
Revised: Noting that all papers report the throughput for ViT-B/DeiT-B, we use this number to align and convert the throughput.

Original: In Table 8, we describe our process and results of standardizing the throughput.
Revised: In Table 8, we describe our process and results of standardizing the throughput.

Original: C. T HE COMPUTATION AND MEMORY OVERHEAD OF THE PROPOSED L R
Revised: C. T HE COMPUTATION AND MEMORY OVERHEAD OF THE PROPOSED L R

Original: We emphasize that adding L R to the multi-stage transformer architectures yields acceptable extra computational cost, while adding L R directly to the monolithic transformer architectures has a huge computational overhead.
Revised: We emphasize that adding L R to the multi-stage transformer architectures yields acceptable extra computational cost, while adding L R directly to the monolithic transformer architectures has a huge computational overhead.

Original: To demonstrate this, we report the cost comparisons in Table 9.
Revised: To demonstrate this, we report the cost comparisons in Table 9.

Original: For each setting, we report [Memory Usage (MB) /
Revised: For each setting, we report [Memory Usage (MB) /

Original: Running time per iteration (second/iteration)].
Revised: Running time per iteration (second/iteration)].

Original: In
Revised: In

Original: Table Table 9 (a), when the batch size is gradually increased ( e.g., , batch-size=12), the memory cost increases nearly 4 times for monolithic architectures, while increases 1.6 times for multi-stage architectures.
Revised: Table Table 9 (a), when the batch size is gradually increased ( e.g., , batch-size=12), the memory cost increases nearly 4 times for monolithic architectures, while increases 1.6 times for multi-stage architectures.

Original: Similar trends are shown for training cost per iteration increase ratio (1.47 vs 1.15).
Revised: Similar trends are shown for training cost per iteration increase ratio (1.47 vs 1.15).

Original: This indicates L R can more naturally ﬁt multi-stage architectures.
Revised: This indicates L R can more naturally ﬁt multi-stage architectures.

Original: Similarly, we compare computational cost comparisons [Memory Usage (MB) /
Revised: Similarly, we compare computational cost comparisons [Memory Usage (MB) /

Original: Running time per iteration (second/iteration)] in Table 9 (b), for other network architecture conﬁgurations .
Revised: Running time per iteration (second/iteration)] in Table 9 (b), for other network architecture conﬁgurations .

Original: From the increased cost ratio, we see that L R adds acceptable cost in terms of both memory and training time, compared with the baseline.
Revised: From the increased cost ratio, we see that L R adds acceptable cost in terms of both memory and training time, compared with the baseline.

Original: C. E XPERIMENTAL SETTINGS OF PRE - TRAINING AND EVALUATION ON I MAGE N ET
Revised: C. E XPERIMENTAL SETTINGS OF PRE - TRAINING AND EVALUATION ON I MAGE N ET

Original: We study unsupervised pre-training performed in ImageNet-1K dataset (Deng et al., 2009) without labels.
Revised: We study unsupervised pre-training performed in ImageNet-1K dataset (Deng et al., 2009) without labels.

Original: The default training details are described as follows, mostly following (Caron et al., 2021).
Revised: The default training details are described as follows, mostly following (Caron et al., 2021).

Original: We train with the Adamw optimizer (Loshchilov & Hutter, 2018), a batch size of 512 , and total epochs 300 .
Revised: We train with the Adamw optimizer (Loshchilov & Hutter, 2018), a batch size of 512 , and total epochs 300 .

Original: Linear warmup of the learning rate is used during the ﬁrst 10 epochs, with its base value
Revised: Linear warmup of the learning rate is used during the ﬁrst 10 epochs, with its base value

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: determined with the linear scaling rule (Goyal et al., 2017): lr = 0 .
Revised: determined with the linear scaling rule (Goyal et al., 2017): lr = 0 .

Original: 0005 ∗ batchsize / 256 .
Revised: 0005 ∗ batchsize / 256 .

Original: After this warmup, the learning rate is decayed with a cosine schedule.
Revised: After this warmup, the learning rate is decayed with a cosine schedule.

Original: We build our systems based on
Revised: We build our systems based on

Original: Swin Transformers (Liu et al., 2021) in our experiments.
Revised: Swin Transformers (Liu et al., 2021) in our experiments.

Original: Swin-B has a model size and computation complexity similar to ViT-B/DeiT-B (patch size 16).
Revised: Swin-B has a model size and computation complexity similar to ViT-B/DeiT-B (patch size 16).

Original: We also considered Swin-T and Swin-S, which have the complexity that are similar to those of ResNet-50 (DeiT-S) and ResNet-101, respectively.
Revised: We also considered Swin-T and Swin-S, which have the complexity that are similar to those of ResNet-50 (DeiT-S) and ResNet-101, respectively.

Original: The default window size is W =7 .
Revised: The default window size is W =7 .

Original: One major common protocol to evaluate SSL is linear probe on ImageNet-1K, where features are extracted from a frozen backbone, and a supervised linear classiﬁer is trained.
Revised: One major common protocol to evaluate SSL is linear probe on ImageNet-1K, where features are extracted from a frozen backbone, and a supervised linear classiﬁer is trained.

Original: For all Transformer models, we use the concatenation of view-level features ¯ z in the last 4 layers (the results are similar to the use of last 3 or 5 layers in our initial experiments).
Revised: For all Transformer models, we use the concatenation of view-level features ¯ z in the last 4 layers (the results are similar to the use of last 3 or 5 layers in our initial experiments).

Original: [Table] [Table]
Revised: [Table] [Table]

Original: C. C OMPARISON WITH A R ES N ET -50 BACKBONE
Revised: C. C OMPARISON WITH A R ES N ET -50 BACKBONE

Original: To compare our EsViT learning method with other SSL algorithms
Revised: To compare our EsViT learning method with other SSL algorithms

Original: [Table]
Revised: [Table]

Original: we conduct experiments with a
Revised: we conduct experiments with a

Original: ResNet-50 backbone, and show the results in Table 12.
Revised: ResNet-50 backbone, and show the results in Table [Table]

Original: C. L INEAR PROBE ON A SUITE OF SMALL DATASETS
Revised: C. L INEAR PROBE ON A SUITE OF SMALL DATASETS

Original: Datasets.
Revised: Datasets.

Original: Table 10 shows details and source of all datasets used for linear probe, including the number of classes, the size of training set and testing set, metrics used in evaluation, as well as a public source of the dataset.
Revised: Table 10 shows details and source of all datasets used for linear probe, including the number of classes, the size of training set and testing set, metrics used in evaluation, as well as a public source of the dataset.

Original: Note that original UCF101 dataset is a video dataset.
Revised: Note that original UCF101 dataset is a video dataset.

Original: Here the middle 
Revised: Here the middle frame

Original: Automatic hyper-parameter tuning.
Revised: Automatic hyper-parameter tuning.

Original: We rigorously follow (Radford et al., 2021) to conduct training and evaluation for linear probe on the downstream datasets.
Revised: We rigorously follow (Radford et al., 2021) to conduct training and evaluation for linear probe on the downstream datasets.

Original: We train a logistic regression classiﬁer using scikit-learn’s L-BFGS implementation, with maximum 1 , 000 iterations, and report the corresponding metric for each dataset.
Revised: We train a logistic regression classiﬁer using scikit-learn’s L-BFGS implementation, with maximum 1 , 000 iterations, and report the corresponding metric for each dataset.

Original: We determine the L 2 regularization strength λ using a hyperparameter sweep on the validation sets over the range between 10 − 6 and 10 6 , withlogarithmically spaced steps.
Revised: We determine the L 2 regularization strength λ using a hyperparameter sweep on the validation sets over the range between 10 − 6 and 10 6 , withlogarithmically spaced steps.

Original: To save compute required for the sweeps, we perform a parametric binary search that starts with λ =
Revised: To save compute required for the sweeps, we perform a parametric binary search that starts with λ =

Original: [10 − 6 , 10 − 4 , 10 − 2 , 1 , 10 2 , 10 4 , 10 6 ] and iteratively halves the interval around the peak until it reaches a resolution of 8 steps per decade.
Revised: [10 − 6 , 10 − 4 , 10 − 2 , 1 , 10 2 , 10 4 , 10 6 ] and iteratively halves the interval around the peak until it reaches a resolution of 8 steps per decade.

Original: The hyperparameter sweeps are performed on a validation split of each dataset.
Revised: The hyperparameter sweeps are performed on a validation split of each dataset.

Original: For the datasets that contain a validation split in addition to a test split, we use the provided validation set to perform the hyperparameter search, and for the datasets that do not provide a validation split or have not published labels for the test data, we split the training dataset to perform the hyperparameter search.
Revised: For the datasets that contain a validation split in addition to a test split, we use the provided validation set to perform the hyperparameter search, and for the datasets that do not provide a validation split or have not published labels for the test data, we split the training dataset to perform the hyperparameter search.

Original: For the ﬁnal result, we combine the validation split back with the training split and report the performance on the unused split.
Revised: For the ﬁnal result, we combine the validation split back with the training split and report the performance on the unused split.

Original: Detailed results.
Revised: Detailed results.

Original: Only the last layer feature is considered for all models for simplicity, though adding features from more layers may potentially improve the results.
Revised: Only the last layer feature is considered for all models for simplicity, though adding features from more layers may potentially improve the results.

Original: Table 11 shows the results for architectures at a similar scale of ResNet-50 or Swin-T.
Revised: Table 11 shows the results for architectures at a similar scale of ResNet-50 or Swin-T.

Original: The ﬁrst two columns are numbers from (Radford et al., 2021).
Revised: The ﬁrst two columns are numbers from (Radford et al., 2021).

Original: CLIP with ResNet-50 is pre-trained on 400 million image-text pairs.
Revised: CLIP with ResNet-50 is pre-trained on 400 million image-text pairs.

Original: Supervised ResNet-50 and Swin-T are pre-trained on ImageNet-1K, on which EsViT with Swin-T is pre-trained as well (Batch Size=512).
Revised: Supervised ResNet-50 and Swin-T are pre-trained on ImageNet-1K, on which EsViT with Swin-T is pre-trained as well (Batch Size=512).

Original: EsViT outperforms its supervised counterpart, and is on par with the performance of CLIP in a similar image encoder architecture scale.
Revised: EsViT outperforms its supervised counterpart, and is on par with the performance of CLIP in a similar image encoder architecture scale.

Original: C. P RE - TRAINING DATASETS
Revised: C. P RE - TRAINING DATASETS

Original: We describe the statistics and training schedule on larger and less curated datasets in Table 13.
Revised: We describe the statistics and training schedule on larger and less curated datasets in Table 13.

Original: The pre-training epochs are chosen so that the model is trained with a similar number of augmented views.
Revised: The pre-training epochs are chosen so that the model is trained with a similar number of augmented views.

Original: [Table]
Revised: [Table]

Original: C. R ESULTS ON CORRESPONDENCE LEARNING
Revised: C. R ESULTS ON CORRESPONDENCE LEARNING

Original: We ﬁrst quantitatively evaluate the correspondence learning results with 50K images in the ImageNet validation dataset.
Revised: We ﬁrst quantitatively evaluate the correspondence learning results with 50K images in the ImageNet validation dataset.

Original: We create a simple evaluation dataset with mild augmentations.
Revised: We create a simple evaluation dataset with mild augmentations.

Original: For a center-crop image, we apply HorizontalFlip , then ColorJitter and RandomGrayscale to create a new augmented view.
Revised: For a center-crop image, we apply HorizontalFlip , then ColorJitter and RandomGrayscale to create a new augmented view.

Original: In this way, ground-truth correspondences are created.
Revised: In this way, ground-truth correspondences are created.

Original: Please see the 1st row of Figure 9 for one such example.
Revised: Please see the 1st row of Figure 9 for one such example.

Original: The top-10 correspondences are used for evaluation.
Revised: The top-10 correspondences are used for evaluation.

Original: Two metrics are considered: (1) Accuracy measures the percentage of correctly matched region pairs, (2) distance error indicates the averaged (cid:96) 2 distance between the predicted matched region and ground-truth region (the value is 0 for perfect matching).
Revised: Two metrics are considered: (1) Accuracy measures the percentage of correctly matched region pairs, (2) distance error indicates the averaged (cid:96) 2 distance between the predicted matched region and ground-truth region (the value is 0 for perfect matching).

Original: The results are reported in Figure 8.
Revised: The results are reported in Figure 8.

Original: DINO with monolithic Transformers shows surprisingly good performance on correspondence learning.
Revised: DINO with monolithic Transformers shows surprisingly good performance on correspondence learning.

Original: The use of multi-stage Transformer architecture reduces this ability, shows a lack of good region correspondence.
Revised: The use of multi-stage Transformer architecture reduces this ability, shows a lack of good region correspondence.

Original: With L R , the region matching ability is signiﬁcantly recovered.
Revised: With L R , the region matching ability is signiﬁcantly recovered.

Original: In Figure 9, we visualize the correspondences for more images.
Revised: In Figure 9, we visualize the correspondences for more images.

Original: Overall, DINO with monolithic
Revised: Overall, DINO with monolithic

Original: Transformers is able to discover most salient correspondences of semantic meaning in the mild augmentation conditions, even without an implicit region matching loss in training.
Revised: Transformers is able to discover most salient correspondences of semantic meaning in the mild augmentation conditions, even without an implicit region matching loss in training.

Original: We believe this previously underestimated property is whole-noting, and has potentials to enable more applications.
Revised: We believe this previously underestimated property is whole-noting, and has potentials to enable more applications.

Original: However, this desired property gets dilated when changing from monolithic to multi-stage Transformer architecture (from column 1 to column 2), then the proposed region level task can alleviate this issue (from column 2 to column 3).
Revised: However, this desired property gets dilated when changing from monolithic to multi-stage Transformer architecture (from column 1 to column 2), then the proposed region level task can alleviate this issue (from column 2 to column 3).

Original: To more speciﬁcally analyze the correspondences, we note the following results.
Revised: To more speciﬁcally analyze the correspondences, we note the following results.

Original: The ﬁrst row shows a simple case, where only images of left-to-right ﬂipped views are presented.
Revised: The ﬁrst row shows a simple case, where only images of left-to-right ﬂipped views are presented.

Original: The ground-truth
Revised: The ground-truth

Original: [Figure]
Revised: [Figure]

Original: correspondences should be horizontal lines that link the two ﬂipped regions.
Revised: correspondences should be horizontal lines that link the two ﬂipped regions.

Original: It reveals that the view-level pre-train task alone is insufﬁcient to learn good correspondences for the multi-stage
Revised: It reveals that the view-level pre-train task alone is insufﬁcient to learn good correspondences for the multi-stage

Original: Transformer architecture, while region matching task can alleviate this issue signiﬁcantly.
Revised: Transformer architecture, while region matching task can alleviate this issue signiﬁcantly.

Original: Similar observations are shown in row 3 and row 4.
Revised: Similar observations are shown in row 3 and row 4.

Original: We further study more cases that requires real-world correspondences in row 2, row 5 and row 6.
Revised: We further study more cases that requires real-world correspondences in row 2, row 5 and row 6.

Original: These views are not generated with data augmentation (as in model pre-training), but are often presented in more practical scenarios: one-to-many mappings, cartoon-to-toy, seasonal changing of the scene, respectively.
Revised: These views are not generated with data augmentation (as in model pre-training), but are often presented in more practical scenarios: one-to-many mappings, cartoon-to-toy, seasonal changing of the scene, respectively.

Original: The proposed region matching task can work particularly well in those cases.
Revised: The proposed region matching task can work particularly well in those cases.

Original: C. M ORE VISUALIZATION RESULTS OF ATTENTION MAPS
Revised: C. M ORE VISUALIZATION RESULTS OF ATTENTION MAPS

Original: We visualize attention maps at the top layer in Figure 10, 11, 12.
Revised: We visualize attention maps at the top layer in Figure 10, 11, 12.

Original: With a monolithic Transformer architecture, DINO can automatically identify the main foreground objects.
Revised: With a monolithic Transformer architecture, DINO can automatically identify the main foreground objects.

Original: Unfortunately, changing from monolithic to the multi-stage Transformer architecture (From left column to middle column), this property gets lost.
Revised: Unfortunately, changing from monolithic to the multi-stage Transformer architecture (From left column to middle column), this property gets lost.

Original: There are more heads in the multi-stage architecture than monolithic architecture (24 heads vs 6 heads in this case) in the last year.
Revised: There are more heads in the multi-stage architecture than monolithic architecture (24 heads vs 6 heads in this case) in the last year.

Original: A fair number of heads in EsViT shows redundant patterns, this issue can be reduced when the region-level matching task is added (From middle column to right column).
Revised: A fair number of heads in EsViT shows redundant patterns, this issue can be reduced when the region-level matching task is added (From middle column to right column).

Original: We observed that DINO with monolithic Transformer architecture only learns to attend the foreground objects, even when the query is a background region (see Figure 12).
Revised: We observed that DINO with monolithic Transformer architecture only learns to attend the foreground objects, even when the query is a background region (see Figure 12).

Original: This is perhaps because
Revised: This is perhaps because

Original: DINO models are trained to learn view-level invariance, the main objects in the pre-train dataset
Revised: DINO models are trained to learn view-level invariance, the main objects in the pre-train dataset

Original: ImageNet tend to be the principle factor that remains invariant across different augmented views.
Revised: ImageNet tend to be the principle factor that remains invariant across different augmented views.

Original: Hence, all backgrounds are ignored, regardless of the query positions.
Revised: Hence, all backgrounds are ignored, regardless of the query positions.

Original: This is improved in EsViT with the region-level pre-train task, as the model is trained to match individual regions.
Revised: This is improved in EsViT with the region-level pre-train task, as the model is trained to match individual regions.

Original: DINO shows high entropy values in all of 6 heads (perhaps a required condition to cover all regions of the main object).
Revised: DINO shows high entropy values in all of 6 heads (perhaps a required condition to cover all regions of the main object).

Original: In EsViT, L R plays an interesting role in modulating the entropy distributions among heads: it increases those with larger entropy values, while decreasing those with lower entropy values.
Revised: In EsViT, L R plays an interesting role in modulating the entropy distributions among heads: it increases those with larger entropy values, while decreasing those with lower entropy values.

Original: In another word, it makes the attention patterns in different heads more diverse.
Revised: In another word, it makes the attention patterns in different heads more diverse.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]