Original: M OSAIC R EPRESENTATION L EARNING FOR S ELF - SUPERVISED V ISUAL P RE - TRAINING
Revised: M OSAIC R EPRESENTATION L EARNING FOR S ELF - SUPERVISED V ISUAL P RE - TRAINING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Self-supervised learning has achieved significant success in learning visual representations without the need of manual annotation.
Revised: Self-supervised learning has achieved significant success in learning visual representations without the need of manual annotation.

Original: To obtain generalizable representations, a meticulously designed data augmentation strategy is one of the most crucial parts.
Revised: To obtain generalizable representations, a meticulously designed data augmentation strategy is one of the most crucial parts.

Original: Recently, multi-crop strategies utilizing a set of small crops as positive samples have been shown to learn spatially structured features.
Revised: Recently, multi-crop strategies utilizing a set of small crops as positive samples have been shown to learn spatially structured features.

Original: However, it overlooks the diversity of contextual backgrounds, which reduces the variance of the input views and degenerates the performance.
Revised: However, it overlooks the diversity of contextual backgrounds, which reduces the variance of the input views and degenerates the performance.

Original: To address this problem, we propose a mosaic representation learning framework (MosRep), consisting of a new data augmentation strategy that enriches the backgrounds of each small crop and improves the quality of visual representations.
Revised: To address this problem, we propose a mosaic representation learning framework (MosRep), consisting of a new data augmentation strategy that enriches the backgrounds of each small crop and improves the quality of visual representations.

Original: Specifically, we randomly sample numbers of small crops from different input images and compose them into a mosaic view, which is equivalent to introducing different background information for each small crop.
Revised: Specifically, we randomly sample numbers of small crops from different input images and compose them into a mosaic view, which is equivalent to introducing different background information for each small crop.

Original: Additionally, we further jitter the mosaic view to prevent memorizing the spatial locations of each crop.
Revised: Additionally, we further jitter the mosaic view to prevent memorizing the spatial locations of each crop.

Original: Along with optimization, our MosRep gradually extracts more discriminative features.
Revised: Along with optimization, our MosRep gradually extracts more discriminative features.

Original: Extensive experimental results demonstrate that our method improves the performance far greater than the multicrop strategy on a series of downstream tasks, e.g., +7 .
Revised: Extensive experimental results demonstrate that our method improves the performance far greater than the multicrop strategy on a series of downstream tasks, e.g., +7 .

Original: and +4 .
Revised: and +4 .

Original: 9% than the multi-crop strategy on ImageNet-1K with 1% label and 10% label, respectively.
Revised: 9% than the multi-crop strategy on ImageNet-1K with 1% label and 10% label, respectively.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: High-quality representation learning (Bengio et al., 2013) is a fundamental task in machine learning.
Revised: High-quality representation learning (Bengio et al., 2013) is a fundamental task in machine learning.

Original: Tremendous number of visual recognition models have achieved promising performance by learning from large-scale annotated datasets, e.g., ImageNet (Deng et al., 2009) and OpenImage (Kuznetsova et al., 2020).
Revised: Tremendous number of visual recognition models have achieved promising performance by learning from large-scale annotated datasets, e.g., ImageNet (Deng et al., 2009) and OpenImage (Kuznetsova et al., 2020).

Original: However, a great deal of challenges exist in collecting large-scale datasets with annotations, e.g., label noise (Liu & Tao, 2015; Natarajan et al., 2013), high cost (Zhu et al., 2019) and privacy concerns (Liang et al., 2020).
Revised: However, a great deal of challenges exist in collecting large-scale datasets with annotations, e.g., label noise (Liu & Tao, 2015; Natarajan et al., 2013), high cost (Zhu et al., 2019) and privacy concerns (Liang et al., 2020).

Original: To address these issues, self-supervised learning (SSL) is proposed to learn generic representations without manual annotation.
Revised: To address these issues, self-supervised learning (SSL) is proposed to learn generic representations without manual annotation.

Original: Recent progress in visual self-supervised learning (Caron et al., 2020;
Revised: Recent progress in visual self-supervised learning (Caron et al., 2020;

Original: He et al., 2020; Grill et al., 2020; Chen & He, 2021) shows remarkable potential and achieves comparable results with supervised learning.
Revised: He et al., 2020; Grill et al., 2020; Chen & He, 2021) shows remarkable potential and achieves comparable results with supervised learning.

Original: Among these SSL methods, a common underlying idea is to extract invariant feature representations from different augmented views of the same input image.
Revised: Among these SSL methods, a common underlying idea is to extract invariant feature representations from different augmented views of the same input image.

Original: Contrastive learning (Dosovitskiy et al., 2015; Wu et al., 2018;
Revised: Contrastive learning (Dosovitskiy et al., 2015; Wu et al., 2018;

Original: Chen et al., 2020a; He et al., 2020) is one of the most commonly used methods.
Revised: Chen et al., 2020a; He et al., 2020) is one of the most commonly used methods.

Original: They define ‘positive’ and ‘negative’ pairs and apply the contrastive loss ( i.e., InfoNCE (H´enaff et al., 2019)) for optimization, where the ‘positive’ pairs are pulled close and the ‘negative’ pairs are pushed away.
Revised: They define ‘positive’ and ‘negative’ pairs and apply the contrastive loss ( i.e., InfoNCE (H´enaff et al., 2019)) for optimization, where the ‘positive’ pairs are pulled close and the ‘negative’ pairs are pushed away.

Original: Another trend of work, such as BYOL (Grill et al., 2020) and Simsiam (Chen & He, 2021), introduces the concept of asymmetry, which is free from designing negatives.
Revised: Another trend of work, such as BYOL (Grill et al., 2020) and Simsiam (Chen & He, 2021), introduces the concept of asymmetry, which is free from designing negatives.

Original: They add an extra ‘predictor’ behind the model and update the parameters using one augmented view, while the feature of another augmented view is used as fixed supervision.
Revised: They add an extra ‘predictor’ behind the model and update the parameters using one augmented view, while the feature of another augmented view is used as fixed supervision.

Original: Besides, clustering methods (Caron et al., 2018; 2020; Asano et al., 2019; Li et al., 2020) adopt two augmented views of the same image as the prediction and the pseudo cluster label and enforce the consistency between the two views.
Revised: Besides, clustering methods (Caron et al., 2018; 2020; Asano et al., 2019; Li et al., 2020) adopt two augmented views of the same image as the prediction and the pseudo cluster label and enforce the consistency between the two views.

Original: We present more related works in the appendix.
Revised: We present more related works in the appendix.

Original: It is worth noting that a carefully-designed data augmentation strategy is an essential part of the above self-supervised learning frameworks.
Revised: It is worth noting that a carefully-designed data augmentation strategy is an essential part of the above self-supervised learning frameworks.

Original: SimCLR (Tian et al., 2020) and InfoMin (Chen et al., 2020a) empirically investigate the impact of different data augmentations and observe that SSL benefits
Revised: SimCLR (Tian et al., 2020) and InfoMin (Chen et al., 2020a) empirically investigate the impact of different data augmentations and observe that SSL benefits

Original: [Figure]
Revised: [Figure]

Original: Qualitatively, MosRep performs better localization methods, demonstrating our method effectively extracts features learns more from strong data augmentations than supervised learning.
Revised: Qualitatively, MosRep performs better localization methods, demonstrating our method effectively extracts features learns more from strong data augmentations than supervised learning.

Original: After that, SwAV (Caron et al., 2020) proposes the multi-crop strategy, which achieves significant performances on downstream tasks.
Revised: After that, SwAV (Caron et al., 2020) proposes the multi-crop strategy, which achieves significant performances on downstream tasks.

Original: As shown in Figure 1 (a) and (b), they use two standard resolution crops and sample several small crops that cover the local regions of the input image in order to encourage the “local-to-global” correspondences.
Revised: As shown in Figure 1 (a) and (b), they use two standard resolution crops and sample several small crops that cover the local regions of the input image in order to encourage the “local-to-global” correspondences.

Original: However, small crops overlook the diverse backgrounds and decrease the variance, where such views with too many similarities are trivial for learning discriminative features.
Revised: However, small crops overlook the diverse backgrounds and decrease the variance, where such views with too many similarities are trivial for learning discriminative features.

Original: Intuitively, if we can take into account both the “local-to-global” correspondences and the diverse contextual backgrounds, the quality of learned representations can be further improved.
Revised: Intuitively, if we can take into account both the “local-to-global” correspondences and the diverse contextual backgrounds, the quality of learned representations can be further improved.

Original: In this paper, we propose a mosaic representation learning framework (MosRep) consisting of a new data augmentation strategy, which can enrich the contextual background of each small crop and encourage the “local-to-global” correspondences.
Revised: In this paper, we propose a mosaic representation learning framework (MosRep) consisting of a new data augmentation strategy, which can enrich the contextual background of each small crop and encourage the “local-to-global” correspondences.

Original: Specifically, we first sample M ( e.g., M = 4 ) small crops of each input image in the current batch.
Revised: Specifically, we first sample M ( e.g., M = 4 ) small crops of each input image in the current batch.

Original: Then, these crops are randomly shuffled and divided into multiple groups.
Revised: Then, these crops are randomly shuffled and divided into multiple groups.

Original: Each group contains M crops and we also ensure that small crops in each group are from different input images.
Revised: Each group contains M crops and we also ensure that small crops in each group are from different input images.

Original: Subsequently, as illustrated in Figure 1 (c), we combine the small crops of the same group into a single view, which terms the mosaic view.
Revised: Subsequently, as illustrated in Figure 1 (c), we combine the small crops of the same group into a single view, which terms the mosaic view.

Original: Finally, we further jitter the mosaic view in order to prevent the model from memorizing the spatial position of each small crop.
Revised: Finally, we further jitter the mosaic view in order to prevent the model from memorizing the spatial position of each small crop.

Original: In the forward process, the mosaic view is fed into the model for feature extraction, and we adopt the RoI Align operator to extract the feature of each crop from the mosaic view and project this feature into an embedding space.
Revised: In the forward process, the mosaic view is fed into the model for feature extraction, and we adopt the RoI Align operator to extract the feature of each crop from the mosaic view and project this feature into an embedding space.

Original: To minimize the loss function ( e.g., contrastive loss), the model gradually learns to capture more discriminative features ( i.e., foreground objects) from the complex backgrounds, improving the quality of visual representations, which is shown in Figure(d).
Revised: To minimize the loss function ( e.g., contrastive loss), the model gradually learns to capture more discriminative features ( i.e., foreground objects) from the complex backgrounds, improving the quality of visual representations, which is shown in Figure(d).

Original: In summary, the main contributions of this paper are as follows:
Revised: In summary, the main contributions of this paper are as follows:

Original: We design a mosaic augmentation strategy that takes into account the diversity of backgrounds and the “local-to-global” correspondences.
Revised: We design a mosaic augmentation strategy that takes into account the diversity of backgrounds and the “local-to-global” correspondences.

Original: Based on the proposed mosaic augmentation, we propose a practical and effective SSL framework, MosRep, which benefits the extraction of discriminative features and improves the quality of visual representations.
Revised: Based on the proposed mosaic augmentation, we propose a practical and effective SSL framework, MosRep, which benefits the extraction of discriminative features and improves the quality of visual representations.

Original: We build our proposed method upon two different SSL frameworks and validate the effectiveness.
Revised: We build our proposed method upon two different SSL frameworks and validate the effectiveness.

Original: Experimental results show that our method achieves superior gains compared to the multi-crop strategy on various downstream tasks.
Revised: Experimental results show that our method achieves superior gains compared to the multi-crop strategy on various downstream tasks.

Original:  M ETHODS
Revised:  M ETHODS

Original: We propose MosRep, a framework to adequately facilitate the learning of discriminative features from large-scale unlabeled datasets.
Revised: We propose MosRep, a framework to adequately facilitate the learning of discriminative features from large-scale unlabeled datasets.

Original: In this section, we first revisit the preliminaries on contrastive learning and the multi-crop strategy.
Revised: In this section, we first revisit the preliminaries on contrastive learning and the multi-crop strategy.

Original: Then we present our mosaic representation learning framework in detail.
Revised: Then we present our mosaic representation learning framework in detail.

Original: [Figure]
Revised: [Figure]

Original: 2. P RELIMINARY
Revised: 2. P RELIMINARY

Original: Contrastive learning.
Revised: Contrastive learning.

Original: Contrastive learning (Dosovitskiy et al., 2015; Chen et al., 2020a;
Revised: Contrastive learning (Dosovitskiy et al., 2015; Chen et al., 2020a;

Original: He et al., 2020) is one of the most popular self-supervised learning frameworks and has achieved great success in recent years.
Revised: He et al., 2020) is one of the most popular self-supervised learning frameworks and has achieved great success in recent years.

Original: Given a set of images X , the goal of contrastive learning is to learn an embedding space such that the embedding of each image x i can be distinguished from a set of negatives N .
Revised: Given a set of images X , the goal of contrastive learning is to learn an embedding space such that the embedding of each image x i can be distinguished from a set of negatives N .

Original: Firstly, two separate augmentations t q and t k are sampled from a set of pre-defined augmentations
Revised: Firstly, two separate augmentations t q and t k are sampled from a set of pre-defined augmentations

Original: T and applied to obtain two different views of each image, i.e., x qi =
Revised: T and applied to obtain two different views of each image, i.e., x qi =

Original: t q ( x i ) , x ki = t k
Revised: t q ( x i ) , x ki = t k

Original: ( x i ) .
Revised: ( x i ) .

Original: Then, these views are fed into an encoder F ( · ) to extract features h and a projector g ( · ) to map features h into a embedding space, i.e., z qi =
Revised: Then, these views are fed into an encoder F ( · ) to extract features h and a projector g ( · ) to map features h into a embedding space, i.e., z qi =

Original: g ( F ( x qi )) , z ki = g ( F ( x ki )) .
Revised: g ( F ( x qi )) , z ki = g ( F ( x ki )) .

Original: Finally, a contrastive objective ( e.g.,
Revised: Finally, a contrastive objective ( e.g.,

Original: InfoNCE) is formulated as:
Revised: InfoNCE) is formulated as:

Original: [Equation]
Revised: [Equation]

Original: where n − , τ , N and sim ( · ) denote a negative sample, a temperature parameter, the number of input images and cosine similarity, respectively.
Revised: where n − , τ , N and sim ( · ) denote a negative sample, a temperature parameter, the number of input images and cosine similarity, respectively.

Original: Besides, without the need of negatives, BYOL (Grill et al., 2020) and Simsiam (Chen & He, 2021) adopt an extra predictor to map the embedding z to the prediction p and minimize their negative cosine similarity of them, i.e.,
Revised: Besides, without the need of negatives, BYOL (Grill et al., 2020) and Simsiam (Chen & He, 2021) adopt an extra predictor to map the embedding z to the prediction p and minimize their negative cosine similarity of them, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where sg ( · ) denotes the stop-gradient trick that is crucial to avoid model collapsing.
Revised: where sg ( · ) denotes the stop-gradient trick that is crucial to avoid model collapsing.

Original: Multi-crop strategy.
Revised: Multi-crop strategy.

Original: SwAV (Caron et al., 2020) proposes a multi-crop strategy using two highresolution views that cover large parts of the image and several low-resolution views that cover small parts of the image.
Revised: SwAV (Caron et al., 2020) proposes a multi-crop strategy using two highresolution views that cover large parts of the image and several low-resolution views that cover small parts of the image.

Original: In doing so, this strategy maximizes the agreement between a high-resolution (global) view and a low-resolution (local) view of the same image, encouraging the model to learn spatially structured representations.
Revised: In doing so, this strategy maximizes the agreement between a high-resolution (global) view and a low-resolution (local) view of the same image, encouraging the model to learn spatially structured representations.

Original: 2. M OSAIC R EPRESENTATION L EARNING
Revised: 2. M OSAIC R EPRESENTATION L EARNING

Original: In the multi-crop strategy, the features of low-resolution and high-resolution crops are enforced to remain constant, which encourages the “local-to-global” correspondences.
Revised: In the multi-crop strategy, the features of low-resolution and high-resolution crops are enforced to remain constant, which encourages the “local-to-global” correspondences.

Original: However, the lowresolution crops only cover small parts of the input images, which overlooks the diverse background information and reduces the variance of these crops.
Revised: However, the lowresolution crops only cover small parts of the input images, which overlooks the diverse background information and reduces the variance of these crops.

Original: Because of this issue, the model is not able to learn sufficient discriminative features, which can cause the degeneration of performance on various downstream tasks.
Revised: Because of this issue, the model is not able to learn sufficient discriminative features, which can cause the degeneration of performance on various downstream tasks.

Original: Motivated by this, we propose a mosaic representation learning framework (MosRep), which consists of a new mosaic augmentation strategy that enriches the backgrounds of each small crop and improves the quality of visual representations.
Revised: Motivated by this, we propose a mosaic representation learning framework (MosRep), which consists of a new mosaic augmentation strategy that enriches the backgrounds of each small crop and improves the quality of visual representations.

Original: Concretely, given an input image x i , we generate two standard views and M small crops by three separate augmentation operators t q , t k and t s , i.e.,
Revised: Concretely, given an input image x i , we generate two standard views and M small crops by three separate augmentation operators t q , t k and t s , i.e.,

Original: [Equation]
Revised: [Equation]

Original: where x qi and x ki denote two standard view and X si denotes a set of small crops.
Revised: where x qi and x ki denote two standard view and X si denotes a set of small crops.

Original: As shown in “Small
Revised: As shown in “Small

Original: Crops” of Figure 2, we randomly shuffle all small crops from images in a batch and divide them into groups.
Revised: Crops” of Figure 2, we randomly shuffle all small crops from images in a batch and divide them into groups.

Original: We set up M crops in each group and ensure that the crops in each group come from different input images.
Revised: We set up M crops in each group and ensure that the crops in each group come from different input images.

Original: Then, we compose the crops of each group into a single view, termed as the mosaic view x vi , and record the coordinates ( t, l, b, r ) of each small crop relative to x vi , where t, l, b, r indicates the top, left, bottom and right position.
Revised: Then, we compose the crops of each group into a single view, termed as the mosaic view x vi , and record the coordinates ( t, l, b, r ) of each small crop relative to x vi , where t, l, b, r indicates the top, left, bottom and right position.

Original: This process is formulated as,
Revised: This process is formulated as,

Original: [Equation]
Revised: [Equation]

Original: where M i denotes a group of shuffled small crops, N denotes all indices of input images and j denotes the index of small crops in a group.
Revised: where M i denotes a group of shuffled small crops, N denotes all indices of input images and j denotes the index of small crops in a group.

Original: That is, x sij is the j -th crop in the set X si .
Revised: That is, x sij is the j -th crop in the set X si .

Original: Since the spatial position of each crop is fixed in the mosaic view x vi , the model can easily memorize the position, resulting in over-fitting.
Revised: Since the spatial position of each crop is fixed in the mosaic view x vi , the model can easily memorize the position, resulting in over-fitting.

Original: In order to tackle this dilemma, we conduct the view jitter operation on the mosaic view.
Revised: In order to tackle this dilemma, we conduct the view jitter operation on the mosaic view.

Original: We first sample offsets of the mosaic view from a beta distribution β ( α, α ) with two identical parameters α , i.e.,
Revised: We first sample offsets of the mosaic view from a beta distribution β ( α, α ) with two identical parameters α , i.e.,

Original: [Equation]
Revised: [Equation]

Original: where θ , ∆ x and ∆ y denote the upper bound of jitter range and the offsets of the mosaic view, respectively.
Revised: where θ , ∆ x and ∆ y denote the upper bound of jitter range and the offsets of the mosaic view, respectively.

Original: We set α < 1 , which indicates a U-shaped distribution.
Revised: We set α < 1 , which indicates a U-shaped distribution.

Original: In this way the mosaic view is more likely to be jittered in a relatively wider range with larger offsets.
Revised: In this way the mosaic view is more likely to be jittered in a relatively wider range with larger offsets.

Original: Meanwhile, we update the coordinates of each small crop with these offsets, which is calculated as,
Revised: Meanwhile, we update the coordinates of each small crop with these offsets, which is calculated as,

Original: [Equation]
Revised: [Equation]

Original: In doing so, the mosaic view effectively enriches the background information of each small crop and facilitates the extraction of discriminative features, improving the quality of learned representations.
Revised: In doing so, the mosaic view effectively enriches the background information of each small crop and facilitates the extraction of discriminative features, improving the quality of learned representations.

Original: Subsequently, we present the forward propagation of our proposed framework in the lower part of
Revised: Subsequently, we present the forward propagation of our proposed framework in the lower part of

Original: Figure 2.
Revised: Figure 2.

Original: Given two standard views x qi , x ki and a mosaic view x vi , an encoder F ( · ) is used to extract the feature h of them, i.e.,
Revised: Given two standard views x qi , x ki and a mosaic view x vi , an encoder F ( · ) is used to extract the feature h of them, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where h qi , h ki and h vi denote the feature of two standard views and a mosaic view, respectively.
Revised: where h qi , h ki and h vi denote the feature of two standard views and a mosaic view, respectively.

Original: It is worth noting that, by resorting to the above-mentioned coordinates, we adopt a RoI Align operator to extract the feature of each small crop in the mosaic view.
Revised: It is worth noting that, by resorting to the above-mentioned coordinates, we adopt a RoI Align operator to extract the feature of each small crop in the mosaic view.

Original: According to the index i of the input images, we rearrange the features of these small crops to correspond to their positive samples, and use H si to denote a set of features of small crops in i -th mosaic view.
Revised: According to the index i of the input images, we rearrange the features of these small crops to correspond to their positive samples, and use H si to denote a set of features of small crops in i -th mosaic view.

Original: After that, all features are mapped into an embedding space by a projector g ( · ) , i.e.,
Revised: After that, all features are mapped into an embedding space by a projector g ( · ) , i.e.,

Original: [Equation]
Revised: [Equation]

Original: (8) where z qi , z ki are the embedding of two standard views, and Z si denotes a set of embeddings of small crops.
Revised: (8) where z qi , z ki are the embedding of two standard views, and Z si denotes a set of embeddings of small crops.

Original: Finally, we define the positive and negative pairs of all embeddings based on the index i of the input images and calculate the contrastive loss, i.e.,
Revised: Finally, we define the positive and negative pairs of all embeddings based on the index i of the input images and calculate the contrastive loss, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where z sij denotes the j -th embedding in the set Z si , Z k denotes a set of embeddings of the key view and n − denotes negative samples in the set Z k and N .
Revised: where z sij denotes the j -th embedding in the set Z si , Z k denotes a set of embeddings of the key view and n − denotes negative samples in the set Z k and N .

Original: The first component in Eq.
Revised: The first component in Eq.

Original: 9 is the standard contrastive loss used in (Chen et al., 2020b).
Revised: 9 is the standard contrastive loss used in (Chen et al., 2020b).

Original: And we design the second component in Eq.to explicitly pull the small crops and its corresponding key view closer and push other irrelevance away.
Revised: And we design the second component in Eq.to explicitly pull the small crops and its corresponding key view closer and push other irrelevance away.

Original: With optimization, the model gradually learns to extract more discriminative features, thus improving the quality of learned representations.
Revised: With optimization, the model gradually learns to extract more discriminative features, thus improving the quality of learned representations.

Original: Besides, we also build our proposed method upon the BYOL (Grill et al., 2020) framework by minimizing the negative cosine similarity between small crops and the key view.
Revised: Besides, we also build our proposed method upon the BYOL (Grill et al., 2020) framework by minimizing the negative cosine similarity between small crops and the key view.

Original: [Equation]
Revised: [Equation]

Original: where B and B ′ denote the batch of each rank and all ranks, p i 2 t c and p t 2 i c denote the image-to-text similarity and the text-to-image similarity of the CLIP model, p i 

Revised: where B and B ′ denote the batch of each rank and all ranks, p i 2 t c and p t 2 i c denote the image-to-text similarity and the text-to-image similarity of the CLIP model, p i

Original: i s and p
Revised: i s and p

Original: i 2 t s denote the image-to-image similarity and the image-to-text similarity between the small model and CLIP model, and τ T and τ S denote the temperature of the CLIP model and small model.
Revised: i 2 t s denote the image-to-image similarity and the image-to-text similarity between the small model and CLIP model, and τ T and τ S denote the temperature of the CLIP model and small model.

Original:  E XPERIMENTAL R ESULTS 3. P RE - TRAINING S ETTINGS
Revised:  E XPERIMENTAL R ESULTS 3. P RE - TRAINING S ETTINGS

Original: Datasets We perform self-supervised pre-training on two datasets, one middle-scale and another large-scale: 1) 100-category ImageNet (IN-100) (Tian et al., 2019), a subset of IN-1K dataset containing ∼ 125k images; and 2) 1000-category ImageNet (IN-1K)
Revised: Datasets We perform self-supervised pre-training on two datasets, one middle-scale and another large-scale: 1) 100-category ImageNet (IN-100) (Tian et al., 2019), a subset of IN-1K dataset containing ∼ 125k images; and 2) 1000-category ImageNet (IN-1K)

Original: (Deng et al., 2009), the standard
Revised: (Deng et al., 2009), the standard

Original: ImageNet training set containing
Revised: ImageNet training set containing

Original: Architectures Following the commonly-used setting in recent unsupervised methods (He et al., 2020; Chen et al., 2020a; Tian et al., 2019), we adopt the ResNet-

Revised: Architectures Following the commonly-used setting in recent unsupervised methods (He et al., 2020; Chen et al., 2020a; Tian et al., 2019), we adopt the ResNet-

Original: (He et al., 2016) model as our encoder.
Revised: (He et al., 2016) model as our encoder.

Original: To study the flexibility of our proposed method, we build on two different frameworks,
Revised: To study the flexibility of our proposed method, we build on two different frameworks,

Original: MoCo-v2 (Chen et al., 2020b) and BYOL (Grill et al., 2020).
Revised: MoCo-v2 (Chen et al., 2020b) and BYOL (Grill et al., 2020).

Original: Data Augmentation During pre-training, we adopt the data augmentation used in MoCo-v2 (He et al., 2020) and BYOL (Grill et al., 2020) for the standard view.
Revised: Data Augmentation During pre-training, we adopt the data augmentation used in MoCo-v2 (He et al., 2020) and BYOL (Grill et al., 2020) for the standard view.

Original: As for the mosaic view, we generate
Revised: As for the mosaic view, we generate

Original: M = 4 small crops with 112 × 112 input size, and other data augmentations are the same with
Revised: M = 4 small crops with 112 × 112 input size, and other data augmentations are the same with

Original: MoCo-v2 and BYOL.
Revised: MoCo-v2 and BYOL.

Original: Optimization Our hyperparameters closely follow these of the adopted self-supervised learning methods, MoCo-v2 (He et al., 2020) and BYOL (Grill et al., 2020).
Revised: Optimization Our hyperparameters closely follow these of the adopted self-supervised learning methods, MoCo-v2 (He et al., 2020) and BYOL (Grill et al., 2020).

Original: As for the MoCo version, we pre-train the network on IN-100 and IN-1K for 400 and 200 epochs, respectively.
Revised: As for the MoCo version, we pre-train the network on IN-100 and IN-1K for 400 and 200 epochs, respectively.

Original: SGD (Loshchilov & Hutter, 2016) optimizer with a cosine learning rate scheduler and lr base = 0 .
Revised: SGD (Loshchilov & Hutter, 2016) optimizer with a cosine learning rate scheduler and lr base = 0 .

Original: 3 is adopted, with a mini-batch size of 256 on 8 NVIDIA V100 GPU.
Revised: 3 is adopted, with a mini-batch size of 256 on 8 NVIDIA V100 GPU.

Original: We utilize a negative queue of 16,384 for IN-100, and 65,536 for IN-1K.
Revised: We utilize a negative queue of 16,384 for IN-100, and 65,536 for IN-1K.

Original: The weight decay is 0.0001 and SGD momentum is 0.9.
Revised: The weight decay is 0.0001 and SGD momentum is 0.9.

Original: As for the BYOL (Grill et al., 2020) version, we pre-train the network on IN-100 for 400 epochs, and on IN-1K for 200 epochs.
Revised: As for the BYOL (Grill et al., 2020) version, we pre-train the network on IN-100 for 400 epochs, and on IN-1K for 200 epochs.

Original: We adopt the SGD optimizer with lr base = 0 .
Revised: We adopt the SGD optimizer with lr base = 0 .

Original: 7 , where the learning rate is linearly scaled with the batch size as lr = lr base × batch/ 256 .
Revised: 7 , where the learning rate is linearly scaled with the batch size as lr = lr base × batch/ 256 .

Original: The batch size is set as 512 for
Revised: The batch size is set as 512 for

Original: IN-1OO and 2048 for IN-1K.
Revised: IN-1OO and 2048 for IN-1K.

Original: The weight decay is 0.000001 and the SGD momentum is 0.9.
Revised: The weight decay is 0.000001 and the SGD momentum is 0.9.

Original: For the momentum encoder, the momentum value starts from 0.99 and is increased to 1, following (Grill et al., 2020).
Revised: For the momentum encoder, the momentum value starts from 0.99 and is increased to 1, following (Grill et al., 2020).

Original: We use batch normalization (Ioffe & Szegedy, 2015) synchronized across devices.
Revised: We use batch normalization (Ioffe & Szegedy, 2015) synchronized across devices.

Original: 3. E VALUATION ON I MAGE N ET
Revised: 3. E VALUATION ON I MAGE N ET

Original: To evaluate the effectiveness of our proposed method, we pre-train the model on IN-100 and IN-1K, respectively, and conduct a series of downstream tasks, including linear probing, nearest neighbor, semi-supervised learning, and transfer learning.
Revised: To evaluate the effectiveness of our proposed method, we pre-train the model on IN-100 and IN-1K, respectively, and conduct a series of downstream tasks, including linear probing, nearest neighbor, semi-supervised learning, and transfer learning.

Original: Linear Probing
Revised: Linear Probing

Original: To evaluate the quality of visual representations, we conduct linear classification on IN-100 and IN-1K, following a standard protocol used in (Chen & He, 2021).
Revised: To evaluate the quality of visual representations, we conduct linear classification on IN-100 and IN-1K, following a standard protocol used in (Chen & He, 2021).

Original: We freeze the
Revised: We freeze the

Original: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: deep features and train a supervised classifier for 90 epochs with a batch size of 1024.
Revised: deep features and train a supervised classifier for 90 epochs with a batch size of 1024.

Original: We adopt the
Revised: We adopt the

Original: SGD optimizer with lr base = 0 .
Revised: SGD optimizer with lr base = 0 .

Original: 1 , where the learning rate is linearly scaled with the batch size as lr = lr base × batch/ 256 .
Revised: 1 , where the learning rate is linearly scaled with the batch size as lr = lr base × batch/ 256 .

Original: We use standard supervised ImageNet augmentations (He et al., 2020) during training.
Revised: We use standard supervised ImageNet augmentations (He et al., 2020) during training.

Original: For a thorough comparison, we set two baselines: 1) a weak baseline with two standard crops.
Revised: For a thorough comparison, we set two baselines: 1) a weak baseline with two standard crops.

Original: 2) a strong baseline with the multi-crop strategy.
Revised: 2) a strong baseline with the multi-crop strategy.

Original: Firstly, we build our MosRep upon the MoCo-v2 baseline.
Revised: Firstly, we build our MosRep upon the MoCo-v2 baseline.

Original: The key difference between the two frameworks is the use of mosaic representation learning.
Revised: The key difference between the two frameworks is the use of mosaic representation learning.

Original: As illustrated in Table 1, MosRep surpasses the weak baseline by an obvious margin of 4.8% at linear probing on IN-100.
Revised: As illustrated in Table 1, MosRep surpasses the weak baseline by an obvious margin of 4.8% at linear probing on IN-100.

Original: Similarly, MosRep achieves a significant gain of 4.6% over the weak baseline at linear probing on IN-1K. More importantly, MosRep shows a considerable improvement over the strong baseline on both IN-100 and
Revised: Similarly, MosRep achieves a significant gain of 4.6% over the weak baseline at linear probing on IN-1K. More importantly, MosRep shows a considerable improvement over the strong baseline on both IN-100 and

Original: IN-1K datasets, which demonstrates that the performance gain is not simply from more small crops.
Revised: IN-1K datasets, which demonstrates that the performance gain is not simply from more small crops.

Original: For each small crop, the mosaic view effectively introduce diverse background information.
Revised: For each small crop, the mosaic view effectively introduce diverse background information.

Original: We argue that, along with the optimization, the model needs to extract more discriminative features from the diverse background to maximize the similarity of positive pairs and minimize the similarity of negative pairs, which can improve the quality of learned representations.
Revised: We argue that, along with the optimization, the model needs to extract more discriminative features from the diverse background to maximize the similarity of positive pairs and minimize the similarity of negative pairs, which can improve the quality of learned representations.

Original: In the BYOL experiments, we observe similar phenomenon as the MoCo-v2 ones.
Revised: In the BYOL experiments, we observe similar phenomenon as the MoCo-v2 ones.

Original: Nearest Neighbor We further evaluate the representations of the pre-trained models by the nearest neighbor classifier.
Revised: Nearest Neighbor We further evaluate the representations of the pre-trained models by the nearest neighbor classifier.

Original: Following (Koohpayegani et al., 2021), we adopt the center crop operation with 256 × 256 on the training and test sets, and calculate l 2 normalized embeddings by forwarding through the backbone.
Revised: Following (Koohpayegani et al., 2021), we adopt the center crop operation with 256 × 256 on the training and test sets, and calculate l 2 normalized embeddings by forwarding through the backbone.

Original: We report Top-1 accuracy on 1-NN and 5-NN classifiers.
Revised: We report Top-1 accuracy on 1-NN and 5-NN classifiers.

Original: For a thorough comparison, we set two baselines: 1) a weak baseline with two standard crops.
Revised: For a thorough comparison, we set two baselines: 1) a weak baseline with two standard crops.

Original: 2) a strong baseline with the multi-crop strategy.
Revised: 2) a strong baseline with the multi-crop strategy.

Original: Table 1 shows the results of nearest neighbor on the IN-100 and IN-1K datasets, respectively.
Revised: Table 1 shows the results of nearest neighbor on the IN-100 and IN-1K datasets, respectively.

Original: MosRep achieves more consistent improvements over the weak and strong baselines on the 1-NN and 5-NN classifiers.
Revised: MosRep achieves more consistent improvements over the weak and strong baselines on the 1-NN and 5-NN classifiers.

Original: Significant gains of2.5% and 5.7% are achieved on the IN-100 and IN-1K datasets when we build our method on the MoCo-v2 framework.
Revised: Significant gains of and 5.7% are achieved on the IN-100 and IN-1K datasets when we build our method on the MoCo-v2 framework

Original: Besides, we also build on BYOL and obtain similar improvements of 2.0% and 0.8% on the IN-100 and IN-1K, respectively. [Table]
Revised: Besides, we also build on BYOL and obtain similar improvements of 0% and 8% on the IN-100 and IN-1K, respectively

Original: Semi-supervised Learning Following the semi-supervised settings (Chen et al., 2020a [Table]
Revised: Semi-supervised Learning Following the semi-supervised settings (Chen et al., 2020a;

Original: Grill et al [Table] [Table] H´enaff et al., 2019)
Revised: Grill et al., 2020; H ´ enaff et al., 2019), we evaluate the pre-trained models on the task of classification with

Original: [Table]
Revised: [Table]

Original: we evaluate the pre-trained models on the task of classification with limited ImageNet labels.
Revised: limited ImageNet labels.

Original: The sizes of annotations are reduced to 1% and 10% on the IN (Deng et al., 2009) training dataset, respectively.
Revised: The sizes of annotations are reduced to 1% and 10% on the IN-1K (Deng et al., 2009) training dataset, respectively.

Original: We freeze the backbone and train a single linear classifier.
Revised: We freeze the backbone and train a single linear classifier.

Original: We utilize the features from the top of the backbone, which are normalized to have unit l 2 norm and then scaled and shifted to have zero mean and unit variance for each dimension.
Revised: We utilize the features from the top of the backbone, which are normalized to have unit l 2 norm and then scaled and shifted to have zero mean and unit variance for each dimension.

Original: The linear classifier is trained with SGD optimizer (lr=0.01, epochs=40, batch size=256, weight decay=0.0005, and momentum=0.9).
Revised: The linear classifier is trained with SGD optimizer (lr=0.01, epochs=40, batch size=256, weight decay=0.0005, and momentum=0.9).

Original: The learning rate is multiplied by 0.1 at 15 and 30 epochs.
Revised: The learning rate is multiplied by 0.1 at 15 and 30 epochs.

Original: We use the standard setting of ImageNet supervised learning 1 during training.
Revised: We use the standard setting of ImageNet supervised learning 1 during training.

Original: Table 2 illustrates the comparisons of our approach against two widely-used frameworks (He et al., 2020; Grill et al., 2020) and their strong variants.
Revised: Table 2 illustrates the comparisons of our approach against two widely-used frameworks (He et al., 2020; Grill et al., 2020) and their strong variants.

Original: Impressively, when we build our proposed MosRep on the MoCo-v2, we achieves considerable improvements over the strong baseline with both 1% and 10% labels, e.g., 7.4% Top-1 accuracy on 1% subset and 4.9% Top-1 accuracy on 10% subset.
Revised: Impressively, when we build our proposed MosRep on the MoCo-v2, we achieves considerable improvements over the strong baseline with both 1% and 10% labels, e.g., 7.4% Top-1 accuracy on 1% subset and 4.9% Top-1 accuracy on 10% subset.

Original: After that, we build on the BYOL and still significantly outperforms the strong baseline by 1.4% and 0.6% at Top-1 accuracy with 1% and 10% labels, respectively.
Revised: After that, we build on the BYOL and still significantly outperforms the strong baseline by 1.4% and 0.6% at Top-1 accuracy with 1% and 10% labels, respectively.

Original: 3. E VALUATION ON T RANSFER L EARNING
Revised: 3. E VALUATION ON T RANSFER L EARNING

Original: To further evaluate the quality of learned representations, we conduct a series of transfer learning, including linear probing on various classification datasets, COCO object detection (Lin et al., 2014),
Revised: To further evaluate the quality of learned representations, we conduct a series of transfer learning, including linear probing on various classification datasets, COCO object detection (Lin et al., 2014),

Original: COCO instance segmentation (Lin et al., 2014) and cityscapes instance segmentation (Cordts et al., 2016).
Revised: COCO instance segmentation (Lin et al., 2014) and cityscapes instance segmentation (Cordts et al., 2016).

Original: Linear Probing
Revised: Linear Probing

Original: Following the procedure adopted in (Chen et al., 2020a; Grill et al., 2020), we perform the linear probing task on the following datasets: CIFAR-10 (Krizhevsky et al., 2009),
Revised: Following the procedure adopted in (Chen et al., 2020a; Grill et al., 2020), we perform the linear probing task on the following datasets: CIFAR-10 (Krizhevsky et al., 2009),

Original: CIFAR-100 (Krizhevsky et al., 2009), STL-10 (Coates et al., 2011), Food-101 (Bossard et al., 2014),
Revised: CIFAR-100 (Krizhevsky et al., 2009), STL-10 (Coates et al., 2011), Food-101 (Bossard et al., 2014),

Original: Flower-102 (Nilsback & Zisserman, 2008), DTD (Cimpoi et al., 2014), Pets (Parkhi et al., 2012) and Cars (Krause et al., 2013).
Revised: Flower-102 (Nilsback & Zisserman, 2008), DTD (Cimpoi et al., 2014), Pets (Parkhi et al., 2012) and Cars (Krause et al., 2013).

Original: More details about the above datasets can be found in the appendix.
Revised: More details about the above datasets can be found in the appendix.

Original: We freeze the pre-trained backbone to extract the deep features of input images and train a single classifier.
Revised: We freeze the pre-trained backbone to extract the deep features of input images and train a single classifier.

Original: The following pre-processing is applied for all input images: resize to 256 × 256 and take a center crop of 224 × 224 .
Revised: The following pre-processing is applied for all input images: resize to 256 × 256 and take a center crop of 224 × 224 .

Original: After that, the features from the top of the backbone are normalized to have unit ℓ 2 norm and then scaled and shifted to have zero mean and unit variance for each dimension.
Revised: After that, the features from the top of the backbone are normalized to have unit ℓ 2 norm and then scaled and shifted to have zero mean and unit variance for each dimension.

Original: The linear classifier is trained with an SGD optimizer for 27 epochs.We set the SGD momentum [Table] [Table]
Revised: The linear classifier is trained with an SGD optimizer for 27 epochs.

Original: as 0.

Original: [Table] the batch size as [Table] and weight decay as 0.
Revised: We set the SGD momentum as 0.9, the batch size as 256, and weight decay as 0.

Original: We search the optimal learning rate for each pre-trained model.
Revised: We search the optimal learning rate for each pre-trained model.

Original: The learning rate is multiplied by 0.1 at 8, 16 and 24 epochs.
Revised: The learning rate is multiplied by 0.1 at 8, 16 and 24 epochs.

Original:  First, we build on the MoCo-v2 framework and compare it with the weak and strong baselines.
Revised: [Table] [Table] First, we build on the MoCo-v2 framework and compare it with the weak and strong baselines.

Original: As shown in the Table
Revised: As shown in the Table

Original: [Table]
Revised: [Table]

Original: it is obvious that our proposed MosRep outperforms both baselines on all eight datasets
Revised: it is obvious that our proposed MosRep outperforms both baselines on all eight datasets

Original: [Table]
Revised: [Table]

Original: and achieves better transfer performances than the supervised pre-trained model on CIFAR-10, CIFAR-100, STL-10, Food-101 and DTD, which demonstrates that our method can effectively improve the generalization.
Revised: and achieves better transfer performances than the supervised pre-trained model on CIFAR-10, CIFAR-100, STL-10, Food-101 and DTD [Table] which demonstrates that our method can effectively improve the generalization.

Original: Besides, we also conduct experiments on the BYOL framework.
Revised: Besides [Table] we also conduct experiments on the BYOL framework.

Original: Although BYOL is a state-of-the-art self-supervised learning framework, which achieves excellent transfer performances on eight datasets, our approach can improve the generalization performance on most datasets, even outperforming the IN-1K supervised model across the board.
Revised: Although BYOL is a state-of-the-art self-supervised learning framework [Table] which achieves excellent transfer performances on eight datasets, our approach can improve the generalization performance on most datasets, even outperforming the IN-1K supervised model across the board.

Original: COCO Object Detection & Instance Segmentation
Revised: COCO Object Detection & Instance Segmentation

Original: Following the procedure outlined in (Chen et al., 2020b;
Revised: Following the procedure outlined in (Chen et al., 2020b;

Original: He et al., 2020; Chen & He, 2021), we fine-tune a Mask-RCNN (R50-FPN)
Revised: He et al., 2020; Chen & He, 2021), we fine-tune a Mask-RCNN (R50-FPN)

Original: (He et al., 2017; Lin et al., 2017; Wu et al., 2019) under 1 × scheduler and add new Batch Normalization layers before the FPN parameters.
Revised: (He et al., 2017; Lin et al., 2017; Wu et al., 2019) under 1 × scheduler and add new Batch Normalization layers before the FPN parameters.

Original: All Batch Normalization statistics are synchronized across GPUs.
Revised: All Batch Normalization statistics are synchronized across GPUs.

Original: Training is conducted on train2017 split with ˜118k images, and testing is conducted on val split.
Revised: Training is conducted on train2017 split with ˜118k images, and testing is conducted on val split.

Original: The short-side length of input images is randomly selected from 640 to 800 pixels during training and fixed at 800 pixels for testing.
Revised: The short-side length of input images is randomly selected from 640 to 800 pixels during training and fixed at 800 pixels for testing.

Original: We report Average Precision on bounding-boxes (AP bb ) and Average Precision on masks (AP mk ).
Revised: We report Average Precision on bounding-boxes (AP bb ) and Average Precision on masks (AP mk ).

Original: Table 4 shows the comparison of our proposed methods and both baselines.
Revised: Table 4 shows the comparison of our proposed methods and both baselines.

Original: Our MosRep can consistently increase the ability of detection and segmentation on the MoCo-v2 and BYOL frameworks.
Revised: Our MosRep can consistently increase the ability of detection and segmentation on the MoCo-v2 and BYOL frameworks.

Revised: Cityscapes Instance Segmentation Following the setting utilized in ReSim Xiao et al.

Original: [Table]
Revised: , we fine-tune a Mask-RCNN (R50-FPN) He et al. ; Lin et al. ; Wu et al.

Revised: and add

Original:  [Figure]
Revised: [Table] [Figure] [Table] [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Original:  [Table]
Revised: [Figure] [Table] [Figure] [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Original:  [Table]
Revised: [Figure] [Table] [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Table] 
Revised: [Figure] [Table] [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table][Figure] [Table] [Figure]
Revised: [Table]

Original: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Cityscapes Instance Segmentation Following the setting utilized in ReSim Xiao et al.

Original: , we fine-tune a Mask-RCNN (R50-FPN)

Original: He et al.

Original: ; Lin et al. ; Wu et al. and add Batch Normalization layers before the FPN, and synchronize all Batch Normalization during training.
Revised: Batch Normalization layers before the FPN, and synchronize all Batch Normalization during training.

Original: Table 5 illustrates the comparisons of MosRep versus the supervised pre-training counterparts and the weak and strong baselines.
Revised: Table 5 illustrates the comparisons of MosRep versus the supervised pre-training counterparts and the weak and strong baselines.

Original: 3. A BLATIONS
Revised: 3. A BLATIONS

Original: In this section, we study the effect of some design choices in the MosRep, including the jittering operation, α in the beta distribution and the upper bound θ of jitter range.
Revised: In this section, we study the effect of some design choices in the MosRep, including the jittering operation, α in the beta distribution and the upper bound θ of jitter range.

Original: We built our method upon the MoCo-v2 and pre-train it on IN-100 for 400 epochs.
Revised: We built our method upon the MoCo-v2 and pre-train it on IN-100 for 400 epochs.

Original: We report the Top-1 accuracy on INlinear probing.
Revised: We report the Top-1 accuracy on INlinear probing.

Original: Effect of α in the Beta distribution We use α values from the set { 0.3, 0.5, 1.0, 2.0 } and set α = 0 .
Revised: Effect of α in the Beta distribution We use α values from the set { 0.3, 0.5, 1.0, 2.0 } and set α = 0 .

Original: 5 for the above experiments.
Revised: 5 for the above experiments.

Original: A smaller α means a greater likelihood of having a large range of jittering and a larger α means the opposite.
Revised: A smaller α means a greater likelihood of having a large range of jittering and a larger α means the opposite.

Original: As illustrated in Figure 3, MosRep with the jittering operation achieves better transfer performance than ones without the jittering operation, which demonstrates that this operation can increase the variance of the mosaic view to prevent the model from memorizing the spatial location of each small crop.
Revised: As illustrated in Figure 3, MosRep with the jittering operation achieves better transfer performance than ones without the jittering operation, which demonstrates that this operation can increase the variance of the mosaic view to prevent the model from memorizing the spatial location of each small crop.

Original: The Jitter Range We further study the effect of different jitter ranges.
Revised: The Jitter Range We further study the effect of different jitter ranges.

Original: As shown in the right part of Figure 3, with an increasing jittering range, MosRep achieves better Top-1 accuracy on the linear probing.
Revised: As shown in the right part of Figure 3, with an increasing jittering range, MosRep achieves better Top-1 accuracy on the linear probing.

Original: However, when the jittering range is too large, more false positive samples with no overlapping area between positive pairs are likely to be observed, which results in the degradation of the quality of visual representations.
Revised: However, when the jittering range is too large, more false positive samples with no overlapping area between positive pairs are likely to be observed, which results in the degradation of the quality of visual representations.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper we design a mosaic data augmentation strategy that focuses on small parts of input images and adequately enriches the background information.
Revised: In this paper we design a mosaic data augmentation strategy that focuses on small parts of input images and adequately enriches the background information.

Original: Through this strategy, we develop a simple and effective mosaic representation learning framework for self-supervised visual pre-training.
Revised: Through this strategy, we develop a simple and effective mosaic representation learning framework for self-supervised visual pre-training.

Original: Our method can be flexibly combined with other self-supervised learning frameworks, e g., MoCo-v and BYOL.
Revised: Our method can be flexibly combined with other self-supervised learning frameworks, e g., MoCo-v and BYOL.

Original: Extensive experimental results demonstrate that our method consistently boosts the performance on a series of downstream tasks, including linear probing, nearest neighbor, semisupervised learning, transfer learning, and etc.
Revised: Extensive experimental results demonstrate that our method consistently boosts the performance on a series of downstream tasks, including linear probing, nearest neighbor, semisupervised learning, transfer learning and etc.

Original: We hope this work could inspire future research on view designing, considering its significant role in self-supervised learning.
Revised: We hope this work could inspire future research on view designing, considering its significant role in self-supervised learning.

Original: The main limitation is that although we achieved promising results on various image-level downstream tasks, we do not observe noticeable gains on pixel-level downstream tasks, especially when pretrained on large-scale datasets.
Revised: The main limitation is that although we achieved promising results on various image-level downstream tasks, we do not observe noticeable gains on pixel-level downstream tasks, especially when pretrained on large-scale datasets.

Original: This phenomenon implies that image-level self-supervised learning is sub-optimal for pixel-level downstream tasks.
Revised: This phenomenon implies that image-level self-supervised learning is sub-optimal for pixel-level downstream tasks.

Original: We will study this problem in our future research
Revised: We will study this problem in our future research

Original: AA PPENDIX - R ELATED W ORK
Revised: A R ELATED W ORK

Original: Self-supervised learning methods have been extensively studied to close the gap with supervised learning.
Revised: Self-supervised learning methods have been extensively studied to close the gap with supervised learning.

Original: These methods can be mainly categorized into several aspects, including contrastive-based, consistency-based, clustering-based and generative-based methods.
Revised: These methods can be mainly categorized into several aspects, including contrastive-based, consistency-based, clustering-based and generative-based methods.

Original: A. C ONTRASTIVE L EARNING
Revised: A. C ONTRASTIVE L EARNING

Original: contrastive self-supervised learning has emerged as a promising approach to unsupervised visual representation learning.
Revised: contrastive self-supervised learning has emerged as a promising approach to unsupervised visual representation learning.

Original: Each image is treated as an individual class in an instance discrimination setting, whose core idea is to pull the positive pairs together and push the negative pairs away in the embedding space.
Revised: Each image is treated as an individual class in an instance discrimination setting, whose core idea is to pull the positive pairs together and push the negative pairs away in the embedding space.

Original: As studied in many previous works (Chen et al., 2020b;
Revised: As studied in many previous works (Chen et al., 2020b;

Original: He et al., 2020; Chen et al., 2020a; Hu et al., 2021), it is extremely essential to construct high-quality positive and negative pairs to achieve higher transfer performance.
Revised: He et al., 2020; Chen et al., 2020a; Hu et al., 2021), it is extremely essential to construct high-quality positive and negative pairs to achieve higher transfer performance.

Original: Specifically, InstDisc (Wu et al., 2018) increases the size of negative pairs by built a memory bank that stores pre-computed embeddings, thus improving the performance.
Revised: Specifically, InstDisc (Wu et al., 2018) increases the size of negative pairs by built a memory bank that stores pre-computed embeddings, thus improving the performance.

Original: Following this line, MoCo (He et al., 2020) utilizes a momentum update mechanism to update a large queue of negatives for contrastive learning.
Revised: Following this line, MoCo (He et al., 2020) utilizes a momentum update mechanism to update a large queue of negatives for contrastive learning.

Original: This momentum encoder greatly improves the quality and consistency of negative pairs and achieves remarkable performance compared to previous works.
Revised: This momentum encoder greatly improves the quality and consistency of negative pairs and achieves remarkable performance compared to previous works.

Original: SimCLR (Chen et al., 2020a) further improves in a straightforward way that directly adopts negative samples in the current batch with much bigger batch size.
Revised: SimCLR (Chen et al., 2020a) further improves in a straightforward way that directly adopts negative samples in the current batch with much bigger batch size.

Original: Besides, researchers carefully construct a rich family of data augmentations on cropped images, significantly boosting classification accuracy.
Revised: Besides, researchers carefully construct a rich family of data augmentations on cropped images, significantly boosting classification accuracy.

Original: MoCo-v2 Chen et al.
Revised: MoCo-v2 Chen et al.

Original: (2020b) also improve the performance than MoCo (He et al., 2020) by using the same data augmentation and MLP layer design.
Revised: (2020b) also improve the performance than MoCo (He et al., 2020) by using the same data augmentation and MLP layer design.

Original: A. C ONSISTENCY R EGULARIZATION
Revised: A. C ONSISTENCY R EGULARIZATION

Original: Different from other contrastive-based methods, BYOL (Grill et al., 2020) trains an online network to predict the output of the target network, with the latter slowly updated with momentum.
Revised: Different from other contrastive-based methods, BYOL (Grill et al., 2020) trains an online network to predict the output of the target network, with the latter slowly updated with momentum.

Original: The authors assume that the additional predictor to the online network and the momentum encoder are essential to avoid collapsed solutions without negatives.
Revised: The authors assume that the additional predictor to the online network and the momentum encoder are essential to avoid collapsed solutions without negatives.

Original: SimSiam (Chen & He, 2021) further explores simple Siamese networks that can learn transferable representations without negative samples, large batches and momentum encoders.
Revised: SimSiam (Chen & He, 2021) further explores simple Siamese networks that can learn transferable representations without negative samples, large batches and momentum encoders.

Original: The role of the stop gradient is emphasized in preventing collapse.
Revised: The role of the stop gradient is emphasized in preventing collapse.

Original: BarlowTwins (Zbontar et al., 2021) avoids collapse by measuring the cross-correlation matrix between the outputs of two identical networks fed with distorted versions of a sample, and making it as close to the identity matrix as possible.
Revised: BarlowTwins (Zbontar et al., 2021) avoids collapse by measuring the cross-correlation matrix between the outputs of two identical networks fed with distorted versions of a sample, and making it as close to the identity matrix as possible.

Original: This causes the embedding vectors of distorted versions of a sample to be similar, while minimizing the redundancy between the components of these vectors.
Revised: This causes the embedding vectors of distorted versions of a sample to be similar, while minimizing the redundancy between the components of these vectors.

Original: A. C LUSTERING
Revised: A. C LUSTERING

Original: These methods, such as DeepCluster (Caron et al., 2018), SWAV (Caron et al., 2020), SELA Ma et al.
Revised: These methods, such as DeepCluster (Caron et al., 2018), SWAV (Caron et al., 2020), SELA Ma et al.

Original: (2019), perform contrastive-like comparisons without the requirement to compute all pairwise distances.
Revised: (2019), perform contrastive-like comparisons without the requirement to compute all pairwise distances.

Original: Specifically, these methods cluster the data simultaneously while enforcing consistency between cluster assignments produced for different distortions of the same image, instead of directly comparing features in contrastive learning.
Revised: Specifically, these methods cluster the data simultaneously while enforcing consistency between cluster assignments produced for different distortions of the same image, instead of directly comparing features in contrastive learning.

Original: Clustering methods are also prone to collapse, e.g., empty clusters in k-means and avoiding them relies on careful implementation details.
Revised: Clustering methods are also prone to collapse, e.g., empty clusters in k-means and avoiding them relies on careful implementation details.

Original: Online clustering methods like SWAV can be trained with large and small batches but require storing features when the number of clusters is much larger than the batch size.
Revised: Online clustering methods like SWAV can be trained with large and small batches but require storing features when the number of clusters is much larger than the batch size.

Original: A. G ENERATIVE & M ASK I MAGE M ODELING
Revised: A. G ENERATIVE & M ASK I MAGE M ODELING

Original: The generative methods typically adopt auto-encoders (Kingma et al., 2019), and adversarial learning to train an unsupervised representation.
Revised: The generative methods typically adopt auto-encoders (Kingma et al., 2019), and adversarial learning to train an unsupervised representation.

Original: Usually, they focused on the pixel-wise information of images to distinguish images from different classes.
Revised: Usually, they focused on the pixel-wise information of images to distinguish images from different classes.

Original: For instance, BigBiGAN (Donahue & Simonyan, 2019) adopted BiGAN to capture the relationship between latent semantic representations and the input images.
Revised: For instance, BigBiGAN (Donahue & Simonyan, 2019) adopted BiGAN to capture the relationship between latent semantic representations and the input images.

Original: Motivated by the great success of BERT, masked image modeling (MIM) (Xie et al., 2022; Bao et al., 2021;
Revised: Motivated by the great success of BERT, masked image modeling (MIM) (Xie et al., 2022; Bao et al., 2021;

Original: He et al., 2022) becomes a new trend in self-supervised visual pre-training, which randomly masks parts of images and reconstructs them based on the corrupted image.
Revised: He et al., 2022) becomes a new trend in self-supervised visual pre-training, which randomly masks parts of images and reconstructs them based on the corrupted image.

Original: ViT (Dosovitskiy et al., 2021) attempts to adopt masked patch prediction for self-supervised learning. 
Revised: ViT (Dosovitskiy et al., 2021) attempts to adopt masked patch prediction for self-supervised learning. [Table] [Table]

Original: BEiT
Revised: BEiT

Original: (Bao et al., 2021) predicts the discrete tokens of masked token resorting to an off-the-shelf discrete VAE.
Revised: (Bao et al., 2021) predicts the discrete tokens of masked token resorting to an off-the-shelf discrete VAE.

Original: Instead of discretizing the visual information, MAE (He et al., 2022) and SimMIM (Xie et al., 2022) propose to directly predict the pixel-level value as the reconstruction target.
Revised: Instead of discretizing the visual information, MAE (He et al., 2022) and SimMIM (Xie et al., 2022) propose to directly predict the pixel-level value as the reconstruction target.

Revised: plugging in MoCo-v2 Chen et al.

Revised: (2020b), we adopt a ResNet-50 (He et al., 2016) as our backbone, and pre-train it on ImageNetfor 400 epochs.

Revised: We report the top 1 accuracy of linear probing and 1 -NN on ImageNet-100.

Revised: [Table]

Revised: B A DDITIONAL E VALUATION ON I MAGE N ET

Revised: We build our method on DINO Caron et al.

Revised: For a fair comparison, we set a standard two-crop version as a weak baseline and a multi-crop version as a strong baseline.

Revised: We perform self-supervised pre-training and linear probing on both ImageNet-100 and ImageNet-1K. Following Caron et al.

Revised: (2021), we adopt ViT-S/16 as the encoder and pre-train for 200 epochs.

Revised: The batch size is 1024.

Revised: We report the top1 accuracy for linear probing and 1-NN.

Revised: As illustrated in Table 8, MosRep surpasses both baselines by clear margins on linear probing and 1-NN.

Revised: C A DDITIONAL A BLATION S TUDY

Revised: Simple background noise.

Revised: We conduct an ablation study that adds simple background noise (e.g., gaussian) in the small crops.

Revised: Experimental results in the following table 1 show that adding simple background noise cannot improve the performance of linear probing.

Revised: We argue that although simple background noise can enrich the background information, it cannot effectively reduce the redundant mutual information and is easy to optimize.

Revised: Different input sizes of small crops.

Revised: We vary the input size of small crops from 64 × 64 to 112 ×and pre-train a ResNet-50 on ImageNet-100 for 400 epochs.

Revised: As illustrated in Table 1, experimental results show that decreasing the input size of small crops causes performance degradation on linear probing and 1-NN.

Revised: Considering the limited resource, we do not further increase the input size of small crops.