Original: M ASKED S IAMESE C ONV N ETS : T OWARDS AN E FFECTIVE M ASKING S TRATEGY FOR G ENERAL - PURPOSE S IAMESE N ETWORKS
Revised: M ASKED S IAMESE C ONV N ETS : T OWARDS AN E FFECTIVE M ASKING S TRATEGY FOR G ENERAL - PURPOSE S IAMESE N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Siamese Networks are a popular self-supervised learning framework that learns useful representation without human supervision by encouraging representations to be invariant to distortions.
Revised: Siamese Networks are a popular self-supervised learning framework that learns useful representation without human supervision by encouraging representations to be invariant to distortions.

Original: Existing methods heavily rely on hand-crafted augmentations, which are not easy to adapt to new domains.
Revised: Existing methods heavily rely on hand-crafted augmentations, which are not easy to adapt to new domains.

Original: To explore ageneralpurpose or domain-agnostic siamese network, we investigate using masking as augmentations in siamese networks.
Revised: To explore a domain-agnostic siamese network, we investigate using masking as augmentations in siamese networks.

Original: Recently, masking for siamese networks has only been shown useful with transformer architectures, e.g. MSN (Assran et al., 2022) and data2vec (Baevski et al., 2022).
Revised: Recently, masking for siamese networks has only been shown useful with transformer architectures, e.g. MSN (Assran et al., 2022) and data2vec (Baevski et al., 2022).

Original: In this work, we identify the underlying problems of masking for siamese networks with arbitrary backbones, including ConvNets.
Revised: In this work, we identify the underlying problems of masking for siamese networks with arbitrary backbones, including ConvNets.

Original: We propose an effectiveand general-purpose masking strategy and demonstrate its effectiveness on various siamese network frameworks.
Revised: We propose an effective masking strategy and demonstrate its effectiveness on various siamese network frameworks.

Original: Our method generally improves siamese networks’ performances in the few-shot image classification, and object detection tasks.
Revised: Our method generally improves siamese networks’ performances in the few-shot image classification, and object detection tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Self-supervised learning aims to learn useful representations from scalable unlabeled data without relying on human annotation.
Revised: Self-supervised learning aims to learn useful representations from scalable unlabeled data without relying on human annotation.

Original: It has widely used in natural language processing (Devlin et al., 2019;
Revised: It has widely used in natural language processing (Devlin et al., 2019;

Original: Zhang et al., 2022; Brown et al., 2020), speech recognition (van den Oord et al., 2018; Hsu et al., 2021; Schneider et al., 2019; Baevski et al., 2020) and other domains (Rives et al., 2021;
Revised: Zhang et al., 2022; Brown et al., 2020), speech recognition (van den Oord et al., 2018; Hsu et al., 2021; Schneider et al., 2019; Baevski et al., 2020) and other domains (Rives et al., 2021;

Original: Rong et al., 2020).
Revised: Rong et al., 2020).

Original: Recently, self-supervised visual representation learning has also become an active research area.
Revised: Recently, self-supervised visual representation learning has also become an active research area.

Original: First introduced by Bromley et al.
Revised: First introduced by Bromley et al.

Original: the siamese network (Chen et al., 2020a;b; He et al., 2020; Chen et al., 20220; 2021; Caron et al., 2020; Grill et al., 2020; Chen & He, 2020; Wang et al., 2022; Zbontar et al., 2021; Bardes et al., 2021) is one promising approach among many selfsupervised learning approaches and outperforms supervised counterparts in many visual benchmarks.
Revised: the siamese network (Chen et al., 2020a;b; He et al., 2020; Chen et al., 20220; 2021; Caron et al., 2020; Grill et al., 2020; Chen & He, 2020; Wang et al., 2022; Zbontar et al., 2021; Bardes et al., 2021) is one promising approach among many selfsupervised learning approaches and outperforms supervised counterparts in many visual benchmarks.

Original: It encourages the encoder to be invariant to human-designed augmentations, capturing only the essential features.
Revised: It encourages the encoder to be invariant to human-designed augmentations, capturing only the essential features.

Original: Practically, in the vision domain, the siamese network methods rely on domainspecific augmentations, such as cropping, color jittering and Gaussian blur, which do not transfer well to other domains.
Revised: Practically, in the vision domain, the siamese network methods rely on domainspecific augmentations, such as cropping, color jittering and Gaussian blur, which do not transfer well to other domains.

Original: Therefore, it is desired to have a general augmentation approach for siamese networks that require minimal domain knowledge and can generalize.
Revised: Therefore, it is desired to have a general augmentation approach for siamese networks that require minimal domain knowledge and can generalize.

Original: Among various augmentations, masking the input is one of the simplest and effective choices, which has been demonstrated to be useful for language (Devlin et al., 2019) and speech (Hsu et al., 2021).
Revised: Among various augmentations, masking the input is one of the simplest and effective choices, which has been demonstrated to be useful for language (Devlin et al., 2019) and speech (Hsu et al., 2021).

Original: However, not until the recent success of vision transformers (ViTs) (Dosovitskiy et al., 2021; Touvron et al., 2021) can vision models leverage masking as a general augmentation.
Revised: However, not until the recent success of vision transformers (ViTs) (Dosovitskiy et al., 2021; Touvron et al., 2021) can vision models leverage masking as a general augmentation.

Original: Self-supervised learning with masked inputs has demonstrated more scalable properties when combined with ViTs (He et al., 2021; Bao et al., 2021;
Revised: Self-supervised learning with masked inputs has demonstrated more scalable properties when combined with ViTs (He et al., 2021; Bao et al., 2021;

Original: Zhou et al., 2021; Baevski et al., 2022).
Revised: Zhou et al., 2021; Baevski et al., 2022).

Original: Unfortunately, siamese networks with naive masking do not work well with most off-the-shelf architecture, e.g., ConvNets (He et al., 2016; Liu et al., 2022).
Revised: Unfortunately, siamese networks with naive masking do not work well with arbitrary architecture, e.g., ConvNets (He et al., 2016; Liu et al., 2022).

Original: We identify the underlying issues behind masked siamese networks with ConvNets.
Revised: We identify the underlying issues behind masked siamese networks with ConvNets.

Original: We argue that
Revised: We argue that

Original: ConvNets do not have a mechanism to encode null information.
Revised: ConvNets do not have a mechanism to encode null information.

Original: In addition, masking will introduce parasitic edges.
Revised: In addition, masking will introduce parasitic edges.

Original: We propose a preprocessing procedure to solve these problems.
Revised: We propose a preprocessing procedure to solve these problems.

Original: We present several general-purpose designs that allow siamese networks to benefit from masked inputs.
Revised: We present several general-purpose designs that allow siamese networks to benefit from masked inputs.

Original: Experiments show that siamese networks with ConvNets backbone can benefit from masked inputs with our masking strategy.
Revised: Experiments show that siamese networks with ConvNets backbone can benefit from masked inputs with our masking strategy.

Original: We summarize our contributions below:
Revised: We summarize our contributions below:

Original: • We discuss the role of augmentations in siamese networks and explore a general-purpose augmentation approach of masking.
Revised: • We discuss the role of augmentations in siamese networks and explore a general-purpose augmentation approach of masking.

Original: We identify the underlying problem of masking for siamese with ConvNets backbones.
Revised: We identify the underlying problem of masking for siamese with ConvNets backbones.

Original: • We propose a preprocessing step to overcome the problem behind masked siamese networks with ConvNets.
Revised: • We propose a preprocessing step to overcome the problem behind masked siamese networks with ConvNets.

Original: We present a series ofgeneral-purpose masking designs which allow masking to benefit the siamese network.
Revised: We present a series of masking designs which allow masking to benefit the siamese network with ConvNets backbones.

Original: • We propose Masked Siamese ConvNets (MSCN), an effective masking strategy for generalpurpose siamese networks.
Revised: • We propose Masked Siamese ConvNets (MSCN), an effective masking strategy for generalpurpose siamese networks.

Original: Our method can be applied to various siamese network frameworks, and it demonstrates competitive performances on few-shot image classification benchmarks and outperforms previous methods on object detection benchmarks.
Revised: Our method can be applied to various siamese network frameworks, and it demonstrates competitive performances on few-shot image classification benchmarks and outperforms previous methods on object detection benchmarks.

Original:  R ELATED W ORKS 2. S IAMESE N ETWORKS
Revised:  R ELATED W ORKS 2. S IAMESE N ETWORKS

Original: Self-supervised visual representation learning has become an active research area since they have shown superior performances over supervised counterparts in recent years.
Revised: Self-supervised visual representation learning has become an active research area since they have shown superior performances over supervised counterparts in recent years.

Original: One promising approach is to learn useful representations by encouraging them to be invariant to augmentations, known as siamese networks or joint-embedding methods (Misra & van der Maaten, 2020;
Revised: One promising approach is to learn useful representations by encouraging them to be invariant to augmentations, known as siamese networks or joint-embedding methods (Misra & van der Maaten, 2020;

Original: Chen et al., 2020a;b;
Revised: Chen et al., 2020a;b;

Original: He et al., 2020; Chen et al., 20220; 2021; Caron et al., 2020; Grill et al., 2020; Chen & He, 2020;
Revised: He et al., 2020; Chen et al., 20220; 2021; Caron et al., 2020; Grill et al., 2020; Chen & He, 2020;

Original: Wang et al., 2022; Zbontar et al., 2021;
Revised: Wang et al., 2022; Zbontar et al., 2021;

Original: Bardes et al., 2021).
Revised: Bardes et al., 2021).

Original: These methods use different mechanisms to prevent collapse, and they all rely on carefully designed augmentations such as random resized cropping, color jittering, grayscale and Gaussian blur.
Revised: These methods use different mechanisms to prevent collapse, and they all rely on carefully designed augmentations such as random resized cropping, color jittering, grayscale and Gaussian blur.

Original: These augmentations prevent the encoder from only using trivial features.
Revised: These augmentations prevent the encoder from only using trivial features.

Original: Empirically, siamese networks with these standard augmentation settings usually work well with off-the-shelf architectures, including ResNets (He et al., 2016) and
Revised: Empirically, siamese networks with these standard augmentation settings usually work well with arbitrary architectures, including ResNets (He et al., 2016) and

Original: ViTs (Dosovitskiy et al., 2021).
Revised: ViTs (Dosovitskiy et al., 2021).

Original: Their representations are label-efficient (Assran et al., 2021; 2022), more robust (Hendrycks et al., 2019), and have improved fairness (Goyal et al., 2022).
Revised: Their representations are label-efficient (Assran et al., 2021; 2022), more robust (Hendrycks et al., 2019), and have improved fairness (Goyal et al., 2022).

Original: In addition,
Revised: In addition,

Original: Siamese networks have been demonstrated to benefit from scalable data (Goyal et al., 2021a).
Revised: Siamese networks have been demonstrated to benefit from scalable data (Goyal et al., 2021a).

Original: 2. R EPRESENTATION LEARNING WITH M ASKED I NPUTS
Revised: 2. R EPRESENTATION LEARNING WITH M ASKED I NPUTS

Original: Masking the input is one of the simplest methods to corrupt the information input and could be applied to a wide range of data types.
Revised: Masking the input is one of the simplest methods to corrupt the information input and could be applied to a wide range of data types.

Original: It has been mostly used in two scenarios.
Revised: It has been mostly used in two scenarios.

Original: The first is for denoising autoencoder frameworks (Vincent et al., 2010; 2008).
Revised: The first is for denoising autoencoder frameworks (Vincent et al., 2010; 2008).

Original: Motivated by its success in NLP (Devlin et al., 2019; Brown et al., 2020) with transformers (Vaswani et al., 2017), various visual representation learning methods (He et al., 2021; Bao et al., 2021;
Revised: Motivated by its success in NLP (Devlin et al., 2019; Brown et al., 2020) with transformers (Vaswani et al., 2017), various visual representation learning methods (He et al., 2021; Bao et al., 2021;

Original: Zhou et al., 2021) using ViTs have also shown benefit from masked inputs.
Revised: Zhou et al., 2021) using ViTs have also shown benefit from masked inputs.

Original: These methods have proven to be a promising general-purpose self-supervised learning approach.
Revised: These methods have proven to be a promising general-purpose self-supervised learning approach.

Original: The second is for siamese networks.
Revised: The second is for siamese networks.

Original: Siamese networks can benefit from masked inputs (Baevski et al., 2022; Assran et al., 2021) where masking serves as an extra augmentation.
Revised: Siamese networks can benefit from masked inputs (Baevski et al., 2022; Assran et al., 2021) where masking serves as an extra augmentation.

Original: These methods are able to learn more transferable representations and have the benefit of being label-efficient.
Revised: These methods are able to learn more transferable representations and have the benefit of being label-efficient.

Original: These works are limited to ViT architectures.
Revised: These works are limited to ViT architectures.

Original: However, no previous work has shown that the masking approach can work equally well with off-the-shelf ConvNets.
Revised: However, no previous work has shown that the masking approach can work equally well with arbitrary ConvNets.

Original:  A UGMENTATIONS FOR S IAMESE N ETWORKS
Revised:  A UGMENTATIONS FOR S IAMESE N ETWORKS

Original: In this section, we discuss the role of augmentations in siamese networks and outline several design principles which will be used to guide our masking strategy.
Revised: In this section, we discuss the role of augmentations in siamese networks and outline several design principles which will be used to guide our masking strategy.

Original: 3. P RELIMINARIES
Revised: 3. P RELIMINARIES

Original: Siamese networks for visual representation learning start with creating two random views x 1 and x 2 from the same input x , with two different sets of random augmentation transformations T ϕ and
Revised: Siamese networks for visual representation learning start with creating two random views x 1 and x 2 from the same input x , with two different sets of random augmentation transformations T ϕ and

Original: T ϕ ′ to each view.
Revised: T ϕ ′ to each view.

Original: These methods then train an encoder f θ ( · )  to minimize L p =
Revised: These methods then train an encoder f θ ( · ) and a projector q θ ( · ) to minimize L p =

Original: E ϕ,ϕ ′
Revised: E ϕ,ϕ ′

Original: ||
Revised: ||

Original:  f θ ( T ϕ ( x 1 ))
Revised: q θ ( f θ ( T ϕ ( x 1 )))

Original: −  f θ ( T ϕ ′ ( x 2 ))
Revised: − q θ ( f θ ( T ϕ ′ ( x 2 )))

Original: || 2 , known as the positive term of the siamese network loss function.
Revised: || 2 , known as the positive term of the siamese network loss function.

Revised: To simplify the notation, we will ignore the projector q θ ( · ) in all of our following discussions, since we could always consider the projector as a part of the encoder.

Original: However, if the encoder f θ ( · ) is trained with the positive term alone, it f θ ( · ) will quickly converge to a collapsed solution that produces a constant representation for all inputs.
Revised: However, if the encoder f θ ( · ) is trained with the positive term alone, it f θ ( · ) will quickly converge to a collapsed solution that produces a constant representation for all inputs.

Original: Preventing collapse can be solved by various frameworks, including contrastive loss Chen et al.
Revised: Preventing collapse can be solved by various frameworks, including contrastive loss Chen et al.

Original: (2020a)
Revised: (2020a)

Original: ; He et al.
Revised: ; He et al.

Original: redundancy reduction Zbontar et al.
Revised: redundancy reduction Zbontar et al.

Original: ; Bardes et al.
Revised: ; Bardes et al.

Original: (2021), clustering Caron et al.
Revised: (2021), clustering Caron et al.

Original: and distillation Grill et al.
Revised: and distillation Grill et al.

Original: ; Chen & He (2020); Caron et al.
Revised: ; Chen & He (2020); Caron et al.

Original: These methods explicitly or implicitly prevent E ϕ,ϕ ′
Revised: These methods explicitly or implicitly prevent E ϕ,ϕ ′

Original: [ || f θ ( T ϕ ( x )) − f θ ( T ϕ ′ ( x ′ ))
Revised: [ || f θ ( T ϕ ( x )) − f θ ( T ϕ ′ ( x ′ ))

Original: || 2 ] becoming too small for x , x ′ coming from different images.
Revised: || 2 ] becoming too small for x , x ′ coming from different images.

Original: This is known as the negative term of the siamese networks loss function.
Revised: This is known as the negative term of the siamese networks loss function.

Original: Although the negative term design is an important topic of siamese network methods, it is relatively independent of augmentation choice.
Revised: Although the negative term design is an important topic of siamese network methods, it is relatively independent of augmentation choice.

Original: In this work, we focus on the positive term to analyize the role of augmentations T ϕ .
Revised: In this work, we focus on the positive term to analyize the role of augmentations T ϕ .

Original: Data augmentation plays an important role in guiding encoders f θ ( · ) to learn useful representations of the input so that they can be used for downstream tasks.
Revised: Data augmentation plays an important role in guiding encoders f θ ( · ) to learn useful representations of the input so that they can be used for downstream tasks.

Original: With weak augmentation, we could find a encoder g based on simple input statistics such that ∥ g ( T ϕ ( x )) − g (
Revised: With weak augmentation, we could find a encoder g based on simple input statistics such that ∥ g ( T ϕ ( x )) − g (

Original: T ϕ ′
Revised: T ϕ ′

Original: ( x ))
Revised: ( x ))

Original: ∥ is small for all ϕ and ϕ ′ , then the encoder f only needs to capture those superficial features to minimize the positive term of the loss function.
Revised: ∥ is small for all ϕ and ϕ ′ , then the encoder f only needs to capture those superficial features to minimize the positive term of the loss function.

Original: With aggressive augmentations, it becomes difficult to find a encoder g such that ∥ g ( T ϕ ( x )) − g (
Revised: With aggressive augmentations, it becomes difficult to find a encoder g such that ∥ g ( T ϕ ( x )) − g (

Original: T ϕ ′
Revised: T ϕ ′

Original: ( x ))
Revised: ( x ))

Original: ∥ is small for all ϕ and ϕ ′ due to the mutual information between two augmented views is destroyed.
Revised: ∥ is small for all ϕ and ϕ ′ due to the mutual information between two augmented views is destroyed.

Original: Then the encoders f θ ( · ) are impossible to capture that information which is potentially useful for downstream tasks.
Revised: Then the encoders f θ ( · ) are impossible to capture that information which is potentially useful for downstream tasks.

Original: Furthermore, other than assessing the augmentations based on how aggressive it is, Huang et al.
Revised: Furthermore, other than assessing the augmentations based on how aggressive it is, Huang et al.

Original: (2021) shows that the generalization ability of contrastive self-supervised learning depends on the concentration of augmented data within the same latent class.
Revised: (2021) shows that the generalization ability of contrastive self-supervised learning depends on the concentration of augmented data within the same latent class.

Original: Intuitively, it means that to increase the performance of downstream tasks that require the semantic information of the data, the augmentations we applied to input data should be able to increase the probability of similar augmented views of images from the same latent class, or decrease the probability of similar augmented views of images from the different latent classes.
Revised: Intuitively, it means that to increase the performance of downstream tasks that require the semantic information of the data, the augmentations we applied to input data should be able to increase the probability of similar augmented views of images from the same latent class, or decrease the probability of similar augmented views of images from the different latent classes.

Original: 3. A UGMENTATION D ESIGNING P RINCIPLE
Revised: 3. A UGMENTATION D ESIGNING P RINCIPLE

Original: Following the discussion in 3.1, we present three design principles of data augmentations for siamese networks:
Revised: Following the discussion in 3.1, we present three design principles of data augmentations for siamese networks:

Original: Prevent easy solutions such that encoder f θ ( · ) only captures certain superficial features; 2.
Revised: Prevent easy solutions such that encoder f θ ( · ) only captures certain superficial features; 2.

Original: Keep as much mutual information between the two views as possible; 3.
Revised: Keep as much mutual information between the two views as possible; 3.

Original: Increase the probability of similar augmented views of images from the same latent class, or decrease the probability of similar augmented views of images from the different latent classes.
Revised: Increase the probability of similar augmented views of images from the same latent class, or decrease the probability of similar augmented views of images from the different latent classes.

Original:  P ROBLEMS IN M ASKED S IAMESE N ETWORKS WITH C ONV N ETS
Revised:  P ROBLEMS IN M ASKED S IAMESE N ETWORKS WITH C ONV N ETS

Original: Siamese networks using masking as an extra augmentation have demonstrated competitive performances (Assran et al., 2022; Baevski et al., 2022) with ViT backbones.
Revised: Siamese networks using masking as an extra augmentation have demonstrated competitive performances (Assran et al., 2022; Baevski et al., 2022) with ViT backbones.

Original: Naively replacing ViTs withoff-the-shelf ConvNets results in significantly worse performances.
Revised: Naively replacing ViTs with ConvNets results in significantly worse performances.

Original: Here, we identify its underlying problems.
Revised: Here, we identify its underlying problems.

Original: See Figure 1.
Revised: See Figure 1.

Original: Masking As Missing Information - The main reason masking as augmentation performs poorly on siamese networks with ConvNets is that masking represents missing information, and there is no good choice of assigned values for continuous input.
Revised: Masking As Missing Information - The main reason masking as augmentation performs poorly on siamese networks with ConvNets is that masking represents missing information, and there is no good choice of assigned values for continuous input.

Original: Figure 1b shows the color histograms for the original input and its masked version.
Revised: Figure 1b shows the color histograms for the original input and its masked version.

Original: Naively assigning zero or noise or even a trainable value will  confuse the network.
Revised: Naively assigning zero or noise or even a trainable value will [Figure] confuse the network.

Original: For ViTs, this problem is automatically avoided by the self-attention mechanism that does not attend to the masked area.
Revised: For ViTs, this problem is automatically avoided by the self-attention mechanism that does not attend to the masked area.

Original: Hence, it is essential for the network to identify masking as missing information with proper preprocessing.[Figure]
Revised: Hence, it is essential for the network to identify masking as missing information with proper preprocessing.

Original: Masking Introduces Parasitic Edges -
Revised: Masking Introduces Parasitic Edges -

Original: The convolutional kernels are well known for their edge detection behaviors (LeCun et al., 1998).
Revised: The convolutional kernels are well known for their edge detection behaviors (LeCun et al., 1998).

Original: Applying masks creates a large number of parasitic edges in the images.
Revised: Applying masks creates a large number of parasitic edges in the images.

Original: The feature maps generated by edge-detecting kernels are drastically distorted; hence, these kernels are suppressed during training by siamese networks.
Revised: The feature maps generated by edge-detecting kernels are drastically distorted; hence, these kernels are suppressed during training by siamese networks.

Original: More severally, these parasitic edges will remain in the intermediate feature maps and affect all the convolutional layers.
Revised: More severally, these parasitic edges will remain in the intermediate feature maps and affect all the convolutional layers.

Original: In Figure 1c, we visualize the encoder’s first convolutional layer kernels pretrained with standard augmentation or masked input.
Revised: In Figure 1c, we visualize the encoder’s first convolutional layer kernels pretrained with standard augmentation or masked input.

Original: Due to the parasitic edges, many kernels collapsed to trivial blank features.
Revised: Due to the parasitic edges, many kernels collapsed to trivial blank features.

Original: ViTs dodge this problem by simply matching the patch boundaries to the mask.
Revised: ViTs dodge this problem by simply matching the patch boundaries to the mask.

Original:  M ETHODS 5. P REPROCESSING
Revised:  M ETHODS 5. P REPROCESSING

Original: To solve the problems caused by masking in siamese networks with ConvNets backbones as discussed in Section 4, we propose to apply a high-pass filter during the preprocessing stage.
Revised: To solve the problems caused by masking in siamese networks with ConvNets backbones as discussed in Section 4, we propose to apply a high-pass filter during the preprocessing stage.

Original: See Figure 2.
Revised: See Figure 2.

Original: High-pass filter allows zeros to represent null information - With an high-pass filter, the zero values in the input image now represent null information instead of a regular pixel.
Revised: High-pass filter allows zeros to represent null information - With an high-pass filter, the zero values in the input image now represent null information instead of a regular pixel.

Original: Therefore, by applying an extra mask with zero value perfectly fit the value distribution and results in minimal information distortion change to the regular pixels.
Revised: Therefore, by applying an extra mask with zero value perfectly fit the value distribution and results in minimal information distortion change to the regular pixels.

Original: High-pass filter elimiates parastic edges - We observe that the masking edges on the high-pass filtered image becomes invisible.
Revised: High-pass filter elimiates parastic edges - We observe that the masking edges on the high-pass filtered image becomes invisible.

Original: See Figure 2.
Revised: See Figure 2.

Original: Empirical results also verify that the ConvNets encoder is able to learn useful edge detection kernels.
Revised: Empirical results also verify that the ConvNets encoder is able to learn useful edge detection kernels.

Original: : A high-pass filter is able to solve the problem caused by masks on ConvNets. The high-pass filter shifts the pixel value distribution so that zeros represent null information. These zeros mean that the original information can be interpolated from surrounding pixels. In addition, the mask edges on the image become invisibleafter applying the high-pass filter.
Revised: be addition invisible

Original: We quantitatively show that a high-pass filter as an extra preprocessing step significantly improves siamese networks’ performance with masked inputs.
Revised: We quantitatively show that a high-pass filter as an extra preprocessing step significantly improves siamese networks’ performance with masked inputs.

Original: We start with applying two random grid masks with grid size 32 on the same random crop with a fixed 30% masking ratio.
Revised: We start with applying two random grid masks with grid size 32 on the same random crop with a fixed 30% masking ratio.

Original: We do not apply any other  augmentations, such as random resized crops or color distortions.
Revised: We do not apply any other [Figure] augmentations, such as random resized crops or color distortions.

Original: This mask-only setting achieves a non-trivial 21 .
Revised: This mask-only setting achieves a non-trivial 21 .

Original: 0% linear probe accuracy on ImageNet.
Revised: 0% linear probe accuracy on ImageNet.

Original: By adding a high-pass filter, the accuracy improves to 30 .
Revised: By adding a high-pass filter, the accuracy improves to 30 .

Original: We empirically find that a smaller high-pass filter kernel σ = 5 is optimal, see
Revised: We empirically find that a smaller high-pass filter kernel σ = 5 is optimal, see

Original: Appendix B.
Revised: Appendix B.

Original: [Figure]

Original: One may suspect that a high-pass filter will eliminate low-frequency information and results in learning worse representations.
Revised: One may suspect that a high-pass filter will eliminate low-frequency information and results in learning worse representations.

Original: We empirically show that a high-pass filter will cause negligible change in terms of performance.
Revised: We empirically show that a high-pass filter will cause negligible change in terms of performance.

Original: We conduct supervised training experiments with or without a high-pass filter and evaluate their image classification accuracy.
Revised: We conduct supervised training experiments with or without a high-pass filter and evaluate their image classification accuracy.

Original: A ResNet-50 with a high-pass filter reaches 76 .
Revised: A ResNet-50 with a high-pass filter reaches 76 .

Original: 2% ImageNet supervised Top-1 accuracy, while the one without a high-pass filter gets 76 . 0% .
Revised: 2% ImageNet supervised Top-1 accuracy, while the one without a high-pass filter gets 76 . 0% .

Original: These two numbers are within an error range.
Revised: These two numbers are within an error range.

Original: 5. M ASKING S TRATEGY
Revised: 5. M ASKING S TRATEGY

Original: Besides standard augmentations, adding masks could prevent the encoder f θ ( · ) from learning some superficial features, as discussed in Sec 5, but at the same time, masking might also introduce new superficial features that confuse the encoder.
Revised: Besides standard augmentations, adding masks could prevent the encoder f θ ( · ) from learning some superficial features, as discussed in Sec 5, but at the same time, masking might also introduce new superficial features that confuse the encoder.

Original: We propose several masking designs to prevent such behaviors.
Revised: We propose several masking designs to prevent such behaviors.

Original: Noisy Mask - Inspired by masked language modeling Devlin et al.
Revised: Noisy Mask - Inspired by masked language modeling Devlin et al.

Original: (2019) where masked tokens were replaced by random tokens with a certain probability, we find that adding Gaussian noise to the masked area is beneficial and reduces the siamese networks’ reliance on augmentations such as color jittering.
Revised: (2019) where masked tokens were replaced by random tokens with a certain probability, we find that adding Gaussian noise to the masked area is beneficial and reduces the siamese networks’ reliance on augmentations such as color jittering.

Original: A noisy mask prevents the network from easily overfitting to use the color histogram as the entire feature, which is similar to the role of applying color-jittering on the unmasked area (Chen et al., 2020a).
Revised: A noisy mask prevents the network from easily overfitting to use the color histogram as the entire feature, which is similar to the role of applying color-jittering on the unmasked area (Chen et al., 2020a).

Original: Channel-Independent Mask -
Revised: Channel-Independent Mask -

Original: In a discrete space, it is natural to mask the entire patch or token.
Revised: In a discrete space, it is natural to mask the entire patch or token.

Original: However, for continuous input with ConvNets backbone, it is essential for the network to attend to not only spatial correlation but local channel-wise dependency.
Revised: However, for continuous input with ConvNets backbone, it is essential for the network to attend to not only spatial correlation but local channel-wise dependency.

Original: In standard siamese networks, this is done by handcrafting augmentations such as color-jittering and grayscale.
Revised: In standard siamese networks, this is done by handcrafting augmentations such as color-jittering and grayscale.

Original: Here, we propose to add such functionality for masks by introducing a channel-independent mask.
Revised: Here, we propose to add such functionality for masks by introducing a channel-independent mask.

Original: In addition to standard spatial-wise masking, where we apply the same mask on three color channels, we generate three random masks and apply them to each color channel separately.
Revised: In addition to standard spatial-wise masking, where we apply the same mask on three color channels, we generate three random masks and apply them to each color channel separately.

Original: Thus, this  is more general-purpose as it blurs the difference between spatial and channel dimensions.
Revised: Thus, this design is agnostic to input format as it blurs the difference between spatial and channel dimensions.

Original: We suspect that this design can further improve high-dimensional input such as RGBD images or videos.
Revised: We suspect that this design can further improve high-dimensional input such as RGBD images or videos.

Original: Combining Random Masks and Focal Masks -
Revised: Combining Random Masks and Focal Masks -

Original: In siamese networks with ViTs, Assran et al.
Revised: In siamese networks with ViTs, Assran et al.

Original: (2022) show that a mixture of random grid masks and focal masks can significantly boost the performance of the siamese network.
Revised: (2022) show that a mixture of random grid masks and focal masks can significantly boost the performance of the siamese network.

Original: This is also similar to (Bao et al., 2021) where larger blockwise masks are combined with randomly smaller ones.
Revised: This is also similar to (Bao et al., 2021) where larger blockwise masks are combined with randomly smaller ones.

Original: Compared to a random grid mask, a focal mask decreases the probability of similar augmented views of images from the different latent classes.
Revised: Compared to a random grid mask, a focal mask decreases the probability of similar augmented views of images from the different latent classes.

Original: Intuitively, the larger a patch is, the smaller the probability that images from the different latent classes could both generate the patch.
Revised: Intuitively, the larger a patch is, the smaller the probability that images from the different latent classes could both generate the patch.

Original: Therefore, the siamese network will be encouraged to learn semantic-level features.
Revised: Therefore, the siamese network will be encouraged to learn semantic-level features.

Original: In our design, we randomly apply focal masks or distributed grid masks for each view.
Revised: In our design, we randomly apply focal masks or distributed grid masks for each view.

Original: Asymmetric Augmentation - Wang et al.
Revised: Asymmetric Augmentation - Wang et al.

Original: (2022) highlighted the importance of asymmetry for siamese networks in augmentations on various siamese network frameworks.
Revised: (2022) highlighted the importance of asymmetry for siamese networks in augmentations on various siamese network frameworks.

Original: Here, we find thatmasking asymmetrically between the two branches in the siamese networks improves learned representations.
Revised: Here, we find that

Original: This has also been used in masked siamese networks with ViTs (Assran et al., 2022) that one branch always takes an unmasked view. Empirically, we find that applying masking on both branches is essential for good representations, especially for frameworks that do not use momentum
Revised: [Figure]

Original: 𝑍 𝑍
Revised: (𝑍

Original: [Figure]
Revised: [Figure]

Original: 𝐿(𝑍, 𝑍’)
Revised: 𝑍’)

Original: [Figure]
Revised: [Figure]

Original: encoders (He et al., 2020; Grill et al., 2020).
Revised: between in rep- masked networks with ViTs ( et al., 2022) do (He et al Grill al

Original: In our design, we will applymasking alongside otheraugmentations onboth branches with different magnitudes.
Revised: In our design we will apply alongside other on with magnitudes

Original: Multicrops - Besides the two main views used in siamese networks, Multicrops (Caron et al., 2020) feed additional views for each image and apply siamese networks between some pairs of them.
Revised: Multicrops - Besides the two main views used in siamese networks, Multicrops (Caron et al., 2020) feed additional views for each image and apply siamese networks between some pairs of them.

Original: Multicrops have been proven beneficial in various frameworks.
Revised: Multicrops have been proven beneficial in various frameworks.

Original: Instead of relying on small resolution views used in previous methods (Caron et al., 2020; Assran et al., 2022), we simply apply multiple views with the same resolution and apply siamese network loss on every pair of them.
Revised: Instead of relying on small resolution views used in previous methods (Caron et al., 2020; Assran et al., 2022), we simply apply multiple views with the same resolution and apply siamese network loss on every pair of them.

Original: Specifically, we propose to use two extra masked views.
Revised: Specifically, we propose to use two extra masked views.

Original: Thus, our strategy does not assume input format.
Revised: Thus, our strategy does not assume input format.

Original: 5. M ASKED S IAMESE C ONV N ETS F RAMEWORK
Revised: 5. M ASKED S IAMESE C ONV N ETS F RAMEWORK

Original: The overall Masked Siamese ConvNets (MSCN) architecture is shown in Figure 4.
Revised: The overall Masked Siamese ConvNets (MSCN) architecture is shown in Figure 4.

Original: This framework can be a simple add-on to various siamese network frameworks with arbitrary backbone architecture.
Revised: This framework can be a simple add-on to various siamese network frameworks with arbitrary backbone architecture.

Original: During preprocessing, the high-pass filter is conducted after normalization.
Revised: During preprocessing, the high-pass filter is conducted after normalization.

Original: During the augmentation stage on each branch, the masking is applied after other augmentations.
Revised: During the augmentation stage on each branch, the masking is applied after other augmentations.

Original:  R ESULTS
Revised:  R ESULTS

Original: In this section, we evaluate the representations obtained by a Masked Siamese ConvNets pretrained on the ImageNet-1K dataset (Russakovsky et al., 2015) with ResNet-50 backbone (He et al., 2016).
Revised: In this section, we evaluate the representations obtained by a Masked Siamese ConvNets pretrained on the ImageNet-1K dataset (Russakovsky et al., 2015) with ResNet-50 backbone (He et al., 2016).

Original: We train MSCN with different joint-embedding losses, namely SimCLR and BYOL.
Revised: We train MSCN with different joint-embedding losses, namely SimCLR and BYOL.

Original: The network is pretrained for 800/1000 epochs, including 10 epochs of warm-up, and uses a cosine learning rate schedule.
Revised: The network is pretrained for 800/1000 epochs, including 10 epochs of warm-up, and uses a cosine learning rate schedule.

Original: We use the LARS optimizer (You et al., 2017) with a batch size of 4096.
Revised: We use the LARS optimizer (You et al., 2017) with a batch size of 4096.

Original: All the hyperparameters, including learning rate, closely follow the original SimCLR and BYOL implementation so that we could have a fair comparison.
Revised: All the hyperparameters, including learning rate, closely follow the original SimCLR and BYOL implementation so that we could have a fair comparison.

Original: For each model, the pretraining is distributed across
Revised: For each model, the pretraining is distributed across

Original: V100 GPUs and takes approximately 180 hours.
Revised: V100 GPUs and takes approximately 180 hours.

Original: We list all the pretraining and evaluation details in supplementary material.
Revised: We list all the pretraining and evaluation details in supplementary material.

Original: 6. I MAGE C LASSIFICATION
Revised: 6. I MAGE C LASSIFICATION

Original: We first evaluate the representations on the ImageNet-1K dataset using linear probe and semisupervised classification.
Revised: We first evaluate the representations on the ImageNet-1K dataset using linear probe and semisupervised classification.

Original: We compare MSCN with baselines in Table 1.
Revised: We compare MSCN with baselines in Table 1.

Original: For linear probe, we train a linear classifier on 100% of the labels with frozen weights.
Revised: For linear probe, we train a linear classifier on 100% of the labels with frozen weights.

Original: MSCN improves SimCLR but performs slightly worse with the BYOL baseline.
Revised: MSCN improves SimCLR but performs slightly worse with the BYOL baseline.

Original: We suspect BYOL may require a different asymmetric masking setting due to its asymmetric nature of the framework.
Revised: We suspect BYOL may require a different asymmetric masking setting due to its asymmetric nature of the framework.

Original: For semi-supervised classification, we finetune the network using 1% of the labels.
Revised: For semi-supervised classification, we finetune the network using 1% of the labels.

Original: MSCN demonstrates superior performances.
Revised: MSCN demonstrates superior performances.

Original: The advantage of masked siamese networks in few-shot image classification tasks has also been observed by Assran et al.
Revised: The advantage of masked siamese networks in few-shot image classification tasks has also been observed by Assran et al.

Original: (2022) using ViTs.
Revised: (2022) using ViTs.

Original: We compare the effect of masking on ConvNets and ViTs in Table 2.
Revised: We compare the effect of masking on ConvNets and ViTs in Table 2.

Original: MSCN with a ConvNet backbone demonstrates similar behaviors to MSN with a ViT backbone.
Revised: MSCN with a ConvNet backbone demonstrates similar behaviors to MSN with a ViT backbone.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: T RANSFER L

Original: [Table]
Revised: [Table]

Revised: T RANSFER L EARNING

Original: We then evaluate the representations by transferring the network to other downstream tasks.
Revised: We then evaluate the representations by transferring the network to other downstream tasks.

Original: We report the transferred image classification results on iNaturalist 2018 (Horn et al., 2018) dataset and
Revised: We report the transferred image classification results on iNaturalist 2018 (Horn et al., 2018) dataset and

Original: Places-205 (Zhou et al., 2014) dataset in Table 3.
Revised: Places-205 (Zhou et al., 2014) dataset in Table 3.

Original: In Table 4, we report the object detection and instance segmentation performance on VOC07+

Revised: In Table 4, we report the object detection and instance segmentation performance on VOC07+

Original: (Everingham et al., 2009) and COCO datasets (Lin et al., 2014).
Revised: (Everingham et al., 2009) and COCO datasets (Lin et al., 2014).

Original: Comparing MSCN with other methods in Table 3.
Revised: Comparing MSCN with other methods in Table 3.

Original: We observe similar results as Table 1, that MSCN improves SimCLR but performs worse with BYOL.
Revised: We observe similar results as Table 1, that MSCN improves SimCLR but performs worse with BYOL.

Original: We still suspect that the worse performance of BYOL could come from the non-optimal masking hyperparameters.
Revised: We still suspect that the worse performance of BYOL could come from the non-optimal masking hyperparameters.

Original: Since most of the objects in Places-205 have a larger scale than the objects in iNaturalist 2018, the non-optimal masking hyperparameters cost more harm on the smaller scale objects than on larger-scale objects.
Revised: Since most of the objects in Places-205 have a larger scale than the objects in iNaturalist 2018, the non-optimal masking hyperparameters cost more harm on the smaller scale objects than on larger-scale objects.

Original: For the object detection and instance segmentation tasks, MSCN demonstrates superior performances over previous siamese network frameworks on VOC07+12 detection task and performs comparably to the state-of-the-art representation learning methods on COCO dataset. 
Revised: For the object detection and instance segmentation tasks, MSCN demonstrates superior performances over previous siamese network frameworks on VOC07+12 detection task and performs comparably to the state-of-the-art representation learning methods on COCO dataset. [Table] [Table] [Table]

Original: 6. A BLATION S TUDY
Revised: A BLATION S TUDY

Original: We conduct ablation experiments to gain insights into our masking design strategy.
Revised: We conduct ablation experiments to gain insights into our masking design strategy.

Original: By default, we pretrain MSCN for 100 epochs with the SimCLR framework.
Revised: By default, we pretrain MSCN for 100 epochs with the SimCLR framework.

Original: We measure the performance by linear probe accuracy on ImageNet-1K.
Revised: We measure the performance by linear probe accuracy on ImageNet-1K.

Original: Masking Ratio - We first explore the optimal masking ratio in Figure 5a.
Revised: Masking Ratio - We first explore the optimal masking ratio in Figure 5a.

Original: A small masking ratio of 15% is optimal for a ResNet-50 backbone.
Revised: A small masking ratio of 15% is optimal for a ResNet-50 backbone.

Original: This matches the observation in (Assran et al., 2022)

Original:  that[Table] [Table] smaller networks (e.g. ViT-S) prefer a smaller masking ratio.
Revised: This matches the observation in (Assran et al., 2022) that smaller networks (e.g. ViT-S) prefer a smaller masking ratio.

Original: We also observe that the accuracy is relatively stable against the masking ratio up to 50% with our masking strategy.
Revised: We also observe that the accuracy is relatively stable against the masking ratio up to 50% with our masking strategy.

Original: Masking Grid Size - Mask grid size is an important hyperparameter that controls the balance between local and global features in the input, and determine what the siamese network will learn.
Revised: Masking Grid Size - Mask grid size is an important hyperparameter that controls the balance between local and global features in the input, and determine what the siamese network will learn.

Original: In masked siamese networks with ViTs, the masking grid size is fixed, and it is always set to match the patch boundaries.
Revised: In masked siamese networks with ViTs, the masking grid size is fixed, and it is always set to match the patch boundaries.

Original: However, an optimal masking grid size can vary.
Revised: However, an optimal masking grid size can vary.

Original: In Figure 5b, we show that the siamese networks can benefit from a more appropriate mask grid size.
Revised: In Figure 5b, we show that the siamese networks can benefit from a more appropriate mask grid size.

Original: We observe a large grid size of 32 is optimal for our current masking strategy with ConvNets backbone.
Revised: We observe a large grid size of 32 is optimal for our current masking strategy with ConvNets backbone.

Original: Focal Mask Probability - We explore the optimal focal mask probability in Figure 5c.
Revised: Focal Mask Probability - We explore the optimal focal mask probability in Figure 5c.

Original: Combining focal masks with random grid masks leads to significant improvements.
Revised: Combining focal masks with random grid masks leads to significant improvements.

Original: It remains an interesting open direction on how to optimally mix local features with global ones.
Revised: It remains an interesting open direction on how to optimally mix local features with global ones.

Original: Independent Mask Probability - We explore the optimal independent mask probability in Table 5d.
Revised: Independent Mask Probability - We explore the optimal independent mask probability in Table 5d.

Original: Independent mask with high probability results in significant accuracy boost.
Revised: Independent mask with high probability results in significant accuracy boost.

Original:  D ISCUSSION AND F UTURE D IRECTIONS
Revised:  D ISCUSSION AND F UTURE D IRECTIONS

Original: General-purpose Siamese Networks - Human-designed augmentations leveraging domain knowledge are essential for siamese networks to learn useful representations.
Revised: General-purpose Siamese Networks - Human-designed augmentations leveraging domain knowledge are essential for siamese networks to learn useful representations.

Original: Similar to masked siamese  networks with ViTs (Assran et al., 2022), our approach still combines standard augmentations with masking by proposing masking as an extra augmentation.
Revised: Similar to masked siamese [Figure] networks with ViTs (Assran et al., 2022), our approach still combines standard augmentations with masking by proposing masking as an extra augmentation.

Original: However, it is desirable to find a domain-knowledge-free augmentation strategy so that this approach can be applied to more general domains or out-of-distribution scenarios.
Revised: However, it is desirable to find a domain-knowledge-free augmentation strategy so that this approach can be applied to more general domains or out-of-distribution scenarios.

Original: And it is reasonable to believe that the importance of domain-knowledge-specific augmentations will diminish with an increasing amount of pretraining data.
Revised: And it is reasonable to believe that the importance of domain-knowledge-specific augmentations will diminish with an increasing amount of pretraining data.

Original: Masked ConvNets for Masked Image Models - Besides siamese networks, masked image modeling based on denoising autoencoders (Vincent et al., 2008; 2010) is another promising approach for visual representation learning.
Revised: Masked ConvNets for Masked Image Models - Besides siamese networks, masked image modeling based on denoising autoencoders (Vincent et al., 2008; 2010) is another promising approach for visual representation learning.

Original: These methods (Bao et al., 2022; He et al., 2021) have demonstrated[Figure] impressive performances in visual representation learning.
Revised: These methods (Bao et al., 2022; He et al., 2021) have demonstrated impressive performances in visual representation learning.

Original: Unfortunately, masked autoencoders also fail to work withoff-the-shelf ConvNets caused by similar problems as mentioned in Secfor siamese networks.
Revised: Unfortunately, masked autoencoders also fail to work with ConvNets caused by similar problems as mentioned in Sec 4 for siamese networks.

Original: Even though the mask here serves a different purpose, we suspect that our design in Masked Siamese ConvNets may also apply to masked image modeling with ConvNets.
Revised: Even though the mask here serves a different purpose, we suspect that our design in Masked Siamese ConvNets may also apply to masked image modeling with ConvNets.

Original: We hope the discovery in this paper may shed light on general self-supervised learning and reduce the requirement for inductive bias of different architectures.
Revised: We hope the discovery in this paper may shed light on general self-supervised learning and reduce the requirement for inductive bias of different architectures.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This work explores whether masking can be applied as an general-purpose augmentation to siamese networks with arbitrary backbones, including ConvNets.
Revised: This work explores whether masking can be applied as an general-purpose augmentation to siamese networks with arbitrary backbones, including ConvNets.

Original: We first present the problems introduced by the use of masking as augmentation.
Revised: We first present the problems introduced by the use of masking as augmentation.

Original: We then propose an effectiveand general-purpose masking strategy and demonstrate its effectiveness on various siamese network frameworks.
Revised: We then propose an effective masking strategy and demonstrate its effectiveness on various siamese network frameworks.

Original: Our method performs competitively on few-shot image classification benchmarks and outperforms previous methods on object detection benchmarks.
Revised: Our method performs competitively on few-shot image classification benchmarks and outperforms previous methods on object detection benchmarks.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: The pretraining code can be found in the supplementary material.
Revised: The pretraining code can be found in the supplementary material.

Original: We also provide a detailed implementation setup for pre-training and downstream experiments in Appendix A. After publication, we will provide pretrained checkpoints and open-source the code on a public repo
Revised: We also provide a detailed implementation setup for pre-training and downstream experiments in Appendix A. After publication, we will provide pretrained checkpoints and open-source the code on a public repo.

Original: A I MPLEMENTATION D ETAIL A. P RETRAINING
Revised: A I MPLEMENTATION D ETAIL A. P RETRAINING

Original: We closely follow the original setting in (Chen et al., 2020a) for our MSCN (w/ SimCLR) pretraining and original setting in (Grill et al., 2020) for our MSCN (w/ BYOL) pretraining.
Revised: We closely follow the original setting in (Chen et al., 2020a) for our MSCN (w/ SimCLR) pretraining and original setting in (Grill et al., 2020) for our MSCN (w/ BYOL) pretraining.

Original: Augmentation - For both methods, we use the same augmentation methods.
Revised: Augmentation - For both methods, we use the same augmentation methods.

Original: Each augmented view is generated from a random set of augmentations from the same input image.
Revised: Each augmented view is generated from a random set of augmentations from the same input image.

Original: We apply a series of standard augmentations for each view, including random cropping, resizing to 224x224, random horizontal flipping, a random color-jittering, randomly converting to grayscale, and a random
Revised: We apply a series of standard augmentations for each view, including random cropping, resizing to 224x224, random horizontal flipping, a random color-jittering, randomly converting to grayscale, and a random

Original: Gaussian blur.
Revised: Gaussian blur.

Original: These augmentations are applied symmetrically on two branches.
Revised: These augmentations are applied symmetrically on two branches.

Original: Architecture - For MSCN (w/ SimCLR), the encoder is a ResNet-50 network without the final classification layer followed by a projector.
Revised: Architecture - For MSCN (w/ SimCLR), the encoder is a ResNet-50 network without the final classification layer followed by a projector.

Original: The projector is a two-layer MLP with input dimension 2048, hidden dimension 2048, and output dimension 256.
Revised: The projector is a two-layer MLP with input dimension 2048, hidden dimension 2048, and output dimension 256.

Original: The projector has ReLU between the two layers and batch normalization after every layer.
Revised: The projector has ReLU between the two layers and batch normalization after every layer.

Original: This 256-dimensional embedding is fed to the infoNCE loss.
Revised: This 256-dimensional embedding is fed to the infoNCE loss.

Original: We use a temperature 0 .
Revised: We use a temperature 0 .

Original: 2 for the infoNCE loss.
Revised: 2 for the infoNCE loss.

Original: For MSCN (w/ BYOL), the online encoder is a ResNet-50 network without the final classification layer.
Revised: For MSCN (w/ BYOL), the online encoder is a ResNet-50 network without the final classification layer.

Original: The online projector is a two-layer MLP with input dimension 2048, hidden dimension 4096, and output dimension 256.
Revised: The online projector is a two-layer MLP with input dimension 2048, hidden dimension 4096, and output dimension 256.

Original: The predictor is a two-layer MLP with input dimension 256, hidden dimension 4096, and output dimension 256.
Revised: The predictor is a two-layer MLP with input dimension 256, hidden dimension 4096, and output dimension 256.

Original: The projector and the predictor have ReLU between the two layers and batch normalization after every layer except the final linear layer.
Revised: The projector and the predictor have ReLU between the two layers and batch normalization after every layer except the final linear layer.

Original: The target encoder and projector are the exponential moving average of the online encoder and projector, with an initial momentum τ = 0 .
Revised: The target encoder and projector are the exponential moving average of the online encoder and projector, with an initial momentum τ = 0 .

Original: 996 with a cosine decay schedule to 1 . 

Revised: 996 with a cosine decay schedule to 1 .

Original: during the pretraining.
Revised: during the pretraining.

Original: Optimization - We follow the training protocol in (Zbontar et al., 2021) .
Revised: Optimization - We follow the training protocol in (Zbontar et al., 2021) .

Original: For MSCN (w/ SimCLR), we use a LARS optimizer and a base learning rate 4.8 using cosine learning rate decay schedule.
Revised: For MSCN (w/ SimCLR), we use a LARS optimizer and a base learning rate 4.8 using cosine learning rate decay schedule.

Original: We pretrain the model for 800 epochs with 10 epochs warm-up with batch size 4096.
Revised: We pretrain the model for 800 epochs with 10 epochs warm-up with batch size 4096.

Original: For MSCN (w/ BYOL), we use a LARS optimizer and a base learning rate 3.2 using cosine learning rate decay schedule.
Revised: For MSCN (w/ BYOL), we use a LARS optimizer and a base learning rate 3.2 using cosine learning rate decay schedule.

Original: We pretrain the model for 1000 epochs with 10 epochs warm-up with batch size 4096.
Revised: We pretrain the model for 1000 epochs with 10 epochs warm-up with batch size 4096.

Original: A. L INEAR P ROBE ON I MAGE N ET
Revised: A. L INEAR P ROBE ON I MAGE N ET

Original: We closely follow the setting used in (Zbontar et al., 2021) for our linear probe evaluation.
Revised: We closely follow the setting used in (Zbontar et al., 2021) for our linear probe evaluation.

Original: The linear classifier is trained for 100 epochs with a base learning rate of 1 . 0 and a cosine learning rate schedule.
Revised: The linear classifier is trained for 100 epochs with a base learning rate of 1 . 0 and a cosine learning rate schedule.

Original: We minimize the cross-entropy loss with the SGD optimizer with momentum and weight decay of 10 − 6 .
Revised: We minimize the cross-entropy loss with the SGD optimizer with momentum and weight decay of 10 − 6 .

Original: We use a batch size of 256.
Revised: We use a batch size of 256.

Original: At training time, we use random resized crops to 224x224, followed by random horizontal flip and a
Revised: At training time, we use random resized crops to 224x224, followed by random horizontal flip and a

Original: High-pass filter.
Revised: High-pass filter.

Original: At test time, we resize the image to 256×256 and center-crop it to a size of 224× and followed by a High-pass filter.
Revised: At test time, we resize the image to 256×256 and center-crop it to a size of 224× and followed by a High-pass filter.

Original: A. F INETUNING
Revised: A. F INETUNING

Original: We closely follow the setting used in (Zbontar et al., 2021) for our finetuning evaluation.
Revised: We closely follow the setting used in (Zbontar et al., 2021) for our finetuning evaluation.

Original: We finetune the ResNet-50 encoder for 20 epochs with a learning rate of 0 .
Revised: We finetune the ResNet-50 encoder for 20 epochs with a learning rate of 0 .

Original: 002 and the classifier with a base learning rate 0 .
Revised: 002 and the classifier with a base learning rate 0 .

Original: The learning rate is multiplied by a factor of 0.2 after the 12th and 16th epoch.
Revised: The learning rate is multiplied by a factor of 0.2 after the 12th and 16th epoch.

Original: We minimize the cross-entropy loss with the SGD optimizer with momentum and do not use weight decay.
Revised: We minimize the cross-entropy loss with the SGD optimizer with momentum and do not use weight decay.

Original: We use a batch size of 256.
Revised: We use a batch size of 256.

Original: The image augmentations are the same as in the linear evaluation setting.
Revised: The image augmentations are the same as in the linear evaluation setting.

Original: A. O BJECT D ETECTION
Revised: A. O BJECT D ETECTION

Original: We use the detectron2 library (Wu et al., 2019) for training the detection models.
Revised: We use the detectron2 library (Wu et al., 2019) for training the detection models.

Original: All the configuration files are from the VISSL library (Goyal et al., 2021b), which are closely follow the evaluation settings from (He et al., 2020).
Revised: All the configuration files are from the VISSL library (Goyal et al., 2021b), which are closely follow the evaluation settings from (He et al., 2020).

Original: The backbone ResNet50 network for Faster R-CNN (Ren et al., 2015) and
Revised: The backbone ResNet50 network for Faster R-CNN (Ren et al., 2015) and

Original: Mask R-CNN (He et al., 2017) is initialized using our pretrained model.
Revised: Mask R-

Original: VOC07+

Revised: VOC07+

Original: We use the VOC07+12 (Everingham et al., 2009) trainval set of 16K images for training a Faster R-CNN C-4 backbone for 24K iterations using a batch size of 16.
Revised: We use the VOC07+12 (Everingham et al., 2009) trainval set of 16K images for training a Faster R-CNN C-4 backbone for 24K iterations using a batch size of 16.

Original: The initial learning rate for the model is 0.085 which is reduced by a factor of 10 after 18K and 22K iterations.
Revised: The initial learning rate for the model is 0.085 which is reduced by a factor of 10 after 18K and 22K iterations.

Original: We use linear warmup (Goyal et al., 2019) with a slope of 0.333 for 1000 iterations.
Revised: We use linear warmup (Goyal et al., 2019) with a slope of 0.333 for 1000 iterations.

Original: COCO We train Mask R-CNN C-4 backbone on the COCO (Lin et al., 2014) 2017 train split for 90K iterations using a batch size of 16 The initial learning rate for the model is 0.05 which is reduced by a factor of 10 after 60K and 80K iterations.
Revised: COCO We train Mask R-CNN C-4 backbone on the COCO (Lin et al., 2014) 2017 train split for 90K iterations using a batch size of 16 The initial learning rate for the model is 0.05 which is reduced by a factor of 10 after 60K and 80K iterations.

Original: We use linear warmup (Goyal et al., 2019) with a slope of 0.333 for 1000 iterations.
Revised: We use linear warmup (Goyal et al., 2019) with a slope of 0.333 for 1000 iterations.

Original: We report results on the 2017 val split.
Revised: We report results on the 2017 val split.

Original: B A DDITIONAL A BLATION S TUDY
Revised: B A DDITIONAL A BLATION S TUDY

Original: We conduct additional ablation experiments to gain insights into our masking design strategy.
Revised: We conduct additional ablation experiments to gain insights into our masking design strategy.

Original: By default, we pretrain MSCN with SimCLR loss for 100 epochs.
Revised: By default, we pretrain MSCN with SimCLR loss for 100 epochs.

Original: We measure the performance by linear probe accuracy on ImageNet-1K.
Revised: We measure the performance by linear probe accuracy on ImageNet-1K.

Original: High-pass Sigma - We explore the optimal high-pass filter σ in Figure 6.
Revised: High-pass Sigma - We explore the optimal high-pass filter σ in Figure 6.

Original: In addition to the varying σ for pretraining, we also update the high-pass filter σ for the transformation during evaluation.
Revised: In addition to the varying σ for pretraining, we also update the high-pass filter σ for the transformation during evaluation.

Original: In practice, we prefer a small σ because there is less computational overhead.
Revised: In practice, we prefer a small σ because there is less computational overhead.

Original: [Figure]
Revised: [Figure]

Original: View Sharing -
Revised: View Sharing -

Original: In our masking strategy, we apply the standard augmentations to generate multiple views and then randomly apply masks on these views.
Revised: In our masking strategy, we apply the standard augmentations to generate multiple views and then randomly apply masks on these views.

Original: One alternative is to apply random masks on the same augmented view.
Revised: One alternative is to apply random masks on the same augmented view.

Original: Figure 5 shows that applying masks on the same view results in significantly worse representations.
Revised: Figure 5 shows that applying masks on the same view results in significantly worse representations.

Original: As discussed in our design principle, the masks are used to prevent superficial solutions based on masked areas.
Revised: As discussed in our design principle, the masks are used to prevent superficial solutions based on masked areas.

Original: It is still important to apply different augmentations to the original image to prevent the superficial solutions based on unmasked areas.
Revised: It is still important to apply different augmentations to the original image to prevent the superficial solutions based on unmasked areas.

Original: [Table]
Revised: [Table]

Revised: C A DDITIONAL V ISUALIZATION