Original: M ASKED F REQUENCY M ODELING FOR S ELF -S UPERVISED V ISUAL P RE -T RAINING
Revised: M ASKED F REQUENCY M ODELING FOR S ELF -S UPERVISED V ISUAL P RE -T RAINING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We present M asked F requency M odeling ( MFM ), a unified frequency-domainbased approach for self-supervised pre-training of visual models.
Revised: We present M asked F requency M odeling ( MFM ), a unified frequency-domainbased approach for self-supervised pre-training of visual models.

Original: Instead of randomly inserting mask tokens to the input embeddings in the spatial domain, in this paper, we shift the perspective to the frequency domain.
Revised: Instead of randomly inserting mask tokens to the input embeddings in the spatial domain, in this paper, we shift the perspective to the frequency domain.

Original: Specifically, MFM first masks out a portion of frequency components of the input image and then predicts the missing frequencies on the frequency spectrum.
Revised: Specifically, MFM first masks out a portion of frequency components of the input image and then predicts the missing frequencies on the frequency spectrum.

Original: Our key insight is that predicting masked components in the frequency domain is more ideal to reveal underlying image patterns rather than predicting masked patches in the spatial domain, due to the heavy spatial redundancy.
Revised: Our key insight is that predicting masked components in the frequency domain is more ideal to reveal underlying image patterns rather than predicting masked patches in the spatial domain, due to the heavy spatial redundancy.

Original: Our findings suggest that with the right configuration of mask-and-predict strategy, both the structural information within high-frequency components and the low-level statistics among low-frequency counterparts are useful in learning good representations.
Revised: Our findings suggest that with the right configuration of mask-and-predict strategy, both the structural information within high-frequency components and the low-level statistics among low-frequency counterparts are useful in learning good representations.

Original: For the first time, MFM demonstrates that, for both ViT and CNN, a simple non-Siamese framework can learn meaningful representations even using none of the following: (i) extra data, (ii) extra model, (iii) mask token.
Revised: For the first time, MFM demonstrates that, for both ViT and CNN, a simple non-Siamese framework can learn meaningful representations even using none of the following: (i) extra data, (ii) extra model, (iii) mask token.

Original: Experimental results on image classification and semantic segmentation, as well as several robustness benchmarks show the competitive performance and advanced robustness of MFM compared with recent masked image modeling approaches.
Revised: Experimental results on image classification and semantic segmentation, as well as several robustness benchmarks show the competitive performance and advanced robustness of MFM compared with recent masked image modeling approaches.

Original: Furthermore, we also comprehensively investigate the effectiveness of classical image restoration tasks for representation learning from a unified frequency perspective and reveal their intriguing relations with our MFM approach.
Revised: Furthermore, we also comprehensively investigate the effectiveness of classical image restoration tasks for representation learning from a unified frequency perspective and reveal their intriguing relations with our MFM approach.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Following the success of Masked Language Modeling (MLM) such as BERT (Devlin et al., 2019) in natural language processing (NLP), Masked Image Modeling (MIM) (Bao et al., 2022;
Revised: Following the success of Masked Language Modeling (MLM) such as BERT (Devlin et al., 2019) in natural language processing (NLP), Masked Image Modeling (MIM) (Bao et al., 2022;

Original: He et al., 2022; Wei et al., 2022; Xie et al., 2022) has shown promising performance in self-supervised pretraining of visual models.
Revised: He et al., 2022; Wei et al., 2022; Xie et al., 2022) has shown promising performance in self-supervised pretraining of visual models.

Original: Both MLM and MIM follow a common corrupt-and-predict paradigm – randomly masking a portion of input data and then learning to predict the missing parts.
Revised: Both MLM and MIM follow a common corrupt-and-predict paradigm – randomly masking a portion of input data and then learning to predict the missing parts.

Original: This simple recipe enables modern Transformer-based deep architectures (Vaswani et al., 2017; Dosovitskiy et al., 2020) to learn generalizable representations from ubiquitous unlabeled text or image data.
Revised: This simple recipe enables modern Transformer-based deep architectures (Vaswani et al., 2017; Dosovitskiy et al., 2020) to learn generalizable representations from ubiquitous unlabeled text or image data.

Original: By default, current MIM methods such as BEiT (Bao et al., 2022), MAE (He et al., 2022) and
Revised: By default, current MIM methods such as BEiT (Bao et al., 2022), MAE (He et al., 2022) and

Original: SimMIM (Xie et al., 2022) perform masking in the spatial domain by excluding image patches randomly, a strategy inspired by MLM that performs masking on words (Figure 1(a-b)).
Revised: SimMIM (Xie et al., 2022) perform masking in the spatial domain by excluding image patches randomly, a strategy inspired by MLM that performs masking on words (Figure 1(a-b)).

Original: However, unlike human-generated language that is succinct and highly semantic, raw pixel values in the spatial domain are of low information density.
Revised: However, unlike human-generated language that is succinct and highly semantic, raw pixel values in the spatial domain are of low information density.

Original: To cope with heavy spatial redundancy in images, MAE (He et al., 2022) shows that one would need to mask a very high proportion ( e.g. , 75%) to encourage the learning of meaningful features.
Revised: To cope with heavy spatial redundancy in images, MAE (He et al., 2022) shows that one would need to mask a very high proportion ( e.g. , 75%) to encourage the learning of meaningful features.

Original: Beyond masking image patches, which is a particular way of corruption, in this paper, we are interested in investigating the effectiveness of other corruption strategies for self-supervised representation learning.
Revised: Beyond masking image patches, which is a particular way of corruption, in this paper, we are interested in investigating the effectiveness of other corruption strategies for self-supervised representation learning.

Original: We first explore the corruption recipes commonly applied in low-level image processing tasks, including image super-resolution (SR), deblurring and denoising.
Revised: We first explore the corruption recipes commonly applied in low-level image processing tasks, including image super-resolution (SR), deblurring and denoising.

Original: As shown in
Revised: As shown in

Original: Figure 1(c), the downsampling, blur, and noise operations can degrade the exemplar image effectively in the spatial domain, thus potentially serving as useful corruption strategies.
Revised: Figure 1(c), the downsampling, blur, and noise operations can degrade the exemplar image effectively in the spatial domain, thus potentially serving as useful corruption strategies.

Original: However, the corruption induced in the spatial domain prevents us from analyzing what specific information is
Revised: However, the corruption induced in the spatial domain prevents us from analyzing what specific information is

Original: [Figure]
Revised: [Figure]

Original: a green grass covered
Revised: a green grass covered

Original: [Figure]
Revised: [Figure]

Original: small flowers.
Revised: small flowers.

Original: [Figure]
Revised: [Figure]

Original: white
Revised: white

Original: [Figure]
Revised: [Figure]

Original: green
Revised: green

Original: [Figure]
Revised: [Figure]

Original: corrupted and needs to be reconstructed.
Revised: corrupted and needs to be reconstructed.

Original: To better understand these low-level corruptions, we shift our attention from the spatial image domain to the frequency domain.
Revised: To better understand these low-level corruptions, we shift our attention from the spatial image domain to the frequency domain.

Original: In the frequency domain, one could observe underlying patterns of an image not conveniently visible from raw pixel values.
Revised: In the frequency domain, one could observe underlying patterns of an image not conveniently visible from raw pixel values.

Original: For example, the downsampling and blur operations dominantly remove the high-frequency image details, while adding noises tends to corrupt the full frequency spectrum of an image globally (Figure 1(c)).
Revised: For example, the downsampling and blur operations dominantly remove the high-frequency image details, while adding noises tends to corrupt the full frequency spectrum of an image globally (Figure 1(c)).

Original: Driven by this observation, we present a simple and effective masking strategy in the frequency domain for self-supervised visual representation learning, dubbed as M asked F requency M odeling ( MFM ).
Revised: Driven by this observation, we present a simple and effective masking strategy in the frequency domain for self-supervised visual representation learning, dubbed as M asked F requency M odeling ( MFM ).

Original: Specifically, we first perform Fast Fourier Transform (FFT) to convert each input image into its frequency representation, i.e. , frequency spectrum.
Revised: Specifically, we first perform Fast Fourier Transform (FFT) to convert each input image into its frequency representation, i.e. , frequency spectrum.

Original: We then mask a portion of frequencies on the frequency spectrum using a low-/high-pass filter.
Revised: We then mask a portion of frequencies on the frequency spectrum using a low-/high-pass filter.

Original: With inverse FFT (iFFT), we finally take the corrupted image with some of the frequencies attenuated as input.
Revised: With inverse FFT (iFFT), we finally take the corrupted image with some of the frequencies attenuated as input.

Original: Our encoder is quite flexible as no mask tokens are inserted.
Revised: Our encoder is quite flexible as no mask tokens are inserted.

Original: Thus, MFM can embrace both the vision Transformer (ViT)
Revised: Thus, MFM can embrace both the vision Transformer (ViT)

Original: (Dosovitskiy et al., 2020) and convolutional neural network (CNN) (LeCun et al., 1989) families.
Revised: (Dosovitskiy et al., 2020) and convolutional neural network (CNN) (LeCun et al., 1989) families.

Original: Our decoder is a lightweight linear layer that reconstructs the masked frequency values on the frequency spectrum via a frequency loss.
Revised: Our decoder is a lightweight linear layer that reconstructs the masked frequency values on the frequency spectrum via a frequency loss.

Original: As shown in Figure 1(d), an image with low or high frequencies attenuated would reveal entirely different patterns: the low-frequency components usually contain object smooth structure such as colors and styles, while the high-frequency counterparts largely depict the object outline or silhouette structure.
Revised: As shown in Figure 1(d), an image with low or high frequencies attenuated would reveal entirely different patterns: the low-frequency components usually contain object smooth structure such as colors and styles, while the high-frequency counterparts largely depict the object outline or silhouette structure.

Original: Such unique properties of the frequency domain make it appealing for reducing information redundancy, thus creating a nontrivial and meaningful self-supervisory task.
Revised: Such unique properties of the frequency domain make it appealing for reducing information redundancy, thus creating a nontrivial and meaningful self-supervisory task.

Original: Our contributions are summarized as follows:
Revised: Our contributions are summarized as follows:

Original: 1) We propose a new masked frequency modeling task to pre-train visual encoders in a selfsupervised manner.
Revised: 1) We propose a new masked frequency modeling task to pre-train visual encoders in a selfsupervised manner.

Original: Our MFM is agnostic to the architectures, and we demonstrate the flexibility of applying MFM for both ViT and CNN families.
Revised: Our MFM is agnostic to the architectures, and we demonstrate the flexibility of applying MFM for both ViT and CNN families.

Original: We contribute the first study of low-level corruption tasks for self-supervised learning (SSL) in frequency domain.
Revised: We contribute the first study of low-level corruption tasks for self-supervised learning (SSL) in frequency domain.

Original: We investigate the effectiveness of corruption strategies commonly adopted in low-level image processing tasks ( i.e. , SR, deblurring and denoising) for SSL from a unified frequency perspective and reveal that the representation learning capability of these corruption tasks actually depends on the architectures: they can achieve comparable and even better results than their supervised counterpart on ViT, but no gains are observed on CNN.
Revised: We investigate the effectiveness of corruption strategies commonly adopted in low-level image processing tasks ( i.e. , SR, deblurring and denoising) for SSL from a unified frequency perspective and reveal that the representation learning capability of these corruption tasks actually depends on the architectures: they can achieve comparable and even better results than their supervised counterpart on ViT, but no gains are observed on CNN.

Original: 3) Extensive experiments show that our MFM can achieve competitive performance among existing MIM approaches on downstream tasks, such as image classification and semantic segmentation, while not using mask tokens or other more complex designs.
Revised: 3) Extensive experiments show that our MFM can achieve competitive performance among existing MIM approaches on downstream tasks, such as image classification and semantic segmentation, while not using mask tokens or other more complex designs.

Original: Further analysis on several robustness benchmarks also exhibits more appealing robustness of the studied corruption tasks than MIM.
Revised: Further analysis on several robustness benchmarks also exhibits more appealing robustness of the studied corruption tasks than MIM.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Masked language modeling and its auto-regressive variants, such as BERT (Devlin et al., 2019) and GPT (Radford et al., 2018; 2019; Brown et al., 2020), have achieved great success in pretraining large-scale language models in the NLP community.
Revised: Masked language modeling and its auto-regressive variants, such as BERT (Devlin et al., 2019) and GPT (Radford et al., 2018; 2019; Brown et al., 2020), have achieved great success in pretraining large-scale language models in the NLP community.

Original: These approaches perform masking on the human-generated language by holding out random words and then predicting the missing content.
Revised: These approaches perform masking on the human-generated language by holding out random words and then predicting the missing content.

Original: This simple mask-word recipe has shown excellent ability in pre-training generalizable representations for broad NLP applications.
Revised: This simple mask-word recipe has shown excellent ability in pre-training generalizable representations for broad NLP applications.

Original: Masked image modeling leverages images corrupted by masking to learn useful representations.
Revised: Masked image modeling leverages images corrupted by masking to learn useful representations.

Original: Pioneered with stacked autoencoders (Vincent et al., 2010) and context encoders (Pathak et al., 2016) using CNNs, recent approaches (Bao et al., 2022;
Revised: Pioneered with stacked autoencoders (Vincent et al., 2010) and context encoders (Pathak et al., 2016) using CNNs, recent approaches (Bao et al., 2022;

Original: He et al., 2022; Xie et al., 2022; Wei et al., 2022;
Revised: He et al., 2022; Xie et al., 2022; Wei et al., 2022;

Original: Chen et al., 2022) follow the mask-word strategy in NLP to randomly mask image patches in the spatial domain using the vision Transformers (Dosovitskiy et al., 2020; Liu et al., 2021).
Revised: Chen et al., 2022) follow the mask-word strategy in NLP to randomly mask image patches in the spatial domain using the vision Transformers (Dosovitskiy et al., 2020; Liu et al., 2021).

Original: Along with this mask-patch strategy, different types of prediction targets have been studied, including discrete tokens (Bao et al., 2022; Dong et al., 2021), raw pixels (He et al., 2022; Xie et al., 2022), and handcrafted features (Wei et al., 2022).
Revised: Along with this mask-patch strategy, different types of prediction targets have been studied, including discrete tokens (Bao et al., 2022; Dong et al., 2021), raw pixels (He et al., 2022; Xie et al., 2022), and handcrafted features (Wei et al., 2022).

Original: Besides, iGPT (Chen et al., 2020a) takes a low-resolution image sequence as input and predicts missing pixels in an auto-regressive manner.
Revised: Besides, iGPT (Chen et al., 2020a) takes a low-resolution image sequence as input and predicts missing pixels in an auto-regressive manner.

Original: Several methods (Zhou et al., 2022; El-Nouby et al., 2021) also integrate MIM into contrastive-based Siamese frameworks.
Revised: Several methods (Zhou et al., 2022; El-Nouby et al., 2021) also integrate MIM into contrastive-based Siamese frameworks.

Original: Our work differs from previous approaches in that we perform masking in the frequency domain, which relies on none of the following: (i) extra data (Bao et al., 2022;
Revised: Our work differs from previous approaches in that we perform masking in the frequency domain, which relies on none of the following: (i) extra data (Bao et al., 2022;

Original: Dong et al., 2021; Fang et al., 2022), (ii) extra model (Zhou et al., 2022; El-Nouby et al., 2021; Fang et al., 2022; Shi et al., 2022; Chen et al., 2022), or (iii) mask token (Bao et al., 2022; He et al., 2022; Xie et al., 2022; Wei et al., 2022; Chen et al., 2022).
Revised: Dong et al., 2021; Fang et al., 2022), (ii) extra model (Zhou et al., 2022; El-Nouby et al., 2021; Fang et al., 2022; Shi et al., 2022; Chen et al., 2022), or (iii) mask token (Bao et al., 2022; He et al., 2022; Xie et al., 2022; Wei et al., 2022; Chen et al., 2022).

Original: CIM (Fang et al., 2022) also does not use mask token.
Revised: CIM (Fang et al., 2022) also does not use mask token.

Original: However, introducing an auxiliary generator to corrupt the input images adds nontrivial pre-training overhead.
Revised: However, introducing an auxiliary generator to corrupt the input images adds nontrivial pre-training overhead.

Original: In contrast, our frequency-domain-based corruption strategy can achieve comparable performance with negligible computational cost.
Revised: In contrast, our frequency-domain-based corruption strategy can achieve comparable performance with negligible computational cost.

Original: Self-supervised learning mainly focuses on designing effective pretext tasks for pre-training (Doersch et al., 2015; Wang & Gupta, 2015; Noroozi & Favaro, 2016;
Revised: Self-supervised learning mainly focuses on designing effective pretext tasks for pre-training (Doersch et al., 2015; Wang & Gupta, 2015; Noroozi & Favaro, 2016;

Original: Larsson et al., 2016; Zhang et al., 2016; 2017c; Noroozi et al., 2017; Bojanowski & Joulin, 2017; Pathak et al., 2017; Gidaris et al., 2018).
Revised: Larsson et al., 2016; Zhang et al., 2016; 2017c; Noroozi et al., 2017; Bojanowski & Joulin, 2017; Pathak et al., 2017; Gidaris et al., 2018).

Original: Contrastive learning (Wu et al., 2018;
Revised: Contrastive learning (Wu et al., 2018;

Original: He et al., 2020; Misra & Maaten, 2020; Chen et al.
Revised: He et al., 2020; Misra & Maaten, 2020; Chen et al.

Original: 2020b;c; Grill et al., 2020; Chen & He, 2021; Chen et al., 2021; Caron et al., 2021) has dominated the field over the past few years.
Revised: 2020b;c; Grill et al., 2020; Chen & He, 2021; Chen et al., 2021; Caron et al., 2021) has dominated the field over the past few years.

Original: Unlike the mask-and-predict pretext task, contrastive learning typically uses a Siamese framework and greatly relies on data augmentation.
Revised: Unlike the mask-and-predict pretext task, contrastive learning typically uses a Siamese framework and greatly relies on data augmentation.

Original: Low-level image processing tasks, such as image super-resolution (Dong et al., 2015), deblurring (Zhang et al., 2022) and denoising (Zhang et al., 2017b), focus on restoring the high-fidelity image from its corrupted input.
Revised: Low-level image processing tasks, such as image super-resolution (Dong et al., 2015), deblurring (Zhang et al., 2022) and denoising (Zhang et al., 2017b), focus on restoring the high-fidelity image from its corrupted input.

Original: The corrupted images are usually generated with degradation transformations, which consist of downsampling, blur, noise and JPEG compression.
Revised: The corrupted images are usually generated with degradation transformations, which consist of downsampling, blur, noise and JPEG compression.

Original: Recent promising results of MIM motivate us to investigate the effectiveness of these corruption operations in the context of representation learning.
Revised: Recent promising results of MIM motivate us to investigate the effectiveness of these corruption operations in the context of representation learning.

Original: Frequency domain analysis has been widely adopted in many computer vision tasks, such as image generation (Jiang et al., 2021), domain adaptation (Xu et al., 2021), and image superresolution (Pang et al., 2020).
Revised: Frequency domain analysis has been widely adopted in many computer vision tasks, such as image generation (Jiang et al., 2021), domain adaptation (Xu et al., 2021), and image superresolution (Pang et al., 2020).

Original: Early studies (Oppenheim et al., 1979; Oppenheim & Lim, 1981;
Revised: Early studies (Oppenheim et al., 1979; Oppenheim & Lim, 1981;

Original: Piotrowski & Campbell, 1982; Hansen & Hess, 2007) have revealed that in the frequency domain, the phase component largely captures high-level semantics of the original signals, while the amplitude component mainly retains low-level statistics.
Revised: Piotrowski & Campbell, 1982; Hansen & Hess, 2007) have revealed that in the frequency domain, the phase component largely captures high-level semantics of the original signals, while the amplitude component mainly retains low-level statistics.

Original: As such, underlying image patterns can be more conveniently observed in the frequency representation, compared with the raw pixel values in the spatial domain.
Revised: As such, underlying image patterns can be more conveniently observed in the frequency representation, compared with the raw pixel values in the spatial domain.

Original: Motivated by the intriguing properties of the Fourier domain, we propose a novel mask-frequency recipe and conduct the first study w.r.t.
Revised: Motivated by the intriguing properties of the Fourier domain, we propose a novel mask-frequency recipe and conduct the first study w.r.t.

Original: masked information modeling in the frequency domain for image data.
Revised: masked information modeling in the frequency domain for image data.

Original:  A PPROACH
Revised:  A PPROACH

Original: Our masked frequency modeling (MFM) is a simple yet effective self-supervised pre-training approach, which masks out a portion of image frequency components and predicts the missing frequencies on the frequency spectrum.
Revised: Our masked frequency modeling (MFM) is a simple yet effective self-supervised pre-training approach, which masks out a portion of image frequency components and predicts the missing frequencies on the frequency spectrum.

Original: Figure 2 shows the overview of our approach.
Revised: Figure 2 shows the overview of our approach.

Original: The framework consists of four components: masking strategy, encoder, decoder, and reconstruction target.
Revised: The framework consists of four components: masking strategy, encoder, decoder, and reconstruction target.

Original: We first
Revised: We first

Original: [Figure]
Revised: [Figure]

Original: detail each component of MFM in Section 3.1, and then discuss the relation of our approach with low-level image processing tasks in Section 3.2.
Revised: detail each component of MFM in Section 3.1, and then discuss the relation of our approach with low-level image processing tasks in Section 3.2.

Original: 3. M ASKED F REQUENCY M ODELING
Revised: 3. M ASKED F REQUENCY M ODELING

Original: Preliminary: Frequency representation of images.
Revised: Preliminary: Frequency representation of images.

Original: Given a single channel image 1 x ∈ R H × W , we can obtain the corresponding frequency representation via 2D Discrete Fourier Transform F ( x ) : 
Revised: Given a single channel image 1 x ∈ R H × W , we can obtain the corresponding frequency representation via 2D Discrete Fourier Transform F ( x ) : [Equation]

Original: [Equation]
Revised: [Equation]

Original:  where x ( h, w ) is the real pixel value at the coordinate of ( h, w ) on the spatial image, F ( x ) ( u, v ) is the complex frequency value at the coordinate of ( u, v ) on the frequency spectrum, e and i are Euler’s number and the imaginary unit, respectively.
Revised: ) where x ( h, w ) is the real pixel value at the coordinate of ( h, w ) on the spatial image, F ( x ) ( u, v ) is the complex frequency value at the coordinate of ( u, v ) on the frequency spectrum, e and i are Euler’s number and the imaginary unit, respectively.

Original: Accordingly, F − 1 ( x ) defines the inverse Fourier transform that maps spectral signals back into original image space.
Revised: Accordingly, F − 1 ( x ) defines the inverse Fourier transform that maps spectral signals back into original image space.

Original: Both the Fourier transform and its inverse can be calculated efficiently using the FFT algorithm (Nussbaumer, 1981).
Revised: Both the Fourier transform and its inverse can be calculated efficiently using the FFT algorithm (Nussbaumer, 1981).

Original: Masking strategy.
Revised: Masking strategy.

Original: We define a mask M ∈ { 0 , 1 } H × W , whose value is determined by a thresholding function that separates the low and high frequency components from F ( x ) according to a hyper-parameter, i.e. , radius r :
Revised: We define a mask M ∈ { 0 , 1 } H × W , whose value is determined by a thresholding function that separates the low and high frequency components from F ( x ) according to a hyper-parameter, i.e. , radius r :

Original: [Equation]
Revised: [Equation]

Original: where ( c h , c w ) denotes the center of the image, d ( · , · ) denotes a certain distance criterion.
Revised: where ( c h , c w ) denotes the center of the image, d ( · , · ) denotes a certain distance criterion.

Original: Here, we use the Euclidean distance, i.e. , a circle mask as default.
Revised: Here, we use the Euclidean distance, i.e. , a circle mask as default.

Original: Note that the mask shape is not solely restricted to a circle one, and we study the effects of different mask shapes in the experiment section.
Revised: Note that the mask shape is not solely restricted to a circle one, and we study the effects of different mask shapes in the experiment section.

Original: With the predefined mask M , we can easily obtain the decomposed low-pass filtered image x l and the high-pass filtered counterpart x h as follows:
Revised: With the predefined mask M , we can easily obtain the decomposed low-pass filtered image x l and the high-pass filtered counterpart x h as follows:

Original: [Equation]
Revised: [Equation]

Original: where 1 is the all-ones matrix, ⊙ is the Hadamard product between matrices.
Revised: where 1 is the all-ones matrix, ⊙ is the Hadamard product between matrices.

Original: These filtered images are then randomly selected  and fed to an encoder as input, following a Bernoulli distribution.
Revised: These filtered images are then randomly selected with a Bernoulli distribution and fed to an encoder as input 2 .

Original: Note that despite performing masking in the frequency domain, we still take the converted spatial images as input such that our model would not suffer an input domain gap between pre-training and finetuning.

Original: MFM encoder.
Revised: MFM encoder.

Original: The architecture of our encoder is quite flexible since we do not insert any mask tokens on the corrupted non-overlapping patch embeddings as in MIM (Bao et al., 2022;
Revised: The architecture of our encoder is quite flexible since we do not insert any mask tokens on the corrupted non-overlapping patch embeddings as in MIM (Bao et al., 2022;

Original: He et al., 2022; Xie et al., 2022; Wei et al., 2022).
Revised: He et al., 2022; Xie et al., 2022; Wei et al., 2022).

Original: Therefore, our MFM can be applied on both ViT and CNN architectures without any special designs.
Revised: Therefore, our MFM can be applied on both ViT and CNN architectures without any special designs.

Original: In this paper, we mainly use a standard ViT (Dosovitskiy et al., 2020) as our encoder for a direct comparison with MIM methods.
Revised: In this paper, we mainly use a standard ViT (Dosovitskiy et al., 2020) as our encoder for a direct comparison with MIM methods.

Original: Specifically, we first divide a filtered  image into regular non-overlapping patches.
Revised: Specifically, we first divide a filtered spatial image into regular non-overlapping patches.

Original: Then, the encoder embeds the patches by linear projection with added positional embeddings.
Revised: Then, the encoder embeds the patches by linear projection with added positional embeddings.

Original: The combined embeddings are then processed via a series of self-attention-based Transformer blocks (Vaswani et al., 2017).
Revised: The combined embeddings are then processed via a series of self-attention-based Transformer blocks (Vaswani et al., 2017).

Original: We also consider a typical CNN architecture, i.e. , ResNet-50 (He et al., 2016), to demonstrate the versatility of MFM.
Revised: We also consider a typical CNN architecture, i.e. , ResNet-50 (He et al., 2016), to demonstrate the versatility of MFM.

Original: To this end, we simply send the filtered  image to the CNN encoder as input.
Revised: To this end, we simply send the filtered spatial image to the CNN encoder as input.

Original: MFM decoder.
Revised: MFM decoder.

Original: The decoder accomplishes the frequency reconstruction task.
Revised: The decoder accomplishes the frequency reconstruction task.

Original: It can be of arbitrary form as long as its input is compatible with the encoder’s output.
Revised: It can be of arbitrary form as long as its input is compatible with the encoder’s output.

Original: The behaviors of different decoder architectures are beyond the focus of this study. Here, we simply adopt a lightweight linear layer as our decoder for efficiency. Note that the decoder is only employed during pre-training and is removed for downstream tasks.
Revised: Here, we simply adopt a lightweight linear layer as our decoder for efficiency, after which we perform FFT to convert each output image into the frequency domain for frequency reconstruction. The effect of different decoders are further studied in Appendix A.

Original: Reconstruction target.
Revised: Reconstruction target.

Original: Our MFM reconstructs the input by predicting the missing frequency values on the frequency spectrum.
Revised: Our MFM reconstructs the input by predicting the missing frequency values on the frequency spectrum.

Original: To faithfully recover the frequency values, we should define a frequency distance metric that considers both amplitude and phase as a loss function.
Revised: To faithfully recover the frequency values, we should define a frequency distance metric that considers both amplitude and phase as a loss function.

Original: Regarding each frequency value F ( x ) ( u, v ) as a two-dimensional Euclidean vector ⃗f , one can easily derive that the magnitude of the vector corresponds to the amplitude while the angle corresponds to the phase.
Revised: Regarding each frequency value F ( x ) ( u, v ) as a two-dimensional Euclidean vector ⃗f , one can easily derive that the magnitude of the vector corresponds to the amplitude while the angle corresponds to the phase.

Original: Inspired by Jiang et al.
Revised: Inspired by Jiang et al.

Original: (2021), we thus define the frequency distance D ( · , · ) as the distance between the reconstructed vector ⃗f r and the original vector ⃗f o at each spectrum coordinate ( u, v ) :
Revised: (2021), we thus define the frequency distance D ( · , · ) as the distance between the reconstructed vector ⃗f r and the original vector ⃗f o at each spectrum coordinate ( u, v ) :

Original: [Equation]
Revised: [Equation]

Original: where R ( x ) and I ( x ) are the real and imaginary part of F ( x ) , respectively, γ is an exponent to control the sharpness of the distance function and is set to 1 by default.
Revised: where R ( x ) and I ( x ) are the real and imaginary part of F ( x ) , respectively, γ is an exponent to control the sharpness of the distance function and is set to 1 by default.

Original: For each image, the final loss function, i.e. , the average frequency distance of all spectrum positions can thus be written as:
Revised: For each image, the final loss function, i.e. , the average frequency distance of all spectrum positions can thus be written as:

Original: [Equation]
Revised: [Equation]

Original: In practice, we compute the loss only on the masked area of the frequency spectrum instead of the full spectrum as the latter tends to decrease the accuracy according to our experiments.
Revised: In practice, we compute the loss only on the masked area of the frequency spectrum instead of the full spectrum as the latter tends to decrease the accuracy according to our experiments.

Original: 3. R ELATION WITH L OW -L EVEL I MAGE P ROCESSING T ASKS
Revised: 3. R ELATION WITH L OW -L EVEL I MAGE P ROCESSING T ASKS

Original: The notion of recovering masked frequency components in MFM is reminiscent to the objectives in low-level image processing tasks, such as image super-resolution (SR), deblurring and denoising.
Revised: The notion of recovering masked frequency components in MFM is reminiscent to the objectives in low-level image processing tasks, such as image super-resolution (SR), deblurring and denoising.

Original: In these tasks, a model takes a degraded image as input, and the aim is to restore the missing components.
Revised: In these tasks, a model takes a degraded image as input, and the aim is to restore the missing components.

Original: Different degradations corrupt different components in the frequency domain.
Revised: Different degradations corrupt different components in the frequency domain.

Original: As discussed in Section 1, for the image SR and deblurring tasks, most of the high-frequency components are removed while the low-frequency counterparts are retained; for the image denoising task, both lowand high-frequencies are significantly altered.
Revised: As discussed in Section 1, for the image SR and deblurring tasks, most of the high-frequency components are removed while the low-frequency counterparts are retained; for the image denoising task, both lowand high-frequencies are significantly altered.

Original: By analyzing the frequency spectrum of these tasks, we can observe how different frequencies of an image contribute to visual representation learning, thus gaining better insights on designing more effective learning objectives.
Revised: By analyzing the frequency spectrum of these tasks, we can observe how different frequencies of an image contribute to visual representation learning, thus gaining better insights on designing more effective learning objectives.

Original: Compared with these tasks, MFM provides a more general and unified frequency perspective to perform these low-level corruptions while being conceptually simpler: we directly remove certain frequencies on the frequency spectrum via a low-/high-pass filter.
Revised: Compared with these tasks, MFM provides a more general and unified frequency perspective to perform these low-level corruptions while being conceptually simpler: we directly remove certain frequencies on the frequency spectrum via a low-/high-pass filter.

Original: Our experiments show that MFM can achieve better performance than these tasks for representation learning.
Revised: Our experiments show that MFM can achieve better performance than these tasks for representation learning.

Original: We will comprehensively study these tasks and show more details in the experiment section.
Revised: We will comprehensively study these tasks and show more details in the experiment section.

Original:  E XPERIMENTS4. I MPLEMENTATION D ETAILS
Revised:  E XPERIMENTS

Revised:  I MPLEMENTATION D ETAILS

Original: We use the vanilla ViT-Small (ViT-S/16), ViT-Base (ViT-B/16) and ResNet-50 models as the backbones in our study.
Revised: We use the vanilla ViT-Small (ViT-S/16), ViT-Base (ViT-B/16) and ResNet-50 models as the backbones in our study.

Original: We perform self-supervised pre-training on the ImageNet-1K (Deng et al., 2009) training set without labels.
Revised: We perform self-supervised pre-training on the ImageNet-1K (Deng et al., 2009) training set without labels.

Original: For ViT, our pre-training setting generally follows BEiT
Revised: For ViT, our pre-training setting generally follows BEiT

Original: (Bao et al., 2022), while we simply use random resized cropping ( 224 × 224 resolution) and flipping as data
Revised: (Bao et al., 2022), while we only use random resized cropping ( 224 × 224 resolution) and flipping as data

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: augmentation, with dropout and stochastic depth not applied.
Revised: augmentation, with dropout and stochastic depth not applied.

Original: We also do not use relative position or layer scaling.
Revised: We also do not use relative position or layer scaling.

Original: After pre-training, we conduct supervised end-to-end fine-tuning on ImageNet-1K image classification and ADE20K
Revised: After pre-training, we conduct supervised end-to-end fine-tuning on ImageNet-1K image classification and ADE20K

Original: (Zhou et al., 2017) semantic segmentation to evaluate the quality of learned representations, following BEiT (Bao et al., 2022).
Revised: (Zhou et al., 2017) semantic segmentation to evaluate the quality of learned representations, following BEiT (Bao et al., 2022).

Original: For ResNet-50, we adopt the same pre-training configuration as that in ViT without further parameter tuning.
Revised: For ResNet-50, we adopt the same pre-training configuration as that in ViT without further parameter tuning.

Original: We provide the detailed pre-training and fine-tuning recipes in Appendix C.
Revised: We provide the detailed pre-training and fine-tuning recipes in Appendix G.

Original: M AIN P ROPERTIES
Revised: M AIN P ROPERTIES

Original: We start by ablating our MFM using ViT-B/16 as the default backbone.
Revised: We start by ablating our MFM using ViT-B/16 as the default backbone.

Original: All experiments are conducted with 300-epoch pre-training and 100-epoch fine-tuning on the ImageNet-1K dataset unless otherwise specified.
Revised: All experiments are conducted with 300-epoch pre-training and 100-epoch fine-tuning on the ImageNet-1K dataset unless otherwise specified.

Original: Several intriguing properties are observed.
Revised: Several intriguing properties are observed.

Original: Masking strategy.
Revised: Masking strategy.

Original: We first study different masking strategies on the frequency spectrum.
Revised: We first study different masking strategies on the frequency spectrum.

Original: We consider two kinds of filters: low-pass filter ( i.e. , mask high frequencies), and high-pass filter ( i.e. , mask low frequencies).
Revised: We consider two kinds of filters: low-pass filter ( i.e. , mask high frequencies), and high-pass filter ( i.e. , mask low frequencies).

Original: As shown in Table 1a, masking and predicting either high frequencies (“lowpass” entry) or low frequencies (“high-pass” entry) perform significantly better than simply encoding and reconstructing the original image (“none” entry).
Revised: As shown in Table 1a, masking and predicting either high frequencies (“lowpass” entry) or low frequencies (“high-pass” entry) perform significantly better than simply encoding and reconstructing the original image (“none” entry).

Original: This indicates that both high-frequency and low-frequency components are useful in representation learning, where the former largely depicts the object structure information such as outline or silhouette and the latter usually captures low-level statistics such as colors and styles.
Revised: This indicates that both high-frequency and low-frequency components are useful in representation learning, where the former largely depicts the object structure information such as outline or silhouette and the latter usually captures low-level statistics such as colors and styles.

Original: A random variant, i.e. , randomly selecting one filter from both low-pass and high-pass filters (“random” entry), benefits from all lens of frequencies, thus further improving the performance.
Revised: A random variant, i.e. , randomly selecting one filter from both low-pass and high-pass filters (“random” entry), benefits from all lens of frequencies, thus further improving the performance.

Original: Mask radius.
Revised: Mask radius.

Original: Table 1b studies the effect of mask radius, which controls the difficulty of our task.
Revised: Table 1b studies the effect of mask radius, which controls the difficulty of our task.

Original: A larger radius leaves more frequencies for a low-pass filter while removes more frequencies for a high-pass filter.
Revised: A larger radius leaves more frequencies for a low-pass filter while removes more frequencies for a high-pass filter.

Original: MFM works the best with a moderate difficulty.
Revised: MFM works the best with a moderate difficulty.

Original: Using a fixed radius ( e.g. , 16) performs slightly better than a random one, i.e. , the radius is uniformly sampled within a range ( e.g. , [8 , 24] ).
Revised: Using a fixed radius ( e.g. , 16) performs slightly better than a random one, i.e. , the radius is uniformly sampled within a range ( e.g. , [8 , 24] ).

Original: Mask shape.
Revised: Mask shape.

Original: We study three centrosymmetric mask shapes in Table 1c.
Revised: We study three centrosymmetric mask shapes in Table 1c.

Original: Different mask shapes focus on different masking directions.
Revised: Different mask shapes focus on different masking directions.

Original: Take low-pass filter as an example, a square shape removes more frequencies in the horizontal and vertical direction, while a rhombus one removes more in the diagonal direction.
Revised: Take low-pass filter as an example, a square shape removes more frequencies in the horizontal and vertical direction, while a rhombus one removes more in the diagonal direction.

Original: The results demonstrate that a circle mask shape that pays an equal attention to each direction on the frequency spectrum performs the best.
Revised: The results demonstrate that a circle mask shape that pays an equal attention to each direction on the frequency spectrum performs the best.

Original: We hypothesize that the effect of different mask shapes is largely correlated with the category statistics of pre-training datasets.
Revised: We hypothesize that the effect of different mask shapes is largely correlated with the category statistics of pre-training datasets.

Original: [Table]
Revised: [Table]

Original: Sampling ratio.
Revised: Sampling ratio.

Original: Table 1d ablates different sampling ratios for low-/high-pass filters.
Revised: Table 1d ablates different sampling ratios for low-/high-pass filters.

Original: Here, the sampling ratio denotes the probability of sampling a low-pass filter, following a Bernoulli distribution.
Revised: Here, the sampling ratio denotes the probability of sampling a low-pass filter, following a Bernoulli distribution.

Original: The results show that simply sampling both filters with an equal probability works the best.
Revised: The results show that simply sampling both filters with an equal probability works the best.

Original: Reconstruction target.
Revised: Reconstruction target.

Original: Table 1e compares two reconstruction targets: 1) predicting only the masked frequencies on the frequency spectrum as in our default setting
Revised: Table 1e compares two reconstruction targets: 1) predicting only the masked frequencies on the frequency spectrum as in our default setting

Original: [Table]
Revised: [Table]

Original: and 2) recovering both the masked and unmasked frequencies on the frequency spectrum.
Revised: and 2) recovering both the masked and unmasked frequencies on the frequency spectrum.

Original: Predicting the masked spectrum performs better than reconstructing the full spectrum by a clear margin (83.1% vs. 82.4%).
Revised: Predicting the masked spectrum performs better than reconstructing the full spectrum by a clear margin (83.1% vs. 82.4%).

Original: This suggests that predicting the invisible signals is a more favourable task in representation learning, which is in accordance with the observation in recent MIM approaches.
Revised: This suggests that predicting the invisible signals is a more favourable task in representation learning, which is in accordance with the observation in recent MIM approaches.

Original: Loss function.
Revised: Loss function.

Original: Table 1f studies the design of loss functions.
Revised: Table 1f studies the design of loss functions.

Original: A frequency loss  performs better than a spatial loss (with γ = 1 working the best).
Revised: A frequency loss (freq.) performs better than a spatial loss ( ℓ 1 , ℓ 2 ), with γ = 1 working the best.

Original: It makes sense as directly predicting the missing frequencies in the frequency domain better aligns to our MFM task.
Revised: It makes sense as directly predicting the missing frequencies in the frequency domain better aligns to our MFM task.

Original: 4. D IAGNOSIS OF L OW -L EVEL I MAGE P ROCESSING T ASKS
Revised: 4. D IAGNOSIS OF L OW -L EVEL I MAGE P ROCESSING T ASKS

Original: In this subsection, we study the representation learning capability of low-level image processing tasks from a unified frequency perspective.
Revised: In this subsection, we study the representation learning capability of low-level image processing tasks from a unified frequency perspective.

Original: We examine three representative tasks: image superresolution (SR), deblurring, and denoising.
Revised: We examine three representative tasks: image superresolution (SR), deblurring, and denoising.

Original: Setup.
Revised: Setup.

Original: To ensure a direct comparison, we adopt the same pre-training and fine-tuning hyperparameters as MFM and only alter the types of image degradation during pre-training.
Revised: To ensure a direct comparison, we adopt the same pre-training and fine-tuning hyperparameters as MFM and only alter the types of image degradation during pre-training.

Original: Specifically, for the SR task, we first use its standard data pre-processing, i.e. , bicubic downsampling, to downsample the input images by a scale factor.
Revised: Specifically, for the SR task, we first use its standard data pre-processing, i.e. , bicubic downsampling, to downsample the input images by a scale factor.

Original: We then upsample them back to the original input size, i.e. , 224 × 224 .
Revised: We then upsample them back to the original input size, i.e. , 224 × 224 .

Original: For the deblurring task, we consider the commonly-used isotropic Gaussian filter and uniformly select the blur kernel size from { 7 , 9 , 11 , 13 , 15 , 17 , 19 , 21 } as suggested in Wang et al.
Revised: For the deblurring task, we consider the commonly-used isotropic Gaussian filter and uniformly select the blur kernel size from { 7 , 9 , 11 , 13 , 15 , 17 , 19 , 21 } as suggested in Wang et al.

Original: For the denoising task, we employ the typical Gaussian noise.
Revised: For the denoising task, we employ the typical Gaussian noise.

Original: The intensity of both deblurring and denoising tasks is controlled by the standard deviation ( i.e. , sigma value) of the Gaussian distribution.
Revised: The intensity of both deblurring and denoising tasks is controlled by the standard deviation ( i.e. , sigma value) of the Gaussian distribution.

Original: For all tasks, the reconstruction target is the original image but in the frequency domain via the same frequency loss as MFM.
Revised: For all tasks, the reconstruction target is the original image but in the frequency domain via the same frequency loss as MFM.

Original: Observations.
Revised: Observations.

Original: Table 2 shows the results with different levels of degradation for each task.
Revised: Table 2 shows the results with different levels of degradation for each task.

Original: We first notice that the optimal degradation level of each task in the context of representation learning is much heavier than its original task setting.
Revised: We first notice that the optimal degradation level of each task in the context of representation learning is much heavier than its original task setting.

Original: For instance, a standard SR task usually has a downsampling factor within × 4 , while we show that a much heavier × 8 setting works the best.
Revised: For instance, a standard SR task usually has a downsampling factor within × 4 , while we show that a much heavier × 8 setting works the best.

Original: With right configuration of the task difficulty, all these tasks can achieve comparable or even better performance than their supervised counterpart ( e.g. , 81.8% in Touvron et al.
Revised: With right configuration of the task difficulty, all these tasks can achieve comparable or even better performance than their supervised counterpart ( e.g. , 81.8% in Touvron et al.

Original: (2021a))
Revised: (2021a))

Original: , indicating that these low-level tasks are more or less helpful in representation learning.
Revised: , indicating that these low-level tasks are more or less helpful in representation learning.

Original: In addition, we observe that representation learning benefits from all lens of frequencies.
Revised: In addition, we observe that representation learning benefits from all lens of frequencies.

Original: This can be verified by the superior performance of denoising over SR and deblurring.
Revised: This can be verified by the superior performance of denoising over SR and deblurring.

Original: As visualized in the frequency spectrum of the example image, denoising tends to intensify all frequencies of the spectrum, while SR and deblurring only removes high-frequency components.
Revised: As visualized in the frequency spectrum of the example image, denoising tends to intensify all frequencies of the spectrum, while SR and deblurring only removes high-frequency components.

Original: Thus, the performance of denoising is much closer to MFM, as both utilize the full frequency spectrum.
Revised: Thus, the performance of denoising is much closer to MFM, as both utilize the full frequency spectrum.

Original: Attenuating and intensifying frequencies on the spectrum are essentially two different ways of performing corruption in the frequency domain.
Revised: Attenuating and intensifying frequencies on the spectrum are essentially two different ways of performing corruption in the frequency domain.

Original: We believe other corruption types may also work well and leave this exploration for future work.
Revised: We believe other corruption types may also work well and leave this exploration for future work.

Original: 4. C OMPARISON WITH P REVIOUS M ETHODS 4.4. I MAGE C LASSIFICATION
Revised: 4. C OMPARISON WITH P REVIOUS M ETHODS 4.4. I MAGE C LASSIFICATION

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: ViT.
Revised: ViT.

Original: In Table
Revised: In Table

Original: [Table]
Revised: [Table]

Original: we compare the ImageNet-1K end-to-end fine-tuning results of self-supervised ViTS/16 and ViT-B/16 models.
Revised: we compare the ImageNet-1K end-to-end fine-tuning results of self-supervised ViTS/16 and ViT-B/16 models.

Original: We fine-tune ViT-S/16 for 200 epochs
Revised: We fine-tune ViT-S/16 for 200 epochs

Original: [Table]
Revised: [Table]

Original: and ViT-B/16 for 100 epochs.
Revised: and ViT-B/16 for 100 epochs.

Original: Other self-supervised models use the same or longer fine-tuning schedule.
Revised: Other self-supervised models use the same or longer fine-tuning schedule.

Original: Compared with other representative self-supervised learners, our MFM can achieve comparable performance with fewer pre-training epochs while using none of the following: (i) extra data, (ii) extra model, (iii) mask token.
Revised: Compared with other representative self-supervised learners, our MFM can achieve comparable performance with fewer pre-training epochs while using none of the following: (i) extra data, (ii) extra model, (iii) mask token.

Original: This demonstrates the great potential of masked frequency modeling.
Revised: This demonstrates the great potential of masked frequency modeling.

Original: ResNet-50.
Revised: ResNet-50.

Original: We demonstrate that MFM can also pre-train a high-capacity ResNet-50 model.
Revised: We demonstrate that MFM can also pre-train a high-capacity ResNet-50 model.

Original: We simply adopt the same pre-training settings as ViT. During fine-tuning, we generally follow the advanced vanilla ResNet “training from scratch” recipe in RSB (Wightman et al., 2021) except that we use the AdamW optimizer (Loshchilov & Hutter, 2017) following Fang et al.
Revised: We simply adopt the same pre-training settings as ViT. During fine-tuning, we generally follow the advanced vanilla ResNet “training from scratch” recipe in RSB (Wightman et al., 2021) except that we use the AdamW optimizer (Loshchilov & Hutter, 2017) following Fang et al.

Original: Table 4 shows the results.
Revised: Table 4 shows the results.

Original: Different from ViT, we observe performance degeneration of low-level image processing tasks like SR, deblurring and denoising compared with the RSB training-from-scratch baseline (Table 4b).
Revised: Different from ViT, we observe performance degeneration of low-level image processing tasks like SR, deblurring and denoising compared with the RSB training-from-scratch baseline (Table 4b).

Original: We hypothesize this discrepancy is due to the architectural difference between ViT and
Revised: We hypothesize this discrepancy is due to the architectural difference between ViT and

Original: CNN.
Revised: CNN.

Original: Compared with ViT, the convolution operation in CNN tends to be more effective in capturing high-frequency components.
Revised: Compared with ViT, the convolution operation in CNN tends to be more effective in capturing high-frequency components.

Original: Thus, encouraging a CNN model to reconstruct high-frequency components of images brings no benefits to the performance.
Revised: Thus, encouraging a CNN model to reconstruct high-frequency components of images brings no benefits to the performance.

Original: Instead, learning high-frequency information can compensate for the ability of ViT models in capturing the high-frequency components.
Revised: Instead, learning high-frequency information can compensate for the ability of ViT models in capturing the high-frequency components.

Original: In contrast, our MFM outperforms its supervised counterparts in both ViT and CNN architectures as it leverages both low- and high-frequency components.
Revised: In contrast, our MFM outperforms its supervised counterparts in both ViT and CNN architectures as it leverages both low- and high-frequency components.

Original: Even under a demanding training procedure, e.g. , fine-tuning for 300 epochs (Table 4c), MFM can still improve the supervised RSB A2 baseline by 0.3% and surpass several representative contrastive-based self-supervised learning methods.
Revised: Even under a demanding training procedure, e.g. , fine-tuning for 300 epochs (Table 4c), MFM can still improve the supervised RSB A2 baseline by 0.3% and surpass several representative contrastive-based self-supervised learning methods.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: S EMANTIC S EGMENTATION
Revised: S EMANTIC S EGMENTATION

Original: We evaluate MFM and low-level image processing tasks on the ADE20K semantic segmentation benchmark.
Revised: We evaluate MFM and low-level image processing tasks on the ADE20K semantic segmentation benchmark.

Original: We use
Revised: We use

Original: UperNet (Xiao et al., 2018) and adopt the same setup following BEiT (Bao et al., 2022).
Revised: UperNet (Xiao et al., 2018) and adopt the same setup following BEiT (Bao et al., 2022).

Original: All models are fine-tuned for 160K iterations with an input resolution of 512 × 512 .
Revised: All models are fine-tuned for 160K iterations with an input resolution of 512 × 512 .

Original: As shown in Table 5, our corruption-based models surpass supervised pre-training and achieve competitive performance compared with other representative self-supervised learners that are usually more expensive to compute.
Revised: As shown in Table 5, our corruption-based models surpass supervised pre-training and achieve competitive performance compared with other representative self-supervised learners that are usually more expensive to compute.

Original: The original ImageNet top tuning results are also appended for reference.
Revised: The original ImageNet top tuning results are also appended for reference.

Original: best are in bold , andresults are underlined
Revised: best are in bold , and

Original: [Table]
Revised: [Table]

Original: 4. R OBUSTNESS E VALUATION
Revised: 4. R OBUSTNESS E VALUATION

Original: We evaluate the robustness of our models on a series of benchmarks in three aspects: (i) adversarial robustness, (ii) common corruption robustness
Revised: We evaluate the robustness of our models on a series of benchmarks in three aspects: (i) adversarial robustness, (ii) common corruption robustness

Original: [Table]
Revised: [Table]

Original: and (iii) out-of-distribution robustness.
Revised: and (iii) out-of-distribution robustness.

Original: For (i), we study the adversarial examples generated by white-box attackers ( e.g. , FGSM (Goodfellow et al., 2014) and PGD (Madry et al., 2017)) on ImageNet-1K validation set as well as natural adversarial examples on ImageNet-A
Revised: For (i), we study the adversarial examples generated by white-box attackers ( e.g. , FGSM (Goodfellow et al., 2014) and PGD (Madry et al., 2017)) on ImageNet-1K validation set as well as natural adversarial examples on ImageNet-A

Original: (Hendrycks et al., 2021b); for (ii), we evaluate on ImageNet-C (Hendrycks & Dietterich, 2019) that includes 15 types of algorithmically generated corruptions with five levels of severity; for (iii), we test on ImageNet-R
Revised: (Hendrycks et al., 2021b); for (ii), we evaluate on ImageNet-C (Hendrycks & Dietterich, 2019) that includes 15 types of algorithmically generated corruptions with five levels of severity; for (iii), we test on ImageNet-R

Original: (Hendrycks et al., 2021a) and ImageNet-Sketch (Wang et al., 2019) that contain images with naturally occurring distribution shifts.
Revised: (Hendrycks et al., 2021a) and ImageNet-Sketch (Wang et al., 2019) that contain images with naturally occurring distribution shifts.

Original: We evaluate the same models fine-tuned on original ImageNet-1K (ViT-B/16 in Table 3 and ResNet-50 in Table 4b) without any specialized fine-tuning on the different validation sets.
Revised: We evaluate the same models fine-tuned on original ImageNet-1K (ViT-B/16 in Table 3 and ResNet-50 in Table 4b) without any specialized fine-tuning on the different validation sets.

Original: As shown in Table 6, we can conclude three observations: 1) Transformer-based models ( e.g. , ViT) are more robust than the CNN counterparts ( e.g. , ResNet-50).
Revised: As shown in Table 6, we can conclude three observations: 1) Transformer-based models ( e.g. , ViT) are more robust than the CNN counterparts ( e.g. , ResNet-50).

Original: 2) Corruption-based tasks ( e.g. , SR, Deblur, Denoise and MFM) are generally more robust than the MIM task ( e.g. , MAE).
Revised: 2) Corruption-based tasks ( e.g. , SR, Deblur, Denoise and MFM) are generally more robust than the MIM task ( e.g. , MAE and SimMIM).

Original: 3) MFM achieves the best trade-off between standard performance and robustness (the robustness of MFM always ranks within the top two, while the standard accuracy is the best).
Revised: 3) MFM achieves the best trade-off between standard performance and robustness (the robustness of MFM always ranks within the top two, while the standard accuracy is the best).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we have studied the effectiveness of low-level image processing tasks for visual representation learning from a new frequency perspective and introduced a unified, flexible and robust self-supervised visual pre-training framework to perform image corruptions in the frequency domain.
Revised: In this work, we have studied the effectiveness of low-level image processing tasks for visual representation learning from a new frequency perspective and introduced a unified, flexible and robust self-supervised visual pre-training framework to perform image corruptions in the frequency domain.

Original: We show that without relying on mask tokens or more complex designs ( e.g. , discrete visual tokens), a simple mask-frequency strategy can achieve competitive performance for both ViT and
Revised: We show that without relying on mask tokens or more complex designs ( e.g. , discrete visual tokens), a simple mask-frequency strategy can achieve competitive performance for both ViT and

Original: CNN.
Revised: CNN.

Original: We hope our unique frequency perspective can motivate the community to rethink the role of low-level tasks for unsupervised representation learning.
Revised: We hope our unique frequency perspective can motivate the community to rethink the role of low-level tasks for unsupervised representation learning.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: The proposed method learns statistics of the training dataset and may reflect the biases in the data.
Revised: The proposed method learns statistics of the training dataset and may reflect the biases in the data.

Original: Debiased measures thus have to be taken.
Revised: Debiased measures thus have to be taken.

Original: The method may be deployed with large-scale models and data, causing negative impacts on the environment.
Revised: The method may be deployed with large-scale models and data, causing negative impacts on the environment.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: We provide detailed hyper-parameter specifications for our experiments in the main text (Section 4) and the supplementary material (Appendix C) to ensure reproducibility.
Revised: We provide detailed hyper-parameter specifications for our experiments in the main text (Section 4) and the supplementary material (Appendix G) to ensure reproducibility.

Original: We will release the code and pre-trained models to the research community for reproducible research
Revised: We will release the code and pre-trained models to the research community for reproducible research

Original: A C OMBINATION OF L OW -L EVEL I MAGE P ROCESSING T ASKS
Revised: A M ORE A BLATIONS

Revised: [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Decoder depth.

Revised: Table 7a ablates the effect of different decoders.

Revised: MFM can work the best with a simple linear layer while benefiting from lower training costs compared with a deeper decoder.

Revised: Frequency masking vs. spatial masking.

Revised: Considering that different design choices in existing masked image modeling (MIM) methods could affect the performance significantly, to eliminate the interference of other factors

Revised: [Table]

Revised: we directly replace our frequency-domain-based masking with spatialdomain-based random patch masking for a fair comparison.

Revised: The results are shown in Table 7b

Revised: [Table]

Revised: Applying MIM to ResNet-50 leads to inferior performance even than the supervised baseline ( i

Revised: [Table]

Revised: e. , 78.1% in RSB A3 (Wightman et al., 2021)).

Revised: In contrast, MFM outperforms the supervised RSB baseline as well as the MIM counterpart regardless of architectures, demonstrating that frequency masking is indeed a more flexible and unified option for different architectures.

Revised: B T RAINING T IME C OMPARISON

Revised: We measure the pre-training time per epoch in Table 8.

Revised: Note that MoCo v3 (Chen et al., 2021) and

Revised: DINO (Caron et al., 2021) need to switch two global views and have four and 14 forward passes in total, respectively.

Revised: BEiT (Bao et al., 2022), MAE (He et al., 2022) and MFM are 1-view methods without switching.

Revised: MFM is relatively efficient compared with other MIM methods ( e.g. , BEiT) except for MAE.

Revised: However, taking only visible patches as input breaks the regular 2D structure of images, which makes MAE only applicable to ViT. In contrast, MFM is agnostic to architectures and can be flexibly applied for both ViT and CNN families.

Revised: Considering the flexibility and universality of MFM, a slightly increasing time over MAE is acceptable.

Original:  In Table 1a of the main text, we have shown that randomly sampling low-/high-pass filters to predict both high and low frequencies benefits as MFM can make full use of the frequency spectrum, thus leading to better performance.
Revised: [Table] C C OMBINATION OF L OW -L EVEL I MAGE P ROCESSING T ASKS In Table 1a of the main text, we have shown that randomly sampling low-/high-pass filters to predict both high and low frequencies benefits as MFM can make full use of the frequency spectrum, thus leading to better performance.

Original: Here, we further study the effect of combining low-level image processing tasks, i.e. , SR, deblurring and denoising.
Revised: Here, we further study the effect of combining low-level image processing tasks, i.e. , SR, deblurring and denoising.

Original: Table 7 reports the ImageNet-1K top-1 finetuning accuracy of ViT-B/16 models.
Revised: Table 9 reports the ImageNet-1K top-1 finetuning accuracy of ViT-B/16 models.

Original: We find that combining these low-level corruptions does not bring similar benefits as MFM and even degrades the performance.
Revised: We find that combining these low-level corruptions does not bring similar benefits as MFM and even degrades the performance.

Original: [Table]
Revised: [Table]

Original: B F URTHER D ISCUSSION
Revised: D F URTHER D ISCUSSION

Original: Mask tokens.
Revised: Mask tokens.

Original: In masked image modeling (MIM) methods, mask tokens are learnable patch embeddings inserted in the position where the input tokens are masked out.
Revised: In MIM methods, mask tokens are learnable patch embeddings inserted in the position where the input tokens are masked out.

Original: They are highly coupled with the Transformer architecture and not directly applicable to CNNs.
Revised: They are highly coupled with the Transformer architecture and not directly applicable to CNNs.

Original: Introducing special mask tokens in any intermediate stage of CNN is infeasible, as the intrinsic dense-sliding-window paradigm in convolution layers brings information leakage between visual features in previous layers (Fang et al., 2022).
Revised: Introducing special mask tokens in any intermediate stage of CNN is infeasible, as the intrinsic dense-sliding-window paradigm in convolution layers brings information leakage between visual features in previous layers (Fang et al., 2022).

Original: Thus, the large CNN family cannot directly benefit from this pre-training scheme like Transformers.
Revised: Thus, the large CNN family cannot directly benefit from this pre-training scheme like Transformers.

Original: In contrast, our MFM performs masking in the frequency domain without relying on mask tokens.
Revised: In contrast, our MFM performs masking in the frequency domain without relying on mask tokens.

Revised: Thus, MFM is agnostic to the architectures and can be flexibly applied for broader ViT and CNN families.

Revised: ) without additional losses of intermediate Transformer layers using their official code.

Original: Thus, MFM is agnostic tothe architectures and can be flexibly applied for broader
Revised: Training costs are counted in relative to our approach. Note that MFM is agnostic to architectures while these hybrid methods are not. [Table] Comparison with hybrid MIM methods.

Original: ViT and CNN families.
Revised: A line of recent research (Zhou et al., 2022; Baevski et al., 2022; Assran et al., 2022) combines MIM with contrastive learning (CL) into a Siamese framework and achieves better performance than a single task. Apart from adopting a Siamese network, additional techniques used in these works include multi-crop augmentation in Zhou et al. Assran et al. and multiple losses of intermediate Transformer layers in Baevski et al.

Revised: (2022), without which their performance will be degraded significantly as shown in Table 10 (we take iBOT and data2vec as examples here since most of these works are concurrent to ours).

Revised: Therefore, to eliminate the interference of other design factors, we mainly compare with pure MIM methods in our study.

Revised: More advanced techniques used in these works may also be incorporated into MFM to further improve the performance, which is beyond the focus of this work.

Original: Concurrent work.
Revised: Concurrent work.

Original: A concurrent work (Liu et al., 2022) also involves self-supervised learning in the frequency domain.
Revised: A concurrent work (Liu et al., 2022) also involves self-supervised learning in the frequency domain.

Original: Our work differs from theirs in the following aspects: 1) It aims at improving upon existing MIM approaches by additionally designing a more complex frequency decoder and computing losses in both spatial and frequency domain.
Revised: Our work differs from theirs in the following aspects: 1) It aims at improving upon existing MIM approaches by additionally designing a more complex frequency decoder and computing losses in both spatial and frequency domain.

Original: In contrast, we aim at exploring an alternative frequency corruption strategy beyond MIM.
Revised: In contrast, we aim at exploring an alternative frequency corruption strategy beyond MIM.

Original: Our method does not rely on any existing MIM approaches and we show that MFM can also work well independently.
Revised: Our method does not rely on any existing MIM approaches and we show that MFM can also work well independently.

Original: 2) It is based on MAE (He et al., 2022) and still performs spatial masking with mask tokens.
Revised: 2) It is based on MAE (He et al., 2022) and still performs spatial masking with mask tokens.

Original: Thus, it is still not applicable to
Revised: Thus, it is still not applicable to

Original: CNNs, whereas our frequency-domain-based masking strategy is agnostic to architectures.
Revised: CNNs, whereas our frequency-domain-based masking strategy is agnostic to architectures.

Original: 3) As opposed to Liu et al.
Revised: 3) As opposed to Liu et al.

Original: (2022) that mainly targets at improving MIM performance, we comprehensively study the effectiveness of low-level image processing tasks for representation learning from a unified frequency perspective and provide rather different insights that other low-level tasks beyond
Revised: (2022) that mainly targets at improving MIM performance, we comprehensively study the effectiveness of low-level image processing tasks for representation learning from a unified frequency perspective and provide rather different insights that other low-level tasks beyond

Original: MIM can also work well.
Revised: MIM can also work well.

Original: Limitations.
Revised: Limitations.

Original: Our study has several limitations: 1) We mainly focus on the architectural flexibility and universality of MFM, while leaving the scaling behaviour under-explored.
Revised: Our study has several limitations: 1) We mainly focus on the architectural flexibility and universality of MFM, while leaving the scaling behaviour under-explored.

Original: We mainly evaluate the quality of learned representations on representative benchmarks, i.e. , ImageNet-1K image classification and ADE20K semantic segmentation, following Bao et al.
Revised: We mainly evaluate the quality of learned representations on representative benchmarks, i.e. , ImageNet-1K image classification and ADE20K semantic segmentation, following Bao et al.

Original: The transferability on more downstream tasks can be studied in future research.
Revised: The transferability on more downstream tasks can be studied in future research.

Original: Despite the intriguing properties of the Fourier domain, the redundancy in frequencies may still exist.
Revised: Despite the intriguing properties of the Fourier domain, the redundancy in frequencies may still exist.

Original: More advanced information suppression strategies can be further explored.
Revised: More advanced information suppression strategies can be further explored.

Original: We believe that our MFM can also complement contrastive learning and MIM approaches to further improve the performance.
Revised: We believe that our MFM can also complement contrastive learning and MIM approaches to further improve the performance.

Original: We leave these explorations for future work. 
Revised: We leave these explorations for future work. Future work.

Revised: In this paper, we have shown that MFM is a simple, unified and flexible selfsupervised pretext task for various architectures.

Revised: Compared with MIM, MFM can embrace broader architectures ( e.g. , ViT, CNN, etc. ) and has more appealing robustness.

Revised: Some possible future research directions may include: 1) More self-supervised learning works in the frequency domain with different modalities ( e.g. , image, video, audio, etc. ).

Revised: 2) Combine MFM with existing contrastive learning and MIM paradigms to further improve the performance.

Original: C I MPLEMENTATION D ETAILS
Revised: 3) Apply MFM for model robustness analysis and calibration. The idea of MFM may also be used in low-level image reconstruction and synthesis tasks. E P SEUDOCODE Algorithm 1 Pseudocode of MFM in a PyTorch

Revised: [Table]

Revised: style

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: N

Revised: [Table]

Revised: aug

Revised: [Table]

Revised: # convert spatial domain into frequency domain x_freq

Revised: = fft2(x)

Revised: # 2D FFT x_freq = fftshift(x_freq, dim=(-2, -1))

Revised: # shift low frequency to the center x_freq = x_freq

Revised: * mask # mask a portion of frequencies x_freq

Revised: = ifftshift(x_freq, dim=(-2, -1))

Revised: # restore the original frequency order # convert frequency domain back into spatial domain x_corrupted =

Revised: ifft2(x_freq).real # 2D iFFT (only keep the real part)

Revised: x_predicted = f(x_corrupted)

Revised: # predicted view, NxCxHxW loss =

Revised: FrequencyLoss(x_predicted, x, gamma)

Revised: # frequency loss # only compute the frequency loss on the masked area loss =

Revised: (loss * (1 - mask)).sum() / (1 - mask).sum() #

Revised: [Table]

Revised: model loss.backward(

Revised: [Table]

Revised: (f

Revised: [Table]

Revised: def FrequencyLoss(x, y, gamma): x_freq, y_freq = fft2(x), fft2(y)

Revised: # 2D FFT # shift low frequency to the center x_freq

Revised: = fftshift(x_freq, dim=(-2, -1))

Revised: y_freq

Revised: = fftshift(y_freq, dim=(-2, -1))

Revised: # stack the real and

Revised: [Table]

Revised: along the last dimension x_freq = stack([x_freq.real, x_freq.imag], -1) y_freq

Revised: = stack([y_freq.real, y_freq.imag], -1)

Revised: # compute the frequency distance

Revised: [Table]

Revised: (x_freq - y_freq)

Revised: ** 2 return (d[..., 0] + d[..., 1]) ** (0.5 * gamma)

Revised: F F REQUENCY S PECTRUM OF MIM

Revised: Figure 3 visualizes some example frequency spectrums of spatial-domain-based random patch masking used in MIM.

Revised: Performing patch-wise masking in the spatial domain incurs grid-wise artifacts on the frequency spectrum, preventing further meaningful observations.

Revised: G I MPLEMENTATION D ETAILS

Original: Pre-training.
Revised: Pre-training.

Original: Table 8 summarizes the pre-training settings for vanilla ViT and ResNet-50 models.
Revised: Table 11 summarizes the pre-training settings for vanilla ViT and ResNet-50 models.

Original: All experiments are conducted on 16 V100 32G GPUs for ViT models and 8 V100 32G GPUs for
Revised: All experiments are conducted on 16 V100 32G GPUs for ViT models and 8 V100 32G GPUs for

Original: ResNet-50.
Revised: ResNet-50.

Original: The configurations are shared by different architectures, without specialized tuning.
Revised: The configurations are shared by different architectures, without specialized tuning.

Original: This demonstrates that MFM is general across architectures.
Revised: This demonstrates that MFM is general across architectures.

Original: Fine-tuning.
Revised: Fine-tuning.

Original: Table 9 and Table 10 summarize the fine-tuning settings for vanilla ViT and ResNet50 models, respectively.
Revised: Table 12 and Table 13 summarize the fine-tuning settings for vanilla ViT and ResNet50 models, respectively.

Original: The configurations for ViT are shared across models, except that smaller models are fine-tuned longer.
Revised: The configurations for ViT are shared across models, except that smaller models are fine-tuned longer.

Original: The configurations for ResNet-50 basically follow Wightman et al.
Revised: The configurations for ResNet-50 basically follow Wightman et al.

Original: (2021), except that we adopt the AdamW optimizer following Fang et al.
Revised: (2021), except that we adopt the AdamW optimizer following Fang et al.

Original: Semantic segmentation on ADE20K.
Revised: Semantic segmentation on ADE20K.

Original: We use UperNet (Xiao et al., 2018) following the configurations in BEiT (Bao et al., 2022).
Revised: We use UperNet (Xiao et al., 2018) following the configurations in BEiT (Bao et al., 2022).

Original: We use AdamW as the optimizer and fine-tune for 160K iterations with a batch size of 16.
Revised: We use AdamW as the optimizer and fine-tune for 160K iterations with a batch size of 16.

Original: We search the learning rate for all the results in Table 5 of the main text.
Revised: We search the learning rate for all the results in Table 5 of the main text.

Original: The input resolution is 512 × 512 , and we use single-scale inference.
Revised: The input resolution is 512 × 512 , and we use single-scale inference.

Original: [Table]
Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Table]

Original: D V ISUALIZATION
Revised: H V ISUALIZATION

Original: We provide more qualitative results of corrupted images in Figure 3 following Table 2 in the main text, as well as recovered images using unseen ImageNet-1K (Figure 4) and COCO (Figure 5) validation images.[Table] [Table]
Revised: We provide more qualitative results of corrupted images in Figure 4 following Table 2 in the main text, as well as recovered images using unseen ImageNet-1K (Figure 5) and COCO (Figure 6) validation images.

Original: [Figure]
Revised: [Figure]

Original: ×
Revised: ×

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: (a) (b) (c) (d) original            (a) (b) (c) (d) original
Revised: (a) (b) (c) (d) original            (a) (b) (c) (d) original

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: (a) (
Revised: (a) (

Original: [Figure]
Revised: [Figure]

Original: ) (c) (d)
Revised: ) (c) (d)

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: c)
Revised: c)

Original: [Figure]
Revised: [Figure]

Original: (a) (
Revised: (a) (

Original: [Figure]
Revised: [Figure]

Original: ) (c) (d) original
Revised: ) (c) (d) original

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: original input output input output input output input output input output
Revised: original input output input output input output input output input output

Original: SR
Revised: SR

Original: Deblur
Revised: Deblur

Original: Denoise
Revised: Denoise

Original: MFM
Revised: MFM

Original: [Figure]
Revised: [Figure]