Original: Masked Autoencoders that Listen
Revised: Masked Autoencoders that Listen

Original: Abstract
Revised: Abstract

Original: This paper studies a simple extension of image-based Masked Autoencoders (MAE) [1] to self-supervised representation learning from audio spectrograms.
Revised: This paper studies a simple extension of image-based Masked Autoencoders (MAE) [1] to self-supervised representation learning from audio spectrograms.

Original: Following the Transformer encoder-decoder design in MAE, our Audio-MAE first encodes audio spectrogram patches with a high masking ratio, feeding only the non-masked tokens through encoder layers.
Revised: Following the Transformer encoder-decoder design in MAE, our Audio-MAE first encodes audio spectrogram patches with a high masking ratio, feeding only the non-masked tokens through encoder layers.

Original: The decoder then re-orders and decodes the encoded context padded with mask tokens, in order to reconstruct the input spectrogram.
Revised: The decoder then re-orders and decodes the encoded context padded with mask tokens, in order to reconstruct the input spectrogram.

Original: We find it beneficial to incorporate local window attention in the decoder, as audio spectrograms are highly correlated in local time and frequency bands.
Revised: We find it beneficial to incorporate local window attention in the decoder, as audio spectrograms are highly correlated in local time and frequency bands.

Original: We then fine-tune the encoder with a lower masking ratio on target datasets.
Revised: We then fine-tune the encoder with a lower masking ratio on target datasets.

Original: Empirically, Audio-MAE sets new state-of-the-art performance on six audio and speech classification tasks, outperforming other recent models that use external supervised pre-training.
Revised: Empirically, Audio-MAE sets new state-of-the-art performance on six audio and speech classification tasks, outperforming other recent models that use external supervised pre-training.

Original: Our code and models is available at https://github.com/facebookresearch/AudioMAE .
Revised: Our code and models is available at https://github.com/facebookresearch/AudioMAE .

Original:  Introduction
Revised:  Introduction

Original: Transformers [2] and self-supervised learning [3, 4, 5, 6, 7, 1] are dominating computer vision (CV) and natural language processing (NLP) research.
Revised: Transformers [2] and self-supervised learning [3, 4, 5, 6, 7, 1] are dominating computer vision (CV) and natural language processing (NLP) research.

Original: The revolution firstly started in NLP with the invention of the Transformer architecture and self-attention [8].
Revised: The revolution firstly started in NLP with the invention of the Transformer architecture and self-attention [8].

Original: Masked autoencoding with BERT [3] set a new state-of-the-art on various NLP tasks by self-supervised pre-training on large-scale language corpus.
Revised: Masked autoencoding with BERT [3] set a new state-of-the-art on various NLP tasks by self-supervised pre-training on large-scale language corpus.

Original: Similarly in the CV community, Vision Transformers (ViT) [9] have become popular for CV tasks and, for self-supervised image representation learning, Masked Autoencoders (MAE)
Revised: Similarly in the CV community, Vision Transformers (ViT) [9] have become popular for CV tasks and, for self-supervised image representation learning, Masked Autoencoders (MAE)

Original: [1] have brought the CV community closer to the success of BERT in NLP.
Revised: [1] have brought the CV community closer to the success of BERT in NLP.

Original: In addition to the existing masked autoencoders that can read (BERT) or see (MAE), in this work we study those that can listen .
Revised: In addition to the existing masked autoencoders that can read (BERT) or see (MAE), in this work we study those that can listen .

Original: Transformer-based models have recently refreshed leaderboards for audio understanding tasks.
Revised: Transformer-based models have recently refreshed leaderboards for audio understanding tasks.

Original: For example, AST [10] and MBT [11] improved the audio classification performance on the AudioSet [12],
Revised: For example, AST [10] and MBT [11] improved the audio classification performance on the AudioSet [12],

Original: Event Sound Classification [13], etc.
Revised: Event Sound Classification [13], etc.

Original: The key technique behind this is initialization of audio model weights with ImageNet pre-trained supervised models ( e.g ., DeiT [14]) by deflating patch embeddings and interpolating positional embeddings for encoding audio spectrograms.
Revised: The key technique behind this is initialization of audio model weights with ImageNet pre-trained supervised models ( e.g ., DeiT [14]) by deflating patch embeddings and interpolating positional embeddings for encoding audio spectrograms.

Original: However, exploiting
Revised: However, exploiting

Original: ImageNet pre-trained models could be sub-optimal.
Revised: ImageNet pre-trained models could be sub-optimal.

Original: Unlike initializing video models with weights from image models ( e.g ., the initial weights of I3D [15] or 3D-ResNets [16] are inflated from
Revised: Unlike initializing video models with weights from image models ( e.g ., the initial weights of I3D [15] or 3D-ResNets [16] are inflated from

Original: ImageNet pre-trained image models), there are clear and notable discrepancies between spectrograms representing audio content and natural images.
Revised: ImageNet pre-trained image models), there are clear and notable discrepancies between spectrograms representing audio content and natural images.

Original: It remains unclear why such heterogeneous image-toaudio transfer is useful beyond arguably similar low-level semantics such as shapes of spectrograms and shapes of visual objects.
Revised: It remains unclear why such heterogeneous image-toaudio transfer is useful beyond arguably similar low-level semantics such as shapes of spectrograms and shapes of visual objects.

Original: Further, any label bias would inevitably be transferred to audio models.
Revised: Further, any label bias would inevitably be transferred to audio models.

Original: Addressing these concerns, self-supervised audio representation learning has recently attracted much research attention.
Revised: Addressing these concerns, self-supervised audio representation learning has recently attracted much research attention.

Original: Based on BEiT [17] that learns to reconstruct image patches or learnt patch tokens, SS-AST [18] extends to the audio domain and exploits spectrograms (akin to 1-channel 2D images) and use both contrastive and reconstruction objective as self-supervision.
Revised: Based on BEiT [17] that learns to reconstruct image patches or learnt patch tokens, SS-AST [18] extends to the audio domain and exploits spectrograms (akin to 1-channel 2D images) and use both contrastive and reconstruction objective as self-supervision.

Original: Without using any labels, the key enabler to effective self-supervised representation learning is large-scale pre-training data.
Revised: Without using any labels, the key enabler to effective self-supervised representation learning is large-scale pre-training data.

Original: In this work we use AudioSet [12] for pre-training, a common dataset containing ∼ 2 million audio recordings.
Revised: In this work we use AudioSet [12] for pre-training, a common dataset containing ∼ 2 million audio recordings.

Original: Performing large-scale training with Transformer architectures is challenging as self-attention in Transformers has quadratic complexity w.r.t.
Revised: Performing large-scale training with Transformer architectures is challenging as self-attention in Transformers has quadratic complexity w.r.t.

Original: the length of input sequence.
Revised: the length of input sequence.

Original: [Figure]
Revised: [Figure]

Original: An encoder then operates on the visible (20%) patch embeddings.
Revised: An encoder then operates on the visible (20%) patch embeddings.

Original: Finally, a decoder processes the order-restored embeddings and mask tokens to reconstruct the input.
Revised: Finally, a decoder processes the order-restored embeddings and mask tokens to reconstruct the input.

Original: Audio-MAE is minimizing the mean square error (MSE) on the masked portion of the reconstruction and the input spectrogram.
Revised: Audio-MAE is minimizing the mean square error (MSE) on the masked portion of the reconstruction and the input spectrogram.

Original: This computational burden has been addressed in different ways.
Revised: This computational burden has been addressed in different ways.

Original: A popular approach is to reduce the sequence length in self-attention.
Revised: A popular approach is to reduce the sequence length in self-attention.

Original: Various ViT-based architectures have been developed to alleviate such issues for image and video understanding.
Revised: Various ViT-based architectures have been developed to alleviate such issues for image and video understanding.

Original: For example, Swin-Transformer [19] only performs local attention within windows that shift across layers.
Revised: For example, Swin-Transformer [19] only performs local attention within windows that shift across layers.

Original: MViT [20] employs pooling attention to construct a hierarchy of Transformers where sequence lengths are downsampled.
Revised: MViT [20] employs pooling attention to construct a hierarchy of Transformers where sequence lengths are downsampled.

Original: For self-supervised learning, MAE [1] efficiently encodes only a small portion (25%) of visual patches while the majority of patches is discarded.
Revised: For self-supervised learning, MAE [1] efficiently encodes only a small portion (25%) of visual patches while the majority of patches is discarded.

Original: The simplicity and scalability in MAE make it a promising framework for large-scale self-supervised learning.
Revised: The simplicity and scalability in MAE make it a promising framework for large-scale self-supervised learning.

Original: In this work, we study MAE for sound recognition and the unique challenges of the audio domain.
Revised: In this work, we study MAE for sound recognition and the unique challenges of the audio domain.

Original: We present Audio-MAE (Fig. 1) as unified and scalable framework for learning self-supervised audio representations.
Revised: We present Audio-MAE (Fig. 1) as unified and scalable framework for learning self-supervised audio representations.

Original: Similar to MAE, it is composed of a pair of a Transformer encoder and decoder.
Revised: Similar to MAE, it is composed of a pair of a Transformer encoder and decoder.

Original: Sound is first transformed and embedded into spectrogram patches.
Revised: Sound is first transformed and embedded into spectrogram patches.

Original: Before feeding them into the
Revised: Before feeding them into the

Original: Transformer encoder, we mask and discard the majority and only feed a small number of non-masked embeddings into the encoder for efficient encoding.
Revised: Transformer encoder, we mask and discard the majority and only feed a small number of non-masked embeddings into the encoder for efficient encoding.

Original: After padding encoded patches with learnable embeddings to represent masked patches, it then restores the order of these patches in frequency and time and propagates them through a Transformer decoder to reconstruct the audio spectrogram.
Revised: After padding encoded patches with learnable embeddings to represent masked patches, it then restores the order of these patches in frequency and time and propagates them through a Transformer decoder to reconstruct the audio spectrogram.

Original: Different from image patches, spectrogram patches are comparably local-correlated.
Revised: Different from image patches, spectrogram patches are comparably local-correlated.

Original: For example, formants, the vocal tract resonances, are typically grouped and continuous locally in the spectrogram.
Revised: For example, formants, the vocal tract resonances, are typically grouped and continuous locally in the spectrogram.

Original: The location in frequency and time embeds essential information that determines the semantics of a spectrogram patch and how it sounds like.
Revised: The location in frequency and time embeds essential information that determines the semantics of a spectrogram patch and how it sounds like.

Original: To this end, we further investigate using localized attention and a hybrid architecture in the Transformer decoder to properly decode for reconstruction.
Revised: To this end, we further investigate using localized attention and a hybrid architecture in the Transformer decoder to properly decode for reconstruction.

Original: This simple-yet-effective upgrade leads to improved performance for Audio-MAE.
Revised: This simple-yet-effective upgrade leads to improved performance for Audio-MAE.

Original: Similar to MAE for images, we minimize the patch-normalized mean square error.
Revised: Similar to MAE for images, we minimize the patch-normalized mean square error.

Original: At the fine-tuning stage, we discard the decoder and fine-tune the encoder with patch-masking.
Revised: At the fine-tuning stage, we discard the decoder and fine-tune the encoder with patch-masking.

Original: Empirically, AudioMAE sets a new state-of-the-art performance on six audio and speech classification tasks.
Revised: Empirically, AudioMAE sets a new state-of-the-art performance on six audio and speech classification tasks.

Original: It is the first audio-only self-supervised model that achieves state-of-the-art mAP on AudioSet-2M, outperforming other recent models with external supervision.
Revised: It is the first audio-only self-supervised model that achieves state-of-the-art mAP on AudioSet-2M, outperforming other recent models with external supervision.

Original: We further provide the visualization and audible examples to qualitatively demonstrate the effectiveness of the Audio-MAE decoder.
Revised: We further provide the visualization and audible examples to qualitatively demonstrate the effectiveness of the Audio-MAE decoder.

Original:  Related Work
Revised:  Related Work

Original: Visual masked pre-training.
Revised: Visual masked pre-training.

Original: Masked/Denoising autoencoders [21, 22, 3] are a general representation learning methodology by reconstructing source from masked or corrupted inputs.
Revised: Masked/Denoising autoencoders [21, 22, 3] are a general representation learning methodology by reconstructing source from masked or corrupted inputs.

Original: In CV, visual masked pre-training has made recent progress [23, 24, 1, 20].
Revised: In CV, visual masked pre-training has made recent progress [23, 24, 1, 20].

Original: Based on ViT [9] that applies
Revised: Based on ViT [9] that applies

Original: Transformers to image patches, BEiT [17] and MAE [1] present masked image modeling frameworks.
Revised: Transformers to image patches, BEiT [17] and MAE [1] present masked image modeling frameworks.

Original: BEiT [17] learns to predict discrete visual tokens generated by VAE [25] in masked patches.
Revised: BEiT [17] learns to predict discrete visual tokens generated by VAE [25] in masked patches.

Original: MAE [1] reduces sequence length by masking a large portion of image patches randomly and encoding only non-masked ones for reconstruction of pixel color information.
Revised: MAE [1] reduces sequence length by masking a large portion of image patches randomly and encoding only non-masked ones for reconstruction of pixel color information.

Original: MaskFeat [20] studies features for masked pre-training and finds that Histograms of Oriented Gradients (HoG) [26], which are in turn related to spectrogram features, perform strongly for image and video classification models.
Revised: MaskFeat [20] studies features for masked pre-training and finds that Histograms of Oriented Gradients (HoG) [26], which are in turn related to spectrogram features, perform strongly for image and video classification models.

Original: Our work extends the MAE framework for representation learning with audio spectrograms.
Revised: Our work extends the MAE framework for representation learning with audio spectrograms.

Original: Out-of-domain pre-training for audio.
Revised: Out-of-domain pre-training for audio.

Original: Transferring ImageNet supervised pre-trained ViT [9] or ResNet [27] has become a popular practice for audio models [10, 28, 11, 29, 30, 31].
Revised: Transferring ImageNet supervised pre-trained ViT [9] or ResNet [27] has become a popular practice for audio models [10, 28, 11, 29, 30, 31].

Original: After pre-training, these models operate over audio spectrograms by deflating from 3-channels (RGB) into 1-channel (spectrogram) in the pre-trained patch embedding in ViT and employing the rest of the transformer blocks on top.
Revised: After pre-training, these models operate over audio spectrograms by deflating from 3-channels (RGB) into 1-channel (spectrogram) in the pre-trained patch embedding in ViT and employing the rest of the transformer blocks on top.

Original: For example, HTS-AT [29] encodes spectrograms with hierarchical
Revised: For example, HTS-AT [29] encodes spectrograms with hierarchical

Original: Transformer initialized from the Swin Transformer [19].
Revised: Transformer initialized from the Swin Transformer [19].

Original: MBT [11] uses ImageNet-21K pre-trained
Revised: MBT [11] uses ImageNet-21K pre-trained

Original: ViT; AST [10] and PaSST [28] employ DeiT [14] as the Transformer backbone.
Revised: ViT; AST [10] and PaSST [28] employ DeiT [14] as the Transformer backbone.

Original: Without using out-of-domain (non-audio) data, the proposed Audio-MAE focuses on audio-only self-supervised pre-training from scratch.
Revised: Without using out-of-domain (non-audio) data, the proposed Audio-MAE focuses on audio-only self-supervised pre-training from scratch.

Original: In-domain pre-training for audio.
Revised: In-domain pre-training for audio.

Original: Existing in-domain ( i.e ., audio-only) self-supervised methods can be broadly categorized by the input signal type ( e.g ., raw waveform [32, 33, 34], frame-level features [35, 36, 37], or spectrogram patches [18, 38]); and the objective used for self-supervision ( e.g ., contrastive [39, 33, 40, 41, 35] or prediction/reconstruction [18, 34, 37, 36]).
Revised: Existing in-domain ( i.e ., audio-only) self-supervised methods can be broadly categorized by the input signal type ( e.g ., raw waveform [32, 33, 34], frame-level features [35, 36, 37], or spectrogram patches [18, 38]); and the objective used for self-supervision ( e.g ., contrastive [39, 33, 40, 41, 35] or prediction/reconstruction [18, 34, 37, 36]).

Original: For example, wav2vec 2.0 [33] takes raw waveform as inputs and exploits contrastive learning to discriminate contextualized representations in different time segments.
Revised: For example, wav2vec 2.0 [33] takes raw waveform as inputs and exploits contrastive learning to discriminate contextualized representations in different time segments.

Original: Mockingjay [42] proposed a masked acoustic model pretext task to reconstruct frame-level Mel-features of masked time frames.
Revised: Mockingjay [42] proposed a masked acoustic model pretext task to reconstruct frame-level Mel-features of masked time frames.

Original: SSAST [18] is the closest work to Audio-MAE and is our main benchmark.
Revised: SSAST [18] is the closest work to Audio-MAE and is our main benchmark.

Original: Inspired by the success of
Revised: Inspired by the success of

Original: BERT [3], SS-AST proposed a self-supervised learning method which operates over spectrogram patches and employs joint contrastive and reconstructive objectives on masked patches.
Revised: BERT [3], SS-AST proposed a self-supervised learning method which operates over spectrogram patches and employs joint contrastive and reconstructive objectives on masked patches.

Original: These previous methods generate audio representations by encoding full-view of both masked and nonmasked time or spectrogram segments for self-supervised pre-training.
Revised: These previous methods generate audio representations by encoding full-view of both masked and nonmasked time or spectrogram segments for self-supervised pre-training.

Original: In contrast, Audio-MAE encodes only the non-masked spectrogram patches.
Revised: In contrast, Audio-MAE encodes only the non-masked spectrogram patches.

Original: Our work is done independently and concurrently with [38, 43, 44] related methods.
Revised: Our work is done independently and concurrently with [38, 43, 44] related methods.

Original: We also compare our model to these concurrent works in the experiments and showcase the superiority of Audio-MAE.
Revised: We also compare our model to these concurrent works in the experiments and showcase the superiority of Audio-MAE.

Original:  Audio Masked Autoencoders (Audio-MAE)
Revised:  Audio Masked Autoencoders (Audio-MAE)

Original: Audio-MAE is a conceptually simple extension of MAE to learn self-supervised representations from audio spectrograms.
Revised: Audio-MAE is a conceptually simple extension of MAE to learn self-supervised representations from audio spectrograms.

Original: Fig.
Revised: Fig.

Original: 1 depicts an overview.
Revised: 1 depicts an overview.

Original: The details of each component are as follows.
Revised: The details of each component are as follows.

Original: Spectrogram Patch Embeddings .
Revised: Spectrogram Patch Embeddings .

Original: Following [10, 18], we transform audio recordings into Melspectrograms and divide them into non-overlapped regular grid patches.
Revised: Following [10, 18], we transform audio recordings into Melspectrograms and divide them into non-overlapped regular grid patches.

Original: These patches are then flattened and embedded by a linear projection.
Revised: These patches are then flattened and embedded by a linear projection.

Original: Similar to MAE [1], we add fixed sinusoidal positional embeddings to the embedded patches.
Revised: Similar to MAE [1], we add fixed sinusoidal positional embeddings to the embedded patches.

Original: [Figure]
Revised: [Figure]

Original: Masking Strategies .
Revised: Masking Strategies .

Original: Audio-MAE masks out a large subset of spectrogram patches.
Revised: Audio-MAE masks out a large subset of spectrogram patches.

Original: As a spectrogram can be viewed as a 2D representation of time and frequency components of a sound, it is reasonable to explore treating time and frequency differently during masking.
Revised: As a spectrogram can be viewed as a 2D representation of time and frequency components of a sound, it is reasonable to explore treating time and frequency differently during masking.

Original: In this work, we investigate both the unstructured ( i.e ., random masking without any prior) and structured ( i.e ., randomly masking a portion of time, frequency, or time + frequency of a spectrogram) in the pre-training and fine-tuning phase.
Revised: In this work, we investigate both the unstructured ( i.e ., random masking without any prior) and structured ( i.e ., randomly masking a portion of time, frequency, or time + frequency of a spectrogram) in the pre-training and fine-tuning phase.

Original: Illustrative examples are shown in Fig.
Revised: Illustrative examples are shown in Fig.

Original: We show masked regions with dark overlay.
Revised: We show masked regions with dark overlay.

Original: The masking mechanism, as introduced in MAE [1], is the key ingredient for efficient self-supervised learning.
Revised: The masking mechanism, as introduced in MAE [1], is the key ingredient for efficient self-supervised learning.

Original: For a input patch sequence, this can be regarded as a Bernoulli process where each patch is masked/dropped with probability p (masking ratio).
Revised: For a input patch sequence, this can be regarded as a Bernoulli process where each patch is masked/dropped with probability p (masking ratio).

Original: Masking reduces input patch sequence length and encourages learning global, contextualized representations from limited “visible” patches.
Revised: Masking reduces input patch sequence length and encourages learning global, contextualized representations from limited “visible” patches.

Original: We observe that akin to images, a large masking rate (80% in our experiments for spectrogram patches, which is similar to 75% in MAE for images) is feasible for learning self-supervised audio representations.
Revised: We observe that akin to images, a large masking rate (80% in our experiments for spectrogram patches, which is similar to 75% in MAE for images) is feasible for learning self-supervised audio representations.

Original: Unlike BERT [3] that uses 15% masking rate for self-supervised learning in NLP, most of the tokens/patches can be discarded for spectrograms as well as images due to high redundancy in these modalities.
Revised: Unlike BERT [3] that uses 15% masking rate for self-supervised learning in NLP, most of the tokens/patches can be discarded for spectrograms as well as images due to high redundancy in these modalities.

Original: Beyond self-supervised pre-training, we further explore the effectiveness of masking in the supervised fine-tuning stage.
Revised: Beyond self-supervised pre-training, we further explore the effectiveness of masking in the supervised fine-tuning stage.

Original: Empirically, we found unstructured (random) masking at a higher ratio for pre-training and structured (time
Revised: Empirically, we found unstructured (random) masking at a higher ratio for pre-training and structured (time

Original: + frequency masking) at a lower ratio for fine-tuning provide best accuracy (ablations are in §4.4).
Revised: + frequency masking) at a lower ratio for fine-tuning provide best accuracy (ablations are in §4.4).

Original: Encoder .
Revised: Encoder .

Original: Audio-MAE uses a stack of standard Transformers [2] as its encoder.
Revised: Audio-MAE uses a stack of standard Transformers [2] as its encoder.

Original: The encoder only processes (20%) non-masked patches to reduce computation overhead which is quadratic to the input sequence length.
Revised: The encoder only processes (20%) non-masked patches to reduce computation overhead which is quadratic to the input sequence length.

Original: We use the 12-layer ViT-Base (ViT-B) [9] Transformer as our default.
Revised: We use the 12-layer ViT-Base (ViT-B) [9] Transformer as our default.

Original: Decoder with Local Attention .
Revised: Decoder with Local Attention .

Original: The decoder is also composed of standard Transformer blocks.
Revised: The decoder is also composed of standard Transformer blocks.

Original: The encoded patches from the encoder are padded with trainable masked tokens.
Revised: The encoded patches from the encoder are padded with trainable masked tokens.

Original: After restoring the original time-frequency order in the audio spectrogram, we add the decoder’s (fixed sinusoidal) positional embeddings and feed the restored sequence into the decoder.
Revised: After restoring the original time-frequency order in the audio spectrogram, we add the decoder’s (fixed sinusoidal) positional embeddings and feed the restored sequence into the decoder.

Original: At the top of the decoder stack, we add a linear head to predict and reconstruct the input spectrogram.
Revised: At the top of the decoder stack, we add a linear head to predict and reconstruct the input spectrogram.

Original: To address the unique characteristics of audio spectrograms, our work investigates an enhancement to the vanilla MAE decoder.
Revised: To address the unique characteristics of audio spectrograms, our work investigates an enhancement to the vanilla MAE decoder.

Original: Image-based MAE uses global self-attention in the Transformer decoder which is appropriate for visual context, because visual objects are typically invariant under translation or scaling, and their exact position may not affect the semantics of an image.
Revised: Image-based MAE uses global self-attention in the Transformer decoder which is appropriate for visual context, because visual objects are typically invariant under translation or scaling, and their exact position may not affect the semantics of an image.

Original: In contrast, the position, scale, and translation of spectrogram features however directly affects the sound or semantics of an audio recording.
Revised: In contrast, the position, scale, and translation of spectrogram features however directly affects the sound or semantics of an audio recording.

Original: Consequently, global self-attention is sub-optimal for spectrograms if the timefrequency components is predominantly local.
Revised: Consequently, global self-attention is sub-optimal for spectrograms if the timefrequency components is predominantly local.

Original: For instance, we would have better success to use the harmonics ( e.g ., Fig. 2a) in lower bands of a vowel to predict the spectrogram patch vertically in a higher frequency band rather than horizontally in the time domain.
Revised: For instance, we would have better success to use the harmonics ( e.g ., Fig. 2a) in lower bands of a vowel to predict the spectrogram patch vertically in a higher frequency band rather than horizontally in the time domain.

Original: Similarly, a frictional sound of a consonant likely only correlates to other part of the consonant, and is without dependency to other silence segments in the audio recording.
Revised: Similarly, a frictional sound of a consonant likely only correlates to other part of the consonant, and is without dependency to other silence segments in the audio recording.

Original: Compared to images, the spectrogram patches are more similar to speech or text tokens where its order and position is more relevant.
Revised: Compared to images, the spectrogram patches are more similar to speech or text tokens where its order and position is more relevant.

Original: [Figure]
Revised: [Figure]

Original: To address the nature of audio spectrograms, in addition to using Transformers with global self-attention as in vanilla MAE, we incorporate the local attention mechanism which groups and separates the spectrogram patches in to local windows in self-attention for decoding.
Revised: To address the nature of audio spectrograms, in addition to using Transformers with global self-attention as in vanilla MAE, we incorporate the local attention mechanism which groups and separates the spectrogram patches in to local windows in self-attention for decoding.

Original: We investigate two types of local attention: (1) Shifted window location: Inspired by the shifted-window in Swin Transformers [19], we shift window attention by 50% between consecutive Transformer decoder layers.
Revised: We investigate two types of local attention: (1) Shifted window location: Inspired by the shifted-window in Swin Transformers [19], we shift window attention by 50% between consecutive Transformer decoder layers.

Original: For padding the margin when shifting, we cyclically shift the spectrogram to the top-left direction.
Revised: For padding the margin when shifting, we cyclically shift the spectrogram to the top-left direction.

Original: Fig.
Revised: Fig.

Original: 3 illustrates the localized decoder attention by shifted windows.
Revised: 3 illustrates the localized decoder attention by shifted windows.

Original: (2) Hybrid window attention (global + local attention): Inspired by [45], to add better cross-window connections, we design a simple hybrid (global+local) attention that computes local attention within a window in all but the last few top layers.
Revised: (2) Hybrid window attention (global + local attention): Inspired by [45], to add better cross-window connections, we design a simple hybrid (global+local) attention that computes local attention within a window in all but the last few top layers.

Original: In this way, the input feature maps for the final reconstruction layer also contain global information.
Revised: In this way, the input feature maps for the final reconstruction layer also contain global information.

Original: For simplicity, we use no pooling or hierarchical structure.
Revised: For simplicity, we use no pooling or hierarchical structure.

Original: Decoders with different attention types are compared in §4.4.
Revised: Decoders with different attention types are compared in §4.4.

Original: Objective .
Revised: Objective .

Original: The Audio-MAE decoder learns to reconstruct the input spectrogram by predicting the values in the spectrogram patches or their per-patch normalized ones.
Revised: The Audio-MAE decoder learns to reconstruct the input spectrogram by predicting the values in the spectrogram patches or their per-patch normalized ones.

Original: The objective is the mean squared error (MSE) between the prediction and the input spectrogram, averaged over unknown patches.
Revised: The objective is the mean squared error (MSE) between the prediction and the input spectrogram, averaged over unknown patches.

Original: Empirically we found employing the reconstruction loss alone is sufficient while including additional contrastive objectives ( e.g ., InfoNCE loss [46]) does not improve Audio-MAE.
Revised: Empirically we found employing the reconstruction loss alone is sufficient while including additional contrastive objectives ( e.g ., InfoNCE loss [46]) does not improve Audio-MAE.

Original: Fine-tuning for Downstream Tasks .
Revised: Fine-tuning for Downstream Tasks .

Original: In the fine-tuning stage, we only keep and fine-tune the AudioMAE encoder and discard the decoder.
Revised: In the fine-tuning stage, we only keep and fine-tune the AudioMAE encoder and discard the decoder.

Original: Different from the original MAE, and inspired by [47, 28], we also explore to employ masking in the fine-tuning stage to remove a portion of patches to further regularize learning from a limited view of spectrogram inputs, which, as a side effect, also reduces computation during fine-tuning.
Revised: Different from the original MAE, and inspired by [47, 28], we also explore to employ masking in the fine-tuning stage to remove a portion of patches to further regularize learning from a limited view of spectrogram inputs, which, as a side effect, also reduces computation during fine-tuning.

Original: Compared to SpecAug [48] which takes full-length input with the masked portion set to zero as data augmentation, Audio-MAE sees only a subset of real-valued input patches without the nullified ones.
Revised: Compared to SpecAug [48] which takes full-length input with the masked portion set to zero as data augmentation, Audio-MAE sees only a subset of real-valued input patches without the nullified ones.

Original: Audio-MAE then encodes these non-masked patches and applies an average pooling layer followed by a linear layer on top for fine-tuning in classification tasks.
Revised: Audio-MAE then encodes these non-masked patches and applies an average pooling layer followed by a linear layer on top for fine-tuning in classification tasks.

Original:  Experiments
Revised:  Experiments

Original: We perform an extensive evaluation on six tasks, including audio classification on AudioSet (AS-2M,
Revised: We perform an extensive evaluation on six tasks, including audio classification on AudioSet (AS-2M,

Original: AS-20K) and Environmental Sound Classification (ESC-50), and speech classification on Speech
Revised: AS-20K) and Environmental Sound Classification (ESC-50), and speech classification on Speech

Original: Commands (SPC-1 and SPC-2) and VoxCeleb (SID).
Revised: Commands (SPC-1 and SPC-2) and VoxCeleb (SID).

Original: We use AudioSet for ablation studies.
Revised: We use AudioSet for ablation studies.

Original: 4.1 Datasets and Tasks
Revised: 4.1 Datasets and Tasks

Original: AudioSet [12] (AS-2M, AS-20K) contains ∼ 2 million 10-second YouTube clips for audio classification.
Revised: AudioSet [12] (AS-2M, AS-20K) contains ∼ 2 million 10-second YouTube clips for audio classification.

Original: 527 types of audio events are weakly annotated [49, 50, 51] for each clip.
Revised: 527 types of audio events are weakly annotated [49, 50, 51] for each clip.

Original: There could be multiple events in a clip.
Revised: There could be multiple events in a clip.

Original: The full training set has 2 subsets: A class-wise balanced (22,176 clips) and an unbalanced (2,042,985 clips) set.
Revised: The full training set has 2 subsets: A class-wise balanced (22,176 clips) and an unbalanced (2,042,985 clips) set.

Original: The eval set has 20,383 clips.
Revised: The eval set has 20,383 clips.

Original: We downloaded and processed around 1.96M unbalanced training, 21K balanced training, and 19K evaluation clips.
Revised: We downloaded and processed around 1.96M unbalanced training, 21K balanced training, and 19K evaluation clips.

Original: For the AS-2M experiments, we use the union of unbalanced and balanced training audio for pretraining and fine-tuning.
Revised: For the AS-2M experiments, we use the union of unbalanced and balanced training audio for pretraining and fine-tuning.

Original: For the AS-20K experiments, we use AS-2M for pre-training and the 20K balanced set for fine-tuning.
Revised: For the AS-20K experiments, we use AS-2M for pre-training and the 20K balanced set for fine-tuning.

Original: We report the testing mAP on the 19K eval set used by AST [10].
Revised: We report the testing mAP on the 19K eval set used by AST [10].

Original: Environmental Sound Classification (ESC-50) [13] is an audio classification dataset consists of 2,000 5-second environmental sound recordings.
Revised: Environmental Sound Classification (ESC-50) [13] is an audio classification dataset consists of 2,000 5-second environmental sound recordings.

Original: There are 50 classes in ESC.
Revised: There are 50 classes in ESC.

Original: We report accuracy under 5-fold cross-validation with the same split used by [10].
Revised: We report accuracy under 5-fold cross-validation with the same split used by [10].

Original: Speech Commands (SPC-2, SPC-1) [52] are two keyword spotting tasks.
Revised: Speech Commands (SPC-2, SPC-1) [52] are two keyword spotting tasks.

Original: In SPC-2, there arespeech commands.
Revised: In SPC-2, there arespeech commands.

Original: The training/validation/testing set has 84,843/9,981/11,005 1-second recordings, respectively.
Revised: The training/validation/testing set has 84,843/9,981/11,005 1-second recordings, respectively.

Original: In SPC-1, there are 10 classes of keywords, 1 silence class, and 1 unknown class that includes all the other 20 common speech commands.
Revised: In SPC-1, there are 10 classes of keywords, 1 silence class, and 1 unknown class that includes all the other 20 common speech commands.

Original: We use the data and split provided in the
Revised: We use the data and split provided in the

Original: SUPERB [53] benchmark to report the testing accuracy.
Revised: SUPERB [53] benchmark to report the testing accuracy.

Original: VoxCeleb (SID) [54] contains 150K utterances from 1,251 speakers.
Revised: VoxCeleb (SID) [54] contains 150K utterances from 1,251 speakers.

Original: The speaker identification task (SID) is to classify the utterances to identify its original speaker.
Revised: The speaker identification task (SID) is to classify the utterances to identify its original speaker.

Original: We use the V1 standard train (138,361), validation (6,904), testing (8,251) sets and report the testing accuracy.
Revised: We use the V1 standard train (138,361), validation (6,904), testing (8,251) sets and report the testing accuracy.

Original: 4.2 Implementation Details
Revised: 4.2 Implementation Details

Original: We use a vanilla 12-layer ViT-B by default as the Transformer encoder.
Revised: We use a vanilla 12-layer ViT-B by default as the Transformer encoder.

Original: For the decoder, we use a 16-layer Transformer with shifted local attention.
Revised: For the decoder, we use a 16-layer Transformer with shifted local attention.

Original: We investigate the vanilla (global attention) and hybrid (global + local attention) decoder variants (see Table. 1c).
Revised: We investigate the vanilla (global attention) and hybrid (global + local attention) decoder variants (see Table. 1c).

Original: Following [10, 11], we transform raw waveform (pre-processed as mono channel under 16, sampling rate) into 128 Kaldi [55]-compatible Mel-frequency bands with a 25ms Hanning window that shifts every 10 ms.
Revised: Following [10, 11], we transform raw waveform (pre-processed as mono channel under 16, sampling rate) into 128 Kaldi [55]-compatible Mel-frequency bands with a 25ms Hanning window that shifts every 10 ms.

Original: For a 10-second recording in AudioSet, the resulting spectrogram is of 1 × 1024 × 128 dimension.
Revised: For a 10-second recording in AudioSet, the resulting spectrogram is of 1 × 1024 × 128 dimension.

Original: For patch embedding, we use convolutional kernels with (16 , 16) size and stride in time and frequency (thus, patches are non-overlapping) to avoid short-cuts via overlap in self-supervision (though, at high masking ratios such short-cuts are less severe).
Revised: For patch embedding, we use convolutional kernels with (16 , 16) size and stride in time and frequency (thus, patches are non-overlapping) to avoid short-cuts via overlap in self-supervision (though, at high masking ratios such short-cuts are less severe).

Original: By default, we use a masking ratio of 0 .
Revised: By default, we use a masking ratio of 0 .

Original: 8 with (unstructured) random masking for pre-training.
Revised: 8 with (unstructured) random masking for pre-training.

Original: During fine-tuning, we employ a lower masking ratio ( 0 . 3 in time and 0 .
Revised: During fine-tuning, we employ a lower masking ratio ( 0 . 3 in time and 0 .

Original: 3 in frequency).
Revised: 3 in frequency).

Original: Ablations on these design choices are given in §4.4.
Revised: Ablations on these design choices are given in §4.4.

Original: 4.3 Pre-training and Fine-tuning
Revised: 4.3 Pre-training and Fine-tuning

Original: We use AudioSet-2M for pre-training and randomly iterate over all audio recordings.
Revised: We use AudioSet-2M for pre-training and randomly iterate over all audio recordings.

Original: We train for 32 epochs with a batch size of 512 and a 0.0002 learning rate.
Revised: We train for 32 epochs with a batch size of 512 and a 0.0002 learning rate.

Original: We distribute the training load over 64 V100 GPUs and the total training time is ∼ 36 hours.
Revised: We distribute the training load over 64 V100 GPUs and the total training time is ∼ 36 hours.

Original: For each audio, we randomly sample the starting time, cyclically extract 10-second audio, and randomly jitter its magnitude by up to ± 6dB.
Revised: For each audio, we randomly sample the starting time, cyclically extract 10-second audio, and randomly jitter its magnitude by up to ± 6dB.

Original: We use only natural audio spectrograms and apply no augmentations ( e.g ., [48, 56, 57]) as we do not find these strong augmentations helpful in the pre-training phase.
Revised: We use only natural audio spectrograms and apply no augmentations ( e.g ., [48, 56, 57]) as we do not find these strong augmentations helpful in the pre-training phase.

Original: In the fine-tuning phase, we remove the decoder and only fine-tune the encoder.
Revised: In the fine-tuning phase, we remove the decoder and only fine-tune the encoder.

Original: For the supervised fine-tuning on AudioSet-2M, since the size of training samples are uneven across classes (unbalanced), we follow the common practice of using a weighted sampling to balance the classes during training.
Revised: For the supervised fine-tuning on AudioSet-2M, since the size of training samples are uneven across classes (unbalanced), we follow the common practice of using a weighted sampling to balance the classes during training.

Original: In each epoch, we sample 200K instances ( ∼ 10% of AudioSet-2M) without replacement.
Revised: In each epoch, we sample 200K instances ( ∼ 10% of AudioSet-2M) without replacement.

Original: We fine-tune unstructured time freq time+freq (a)
Revised: We fine-tune unstructured time freq time+freq (a)

Original: Pre-training masking 46.
Revised: Pre-training masking 46.

Original: unstructured time freq time+freq
Revised: unstructured time freq time+freq

Original: [Figure]
Revised: [Figure]

Original: for 100 epochs, which aggregate to ∼ 10 full epochs of AudioSet-2M.
Revised: for 100 epochs, which aggregate to ∼ 10 full epochs of AudioSet-2M.

Original: The probability of sampling an instance is inversely proportional to the dataset-wise occurrences of its classes.
Revised: The probability of sampling an instance is inversely proportional to the dataset-wise occurrences of its classes.

Original: Fine-tuning on 64 GPUs takes ∼ 12 hours.
Revised: Fine-tuning on 64 GPUs takes ∼ 12 hours.

Original: For the smaller balanced AudioSet-20K, we fine-tune on 4 GPUs forepochs without weighted sampling.
Revised: For the smaller balanced AudioSet-20K, we fine-tune on 4 GPUs forepochs without weighted sampling.

Original: Please see Supplementary for the details on other datasets.
Revised: Please see Supplementary for the details on other datasets.

Original: 4.4 Ablations and Model Properties
Revised: 4.4 Ablations and Model Properties

Original: Masking Strategies in Pre-training and Fine-tuning.
Revised: Masking Strategies in Pre-training and Fine-tuning.

Original: In Fig.
Revised: In Fig.

Original: 4, we compare different pre-training and fine-tuning masking strategies for Audio-MAE.
Revised: 4, we compare different pre-training and fine-tuning masking strategies for Audio-MAE.

Original: First, in Fig.
Revised: First, in Fig.

Original: 4a we explore the pre-training masking ratio .
Revised: 4a we explore the pre-training masking ratio .

Original: We observe, similar as in MAE for images [1], that a high pre-training masking ratio (80% in our case) is optimal for audio spectrograms.
Revised: We observe, similar as in MAE for images [1], that a high pre-training masking ratio (80% in our case) is optimal for audio spectrograms.

Original: This is due to the fact that both audio spectrograms and images are continuous signals with significant redundancy.
Revised: This is due to the fact that both audio spectrograms and images are continuous signals with significant redundancy.

Original: Further, we find the unstructured random masking works the best for self-supervised pre-training over more structured masking ( e.g ., time + frequency).
Revised: Further, we find the unstructured random masking works the best for self-supervised pre-training over more structured masking ( e.g ., time + frequency).

Original: Unlike MAE for images, there are clear performance differences among masking strategies when pre-training with audio spectrograms.
Revised: Unlike MAE for images, there are clear performance differences among masking strategies when pre-training with audio spectrograms.

Original: Comparing Audio-MAE reconstructions between Fig.
Revised: Comparing Audio-MAE reconstructions between Fig.

Original: 6a to 6e and 6d to 6h, under the same masking ratio, we observe the unstructured random masking is comparably easier than structured masking ( i.e ., time and/or frequency) as the model can guess the missing component by extrapolating nearby context ( e.g ., formants in vowels and frictional sounds in consonants around).
Revised: 6a to 6e and 6d to 6h, under the same masking ratio, we observe the unstructured random masking is comparably easier than structured masking ( i.e ., time and/or frequency) as the model can guess the missing component by extrapolating nearby context ( e.g ., formants in vowels and frictional sounds in consonants around).

Original: We also observe that for higher masking ratios, the structured masking alternatives drop in performance, presumably because the task becomes too difficult while random masking improves steadily up to 80%.
Revised: We also observe that for higher masking ratios, the structured masking alternatives drop in performance, presumably because the task becomes too difficult while random masking improves steadily up to 80%.

Original: This result show that designing a pretext task with proper hardness is important for effective self-supervised learning of audio representations.
Revised: This result show that designing a pretext task with proper hardness is important for effective self-supervised learning of audio representations.

Original: We therefore use random masking with ratio of 80% as our default for pre-training.
Revised: We therefore use random masking with ratio of 80% as our default for pre-training.

Original: Fig.
Revised: Fig.

Original: 4b studies the effect of masking during the fine-tuning phase.
Revised: 4b studies the effect of masking during the fine-tuning phase.

Original: We see that in this case, it is more beneficial to use structured masking: time + frequency performs better than time- or frequency-based masking, and these perform better than unstructured masking.
Revised: We see that in this case, it is more beneficial to use structured masking: time + frequency performs better than time- or frequency-based masking, and these perform better than unstructured masking.

Original: Overall, we see that the optimal masking ratios are lower than for pre-training and we use 0.3 as our default in the fine-tuning phase.
Revised: Overall, we see that the optimal masking ratios are lower than for pre-training and we use 0.3 as our default in the fine-tuning phase.

Original: In general, we observe that for task-agnostic pre-training, unstructured masking with a higher ratio is preferred.
Revised: In general, we observe that for task-agnostic pre-training, unstructured masking with a higher ratio is preferred.

Original: While in task-specific fine-tuning, structured masking with lower ratios performs better.
Revised: While in task-specific fine-tuning, structured masking with lower ratios performs better.

Original: Impact of Patch Size and Stride.
Revised: Impact of Patch Size and Stride.

Original: We compare the performance of Audio-MAE trained with different patch sizes and strides in Table 1a.
Revised: We compare the performance of Audio-MAE trained with different patch sizes and strides in Table 1a.

Original: A non-zero overlap ( i.e ., stride < patch size) between patches will increase the number of patches and quadratically increase computation in floating point operations (FLOPs), as reported in the table.
Revised: A non-zero overlap ( i.e ., stride < patch size) between patches will increase the number of patches and quadratically increase computation in floating point operations (FLOPs), as reported in the table.

Original: Most prior works follow AST [10] to use overlapped patches (patch = 16 and stride = 10 ) to boost end task performance.
Revised: Most prior works follow AST [10] to use overlapped patches (patch = 16 and stride = 10 ) to boost end task performance.

Original: As shown in Table 1a, we do not observe a performance improvement using overlapped patches for Audio-MAE (both 47.3 mAP), presumably because due to overlap, the patch embedding can leak information into the masked patches.
Revised: As shown in Table 1a, we do not observe a performance improvement using overlapped patches for Audio-MAE (both 47.3 mAP), presumably because due to overlap, the patch embedding can leak information into the masked patches.

Original: The non-overlapped 16 × 16 patches achieve a good balance between computation and performance.
Revised: The non-overlapped 16 × 16 patches achieve a good balance between computation and performance.

Original: By default, we use this setup in our experiments.
Revised: By default, we use this setup in our experiments.

Original: Encoder.
Revised: Encoder.

Original: We investigate the design choices of encoder and decoder architectures in Audio-MAE.
Revised: We investigate the design choices of encoder and decoder architectures in Audio-MAE.

Original: Table 1b shows the trade-off between encoder model size and performance.
Revised: Table 1b shows the trade-off between encoder model size and performance.

Original: As expected, larger models achieve better performance, at a cost of computation and memory.
Revised: As expected, larger models achieve better performance, at a cost of computation and memory.

Original: The accuracy gain of ViT-L over ViT-B/S is more significant on the smaller and balanced AS-20K. For ViT-S, the performance
Revised: The accuracy gain of ViT-L over ViT-B/S is more significant on the smaller and balanced AS-20K. For ViT-S, the performance

Original: [Table]
Revised: [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: gap to ViT-B can be significantly closed (5.0 → 2.3 mAP) when fine-tuning with more in-domain data (AS-20K → AS-2M).
Revised: gap to ViT-B can be significantly closed (5.0 → 2.3 mAP) when fine-tuning with more in-domain data (AS-20K → AS-2M).

Original: [Figure]
Revised: [Figure]

Original: Decoder.
Revised: Decoder.

Original: Table 1c compares decoder attention types in
Revised: Table 1c compares decoder attention types in

Original: Audio-MAE.
Revised: Audio-MAE.

Original: Note that decoders are discarded after pretraining and only the equal-sized ViT-B encoders are fine-tuned for the end task.
Revised: Note that decoders are discarded after pretraining and only the equal-sized ViT-B encoders are fine-tuned for the end task.

Original: Our results show that local attention with shifted window achieves the best performance.
Revised: Our results show that local attention with shifted window achieves the best performance.

Original: Combining local and global attention ( i.e ., hybrid attention, Hwin) also improves vanilla global self-attention.
Revised: Combining local and global attention ( i.e ., hybrid attention, Hwin) also improves vanilla global self-attention.

Original: Fig.
Revised: Fig.

Original: 5 shows the qualitative reconstruction comparison.
Revised: 5 shows the qualitative reconstruction comparison.

Original: In the spectrogram of vowels, the decoder with local attention reconstructs better harmonics and recovers more context in the spectrogram.
Revised: In the spectrogram of vowels, the decoder with local attention reconstructs better harmonics and recovers more context in the spectrogram.

Original: Similar phenomena are observed in the frictional sound in the middle consonant.
Revised: Similar phenomena are observed in the frictional sound in the middle consonant.

Original: Table 1d ablates the impact of decoder depth on mAP.
Revised: Table 1d ablates the impact of decoder depth on mAP.

Original: A deeper 16-layer decoder achieves better performance against its shallower variants.
Revised: A deeper 16-layer decoder achieves better performance against its shallower variants.

Original: Note that our decoder uses local window attention by default where only a fraction of tokens (4 × 4 local windows vs. 64 × 8 with global attention) are attended.
Revised: Note that our decoder uses local window attention by default where only a fraction of tokens (4 × 4 local windows vs. 64 × 8 with global attention) are attended.

Original: For global attention we find 8-layer decoders to perform better than 16-layer.
Revised: For global attention we find 8-layer decoders to perform better than 16-layer.

Original: Table 1e compares decoder width (embedding dimension).
Revised: Table 1e compares decoder width (embedding dimension).

Original: A 512-dimension decoder achieves a good trade-off between computation and performance as a wider one is not better.
Revised: A 512-dimension decoder achieves a good trade-off between computation and performance as a wider one is not better.

Original: Pre-training Data and Setup.
Revised: Pre-training Data and Setup.

Original: Table 1f summarizes the impact of pre-training dataset size.
Revised: Table 1f summarizes the impact of pre-training dataset size.

Original: Overall the model performance is monotonically increasing when using more data for pre-training.
Revised: Overall the model performance is monotonically increasing when using more data for pre-training.

Original: Comparing the performance of using 1% well-annotated AS-20K balanced data to using randomly sampled 20K unbalanced data for pre-training, the similar mAPs (39.4 vs 39.6) suggest that the distribution of data classes (balanced vs. unbalanced) is less important for pre-training.
Revised: Comparing the performance of using 1% well-annotated AS-20K balanced data to using randomly sampled 20K unbalanced data for pre-training, the similar mAPs (39.4 vs 39.6) suggest that the distribution of data classes (balanced vs. unbalanced) is less important for pre-training.

Original: Meanwhile, as shown in
Revised: Meanwhile, as shown in

Original: Table 1g, training for longer is beneficial yet the performance saturates after the 24- th epoch.
Revised: Table 1g, training for longer is beneficial yet the performance saturates after the 24- th epoch.

Original: Out-of-domain Pre-training on ImageNet.
Revised: Out-of-domain Pre-training on ImageNet.

Original: Initializing audio models from ImageNet pre-trained weights has become popular for audio classification.
Revised: Initializing audio models from ImageNet pre-trained weights has become popular for audio classification.

Original: However, as there are significant discrepancies between image and audio modalities, it is questionable if out-of-domain pre-training benefits audio representation learning.
Revised: However, as there are significant discrepancies between image and audio modalities, it is questionable if out-of-domain pre-training benefits audio representation learning.

Original: In Table 1h we design 3 scenarios to investigate this for Audio-MAE: (1)
Revised: In Table 1h we design 3 scenarios to investigate this for Audio-MAE: (1)

Original: [Table]
Revised: [Table]

Original: Audio-only pre-training (AS-SSL) from scratch.
Revised: Audio-only pre-training (AS-SSL) from scratch.

Original: We consider this the ideal schema for learning audio representations as it is a simple and clean setup that prevents uncontrollable bias transfer from other modalities.
Revised: We consider this the ideal schema for learning audio representations as it is a simple and clean setup that prevents uncontrollable bias transfer from other modalities.

Original: (2) Directly using self-supervised ImageNet MAE models (IN-SSL) and its fine-tuned variant (IN-SL).
Revised: (2) Directly using self-supervised ImageNet MAE models (IN-SSL) and its fine-tuned variant (IN-SL).

Original: (3) Audio-MAE self-supervised pre
Revised: (3) Audio-MAE self-supervised pre

Original: -training on top of these ImageNet weights.
Revised: -training on top of these ImageNet weights.

Original: The results show that (1) from-scratch audio-only pre-training is the best.
Revised: The results show that (1) from-scratch audio-only pre-training is the best.

Original: For scenarios (2) and (3), we observe that ImageNet pre-training alone (2) is not sufficient (especially when the downstream data is smaller, AS-20K), and, in self-supervised pre-training on AudioSet, ImageNet initialization (3) does not help but degrades accuracy.
Revised: For scenarios (2) and (3), we observe that ImageNet pre-training alone (2) is not sufficient (especially when the downstream data is smaller, AS-20K), and, in self-supervised pre-training on AudioSet, ImageNet initialization (3) does not help but degrades accuracy.

Original: Also in (3), supervised ImageNet pre-training (IN-SL) seems harmful.
Revised: Also in (3), supervised ImageNet pre-training (IN-SL) seems harmful.

Original: Consequently, the result suggests that out-of-domain pre-training ( i.e ., ImageNet) is not helpful for Audio-MAE, possibly due to domain shift.
Revised: Consequently, the result suggests that out-of-domain pre-training ( i.e ., ImageNet) is not helpful for Audio-MAE, possibly due to domain shift.

Original: 4.5 Comparison with the State-of-the-art
Revised: 4.5 Comparison with the State-of-the-art

Original: Table 2 compares Audio-MAE (with 3-run error bars) to prior state-of-the-art.
Revised: Table 2 compares Audio-MAE (with 3-run error bars) to prior state-of-the-art.

Original: We categorize the comparison into 3 groups.
Revised: We categorize the comparison into 3 groups.

Original: For fair comparison, our main benchmark is the models in the middle group with self-supervised pre-training on in-domain (audio) datasets (AudioSet and LibriSpeech).
Revised: For fair comparison, our main benchmark is the models in the middle group with self-supervised pre-training on in-domain (audio) datasets (AudioSet and LibriSpeech).

Original: For reference we also list other models without pre-training (the top group) and other models with supervised pre-training on out-of-domain ImageNet (the bottom group), where the latter contains previous best systems on the datasets.
Revised: For reference we also list other models without pre-training (the top group) and other models with supervised pre-training on out-of-domain ImageNet (the bottom group), where the latter contains previous best systems on the datasets.

Original: Pre-trained on AudioSet, Audio-MAE achieves the best performance across all tasks compared to other models with in-domain self-supervised pre-training.
Revised: Pre-trained on AudioSet, Audio-MAE achieves the best performance across all tasks compared to other models with in-domain self-supervised pre-training.

Original: On AudioSet-20K, its 37.1 mAP significantly outperforms all other approaches including concurrent works and other models with outof-domain pre-training.
Revised: On AudioSet-20K, its 37.1 mAP significantly outperforms all other approaches including concurrent works and other models with outof-domain pre-training.

Original: On AudioSet-2M and ESC-50, our method also outperforms Conformer [37] and SS-AST [18].
Revised: On AudioSet-2M and ESC-50, our method also outperforms Conformer [37] and SS-AST [18].

Original: Notably, unlike SS-AST and concurrent MAE-AST [38], which trained with additional 1,000 hours of speech in Librispeech, we use only AudioSet for pre-training.
Revised: Notably, unlike SS-AST and concurrent MAE-AST [38], which trained with additional 1,000 hours of speech in Librispeech, we use only AudioSet for pre-training.

Original: In the bottom group of Table 2, Audio-MAE also outperforms previous state-of-the-art models with
Revised: In the bottom group of Table 2, Audio-MAE also outperforms previous state-of-the-art models with

Original: ImageNet supervised pre-training.
Revised: ImageNet supervised pre-training.

Original: Note that the proposed Audio-MAE does not rely on any out-ofdomain data and labels, nor using knowledge distillation ( e.g ., DeiT) from additional CNN-based models.
Revised: Note that the proposed Audio-MAE does not rely on any out-ofdomain data and labels, nor using knowledge distillation ( e.g ., DeiT) from additional CNN-based models.

Original: Also, compared to HTS-AT [29] and PaSST [28], Audio-MAE is trained with audio under 16K sampling rate.
Revised: Also, compared to HTS-AT [29] and PaSST [28], Audio-MAE is trained with audio under 16K sampling rate.

Original: As experimented in [59], there could be up to 0.4 potential mAP improvement for
Revised: As experimented in [59], there could be up to 0.4 potential mAP improvement for

Original: Audio-MAE if audio with 32K sampling rate are available.
Revised: Audio-MAE if audio with 32K sampling rate are available.

Original: [Figure]
Revised: [Figure]

Original: For the speech tasks (SPC-1, SPC-2, and SID), Audio-MAE outperforms other models without pre-training (ERANN [58], PANN [59]), supervised (AST) and self-supervised models (SS-AST,
Revised: For the speech tasks (SPC-1, SPC-2, and SID), Audio-MAE outperforms other models without pre-training (ERANN [58], PANN [59]), supervised (AST) and self-supervised models (SS-AST,

Original: MAE-AST).
Revised: MAE-AST).

Original: We further list other works (marked with
Revised: We further list other works (marked with

Original: ) to include the latest results introduced in the SUPERB [53] benchmark.
Revised: ) to include the latest results introduced in the SUPERB [53] benchmark.

Original: But note that these results are not strictly comparable since SUPERB employs linear evaluation where the underlying pre-trained models are not end-to-end fine-tuned.
Revised: But note that these results are not strictly comparable since SUPERB employs linear evaluation where the underlying pre-trained models are not end-to-end fine-tuned.

Original: In summary, with audio-only from-scratch pre-training on AudioSet, our Audio-MAE performs well for both the audio and speech classification tasks.
Revised: In summary, with audio-only from-scratch pre-training on AudioSet, our Audio-MAE performs well for both the audio and speech classification tasks.

Original: 4.6 Visualization and Audible Examples by Audio-MAE Decoder
Revised: 4.6 Visualization and Audible Examples by Audio-MAE Decoder

Original: For better visualization, we follow MAE [1] to use MSE over non-normalized spectrograms as the selfsupervised objective.
Revised: For better visualization, we follow MAE [1] to use MSE over non-normalized spectrograms as the selfsupervised objective.

Original: We use ViT-L as the Audio-MAE encoder for visualization.
Revised: We use ViT-L as the Audio-MAE encoder for visualization.

Original: Fig.
Revised: Fig.

Original: 6 illustrates the reconstruction results sampled from the AudioSet-2M eval set.
Revised: 6 illustrates the reconstruction results sampled from the AudioSet-2M eval set.

Original: We further reconstruct .wav
Revised: We further reconstruct .wav

Original: s using the Griffin-Lim [61] algorithm, audible under the anonymous links (accessible in respective 1 2 3).
Revised: s using the Griffin-Lim [61] algorithm, audible under the anonymous links (accessible in respective 1 2 3).

Original: As can be seen and heard, for various masking strategies and different sounds, our Audio-MAE generates reasonable reconstruction.
Revised: As can be seen and heard, for various masking strategies and different sounds, our Audio-MAE generates reasonable reconstruction.

Original: It works well for noisy event sounds ( e.g ., the reconstructed siren in Fig.
Revised: It works well for noisy event sounds ( e.g ., the reconstructed siren in Fig.

Original: 6c-3), as well as speech and music ( e.g ., the reconstructed singing in Fig.
Revised: 6c-3), as well as speech and music ( e.g ., the reconstructed singing in Fig.

Original: 6b-3).
Revised: 6b-3).

Original: Notably, unlike visual contents that are typically scale/translation/position invariant [19], absolute positions and arrangement of spectrogram components are critical for humans to understand sound [62].
Revised: Notably, unlike visual contents that are typically scale/translation/position invariant [19], absolute positions and arrangement of spectrogram components are critical for humans to understand sound [62].

Original: For example, shifting a pitch will make an audio sounds completely different.
Revised: For example, shifting a pitch will make an audio sounds completely different.

Original: Also, phoneme sequences in time are important cues for speech understanding.
Revised: Also, phoneme sequences in time are important cues for speech understanding.

Original: Consequently, unstructured masking produces better aligned outputs that are closer to the ground-truth (top row in each subfigure) as the model can make better predictions based on nearby spectrogram patches; while structured masking is harder (less accurate or with words missing), especially when masking is performed over the time axis.
Revised: Consequently, unstructured masking produces better aligned outputs that are closer to the ground-truth (top row in each subfigure) as the model can make better predictions based on nearby spectrogram patches; while structured masking is harder (less accurate or with words missing), especially when masking is performed over the time axis.

Original: A failure example (missing words) is the reconstructed speech in Fig.
Revised: A failure example (missing words) is the reconstructed speech in Fig.

Original: 6e-3.
Revised: 6e-3.

Original:  Conclusion
Revised:  Conclusion

Original: We have explored a simple extension of MAE [1] to audio data.
Revised: We have explored a simple extension of MAE [1] to audio data.

Original: Our Audio-MAE learns to reconstruct masked spectrogram patches from audio recordings and achieves state-of-the-art performance on six audio and speech classification tasks.
Revised: Our Audio-MAE learns to reconstruct masked spectrogram patches from audio recordings and achieves state-of-the-art performance on six audio and speech classification tasks.

Original: We have drawn four interesting observations: First, a simple MAE approach works surprisingly well for audio spectrograms.
Revised: We have drawn four interesting observations: First, a simple MAE approach works surprisingly well for audio spectrograms.

Original: Second, we find that it is possible to learn stronger representations with local self-attention in the decoder.
Revised: Second, we find that it is possible to learn stronger representations with local self-attention in the decoder.

Original: Third, we show that masking can be applied to both pre-training and fine-tuning, improving accuracy and reducing training computation.
Revised: Third, we show that masking can be applied to both pre-training and fine-tuning, improving accuracy and reducing training computation.

Original: The optimal strategy depends on the nature of the data (audio, image, etc .) and the learning type (self-/supervised).
Revised: The optimal strategy depends on the nature of the data (audio, image, etc .) and the learning type (self-/supervised).

Original: Fourth, the best performance can be achieved by pre-training and fine-tuning under the same modality, without reliance on cross-modality transfer learning.
Revised: Fourth, the best performance can be achieved by pre-training and fine-tuning under the same modality, without reliance on cross-modality transfer learning.

Original: In future work, we aim to explore multimodal self-supervised learning with a joint audio-visual MAE approach as these domains share natural correspondences in video data.
Revised: In future work, we aim to explore multimodal self-supervised learning with a joint audio-visual MAE approach as these domains share natural correspondences in video data.

Original: Acknowledgements.
Revised: Acknowledgements.

Original: We thank Kaiming He and Luke Zettlemoyer for their feedback and discussions.Checklist
Revised: We thank Kaiming He and Luke Zettlemoyer for their feedback and discussions.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] Ans: Please refer to the limitation discussion in the supplemental material.

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] .

Original: Ans: Our paper conforms the ethics requirement.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: Data: [Yes] - the datasets we used are publicly available §4.1.

Original: Code and model:

Original: [Yes] - the code and pre-trained model will be released at the url specified in the Abstract.

Original: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] Ans: part of them are in §4.2 and §4.3 and the rest are specified in the Appendix.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] Ans: Please see Table 2.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] Ans: please check §4. 4.

Original: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]

Original: Appendix

Original: The appendix is organized as follows:

Original: In §A, we first demonstrate additional audible visualizations with anonymous URL links.

Original: In §B, we provide the complete experimental details and hyperparameter configurations for pre-training and fine-tuning on each dataset.

Original: Then in §C, we conduct extra experiments on ESC-50 (§C.1) with additional supervised pre-training on AudioSet to complete the comparison with the models marked with † in Table 2 of the main paper.

Original: We then study a case how

Original: Audio-MAE could be applied to a practical speech generation task (§C.2); and share some negative results and insights on directions we tried that did not work well (§C.3).

Original: Finally, we discuss the limitations (§D) of Audio-MAE.

Original: A Additional Reconstruction Details and Results by Audio-MAE Decoder

Original: Fig.

Original: 7 illustrates additional reconstruction results on the AudioSet-2M eval set.

Original: Audible examples are under the anonymous links, accessible by clicking on respective 1 2 3.

Original: (1 is the ground truth reference, 2 is the masked input for Audio-MAE, and 3 is the reconstruction output by Audio-MAE.)

Original: We use an Audio-MAE model with a ViT-L encoder and a 16-layer decoder with local attention for visualization.

Original: The model is trained under 80% unstructured (random) masking on AudioSet.

Original: We inverse Mel-spectrograms and exploit the Griffin-Lim [61] algorithm to reconstruct waveform.

Original: There could be perceivable artifacts due to imperfect phase estimation in [61].

Original: Note that the default masking ratio in Fig.

Original: 7 is 70% for better visualization.

Original: We also show reconstruction results under 80% masking ratio in Fig.

Original: 7e-7h for comparison.

Original: Comparing 2 and 3 under the each caption in Fig.

Original: 7, even with 70%-80% masking ratio, Audio-MAE can still create reasonable reconstructions.

Original: Music and event sound are easier for Audio-MAE due to their relatively predictable spectrogram patterns.

Original: For example, the repeating tempos across time domain ( e.g ., the music in Fig.

Original: 7b and Fig.

Original: 7l) and the harmonics across frequency domain ( e.g ., the siren in Fig.

Original: 7c and the trumpeting elephant in Fig.

Original: 7d) are very well reconstructed.

Original: Speech recordings are more challenging as shown in Fig.

Original: 7a and Fig.

Original: 7e.

Original: In most cases, Audio-MAE successfully restores audio from masked/corrupted inputs.

Original: With these encouraging results, we envision that Audio-MAE can also be applied to other speech generation tasks and qualitatively case-study an application in §C.2.

Original: [Figure]

Original: [Table]

Original: B Experimental Details and Hyperparameter Settings

Original: In this section we provide additional experimental details.

Original: For audio recordings in each dataset, we pre-process all of them into mono channel under 16K sampling rate for simplicity and consistency between pre-training and fine-tuning tasks.

Original: Note that their native sampling rate may not be 16K (there are many 8K or higher sampling rate recordings in AudioSet.

Original: Also, video compression by YouTube may up-samples or down-samples the audio tracks of user-uploaded videos).

Original: During data loading, we pad or trim the audio length (in seconds) on each dataset as follows:

Original: AudioSet: 10, ESC: 5, SPCand SPC-2: 1, SID: 10 seconds.

Original: We use a window of 25 ms with a hop length of 10 ms to transform waveform into 128 mel-bank features.

Original: The resulting input shapes are: AudioSet: 1 × 1024 × 128 ,

Original: ESC: 1 × 512 × 128 , SPC: 1 × 128 × 128 , SID: 1 × 1024 × 128 .

Original: With different input shapes and audio types, we adjust the hyperparameters and data augmentation for each task respectively.

Original: We summarize the pre-training (AS-2M PT) and fine-tuning details on each dataset in Table 3.

Original: We adopt most of the default hyper-parameters used in MAE [1].

Original: Note that the effective learning rate ( lr eff ) depends on the base learning rate ( lr base ) and the batch size.

Original: Precisely, lr eff = lr base ∗ batch size 256 .

Original: When the dataset is multi-label or the mixup [57] augmentation is enabled, we use binary crossentropy loss (BCE) as the fine-tuning objective without label smoothing [67].We also experimented using strong data augmentations ( e.g ., mixup [57], SpecAug [57], and CutMix [56]) for pre-training but found the resulting performance similar or worse (especially for CutMix which resulted in ∼ 0.5 mAP degrade in AudioSet-2M).

Original: Therefore we discard these strong data augmentations in the pre-training phase by default.

Original: To perform importance sampling when fine-tuning on the unbalanced AudioSet-2M, following prior works, we apply a weighted sampler.

Original: We set the probability of sampling a sample proportional to the inverse frequency of its labels, where the label frequency is estimated over the training set.

Original: Specifically, for a instance i in a dataset D with a label pool C , its sampling weight is proportional to (cid:80) c i ∈ C w c , where w c = 1000 (cid:80) i ∈

Original: D c i + ϵ and ϵ = 0 .

Original: 01 is set to avoid underflow in majority classes as in [10].

Original: In each fine-tuning epoch on AS-2M, we sample 200K instances ( ∼ 10% of AudioSet-2M) without replacement in avoidance of duplicated samples in a batch and repeating samples within an epoch.

Original: We fine-tune for 100 epochs, which aggregate to ∼ 10 full epochs of AudioSet-2M. Proper normalization for audio is important to avoid pre-training fine-tuning discrepancy.

Original: We use the training split of each end task to estimate dataset-wise mean and standard deviation The code, scripts, and pretrained models for reproducibility are at https://github.com/facebookresearch/AudioMAE .

Original: C Additional Experiments

Original: In this section, we extend our experimental investigation of Audio-MAE to include additional results that are not covered in the main paper.

Original: First (§C.1), on ESC-50, we report and compare model performance under an additional round of supervised pre-training on labeled AudioSet-2M (models marked with † in Table 2 of the main paper).

Original: Second (§C.2), we include additional qualitative results on packet loss concealment (PLC) as a preliminary case study on practically useful downstream tasks for the decoder in Audio-MAE, and demonstrate its potential impact for generative applications.

Original: Third (§C.3), we share some negative results when we tried incorporating contrastive objectives for

Original: Audio-MAE.

Original: Our findings suggest that using reconstruction objective alone is sufficient.

Original: C.1 ESC-50 with AudioSet-2M Supervised Pre-training

Original: ESC-50 is designed for environmental sound classification.

Original: Besides the pre-training setup introduced in the original paper, we further study a widely compared setup where the models are additionally supervisedly pre-trained with AudioSet data and labels before fine-tuning on ESC-50.

Original: Tablesummarizes the results under this setup where our Audio-MAE achieves state of the art accuracy with the additional AudioSet-2M supervised pre-training.

Original: Note that our model is still audio-only and uses no ImageNet data (IN-SL).

Original: [Table]

Original: C.2 Qualitative Results for a practical generation task

Original: Packet Loss Concealment (PLC) is a widely deployed technique to alleviate side effects from missing or corrupted packets in Voice over IP (VoIP) applications ( e

Original: [Table]

Original: g ., video conferencing, Bluetooth earbuds, wireless virtual reality headset

Original: [Table]

Original: etc .)

Original: When an encoded speech is sent as a sequence of VoIP packets over a network, these packets may get lost or be corrupted during the transmission, resulting in undesirable low quality speech.

Original: To this end, various PLC techniques has been developed.

Original: The recent approaches substitute the corrupted waveform segments by either replacing the corrupted waveform segments with other intact segments base on the acoustic pitch detected

Original: [Table]

Original: or via inpainting with RNN-based [68], CNN-based [69]

Original: [Table]

Original: or autoencoding-based [70, 71] reconstruction.

Original: In this section

Original: [Table]

Original: we qualitatively demonstrate how Audio-MAE could potentially be applied for PLC to recover corrupted waveform segments with its encoder-decoder architecture.

Original: In Fig.

Original: [Table]

Original: we simulate two time-corrupted speech recordings by masking speech in time and perform reconstruction with

Original: Audio-MAE.

Original: In practice, a PLC system may exploit packet checksums to identify corrupted or missing packets and mask them.

Original: The PLC problem then can be viewed as a special case (time-only, structured masking) of Audio-MAE.

Original: As shown in both cases, the Audio-MAE decoder produces reasonable speech reconstruction.

Original: We leave the in-depth study and analysis of generative tasks ( e.g .

Original: PLC and speech bandwidth expansion (BWE) [72, 54]) as the future work.

Original: [Figure]

Original: C.3 Negative Results: Directions that did not work well

Original: Additional Contrastive Objective We examined using additional contrastive objectives in the pretraining phase but do not find them helpful empirically.

Original: Similar to SS-AST [18] and Wave2vec 2.0 [33], we apply InfoNCE [46] loss over masked tokens of an instance.

Original: Specifically, let x i , i = 1 . . .

Original: N denotes the values of i -th masked spectrogram patch where N is the number of masked patches in an instance.

Original: ( e.g ., rounded N = 102 under 80% masking over 64 × 8 spectrogram patches of a 10-second audio recording.)

Original: And let c i denotes its corresponding contextualized embedding projected by a separated decoder head.

Original: We investigate the following contrastive objective:

Original: [Equation]

Original: Intuitively, L c draws closer patches with their contextualized embeddings (positive pairs) at each masked position while contrasting and pushing away mismatched ones (negative pairs) from all masked patches.

Original: For the reconstructive objective, let ˆ x

Original: i

Original: , i = 1 . . .

Original: N be the reconstruction of i -th masked spectrogram patch generated by the reconstruction head of our Audio-MAE decoder.

Original: The original reconstruction objective L r in Audio-MAE is formally defined as:

Original: [Equation]

Original: We consider three setups: (1) Using the reconstructive objective ( L r ) alone (the default setup); (2) using the contrastive objective ( L c ) alone; (3) multi-tasking with both the reconstructive and contrastive objectives ( L r + αL c ), where α is the hyper-parameter that balances two objectives.

Original: Table 5 shows the results: We see that the reconstruction objective L r alone is sufficient and yields the best performance.

Original: Empirically, we do not observe improvement with contrastive objectives alone or under the multi-task setup (the best α is 0.2 in our experiments).

Original: L c and L r do not work complementarily in Audio-MAE.

Original: [Table]

Original: D Limitations

Original: We think there are few direct limitations of this work.

Original: The data scale is one of them.

Original: AudioSet used by Audio-MAE is around two orders of magnitude smaller than the text corpus used in the language [3, 5, 4] counterparts.

Original: Another limitation is duration of each sample: the 10-second recordings in AudioSet are short and thus distant temporal dependencies in audio may not be properly learned yet.

Original: Further, as AudioSet is unbalanced and there are many audio types beyond the 527 classes annotated in AudioSet, Audio-MAE could be sub-optimal when transferring to tasks concerning rare or unseen audio events.

Original: Lastly, while Audio-MAE has greatly improved the efficiency of large-scale self-supervised learning, modeling lengthy audio and high-dimensional data with Transformers is computationally demanding.