EH - MAM : Easy-to-Hard Masked Acoustic Modeling for Self-Supervised Speech Representation Learning (2024)

Abstract: In this paper, we present EH-MAM (Easy-to-Hard adaptive Masked Acoustic Modeling), a novel self-supervised learning approach for speech representation learning. In contrast to the prior methods that use random masking schemes for Masked Acoustic Modeling (MAM), we introduce a novel selective and adaptive masking strategy. Specifically, during SSL training, we progressively introduce harder regions to the model for reconstruction. Our approach automatically selects hard regions and is built on the observation that the reconstruction loss of individual frames in MAM can provide natural signals to judge the difficulty of solving the MAM pre-text task for that frame. To identify these hard regions, we employ a teacher model that first predicts the frame-wise losses and then decides which frames to mask. By learning to create challenging problems, such as identifying harder frames and solving them simultaneously, the model is able to learn more effective representations and thereby acquire a more comprehensive understanding of the speech. Quantitatively, EH-MAM outperforms several state-of-the-art baselines across various low-resource speech recognition and SUPERB benchmarks by 5%-10%. Additionally, we conduct a thorough analysis to show that the regions masked by EH-MAM effectively capture useful context across speech frames.

## introduction self-supervised learning (ssl) has emerged as one of the most effective paradigms of speech representation learning when labeled data is scarce @xcite . the task is to learn general-purpose speech representations from unlabeled data that can then be transferred to spoken language processing (slp) tasks like automatic speech recognition (asr), speech emotion recognition (ser), etc @xcite . progress in ssl for speech has led to significant performance improvements in a range of low-resource slp tasks including phoneme recognition (pr), keyword spotting (ks), etc @xcite . masked acoustic modeling (mam) has been one the most prevalent pretext tasks for ssl-based speech representation learning wherein the model tries to reconstruct frames that are masked at the input, utilizing the context of the surrounding frames @xcite . although a considerable amount of research in mam has been performed, most has focused on improving model architectures @xcite @xcite and pretext tasks @xcite @xcite , with very limited progress in improving the masking algorithm @xcite . most mam algorithms still perform random masking of input frames. on the other hand, selective masking strategies for other domains, like computer vision (cv) @xcite @xcite the former consistently shows a significant increase in relative wer than the later, thereby indicating that these frames capture more useful context for speech reconstruction as a result of capturing more information, thus building on this result we hypothesize that asking a model to reconstruct these frames will result in stronger learning signals. and natural language processing (nlp) @xcite @xcite that focuses on masking useful context, have shown significant improvements over random masking. this can be attributed to multiple factors, including: (1) variable information content: variable information content in data translates to variable learning signals for the reconstruction task. for instance, in masked language modeling (mlm) @xcite , the reconstruction of high-frequency stop words such as "the" or "is" offers minimal discriminative power due to the ubiquity and low semantic load of these words @xcite . in speech, for example, this can be translated to reconstructing frames corresponding to random noise or partial phonemes, where much of the frames is already available as context. ## related work self-supervised learning. ssl has emerged as a prevalent speech representation learning paradigm, demonstrating impressive downstream performance under low-resource settings @xcite . at its core, ssl relies on the quality of pretext tasks for capturing varied learning signals from unlabeled data sources. based on the nature of the pretext tasks, the ssl frameworks are further categorized into the following sub-categories: 1) contrastive approaches: the pretext task is designed to maximize latent space similarity between the anchor and positive samples while minimizing the similarity between the anchor and negative samples. 2) generative approaches: these methods primarily focus on first building a target by randomly masking multiple speech frames and then reconstructing them by optimizing a similarity measure (mse or cross entropy) between the predicted frames and the targets. the pretext task includes predicting future input from past inputs @xcite , masked from unmasked @xcite or the original from some other corrupted view @xcite . masked acoustic modeling has undoubtedly seen the most success for speech representation learning. ## methodology in this section, we explain the eh-mam methodology. we first provide an overview of the eh-mam learning paradigm (in section 3.1), followed by details on the reconstruction and auxiliary loss formulations (in sections 3.2.1, 3.2.2). finally, we introduce the easy-to-hard masking algorithm (in section 3.2.3). ## results and analysis in this section, we present the quantitative and qualitative results. for quantitative evaluation, we first fine-tune eh-mam on librilight @xcite and evaluate across all the test splits. next, to show the scalability of the speech representations learned by eh-mam, we conduct a downstream evaluation on superb benchmark (wen @xcite . additionally, we also perform a qualitative analysis on the masked regions predicted by the eh-mam. all the results reported for eh-mam are averaged across five runs. ## conclusion in this paper, we propose eh-mam, a novel ssl framework for learning robust speech representations. in contrast to prior work that relies on random masking schemes for creating mam pretext tasks, eh-mam first identifies hard regions to reconstruct using a teacher network and then challenges the student to reconstruct them by progressively introducing hard regions throughout the learning process. next, we introduce an easy-tohard masking scheme that guides the eh-mam to mask harder regions to reconstruct step-by-step. eh-mam outperforms all the other models on popular low-resource asr benchmarks and downstream evaluation on superb.