R -Spin: Efficient Speaker and Noise-invariant Representation Learning with Acoustic Pieces (2024)

Abstract: This paper introduces Robust Spin (R-Spin), a data-efficient domain-specific self-supervision method for speaker and noise-invariant speech representations by learning discrete acoustic units with speaker-invariant clustering (Spin). R-Spin resolves Spin’s issues and enhances content representations by learning to predict acoustic pieces. R-Spin offers a 12X reduction in computational resources compared to previous state-of-the-art methods while outperforming them in severely distorted speech scenarios. This paper provides detailed analyses to show how discrete units contribute to speech encoder training and improving robustness in diverse acoustic environments.

## introduction self-supervised learning (ssl) for encoder pretraining has emerged as a foundational element in speech processing, outperforming conventional approaches across various applications @xcite . given the substantial cost associated with human annotation of speech data, ssl methods leverage unlabeled audio data to pre-train encoders, generating good representations for downstream tasks like automatic speech recognition (asr) and speaker identification @xcite . the application of ssl models has notably concentrated on asr, aiming to mitigate the dependence on large transcribed corpora @xcite @xcite . thus, extracting content representations has become a crucial aspect of speech ssl research @xcite @xcite . prior studies have devised objective functions to disentangle content from speech, fostering the ability of ssl models to generate speakerinvariant representations through domain-specific self-supervision (ds). in ds, a pre-trained ssl model is fine-tuned with unlabeled data for spe-cific applications. @xcite propose con-tentvec by disentangling speaker and content information, demonstrating promising results. however, contentvec suffers from the requirement of a voice conversion model and substantial computational costs exceeding 600 gpu hours. alternatively, @xcite propose speaker-invariant clustering (spin) to produce content representations with minimal fine-tuning resources. nonetheless, spin is constrained to fine-tuning only the top layers, thereby lacking the flexibility to adapt to diverse acoustic domains. parallel to modeling content information in speech, numerous studies are dedicated to investigating the robustness of speech ssl. while current methods perform well on clean speech datasets, they are vulnerable to out-of-domain data like distorted audio signals @xcite . to mitigate this vulnerability, researchers have proposed noise-invariant training techniques. @xcite proposes hubert-mgr via domain adversarial training to render the fine-tuned hubert model @xcite invariant to domain shifts. wavlm @xcite integrates denoising with the hubert pre-training framework, achieving state-of-the-art performance in many speech processing downstream tasks. similarly, @xcite propose robust data2vec, introducing perturbations to the input to predict the exponential moving average teacher model's representations. in dehubert @xcite , the barlow twins loss @xcite is applied to encourage representation invariability to input perturbations. although many methods have shown success in noisy speech recognition @xcite @xcite , to our knowledge, none have concurrently addressed the disentanglement of speaker and noise while enhancing content information. furthermore, these approaches exhibit inefficiency, often requiring high computation costs and iterating large corpora over numerous epochs. to effectively acquire high-quality content and robust representations for real-world applications, this paper extends spin with noise-invariant training and acoustic piece pseudo-label learning, coined robust spin (r-spin). during training, two utterances of the same content with different distortions are fed into a speech ssl encoder. the outputs are frame-wise vector-quantized with a learnable codebook via online clustering, as in spin. the model is trained to match cluster id distributions between the utterances. to prevent codebook collapse, an additional pseudo-label prediction loss is introduced. the pseudo-labels are generated by learning acoustic pieces @xcite on top of the discrete units produced by a pre-trained spin model, offering better training targets that closely align with phonemes and characters. within this framework, the speech encoder learns speaker and noise-invariant representations, benefiting robustness and content extraction simultaneously. the contributions are summarized as follows: ## conclusion this paper proposes r-spin, a domain-specific self-supervision method with speaker and noiseinvariant clustering for robust content representations. results illustrate the efficacy and broad applicability of r-spin across various acoustic scenarios, even within constrained computation budgets. the acoustic analyses presented in this study offer insights into the characteristics of discrete units of this nature and strategies for their utilization. future directions involve scaling to larger models and exploring its application in diverse downstream tasks like robust voice conversion.