Towards efficient self-supervised representation learning in speech processing (2024)

Abstract: Self-supervised learning has achieved impressive results in speech processing, but current models are computationally expensive, generating environmental concerns because of their high energy consumption. Therefore, we propose an efficient self-supervised approach to address high computational costs, using a single GPU during 24 to 48 hours of pretraining. The proposed approach combines linear, convolutional, and self-attention layers with several optimizations, including dynamic batching, flash attention, mixed-precision training, gradient accumulation, and acoustic feature extraction with input preprocessing. Computational cost estimations for our proposed model represent up to two orders of magnitude improvements in computational efficiency against existing speech models.

## introduction self-supervised models generate impressive results when learning latent representations, but their training is computationally expensive @xcite . yet, their results in speech processing are astounding because downstream tasks strongly benefit from their learned representations @xcite . self-supervised approaches for speech representation learning can be based on consistency or selftraining @xcite . whether using consistency or self-training, large training costs represent a challenge. indeed, most existing models require several gpus for days to pretrain their neural architectures. this requirement causes several limitations. first, it hinders the training and deployment of speech models in computing platforms with low resources, such as edge devices and mobile platforms @xcite . secondly, reproducibility is challenging, as few labs can afford large computational resources @xcite . last but not least, it creates environ-mental concerns because of the high energy consumption during training @xcite . to address those limitations, we propose an efficient self-supervised model to learn speech representations. instead of focusing on the model performance in downstream tasks, the proposed model focuses primarily on computational costs, limiting the resources available for pretraining. we set a pretraining limit based on cramming @xcite : we use a single gpu for 24 to 48 hours to train the model. ## related work several models have been recently proposed for self-supervised learning of speech representations, including combinedssl @xcite , mockingjay @xcite , spiral @xcite , data2vec2 @xcite , and di-nosr @xcite . but two approaches have clearly emerged @xcite : hidden unit bert (hubert) @xcite and wav2vec2 @xcite . however, selfsupervised models are quite costly, requiring a lot of computational resources for training. one alternative to reduce training costs is knowledge distillation @xcite , where a small student model learns from a large teacher model, which has been pretrained previously @xcite . using knowledge distillation, lighthubert @xcite improves hubert with a oncefor-all transformer model. the teacher is a hu-bert base model, while the student learns by predicting masked inputs in an iterative process. the transformer in lighthubert comprises subnets with sharable weights and several configuration parameters, enabling a random search to adjust the model to different resource constraints. the student architecture in knowledge distillation methods is manually designed, and it does not change during training. however, modifying student architectures can have a considerable impact on model results, even for student architectures with similar sizes @xcite . therefore, a joined distillation and pruning approach for speech ssl has been recently proposed, using hu-bert (dphubert) or wavlm (dpwavlm) as the teacher models @xcite . yet, knowledge distillation approaches need a pretrained teacher model because student models can not be trained standalone @xcite . thus, computational costs do not improve as they should include teacher model training. in contrast, @xcite proposes a simplified version of hubert that has twelve self-attention layers and a weighted sum of all the layers for downstream tasks. the input is a 40-dimensional mel log spectrogram, so input sequences are shorter, reducing the multiplication and addition calculations by @xcite . there are also efforts to improve the wav2vec architecture. proposed approaches improving wav2vec include squeezed and efficient wav2vec2 with disentangled attention (sew-d) @xcite and stochastic squeezed and efficient wav2vec2 (s-sew) @xcite . despite existing efforts to improve selfsupervised model efficiency, there is still room to reduce the computational costs of self-supervised models. computational costs create challenges when using these models in mobile devices and for training on very large datasets @xcite . they also hinder the development of new approaches, the study of other training recipes, and the reproduction of experimental results, as few researchers can afford the cost @xcite @xcite . besides, computational costs have environmental implications, as training requires considerable amounts of energy @xcite . likewise, few existing self-supervised models use half-precision numbers, even though this technique can half the memory requirements and accelerate the arithmetic computations on recent gpus @xcite . a similar issue happens with dynamic batching @xcite , a procedure that avoids wasting computing resources on the padded portion of speech mini-batches. also, most models use standard self-attention layers, though efficient alternatives have been proposed recently, without using approximations @xcite . we address these limitations in the following section, proposing an efficient model for selfsupervised learning of speech representations. ## efficient self-supervised approach in this section, we describe our proposed model: efficient self-supervised learning (essl). we also summarize the optimizations used to improve model efficiency. ## results and discussion all experiments run on a single gpu, an nvidia geforce rtx 3090 ti, with 24 gb of memory and 1.56ghz of base clock rate. considering training data, librispeech 960h provides speech utterances for unsupervised pretraining. finetuning for automatic speech recognition (asr) is performed with librispeech 100h @xcite , using a ctc loss @xcite . regarding training configuration, pretraining requires 60k iterations, which is equivalent to 15k pretraining steps because we do four gradient accumulations. the learning rate warms up the first 8% of the iterations to a maximum of 3e-4. for finetuning, 160k iterations are performed. this is equivalent to 40k finetuning steps with four gradient accumulations. the learning rate warms up the first 10% of the iterations to a maximum of 3e-5 @xcite . code is publicly available 1 to facilitate the replication of experimental results. efficiency gains of essl are remarkable (figure 2 ). though metrics degrade against large speech models ( ## limitations very-low data settings are challenging. the limited availability of data hinders research in speech processing for under-resourced languages @xcite . we tested finetuning essl for asr with the librilight dataset @xcite . librilight has 10 hours, 1 hour, and 10 minutes datasets to finetune models, in contrast with the 100 hours available in librispeech. results indicate essl struggles in very-low data settings, with a wer of 97.30% in librispeech dev-other ( ## conclusion in this work, we proposed essl, an efficient approach for self-supervised learning of speech representations. essl addresses high computational costs by combining several model optimizations and fixing a limit on computational resources available for pretraining. estimations of computational cost reduction reveal up to two orders of magnitude improvements against existing speech ssl models. overall, essl is a step in the process of reducing computational costs in ssl models, enabling their training in edge devices, facilitating the development of new approaches, and making them more environmentally friendly. for future work, we will investigate our efficient approach for other speech processing tasks, including intent classification, keyword spotting, query by example, and other downstream tasks. we will also explore architectural modifications to improve model performance in very-low data settings.