SUPERB - SG : Enhanced Speech processing Universal PER formance Benchmark for Semantic and Generative Capabilities (2022)

Abstract: Transfer learning has proven to be crucial in advancing the state of speech and natural language processing research in recent years. In speech, a model pre-trained by self-supervised learning transfers remarkably well on multiple tasks. However, the lack of a consistent evaluation methodology is limiting towards a holistic understanding of the efficacy of such models. SUPERB was a step towards introducing a common benchmark to evaluate pre-trained models across various speech tasks. In this paper, we introduce SUPERB-SG, a new benchmark focusing on evaluating the semantic and generative capabilities of pre-trained models by increasing task diversity and difficulty over SUPERB. We use a lightweight methodology to test the robustness of representations learned by pre-trained models under shifts in data domain and quality across different types of tasks. It entails freezing pre-trained model parameters, only using simple task-specific trainable heads. The goal is to be inclusive of all researchers, and encourage efficient use of computational resources. We also show that the task diversity of SUPERB-SG coupled with limited task supervision is an effective recipe for evaluating the generalizability of model representation.

## introduction transfer learning is a paradigm in machine learning that has been very effective for natural language processing (nlp) @xcite @xcite @xcite @xcite @xcite , and speech processing @xcite @xcite @xcite @xcite @xcite . self-supervised learning (ssl) is the main driver of this paradigm, an effective and scalable way to learn high-level representation of language that transfers to a variety of tasks. ssl entails learning from the input or some perturbation of it without the need for labelled data. this has unlocked the usage of large amounts of cheaply available unlabelled data. it lends naturally to neural network models that have been shown to possess impressive scaling characteristics such that it is often enough to increase the model and data sizes to improve downstream performance @xcite @xcite @xcite . speech signal consists of acoustic, linguistic, prosodic, and speaker characteristics. ssl algo-rithms in speech must be evaluated in their ability to produce representations that are useful for tasks that demand understanding of linguistic, speaker, and prosodic elements of spoken language as well as high-level semantics. researchers have used auto-regressive, contrastive, discriminative and multi-task learning objectives to pre-train models, and have investigated their capabilities across tasks like phoneme recognition @xcite , automatic speech recognition (asr) @xcite @xcite @xcite , speaker verification @xcite , speaker identification @xcite , emotion recognition @xcite , speech translation @xcite , voice conversion @xcite , spoken language understanding @xcite , and text-tospeech @xcite . however, the methodologies in such studies vary in the use of datasets, fine-tuning strategies and task-specific model architectures. to bridge this gap, superb @xcite introduced a standardized benchmark of 10 speech tasks to compare 13 pre-trained models and a log mel-filterbank baseline. it studied the models' performance in tasks focusing on linguistic (phoneme recognition and automatic speech recognition, keyword spotting and query by example), shallow semantic (intent classification and slot filling), speaker (speaker identification, speaker verification and speaker diarization), and prosodic (emotion recognition) characteristics. in this paper, we introduce superb-sg, a benchmark with 5 new tasks, which are speech translation, out-of-domain asr, voice conversion, speech separation, and speech enhancement, with an emphasis on evaluating the semantic and generative capabilities of pre-trained models that require high-level representations to capture linguistic, semantic, and speaker characteristics. these tasks go beyond speech recognition by focusing on various other aspects that are essential to building intelligent speech interfaces. further, we show that while ssl models achieve close to state-of-the-art performance on many tasks, there isn't one model that outperforms all others, and that a simple log mel-filterbank can perform competitively on some tasks. we also demonstrate the robustness of our methodology with an ablation study over different task-specific model architectures and data sizes. the introduction of these new tasks of varying difficulty takes us closer to a more comprehensive unified standard speech benchmark. we hope that this will motivate the development of more powerful, generalizable, and reusable pre-trained models to democratize the advancement of speech research. to facilitate this, we released the codes foot_0 and integrated the tasks with the superb benchmark. ## related work as more powerful ssl models are proposed with promising performance on various tasks, researchers continually try to find extensive evaluation methods to assess model performance, and wish to holistically understand the capability of the learned representations in these models. superb @xcite is a framework to benchmark the ssl models on 10 speech tasks by learning task-specific prediction heads on top of the frozen shared ssl models. although the tasks in superb span across different domains, most of them are simple classification problems, or only require utilization of shallow semantics. in contrast, we focus on harder semantic and generative tasks. another recently proposed benchmark is the lebenchmark @xcite , investigating the performance of ssl models trained on french data with three semantic tasks. however, they only consider wav2vec 2.0 @xcite with different architectures as their upstream models (i.e., networks pre-trained with ssl). here, we evaluate a diverse set of ssl models, and offer a more comprehensive analysis. the zero resource speech benchmark 2021 @xcite introduces unsupervised speech processing tasks, particularly the spoken language modeling problem. they evaluate the ssl models via zero-shot probings at four linguistic levels. while their benchmark task is specific for certain domain, we use various tasks to evaluate different aspects of ssl models. the hear 2021 challenge foot_1 aims to develop general-purpose audio representation by focusing on audio tasks beyond speech that include sound event detection, speech commands and pitch & chroma classification. we specifically focus on various aspects of speech processing, thus providing a wide variety of spoken language tasks. ## experimental setup following superb, we fix upstream models parameters for all downstream tasks' training. we extract the frame-level representations for each hidden layer of the upstream models from raw waveform, and use a trainable task-specific weightedsum mechanism to summarize all layers' representations into a sequence of vectors. the summarized representations are then used as the downstream model's input. an overview of the training procedure is demonstrated in figure 1 . each experiment is done by one single run with the same seed. this procedure is consistent for all experiments, offering a fair and simple evaluation strategy for all upstream models. ## conclusion we introduce superb-sg, a set of 5 new tasks that include speech translation, out-of-domain asr, voice conversion, speech separation, and speech enhancement to evaluate the deep semantic and generative capabilities of ssl models. we evaluate 15 ssl models, and do a comprehensive analysis of the task correlations to demonstrate the reliability of our methodology. we test and confirm the robustness of superb-sg in terms of the downstream model architecture as well as the training data size. the latest introduction of the semantic and generative tasks increases the diversity and difficulty of superb, which can boost a more comprehensive understanding of the capability of various ssl models' representations, and help researchers discover the hidden properties of ssl techniques in development. we have open-sourced all the codes 1 and released a challenge foot_2 to encourage further research of ssl in speech. we welcome the community to participate and advance the research frontier together.