On the Evaluation of Speech Foundation Models for Spoken Language Understanding (2024)

Abstract: The Spoken Language Understanding Evaluation (SLUE) suite of benchmark tasks was recently introduced to address the need for openresources and benchmarking of complex spoken language understanding (SLU) tasks, including both classification and sequence generation tasks, on natural speech. The benchmark has demonstrated preliminary success in using pre-trained speech foundation models (SFM) for these SLU tasks. However, the community still lacks a fine-grained understanding of the comparative utility of different SFMs. Inspired by this, we ask: which SFMs offer the most benefits for these complex SLU tasks, and what is the most effective approach for incorporating these SFMs? To answer this, we perform an extensive evaluation of multiple supervised and self-supervised SFMs using several evaluation protocols: (i) frozen SFMs with a lightweight prediction head, (ii) frozen SFMs with a complex prediction head, and (iii) fine-tuned SFMs with a lightweight prediction head. Although the supervised SFMs are pre-trained on much more speech recognition data (with labels), they do not always outperform self-supervised SFMs; the latter tend to perform at least as well as, and sometimes better than, supervised SFMs, especially on the sequence generation tasks in SLUE. While there is no universally optimal way of incorporating SFMs, the complex prediction head gives the best performance for most tasks, although it increases the inference time. We also introduce an open-source toolkit and performance leaderboard, SLUE-PERB, for these tasks and modeling strategies.

## introduction spoken language understanding (slu) refers to tasks that require extracting semantics from spoken utterances. slu systems have important applications, for example, in voice assistants and conversational agents, and have attracted increasing interest in recent years @xcite . slu encompasses a wide range of tasks, such as predicting intents and slots @xcite @xcite , recognizing entity mentions and labels @xcite , detecting the speaker's sentiment @xcite and modeling the topic of a spoken dialogue @xcite . more recently, there has been significant interest in tackling more complex tasks like question answering @xcite or summarization @xcite . the spoken language understanding evaluation (slue) @xcite suite of benchmark tasks was recently proposed to address the lack of sufficiently complex and varied tasks on natural (rather than synthetic or read) speech from public datasets. slue uses annotated natural speech from conversations and monologues and includes both classification and sequence generation tasks. traditional slu models use a pipeline @xcite @xcite of an automatic speech recognition (asr) system followed by a natural language understanding (nlu) system. end-to-end (e2e) slu systems @xcite have also been explored to mitigate the impact of error propagation observed in pipeline approaches and take advantage of the information in the audio signal beyond the word content. a recent trend in e2e models has been the use of pre-trained speech foundation models (sfm) @xcite @xcite @xcite that can learn useful representations for a large number of tasks. due to the increasing diversity of models, benchmarks are important to compare the performance of sfms on multiple downstream tasks. performance benchmarks like superb (speech processing universal perfor-mance benchmark) @xcite have facilitated standardized comparison of pre-trained sfms across a diverse range of speech-processing tasks. however, such benchmarks lack coverage of challenging and realistic slu tasks. hence, the community lacks a fine-grained understanding of the relative merits of different sfms and different ways to use them for downstream slu tasks. motivated by these shortcomings, we introduce slue-perb (spoken language understanding evaluation performance benchmark), specifically designed to evaluate representations extracted from pre-trained sfms on complex slu tasks. we use this benchmark to answer two main questions: (i) which sfms are most useful for these tasks, and (ii) how do different ways of using these sfms, varying in their compute budget, compare. our study addresses various questions concerning slu systems, such as whether supervised sfms are more beneficial than self-supervised sfms, whether sfms are effective as frozen feature extractors or should be fine-tuned on downstream tasks, and whether the complexity of prediction heads affects the performance trends. we conduct a comprehensive analysis by examining three types of sfms: (i) self-supervised learning (ssl) speech models @xcite @xcite trained on unlabeled speech data; (ii) (weakly) supervised asr (and speech translation) models @xcite pre-trained on large labeled corpora; and (iii) supervised slu models pretrained on external slu corpora @xcite . our extensive experiments are performed on the slue benchmark @xcite , which provides curated data for sentiment analysis (sa), named entity recognition (ner), named entity localization (nel), dialogue act classification (dac), question answering (qa) and summarization (summ). the key contributions are: ## experiments we conduct our analysis by examining various sfms as introduced in tab. 2. training hyperparameters are selected based on validation performance. more details can be found in sec. a.3 in the appendix. all our models and config files will be publicly available upon acceptance of the paper. lightweight prediction head: for the sa task, we mean-pool the extracted features from the sfms across time, and then pass the pooled representation through a linear layer to compute the probability for each sentiment class. the lightweight classification layers are trained using cross-entropy loss. in the case of dac, we follow a similar procedure of mean-pooling followed by a linear layer. as this is a multi-label classification task, we use a sigmoid ## results in this section, we analyze the performance of various sfms on our performance leaderboard slue-perb, as detailed in sec. 3. this analysis provides insights into the types of sfms that prove most effective for complex understanding tasks and how this trend varies across tasks and modeling settings. figs. 1-7 summarize our results. in all figures, bars with sparse stripes correspond to the "lightweight prediction head" setting, dense striped bars correspond to "complex prediction head", and solid bars correspond to "fine-tuned representations". test and development set results for all experiments are shown in tabs. 3 and 5 (in the appendix) respectively. ## conclusion in this paper, we address the lack of performance benchmarks for evaluating pre-trained sfms on slu tasks. we introduce slue-perb to compare multiple pre-trained ssl and supervised sfms on complex slu tasks. our experiments demonstrate that supervised asr sfms like owsm produce the best performing representations for classification tasks, while ssl sfms like wavlm can outperform or perform comparably to supervised asr sfms on temporal alignment and sequence generation tasks. the trends generally remain similar across different evaluation settings, but the performance gap between different sfms decreases as we increase the size of the prediction head or finetune the sfms. we also find that while there is no universal best approach for incorporating sfms, a complex prediction head gives the best performance for most tasks, at the price of higher inference latency. in addition to providing guidance for researchers working on slu tasks, we believe that our findings will spark innovation in developing sfms for slu tasks. (i) ssl sfm representations either outperform or perform comparably to supervised sfm representations for sequence generation tasks. this suggests that supervised sfms, which employ an encoder-decoder architecture, may retain meaningful information within their decoder, which is not straightforward to use for feature extraction. hence, developing encoder-only supervised sfms @xcite ) could be a promising future research direction. (ii) sfms like whisper demonstrate notably poor performance on qa since whisper's pretraining is on 30-second speech segments, while the input audios for qa tasks are typically longer than 30 seconds. this suggests the need for sfms pre-trained on longer speech utterances @xcite . (iii) using a complex prediction head with a frozen sfm outperforms full fine-tuning on most tasks, which suggests the exploration of modeling strategies that can utilize sfms without significantly changing the pre-trained parameters, such as using parameter-efficient tuning approaches @xcite . by making all our code public, we aim to facilitate future research and development on slue tasks. in future work, we plan to extend slue-perb to include more data and models, including pipeline systems.