SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Tasks (2023)

Abstract: Spoken language understanding (SLU) tasks have been studied for many decades in the speech research community, but have not received as much attention as lower-level tasks like speech and speaker recognition. In this work, we introduce several new annotated SLU benchmark tasks based on freely available speech data, which complement existing benchmarks and address gaps in the SLU evaluation landscape. We contribute four tasks: question answering and summarization involve inference over longer speech sequences; named entity localization addresses the speech-specific task of locating the targeted content in the signal; dialog act classification identifies the function of a given speech utterance. In order to facilitate the development of SLU models that leverage the success of pre-trained speech representations, we will release a new benchmark suite, including for each task (i) curated annotations for a relatively small fine-tuning set, (ii) reproducible pipeline (speech recognizer + text model) and end-to-end baseline models and evaluation metrics, (iii) baseline model performance in various types of systems for easy comparisons. We present the details of data collection and annotation and the performance of the baseline models. We also analyze the sensitivity of pipeline models’ performance to the speech recognition accuracy, using more than 20 publicly availablespeech recognition models.

## introduction spoken language understanding (slu) tasks involve inferring the linguistic structure or semantic meaning of a speech signal beyond its text transcript. we use this term broadly to include any natural language processing (nlp) task applied to speech, and tasks that involve linguistic understanding but also localization in the signal of relevant segments or producing speech as output. slu * core contributors in alphabetical order has been an active area throughout the history of speech research @xcite @xcite @xcite @xcite . however, compared to "lower-level" tasks like automatic speech recognition (asr) and speaker identification, slu has received much less attention and resources, and specifically there are much fewer benchmarks with freely available data. slu tasks can in principle be addressed via a pipeline approach -using asr to map speech to text and an nlp (text) model to map text to the desired output. the alternative is an end-toend (e2e) model, which maps directly from the input speech to the target output. while pipeline approaches can take advantage of existing strong asr and nlp models, e2e models can be more efficient at inference time, can avoid asr error propagation, and can directly use aspects of the speech signal beyond the text that are useful for the task (e.g., prosody) @xcite @xcite . in addition, for tasks whose output includes speech segments or time spans, there is no direct combination of an asr model and an nlp model that produces precisely the desired type of output. for some slu tasks, the current state of the art is a pipeline model @xcite , whereas for others e2e models are better @xcite @xcite @xcite . in order to better understand the pros and cons of pipeline and e2e approaches, more public benchmarks are sorely needed. while collecting large amounts of labeled speech data for many slu tasks may be prohibitively costly, recent advances in pre-trained models @xcite @xcite @xcite @xcite make it feasible to use relatively small fine-tuning sets for each task. there have been several recent efforts to introduce new benchmark slu tasks @xcite @xcite @xcite @xcite , most (but not all) using fairly small training sets of several hours to several dozens of hours of speech. among them, the spoken language understanding evaluation (slue) 1 @xcite ) motivated us since it pursues a natural speech, rather than a short command type of speech that is populated in other benchmarks. however, there are only two slue tasks (sentiment analysis and named entity recognition), thus more tasks with different complexities are needed to cover the diverse application of slu. we introduce slue phase-2, a set of slu tasks that complement the existing slu datasets or benchmarks. the new tasks include dialog act classification (dac), question answering (qa), summarization (summ), and named entity localization (nel) , applied to english speech data. slue phase-2 has several advantages compared to other recent work introduced in section 2: more diverse tasks: slue phase-2 not only include utterance or word-level classification task but also includes qa and summ task. more challenging tasks: the complexity of the task is influenced by the type of input and the type of output. slue phase-2 uses conversational or longer discourse speech as input. the type of output is not limited to labels or text, but also includes the speech span time stamp. new human annotation: a new annotation was collected by a human annotator. human annotator validated an automatically-collected data if needed. natural speech: we do not use synthesized speech. we only include conversational or considerably long discourse speech rather than short speech commands. cc license: creative common licensed dataset to give the best freedom of use. for each task, we provide publicly available foot_1 datasets, annotations, models, and code. we provide both pipeline and e2e baseline models and, for pipeline models, we use multiple asr systems to analyze the effect of the asr error rate on the final task performance. speech summarization has gained interest over the past few years with tasks such as abstractive summarization of instructional how-2 videos @xcite and ted talks @xcite , but the raw audio for these tasks is not publicly available. other corpora, such as the icsi @xcite and @xcite meeting summarization corpora, contain rel-atively less annotated data. named entity localization (nel) is a fairly new task. a similar task, audio de-identification (audio de-id), has been introduced with annotations for conversational data from switchboard and fisher @xcite , but these datasets are not free. audio de-id is a special case of nel where the entities of interest are related to personal identifiers. we focus on english speech-related work (most comparable with our work), but there are also ongoing efforts for other languages @xcite . ## slue phase-2: tasks and data this section introduces the tasks and metrics in slue phase-2. the slue phase-1 introduced the "slue score", a numerical summary of model performance across tasks. however, as we consider a more diverse set of tasks, using the same pretrained model for all tasks is difficult, and evaluation via a single slue score may discourage building systems for individual tasks. in slue phase-2, therefore, we do not adopt the single slue score, and evaluate each task individually. ## discussion among the baseline models, our pipeline models generally outperform their end-to-end counterparts. however, as shown in prior work (e.g., @xcite ), end-to-end models often have more room for improvement with careful and creative modeling ideas, and we hope that this new testbed helps spur such research. in addition, the wer sensitivity analysis in figure 1 suggests different strategies are needed for the pipeline system depending on the slu task. for example, fine-tuned asr (pipeline-w2v2) plays a significant role in the dac task while the qa task is not, and asr model architecture is critical for the nel task while wer is more matter for dac and summ tasks.