U niver SLU : Universal Spoken Language Understanding for Diverse Tasks with Natural Language Instructions (2024)

Abstract: Recent studies leverage large language models with multi-tasking capabilities, using natural language prompts to guide the model’s behavior and surpassing performance of task-specific models. Motivated by this, we ask: can we build a single model that jointly performs various spoken language understanding (SLU) tasks? We start by adapting a pre-trained automatic speech recognition model to additional tasks using single-token task specifiers. We enhance this approach through instruction tuning, i.e., finetuning by describing the task using natural language instructions followed by the list of label options. Our approach can generalize to new task descriptions for the seen tasks during inference, thereby enhancing its user-friendliness. We demonstrate the efficacy of our single multi-task learning model “UniverSLU” for 12 speech classification and sequence generation task types spanning 17 datasets and 9 languages. On most tasks, UniverSLU achieves competitive performance and often even surpasses task-specific models. Additionally, we assess the zero-shot capabilities, finding that the model generalizes to new datasets and languages for seen task types.

## introduction spoken language understanding (slu) refers to the task of inferring semantic meaning or linguistic structure from spoken utterances. like prior work @xcite , we use the term broadly to refer to understanding content @xcite , speaker characterstics @xcite , paralinguistics @xcite and semantics @xcite directly from speech. 1 in this work, we define a task as a specific combination of task type and dataset. a task type refers to the type of slu operation being carried out, like intent classification (ic) or emotion recognition (er). for instance, performing the same task type (ic) on the snips @xcite and fsc @xcite datasets are different tasks. slu tasks are traditionally approached with "task-specific" models @xcite , i.e. having a separate model for each task. on the other hand, the use of multi-task learning (mtl) approaches in natural language processing (nlp) @xcite and speech processing @xcite has garnered significant interest. such approaches offer cost-efficiency by eliminating the need for separate models for each task. mtl has also been employed for slu tasks @xcite , primarily aiming to improve the performance of tasks with limited labeled data using an auxiliary automatic speech recognition (asr) objective. the rise of large language models (llms) @xcite has sparked interest in mtl, where the selection of appropriate "prompts" @xcite enables manipulation of the model's behavior to perform specific tasks without additional task-specific parameters. llms fine-tuned with natural language instructions @xcite describing the task show strong zero-shot performance. building upon llm's success in solving various nlp tasks, there is growing interest in leveraging the "pre-train, prompt, and predict" @xcite paradigm. prompt-based paradigms have also been explored for speech processing tasks @xcite . speechprompt @xcite employs trainable task-specific prompt vectors with generative spoken language models (gslms) @xcite . however, this model still lags behind task-specific baselines in most speech classification tasks and par2) or natural language phrases (i r in eq. 4) to describe the task, as shown in figures 1a and 1b , respectively. colored boxes denote a sequence of tokens, while white boxes denote the functionality enabled by the sequence of tokens. sop, sot, nt, and trans are standard whisper tokens that specify start-of-prev, start-of-transcript, no-timestamps, and transcribe, respectively. recently, there has been work on instruction tuning for speech processing tasks @xcite . however, these approaches mainly concentrate on asr and related tasks and are often not able to perform diverse slu tasks. motivated by the potential of prompt-based mtl, there has been interest in building "universal" asr models @xcite that can perform asr across many languages. we extend this work by building a "universal" slu model that can match or even outperform stateof-the-art (sota) task-specific slu models. we leverage the whisper @xcite model, which has been pre-trained on labeled data for asr and speech translation (st) tasks using only 2 task specifiers, namely ⟨|transcribe|⟩ and ⟨|translate|⟩ respectively. we first generalize the whisper model to a wide range of slu tasks by utilizing single token task specifiers, as shown in fig. 1a , drawing inspiration from whisper's pre-training formulation. we show that unlike the most comparable prior multi-task approach, speechprompt v2 @xcite , our mtl model can outperform or achieve competitive performance to taskspecific baselines across many classification and sequence generation tasks. however, the use of single-token task specifiers not only constrains our model's ability to handle variations in input format, affecting its user-friendliness, but also prevents it from performing unseen tasks in a zero-shot manner. hence, we experiment with utilizing natural language instruction @xcite , such as "classify speechbased commands" that describe each task as a prompt (fig. 1b ). for classification tasks, we include a list of options within the instruction. our approach generalizes to new paraphrases describing the same task, enhancing the model's userfriendliness. we assess our model's ability to handle unseen tasks, covering new datasets and languages for the known task types, as well as fully unseen task types. this ability is achieved by conducting inference using a task type description accompanied by the list of options specific to the given dataset as the instruction. our model surpasses random and majority baselines in zero-shot inference on unseen datasets and languages, although it does not match supervised performance. our experiments encompass 10 speech classification and 2 sequence generation task types, covering 17 publicly available datasets based on 9 languages. our proposed approach using natural language instruction advances the sota on 10 classification tasks and achieves competitive performance to taskspecific baselines on sequence generation tasks. we will make our source code and models publicly available as part of the espnet-slu toolkit foot_1 @xcite for reproducibility. the key contributions of our work are: ## related work multi-task learning (mtl) @xcite concurrently trains a model to perform multiple tasks, allowing the model to learn generalizable features instead of task-specific ones. mtl has been extensively explored in various nlp settings @xcite @xcite , ranging from similar task types @xcite to the same task type across multiple domains and languages @xcite . mtl can surpass task-specific models in certain scenarios, especially for tasks with limited labeled data @xcite . similar investigations have been conducted in speech processing, such as adding auxiliary tasks in asr @xcite @xcite and for other tasks @xcite @xcite , primarily using an auxiliary asr objective. however, these approaches covered very few slu benchmarks, and we extend this line of research by mtl across 17 slu datasets. llms for speech processing: recently, there has been significant interest in enhancing llms with speech processing and understanding capabilities @xcite @xcite . these approaches typically involve multi-tasking on a wide range of speech processing tasks, demonstrating robust performance across multiple benchmarks. lauragpt @xcite ) is one such approach that uses whisper-style task specifiers with qwen @xcite to handle various speech processing and generation tasks. some works have explored preserving llm capabilities, such as understanding "instructions" to perform tasks in a zero-shot manner @xcite @xcite . ltu-as @xcite ) concatenates text instruction with whisper output and feeds them into vicuna llm @xcite to perform a range of speech and audio tasks. salmonn @xcite employs both speech and audio representation into vicuna llm for "instruction tuning" across various tasks. however, while these efforts primarily concentrate on asr @xcite @xcite and st @xcite , our approach focuses on a diverse array of slu tasks (see sec. 6.2 and a.4). furthermore, our novel approach, involving the addition of the list of options in the instruction, not only improves performance but also enhances model generalization to new datasets and languages for the trained task types. our results in sec. 6.2 demonstrate our ability to outperform llm-based approaches on the subset of slu tasks they are trained on. moreover, we distinguish ourselves as the first to exhibit instruction tuning capabilities with a decoder of speech foundation model to the best of our knowledge, that has the important advantage of using much fewer parameters and lower computation. ## problem formulation our objective is to build a prompt-based mtl framework capable of performing various speech classification and sequence generation tasks. our model takes a sequence of speech features, denoted x, as input for all downstream tasks. for a specific downstream task r, we aim to predict the corresponding o r -length label sequence y @xmath0, . . . , o r }, where l r represents the label set for that particular task. prior studies like speechprompt @xcite have modelled the prompting formulation using gslms. they prepend the embedding task type dataset language generated prompt (s task_type ) (s data ) (s lang ) task specifier (s r ) natural language instruction (i r ) speech command google sc v1 en sot ⟨en⟩ ⟨scr⟩ ⟨google_scr⟩ nt sop classify speech-based commands. the options are 0."down", . . . sot ⟨en⟩ trans nt recognition lithuanian sc lt sot ⟨lt⟩ ⟨scr⟩ ⟨lt_scr⟩ nt sop classify speech-based commands. the options are 0."iki", . . . sot ⟨lt⟩ trans nt fake speech detection asvspoof en sot ⟨en⟩ ⟨fsd⟩ ⟨asvspoof⟩ nt sop tell apart speech synthesis and voice conversion from authentic speech. the options are 0."bonafide", 1."spoof". sot ⟨en⟩ trans nt accent classification accentdb en sot ⟨en⟩ ⟨accent_rec⟩ ⟨accentdb⟩ nt sop accent identification in speech. the options are 0."telugu", . . . sot ⟨en⟩ trans nt audio classification esc-50 ✗ sot ⟨audio⟩ ⟨auc⟩ ⟨esc50⟩ nt sop specify the type of environmental noise in the audio. the options are 0."audio_class:17", . . . sot ⟨audio⟩ trans nt in our approach, instead of relying on a pretrained model solely for generating discrete tokens for gslm, we fine-tune an entire pre-trained speech foundation model. furthermore, we use discrete prompts corresponding to a sequence of tokens. starting from a natural extension of the pretraining formulation in speech foundation models like whisper @xcite , we first investigate using single token "task specifiers" as prompts. during pre-training, whisper utilizes only two task specifiers, namely ⟨transcribe⟩ and ⟨translate⟩, to facilitate mtl across asr and st tasks, respectively. we experiment with the use of a single token as specifier s task_type for each task type, s lang for each language, and s data for each dataset as shown in fig. 1a to represent s r (eq. 1): we then propose to incorporate humaninterpretable natural phrases to enhance the attractiveness of our model for human interaction, as shown in the example of fig. 1b . we use a proba-bilistic formulation to provide a new training objective, which handles various natural phrase prompts. let i r be the natural language instruction describing downstream task r. we can model the posterior distribution in eq. 1 as: the samples of natural language task descriptions i r are generated by an llm that models the distribution p (i r |s r ), as discussed in sec. 4.2. by using this training scheme, the model can accept various natural phrase prompts provided by the user as a way of describing s r (eq. 2). unlike prior studies @xcite , our approach allows for potential generalization to various paraphrases i r that specify the same task, thereby enhancing the user-friendliness and adaptability of our model for human interaction. ## prompt-based multi-tasking to achieve the formulation described in eq. 4, this work proposes a prompt-based mtl training format, as illustrated in fig. 1b . the input speech x is passed through an encoder-decoder architecture. we prepend instruction i r to the preceding tokens generated by the decoder (shown as "instruction tuning" in fig. 1b and described in further detail in sec. 4.2). the likelihood of the entire label sequence i.e. p (y r |x, i r ) in eq. 4, can be computed using task-specific decoder representations h r o as: ## conclusion we introduce an "instruction tuning" approach that utilizes human-interpretable natural language instructions, combined with a list of option labels as prompts, to fine-tune a speech foundation model on slu benchmarks. our experiments demonstrate the model's ability to understand diverse expressions as instructions during inference. leveraging this approach, we train a single mtl model named universlu, capable of performing 12 slu task types, encompassing both classification and sequence generation tasks. universlu not only provides cost efficiency but also achieves competitive performance compared to task-specific baselines, outperforming sota models on several slu benchmarks. additionally, we find that the model has some generalization ability on new datasets and languages for seen task types. while our current approach faces challenges in generalizing to new task types, it demonstrates strong generalization to new paraphrases describing known tasks (sec. 6.1) and achieves non-trivial performance on new datasets and languages with entirely different label sets in a zero-shot scenario. thus, we believe our approach possesses some instruction-following capabilities. prior work, exemplified by instruction tuning models like flan @xcite , has shown that generalization improves with an increase in the size of the instruction tuning dataset. we plan to explore this further in future work, along with integrating an llm-based decoder and implementing few-shot inference techniques.