MTL - SLT : Multi-Task Learning for Spoken Language Tasks (2022)

Abstract: Language understanding in speech-based systems has attracted extensive interest from both academic and industrial communities in recent years with the growing demand for voice-based applications. Prior works focus on independent research by the automatic speech recognition (ASR) and natural language processing (NLP) communities, or on jointly modeling the speech and NLP problems focusing on a single dataset or single NLP task. To facilitate the development of spoken language research, we introduce MTL-SLT, a multi-task learning framework for spoken language tasks. MTL-SLT takes speech as input, and outputs transcription, intent, named entities, summaries, and answers to text queries, supporting the tasks of spoken language understanding, spoken summarization and spoken question answering respectively. The proposed framework benefits from three key aspects: 1) pre-trained sub-networks of ASR model and language model; 2) multi-task learning objective to exploit shared knowledge from different tasks; 3) end-to-end training of ASR and downstream NLP task based on sequence loss. We obtain state-of-the-art results on spoken language understanding tasks such as SLURP and ATIS. Spoken summarization results are reported on a new dataset: Spoken-Gigaword.

## introduction the wide deployment of voice controlled computing has led to extensive interest in spoken language tasks in recent years @xcite @xcite . for instance, spoken language understanding aims to extract the semantics from user queries @xcite @xcite , spoken question answering aims to predict the answer given the spoken context @xcite . the rapid development of spoken language tasks have followed dataset releases @xcite and the evolution of pre-trained [house_place : bedroom], ans1. nine thirty pm ans2. tomorrow multi-task learning (mtl) @xcite focuses on simultaneously solving multiple related tasks and has attracted much attention in recent years. compared with single-task learning, it can reduce the training and inference time while improving generalization performance and prediction accuracy by learning a shared representation across related tasks. prior works show the effectiveness of mtl while they only focus on multiple text-based tasks/datasets (e.g., mt-dnn @xcite ) or multiple speechbased tasks/datasets (e.g., speechstew @xcite ). also, some works @xcite prove the effectiveness of considering speech information when performing nlp tasks. thus, as can be seen in figure 1 , we argue that it is helpful when extend these mtl approaches to spoken language tasks (i.e., asr-nlp-shared). in this paper, we develop multi-task learning methods to optimize spoken summarization, spoken question answering, spoken language understanding (intent classification and slot filling), as well as speech recognition on multiple spoken language datasets. an example of an application with these four tasks can be seen in a primary challenge with audio as an input modality is the impact of speech recognition errors and acoustic noise on spoken language tasks. to mitigate this, our approach jointly optimizes pretrained speech recognition and language models for semantic metrics of interest and we train across multiple language tasks. the various language tasks and the impact of multi-task training can be visualized in the clustering plot of the hidden state of a pretrained language model in figure 2 . we demonstrate our results using listen-attend-spell (las) @xcite speech recognition model and a bart @xcite based nlp model. overall, the main contributions are as follows: pre-trained models the paradigm of pre-training a language model (lm) followed by task-specific fine-tuning has been shown to obtain remarkable performance on many nlp tasks. bert @xcite pre-trains deep bidirectional representations from unlabeled text and showed competitive performance on the glue @xcite benchmark. this provided a base for researchers to build upon, leading to several extensions and rapid progress in the space of pre-trained lms. the multitask deep neural network @xcite is one such extension with multi-task learning across all glue tasks. the paper argues for improved domain transfer by performing standard bert pretraining, followed by multi-task learning and task-specific fine-tuning. bert has been leveraged for various nlp tasks, for e.g. the effectiveness of bert for the summarization task was explored by @xcite . the performance of text generation tasks have been approaching a near-human level by virtue of pre-trained encoderdecoder models, such as bart @xcite and t5 @xcite . spoken language tasks spoken language tasks include standard nlp tasks with speech-input instead of text-input. speech recognition errors can impact the performance of downstream nlp systems. recently, @xcite proposed the asr-glue benchmark, augmented 6 nlp tasks from glue with speech generated from google tts, and analyzed the robustness of nlp to asr errors. however, all 6 tasks are sentence-level classification problems, and the models did not utilize mtl framework. @xcite introduced a speech-language joint pre-training framework for slu tasks. the paper showed the effectiveness of the joint pre-training method with experiments on four classification tasks, i.e., intent detection, dialog act classification, spoken sentiment analysis and spoken question answering. prior works for slu show the impact of speech recognition errors on downstream natural language understanding (nlu) performance and propose joint training of asr and nlu to improve overall performance @xcite . @xcite introduced a speech-based benchmark for task-oriented dialogue systems, specifically targeting the problems of multi-domain dialogue state tracking and knowledge grounded dialogue modeling, and showed that well-behaved models trained on written conversations do not perform well on spoken data. asr model unlike previous works on spoken language tasks (slt) that obtain transcriptions using existing asr systems/tools @xcite , in our approach, the asr model is updated with the training of end-to-end spoken language tasks. to address this, we generate the asr transcriptions from a las model explained in @xcite , and pre-trained it on the librispeech dataset @xcite following previous works @xcite . enc-decoder nlp model bidirectional and auto-regressive transformers (bart) @xcite uses a separate bidirectional encoder and autoregressive decoder similar to bert @xcite except that (1) bart's decoder incorporates cross attention over the final encoder layer and (2) bart's encoder does not use a feedforward dense layer for word prediction. the bart model can be used to perform both language understanding (i.e., intent classification) and language generation (i.e., summarization) problems at the same time, we refer to it as an nlp model in this work. we use the same pre-trained bartbase model as the original paper, which includes 6 transformer layers in the encoder and decoder. ## conclusion we proposed a multi-task learning framework for spoken language understanding tasks that take speech as input and produces (1) intents and namedentities in language understanding tasks, (2) abstract text summaries, or (3) question answering. this framework can be extended to other language tasks such as translation. in this framework, we make use of pretrained asr models and language models like bart and jointly train these layers across multiple language tasks. we demonstrate that this training across tasks coupled with task-specific post-finetuning produces significantly better results for asr and bart separately. we made use of the sequence loss training framework to enable end-to-end training of asr and bart to optimize for metrics of interest for the classification, sequence tagging, and generation tasks. this made the downstream language task robust to errors in asr hypotheses that otherwise leads to performance degradation in pipelined asr and language task systems. we demonstrate state-of-the-art results on public corpora of slurp and atis for spoken language understanding. we also prepare the spoken-gigaword dataset for abstractive summarization of speech.