Multi-task Learning of Spoken Language Understanding by Integrating N-Best Hypotheses with Hierarchical Attention (2020)

Abstract: Currently, in spoken language understanding (SLU) systems, the automatic speech recognition (ASR) module produces multiple interpretations (or hypotheses) for the input audio signal and the natural language understanding (NLU) module takes the one with the highest confidence score for domain or intent classification. However, the interpretations can be noisy, and solely relying on one interpretation can cause information loss. To address the problem, many research works attempt to rerank the interpretations for a better choice while some recent works get better performance by integrating all the hypotheses during prediction. In this paper, we follow the way of integrating hypotheses but strengthen the training mode by involving more tasks, some of which may be not in existing tasks of NLU but relevant, via multi-task learning or transfer learning. Moreover, we propose the Hierarchical Attention Mechanism (HAM) to further improve the performance with the acoustic-model features like confidence scores, which are ignored in the current hypotheses integration models. The experimental results show that compared to the standard estimation with one hypothesis, the multi-task learning with HAM can improve the domain and intent classification by relatively 19% and 37%, which are much higher than improvements with current integration or reranking methods. To illustrate the cause of improvements brought by our model, we decode the hidden representations of some utterance examples and compare the generated texts with hypotheses and transcripts. The comparison shows that our model could recover the transcription by integrating the fragmented information among hypotheses and identifying the frequent error patterns of the ASR module, and even rewrite the query for a better understanding, which reveals the characteristic of multi-task learning of broadcasting knowledge.

## introduction in an slu system @xcite , the domains and intents are usually inferred by natural language understanding (nlu) modules with the hypotheses mapped from input speech by asr module. for each speech audio, the transferred hypothesis is the one with the highest recognition score. however, due to the unsatisfactory asr accuracy @xcite , the 1-best hypothesis may contain errors. to solve the problem, there are some research works rescoring (reranking) the -best hypotheses to reduce the word error rate (wer) by dual comparison with a discriminative language model @xcite ; or involving morphological, lexical, syntactic or confidence score features for reranking @xcite @xcite @xcite . in contrast to the reranking models, which predict only one hypothesis with the lowest wer and transfer that hypothesis to nlu modules, there is recently another attempt to integrate the fragmented information among the -best hypotheses by feeding all the hypotheses together to nlu modules @xcite . the proposed approaches to integrating hypotheses include hypothesized text concatenation (combined sentence) and hypotheses embedding concatenation @xcite . compared to the accuracy on the oracle reranking results (i.e., picking the hypothesis most similar to transcription), the poolingavg achieves much higher improvements for the nlu tasks. however, the integration framework can be further improved by introducing more tasks with multi-task learning (mtl) or transfer learning and involving more features to optimize the integration process. mtl @xcite @xcite is a widely used machine learning paradigm for simultaneously training related tasks. in the mtl training, one task can apply the knowledge learned from others. mtl can improve the generalization of the trained model by avoiding overfitting to a single task and make full use of all the labeled data from all tasks to solve the issue of insufficient training data. the mtl has been shown efficient for some natural language processing tasks outside the slu system like text similarity, pairwise text classification @xcite . in contrast to multi-task learning, by transfer learning or domain adaption @xcite @xcite , some tasks (source tasks) can be trained in the first stage knowing nothing about the other tasks (target tasks). while in the second stage, the embeddings from pre-trained model are fine-tuned according to the target mission. the transfer learning cares more about the target tasks. some popular fine-tunable pre-trained models like bert @xcite , elmo @xcite nail down the transfer learning in nlp. the rest of the paper is organized as follows. sec. 2 presents various models and training paradigms explored in this work. sec. 3 describes our experimental details, results and analysis. sec. 4 concludes our findings and discusses the future directions. ## models we start by reviewing different categories of slu system designs in sec. 2.1. those designs have achieved great success, but they are trained solely on one task and cannot borrow the knowledge from some relevant tasks like transcription reconstruction. to involve more tasks during training, we explore two paradigms to: 1) train them simultaneously in a single stage (sec. 2.2), which is actually multi-task learning; 2) train them asynchronously (sec. 2.3) in multiple stages, which includes two ways of using the pre-trained model from the first stage (transfer learning or text generation). in sec. 2.4, we illustrate the importance of acoustic-model features and the way of utilizing them hierarchically. we firstly review the current designs for the slu system in figure 1 , which include the standard pipeline (baseline), reranking and integration models. in production, the input audio is transcribed by asr to get -best hypotheses. then, the baseline model will take the one with the highest confidence score for nlu tasks. nevertheless, the reranking models do not solely rely on the confidence scores generated by the asr module. they prefer to rescore the interpretations based on more features like semantic information and choose the most reliable one. both baseline and reranking models transfer one sentence to the nlu module for classification. however, some recent works like @xcite indicate this causes information loss and attempt to use all the hypotheses during classification. they embed each hypothesis to one vector and unify the vectors to one by a pooling layer, which becomes the input to the nlu task. ideally, we can make the hypothesis close to the transcribed sentence by humans. to know the ceiling point of performance, there is always the oracle model predicting with human transcriptions.