N-Best ASR Transformer: Enhancing SLU Performance using Multiple ASR Hypotheses (2021)

Abstract: Spoken Language Understanding (SLU) systems parse speech into semantic structures like dialog acts and slots. This involves the use of an Automatic Speech Recognizer (ASR) to transcribe speech into multiple text alternatives (hypotheses). Transcription errors, ordinary in ASRs, impact downstream SLU performance negatively. Common approaches to mitigate such errors involve using richer information from the ASR, either in form of N-best hypotheses or word-lattices. We hypothesize that transformer models will learn better with a simpler utterance representation using the concatenation of the N-best ASR alternatives, where each alternative is separated by a special delimiter [SEP]. In our work, we test our hypothesis by using the concatenated N-best ASR alternatives as the input to the transformer encoder models, namely BERT and XLM-RoBERTa, and achieve equivalent performance to the prior state-of-the-art model on DSTC2 dataset. We also show that our approach significantly outperforms the prior state-of-the-art when subjected to the low data regime. Additionally, this methodology is accessible to users of third-party ASR APIs which do not provide word-lattice information.

## introduction spoken language understanding (slu) systems are an integral part of spoken dialog systems. they parse spoken utterances into corresponding semantic structures e.g. dialog acts. for this, a spoken utterance is usually first transcribed into text via an automated speech recognition (asr) module. often these asr transcriptions are noisy and erroneous. this can heavily impact the performance of downstream tasks performed by the slu systems. to counter the effects of asr errors, slu systems can utilise additional feature inputs from asr. a common approach is to use n-best hypotheses where multiple ranked asr hypotheses are used, instead of only 1 asr hypothesis. a few asr systems also provide additional information like wordlattices and word confusion networks. word-lattice information represents alternative word-sequences that are likely for a particular utterance, while word confusion networks are an alternative topology for representing a lattice where the lattice has been transformed into a linear graph. additionally, dialog context can help in resolving ambiguities in parses and reducing impact of asr noise. n-best hypotheses: @xcite work with 1-best asr hypothesis and exploits unsupervised asr error adaption method to map asr hypotheses and transcripts to a similar feature space. on the other hand, @xcite uses multiple asr hypotheses to predict multiple semantic frames per asr choice and determine the true spoken dialog system's output using additional context. wordlattices: @xcite propose using recurrent neural networks (rnns) to process weighted lattices as input to slu. @xcite presents a method for converting word-based asr lattices into word-semantic (w-se) which reduces the sparsity of the training data. @xcite provides an approach for adapting lattices with pretrained transformers. word confusion networks (wcn): jagfeld and vu (2017) proposes a technique to exploit word confusion networks (wcns) as training or testing units for slot filling. @xcite models wcn as sequence of bag-of-weighted-arcs and introduce a mechanism that converts the bag-of-weighted-arcs into a continuous representation to build a neural network based spoken utterance classification. @xcite proposes a bert based slu model to encode wcns and the dialog context jointly to reduce ambiguity from asr errors and improve slu performance with pre-trained models. the motivation of this paper is to improve performance on downstream slu tasks by exploiting transfer learning capabilities of the pre-trained transformer models. richer information representations like word-lattices @xcite ) and word confusion networks @xcite ) have been used with gpt and bert respectively. these representations are non-native to transformer models, that are pre-trained on plain text sequences. we hypothesize that transformer models will learn better with a simpler utterance representation using concatenation of the n-best asr hypotheses, where each hypothesis is separated by a special delimiter @xcite . we test the effectiveness of our approach on a dialog state tracking dataset -dstc2 @xcite , which is a standard benchmark for slu. contributions: (i) our proposed approach, trained with a simple input representation, exceeds the competitive baselines in terms of accuracy and shows equivalent performance on the f1-score to the prior state-of-the-art model. (ii) we significantly outperform the prior state-of-the-art model in the low data regime. we attribute this to the effective transfer learning from the pre-trained transformer model. (iii) this approach is accessible to users of third party asr apis unlike the methods that use word-lattices and word confusion networks which need deeper access to the asr system. ## n-best asr transformer n-best asr transformer foot_0 works with a simple input representation achieved by concatenating the n-best asr hypotheses together with the dialog context (system utterance). pre-trained transformer models, specifically bert and xlmroberta, are used to encode the input representation. for output layer, we use a semantic tuple classifier (stc) to predict act-slot-value triplets. the following sub-sections describe our approach in detail. ## results in this section, we compare the performance of our approach with the baselines on the dstc2 dataset. to compare the transfer learning effectiveness of pre-trained transformers with n-best asr bert (our approach) and the previous state-of-the-art model wcn-bert stc, we perform comparative analysis in the low data regime. additionally, we perform an ablation study on n-best asr bert to see the impact of modeling dialog context (last system utterance) with the user utterances. ## conclusion in this work, building on a simple input representation, we propose n-best asr transformer, which outperforms all the competitive baselines on utterance-level accuracy for the dstc2 dataset. however, the highlight of our work is in achieving significantly higher performance in an extremely low data regime. this approach is accessible to users of third-party asr apis, unlike the methods that use word-lattices and word confusion networks. as future extensions to this work, we plan to :