Using multiple ASR hypotheses to boost i18n NLU performance (2020)

Abstract: Current voice assistants typically use the best hypothesis yielded by their Automatic Speech Recognition (ASR) module as input to their Natural Language Understanding (NLU) module, thereby losing helpful information that might be stored in lower-ranked ASR hypotheses. We explore the change in performance of NLU associated tasks when utilizing five-best ASR hypotheses when compared to status quo for two language datasets, German and Portuguese. To harvest information from the ASR five-best, we leverage extractive summarization and joint extractive-abstractive summarization models for Domain Classification (DC) experiments while using a sequence-to-sequence model with a pointer generator network for Intent Classification (IC) and Named Entity Recognition (NER) multi-task experiments. For the DC full test set, we observe significant improvements of up to 7.2% and 15.5% in micro-averaged F1 scores, for German and Portuguese, respectively. In cases where the best ASR hypothesis was not an exact match to the transcribed utterance (mismatched test set), we see improvements of up to 6.7% and 8.8% micro-averaged F1 scores, for German and Portuguese, respectively. For IC and NER multi-task experiments, when evaluating on the mismatched test set, we see improvements across all domains in German and in 17 out of 19 domains in Portuguese (improvements based on change in SeMER scores). Our results suggest that the use of multiple ASR hypotheses, as opposed to one, can lead to significant performance improvements in the DC task for these non-English datasets. In addition, it could lead to significant improvement in the performance of IC and NER tasks in cases where the ASR model makes mistakes.

## introduction recent years have seen a dramatic increase in the adoption of intelligent voice assistants such as amazon alexa, apple siri and google assistant. as use cases expand, these assistants are expected to process ever more complex user utterances and perform many different tasks. some of the key components that enable the performance of these tasks are housed within the spoken language understanding (slu) system; one being the automatic speech recognition (asr) module which transcribes the users' vocal sound wave into text and another being the natural language understanding module which performs a variety of downstream tasks that help identify the actions requested by the user @xcite . these modules perform in tandem and are crucial for the successful processing of user utterances. typical asr models generate multiple hypotheses for an input audio signal, that are ranked by their confidence scores @xcite . however, only the top ranked hypothesis (referred to hereafter as the asr 1-best) is usually processed by the nlu module for downstream tasks @xcite . three major tasks performed by the nlu module are domain classification (dc), intent classification (ic) and named entity recognition (ner). dc predicts the domain relevant to the utterance (weather, shopping, music etc.) and ic extracts actions requested by users (some examples are, buy an item, play a song or set a reminder). ner is focused on identifying and extracting entities from user requests @xcite . current nlu models usually take in the asr 1-best hypothesis as input to perform nlu recognition @xcite . however, the highest-scored asr hypothesis is not always correct and, at times, can lead to downstream failures including incorrect nlu hypotheses. these errors can be mitigated by uti-lizing multiple top-ranked asr hypotheses (asr n-best hypotheses) in nlu modeling, which have a higher likelihood of containing the correct hypothesis. even in the case of all n-best hypotheses being incorrect, the nlu models may be capable of recovering the correct hypothesis by integrating the information contained within the n-best hypotheses. hence, the use of multiple hypotheses should help obtain firmer predictions from asr modules for their corresponding nlu module and result in improved performance. in this study we focus on two non-english internal datasets, german and portuguese, and evaluate the use of asr n-best hypotheses for improving nlu modeling within these contexts. given that the asr models we use in this experiment produce a maximum of five (or less) hypotheses per input utterance, we utilize all available hypotheses (referred to hereafter as the asr 5-best) for our work. we leverage two bert-based summarization models @xcite @xcite and a sequence-to-sequence model with a pointer generator network @xcite to extract the information from the asr 5-best hypotheses. we show that using multiple hypotheses, as opposed to just one, can significantly improve the overall performance of dc, and the performance of ic and ner in cases where the asr model makes mistakes. we describe relevant work in section 2 and present a description of our data set and opportunity cost analysis in section 3. in section 4 we describe the architecture of our models. in section 5, we present our experimental results followed by our conclusions in section 6. ## related work using deep learning models for summarization has been an active area of research in the recent past. two popular types in current literature have been extractive summarization and abstractive summarization. extractive summarization systems summarize by identifying and concatenating the most important sentences in a document whereas abstractive summarization systems conceptualize the task as a sequence-to-sequence problem and generate the summary by paraphrasing sections of the source document. extensive work has been done on extractive summarization @xcite @xcite @xcite @xcite and abstractive summa-rization @xcite @xcite used in isolation. furthermore, studies have shown improvement in summary quality when extractive and abstractive objectives have been used in combination @xcite @xcite . @xcite proposed a simple, yet powerful, variant of bert for extractive summarization in which they modified the input sequence of bert from its original two sentences to multiple sentences. they used multiple classification tokens ([cls]) combined with interval segment embeddings to distinguish multiple sentences within a document. they appended several summarization specific layers (either a simple classifier, a transformer or an lstm) on top of the bert outputs to capture document level features relevant for extracting summaries. following this work, @xcite proposed a model that comprises of the pre-trained bert extractive summarization model @xcite as the encoder and a decoder which consists of a 6-layered transformer @xcite . the encoder was fine-tuned in two stages, first on the extractive summarization task and then again on an abstractive summarization task resulting in a joint extractive-abstractive model that showed improved performance on summarization tasks. the utilization of multiple asr hypotheses for improved nlu model performance across dc, ic tasks was first introduced by @xcite . they proposed the use of 5-best asr hypotheses to train a bilstm language model, instead of using a single 1-best hypothesis selected using either majority vote, highest confidence score or a reranker. they explored two methods to integrate the n-best hypothesis: a basic concatenation of hypotheses text and a hypothesis embedding concatenation using max/avg pooling. the results show 14%-25% relative gains in both dc and ic accuracy. in our work, we explore the performance improvement offered by utilizing the asr 5-best hypotheses in previously unexplored languages, german and portuguese. we also differ from previous studies due to our use of the superior bertbased extractive @xcite and joint extractiveabstractive @xcite summarization models to extract a summary hypothesis for the dc task, from the asr 5-best. voice assistants traditionally handle ic and ner tasks using semantic parsing components which typically comprise of statistical slot-filling systems for simple queries and, in more recent time, shiftreduce parsers @xcite for more complex utterances. @xcite proposed a unified architecture based on sequence-to-sequence models and pointer generator networks to handle both simple and complex ic and ner tasks with which they achieve stateof-the-art results. in this work, we use a model that expands this approach to consume the 5-best asr hypotheses and evaluate its performance on ic/ner tasks for the two language datasets considered. ## data our experiments focus on two non-english internal datasets; german and portuguese. we run all utterances in each language through one languagespecific asr model and take the top-ranked asr hypothesis for each utterance as asr 1-best and all available hypotheses for each utterance (a maximum of five in our models) as asr 5-best. in addition, we also obtain a human transcribed version of each utterance. for german, we use 1.48 million utterances from 21 domains for training and validation. we split the data randomly within each domain, with 85% used for training and 15% for validation. an independent set of 193k utterances are used for testing. within the independent test set we find 17k utterances where the asr 1-best did not match the transcribed utterance exactly and mark them as the "mismatched" test set. ( ## conclusions and future work in this study, we explore the benefits of using asr 5-best hypotheses for the nlu tasks in the german and portuguese datasets. we explore several models to perform dc and ic/ner tasks and evaluate their performance against baseline models that use asr 1-best. we find significant overall improvement in performance for the dc task. we also find significant improvement in performance of the jointly evaluated ic/ner tasks in cases where the asr 1-best hypothesis is not an exact match to the transcribed utterance. for the dc task, our results suggest that the use of asr 5-best helps produce better hypotheses and thereby greater improvements in the case of slight lower quality asr models. our next steps will include exploring how different data splits based on asr confidence scores might affect the sequence-to-sequence model performance. furthermore, we will explore performance improvements in ic and ner tasks, using different model architectures and training schedules. we will also expand our study to a larger set of languages in order to understand how the use of multiple asr hypotheses might affect languages with different lexical distributions. languages which use multiple scripts (japanese, hindi, arabic etc.) or which are more opaque and likely to have heterographs (e.g., "serial, "cereal") and those that have less standardized spelling systems (hindi etc) are more likely to have asr errors. they may have different levels of improvement with the use of asr 5-best hypotheses and we hope to analyze this in our future work. yue dong, yikang shen, eric crawford, herke van hoof, and jackie chi kit cheung. 2018. bandit-sum: extractive summarization as a contextual bandit. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 3739-3748, brussels, belgium. association for computational linguistics.