Data Augmentation for Training Dialog Models Robust to Speech Recognition Errors (2020)

Abstract: Speech-based virtual assistants, such as Amazon Alexa, Google assistant, and Apple Siri, typically convert users’ audio signals to text data through automatic speech recognition (ASR) and feed the text to downstream dialog models for natural language understanding and response generation. The ASR output is error-prone; however, the downstream dialog models are often trained on error-free text data, making them sensitive to ASR errors during inference time. To bridge the gap and make dialog models more robust to ASR errors, we leverage an ASR error simulator to inject noise into the error-free text data, and subsequently train the dialog models with the augmented data. Compared to other approaches for handling ASR errors, such as using ASR lattice or end-to-end methods, our data augmentation approach does not require any modification to the ASR or downstream dialog models; our approach also does not introduce any additional latency during inference time. We perform extensive experiments on benchmark data and show that our approach improves the performance of downstream dialog models in the presence of ASR errors, and it is particularly effective in the low-resource situations where there are constraints on model size or the training data is scarce.

## introduction speech-based virtual assistants, such as amazon alexa, google assistant, and apple siri, have become increasingly powerful and popular in our everyday lives, offering a wide range of functionality including controlling smart home devices, booking movie tickets, and even chit-chatting. these speech-based virtual assistants typically contain the following components: an automatic speech recognition (asr) module that converts audio signals from a user to a sequence of words, a natural language understanding (nlu) module that extracts semantic meaning from the user utterance, a dialog management (dm) module that controls the dialog flow and communicates with external applications if necessary, a natural language generation (nlg) module that converts the system response to natural language, and a text-to-speech (tts) module that converts the text response to an audio response @xcite . the errors made by the asr module can propagate to the downstream dialog models in nlu and dm and degrade their performances @xcite . one straightforward approach to improve the downstream dialog models' robustness to asr errors is to train them with asr hypotheses with potential asr errors in addition to the error-free reference texts. however, the training data might not always have corresponding asr hypotheses available, for example, when the training data are created in written forms from the beginning. such training data include online reviews, forums, and data collected in a wizard-of-oz (woz) setting @xcite . additionally, even when there are asr hypotheses available, transcribing and annotating the asr hypotheses to create the training data is a slow and expensive process due to human involvement, limiting the size of available training data. to address these challenges, we propose a simple data augmentation method leveraging a confusionmatrix-based asr error simulator @xcite . our method can be used on training data with or without existing asr hypotheses, does not require modifying the asr model or downstream dialog models, and consequently does not introduce additional latency during inference time. we assess the method's effectiveness on a multi-label classification task on a public dataset from dstc2 @xcite . we show that our method can improve the dialog models' performances in the presence of asr errors, particularly in the low resource situations where there are model size or latency constraints, or the training data is scarce. ## related work existing approaches for handling asr errors generally fall into four categories: 1) pre-training word embeddings such that acoustically similar words are close to each other in the embedding space @xcite @xcite ; 2) using multitask training to jointly correct asr errors in addition to performing the original nlu tasks @xcite ; 3) using n-best asr hypotheses, word confusion networks, or asr lattice produced by the asr system as the input for the downstream dialog models, allowing the models to consider all the alternatives instead of only the 1-best asr hypothesis @xcite @xcite ; and 4) using an end-to-end approach that combines asr and nlu systems into one, extracting semantics directly from audio signals @xcite . these approaches often either require significant modifications to asr model and/or downstream dialog models, or require access to additional information from the asr model, such as asr n-best or asr lattice, during inference time. in comparison, data augmentation is much simpler because it does not modify the existing model architecture or introduce additional latency during inference time. data augmentation has a long history in image processing @xcite . in language processing, researchers have proposed back-translation @xcite and simple operations such as synonym replacement and random swap @xcite to increase the variations of training data. these data augmentation approaches aim to improve dialog models' robustness to surface form variations in general, whereas our approach focuses on robustness to asr errors in particular. note that our data augmentation approach can be complimentary to using acoustic embeddings (first category), multi-task training (second category), and the other mentioned data augmentation approaches, and it is possible to combine them for further performance gains. ## method we propose to use simulated asr hypotheses to augment the training data of dialog models. to this end, we adopt the confusion-matrix-based asr error simulator initially proposed by @xcite and improved by @xcite . here we describe the error simulator at a high level, while leaving the details to the mentioned references. the main component of the error simulator is an n-gram confusion matrix constructed from a corpus of asr hypotheses and corresponding reference texts: each asr hypothesis and its reference text are aligned at the word level by minimizing the levenshtein distance between them, then the frequencies of n-gram confusions are added to the confusion matrix for @xmath0, m ], where m is a pre-specified constant. during inference time, the error simulator first partitions a reference text into n-grams where n can vary, then for each n-gram it samples a replacement from the confusion matrix with sampling probabilities proportional to the frequencies of confusions. note that the sampled "confusion" can be the original n-gram itself, which means correct recognition for this n-gram in the simulated hypothesis. we refer to the corpus used to construct the ngram confusion matrix as the asr corpus to distinguish it from the training data for the dialog models that we want to apply the error simulator to. by design, if the reference texts that the error simulator is applied to have the same distribution as the reference texts in the asr corpus, then the simulated asr hypotheses will have the same error distribution as the asr hypotheses in the asr corpus @xcite , where the error distribution includes word-error-rate (wer) and proportions of insertion, deletion, and substitution errors. however, in practice it can be useful to simulate asr hypotheses with a pre-specified wer different from that of the asr corpus. adjusting the wer is non-trivial, because each word's individual wer is often different from the overall wer of the asr corpus; i.e., some words are more easily confusable than others. we introduce a heuristic to adjust each word's individual wer during inference time of the error simulator so that the overall wer in the simulated asr hypotheses is close to the pre-specified target overall wer based on the following formula (see appendix a): this heuristic has the following desired properties: 1) if w 1 has a higher original individual wer than that of w 2 before the adjustment, then w 1 will have a higher target individual wer than that of w 2 from this adjustment, for arbitrary words w 1 and w 2 under certain simplifying conditions (appendix a); i.e., we mostly preserve the property that some words are more easily confusable than others. 2) in the trivial case where all words have the same individual wer as the overall wer, this heuristic is equivalent to setting all individual wer to the target overall wer. we apply data augmentation to the training data of dialog models in two different cases: s1) the training data only have reference texts and no corresponding asr hypotheses. in this case, we construct the confusion matrix used by the asr error simulator with an asr corpus ideally close to the training data of dialog models in terms of vocabulary overlap, or a large generic asr corpus such as fisher english training speech corpora @xcite . we simulate multiple asr hypotheses for each reference sentence with different wer, and combine all the simulated asr hypotheses with the reference text as the augmented training data -the motivation behind this is to create more variations, make dialog models robust to different levels of wer, and avoid degradation on error-free data. s2) the training data have both reference texts and corresponding asr hypotheses. in this case, we can directly use the training data as the asr corpus to construct the confusion matrix, then simulate asr hypotheses with different wer and combine them with the original asr hypotheses and reference texts as the augmented training data. note that during inference time of the asr error simulator, it partitions a sentence and samples n-gram replacements probabilistically, so even though we use the training data of dialog models as the asr corpus, the error simulator can still create new variations in the simulated asr hypotheses. ## experiments we experiment our proposed data augmentation method on the dialog act classification task. ## conclusion in this paper, we proposed a method for data augmentation in order to make downstream dialog models more robust to asr errors. we leveraged a confusion-matrix-based asr error simulator to inject noise into the error-free text data, and subsequently trained dialog act classification models with the augmented data. compared to other approaches of handling asr errors, our data augmentation approach does not require any modification to the asr models or downstream dialog models, thus our approach also does not introduce any additional latency during inference time of the dialog models. we performed extensive experiments on benchmark data and showed that our approach improves the performance of downstream dialog models in the presence of asr errors, and it is particularly effective in the low-resource situations where the model size needs to be small or the training data is scarce. for future work, we plan to investigate the effect of our proposed method on additional tasks such as dialog state tracking and response generation. additionally, we believe that our data augmentation approach is complimentary to using acoustic embeddings, multi-task training, and other mentioned data augmentation approaches, and we plan to combine them for further performance gains.