Back Transcription as a Method for Evaluating Robustness of Natural Language Understanding Models to Speech Recognition Errors (2023)

Abstract: In a spoken dialogue system, an NLU model is preceded by a speech recognition system that can deteriorate the performance of natural language understanding. This paper proposes a method for investigating the impact of speech recognition errors on the performance of natural language understanding models. The proposed method combines the back transcription procedure with a fine-grained technique for categorizing the errors that affect the performance of NLU models. The method relies on the usage of synthesized speech for NLU evaluation. We show that the use of synthesized speech in place of audio recording does not change the outcomes of the presented technique in a significant way.

## introduction regardless of the near-human accuracy of automatic speech recognition in general-purpose transcription tasks, speech recognition errors can still significantly deteriorate the performance of a natural language understanding model that follows the speech-to-text module in a conversational system. the problem is even more apparent when an automatic speech recognition system from an external vendor is used as a component of a virtual assistant without any further adaptation. the goal of this paper is to present a method for investigating the impact of speech recognition errors on the performance of natural language understanding models in a systematic way. the method that we propose relies on the use of back transcription, a procedure that combines a text-to-speech model with an automatic speech recognition system to prepare a dataset contaminated with speech recognition errors. the augmented dataset is used to evaluate natural language understanding models and the outcomes of the evaluation serve as a basis for defining the criteria of ## related work data augmentation is a commonly employed method for improving the performance of neural models of vision, speech and language. @xcite developed nlpaug, a tool that encompasses a wide range of augmentation techniques for text and audio signal. @xcite presented a framework for adversarial attacks for the nlp domain, called textattack, that can be utilized for data augmentation and adversarial training. the framework has a modular design -attacks can be built from four components: a goal function, a set of constraints, a transformation, and a search method. it also provides out-of-the-box implementations for tts reference audio asr hypothesis backtranscription robustness assessment r(s) evaluate & compare h(s) b(s) a(s) e(s) criterion (1): correct -> incorrect criterion (2): incorrect->incorrect criterion (3): incorrect -> correct negative irrelevant positive error analysis whole word split/join affix text alignment rules error model nlu model several popular attacks described in the literature. back translation @xcite is a technique developed to improve the performance of neural machine translation. in this method, additional monolingual data for training the model are obtained by translating utterances from the target language to the source language. the first experiments with augmenting asr data with text-to-speech tools were conducted by @xcite . they generated speech data from unpaired texts to create a paired speech-text corpus. @xcite proposed a novel data augmentation method for end-to-end asr. their method uses a large amount of text not paired with speech signals in a back-translation-like manner. the authors claim to achieve faster attention learning and reduce computational costs by using hidden states as a target instead of acoustic features. @xcite built a tts system on an asr training database, then synthesized speech to extend the training data. the authors distinguish two main approaches to asr: hybrid (dnn+hmm) and end-to-end. they noted that both approaches achieve similar performance when there is a large amount of training data @xcite , but hybrid models perform better than end-to-end models when the amount of data is smaller @xcite . @xcite construct a parallel corpus of texts and their back-transcribed counterparts for the purpose of training a post-processor for an asr system. they show that their method is effective in correcting spacing, punctuation and foreign word conversions. recently, there have been several papers addressing the issue of the robustness of natural language understanding systems to various types of input errors. @xcite defined three aspects of nlu robustness (language variety, speech characteristics, and noise perturbation) and proposed a method of simulating natural language perturbations for assessing the robustness in task-oriented dialog systems. the influence of the noisy text input on the nlu performance was also studied by @xcite , who tested intent classification and slot labeling models' robustness to seven types of input data noise (abbreviations, casing, misspellings, morphological variants, paraphrases, punctuation, and synonyms). @xcite created a benchmark for evaluating the performance of task-oriented dialog models, designed to favor models with a strong generalization ability, i.e., robust to language variations, speech errors, unseen entities, and out-of-domain utterances. ## method the proposed method of evaluation consists of three stages: the execution of the back transcription procedure that transfers nlu data between text and audio domains, the automatic assessment of the outcome from the nlu model on a per-sample basis, and the fine-grained method of inspecting the results with the use of edit operations (see figure 1 ). the method relies on the availability of an nlu dataset containing user utterances along with the semantic representation of the uttered commands. we do not assume any particular annotation scheme for the nlu commands. however, in the experiments discussed in section 4, we work with the dataset that provides conventional annotations for domains, intents, and slots. ## conclusion in this paper, we proposed a method for assessing the robustness of nlu models to speech recognition errors. the method repurposes the nlu data used for model training and does not depend on the availability of spoken corpora. we introduced criteria for robustness that rely on the outcome of the nlu model but do not assume any particular semantic representation of the utterances. we showed how these criteria can be used to formulate summary metrics and constructed an analytical model that prioritizes individual categories of speech recognition errors on the basis of their impact on the (non-)robustness of the nlu model. finally, we performed an experimental evaluation of the robustness of transformer-based models and investigated the impact of using text-to-speech models in place of audio recording.