Interventional Speech Noise Injection for ASR Generalizable Spoken Language Understanding (2024)

Abstract: Recently, pre-trained language models (PLMs) have been increasingly adopted in spoken language understanding (SLU). However, automatic speech recognition (ASR) systems frequently produce inaccurate transcriptions, leading to noisy inputs for SLU models, which can significantly degrade their performance. To address this, our objective is to train SLU models to withstand ASR errors by exposing them to noises commonly observed in ASR systems, referred to as ASR-plausible noises. Speech noise injection (SNI) methods have pursued this objective by introducing ASR-plausible noises, but we argue that these methods are inherently biased towards specific ASR systems, or ASR-specific noises. In this work, we propose a novel and less biased augmentation method of introducing the noises that are plausible to any ASR system, by cutting off the non-causal effect of noises. Experimental results and analyses demonstrate the effectiveness of our proposed methods in enhancing the robustness and generalizability of SLU models against unseen ASR systems by introducing more diverse and plausible ASR noises in advance.

## introduction pre-trained language models (plms) have demonstrated a robust contextual understanding of language and the ability to generalize across different domains. thus, plms have gained widespread acceptance and been employed within the realm of spoken language understanding (slu), such as voice assistants @xcite . a concrete instance of this application is a pipeline where an automatic speech recognition (asr) system first transcribes audio inputs, and the transcriptions are then processed by plms for downstream slu tasks @xcite . * corresponding author. figure 1 : different asr systems generate different asr errors (asr 1 : blue, asr 2 : red, common, or, asr * : green). biased toward a specific asr, asr 1 , baseline sni generates noises plausible only for asr 1 , or even some noise that are not plausible to any (cue to sue). our distinction is 1) removing its bias to a specific asr (read to lead), and 2) generating asr * -plausible noises (cue to queue). however, such pipelines encounter challenges when faced with inaccuracies from asr systems. we refer to these inaccuracies as asr error words, which are phonetically similar but semantically unrelated @xcite . for instance, as shown in fig. 1 , asr systems might confuse words like "cereal" and "serial" or "quarry" and "carry", resulting in incorrect transcriptions such as "workers eat serial in the carry". despite their phonetic resemblance, these asr error words convey unintended meanings, thereby impeding the semantic understanding of speech in slu tasks @xcite . a well-known solution is speech noise injection (sni) which generates likely incorrect transcriptions, namely pseudo transcriptions, then exposes plms to the generated pseudo transcriptions while training slu tasks @xcite . thus, the injected noises should be asr-plausible, being likely to be gen-erated by asr systems from real audio input, for which the traditional methods attempted to replicate the asr errors from written text. however, their asr-plausibility is conditional on a particular asr system, or asr i , where the error distribution from the collected transcriptions follows only the observed error distribution @xcite @xcite . however, different asr systems have distinct error distributions @xcite , which hinders the trained slu models to be used with other asr systems, asr j . a straightforward solution to this issue might be to employ multiple asr systems or to build multiple slu models, but this approach incurs significant overheads and cannot account for the diversity of real-world asr errors, e.g., errors due to environmental sounds. instead, we investigate a novel but easier solution of introducing better generalizable noises from a single asr system. for this purpose, we first identify the gap between the asr transcription in slu and sni from a causality perspective. first, slu tasks aim to handle real audio input, where written groundtruths(gts) are recorded as audio by humans, such that asr errors are causally affected by recorded audio, as shown in figure 2a . however, sni, when replicating error patterns, may discard the correctly transcripted texts, biased towards the observed errors. inspired by causality literature, we introduce two technical contributions: 1) interventional noise injection and 2) phoneme-aware generation. specifically, we adopt do-calculus to intervene the "revised" in sni to deviate from the observed error distribution, thereby broadening the error patterns in the resulting pseudo transcripts. for instance, as shown in fig. 1 , our sni model can corrupt the word 'read' with the phoneme 'ô', which was corrupted in asr 2 but not in asr 1 , in addition to the errors introduced by baseline sni. next, we ensure that the debiased noises are plausible for any asr system, referred to as asr * . this involves making gt words and asr noises phonetically similar based on the common characteristics shared by asr * @xcite . along with the textual input, we incorporate information on how words are pronounced. by being aware of pronunciation, we can introduce asr noises that are plausible regardless of the specific asr system used, making them asr * -plausible. experiments were conducted in an asr zeroshot setting, where sni models were trained on asr i and tested on slu tasks using another asr system, asr j , on the dstc10 track2 and asr glue benchmarks. results show that our proposed methods effectively generalize across different asr systems, with performance comparable to, or even exceeding, the in-domain setting where asr j is used to train the sni model. ## method before introducing isni, we first formally define the problem of sni and outline its causal diagram. following this, we provide an overview of isni and detail the methods used to generate pseudo transcriptions that enhance the robustness of slu tasks against asr * . ## experiments in this section, we delineate our experimental setup, datasets, and baseline, for sni and slu. our experiments, conducted in an asr zero-shot setting, train sni models on one asr system (asr i ) and test slu models on another (asr j ), to evaluate generalizability across different asr systems. ## results we now present our experimental results, addressing the following research questions: rq1: is the asr zero-shot setting valid and how effective are isni in the asr zero-shot setting? rq2: can isni robustify the various slu tasks in the asr zero-shot setting? rq3: does each of methods contribute to robustification? ## conclusion in this paper, we address the challenge of asr generalizability within the context of sni. we focus on enhancing the robustness of slu models against asr errors from diverse asr systems. our contributions are two-fold: firstly, isni significantly broadens the spectrum of plausible asr errors, thereby reducing biases. second, not to lose the generality to any asr, we generate noises that are universally plausible, or asr * -plausible, which is empirically validated through extensive experiments across multiple asr systems. limitations. one limitation of isni is reducing the chance of making substitution errors for entire sequences of 2-3 words. as in previous works in sni, isni consumes one token at a time when producing outputs. while such consumption does not restrict to make substitution errors for entire sequences, it may reduce the chance of making substitution errors for such long sequences, as isni decides to corrupt one token at a time. secondly, as isni is trained with speech corpora collected for academic purposes, they may face challenges when adopted for real-world applications, including diverse spoken language variations such as dialects and accents. these variations can introduce noises that are not phonetically similar, which are different from the speech data used during isni training. this discrepancy may cause isni to fail in robustifying slu models as isni is not prepared to handle asr errors from those speech variations. addressing this limitation may require enlarging the training dataset for isni to cover the diverse noises from spoken language variations. 7 appendix 7.1 related works previously proposed methods can be categorized into three categories. first, tts (text-to-speech)-asr pipeline @xcite adopts the tts engine, the reverse of asr, to convert written text into audio. the asr i transcribes it into pseudo transcription. however, the human recording and tts-generated audio differ in their error distributions, which makes the resulting pseudo transcriptions different from asr transcriptions @xcite . second is textual perturbation, replacing the words x @xmath0: a representative of the above two categories is tod-da which embodies both categories. third is the auto-regressive generation, where auto-regressive plms such as gpt-2 or bart are supervised to maximize the likelihood of the asr transcription t i given its written text x. adopting plms, the auto-regressive generation can consider contextual information and generate more asr iplausible noise words @xcite . auto-regressive generation is biased to asr i , limiting the slu model used for asr i . our distinction is generalizing sni so that the slu tasks can be conducted with asr * .