Enhancing Dialogue Speech Recognition with Robust Contextual Awareness via Noise Representation Learning (2024)

Abstract: Recent dialogue systems typically operate through turn-based spoken interactions between users and agents. These systems heavily depend on accurate Automatic Speech Recognition (ASR), as transcription errors can significantly degrade performance in downstream dialogue tasks. To alleviate this challenge, robust ASR is required, and one effective method is to utilize the dialogue context from user and agent interactions for transcribing the subsequent user utterance. This method incorporates the transcription of the user’s speech and the agent’s response as model input, using the accumulated context generated by each turn. However, this context is susceptible to ASR errors because the ASR model generates it auto-regressively. Such noisy context can further degrade the benefits of context input, resulting in suboptimal ASR performance. In this paper, we introduce context noise representation learning to enhance robustness against noisy context, ultimately improving dialogue speech recognition accuracy. To maximize the advantage of context awareness, our approach involves decoder pre-training with text-based dialogue data and noise representation learning for a context encoder. Evaluated on DSTC11 (MultiWoZ 2.1 audio dialogues), it achieves a 24% relative reduction in Word Error Rate (WER) compared to wav2vec2.0 baselines and a 13% reduction compared to Whisper-large-v2. Notably, in noisy environments where user speech is barely audible, our method proves its effectiveness by utilizing contextual information for accurate transcription. Tested on audio data with strong noise level (Signal Noise Ratio of 0dB), our approach shows up to a 31% relative WER reduction compared to the wav2vec2.0 baseline, providing a reassuring solution for real-world noisy scenarios.

## introduction automatic speech recognition (asr) is central in accurately interpreting human speech, serving as a fundamental resource for numerous subsequent downstream tasks. the advent of robust asr modules, such as wav2vec2.0 @xcite and whisper @xcite , has significantly enhanced the capabilities of asr systems, facilitating their integration into a wide array of * equally contributed research and application domains. the integration of asr modules into various works highlights the pivotal role of asr in enhancing human-computer interaction, signifying a notable development in interactive technologies. despite the successful advancement of the asr system, its inaccuracy poses significant risks to the efficacy of downstream tasks, such as speech-totext translation @xcite @xcite and spoken language understanding @xcite @xcite . these tasks predominantly rely on the textual output generated by asr systems, highlighting the importance of accuracy in the initial speech recognition process. especially for the dialogue system, the quality of the asr system is paramount to ensure seamless interaction between user and dialogue agent, as models trained on written conversations perform poorly on spoken data @xcite . to minimize the impact of asr error on the dialogue model, various endeavors have been made. @xcite used an asr correction module which employs multiple asr models, while others focused on augmenting data with plausible asr errors @xcite @xcite . however the limitation is evident as they primarily focus on the robustness of dialogue models, which may not address the core issue compared to directly rectifying asr models. conversely, incorporating a context encoder for dialogue history to improve the asr model has been proposed, resulting in notable performance enhancements @xcite @xcite . nevertheless, since the context is transcribed at each turn by the asr model, it may contain errors, potentially disrupting the use of contextual information. in this work, we present a novel context noise representation learning (cnrl) method to encode accurate contextual information, even from noisy asr transcriptions. this approach aims to improve the performance of speech recognition in task oriented dialogue (tod) by minimizing the impact of asr errors in dialogue history as context. furthermore, we explore the advantages of decoder pre-training in context-aware asr systems, emphasizing their improved robustness in noisy environments. the overall training pipeline can be decomposed by three steps: 1) decoder pretraining on text-based dialogue data between user and agent. 2) asr fine-tuning with speech encoder and context encoder jointly. 3) cnrl on context encoder to minimize the impact of asr-noise context. our contributions are as follows: ## conclusion this study introduced context noise representation learning (cnrl) to improve context-aware asr systems, especially in noisy environments. by integrating decoder pre-training with dialogue data, asr fine-tuning, and cnrl, we significantly reduced transcription errors. our training pipeline demonstrated significant improvements in dialogue speech recognition, even in noisy environments where speech input is defective. experiments showed cnrl's efficacy, reducing word error rate (wer) by up to 11.1% in clean conditions and 19.1% in noisy settings. by making the model more robust against noisy context, our approach consistently outperformed baselines in various settings.