ML - LMCL : Mutual Learning and Large-Margin Contrastive Learning for Improving ASR Robustness in Spoken Language Understanding (2023)

Abstract: Spoken language understanding (SLU) is a fundamental task in the task-oriented dialogue systems. However, the inevitable errors from automatic speech recognition (ASR) usually impair the understanding performance and lead to error propagation. Although there are some attempts to address this problem through contrastive learning, they (1) treat clean manual transcripts and ASR transcripts equally without discrimination in fine-tuning; (2) neglect the fact that the semantically similar pairs are still pushed away when applying contrastive learning; (3) suffer from the problem of Kullback–Leibler (KL) vanishing. In this paper, we propose Mutual Learning and Large-Margin Contrastive Learning (ML-LMCL), a novel framework for improving ASR robustness in SLU. Specifically, in fine-tuning, we apply mutual learning and train two SLU models on the manual transcripts and the ASR transcripts, respectively, aiming to iteratively share knowledge between these two models. We also introduce a distance polarization regularizer to avoid pushing away the intra-cluster pairs as much as possible. Moreover, we use a cyclical annealing schedule to mitigate KL vanishing issue. Experiments on three datasets show that ML-LMCL outperforms existing models and achieves new state-of-the-art performance.

## introduction spoken language understanding(slu) is an important component of various personal assistants, such as amazon's alexa, apple's siri, microsoft's cortana and google's assistant @xcite . slu aims at taking human speech input and extracting semantic information for two typical subtasks, mainly including intent detection and slot filling @xcite . pipeline approaches and end-to-end approaches are two kinds of solutions of slu. pipeline slu methods usually combine automatic speech recognitgion (asr) and natural language understanding (nlu) in a cascaded manner, so they can easily apply external datasets and external pre-trained language models. however, error propagation is a common problem of pipeline approaches, where an inaccurate asr output can theoretically lead to a series of errors in subtasks. as shown in figure 1 , due to the error from asr, the model can not predict the intent correctly. following @xcite , this paper only focuses on intent detection. learning error-robust representations is an effective method to mitigate the negative impact of errors from asr and is gaining increasing attention. the remedies for asr errors can be broadly categorized into two types: (1) applying machine translation to translate the erroneous asr transcripts to clean manual transcripts @xcite @xcite ; (2) using masked language modeling to adapt the model. however, these methods usually requires additional speechrelated inputs @xcite @xcite , which may not always be readily available. therefore, this paper focuses on improving asr robustness in slu without using any speech-related input features. despite existing error-robust slu models have achieved promising progress, we discover that they suffer from three main issues: in this paper, we propose mutual learning and large-margin contrastive learning (ml-lmcl), a novel framework to tackle above three issues. for the first issue, we propose a mutual learning paradigm. in fine-tuning, we train two slu models on the manual and asr transcripts, respectively. these two models are collaboratively trained and considered as peers, with the aim of iteratively learning and sharing the knowledge between the two models. mutual learning allows effective dual knowledge transfer @xcite @xcite , which can improve the performance. for the second issue, our framework implements a large-margin contrastive learning to distinguish between intra-cluster and inter-cluster pairs. specifically, we apply a distance polarization regularizer and penalize all pairwise distances within the margin region, which can encourage polarized distances for similarity determination and obtain a large margin in the distance space in an unsupervised way. for the third issue, following @xcite , we mitigate kl vanishing by adopting a cyclical annealing schedule. the training process is effectively split into many cycles. in each cycle, the coefficient of kl divergence progressively increases from 0 to 1 during some iterations and then stays at 1 for the remaining iterations. experiment results on three datasets slurp, atis and trec6 @xcite @xcite demonstrate that our ml-lmcl significantly outperforms previous best models and model analysis further verifies the advantages of our model. the contributions of our work are four-fold: ## approach our framework includes four elements: (1) selfsupervised contrastive learning with a distance polarization regularizer in pre-training. (2) mutual learning between the model trained on clean manual transcripts and the model trained on asr transcripts in fine-tuning. (3) supervised contrastive learning with a distance polarization regularizer in fine-tuning. (4) self-distillation with the cyclical annealing schedule in fine-tuning. ## related work error-robust spoken language understanding slu usually suffers from asr error propagation and this paper focus on improving asr robustness in slu. @xcite makes the first attempt to use contrastive learning to improve asr robustness with only textual information. following @xcite , this paper only focuses on intent detection in slu. intent detection is usually formulated as an utterance classification problem. as a large number of pre-trained models achieve surprising results across various tasks @xcite @xcite , some bertbased @xcite ) pre-trained work has been explored in slu where the representation of the special token [cls] is used for intent detection. in our work, we adopt roberta and try to learn the invariant representations between clean manual transcripts and erroneous asr transcripts. ## conclusion in this paper, we propose ml-lmcl, a novel framework for improving asr robustness in slu. we apply mutual learning and introduce the distance polarization regularizer. moreover, cyclical annealing schedule is utilized to mitigate kl vanishing. experiments and analysis on three benchmark datasets show that our model significantly outperforms previous models whether clean manual transcriptions is available in fine-tuning or not. future work will focus on improving asr robustness with only clean manual transcriptions.