PCAD : Towards ASR -Robust Spoken Language Understanding via Prototype Calibration and Asymmetric Decoupling (2024)

Abstract: Spoken language understanding (SLU) inevitably suffers from error propagation from automatic speech recognition (ASR) in actual scenarios. Some recent works attempt to alleviate this issue through contrastive learning. However, they (1) sample negative pairs incorrectly in pre-training; (2) only focus on implicit metric learning while neglecting explicit erroneous predictions; (3) treat manual and ASR transcripts indiscriminately. In this paper, we propose a novel framework termed \textbf{PCAD} , which can calibrate bias and errors and achieve adaptive-balanced decoupling training. Specifically, PCAD utilizes a prototype-based loss to aggregate label and prediction priors and calibrate bias and error-prone semantics for better inter-class discrimination and intra-class consistency. We theoretically analyze the effect of this loss on robustness enhancement. Further, we leverage a teacher-student model for asymmetric decoupling training between different transcripts and formulate a novel gradient-sensitive exponential moving averaging (GS-EMA) algorithm for adaptive balance of accuracy and robustness. Experiments on three datasets show that PCAD significantly outperforms existing approaches and achieves new state-of-the-art performance.

## introduction spoken language understanding (slu) is the core component of intelligent assistants, aiming to capture the semantics information of user queries @xcite . the mainstream slu solutions are divided into two categories: pipeline and endto-end methods @xcite . pipeline slu methods combine automatic speech recognition (asr) and natural language understanding (nlu) in a cascaded manner, enabling them to easily employ pre-trained language models (cheng et al., 2023a, 2024; zhuang et al., 2024; zhu et al., * corresponding author. 2024a). however, pipeline slu suffers from error propagation from asr in actual scenarios, which inevitably damages the performance of cascaded nlu components. some studies explore to address this issue through directly correcting asr errors @xcite or adapting networks via masked language modeling (mlm) @xcite @xcite , which usually introduces extra speechrelated information for correction. recent works @xcite enhance representation through contrastive learning @xcite to mitigate the impact of asr errors. following @xcite ; @xcite , we focus on pipeline slu methods and attempt to improve asr robustness only by using textual information. although these works @xcite have achieved promising progress through contrastive learning on asr and clean transcripts, we discover that they still suffer from (1) sample bias: in the label-agnostic pre-training stage, contrastive learning @xcite ) may mistakenly sample negative pairs @xcite , resulting in the separation of features with the same semantics @xcite . (2) semantic error: chang and chen (2022); @xcite attempt to alleviate asr errors by implementing feature alignment in an implicit space. however, they ignore explicit erroneous intentions and treat correct and incorrect predictions equally during metric learning. (3) coupling tendency: clean and asr transcripts make different contributions to the accuracy and robustness. @xcite treat different transcripts indiscriminately and use a mixed set of all transcripts for fine-tuning simultaneously. despite being aware of this issue, @xcite still symmetrically trains different transcripts through dual models. in this paper, we propose prototype calibration and asymmetric decoupling (pcad) to tackle the above issues. for sample bias, we adopt a prototype contrastive loss as a regularizer to calibrate errors between all features and corresponding prototype centroids @xcite for alleviating representation deterioration. class prototypes can be used to include label supervision, thereby alleviating sampling bias caused by labelagnostic pre-training. for the second issue, we introduce an error-sensitive prototype loss which adds a calibration term to the regular prototype loss to punish the misclassified predictions during fine-tuning. we emphasize on learning more accurate prototypes by removing misclassified intentions. figure 1 illustrates our main idea. we further analyze the theoretical effectiveness of our prototype-based approach for calibrating bias and improving robustness from the perspectives of gradient and robust decision-making in section 3.6. for coupling tendency, we innovatively propose an asymmetric algorithm termed gs-ema for decoupling training, which utilizes the average gradient saliency to achieve the adaptive balance of accuracy and robustness. to our best knowledge, it is the first to calibrate error-prone slu representation from a prototype perspective and consider adaptivebalanced decoupling training for improving asr robustness. our contributions can be summarized as: (1) we propose an asr-robust slu framework termed pcad, which introduces a prototype-based calibration loss and its error-sensitive variant for calibrating bias in pre-training and reducing errors in fine-tuning. (2) we introduce a teacher-student model for asymmetric decoupling training and propose a novel gs-ema algorithm to achieve adaptive balance. (3) theoretical analysis and extensive experiments on three datasets slurp @xcite , atis @xcite and trec6 @xcite show that pcad can effectively handle asr noise and improve robustness in slu. ## method in this section, we will elaborate on the specific pipeline of our pcad. as shown in figure 2 , our proposed pcad includes three components: (1) prototype calibration loss in pre-training (in section 3.2); (2) error-sensitive calibration loss in finetuning (in section 3.3); (3) adaptive-balanced decoupling training in fine-tuning (in section 3.4). further, we provide the total training objective (in section 3.5) and the theoretical analysis (in section 3.6) of our method in asr robustness. ## conclusion in this paper, we introduce prototype-based losses for pre-training and fine-tuning, namely pcl and es-pcl, and analyze their remedial effects on sampling bias and semantic errors from a theoretical perspective. it is the first time to improve asr ro-bustness from the perspective of prototype calibration. in addition, in order to achieve high-quality decoupling training, we propose a novel update algorithm based on gradient saliency for adaptive balance. extensive experiments on three datasets demonstrate that pcad outperforms the state-ofthe-art approaches by a large margin.