Contrastive and Consistency Learning for Neural Noisy-Channel Model in Spoken Language Understanding (2024)

Abstract: Recently, deep end-to-end learning has been studied for intent classification in Spoken Language Understanding (SLU). However, end-to-end models require a large amount of speech data with intent labels, and highly optimized models are generally sensitive to the inconsistency between the training and evaluation conditions. Therefore, a natural language understanding approach based on Automatic Speech Recognition (ASR) remains attractive because it can utilize a pre-trained general language model and adapt to the mismatch of the speech input environment. Using this module-based approach, we improve a noisy-channel model to handle transcription inconsistencies caused by ASR errors. We propose a two-stage method, Contrastive and Consistency Learning (CCL), that correlates error patterns between clean and noisy ASR transcripts and emphasizes the consistency of the latent features of the two transcripts. Experiments on four benchmark datasets show that CCL outperforms existing methods and improves the ASR robustness in various noisy environments. Code is available at https://github.com/syoung7388/CCL

## introduction understanding a conversation is critical for reaching a user's goal in a spoken dialogue system. spoken language understanding (slu) has been investigated by applying an end-to-end approach with a singular speech input model for training. the end-to-end approach requires an extensive volume of labeled speech data, which can be challenging to collect. as dialogue systems are commonly used in many real-world scenarios, the end-to-end slu models are susceptible to unseen instances that mismatch the training data due to limited language expression and acoustic conditions. utilizing a module-based approach, which leverages the capabilities of pre-trained language models, is an appealing option for developing gen-eral spoken dialogue systems. however, this option depends on the premise that the model pre-trained on clean text can effectively handle errors caused by the automatic speech recognition (asr) module. consequently, it is necessary to solve the problem of inconsistencies in noisy asr transcripts that contain speech recognition errors. recent studies have addressed the challenge of discrepancies between clean and noisy asr transcripts. @xcite minimized the kullback-leibler (kl) divergence so that the prediction distributions of both transcripts become similar in a noisy environment. in contrast, @xcite demonstrated a technique that utilizes a contrastive learning to align the implicit features of paired clean and noisy asr transcripts. however, most previous studies still need to consider the alignment for fine-grained errors such as insertion, deletion, and substitution caused by asr modules. we expect that using an improved noisy-channel model will mitigate the sensitivity of the pretrained language model to discrepancies between clean and noisy asr transcripts. the noisychannel approach aims to identify the target word, even if the input becomes scrambled or distorted. we present the neural noisy-channel model to predict error-free intent by (i) finding the error portions of given noisy input relative to clean transcript and (ii) matching the latent representations from two transcripts. in this paper, we introduce contrastive and consistency learning (ccl) to perform this two-stage. as shown in figure 1 (a), reference and inference networks receive clean and noisy asr transcripts as inputs, respectively. during the ccl, two networks simultaneously perform a token-based contrastive learning followed by a consistency learning. the purpose of token-based contrastive learning is to correlate errors in the noisy asr transcript with the corresponding clean transcript at both word and utterance token-levels. we select the word tokens in figure 1 (b), we only use the inference network during the evaluation process, and the reference network plays a supplementary role in inference network training. this allows us to solve the asr error problem without any increase in inference time. to demonstrate the effectiveness of the proposed ccl, we experiment with the slu benchmark datasets: slurp @xcite , timers @xcite , fsc @xcite , and snips @xcite . on the slurp benchmark dataset, the proposed ccl method improves the intent classification performance by 2.59% under severe asr errors. we also observe that our method outperforms previous module-based or end-to-end approaches on snips. to summarize, our contributions are as follows: ## related work in this section, we introduce the related work from two aspects. ## method in this section, we analyze the asr error problems. we also introduce the noisy-channel model and the ccl method to address these issues. our method performs the token-based contrastive learning followed by the consistency learning. ## conclusion we have introduced ccl, a novel method to solve the asr error problem of pre-trained language models. we designed the two-stage method: (i) token-based contrastive learning to correlate the asr errors at both the word and utterance tokenlevels and (ii) consistency learning to maintain coherence with the latent feature of relatively better