MCLF : A Multi-grained Contrastive Learning Framework for ASR -robust Spoken Language Understanding (2023)

Abstract: Enhancing the robustness towards Automatic Speech Recognition (ASR) errors is of great importance for Spoken Language Understanding (SLU). Trending ASR-robust SLU systems have witnessed impressive improvements through global contrastive learning. However, although most ASR errors occur only at local positions of utterances, they can easily lead to severe semantic changes, and utterance-level classification or comparison is difficult to distinguish such differences. To address the problem, we propose a two-stage multi-grained contrastive learning framework dubbed MCLF. Technically, we first adapt the pre-trained language models to downstream SLU datasets via the proposed multi-grained contrastive learning objective and then fine-tune it on the corresponding dataset. Besides, to facilitate contrastive learning in the pre-training stage, we explore several data augmentation methods to expand the training data. Experimental results and detailed analyses on four datasets and four BERT-like backbone models demonstrate the effectiveness of our approach.

## introduction the rise in popularity of intelligent assistants like apple siri and amazon alexa has aroused interest in intelligent speech technology. a fundamental component of these smart assistants is the spoken language understanding (slu), a framework that aims to comprehend the semantics of human speech @xcite . traditionally, semantic information from utterances are obtained by initially applying automatic speech recognition (asr) to convert human speech into text. afterwards, the slu model is trained using this textual transcript as input @xcite @xcite . since speech-to-text has a wide range of application scenarios in both academia and industry, many asr systems based on neural networks have been proposed, e.g., speech transformer @xcite , wav2vec @xcite , and spiral @xcite . despite the success, asr systems still generate transcripts with errors (mainly insertions, deletions, and substitutions errors) when facing more complicated realworld scenarios @xcite . these asr errors may confuse the downstream slu tasks and lead to performance degradation. previous works towards asr-robust slu can be divided into two categories: 1) detecting and correcting asr errors directly @xcite , or injecting additional asr output information such as n-best hypothesis and phoneme sequence to the downstream slu models @xcite @xcite . however, both hypotheses with asr errors and extra asr information are hard to obtain due to the limitation of the asr system @xcite . 2) adapting the asr system architecture towards improving language correctness and fluency @xcite . an obvious shortcoming is the additional technical burden of modifying and retraining existing asr systems according to specific tasks. recent pre-trained language models (plms) like bert @xcite have shown appealing performance on various downstream nlp tasks. however, most plms are trained on clean texts and are compromised inevitably to asr errors. as shown in figure 1 , as wer gets higher, the slu model suffers severe classification performance degradation (from 95.69% to 56.71%). taking asr hypotheses as inputs may introduce an issue that words in an utterance may be misrecognized. for instance, fair and fare are acoustically similar, so an asr system may fail to distinguish between them, resulting in a substitution error. such substitution errors might be recovered by humans because humans are aware of the acoustic confusability of words. however, the errors may significantly degrade the testing performance when the models are trained on oracle transcripts (i.e., clean texts). in order to enhance the asr robustness in contextualized word embeddings, chang and chen (2022) first introduced contrastive pre-training objectives with only textual information as input. however, they neglect finer-level alignment between the oracle transcript and the asr hypothesis. in this work, we propose a novel two-stage multigrained contrastive learning framework (mclf) for asr-robust slu (illustrated in figure 2 ). specifically, in the first stage, we devise a finegrained interactive module to align the tokens of the oracle transcript with the ones of the asr hypothesis. we also leverage contrastive learning to facilitate global representation learning at the utterance level. inspired by @xcite that consider the phonetic confusion, we explore various task-specific data augmentation (da) methods to mitigate the deficiency of large amounts of labeled data for the plm training procedure. 1 in the second stage, we fine-tune the pre-trained mclf model with several robust training techniques. we conduct extensive experiments on four benchmark datasets (i.e., slurp, atis, trec6, and trec50) and four bert-like backbone models. experimen- in summary, the contributions of this paper are three-fold: 1) we propose mclf, which is the first attempt to combine global contrastive and fine-grained contrastive objectives for establishing asr-robust slu models; 2) experiments on four backbone models and four datasets demonstrate the effectiveness of our proposed method; 3) detailed analysis shows that our proposed framework brings effectiveness to the robustness of noise data. ## approach in this section, we first detailedly elaborate on the pre-training tasks used in mclf (figure 3 ) in section 2.1. then we introduce the proposed data augmentation methods for contrastive learning in the pre-training stage (section 2.2). finally, we briefly describe the fine-tuning stage in section 2.3. ## experiments in this section, we evaluate the effectiveness of the proposed method on four benchmark datasets, i.e., slurp @xcite , atis @xcite , trec6 and trec50 @xcite . data statistics are shown in ## discussions 4.1 umap visualization to further understand how the proposed method plays an important role, we extract the 768-d utterance embeddings from the trec6 dataset and use the uniform manifold approximation and projection (umap) (mcinnes and healy, 2018) algorithm to project these embeddings onto a 2-d plane in figure 4 . first, from figure 4a and figure 4b , we can see that the semantics of asr utterances are very scattered, farther than oracle utterances. second, figure 4a and 4c show the representations of asr hypothesis on roberta base and mclf, respectively. we can see that most of the asr utterances with same class are pulled together in both models. in roberta base , there are some asr utterances that cannot be assigned to specific classes, while nearly all the utterances in mclf model are pulled closer. third, from figure 4c and figure 4d , even in mclf, there are still some asr utterances that are indistinguishable, which we believe that it is due to the asr errors that make them indistinguishable, for example, we found two oracle utterances with different labels but they share same asr assumption in the dataset. ## related work slu is an essential task for machines to infer correct semantic meaning (e.g., intent detection, slot filling) from human speech @xcite @xcite @xcite @xcite . traditionally, it can be solved by fine-tuning nlp models (especially plms) with the asr hypothesis as input @xcite . however, the asr hypothesis often contains errors caused by asr systems. if the slu models are trained with clean texts, the performance will be poor when predicting using noisy text as input. also, models trained with noisy texts may suffer performance degradation over datasets without asr errors. the core problem lies in improving the asr robustness for slu models, and several types of methods have been proposed to address this issue. one stream treats texts with or without noise as two modalities. for example, @xcite proposed to rebuild tuples between asr results and correct candidates by leveraging the minimum edit distance (med). @xcite introduced a novel loss to minimize the distribution differences between correct and incorrect output texts. spo-kencse @xcite aimed to learn the invariant representations between clean text and erroneous hypothesis by utilizing a contrastive objective to adapt plm to asr results. another stream considers making full use of the asr output information (e.g., n-best hypothesis, phoneme sequence). for example, @xcite proposed two novel approaches (i.e., n-best plain, n-best alignment) that combine the n-best hypothesis of asr as input for error correction. @xcite proposed to trigger the fine-grained interactions between phoneme and word embeddings to derive phonetic-aware asr-robust features. @xcite used manual transcripts to improve asr robustness by mutual learning. @xcite proposed to incorporate syntax information into slu system to strengthen asr-robust ability by constraining attention scopes based on relationships within the syntactic structure. @xcite utilized cross attention to fuse the features between manual transcripts and asr transcripts and contrastive attention to capture unique features of asr transcripts. considering that the asr output information may not be easily obtained due to the constraint of asr systems, we adopt the first stream in this paper. in addition to following @xcite that builds alignment between asr hypotheses and the oracle transcripts through a contrastive learning objective, we design the fine-grained contrastive learning which strengthens the model's robustness towards the local word error caused by phonetic confusion, and it serves as a complement to the global contrastive learning. the combination of both types of contrastive objectives is called multigrained contrastive learning, aiming to learn both utterance and token level alignment between asr hypotheses and the oracle transcripts, which mines fruitful utterance-level and token-level information from downstream datasets. ## conclusion in this paper, we propose mclf, an asr-robust slu model towards multi-grained oracle-asr contrastive pre-training. besides, several data augmentation techniques are used to facilitate the contrastive pre-training stage. to our best knowledge, this is the first attempt to improve the asr robustness of slu models by simultaneously exploiting both global and fine-grained features alignment between asr hypothesis and oracle transcripts. empirical results on four benchmark datasets show the effectiveness of the proposed mclf and the superiority compared with several baseline works.