Continual Contrastive Spoken Language Understanding (2024)

Abstract: Recently, neural networks have shown impressive progress across diverse fields, with speech processing being no exception. However, recent breakthroughs in this area require extensive offline training using large datasets and tremendous computing resources. Unfortunately, these models struggle to retain their previously acquired knowledge when learning new tasks continually. In this paper, we investigate the problem of learning sequence-to-sequence models for spoken language understanding in a class-incremental learning (CIL) setting and we propose COCONUT, a CIL method that relies on the combination of experience replay and contrastive learning. Through a modified version of the standard supervised contrastive loss, COCONUT preserves the learned representations by pulling closer samples from the same class and pushing away the others. Moreover, we leverage a multimodal contrastive loss that helps the model learn more discriminative representations of the new data by aligning audio and text features. We also investigate different contrastive designs to combine the strengths of the contrastive loss with teacher-student architectures used for distillation. Experiments on two established SLU datasets reveal the effectiveness of our proposed approach and significant improvements over the baselines. We also show that COCONUT can be combined with methods that operate on the decoder side of the model, resulting in further metrics improvements.

## introduction with the rapid progress of intelligent voice-enabled personal assistants, the significance of spoken language understanding (slu) has gained substantial recognition in recent years @xcite . conventional slu models deploy a cascaded pipeline of an automatic speech recognition (asr) system followed by a natural language understanding (nlu) module @xcite . asr maps the input speech into text representations, and nlu extracts the target intent labels from the intermediate text. even though these approaches can leverage a vast abundance of asr and nlu data, they suffer from asr error propagation. conversely, end-to-end (e2e) slu @xcite @xcite has received more attention in recent research because it uses a single trainable model to map the speech audio directly to the intent labels, bypassing the text transcript and reducing latency and error propagation. the assumption that the data distribution the model will face after deployment aligns with what it encountered during the training phase is brittle and unrealistic. in fact, real-world scenarios entail evolving streams of data where novel categories (e.g., new vocabulary or intents) emerge sequentially, known as continual learning (cl). unfortunately, while neural networks thrive in a stationary environment, the situation is reversed in cl, resulting in the "catastrophic forgetting" (cf) of the existing knowledge in favor of fresh new information @xcite . although the majority of cl works have focused on computer vision tasks like image classification @xcite and semantic segmentation @xcite , a few works have recently turned their attention towards text @xcite and speech @xcite , as well as vision-language @xcite and vision-audio @xcite . while most slu works consider offline settings, a thorough study of slu under a class-incremental learning (cil) setup still lacks. in cil, one single model is adapted to a sequence of different tasks as incremental labels emerge sequentially. recently, @xcite studied the problem of cil in asr-slu, where slu is carried out in a sequence-to-sequence (seq2seq) fashion, thus computing the intent labels in an auto-regressive way together with the asr transcriptions. by doing this, the model comprises three blocks: text and audio encoders, and an asr decoder. while in that work the knowledge distillation (kd) principle applied to the asr decoder is used, in this paper, we exploit the multi-modal audio-text setting and propose coconut : continual contrastive spoken language understanding. coconut combines experience replay (er) and contrastive learning principles. whereas er is a well-established approach in cl, whereby a bunch of old training samples are collected into a dedicated rehearsal memory buffer and interleaved with the data from the new task @xcite , only recently has contrastive learning been harnessed to learn representations continually. both supervised @xcite and self-supervised @xcite contrastive learning have proven useful to lessen the cf issue. specifically, coconut relies on two contrastive learning-based losses that operate on a shared embedding space where the audio and text features are projected. the first loss coined negative-student positive-teacher (nspt), is a modified version of the supervised contrastive learning loss that aims to consolidate what the model has learned in the previous tasks. it also exploits kd @xcite to guide the current model (student) to produce representations that resemble the ones obtained with the model from the previous tasks (teacher). for this reason, this loss is computed only on the rehearsal data (i.e., the anchors). a key difference between our loss and the standard contrastive one is that the positive samples are computed using the teacher (the positives only come from the rehearsal data), whereas the negatives are computed with the student. in this way, we avoid stale and scattered representations for the new data. the second loss is inspired by the recent progress in multi-modal representation learning. considering that for audio-text paired data, audio and text represent the same information but in different ways, it has been shown that aligning their representations results in better performance for various speech-related problems @xcite @xcite . therefore, we propose a multi-modal (mm) supervised contrastive loss that, exclusively applied to the current task's data, brings audio and text representations belonging to the same class into closer proximity in the shared feature space, resulting in features that are more transferable and resilient to cf. an overview of coconut is illustrated in figure 1 . in summary, our contributions are three-fold: ❶ we introduce coconut , a cl method that makes use of two supervised contrastive learning objectives to mitigate cf for seq2seq slu models. in particular, through our proposed nspt loss we provide a detailed study of which models (student/teacher) should be used at the numerator/denominator (positives/negatives) of the contrastive loss tailored for class-incremental learning. ❷ we conduct extensive experiments on two popular slu benchmarks demonstrating that coconut achieves consistent improvements over the baselines. we also show that it can be combined with kd applied to the asr decoder, leading to further improvements. finally, ❸ we ablate the contribution of each loss and its components, showcasing their pivotal role in coconut. ## conclusion in this work, we study the problem of e2e slu using a seq-2-seq model for class-incremental learning. in order to mitigate catastrophic forgetting we propose coconut , a cl approach that exploits experience replay and contrastive learning paradigms. on the one hand, it preserves the previously learned feature representations via an ad-hoc supervised contrastive distillation loss, on the other it contributes to aligning audio and text representations, thus resulting in more transferable and robust to catastrophic forgetting representations. we show that coconut outperforms the other baselines and that synergizes with other kd techniques operating on the decoder side. we finally dissect the design choices of coconut through specific ablation studies, showcasing that each component is pivotal to attain the best results.