INT apt: Information-Theoretic Adversarial Prompt Tuning for Enhanced Non-Native Speech Recognition (2023)

Abstract: Automatic Speech Recognition (ASR) systems have attained unprecedented performance with large speech models pre-trained based on self-supervised speech representation learning. However, these pre-trained speech models suffer from representational bias as they tend to better represent those prominent accents (i.e., native (L1) English accent) in the pre-training speech corpus than less represented accents, resulting in a deteriorated performance for non-native (L2) English accents. Although there have been some approaches to mitigate this issue, all of these methods require updating the pre-trained model weights. In this paper, we propose Information Theoretic Adversarial Prompt Tuning (INTapt), which introduces prompts concatenated to the original input that can re-modulate the attention of the pre-trained model such that the corresponding input resembles a native (L1) English speech without updating the backbone weights. INTapt is trained simultaneously in the following two manners: (1) adversarial training to reduce accent feature dependence between the original input and the prompt-concatenated input and (2) training to minimize CTC loss for improving ASR performance to a prompt-concatenated input. Experimental results show that INTapt improves the performance of L2 English and increases feature similarity between L2 and L1 accents.

## introduction self-supervised learning has improved input data representation without requiring extensive humanlabeled data @xcite . based on this advancement, powerful pre-trained models providing high-performing representations for various data types (e.g., text, images, and audio) have been proposed. for instance, in speech, self-supervised pre-trained models such as hu-bert @xcite art performance of automatic speech recognition (asr). however, one major challenge in using pretrained speech models for asr is the representational bias towards prominent accents present in the dataset during pre-training. consequently, there will be a disparity in asr performance between native and non-native speakers. more specifically, pre-training using a large dataset such as the lib-rispeech @xcite , which comprises a large proportion of utterances from native (l1) english speakers, leads to a less satisfactory recognition rate for non-native (l2) english accented speech. this phenomenon can curtail the effectiveness of current high-performing asr systems for real-world applications. there have been several ways to address this issue, including fine-tuning the model on diverse accents @xcite , having a separate model for each accent @xcite or using regularization losses that guide the fine-tuning process to achieve robustness to accents @xcite , all of which require updating the pre-trained model. we propose a different solution for improving l2 speech recognition in transformer-based speech models that introduces a small number of learnable parameters into the input space while keeping the backbone weights of the model untouched. our approach is guided by information-theoretic adversarial learning; thus, we refer to it as in-tapt (information-theoretic adversarial prompt tuning). intapt aims to introduce auxiliary embeddings (i.e., prompt) concatenated to the original input, which can re-modulate the attention and adapt the pre-trained weights so that the corresponding input looks like speech with an accent seen during pre-training (figure 1 ). to achieve this, intapt incorporates (1) adversarial training, which tries to minimize the mutual information between the accent feature of the original input and that obtained by concatenating the prompt embeddings in front of the initial input, and (2) ctc loss training to improve the asr performance of the prompt-concatenated input. essentially the prompt is trained such that the accent of the concatenation is pushed away from the input accent and the concatenation achieves native ctc loss performance. unlike the previous use-case of prompts in nlp or computer vision (cv), where a single prompt embedding is learned for each discrete task or input domain, the intensity of an accent is continuous. thus, we propose an input-dependent prompt embedding by training a prompt generator that outputs an input-specific prompt. through extensive experiments, we show that the proposed dual objectives of intapt not only lead to better performance on l2 english accents but result in a higher similarity between the accent feature of the promptconcatenated input and that of l1 english accents. ## intapt figure 2 depicts the overall process of intapt. in-tapt incorporates a two-step training process. in the first step, we train an accent module (am) capable of isolating the accent feature from a given audio feature a of an input speech x. in the second step, we train a prompt generator (pg), which outputs a prompt p for a given audio feature a, using two objectives: (1) minimize the mutual information between the accent feature z ′ and z, where the former is obtained using the prompt-concatenated input (p; a) and the latter is obtained from the original audio feature a, (2) minimize ctc loss to improve the asr performance of the input (p; a). ## ablation study we analyze whether intapt allows the l2 speech input to resemble the accent of l1 speech (figure 3 ). using the accent module, we extract the l1 accent feature, and l2 accent features obtained using the backbone model, prompt ctc , and intapt. intapt showed the highest cosine similarity for all l2 groups, meaning that intapt effectively adjusts the attention of the pre-trained model so that l2 speech resemble l1 speech in terms of accent. ## conclusion we introduced information theoretic adversarial prompt tuning (intapt) for improving non-native asr performance. to achieve this, intapt remodulates the attention of the pre-trained speech models by concatenating input-dependent prompt embeddings to the original input, without updating the model weights. throughout the experiment, we show that intapt is capable of outperforming standard finetuning of the pre-trained model on l2 speech, without degradation on l1 speech, by allowing the l2 input to resemble a l1 accent.