A Unified Speaker Adaptation Approach for ASR (2021)

Abstract: Transformer models have been used in automatic speech recognition (ASR) successfully and yields state-of-the-art results. However, its performance is still affected by speaker mismatch between training and test data. Further finetuning a trained model with target speaker data is the most natural approach for adaptation, but it takes a lot of compute and may cause catastrophic forgetting to the existing speakers. In this work, we propose a unified speaker adaptation approach consisting of feature adaptation and model adaptation. For feature adaptation, we employ a speaker-aware persistent memory model which generalizes better to unseen test speakers by making use of speaker i-vectors to form a persistent memory. For model adaptation, we use a novel gradual pruning method to adapt to target speakers without changing the model architecture, which to the best of our knowledge, has never been explored in ASR. Specifically, we gradually prune less contributing parameters on model encoder to a certain sparsity level, and use the pruned parameters for adaptation, while freezing the unpruned parameters to keep the original model performance. We conduct experiments on the Librispeech dataset. Our proposed approach brings relative 2.74-6.52% word error rate (WER) reduction on general speaker adaptation. On target speaker adaptation, our method outperforms the baseline with up to 20.58% relative WER reduction, and surpasses the finetuning method by up to relative 2.54%. Besides, with extremely low-resource adaptation data (e.g., 1 utterance), our method could improve the WER by relative 6.53% with only a few epochs of training.

## introduction end-to-end models yield state-of-the-art performance on automatic speech recognition (asr) in the past decade, such as connectionist temporal classification (ctc) model @xcite , attention-based encoder-decoder model @xcite , recurrent neural network transducer (rnn-t) @xcite , transformer model @xcite and conformer model @xcite . however, model performance deteriorates due to speaker mismatch between training and test data. given the target speaker, finetuning the trained model could alleviate the speaker mismatch problem to some extent, but finetuning the entire model requires large amounts of compute to be effective, and it could in turn bring catastrophic forgetting @xcite to the existing speakers. currently, there are two lines of studies to address the speaker mismatch problem in neural network based models. one category is working on the acoustic features, i.e., either by normalizing acoustic features to be speaker-independent @xcite @xcite or by introducing additional speaker related knowledge (e.g., i-vector) to adapt the acoustic model @xcite @xcite . a summary vector of each utterance can be trained to replace speaker i-vector @xcite . to adapt to acoustic variability, @xcite add shifting and scaling parameters in the layernormalization layer. the other category belongs to model adaptation, i.e., to train the speaker-dependent model from speaker-independent model parameters with extra adaptation data. to avoid overfitting, techniques such as l2 regularization @xcite , kullback-leibler divergence @xcite and adversarial multitask learning @xcite have been used. because finetuning the entire model is computationally expensive, @xcite ; @xcite ; @xcite only adapt specific layers or a subset of parame-ters. in particular, @xcite ; samarakoon and sim (2016b); @xcite reparameterize each hidden unit with speaker-dependent amplitude function in fully-connected or convolutional neural network layers. however, it is difficult to determine which model parameters to adapt for target speaker, and choosing certain sub-layer(s) intuitively may not be optimal. in this work, we propose a unified speaker adaptation model by making use of both feature adaptation and model adaptation. for feature adaptation, we propose the speaker-aware persistent memory model to generalize better to unseen test speakers. in particular, speaker i-vectors from the training data are sampled and concatenated to speech utterances in each encoder layer, and the speaker knowledge is learnt through attention computation with speaker i-vectors. our method learns utterance level speaker knowledge, which is more effective than learning time step dependent speaker knowledge @xcite since it is more robust to various variability factors along an utterance. for model adaptation, we explore gradual pruning @xcite , which to the best of our knowledge, is the first time being studied for speaker adaptation. we gradually prune less contributing parameters on model encoder, and then use the pruned parameters for target speaker adaptation while freeze the unpruned parameters to retain the model performance on general speaker data. in this way, our model could adapt to target speakers very fast by updating only a small percentage (10%) of encoder parameters, and it does not change the model architecture. freezing unpruned parameters alleviates the catastrophic forgetting problem as well. our proposed approach brings relative 2.74-6.52% wer reduction on general speaker adaptation. on target speaker adaptation, our method outperforms the baseline with up to 20.58% relative wer reduction, and surpasses the finetuning method by up to relative 2.54% 1 . ## proposed method we propose an efficient speaker adaptation model by making use of both feature adaptation and model adaptation. for feature adaptation, we embed speaker knowledge represented by a number of fixed speaker i-vectors into each input utterance @xcite . this aims to capture speaker information through attention computation between each utterance and speaker i-vectors. for model adaptation, an effective method is employed that can adapt to target speaker very fast without sacrificing performance on existing speakers. in particular, we prune the model gradually and finetune a small subset of parameters to be speaker-specific. ## experiments in this section, we present our experiments using the proposed speaker-aware persistent memory model and the gradual pruning method. ## analysis in this section, we revisit our approach to reveal more details and explore the effectiveness of the gradual pruning method in combination with the speaker-aware persistent memory model. ## conclusion in this paper, we have proposed a unified speaker adaptation approach consisting of feature adaptation and model adaptation. speaker-aware persistent memory model makes use of speaker i-vectors to adapt at the feature level, and we use the gradual pruning approach to retrieve a subset of model parameters for adaptation at the model level. gradual pruning is found to be better than one-time pruning because gradual pruning could iteratively learn based on pruned parameters. it can alleviate catastrophic forgetting problem as well by retaining a subnetwork whose performance matches the original network. we find that our proposed method is effective in both general speaker adaptation and specific target speaker adaptation. in particular, our method brings relative 2.74-6.52% wer reduction on general speaker adaptation, and outperforms the baseline with up to 20.58% relative wer reduction on target speaker adaptation. even with extremely low-resource adaptation data, our method could bring 6.53% relative improvement with only a few training epochs. in the future, we are interested in the overfitting issue with low-resource data, as well as multi-speaker adaptation with our method.