Residual Adapters for Parameter-Efficient ASR Adaptation to Atypical and Accented Speech (2021)

Abstract: Automatic Speech Recognition (ASR) systems are often optimized to work best for speakers with canonical speech patterns. Unfortunately, these systems perform poorly when tested on atypical speech and heavily accented speech. It has previously been shown that personalization through model fine-tuning substantially improves performance. However, maintaining such large models per speaker is costly and difficult to scale. We show that by adding a relatively small number of extra parameters to the encoder layers via so-called residual adapter, we can achieve similar adaptation gains compared to model fine-tuning, while only updating a tiny fraction (less than 0.5%) of the model parameters. We demonstrate this on two speech adaptation tasks (atypical and accented speech) and for two state-of-the-art ASR architectures.

## introduction automatic speech recognition (asr) systems have achieved great success on a diverse set of acoustic and linguistic conditions, domains and speech patterns. state-of-the-art asr systems are typically trained on tens of thousands of hours of speech data, and they perform well as long as these domains and conditions are well represented in the training data. understandably, the distribution of such data typically focuses on the canonical and typical spoken language patterns of the target language, i.e., regional dialects, common accents and frequent non-native accents. as a result, these systems may perform poorly on the tail of the distribution which may include "heavily" accented speech and/or speech with atypical speech patterns @xcite . atypical speech includes dysarthric speech, speech impairments (due to, for example, als, stroke, traumatic brain injury, down syndrome, cerebral palsy, and ms), stuttering, deaf * equal contribution speech, or severe hyper-nasality due to cleft lip and palate. the lack of sufficient training data for these accents and atypical speech in the training distribution may result in a poor experience for a large segment of the population, leaving the less fortunate communities behind when it comes to speech-enabled technologies @xcite . studies on accented speech showed word error rates (wer) twice or three times as high for accented speech compared to the more standard us accent @xcite . even worse performance is observed for speakers with speech impairments @xcite . our goal, in this paper, is to efficiently build scalable models that can adapt to non-canonical or atypical speech. it has been shown that speech models originally developed for typical speech can be successfully fine-tuned with limited amounts of data to accented or impaired speech @xcite @xcite @xcite @xcite . nevertheless, one major challenge with adapting models to either individuals or small groups of speakers is that it is necessary to scale the number of models that need to be maintained and hosted. for example, for smart devices powerful enough to run asr models on-device, having to deploy and store an additional (potentially large) model may take up valuable on-device resources. similarly, providing personalized models for a large population of speakers in a centralized/server-based scenario is not feasible. we propose to mitigate this issue by injecting residual adapter layers into the architecture. particularly, we use a bottleneck architecture that requires a tiny number of parameters (< 0.5% in our scenario) compared to the full model update via fine-tuning. then, while keeping the original pretrained model parameters frozen, we update only the parameters of the adapter layers as we train on the custom data of interest. this provides an easy way to deploy and store adapted models: a (generic) base model is deployed to all clients, and each individual or group can receive a personalized set of trained adapter layers that is small in size. the main contributions of the paper are as follows. we show that residual adapters work extremely well for acoustic adaptation of different speech models. we present extensive experiments with adapter layers in two very different asr usecases: personalized models for atypical speech, and group models for accented speech. we also demonstrate that adapter layers work well in two different, state-of-the-art end-to-end asr architectures, neural network transducers (rnn-t), and transformer transducers (t-t). this emphasizes the flexibility of this approach and its suitability as a standard alternative to full fine-tuning for arbitrary models. our results clearly demonstrate how adaptation via adapter layers solves the issue of parameter inefficiency while largely retaining the significant adaptation gains achievable through model adaptation with in-domain data. ## related work model fine-tuning has been successfully applied to domain adaptation for a variety of nlp tasks @xcite , machine translation @xcite , and speech recognition and conversation systems, including accented and atypical speech @xcite @xcite @xcite a major disadvantage of model fine-tuning is its parameter inefficiency since it retrains all (or a large portion of) the model parameters on given task-or domain-specific data, resulting in a copy of the model for that task/domain. this is especially problematic for personalization of models due to the resulting high number of specialized models. concatenating input features and speakerdependent vectors, such as i-vectors, is a parameterefficient speaker adaptive approach that has been applied to both acoustic models as well as endto-end asr models @xcite . however, only moderate improvements have been achieved, even on typical speech. we speculate that such a static, low dimensional representation may not be sufficient to capture the complex acousticphonetic patterns (e.g., consonant dropping and vowel dropping, extreme vowel reduction or length-ening, missing phonemes and even syllables, very irregular speaking rate and rhythm) often found in impaired speech. residual adapters were originally introduced by @xcite for computer vision tasks as an alternative to fine-tuning. these first residual adapter modules consisted of a single projection layer added between layers of a pre-trained network. @xcite proposed a variation consisting of a bottleneck structure (downprojection through feed forward layer, relu, upprojection) for task-specific adaptation of bert models. adapter modules were added after each sub-layer within a transformer layer, and the weights of the residual adapters as well as existing layer normalization parameters were updated during training. finally, @xcite have formulated a simplification of residual adapters in the context of domain-adaptation for machine translation. each residual adapter module has its own layer normalization block, followed by a down-and up-projection feed forward network. they argued that by including layer normalization in the residual adapter block, these modules are plug-able into arbitrary blocks of pre-trained modules because they learn the activation pattern of the layer into which they are injected. @xcite proposed to use adapters on top of multi-lingual asr models to further improve their performance (they report up to 9% wer improvement for some of the 5 languages of the multilingual model). our focus is different in that we consider residual adapters in speech personalization scenarios where the number of adapted models is several orders of magnitudes higher (e.g., tens of thousand of speakers with atypical speech and potentially hundreds of accents and dialects) and also not static (e.g. speech impairments often progress over time). learning hidden unit contribution (lhcu) @xcite is another approach to more parameter efficient speaker adaptation. instead of updating all weights of a model, lhuc adds learned factors to the output of each hidden unit modulating their amplitude. however, @xcite have shown that using residual adapters is much more effective. ## methods for our experiments, we chose two state-of-the-art end-to-end asr architectures: the recurrent neural network transducers (rnn-t) @xcite @xcite and the transformer transducer (t-t) @xcite . both architectures enable deployment on mobile devices, support streaming @xcite , and have demonstrated high performance. both architectures consist of three main components: an encoder, a prediction network that incorporates label history and serves as a language model component (decoder), and a joint layer that combines predictions made by the encoder and the prediction network and feeds into a softmax. all components of the two architectures are identical except the encoder stack. the prediction network consists of 2 uni-directional lstm layers. inputs are 128-dimensional log mel features computed every 10 milliseconds. 4 consecutive features are stacked with a stride of 3 frames to yield a 512dimensional input to the encoder every 30 milliseconds. our output vocabulary consists of 4096 word piece tokens. figure 1a shows a high-level overview of both architectures. for rnn-t, the encoder consists of 8 lstm layers; for t-t, we use 15 transformer layers in the encoder. both architectures are trained with the rnn-t loss @xcite . to make t-t streamable, the attention calculation pays attention to past contexts only, which makes this architecture analogous to a uni-directional rnn. we propose to utilize residual adapter modules as outlined by @xcite for our adaptation approach. each residual adapter block starts with layer normalization applied to the inputs, followed by a feed-forward layer with downprojection to dimension d b , a non-linear activation (relu), and another feed-forward layer with upprojection to the original input dimension d i . all weights of the residual adapter module are randomly initialized. figure 1b shows such a residual adapter module and its integration within the transformer encoder. we add residual adapters to each encoder layer, resulting in 8 adapter layers for rnn-t and 15 adapter layers for t-t. the bottleneck dimension d b enables control of the number of parameters of each residual adapter module and thus the capacity available during adaptation. ## experiments we analyze the performance of residual adapters as an alternative to model fine-tuning in two scenarios: adaptation to (a) atypical speech and (b) accented speech. for atypical speech, we build a personalized, speaker-dependent model for each speaker based on their data. for accented speech, we build per-accent models (i.e. speaker-independent models) and also experiment with a multi-accent adaptation scenario where one model is used for all covered accents. we conduct experiments using both asr transducer architectures (rnn-t and t-t). ## conclusions and future work in this work, we have shown that adaptation of asr models using residual adapter layers leads to substantial wer improvements over unadapted models across two tasks: atypical speech (up to 77% relative wer reduction) and accented speech (up to 31% relative reduction) and in two architectures (rnn-t and t-t). in comparison, finetuning the entire encoder for each speaker or accent yields only small improvements compared to residual adapter training. while similar in adaptation performance, residual adapters are much more parameter efficient than model fine-tuning. in our scenario, using residual adapters on each encoder layer, less than 0.5% of the overall model parameters need to be trained and maintained per speaker or accent. on the other hand, fine-tuning the entire encoder affects over 80% of the model parameters. in addition to substantially improved parameter efficiency, we also observed a dramatic training time speed up of about 40% due to the reduced number of parameter updates. overall, these findings demonstrate a feasible and scalable solution for personalized, speakerdependent models as well as domain-specific or dialect/accent-focused models. in future work, we plan to study to which encoder layers we need to add adapters for best performance and to potentially make residual adapters even more parameter efficient. similarly, we plan to apply residual adapters with different bottleneck dimensions depending on the position in the encoder layer stack (bottom and middle layers likely require larger, top layers smaller capacity). finally, we also plan to directly compare the effectiveness of residual adapters to approaches using statically fed speaker-dependent vectors for speaker adaptation, especially in the context of accent adaptation.