A da BERT - CTC : Leveraging BERT - CTC for Text-Only Domain Adaptation in ASR (2023)

Abstract: End-to-end (E2E) automatic speech recognition (ASR) models are becoming increasingly popular in commercial applications, such as virtual assistants, closed captioning, and dictation systems. The accuracy of the ASR is crucial to their success. However, E2E models still struggle to recognize out-of-domain words such as proper nouns and domain-specific terms. In this paper we introduce AdaBERT-CTC, a domain adaptation technique that relies solely on textual data. Our method allows for text-only adaptation by fine-tuning a pre-trained self-supervised text encoder model. Additionally, we show that our method can be made parameter-efficient by adding bottleneck adapters to the pre-trained model. This allows for adaptation with less than a 5% increase in parameters and minimal computational overhead during inference. We demonstrate that our approach outperforms the base BERT-CTC model by up to 14% relative word error rate improvement on several out-of-domain, publicly available datasets.

## introduction end-to-end (e2e) automatic speech recognition (asr) models such as connectionist temporal classification (ctc) @xcite have become popular due to their ability to map acoustic features to text sequences using a single model. these architectures do not require an acoustic model (am), language models (lm), nor explicit alignment information during training. these characteristics make them an attractive choice for large-scale production settings @xcite . however, performance deteriorates on out-of-domain datasets not seen during training @xcite , and adapting these models to out-of-domain data is a * work done while at aws ai labs. due to the aforementioned challenges, text-only adaptation methods are gaining popularity for asr. in @xcite , the authors use a separate text-encoder network and additional encoder layers on top of an acoustic encoder to infuse text information. in @xcite , the authors use random encoder features in place of real audio in a transducer encoder and showcase improvements using text-only data. in @xcite , the predictor network of the rnn-t model is adapted with text-only data using textogram representations. a different line of work where text-only adaptation is possible uses self-supervised learning (ssl) to train models with joint speech/text representations, such as joist @xcite , maestro @xcite , and mslam @xcite . even though these models show promising results in downstream tasks, text-only adaptation remains challenging. bridging the gap between supervised and ssl models, the authors in @xcite combined fully supervised architectures with bert @xcite . specifically in @xcite , the authors proposed a bert-ctc asr model that expands the ctc model by combining the acoustic representations and bert representations to perform asr using self-attention @xcite . the bert-ctc model conditions ctc outputs on bert embeddings, incorporating explicit linguistic information into training/inference, while maintaining an iterative, non-autoregressive decoding. in this paper, we present adabert-ctc, a method to adapt bert-ctc to out-of-domain data using text only. we do so by training bert with domain-specific text (fig. 1 ) using the bert-ctc loss. we further explore adding parameterefficient adapters @xcite to bert and train only these adapters during the text adaptation phase to make the approach parameter-efficient and deployment friendly @xcite . at inference time, we replace the original bert used to train the bert-ctc model with our adapted bert. the main contributions of our work are: (1) we propose a simple yet effective text-only domain adaptation method for asr that leverages the recently proposed bert-ctc architecture, (2) we explore the behaviour of the bert-ctc architecture when bert is further fine-tuned with domain data using masked language model (mlm) loss, (3) we demonstrate the advantages of parameter-efficient adapters to fine-tune the bert module, and (4) we present that our text-only domain adaptation approach complements the use of a domain-specific language model. ## adabert-ctc: adapted bert-ctc the bert-ctc model conditions the output prediction on both the acoustic information and the bert embeddings of the predicted sequence from the previous iteration. our adaption approach uses text to modify the bert module of a trained bert-ctc model without changing the other parameters. our adabert-ctc method is comprised of three steps: (1) training a base bert-ctc model with minimal changes to the original training method; (2) adapting only the bert model with our text-only adaptation methods while keeping the rest of the trained bert-ctc network frozen (which we call adapt-bert); and (3) at inference time, we replace the original bert model with our adapted bert model, making no other changes to the original bert-ctc inference framework. we call this method adabert-ctc. in the next subsections, these three steps are further described. ## experimental setup in this section, we describe the data and experiments performed to showcase the use of our pro-posed model and method. ## conclusions in this work, we propose adabert-ctc, a method for adapting bert-ctc using text-only data. the adaptation approach modifies a trained bert-ctc model by fine-tuning bert (while keeping everything else frozen) using bert-ctc loss with textonly input. our results show that we achieve up to 14% werr without the use of an external language model on publicly available datasets. furthermore, we show that adapting bert-ctc using our approach in a parameter-efficient manner with bottleneck adapters achieves comparable performance to fully fine-tuning bert. to understand if our method complements the use of an external language model, we show that combing adabert-ctc with sf improves gives a werr of 6.6% compared to bert-ctc with sf. for future work, we plan to evaluate our adaptation performance on real-world datasets.