A Simple Baseline for Domain Adaptation in End to End ASR Systems Using Synthetic Data (2022)

Abstract: Automatic Speech Recognition(ASR) has been dominated by deep learning-based end-to-end speech recognition models. These approaches require large amounts of labeled data in the form of audio-text pairs. Moreover, these models are more susceptible to domain shift as compared to traditional models. It is common practice to train generic ASR models and then adapt them to target domains using comparatively smaller data sets. We consider a more extreme case of domain adaptation where text-only corpus is available. In this work, we propose a simple baseline technique for domain adaptation in end-to-end speech recognition models. We convert the text-only corpus to audio data using single speaker Text to Speech (TTS) engine. The parallel data in the target domain is then used to fine-tune the final dense layer of generic ASR models. We show that single speaker synthetic TTS data coupled with final dense layer only fine-tuning provides reasonable improvements in word error rates. We use text data from address and e-commerce search domains to show the effectiveness of our low-cost baseline approach on CTC and attention-based models.

## introduction end-to-end speech recognition models simplify the speech recognition process by folding multiple components into a single model. these models directly convert the speech utterance into the spoken text @xcite . the major end-to-end architectures include ctc-based, attention-based, and transducer-based approaches @xcite @xcite . these all neural approaches are competitive in terms of performance however they require a large amount of supervised data to achieve generalization. a model trained on a single application domain doesn't work well on other target domains. examples of such applications domains include e-commerce, voice search, medical, etc. since it is not feasible to prepare supervised data for all the application domains, it is common to train models on large out of domain corpus followed by a small amount of in-domain finetuning @xcite . however, this approach still requires the availability of small labeled data. in the most basic form, the unlabelled text data from the target domain can be used to build domain-specific language models (lms). the domain lms are combined with the end-to-end asr model using shallow fusion @xcite @xcite . this approach has limited benefits since the main asr model is not tuned to the target domain. another popular technique is to prepare synthetic data using a text to speech (tts) system and the target domain text data @xcite . this requires a sophisticated multi-speaker tts system followed by the addition of representative noise to make the data usable. the idea is to make synthetic data as close as the real-world data. however, this approach is prone to overfitting as the synthetic data does not exactly resemble real-world noisy conditions. different fine-tuning approaches have been explored using synthetic data to alleviate the overfitting problem. in this work, we are concerned with domain adaptation techniques when a text-only corpus from the target domain is available @xcite . we present a simple baseline approach using single speaker synthetic tts data followed by final dense layer only fine-tuning. the synthetic data is created using a single speaker tts system which is commonly available and also easier to build in-house. the data is not subjected to any noise and is directly used to fine-tune the neural network. although such single speaker data is easy to build it is not usable for the training of endto-end networks. we, therefore, propose denseonly fine-tuning for effective fine-tuning. the approach solely relies on final dense layer fine-tuning to avoid over-fitting on single speaker and acoustic conditions. we refer to the dense layer projecting the intermediate embedding onto vocabulary space as the final dense layer. since the acoustic encoder of the neural network is frozen, the network only learns about the linguistic characteristic of the target domain. similar approaches have been explored in literature where only the decoder part of the neural network is fine-tuned. however, this approach is not applicable to ctc-based neural networks @xcite which do not follow an encoder-decoder architecture. we present our approach in the context of ctc and listen-attend-spell (las) based neural network architectures. for las-based network, we also compare dense only and decoder only fine-tuning. we consider the text from address (for delivery of e-commerce products) domain and voice search (of e-commerce products) domain @xcite for fine-tuning the model trained on a generic multidomain dataset. although encoder only fine-tuning has been widely studied in the literature @xcite , this is the first work to exploit denseonly fine-tuning which is more relevant to the ctcbased systems. moreover, we demonstrate a way to build an asr system for the address domain which is not explored in the literature. ## related work our work is at the intersection of data augmentation using the tts system and domain adaptation. in this section, we review the recent work in these two areas. the synthetic data generated using the tts system was used to improve the recognition of out of vocabulary (oov) words in @xcite . both synthetic data containing oov words and original data were used together to train the best rnn-t model. encoder freezing and elastic weight consolidation were further shown to provide extra benefits. similarly, @xcite ) used a tts system to generate numeric training data and improve the asr performance on the out of vocabulary numeric sequences. the importance of data augmentation over semi-supervised learning was shown in @xcite . in this work, the tts system was trained on the same supervised asr data set and used to generate synthesized samples on a wider set. the work also highlights the importance of multi-speaker tts systems and noise addition to build usable systems. other data augmentation techniques like spec augment @xcite were shown to be complementary with tts based augmentation in @xcite . effective training strategies for using synthetic data were proposed in @xcite . in order to avoid catastrophic forgetting, multi-stage training was used. the encoder layers were frozen in the initial stage followed by full fine-tuning in later stages. an elastic penalty was also added to the loss function so to avoid large deviation in learned parameters. similar approaches have been proposed for domain adaptation as well with a bais towards finetuning based transfer learning approaches. an lstm-based domain classifier was trained to select an appropriate domain adapted language model in @xcite . the corresponding domainspecific language model was used for second pass re-scoring. the transfer learning approaches for domain adaptation and cross-language adaptation were evaluated in @xcite . they compare the fine-tuning of the pre-trained quartznet model with the corresponding model trained from scratch. they concluded that large pre-trained models performed better than small pre-trained models and the models trained from scratch. another form of transfer learning involves partial fine-tuning of the model instead of the entire model. the decoder only fine-tuning for domain adaptation in listen-attend-spell (las) based model was evaluated in @xcite . the model is first trained on the source domain followed by decoder only finetuning on the target domain. the partial fine-tuning is shown to work better than the full fine-tuning and from the models trained from scratch. an adaptation technique specific to rnn-t networks using text-only data was proposed in @xcite . the prediction network of rnn-t is viewed as a neural language model and is adapted using text-only corpus while keeping the encoder and joint network fixed. another approach for adapting rnn-t network using text-only data was proposed in @xcite . the fine-tuning of prediction and the joint network was performed using synthetic tts domain-specific data. partial fine-tuning was shown to work better than full fine-tuning approaches. these works mainly used rnnt-based systems and employ a multi-context multi-speaker tts system. in this work, we use a single speaker tts system with a focus on ctc and attentionbased models. moreover, we focus on dense only fine-tuning instead of decoder fine-tuning studied in these works. ## methodology the flow of our process is depicted in figure 1 . we follow a simple pre-training and fine-tuning approach. the model is first trained on general out-of-domain data. the target domain text data is converted into audio using a single speaker tts engine. the synthetic samples are then used to fine-tune the final dense layers of asr models. we consider two model types i.e ctc based models and attention-based models. the model architecture and tts system description are provided in the following sub-sections. ## conclusion in conclusion, we demonstrate a simple baseline approach for domain adaptation using a text-only corpus from the target domain. we show that the final dense layer only fine-tuning using single speaker tts data provides considerable improvements over the generic model. the results are shown on two different domains of voice search and address domain. for both ctc and attention-based models we show that dense-only fine-tuning is a reasonable approach for domain adaptation. although the technique is more relevant to ctc-based models it can also be used with encoder-decoder type models. for encoder-decoder models, the decoder only fine-tuning performs better.