Supervised Adaptation of Sequence-to-Sequence Speech Recognition Systems using Batch-Weighting (2020)

Abstract: When training speech recognition systems, one often faces the situation that sufficient amounts of training data for the language in question are available but only small amounts of data for the domain in question. This problem is even bigger for end-to-end speech recognition systems that only accept transcribed speech as training data, which is harder and more expensive to obtain than text data. In this paper we present experiments in adapting end-to-end speech recognition systems by a method which is called batch-weighting and which we contrast against regular fine-tuning, i.e., to continue to train existing neural speech recognition models on adaptation data. We perform experiments using theses techniques in adapting to topic, accent and vocabulary, showing that batch-weighting consistently outperforms fine-tuning. In order to show the generalization capabilities of batch-weighting we perform experiments in several languages, i.e., Arabic, English and German. Due to its relatively small computational requirements batch-weighting is a suitable technique for supervised life-long learning during the life-time of a speech recognition system, e.g., from user corrections.

## introduction when building an automatic speech recognition (asr) system for a specific domain, one is often faced with the fact that only very limited amounts of training data for the target domain are available. this problem has become bigger with the advent of end-to-end speech recognition systems. the old asr systems used the bayes theorem to solve the speech recognition with the help of an acoustic model (am) and a language model (lm). the acoustic model needed to be trained on transcribed speech recordings, the language model was trained on text only. therefore, topic adaptation, could be done with the help of textual training data only, by adapting or training a language model on topic specific data. as text only data is easier to come by than transcribed speech, topic adaptation was often feasible. for end-to-end speech recognition systems this option is no longer available, as they only accept transcribed speech as training data. however, transcribed speech for a specific domain is often more difficult to find than text data, thus making it more difficult or expensive to find or create fitting adaptation data. for hmm based asr systems that use gaussian mixture models (gmms) for estimating the emission probabilities of the hmm, several techniques for adapting to speakers or channels were available @xcite @xcite . these techniques often could also be applied in an unsupervised or semi-supervised manner during inference. for end-to-end asr systems such techniques need to be newly created. in this paper we are examining the use of fine-tuning for end-to-end asr for adapting them to different domains. we thereby examine different dimensions of domain adaptation, such as adapting to topics, accents and vocabulary. in the experiments we compare fine-tuning, i.e., continuing to train an end-to-end system on adaptation data, to a technique called batch-weighting, in which we mix adaptation data with the data for training the background model in a certain ratio at mini-batch level. batch-weighting was thereby inspired by a technique from machine translation @xcite and is explained in detail in section 5. we adapt this technique for automatic speech recognition. further, for the different dimensions of domain adaptation, we examine the fine-tuning of different parts of the end-to-end asr systems, e.g., only the encoder or only the decoder, in order to test the hypothesis that encoder layers are mainly con-cerned with learning features and acoustic properties, while the decoder models the linguistic properties of the recognizer's domain. we performed our experiments in several languages -arabic, english and german -and in different domain scenarios in order to show batchweighting's generalization capabilities to new adaptation scenarios. our experiments thereby show that batch-weighting consistently outperforms simple fine-tuning. we also report the computational time needed for performing batch-weighting in the different scenarios. the low computation times (less than six hours in all cases) makes this technique suitable for life-long learning, when small amounts of supervised adaptation data can be collected during the life of a system, e.g., by user corrections. ## related work there are a several studies about adaptation of neural machine translation (nmt) systems. @xcite divided nmt domain adaptation methods into four categories: data centric, training objective centric, architecture centric and decoding centric: chen et al. (2017) used sentence weighting for adaptation of part-of-speech (pos) tagging, a named entity (ne) recognition task. wang et al. (2017) used sentence weighting, domain weighting and batch weighting for nmt, and yan et al. (2019) used word weighting. (luong and manning, 2015; freitag and al-onaizan, 2016; neubig and hu, 2018) used fine-tuning, and chu and dabre (2019) used mixed fine-tuning, while kobus et al. (2016); chu et al. (2017) combined mixed fine-tuning and adding domain tag. in the area of end-to-end speech recognition @xcite trained a speech recognition system on a multi-domain corpus. by using a domain identification (di) vector derived from the activation of a bottle-neck layer in a domain classifying network they prime their speech recognition system to different domains present in the training data. the paper then shows improvements in word error rate (wer) when adapting the speech recognition system in an unsupervised manner using the di vector to a domain for which no training or adaptation data is available. in contrast in our experiments we work with small amounts of adaptation data that are not sufficient for training a complete system, but can be exploited for adapting an existing system. ## dimensions of adaption in our work we examine the adaptation of our system to different dimensions of variability. sometimes these different dimensions are subsumed under the term domain @xcite . however, on other occasions domain is used to describe the topic of speech data only. we will follow the latter use of domain in this paper and will explicitly address the different dimensions of the speech data for which we examined suitable adaptation techniques: topic adaptation, accent adaptation and vocabulary adaptation. ## data we conduct experiments on the languages english, german and arabic, in order to make sure that batch-weighting generalizes across languages. tables 1, 2 and 3 contain a summary of the speech data-sets that we used as general, out-of-domain training data-sets (out), and of the in-domain datasets (in) that match our target domain. ## experiments and results for all experiments, we first trained a general asr seq2seq model on the out-of-domain data-set, then adapted the model with the in-domain data-set. for the experiments in english topic adaptation and in german we used a transformer based seq2seq model @xcite , for arabic and english accent adaptation we used an encoder-decoder plus attention based system @xcite . for the adaptation we compare fine-tuning and batch-weighting @xcite . in the batchweighting strategy the training data-set is a combination of in-and out-of-domain data. to describe how the data of both data-sets is combined we report the out-of-domain ratio, i.e., the number of tokens in a mini batch from the out-of-domain dataset divided by the total number of tokens in the mini batch. a ratio of 0 is equivalent to conventional fine-tuning using only in-domain data and 1 is equivalent to training without in-domain data. furthermore, we combined these methods with freezing layers. we froze the encoder and all layers except the softmax-layer. for all the experiments we report the time for the adaptation. note that it is possible for methods with frozen layers to take longer compared to their counterparts with no frozen layers since different methods can require different amounts of update steps to obtain the best performance. english for english the baseline model achieved a word error rate (wer) of 11.0% on the out-of-domain test set, i.e., ted talks. on the out-of-domain test set, atis, the baseline system achieves a wer of 43.1%. ## conclusion in this paper we examined the supervised adaptation of end-to-end speech recognition systems on small amounts of adaptation data when large amounts of general, out-of-domain training data are available. we used a technique called batchweighting and contrasted it against regular finetuning, showing that batch-weighting delivers consistently better performance. for this we performed experiments on several dimensions of domain adaptation: topic, accent and vocabulary. we also performed experiments on three languages -arabic, english and germanto show that batch-weighting generalizes across different languages and scenarios. for a rule of thumb to choose a good mixing ratio further experiments have to be conducted. due to its comparatively short run-time and computational resources necessary batch-weighting is suitable for life-long learning of an asr systems during deployment, e.g., from user corrections.