Joint Audio/Text Training for Transformer Rescorer of Streaming Speech Recognition (2022)

Abstract: Recently, there has been an increasing interest in two-pass streaming end-to-end speech recognition (ASR) that incorporates a 2nd-pass rescoring model on top of the conventional 1st-pass streaming ASR model to improve recognition accuracy while keeping latency low. One of the latest 2nd-pass rescoring model, Transformer Rescorer, takes the n-best initial outputs and audio embeddings from the 1st-pass model, and then choose the best output by re-scoring the n-best initial outputs. However, training this Transformer Rescorer requires expensive paired audio-text training data because the model uses audio embeddings as input. In this work, we present our Joint Audio/Text training method for Transformer Rescorer, to leverage unpaired text-only data which is relatively cheaper than paired audio-text data. We evaluate Transformer Rescorer with our Joint Audio/Text training on Librispeech dataset as well as our large-scale in-house dataset and show that our training method can improve word error rate (WER) significantly compared to standard Transformer Rescorer without requiring any extra model parameters or latency.

## introduction streaming end-to-end automatic speech recognition (asr) models aim to transcribe the user's voice with minimal latency and have been widely used in numerous interactive asr applications that support direct user interaction in real-time. unlike non-streaming end-to-end asr @xcite @xcite @xcite , streaming endto-end asr, such as rnn-t @xcite @xcite @xcite , are limited to use short audio context or not use future context to satisfy low latency constraints and suffer from higher word error rates (wer). to address this issue of streaming asr, a two-pass architectures has been recently proposed to improve wer while keeping latency low @xcite @xcite . the main idea of the two-pass architectures is to use a non-streaming model, so-called rescorer, (2ndpass) on top of the conventional streaming model (1st-pass) to re-score and choose the best output among the initial n-best outputs generated from 1stpass model. specifically, the latest transformerbased @xcite rescorer @xcite has been shown promising results in both accuracy and latency improvement. the rescorer can improve wer and keep low latency because 1) it does not need to perform expensive beam search decoding process which is done already with 1stpass model, and 2) it exploits full context of audio embeddings generated from the 1st-pass model. while there are performance advantages to exploit full context of audio embeddings as input, training transformer rescorer needs expensive paired audio/text data similar to asr training and thus cannot leverage unpaired text-only data. many previous studies have investigated approaches to leverage unpaired text-only data including fusion with an external neural network language model (nnlm) @xcite ; kannan et al., 2018; sriram et al., 2017; shan et al., 2019; toshniwal et al., 2018; kannan et al., 2018; mcdermott et al., 2019; variani et al., 2020 @xcite have been proposed @xcite @xcite . in this work, we present our joint audio/text training method for transformer rescorer to leverage unpaired text-only data without requiring any extra parameter or latency. unlike previous studies @xcite , our method does not need to use text-to-speech (tts) or prior information (domain id) or tuning parameter, and it is based on transformer decoder. we evaluate our transformer rescorer with our joint audio/text training method on librispeech dataset as well as large-scale in-house dataset and show that our model significantly outperforms over transformer rescorer with standard training method wer without requiring extra training parameters or computational latency. ## transformer rescorer the architecture of the two-pass system @xcite consists of streaming 1st pass rnn-t model @xcite and nonstreaming 2nd pass transformer rescorer @xcite is illustrated in figure 2. then, transformer rescorer is trained with audio embeddings h generated from the fixed encoder of rnn-t and true transcription text y to maximize u logp (y u |h, @xmath0. @xcite . in step 2, transformer rescorer is based on transformer decoder setup @xcite , and the true transcription, y, is the query and audio embeddings, h, is the key in cross attention. ## joint audio/text training our joint audio/text training method aims to leverage large amount of unpaired text-only data without requiring extra model parameter or latency. to do so, we allow the model to be trained on either 1) paired audio/text data where both modality inputs are available, or 2) unpaired text data where only text inputs are available. our training method is also two-step as described in 2. in step 2, for training on the unpaired textonly example, we use the estimated averaged audio embeddings, h avg , of paired audio/text data in the training set. this averaged audio embeddings, h avg , can be simply obtained by passing a 0-dummy audio sequence to the well-trained encoder of rnn-t from the step 1. our transformer rescorer takes this averaged audio embeddings instead as the keys and values in cross attention. unlike @xcite , our approach is based on transformer architecture and it does not need to change the original objective function, nor does it require a tuning parameter for multiple losses or any prior knowledge of inputs. figure 2 illustrates our proposed joint audio/text training method for transformer rescorer. note that the inference process is the same as general rescorer as described in 2. ## conclusions we have introduced joint audio/text training method for transformer rescorer of the streaming two-pass end-to-end asr. unlike standard training method for transformer rescorer, our method can leverage unpaired text-only data and consequently improves recognition accuracy without requiring extra model parameters or computational latency. we evaluated our approach on the librispeech dataset as well as large-scale in-house dataset and showed that transformer rescorer with our proposed method obtained 3% -7% relative improvement in wer compared to the standard transformer rescorer model.