Massive End-to-end Speech Recognition Models with Time Reduction (2024)

Abstract: We investigate massive end-to-end automatic speech recognition (ASR) models with efficiency improvements achieved by time reduction. The encoders of our models use the neural architecture of Google’s universal speech model (USM), with additional funnel pooling layers to significantly reduce the frame rate and speed up training and inference. We also explore a few practical methods to mitigate potential accuracy loss due to time reduction, while enjoying most efficiency gain. Our methods are demonstrated to work with both Connectionist Temporal Classification (CTC) and RNN-Transducer (RNN-T), with up to 2B model parameters, and over two domains. For a large-scale voice search recognition task, we perform extensive studies on vocabulary size, time reduction strategy, and its generalization performance on long-form test sets, and show that a 900M RNN-T is very tolerant to severe time reduction, with as low encoder output frame rate as 640ms. We also provide ablation studies on the Librispeech benchmark for important training hyperparameters and architecture designs, in training 600M RNN-T models at the frame rate of 160ms.

## introduction there has been a recent focus on scaling up endto-end (e2e) asr models, such as connectionist temporal classification (ctc, @xcite and neural transducers (rnn-t, @xcite , to extremely large sizes @xcite @xcite @xcite , to unlock new levels of accuracy on more challenging tasks. many of these models are motivated by the goal of training a single multilingual asr model which can perform well across a range of languages. for example, the universal speech model (usm, @xcite is a 2b parameter full-context ctc model trained on youtube data from 300+ languages; pratap et al. the present work follows in the footsteps of these previous works, and addresses the important practical challenges of training massive e2e models: as model size increases, the cost of computing the outputs of the encoders in the model (most of the model parameters are typically in the encoder) scales poorly in attention-based encoders such as transformers @xcite and conformers @xcite . the rnn-t model, in particular, requires additional computation and memory for the prediction and joint network, which further compounds the problem. in this work, we investigate the aforementioned issues and find that encoder output frame rate reduction can be applied repeatedly at multiple layers in the encoder to obtain a large output frame rate reduction; this in turn is critical for training models and performing inference efficiently. on a large scale voice search task with short queries, while past research incorporated time reduction to lower the final frame rate to 60ms @xcite , or 120ms @xcite for voice search we find that we can increase time reduction to 6x with large ctc, and all the way up to 16x with large rnn-t, of the 40ms base frame rate with funnel pooling @xcite , without sacrificing much accuracy. we also propose a few practical techniques to mitigate the potential accuracy loss due to time reduction while still enjoying most efficiency improvements. we provide ablation studies of these techniques on the librispeech benchmark, for training 600m rnn-t model with a frame rate of 160ms. our results show that time reduction is generally useful for asr in different domains. more information in recent overview articles @xcite . notations we assume that the input audio signal has been parameterized into suitable acoustic feature vectors: @xmath0, . . . , x t ], of length t , where x @xmath1, in this work). the input acoustic features are processed using an encoder which transforms the input into a higher-level representation: @xmath2, . . . , h t ], where the length of the encoded representation is typically shorter than the original input length (@xmath3. we assume that we have an input transcript corresponding to each utterance: @xmath4, y 1 , . . . , y u ], where each y @xmath5, the set of output symbols (word-pieces @xcite , in this work), and s represents a special start-of-sentence symbol. ## massive models with time reduction in this work, we boost the asr performance by tremendously scaling up the size of the e2e models. recognizing the encoder's crucial role in e2e speech recognition, we dramatically expand encoder parameters (up to 1.8b), enabling the model to effectively leverage the vast amount of training data and improve the recognition accuracy. the usm encoder of our e2e model is built on the convolution-augmented transformer @xcite , or conformer architecture. each conformer block consists of a feed-forward module (ffn), a multi-head self-attention module (mhsa), a convolution module (conv), and a second feed-forward module. if the input to l-th conformer block is x (l) , then its output (and, the input to the next block), x (l+1) , is computed as: for the standard conformer layer, the input length and output length are the same. ## related work in time reduction time reduction has been a useful technique for achieving a balance between input and output lengths for asr, and has evolved over time with the choice of neural architecture and modeling units. in @xcite ; @xcite , the lower frame rates were achieved by concatenating input frames and striding, for dnn or lstm models predicting context-dependent hmm states. @xcite ; @xcite ; @xcite proposed to use hierarchical/pyramidal rnns where outputs of consecutive steps are combined before feeding to the next layer. after the community switched to end-to-end systems and word-piece type modeling units, a popular frame rate is 40ms as achieved by convolutional subsampling, and is adopted by widely-used open-source libraries @xcite . in this work, we also use such a configuration for the input to our encoder, and perform funnel pooling atop. with the similar intuition of reducing sequence lengths at encoder layers, efficient conformer @xcite used convolution and strided attention to produce shortened sequences in a progressive manner. squeezeformer @xcite applied progressive downsampling in the decoder and included upsampling at the end to recover the original sequence length with a u-net-like design; a similar idea was studied in @xcite . compared to these prior work, we investigate the use of time reduction with much larger model sizes and more extreme reduction rates, on multiple datasets from different domains, and explore additional techniques to make up potential accuracy loss. @xcite explore extreme encoder output reduction rates for rnn-t on a large-scale voice search task with as low frame rate as 2.56 secs, by using funnel pooling at later encoder layers. another line of work @xcite @xcite proposed to use ctc to (irregularly) select encoder output frames (or to skip confident blank frames) for rnn-t modeling, effectively reducing the frame rate for the decoder. however, such a ctc-based encoder output selection does not save computations in the encoder, but only reduces the computation in downstream modules. our goal in this work is to aggressively reduce the sequence length early in the architecture for computational savings. ## experiments on voice search queries our first set of experiments focus on a short voice search task with in-house datasets. ## experiments on librispeech we now turn to the librispeech benchmark @xcite and provide ablation studies for other techniques that mitigate accuracy loss. all the encoders we considered have the same number of trainable parameters (626.6m), with 24 layers, encoder dimension 1024, and convolution kernel size of 5. the attention module has a left context of 129 tokens and a right context of 128 tokens. for funnel encoders, we used a total of 4x reduction, done through a 2x reduction in layer 4 and another in layer 5. for encoders with upsampling, a 2x or 4x upsampling is done in the last layer. we focus on rnn-t models in this section which are equipped with a single-layer lstm decoder, with an output vocabulary size of 1024. ## conclusions in this paper, we have investigated the use of time reduction by funnel pooling with massive e2e asr models. our experiments on a large-scale voice search task demonstrate its effectiveness with two major e2e asr models, ctc and rnn-t, and show that for large models, funnel pooling significantly reduces inference and training costs without sacrificing much accuracy, achieving encoder output frame rate as low as 640ms. we have similar success on the librispeech benchmark with a reduced frame rate of 160ms, given proper training hyperparameters and carefully designed architectures. we believe our techniques are generally useful for speech recognition and will further extend their usage to much longer audio.