Fast Streaming Transducer ASR Prototyping via Knowledge Distillation with Whisper (2024)

Abstract: The training of automatic speech recognition (ASR) with little to no supervised data remains an open question. In this work, we demonstrate that streaming Transformer-Transducer (TT) models can be trained from scratch in consumer and accessible GPUs in their entirety with pseudo-labeled (PL) speech from foundational speech models (FSM). This allows training a robust ASR model just in one stage and does not require large data and computational budget compared to the two-step scenario with pre-training and fine-tuning. We perform a comprehensive ablation on different aspects of PL-based streaming TT models such as the impact of (1) shallow fusion of n-gram LMs, (2) contextual biasing with named entities, (3) chunk-wise decoding for low-latency streaming applications, and (4) TT overall performance as the function of the FSM size. Our results demonstrate that TT can be trained from scratch without supervised data, even with very noisy PLs. We validate the proposed framework on 6 languages from CommonVoice and propose multiple heuristics to filter out hallucinated PLs.

## introduction there are many challenges when developing automatic speech recognition (asr) engines for industrial applications, including (1) large-scale databases that generalize across multiple domains; the transformer-transducer architecture @xcite is widely exploited for industrial uses that require streaming decoding because the transducer decoder naturally supports streaming @xcite . however, the transducer used to be harder to train compared to aed and ctc, thus, it was less explored in the community, until it was shown to achieve a performance as close as aed models @xcite . the transducer models consist of an encoder, predictor and joint networks. using a transformer @xcite encoder leads to a transformer-transducer (tt) architecture @xcite @xcite . when trained from scratch, the tt models require sufficient amounts of supervised datasets in the target language and domain @xcite . at the same time, fine-tuning a large pre-trained model, even when using a transducer decoder, would not allow streaming decoding. in this work, we focus on two questions partly unanswered by the research community: (1) could we quickly prototype a streaming tt model on a single accessible gpu? (2) can we train tt models with only pseudo-labeled (pl) data? we target the streaming scenario, which is by nature more challenging than standard offline (full attention) decoding @xcite . despite the robustness of aed models in the offline scenario, they still require a large amount of supervised data. here, we use tt models @xcite , where the challenge arises on the fact that these do not include a self-supervised stage, 1 i.e., needing audiotext pairs always. we demonstrate that tt models can be trained entirely from scratch with pls generated from whisper @xcite while attaining competitive performance in streaming scenarios. the overall proposed approach is illustrated in figure 1 . ## related work developing robust asr systems for low-latency online settings with little to no supervised data is still an open challenge in the community. in this section, we introduce the most prominent approaches to overcome these issues. ## experimental setup this section describes the datasets, tt architecture, details for training with pseudo-labeled data, effective integration of language model and contextual biasing with shallow fusion, and metrics we use for evaluation. ## results we report our results in two parts. first, we present the overall performance of models trained with pl and with different settings. the following configurations are compared: (1) offline vs streaming tt models, (2) models trained on pl-only vs models with supervised data regularization, and (3) models with different chunk sizes. second, we report the performance of models with sf. for the baseline, we use a zipformer streaming model trained per each language on pl only. in addition, we also compare our results to the offline zipformer trained on the same data and include the reference to the previous results reported on commonvoice in @xcite ( ## conclusions in this work, we propose a framework to meet the challenge of training streaming asr systems with few-to-none supervised data by leveraging pls from foundational speech models. we conduct a thorough examination of the efficacy of pl-based tt models across various dimensions, including offline and chunk-wise decoding for streaming applications, and the influence of fsm size on the tt model's wers. we introduce robust heuristics to filter out unreliable and hallucinated pls. our findings reveal that tt models can be effectively trained from scratch on noisy pls. we managed to further improve the performance of models trained with weak pseudo-labels (generated by whispertiny, -base, and -small) by adding regularization with different amounts of supervised data. additionally, we prove that decoding with the shallow fusion of external n-gram lm and automatically generated named entities always improves the performance of models, independent of the quality of pseudo-labels.