S peech N et: Weakly Supervised, End-to-End Speech Recognition at Industrial Scale (2022)

Abstract: End-to-end automatic speech recognition systems represent the state of the art, but they rely on thousands of hours of manually annotated speech for training, as well as heavyweight computation for inference. Of course, this impedes commercialization since most companies lack vast human and computational resources. In this paper, we explore training and deploying an ASR system in the label-scarce, compute-limited setting. To reduce human labor, we use a third-party ASR system as a weak supervision source, supplemented with labeling functions derived from implicit user feedback. To accelerate inference, we propose to route production-time queries across a pool of CUDA graphs of varying input lengths, the distribution of which best matches the traffic’s. Compared to our third-party ASR, we achieve a relative improvement in word-error rate of 8% and a speedup of 600%. Our system, called SpeechNet, currently serves 12 million queries per day on our voice-enabled smart television. To our knowledge, this is the first time a large-scale, Wav2vec-based deployment has been described in the academic literature.

## introduction training an end-to-end automatic speech recognition (asr) model requires hundreds, if not thousands, of hours of hand-labeled speech. with the rise of silicon-hungry pretrained transformers, these models additionally need increasing amounts of computational power just to perform inference. together, these two hurdles impede effective model deployment at all but the largest technology companies and specialized speech processing startups. the hurdles certainly apply to us at comcast, the main stage of this work. our industrial challenge is to fine-tune and deploy a large, pretrained speech recognition model, without an army of annotators (as in amazon) or mammoth gpu farms (e.g., google). our end application is the xfinity x1, a voice-enabled smart television serving millions of active devices in the united states. evidently, cloud asr services are cheaply available. foot_0 google cloud, for example, charges @xmath090 per hour, and our in-house annotators, whom comcast must use to protect user privacy, cost even more. thus, cloud asr's comparatively low pricing, combined with its decent quality, suggests its utility as an annotation source in the absence of substantial human-labeled data. nevertheless, cloud asr still falls short of human parity and hence demands label denoising. to do this, we propose to use implicit user feedback to remove incorrectly labeled examples, bootstrapping an existing cloud asr service. we derive these labeling functions using signals from query repetition, session length, and asr confidence scores. we model them in snorkel @xcite , a popular data programming framework, producing a 1400-hour weakly labeled dataset. trained on this, our models improve over those using unfiltered data by an average 0.97 points in word-error rate (wer), as presented in section 4. as for the second hurdle of resource efficiency, many model acceleration methods exist. however, few meet our productionization criteria: we seek to preserve the quality, ruling out structured pruning @xcite ; we wish to preserve the pretrained architectural structure, eliminating knowledge distillation @xcite ; and we require stable software-gpu support, disqualifying low bit-width quantization @xcite and other cpu-oriented approaches. all things considered, the prime candidates are medium bit-width quantization, decoder optimizations @xcite , and cuda computation graphs @xcite . the first two follow the literature, but the third is more open ended. in spite of their record-breaking performance, cuda graphs work only with fixed-length input, not variable length. toward this, we propose to allocate a pool of cuda graphs of varying lengths, altogether matching the production-time traffic length distribution. during inference, we route each query to the graph with the least upper-bound in length. as we show in section 4, this yields a 3-5× increase in throughput. we claim the following contributions: first, we derive novel labeling functions for constructing weakly labeled speech datasets from in-production asr systems, improving our best model by a relative 8% in word-error rate. second, we propose to accelerate model inference using a pool of cuda graphs, attaining a 7-9× inference speed increase at no quality loss. the resulting system, speech-net, currently serves more than 20 million queries per day on our smart television. to our knowledge, we are the first to describe a large-scale, wav2vecbased deployment in the academic literature. ## our speechnet approach our task is to train and deploy a state-of-the-art, end-to-end asr system, without using humanannotated data. the context of this deployment is a smart tv, which users interact with using a speechdriven remote control. to issue a voice query, users hold a button, speak their command, and release the button. we initially serve them with a third-party cloud asr service, bootstrapping it for the development of speechnet. data-wise, we store thousands of hours of utterances per day, complete with session ids, transcripts, and device ids. resourcewise, we have 30 deployment nodes, each hosting an nvidia tesla t4 gpu and receiving 120 queries per second (qps) at peak time; thus, our model's real-time factor must exceed 120. ## experimental setup our key experiments are to validate the model effectiveness of our labeling functions (section 2.2) and the computational savings of our cuda graph pool (section 2.3). we trained every run on one p3.2xlarge amazon web services (aws) instance, which has an nvidia v100 gpu and eight virtual cpu cores. we implemented our models in pytorch using the huggingface transformers library @xcite ) and nvidia's nemo @xcite ; see the appendix for more details. ## conclusions and future work in this paper, we explore commercializing a transformer-based, end-to-end speech recognition system without human annotation and with less computational power. we design three novel labeling functions, derived from implicit user feedback, for snorkel to construct weakly labeled, indomain speech datasets from production traffic. we also propose cuda graph pools, a novel model acceleration method especially suited for singleexample inference, as frequently encountered in production. our system, speechnet, improves the word-error rate by a relative 8% and the inference speed by 600%, compared to our third-party asr service. one promising research direction is to extend speechnet to the recently released openai whisper @xcite , an ultra large-scale asr model trained on 680,000 hours of speech, representing the longest corpus to date.