  End-to-end neural network models achieve improved performance on various
automatic speech recognition (ASR) tasks. However, these models perform poorly
on edge hardware due to large memory and computation requirements. While
quantizing model weights and/or activations to low-precision can be a promising
solution, previous research on quantizing ASR models is limited. Most
quantization approaches use floating-point arithmetic during inference; and
thus they cannot fully exploit integer processing units, which use less power
than their floating-point counterparts. Moreover, they require
training/validation data during quantization for finetuning or calibration;
however, this data may not be available due to security/privacy concerns. To
address these limitations, we propose Q-ASR, an integer-only, zero-shot
quantization scheme for ASR models. In particular, we generate synthetic data
whose runtime statistics resemble the real data, and we use it to calibrate
models during quantization. We then apply Q-ASR to quantize QuartzNet-15x5 and
JasperDR-10x5 without any training data, and we show negligible WER change as
compared to the full-precision baseline models. For INT8-only quantization, we
observe a very modest WER degradation of up to 0.29%, while we achieve up to
2.44x speedup on a T4 GPU. Furthermore, Q-ASR exhibits a large compression rate
of more than 4x with small WER degradation.
