Conformer-Based Speech Recognition On Extreme Edge-Computing Devices (2024)

Abstract: With increasingly more powerful compute capabilities and resources in today’s devices, traditionally compute-intensive automatic speech recognition (ASR) has been moving from the cloud to devices to better protect user privacy. However, it is still challenging to implement on-device ASR on resource-constrained devices, such as smartphones, smart wearables, and other small home automation devices. In this paper, we propose a series of model architecture adaptions, neural network graph transformations, and numerical optimizations to fit an advanced Conformer based end-to-end streaming ASR system on resource-constrained devices without accuracy degradation. We achieve over 5.26 times faster than realtime (0.19 RTF) speech recognition on small wearables while minimizing energy consumption and achieving state-of-the-art accuracy. The proposed methods are widely applicable to other transformer-based server-free AI applications. In addition, we provide a complete theory on optimal pre-normalizers that numerically stabilize layer normalization in any L_p - norm using any floating point precision.

## introduction conformer-based @xcite end-to-end (e2e) automatic speech recognition (asr) @xcite with streaming capabilities @xcite have made numerous advances recently. this has paved the way for fully neural speech recognition on resource-constrained mobile devices. these systems also have numerous advantages over conventional hybrid-hmm asr @xcite . first, the training procedure is simplified; the entire system can be defined in a single deep learning framework such as pytorch or tensorflow. second, recent work (e.g. miao et al., 2019; sainath * equal contribution. in this paper, we present optimizations to enable fully e2e neural network based asr system under resource-constrained environments, such as smartphones, wearables, and home automation devices. operating fully offline saves cloud computing resources while providing stronger user privacy @xcite guarantees, as the user's speech does not need to be transmitted outside of the device. when targeting resource constrained devices, hardware limitations present many challenges. we describe several multidisciplinary solutions we explored, including memory-aware network transformation, model structural adjustment, and numerical optimizations to address inference stability. we specifically focus on our efforts to take advantage of the inference efficiency provided by specialty hardware accelerators. we derive a theory to numerically stabilize computation of layer normalization on hardware accelerators. this stabilization technique does not require model retraining and is applicable to the computation of any l p -norm. ## prior work improving the efficiency of the transformer architecture has seen substantial interest. @xcite provides a comprehensive survey primarily concentrating on model architecture improvements. @xcite is another noteworthy resource which delves deeper into considerations specific to hardware configurations. linear transformer @xcite ) is a key technique, mitigating the computationally expensive softmax function @xcite within the attention mechanism. softmax is also susceptible to numeric overflow problems when computing with limited numerical range. @xcite ; @xcite discuss alternative normalization methods other than batchnorm @xcite and layernorm @xcite to improve computational efficiency and numerical stability in low precision environments. principles for optimizing transformers have been described in apple (2022) which target apple hardware, but are generally applicable for similar devices. within the domain of speech recognition, squeezeformer @xcite stands as a seminal work focusing on efficiency optimization, particularly with respect to the conformer architecture. the paper uses depthwise separable convolution subsampling to substantially save computation which is central to mobilenet @xcite . it's worth mentioning that the majority of prior work focuses on improving training efficiency by making modifications to the existing model architecture. as a result, these changes require model retraining to achieve efficiency improvements. in contrast, our research primarily concentrates on post-training, inference-only processes while avoiding model retraining whenever possible. ## backbone model our backbone model is built upon the conformer neural architecture @xcite as shared acoustic encoder while connectionist temporal classification @xcite (ctc) and attention-based encoder decoder (aed) @xcite as dual decoders trained with multitask learning mechanism @xcite . similar to prior work (e.g. gulati et al., 2020), we stack transformer @xcite layers and convolution @xcite layers alternatively to convert speech frames into high-level representation. we use a relative sinusoidal positional encoding @xcite into transformer layers. since our goal is to stream asr on edge devices, we adopt the chunk-based attention strategy to better balance accuracy and dependency of future audio frames @xcite . ## conclusions through architectural and numerical optimizations, we demonstrate that conformer ctc asr models are capable of running on resource-constrained devices such as mobile phones, and wearables. the optimizations preserve recognition accuracy while performing faster than real time and consuming lesser energy. our theoretical findings of techniques in numerical stabilization is applicable to a wide range of deep learning models and computing tasks.