Breaking Down Power Barriers in On-Device Streaming ASR : Insights and Solutions (2025)

Abstract: Power consumption plays a crucial role in on-device streaming speech recognition, significantly influencing the user experience. This study explores how the configuration of weight parameters in speech recognition models affects their overall energy efficiency. We found that the influence of these parameters on power consumption varies depending on factors such as invocation frequency and memory allocation. Leveraging these insights, we propose design principles that enhance on-device speech recognition models by reducing power consumption with minimal impact on accuracy. Our approach, which adjusts model components based on their specific energy sensitivities, achieves up to 47% lower energy usage while preserving comparable model accuracy and improving real-time performance compared to leading methods.

## introduction streaming automatic speech recognition (streaming asr) enables real-time transcription of speech to text with latency typically under 500 milliseconds, supporting applications such as interface navigation, voice commands, real-time communication, and accessibility on mobile and wearable devices. however, high power consumption poses a significant challenge, limiting usability by requiring frequent recharges. improving the energy efficiency of on-device streaming asr is therefore essential for enhancing user experience. we focus on on-device streaming asr models, particularly the neural transducer @xcite , which combines an encoder for acoustic modeling, a predictor for language modeling, and a joiner to integrate their outputs (see figure 1 ). widely regarded as the standard for on-device streaming asr @xcite @xcite , the neural transducer excels in balancing computational efficiency and accuracy. we train and evaluate over 180 neural transducer models foot_0 , exploring architectures including emformer @xcite and conformer @xcite while varying component sizes. this extensive study reveals how the components impact accuracy, real-time factor (rtf), foot_1 and power consumption. our analysis reveals several key findings: (1) energy usage in streaming asr models is driven by memory traffic for loading weights, which depends on the invocation frequency of components and their memory hierarchy placement. (2) invocation frequencies vary widely, with the joiner being called far more often than the predictor, and the predictor more than the encoder. despite comprising only 5-9% of the model's size, the joiner accounts for 48-73% of its power consumption. (3) we identify an exponential relationship between model accuracy and encoder size, suggesting new directions for streaming asr research. building on these insights, we propose a targeted compression strategy to optimize energy efficiency with minimal accuracy loss. this approach evaluates power and accuracy sensitivity for each component, prioritizing compression of components with higher power sensitivity and lower accuracy sensitivity. specifically, we focus on compressing the joiner first, followed by the predictor and encoder, and aim to store the joiner's weights in energy-efficient local memory. experiments on librispeech @xcite and public video datasets show our method reduces energy usage by up to 47% and lowers rtf by up to 29%, while maintaining comparable accuracy to stateof-the-art compression strategies. unlike previous approaches, our method effectively leverages the diverse runtime characteristics of asr components, showcasing its superior efficiency. this paper makes the following contributions: an earlier version of this paper was released as a preprint on arxiv @xcite . ## power and accuracy analysis of on-device streaming asr in this section, we use adam-pruning @xcite , a state-of-the-art weight pruning technique for speech recognition, foot_2 to adjust the sizes of the encoder, predictor, and joiner in asr models. this generates asr models of varying sizes, enabling analysis of their power consumption and accuracy, yielding key insights. ## asr energy efficiency optimization we aim to minimize the power consumption of streaming asr models with minimal performance impact by evaluating the power and accuracy sensitivities of the encoder, predictor, and joiner components. these sensitivities quantify the change in power consumption and performance, respectively, for a unit reduction in component size: power sensitivity component := ∆power ∆size component accuracy sensitivity component := ∆accuracy ∆size component (2) 4 variability in predictor and joiner compression curves stems from randomness in training and pruning. here, component refers to the encoder, predictor, or joiner, and accuracy is inversely related to the word error rate. the power consumption of on-device streaming asr is primarily due to loading model weights from memory. power sensitivity is therefore expressed as: power sensitivity @xmath0, we adopt 1.5pj/byte for local memory and 120pj/byte @xcite for off-chip memory. component size determines whether weights fit in energy-efficient local memory or power-hungry off-chip memory, influencing power sensitivity. accuracy sensitivity is calculated by progressively reducing a component's size, observing the effect on model accuracy, and fitting an exponential function to describe the relationship. the derivative of this function quantifies accuracy sensitivity. finally, we use the power-to-accuracy sensitivity ratio to prioritize compression decisions: power-to-accuracy sensitivity @xmath1, helping determine the optimal compression order for on-device asr models. our compression algorithm starts with a fully uncompressed model and iteratively reduces its size to achieve a user-defined power reduction target (e.g., "reduce power by 60 mw"). at each step, we calculate the power-to-accuracy sensitivity ratio for each component and compress the one with the highest ratio. in neural transducer models, the joiner typically starts with the highest ratio due to its high power sensitivity from frequent invocation. once its size is reduced enough to fit into energy-efficient local memory, its ratio decreases, and the predictor becomes the next priority. the predictor is compressed until it reaches its userdefined minimum size, beyond which further compression would cause significant accuracy loss due to the exponential relationship between accuracy and size. the encoder is then compressed similarly, followed by additional compression of the joiner if more power reduction is required. the compression order is thus: @xmath2. our algorithm determines only the compression order between components, delegating the pruning of weight parameters within a selected component to existing compression methods. this makes our approach compatible with any existing compression algorithm. ## related work this study is the first to analyze the operational dynamics and memory placement of model components to enhance energy efficiency in on-device streaming asr. the most relevant prior works focus on asr compression and power optimization. ## conclusion power consumption is a critical challenge for ondevice streaming asr, impacting device recharge frequency and user experience. this study analyzed power usage in asr models, revealing its dependence on model size, invocation frequency, and memory placement. notably, the joiner consumes more power than the larger encoder and predictor due to its higher invocation frequency and off-chip memory usage. we also identified an exponential relationship between word error rate and encoder size. based on these insights, we developed guidelines for model compression to enhance energy efficiency. applying these guidelines to the lib-rispeech and public video datasets achieved up to 47% energy savings and a 29% reduction in rtf, maintaining accuracy comparable to state-of-theart methods. these findings highlight the potential of targeted optimizations to advance sustainable and energy-efficient on-device streaming speech recognition.