Methods to Optimize W av2 V ec with Language Model for Automatic Speech Recognition in Resource Constrained Environment (2022)

Abstract: Automatic Speech Recognition (ASR) on resource constrained environment is a complex task since most of the State-Of-The-Art models are combination of multilayered convolutional neural network (CNN) and Transformer models which itself requires huge resources such as GPU or TPU for training as well as inference. The accuracy as a performance metric of an ASR system depends upon the efficiency of phonemes to word translation of an Acoustic Model and context correction of the Language model. However, inference as a performance metric is also an important aspect, which mostly depends upon the resources. Also, most of the ASR models uses transformer models at its core and one caveat of transformers is that it usually has a finite amount of sequence length it can handle. Either because it uses position encodings or simply because the cost of attention in transformers is actually O(n²) in sequence length, meaning that using very large sequence length explodes in complexity/memory. So you cannot run the system with finite hardware even a very high-end GPU, because if we inference even a one hour long audio with Wav2Vec the system will crash. In this paper, we used some state-of-the-art methods to optimize the Wav2Vec model for better accuracy of predictions in resource constrained systems. In addition, we have performed tests with other SOTA models such as Citrinet and Quartznet for the comparative analysis.

## introduction speech is the most natural way of human communication; it gives humans a medium to understand and communicate their feelings and emotions. understanding speech or speech recognition is a critical part of modern applications even plays a significant role for empowering ai enabled smart devices such as amazon's echo, google's nest, apple's homepod etc. in this paper, we have proposed a method called wslr (wav2vec with stride chunking and language model for resource-constrained devices) to create an offline speech recognition system using wav2vec model which achieved a word error rate (wer) of 0.85 in an environment of 2 core cpu and 4 gigabytes of ram on librispeech test set, which surpasses the score of original wav2vec model i.e. 1.8 and other sota models. to recognize speech of an audio file we first start off by converting it into a digital format since audio data could have a lot of variations like different channels or sample rate. we further have must standardize the dataset by re-sampling the data to a specific sample rate as any model requires data to follow common standards. we have used wav2vecbase model trained on librispeech data for 960 hours as our main asr model for the speech recognition task where it encodes sound files via multi-layer convolutional neural network (cnn) to produce latent speech representation and then feed those masked representations to a transformer network, the output of which can be decoded to word vectors using a connectionist temporal classification (ctc) algorithm. the following model can further be optimized when the output logits are fed to a language model (in this paper we used a 4-gram language model). @xcite foot_0 to deploy our model to a resource constrained environment we use something called stride chunking, here the chunking is sequential, and we chunk the audios such that there is some overlapping length at borders defined by 'stride length' and the main context stay intact within the center. then while inference, overlapped segments or logits (as output) are dropped to keep the results of the transcriptions as similar as they would be while inferenced with full audio. we also use some rule-based postprocessing steps for our transcription to generalize even better for spoken numbers or symbols in-case. ## proposed method in this section, we discuss the proposed approach i.e. wslr to optimize the state-of-the-art wav2vec model introduced by facebook and that achieves such an optimal performance when inferenced in constrained environments such as, constrained docker container or embedded/micro systems like raspberry pi or nvidia's jetson board. constrained inference along with good scores is quite a big deal for a huge deep learning model. since, more the size of the model the more resources it demands and wav2vec is itself a combination of huge models such as cnns and transformers. therefore, it is quite a challenging task. the slight drop in scores while chunking is comprehended with preprocessing, language model, and post-processing optimizations. further, the details of the optimization methods and proposed functionalities are discussed in the following sub-sections. ## experiments and comparative analysis this section presents the experimental evaluations of the optimized approach, using wav2vec model with optimization methods mentioned in the former sections. for the experiments, we have used docker containers to reflect the constraints of mobile devices and the experiments are performed with 2 most common mobile configurations i.e., 2 cores cpu with 8 gigabytes of ram and 2 cores cpu with 4 gigabytes of ram. it has additionally been tested with a gpu configuration comprised of tesla p100 for a better intuition of the performance. we have done a cross-comparative analysis of the system proposed with general wav2vec model inference as well as other state-of-the-art models like quartznet by @xcite and citrinet model by @xcite . for the experiments, we used librispeech's test set and pre-processed it further to generalize it for metric calculations. ## conclusion this paper presents, an optimized way of scaling offline asr system to resource constrained environments, in our experiments we have noted a very significant improvement such as 0.85 of wer and pipeline inference on environment with 4 gigabytes of ram and 2 cpu cores without facing any system crashes also the results on resource constrained environment are mostly similar to a high-end resource environment, some related work of speech recognition in low end systems is also presented by @xcite and @xcite . the proposed flow showed a significant improvement in terms of scalability and performance and can be integrated and used in the advancement of the applications such as smart devices like voice assistant enabled speakers, offline voice typing in mobile devices, speech summarization and speech context moderation etc. however, there is always a room for improvement, some future research can still be performed on improving the inference of the proposed method in the constrained systems since faster inference with least resources is always the call of state-of-the-art methods. exploring and researching to add novelties such as speech content moderation in asr systems seems one of the promising directions of future work.