Building Accurate Low Latency ASR for Streaming Voice Search in E -commerce (2023)

Abstract: Automatic Speech Recognition (ASR) is essential for any voice-based application. The streaming capability of ASR becomes necessary to provide immediate feedback to the user in applications like Voice Search. LSTM/RNN and CTC based ASR systems are very simple to train and deploy for low latency streaming applications but have lower accuracy when compared to the state-of-the-art models. In this work, we build accurate LSTM, attention and CTC based streaming ASR models for large-scale Hinglish (blend of Hindi and English) Voice Search. We evaluate how various modifications in vanilla LSTM training improve the system’s accuracy while preserving the streaming capabilities. We also discuss a simple integration of end-of-speech (EOS) detection with CTC models, which helps reduce the overall search latency. Our model achieves a word error rate (WER) of 3.69% without EOS and 4.78% with EOS, with ~1300 ms (~46.64%) reduction in latency.

## introduction as an e-commerce platform in india, we need to cater to a variety of user bases, and a big part of that consists of users who cannot or do not want to type while interacting with the app, e.g., while searching for a product. for such users, interaction via a voice-based interface becomes an essential feature requiring an accurate and efficient automatic speech recognition (asr) system. recent years have witnessed the popularity of end-to-end asr models, which have achieved state-of-the-art results @xcite . these models offer simplified training and inference processes and have demonstrated higher accuracy compared to traditional pipelines with separate acoustic, pronunciation, and language models. common approaches for end-to-end asr models include ctc (connectionist temporal classification), aed (attention-based encoder-decoder), and rnnt (rnn-transducer) @xcite @xcite . however, streaming capability plays a pivotal role in choosing the most suitable asr model. while non-streaming models can leverage the entire audio for text inference, streaming models have access only to past context, which can result in reduced accuracy. nevertheless, streaming models provide immediate feedback, a critical requirement for consumer-facing applications like voice search. additionally, low inference latency is essential to ensure a user-friendly experience, as delayed feedback can adversely impact usability. another challenge in streaming asr applications is accurately detecting the end of speech (eos). conventional methods rely on standalone voice activity detection (vad) models, which operate independently from the asr system and may not offer optimal accuracy. in this work, we focus on developing a streaming asr system for large-scale hinglish voice search. our objective is to enhance accuracy and reduce latency while preserving streaming capabilities. specifically, we propose modifications to an lstm and ctc based asr system, aiming to bridge the gap between streaming and nonstreaming asr models. we also present a simple training and inference strategy that enables joint asr and eos detection within end-to-end ctc models, effectively reducing user-perceived latency in voice search. the contributions of this research can be summarized as follows: next, we discuss some related work in section 2. section 3 describe the model architecture we use, eos integration and the inference method. we talk about the dataset and experimental setup in section 4. finally, we conclude with a discussion on results and limitations in section 5. ## related work ctc, the first e2e approach developed for asr @xcite , has been widely used over the last few years @xcite . although it provides simplicity, it makes a conditional independence assumption, that output token at any time doesn't depend on past tokens, which can make it sub-optimal. aed and rnnt models relax this assumption by leveraging past output tokens. while aed models like las (listen, attend and spell) @xcite work very well for non-streaming tasks, they require complex training strategies for streaming scenarios @xcite . rnnt @xcite provides a natural alternative in streaming scenarios but has high training complexity and inference latency rendering it difficult to use in a real-world setting without complex optimizations/modifications @xcite . there have been many attempts to improve the accuracy of ctc models that preserve their training and inference simplicity. @xcite leverages hierarchical structure in the speech by adding auxiliary losses to train a ctc-based acoustic-to-subword model. their hierarchical ctc (hctc) model predicts different text segmentations in a fine-to-coarse fashion. recent studies have explored the use of attention in ctc models to implicitly relax the conditional independence assumption by enriching the features using other time frames. @xcite uses component attention and implicit language model to enrich the context while @xcite evaluates a fully self-attention-based network with ctc. in this work, we explore how augmenting an lstmbased network with windowed self-attention can help improve the transcription while preserving streaming capability. another line of work in improving the output of streaming models is the second pass rescoring that uses an additional (usually non-streaming) component to re-rank the streaming model's hypotheses @xcite . while we also rescore the candidate hypotheses at the last step, our system doesn't employ any external acoustic model to do so and leverages the hierarchical losses that are part of the model itself. for addressing eos detection, conventional approaches use vad models with a threshold on silence amount. this may lead to early termination of user speech. @xcite addresses this by training an eoq (end-of-query) classifier which performs better than vad but is still optimized independent of the asr system. vad based on output ctc labels has also been explored to detect eos based on the length of non-speech (blank) region @xcite . @xcite jointly train an rnnt model for eos detection by using and @xmath0. prediction @xmath1. we follow a similar approach where we train the model with early and late penalties. during inference, we use a dynamic threshold @xmath2. ## dataset and training setup queries from e-commerce voice search are our primary source of data. we also collect speech from other sources like on-call customer support, crowdsourced read-speech, etc., to augment training data. we transcribe all the utterances, except read-speech, using an existing asr system and manually correct them. the asr system that generates reference transcripts progressively improves as part of model iterations. collectively, the training datasets amount to ∼14m audio-text pairs (8m from the target domain and 6m from other) or roughly 22.5k hours of audio. for evaluation, we randomly sample ∼19k audios from e-commerce voice search queries, transcribe it manually (without any reference text) and reduce the human error by using multiple iterations of verification. we categorize the test set into clean and noisy subsets, containing ∼16k and ∼3k samples resp. clean utterances are audios where only one speaker's speech is intelligible. noisy utterances are those where more than one speaker has intelligible speech (overlapping or non-overlapping). in noisy utterances, the primary speaker is the user whose utterance is more relevant for the ecommerce voice search application. note that clean utterances may also have non-intelligible secondary speakers. we train and evaluate the model to transcribe only the primary speaker's speech while ignoring the rest. for training kenlm and sentencepiece models, we use a large corpus comprising text from various sources like transcribed voice search queries and on-call customer support queries, customer support chatbot queries, and product catalogues. we use a cyclical learning rate (lr) (smith, 2017) with adam optimizer to train the asr model for 200k iterations with a batch size of ∼55 minutes. training the model on two a100 (40 gb) gpus takes ∼50 hours. for eos detection, we fine-tune the model with el penalties for an additional 48k iterations (∼12 hours). we report wer and mean eos latency on the test set for evaluating the performance of our model in ## results and discussion to understand how modifications in the architecture contribute to improving the accuracy of the vanilla lstm ctc model, we conduct an ablation study and report the wer in