End-to-End Speech to Intent Prediction to improve E -commerce Customer Support Voicebot in H indi and E nglish (2022)

Abstract: Automation of on-call customer support relies heavily on accurate and efficient speech-to-intent (S2I) systems. Building such systems using multi-component pipelines can pose various challenges because they require large annotated datasets, have higher latency, and have complex deployment. These pipelines are also prone to compounding errors. To overcome these challenges, we discuss an end-to-end (E2E) S2I model for customer support voicebot task in a bilingual setting. We show how we can solve E2E intent classification by leveraging a pre-trained automatic speech recognition (ASR) model with slight modification and fine-tuning on small annotated datasets. Experimental results show that our best E2E model outperforms a conventional pipeline by a relative ~27% on the F1 score.

## introduction spoken language understanding (slu) systems that extract the intent from a spoken utterance are integral in various voicebot applications such as automated on-call customer support, voice assistants, home or vehicle automation systems, etc. the extracted intent triggers a standard operating procedure (sop) as defined by the respective application, e.g. an e-commerce customer query "i want to return my phone" maps to "return" intent which triggers the sop to help the user with returns. it helps us reduce the reliance on human agents and provide faster resolutions. more elaborate examples are shown in conventionally, such systems consist of two components -an automatic speech recognition (asr) system followed by a natural language understanding (nlu) unit. asr converts audio to text, and nlu performs intent classification. further, each component can have multiple submodels. typically, both these components are developed and optimized independently. asr optimizes word error rate (wer) with equal weightage to individual words. this might not be optimal for an s2i system since all words are not equally relevant for intent classification. also, due to the broad diversity in speech, training reliable asr models can be very data intensive and strenuous. an errorprone asr results in noisy inputs to nlu models, typically trained on clean text. this causes error accumulation which reduces the pipeline's performance. data-intensive training of multiple models, high complexity & maintenance, and higher overall latency make this pipeline approach sub-optimal. the end-to-end s2i model is an intuitive alternative to overcome these limitations. it eliminates the problem of error accumulation, is simple and faster, and reduces the efforts required for independent models. modelling the problem as audio-to-intent classification simplifies the task since the number of intents is usually much less than the vocabulary size used in asr and nlu. it helps us reduce the requirement of manually annotated training data. in this work, we adapt an e2e asr model to build an e2e s2i model for flipkart's on-call customer support. an overview of our contributions is as follows: next, we discuss some related work in section 2. section 3 & 4 describe the baseline s2i pipeline and our e2e approach respectively. we talk about datasets, preprocessing and experimental setup in section 5. finally, we conclude with a discussion on results and limitations in section 6. 2 related work there have been several attempts to mitigate asr error propagation in text-based pipelines. one straightforward idea is to correct the asr output, using error correction models @xcite or by ranking n-best hypotheses @xcite @xcite . other approach is to leverage extra information from asr -output lattice @xcite @xcite , nbest hypotheses @xcite @xcite or word confusion networks/embeddings @xcite . though these approaches make nlu robust to some asr errors, they still use a strict multi-component pipeline. there have been an increasing number of attempts toward building end-to-end slu models. qian et al. (2017); serdyuk et al. (2018); chen et al. (2018) investigate end-to-end slu models which do not use asr at all whereas @xcite optimizes asr and nlu in a joint setup. such endto-end models can require a large amount of paired speech and intent data which may not always be available. wang et al. (2020); morais et al. (2021) explore unsupervised pre-training which helps in low-resource settings but is usually very compute intensive. an alternative approach is to initialize slu models using weights trained for asr @xcite @xcite . since asr datasets are more easily available, this approach presents a much easier method of pre-training than unsupervised methods. inspired by asr pre-training, we explore how to augment a pre-trained asr model for end-to-end s2i task for flipkart customer support voicebot in hindi and english languages. ## text-based pipeline our baseline consists of 3 components -asr, transliteration and text-to-intent as shown in fig. 1 . we use a bilingual asr system which predicts text in devanagari script for both hindi and english. the transliteration model converts this text into roman script. finally, the text-to-intent model extracts intent from roman text. ## e2e speech-to-intent for the s2i task, we augment the pre-trained asr model (same as used in the baseline) with intent prediction head as shown in fig. 2 . we summarize the hidden features from the last block of the asr model using a dot-product based multi-headed selfattention (mha) layer. we use the output sequence of the last block as key-value vectors and the final cell state of the last bilstm layer as the query vector. a linear layer then predicts probability distribution over the intent classes. since there's no text output from the model, the "blank" intent is also predicted the by e2e s2i model. we train the intent prediction head (and fine-tune the bilstm blocks) using cross entropy loss. ## results we compare the baseline and e2e model on 14606 voicebot queries manually transcribed and annotated for text and target intent resp. we report accuracy and f1 score for intent classification and word error rate (wer) for asr in the s2i model, fine-tuned on just 10k manually annotated audio-intent pairs (v1), outperforms the baseline by an absolute 3.07% on the f1 score. using this, we predict intent on an unlabeled set and get a random sample of 25k audios where the model has low confidence (prediction probability as given by softmax on the last layer). we correct this set manually and re-train the model using all 35k samples (v2), improving the f1 score by 1.28%. we then re-train the model on the complete set of voicebot queries (∼920k audios from asr dataset) using pseudo labels, further improving the score marginally. our final e2e model outperforms the baseline by an absolute 4.59% on the f1 score. the e2e model has a median latency of ∼41ms, which is 1/3rd of the baseline latency (∼123ms). since we can deploy the complete model on a gpu, it can handle inference at a much larger scale than the baseline -more than 1000 queries per second using a single a100 gpu. whereas the decoder in the asr system used for the baseline, which is the bottleneck, can only handle about 90 queries per second. thus, the e2e model outperforms the baseline on accuracy, latency, and scalability.