ASR Adaptation for E -commerce Chatbots using Cross-Utterance Context and Multi-Task Language Modeling (2021)

Abstract: Automatic Speech Recognition (ASR) robustness toward slot entities are critical in e-commerce voice assistants that involve monetary transactions and purchases. Along with effective domain adaptation, it is intuitive that cross utterance contextual cues play an important role in disambiguating domain specific content words from speech. In this paper, we investigate various techniques to improve contextualization, content word robustness and domain adaptation of a Transformer-XL neural language model (NLM) to rescore ASR N-best hypotheses. To improve contextualization, we utilize turn level dialogue acts along with cross utterance context carry over. Additionally, to adapt our domain-general NLM towards e-commerce on-the-fly, we use embeddings derived from a finetuned masked LM on in-domain data. Finally, to improve robustness towards in-domain content words, we propose a multi-task model that can jointly perform content word detection and language modeling tasks. Compared to a non-contextual LSTM LM baseline, our best performing NLM rescorer results in a content WER reduction of 19.2% on e-commerce audio test set and a slot labeling F1 improvement of 6.4%.

## introduction task-oriented conversations in voice chatbots deployed for e-commerce usecases such as shopping @xcite , browsing catalog, scheduling deliveries or ordering food are predominantly shortform audios. moreover, these dialogues are restricted to a narrow range of multi-turn interactions that involve accomplishing a specific task @xcite . the back and forth between a user and the chatbots are key to reliably capture the user intent and slot entities referenced in the spoken utterances. as shown in previous works @xcite @xcite , rather than decoding each utterance independently, there can be benefit in decoding these utterances based on context from previous turns. in the case of grocery shopping for example, knowing that the context is "what kind of laundry detergent?" should help in disambiguating "pods" from "pause". another common aspect in e-commerce chatbots is that the speech patterns differ among sub-categories of usecases (eg. shopping clothes vs ordering fast food). hence, some chatbot systems allow users to provide pre-defined grammars or sample utterances that are specific for their usecase @xcite . these user provided grammars are then predominantly used to perform domain adaptation on an n-gram language model. recently @xcite showed that these can be leveraged to bias a transformer-xl (txl) lm rescorer on-the-fly. while there has been extensive previous work on improving contextualization of txl lm using historical context, none of the approaches utilize signals from a natural language understanding (nlu) component such as turn level dialogue acts. this paper investigates how to utilize dialogue acts along with user provided speech patterns to adapt a domain-general txl lm towards different ecommerce usecases on-the-fly. we also propose a novel multi-task architecture for txl, where the model jointly learns to perform domain specific slot detection and lm tasks. we use perplexity (ppl) and word error rate (wer) as our evaluation metrics. we also evaluate on downstream nlu metrics such as intent classification (ic) f1 and slot labeling (sl) f1 to capture the success of these conversations. the overall contributions of this work can be summarized as follows : ## related work incorporating cross utterance context has been well explored with both recurrent and non-recurrent nlms. with lstm nlms, long span context is usually propogated without resetting hidden states across sentences or using longer sequence lengths @xcite @xcite . in @xcite , along with longer history, information about turn taking and speaker overlap is used to improve contextualization in human to human conversations. with transformer architecture based on self attention @xcite showed that by utilizing segment wise recurrence transformer-xl (txl) @xcite is able to effectively leverage long span context while decoding. more recently, improving contextualization of the txl models included adding a lstm fusion layer to complement the advantages of recurrent with non-recurrent models @xcite . @xcite incorporated a non-finetuned masked lm fusion in order to make the domain adaptation of txl models quick and on-the-fly using embeddings derived from customer provided data and incorporated dialogue acts but only with an lstm based lm. while @xcite tried to fuse multi-model features into a seq-to-seq lstm based network. in (sharma, 2020) cross utterance context was effectively used to perform better intent classification with e-commerce voice assistants. for domain adaptation, previous techniques explored include using an explicit topic vector as classified by a separate domain classifier and incorporating a neural cache @xcite @xcite . @xcite used a mixture of domain experts which are dynamically interpolated. it is also shown in @xcite , that using a hybrid pointer network over contextual metadata can also help in transcribing long form social media audio. joint learning nlu tasks such as intent detection and slot filling have been explored with rnn based lms in @xcite and more recently in @xcite , where they show that a jointly trained model consisting of both asr and nlu tasks interfaced with a neural network based interface helps incorporate semantic information from nlu and improves asr that comprises a lstm based nlm. in @xcite tried to incorporate joint slot and intent detection into a lstm based rescorer with a goal of improving accuracy on rare words in an end-to-end asr system. however, none of the previous work utilize dialogue acts with a non-recurrent based lm such as transformer-xl nor optimize towards improving robustness of in-domain slot entities. in this paper we experiment and study the impact of utilizing dialogue acts along with a masked language model fusion to improve contextualization and domain adaptation. additionally, we also propose a novel multi-task architecture with txl lm that improves the robustness towards in-domain slot entity detection. ## approach a standard language model in an asr system computes a probability distribution over a sequence of words @xmath0, ..., w n auto-regressively as: in our experiments, along with historical context, we condition the lm on additional contextual metadata such as dialogue acts : where c 1 , c 2 , ...c k are the turn based lexical representation of the contextual metadata. for baseline, we use a standard lstm lm as summarized below : utterance i want my shoes delivered to seattle next thursday annotation o o o slot o o slot slot slot figure 2: example utterance with slots annotated ## results and discussion ## conclusion in this paper we explored different ways to robustly adapt a domain-general transformer-xl nlm to rescore n-best hypotheses from a hybrid asr system for task-oriented e-commerce speech conversations. we demonstrated that transformer-xl lm trained with turn level dialogue acts benefits intent classification by improving the recognition of content words. additionally, we show that using semantic embeddings derived from a masked language model finetuned on e-commerce domain can be effectively used to adapt a domain-general txl lm for e-commerce domain utterance rescoring task. finally, we introduced a new txl training loss function to jointly predict content words along with language modeling task, this when combined with bert fusion and dialogue acts, amplifies the wer, ic f1 and sl f1 gains. we have also shown these improvements to be statistically significant. future work can look at integrating these methods into an end-to-end asr system for both rescoring task and first pass lm fusion.