E 2 E Spoken Entity Extraction for Virtual Agents (2023)

Abstract: In human-computer conversations, extracting entities such as names, street addresses and email addresses from speech is a challenging task. In this paper, we study the impact of fine-tuning pre-trained speech encoders on extracting spoken entities in human-readable form directly from speech without the need for text transcription. We illustrate that such a direct approach optimizes the encoder to transcribe only the entity relevant portions of speech ignoring the superfluous portions such as carrier phrases, or spell name entities. In the context of dialog from an enterprise virtual agent, we demonstrate that the 1-step approach outperforms the typical 2-step approach which first generates lexical transcriptions followed by text-based entity extraction for identifying spoken entities.

## introduction enterprise virtual agents (eva) provide automated customer care services that rely on spoken language understanding (slu) in a dialog context to extract a diverse range of intents and entities that are specific to that business @xcite . gathering various entities like names, email, street address from human callers become a part of large range of virtual agents. in order to minimize the error in recognition and extraction of names, designers of speech interfaces often design prompts that request the user not only to say their name but spell it as well to address issues of homophones. (eg. catherine or katheryn). such a behavior to spell carries over to other entities such as street and email addresses, without the users being explicitly prompted to do so. extensive research has been done to recognize entities in spoken input @xcite @xcite @xcite . similar to text-based ner, approaches for spoken ner often involve predicting entity offsets and type in text provided by an automatic speech recognizer (asr) @xcite @xcite or recognizing directly as a part of e2e asr output. significantly limited research has been done on spoken entity extraction in dialogs @xcite , and even fewer in enterprise virtual agents @xcite . some methods proposed include using a predefined list of entity names @xcite in a speech recognizer, fuzzy refinement by exploiting knowledge graphs @xcite , or to using a large vocabulary speech recognizer to obtain the transcript further processed using text-based ner tools. such techniques are difficult to adapt to caller responses to the prompt "say and spell your first/last name" in a spoken dialog system, as illustrated by the following example. until recently, where @xcite adapt a standard seq-2-seq architecture to extract person names from automatically transcribed text. however, this 2-step approach means two sys-tems to maintain and adapt, but also, possibly loss of acoustic-prosodic information. in this paper, we propose a novel method for extracting humanreadable spoken entities directly from speech with a single model (1-step approach) that is optimized for the entity extraction task. we hypothesize ctc loss @xcite , widely used for training e2e asr systems in an non-autoregressive manner, can be re-imagined to map audio events to text events. by generating only entity relevant tokens, our system learns to perform more intelligent entity extraction, instead of just performing literal lexical transcription as done by all existing asr systems. we believe this opens the door to more interesting use-cases where e2e asr systems learns to perform task-specific generation directly from speech. we acquire data from a production eva system which has human-in-the-loop for automation and data-collection purposes. section 3 describes dataset in detail. we found that our proposed 1step approach significantly outperforms the 2-step approach for extracting names, address and emails from users of an eva system. the contributions of our paper are as follows: ## related work a common practice is to convert normalized token sequence in spoken form produced by asr into a written form better suited to processing by downstream components in dialog systems @xcite . this written form is then used to extract structured information in the form of intent and slot-values to continue a dialog @xcite . recently, there is a growing tread to use neural encoders optimizing directly using speech input, popularly known as e2e slu approaches @xcite . inverse text normalization: information extraction systems generally use an inverse text normalization (itn) component to convert a token sequence in spoken form produced by asr into a written form suitable for downstream components -nlu and dialog. transforming spoken language to written form involves altering entities like cardinals, ordinals, dates, times, and addresses @xcite . methods proposed for itn include: using language models (lm) to decode written-form hypothesis @xcite , a finite-state verbalization model @xcite , leveraging rules and handcrafted grammars to cast itn as a labeling problem @xcite . e2e slu: several e2e approaches which directly act on speech, have been proposed for named entity recognition, a closely related task to the entity extraction studied in this work @xcite @xcite @xcite . ghannay et al. @xcite fine-tune an e2e asr pre-trained with the ctc loss @xcite with a set of special character labels enclosing the named entities in the transcription using ctc @xcite @xcite @xcite . unlike these previous works that typically need both the text transcript along with entity type and offset tags, our approaches only need the normalized entity for supervision. our system transparently only use pairs of audio and the target normalized entities to extract. thus, removing a significant amount of cost and effort needed to obtain transcription and entity tags. ## method we rethink asr not only to be a transcription system but an e2e speech based encoder that can extract human-readable entities thus, learning to ignore, normalize and generate only target entity tokens directly from speech. ## dataset while some public datasets like the ogi collection @xcite include a small subset of spelled names. evas for multiple industry verticals, record millions of user utterances responding to different prompts. in these dialog systems customers are prompted to provide information at various dialog turns using different authored prompts. what's your name? h: a l e x u s last name k i n g m: alexus king h: l i s a s t a n t as in tom o n m: lisa staton what's your street address? h: four three eight three remo crescent road m: 4383 remo crescent rd. h: six forty six eighteenth street apartment one m: 646 state st. apt 1 what's your email-id ? h: k as kite i n as nancy nine one five at gmail.com m: kin915@gmail.com h: a n as in nancy girl tower e e at outlook dot com m: angtee@outlook.com for testing purposes, we randomly sampled audio from a large pool of data, which is collected at different time frames than train data, but from same set of applications. we imitate an human-inthe-loop scenario where annotators listen to user inputs, type the entity by listening to audio only once in the limited time. our test participants are a mix of native and non-native speakers who could be less exposed to european names. we merge transcriptions of training data obtained using an e2e asr for all entities to create a list of most frequent keywords (excluding characters, number words, email-address-providers). ## experiments and evaluation we use only 1 nvidia a100 gpu for all finetuning purposes. we keep batch-size of 32 with starting learning rate of .001. we use weight decay of .001 and update the model with 8 accumulated batches with adam back-propagation algorithm. we will share our experiment configuration in the final revision. we report results for average of 5 runs. we provide entity confidence score by summing over the posterior probability of non-blank predicted tokens, a method originally proposed by @xcite . results for our proposed 1-step extraction outperforms 2-step entity extraction approach, as shown at ## conclusions in this paper we show high-quality spoken entities can be extracted directly from speech by finetuning e2e asr systems. the proposed 1-step model may not be influenced by asr mistakes while carrying the critical token sequence to the final entity extraction phase. we didn't do hyperparameter search for the models, due to gpu limitations. for complete automation of prompts in customer calls a system also needs to extract intent for samples (10-15%) with no entities in it. our early experiments suggest this can be done by mixing intent labelled data (intent label used as single vocab token) or transcriptions of samples with no entities along with entity extraction data. this leads to minor loss in performance for each entity.