Improving Named Entity Recognition in Spoken Dialog Systems by Context and Speech Pattern Modeling (2021)

Abstract: While named entity recognition (NER) from speech has been around as long as NER from written text has, the accuracy of NER from speech has generally been much lower than that of NER from text. The rise in popularity of spoken dialog systems such as Siri or Alexa highlights the need for more accurate NER from speech because NER is a core component for understanding what users said in dialogs. Deployed spoken dialog systems receive user input in the form of automatic speech recognition (ASR) transcripts, and simply applying NER model trained on written text to ASR transcripts often leads to low accuracy because compared to written text, ASR transcripts lack important cues such as punctuation and capitalization. Besides, errors in ASR transcripts also make NER from speech challenging. We propose two models that exploit dialog context and speech pattern clues to extract named entities more accurately from open-domain dialogs in spoken dialog systems. Our results show the benefit of modeling dialog context and speech patterns in two settings: a standard setting with random partition of data and a more realistic but also more difficult setting where many named entities encountered during deployment are unseen during training.

## introduction named entity recognition (ner) is the task of extracting proper names of people, locations, and so on from text or speech @xcite . there has been a lot of work on ner from written text with many systems achieving impressive results @xcite . although, ner from speech has been around for the same time as ner from text (starting with work by @xcite ), accuracy of ner from speech still lags behind the accuracy of ner from text. the rise in popularity of spoken dialog systems such as siri or alexa highlights the need for more accurate ner from speech because ner is a core component for understanding what users said in dialogs. in spoken dialog systems, humans interact with the systems using natural speech to accomplish certain tasks (task-oriented dialog) or just to be entertained (chitchat or open-domain dialog) @xcite . these systems require speech transcripts as input in real-time and the transcripts are obtained using automatic speech recognition (asr) components @xcite . much previous work on ner from speech data, such as broadcast news, applied text-based ner systems to the output of an asr system @xcite . however, ner performance degraded significantly (20 points drop in f1 score) when applying a ner trained on written data to transcribed speech @xcite . this could be because applying text-based ner system to asr output ignores the differences in styles and conventions in written and spoken language @xcite . for example, spoken utterances in spontaneous speech are usually much shorter than written prose so the utterances could be ambiguous when taken out of context. in addition, speech also contains disfluencies, repetitions, restarts and corrections @xcite . besides, text-based ner system may depend on cues such as sentence punctuation and capitalization which are not present in asr transcripts @xcite . furthermore, asr is not error-free and errors in asr transcripts lead to cascading errors in ner @xcite . due to factors such as greater variation in speakers, greater variation in content because of the openended nature of open-domain dialogs, and less professional recording environment, asr transcripts from spoken dialog systems often contain more errors than that from broadcast news, making ner in dialogs a much more challenging task. we propose two models that exploit dialog context and speech patterns which are available in open-domain dialogs from spoken dialog systems to achieve more accurate ner. our results show the benefit of modeling dialog context and speech patterns in two settings: a standard setting with random partition of data and a more realistic but also more difficult setting where there is little overlap between named entities during training and testing. ## related work recent ner models perform well on clean text datasets such as conll @xcite and ontonotes @xcite , but less well on noisy data @xcite such as the wnut dataset @xcite . in term of f1 score, the current state-of-theart model @xcite achieves 93% on the conll dataset but only 49% on the wnut dataset. the overreliance of ner models on the convention of capitalizing named entities @xcite partly explains why they perform poorly on text where capitalization is absent or noisy. in spoken dialog systems, inputs to ner models are asr transcripts which not only lack capitalization and punctuation but also contain transcription errors @xcite . although, joint decoding of asr transcript and ner output @xcite partly lessens the impact of asr errors on ner, detecting named entities in asr transcripts remains a challenging problem @xcite . prior work on ner from asr transcripts focus on reducing asr errors @xcite , exploiting multiple asr hypotheses @xcite , or exploiting additional information such as speech pattern features @xcite . examples of speech pattern features are asr confidence @xcite , pauses, and word durations @xcite . recently, cervantes and ward (2020) used solely prosidic speech features to spot location mentions. our work is similar to @xcite in that we also utilize speech pattern features. however, while @xcite focused on broadcast news speech, our work focuses on spoken dialogs. thus, besides speech pattern features, our models also exploit dialog context for more accurate ner. in addition, @xcite used a separate classifier trained on data from a small set of speakers to derive speech pattern features, so the predicted features may not generalize to more diverse populations. in contrast, our approach is more integrated since the speech pattern features encoder is part of the proposed models thereby encouraging the models to learn features that are more generalizable. ## conclusions named entity recognition for dialogs is difficult because utterances are ambiguous out of context and asr transcripts are noisy due to asr errors and the lack of punctuation and capitalization. we proposed two ner models exploiting dialog context and speech patterns to address the ambiguity issue and asr noise. our results show that context usually improves ner accuracy while speech patterns help in the more difficult but more realistic scenario with many unseen named entities. further studies on exploiting features from non-text modalities are warranted to enhance ner in dialog systems.