Why Aren’t We NER Yet? Artifacts of ASR Errors in Named Entity Recognition in Spontaneous Speech Transcripts (2023)

Abstract: Transcripts of spontaneous human speech present a significant obstacle for traditional NER models. The lack of grammatical structure of spoken utterances and word errors introduced by the ASR make downstream NLP tasks challenging. In this paper, we examine in detail the complex relationship between ASR and NER errors which limit the ability of NER models to recover entity mentions from spontaneous speech transcripts. Using publicly available benchmark datasets (SWNE, Earnings-21, OntoNotes), we present the full taxonomy of ASR-NER errors and measure their true impact on entity recognition. We find that NER models fail spectacularly even if no word errors are introduced by the ASR. We also show why the F1 score is inadequate to evaluate NER models on conversational transcripts.

## introduction the performance of nlp models tends to deteriorate significantly when the models are applied to the raw outputs of the automatic speech recognition (asr) system. we coin the term asr-nlp gap to describe this phenomenon. despite unprecedented advances in modern language models, the transcript of a spontaneous human-human conversation remains an insurmountable challenge for most models. this is particularly true for named entity recognition (ner) models, which struggle to retrieve even the most basic entity mentions from spontaneous speech. three primary factors contribute to the existence of the asr-nlp gap. firstly, the structure of spontaneous human conversations is diametrically different from the prescriptive written language used to train language models. these models can use the grammatical structure present in the training corpora, such as part-of-speech sequences, dependency trees, and dialog acts. on the other hand, spontaneous conversations lack sentence structure. they contain repetitions, back-channeling, phatic expressions, and other artifacts of turn-taking. the second challenge comes from the original asr output containing neither punctuation nor sentence segmentation. these have to be restored by an auxiliary downstream model. thus, nlp models trained on prescriptive written text or scripted conversations already have to process the out-ofdomain input. the third problem stems from asr systems injecting word errors into the transcript. due to efficiency requirements, most asr systems use unsophisticated language models such as ngram models with limited vocabulary. thus, many utterances in the input audio may be unrecognized and deleted from the output, while other utterances may cause substitutions or insertions of erroneous tokens into the output. consider the following sentence: "i am to see [dr smith] person at [9 am] time on [monday, may 14th] date ". the ner model 2 correctly recognizes three entity spans in the sentence. compare this to the ner spans recognized in the sentence, which is far more likely to be produced by the asr: "i am to see doctor @xcite person at nine i am on [monday] date [uhm] org yeah [monday] date may for teen." two entity spans have been cut short, an incorrect label has replaced one span's label, and the model recognized a filler uhm as the entity org! with a few more asr errors and lowercase output, the model does not recognize a single entity in the output of the asr: "i am to see doctor uhm doctor smith at nine i am on man day may for teen." the main problem is that asr errors are very "unnatural" from the point of view of the ner model because they tend to break the grammar of the sentence on which the ner model depends. one of the most consequential errors made by the asr is the confusion about the part-of-speech tag. consider possible asr errors in the sentence "my [second] ordinal visit is [wednesday] date at [half past one] time ." changing the personal pronoun "my" to the noun "may" forces the ner model to recognize a date span, which is reasonable. but if the asr changes the preposition "at" into a verb "add," the ner model loses the ability to recognize the utterance "half past one" as time because of the lack of the preceding preposition. similarly, changing "half past one" to "[one thirty] time " retrieves the time span, but an asr error confusing the numeral "one" with the conjunction "when" produces "[wednesday] date at when [thirty] date ." if, however, the same word is mistakenly recognized as the verb "want," the ner model produces "[wednesday] date at want [thirty] cardinal ". unfortunately, the problems mentioned above cannot be easily solved. word error rates (wer) of asr systems remain high for spontaneous human conversations @xcite . recently announced results claiming wers at the level of 5% apply to conversations with digital assistants, where spoken utterances are imperative phrases with limited vocabulary. these results are not representative of spontaneous human open dialogues, which lack the rigid grammatical phrase structure and contain fillers, back-channeling, repetitions, hesitation markers, and other elements which are a part of spontaneous speech. the interplay of two phenomena makes the processing of spontaneous speech transcripts with nlp models so challenging. on the one hand, every nlp model is inherently flawed and produces errors (such as not recognizing an instance of an entity). on the other hand, the asr system injects errors in the form of insertions, deletions, and substitutions. this changes the structure and semantics of transcribed speech and introduces yet another source of errors: alignment. in order to measure the quality of the ner model on the transcript, one has to align tokens between gold transcripts and the asr output to match entity spans. this process may produce artifacts that significantly skew the results of the evaluation. the evaluation of the ner task is usually performed using precision, recall, and the f 1 score. unfortunately, these measures are of limited use for processing spontaneous conversation transcripts because they confound two independent factors contributing to the errors mentioned above: the inability of the ner model to recognize a span as an entity and the word error introduced by the wrong transcription of a token. our paper is a reality check on the state of named entity recognition in spontaneous speech transcripts. using popular benchmark datasets, we show how state-of-the-art language models fail to discover entity spans in transcripts of spontaneous speech. we identify several artifacts of asr errors with respect to entity recognition. we measure the propensity of each type of artifact to influence the recognition of named entities. this approach brings us closer to understanding the true reasons for ner model failures on spontaneous speech transcripts. we argue that misalignment artifacts are essential characteristics of the performance of nlp models and should be considered when evaluating downstream nlp models on spontaneous speech transcripts. ## entity span alignment we measure the loss of entity spans recognized in the asr output compared to those recognized in the gold transcript. thus, we must perform token alignment between the asr output and the gold transcript, as they may differ in the number of tokens. alignment is performed after diarisation (separating speakers' utterances into separate channels) for each channel independently. we use a greedy alignment procedure. we begin by running the ner model on the gold transcript and tagging each token in the transcript using the iob scheme (b -beginning of an entity span, i -inside an entity span, o -outside of an entity span). next, we collapse all adjacent i-tags so that each channel is represented by a sequence of b-tags and o-tags. we repeat the same procedure for the asr output and then align both transcripts. the alignment of gold transcripts, normalized gold transcripts, and the asr output is performed by the fstalign @xcite and the kaldialign ( żelasko and guo, 2021) libraries, with minor additional corrections. all transcripts are matched at the level of tokens. in the remainder of the paper, we will use the following terminology @xcite . for the asr errors, we will distinguish the following types of errors: in parallel, the ner model can introduce the following errors: let us now describe in detail all possible combinations of the above asr and nlp errors and their impact on the recognition of named entities. for the sake of clarity, we will only consider artifacts of the asr-nlp gap within a single entity span. detailed examples of every combination of asr-nlp errors discovered in the earnings-21 dataset are presented in appendix a. firstly, let us consider a scenario where the gold transcript and the asr output are perfectly aligned, i.e., all tokens are correctly recognized. the gold transcript contains the utterance "second b-date quarter b-date twenty b-date twenty b-date ." the following entity span errors are possible ( consider a situation where the asr inserts a token into the gold transcript. obviously, there is a mismatch in the number of tokens in the gold transcript and the transcription. let us assume that the utterance "nextstart b-org group i-org " has been mistakenly transcribed as "next door group." the asr can delete a token from the gold transcript, resulting in a possible misalignment. in this scenario, full matching is impossible because the gold transcript will contain an unmatched token. similarly, an entity span cannot be hallucinated or fully substituted. let us assume that the gold transcript utterance "next b-org door i-org group i-org " has been mistakenly transcribed as "@xmath0.e., the asr failed to recognize the "door" token). finally, the ner model can hallucinate an entity span where the gold transcript has no entities. as we can see, the number of possible mistakes is large, and it is not obvious which scenarios are common or rare. in other words, if we are to develop more robust models for named entity recognition in the transcripts of spontaneous speech, we need to understand which scenarios are the most impactful for the ner task. in the next sections, we present experiments that try to present a much more detailed and nuanced view of asr and ner errors. ## datasets we use three datasets in our experiments. we decided to omit the conll-2003/conll++ @xcite dataset because it is annotated with only four classes of entities. unfortunately, the three listed datasets are the only publicly available datasets that contain audio segments and transcripts annotated with entity types. one may argue that these datasets are not representative of spontaneous conversations. for instance, earnings-21 transcripts sound heavily scripted, and the interlocutors present speeches rather than a free exchange of utterances. while this is true, at the same time, these three datasets present the closest that researchers can get to conversational audio transcripts with annotated entity spans. there are datasets with audio recordings annotated with entity spans, but these datasets are not in the domain of spontaneous speech. in recent years we are observing significant progress in named entity recognition in transcripts of scripted speech. this progress is made possible mostly due to the publication of annotated datasets. yadav et al. present a dataset consisting of ted talks, mozilla common voice recordings, librispeech audiobook recordings, and voxforge recordings. as the authors observe, ner models achieve promising results on these transcripts (probably due to the fact that the input transcript is semantically similar to the typical training data for ner models). the same dataset is used by @xcite to illustrate the error correction model. recently, annotated transcripts of speech (albeit non-conversional) have been released for scandinavian languages @xcite , for french @xcite , and for chinese @xcite . it is worth mentioning that ner task has been added to the recent spoken language understanding evaluation (slue) benchmark @xcite . unfortunately, the annotation covers a small subset of the voxpopuli dataset, which is not representative of spontaneous speech, the voxpopuli is the set of recorded speeches in the european parliament. entity classes annotated in the above datasets can be broadly divided into closed-domain and opendomain types. closed-domain entity classes can be regarded as almost gazetteers, i.e., these are classes for which a vast majority of entities can be listed. examples of closed-domain entity classes include geographical locations or first names (since the distribution of us first names follows a power law distribution @xcite , a relatively small number of first names represents the majority of first names encountered in the dataset). on the other hand, open-domain entity classes cannot be summarized using a gazetteer. this is the case with numbers, product names, money, or organizations. ## experiments one might argue that the most important variable influencing the performance of downstream nlp tasks on a transcript is the choice of a particular asr system. however, we do not find this to be the case. the asr-nlp gap is equally pronounced for all major commercial asr systems. in our experiments, we choose the asr offered by microsoft due to its lowest reported wer on the earnings-21 dataset @xcite . ## related work in our opinion, the nlp research community has an overly optimistic view of the wers introduced by asr systems. recent experiments show that wers in transcripts of spontaneous human speech is much higher than expected. for instance, @xcite showed that a transcript of a standard gsm phone call conversation is subject to a 16%-20% error rate. del @xcite confirm this result and report how wers differ between different types of entity spans. spans related to date, time, and ordinal numbers were observed to have a lower wer than entities related to proper names. facility names, organizations, and personal names demonstrate a very high wer of 30%-50%. @xcite also released a library for using finite state transducers (fsts) to account for different representations of the same entity (2020 vs. twenty twenty) among asrs. these findings are in stark contrast to initial reports. for instance, @xcite reported named entity recognition in switchboard corpus to be within 5% from a system evaluated on clean textual data. similarly, @xcite claims to have achieved approximately 0.90 f 1 for recognizing phone numbers and 0.70 f 1 for recognizing money mentions in the transcripts from the at&t how may i help you? system under 27.4% wer ratio. @xcite apply ner models to french corpora and achieve 0.74 f 1 for a relatively broad set of named entities. precision, recall, and f 1 scores are standard metrics for reporting ner model performance in nlp. however, these metrics can produce unreliable scores where entity spans are marked on spontaneous human conversation transcripts due to the presence of conversational artifacts (repetitions mentioned above, backchanneling, phatic expressions). an example of entity span tagging where the f 1 metric produces highly misleading scores is presented in section 6. to account for the presence of these artifacts, message understanding conference (muc) @xcite ; @xcite ) introduced metrics that allow for partial matching of an entity span. muc defines six categories of partial matching based on the degree of span overlap, the type of the matched entity, and the strictness of expectations, as outlined by @xcite . recently, this problem has been addressed by @xcite who argues for the use of slot error rates. to the best of our knowledge, @xcite was the first to attempt to incorporate named entity recognition into the automatic speech transcription process. the authors tagged the asr dictionary with named entity tags (since asr cannot produce any words not present in its dictionary). this initial approach has been superseded by methods aiming at training end-to-end joint models for asr and ner, as proposed by @xcite , @xcite @xcite . the authors train asr systems to predict transcription tokens and their part-of-speech or named entity tags in these works. ## limitations obviously, the work presented in this paper is limited to transcripts of spontaneous conversations in english. since we are investigating the problem of named entity recognition, we have to point out that there are practically no datasets of human conversations (both audio and transcripts) annotated with entity spans apart from swne, ontonotes and earnings-21, the three datasets used in our paper. these datasets are relatively small, and the distribution of the frequency of appearance of entity classes is extremely skewed, with several entity classes represented by a handful of examples. another significant limitation of the results reported in this paper is the choice of metric. following the common practice in the nlp community, we have chosen the f 1 score as the primary metric of entity recognition. however, this metric is questionable in the context of ner recognition in asr transcripts because it is highly dependent on two factors: the wer produced by the asr and the definition of span alignment. consider a gold transcript annotation "john b-person f. i-person kennedy i-person " and the asr output with "f." transcribed as "eh" annotated as follows: "john b-person eh kennedy b-person ." should this annotation be considered correct? the original person entity starting at "john" is only partially matched, and a new person entity starting at "kennedy" is introduced in the asr output. consider another gold annotation of the following transcript: "second b-date quarter i-date twenty i-date twenty i-date ," which the ner model tags as follows: we conduct experiments with the asr on audio files from the earnings-21 dataset. these files are recorded at 11 khz-44 khz, while typical call center conversations are recorded at 8 khz-16 khz. unfortunately, training datasets with recording characteristics resembling real-world usage scenarios are unavailable. we also do not address the problem of racial, gender, and age disparity @xcite due to the lack of availability of sufficiently representative and inclusive datasets. it is, however, to be expected that the performance of the asr deteriorates for the recordings of speakers other than male speakers of general american. ## conclusions our work provides a thorough, albeit pessimistic, reality check on the named entity recognition in conversational transcripts. our first conclusion is straightforward: currently available ner models are not trained on representative data (due to the lack of annotated datasets), and their performance on transcripts of spontaneous conversations is much worse than their performance on written language. importantly, this failure cannot be attributed solely to the presence of asr word errors. as we show, ner models exhibit very high entity wers even on gold transcripts, where no asr errors are present. when the transcript contains asr insertions, substitutions, or deletions, the entity recognition rates fall to the level where ner models become unusable in downstream tasks. secondly, we conclude that a completely new approach is required to meaningfully measure the quality of ner models on conversational transcripts. traditional metrics, such as f 1 score or entity wer do not account for the intricate interplay of factors (ner errors, asr errors, artifacts of spontaneous speech) and do not provide a useful insight into the model's performance. we need to design a more complex evaluation scheme that would take into account the token alignment errors, partial entity span matchings, asr word errors, and ner errors.