On the Use of External Data for Spoken Named Entity Recognition (2022)

Abstract: Spoken language understanding (SLU) tasks involve mapping from speech signals to semantic labels. Given the complexity of such tasks, good performance is expected to require large labeled datasets, which are difficult to collect for each new task and domain. However, recent advances in self-supervised speech representations have made it feasible to consider learning SLU models with limited labeled data. In this work, we focus on low-resource spoken named entity recognition (NER) and address the question: Beyond self-supervised pre-training, how can we use external speech and/or text data that are not annotated for the task? We consider self-training, knowledge distillation, and transfer learning for end-to-end (E2E) and pipeline (speech recognition followed by text NER) approaches. We find that several of these approaches improve performance in resource-constrained settings beyond the benefits from pre-trained representations. Compared to prior work, we find relative improvements in F1 of up to 16%. While the best baseline model is a pipeline approach, the best performance using external data is ultimately achieved by an E2E model. We provide detailed comparisons and analyses, developing insights on, for example, the effects of leveraging external data on (i) different categories of NER errors and (ii) the switch in performance trends between pipeline and E2E models.

## introduction named entity recognition (ner) is a popular task in natural language processing. it involves detecting the named entities and their categories from a text sequence. ner can be used to extract information from unstructured data, which can also be used as features for other tasks like question answering @xcite and slot filling for taskoriented dialogues @xcite . by a text ner model; "e2e" refers to approaches that directly map from speech to ner-tagged text. the "baseline" and "text ner" numbers are from previously established baselines @xcite . thanks to pre-trained text representations such as bert @xcite , text-based ner has recently improved greatly @xcite . spoken ner, on the other hand, is not as well-studied. it has the added challenges of continuous-valued and longer input sequences and, at the same time, provides opportunities to take advantage of acoustic cues in the input. a recent study @xcite shows that there is still 10-20% absolute degradation in the f1 scores of spoken ner models compared to text-based ner using gold transcripts (see figure 1 ) despite using large pre-trained speech representation models. closing this gap remains a critical challenge. in this work, we study the potential benefits of using a variety of external data types: (a) plain speech audio, (b) plain text, (c) speech with transcripts, and (d) text-based ner data. we benchmark our findings against recently published baselines for ner on the voxpopuli dataset of european parliament speech recordings @xcite and also introduce baselines of our own. we observe improvement from leveraging every type of external data. our analysis also quantifies the pros and cons of the pipeline (speech recognition followed by text ner) and end-to-end (e2e) approaches. the key improvements are summarized in figure 1 . specific contributions include: (i) unlike previous work, we devote equal effort to improving both pipeline and e2e approaches. (ii) we present experiments using various external data types and modeling approaches. (iii) overall, we obtain f1 improvements of up to 16% for the e2e model and 6% for the pipeline model over previously published baselines, setting a new state of the art for ner on this dataset. (iv) we benchmark the advantage of selfsupervised representations (ssr) over a baseline that uses standard spectrogram features. ssr gives relative improvements of 36%/31% for pipeline/e2e models, respectively. to our knowledge, prior work has not directly measured this improvement over competitive baselines tuned for the task. (v) we establish that e2e models outperform pipeline approaches on this task, given access to external data, while the baseline models without the external data have the opposite relationship. (vi) we provide a detailed analysis of model behavior, including differences in error types between pipeline and e2e approaches and the reasoning for the superiority of e2e over pipeline models when using external data but not in the baseline setting. ## methods spoken ner involves detecting the entity phrases in a spoken utterance along with their tags. the annotations include the text transcripts for the audio and the entity phrases with their corresponding tags. spoken ner, like any other slu task, is typically tackled using one of two types of approaches: (i) pipeline and (ii) end-to-end (e2e). as shown in fig. 2 , a pipeline approach decodes speech to text using asr and then passes the decoded text through a text ner module, whereas an e2e system directly maps the input speech to the output task labels. each approach has its own set of advantages and shortcomings. pipeline systems can enjoy the individual advances from both the speech and the text research communities, whereas combining two modules increases inference time, and propagation of asr errors can have unexpected detrimental effects on the text ner module performance. on the other hand, e2e models directly optimize a task-specific objective and tend to have faster inference. however, such models typically require a large amount of task-specific labeled data to perform well. this can be seen from previous papers on e2e ner @xcite , where using at least 100 hours of labeled data is typical. ## discussion and analysis the baseline results are not surprising: the limited labeled data is not enough for the baseline e2e approach, but the pipeline model can leverage a strong text representation model, which gives it an edge. improvements to these models can be attributed to either (i) a better speech-to-text conversion or (ii) a better semantic understanding of the input content. next, we use this distinction to understand the observed improvements from using external data. ## conclusion we have explored various ways to use different external data types that improve both pipeline and e2e methods for spoken ner. the bestperforming model when using external data is an e2e approach. this is one of the few results in the literature thus far showing better performance for e2e over pipeline methods that use state-of-the-art modules for spoken language understanding. we develop some insights into this difference; we notice that pipeline models are adversely affected by false positives and that leveraging external data improves the semantic understanding capability of the e2e models. we hope that our work provides guiding principles for researchers working on slu tasks in similar low-resource domains when some form of external data is abundant. this work also leaves some interesting research questions for future work. for example, we see minor improvements between 100h and 500h of external data (see tab. 5 and 6 ), which suggests the question: what is the smallest amount of external data needed to obtain significant improvements in ner performance? additionally, one preliminary experiment with external, out-of-domain text ner data (ontonotes 5.0) fails to improve the text ner performance, suggesting the challenges of dealing with out-of-domain datasets. the scenario where we have access to out-of-domain external data is common but challenging, and warrants further study. from the modeling perspective, better fine-tuning strategies for wav2vec2.0 in low supervision settings have been proposed for asr @xcite ; it would be interesting to explore how these findings may transfer to an slu task. error category outputs from e2e model gt predicted correct asr, over detection and this means that you look and tell us honestly what does it mean if you start @ three years ] later [('when', 'three years')] this means that you look and tell us honestly what does it mean if you start @ three years later ] [('when', 'three years later')] correct asr, missed detection the situation in the % drc ] is indeed terrible and it has been this way for quite a while and i am deeply concerned about the handling of the current issue with regard to the % kasai ] province [('place', 'drc'), ('place', 'kasai')] the situation in the drc is indeed terrible and it has been this way for for quite a while and i am deeply concerned about the handling of the current issue with regard to the a province [] correct asr, false detection and yet @ one month ] after we adopted our compromise the council did not put it on the agenda did not even present it i used this time to talk to the member states and the presidencies [('when', 'one month')] still @ one month ] after we voted a compromise the ' council ] did not put it on the agenda did not even present i use this time to talk with the member states and the presidency [('when', 'one month'), ('org', 'council')] incorrect asr, false detection it has nothing to do with religion but it has all to do with patriarchy [] it has nothing to do with religion but it has all to do with % turkey ] [('place', 'turkey')]