C hinese Spoken Named Entity Recognition in Real-world Scenarios: Dataset and Approaches (2024)

Abstract: Spoken Named Entity Recognition (NER) aims to extract entities from speech. The extracted entities can help voice assistants better understand user’s questions and instructions. However, current Chinese Spoken NER datasets are laboratory-controlled data that are collected by reading existing texts in quiet environments, rather than natural spoken data, and the texts used for reading are also limited in topics. These limitations obstruct the development of Spoken NER in more natural and common real-world scenarios. To address this gap, we introduce a real-world Chinese Spoken NER dataset (RWCS-NER), encompassing open-domain daily conversations and task-oriented intelligent cockpit instructions. We compare several mainstream pipeline approaches on RWCS-NER. The results indicate that the current methods, affected by Automatic Speech Recognition (ASR) errors, do not perform satisfactorily in real settings. Aiming to enhance Spoken NER in real-world scenarios, we propose two approaches: self-training-asr and mapping then distilling (MDistilling). Experiments show that both approaches can achieve significant improvements, particularly MDistilling. Even compared with GPT4.0, MDistilling still reaches better results. We believe that our work will advance the field of Spoken NER in real-world settings.

## introduction as one of the core tasks in spoken language understanding, spoken named entity recognition (ner) aims to extract entities like person names (per), locations (loc), and organizations (org) from speeches @xcite . the extracted entities can help voice assistants better grasp the intent behind users' questions or instructions, thereby benefiting various downstream natural language processing (nlp) tasks such as information retrieval @xcite and question * corresponding author answering @xcite . while significant advancements have been made in chinese text-based ner @xcite , chinese spoken ner still faces substantial challenges. the main issue lies in the disparity between current research in chinese spoken ner and its applications in real-world scenarios. this gap hinders the application and development of spoken ner. firstly, a key challenge is that current chinese spoken ner datasets do not closely match real-world scenarios. for chinese spoken ner, the only significant dataset currently available is aishell-ner, introduced by @xcite . this dataset was built upon the automatic speech recognition (asr) dataset aishell-1, which already included paired speech-text data. they utilized the msra @xcite guidelines to annotate entities within the text. however, aishell-ner does not accurately reflect real-life conditions. on the one hand, speeches in aishell-ner are reading speeches recorded in quiet environments. in contrast, in applications like using siri or other voice assistants, human speech is natural and often includes various background noises and informal expressions like interjections, stutters, and grammatical errors. on the other hand, the aishell-ner dataset primarily encompasses topics related to finance and news, whereas conversations in real life cover a broader range of subjects. these differences lead to a gap between the research on aishell-ner and its applications in real-world scenarios. in this paper, we address the issue of the lack of annotated spoken ner datasets for real-world scenarios by introducing a real-world chinese spoken ner dataset, rwcs-ner. it covers two scenarios: open-domain daily conversation (dc) and task-oriented intelligent cockpit instructions (ici). rwcs-ner facilitates the evaluation of spoken ner models in real-world contexts, thereby bridging the gap between spoken ner researches and its applications. secondly, beyond the data shortage, current spoken ner models are not specifically tailored for real-world scenarios and often underperform in such environments. spoken ner typically follows a pipeline process where speech is first converted into text by an asr model, followed by the use of an ner model to identify entities within the transcribed text. considering the scarcity of annotated spoken ner data, researchers have sought to leverage unlabeled data to help spoken ner. @xcite successfully used the self-training with unlabeled text data to help the ner model in pipeline. however, their approach does not fully align with real-world conditions. in practice, ner is executed on asr outputs, which inevitably contain transcription errors from asr. in contrast, in the self-training approach by @xcite , the ner models within the pipeline are trained on clean texts, devoid of any asr inaccuracies. the discrepancy, known as exposure bias @xcite , will lead the model trained on clean text to struggle with the error-prone text from asr in real-world scenarios. furthermore, in the pipeline workflow, transcription errors from asr can propagate to the ner stage, disrupting the ner model's ability to accurately identify entities. this issue is also not considered in @xcite . to help spoken ner on texts with asr errors in real-world scenarios, we first introduce selftraining-asr by conducting self-training on asr outputs to mitigate the exposure bias issue. furthermore, to reduce the impact of asr errors on the ner model in the pipeline process, we propose a novel mapping then distilling (mdistilling) approach. we then evaluate various approaches on rwcs-ner. experiments demonstrate that both our self-training-asr and mdistilling approaches achieve significant improvements. notably, mdistilling effectively alleviates the impact of asr errors on ner in the pipeline workflow. finally, we also evaluate the performance of llms, i.e., including qwen1.5 @xcite , baichuan2 @xcite , gpt3.5, and gpt4.0 @xcite , on our rwcs-ner. results show that gpt4.0 has reached performance comparable to that of supervised models on clean texts. but our mdistilling is more competitive than gpt4.0 in real-world scenarios where texts come from the asr model. in summary, our work makes the following contributions: ## construction of rwcs-ner in this section, we detail the construction process of our dataset. we consider two typical real-world scenarios: open-domain daily conversation (dc) and task-oriented intelligent cockpit instruction (ici). dc typically involves dialogues between individuals about various topics. in contrast, ici involve interactions between humans and machines, where humans interact with machines to accomplish certain tasks during driving. our selection of dc is driven by its prevalence in the conversation scenarios, encompassing a range of spoken expressions like interjections, stuttering, and grammatical errors, which are absent in formal written text. ici is chosen due to the rising trend of intelligent driving and the increasing reliance on speech instructions for controlling in-car devices. effective entity recognition in these speech instructions is pivotal for in-car systems to assist users. in addition, ici data is relatively scarce and, to our knowledge, no publicly accessible ner dataset for intelligent cockpit instruction exists. ## conclusion in this paper, we present rwcs-ner, a chinese spoken ner dataset designed for real-world scenarios, focusing on daily conversation and intelligent driving. to boost the performance of ner in the pipeline, especially when dealing with asr noise, we introduce two novel approaches: selftraining-asr and mdistilling. we compared our approaches with mainstream pipeline models, including gpts, on rwcs-ner. our findings indicate that both our approaches, particularly mdistilling, significantly enhance performance. however, on the whole view, these approaches still leaves room for improvement on rwcs-ner. our work contributes to the advancement of chinese spoken ner by providing new data and approaches, aiming to spotlight the importance of spoken ner in real-world applications.