Medical Spoken Named Entity Recognition (2025)

Abstract: Spoken Named Entity Recognition (NER) aims to extract named entities from speech and categorise them into types like person, location, organization, etc. In this work, we present *VietMed-NER* - the first spoken NER dataset in the medical domain. To our knowledge, our Vietnamese real-world dataset is the largest spoken NER dataset in the world regarding the number of entity types, featuring 18 distinct types. Furthermore, we present baseline results using various state-of-the-art pre-trained models: encoder-only and sequence-to-sequence; and conduct quantitative and qualitative error analysis. We found that pre-trained multilingual models generally outperform monolingual models on reference text and ASR output and encoders outperform sequence-to-sequence models in NER tasks. By translating the transcripts, the dataset can also be utilised for text NER in the medical domain in other languages than Vietnamese. All code, data and models are publicly available.

## introduction named entity recognition (ner) targets extracting named entities (ne) from text and categorizing them into types like person, location, organization, etc. initially studied in written language, recent attention has turned to study spoken ner @xcite , which aims to extract semantic information from speech. however, spoken ner has limited literature compared to ner on written text data @xcite . spoken ner is particularly challenging, firstly due to the impact of word segmentation on results. the medical vocabulary poses difficulties with numerous confused monosyllabic and polysyllabic words. for instance, the word "đường" alone could denote "sugar" (chemical), "street" (location), or be part of a compound word like "đường tiêu hóa" -"gastrointestinal" (anatomy). this confusion has also been reported in chinese spoken ner by @xcite . further, data quality control and annotation consistency have been problematic, with some entities tagged in one sentence but not in others, and full nes inconsistently tagged as multiple sub-nes @xcite @xcite . finally, obtaining accurate medical ner from natural speech is challenging due to the lack of punctuation @xcite , speech disfluencies @xcite , context, and the complexity of medical terms. as for the medical domain, to the best of our knowledge, there is no dataset available for medical spoken ner. the only related work we found, @xcite , published a ner evaluation benchmark using an english general-domain conversational dataset, switchboard @xcite and @xcite , for the task of audio de-identification specifically targeting personal health identifiers. to address this gap, we introduce vietmed-ner, a medical spoken ner dataset built on the real-world medical automatic speech recognition (asr) dataset vietmed (le-duc, 2024), featuring 18 medically-defined entity types. in the era of the advanced in-context learning capabilities of large language models (llms) and human-level text-to-speech technologies, the dataset, with entity positional labels maintained during translation, is applicable not only to vietnamese but also to other languages (see appendix c). this enables various real-world applications, including: search engines @xcite , content classification for news providers @xcite , medical asr error correction @xcite , audio @xcite and content recommendation systems @xcite . our contributions are as follows: ## related works traditionally, spoken ner has been done using a pipeline methodology, also known as cascaded approach, starting with an asr stage, followed by ner applied to the generated transcriptions @xcite . another variant of the cascaded approach involves embedding specific entity expressions into the lexicon, thereby improving the language model's ability to accurately recognize these expressions @xcite . besides, end-to-end ner has recently garnered some attention within the research community. this approach seeks to optimize asr and ner processes simultaneously, offering a potentially more efficient alternative to traditional pipeline methods by harnessing the ability of trainable acoustic features. however, its accuracy advantage over the cascaded approach remains a subject of debate, and the end-to-end training setup introduces additional complexity @xcite . ## experimental setups we employ the cascaded (two-stage) pipeline for spoken ner: a hybrid asr model transcribes audio into text and then the transcribed text is fed into a text ner model. ## experimental results phobert_large, it performed similarly, likely due to more pre-training data. the pre-trained multilingual model xlm-r_large achieved the best performance with an f1 score of 0.74 on reference text and 0.58 on asr output, while xlm-r_base performed worse than phobert_base-v2. this gap is explained by the larger pre-training data (2.5tb multilingual data for xlm-r vs. 140gb monolingual data for phobert_base-v2). our results with phobert_base-v2 and xlm-r_large confirmed that pre-trained multilingual representations improve performance on medical spoken ner tasks, similar to other language-specific downstream tasks by @xcite . however, multilingual models may face a transfer-dilution trade-off (conneau et al., 2020), where they lack the capacity to learn effective multilingual representations. in other words, for a fixed sized model, the per-language performance decreases as we increase the number of languages @xcite . to address this trade-off, multilingual models should possess sufficient capacity, necessitating an adequately large model size (chen and chen, 2024). this is evident in the performance comparison between phobert_base-v2 and xlm-r_base, as seen in other language-specific downstream tasks by @xcite . ## error analysis we performed an error analysis using the bestperforming models. ## conclusion in this work, we present vietmed-ner -the first spoken ner dataset in the medical domain. our dataset contains 18 entity types, including both conventional and newly defined entity types for real-world medical conversations. our results show that pre-trained multilingual models typically outperform monolingual models on both reference text and asr output if the multilingual models are sufficiently large to learn multilingual representations. additionally, encoders generally demonstrate better performance than seq2seq models in the ner task. finally, while pre-trained audio data impacts asr output, it does not significantly impact ner performance in the cascaded setting.