Breaking the Boundaries: A Unified Framework for C hinese Named Entity Recognition Across Text and Speech (2024)

Abstract: In recent years, with the vast and rapidly increasing amounts of spoken and textual data, Named Entity Recognition (NER) tasks have evolved into three distinct categories, i.e., text-based NER (TNER), Speech NER (SNER) and Multimodal NER (MNER). However, existing approaches typically require designing separate models for each task, overlooking the potential connections between tasks and limiting the versatility of NER methods. To mitigate these limitations, we introduce a new task named Integrated Multimodal NER (IMNER) to break the boundaries between different modal NER tasks, enabling a unified implementation of them. To achieve this, we first design a unified data format for inputs from different modalities. Then, leveraging the pre-trained MMSpeech model as the backbone, we propose an **I**ntegrated **M**ultimod**a**l **Ge**neration Framework (**IMAGE**), formulating the Chinese IMNER task as an entity-aware text generation task. Experimental results demonstrate the feasibility of our proposed IMAGE framework in the IMNER task. Our work in integrated multimodal learning in advancing the performance of NER may set up a new direction for future research in the field. Our source code is available at https://github.com/NingJinzhong/IMAGE4IMNER.

## introduction named entity recognition (ner) @xcite is a fundamental and significant task in the field of natural language processing and has been extensively studied to address the challenges posed by real-world text data. chinese ner (cner) @xcite , a significant subdomain of ner, specifically deals with challenges unique to chinese, such as no clear word boundaries and the ambiguity from homophones and polyphones, drawing significant academic focus @xcite @xcite . traditionally, named entity recognition (ner) tasks have concentrated on text-based ner (tner) @xcite @xcite . however, as the volume of audio data increases, there has been a growing interest in speech ner (sner) @xcite @xcite , which focuses on extracting named entities from speech, and multimodal ner (mner) @xcite , which involves extracting entities from both speech and text. ## methodology in this section, we first introduce the imner task definition. then we detail the implementation of image, including its overal structure as depicted in figure 2 . ## conclusions in our study, we introduce the integrated multimodal ner (imner) task, bridging the gap between text-based ner, speech ner, and multimodal ner to enable a unified approach to these three distinct tasks. by designing a novel unified data format and leveraging the pre-trained mm-speech as backbone, we introduced the image framework, transforming the chinese imner task into an entity-aware text generation task. experimental results reveal the effectiveness of image, marking a significant step forward in integrated multimodal learning for ner, which may shed light on future research in this research domain.