Overview of the ROCLING 2022 Shared Task for C hinese Healthcare Named Entity Recognition (2022)

Abstract: This paper describes the ROCLING-2022 shared task for Chinese healthcare named entity recognition, including task description, data preparation, performance metrics, and evaluation results. Among ten registered teams, seven participating teams submitted a total of 20 runs. This shared task reveals present NLP techniques for dealing with Chinese named entity recognition in the healthcare domain. All data sets with gold standards and evaluation scripts used in this shared task are publicly available for future research.

## introduction named entity recognition (ner) is a traditional and fundamental nlp task in the information extraction domain that locates and identifies mentions of named entities (e.g., person, organization, and location) in unstructured texts. the ner task is usually regarded as a sequence labeling problem, where entity boundaries and category labels are jointed predicted. chinese ner is correlated with word segmentation, since named entity boundaries are also word boundaries. due to a lack of delimiters between characters and a lack of conventional features like capitalization, chinese ner is more difficult to process than english ner. incorrect word segmentation will cause error propagation in ner. for example, "思覺失調症" (schizophrenia) is a kind of mental disorder that affects the way a person thinks, feels, perceives reality, and relates to others. this named entity may be incorrectly segmented into three words: "思覺" (thinking and feeling), "失調 " (disorder) and "症 " (disease), resulting in fail to recognize it as a named entity belonging to disease type. character-based methods have been found to outperform wordbased approaches for breaking through this word segmentation limitation in chinese ner @xcite @xcite . various methods have been proposed to tackle chinese ner tasks. in addition to machine learning approaches, such as hmm (hidden markov model) @xcite , markov logistic network @xcite , and crf (conditional random field) @xcite , deep learning techniques have been widely used, with mostly promising results. a character-based lstm (long short-term memory)-crf model with radicallevel features was proposed for chinese ner @xcite . the bilstm (bidirectional lstm)-crf model was trained based on character-word mixed embeddings to improve the recognition effectiveness of chinese ner @xcite . a bilstm-crf model with a selfattention mechanism was proposed to integrate part-of-speech labeling information to capture the semantic features of input sequences for chinese clinical ner @xcite . a residual dilated cnn (convolution neural network) with crf was also presented to enhance chinese clinical ner in terms of computational performance and training time @xcite . a bert-bilstm-crf model was proposed to use bert embedding for character representation and to train the bilstm-crf model to recognize complex named entities @xcite . prior to scheduling a doctor's appointment for diagnosis and treatment of a perceived medical issues, people frequently seek healthcare-related information online from health-related news articles, digital health services, and medical question-answering forums. domain-specific healthcare information usually includes many proper names. these often take the form of named entities such as "三酸甘油酯" (triglyceride), "電 腦斷層掃描" (computer tomography, ct) and "靜 脈 免 疫 球 蛋 白 注 射 " (intravenous immunoglobulin, ivig), presenting language processing challenges for healthcare-related applications. responding to this pronounced challenge in the healthcare domain, the rocling-2022 conference features a chinese healthcare ner task, providing an evaluation platform for the development and implementation of chinese healthcare ner system. given a chinese sentence, the ner system is expected to automatically recognize healthcare entities such as symptoms, chemicals, diseases, and treatments. the rest of this article is organized as follows. section 2 provides a description of the chinese healthcare ner shared task. section 3 introduces the constructed data sets. section 4 describes the evaluation metrics. section 5 compares evaluation results from the various participating teams. finally, we conclude this paper with findings and offer future research directions in section 6. ## task description the goal of this shared task is to develop and evaluate the capability of a chinese healthcare ner recognizer. a sentence containing at least one named entity is given as the input. the recognizer should predict the named entity's boundaries and category for each given sentence. we use the common bio (beginning, inside, and outside) format for the ner task. the b-prefix before a tag indicates that the character is the beginning of a named entity and the i-prefix before a tag indicates ## data preparation the chinese healthner corpus @xcite was used as the training set. it includes 30,692 sentences with a total around 1.5 million characters or 91,700 words. the data was sourced from articles on websites that provide healthcare information, on-line health news and medical question/answer forums. after manual annotation, this corpus consists of 68460 named entities across 10 defined entity types. we use the existing named entities in the chinese healthner corpus as the query terms and to find the corresponding articles in chinese wikipedia (zh_tw version). the first paragraph in the wiki articles was segmented into sentences for manual annotation. three graduate students majoring in electrical engineering were trained in the named entity tagging task, producing a fleiss' kappa value of inter-annotator agreement of 89%. all annotators were asked to discuss differences and seek consensus. when agreement was reached, each annotator was then asked to process sentences individually. as a result, our constructed test set includes 3,205 sentences with a total of 118,116 characters and 13,369 named entities. in addition, sentences in the training set may contain named entities or not, each with an average of 49.31 characters and 2.23 named entities. however, all sentences in the test set contained at least one named entity, each with an average of 36.85 characters and 4.17 named entities. in summary, the average sentence length is short in the test set, but named entity density is relatively high. ## performance metrics performance is evaluated by examining the difference between the machine-predicted and human-annotated bio tags. standard precision, recall and f1-score are the most typical evaluation metrics of ner systems at a character level, and are used here. if the predicted tag of a character in terms of bio format was completely identical with the gold standard, the character in the testing instances was regarded as correctly recognized. entity type #train (%) #test (%) body 26411 (38.58%) 5315 (39.76%) symptom 12904 (18.85%) 1944 (14.54%) instrument 1089 (1.59%) 250 (1.87%) examination 2622 (3.83%) 207 (1.55%) chemical 6834 (9.98%) 1718 (12.85%) disease 10079 (14.72%) 2609 (19.52%) drug 2225 (3.25%) 481 (3.60%) supplement 1525 (2.23%) 183 (1.37%) treatment 3108 (4.54%) 468 (3.50%) time 1663 (2.43%) 194 (1.44%) total 68460 (100%) 13,369 (100%) precision is defined as the percentage of named entities found by the ner system that are correct. recall is the percentage of named entities present in the test set found by the ner system. the f1score is the harmonic mean of precision and recall. ## evaluation results the policy of this shared task is an open test. participating systems are allowed to use other publicly available data for this shared task, but the usage should be specified in their system description paper. each team was allowed to provide at most three submissions during the evaluation period. among ten registered teams, seven submitted their testing results, providing a total of 20 submissions, from which the submission with the best f1-score of each team was kept in the leaderboard for performance ranking. in summary, the overall best results came from the migbaseline team @xcite , whose approach achieved the best scores across all the evaluation metrics, followed by scu-mesclab @xcite and crowner @xcite . the most frequently used neural architecture in this shared task is bilstm-crf, which usually achieved promising results, matching findings from related studies for named entity recognition in the english language (chiu and nichols, 2016; @xcite @xcite . ## conclusions and future work this paper provides an overview of the rocling-2022 shared task for chinese healthcare named entity recognition, including task design, data preparation, performance metrics and evaluation results. we received a total of 20 testing submissions from seven participating teams. regardless of actual performance, all submissions contribute to the development of an effective named entity recognition solution in the healthcare