YNU - HPCC at ROCLING 2022 Shared Task: A Transformer-based Model with Focal Loss and Regularization Dropout for C hinese Healthcare Named Entity Recognition (2022)

Abstract: Named Entity Recognition (NER) is a fundamental task in information extraction that locates the mentions of named entities and classifies them in unstructured texts. Previous studies typically used hidden Markov model (HMM) and conditional random fields (CRF) for NER. To learn long-distance dependencies in text, recurrent neural networks, e.g., LSTM and GRU can extract the semantic features for each token with a sequential manner. Based on Transformers, this paper describes the contribution to ROCLING-2022 Share Task. This paper adopts a transformer-based model with focal Loss and regularization dropout. The focal loss is to overcome the uneven distribution of the label. The regularization dropout (r-drop) is to address the problem of vocabulary and descriptions that are too domain-specific. The ensemble learning is to improve the performance of the model. Comparative experiments were conducted on dev set to select the model with the best performance for submission. That is, BERT model with BiLSTM-CRF, focal loss and R-Drop has achieved the best F1-score of 0.7768 and rank the 4th place.

## introduction providing computer the ability to understand the abstract meaning of real world is a fundamental task. the shared task of rocling-2022 is chinese healthcare named entity recognition task. given a sentence about chinese healthcare, the intelligent model is required to produce the entities in this sentence. previous studies used probabilistic model for named entity recognition on text, such as hidden markov model (hmm) @xcite and conditional random field (crf) @xcite . recent advances in deep neural networks (dnn) @xcite and representation learning @xcite have considerably improved the ability of ner models. it mainly consists of an encoder to learn hidden representation for each token, as well as a classifier to assign a label for the token. for encoders, traditional models are usually used recurrent neural networks (rnn), such as long short-term memory (lstm) @xcite or gated recurrent units (gru) to learn longdistance dependencies. furthermore, attention mechanisms can be applied to improve the performance of rnn models to extract more task-specific features between tokens to provide meaningful information. several effective approaches apply the pre-trained language models (plm), such as bert @xcite , roberta @xcite and albert @xcite , to provide powerful representation to boost the performance of the whole physical structure that forms a person or animal including biological cells, organizations, organs and systems. symptom(symp) any feeling of illness or physical or mental change that is caused by a particular disease. instrument(inst) a tool or other device used for performing a particular medical task such as diagnosis and treatments. examination(exam) the act of looking at or checking something carefully in order to discover possible diseases. ## model description this section will describe the architecture of the proposed model in details. there are several components in this section, including bert, bilstm-crf, focal loss, and r-drop. the model architecture is shown in figure 2 . ## experimental results in this section, comparative experiments were conducted to select the best model as the final submission. the details of the experiments are presented as follows. ## conclusions in this paper, we describe our entire experimental procedure, and finally achieve the best f 1 -score of 0.7768 and rank the 4th place. for implementation, several different approaches were applied, such as mrc and label embedding. unfortunately, they didn't perform well. we applied a bert-bilstm-crf architecture with warm up strategy and r-drop, to get the best score. future works will attempt to explore more different span-based extraction methods for the chinese healthcare ner task.