MRC -based Medical NER with Multi-task Learning and Multi-strategies (2022)

Abstract: “Medical named entity recognition (NER), a fundamental task of medical information extraction, is crucial for medical knowledge graph construction, medical question answering, and automatic medical record analysis, etc. Compared with named entities (NEs) in general domain, medical named entities are usually more complex and prone to be nested. To cope with both flat NEs and nested NEs, we propose a MRC-based approach with multi-task learning and multi-strategies. NER can be treated as a sequence labeling (SL) task or a span boundary detection (SBD) task. We integrate MRC-CRF model for SL and MRC-Biaffine model for SBD into the multi-task learning architecture, and select the more efficient MRC-CRF as the final decoder. To further improve the model, we employ multi-strategies, including adaptive pre-training, adversarial training, and model stacking with cross validation. Experiments on both nested NER corpus CMeEE and flat NER corpus CCKS2019 show the effectiveness of the MRC-based model with multi-task learning and multi-strategies.”

## introduction with the fast development of medical digitalization, more and more medical documents are generated, including electronic medical records, medical reports, etc. medical information extraction, including medical named entity recognition (ner), becomes increasingly important to applications like knowledge graph construction, question answering system, and automatic electronic medical record analysis. medical ner is a task to automatically recognize medical named entities, like body (bod), disease, clinical symptom (sym), medical procedure, medical equipment, drug, medical examination item, etc., from medical texts. medical named entities are usually long, nested and polysemous, which pose great challenges to medical ner. for example, in fig 1 , the two "bod" entities "延髓"(medulla oblongata) and "脊髓"(spinal cord) are nested in the "sym" entity "延髓和脊髓受损"(damage to the medulla oblongata and spinal cord). ## related work just like ner in other application domains, medical ner borrows methods from ner in general domain. the methods evolve from rule-based methods, traditional machine learningbased methods, deep learning-based methods, to the present mainstream pre-training-based methods. pre-trained models like bert @xcite @xcite , elmo @xcite @xcite , etc., have become a standard module to encode the input texts. to better represent a text, rnn @xcite , lstm @xcite , gru @xcite , cnn @xcite and other neural networks are usually employed after the pre-trained model. taking the ner as a sequence labeling problem, crf @xcite is finally used to generate the sequence labels. for chinese, characters @xcite , radicals, strokes @xcite ) and glyphs @xcite can provide useful information besides words. thus such linguistic units are encoded together with words using latticelstm @xcite , elmo @xcite @xcite and other networks. domain data can be used to improve medical ner. @xcite pre-train a med-bert based on medical texts to boost the performance significantly. @xcite integrate domain dictionary and rules with bi-lstm-crf. multi-task learning is another way to improve the performance. ner model can be enhanced by parameter sharing with models of other tasks. @xcite take ner and pos tagging as two tasks. @xcite ) take ner on two different datasets as two tasks. to tackle nested ner problem and encode knowledge from entity types, ner is formulated as a machine reading comprehension task @xcite , and two binary classifiers are used to detect the span of a named entity. to enhance the information interaction between the head and tail of an entity, @xcite introduce biaffine to mrc. @xcite ensemble sequence labeling and span boundary detection by voting strategies while @xcite ensemble crf and mrc. ## the mrc-mtl-ms model mrc model extracts answer fragments from paragraphs by a given question. suppose x is the input text, for each entity type y, designing a query q y , extracting a subsequence x of type y from x, and we can get the triple (q y , x, x), which is exactly the (question, answer, context) a mrc model needs. the model only calculates the loss of context during training, and masks the loss of query and padding. 3.1 multi-task learning (mtl) the overall architecture of the model is shown in fig 2 . the multi-task learning architecture consists of the main task of sequence labeling by crf and the auxiliary task of span boundary detection by biaffine. for each entity type y, the input to the model is context x followed by query q y , which is proved experimentally better than reversed concatenating way. the input is encoded by an adaptive pre-trained model cme-nezha, then goes through a conditional layernorm guided by entity label embedding to further untilize entity type knowledge, and finally is decoded by crf and biaffine respectively. ## datasets two public datasets are used for experiments, cmeee for nested ner and ccks2019 for flat ner. statistics of the two datasets are shown in flat entity nested entity percent of nested percent of nested in sym 84119 10063 10.68% 30.21% ## mrc-mtl-ms(ours) as shown in ## conclusion this paper proposes a mrc-based multi-task model for chinese medical ner, enhancing mrc-crf with biaffine to recognize the named entities more accurately. to further improve the model, we introduce multi-strategies, including adaptive pre-training, adversarial training and model stacking with cross validation. our model can cope with both flat ner and nested ner. experiments on the nested ner corpus cmeee and the flat ner corpus ccks2019 show the effectiveness of our model. in the future, we will incorporate domain knowledge to improve the recognition performance on hard named entities.