SM ed BERT : A Knowledge-Enhanced Pre-trained Language Model with Structured Semantics for Medical Text Mining (2021)

Abstract: Recently, the performance of Pre-trained Language Models (PLMs) has been significantly improved by injecting knowledge facts to enhance their abilities of language understanding. For medical domains, the background knowledge sources are especially useful, due to the massive medical terms and their complicated relations are difficult to understand in text. In this work, we introduce SMedBERT, a medical PLM trained on large-scale medical corpora, incorporating deep structured semantic knowledge from neighbours of linked-entity. In SMedBERT, the mention-neighbour hybrid attention is proposed to learn heterogeneous-entity information, which infuses the semantic representations of entity types into the homogeneous neighbouring entity structure. Apart from knowledge integration as external features, we propose to employ the neighbors of linked-entities in the knowledge graph as additional global contexts of text mentions, allowing them to communicate via shared neighbors, thus enrich their semantic representations. Experiments demonstrate that SMedBERT significantly outperforms strong baselines in various knowledge-intensive Chinese medical tasks. It also improves the performance of other tasks such as question answering, question matching and natural language inference.

## introduction pre-trained language models (plms) learn effective context representations with self-supervised tasks, spotlighting in various nlp tasks @xcite @xcite . in addition, knowledge-enhanced plms (keplms) @xcite @xcite further benefit language understanding by in the literatures, a majority of keplms @xcite @xcite inject information of entities corresponding to mention-spans from knowledge graphs (kgs) into contextual representations. however, those keplms only utilize linked-entity in the kgs as auxiliary information, which pay little attention to the neighboring structured semantics information of the entity linked with text mentions. in the medical context, there exist complicated domain knowledge such as relations and medical facts among medical terms @xcite , which are difficult to model using previous approaches. to address this issue, we consider leveraging structured semantics knowledge in medical kgs from the two aspects. (1) rich semantic information from neighboring structures of linked-entities, such as entity types and relations, are highly useful for medical text understanding. as in figure 1 , "新型冠状病毒" (novel coronavirus) can be the cause of many diseases, such as "肺炎" (pneumonia) and "呼吸综合征" (respiratory syndrome). 2 (2) additionally, we leverage neighbors of linked-entity as global "contexts" to complement plain-text contexts used in @xcite . the structure knowledge contained in neighbouring entities can act as the "knowledge bridge" between mention-spans, facilitating the interaction of different mention representations. hence, plms can learn better representations for rare medical terms. in this paper, we introduce smedbert, a ke-plm pre-trained over large-scale medical corpora and medical kgs. to the best of our knowledge, smedbert is the first plm with structured semantics knowledge injected in the medical domain. specifically, the contributions of smedbert mainly include two modules: mention-neighbor hybrid attention: we fuse the embeddings of the node and type of linkedentity neighbors into contextual target mention representations. the type-level and node-level attentions help to learn the importance of entity types and the neighbors of linked-entity, respectively, in order to reduce the knowledge noise injected into the model. the type-level attention transforms the homogeneous node-level attention into a heterogeneous learning process of neighboring entities. mention-neighbor context modeling: we propose two novel self-supervised learning tasks for promoting interaction between mention-span and corresponding global context, namely masked neighbor modeling and masked mention modeling. the former enriches the representations of "context" neighboring entities based on the well trained "target word" mention-span, while the latter focuses on gathering those information back from neighboring entities to the masked target like low-frequency mention-span which is poorly represented @xcite . in the experiments, we compare smedbert against various strong baselines, including mainstream keplms pre-trained over our medical resources. the underlying medical nlp tasks include: named entity recognition, relation extraction, question answering, question matching and natural language inference. the results show that smedbert consistently outperforms all the baselines on these tasks. ## conclusion in this work, we address medical text mining tasks with the structured semantics keplm proposed named smedbert. accordingly, we inject entity type semantic information of neighboring entities into node attention mechanism via heterogeneous feature learning process. moreover, we treat the neighboring entity structures as additional global contexts to predict the masked candidate entities based on mention-spans and vice versa. the experimental results show the significant improvement of our model on various medical nlp tasks and the intrinsic evaluation. there are two research directions that can be further explored: (1) injecting deeper knowledge by using "farther neighboring" entities as contexts; (2) further enhancing chinese medical long-tail entity semantic representation. the pre-training corpora is crawled from dxy bbs (bulletin board system) foot_7 , which is a very popular chinese social network for doctors, medical institutions, life scientists, and medical practitioners. the bbs has more than 30 channels, which contains 18 forums and 130 fine-grained groups, covering most of the medical domains. for our pre-training purpose, we crawl texts from channels about clinical medicine, pharmacology, public health and consulting. for text pre-processing, we mainly follow the methods of @xcite . additionally, (1) we remove all urls, html tags, e-mail addresses, and all tokens except characters, digits, and punctuation (2) all documents shorter than 256 are discard, while documents longer than 512 are cut into shorter text segments.