M ed JE x: A Medical Jargon Extraction Model with W iki’s Hyperlink Span and Contextualized Masked Language Model Score (2022)

Abstract: This paper proposes a new natural language processing (NLP) application for identifying medical jargon terms potentially difficult for patients to comprehend from electronic health record (EHR) notes. We first present a novel and publicly available dataset with expert-annotated medical jargon terms from 18K+ EHR note sentences (MedJ). Then, we introduce a novel medical jargon extraction (MedJEx) model which has been shown to outperform existing state-of-the-art NLP models. First, MedJEx improved the overall performance when it was trained on an auxiliary Wikipedia hyperlink span dataset, where hyperlink spans provide additional Wikipedia articles to explain the spans (or terms), and then fine-tuned on the annotated MedJ data. Secondly, we found that a contextualized masked language model score was beneficial for detecting domain-specific unfamiliar jargon terms. Moreover, our results show that training on the auxiliary Wikipedia hyperlink span datasets improved six out of eight biomedical named entity recognition benchmark datasets. MedJEx is publicly available.

## introduction allowing patients to access their electronic health records (ehrs) represents a new and personalized communication channel that has the potential to improve patient involvement in care and assist communication between physicians, patients, and other healthcare providers @xcite . however, studies showed that patients do not understand medical jargon in their ehr notes @xcite . to improve patients' ehr note comprehension, it is important to identify medical jargon terms that are difficult for patients to understand. unlike the traditional concept identification or named 1 https://github.com/mozzitastebitter/ medjex entity recognition (ner) tasks, where the tasks mainly center on semantic salient entities, detecting such medical jargon terms takes into consideration the perspective of user comprehension. traditional ner approaches such as using comprehensive clinical terminological resources (e.g., the unified medical language system (umls) @xcite ) would identify terms such as "water" and "fat", which are not considered difficult for patients to comprehend. meanwhile, using term frequency (tf) as the proxy for medical jargon term identification will miss outliers such as "shock," which is a term frequently used in the open domain with its common sense: "a sudden upsetting or surprising event or experience." however, ehr notes incorporate its uncommon sense: "a medical condition caused by severe injury, pain, loss of blood, or fear that slows down the flow of blood." @xcite . thus, "shock" should be identified as a jargon term from ehr notes since it would be difficult for patients to comprehend, even though its tf is high. in this study, we propose a natural language processing (nlp) system that can identify such outlier jargon from ehr notes through a novel method for homonym resolution. we first expert-annotated de-identified ehr note sentences for medical jargon terms judged to be difficult to comprehend. this resulted in the medical jargon extraction for improving ehr text comprehension (medj) dataset, which comprises 18,178 sentences and 95,393 medical jargon terms. we then present a neural network-based medical jargon extraction (medjex) model to identify the jargon terms. to ameliorate the limited training-size issue, we propose a novel transfer learning-based framework @xcite utilizing auxiliary wikipedia (wiki) hyperlink span datasets (wikihyperlink), where the span terms link to different wiki articles @xcite . although medical jargon extraction and wikihyperlink recognition seem to be two different applications, they share similarities. the role of hyperlinks is to help a reader to understand an wiki article. thus, "difficult to understand" concepts in the wiki article may be more likely to have hyperlinks. therefore, we hypothesize that large-scale hyperlink span information from wiki can be advantageous for our models of medical jargon extraction. our results show that models trained on wikihyperlink span datasets indeed substantially improved the performance of medjex. moreover, we also found that such auxiliary learning improved six out of the eight benchmark datasets of biomedical ner tasks. to detect outlier homonymous terms such as "shock", we deployed an approach inspired by masking probing @xcite , a method for evaluating linguistic knowledge of large-scale pre-trained language models (plms). @xcite suggests plms are beneficial for predicting the reading time, with longer reading time indicates difficult for indicating difficulty in understanding. in our work, we propose a contextualized masked language model (mlm) score feature to tackle the homonym challenge. note that models will recognize the sense of a word or phrase using contextual information. since plms calculate the probability of masked words in consideration of context, we hypothesize that plms trained in the open-domain corpus would predict poorly masked medical jargon if senses are distributed differently between the open domain and clinical domain corpora. we conducted experiments on four state-of-theart plms, namely bert @xcite , roberta @xcite , bioclinicalbert @xcite and biobert @xcite . experimental results show that when both of the methods are combined, the medical jargon extraction performance is improved by 2.44%p in bert, 2.42%p in roberta, 1.56%p in bioclini-calbert, and 1.19%p in biobert. our contributions are as follows: ## related work in principle, medjex is related to text simplification @xcite . none of the previous work @xcite @xcite identified terms that important for comprehension. on the other hand, medjex is relevant to bioner, a task for identifying biomedical named entities such as disease, drug, and symptom from medical text. there are several benchmark corpora, including i2b2 2010 @xcite , share/clef 2013 @xcite , and made @xcite , all of which were developed solely based on clinical importance. in contrast, m edj is patient-centered, taking into consideration of patients' comprehension. identifying bioner from medical documents has been an active area of research. earlier work such as the metamap @xcite , used linguistic patterns, either manually constructed or learned semiautomatically, to map free text to external knowledge resources such as umls @xcite . the benchmark corpora have promoted supervised machine learning approaches including conditional random fields and deep learning approaches @xcite . key phrase extraction in the medical domain is another related task. it identifies important phrases or clauses that represent topics @xcite . in previous studies, key phrases were extracted using features such as tf, word stickiness, and word centrality @xcite . @xcite proposed an unsupervised learning based method to elicit important medical terms from ehr notes using metamap @xcite and various weighting features such as textrank @xcite and term familiarity score @xcite . in another work, @xcite proposed an adaptive distant supervision based medical term extraction approach that utilizes consumer health vocabulary @xcite and a heuristic rule to distantly label medical jargon training datasets. a key phrase extraction method using a large-scale pretrained model is being actively studied @xcite . unlike the previous bioner or key phrase identification applications, identifying medical jargon terms is important for patients' comprehension of their ehr notes and represents a novel nlp application. however, not all medical entities are unfamiliar to patients. the brute force approach of capturing every medical entity, the approaches of existing bioner and key phrase identification applications, may bring about confusion to patients. on the other hand, undetected medical jargon terms will reduce patients' ehr note comprehension. in this paper, we propose medjex, a novel application that identifies medical jargon terms important for patients' comprehension. once jargon terms are identified, interventions such as linking the jargon terms to lay definitions can help improve comprehension. ## dataset construction this work has two different datasets: 1) medj for medical jargon extraction and 2) wiki's hyperlink span (wikihyperlink) dataset for transfer learning. ## medjex model figure 1 is an overview of medjex. first, we trained plms with wikihyperlink (wiki-trained). then, the wiki-trained model was transferred to the target model that we propose by initializing the target model with the weight parameters of the wiki-trained model. finally, we fine-tuned the target model with our expert-annotated dataset. note that, since the pretrain corpora of plms used in this work include the wiki corpus, we noticed that the performance change should derive from the added labels (hyperlink spans). herein, we extracted umls concepts and used them as auxiliary features. ## conclusion we introduce a novel nlp tasked named medjex and present an expert-curated medj dataset for the task. we propose two innovative methods: 1) pretraining wiki's hyperlink span, and 2) contextualized mlm score feature for extracting medical jargon from ehr notes. the experimental results show that the wiki's hyperlink span can be effectively transferred to the medical jargon extraction model, leading to a significant performance improvement. wiki's hyperlink span training also beneficial in six out of eight bioner benchmarks. finally, in a qualitative evaluation, the mlm score feature complements the tf feature to identify common terms (or terms with high tfs) used in the clinical domain (homonyms).