Evaluation of Transfer Learning for Adverse Drug Event ( ADE ) and Medication Entity Extraction (2020)

Abstract: We evaluate several biomedical contextual embeddings (based on BERT, ELMo, and Flair) for the detection of medication entities such as Drugs and Adverse Drug Events (ADE) from Electronic Health Records (EHR) using the 2018 ADE and Medication Extraction (Track 2) n2c2 data-set. We identify best practices for transfer learning, such as language-model fine-tuning and scalar mix. Our transfer learning models achieve strong performance in the overall task (F1=92.91%) as well as in ADE identification (F1=53.08%). Flair-based embeddings out-perform in the identification of context-dependent entities such as ADE. BERT-based embeddings out-perform in recognizing clinical terminology such as Drug and Form entities. ELMo-based embeddings deliver competitive performance in all entities. We develop a sentence-augmentation method for enhanced ADE identification benefiting BERT-based and ELMo-based models by up to 3.13% in F1 gains. Finally, we show that a simple ensemble of these models out-paces most current methods in ADE extraction (F1=55.77%).

## introduction adverse drug events (ade) arising from the medical intervention of drugs account for 1.3 million visits to the emergency department in the united states alone @xcite . randomized controlled trials (rcts), the primary mechanism for monitoring and identifying ades, are hampered by insufficient sample sizes of clinical trials @xcite . pharmacovigilance databases such as the food and drug administration's adverse event reporting system (faers) strive to be authoritative sources for physicians; however, they require regular manual data entry @xcite . electronic health records (ehrs) contain valuable information about patient medication history: drugs prescribed, reasons for administration, dosages/strengths, and ades. automated extraction of these medication entities by natural language processing (nlp) techniques can facilitate wide-scale pharmacovigilance @xcite . incorporating such a predictive system within the clinical note-taking interface may help the physician by alleviating the need to access external clinical decision support applications @xcite . for instance, if a physician notes down 'started on dilantin for seizure prophylaxis for a few days', the text could be quickly parsed -highlighting 'dilantin' as a drug, 'seizure prophylaxis' as the reason for administration, 'few days' as the duration, and warnings of 'eye discharge', 'oral sores', etc. as potential ades. in the example given, 'seizure prophylaxis' and 'few days' may occur any where in the clinical text, but only in the context of 'dilantin' they indicate reason / duration for administration. besides, such 'dynamic' interfaces can aid medical students to learn from their collective experiences. among medication entities, ade and reason are challenging to disambiguate @xcite . frequently, the specific reason for drug administration may appear in a subsequent sentence @xcite . besides, ade data-sets include goldannotations for these entities, only if they are associated with a drug. doing so leads to a significant reduction in the number of gold annotations @xcite . as part of our work in uniting clinical decision support functions and note-taking interfaces, we needed to develop a high-performing medication extraction model using open-source nlp frameworks. following @xcite , we modeled this as a named-entity recognition task (uzuner @xcite and experimented with transfer learning using openly available biomedical contextual embeddings. it is in this context, 1. we evaluate transfer learning models incorporating: biobert @xcite , clinical-bert @xcite , elmo @xcite and flair @xcite contextual embeddings pre-trained on pubmed abstracts @xcite . our main intention was to get a transfer learning pipeline working with these embeddings and therefore we did not perform any detailed hyperparameter optimization. despite this, we were able to achieve strong performance with all the embeddings. standalone models achieved f1-scores of 53.08% in ade extraction and 92.91% in the overall task with default features. a basic ensemble constructed from these standalone models achieved f1scores of 55.77% in ade extraction and 92.82% in the overall task confirming the viability of the overall strategy. ## related work classical research in this area focused on rulebased systems (such as medex @xcite , adept @xcite ) and crf-based machine-learning leveraging hand-crafted features @xcite @xcite . the 2018 n2c2 adverse drug events and medication extraction in ehr data-set @xcite tion. most participants leveraged the bilstm-crf neural model in their work @xcite . we have listed the top performing methods from the 2018 n2c2 ade challenge in relationship association for medication entities is complementary to our work and can be implemented either jointly or in a pipeline. such a joint architecture utilizes the signals from the relations task to filter out unwanted medication entities. @xcite adopted such a joint-approach with a three-classifier ensemble achieving 52.95% in ade extraction. @xcite also used a joint-architecture supplemented by umls @xcite ) concept lookups and unique modeling of temporal entities. @xcite cascaded classifiers sequentially to widen the contextual information available for ade identification. this model also facilitates improved identification when spans overlap. they evaluated ten pre-trained embedding models: half of them were based on mimic-iii while the rest were general-purpose. kim and meystre (2020) uniquely leveraged searn @xcite , a search-based prediction algorithm for its preference of precision over recall. our work is most similar to @xcite ; they demonstrate that strong medication extraction models can be constructed with minimal engineering using contextual embeddings. the main differences from above mentioned studies are the evaluation of a broader array of contemporary biomedical embeddings, detailed study of fine-tuning strategies, and augmentation methods for ade extraction. ## model selection procedures in transfer learning, the linguistic-information encoded by contextual embedding acts as a primary input to the downstream task layer (bilstm). fine-tuning is generally accepted to be beneficial. however, it requires familiarity with the scripts / associated frameworks specific to the embedding and data-set adaptation. ## discussion tables 6 and 7 show the overall performance of the various models. the prefixes (bb, cb, ep, fp) shows the contextual embedding used; and the suffix (pr, re, f1) shows the precision, recall, f1 metrics. the two highest f1 score for each entity are indicated via subscripts. the three most challenging entities are underlined. ## limitations and future work there are a few limitations in this study that we plan to address in future works: ## conclusion in this study, we presented strong performing transfer learning models for the extraction of medication entities using several biomedical contextual embeddings. our experiments shed light on the strengths of the various embeddings: flair-pubmed embedding out-performs in ade extraction. biobert and clinicalbert embeddings outperform in recognition of drug and form medication entities. elmo-pubmed embedding delivers competitive performance in all medication entities. we showed that sentence-augmentation and ensembling are viable strategies to enhance ade performance. our approach is free of hand-generated features and built using off-the-shelf neural models, default hyper-parameters, and training procedures. these factors decrease the development effort. a detailed analysis of embedding-specific factors contributing to mis-classification and inclusion of finetuning procedures are part of our ongoing work.