Evaluating Data Augmentation for Medication Identification in Clinical Notes (2023)

Abstract: We evaluate the effectiveness of using data augmentation to improve the generalizability of a Named Entity Recognition model for the task of medication identification in clinical notes. We compare disparate data augmentation methods, namely mention-replacement and a generative model, for creating synthetic training examples. Through experiments on the n2c2 2022 Track 1 Contextualized Medication Event Extraction data set, we show that data augmentation with supplemental examples created with GPT-3 can boost the performance of a transformer-based model for small training sets.

## introduction natural language processing (nlp) is an active area of research in healthcare, especially due to the proliferation of electronic health records (ehr). ehrs contain extensive information about individual patients, such as diagnoses with their corresponding international classification of disease (icd) codes, treatment records and test results. while some medication information can be extracted from the structured data in the ehrs, a substantial amount of the medication information resides in text-based narrative clinical notes @xcite . the information contained in clinical notes can be useful for pharmacovigilance, comparative effectiveness studies, and adverse event detection @xcite . the objective of the n2c2 2022 track 1 contextualized medication event extraction was to capture multi-dimensional context of medication changes documented in clinical notes. the track was comprised of three subtasks: first, we detail our model architecture comprised of a transformer-based language model and a conditional random fields (crf) @xcite component for identifying mentions of medication in clinical documents that obtained competitive results on the n2c2 2022 [ner] medication extraction subtask. next, we detail our data augmentation methodology for creating synthetic training examples. finally, we evaluate the effectiveness of using data augmentation for the task of medication extraction in clinical documents. moreover, we evaluate the effectiveness of using data augmentation for low-resource medication extraction, i.e. a scenario in which the size of a training set is small. ## background early systems for medication identification relied chiefly on rule-based techniques. @xcite combine natural language processing (nlp) pre-processing techniques and regular expressions to extract drug-dosage information from clinical narratives. the authors achieve an approximate 80% rate of exact and partial matches on target phrases. later, machine learning demonstrated effectiveness for the task of medication identification. @xcite used a crf model to identify medications for the 2009 i2b2 medication extraction task. the model used six feature sets, many of them requiring external knowledge (e.g. gazetteers) and hand-crafted features (e.g. morphological patterns). currently, neural network architectures, namely transformers, demonstrate state-of-the-art results for medication identification. @xcite fine-tune their domain-specific bio+clinical bert model on the i2b2 2010 concept extraction task @xcite , achieving an f1 score of 0.872 for exact matching, outperforming nondomain-specific variants such as bert @xcite . @xcite combine bert with a final crf layer for pharmaconer @xcite , the first shared task on detecting drug and chemical entities in spanish medical documents. @xcite present an evaluation of finetuning pre-trained language models for the task of biomedical entity recognition, namely drug names and symptoms. the authors compare five language models on two biomedical data sets, cadec and ade-corpus. their evaluation results demonstrate that biobert @xcite , a language model pretrained on in-domain (biomedical) corpora, outperformed all other models on both data sets and obtrained f1-scores of 0.903 and 0.6873 in the ade and cadec corpora, respectively. for the 2022 n2c2 medication extraction subtask, we sought to leverage both an indomain transformer-based language model, namely bio+clinical bert and a crf. ## results f1-scores, calculated at micro and macro averaged levels, were used in the evaluation using the n2c2 prompt: @xmath0: we have prescribed ziac for her blood pressure control results for our submission to the n2c2 2022 challenge are presented in later, we achieved significant improvements by tuning hyper-parameters and by modifying our postprocessing of the data (e.g. conversion from iob to brat standoff format). improved results post-n2c2 are also included in once again, our top-performing model, in terms of f1 lenient matching, was our baseline model (without augmentation), with a result of 96% for lenient matching. the model trained with synthetic examples from the cwlc remained the least ef-fective model and it achieved only a modest 1% increase in f1 lenient matching score (90.11%) on the test set with the optimized hyper-parameters. moreover, we observed a significant difference between our f1 strict and lenient scores. for all models, we achieved higher f1 lenient scores than strict matching scores. the smallest margin between scores on the test set was for our baseline, with a difference of 4.12%. the differences between the f1 lenient and strict scores were 4.78% and 4.85% for mimic-iii and cwlc variants respectively. ## discussion fine-tuning the bio+clinical bert language model in conjunction with a crf, without data augmentation, produces an effective medication identification model, corroborated by our competitive f1 lenient matching score (96%) using approach ii on the n2c2 track 1 ner substask test set. however, our top-performing model still exhibits some weaknesses, such as its handling of abbreviations. for example, for the target medication niacin sr in the test data set, our model identifies niacin while excluding sr (sustained release). given the input sentence "phoslo 1 tab po tidac" from the test data set, our model identifies tidac as a medication mention. notwithstanding that a tidac tablet is a medication used to treat and prevent stomach ulcers, in this context, tidac translates to t.i.d.a.c, i.e. "three times a day before meals". in addition to abbreviations, we also observed occurrences in which our model struggled to handle multi-word medication mentions. for instance, given the target medication multivitamin with betacarotene, our model instead identified two unique medications multivitamin and betacarotene. we also found that the use of data augmentation, when using the full training set, did not improve the performance our model. we achieved f1 lenient matching scores of 94%, 90%, and 92% for our mimic, cwlc, and gpt-3 model variants respectively. there are several variables that may have stymied the effectiveness of our data augmentation strategy. for example, dai and adel (2020) demonstrate that a mention-replacement data augmentation method is most effective on the i2b2 2010 concept extraction task when training on a small training corpus comprised of 50 instances. provided that the cmed data set is comprised of 9,012 annotated medication mentions across 500 clinical notes (400 for training), the baseline training corpus is perhaps ample for training effective medication identification models. moreover, our augmentation method may have introduced a significant amount of noise that was ultimately harmful. applying our baseline model to unannotated discharge summaries resulted in the collection of incorrect and problematic medication mentions. for example, our baseline model recognized kaopectate / benadryl / lidocaine as a single medication instead of three unique medications. our baseline model also identified abstract concepts in the discharge summaries, such as narcotic pain medications, as medication mentions. terms such as safetyglide, cranberry, suction, and banana were incorrectly identified as medications. the quality (e.g. the presence of special characters or medications concatenated with dosage information) of many identified medications in the discharge summaries were also problematic, e.g. caltrate 600 ]and simvistatin80mg. the effectiveness of our models trained with data augmentation may have also been affected by the randomness of the mention-replacement method. concretely, the augmentation method makes contextually inappropriate replacements of medication mentions, highlighted in the use of code-switched resources also failed to improve the generalization ability of our baseline model. notwithstanding that only 92 medication mentions were collected from the cwlc, and hence fewer synthetic examples created than from mimic-iii, our model trained on the codeswitched training corpus resulted in significantly worse results than our baseline model. on the other hand, we find that data augmentation using instances generated by gpt-3 can improve f1-scores in a low-resource setting. concretely, there are two characteristics of gpt-3 that may have contributed to its effectiveness: first, its ability to generate novel human-like sentences, and second, its ability to generate contextually correct sentences (unlike our mention-replacement method). for example, for the input medication phenylephrine, gpt-3 generated the sentence "we can add phenylephrine to help to reduce the congestion". phenylephrine is a medication used to relieve nasal discomfort caused by colds, allergies, and hay fever, and therefore gpt-3 is able to create a novel training example with the medication mention used in the proper context. ## conclusions we have described an architecture based on a transformer-based language model (bio+clinical bert) and a crf for the task of medication identification in clinical notes. additionally, we have presented a data augmentation strategy for creating synthetic training instances. models trained with our proposed data augmentation strategy yielded mixed results on the n2c2 2022 medication identification sub-task. our model using synthetic examples from mimic-iii achieved an f1 lenient score of 94% (which places it above the mean score shared by the task organizers), albeit lower than the score obtained by our baseline model. our model trained with synthetic examples containing medication mentions in spanish from the cwlc failed to produce competitive results. this model obtained an f1 lenient score of 90% on the test data set, placing it below the mean score shared by the task organizers. on the other hand, our baseline model (without augmentation) achieved competitive results in terms of our f1 lenient matching score (96%) on the n2c2 2022 track 1 test set. provided that our chief motivation was to produce an automated data augmentation system (reducing a dependency on costly domain experts), our mention-replacement technique did not contain constraints to ensure the semantic correctness of the substitutes. as a result, errors and biases were likely reinforced during the training of the models with mention-replacement augmentation. future work should also explore techniques to add restrictions that ensure the semantic correctness of synthetic instances. for example, using publicly available lists of medication names could help to ensure the correctness of the synthetic instances. the use of such lists could also permit an introduction of new medication names in the data for continuous training of models. even though hyper-parameters were tuned, there are also some architectural changes that may be adjusted in future work. for example, freezing the weights of bio+clinical bert, and hence only training the token classifier and crf, may be evaluated. moreover, removing the crf should also be assessed. in a low-resource setting, we demonstrate that data augmentation can boost the performance of a medication recognition model. concretely, we demonstrate that zero-shot learning with gpt-3 is an effective technique for creating novel and contextually correct training examples in the clinical domain for medication identification. this technique could be particularly beneficial in situations where the use of annotators with clinical domain expertise is not feasible. additionally, a strength of gpt-3 is its ability to generate coherent text in multiple languages, such as spanish, german, japanese, and russian. the generation of synthetic training instances with gpt-3 for medication identification in multiple languages should evaluated in future work. on the other hand, one known weakness of generative models such gpt-3 is their tendency to hallucinate, i.e. produce factually incorrect text. the ability generate correct medication names from large language models, such as gpt-3, should also be evaluated. the ability of a language model to produce a list of medications related to a given medical problem could reduce dependencies on annotated corpora and external data sources. gpt-3 has other disadvantages, e.g. a pay-per-use system and the collection of user data. therefore, an evaluation of open-source large language models for the creation of synthetic training instances should be conducted in future work.