Large language models are few-shot clinical information extractors (2022)

Abstract: A long-running goal of the clinical NLP community is the extraction of important variables trapped in clinical notes. However, roadblocks have included dataset shift from the general domain and a lack of public clinical corpora and annotations. In this work, we show that large language models, such as InstructGPT (Ouyang et al., 2022), perform well at zero- and few-shot information extraction from clinical text despite not being trained specifically for the clinical domain. Whereas text classification and generation performance have already been studied extensively in such models, here we additionally demonstrate how to leverage them to tackle a diverse set of NLP tasks which require more structured outputs, including span identification, token-level sequence classification, and relation extraction. Further, due to the dearth of available data to evaluate these systems, we introduce new datasets for benchmarking few-shot clinical information extraction based on a manual re-annotation of the CASI dataset (Moon et al., 2014) for new tasks. On the clinical extraction tasks we studied, the GPT-3 systems significantly outperform existing zero- and few-shot baselines.

## introduction clinical text contains a large amount of valuable information that is not captured by the structured data fields in electronic health records @xcite . however, there are significant challenges to clinical information extraction. because clinical text contains irregularities such as ambiguous jargon and nonstandard phrasal structure, most off-the-shelf nlp tools perform poorly, and clinical text annotation requires domain expertise @xcite . further, due to the sensitive nature of clinical text, public corpora are rare and restrictively licensed. as a result, clinical 1 https://huggingface.co/datasets/mitclinicalml/clinical-ie input: the patient takes coumadin 5 mg @xcite . prompt: create a list of medications. she takes 5 mg of coumadin and aspirin the patient takes coumadin 5 mg daily for a tia and she has an aspirin allergy. ## clinical sense disambiguation overview. clinical notes are rife with overloaded jargon and abbreviations. pt can mean patient, prothrombin time, physical therapy, or posterior tibial @xcite . this ambiguity impacts the utility of notes for patients, clinicians, and algorithms @xcite . in this section, we first evaluate clinical sense disambiguation on the casi dataset directly and then transfer a model distilled via weak supervision to another dataset. dataset 1. the clinical acronym sense inventory dataset consists of 500 text examples for each of 75 acronyms @xcite . due to noise in the dataset (e.g. duplications), it is common to filter to a subset of the dataset; we follow the filtering from @xcite , leading to a subset of 18,164 examples and 41 acronyms for evaluation. similar to other works, we treat the task as multiple-choice. dataset 2. we additionally use a reverse substitution dataset @xcite generated over the mimic-iii critical care database @xcite . in reverse substitution, labeled data is generated from unlabeled text by replacing expansions (e.g. physical therapy) with their acronyms (pt) and using the original expansion as the label. we evaluate on their 8912 test examples over the same 41 acronyms as the casi subset. since we cannot query gpt-3 on this dataset, we distill and transfer a model trained on the outputs from dataset 1. prompting + resolver. we used gpt-3 edit (using engine text-davinci-edit-001) with greedy decoding (@xmath0. for each example, we provided the full clinical snippet and appended the single instruction expand the abbreviation:{abbr}. since we did not provide the llm with the answer choices, the form of the output string could still differ slightly from all the candidate answers (e.g. editing "ra" to "right atria" when "right atrium" was expected). in the resolver, we choose the answer choice with the highest contiguous character overlap with the llm generated output. model distillation via weak supervision. direct deployment of large language models can be difficult due to model size and data privacy. to remedy these issues, we follow several recent works @xcite @xcite and show that we can instead view the llm + resolver system as a labeler rather than as a classifier, and that this can even boost performance. in particular, we use outputs of this system on casi as weak supervision (e.g., @xcite to train a smaller, task-specific model. here we fine-tune pubmedbert @xcite and follow @xcite ; details and hyperparameters are found in the appendix. baselines. we compare the performance of our approach to other zero-shot language modeling methods: (i) latent meaning cells (lmc), a deep latent variable model from @xcite which is pre-trained on millions of notes from mimic, (ii) elmo pre-trained on the same dataset @xcite , and (iii) clinical biobert @xcite . numbers for these three baselines are taken from @xcite ; for all three, they choose the answer choice with the most similar representation to the contextual representation of the acronym. we also show performance for random guessing and choosing the most common answer choice per acronym (since the expansions of many acronyms follow a long-tailed distribution). evaluation. accuracy and macro f1 are calculated per acronym and averaged over all acronyms (see left of since the casi dataset is publicly accessible, one possible caveat is that the dataset could have been in the language model's training data; to investigate further (see section c.1), we prompt the llm on acronyms not in the original annotations. ## biomedical evidence extraction task overview. evidence-based medicine (ebm) involves synthesizing findings from across clinical research studies, but the current rapid clip of research makes it nearly impossible to keep up with all studies @xcite . therefore, automated approaches for parsing clinical abstracts could aid the adoption of ebm @xcite . here, we focus on extracting interventions and controls (which we will refer to just as intervention), where the underlying goal is to identify the distinct arms of a clinical trial @xcite . token-level classification is often used as a proxy for this goal, but distilling identified spans into distinct interventions is non-trivial and often requires significant domain knowledge. prior work on arm identification has attempted to use coreference resolution @xcite and to identify of pairs of spans with redundant information @xcite . dataset. we assess intervention identification from the angles of (i) the token classification proxy task and (ii) the underlying task of arm identification. for (i), we use the token-level annotations provided in version 2 of the dataset from @xcite and evaluate on the 187 test abstracts provided. the average cohen's 𝜅 was only 0.59 on this set. for (ii), the two annotators from section 3.2 manually derived a list of the intervention-control arms for 20 abstracts in the test set, with perfect agreement. prompting + resolvers. we use a single prompt with instructgpt (engine text-davinci-002) and greedy decoding. the resolver for the tokenlabeling task removes noisy tokens (stop words) from the llm output, maps remaining tokens in the output to the original input and labels those as 1, and merges fractured spans. the full process can be found in appendix c.2. for the arm identification task, resolving simply involved splitting the output string on new lines. comparison. we compare to supervised approaches that train on the 4800 labeled training examples from @xcite . pubmedbert with an additional classification layer (lstm or crf) achieves close to state-of-the-art performance on the full task @xcite . since prior works report numbers combined over multiple classes, we re-run training on only the intervention label using pubmedbert-crf. we also include the best supervised baseline from @xcite , an lstm-crf over word and character-level embeddings. token-level results (proxy task). we first evaluate sequence labeling precision at the token-level (f1 in a clarifying example can be found in section c.2. ## coreference resolution task overview. coreference resolution involves grouping noun phrases that refer to the same underlying entity (e.g. a person, a medical concept), and it is considered particularly important for clinically accurate information retrieval and summarization @xcite . for example, when surfacing past medical history, it is critical to correctly parse pronouns to understand whether the history describes the patient or a family member. dataset description. in clinical nlp, coreference resolution has been largely evaluated on the 2011 i2b2/va challenge, which consists of thousands of coreference chains @xcite . due to i2b2's data use agreement, the two annotators annotated a new dataset using casi snippets, with 5 coreference pairs for prompt design and 100 pairs for evaluation @xcite . we prioritized ## medication extraction the recognition of clinical concept mentions (problems, treatments, etc.), their modifiers (e.g., negation), and relations (e.g., dosage) is a fundamental building block in clinical nlp @xcite . here we examine the extraction of medication concepts with two different schemas. ## conclusion in this work, we introduced new annotated datasets to show that (i) large language models have great promise at diverse clinical extraction tasks and (ii) we can guide generations to map to complex output spaces with only light post-processing. we also demonstrated how weak supervision over the system's outputs can be used to train smaller, taskspecific models that are more deployable. the scope of clinical nlp extends past what we studied here, and important next steps involve experimenting with llms such as opt (zhang et al., 2022) for which we can run inference locally, enabling evaluation on existing benchmarks and fine-tuning. another important direction involves leveraging the outputs from several prompts (e.g. 1-shot prompts with different examples) to learn to determine when gpt-3 is uncertain; this increased reliability will be vital given the high-stakes in clinical information extraction. taken as a whole, our work indicates a new paradigm for clinical information extractionone that can scale to the lofty goals of clinical nlp.