Prompt-based Extraction of Social Determinants of Health Using Few-shot Learning (2023)

Abstract: Social determinants of health (SDOH) documented in the electronic health record through unstructured text are increasingly being studied to understand how SDOH impacts patient health outcomes. In this work, we utilize the Social History Annotation Corpus (SHAC), a multi-institutional corpus of de-identified social history sections annotated for SDOH, including substance use, employment, and living status information. We explore the automatic extraction of SDOH information with SHAC in both standoff and inline annotation formats using GPT-4 in a one-shot prompting setting. We compare GPT-4 extraction performance with a high-performing supervised approach and perform thorough error analyses. Our prompt-based GPT-4 method achieved an overall 0.652 F1 on the SHAC test set, similar to the 7th best-performing system among all teams in the n2c2 challenge with SHAC.

## introduction and related work social determinants of health (sdoh) are the conditions in which people work and live that impact quality of life and health (centers for disease @xcite . understanding sdoh can assist in clinical decision-making @xcite . sdoh is documented in the electronic health record (ehr) through unstructured clinical narratives and structured data; however, the clinical narrative includes a more detailed description of many sdoh events. to utilize the text-encoded sdoh information in secondary use applications, including clinical decision-support systems, the sdoh information must be automatically extracted @xcite . sdoh extraction has been explored using rulebased systems and data-driven models that use supervised learning (hatef et al., 2019; patra et al., * equal contribution 2021; @xcite on a variety of corpora @xcite @xcite . recent sdoh extraction work utilizes large language models (llms) like bert and t5, where models are fine-tuned to the sdoh extraction task @xcite . recent advancements in llms, including larger models like generative pretrained transformer (gpt)-based models @xcite , allow for new training paradigms, including few-shot or zero-shot learning. recent developments in llms like the gpt-4 (openai, 2023) and med-palm models have shown their capability to understand the clinical text and achieve/exceed human-level performance in us medical licensing exams @xcite . this high performance may be attributed to (1) high model parameter counts, (2) large pre-training datasets, and (3) instruction tuning and optimization with reinforcement learning human feedback (rlhf) @xcite . recent clinical information extraction (ie) work @xcite comparing bertbased fine-tuning approaches to zero-shot learning indicates gpt models can extract entities and relations with reasonable performance; however, there are many open questions related to the use of recent llms, like gpt-4, in clinical ie tasks. in this work, we explore the extraction of sdoh using gpt-4 in a one-shot prompting setting with event-based shac @xcite . we compare prompt-based extraction approaches with a high-performing supervised bert-based model @xcite that has been finetuned to sdoh extraction from shac. we investigate two different one-shot prompting strategies for gpt-4, including prompts aimed at generating brat standoff format and inline annotations. we report an overall performance of 0.861 f1 from the fine-tuned model, evaluated on the withheld test set. the highest-performing one-shot gpt-4 approach achieved an overall 0.652 f1 for sdoh event extraction. our initial study shows that gpt-4 can extract sdoh information from text with limited training examples. ## data, task, & evaluation the 2022 national nlp clinical challenges sdoh extraction task (n2c2/uw sdoh challenge) used shac for model development and evaluation @xcite . shac contains 4405 deidentified social history sections of notes from mimic-iii @xcite and the university of washington (uw). shac includes training, development, and test partitions for both sources (mimic-iii and uw). shac was annotated using brat @xcite , a web-based annotation tool, to capture five sdoh event types: substance use (alcohol, drug, tobacco), employment status (employment), and living status (livingstatus). figure 1 (a ) presents an annotated sample in brat from the shac uw training set. the n2c2/uw sdoh challenge evaluation criteria interpret the extraction as a slot-filling task @xcite . each event comprises a single trigger span and at least one required argument. trigger any overlap equivalence requires the predicted trigger to overlap with the true trigger of the same event type. arguments can be classified into two categories: span-only (a multi-word span and argument type) and labeled (a multi-word span, argument type, and subtype label). arguments can be equivalent only when attached to equivalent triggers. in addition to trigger any overlap equivalence, span-only argument equivalence is evaluated by exact match, and labeled arguments equivalence requires the correct argument and subtype labels (span agnostic) @xcite . we evaluated performance using the n2c2/uw sdoh challenge criteria, as well as on more lenient evaluation criteria that still assess the clinical meaning of extraction. in the lenient criteria, trigger equivalence is relaxed to a minimum-distance metric (minimum distance), where gold triggers are paired (aligned) with the closest predicted trigger of the same event type, and the closest predicted trigger is counted as a true positive. in the lenient criteria, the span-only arguments use the any overlap criteria and the labeled arguments are evaluated as previously described. ## methods we benchmark the sdoh extraction task using two methods: (1) multi-label variation of the span-based entity and relation transformer (spert)(eberts and ulges, 2020) architecture, mspert @xcite benchmarked for shac as a high-performing fine-tuned baseline, and ( 2 ) prompt-based one-shot learning with gpt-4. inspired by performance gains of few-shot learning, relative to zero-shot learning in prior work @xcite , we use one-shot prompting with gpt-4 for the sdoh extraction task in this short study. we experiment with two distinct output formats -(1) brat-style standoff annotations (gpt-standoff) and ( 2 ## results and discussion ## conclusions we investigate the efficacy of two prompt-based approaches for extracting sdoh from social history sections using gpt-4. although the supervised model achieves higher performance, our findings indicate that gpt-4's one-shot learning capabilities serve as a promising starting point for extracting sdoh events without the need for annotated data. possible gains in future work may be achieved with a combination of few-shot and active learning.