SDOH - NLI : a Dataset for Inferring Social Determinants of Health from Clinical Notes (2023)

Abstract: Social and behavioral determinants of health (SDOH) play a significant role in shaping health outcomes, and extracting these determinants from clinical notes is a first step to help healthcare providers systematically identify opportunities to provide appropriate care and address disparities. Progress on using NLP methods for this task has been hindered by the lack of high-quality publicly available labeled data, largely due to the privacy and regulatory constraints on the use of real patients’ information. This paper introduces a new dataset, SDOH-NLI, that is based on publicly available notes and which we release publicly. We formulate SDOH extraction as a natural language inference task, and provide binary textual entailment labels obtained from human raters for a cross product of a set of social history snippets as premises and SDOH factors as hypotheses. Our dataset differs from standard NLI benchmarks in that our premises and hypotheses are obtained independently. We evaluate both “off-the-shelf” entailment models as well as models fine-tuned on our data, and highlight the ways in which our dataset appears more challenging than commonly used NLI datasets.

## introduction there has been growing recognition that social and behavioral determinants of health (sdoh) play a significant role in shaping health outcomes for individuals and populations. the ability to accurately identify and extract social and behavioral determinants of health from clinical notes can provide valuable insights that can enable healthcare providers to better understand and address the underlying determinants of health that contribute to poor health outcomes and health disparities. social determinants of health are frequently recorded in clinical notes as unstructured text, so natural language processing (nlp) can be a valuable tool for extracting actionable insights for care teams. however, research in this area often uses patient records from private health systems' electronic health records (ehrs), which makes it difficult to compare results from other health systems or even replicate the studies. the development and release of high-quality publicly available datasets could enable more reproducible research in this area. in this work, we introduce a new, public sdoh dataset based on mtsamples.com, an online collection of transcribed medical reports. our setup is motivated by the use cases of slicing patient populations along social determinant dimensions for population analytics, and retrieving patients with certain social determinants of health to allow for more targeted outreach and intervention. given a large set of social determinant factors, our goal is to make binary determinations for each patient about whether that patient's notes imply a particular sdoh factor. in other words, for example, we want to be able to find all patients who lack access to transportation, as opposed to just tagging transportation-related spans in their notes, as done in some previous work. to achieve this goal, we formulate the task as a textual entailment problem, with patient note snippets as the premises, sdoh factors as the hypotheses, and binary entailment labels. we use human annotators to label 1,398 social history snippets according to a curated list of 60 sdoh statements, resulting in a dataset of 29,635 labeled premisehypothesis examples after some filtering (see section 3 for details). we release this dataset publicly. we also evaluate state-of-the-art publicly available large language models on our data in a range of different settings (see section 4). a notable feature of our entailment dataset is that unlike other entailment datasets, our premises and hypotheses were obtained independently, and we label the full cross product of premises and hypotheses. in traditional entailment datasets, the hypotheses are constructed to be tied to a particular premise; however, in our formulation, the hypotheses are drawn from a large set of sdoh factors that may or may not be discussed in a particular premise (drawn from a clinical note). since all our text comes from the same domain, we still have a non-negligible fraction of positive entailment labels (albeit with a much larger label imbalance than in standard nli benchmark datasets). this requires nli methods to understand both the premise and the hypothesis, and defeats typical shortcuts that have been observed to work for other entailment datasets, such as guessing the label based on the hypothesis alone, or relying on simple syntactic clues such as the presence of negations (see section 2.2). indeed, even though our task does not require domain-specific knowledge, we observe that state-of-the-art models struggle to generalize from common nli benchmarks to our dataset, and highlight typical failure cases (see section 4). we evaluate both off-the-shelf and fine-tuned models in different setups: dual encoders, zero/fewshot prompting, and binary classification. we show that state-of-the-art off-the-shelf models, even if they were fine-tuned on various nli datasets, do not reliably solve our problem; on the other hand, models fine-tuned on our training set robustly generalize both to unseen notes and to unseen factors, providing evidence for the usefulness of our dataset for model fine-tuning and evaluation. ## dataset construction we scraped all 5,003 medical reports from mtsamples. within these reports, we obtained 1,030 note sections related to social history by searching for a collection of note section titles identified as synonyms of social history. we then split each note section into sentences, resulting in 3,281 sentences. many sentences (such as "he is married") appear in multiple notes; after deduplication, we have 1,398 unique sentences. we manually curated a collection of sdoh factors primarily from two sources, the ahrq social determinants of health database (for healthcare research and quality, 2020) and ucsf siren's compendium of medical terminology codes for social risk factors @xcite . we rephrased each factor as a full english sentence stating a fact about a person; e.g. "the person is employed." for binary factors, we included a statement and its negation; e.g. "the person currently drinks alcohol" and "the person currently does not drink alcohol." for factors with multiple potential values (such as housing status), we aimed to list all the common options. we grouped these 60 statements into 10 categories: smoking, alcohol use, drug use, employment, housing, food, transportation, health insurance, social support, and financial situation. see the full list in appendix a. for each social history sentence, we asked human raters to select all the relevant categories and, within each category, all the statements that are supported by the social history snippet. each snippet was rated by at least three raters. for each (snippet, statement) pair, we took the majority vote of raters to get binary entailment labels. rater agreement was high, with a krippendorff's alpha of 0.97 (computed over the binary entailment labels provided by different raters for (premise, hypothesis) pairs). we removed all sdoh statements which were not entailed by any of the social history snippets as well as all note sentences that were not relevant to any sdoh categories. finally, after inspecting the dataset, we removed three sdoh factors ("the person is stably housed.", "the person has social support.", "the person does not have social support.") because raters weren't able to consistently give correct ratings. that left us with 787 unique sentences and 38 sdoh factors. since each snippet would typically only entail one factor or a small number of factors, the resulting dataset is heavily imbalanced: only 4.6% of the labels are positive. we split the dataset along the snippets into training, validation, and test sets with a 70:15:15 ratio, with the following modification: we remove a single pair of factors, "the person lives alone" and "the person does not live alone," from the training and validation sets, in order to evaluate fine-tuned models' ability to generalize to unseen factors. in other words, the training, validation, and test sets have disjoint note snippets but the same sdoh factors, except for the test set which also contains an additional two factors that are not present in the training or validation sets. ## model evaluation since the typical use case is retrieving patients with a particular sdoh factor, and because of the heavy label imbalance in our dataset, we use precision, recall, and f1 score as our evaluation metrics. we evaluate state-of-the-art public models in four different setups: ## conclusion in this work, we introduced sdoh-nli, a new entailment dataset containing social history snippets as premises and social and behavioral determinants of health as hypotheses. our dataset was designed both to reflect realistic use cases of sdoh extraction in clinical settings as well as to provide a high quality entailment dataset to support broader nli research. we evaluated baseline methods using state-of-the-art public models in a variety of setups, and highlighted novel and challenging features of our dataset.