CARE : Extracting Experimental Findings From Clinical Literature (2024)

Abstract: Extracting fine-grained experimental findings from literature can provide dramatic utility for scientific applications. Prior work has developed annotation schemas and datasets for limited aspects of this problem, failing to capture the real-world complexity and nuance required. Focusing on biomedicine, this work presents CARE—a new IE dataset for the task of extracting clinical findings. We develop a new annotation schema capturing fine-grained findings as n-ary relations between entities and attributes, which unifies phenomena challenging for current IE systems such as discontinuous entity spans, nested relations, variable arity n-ary relations and numeric results in a single schema. We collect extensive annotations for 700 abstracts from two sources: clinical trials and case reports. We also demonstrate the generalizability of our schema to the computer science and materials science domains. We benchmark state-of-the-art IE systems on CARE, showing that even models such as GPT4 struggle. We release our resources to advance research on extracting and aggregating literature findings.

## introduction it is surely a great criticism of our profession that we have not organised a critical summary, by specialty or subspecialty, adapted periodically, of all relevant randomised controlled trials. @xcite though this critique focused on clinical trials, the statement arguably applies to much of science today. there is tremendous potential utility in extracting, structuring and aggregating finegrained information about experimental findings and the conditions under which they were achieved, across scientific studies. once extracted and aggregated, scientific findings can power many critical applications such as producing literature reviews @xcite , supporting evidence-based decision-making @xcite , and generating new hypotheses @xcite . while there have been efforts on building resources and tools to capture findings in various domains such as clinical trials @xcite , computer science @xcite and social and behavioral sciences (magnusson and friedman, 2021)-a major obstacle has been creating a representation that is expressive enough to capture complex and nuanced information about findings. we propose a new representation schema that makes important progress in capturing the realworld complexity of scientific findings in papers, and use it to build a high-quality annotated dataset focusing on biomedical (clinical) findings. our schema represents fine-grained information about experimental findings and conditions as n-ary relations between entities and attributes, and includes several structural complexities such as discontinu-ous span annotation, variable arity in relations and nestedness in relations. these aspects have been studied individually in previous datasets @xcite , but our schema is the first to unify them. our dataset also captures numeric findings in addition to their interpretation (e.g., significance, utility, etc.); prior datasets typically focus solely on the latter (e.g., @xcite captures increases/decreases in outcomes but not their magnitudes). though these factors make our annotation schema more complex than prior work, the additional nuance it affords can power several highimpact downstream use cases. for instance, the process of conducting a systematic review 2 includes extracting data about specific outcomes from a large number of relevant studies. unlike prior schemas which only capture interpretation and not precise numeric outcomes, extractors trained on our schema can extract relevant data to assist the review process. moreover, fine-grained population and treatment details captured by our schema makes it capable of answering highly complex clinical questions (e.g., "does vaccination significantly improve mortality outcomes for female patients over 65 who required ventilation?"). therefore, models trained on our schema can assist physicians in developing more personalized treatment plans, especially for patients with multiple comorbidities or health conditions. our schema can also help with the development and exploration of richer clinical hypotheses due to its additional granularity. to build our dataset, named care (clinical aggregation-oriented result extraction), we collect extensive annotations for 700 abstracts (clinical trials and case reports). we also conduct annotation studies demonstrating that our schema generalizes to computer science and materials science, using minor updates based on analogies between aspects across experimental domains (e.g., populations/@xmath0. this reflects the expressive power of our schema to generalize across domains while capturing granular and useful information, making it a strong "backbone schema" for research efforts on resultoriented scientific ie. we achieve good agreement scores (0.74-0.78 partial f1) comparable to prior work that used simpler schemas that are easier to annotate (luan et al., 2 https://training.cochrane.org/handbook/ current 2018; @xcite , and at the same time our resulting dataset is larger in size than previous corpora. our final dataset annotation is extremely rich; at 16.23 relations per abstract, our relation density is nearly 4x that of prior work on annotating findings from clinical trials @xcite . we evaluate a wide range of ie models on our dataset, including both extractive systems and generative llms. given the high annotation burden, we test generative llms in both fully supervised as well as zero-shot and few-shot settings. our results demonstrate the difficulty of our dataset, with even sota models such as gpt4 struggling to accurately extract clinical findings. as a highly challenging new dataset designed to be reflective of real-world nuance and informational needs, we hope care foot_0 is an important resource for the scientific nlp and ie research community to pursue. ## annotation schema we develop a new annotation schema to represent fine-grained clinical findings present in biomedical abstracts, and later demonstrate its broader applicability to domains beyond biomedicine ( §6.2). our schema captures this knowledge via three main elements, commonly used in ie tasks: type arity ei ctkg example attributeof n-ary ✗ ✗ (subpopulation: 144 had the u-type method, size: 144) subpopulationof n-ary ✗ ✗ (p opulation: 285 women, subpopulation: 144 had the u-type method, subpopulation: 141 had the h-type method) treatmentof binary ✗ ✓ (subpopulation: 144 had the u-type method, t reatment: u-type method) result n-ary ✓ ✓ (subpopulation: 144 had the u-type method, m easurement: objective cure rates, n umericf inding: 87.5%) ## dataset collection annotation tool: we use teamtat foot_1 @xcite , a web-based tool for team annotation since it allows for n-ary and nested relation annotation, a core component of our schema. annotator background: we recruit two in-house annotators foot_2 with backgrounds in data analytics and data science, both having extensive experience in reading and annotating scientific papers. one of our annotators has a background in biology. both annotators went through several pilot rounds to gain familiarity with our task and schema. additionally, we used their feedback and insights from pilots to solidify our schema design (see §4.1). we also solicited feedback from two medical students and an md to validate our final schema. data sources: care covers two categories of biomedical literature: (i) clinical trials, and (ii) case reports. clinical trials are research studies that test a medical, surgical, or behavioral intervention in people to determine whether a new form of treatment or prevention or a new diagnostic device is effective. case reports are detailed reports of the symptoms, signs, diagnosis, treatment, and followup of an individual patient, usually motivated by unusual or novel occurrences. we sample clinical trials from the ebm-nlp @xcite dataset, which consists of 4993 abstracts annotated with pico spans, only retaining abstracts containing at least one number (4685 in total). to sample case reports, we extract all reports with at least one number in the abstract from pubmed (907,862 in total) and randomly sample from this pool. we sample 350 abstracts from each source, resulting in our final dataset size of 700 abstracts, which is slightly larger than other prior corpora that perform finegrained annotation ( § 4.3). further characteristics of our abstract sample are detailed in appendix c. ## benchmarking ie models we benchmark the performance of two categories of models on care: (i) extractive models, and (ii) generative llms. we also test generative llms in two settings: (i) finetuning on the full training set, and (ii) zero-shot and in-context learning. ## conclusion in this work, we presented care, a new ie dataset for the task of extracting clinical findings from biomedical literature. to collect this dataset, we first developed a new annotation schema capable of capturing fine-grained information about experimental findings, which unified several challenging ie phenomena such as discontinuous spans, nested relations and variable arity n-ary relations. using this annotation scheme, we collected an extensively annotated dataset of 700 abstracts from clinical trials and case reports. our benchmarking experiments showed that state-of-the-art extractive and generative llms including gpt4 still struggle on this task, particularly on relation extraction. we release both our annotation schema and care as a challenging new resource for the ie community and to encourage further research on extraction and representation of findings from scientific literature.