Intervention extraction in preclinical animal studies of A lzheimer’s Disease: Enhancing regex performance with language model-based filtering (2024)

Abstract: We explore different information extraction tools for annotation of interventions to support automated systematic reviews of preclinical AD animal studies. We compare two PICO (Population, Intervention, Comparison, and Outcome) extraction tools and two prompting-based learning strategies based on Large Language Models (LLMs). Motivated by the high recall of a dictionary-based approach, we define a two-stage method, removing false positives obtained from regexes with a pre-trained LM. With ChatGPT-based filtering using three-shot prompting, our approach reduces almost two-thirds of False Positives compared to the dictionary approach alone, while outperforming knowledge-free instructional prompting.

## introduction and related work biomedical information extraction is the task of automatically extracting entities, relations, and events from biomedical literature @xcite . this information is in turn relevant to writing of systematic reviews, which support evidencebased decision making by identifying, integrating, and assimilating relevant articles on a given clinical question @xcite . a standard framework used for defining review questions is pico, standing for population (or patient), intervention (or exposure), comparison, and outcome @xcite . we examine information extraction in alzheimer's disease (ad), which has affected more than 55 million people around the world 1 . we focus on detecting the pico dimension of intervention in the ad literature, where interventions are typically drugs. this task is sometimes referred to as intervention extraction. it suffers from having low precision compared to extraction of other pico elements (hair et al., 1 https://www.alzint.org/about/dementia-factsfigures/dementia-statistics/ 2023a). more precise extraction of interventions will support more effective systematic reviewing, and can help to prioritize drugs for clinical trials in literature-based discovery @xcite . standard methods for intervention extraction include dictionary-based approaches @xcite and machine learning models @xcite . the recent advent of generative large language models (llms) and the prompting-based paradigm for information extraction @xcite raise questions of how to better leverage them in this task and whether they outperform previous methods. this is particularly interesting in domain-specific scenarios such as ad, where limited data is available for training models @xcite . we address these questions, with two main contributions: 1) we show that while generative llms improve intervention extraction precision, they suffer from low recall compared to dictionary-based methods, and 2) we propose a two-stage architecture combining both dictionaries and llms that better balances precision and recall and reaches a new state-of-the-art on the ad dataset. ## results and discussion our zero-shot filtering approach outperformed the baselines in f1 score, with only a small decrease in recall. it filtered 30% of the false positives (fps) of the regex-based method. a three-shot variant, adding three true positive (tp) examples to the prompt, gave even better precision and f1 score, filtering almost two-thirds of all fps. ## conclusion in this work, we proposed a two-stage approach for intervention extraction that combined a regexbased method with a filtering step done by prompting a generative llm. this approach outperformed strong baselines, including standalone use of llms. effectively, we show that llms can augment regex/dictionary-based methods by removing context recognition errors. future work involves extending our approach to all pico entities, beyond just interventions. this will help automate important tasks in the literature review for ad, such as collecting data for systematic reviews, and support creating more precise knowledge graphs for literature-based discovery. the same approach could also be adapted with specific resources and be applied to other datasets and domains, such as clinical trials @xcite . finally, different strategies to employ llms in the filtering step could be investigated, such as finetuning. betty törnkvist. 2024. named entity recognition for detecting trends in biomedical literature. master's thesis, umeå university, department of computing science. benyou wang, qianqian xie, jiahuan pei, zhihong chen, prayag tiwari, zhao li, and jie fu. 2023. pretrained language models in biomedical domain: a systematic survey. acm comput. surv., 56(3). qianying wang, jing liao, mirella lapata, and malcolm macleod. 2021. pico entity extraction for preclinical animal literature. systematic reviews, 11. chih-hsuan wei, alexis allot, po-ting lai, robert leaman, shubo tian, ling luo, qiao jin, zhizheng wang, qingyu chen, and zhiyong lu. 2024. pubtator 3.0: an ai-powered literature resource for unlocking biomedical knowledge. nucleic acids research. a prompting with negative shot for a two-stage filtering method #examples #tp #fp tp price fp reduction baseline 67 288 0 (0%) 0 (0%) 0 64 203 3 (4%) 85 (30%) 1 41 104 26 (38%) 184 (63%) 2 52 118 15 (22%) 170 (59%) 3 58 167 9 (13%) 121 (42%) 4 61 167 6 (8%) 121 (42%) 5 53 106 14 (20%) 182 (63%) c a prompting template for a two-stage filtering method c.1 zero-shot learning task description: you will be provided with a text span and a block of text. your task is to decide an entity type for the text span by considering the block of text as a context. task content: text span [x]: glutathione. a block of text: moreover the reduced activities or contents of glutathione reductase superoxide dismutase (sod) and reduced gsh within the cortex and hippocampus caused by scopolamine were elevated by the treatment of kd-501. please fill in the slot [z]: [x] belongs to an entity type @xcite . choose an entity type [z] from the ['intervention', 'not an intervention']