B io DEX : Large-Scale Biomedical Adverse Drug Event Extraction for Real-World Pharmacovigilance (2023)

Abstract: Timely and accurate extraction of Adverse Drug Events (ADE) from biomedical literature is paramount for public safety, but involves slow and costly manual labor. We set out to improve drug safety monitoring (pharmacovigilance, PV) through the use of Natural Language Processing (NLP). We introduce BioDEX, a large-scale resource for Biomedical adverse Drug Event eXtraction, rooted in the historical output of drug safety reporting in the U.S. BioDEX consists of 65k abstracts and 19k full-text biomedical papers with 256k associated document-level safety reports created by medical experts. The core features of these reports include the reported weight, age, and biological sex of a patient, a set of drugs taken by the patient, the drug dosages, the reactions experienced, and whether the reaction was life threatening. In this work, we consider the task of predicting the core information of the report given its originating paper. We estimate human performance to be 72.0% F1, whereas our best model achieves 59.1% F1 (62.3 validation), indicating significant headroom. We also begin to explore ways in which these models could help professional PV reviewers. Our code and data are available at https://github.com/KarelDO/BioDEX.

## introduction in the united states, the food and drug administration (fda) mandates drug producers to monitor and report adverse drug events (ade) described in the biomedical literature. such a report, called an individual case safety report (icsr), is stored in the fda adverse event reporting system (faers; food and drug administration 2017), which is a cornerstone resource for drug safety research, also called pharmacovigilance @xcite . figure 1 briefly summarizes the core information pv workers must extract from papers while constructing these reports. this includes a description of the patient in terms of reported weight, age, figure 1 : biodex consists of 65k pubmed abstracts and 19k full text papers, accompanied by 256k document-level drug safety reports. the schematic illustrates the core information that constitutes a drug safety report (they often contain much more detailed information as well). these reports are created by pharmacovigilance experts and are vital for drug safety monitoring. and biological sex, a list of drugs taken by the patient, and a list of adverse reactions experienced and whether they are considered serious. drug manufacturers employ teams of experts to continually triage new papers and submit these reports. this is challenging work since it requires experts to survey entire biomedical papers and utilize their pre-existing knowledge about a drug of interest, its conventional indications, and its known adverse reactions. furthermore, manufacturers are placed under constant time pressure to keep up with the latest publications, since failure to report in a timely manner can lead to hefty fines and compromise public safety. this pressure has potential to increase in the near future: there has been a steady acceleration of biomedical research over the last few years (figure 2 ), and drug events are consistently under-reported @xcite . in this work, we set out to improve the scalability and accuracy of pv using natural language processing (nlp). as a first step, we introduce biodex, a large-scale dataset for documentlevel biomedical adverse drug event extraction. biodex consists of biomedical papers with associated expert-created drug safety reports. these reports were submitted to the fda between 2012 and 2022 as part of real-world pv efforts. thus, biodex is grounded in the historical and regulatory context of drug safety monitoring in the u.s. biodex contains pubmed articles published between 1968 and 2022, with 65,648 articles having an abstract available and 19,433 featuring a fulltext paper. in total, 256,240 reports are included (there can be multiple reports per article). we evaluate the ability of language models (lms) to fill out the core information of a report given a full-text article that is known to describe at least one ade. we estimate a lower bound on human performance to be 72.0% f1. our best model (a fine-tuned flan-t5-large; chung et al. 2022) attains 59.1% f1, indicating substantial additional room for improvement while also suggesting that models trained on biodex are on a path to being useful tools for pv workers. additionally, we evaluate the capability of openai's gpt models (text-davinci-002, text-davinci-003, gpt-3.5-turbo, gpt-4; brown et al. 2020) but find that they severely struggle with this task, attaining at most 53.1% f1. our models can aid drug safety research efforts today. an important use-case for drug safety research is efficiently finding papers that describe an adverse event with regard to a specific drug or reaction. conventional search baselines suffer from low precision, since mentioned drugs and reactions are only rarely involved in an adverse event. our models are specifically trained to extract adverse events, leading to better performance. all our code and data are available as supplementary material. ## pharmacovigilance reporting pharmaceutical companies are required to participate in drug safety reporting for the drugs they produce. regulations differ across regions of the world. in this work, we focus on the pharmacovigilance process as defined by u.s. regulations. the reporting process starts with a pv literature review stage. periodically, a vast database of biomedical literature is queried to retrieve new publications that could describe an adverse event with regard to a drug of interest. conventionally this is done by matching the trade name of the drug or names of its active substances. these queries are designed by experts and depend on the specific usecase, but they always aim for wide coverage; there are strong regulatory fines associated with missing reports, which creates strong incentives for very high recall. reports can also originate from other modalities such as forms, emails, and social media. in this work, we only focus on reports originating from biomedical publications. once a set of candidate publications is found, a triaging process begins. for example, papers that mention a serious adverse event should be prioritized, as these reports need to be submitted in a strict time window. this is often done via another high recall system that matches words such as 'serious' and 'life threatening' via a lexicon-based approach. each resulting publication is investigated by expert pv workers in a multi-stage pipeline, which can differ across companies. typically, the initial flagging of potential ades is done by nonclinician pv workers. evidence is flagged and can be mapped to a standardized ontology to introduce uniformity in downstream stages. subsequently, clinicians review the report and refine the event details before the report is submitted. in this work, we abstract away the details of this human-based workflow and model the task as taking in a biomedical publication and outputting the final pharmacovigilance report. systems that perform well at this task could go a long way towards automating pharmacovigilance. a key challenge is that pv requires processing entire biomedical publications, which pubmedqa does not support but biodex does. recently, @xcite introduced pmc-patients, a large-scale dataset for patient-to-patient or patient-to-article retrieval built on top of pubmed. biodex can be seen as complementing this effort; instead of retrieving relevant papers, biodex aims to extract structured patient information from biomedical publications for pharmacovigilance purposes. both the extraction of the information as well as the retrieval of relevant articles are highly relevant for evidence-based medicine (ebm; sackett 1997) and pharmacovigilance. ## task and metrics in this work, we focus on the task of predicting the core information of a report given a full-text paper, which we call report-extraction. accurate and autonomous extraction of drug safety reports can have a large impact on pv by increasing the quality of safety signals and decreasing the time required to surface new signals. ## experiments and results motivated by the recent success of llms, we choose to model report-extraction as a sequenceto-sequence problem. 3 given a full-text paper as input, we train models to predict the core report in a stringified format, such as "serious: 1 patientsex: 1 drugs: azathioprine, infliximab, mesalamine, prednisolone reactions: epstein-barr virus infection reactivation, idiopathic interstitial pneumonia". we report validation results for all models considered. only the best models are subsequently evaluated on the test split. ## improving pharmacovigilance our primary goal is to improve the scalability and accuracy of pv using nlp. the above experiments highlighted the potential for lms to autonomously fill in ade reports. however, fully autonomous drug event reporting systems are unlikely to achieve widespread adoption today. mainly because of the challenging nature of this task and the high cost of errors, human experts will remain vital for effective solutions in the years to come. however, our models can still deliver tangible value by augmenting existing expert-based workflows. given the vast number of biomedical papers published, it is increasingly impractical to thoroughly vet every candidate publication (as is currently being done). drug manufacturers are looking to more efficiently triage the literature to prioritize efforts, as this minimizes risk from regulatory fines and maximizes public safety. such a triaging system is typically based on a naive lookup: finding all papers that match a drug name is likely to find all papers where that drug engages in an adverse event. unfortunately, such a system has low precision, causing human effort to be wasted investigating irrelevant papers. we find that our model predictions achieve a higher performance at finding adverse events concerning specific reactions, compared to the lookup baseline. we measure this through the macro average f1 score on the binary classification task of predicting per paper if a reaction was part of an adverse events. figure 6 shows the results for the 30 most frequent reactions in the validation split. high-recall baselines still have a valuable place in the pv review process, but our system could be used to more efficiently prioritize effort. appendix f describes the same experiment for drugs. future work could utilize all details of biodex reports or incorporate the 65k abstract-level datapoints during training to further improve utility for pv. for example, biodex would support finetuning a question answering model for pv. ## conclusion we introduced biodex, a large-scale documentlevel biomedical adverse drug event extraction dataset. biodex covers an important and challenging real-world task: extracting detailed drug safety reports from full-text biomedical publications. we find that llms struggle to get traction on this task using in-context learning. fine-tuned models are more successful, but expert-level performance remains elusive. nevertheless, our models have the potential to make drug safety research more efficient, and we demonstrated their utility in a conventional pv use-case. we release all data and models. we hope that biodex stimulates new research in the high-impact area of drug safety monitoring.