SUPP . AI : finding evidence for supplement-drug interactions (2020)

Abstract: Dietary supplements are used by a large portion of the population, but information on their pharmacologic interactions is incomplete. To address this challenge, we present SUPP.AI, an application for browsing evidence of supplement-drug interactions (SDIs) extracted from the biomedical literature. We train a model to automatically extract supplement information and identify such interactions from the scientific literature. To address the lack of labeled data for SDI identification, we use labels of the closely related task of identifying drug-drug interactions (DDIs) for supervision. We fine-tune the contextualized word representations of the RoBERTa language model using labeled DDI data, and apply the fine-tuned model to identify supplement interactions. We extract 195k evidence sentences from 22M articles (P=0.82, R=0.58, F1=0.68) for 60k interactions. We create the SUPP.AI application for users to search evidence sentences extracted by our model. SUPP.AI is an attempt to close the information gap on dietary supplements by making up-to-date evidence on SDIs more discoverable for researchers, clinicians, and consumers. An informational video on how to use SUPP.AI is available at: https://youtu.be/dR0ucKdORwc

## introduction more than half of us adults use dietary supplements @xcite . supplements include vitamins, minerals, enzymes, and other herbal and animal products. supplements and pharmaceutical drugs, when taken together, can cause adverse interactions @xcite @xcite . some studies describe the prevalence of supplement-drug interactions (sdis) in the hospital setting @xcite or among groups such as patients with cancer @xcite , cardiac disease @xcite , hiv/aids @xcite , or alzheimer's disease @xcite . however, these studies largely rely on manual curation of the literature, and are slow and expensive to produce and update. it is also difficult to aggregate their results, and researchers, clinicians, and consumers can lack appropriate upto-date information to make informed decisions about supplement use. a resource that provides experimental evidence for sdis could serve as a good intermediary tool, allowing experts to quickly access information and translate it for healthcare providers and consumers. such a tool could ease the bottleneck of manual curation by directing researcher attention to the most pertinent and novel interactions appearing in recent trials and case reports. our goal is to create such a resource using state-of-the-art methods in nlp and ie, and allow users to better identify appropriate uses of supplements as well as risks for sdis. automated approaches have been used to extract drug-drug interactions (ddis) from literature and other documents @xcite @xcite @xcite @xcite , complementing broadly-used but primarily manual methods @xcite . we expand upon this work to automatically extract evidence for sdis, as well as supplement-supplement interactions (ssis), from a large corpus of 22m biomedical and clinical texts derived from semantic scholar. 1 we leverage labeled datasets for ddi identification for supervision, and train a model that transfers to the related task of identifying supplement interactions. we surface the resulting evidence on supp.ai for browsing and search. to summarize, our contributions are: ## supplement interaction browser information on supplement interactions have immediate implications on public health, which can only be realized by making the data easily accessible to any interested researcher, clinician or consumer. we note that many medical providers in developing countries do not have subscriptions to clinical databases such as trc foot_1 and uptodate, foot_2 and may lack an easy way to identify possible supplement interactions before prescribing drugs to their patients. to fill this gap, we develop supp.ai (available at https://supp.ai/ ), an application for browsing evidence of supplement interactions extracted from clinical and biomedical literature. supp.ai allows users to: we design supp.ai to be a rapid way for users to access and search extracted sdi and ssi evidence. our goal for this application is to provide a high quality, broadly-sourced, up-to-date, and easily accessible platform for searching through sdi and ssi evidence, while providing sufficient information for users to judge the quality of each piece of evidence. in section 3, we describe the nlp pipeline used to extract evidence from scientific papers. below, we describe the user interface and data features of supp.ai. ## methods an overview of our nlp pipeline is given in figure 2 . we first retrieve medline-indexed articles using the semantic scholar api, 4 and pre-process the text to generate candidate evidence sentences (section 3.1). we then use our ddi-detection model, a neural network classifier based on bert @xcite and fine-tuned on labeled ddi data from @xcite (section 3.2), to classify sentences for the existence of an interaction. sentences classified as positive by our model are collated and surfaced on supp.ai (section 2). ## results & evaluation of the 22m articles we retrieve, around 4.6m abstracts contain candidate sentences. after initial filtering, 33.0m candidate sentences containing supplement entity mentions are classified by roberta-ddi. around 625k (1.9%) of these sentences are classified as positive for an interaction. we perform entity normalization across positive sentences based on cui clusters, and perform additional ad hoc filtering of evidence to eliminate incorrectly detected spans resulting from poor ner and linking, such as the span "retina" linking to vitamin a (c0040845). the resulting 195k sentences contain mentions of 2044 unique supplements and 2772 unique drugs, and provide evidence sentences for 60k interactions sourced from 133k papers. comparisons of model variants on ddi classification and detection (including sota results on both tasks) are given in appendix b. to evaluate the transferability of ddi detection to the related task of sdi/ssi detection, we use a test set consisting of 500 sentences annotated for the presence or absence of a supplement interaction. to obtain a balanced test set despite the rare presence of a positive interaction, we sample half the instances from the set of sentences labeled as positive by a previous variant of our model based on fine-tuning bert-large, and the other half from those labeled as negative. after manual annotation, 40% of the sampled instances were positive for an interaction. annotation was performed by two authors without seeing model predictions, with an inter-annotator agreement of 94%. this test set was used for final evaluation, and never for model development or tuning. decrease in recall can be attributed to a larger percentage of positive instances in the sdi test set (roughly 40%, compared to 20% in the ddi training data). another factor is the presence of incorrectly labeled entity spans in the supplements test set due to ner/linking errors. to better understand this second source of errors, we attempt to evaluate the performance of the scispacy entity linker. processing each sentence from the two ddi training sets using scispacy, we determine that only 80% of drug entities from ddi-2013 and 76% from nlm-dailymed are recognized and linked. the likelihood of supplement entities being successfully linked is likely lower, due to sparse training data for supplement ner and linking. these numbers provide an estimate of the global ceiling on recall for our model. in future work, we aim to explore ways to improve ner and linking and assess their impact on the results of sdi detection. sdi/ssi sentences in our output set can also be labeled by biomedical expert annotators and used to further tune the model for sdi/ssi detection. ## discussion information describing the safety and efficacy of dietary supplements can be difficult to find. the inability to locate evidence of sdis can challenge clinician ability to advise patients and cause risks for consumers of dietary supplements. it is our hope that extracting evidence for sdis/ssis from a large corpus of scientific literature and making the evidence available through an easily accessible search interface can offset some of these risks. this work demonstrates how nlp techniques can be extraordinarily useful for extracting information and relationships specific to an application domain in healthcare. re-purposing existing labeled data from related domains (that would be expensive to generate in a new domain) can be a way to derive maximum utility from curation efforts. continuing, we look to investigate fine-grained interaction types, and provide better classification of the level of evidence provided by each sentence or document towards a particular sdi or ssi. we also aim to leverage similar techniques for identifying evidence of indications, contraindications, and side effects of dietary supplements from the biomedical and clinical literature, and make these discoverable on supp.ai.