Detecting Contradictory COVID -19 Drug Efficacy Claims from Biomedical Literature (2023)

Abstract: The COVID-19 pandemic created a deluge of questionable and contradictory scientific claims about drug efficacy – an “infodemic” with lasting consequences for science and society. In this work, we argue that NLP models can help domain experts distill and understand the literature in this complex, high-stakes area. Our task is to automatically identify contradictory claims about COVID-19 drug efficacy. We frame this as a natural language inference problem and offer a new NLI dataset created by domain experts. The NLI framing allows us to create curricula combining existing datasets and our own. The resulting models are useful investigative tools. We provide a case study of how these models help a domain expert summarize and assess evidence concerning remdisivir and hydroxychloroquine.

## introduction the covid-19 pandemic caused by the novel sars-cov-2 virus completely changed modern life. according to the world health organization nov. 16, 2022, situation report, more than 6.5 million people have died as a result of this disease @xcite . during times of pandemic, treatment options are limited, and developing new drug treatments is infeasible in the short-term @xcite . however, if a novel disease shares biological underpinnings with another disease for which a drug treatment already exists, a doctor may be able to repurpose that drug as a treatment for the new disease with positive therapeutic effect @xcite . this strategy has been successful in several contexts (corsello et al., 2017; himmelstein et al., 1 our covid-19 nli dataset and code are available at https://github.com/dnsosa/covid_lit_contra_ claims 2022; @xcite and may be the only viable strategy during an emerging pandemic. decisions about repurposing drug treatments are predicated on scientific knowledge. making predictions about how to repurpose an existing drug requires understanding the target disease's mechanism. because sars-cov-2 was a new virus, our knowledge of covid-19's mechanism rapidly evolved. the biomedical literature about the virus and disease proliferated at an unprecedented rate @xcite . the need for knowledge about the virus and the bottleneck of limited peer reviewers led to many cases of circumventing typical quality control mechanisms for research. to inform their clinical practice, healthcare professionals relied on knowledge sources of lower scientific quality including early clinical reports with small sample sizes and non-peer reviewed manuscripts posted on preprint servers @xcite . this deluge of rapidly changing information became an "infodemic", and it became infeasible for the average clinician to stay up-to-date with the growing literature @xcite . automated methods have great potential to help domain experts fight such an infodemic. we illustrate this potential with a case study focused on automatically detecting contradictory research claims in the covid-19 therapeutics literature. we frame this as a natural language inference (nli) problem: given pairs of research claims in biomedical literature, we develop models that predict whether they entail, contradict, or are neutral with respect to each other. our models are trained on a new dataset of these claim pairs extracted from the cord-19 dataset @xcite and annotated by domain experts. our best models are trained on curricula @xcite of existing nli datasets and our domain-specific one. these models are effective at the nli task, but the ultimate test of their value is whether they can help domain experts. we show how these models could help a domain expert to see early on that hydroxychloroquine was an ineffective covid-19 treatment and how the story of remdisivir was still emerging. ## covid-19 nli dataset our new covid-19 nli dataset consists of pairs of research claims describing covid-19 drug treatment efficacy and safety. these claims came from the subset of the june 17, 2020 (v33) cord-19 @xcite ) manuscripts containing a covid-19-related term (e.g., "sars-cov-2", "2019-ncov"). claims were extracted from the articles' full text using the lstm approach of @xcite . false positive research claims were manually removed. to begin the annotation process, we inspected pairs of claims on common drugs and topics. this led us to a set of five categories: strict entailment, entailment, possible entailment, strict contradiction, contradiction, and neutral. our annotation guidelines were developed and refined by clinically trained annotators (nurses and a biomedical researcher) over two preliminary rounds of annotation. in round 1, four annotators labeled 64 claim pairs (fleiss'  = 0.83). the team discussed this process and refined the guidelines. in round 2, three annotators (a subset of those from round 1) annotated 75 claim pairs (fleiss'  = 0.84) using the new guidelines, and then determined that they were ready to scale (appendix a.1). for the dataset itself, 1000 pairs of claims were sampled for annotation using three criteria: (1) both claims mention at least one of 7 treatment candidates ({"hydroxychloroquine", "chloroquine", "tocilizumab", "remdesivir", "vitamin d", "lopinavir", "dexamethasone"}), (2) high similarity between the claim's embedding and the embedding for a word in a predefined topic list ({"mortality", "effective treatment", "toxicity"}), using usif embeddings @xcite , and (3) non-zero polarities of equal or opposite sign using vader @xcite . appendix a.3 provides further details. each annotation requires a large time investment from the annotator and draws heavily on their domain expertise, so each example was annotated by a single annotator, with the inter-annotator agreement rounds and guidelines serving to ensure consistency across the dataset. because some claims are present in multiple claim pairs, we selected a subset of pairs such that no claim is present in more than one train, validation, or test split to prevent test-set leakage. from the network of claim pairs (claims are nodes, and co-occurrences in an annotated pair are edges), we selected 3 disjoint subnetworks to comprise the train, validation, and test splits. the resulting dataset contains 778 total claim pairs. dataset distributions are found in ## model development our goal is to develop a model to help domain experts find and adjudicate contradictory claims in the covid-19 literature. we explored a wide range of techniques for developing the best model given our available data. the appendices provide a full overview of all these experiments and comparisons. here, we provide a high-level summary. pretrained parameters all our models begin with pretrained parameters created using the general architecture of bert @xcite . five pre-trained bert models were evaluated for further fine-tuning: pubmedbert @xcite , scibert @xcite , biobert @xcite , bioclinbert @xcite , and roberta @xcite . we found that pubmedbert was the best for our task across all fine-tuning regimes (appendix d). fine-tuning curricula for fine-tuning these parameters, we use multinli @xcite , mednli @xcite , mancon-corpus @xcite , and our new covid-19 nli dataset (with our six labels collapsed to three as in the other datasets). we found that the best models were achieved with a curriculum that arranged these in the order we gave above. this is intuitively an arrangement from most general to most domain-specific, which aligns with existing results and intuitions for curriculum learn- results to contextualize our results on this hard, novel task, we evaluated a number of baselines using sparse feature representations and simple similarity calculations, as well as hypothesis-only variants of these models and our bert models. these baselines are described in appendix f. ## case study: wading through the sea of drug treatment literature the value of our model lies in its potential to help domain experts tackle an infodemic. we used the model to understand the state of knowledge about the efficacy and mechanism of two controversial treatments, hydroxychlorouqine and remdesivir, from the perspective of june 2020. we first extracted all claims identified from covid-19 manuscripts concerning a drug treatment, using the same procedure as for our covid nli dataset (section 2), and we filtered that set to pairs of claims that were (1) sufficiently similar (usif @xmath0.5) and (2) both mentioned remdesivir or hydroxychloroquine. we sampled pairs from 50 papers yielding 5,336 total pairs. we then used our best model to make predictions about all these pairs resulting in 322 predicted contradictions. we ranked these by the model's predicted probability of this class, and we inspected the highest probability predictions. for remdesivir, one claim of limited efficacy from an clinical trial of 233 participants yielded several predicted contradictions: nineteen unique papers contained a claim that was predicted to contradict this claim -already a striking pattern that might have taken a researcher days to discover by hand by probing full-text articles. the specific claims that contradict our core claim are illuminating. one reads, from another ongoing clinical trial including 1,064 patients, authors note: ## stakeholders there are several biomedical stakeholders who would benefit from models like ours. epidemiologists epidemiologists survey public health data to inform policy decisions in collaboration with authoritative bodies like the nih and who. their recommendations must be conservative, so surfacing results that dispute claims of drug efficacy is critical. their gold standard resource for aggregating evidence is the meta-analysis, but in the early stages of the pandemic, large randomized controlled trials (rcts) had not completed, and review articles quickly became outdated. fda regulators regulators too need to make conservative recommendations, as fda approval signals to clinicians that a treatment is standard-ofcare. surfacing contradictory claims of drug efficacy and safety is essential @xcite ). researchers by identifying areas of scientific uncertainty via contradictory evidence at all stages of the pipeline (in silico, in vitro, in vivo, clinical), researchers could have more quickly identified fruitful areas of investigation @xcite . drug manufacturers manufacturers of repurposing candidates were incentivized to understand in what settings their drug seemed to be effective and by what mechanism. for claims of inefficacy, they were interested in surfacing any mitigating factors qualifying these claims or motivating followup analyses. we note that these models are not intended as the sole source of decision making in clinical or epidemiological settings. to be clinically translatable, further work would need to be conducted on assessing the quality of research claims by relying on contextual information including research setting, demographics, size, and level of evidence. rather, this work is intended to augment a manual curator's capacity to distill and synthesize a large corpus of literature. this allows trained researchers to use their judgment and conduct last-mile diligence of surfaced research contradictions or corroborations, which will be beneficial to these stakeholders downstream. ## discussion and conclusion in settings where the scale of literature is insurmountable for human readers, as is the case during a pandemic, automated curatorial assistants can be transformative @xcite . during covid-19, meta-analyses and review articles, which are written to synthesize a large body of literature, could not be comprehensive or quickly became outdated. in some cases, it was necessary to create meta-meta-analyses involving hundreds of papers @xcite . our work shows the value of integrating nlp into the domain of meta-science, embracing all the complexities of biomedical research as it naturally exists in literature. we presented an nli framing for identifying contradictory or corroborating research claims in the challenging domain of covid-19 drug efficacy. we created a new dataset and designed curricula for optimizing language model fine-tuning for the task. to illustrate the potential of our model, we showed that we were quickly able to distill the state of knowledge about hydroxychlorouqine and remdesivir efficacy as of june 2020, arriving at conclusions that are extremely well-supported in 2022. identifying where science is inconsistent is necessary for understanding the current state of human knowledge and reveals frontiers for further research. significant contradictions can often be found buried in biomedical articles; surfacing these instances nearly as quickly as research is publicly disseminated can generate leads that researchers and curators should pursue. beyond facilitating search and discovery, our method can help estimate confidence in the consensus of facts in science when creating general knowledge representations (sosa and altman, 2022) for downstream applications like predicting novel drug repurposing opportunities in silico @xcite .