Clinical Contradiction Detection (2023)

Abstract: Detecting contradictions in text is essential in determining the validity of the literature and sources that we consume. Medical corpora are riddled with conflicting statements. This is due to the large throughput of new studies and the difficulty in replicating experiments, such as clinical trials. Detecting contradictions in this domain is hard since it requires clinical expertise. We present a distant supervision approach that leverages a medical ontology to build a seed of potential clinical contradictions over 22 million medical abstracts. We automatically build a labeled training dataset consisting of paired clinical sentences that are grounded in an ontology and represent potential medical contradiction. The dataset is used to weakly-supervise state-of-the-art deep learning models showing significant empirical improvements across multiple medical contradiction datasets.

## introduction determining whether a pair of statements is contradictory is foundational to fields including science, politics, and economics. detecting that statements contradict sheds light on fundamental issues. for instance, mammography is an integral routine in modern cancer risk detection, but there is controversy regarding if certain mammography patterns indicate a cancer risk @xcite . recognizing that a certain topic has opposing viewpoints, signifies that the issue may deserve further investigation. medicine is an interesting domain for contradiction detection, as it is rapidly developing, of high impact, and requires an in-depth understanding of the text. per the national library of medicine, the pubmed @xcite database averaged 900k citations for the years 2018-2021, with a growing trajectory @xcite . the publication of contradictory papers is not uncommon in scientific research, as it is part of the process of validating or refuting hypotheses and advancing knowledge. a study on high impact clinical research found that 16% of established interventions had their outcome refuted @xcite . extending this to pubmed, over 5 million articles would disagree with a previous finding. the problem of contradiction detection has been studied in the task of natural language inference (nli) on a sentence level. nli aims to determine whether a pair of sentences are contradictory, entailing, or neutral. deep learning approaches reach impressive results for this task. specifically, large language models (llms) such as debertav3 @xcite and @xcite , are considered the state-of-the-art (sota) for nli. however, in medical research, detecting a contradiction is more difficult. sometimes more context is needed to detect contradiction due to the difficulty of the material. consider the example below (both interventions are ace inhibitors): deep learning methods for nli require large datasets @xcite . however, few datasets exist to train such algorithms in the clinical contradiction domain. time and cost of labeling complex medical corpora, could be a potential reason for this. the mednli dataset @xcite for instance, required the expert labeling of 4 clinicians over the course of 6 weeks 1 . yet, mednli is fabricated since each of the clin-1 mimic-iii certified users can find the mednli dataset 1248 icians was given a clinical description of a patient and came up with a contradicting, entailing, and neutral sentence to pair up with that description. in this work, we are interested in naturally occurring sentences in clinical literature as opposed to manually curated texts. we focus on sentences containing clinical outcomes and attempt to identify whether outcomes contradict. we will show that even this step, which does not address the intervention and population, yields an initial system for identifying medical contradictions. one approach to overcome the lack of large data is distant supervision @xcite . distant supervision uses existing knowledge sources to automatically label large amount of data. the labels can be noisy, so the goal is to train robust models that learn meaningful patterns. we propose a novel methodology leveraging distant supervision and a clinical ontology, the systematized nomenclature of medicine clinical terms (snomed) @xcite . snomed is developed by a large and diverse group of medical experts @xcite and it contains extensive information about clinical terms and their relations. our method uses knowledge extracted from snomed to classify pairs of "naturally occurring", potentially contradictory sentences. pubmed's abstract database is our source for naturally occurring sentences. we fine-tune sota deep learning models on the aforementioned ontology-driven created dataset. we test the results using manually labeled clinical contradiction datasets. the results demonstrate that our distant-supervision-based methodology yields statistically significant improvements of the models for contradiction detection. the average results of 9 different models see an improvement on our main evaluation set (section 4.1.1) over previous sota. specifically, we find that the improvement is consistent across both small models and those that are considered to be sota on nli tasks, which is the closest task to that of contradiction detection. the contribution of our work is threefold: (1) we present the novel problem of contradiction analysis of naturally occurring sentences in clinical data. (2) we create a clinical contradiction dataset by using distant supervision over a clinical ontology, yielding improvements of sota deep learning models when fine-tuning on it. (3) we empirically evaluate on numerous manually labeled clinical contradiction datasets and show improvements of sota here: https://physionet.org/content/mednli/1.0.0/ models when fine-tuned on the ontology-driven dataset. we make our code publicly available foot_0 . despite the complexity of medical literature and the reality of contradictions in publications, there is surprisingly little work in this area. large nli corpora contain relatively easy contradiction pairs, partly due to the cost of annotating complex contradictions. the contradiction is often a negation through words like 'not'. an example from a large nli corpus, @xcite is: scientific fact-checking is a related task, where a claim is verified against evidence @xcite . the work in this field deals with popular claims justified by evidence from a corpus, as opposed to naturally occurring sentences in medical literature. @xcite 's data comes from popular media sources such as the associated press and reuters news. @xcite developed a labeled dataset, manconcorpus, for contradictory cardiovascular research claims in abstracts. this corpus has pairings between a fabricated query and a naturally occurring claim, thus not dealing with naturally occurring sentences. it is annotated by domain experts. there are works which address contradiction of a clinical query and a claim. tawfik and spruit (2018) use hand-crafted features to build a classifier, whereas yazi et al. (2021) use deep neural networks (see baseline in semantic predications from semmeddb @xcite are commonly used in clinical contradictions. @xcite uses semmeddb to construct autoconcorpus by querying subjectpredicate-object tuples. predicates are 'incompatible' if they belong to different groups. the dataset is in the same query-claim format as manconcorpus and the 'retrieval' stage is manual. @xcite also use semmeddb by finding relevant semantic predications with their corresponding opposing predicate pairs. unlike these approaches, we do not use predicate logic. in addition, our dataset creation is fully automated. we show its efficacy by fine-tuning models on it, which is not done by either of the semmeddb datasets. @xcite introduced distant supervision, which was later extended to knowledge bases like yago by @xcite . neural networks became popular for distantlysupervised relation extraction @xcite . we apply distant supervision to detect contradictions between sentence pairs representing clinical outcomes. we use weak supervision during fine-tuning with sota models and leverage the relational knowledge of a clinical ontology. unlike common approaches @xcite , we infer contradictions using the structure and attributes of a clinical ontology instead of relying on known relation labels. our approach yields positive and negative term relations, distinguishing it from traditional distant supervision models @xcite . to our knowledge, we are first to use distant supervision for detecting contradictions in the clinical domain. ## methods our method creates a dataset for training or finetuning using ontology knowledge, specifically fo-cusing on ontology parts related to clinical findings, which represent the outcomes of clinical observations @xcite . unlike mutually exclusive nouns like 'cat' or 'dog' the terms we encounter in this context may have conflicting clinical findings, suggesting a difference in clinical outcome. ## conclusions contradiction detection is central to many fields, but is especially important in medicine due to human impact and rapid clinical research growth. though contradictions are a subfield of nli, there is less work in the clinical domain. medical contradictions require context and domain knowledge, making them complex. labeling datasets which produce effective deep learning models is costly. we introduce a novel method of using a clinical ontology to weakly-supervise the creation of a potentially contradicting dataset with naturally occurring sentences. we coin it the snomed dataset. empirical results suggest that fine-tuning on the snomed dataset results in consistent improvement across sota models over diverse evaluation datasets spanning multiple medical specialties. we find a balance between term group size and the number of pubmed sentence samples per pairing. additionally, better results are achieved by filtering the pubmed sentences included in our dataset. for future exploration we suggest investigating more robust sentence filtration methods, such as topic modeling or sentence embedding similarity. looking into how other clinical ontologies can be paired with snomed may also be fruitful.