Extraction of Diagnostic Reasoning Relations for Clinical Knowledge Graphs (2022)

Abstract: Clinical knowledge graphs lack meaningful diagnostic relations (e.g. comorbidities, sign/symptoms), limiting their ability to represent real-world diagnostic processes. Previous methods in biomedical relation extraction have focused on concept relations, such as gene-disease and disease-drug, and largely ignored clinical processes. In this thesis, we leverage a clinical reasoning ontology and propose methods to extract such relations from a physician-facing point-of-care reference wiki and consumer health resource texts. Given the lack of data labeled with diagnostic relations, we also propose new methods of evaluating the correctness of extracted triples in the zero-shot setting. We describe a process for the intrinsic evaluation of new facts by triple confidence filtering and clinician manual review, as well extrinsic evaluation in the form of a differential diagnosis prediction task.

## introduction knowledge graphs (kgs) are increasingly utilized in key knowledge-intensive applications, such as recommendation and question answering. however, their utility in these systems can be limited due to missing facts (triples) among entities @xcite . the missing knowledge in kgs largely comes from three main sources: missing unknown entities, missing unknown relations, and missing existing relations between known entities. significant advances have been made in the general and biomedical domains in recent years to tackle each of these problems, using techniques from the nlp and graph communities such as entity linking (el) @xcite , relation extraction (re) @xcite , and link prediction @xcite . in this work, we define a set of missing clinically meaningful diagnostic relations based on an existing clinical reasoning ontology (cro). we then propose two methods of adding such relations to the snomed-ct knowledge graph (kg) using distinct and complementary data sources. we describe a relation extraction task using a semistructured emergency department (ed)-focused wiki and a zero-shot relation classification (rc) task using a newly gathered corpus of consumer health information derived from medlineplus foot_0 and merck manuals (bullers, 2016), described in detail in section 3.2. throughout this work, we will address the following research questions: rq1. can we leverage the semi-structured form of a wiki to extract reasoning relations? rq2. do consumer health resources provide distinct missing relations from those found in rq1? rq3. how do we evaluate the accuracy of new relations in a clinical kg? in rq1 and rq2, we limit ourselves to a predetermined set of relations to minimize hand curation by domain experts, such as would be required with free text relations extracted using an open information extraction system @xcite . however, we still need to determine the accuracy of our new facts to evaluate the model. this leads us to rq3, in which we determine how to evaluate the accuracy of new relations in a clinical kg, given that such relations don't currently exist. we propose intrinsic and extrinsic methods of evaluation in this zero-shot setting. the rest of this proposal will be structured as follows. in section 2, we will describe the existing biomedical and clinical relation extraction datasets and methods, as well as the cro we define our relation label set on. in section 3, we describe our methodology for rq1 and rq2. finally, in section 4, we discuss two strategies to address rq3: manual evaluation by clinician review after pruning low confidence triples and prediction on a proxy clinical diagnostic reasoning task. ## evaluation from rq1 and rq2, we have a set of new clinical reasoning triples, grounded in snomed-ct entities and the cro relation set. however, without existing labeled reasoning relations, we have no way to use conventional confusion matrix-based evaluation measures. therefore, to tackle rq3, we propose two evaluation approaches. filtering and evaluation by clinicians as a first step, we plan to evaluate our zero-shot re system on biorel @xcite , a large distantlysupervised re dataset for the biomedical domain, carefully selecting train/test splits to model the zero-shot setting. while this provides a comparison to baselines, the noisy nature of distant supervision and lack of external validation of the training data by the authors may obscure the accuracy of the model. furthermore, this evaluation scheme doesn't measure our final goal of contributing new facts to snomed-ct. to that end, we will first calibrate our extraction model and filter out any low confidence triples. this reduces potentially noisy triples and allocates clinical resources to the most promising triples. then, we randomly sample triples from the model and have several clinicians determine the proportion of accurate predicted relations, measuring inter-rater reliability. ## societal impact extraction of triples using the relation set described in this proposal and their alignment with an existing clinical kg has the potential to significantly improve automated diagnostic reasoning. for instance, a kg-augmented system may be able to remind the physician to order labs based on probable diagnoses or extract disease-specific, relevant past medical history from patient records in realtime. we can also expect improvement in conventional nlp tasks such as reading comprehension of progress notes and reports (e.g. radiology summaries) and clinical knowledge question answering (qa) involving multi-hop reasoning. benchmarks to evaluate such tasks exist, like mmlu-professional medicine @xcite and medqa @xcite , both of which draw qa pairs from medical licensure exams. however, such benchmarks are abstractions that do not fully align with complex real-world use cases. to better model the challenges of clinical reasoning, we suggested the particular task of differential diagnosis prediction, which involves incorporating additional clinical data modalities. however, additional considerations may be necessary for evaluating kg use in real-world applications, such as modeling temporality. a task involving prediction of changing disease states over time may focus on the longitudinal nature of diagnostic reasoning. a portion of this work will involve continuing to define tasks that consider the challenges of real-world clinical reasoning use cases. ## summary in this thesis proposal, we suggest methods to address the problem of missing reasoning relations in clinical knowledge graphs. we select a subset of relations from a clinical reasoning ontology and extract relations from two data sources: a pointof-care reference for ed physicians and a newly created consumer health resource corpus. we plan to train a t5-based el model to link entities and develop a zero-shot re method to extract relations. finally, we discuss methods of evaluation in the real world context of zero-shot relation extraction using filtered expert review and a proxy diagnostic reasoning prediction task. our work should provide a case study for the complex task of introducing new types of knowledge into an existing structured knowledge base.