Demand for clinical decision support systems in medicine and self-diagnostic symptom checkers has substantially increased in recent years. Existing platforms rely on knowledge bases manually compiled through a labor-intensive process or automatically derived using simple pairwise statistics. This study explored an automated process to learn high quality knowledge bases linking diseases and symptoms directly from electronic medical records. Medical concepts were extracted from 273,174 deidentified patient records and maximum likelihood estimation of three probabilistic models was used to automatically construct knowledge graphs: logistic regression, naive Bayes classifier and a Bayesian network using noisy OR gates. A graph of disease-symptom relationships was elicited from the learned parameters and the constructed knowledge graphs were evaluated and validated, with permission, against Google's manually-constructed knowledge graph and against expert physician opinions. Our study shows that direct and automated construction of high quality health knowledge graphs from medical records using rudimentary concept extraction is feasible. The noisy OR model produces a high quality knowledge graph reaching precision of 0.85 for a recall of 0.6 in the clinical evaluation. Noisy OR significantly outperforms all tested models across evaluation frameworks (p < 0.01). Automated tools to support medical diagnostic reasoning are used by patients seeking information about their symptoms [1] [2] [3] [4] , as well as by clinicians when faced with a difficult case or to avoid prematurely focusing on a small number of potential diagnoses 5 . Considerable effort has been put into building diagnostic reasoning systems and encoding relevant information to drive their inference capabilities [6] [7] [8] [9] [10] [11] . These models showed significant success in improving didactic practices 12, 13 , assisting with diagnosis 6, 7, 9, 11, 14 , and at times even outperforming experienced doctors 15 . Historically, the models used by diagnostic reasoning systems were specified manually, requiring tremendous amounts of expert time and effort. For example, it was estimated that about fifteen person-years were spent building the Internist-1/QMR knowledge base for internal medicine 10 . However, the manual specification made these models extremely brittle and difficult to adapt to new diseases or clinical settings. Automatic compilation of a graph relating diseases to the symptoms that they cause has the potential to significantly speed up the development of such diagnosis tools. Moreover, such graphs would provide value in and of themselves. For example, given that general-purpose web-search engines are among the most commonly consulted sources for medical information 4, 16 , health panels such as those provided by Google using their health knowledge graph have a tremendous potential for impact [17] [18] [19] . Previous work considered the use of natural language processing to find relationships between diseases and symptoms from unstructured or semi-structured data. For example, IBM's WatsonPaths and the symptom checker Isabel made use of medical textbooks, journals, and trusted web content 8, 9 . However, another potential source of data, currently underutilized, is the electronic medical record (EMR), which has become increasingly prevalent in the United States and worldwide 20 .
REF Health knowledge graphs were extracted from the clinical records and maximum likelihood estimation of three models: logistic regression, naive Bayes, and noisy OR.
Learning a Health Knowledge Graph from Electronic Medical Records
Scientific Reports
Scientific Reports
Computer Science Medicine