In 2007, IBM Research took on the grand challenge of building a computer system that can perform well enough on opendomain question answering to compete with champions at the game of Jeopardy! In 2011, the open-domain question answering system dubbed Watson beat the two highest ranked players in a two-game Jeopardy! match. But, to what degree can the QA technology underlying Watson, called DeepQA, which was tuned for answering Jeopardy! questions, succeed in a dramatically different and extremely specialized domain such as medicine? This paper describes the steps needed to adapt and improve performance in this as well as other domains. In addition, whereas Jeopardy! allows only -question in, single answer out‖ with no explanation, we elaborate upon a vision for an evidence-based clinical decision support system, based on the DeepQA technology, that affords exploration of a broad range of hypotheses and their associated evidence, as well as uncovers missing information that can be used in mixed-initiative dialog. Jeopardy! is a quiz show that pits three contestants against each other testing their ability to understand and answer rich natural language questions very quickly. These questions often contain complex language, ambiguities, puns, and other opaque references. For any given question, the contestants compete for the first chance to answer via a handheld buzzer. To be successful at Jeopardy!, players must retain enormous amounts of information, must have strong language skills, must be able to understand precisely what is being asked, and must accurately determine the likelihood they know the right answer. Confidence in the answer is critical, because the first player to buzz in gets the opportunity to answer the question; however, if the player answers incorrectly, the player loses the dollar value associated with the clue. The challenges in the Jeopardy! task are: 1) Questions come from a broad domain: Jeopardy! asks questions about hundreds of thousands of things, using rich and varied natural language expressions. 2) Players must answer questions with high precision and with accurate confidence: On average, champion players must be able to correctly answer more than 85% of the questions they buzz in for and they must be confident enough to buzz in for at least 70% percent of them. 3) Answering must be very fast: Winning players must quickly determine an accurate confidence in a correct answer and buzz in quickly enough to beat their competitors consistently to the buzz. Over a four year period, the team at IBM developed the Watson system that competed on Jeopardy! and the underlying DeepQA question answering technology [15] . Watson played many games of Jeopardy! against celebrated Jeopardy! champions and, in games televised in February 2011, won against the greatest players of all time, Ken Jennings and Brad Rutter. But, DeepQA has application well beyond Jeopardy!. Contrary to some popular misconceptions, DeepQA does not map the question to a database of questions and simply look up the answer. DeepQA is a software architecture for analyzing natural language content in both questions and knowledge sources. DeepQA discovers and evaluates potential answers and gathers and scores evidence for those answers in both unstructured sources, such as natural language documents, and structured sources, such as relational databases and knowledge bases. Figure 1 presents a high-level view of DeepQA architecture [15] . DeepQA has a massively parallel, component-based pipeline architecture [16] , which uses an extensible set of structured and unstructured content sources as well as broad range of pluggable search and scoring components that allow integration of many different analytic techniques. Machine-learning is used to learn the weights for combining scores from different scorers. Each answer is linked to its supporting evidence. DeepQA is informed by extensive research in question answering systems, e.g., [7, 23, 28] . These systems analyze an input question and generate and evaluate candidate answers using a variety of techniques. DeepQA analyzes an input question to determine precisely what it is asking for and generates many possible candidate answers through a broad search of large volumes of content. For each of these candidate answers, a hypothesis is formed based on considering the candidate in the context of the original question and topic. For each hypothesis, DeepQA spawns an independent thread that attempts to prove it. DeepQA searches its content sources for evidence that supports or refutes each hypothesis. For each evidence-hypothesis pair, DeepQA applies hundreds of algorithms that dissect and analyze the evidence along different dimensions of evidence such as type classification, time, geography, popularity, passage support, source reliability, and semantic relatedness. This analysis produces hundreds of features. These features are then combined based on their learned potential for predicting the right answer. The final result of this process is a ranked list of candidate answers, each with a confidence score indicating the degree to which the answer is believed correct, along with links back to the evidence. Figure 2 shows the dimensions of evidence for the Jeopardy! clue -Chile shares its longest land border with this country.‖ Each dimension combines the features produced by many algorithms. Each algorithm uses different resources and algorithmic techniques, each with different precision-recall tradeoffs. To form a consumable set of evidence dimensions, features are grouped according to taxonomy of evidence types (e.g., location and popularity as shown in fig 2) we defined. The Figure 2 illustrates a comparative evidence profile highlighting some of the dimensions defined for Watson. Evidence profiles were used by developers for debugging, and we imagine they will be useful for end users in many applications to understand and explore evidence associated with a candidate answer. DeepQA's approach to Jeopardy! and the success of Watson suggest a powerful new architecture for reasoning over unstructured content. Traditional expert systems use forward reasoning that follows rules from data to conclusions or backward reasoning that follows rules from conclusions to data. To build these systems hand-crafted IF-THEN rules for every bit of domain knowledge are manually developed and maintained by skilled engineers or domain experts. An example of a rule taken from the Mycin system is: IF: The stain of the organism is grampos and the morphology of the organism is coccus and the growth conformation of the organism is chains, THEN: There is suggestive evidence (.7) that the identity of the organism is streptococcus. As a result, expert systems are costly and difficult to develop and maintain as new knowledge is discovered. Expert systems are also brittle, because the underlying reasoning engine requires a perfect match between the input data and the existing rule forms. Additionally, not all rule forms can be known in advance for all the forms that input data may take, which further contributes to their brittleness. In contrast to traditional Expert Systems, DeepQA exploits natural language processing (NLP) and a variety of search techniques to analyze unstructured information to generate likely candidate answers in hypothesis generation (analogous to forward chaining). In evidence collection and scoring (analogous to backward chaining), DeepQA also uses NLP and search over unstructured information to find evidence for ranking and scoring answers based on natural language content. DeepQA's direct use of readily available knowledge in natural language content makes it more flexible, maintainable, and scalable as well as cost efficient in considering vast amounts of information and staying current with the latest content. What this approach lacks in hand-crafted precision using specific rules, it gains in breadth and flexibility. In a clinical setting, for example, it can be used to develop a diagnostic support tool that uses the context of an input case-a rich set of observations about a patient's medical condition-and generates a ranked list of diagnoses (differential diagnosis) with associated confidences based on searching and analyzing evidence from large volumes of content. Physicians and other care providers may evaluate these diagnoses along many different dimensions of evidence that DeepQA has extracted from a patient's electronic medical record (EMR) and other related content sources. For medicine, the dimensions of evidence may include symptoms, findings, patient history, family history, demographics, current medications, and many others. Each diagnosis in the differential diagnosis includes links back to the original evidence used by DeepQA to produce its confidence scores and supports the adoption of evidence-based medicine (EBM) -which aims to apply the best available evidence gained from the scientific method to clinical decision making.‖ [14] When the answers provided by DeepQA are diagnoses of the underlying causes of problems, as in the case of medical diagnosis, then the DeepQA architecture can be thought of as implementing a form of abductive reasoning [25] . As a simple example of abduction, suppose that some piece of structured or unstructured knowledge in the system represents that patients with disease D have symptom S. Then, if the input to the system is that the patient has symptom S, the system will generate the hypothesis that the patient has disease D. The system will then look for evidence to support or refute this hypothesis. For a more complex example of abduction, consider that the system has numerous pieces of knowledge about diseases and their symptoms. Then, given the input that the patient has some set of symptoms, the system's task is to find the best explanation of those symptoms in terms of one or more diseases. The DeepQA architecture does this by generating hypotheses and then, in parallel, evaluating how much evidence supports each hypothesis. In effect, DeepQA is a massive abduction machine. The use of abduction for medical diagnosis has a long history in the field of artificial intelligence. Pople proposed applying abduction to medical diagnosis and provided algorithms for computing explanations of data (like symptoms) in the context of a collection of axioms (medical knowledge) [27] . Goebel, Furukawa, and Poole presented algorithms for generating diagnoses given medical axioms of the form disease ‫כ‬ symptom [19] . They further proposed the use of probabilistic logic for preferring one diagnosis over another. They discuss the addition of a probability to disease ‫כ‬ symptom axioms (sensitivity). Console, Portinale, and Dupré presented an extensive Figure 3 shows a proposed set of clinical dimensions of evidence for a patient from Connecticut with a chief complaint of eye pain and inflammation, blurred vision, headache, fever, and circular rash. Each dimension of evidence-findings, demographics, symptoms, and family history-aggregates individual pieces of evidence. A healthcare provider can observe the contribution of each dimension of evidence, as well as drill down into a particular dimension to see the contributing pieces of evidence and provenance information. Accessing this information would help refine their thinking in an evidence-based manner. The ability to explore alternative hypothesis (diagnoses), along with the confidence values and associated supporting evidence is a key differentiating feature of DeepQA compared to previous systems. This general view of DeepQA, as an architecture for building lower cost, more flexible expert-system technology over readily available knowledge, led us to consider applications beyond Jeopardy! and specifically to healthcare. A special edition of the IBM Journal of Research and Development is currently in preparation and devoted to providing a detailed technical description of the Watson system and the underlying DeepQA architecture [10] . In this paper, we motivate the application of DeepQA to healthcare, specifically in clinical decision support. We discuss the first steps we took to adapt DeepQA to the medical domain and how evidence profiles provide a powerful foundation for communicating with healthcare providers. In the following section, we motivate our work by discussing the problems clinicians face in diagnosis, and review past and current clinical decision support systems along with their strengths and weaknesses. This discussion is followed by our vision of how a system based on DeepQA can become a novel evidence-based decision support tool and, finally we report results on the first steps we have taken toward this goal. Improving diagnostic accuracy and speed can directly improve quality of care in patients as well as reduce the overall cost incurred in this process by healthcare systems. Schiff [32] reported diagnostic errors far outnumbering other medical errors by two to four times. Elstein [12] estimated a diagnostic error rate of about 15%, which is in line with findings in a number of autopsy studies [21, 33] . Singh and Graber [36] assert that -diagnostic errors are the single largest contributor to ambulatory malpractice claims (40% in some studies) and cost approximately $300,000 per claim on average.‖ Results published from these papers and others highlight the frequency and consequence of diagnostic error in healthcare systems today and motivate the need for approaches that can reduce them. A recent study by Graber [20] reviews literature related to the causes of diagnostic error and discusses results obtained in a study of 100 -error cases.‖ They report that 65% of these cases had system-related causes and 75% had cognitive-related causes. System errors were -most often related to policies and procedures, inefficient processes, and difficulty with teamwork and communication, especially communication of test results.‖ Graber [20] reported that cognitive errors were primarily due to -faulty synthesis or flawed processing of the available information.‖ The predominant cause of cognitive error was premature closure, defined as -the failure to continue considering reasonable alternatives after an initial diagnosis was reached.‖ Graber additionally identified four more major contributors to the cognitive errors: faulty context generation, misjudging the salience of a finding, faulty detection or perception, and failed use of heuristics. Graber [20] concluded that the cognitive errors -overwhelmingly reflect inappropriate cognitive processing and/or poor skills in monitoring one's own cognitive processes (metacognition)‖ and suggested 1) -compiling a complete differential diagnosis to combat the tendency to premature closure,‖ 2) using the -crystal ball experience: The clinician would be told to assume that his or her working diagnosis is incorrect, and asked, What alternatives should be considered?‖ and 3) augmenting -a clinician's inherent metacognitive skills by using expert systems.‖ In a recent paper, Singh and Graber [36] also noted that -clinicians continue to miss diagnostic information … one likely contributing factor is the overwhelming volume of alerts, reminders, and other diagnostic information in the Electronic Health Record (EHR). Better techniques to summarize and present data are needed to enable clinicians to find the proverbial ‗needle in the haystack' in the midst of voluminous data.‖ To compound these problems, published medical information is growing and changing extremely quickly, making the information difficult for the healthcare professional to read, process, and remember. Many emergency medical or critical situations require very rapid assessment and correct and timely action. These challenges require mentally weighing many variables and exploring alternatives rapidly, which contributes to the cognitive overload inherent in many aspects of this practice. Our vision for DeepQA is motivated by the problems and suggested solution outlined above. Our approach is to provide a decision support tool that will help the physician overcome the cognitive challenges described above by providing 1) the automatic extraction and presentation of relevant information from the EMR, 2) an extensive differential diagnosis with associated confidences and evidence profiles, and tooling to explore supporting evidence, and 3) a mixed initiative dialogue to suggest exploration of missing information and inform decisions based on evidence gathered from vast amounts of structured and unstructured information such as medical texts, encyclopedias, journals, and guidelines. We review some past and current medical diagnostic systems below as a backdrop for a discussion of system related issues and a comparison with our approach. Diagnosis systems can be classified into systems that use structured knowledge, systems that use unstructured knowledge, and systems that use clinical decision formulas, rules, trees, or algorithms. The earliest diagnosis systems used structured knowledge or classical, manually constructed knowledge bases. The Internist-I system developed in the 1970s used disease-finding relations and disease-disease relations, with associated numbers such as sensitivity-the fraction of patients with a disease who have a finding [24] . The MYCIN system for diagnosing infectious diseases, also developed in the 1970s, used structured knowledge in the form of production rules stating that, if certain facts are true, then one can conclude certain other facts with a given certainty factor [4] . DXplain, developed starting in the 1980s, used structured knowledge similar to that of Internist-I, but added a hierarchical lexicon of findings [2] . The Iliad system developed in the 1990s added more sophisticated probabilistic reasoning. Each disease has an associated a priori probability of the disease (in the population for which Iliad was designed) and list of findings along with the fraction of patients with the disease who have the finding (sensitivity) and the fraction of patients without the disease who have the finding (1 -specificity) [40] . In 2000, diagnosis systems using unstructured knowledge started to appear. These systems use some structuring of knowledge as well. For example, entities such as findings and disorders may be tagged in documents to facilitate retrieval. ISABEL uses Autonomy information retrieval software and a database of medical textbooks to retrieve appropriate diagnoses given input findings [31] . Autonomy Auminence uses the Autonomy technology to retrieve diagnoses given findings and organizes the diagnoses by body system [1] . First CONSULT allows one to search a large collection of medical books, journals, and guidelines by chief complaints and age group to arrive at possible diagnoses [17]. PEPID DDX is a diagnosis generator based on PEPID's independent clinical content [26]. Clinical decision rules have been developed for a number of disorders, and computer systems have been developed to help practitioners and patients apply these rules. The CaseWalker system uses a four-item questionnaire to diagnose major depressive disorder [5] . The PKC Advisor provides guidance on 98 patient problems such as abdominal pain and vomiting [29] . The strengths of current diagnosis systems are that they can improve clinicians' diagnostic hypotheses [18] and can help clinicians avoid missing important diagnoses [30] . But, current diagnosis systems aren't widely used [3] for the following reasons: 1) They aren't integrated into the day-to-day operations of health organizations [8, 34] . A patient may be seen by many different healthcare workers, and patient data may be scattered across many different computer systems in both structured and unstructured form. 2) They are difficult to interact with [3, 34] . Entry of patient data is difficult, the list of diagnostic suggestions may be too long, and the reasoning behind diagnostic suggestions is not always transparent. 3) They aren't focused enough on next actions; they don't help the clinician figure out what to do to help the patient [34] . They are unable to ask the practitioner for missing information that would increase confidence in a diagnosis. 4) They aren't always based on the latest, high-quality medical evidence and are difficult to keep up-to-date [35] . Our goal with DeepQA is to address some of the weaknesses of prior approaches and to help healthcare professionals overcome the cognitive challenges they face in differential diagnosis, treatment, and other aspects of patient care outlined above. A key differentiating characteristic of DeepQA is its strength in using search and NLP techniques to process knowledge present in natural language content. These techniques can be used to extract relevant information from EMRs to provide the context for solving individual cases. The same techniques used by DeepQA for Jeopardy! may be adapted to generate diagnoses and treatment options and then collect evidence from huge volumes of information to support or refute those diagnoses and treatments. The ability to effectively process unstructured content found in medical resources and EMRs allows the practitioner to work with the most current knowledge available and reduces the burden associated with reading and synthesizing vast amounts of data stored in a patient record. It also helps ensure that the evidence provided in support of a set of possible solutions is readable and consumable by human users because the content is typically created by other experts in natural language rather than by knowledge engineers in formal rules. We started to envision how DeepQA can be used in interaction with healthcare professionals. Physicians we have interviewed all stress the need for ease of use in medical decision support systems, especially those that are used during a patient encounter. System input must be minimal and efficient, and information provided must be unobtrusive and relevant. Our goal is to minimize the input required, by automating the extraction of EMR information relevant to the current situation and providing information at a glance as new suggestions are obtained. Standing queries for diagnosis or treatment will be running as a background process, further minimizing the input required. A history of the practitioner's interaction with the system on a particular case will provide a context for future interactions. This interaction will support system-generated suggestions as well as give practitioners the opportunity to ask directed natural language medical questions to obtain additional information they seek and will help them overcome many of the cognitive challenges discussed above, such as premature closure and faulty context generation. Figure 4 illustrates a design for a user interface for clinical decision support. The left side presents information (labeled Factors) related to the patient's chief complaint, which would be automatically extracted from an EMR. Each factor is shown under its respective dimension of evidence, i.e., symptoms, findings, family history, and demographics. The top left side of Figure 4 shows the current differential diagnosis (only the top 3 of a long list are shown in this example) and associated confidence values for each candidate. A practitioner can select a particular candidate diagnosis-in this case Uveitis is selected-and explore the contribution of each dimension of evidence. A particular dimension (e.g. Symptoms) is selected revealing the contributing pieces of evidence as well as where they came from in the Sources tab area. The complete text is accessible via links. In addition, the Factors tab on the bottom right may be selected to explore factors that are present as well as missing from the current case as illustrated in Figure 5 . The ability to explore alternative hypotheses (diagnoses), along with the confidence values and associated evidence is another key differentiating feature of DeepQA. This ability to gather evidence surrounding a hypothesis can also be used to discover information that is missing from the current clinical context and can drive mixed-initiative dialog that help clinicians gather additional information and refine their thinking in an evidence based manner. We are beginning to explore this kind of information and interaction, and we believe it will be an essential feature of our clinical decision support system. Several challenges need to be addressed to apply DeepQA to clinical decision support. We divide them into the challenge of embedding the DeepQA capability into a clinical decision support system and the challenge of adapting the internal components of DeepQA to the medical domain. The decision support system must be able to extract relevant clinical information from EMR systems. We expect that certain portions of the clinical information such as admission notes, consults, clinical assessments, and discharge summaries will continue to be best expressed and communicated in natural language. Our challenge is to apply natural language and reasoning techniques to extract, relate, and structure this information along a timeline of medical encounters. Questions in the medical domain bring along a broader context that is described in the patient's medical history. Patient history comprises (1) a description of the chief complaint, (2) a history of the present illness, (3) a review of the major body systems, and (4) social and family history [13] . Using named entity and relation annotators, we need to extract key clinical concepts that form the context for decision support. These include signs, symptoms, findings, active and past diseases, current medications, allergies, demographics, family history and many others. The concepts need to be broad enough to capture the descriptive intent of the clinician. For example, rather than just extracting -heart murmur‖ as a finding, we should also extract the related modifiers as well: -heart murmur is harsh, systolic, diamond-shaped and increases in intensity with Valsalva.‖ Relations, for example, that indicate a specific family member had a particular disease, or that a symptom is mentioned in negation, need to be accurately captured from the language parse results. Laboratory test results need to be interpreted and evaluated for clinical significance. The extraction of this information from the patient's EMR provides context necessary for hypothesis generation and evaluation performed by DeepQA. Significant challenges also exist in developing the manner in which the output of DeepQA is presented to the healthcare professionals. A clinical decision support system must help a practitioner overcome cognitive errors discussed above by explaining how a certain hypothesis was generated and what factors were considered in its evaluation. As described in our approach, we must be able to decompose the confidence in a hypothesis into its constituent dimensions of evidence and compare them across multiple competing hypotheses so that practitioners can arrive at their own conclusions. A useful capability to improve the quality of decision making is to identify the missing information about the patient that detracts from the confidence for a hypothesis, as a whole or along a specific dimension. This missing information offers a set of questions back to the healthcare professional to explore and answer. Significant opportunities for research remain in evaluating the potential informativeness of such missing information from the large amounts of information that is not recorded in a patient's EMR. When comparing across competing hypotheses, the missing information can also be evaluated and ranked according to its discriminitiveness among these hypotheses. This feedback could focus additional data gathering activities, such as diagnostic tests, to those that are more likely to confirm or reject hypotheses, sharpening the differential diagnosis in the process. Significant areas of research remain within the natural language processing capability of DeepQA itself. We are beginning to address these challenges, starting from those within DeepQA's hypothesis generation and verification components. The following section describes our efforts to date and their impact on medical question answering performance.
REF A comparison of state of the art work in biomedical question answering systems is shown in Table 1 and can be summarized as: 1) QA systems employ question classification or LAT prediction, and most of the systems use pre-defined rules to determine a single LAT of a biomedical question.
Watson: Beyond Jeopardy!
Artif. Intell.
Artif. Intell.
Computer Science Mathematics