The AI Doctor Is In: A Survey of Task-Oriented Dialogue Systems for Healthcare Applications (2022)

Abstract: Task-oriented dialogue systems are increasingly prevalent in healthcare settings, and have been characterized by a diverse range of architectures and objectives. Although these systems have been surveyed in the medical community from a non-technical perspective, a systematic review from a rigorous computational perspective has to date remained noticeably absent. As a result, many important implementation details of healthcare-oriented dialogue systems remain limited or underspecified, slowing the pace of innovation in this area. To fill this gap, we investigated an initial pool of 4070 papers from well-known computer science, natural language processing, and artificial intelligence venues, identifying 70 papers discussing the system-level implementation of task-oriented dialogue systems for healthcare applications. We conducted a comprehensive technical review of these papers, and present our key findings including identified gaps and corresponding recommendations.

## introduction dialogue systems foot_0 have a daily presence in many individuals' lives, acting as virtual assistants (hoy, 2018), customer service agents @xcite , or even companions @xcite . while some systems are designed to conduct unstructured conversations in open domains (chatbots), others (task-oriented dialogue systems) help users to complete tasks in a specific domain @xcite . task-oriented dialogue systems can potentially play an important role in health and medical care @xcite , and they have been adopted by growing numbers of patients, caregivers, and clinicians @xcite . nonetheless, there remains a translational gap @xcite between cuttingedge, foundational work in dialogue systems and prototypical or deployed dialogue agents in healthcare settings. this limits the proliferation of scientific progress to real-world systems, constraining the potential benefits of fundamental research. we move towards closing this gap by conducting a comprehensive, scientifically rigorous analysis of task-oriented healthcare dialogue systems. our underlying objectives are to (a) explore how these systems have been employed to date, and (b) map out their characteristics, shortcomings, and subsequent opportunities for follow-up work. importantly, we seek to address the limitations of prior systematic reviews by extensively investigating the included systems from a computational perspective. our primary contributions are as follows: in the long term, we hope that the gaps and opportunities identified in this survey can stimulate more rapid advances in the design of task-oriented healthcare dialogue systems. we also hope that the survey provides a useful starting point and synthesis of prior work for nlp researchers and practi-tioners entering this critical yet surprisingly understudied application domain. ## related work dialogue systems in healthcare have been the focus of several recent surveys conducted by the medical and clinical communities @xcite @xcite . these surveys have investigated the real-world utilization of deployed systems, rather than examining their design and implementation from a technical perspective. in contrast, studies examining these systems through the lens of ai and nlp research and practice have been limited. @xcite and @xcite presented surveys of recent advances in general-domain task-oriented dialogue systems. although they provide an excellent holistic portrait of the subfield, they do not delve into aspects of particular interest in healthcare settings (e.g., system objectives doubling as clinical goals), limiting their usefulness for this audience. @xcite , @xcite , and @xcite conducted systematic reviews of dialogue systems deployed in mental health @xcite or general healthcare @xcite settings. @xcite examined 10 articles, and @xcite and @xcite examined 17 and 46 articles, respectively. all surveys were written for a medical audience and focused on healthcare issues and impact, covering few articles from ai, nlp, or general computer science venues. @xcite and tudor car et al. (2020) recently reviewed 40 and 47 articles, respectively, covering conversational agents in the healthcare domain. these two surveys are the closest to ours, but differ in important ways. first, our focus is on a specific class of conversational agents: task-oriented dialogue systems. the surveys by @xcite used a wider search breading their ability to provide extensive technical depth. we also reviewed more papers (70 articles), which were then screened using a more thorough taxonomy as part of the analysis. some aspects that we considered that differ from these prior surveys include the overall dialogue system architecture, the dialogue management architecture, the system evaluation methods, and the dataset(s) used when developing and/or evaluating the system. ## search criteria and screening we designed search criteria in concert with our goal of filling a translational information gap between fundamental dialogue systems research and applied systems in the healthcare domain. to do so, we retrieved articles from well-respected computer science, ai, and nlp databases and screened them for focus on task-oriented dialogue systems designed for healthcare settings. our target databases were: although a narrower scope-e.g., developing improved methods for slot-filling-is common when publishing in the dialogue systems community, these papers tend to place more emphasis on technical design irrespective of application context, offering less coverage of the system-level characteristics that are the target of this survey. we followed four steps in our screening process. first (initial search), we applied a predefined search query to the databases to populate our initial list of papers. to generate the query, we used the keywords "task-oriented," "dialogue system," "conversational agent," "health," and "healthcare," and synonyms and abbreviations of these keywords. we shortlisted papers using these keywords individually as well as in combination with one another. next (title screening), we performed a preliminary screening through the initial list of papers by reading the titles, keeping those that satisfied the inclusion criteria. then (abstract screening), we went through the list of papers remaining after the title screening and read the abstracts, keeping those that satisfied the inclusion criteria. lastly (final screening), we read the body of the papers remaining after the abstract screening and kept those that satisfied the inclusion criteria. these funnel filtering processes were conducted by a computer science graduate student (a fluent l2 english speaker) using predefined search and screening guidelines. questions or uncertainties regarding a paper's compliance with inclusion criteria were forwarded along to the senior project lead (a computer science professor and fluent l1 english speaker with expertise in nlp) and final consensus was reached via discussion among the two parties. we detail the number of papers remaining after each screening step in in total, 70 papers (21 from acm, 31 from ieee, 16 from acl, and 2 from aaai 7 ) satisfied the inclusion criteria. we survey papers meeting our inclusion criteria according to a wide range of parameters, and present our findings in the following 7 papers about task-oriented dialogue systems published at aaai often focus on one specific component of the system from a technical perspective, rather than proposing a conversational agent as a whole. therefore, only two papers from the aaai digital library satisfied the inclusion criteria. subsections, grouped into thematic categories: ontology ( §4), system architecture ( §5), system design ( §6), dataset ( §7), and system evaluation ( §8). ## ontology we map each paper to its domain of research ( §4.1), system objective ( §4.2), target audience ( §4.3), and language ( §4.4), and present our findings. ## system architecture we investigate both the general architecture of the system ( §5.1), and if applicable, the dialogue mansystem architecture # papers pipeline 58 end-to-end 2 not specified 10 ## dataset data is crucial for effective system development @xcite , but many datasets for training dialogue systems are smaller than those used for other nlp tasks @xcite . this is even more pronounced in the healthcare domain, in part due to the risk of data misuse by others or the lack of data sharing incentives @xcite . ## system evaluation we examined the means through which systems were evaluated both qualitatively and quantitatively @xcite . we defined human evaluation, often implemented in prior work through questionnaires @xcite @xcite or direct feedback from realworld users @xcite , as an evaluation that relies on subjective, first-hand, human user experience. in contrast, automated evaluation provides an objective, quantitative measurement of one or more dimensions of the system from a mathematical perspective @xcite . some metrics used for automated evaluation of the reviewed systems include measures of task performance (ali et al., 2020) and completion rates @xcite , response correctness (rosruen and samanchuen, 2018), and response time @xcite . in ## discussion we identify common limitations across many surveyed systems, accompanied by recommendations for addressing them in future work. ## conclusion in this work, we conducted a systematic technical survey of task-oriented dialogue systems used for health-related purposes, providing much-needed analyses from a computational perspective and narrowing the translational gap between basic and applied dialogue systems research. we comprehensively searched through 4070 papers in computer science, nlp, and ai databases, finding 70 papers that satisfied our inclusion criteria. we analyzed these papers based on numerous technical factors including the domain of research, system objective, target audience, language, system architecture, system design, training dataset, and evaluation methods. following this, we identified and summarized gaps in this existing body of work, including an incomplete exploration of system design, replicability concerns, potential ethical and privacy issues, room for increased language diversity, and minimal investigation of usability or user engagement. finally, we presented evidence-based recommendations stemming from our findings as a launching point for future work. it is our hope that interested researchers find the information provided in this survey to be a unique and helpful resource for developing task-oriented dialogue systems for healthcare applications.