Bootstrapping a Conversational Guide for Colonoscopy Prep (2023)

Abstract: Creating conversational systems for niche domains is a challenging task, further exacerbated by a lack of quality datasets. We explore the construction of safer conversational systems for guiding patients in preparing for colonoscopies. This has required a data generation pipeline to generate a minimum viable dataset to bootstrap a semantic parser, augmented by automatic paraphrasing. Our study suggests large language models (e.g., GPT-3.5 and GPT-4) are a viable alternative to crowd sourced paraphrasing, but conversational systems that rely upon language models’ ability to do temporal reasoning struggle to provide accurate responses. A neural-symbolic system that performs temporal reasoning on an intermediate representation of user queries shows promising results compared to an end-to-end dialogue system, improving the number of correct responses while vastly reducing the number of incorrect or misleading ones.

## introduction colorectal cancer is the second leading cause of cancer-related deaths worldwide. colonoscopy is a safe and effective strategy to screen asymptomatic individuals for precursors of colorectal cancer, but it requires a precisely timed multi-day, multi-step procedure to clear the colon. in today's standard practice, patients are given information sheets to help them prepare for the procedure, which instruct them to follow a low-fiber diet for several days prior to the procedure (among other restrictions) and to drink a preparatory mix that cleanses the colon. unfortunately, these information sheets are frequently ineffective, resulting in rescheduled procedures with large economic, health-related and social costs. in this paper, we report on our initial steps to develop a conversational assistant to improve the ease of following colonoscopy preparation instructions. to avoid information overload, the assistant is designed to coach patients through the process (known as "prep"), reminding patients when it is time to carry out each step in the instructions and allowing them to ask questions at any time about the procedure and the diet changes they need to make at different stages of the preparatory period. additionally, the assistant will escalate questions to health-care providers when necessary to answer complex questions or reschedule. existing efforts to make it easier to follow colonoscopy prep instructions give strong evidence that our approach can greatly enhance patient success. engaging patients with automatic text reminders greatly improved colonoscopy prep adherence (90% vs. 62%) when patients were invited to ask follow up questions with health-care providers @xcite , but a larger scale trial where patients were not invited to reply to the text messages (for lack of personnel) found no improvements over the control group @xcite . the capacity to answer questions-which we seek to automate for the first time-appears to have been the crucial difference @xcite . embodied conversational agents (eca) from the northeastern relational agents lab have been developed for a variety of health-care communication scenarios over many years. in particular, @xcite develop an eca for counseling patients on their options for anesthesia prior to surgery, but the system cannot answer specific questions patients ask in their own words. with no existing data in this domain, we seek to take advantage of pretrained and large language models (plms/llms) to develop our system in a data-efficient way while robustly avoiding unsafe behavior. recent years have witnessed enormous progress on a wide range of nlp tasks, including conversational ai ones, thanks to engineering advances in training large scale, transformer-based neural language models @xcite @xcite @xcite . however, their deployment for practical tasks has been hindered by concerns about safety, such as the propensity of these models to regurgitate toxic language or hallucinate fake news @xcite @xcite . in healthcare settings, these concerns are especially problematic, as with insufficient controls plms could give harmful or even deadly advice @xcite . to address these safety concerns, we have designed a neuro-symbolic system that uses plms for contextual natural language understanding (nlu) together with a rule-based dialogue manager and knowledge base. to bootstrap the system, we have used state machines to create simulated dialogues @xcite together with llms for paraphrasing, rather than crowdworkers as in the overnight method @xcite ; further enhancement using wizard-of-oz @xcite methods is left to future work. ## conclusions and future work our initial development of a neuro-symbolic conversational guide for colonoscopy prep demonstrated that automatic paraphrasing of simulated conversations using gpt models can be successfully used to generate a diverse dataset for drawing meaningful insights into model behavior. we found that gpt and bart language models struggle with temporal reasoning; thus systems that rely upon explicit nlu and temporal reasoning are better suited for answering critical, time-sensitive questions. further, we found few incorrect responses generated from our system under novel and outof-scope situations, but misleading ones remain a challenging concern. in future work, we plan to enhance and extend the system after collecting wizard-of-oz data with patients. we expect these system improvements to greatly reduce the prevalence of nonresponsive answers when patients use more contextually varied language, as in our current challenge set experiments. we will also re-evaluate the prevalence of misleading responses and consider implementing steps to filter out such responses. we also plan to experiment with making the system more proactive by quizzing patients on their understanding of the instructions, in order to investigate whether this yields improved understanding leading to improved adherence to the prep protocol.