A Dialogue System for Assessing Activities of Daily Living: Improving Consistency with Grounded Knowledge (2023)

Abstract: In healthcare, the ability to care for oneself is reflected in the “Activities of Daily Living (ADL),” which serve as a measure of functional ability (functioning). A lack of functioning may lead to poor living conditions requiring personal care and assistance. To accurately identify those in need of support, assistance programs continuously evaluate participants’ functioning across various domains. However, the assessment process may encounter consistency issues when multiple assessors with varying levels of expertise are involved. Novice assessors, in particular, may lack the necessary preparation for real-world interactions with participants. To address this issue, we developed a dialogue system that simulates interactions between assessors and individuals of varying functioning in a natural and reproducible way. The dialogue system consists of two major modules, one for natural language understanding (NLU) and one for natural language generation (NLG), respectively. In order to generate responses consistent with the underlying knowledge base, the dialogue system requires both an understanding of the user’s query and of biographical details of an individual being simulated. To fulfill this requirement, we experimented with query classification and generated responses based on those biographical details using some recently released InstructGPT-like models.

## introduction conversational ai is expanding beyond use in general applications like virtual assistants @xcite to use in specialized domains such as healthcare and finance where it can aid patients or customers in various scenarios. specifically, there is interest in applications of this technology for patient care and monitoring after hospital discharge @xcite . assessing functioning is crucial in clinical and non-clinical fields, such as nursing, physical and occupational therapy, geriatric medicine, neurology, rheumatology, disability, and human services. a person's ability to perform dayto-day activities independently depends on their cognitive, motor, and perceptual abilities, which are collectively referred to as activities of daily living (adl) @xcite . impairments in these abilities often require assistive devices, external supervision, assistance, or a longterm support plan. the minnesota department of human services (mndhs) provides significant public resources to assist individuals with impaired functioning based on their specific needs. certified assessors conduct face-to-face interviews with individuals to determine the level of support required to meet their needs, covering a wide range of areas related to adls. the goal of these assessments is to determine an individual's level of independence in performing adls. however, ensuring consistency across numerous assessors (e.g., 1,700 in the state of minnesota) and preparing novice assessors for diverse field interactions poses a challenge to the state's intake process. despite the availability of free corpora for training end-to-end neural models, most dialogue systems in healthcare are still rule-based @xcite . with the proliferation of neural models, there has been an increasing concern about the factual consistency of ai-powered applications. factual consistency with a knowledge source, explicated in prior work as knowledge-grounding or attributability @xcite , refers to the ability of a model to generate responses that are accurate and consistent with the information present in a verified knowledge base. knowledge grounding is particularly important in language models used for tasks that require accurate information, such as question-answering, dialogue systems, and chatbots @xcite @xcite . to tackle these challenges and facilitate the training of certified assessors in conducting adl as-sessments, we propose a coaching dialogue system presented in this paper. specifically, our contributions can be summarized as follows: 1. we created a novel dataset for developing and evaluating dialogue systems focused on adl assessments. 2. we compared several statistical models used for natural language understanding. 3. we experimented with several approaches to grounding language generation with large language models by using knowledge contained in a manually constructed knowledge base. ## related work relevant previous work has been conducted on dialogue systems developed for use in healthcare settings @xcite @xcite . these dialogue systems simulate a virtual standardized patient to deliver healthcare education from structured encounter data and rely on matching algorithms to extract scripted answers. in contrast, the dialogue system we have created generates responses that are dependent on various synthetic profile characteristics and allows off-topic conversations, thus offering a greater degree of variability while still remaining grounded in the knowledge contained in the synthetic profile. this allows the system to simulate several possible patients with different attributes, with the goal of giving assessors a chance to practice their interview skills with simulated patients of different functioning levels and communication styles. knowledge grounding using external knowledge graphs in combination with transformer models was previously explored @xcite @xcite . open domain conversational dataset and dialogue systems based on factual knowledge have also been developed @xcite . in this paper we focus on the fine-tuning of instructgpt-like models, and also on the combination of these fine-tuned models with a knowledge base of pre-written natural language facts using query classification and information retrieval via similarity matching. we previously published on earlier versions of this system, referred to as conversational agent for daily living assessment coaching(cadlac) @xcite . these earlier versions as well as the current version of the system were deployed as a demonstration with a web-based interface @xcite . ## conclusions and limitations in this paper, we present a comprehensive framework for measuring the quality of a dialogue system dedicated to activities of daily living assessments. we have created a new high-quality dataset of human-written questions and answers with corresponding profile information. we are currently working on expanding the dataset by adding more profiles and removing any factual inconsistencies resulting from human error. although more complex models showed better query classification performances, we need to consider the trade-offs between model size and generation time in the deployment environment to ensure a smooth user experience. we also identify areas where llm performance can be augmented by a knowledge base filled with human written natural language facts, and that this augmentation need not come at a penalty to sensibleness, specificity, or the realistic quality of conversation. general conclusions based on our initial work here may not be possible given the limited number of evaluators and small amount of evaluated dialogues, and this is a major limitation of our contribution. future work is needed to develop a more robust and replicatable evaluation framework, especially to perform evaluations of long and complex conversations like the type that assessors perform in the field. such an evaluation will need to include larger numbers of human raters to improve the statistical power of the surveys. recent automatic evaluations may also help improve development efforts, as a sufficiently powerful llm such as gpt-4 may be able to monitor the chatbot for regressions in its ability to speak fluently, sensibly or specifically. this assessment, known informally as the "vicuna assessment" @xcite , cannot give an evaluation of the chatbot's fit-for-purpose, but could be used to compare short conversations from several versions of the same chatbot. this could free up more human resources to evaluate the knowledgegroundedness and fit-for-purpose of future versions. in addition, given more computing budget and more time to engineer prompts, larger language models beyond llama 7b could be further studied or fine-tuned while experimenting with fine-tuning datasets and process. there are also many thresholds and parameters that could be further tested in the development of the knowledge-grounding system, wherein similarity measures inform the system's decision to answer using a generative model versus responding with language directly from the knowledge base.