Conversational Agent for Daily Living Assessment Coaching Demo (2021)

Abstract: Conversational Agent for Daily Living Assessment Coaching (CADLAC) is a multi-modal conversational agent system designed to impersonate “individuals” with various levels of ability in activities of daily living (ADLs: e.g., dressing, bathing, mobility, etc.) for use in training professional assessors how to conduct interviews to determine one’s level of functioning. The system is implemented on the MindMeld platform for conversational AI and features a Bidirectional Long Short-Term Memory topic tracker that allows the agent to navigate conversations spanning 18 different ADL domains, a dialogue manager that interfaces with a database of over 10,000 historical ADL assessments, a rule-based Natural Language Generation (NLG) module, and a pre-trained open-domain conversational sub-agent (based on GPT-2) for handling conversation turns outside of the 18 ADL domains. CADLAC is delivered via state-of-the-art web frameworks to handle multiple conversations and users simultaneously and is enabled with voice interface. The paper includes a description of the system design and evaluation of individual components followed by a brief discussion of current limitations and next steps.

## introduction a person's ability to function independently in everyday life depends on multiple factors including, but not limited to, intact physical and mental capacity. in the united states, significant public resources are dedicated to providing assistance to those in need. a key aspect of assistance programs is to provide ongoing assessment of individuals * equal contribution. to determine their level of functioning (e.g. independent, needs supervision, needs physical assistance, or dependent) and their specific needs in order to provide assistance appropriately. these assessments are conducted by certified assessors specifically trained for this purpose. a challenge in the assessment process is to ensure consistency across large numbers of assessors with various degrees of experience and interview skills and to prepare novice assessors for the variety of interactions they will experience in the field. the conversational agent for daily living assessment coaching (cadlac) is designed to coach certified assessors to conduct their assessment interviews in a natural conversational style that simulates real interactions. previously, dialogue systems similar to cadlac have been developed @xcite @xcite . these systems simulate "virtual patients", which are used in healthcare education. cadlac is tailored to support novel application domains of function and disability. an example of the interaction with the conversational agent is shown in figure 1 . the interface and a video highlighting the system can be found here 1 2 . ## data we used two sources of data in order to inform cadlac system design, train machine learning models, and to develop a database to support rulebased approaches used by the system. one source of data consisted of a survey that was administered to certified assessors, and the other consisted of anonymized historical assessment data shared by the minnesota department of human services (dhs). ## system design cadlac is implemented on the mindmeld platform for conversational ai applications @xcite that relies on a commonly used modular dialogue system design consisting broadly of natural language understanding (nlu), natural language generation (nlg) and a dialogue state tracker/manager (dm) components. these components of cadlac prototype have been developed using a hybrid machine learning and rule-based approaches. the prototype is currently deployed via a web service written in python with the modern asynchronous web responder framework. this web service is responsible for accepting requests from a user-facing web client, managing user sessions, and passing conversation objects into the dialogue parser. the web-based client supports text-only, voice-only or hybrid modalities. this demonstration will focus on showing the natural dialogue between human users and cadlac aimed at assessing the level of functioning of the "individual" impersonated by cadlac and the feedback provided to the users regarding their assessments. the system architecture is shown in figure 2 . ## natural language generation we use a rule-based approach in which we first look up a field in the knowledge-base of historical assessments that corresponds to the identified topic and intent for a specific synthetic profile (e.g., challenges[intent] with dressing @xcite ). information contained in historical assessments is underspecified and is not usable as a natural language response. for example, it may contain a note "behavioral issues" for challenges with dressing. we manually annotated a subset of over 100 assessments, where the annotators were instructed to become familiar with the person's level of functioning in various domains and use that knowledge to convert the historical notes to a format that would sound more natural yet still consistent with the synthetic profile (e.g., "behavioral issues" note for a 5 year old child's assessment would be converted to "he can't dress by himself because he throws a tantrum each time he has to change clothes.") the current prototype of cadlac's dialogue manager queries the knowledge base for these manually converted responses and returns a response that most closely matches the named entities mentioned in the user's question. if no natural language response is found, cadlac generates a generic response randomly chosen from a set of responses consistent with the synthetic profile (e.g., for a profile of a person who requires intermittent physical assistance with dressing, the response may be "i need someone to help me with this"). we are currently experimenting with transformer neural models used in machine translation in order to determine if they can "learn" the mapping between the original historical assessment notes and the natural language responses; however, the current demo does not include these models yet. ## feedback the feedback to users being trained to perform assessments is provided via a visual interface designed to compare users' assessments to those stored in synthetic profiles as illustrated in figure 5 . ## voice services in order to enable voice input-output capabilities in cadlac we implemented a automatic speech recognition (asr) and a text-to-speech (tts) web services. both services are implemented using pytorch. voice activity is streamed from the web client to the web server in real time using an implementation of webrtc peer connections. the webrtc protocols are available in most modern browsers, and include hooks to access media devices, standards for establishing peer connections, and asynchronous data channels. the implementation of webrtc that was used for the python web server was aiortc. after voice data arrive at the server they are passed to the asr service, which transcribes english words from the speech utterance. these words take the place of the text from the chat interface for the rest of the conversational turn. ## limitations and future steps one of the limitations of the current implementation of cadlac is that it does not currently learn from user input. one of the next key steps in further development of this system is to implement active learning components for domain and intent classification, asr, and other supervised components of the system. we are also currently developing a formal evaluation of the usability of this system with human end-users. specifically, we plan to use metrics of sensibility and specificity for each system response as proposed by @xcite in addition to overall subjective measures of dialogue success, conversation naturalness, and intelligibility of responses. we also plan to evaluate the system for any potential bias in responses generated by the system and develop ways of un-biasing the system via hybrid rule-based and data-driven approaches @xcite .