Design Considerations for an NLP -Driven Empathy and Emotion Interface for Clinician Training via Telemedicine (2022)

Abstract: As digital social platforms and mobile technologies become more prevalent and robust, the use of Artificial Intelligence (AI) in facilitating human communication will grow. This, in turn, will encourage development of intuitive, adaptive, and effective empathic AI interfaces that better address the needs of socially and culturally diverse communities. In this paper, we present several design considerations of an intelligent digital interface intended to guide the clinicians toward more empathetic communication. This approach allows various communities of practice to investigate how AI, on one side, and human communication and healthcare needs, on the other, can contribute to each other’s development.

## introduction recent years brought both challenges and opportunities to interpersonal communication in all areas of life, especially healthcare. the covid-19 pandemic, for instance, took an enormous toll on people's mental health. effective empathic communication is now even more vital. in healthcare, and telemedicine (tm) in particular, expression of empathy is essential in building trust with patients. yet, physicians' empathic communication in tm encounters has remained largely unexplored and not measured. despite considerable research establishing the clinical efficacy of tm (e.g. in acute stroke care), there is limited research on how tm technology affects physicianpatient communication @xcite . research on how to decode human behaviors with respect to empathy expression, perception and action is still nascent @xcite @xcite @xcite @xcite . of all the components of professionalism, empathy may be the most challenging to communicate via tm given the physical separation of participants. ai systems with simple, intuitive, flexible and efficient emotionally-intelligent interfaces to support empathic provider-patient communication during digital visits are urgently needed. with its current developments, ai can help us understand how to implement empathy and compassion in effective patient-provider interactions and guide training for medical personnel. in healthcare, ai initiatives must also be multidisciplinary, using/developing a variety of core sets of requirements and expertise and engaging many participants, e.g. ai designers, developers, health care leadership, frontline clinical teams, ethicists, humanists, patients and caregivers. health care professional training programs should also incorporate core curricula that trains on using such ai tools appropriately @xcite . with this research, we aim to offer a solution to improve empathic patient-physician communication. specifically, part of a larger inter-disciplinary initiative, we propose to develop a digital interface that integrates with various tm platforms to monitor the emotional state of providers/patients and to guide/train them on how to improve their expression of empathic communication. we use state-ofthe-art multimodal natural language processing (nlp) built on cognitive science communication theories @xcite , operating as a plug-andplay across tm platforms for future scaling. our goals are to: (1) design, build, and test an intelligent digital interface that guides clinicians toward more empathetic communication; (2) develop a set of objective measures to assess the system's ability to positively impact clinicians @xcite (3) design a scalable plug and play architecture agnostic to tm platforms. beyond serving as a tool to improve empathic communication towards increased patient satisfaction, this project lays the groundwork for additional research in helping different professions work together effectively in the tm environment. we believe our research and investigation come at the right time. collaborative nlp + hci developments have been largely unexplored @xcite , yet critical for the next-generation ai-driven immersive environments, especially in healthcare. ## methodology our nlp-driven empathy system is a multitask multimodal (video, speech, text) machine learning (ml) framework to train a classifier to recognize empathetic language in patient-physician communication. it automatically labels dialogues with sentiment and emotions, recognizes different types of empathy (i.e., cognitive, affective, and prosocial behavior) @xcite at the utterance level, and computes an overall empathy score. throughout the dialogue, when the empathy score falls below a critical level, the system automatically recommends the top three most plausible empathetic response suggestions (predicated on the sentiment and emotion labels, and the dialogue history). 1 with this research, we propose an intelligent interface system design, then present various ways to evaluate it along a number of relevant dimensions. we assume an ideal nlp system that operates at the human-level (i.e., gold standard). data. in the first phase of the project, we test the interface design on a dataset of six recorded doctorpatient interaction videos (three empathetic and three non-empathetic) collected from a healthcare training initiative 2 (haglund et al., 2015). the dialogues are professionally designed simulations of five to seven minute interactions, where a doctor, breaking bad news, is expected to use layperson terms in a highly empathic language to console and guide the patient/family. the dataset was already analyzed and annotated for emotion and empathy content by trained third-party annotators (undergraduate psychology and social work students at the university of illinois trained in the spikes protocol @xcite ) using annotation guidelines consistent with established practices in nlp @xcite and socio-behavioral research on empathy @xcite . in this step, we transcribed the interactions, converted the audio into .wav format, single-channel recordings, normalized the intensity by -3db, using audacity (audacityteam, 2017), and annotated the audio files with praat @xcite . the annotators marked utterance boundaries and segmented them by speaker turns, topic changes, and major syntactic boundaries (i.e., sentence and clause breaks) as needed. they, then completed utterance-level empathy annotation on the identified utterances, labeled each dialogue with emotion and sentiment, and suggested a set of three plausible empathetic responses at every time-stamp in the non-empathetic dialogue in need of empathetic intervention (guided by the positive interaction). this dataset/setting was used in developing the user interface and will be used for its evaluation. ## intelligent empathic interface design given our task and data, we show the proposed intelligent interface with its five major functional regions in figure1. this is the doctor's view. foot_2 r1: top left shows the basic function icons: account settings; stats; interface modality selection @xcite . it also includes several standard icons to control the sound and video. r2: top center shows the account owner's info (r2a): picture; basic credentials. top right shows the other participant's (interlocutor) info (r2b). r3: in the center of the screen, there is the text dialogue. the default window is limited to twoturn history of the selected time-stamp; with option to see the entire raw/annotated transcript. r4 and r5, in the bottom half of the user interface, give the audio and video streams, respectively. r5 (empathy statistics) visually shows measured patient's distress and doctor's empathetic score throughout the dialogue interaction. the user can stop, replay, and select various timestamps, etc. for a given time-stamp, a pop-up window suggests more empathetic responses. our long-term plan is to use the interface to evaluate the nlp system, for example to identify statistically significant acoustic differences between empathetic/non-empathetic speech; the extent to which emotion/empathy perception is encoded across modalities, etc. ## proposed evaluation it is important to recognize that effective skills for expressing empathy through tm differ from those used in in-person encounters. virtual environments force healthcare professionals to adapt communication skills in a way that maintains professionalism and fosters the trust needed in medical care. in this study, we propose to investigate how tm can be used to assist rather than hinder patient-provider interactions, and to identify how the technology can support rather than diminish participants' perceptions of expression of understanding, compassion and willingness to help. delivering emotional and empathic suggestions visually as well as presenting them in an approachable way through a minimal interface is not a trivial task. the user must be able to relate to the interface and feel supported. empathy, however, is a complex construct, its interpretation and significance being task-specific. to address the challenges of the empathic construct, we start by focusing on specific tasks of empathic behaviors: breaking bad news to a patient. our primary focus then becomes identifying elements of both affective and cognitive empathy, or perspective taking, in which one person attempts to view the scenario from another person's perspective. we analyze empathy as it reflects in the verbal and nonverbal aspects of the conversation. to investigate the interface efficacy as an assistive, communication mediation tool, we will conduct specific human-computer interaction user studies of the intelligent interface. we propose to evaluate the interface along a number of dimensions with two third-party evaluators: (1) healthcare app developers; (2) doctors and patients. they are first briefed on the task and shown a demo of the interface. they will then watch the video interactions as they use the interface. their findings will be recorded and transcribed, and their answers to all our dedicated questions and open ended ques-tions (i.e., their concerns and suggestions) will be captured. we will then analyze their work and use it to better design and implement nlp-powered tools that can give both the doctor and the patient a frictionless and more accessible healthcare experience. our focus is on making mainstream tm healthcare interfaces accessible and easy to use which, in turn, can lower development costs, increase availability, and lead to better tech acceptability @xcite . paying attention to providers' interactions with patients can encourage not only empathy but also the formation of professional identities that embody desirable values such as integrity and respect. here, we want to build an ai communication mediation system that takes an experiential approach, putting experience and functionality on the same level. besides ease of use, efficiency, and computational aspects, we also want to explore the felt experience and what really matters to human users and what it takes to make technology more meaningful. we intend to design a tool that does not only mediate communication, but also shapes experience. most theoretical and practical hci @xcite and nlp @xcite systems and models focus primarily on quantitative metrics of evaluation. however, experience is subjective and dynamic, and thus, it emerges, shapes and reshapes through interactions with objects, people, environment and how these respond back to the experiencer @xcite . we believe that, besides required specific medical training, there is a need to create a space for clinicians to increase emotional awareness and discuss distressing aspects of their work. ## limitations and potential risks a. privacy, data concerns, accessibility, and personalization need to be addressed because ai models often rely on sensitive patient data to make decisions and predictions. emotion ai models are increasingly better at understanding patient emotions, but expert human supervision is necessary, hence part of the interface design. without earning users' trust and confidence, ai for emotional support will not achieve its potential to help people. in our system, we will make it clear that we use aggregate, de-identified user data collected solely for research purposes (subjects decide to participate). the pandemic drove people of all abilities to use digital products they never used, products where accessibility was often overlooked. most tm platforms do not have custom features to ease healthcare communications @xcite . moreover, tm providers may not be able to understand/address the accessibility issues with their patients even if the system was designed properly. web accessibility standards also need to be adjusted to tm platforms (w3c, 2021). we plan to make our digital experience accessible, and also consider aspects that were less explored in tm. our system allows interface developers to customize the default visualizations/feedback to match the system's aesthetics and goals. customization should further be available to the end-user and meet her individual healthcare preferences and needs (i.e., privacy controls around revealing one's abilities, security controls towards third-party devices combined with personal assistive technologies). b. limitations of tm setting. the tm technology brings benefits to medical care but also adds limitations, as it changes the verbal/nonverbal doctor-patient communication, and mandates focused attention of doctor and patient. unlike in traditional medical visits, where doctors/patients have physical proximity and communicate fully, with tm, non-verbal communication is limited and visual communication might be obstructed/distorted. to counter this loss of patient-doctor information, both the doctor and the patient need to be intentionally focused. doctors must address patients/family by name, nod, smile and provide auditory feedback to show they understand and empathize. both doctors and patients must avoid disruptions outside the medical tm visual field. however, even with the tm limitations, research so far found no reduction in patients' perceived level of physician empathy @xcite . in fact, in tm visits, with the doctor driving about two-thirds of the medical dialog @xcite , tm patients reported higher satisfaction. we argue that, in order to make up for the lack of non-verbal communication, in tm visits, doctors increase verbal communication, voicing agreement more and overall, providing more varied verbal feedback that improves the socio-emotional connection with patients. even though more research is needed over a longer period of time, we believe there is a tm technology paradox: the limitations introduced by the tm technology (reduce communication -non-verbal) in fact force developing the very behaviors they were expected to hinder (increase communication -verbal). c. potential risks in emotional ai. ai is a necessary tool in tm solutions to assist with emotion detection. at the same time, it increases the risk for emotion mis-identification and, worse, has the potential to generalize this across large groups of patients. for example, emotional ai can fail to capture how neurodivergent and neurotypical patients @xcite , or patients of various ethnicities/cultures, ages, genders express emotions, and thus easily mis-identify negative for positive emotions ( @xcite . the smile of a japanese patient might be used to show respect or hide her true emotion, while for an american/australian/canadian patient it might be a sign of happiness. some research found that, as compared to men, women not only seem to smile more but they might also do so on purpose, to diffuse a negative situation @xcite . without intentional, situated research and implementation, the tm solution can easily stereotype some patients and miss-qualify the experience of others. for successful tm, the emotional ai algorithms must account at least for cultural, age and gender differences in patient behaviors. they must also be able to identify extreme views, (e.g. racism, xenophobia, homophobia or ageism) that can lead to miss-interpreting doctor-patient communications in tm visits. this is possible only when intentionally hiring diverse teams to develop the tm solution, e.g. psychologists, ethicists, healthcare professionals and software engineers. allowing for multi-modal inputs, e.g. not only facial recognition (of smiles) but also voice inflections, tone, or choice of words, is crucial to correctly identify emotions and avoid bias and stereotyping. previous research has shown that multi-modal information, grace to complementarity benefits, is much more valuable than individual information -e.g. when used individually, accuracy in facial coding, biometrics, and electroencephalography (eeg) was 9% -62%, but increased to 77%-84% when combined @xcite . in ai, multimodal emotion and empathy detection architectures are still in their infancy with their own challenges (for a survey, see @xcite ). in our study, we intend to contribute to multi-model tm solution development. ## future considerations as healthcare technologies advance, nlp solutions also need to evolve to address the changing needs of tm providers, in particular, to improve the patient/family/caregiver -clinician communication with empathy and compassion. our proposal to design and build an ai-powered interface to better guide/train medical professionals is timely. with our reliably-evaluated interface, we can develop objective data-driven measures of empathy and foresee that they can leverage the promise of data analytics, thus shedding new light, from a novel quantitative perspective, on the construct of empathy (as a psychological and socio-behavioral phenomenon) and its indicators in linguistic behavior. these resources have the potential to present an entirely new framework to investigate, analyze, understand, and automatically detect empathy using advanced language processing technologies. our proposed emotionally-intelligent interface contributes to research on how to decode human behaviors with respect to empathy expression, perception and action. we combine computer science, engineering, language, medicine, human-centered design and education to extend our understanding of one another during the two-way audiovisual communication that has become ubiquitous in the lives of many patients seeking health care. such a system is a novel knowledge-rich resource that could unlock new breakthroughs in our understanding of linguistic discourse-analytic and behavioral indicators of empathy to help shape communication training for physicians and others. for future successful tm solutions, in our opinion, the following system, doctor and patient needs will drive continued development. first, we see a multi-country trend to develop healthcare systems that provide both traditional and modern medicine intentionally integrated (healthcare system need). for this, tm would greatly benefit from a multimodal and multisensory patient evaluation, the basis of traditional medical practices @xcite . second, with increasing invested interest and medical knowledge, patients and their families want to be active co-contributors in the healthcare process (patient need). the future tm interface must welcome patients to share private pictures, videos, notes about their health journey. third, doctors need future tm solutions to be best training tools not only to meet their varied individual learning styles (visual, auditory, kinestetic) but also tools that they can use for self-training on demand (healthcare professional need). to meet all these needs, we believe only tm solutions with smart, immersive and empathic interfaces designed as interactive, adaptive environments that facilitate versatile multimodal and multisensory engagement for more efficient, aesthetic, memorable, and healing medical experiences will be successful.