M ed N gage: A Dataset for Understanding Engagement in Patient-Nurse Conversations (2023)

Abstract: Patients who effectively manage their symptoms often demonstrate higher levels of engagement in conversations and interventions with healthcare practitioners. This engagement is multifaceted, encompassing cognitive and social dimensions. Consequently, it is crucial for AI systems to understand the engagement in natural conversations between patients and practitioners to better contribute toward patient care. In this paper, we present a novel dataset (MedNgage), which consists of patient-nurse conversations about cancer symptom management. We manually annotate the dataset with a novel framework of categories of patient engagement from two different angles, namely: i) socio-affective engagement (3.1K spans), and ii) cognitive engagement (1.8K spans). Through statistical analysis of the data that is annotated using our framework, we show a positive correlation between patient symptom management outcomes and their engagement in conversations. Additionally, we demonstrate that pre-trained transformer models fine-tuned on our dataset can reliably predict engagement categories in patient-nurse conversations. Lastly, we use LIME (Ribeiro et al., 2016) to analyze the underlying challenges of the tasks that state-of-the-art transformer models encounter. The de-identified data is available for research purposes upon request.

## introduction due to the ease of use and efficiency of digital health interventions (dhis) @xcite , we are witnessing a surge in online conversations between patients and healthcare providers. literature suggests that actively engaged patients are more likely to obtain the full benefits of an intervention and exhibit better outcomes @xcite . therefore, it is critical to understand patients' engagement in online conversations and extract insights to aid healthcare professionals in providing in-time support to the patients. while previous research has made progress in modeling engagement in human-human and human-agent conversations @xcite @xcite , these works do not translate well to modeling engagement in patient-provider conversations. to bridge this gap, we introduce a novel resource called medngage, which consists of patient-private asynchronous message boards from an online intervention led by study nurses for symptom management. the dataset encompasses 2.1k turns between 68 patients and the nurses. we hypothesize that patient socio-affective engagement (e.g., sharing cancerrelated experiences) and cognitive engagement (e.g., information-seeking question) can predict their symptom management outcomes. it is important to note that datasets containing patientprovider conversations are rarely found in the ex-isting literature. to effectively model engagement in our proposed dataset, we create a novel framework called socio-affective cognitive engagement (sace), drawing inspiration from linguistic theories of discourse @xcite , cognitive science of grounding in communication @xcite , and the model of social presence @xcite . with sace, we manually annotate the dataset, classifying patient engagement into eight categories of socio-affective engagement (e.g., personal information @xcite ) across 3.1k spans and seven categories of cognitive engagement (e.g., information-seeking question, clarification question @xcite ) across 1.8k spans within conversations. additionally, we investigate the extent to which this framework can predict patient symptom management outcomes. figure 1 showcases excerpts from our dataset, which have been annotated using the sace framework. we conduct spearman's rank correlation and kruskal-wallis tests to analyze the relationship between engagement categories and changes in patients' perceived symptom control. the tests provide empirical evidence that patients who have a higher level of engagement in certain categories gain more control over their symptoms. thus, automated prediction of engagement from conversations can help practitioners to detect low patient engagement levels and identify effective intervention strategies so that they could tailor content for improved outcomes. to facilitate this, we conduct a range of experiments evaluating the efficacy of both traditional machine learning (e.g., svms) and state-of-the-art transformer models in predicting engagement, followed by an analysis using lime @xcite to identify the challenges of the tasks. thus, our contributions to this paper are: ## framework inspired by theories of discourse coherence, we explore linguistic and conceptual models of engagement across various disciplines. through iterative collaboration with expert nurses and linguists during the initial content analysis phase, we have generated a set of categories that effectively capture the nuances of our data. therefore, our categorization of engagement combines existing literature with data-driven adaptation to match the distinctive features of our dataset, medngage. we will discuss the theoretical underpinnings behind our proposed sace framework and the categories of engagement for annotating our dataset. ## dataset in this section, we describe our data source, followed by our annotation protocol and a brief discussion about the patient characteristics in our dataset. ## analysis in this section, we first report the distribution of different engagement categories in our data. then we illustrate the relationship between engagement and symptom controllability. the significance level for statistical analysis is @xmath0.05. ## experiments in order to assess the feasibility of predicting cognitive and socio-affective engagement, we train traditional svms and fine-tuned pre-trained transformer models on our dataset. ## implications on dhis by examining the relationship between patients' engagement and their symptom controllability, our study provides empirical evidence of the potential pathways of patient learning and behavior change to gain better control over symptoms through meaningful engagement. the classification models presented in this paper can be used to track patient engagement based on patient-provider interactions on asynchronous message boards. the models could be used as complementary tools to augment clinicians' capabilities to identify patients with a low engagement level so that they can focus their energy and time (i.e., tailoring content and communication accordingly) to enhance engagement and help those who struggle the most to obtain the maximum benefits of a given dhi. based on predicted engagement markers, or lack thereof, the provider can tailor the intervention content (e.g., suggest more strategies based on professional experience, segment a series of questions into a few posts, provide timely emotional support) to achieve common ground and cultivate an affective connection with the patient. for example, nurses can encourage the patients to i) provide relevant information to complete intervention activities and tasks (e.g., describing their symptom beliefs), and ii) share their feelings, cancer treatment, and symptom experiences. for patients participating in dhis to achieve desired outcomes, it is also important to feel close to the provider, for example, addressing the provider by name, acknowledging the provider's message, and recognizing their contributions to the care and support. our dataset, medngage contains rich narratives of various symptom experiences and management processes from women who were fighting advanced cancer in an online cognitive behavior intervention. the representational approach @xcite underlying the intervention is diseaseagnostic and designed to help clients understand how their disease representations relate to behaviors. therefore, the annotation system and model developed in this study can be applied to various healthcare scenarios involving the promotion of healthy behaviors and the enhancement of clients' mental and physical well-being. the corpus includes comprehensive representations of patient symptoms, concerns, and the obstacles they faced while seeking the best symptom management. this presents an excellent opportunity for various clinical nlp tasks, such as developing a dialog system, optimizing patient self-report/expression review, generating text summaries of patient conversations, and customizing automated relevant responses to improve patient engagement, as suggested in recent works on summarization @xcite and style transfer @xcite . for example, the system can use patient-specific language or generate text that encourages patients to ask questions or express their concerns. ## related work engagement in human-human conversation has been studied from both socio-affective and cognitive aspects, resulting in numerous frameworks and theories for modeling engagement. one of the inspirations behind our work, the community of inquiry framework @xcite , which builds on the educational philosophy and practice of dewey, models conversations in the online learning environment in terms of social presence, cognitive presence, and teaching presence. @xcite , proposes a community of inquiry framework that synthesizes pedagogical principles with the dynamics of computer conferencing, focusing on social presence. @xcite , the inspiration behind the cognitive aspect of engagement in our work, studies how grounding in conversation is shaped by purpose and medium of the conversation. the authors highlight different grounding references that we adapt to our work. previous research has investigated engagement in medical interactions, including conversations. studies have specifically examined the cognitive engagement of individuals with schizophrenia in conversations, with a focus on medication adherence @xcite . in contrast, our study employs various methods to identify and predict patient engagement in a complex intervention aimed at modifying multiple health behaviors, including diet, physical activity, relaxation, and medication adherence. further, we investigate the impact of engagement on patient symptom outcomes, providing practical strategies for nurses and other practitioners to effectively engage patients and deliver optimal care. other studies have explored engagement in different contexts such as political argument settings @xcite , conversations around terrorist attacks @xcite , socio-affective aspects of conversations such as emotion @xcite , student engagement in online discussion forums @xcite , cognitive engagement in mooc forums @xcite , real-time engagement in reducing binge drinking through intervention text messages @xcite , user engagement in online health communities @xcite . however, none of these studies specifically consider the dynamics of socio-affective and cognitive engagement in online conversations between patients and healthcare providers. our work is the first to develop a dataset with aggregated annotations and models for computationally modeling engagement in patient-provider scenarios. ## conclusions we have developed a framework (sace) that effectively captures patient engagement in patient-nurse conversations. through the analysis of a unique dataset (medngage) consisting of online patientnurse conversations, we have identified eight categories for socio-affective engagement and seven categories for cognitive engagement. these findings provide valuable insights for behavioral scientists to understand and monitor patients' engagement in healthcare interactions. the approach could be applied in other fields, such as online education, where engagement plays a crucial role. our analysis confirms that higher levels of engagement, including the increased coordination and emotional connections between patients and healthcare practitioners during the intervention, result in improved symptom control. additionally, we have demonstrated that fine-tuned transformer models can reliably predict fine-grained engagement in conversations so that practitioners can adjust their communication style and tailor strategies promptly, to promote patient engagement for improved outcomes. our analysis of model output using lime has shown the challenges that state-of-the-art transformer models encounter for the two tasks, which can be used to improve these models for similar tasks in the future. we expect our system could also aid subsequent text generation tasks, such as summarization of patient conversations, and tailoring automated responses in clinical settings.