Summarizing Behavioral Change Goals from SMS Exchanges to Support Health Coaches (2021)

Abstract: Regular physical activity is associated with a reduced risk of chronic diseases such as type 2 diabetes and improved mental well-being. Yet, more than half of the US population is insufficiently active. Health coaching has been successful in promoting healthy behaviors. In this paper, we present our work towards assisting health coaches by extracting the physical activity goal the user and coach negotiate via text messages. We show that information captured by dialogue acts can help to improve the goal extraction results. We employ both traditional and transformer-based machine learning models for dialogue acts prediction and find them statistically indistinguishable in performance on our health coaching dataset. Moreover, we discuss the feedback provided by the health coaches when evaluating the correctness of the extracted goal summaries. This work is a step towards building a virtual assistant health coach to promote a healthy lifestyle.

## introduction physical activity (pa) is extremely beneficial to one's health, as it reduces the risk for serious health problems like type 2 diabetes and heart diseases, and also helps to improve mood and reduce depression and anxiety @xcite ). yet, only 45% of the us adult population met the federal guidelines for pa in 2016 @xcite . such findings suggest that a majority of people are unable or not motivated enough to engage in pa @xcite . health coaching has been identified as a successful method for facilitating health behavior changes: a professional provides evidence-based interventions, support for setting realistic goals and encouragement for goal adherence @xcite . health coaching has its origin in motivational interviewing (mi) and is guided by the principle that patients need to identify their own motivation to be successful in achieving health behavior changes @xcite . unfortunately, personal health coaching is expensive, time-intensive, and may have limited reach because of distance and access. as a consequence, researchers have been exploring the use of technology such as computers and (mobile) phones @xcite @xcite to promote health behavior changes for a while now. mobile health technologies (mhealth) have been particularly effective due to their accessibility and ability to reach large populations at low cost. currently, about 96% of the us population owns a cellphone and 81% owns a smartphone @xcite . therefore, we aim to build a dialogue-based virtual assistant health coach to help patients set physical activity goals via text messages (sms) @xcite . studies have shown that setting specific and challenging goals leads to better performance than setting abstract or easy goals @xcite . smart (specific, measurable, attainable, realistic, and timebound) is one such goal setting approach that helps to create specific, measurable, and manageable goals and provides a clear path to success. in this paper, we focus on the natural language understanding (nlu) module of the dialogue system, grounded in two health coaching datasets we collected, and show its application to goal summarization. these goal summaries will help the health coaches to easily recall patients' past goals and use them to suggest a realistic future goal. currently, our health coaches use external documents like microsoft excel to keep track of patients' goals. we conducted an evaluation with the health coaches where they assessed the correctness and usefulness of automatically generated goal summaries in an offline setting. in our future work, health coaches will use the goal summarization module during real-time health coaching and we will evaluate its usefulness in the real world. the main contributions of this paper are: • we propose a two-step goal extraction process that uses phases and dialogue acts to identify the correct goal attributes and show that dialogue acts help more than phases. dialogue act (da) modeling. this task involves finding the intent behind the speaker's utterance in a dialogue such as request, clarification, and acknowledgment. the da tags may differ depending on the dialogue's domain. e.g., negotiation dialogues might involve tags like offer, accept, and suggest. as a result, numerous da schemas have emerged over time @xcite @xcite . however, the majority of them are difficult to reuse due to their complexity and lack of generalizability to other domains. efforts have been devoted to create a standardized schema that can be used for multiple datasets in different domains. one such effort led to the formation of the iso 24617-2, the international iso standard for da annotations @xcite . it provides a domain-and task-independent da schema with 56 das organized into nine dimensions. @xcite proposed a universal da schema by aligning tags from different datasets such as the dialogue state tracking challenge 2 @xcite , google simulated dialogue @xcite , and multiwoz 2.0 @xcite together. @xcite reduced the iso schema to 10 das and showed their applicability to datasets like @xcite , maptask @xcite , and verbmobil @xcite . on account of not reinventing the wheel, we used the iso schema for our dialogues @xcite . since many of the das didn't apply to our dataset such as turn take/grab, stalling, and pausing, we reduced the schema to only 12 das, mostly following @xcite . early work for da modeling involved treating the task as a structured prediction or text classification problem. @xcite used hidden markov models (hmm) to model the dialogue structure, where individual das were treated as observations and n-grams were used to model the probability of the da sequence. they also used acoustic correlates of prosody as raw features in the hmm model. researchers have also explored non-verbal cues such as body postures to better understand a user's intent during a tutorial dialogue @xcite . since then, deep learning techniques have also been applied to the task @xcite . convolutional neural networks (cnn) were also used for intent classification of a query @xcite . however, queries can be treated as individual sentences without any context. given context is important in a dialogue, we experiment with approaches that can take dialogue history into account such as conditional random fields (crf) @xcite and recent transformer-based bert (bidirectional encoder representations from transformers) models @xcite . in particular, we use the work by @xcite and @xcite as the guide for our bert-based da prediction models. ## datasets and annotations no health coaching dialogue dataset is publicly available. therefore, to understand the feasibility of using sms for health coaching, the challenges with patient recruitment and retention, and conversation flow between the coaches and the pa-tients, we collected two health coaching datasets (dataset 1 and dataset 2; dataset 2 is available upon request foot_2 , while dataset 1 cannot be shared due to lack of subject consent). to collect dataset 1, we hired one health coach who coached 28 patients, recruited at one of ui health's internal medicine clinics, for 4 weeks (since one patient didn't finish the study, we exclude their data). the health coach, trained in smart goal setting, helped patients to set a new smart physical activity goal every week. the health coach used mytapp, a web-based application developed by one of our collaborators and validated in other text-based health monitoring studies @xcite , to text the patients, who used their smartphones' texting service. the patients were also given a fitbit to track their progress and the coach could access patients' fitbit data on mytapp. the data collection process was similar for dataset 2, except we hired three new health coaches and 30 different patients, and doubled the duration to 8 weeks. since one patient lost their fitbit and one almost stopped responding after 2 weeks, we only consider 28 patients' data for dataset 2. dataset 1 comprises 2853 text messages (54% coach, 46% patients) and dataset 2 comprises 4134 text messages (58% coaches, 42% patients). in dataset 1, the average number of words per message for coach is 13.74±9.76 and for patient is 7.68±8.19, while in dataset 2, the average number of words per message for coach is 19.27±10.37 and for patient is 9.28±10.74. all our models in this paper are built on dataset 1 as it was collected first and hence, we only had gold standard annotations for that. dataset 2 was collected two years later. we did however annotate a subset of dataset 2 to evaluate the performance of our models on it. more information on the two datasets is available in @xcite . we performed a three-tier annotation on dataset 1: (1) stages and phases (2) dialogue acts, and (3) smart attributes. an example annotated with all the three schemas is shown in figure 1 . the stages and phases schema captures the higher-level conversation structure and consists of two stages: goal setting and goal implementation. the goal setting stage consists of five phases: identification, refining, negotiation, solve barrier, and anticipate barrier. the goal implementation consists of the stage: goal setting phase: goal identification coach: think about this week, let's call it week 1 of 4. [directive] now what goal could you make that would allow you to do more walking (specific activity)? [set question] patient: i can take the stairs (specific activity) at work (specific location) for the work week. [answer] phase: goal refining coach: sounds good. [feedback] so will this be for coming, going and at lunch time (specific time)? [propositional question] patient: i will do at least twice (measurable repetition) during workdays (measurable days name). [answer] phase: anticipate barrier coach: what might get in the way of you accomplishing your goal? [set question] patient: well if im pressed for time that could stop me. [answer] but i think i can fit in at least twice a day (measurable repetition). [inform] coach: on a scale of 1-10 with 10 being very sure and 1 not at all sure. [directive] how sure are you that you will accomplish your goal? [set question] patient: 9 (attainability score) [answer] phase: solve barrier coach: that is pretty sure but i want you to succeed... [inform] so how can you make that a 10? [set question] patient: well, its my first. lol. im not exactly an exercise pro, but with the coaching help im sure i can have that at 10 by next week. [answer] stage: goal implementation phase: follow up coach: good morning! [salutation] how is your goal for this week going so far? [set question] patient: going great. [answer] same phases, minus the identification phase, and plus an additional follow up phase. das capture the general intent of the sender's message at the utterance level (a message can contain one or more utterances). we use a set of 12 tags: set question, choice question, propositional question, inform, answer, commissive, directive, feedback, apology, salutation, thanking, and self correction. this is the same set of tags used by @xcite , except we added the answer and self correction tags from the original iso standard schema. this is because it is important for us to differentiate between inform, answer, and self correction tags for the goal summarization pipeline. the smart attributes schema captures the domain-specific slot values at the word-level and consists of 10 attributes: specific activity, specific time, specific location; measurable quantity amount, measurable quantity distance, measurable quantity duration, measurable days name, measurable days number, measurable repetition; and attainability score. to measure intercoder agreement, two annotators annotated four patients' data (447 messages) and obtained an excellent κ = 0.93 for phases; for smart attributes, κ ranges between ≈ 0.5 for attainability to ≈ 0.9 for specificity and measurability. 3 3 we didn't calculate kappa for dialogue acts as this schema has been validated on many other datasets @xcite . ## goal extraction approach it is usually assumed that users have a specific goal in mind when interacting with a goal-oriented dialogue system. as the user attempts to complete one sub-task after another in order to achieve the final goal, the dialogue becomes easy and sequential. however, some use-cases involve a decisionmaking process. e.g., when booking a flight ticket, the user might want to compare prices for different days, times, destinations, etc. in such cases, the system must keep all options available instead of simply replacing the slots in order. similarly, in our dataset, we noticed complex decision-making behavior where different entities are introduced by both the coach and the patient, some of these entities are then accepted, others rejected or forgotten. the conversation also consists of various smart attribute values that refer to the patient's current progress towards the goal. hence, the coaches need to scroll back through the patient conversations to recall the original goal and determine if the goal was met. a goal summary readily available for each patient can save time for health coaches and provide an idea for a realistic future goal. the correct goal summary for the conversation in figure 1 will beactivity: 'stairs', location: 'at work', days name: 'workdays', repetition: 'twice a day', score: '9' figure 2 shows the overall architecture of our pipeline, which consists of two steps: (1) the nlu module which infers smart attributes, dialogue acts, and phases; and (2) the goal extraction module which selects the smart attribute values included in the patient's agreed-upon goal. in figure 2, 'goal-c' refers to the current goal; it starts with empty smart attribute values and is updated as the week's messages are processed utterance-byutterance. below we will discuss the prediction models in the nlu module followed by the heuristics in the goal extraction module. foot_3 4.1 modeling smart attributes this task involves predicting one of the 10 smart attributes for each word or 'none'. we used dataset 1 (27 patients) for modeling and performed 5-fold cross-validation (train/test: 22/5 patients). we experimented with both sequential and nonsequential classifiers such as crf, structured perceptron (sp) (collins, 2002), logistic regression (lr) (grimm and yarnold, 1995), support vector machines (svm) (cortes and vapnik, 1995), and decision trees (dt) (quinlan, 1986). for features, we tried different combinations of -the current word, left and right context words, part-ofspeech (pos) tags, left and right context words' pos tags, spacy named entity recognizer (ner), current word's phase, and elmo word embeddings @xcite . the crf, sp, and lr models performed the best without a significant difference between them using the current and context words, elmo embeddings, and spacy ner. we decided to use the crf model with an f1 macro score of 0.81. in our previous work @xcite , we used models with word2vec embeddings @xcite but found elmo embeddings to perform better. ## human evaluation evaluating models with automatic metrics is important, but it is equally important to evaluate the usefulness and usability of these models with their users. we performed a pilot evaluation with the help of three health coaches to answer two main questions: (1) what is the health coaches' understanding of a correct goal summary? and (2) are these goal summaries helpful? we created an assessment using google forms and presented the three health coaches, who coached the patients during the dataset 2 collection, the same 12 <set of messages-goal @xmath0, where each pair consisted of a full set of weekly messages and the goal summary generated by our pipeline. the 12 pairs were chosen from 12 different patients, where each health coach had coached 4 of these patients. the summaries were generated by the smart+phases model as the evaluation took place before the da prediction model was built. but we can expect the same if not better results in terms of coaches' feedback as the goal summaries have improved with das. for @xmath1, the coaches were asked to judge the given goal summary as correct, partially correct, or incorrect. in case of partially correct or incorrect, they were asked to write the correct goal summary. partially correct meant some of the smart attributes were missing a value whereas incorrect meant that some of the attributes had an incorrect value. the evaluation results are shown in at the end of the assessment, the coaches were asked on a scale of 1 to 3, how useful a correct, partially correct, or incorrect goal summary would be to them. to this, all the three coaches said 3 (helpful) for correct goal summaries, 2 (neutral) for partially correct summaries, and 1 (not helpful) for incorrect summaries. this means that higher accuracy is required for the health coaches to feel comfortable in using goal summaries. the assessment form also consisted of an open-ended feedback field to write their overall impression of these goal summaries. one of the coaches said, "it would be nice to have the goal (summarized correctly) available and easily viewable, so that we would not have to scroll all the way backwards through our conversation and reread texts to figure out what the goal was. so thank you for doing this!". ## conclusions and future work many applications exist to promote a healthy lifestyle but they lack coaching components that are essential to keep the user motivated long term. in this paper, we discussed our work towards building a virtual assistant health coach that can help patients to set specific and realistic physical activity goals. mainly, we focused on the goal summarization pipeline that is built upon the nlu module of the system and showed its usefulness for health coaches. we found that utterance-level information captured by dialogue acts improves goal summarization performance. next, we will test its usability and helpfulness in an online setting while coaches are communicating with the patients in real-time. following that, we will use phases, dialogue acts, and smart attributes prediction models to generate possible responses for the coaches. in this paper, we have presented an approach that takes advantage of traditional machine learning models, contemporary deep learning ones, and heuristics. we believe that for certain domains where accuracy of information is important, and data is scarce, such as the health coaching exchanges we have discussed here, end-to-end approaches are neither feasible, because of lack of large datasets, nor appropriate, since usability and usefulness for different types of stakeholders are crucial. we cannot claim that our mixed approach would work for any conversational agent in a health care or legal domain where scarce data is available; however, we would encourage researchers who work on such applications, to experiment with a variety of methods as we do here.