Human-Human Health Coaching via Text Messages: Corpus, Annotation, and Analysis (2020)

Abstract: Our goal is to develop and deploy a virtual assistant health coach that can help patients set realistic physical activity goals and live a more active lifestyle. Since there is no publicly shared dataset of health coaching dialogues, the first phase of our research focused on data collection. We hired a certified health coach and 28 patients to collect the first round of human-human health coaching interaction which took place via text messages. This resulted in 2853 messages. The data collection phase was followed by conversation analysis to gain insight into the way information exchange takes place between a health coach and a patient. This was formalized using two annotation schemas: one that focuses on the goals the patient is setting and another that models the higher-level structure of the interactions. In this paper, we discuss these schemas and briefly talk about their application for automatically extracting activity goals and annotating the second round of data, collected with different health coaches and patients. Given the resource-intensive nature of data annotation, successfully annotating a new dataset automatically is key to answer the need for high quality, large datasets.

## introduction a sedentary lifestyle significantly increases the risk of numerous diseases such as type 2 diabetes, cardiovascular disease, and depression @xcite . unfortunately, physical inactivity has progressively increased over the past several decades. it can be attributed to using modes of transportation for short distances, labor-saving devices, and less active occupations among various other reasons. however, the underlying problem is a lack of motivation. successfully implementing healthy behaviors require significant motivation that most people, individually, find difficult to initiate and maintain @xcite . health coaching (hc) has been identified as a successful method for facilitating health behavior changes by having a professional provide evidence-based interventions, support for setting realistic goals, and encouragement for goal adherence @xcite . but hc has its limitations such as it is expensive, time-intensive, and not available around the clock. therefore, we aim to build a dialogue-based virtual assistant health coach that will converse with the patients via text messages and help them to set specific, measurable, attainable, realistic and time-bound (s.m.a.r.t.) goals @xcite . the smart goal-setting approach has been rigorously adopted to set realistic and manageable goals in different fields such as health behavior change and software engineering. it has been shown that goal setting and action planning help patients adopt healthy behaviors and manage chronic diseases @xcite . also, text messages have been shown to help patients follow healthy behaviors as they provide a continuous means of education, support, and motivation @xcite ; currently, a majority of the population owns a cellphone (96% are cellphone users and 81% are smartphone users foot_0 ). most goal-oriented dialogue systems assume that a user has a predefined goal that needs to be accomplished using the system. however, that is not the case in hc dialogues. instead of the usual information-seeking dialogues, where the user requests information from the system, hc dialogues are collaborative where both the coach and the patient negotiate a goal that best suits the patient's lifestyle and seems realistic based on their previous activity patterns. an excerpt from dataset 1 is shown in figure 1 . the patient starts with an (1) patient: good morning, my goal is to aim for 30,000 steps, 40 flights, 12000 calories and 12 miles by the end of friday this week (2) coach: wow that's a lot. figure 1 : example of a conversation between the health coach and a patient ambitious goal in (1), and the coach helps to make it more realistic through (2)-( 5 ). as the conversation takes place over the text messages, a dialogue system will also need to take care of abbreviations and typing errors such as 'goals.ill' in (6). moreover, most existing dialogue datasets do not involve any follow-up conversations. for instance, once a flight is booked, the system doesn't follow-up on how the trip was or if the user would like to modify the booking. however, it is a crucial step in hc conversations as patients tend to change their goals on encountering a barrier. lastly, hc conversations happen over multiple days. some days no messages are exchanged and some days more than 10 messages get exchanged. most publicly available datasets assume that the task will be finished in one sitting. due to this collaborative negotiation setting over multiple days in our corpus, goal information is spread throughout the dialogue. motivated by these complexities, we decided to annotate our data for two types of information: (1) the smart goal attributes in the dialogues to track patients' goals, and (2) different stages and phases that model the conversation flow in hc dialogues. for our domain, smart goal attributes are the slotvalues pertaining to a patient's goal. stages and phases are more abstract, but otherwise analogous to tasks and sub-tasks as defined in task-oriented dialogue systems @xcite . we believe the smart annotation schema that we designed can be applied to any task where smart goal setting is being used and not just physical activity. similarly, the stages-phases annotation schema can be used to model the flow of any collaborative decision making counseling dialogue. in this paper, we will discuss the two rounds of data collection process, the subsequent analysis of the dialogues, which includes developing schemas and annotating the data, and application of models trained on these annotations. our contributions can be summarized as follows: ## related work one cannot build a good domain-specific dialogue system without having any insights into how users will interact with the system. therefore, first we need data that represents at least some range of actions that are found in human-human or humanmachine conversations in the given domain. initiatives such as the dialogue state tracking challenge (dstc) started in 2013 to provide a common testbed for different tasks related to domainspecific dialogue systems such as dialogue state tracking, dialogue act prediction, and response generation; labeled datasets for each of these tasks were provided @xcite . however, most of these datasets focused on traveling and restaurant booking domains @xcite . moreover, for data collection, predefined scenarios are given to the users and thus, the users' responses are not as spontaneous as they would be in a real-life situation @xcite . unfortunately, there are no such publicly available datasets for dialogue systems in the health domain. the idea of automated conversational agents to promote healthy behaviors has recently gained considerable interest. researchers such as @xcite and @xcite respectively worked on promoting physical activity adherence and supporting psychotherapy for adults using automated systems. but internally most of these systems rely on a predefined set of input/output mappings, focus more on general goal setting, and do not provide follow-up during goal accomplishment. researchers have also focused on computational analysis of conversations in the health domain. @xcite collected motivational interviewing (mi) based counseling interviews from public sources such as youtube and built models to predict the overall counseling quality using linguistic features. before the youtube data, the authors also worked on data collected in clinical settings, graduate student training and such, but didn't release it due to privacy reasons @xcite . the authors used the well established motivational interviewing treatment integrity (miti) coding system to annotate the data and score how well or poorly a clinician used mi @xcite . the miti coding system was also used by @xcite to annotate reflections in the health behavior change therapy conversations. since mi based interventions focus on understanding patient's attitudes towards the problem and persuading them to change, the miti coding system supports assessing clinicians based on how well they bring forth patient's experiences, cultivate change talk, provide education, persuade them through logical arguments, and such. however, specific goal setting is not the main focus of these interviews and is rarely discussed. a framework for health counseling dialogue systems closest to ours is by @xcite . their task model comprises opening, small talk, review tasks, assess, counseling, assign task, preclosing, and closing. conversely, our stages-phases schema looks at the fine-grained decomposition of review-tasks, counseling, and assign task, which @xcite did not do. as far as we know, no other work models hc dialogues collected in a smart goal setting, focusing on slotvalues and higher-level conversation flow. ## smart goal setting based on the domain, practitioners modify the definition of smart components to fit the task at hand. for physical activity, we define them as follows: an example of a well-defined smart goal is, i will walk 5000 steps three days a week on monday, wednesday, and friday for 2 weeks, where walk is a specific activity; 5000 steps and 3 days are measurable quantities; 2 weeks is the total time frame. as concerns attainability and realism, they are not immediately available from this goal statement and will depend on the person's circumstances. on the other hand, i will start walking more is a poorly defined, vague, and unquantified goal and is not likely to lead to success. ## data collection and analysis dataset 1: we recruited 28 patients between the ages of 21 to 65 years who were interested in increasing their physical activity at our university's internal medicine clinic. a health coach, trained in smart goal setting, conversed with the patients to set goals every week for four weeks via 1 2 3 4 5 6 7 8 dataset 1 c 19.6 13.8 12.3 11.8 -na-p 15.3 12.0 10.7 11.0 -na-t 34.9 25.8 23.0 22.8 -na-dataset 2 c 14.9 14.2 11.1 11.3 8.7 9.5 9.2 9.2 p 11.0 10.9 7.8 7.1 5.6 6.9 6.6 6.7 t 25.9 25.1 18.9 18.4 14.3 16.4 15.8 15. 9 the patients were also given fitbits to monitor their progress. the coach monitored patients' progress using the mytapp application, as it can fetch the most up-to-date activity data from a patient's fitbit account and show it at one place along with text messages. this reduces the workload for the coach as at any point in time during the study the coach had at least 3 patients and would have had to login into their respective accounts to access the fitbit data without the application. the coach needed all this information to help patients set realistic goals based on previous weeks' performance. the hc conversations involved setting a specific, measurable and realistic goal, and solving any barriers to goal attainment. the coach sent reminders based on patients' preferences and provided motivational feedback on their progress. out of 28 patients, only one did not finish the study due to health problems. therefore, we only considered 27 patients' data for analysis and building models. dataset 1 comprises 2853 messages, where 54% of messages were sent by the coach and 46% by the patients. this tells us that both the coach and the patients were equally involved. an excerpt was shown earlier in figure 1 . lessons from dataset 1 collection: during the initial face-to-face recruitment process at the university clinic, patients were given information about the study and the concept of smart goal setting was explained to them. to help them understand it clearly, the goal for the first week was sometimes discussed during that initial interaction. hence, we found that portions of the initial goal setting conversation may have been missing from the text messages, including: the patient's goal for the first week, discussion that led to that goal, and any time preferences for the text messages. therefore, during dataset 2 collection, we asked the recruiters to take notes about what was discussed face-toface, and asked health coaches to reiterate the first goal in text messages even if it was already known. in cases where patients didn't have information about their current activity level, a goal of one mile (2000 steps) a day was suggested. the recruiters also helped patients with setting up fitbit trackers, downloading the fitbit app, and linking the two together during the initial recruitment process (same as dataset 1). however, based on dataset 1 collection, recruiters had a better understanding of the issues that might arise with fitbit and also met the patients again during the study (if possible) to fix the issues. lastly, during the dataset 2 collection, we also collected audio-recorded feedback at the end of the study if the patients came back to the clinic, else feedback was taken over a phone call and notes were recorded. dataset 2: we recruited three different individuals trained in smart goal setting to be health coaches. we also recruited 30 different patients and conducted the study for eight weeks instead of four to analyze changes in messaging behavior over a longer period. the same mytapp application was used to text the patients and fitbits were given to the patients. out of 30 patients, one patient withdrew after 5 weeks, one lost their fitbit after 2 weeks, and one set goals for only 2 weeks and then almost stopped responding. since the latter two were in the study for fewer than 4 weeks, we only consider the data from 28 patients. we also removed all the messages discussing an appointment time for the exit interview, which comprises more than 600 messages. this resulted in 4134 messages among which 58% were sent by coaches and 42% by the patients. dataset 1 only included about 30 stage phase description phase boundary examples goal setting identification during the beginning of the week when the coach asks the patients about their goal or when the patients inform their goal to the coach coach: now what goal could you make that would allow you to do more walking? refining when the coach asks (or the patient informs) the specifics of the goal such as time, location, frequency to make the goal more effective coach: what time do you plan to do so i can set up a reminder? anticipate barrier when the coach asks the patients (or the patients specify) their confidence in achieving the goal (range 1-10) or if they see any upcoming barriers coach: do you think the weather will make it hard for you to take 50 min walks everyday this week? solve barrier when the coach tries to help patients overcome a barrier or increase their attainability score to 10 without modifying the quantity coach: what do you think will make it easy to accomplish/achieve your goal? negotiation when the patient chooses a goal that the coach thinks might be too much/less or vice-versa coach: another 8 . what if you were to try for 8000 steps again this week would the answer be a 10? goal implementation refining same as the previous stage; here it usually follows solve barrier or goal negotiation phase to make the goal more specific coach: have you decided when you would like to get your walk in? anticipate barrier similar to the previous stage, but here it indicates the barrier that has been encountered patient: good morning [name]. i probably won't be able to make my goal this week. i'm at a professional development all day today and there are no stairs in this building solve barrier same as the previous stage coach: do you want to try to make your goal over the weekend? negotiation when the patient is unable to accomplish the goal or wants to do more, the coach or the patient asks to modify the goal patient: please change my safety goal to three days per week. follow up when the coach asks the patient (or patients themselves inform) about their progress and if they can accomplish the goal coach: good afternoon! how is your goal for this week going so far? ## annotation of the coaching dialogues in this section, we will look at the two types of annotations: smart goal annotations and stagesphases annotations. since no work exists that has used smart criteria to set physical activity goals via sms, we designed the schemas that were in-spired by the literature on goal setting @xcite . we used the general architecture for text engineering (gate) tool for annotations @xcite . stages and phases annotation schema: 15 patient-coach conversations from dataset 1 were used to design stages-phases schema. this annotation aims to understand how the conversation unfolds in hc dialogues. stages and phases respectively help to capture the coaching tasks and sub-tasks being performed throughout the communication dialogue. the annotation schema along with descriptions is shown in stage: goal setting phase: goal identification coach: what would you like to set as your smart goal this week? patient: smart goal 12k steps a day? phase: goal negotiation coach: ok, just something to think about... you got 12k steps 3 out of 7 days in the last week. that was saturday, sunday and monday. how many days out the week do you want to do 12k step? everyday? patient: let's do 15k coach: that's more patient: 12k tu ,w, th coach: are you sure? if you think 12k everyday is realistic for you , go for it! patient: it's a challenge i'll try coach: let's keep it at tue, wed. and thurs then. patient: ok phase: solve barrier coach: what do you think will make it easy to accomplish/achieve your goal? patient: use stairs more and less elevator phase: anticipate barrier coach: on a scale of 1-10 with 10 being very sure and 1 not at all sure. how sure are you that you will accomplish your goal? patient: 8 coach: what do you think will make it difficult? patient: not being able to walk during my lunch hours because it's busy at work. so time. phase: goal negotiation coach: i see maybe you should pick weekend days. that's when you have been most active according to fitbit coach: last sat and sunday you got well over 12k steps coach: or maybe cut down on the amount of steps on those days. how can you change your goal to make that a 10 on the scale? patient: ok. 12k on weekends coach: sounds great good luck!! stage: goal implementation phase: follow up coach: good morning! how is your goal for this week going so far? patient: good morning. it's going great stage phase message count boundary count goal setting identification 408 109 refining 344 85 anticipate barrier 363 82 solve barrier 158 52 negotiation 92 19 goal implementation refining 16 4 anticipate barrier 8 4 solve barrier 25 7 negotiation 23 6 follow up 1348 120 an annotated conversation is shown in figure 2 . two annotators annotated four previously unseen patients' data for stages and phases (447 messages). inter annotator agreement (iaa) was measured using cohen's kappa coefficient (κ) @xcite ; we obtained an excellent κ=0.93. this may be partially due to the stages-phases being bound to occur in a particular sequence: our hc conversations follow a particular structure, which involves phases such as identification, refining, and negotiation. therefore, we analyzed the hc conversations as concerns likely transitions, and their frequencies. tag feature description slot example intent example specificity activity the activity that will be done by the patient patient: ok. i'll walking the stairs in the mornings from 8 to 10 monday -friday time the time of the day when the patient will be doing the activity patient: ok. i'll walking the stairs in the mornings from 8 to 10 monday -friday coach: like how many days next week and at what time of day? location the location where the patient will be doing the activity patient: i can also plan to walk the stairs at home. after work coach: could you maybe get your steps done in the house? measurability quantity (amount/ distance/ duration) quantifies the activity in some way to show what patients are planning to accomplish. it can be number of steps or stairs, distance or duration patient: yes, i'm going for 6000 step (amount) patient: i will walk 3 blocks (distance) patient: i do 40 min of walk (duration) coach: how many would you like to try for? days (name/ number) the number of days or the name of the days the patient will be working on the chosen activity patient: ok. i'll walking the stairs in the mornings from 8 to 10 monday -friday (daysname) patient: i will try 3 days. (daysnumber) coach: will you walk 4 block on the same days mon, wed and fri?(days-name) coach: like how many days next week and at what time of day? (days-number) repetition the number of times the activity will be done in the same day patient: i will attempt to spend 15 mins 3times a day walking up and down two flights of stairs. 8am attainability score specifies how confident a patient is about accomplishing the goal on a scale of 1-10 patient: 8 coach: on a scale of 1-10 with 10 being very sure and 1 not at all sure. how sure are you that you will accomplish your goal? realism helps to indicate statements that judge the realism of a goal for the patient. it is usually based on their previous performance coach: sounds like a very doable goal you are averaging over 9k steps during the weekdays, now annotations in dataset 1 minus the first two to three introduction messages about the study in each conversation as they were the same. other than follow up, all other phases in gi stage rarely occur. focusing on transitions now, a priori, 121 different transitions are possible in a given week, as we have 10 unique stage-phase categories plus the beginning and end of the week (start, stop). however, only 39 unique transitions occur in our dataset, given that a week always starts with the goal setting stage, which in turn starts with the goal identification phase. on further analysis, we found that only 13 of those 39 transitions have a probability above 0.3, as shown in figure 3 . smart tag annotation schema: similar to stage-phase annotations, 15 patient-coach conversations were used to design the smart goal annotation schema. the schema is described in in total, 447 messages were annotated for iaa. there were 128 messages with specificity (s) tag, 120 with measurability (m) tag, 45 with attainability (a) tag and 13 with realism (r) tag. we achieved κ ≈ 0.9 for {s, m} and κ ≈ 0.5 for {a, r}. this is mostly because {s, m} tags have a higher number of occurrences in the data as compared to {a, r} which are hard to distinguish from each other and have very few occurrences. it should also be noted that for {s, m} word-level annotation is more important whereas for {a, r} message level annotation makes more sense. ## development on dataset 1 dataset 1 has been our foundation to develop the computational models we are interested in, namely smart tag and phase prediction. before building these models, we wanted to see if smart tags and phases exhibit any sort of relationship that can be leveraged as features. we plotted the number of smart tags in each phase and obtained the graph shown in figure 5 . smart tags are unevenly distributed across phases, with identification, refining and follow up containing the majority of smart tags. therefore, we experimented with smart tags as a feature in the phase prediction model and vice versa, and found that smart tags helped to predict phases better, than phases help predict smart tags @xcite . we achieved an average (macro) f1 score of 0.80 on smart tag prediction using structured perceptron with feature combination of the current and surrounding words, pre-trained google word embeddings foot_2 , and spacy foot_3 named entity recognizer output. similarly, we achieved an average (macro) f1 score of 0.71 on phase prediction using conditional random fields with feature combination of unigrams, distance of the message from the top in a given week, and human-annotated smart tag counts. importantly, an almost similar performance (f1 @xmath0.69) was achieved using automatically predicted smart tags. unfortunately, use of deep learning is not suitable due to our very small dataset; only 2853 messages in total. one can also notice rare occurrences of classes such as anticipate barriers, solve barriers, and negotiation in figure 5 . ## applications of models developed on dataset 1 so far the models developed in section 6 have been used for two applications: annotating dataset 2 for both smart tags and phases, and goal extraction. goal extraction on dataset 1: goal extraction can help health coaches to recall a goal discussed during the conversation and save their time. since smart tags helped predict phases better, we built a pipeline where smart tags were predicted first, then they were used as one of the features in phase prediction. after smart tag and phase prediction, we extracted the smart tags as long as they were not from the follow-up phase to avoid extracting accomplished and remaining measurable quantity. 65% of the goals we extracted correctly identified at least 8 out of 10 smart attributes of the gold standard goal. detailed results for goal extraction and the two models are available in @xcite . we are currently evaluating our goal extraction model on dataset 2 with the help of health coaches and automatic evaluation. dataset 2 annotation: we used the same pipeline for annotating dataset 2, except we changed google word embeddings to pre-trained elmo word representations for smart tag prediction @xcite . to measure performance, we manually annotated three randomly chosen pa-tients' data, one from each coach. we achieved an f1 score of 0.81 (macro) and 0.98 (weighted) on smart tag annotations and 0.37 (macro) and 0.61 (weighted) on phase annotations. the results for smart tag prediction on dataset 2 is equal to what we achieved on dataset 1. this means that smart tag annotations are transferable even if the dialogues are between different coaches and patients. a sample output for smart tags is shown in figure 6 . our model correctly annotated (1), but missed m days number in (2). more specifically, for the three patients that we automatically annotated, only 113 words (2%) were incorrectly labelled or had a missing label; 390 words (6%) were correctly labelled with a smart tag; and 5959 words (92%) were correctly labelled with 'none' tag. because performance on automatic phase annotation was not as high as we had hoped, we will adopt a semi-automatic approach, with a round of manual edits following automatic annotation of phases. we see semi-automatic annotation as crucial, especially given that state-of-the-art deep learning models require large labeled training data. semi-automatic annotation can still save thousands of hours of manual labor. ## conclusions and future work we envision a virtual assistant health coach that can help people to increase their physical activity by motivating them to set smart goals. to this end, we collected a health coaching dialogue dataset and developed two annotation schemas, one that captures the slot-values of a smart goal and the other that captures the higher-level conversation flow of the health coaching dialogues. we briefly discussed the models built using the two annotations and their application for automatic goal extraction. we also collected a second round of dataset and showed that it can be reliably annotated using the models built on the first dataset. our immediate next steps are to perform extrinsic evaluation of the goal extraction pipeline with the help of our health coaches and integrate it into the mytapp application used by the health coaches for round three of the data collection.