Evaluation of In-Person Counseling Strategies To Develop Physical Activity Chatbot for Women (2021)

Abstract: Artificial intelligence chatbots are the vanguard in technology-based intervention to change people’s behavior. To develop intervention chatbots, the first step is to understand natural language conversation strategies in human conversation. This work introduces an intervention conversation dataset collected from a real-world physical activity intervention program for women. We designed comprehensive annotation schemes in four dimensions (domain, strategy, social exchange, and task-focused exchange) and annotated a subset of dialogs. We built a strategy classifier with context information to detect strategies from both trainers and participants based on the annotation. To understand how human intervention induces effective behavior changes, we analyzed the relationships between the intervention strategies and the participants’ changes in the barrier and social support for physical activity. We also analyzed how participant’s baseline weight correlates to the amount of occurrence of the corresponding strategy. This work lays the foundation for developing a personalized physical activity intervention chatbot.

## introduction physical inactivity is a leading risk factor for premature death from noncommunicable diseases such as heart disease, stroke, and type 2 diabetes @xcite . despite the known benefits of physical activity (pa) in reducing morbidity and mortality @xcite , physical inactivity is common among americans. about 80% of american adults do not meet the guidelines for both aerobic and musclestrengthening activities @xcite . common reasons women are more likely than men to 1 the dataset and code are available at https://github.com/kaihuiliang/ physical-activity-counseling not meeting physical activity guidelines include lack of motivation, lack of social support, lack of time in exercising, etc. effective interventions that can help women overcome these barriers and engage in more regular activity are needed to reduce multiple health risks. physical activity intervention programs have evolved with emerging digital and communication technologies @xcite @xcite @xcite . recently, effective technology-based interventions have been published. for example, a pilot randomized clinical trial (rct) of a mobile app-based online group intervention for african american young women @xcite showed the online tracking and social support increased objectively measured daily physical activity in comparison to a control condition where participants only used the fitbit for self-monitoring. another rct tested the use of a mobile app in conjunction with brief in-person counseling and found the combination increased objectively measured physical activity over three months compared to a control condition in which participants only used accelerometers @xcite . these interventions lack the capacity to tailor the intervention messages to accommodate different individuals' needs and circumstances and automate such personalized messages through mobile technologies. artificial intelligence (ai)-based chatbots are the vanguard in technology-based interventions, and they can deliver intervention messages and tailor contents to meet individual needs through natural conversations with no spatial or time restraints. the first step to develop physical activity intervention chatbots is to learn natural language conversation strategies from human-human conversations in physical activity intervention domains. specifically, it is vital to understand how participants' and trainers' conversation strategies influence the outcomes and how trainers could adapt to different physical activity statuses, socio-demographics, and conversation behaviors to achieve better results. in this research, we aim to address this question by analyzing a real-world intervention conversation dataset collected as a part of an effective physical activity intervention program for women @xcite . unlike the commonly used role-play dialog datasets, our dataset consists of actual dialogs between research staff (trainer) and study participants. we developed a comprehensive annotation scheme based on how the original intervention was organized to extract both social and persuasive conversational strategies. then we manually annotated a set of 17 conversations with 7,808 sentences. after achieving high interrater reliability levels, we developed a bert-based classifier to detect the whole unannotated dataset's strategy. lastly, we analyzed which and to what extent specific conversational strategies decrease physical activity barriers and increase social support among the intervention participants from the first visit (baseline) to the 3-month visit. the following research questions guide our analysis: rq1: does using more barrier strategies by trainers and participants in the intervention session decrease participants' physical activity-related barriers? rq2: does using more support strategies by trainers and participants in the intervention session increase participants' physical activity-related social support? rq3: do participants with a heavier weight at baseline use more weight strategies in the intervention session than participants with lighter weight? this work's main contribution is that we created a real-world human-human intervention dialog dataset that can be used to build physical activity promotion dialog systems. we also developed and designed a set of comprehensive four dimension annotation schemes that can be leveraged to behaviorchange dialogs. lastly, our analysis revealed how trainers' and participants' usage of conversational strategies influence the outcome and how a physical activity intervention chatbot could better adapt to participants' individual needs. ## related work applying ai chatbots to lifestyle modification programs (e.g., physical activity and diet promotion) has great potential to provide cost-effective, sus-tainable, and broadly applicable solutions and is anticipated to benefit health across application domains @xcite . previous studies that developed and tested the efficacy of chatbot-delivered physical activity and diet interventions have demonstrated the potential of using chatbots as a practical solution to promote positive behavior changes @xcite @xcite @xcite . among these studies, some have demonstrated how theory-driven intervention strategies combined with ai chatbot technologies can effectively yield behavioral changes @xcite @xcite . it was also shown that a chatbot could provide richer intervention when combined with behavior monitoring technology, such as mobile or wearable tracking tools that enable real-time monitoring of user activity @xcite . such developments in physical activity and diet change promotion chatbots have contributed to our understanding of the feasibility and effectiveness of chatbot-delivered interventions. to this extent, the existing studies using chatbots for interventions mainly focused on examining the effectiveness of chatbot-delivered strategies (e.g., intervention messages) on physical activity and diet outcomes. although users' conversational inputs can be valuable to successful interventions, previous studies lacked discussion of how users' conversational inputs during the interventions, such as their reflections of behaviors and environments, may have affected the outcomes @xcite . hence, a quantitative analysis of user responses to the chatbot's messages is necessary to better grasp the bot and users' conversational patterns and how they lead to positive outcomes. in this study, we investigate the effects of barrier and support strategies used by the trainer and participants during a 3-month physical activity intervention program and on the intervention outcomes (i.e., changes in participants' physical activity-related barriers and social support). in addition, we explore whether participants' baseline weight (i.e., one's weight before the intervention) would influence the amount of weight-related strategies they mentioned in the conversations. ## dataset this paper used the data collected from the mobile phone-based physical activity education program (mped) study in community-dwelling women aged 25 to 69. the study protocol was approved by the university of california, san francisco, committee on human research, and the mped data and safety monitoring board. detailed descriptions of the study design and outcomes have been previously published @xcite . in brief, the mped trial was an unblinded, parallel randomized clinical trial (rct) conducted with three groups (control, regular, and plus groups). in this study, we used the data from the intervention groups (regular and plus groups) who received the identical physical activity intervention, consisting of brief in-person counseling sessions, an accelerometer, and the mped trial app for the first three months. at the baseline visit, research staff collected participants' sociodemographic information (e.g., age, education, marital status, employment, and racial/ethnicity), assessed participants' weight, and administered the barriers to being physically active quiz and the social support and exercise survey. the barriers to being physically active quiz developed by the centers for disease control and prevention (cdc) @xcite is a 21-item measure assessing the following barriers to physical activity: 1) lack of time, 2) social influence, 3) lack of energy, 4) lack of willpower, 5) fear of injury, 6) lack of skill, and 7) lack of resources (e.g., recreational facilities, exercise equipment). each domain contains three items, with a total score range of 0 to 63, with higher scores indicating more barriers. respondents rate the degree of activity interference on a 4-point scale, ranging from 0="very unlikely" to 3 = "very likely." the social support and exercise survey was used to assess both friend and family social support related to physical activity during the past three months @xcite . the measure consists of two subscales (friend and family support subscales). each subscale has 13 items with 5-point likert scales (ranging from 1="none" to 5="very often"). the ratings of all 13 items were summed for a subtotal score. scores can range from 13 to 65, with higher scores indicating more support. women who met eligibility criteria (a.1) and were randomized to the intervention groups received brief in-person physical activity counseling by trained research staff. all counseling sessions were digitally recorded. the average length of the counseling was 28.8 (sd 6.6) minutes. we randomly selected 107 sessions and had the audio recordings transcribed verbatim by a professional transcriptionist. on average, the trainers and participants spoke 213.91 and 209.63 turns respectively per session. the average sentence length and the average words per sentence from the trainers (397.07 sentences and 10.02 words/sentence) are longer than the participants' (277.67 sentences and 5.99 words/sentence). this is understandable as the trainers were supposed to deliver physical activity educational content during the counseling. after three months, the barriers to being physically active quiz and the social support and exercise survey were administered again to assess the changes (from 3 months to baseline) in these measures. among the 107 transcribed dialogs, two dialogs were dropped due to missing survey results, 17 dialogs (7,808 sentences) were randomly picked for annotation, and the remaining 88 dialogs (63,288 sentences) were used for classifier pretraining and data analysis. since releasing the original interview data is not approved by our irb and hippa, we created and released 44 simulated dialogs (772 sentences) based on the original interview data for our community to use. (more statistics are listed in appendix a.3). ## annotation scheme after the data collection, we developed an annotation scheme to categorize different conversational behaviors used by trainers and participants systematically. the annotation scheme largely consisted of intervention-related categories and general conversational categories. intervention-related categories included domain categories which were used to segment larger stretches of the conversations by topic. in addition, categories pertaining to specific strategies used during the intervention were included. for general conversational categories, we included social exchange and taskfocused exchange categories that were borrowed from the roter method of interaction process analysis @xcite . based on our annotation scheme, we annotated the in-person counseling sessions on a per-sentence level (sentences have been obtained using nltk's punktsentencetokenizer) across four different dimensions: domain, strategy, social exchange, and task-focused exchange. a utterance domain strategy 1 strategy 2 social exchange task-focused t: so again your long-term goal, you'll reach ten thousand steps at week seven and to maintain it from there. goal goal none none give-geninfo p: okay. goal none none agree none t: so how confident do you feel that you can meet your longterm each week? goal selfefficacy goal none ask-opinion p: i feel confident. goal selfefficacy none none give-opinion t: okay, great. goal none none agree none t: so to break it down a little bit more for you, ten minutes brisk walking is gonna give you about a thousand to twelve hundred steps. goal monitoring none none give-geninfo p: okay. goal none none agree none t: so, think about brisk walking as a pace where you can still carry a conversation, but you can't sing. ## strategy classifier to built a dialog system capable of delivering physical activity interventions, it was first necessary to understand patterns in human-delivered intervention counseling sessions. since the strategy dimension is intervention-related and represents each sentence's intention, in this study, we focused on examining how the strategy dimension influenced people's physical activity-related barriers and social support. therefore, we built a bert-based strategy classifier to leverage a large number of unannotated dialogs. we started with the bert-based model pre-trained on wikipedia. we fine-tuned the model with 63,288 unannotated utterances from the physical activity counseling sessions before training on the classification task. we then trained a singlelabel prediction model with the 17 annotated counseling sessions (7,808 sentences in total) using leave-one-out cross-validation, where each training unit was composed of one session. contextual information is crucial in dialog act predictions @xcite . hence, we considered the previous ten sentences as the dialog history. as an input to the model, we appended the history to the current sentence and used a special separate token to separate them. we then plotted the confusion matrix in figure 1 to analyze the results. we found that the main error came from the misclassification of relapse. relapse was sometimes classified as motivation mostly because people talked about recovering from relapse or staying motivated without giving up. for example, "i was doing yoga and pilates and needed to pick that up." mentions activities that motivate the participant to recover from relapse. another error was that motivation was sometimes mistaken as none due to the diverse activities trainers mentioned to motivate the participants. we used the model to classify all the 88 unannotated dialogs with 63,288 sentences. the statistics are shown in ## results we conducted pearson's correlation analysis to assess the relationship between the amount of barrier and support strategies and the changes in their cor-responding survey scores. we also performed multiple linear regression analysis to see the strategy's effect after controlling for social-demographic factors and baseline survey scores @xcite . the results are shown in as shown in dependent variable independent variable trainer + participant trainer participant pearson's r multple coeff. pearson's r multple coeff. pearson's r multple coeff. changes in barrier survey #. barrier strategy 0.28** 0.29** 0.19 0.40 0.27** 0.34* barrier survey baseline --0.57*** --0.58*** --0.56*** changes in support from friend survey #. support strategy 0.17 -0.06 0.06 -0.11 0.23* -0.08 support from friend survey baseline --0.28*** --0.19* --0.28*** changes in support from family survey #. support strategy -0.16 -0.08 -0.18 -0.11 0.10 -0.17 support from family survey baseline --0.19* --0.19* --0.18* marriage (married) -3.57* -3.68* -2.33 ethnicity (multi-race, black and hispanic) --4.84* --4.95* --6.59* #. weight strategy weight baseline 0.13 0.012 0.00 0.00 0.21* 0.01 the results also showed a higher number of barrier strategies from the participants significantly predicted fewer decreases in barrier survey score (@xmath6.27, @xmath7.01). the multiple regression analysis (r 2 = .38, f (8, 79) = 6.16, @xmath8.001) showed similar results (β = 0.34, @xmath9.032). this was interesting since the more the participants talked about their barriers, they were less likely to overcome their barriers in the end. this could mean that talking about barriers may not necessarily help them overcome them. rather, turning the conversation to more future-directed, action-based suggestions may be more beneficial. thus, for future chatbot development, if a participant tends to talk too much about barriers, the bot should stop discussing barriers to avoid negative effects. we also found that the participants with a higher barrier score at baseline visit overcame more barriers after three months (β = -0.56, @xmath10.001 adjusted with bonferroni correction). as discussed above, this may be due to the fact that they had more room for improvement. however, there was no significant interaction between the amount of barrier strategy and barrier survey baseline score (f (9, 78) = 6.18, @xmath11.s.). the effect of the barrier strategy from the combination of both trainer and participant showed similar results to the participants only. ## conclusions and future work in this work, we presented the foundation work on building an automatic physical activity intervention chatbot. a human-human physical activity intervention dialog dataset was created from a real intervention setting. we also designed a set of com-prehensive annotation schemes and annotated the dataset at the sentence level. a strategy classifier with context embedding was shown to achieves good results on intervention strategy detection. the analyses showed that the amount of barrier and support strategies used in the intervention were correlated with the changes in the corresponding score, and the effects differed based on participants' baseline score and socio-demographic. we also found that people with a heavier weight at the beginning tend to talk more about weight. given the analysis result, we provided suggestions on designing a behavior-change intervention chatbot that could adapt to different individuals to yield better outcomes. this project lays the ground for the next step, which is to build a physical activity intervention chatbot that can effectively choose appropriate strategies based on user profiles and survey baseline result information to increase the intervention's effectiveness. in addition, although the main focus of this study was to investigate the association between intervention strategies and physical activity outcomes, social exchange and task-focused categories would also provide useful insights for identifying more effective conversational patterns in future studies. for example, social-exchange categories provide information on patients' acceptance towards strategies used by healthcare providers. task-focused categories inform the exchange of information and opinions. by combining social exchange and task-focused categories with strategy information, we will be able to provide richer content and context to our interpretation of the conversation. since the findings in our study are exploratory, we will also confirm the multiple hypotheses in the following study as pre-hoc hypotheses.