Crisis counselor language and perceived genuine concern in crisis conversations (2024)

Abstract: Although clients’ perceptions of therapist empathy are known to correlate with therapy effectiveness, the specific ways that the therapist’s language use contributes to perceived empathy remain less understood. Natural Language Processing techniques, such as transformer models, permit the quantitative, automated, and scalable analysis of therapists’ verbal behaviors. Here, we present a novel approach to extract linguistic features from text-based crisis intervention transcripts to analyze associations between specific crisis counselor verbal behaviors and perceived genuine concern. Linguistic features associated with higher perceived genuine concern included positive emotional language and affirmations; features associated with lower perceived genuine concern included self-oriented talk and overuse of templates. These findings provide preliminary evidence toward pathways for automating real-time feedback to crisis counselors about clients’ perception of the therapeutic relationship.

## introduction in the context of mental health interventions, an extensive body of research has found significant associations between therapists' behavioral traits and clinical effectiveness. in particular, clients' perceptions of therapist empathy correlate with positive therapy outcome, as documented in various contexts, including addiction treatment, suicide intervention, and cognitive behavioral therapy for anxiety @xcite @xcite . therapist empathy and genuineness are closely connected and are frequently evaluated together in relation to therapy outcomes @xcite . in an analysis of survey data from crisis text line's text-based crisis interventions, @xcite found significant associations between texters' ratings of the overall effectiveness of the intervention and their perceptions of the counselor's genuine concern. higher genuine concern ratings also were associated with greater texter reports of reduced suicidality over the course of the conversation. despite its relevance for therapy effectiveness, developing methods to recognize and improve perceived empathy remains technically demanding due to its complex, context-dependent nature @xcite . a few studies have explored how empathy manifests in therapists' language. however, many of these rely on qualitative conversation analyses @xcite , time-intensive manual annotation of data @xcite , or observer ratings rather than client feedback to gauge empathy @xcite . recent advancements in natural language processing (nlp) offer a new avenue for quantitative, automated, and scalable analysis of therapists' verbal behaviors in relation to perceived empathy. yet, academic investigation with computational approaches remains sparse and often depends on proprietary linguistic analysis tools like linguistic inquiry and word count (liwc) @xcite , which do not offer the flexibility to extend beyond predefined word categories. leveraging a large pool of text-based crisis conversations, we employed nlp techniques to extract over 100 linguistic features and examine their relationship with texter-rated genuine concern in conversations with trained volunteer crisis counselors. ## data we used data from crisis text line, a non-profit organization that provides free, confidential mental health support and crisis intervention through text-based communication. all conversations are de-identified before analysis. data are automatically scrubbed of identifying information prior to access by this research team. this research was evaluated by sterling irb, which issued an exempt determination. after each conversation, texters complete an optional post-conversation survey where they evaluate crisis counselors' genuine concern. the survey item asks respondents to rate their agreement with the statement: "in the conversation, i believe my crisis counselor was genuinely concerned for my well-being". response options include 'strongly agree', 'somewhat agree', 'neither agree nor disagree', 'somewhat disagree' and 'strongly disagree'. we combined multiple message-level and conversation-level data sources from crisis text line's dataset to identify features associated with these ratings. the analysis was restricted to english-language conversations, excluding cases where the counselor indicated that the texter's intent was to prank or test the system. additionally, because some texters have multiple conversations, we included the texters' first conversation, starting from 2017. we considered only conversations featuring at least five counselor turns (defined as concatenated consecutive messages sent by one conversational participant). 404,017 conversations met these criteria and had post-conversation genuine concern data between 2017 and 2023. ## methods the primary goal of this analysis was to identify the relationship between crisis counselor (cc) linguistic cues and the perception of genuine concern by individuals in crisis. features were selected a priori on the basis of a literature review regarding language efficacy in counseling relationships and existing psychotherapy-related nlp work @xcite @xcite @xcite @xcite @xcite @xcite , in consultation with clinical experts (eao). we included many features related to alignment between the texter's and cc's language use, at the turn-by-turn conversational level (for a full list of features, see the appendix). we used natural language processing (nlp) methods to extract linguistic features, including: we converted the dependent variable, perceived genuine concern, into a binary format by labeling "strongly agree" and "somewhat agree" responses as positive, and all other response options as negative, similar to the approach used by @xcite . the data showed a significant imbalance with 86% of responses being positive. to address this, we employed stratified random sampling to select 50,000 conversations for the training set, balanced across the five original likertscaled answer options. for the test set, our sampling mirrored the overall base rates of each response option, resulting in an imbalanced set of 10,000 conversations. we trained logistic regression with lasso penalty @xcite and xgboost @xcite models with bayesian optimization over hyper parameters in a 5-fold cross-validation setup 7 . the core goal of the project was to identify which linguistic features of cc utterances most strongly contribute to perceptions of genuine concern. we chose these models specifically because they allow us to readily identify how much each feature contributes to the prediction. given the imbalanced nature of the test set and the importance of correctly classifying both classes, we assessed model performance using balanced accuracy. this metric represents the arithmetic mean of sensitivity and 7 we used the scikit-learn (v1.1.1) library for statistical models and the scikit-optimize (v0.10.1) library for parameter tuning. parameter tuning was conducted using the bayessearchcv module with uniform priors. the optimal penalty for the lasso model was found to be @xmath0.018. ## results we used the 80 out of 140 features that the lasso model did not shrink to zero and refitted the logistic model using these features. the variables were standardized. conversations with a more positive emotional tone from the cc (mean and maximum valence) were associated with greater perceived genuine concern. at the same time, maintaining alignment with the texter's valence throughout the conversation is crucial: a substantial difference in turn-by-turn valence between cc and texter correlates with poorer perceptions of genuine concern. conversations with a higher cc word count tend to receive better ratings. moreover, high mismatch between cc and texter in overall and turn-by-turn word count are associated with lower genuine concern. ccs maintaining a level of formality equal to or exceeding that of texters tend to be rated more positively. however, using significantly more complex language than the texter may lead to perceptions of lower genuineness. the clinical content of cc messages reveals significant associations. affirmations, including messages such as "you should be proud of yourself for [...]", are associated with higher ratings of genuine concern. conversely, self-disclosures, where ccs disclose personal information like "i've also been in situations where [...]", tend to decrease the likelihood of a positive rating. similarly, the use of firstperson singular (e.g., 'i,' 'me') and first-person plural (e.g., 'we,' 'us') pronouns was associated with lower genuine concern, indicating that ratings tend to decrease when ccs talk about themselves. ## discussion our results indicate that the relationship between perceived genuine concern and counselors' language is often context-dependent, with significant features reflecting differences between counselor and texter language. this conclusion is consistent with literature showing significant associations between perceived empathy and and therapist-client synchrony in specific aspects of language style @xcite . additionally, our results demonstrate that maintaining conversational boundaries between counselor and texter can help perceived genuineness. language that is less formal than the texter's corresponds with lower perceptions of empathy, in line with the findings of @xcite . the literature shows mixed findings on the helpfulness of counselor self-disclosure @xcite . in this unique text-based counseling setting, the use of self-disclosing messages and firstperson pronouns were negatively associated with perceived genuine concern. this work demonstrates the feasibility of using nlp techniques to extract clinically relevant linguistic features from text-based mental health interventions. specifically, we fine-tuned complex transformer models for regression and classification tasks to extract features. many of these features showed significant associations with perceived genuine concern. our findings align with those of @xcite , who documented the positive effect of using affirmations. moreover, our results on the positive associations between counselor-texter synchrony in emotional valence and genuineness complement similar findings on synchrony in vocally encoded arousal and empathy @xcite . this foundational work identifies which linguistic cues are most strongly associated with genuine concern; future work could evaluate whether training ccs in how to increase these cues drives perceptions of higher genuine concern. ## limitations first, while our study found significant associations between linguistic features and perceived genuine concern, establishing causality requires further investigation. for example, a higher mean counselor valence might occur in conversations where texters are already feeling better, leading counselors to make more positive statements. second, this analysis incorporated results from an accessory model that evaluates emotional valence. formal quantitative evaluation of that accessory model's performance is still pending; this requires extensive hand coding. the emotional valence model was trained using a synthetic dataset produced by gpt 3.5; therefore, the synthetic data generation process may not be fully reproducible. third, regarding data sharing: while the dataset is de-identified, due to the highly sensitive content in messages, it is not publicly shareable, limiting reproducibility. crisis text line has a research collaboration program that allows research teams to work with the dataset through an application process. fourth: the features are interrelated (correlation matrix: in appendix). we evaluated multicollinearity and used a lasso approach for the logistic regression. however, we did not examine potential moderation effects, which could be an important direction for future work. fifth: because the purpose of the current analysis was to identify features that relate to genuine concern, we did not compare our model to other approaches such as zero-shot and few-shot llm-based models. finally, we binarized the outcome variable (genuine concern). we initially planned to approach this using the fully granular (5-point) outcome. however, lack of sufficient data populating the lower ranges of the outcome limited our ability to take this approach. ## ethical considerations despite the lack of clarity regarding causality, our model offers a foundation for developing tools to predict genuine concern in real time. however, there are potential risks associated with using the model to guide counselors. inaccurate predictions could mislead counselors into taking actions that may not effectively improve perceived genuine concern. additionally, if counselors are incentivized to achieve high scores, they may focus on enhancing specific linguistic features while neglecting other important aspects not addressed in this study that contribute to perceived empathy. our sample of conversations includes texters from a wide range of ages, races, ethnicities, genders, and sexual orientations. we measured racial and gender bias by evaluating model performance across different demographic groups, using selfreported data from survey items where multiple selections were allowed. we drew 2,000 bootstrap samples from the test set and constructed a 95% confidence interval for balanced accuracy. figure 1 shows that no significant difference in model performance was observed across these groups. we acknowledge the potential for bias in the datasets used to train our feature-generating models, such as those for emotional valence detection and miti technique labeling. a formal assessment of these biases has yet to be conducted. for further discussion of the replicability of our findings across race/ethnicity and gender, please see the appendix.