Comparing emotion feature extraction approaches for predicting depression and anxiety (2022)

Abstract: The increasing adoption of message-based behavioral therapy enables new approaches to assessing mental health using linguistic analysis of patient-generated text. Word counting approaches have demonstrated utility for linguistic feature extraction, but deep learning methods hold additional promise given recent advances in this area. We evaluated the utility of emotion features extracted using a BERT-based model in comparison to emotions extracted using word counts as predictors of symptom severity in a large set of messages from text-based therapy sessions involving over 6,500 unique patients, accompanied by data from repeatedly administered symptom scale measurements. BERT-based emotion features explained more variance in regression models of symptom severity, and improved predictive modeling of scale-derived diagnostic categories. However, LIWC categories that are not directly related to emotions provided valuable and complementary information for modeling of symptom severity, indicating a role for both approaches in inferring the mental states underlying patient-generated language.

## introduction almost 10% of adults in the united states receive mental health counseling @xcite . the principle of measurement-based care dictates that medical treatments should be initiated and evaluated over time based on repeated assessments of patient symptoms and symptom trajectory @xcite . in the context of talk therapy, mental health practitioners estimate treatment progress based on patients' current and historical verbal communications. for evaluating depression and anxiety severity, expressions of emotional state are key aspects of such communications @xcite @xcite . while prior work predominantly focused on sentiment, i.e. positive/negative polarity, expression of fine-grained emotions @xcite may give further insights into depression and anxiety symptomatology. for example, pride may be impacted by depression in a unique way. @xcite showed that pride, a positive emotion relating to the self, is inversely correlated with depression, which is often associated with a poor self-image. at the same time, they found a smaller effect on joy and amusement, concluding that grouping these emotions into "positive affect" may result in a loss of nuance. the increasing adoption of digital mental health tools and services, particularly message-based therapy, has afforded new opportunities to assist practitioners in quantifying depression and anxiety severity by assessing emotion in patient-generated text. linguistic inquiry and word count (liwc) @xcite is a software package designed to count words belonging to pre-defined categories with an extensive track record of validation for the detection of linguistic indicators of mental state @xcite . it is commonly used to measure positive and negative affect, a limited set of specific emotions (sadness, anxiety, and anger), and other linguistic dimensions related to style and topic. several liwc categories have established relationships with depression, including the affect category sadness (e.g. "sad", "cry", "suffer"), the topic category health (e.g. "alcohol", "rash", "self-care"), and the syntactic category firstperson pronouns (e.g. "i", "me", "my"). liwc has been used to measure depression levels in social media posts @xcite @xcite , therapy conversations @xcite , and other written texts @xcite . liwc measurements have also been shown to distinguish between patients with depression and those with anxiety disorders @xcite , correlate with self-reported measures of anxiety and worry in written descriptions of emotional responses to covid-19 @xcite , and predict whether posts emanated from anxiety-related subreddits @xcite . however, word counting methods cannot address linguistic phenomena such as negation ("not bad"), sarcasm, and context-dependence (for example, in the case of polysemy, words have multiple meanings that can only be disambiguated in context), and manually defined dictionaries may omit synonyms for terms they encode. prior work suggests that neural network (nn)-based natural language processing (nlp) techniques can account for such phenomena and may therefore improve upon this straightforward word-counting method in their ability to identify concepts related to symptom severity. shen and rudzicz found that the performance of machine learning models identifying whether or not reddit posts were drawn from anxiety-related subreddits improved when these models included neural word embeddings rather than liwc-derived features (2017). however, the distributed representations of posts used in this work do not relate directly to interpretable emotion features. further, contemporary transformer-based nn language models offer advantages over neural word embeddings in their ability to leverage proximal cues (such as "not") when interpreting the contextual meaning of a word. as noted by the authors, this work suggests a need for further research on automated assessments of linguistic indicators of anxiety disorders, involving larger data sets and explicit diagnostic assessments. therefore, using a large set of messages from text-based therapy session, we investigated if emotions extracted using a bidirectional encoder representations from transformers (bert) @xcite based model trained on goemotions, a large dataset of reddit posts annotated with 27 fine-grained emotions @xcite , are stronger predictors of depression and anxiety status than counts of emotion-related word categories (liwc). to this end, we first determined the association of each feature with the outcomes of interest in univariate regression analyses. further, in order to provide clinical decision support to mental health practitioners, it is paramount to be able to classify previously unseen messages as indicating depression and/or anxiety. we therefore proceeded to train and evaluate a machine learning classifier using emotion features in conjunction with established depression-related liwc features to predict depression and anxiety status in a held-out test set. ## discussion in this work, we showed that neural network models such as the bert-based goemotions classifier can outperform liwc, a straightforward, broadly adopted word-counting method for extracting emotion features from natural language. we further confirmed that some emotions not traditionally associated with depression and anxiety can be predictive of these diagnoses; specifically, pride. finally, we showed that using liwc features together with emotion features derived using goemotions predict depression/anxiety status with reasonable accuracy. this finding is important, in that further development of such tools could lead to better detection of emotional change during treatment in a way that could be derived naturally in the client/clinician encounter. nlp applied to such naturalistic data has been used for measuring clinician skills in delivering psychotherapy with some success @xcite ; here, rather than using such tools for quality measurement, linguistic analysis of affect could be used to detect depression/anxiety severity and client response to treatment. depression affects individuals in many ways and expresses itself in various behavioral and thought patterns that may not be fully captured with the high-level categories of positive and negative affect. goemotions' main strength therefore lies in its ability to extract fine-grained features spanning the breadth of human emotion, capturing depressed individuals' emotional experiences comprehensively. the different emotion feature sets appeared to be somewhat complementary, as evidenced by the additive performance metrics shown in depression severity was associated with large differences in grief, pride, excitement, relief, and disgust. in agreement with generally lower reactivity @xcite , less excitement was predictive of depression. grief manifestations are similar to depression symptoms; though grief in itself is not pathological, it often co-occurs with depression @xcite . additionally, depressed individuals expressing less pride than their non-depressed counterparts might be expected on account of lower self-image, and matches findings presented by @xcite . caused by a perception of violations of moral and social norms, internally directed disgust, also termed self-disgust or self-loathing, has been reported to be associated with both depression and anxiety symptoms @xcite . we further found that increased disapproval -and conversely, decreased approval -were associated with anxiety symptoms. this may be explained by disturbances in interpersonal sensitivity and an inclination to be self-critical, which have been described as characteristic of anxiety non-emotion liwc features have established utility for predicting depression and anxiety. these features capture aspects of symptomatology outside emotion, such as increased self-focus, social isolation, and usage of health-related words. nonemotion liwc features would therefore be expected to be complementary to emotion features, and our work confirms that and leveraging both may achieve the best results. we trained a machine learning model using these features in conjunction with emotion features to predict depression (au-roc 0.671) and anxiety (auroc 0.657). that these models show similar performance using the same features to predict different outcomes may be explained by the large overlap in symptoms between anxiety and depression, e.g. both are characterized by negative self-talk and hopelessness. additionally, depression and anxiety are often comorbid; indeed, in this dataset, 74.5% of assessments with a gad-7 score above the diagnosis threshold also had a positive depression finding, and 70.6% of positive anxiety questionnaires also had a positive anxiety finding. there are important ethical considerations when analyzing patient-generated natural language to infer mental state. any passive monitoring of patientgenerated data may be considered invasive. due to the sensitive nature of personal health data, such data are subject to protections that do not apply to non-health data. when health-related insights are derived from data that may be neither private nor health-related (e.g. social media posts), obtaining informed consent and handling inferences with appropriate care is paramount. while academic studies such as the current work are governed by rigorous institutional ethics guidelines regarding consent and data sharing, different rules apply to healthcare organizations and commercial entities. the use of technologies such as the ones presented here may be acceptable if conducted by trusted entities, such as healthcare providers, in order to support care @xcite ; on the other hand, consumers may be wary of commercial entities conducting such analyses. further research, as well as applications of the findings presented here, must take such considerations into account. this work has several limitations. the data used here stem from predominantly female, young, and well-educated participants, and results may therefore not generalize to populations with a differ-ent makeup. if predictive algorithms were to be deployed in practice, fairness may be a concern if predictive performance differs for underrepresented groups. in addition, the goemotions dataset used to train the bert-based models is drawn from reddit, which has been shown to have a disproportionately high representation of young male users @xcite . though it is encouraging that models trained on these data produce features that correlate well with symptom severity in the current study, the development of annotated datasets drawn from a more diverse population may lead to models that better address linguistic and cultural differences in the ways in which emotions are expressed. several features used in the random forest classification model are expected to be highly redundant (e.g. goemotions cower sadness, goemotions ekman sadness, and liwc sadness; calculated negative emotion variables which are calculated using sadness). however, interdependent features should not affect the random forest's ability to leverage all features optimally to optimize predictive performance. this work enables and informs future work. we showed that bert-based emotion features are associated with depression and anxiety status; however, this work did not assess longitudinally if changes in emotion track with changes in depression and anxiety. while existing work demonstrated this relationship for depression-related liwc features @xcite , future work may aim to ascertain whether changes in emotion features over time also predict longitudinal patient trajectories. this work also informs feature selection for future work in depression and anxiety prediction. emotion variables can be obtained with a range of extraction approaches. our results indicate the goemotions variables may be a better choice than liwc for emotions. nevertheless, liwc features have a place in future work. liwc's syntactic and topic features were shown in prior work to be associated with depression scores as well as longitudinal patient trajectories and continued to demonstrate utility in this work. we determined that fine-grained emotions measured in the language of individuals are associated with and predict anxiety and depression status. the associations we found reflect previous findings. this work thus contributes evidence of the reliability of such measurement approaches, supporting the use of these methods in future work investigating the nature of depression and anxiety. for example, these features could aid investigations into depression phenotypes through cluster analysis, as well as psychology research investigating the differential expression of similarly-valenced emotions in depression and anxiety, e.g. by aiding data collection. additionally, this work has important clinical implications. measurement-based care is facilitated by periodic progress assessments, but additional data collection incurs additional workload. in textbased therapy, depression and anxiety status may instead be automatically determined from alreadyavailable patient messages. in clinical settings, interpretability is essential; thus, models based on interpretable features such as emotions may be preferred over black-box models classifying raw text directly. future work may therefore investigate opportunities to leverage emotion-based predictive models for clinical decision support. ## conclusion extraction methods differ in the quality of emotion features extracted. with the data and approaches presented here, emotion features extracted by the goemotions bert-based model not only explained more variance in univariate mixed-effect regressions, but also contributed significantly to predictions of depression and anxiety status by a random forest classifier. further, while non-emotion variables obtained from liwc remain valuable in linguistic modeling tasks, goemotions' level of granularity offers clinically relevant nuance that prevailing tools cannot capture.