Do Models of Mental Health Based on Social Media Data Generalize? (2020)

Abstract: Proxy-based methods for annotating mental health status in social media have grown popular in computational research due to their ability to gather large training samples. However, an emerging body of literature has raised new concerns regarding the validity of these types of methods for use in clinical applications. To further understand the robustness of distantly supervised mental health models, we explore the generalization ability of machine learning classifiers trained to detect depression in individuals across multiple social media platforms. Our experiments not only reveal that substantial loss occurs when transferring between platforms, but also that there exist several unreliable confounding factors that may enable researchers to overestimate classification performance. Based on these results, we enumerate recommendations for future mental health dataset construction.

## introduction in the last decade, there has been substantial growth in the area of digital psychiatry. automated methods using natural language processing have been able to detect mental health disorders based on a person's language in a variety of data types, such as social media @xcite , speech @xcite and other writings @xcite . as in-person clinical visits are made increasingly difficult by socioeconomic barriers and public-health crises, such as covid-19, tools for measuring mental wellness using implicit signal become more important than ever @xcite . early work in this area leveraged traditional human subject studies in which individuals with clinically validated psychiatric diagnoses volunteered their language data to train classifiers and perform quantitative analyses @xcite . in an effort to model larger, more diverse populations with less overhead, a substantial portion of research in the last decade has instead explored data annotated via automated mechanisms @xcite . studies leveraging proxy-based annotations have supported their design by demonstrating alignment with existing psychological theory regarding language usage by individuals living with a mental health disorder @xcite . for example, feature analyses have highlighted higher amounts of negative affect and increased personal pronoun prevalence amongst depressed individuals @xcite . given these consistencies, the field has largely turned its attention toward optimizing predictive power via state of the art models @xcite . the ultimate goal of these efforts has been threefold-to better personalize psychiatric care, to enable early intervention, and to monitor population-level health outcomes in real time. nonetheless, research has largely trudged forward without stopping to ask one critical question: do models of mental health conditions trained on automatically annotated social media data actually generalize to new data platforms and populations? typically, the answer is no-or at least not without modification. performance loss is to be expected in a variety of scenarios due to underlying distributional shifts, e.g. domain transfer @xcite . accordingly, substantial effort has been devoted to developing computational methods for domain adaptation @xcite . outcomes from this work often provide a solid foundation for use across multiple natural language processing tasks @xcite . however, it is unclear to what extent factors specific to mental health require tailored intervention. in this study, we demonstrate that at a baseline, proxy-based models of mental health status do not transfer well to other datasets annotated via automated mechanisms. supported by five widely used datasets for predicting depression in social media users from both reddit and twitter, we present a combination of qualitative and quantitative experiments to identify troublesome confounds that lead to poor predictive generalization in the mental health research space. we then enumerate evidencebased recommendations for future mental health dataset construction. ## domain adaptation in mental health domain adaptation (or "transfer") of statistical classifiers is a well-studied computational problem with high relevance across several areas of natural language processing @xcite . it is particularly useful in situations where acquiring ample training data for a target application is intractable (e.g. monetary, time constraints) or impossible (e.g. privacy constraints) @xcite . for example, in the sub-field of machine translation, significant effort is devoted to finding ways to effectively use large corpora of formal parallel text to train models for application in domains with informal and dynamic language, such as social media and conversational speech @xcite . traditional challenges encountered when transferring models between domains include variance in source and target class distributions @xcite , semantic misalignment @xcite , and sparse vocabulary overlap @xcite . fortunately, once these issues are identified, it is typically possible to decrease the transfer performance gap via methods such as structural correspondence learning, feature subspace mapping, and adversarial training @xcite @xcite . domain adaptation is of particular interest in the mental health space, where there exist numerous complexities in obtaining a sufficient sample of training data. for instance, the sensitive nature of mental health data necessitates extra care when creating and supporting new datasets @xcite . additionally, behavioral disorders are known to display variable clinical presentations amongst different populations, which can make identification of ground truth difficult @xcite . the latter point highlights the presence of label noise inherent in mental health data @xcite . this facet serves as one of two primary issues unique to this research space that may hinder attempts at domain transfer. indeed, prior work found that diverse and sometimes conflicting views humans have regarding suicidal ideation can make obtaining reliable gold-standard labels fundamentally challenging and lead to degradation in model performance @xcite . sampling-related biases present the other main area of concern for successful domain transfer by mental health classifiers. attributes such as personality, gender, age, and disorder co-morbidity have been found to significantly affect the presentation of mental health disorders in language data @xcite . moreover, the proxy-based annotation mechanisms used to label large social media data sets with mental health status invite the introduction of selfdisclosure bias into the modeling task @xcite . specifically, labels sourced from populations of individuals who self-disclose certain attributes may contain activity-level and thematic biases that cause poor generalization in larger populations @xcite . research leveraging text data for mental health status classification has primarily only considered a constrained form of domain transfer. in a withinsubject analysis, ireland and iserman (2018) examined differences in language usage by reddit users who had posted in an anxiety support forum within and outside mental health forums. similarly, wolohan et al. (2018) explored the predictive power of models trained to detect depression within reddit users as a function of access to text from explicit mental health related subreddits. both studies highlighted a mitigation of overt mental health discussion outside of the support forums, but still detected linguistic nuances in individuals with an affiliation to the mental health forums. @xcite in conjunction with other modalities of social data (e.g. profile metadata, images), the authors showed that signal from twitter was useful for classification on weibo. recent work from @xcite was the first to explore some of aforementioned difficulties with domain transfer in the mental health space. multiple different annotation mechanisms were used to train twitter-based models for identifying schizophrenia and then applied to facebook data from an independent population of clinically diagnosed schizophrenia patients. three different types of proxy signals with varying degrees of manual supervision were each found to generalize poorly to the clinical population. while the authors' analysis suggested the domains were similar enough to justify transfer attempts, only limited post-hoc analysis of the data platform effect was carried out. thus, it remains unclear to what extent the annotation methodologies as opposed to platform effects (or other confounds) caused the degradation. ## data we select depression classification as our task because it is perhaps the most widely studied, has multiple datasets from different platforms, and is of critical importance to society. estimated to affect 4.4% of the global population, depression presents a significant economic burden and remains the most common psychiatric disorder associated with deaths by suicide @xcite . occupying a lion's share of the computational literature, depression classification is a critical first target for evaluating generalization of mental health models in social media @xcite . to quantify the nature of domain transfer loss, we consider five datasets. datasets were selected based on their common adoption in the literature @xcite and their use of proxy-based annotations @xcite . we use two twitter-clpsych 2015 shared task @xcite , multi-task learning @xcite -and three reddit datasets-rsdd @xcite , smhd @xcite , and topic-restricted text @xcite . ## models we begin by training classification models for predicting depression on each dataset. all classification experiments leverage the same training procedure and features (see appendix d for details). as a classifier, we use ℓ 2 -regularized logistic regression. despite our model's relative simplicity we are able to achieve respectable within-domain classification performance while maintaining an ability to interpret learned parameters. logistic regression has served as a difficult benchmark to beat given access to appropriate engineered features for prior mental health studies @xcite . ## transfer experiments we conduct a series of experiments to measure the generalization of models between depression datasets and explain sources of model degradation. ## post-hoc analysis in the previous section, we identified the degree to which loss occurs under a variety of domain transfer settings. however, these settings do not account for all performance disparities. in this section, we measure differences between the datasets to understand the source of loss. ## recommendations we have demonstrated that issues of transfer loss persist in the mental health space, at least for the proxy-based social media datasets considered in our study. importantly, we identified confounds that emerge as a result of each dataset's respective design. critically, existing datasets have flaws that make them difficult to use for constructing models for new data types and populations. topical alignment. researchers must account for self-disclosure bias and confounds of personality when curating new datasets. first discussed in §6.2, models trained on the reddit datasets learn dependencies between support-driven topics, such as medication usage and relationship advice, and depression. in contrast, models trained on the twitter datasets identify the same correlations between sexuality, gender, and depression that redditbased models detect, but also learn about the recreational outlets (i.e. fantasy) and social concerns (i.e. racism, sexism) common in depressed individuals. we hypothesize that semantic divergences reflect self-disclosure bias and differences in platform interaction patterns @xcite . twitter's status-and reply-based structure serves as a place for individuals to share personal thoughts and experiences in reaction to their daily life. meanwhile, reddit's community-based forums require active engagement with specific topics and may silo individuals who wish to discuss their mental health beyond defined areas. the latter gains support from our analysis of subreddit distributions in the topic-restricted text data ( §6.4). topical nuances in language may appropriately reflect elements of identity associated with mental health disorders (i.e. traumatic experiences, coping mechanisms). however, if not contextualized during model training, this type of signal has the potential to raise several false alarms upon application to new populations. accordingly, we urge researchers to minimize the presence of overt topical disparities between classes in their datasets. ## limitations and future work though our study provides a robust perspective toward understanding generalization capabilities of mental health classifiers for social media, we recognize that more learning opportunities exist. our study only considers a handful of datasets, two platforms, a single mental health disorder, and homogeneous annotation mechanisms. still unexplored, in large part due to the precautions necessary for securing sensitive mental health data, is how well models trained on data from actual clinical populations generalize to proxy-based datasets and other clinical populations. while high co-morbidity rates between depression and other mental health disorders may allow us to infer model behavior for alternative conditions, we also recognize that presentations of different psychiatric disorders can be quite variable and warrant their own research @xcite . another limitation in our work is the lack of depression to control group matches from original reference material. @xcite and @xcite demonstrate that mental health disorders such as depression can have variable presentations based on demographic attributes. the attributes used to construct our twitter datasets originally were inferred via now-outdated text-based models. accordingly, demographic inference errors may be propagated to and correlated with depression classification errors. moreover, these attributes were not considered within the construction of any of the reddit datasets we explored. the effect of demographics on generalization remains a valuable insight for future exploration. finally, our attempts at domain transfer are constrained. namely, we do not invoke explicit domain adaptation methods @xcite @xcite . moving forward, we plan to explore algorithmic strategies to mitigate the biases discovered in this study. group were required not to have posted in a list of 24 mental health related subreddits or to have used any of 19 mental health terms. to align the theme of language generated by individuals across classification groups, each individual in the depression group was greedily matched with 12 individuals from the candidate control pool based on hellinger distance between each individual's post distribution over subreddits. to preserve privacy of individuals within the dataset, usernames were anonymized and post metadata was redacted. accordingly, linkages between each individual within the depression group and their respective control group pairs could not be recreated.