Detection of Mental Health from R eddit via Deep Contextualized Representations (2020)

Abstract: We address the problem of automatic detection of psychiatric disorders from the linguistic content of social media posts. We build a large scale dataset of Reddit posts from users with eight disorders and a control user group. We extract and analyze linguistic characteristics of posts and identify differences between diagnostic groups. We build strong classification models based on deep contextualized word representations and show that they outperform previously applied statistical models with simple linguistic features by large margins. We compare user-level and post-level classification performance, as well as an ensembled multiclass model.

## introduction global prevalence of mental disorders has been estimated at 29.2% in a meta-study of 174 surveys across 63 countries @xcite . mental illness is one of the leading causes of disability globally and the costs of mental health treatment have run into the trillions of dollars @xcite @xcite . additionally, individuals suffering from mental illness are estimated at forming 14.3% of deaths worldwide, significantly higher than a control population @xcite . limited mental health resources and funding have necessitated new approaches to addressing the global impact of this problem. however, early detection of mental illness and early intervention have shown promising results for improving treatment and long-term outcome results for many psychiatric disorders; these have the potential to reduce the costly burden that mental illness has placed on our society as well as our global economies @xcite @xcite . advances in artificial intelligence in general and computational linguistics in particular have made important contributions to detecting and predicting mental illness among the population, particularly in social media @xcite . using computational linguistics, researchers have been able to leverage the widespread use of social media to analyze large, publicly available datasets for identifying linguistic markers of mental illness. to date, unique linguistic markers and patterns have been identified for several psychiatric conditions, such as major depressive disorder (mdd) @xcite , general anxiety disorder (gad) @xcite , bipolar disorder (bd) @xcite , eating disorders (ed) @xcite , schizophrenia (sz) @xcite @xcite , obsessive compulsive disorder (ocd) @xcite , posttraumatic stress disorder (ptsd) @xcite , as well as others @xcite . linguistic findings have spanned various domains of language, including the use of pronouns, emotion words, tentative language, tangentiality, punctuation, and content analysis. the majority of these models have been developed to successfully predict if a given user has self-disclosed receiving a diagnosis for a psychiatric condition and is currently suffering with mental illness. however, much of this previous research on social media and mental health has focused on comparing users with particular disorders with control users. in this work we expand this focus to compare across a wide set of common disorders. this is directly applicable to real-world diagnostic scenarios, where clinicians select a diagnosis from a large set of disorders, rather than simply diagnosing an individual as healthy or not. in addition, prior work has focused on data collection and analysis, with less emphasis on building strong predictive models. in this work, we apply state-of-the-art neural network models developed for other natural language tasks to the problem of mental health detection from social media. ## related work in recent years, there has been increased interest in the nlp community in the automatic detection of psychiatric conditions from language. many researchers have focused on analyzing vast amounts of language from social media posts to study mental health @xcite @xcite . with the advent of social media, many people who suffer from various forms of mental illness have found a sense of community and support, and these platforms offer a mode of expression for discussing their experiences openly online. additionally, many online platforms allow users to post anonymously, giving them a sense of security and anonymity to discuss their experiences and struggles without the fear of being stigmatized or discriminated against @xcite @xcite . in order to analyze language patterns related to various disorders from social media data, researchers have developed innovative approaches for automatically labeling this data. @xcite developed a widely used approach for gathering data for a range of psychological disorders, using regular expressions to identify public self-disclosures of diagnoses on social media. they tested this approach using twitter data and collected a dataset of tweets from individuals with bipolar, depression, ptsd, sad, and a control group. they analyzed several linguistic features across conditions using a clustering algorithm and built predictive classifiers to distinguish between diagnosed and control users. @xcite expanded this approach to study a larger set of disorders using reddit data. reddit is one of the fastest growing and widely used social media platforms, averaging over 330 million active monthly users, and, as of 2018, was the fourth most visited website in the us @xcite . unlike twitter, reddit imposes no limits on the length of posts, enabling an analysis of longer language samples. in addition, reddit is composed of subreddits, which are forums dedicated to specific topics, and there are many subreddits related to specific mental health conditions. they collected a large dataset of reddit posts and analyzed linguistic features between different conditions and a control group. they also trained binary classifiers to distinguish between each condition and the control. our work directly builds on this prior work. following @xcite and @xcite , we collect a large expanded dataset of reddit posts. unlike prior work, we do not focus on pairwise analyses of linguistic features between conditions and the control group; rather, we compare features between conditions to highlight important differences that can distinguish between various disorders. while others have trained simple predictive models of these disorders, we instead use state-of-the-art deep contextualized models that have been highly successful across several nlp tasks. our work makes important contributions to the problem of mental health detection from social media data and provides insights for others to further build on this work. ## data collection we focus in this study on 8 mental health conditions: schizophrenia (sz), borderline personality disorder (bpd), post-traumatic stress disorder (ptsd), eating disorder (ed), major depression disorder (mdd), general anxiety disorder (gad) and bipolar disorder. while datasets for many of these conditions have been collected on varying scales, to the best of our knowledge our dataset includes the largest cohort of users whose posts have been collected for many of these conditions. to build this cohort, we collect users with self-identified mental health conditions from reddit using the pushshift api foot_0 . we search for users in mental health related subreddits and use keywords to search for mental health related words. our distant labeling approach is further explained below, in section 3.1. we also identify a group of control users who do not have any of the targeted conditions. we first collect a large scale user pool by scraping posts from common subreddits like r/askreddit, and filter the control users by process described in subsection 3.2. the number of posts collected in each condition is shown in ## linguistic indicators of mental health after collecting and preprocessing the data, we analyzed linguistic characteristics of mental health using linguistic inquiry and word count @xcite . liwc is a text analysis program that computes word counts for semantic classes and structural features. it relies on an internal dictionary that maps words to psychologically motivated categories. these include standard linguistic features (e.g. percentage of words that are pronouns, articles), markers of psychological processes (e.g. affect, social, cognitive words), and punctuation categories (e.g. periods, commas). liwc dimensions have been used in many studies to predict outcomes including personality (pennebaker and king, 1999), deception @xcite , and health @xcite . we extracted 73 features using liwc 2015; a full description of these features is found in @xcite . to construct a single feature vector per user, we concatenated all posts per user and then extracted the liwc features from the combined posts and performed length normalization. prior work on identifying linguistic indicators of mental health has compared liwc features from users' individual disorders with healthy control users. however, it is often unclear whether the findings are specific to the disorder, or if they are indicative of mental disorders more generally. for example, in pairwise analyses, personal pronoun usage has been found to be increased in individuals with schizophrenia @xcite . however, this pattern might or might not be specific to schizophrenia, but may be indicative of other mental disorders as well. because of this gap in prior work, we began by comparing liwc features directly across the 8 diagnostic groups and the control group. figure 2 shows a heatmap of the z-score normalized average liwc features across users in each group. the x-axis shows the 8 diagnostic groups and the control group, and the y-axis shows the normalized liwc feature values. the color of each cell indicates whether the scaled value is high (blue), low (red) or average (white). as shown in this figure, the control group has the greatest number of red features, or liwc features which have a low frequency. it is clear from the figure that the 8 diagnostic groups have different language usage patterns from the control group, and particularly show a higher frequency for several linguistic dimensions. further, there seem to be several interesting similarities and differences in linguistic patterns across the diagnostic groups. to further investigate these differences, we ran one-way anovas comparing each liwc feature across the 8 diagnostic groups and the control group. to correct for family-wise type i errors, we used bonferroni correction. the results indicated that there were significant differences across groups for all 73 liwc variables. we ran tukey posthoc tests to identify which pairs of conditions were most similar and most different. because of limited space, we focus here on the linguistic dimensions with the greatest variance among the groups, indicated by the highest f-statistics. these categories were anx, the use of anxiety words (f(8, 24442) = 531.911, p<.0001), and i, the use of the first person singular pronoun (f(8, 24442) = 438.738, p<.0001). figure 3 shows the results of the posthoc analysis. pairwise comparisons among psychiatric conditions revealed several interesting findings. while each condition differed significantly for both features when compared with the control group (users in the control group were significantly less likely to use anxiety related words and "i" when compared to each condition), when comparing between the psychiatric conditions differences varied. for example, users with sz were significantly less likely to use anxiety related words in comparison with other groups. another interesting finding was that users with bpd used 1st person singular pronouns significantly more than all other psychiatric conditions with the exception of ed. these findings shed light on linguistic variation across different psychiatric conditions, and provide further motivation for developing methods to distinguish between individuals with these disorders by leveraging social media posts. ## methods for classification experiments having identified significant differences in linguistic features between the disorders, and between the control users, we next explore several classification methods for automatically identifying different mental health conditions. previous efforts to identify such conditions in reddit have primarily employed simple logistic regression or svms using a bag-of-words representation or liwc features, and some have explored rnn/cnn based text encoder models @xcite @xcite . however, recent advances in contextual representations like elmo @xcite and bert @xcite , which have enabled substantial performance increases across many nlp task, have not been well integrated into mental health identification tasks. this is due to model size and scalability issues of the large number of posts generated by a user. in this work we focus on methods utilizing contextual representations for mental health identification and compare their effectiveness to a logistic regression baseline trained on liwc features. we present an attention-based model using bert representations as input features, as well as a realmlike model @xcite inspired by recent advances in open domain question-answering. all of these models are trained for a user-level classification task, to detect whether a user has a particular diagnosis, based on an aggregated representation of their posts. in addition, we conduct post-level classification experiments with classic bert finetuning settings. this experiment is done to discover the importance of global context in classification. finally, in addition to these binary classifiers (diagnosis vs. control), we ensemble all our binary classification models as a multi-label classifier among different diagnostic groups, which is the ultimate goal for the application of this work. for user-level classification, we select users not belonging to the co-morbidity group as a control group. to reduce the size of the data, we remove posts less than 50 characters long from both the mental group and the control group, as we hypothesize that these may not provide enough information for classification. also we do not include control users with fewer than 20 non-sensitive posts. when pairing with mental health users, we select control users who have a similar total number of posts, who do not have mental health sensitive posts, and who have at least some subreddit overlap with the mental health users, as described by @xcite . we consider two experimental settings: for the cl (clean) experiments we exclude all mental health sensitive posts for mental health users, and for the uncl (unclean) experiments we include these posts for their corresponding users. the intuition is that under the uncl setting, our model should be able to make predictions based on some explicit semantic triggers, thus resulting in better performance. however, under the cl setting, the model may rely on underlying syntactic differences that may generalize better than explicit semantic features. below we describe the attention-based model and the realm model that we adapt for this work. ## mental health detection results in this section we present the results for user-level and post-level binary classification, and for the multiclass ensemble classification. ## conclusions and future work in this paper we collect and analyze a large scale dataset of social media posts from users various mental health conditions. we analyze linguistic characteristics of the posts, directly comparing the features of the various conditions. we build strong classification models based on deep contextualized representations and demonstrate that they outperform the liwc feature based logistic regression baseline by a large margin. although the liwc feature representation is not as useful for classification, it is a useful representation for analysis of posts to gain insight about the differences between groups. our experimental results show that linguistic traits for mental health detection are more easily recognized at the user-level and thus effectively aggregating post-level signals is crucial to accurate prediction. also, we find that these contextualized representations rely heavily on semantic content and always perform better when semantic indicators are obvious. we also show that the prediction scores of our classification models, even the accurate ones, are not well calibrated and thus are not an accurate uncertainty estimator of mental health risk. these results call for a more interpretable model for mental health detection. future research may look into the direction of learning better deep features and exploring additional classification paradigms to further improve performance for this impactful problem.