Exploring Hybrid and Ensemble Models for Multiclass Prediction of Mental Health Status on Social Media (2022)

Abstract: In recent years, there has been a surge of interest in research on automatic mental health detection (MHD) from social media data leveraging advances in natural language processing and machine learning techniques. While significant progress has been achieved in this interdisciplinary research area, the vast majority of work has treated MHD as a binary classification task. The multiclass classification setup is, however, essential if we are to uncover the subtle differences among the statistical patterns of language use associated with particular mental health conditions. Here, we report on experiments aimed at predicting six conditions (anxiety, attention deficit hyperactivity disorder, bipolar disorder, post-traumatic stress disorder, depression, and psychological stress) from Reddit social media posts. We explore and compare the performance of hybrid and ensemble models leveraging transformer-based architectures (BERT and RoBERTa) and BiLSTM neural networks trained on within-text distributions of a diverse set of linguistic features. This set encompasses measures of syntactic complexity, lexical sophistication and diversity, readability, and register-specific ngram frequencies, as well as sentiment and emotion lexicons. In addition, we conduct feature ablation experiments to investigate which types of features are most indicative of particular mental health conditions.

## introduction mental health is a major challenge in healthcare and in our modern societies at large, as evidenced by the topic's inclusion in the united nations' 17 sustainable development goals. the world health organization estimates that 970 million people worldwide suffer from mental health issues, the most common being anxiety and depressive disorders foot_0 . the problem is compounded by the fact that the rate of undiagnosed mental disorders has been estimated to be as high as 45% @xcite . the societal impact of mental health disorders requires prevention and intervention strategies focused primarily on screening and early diagnosis. in keeping with the who mental health action plan @xcite , natural language processing and machine learning can make an important contribution to gathering more comprehensive information and knowledge about mental illness. in particular, an increasing use of social media platforms by individuals is generating large amounts of high-quality behavioral and textual data that can support the development of computational solutions for the study of mental disorders. an emerging, interdisciplinary field of research at the intersections of computational linguistics, health informatics and artificial intelligence now leverages natural language processing techniques to analyze such data to develop models for early detection of various mental health conditions. systematic reviews of this research show that the vast majority of the existing work has focused primarily on automatic identification of specific disorders, with depression and anxiety being the most commonly studied target conditions @xcite @xcite . as a result, existing work has focused on developing binary classifiers that aim to distinguish between individuals with a particular mental illness and control users. the current work addresses the more complex problem of distinguishing between multiple mental states, which is essential if we are to uncover the subtle differences among the statistical patterns of language use associated with particular disorders. specifically, in this paper we make the following contributions to the existing literature on health text mining based on social media data: (1) we frame the mhc detection tasks as a multiclass prediction task aimed to determine to what extent six mental health conditions (anxiety, attention deficit hyperactivity disorder, bipolar disorder, post-traumatic stress disorder, depression, and psychological stress) can be predicted on the basis of social media posts from reddit. (2) we explore and compare the performance of hybrid and ensemble models leveraging transformer-based architectures (bert and roberta) and bilstm neural networks trained on within-text distributions of a diverse set of linguistic features. (3) we conduct feature ablation experiments to investigate which types of features are most indicative of particular mental health conditions. this paper is organized into five sections. section 2 provides a concise overview of the current state of research on mental health detection from reddit social media posts. section 3 presents the experimental setup including descriptions of the data, the type of linguistic features used and their computation, and the modeling approach. the main results are presented and discussed in section 4. in section 5 general conclusions are drawn and an outlook is given. ## related work a growing body of research has demonstrated that nlp techniques in combination with text data from social media provide a valuable approach to understanding and modeling people's mental health and have the potential to enable more individualized and scalable methods for timely mental health care (see @xcite ; chancellor and de choudhury (2020); @xcite , for systematic reviews). a surge in the number of research initiatives by way of workshops and shared tasks, such as computational linguistics and clinical psychology (clpsych) workshop, social media mining for health applications (smmh) and international workshop on health text mining and information analysis (louhi), are advancing this research area: it fosters an interdisciplinary approach to automatic methods for the collection, extraction, representation, and analysis of social media data for health informatics and text mining that tightly integrates insights from clinical and cognitive psychology with natural language processing and machine learning. it actively contributes to making publicly available large labeled and high quality datasets, the availability of which has a significant impact on modeling and understanding mental health. while earlier research on social media mining for health applications has been conducted primarily with twitter texts @xcite , a more recent stream of research has turned towards leveraging reddit as a richer source for constructing mental health benchmark datasets @xcite . reddit is an interactive, discussion-oriented platform without any length constraints like twitter, where posts are limited to 280 characters. its users, the redditors, are anonymous and the site is clearly organized into more than two million different topics, subreddits. another crucial fact that makes reddit more suitable for health text mining is that, unlike twitter (with its limited text length), extended text production provides a richer linguistic signal that allows analysis at all levels of organization (morpho-syntactic complexity, lexical and phrasal variety, and sophistication and readability). @xcite , for instance, proposed an approach for automatically labeling the mental health status of reddit users. reflecting the topic organization of reddits with its subreddits, the authors created high precision patterns to identify users who claimed to have been diagnosed with a mental health condition (diagnosed users) and used exclusion criteria to match them with control users. to prevent easy identification of diagnosed users, the resulting dataset excluded all obvious expressions used to construct it. this approach was also adapted to other mental health conditions @xcite . previous research on health text mining from social media posts has primarily focused on the automatic identification of specific mental disorders and has treated it as a binary classification task aimed at distinguishing between users with a target mental condition and control ones (see the systematic reviews mentioned above). to the best of our knowledge, the only two exceptions are @xcite and @xcite . @xcite proposed an approach to classify mental health-related posts according to theme-based subreddit groupings using deep learning techniques. the authors constructed a dataset of 458,240 posts from mental health related subreddits paired with a control set approximately matched in size (476,388 posts). the mental healthrelated posts were grouped into 11 mhc themes (addiction, autism, anxiety, bipolar, bpd, depression, schizophrenia, selfharm, suicidewatch, cripplingalcoholism, opiates) based on a combination of manual assessment steps and automated topic detection. their best performing model, a convolutional neural network classifier trained on word embeddings, was able to identify the correct theme with a weighted average accuracy of 71.37%. the approach taken in this work was primarily aimed at identifying posts that are relevant to a mental health subreddit, as well as the actual mental health topic to which they relate. another more recent exception similar to our work is @xcite . the authors used roberta (robustly optimized bert pretraining approach, @xcite al. (2019) ) to build multiclass models to identify five mental health conditions from reddit posts (adhd, anxiety, bipolar disorder, depression, and ptsd). the model was trained on a dataset consisting of reddit subreddits with 17,159 posts. the roberta-based model achieved a macro-averaged f1 value of 89%, with f1 values for individual conditions ranging from 84% for depression to 91% for adhd. although these results appear impressive, they should be interpreted with caution: to obtain data for each of the mental health conditions, the authors extracted posts from five subreddits (r/adhd, r/anxiety, r/bipolar r/disorder, r/depression, r/ptsd) and assigned them a class label corresponding to the name of the condition with which they were associated. posts for the control group were selected from subreddits with a wide range of general topics (music, travel, india, politics, english, datasets, mathematics and science). the way the datasets in @xcite and @xcite are constructed rendered the classification tasks relatively easy, as it allows the classifier to use explicit mentions of mental health terms associated with a particular mental health condition. however, there is growing recognition that careful dataset construction is critical to developing robust and generalizable models for detecting mental health status on social media. this requires the removal of expressions indicating mental health status for both diagnosed and control users (see @xcite or smhd @xcite ; see also chancellor and de choudhury (2020) and @xcite for discussions on obtaining ground truth labels for the positive classes and data preprocessing/selection). the existing research on the detection of mental health conditions in social media mainly follows one of two approaches: one focuses on linguistic features, mainly in the form of unigrams with tf-idf (term frequency-inverse document frequency) weighting, or on specialized dictionaries, especially the categories from the linguistic inquiry and word count (liwc) dictionaries @xcite @xcite . the second centers on leveraging contextualized embedding techniques and pre-trained language models such as bert @xcite , elmo @xcite , and roberta (?), minimizing the need for tasks such as feature engineering or feature selection @xcite ; @xcite , see also @xcite for a review). however, less work has been undertaken to date to explore hybrid and ensemble models for mental illness recognition that integrate engineered features with transformer-based language models. such hybrid models have recently been successfully applied in the neighboring research area of personality recognition @xcite . ## results and discussion as for the error analysis, figure 5 shows the confusion matrix of our best model (stacking) normalized over the actual classes (in rows). we found that for five of the seven mental health conditions, the majority of model predictions applied to the correct class (adhd 25%, bipolar 23%, depression 34%, stress 76%, control 54%). bipolar disorder was frequently misclassified as ptsd (23%). anxiety was most often classified as adhd (18%), followed by bipolar disorder and correct classification (both 16%). depression posts were most frequently confused with adhd (19%), bipolar disorder (16%) and anxiety (14%). at the same time, depression was by far the most frequently predicted class overall, with an average prediction rate of 24.4%. these findings reflect evidence in the psychiatric literature indicating that there is considerable overlap in clinical symptoms and pathophysiological processes and that depressive symptoms may also occur in the context of another psychiatric disorder (e.g., bipolar disorder) @xcite . furthermore, psychiatric data suggest that depressive disorders (i.e., major depressive disorder and dysthymia) are highly comorbid with other common mental disorders @xcite . in contrast, misclassifications in the stress category were almost exclusively controls (22% of all predictions), indicating that statistical patterns of language use reflecting stress differ from those for diagnosed mental health disorders. controls were in turn most frequently confused with adhd (13% of all predictions). this finding is consistent with the prevalence of overdiagnosis of adhd in children and adolescents @xcite . finally, ptsd was correctly classisfied in only 1% of the cases, and typically misclassified as depression (36%) or bipolar (23%). that said, user posts were predicted by the stacking model to be ptsd only 6.5% (21/320) of the time, suggesting that the classifier is sensitive to the slightly lower frequency of this mental disorder. in view of the model's tendency to avoid predictions for the less populated class, we conducted additional multiclass experiments without the ptsd class to determine how this would affect the overall pattern of findings. the results of these experiments revealed that the exclusion of ptsd yielded a slight improvement in overall classification accuracy, with the improvement over chance increasing from 18.9% f1 to 23.65% f1. in regards to rank order, the performances of the models mirror those of the models with ptsd: the hybrid model still outperformed both transformer-based models (+3.6% f1 over bert and +3.37% f1 over roberta) and the stacked generalization still yielded highest classification accuracy (+2.05% f1 over the hybrid model). the general patterns of misclassification remained the same (for further details, see the results of the feature ablation experiments are presented in ## conclusion and outlook in this paper, we reported on multiclass classification experiments aimed at predicting six mental health conditions from reddit social media posts. we explored and compared the performance of hybrid and ensemble models leveraging transformerbased architectures (bert and roberta) and bil-stm networks trained on within-text distributions of a diverse set of linguistic features. our results show that the proposed hybrid models significantly improve both model robustness and model accuracy compared to transformer-based baseline models. the use of model stacking proved to be an effective technique to further improve model accuracy. ablation experiments revealed that the importance of textual features concerning readability, registerspecific n-gram frequency and lexical richness far outweighs the importance of closed vocabulary features. in future work, we intend to perform comprehensive feature analysis based on within-text distribution to identify most distinctive indicators of diverse depressive disorders. we also intend to extend the approach presented here to incorporate features of textual cohesion. in addition, we intend to integrate the proposed approach with data on the behavioral activity of the individual, such as the frequency of posting and the temporal distribution of posting histories.