Classification of mental illnesses on social media using R o BERT a (2021)

Abstract: Given the current social distancing regulations across the world, social media has become the primary mode of communication for most people. This has isolated millions suffering from mental illnesses who are unable to receive assistance in person. They have increasingly turned to online platforms to express themselves and to look for guidance in dealing with their illnesses. Keeping this in mind, we propose a solution to classify mental illness posts on social media thereby enabling users to seek appropriate help. In this work, we classify five prominent kinds of mental illnesses- depression, anxiety, bipolar disorder, ADHD and PTSD by analyzing unstructured user data on Reddit. In addition, we share a new high-quality dataset1 to drive research on this topic. The dataset consists of the title and post texts from 17159 posts and 13 subreddits each associated with one of the five mental illnesses listed above or a None class indicating the absence of any mental illness. Our model is trained on Reddit data but is easily extensible to other social media platforms as well as demonstrated in our results. We believe that our work is the first multi-class model that uses a Transformer based architecture such as RoBERTa to analyze people’s emotions and psychology. We also demonstrate how we stress test our model using behavioral testing. Our dataset is publicly available and we encourage researchers to utilize this to advance research in this arena. We hope that this work contributes to the public health system by automating some of the detection process and alerting relevant authorities about users that need immediate help.

## introduction during these unprecedented times when the world is plagued by covid19, a large number of people have been showing symptoms of clinical anxiety or depression foot_0 . this can be attributed to a myriad of reasons including lock down, mandatory social distancing, higher unemployment, economic depression and work-related stress. in a report published earlier this year, the american foundation for suicide prevention found that people experience anxiety (53%) and sadness (51%) more often now than before the coronavirus pandemic. in the past decade, social media has transformed how people interact with each other. apart from sharing factual information and news, people actively partake in sharing their day to day activities, experiences, feelings, opinions, hopes, desires, and emotions online. these texts provide information which can be used to identify the mental health individuals. furthermore, the current state of enforced social distancing and isolation has propelled more people to express their emotions on social media as it provides them with an accessible platform to share their thoughts with others, many a times, in search for help. our research work utilizes user data, especially the kind pertaining to emotions as this class of data can give us valuable insights about the mental state of a person; and, in turn, our work has the potential to assist in the diagnosis and analysis of various mental disorders. this study aims to bridge the gap between people in search of help and experts who can provide the needed help. due to the paucity of adequate annotated and structured user data in this domain, we decided to generate our own dataset by crawling subreddits on reddit.com foot_1 pertaining to our use case as a lot of users were found to have shared their feelings there. although reddit was the sole source of our dataset, we believe that this study can be seamlessly extended to other social media platforms as well because of the presence of similar unstructured user data online. the advent of transformers and @xcite has caused quite a stir in the nlp community because of the state-of-the-art results it was able to produce in various nlp tasks. in this work, we use a roberta @xcite based classifier, which has a similar architecture to bert with an improved pre-training procedure. roberta's effective and efficient performance on unstructured data and its ability to learn contextual information compelled us to explore and capitalize its power to categorize online user generated texts into various classes of mental illness. we identify five broad classes of mental illnesses -depression, anxiety, bipolar disorder, adhd (attention deficit hyperactivity disorder), ptsd (post traumatic stress disorder) and an additional 'none' class (which does not pertain to any mental illness). we train a multi-class classifier on the data crawled from online user data. based on our experiments, we present encouraging results that demonstrate that social media data has the potential to complement standard clinical procedures in the prognosis of mental health amongst two broad categories of users -ones who are seeking help online and ones who are unbeknownst of their condition. ## related work in the recent past, people have increasingly turned to social media to share and seek counsel on the topic of mental health. this has prompted researchers to utilize the information and apply a plethora of techniques in nlp and machine learning in order to assist people who might require help. most of the recent such research, as described in the subsequent paragraphs, has revolved around reddit data: @xcite al. (2020), gkotsis et al. (2017 there have been a wide variety of approaches ranging from classical nlp techniques to neural network based deep learning methods. @xcite used character level language models to examine how likely a sequence of characters is to be generated by a user with mental health issues. @xcite evaluated a standard regression model, a multilayer perceptron single-task learning (stl) model, and a neural mtl model on detecting multiple types of mental health issues. orabi et al. (2018) utilized word embeddings in tandem with a variety of neural network models like cnns and rnns to detect depression. gkotsis et al. (2017) experimented with feed forward neural networks, cnns, svms and linear classifiers to perform binary classification on mental health posts. sekulic and strube (2019) came up with the approach of using hierarchical attention networks(hans) to detect a wide range of mental health issues like depression, adhd, anxiety etc. and trained a binary classifier for each of the disorders. the most recent work on this was by kim et al. (2020) who proposed a cnn-based classification model. once again though, each disorder had its own separate binary classifier to perform the detection. to our knowledge, this is the first attempt at treating this problem as a multi-class classification problem, where a single classifier can accurately classify the type of disorder that the person is referring to in their post. in addition, this is also the first work that harnesses the incredible capabilities of an advanced transformer based algorithms like roberta to solve this difficult problem. ## dataset the reddit api was used to crawl 13 reddit subreddits for a total of 17159 posts (text and title) to obtain the data for this work. the text from comment threads of these posts was not collected as it tended to diverge from the main topic of the subreddit. even though there are a lot of mental illnesses that need addressing, only 5 of them had sufficient data for our purposes that were chosen for the purposes of this paper. they are: bipolar, adhd, anxiety, depression and ptsd. we do plan to extend this solution to tackle the remaining illnesses as well in due time. the posts in these subreddits were assigned a class label corresponding to the name of the mental illness they were associated with. all the remaining subreddits were carefully chosen as to minimize any chances of thematic content overlap between them and the illness classes i.e to enable the model to differentiate between posts discussing mental illness from those that were not, we chose a few subreddits from a wide range of common topics such as music, travel, india, politics, english, datasets, mathematics and science. the text from these subreddits were combined and assigned the class label none. while we are currently treating this task as a multi-class classification problem, we duly acknowledge the fact that mental illnesses are generally co-related and require multi-label classification techniques. we are actively working on converting this task to a multi-label classification problem. while collecting data, we ensured that the number of upvotes for each post in all subreddits is more than 10. we also set a minimum post token length of 30 tokens. these numbers were chosen after carefully perusing through the data and going over the post text in order to retain quality in the dataset. we initially crawled the subreddits pertaining to mental illness and were able to collect about 3000 posts in total. we then crawled the other subreddits corresponding to the none class label and collected about 300 posts per subreddit for 8 subreddits. this also ensured a good balance of class labels. while selecting the eight general topic subreddits, we not only selected subreddits that have sufficiently high number of posts, but also ensured that we cover a broad range of topics. to gauge the data quality we ran some analysis. we manually went over the lowest voted posts for each mental illness subreddit. we wanted to establish that texts from these posts expressed emotions from people discussing corresponding mental illness it is labelled as. we also certified that the general topic subreddits did not have a high similarity with the posts corresponding to other the other 5 subreddits. this was done to ensure that we do not have any false negatives while assigning truth labels. we counted the number of posts the mental illness terms appeared in, for each subreddit. the subreddits corresponding to mental illnesses had a much higher count of these words. in addition to this, we compared the cosine similarity between some of the highest/lowest posts of mental illness subreddits and the general topic subreddits and manually compared the results to find that this distance was higher than the distance between two posts of the mental illness subreddits. we also attempted to augment the data using easy data augmentation @xcite to boost the performance of our model. however, we did not observe an apparent shift in our evaluation metrics-explained by the fact the eda is meant to perform best for smaller datasets (≤ 5000 sample sizes). ## model in this section, we describe our model architecture for the multi-class mental illness classification task. we propose a roberta based classifier in order to accomplish this. in addition, we also compare the proposed model against an @xcite based classifier and a @xcite based classifier to demonstrate the superiority of our approach. since this is an entirely new dataset, there is no established baseline, so the lstm model will serve as the baseline for our experiments. we also showcase our gains over bert, the most widely used transformer model today for text classification. all our models were implented in pytorch @xcite . the transformer models were implemented with the help of the huggingface transformers @xcite library. ## result analysis as described earlier, in addition to our primary roberta classifier, we also run experiments on an lstm classifier and a bert classifier for the sake of comparison. we fine-tune each of the aforementioned models on just the titles, just the posts and a combination of both in order to perform comprehensive tests and comparisons. when combining the titles and posts for our transformer models, we convert the problem into a sequence-pair classification task. this allows the model to give more importance to the title which would otherwise be lost when combining the title and the post into one single input given the relative difference in their lengths (the average number of tokens in titles is roughly 6% that of posts). the results from our experiments are documented in tables 3 through 5 . as can be observed from the rest of this section will focus solely on the results of our best performing roberta model. the first strikingly obvious result is the high accuracy with which the model is able to detect non-illness related posts. even with just the titles, the model is able to classify the none class with an f1 score of more than 0.9. this gives us hope that this model will suffer from very few false positives when it comes to mental illness detection on social media. an even more crucial property of our model can be noticed in the confusion matrices for posts and posts+titles in figure 1 . when using posts, just 3 illness related posts across the entire test dataset were misclassified as non-illness posts. this number further reduces to 0 when using titles+posts. this shows that the model will detect mental illness posts correctly nearly every single time, thus ensuring that posts from people who are seeking help never go unnoticed when this solution is deployed in the real world. when it comes to the class wise performance amongst the mental illnesses, the two best performing classes are adhd and ptsd whereas the two worst performing classes are depression and anxiety. the performance of depression and anxiety classes can be attributed to a few factors. the average number of words per post for depression and anxiety are the least for any given class. for instance, depression posts have roughly 53% lesser textual data when compared to ptsd posts. in addition, studies show that depression might often occur in tandem with another mental illness and our data and results back this up as well. the depression word occurs in 12% of anxiety posts, 12% of ptsd posts and 31% of bipolar posts. similarly, anxiety occurs in 20% of ptsd posts, 12% of adhd posts and 14% of bipolar posts. this implies that the model cannot give high importance to the mention of these class names like it can with rest of the illnesses, thus making the classification of these 2 classes that much harder. in figure 1 . in the same figure, we can see that depression and anxiety are often misclassified as each other due to the reason that they commonly occur together. there are more posts in the adhd and ptsd classes that mention the words depression and anxiety than their respective class names itself. one would assume that this would result in subpar results, but, these classes actually perform the best. this really showcases the true potential of our model, where it doesn't just rely on mention of class names, but has a strong understanding of the context of the post itself. additionally, the symptoms or descriptions provided for these classes could be strong, unique and discriminative enough for the model to be able to classify them correctly even with all the mentions of other class names. in ## conclusion and future work our chief motivation behind this work is the current pandemic and mandatory confinement worldwide. we believe that social media has become the prime mode of communication for people and has paved way for them to vent freely without judgement. our roadmap includes getting all our data annotated by mental health experts in order to verify our annotations. this would also assist us in creating a multi label dataset which is more representative of this problem when compared to a multi class one. we would also like to work on bettering our model on the behavioral tests. our work involves two kinds of texts-long and short -both of which are common to the internet community. hence, our work can easily be extended to many websites. in conclusion, we believe that our work explores an interesting line of research where nlp is used to bridge the gap between virtual and real life of users and help those in need of medical attention.