Automatic Detection and Classification of Mental Illnesses from General Social Media Texts (2021)

Abstract: Mental health is getting more and more attention recently, depression being a very common illness nowadays, but also other disorders like anxiety, obsessive-compulsive disorders, feeding disorders, autism, or attention-deficit/hyperactivity disorders. The huge amount of data from social media and the recent advances of deep learning models provide valuable means to automatically detecting mental disorders from plain text. In this article, we experiment with state-of-the-art methods on the SMHD mental health conditions dataset from Reddit (Cohan et al., 2018). Our contribution is threefold: using a dataset consisting of more illnesses than most studies, focusing on general text rather than mental health support groups and classification by posts rather than individuals or groups. For the automatic classification of the diseases, we employ three deep learning models: BERT, RoBERTa and XLNET. We double the baseline established by Cohan et al. (2018), on just a sample of their dataset. We improve the results obtained by Jiang et al. (2020) on post-level classification. The accuracy obtained by the eating disorder classifier is the highest due to the pregnant presence of discussions related to calories, diets, recipes etc., whereas depression had the lowest F1 score, probably because depression is more difficult to identify in linguistic acts.

## introduction an analysis performed by @xcite estimates that approximately 10% of the world's population is living with a mental illness. the global burden of disease @xcite states that depression is a very common illness and there are more than 264 million people affected by it. at its worst, the illness can lead to suicide and it is the second highest cause of death for people between 15 and 29. between 76% and 85% of the potentially diagnosed people, do not benefit from any treatment for their illness due to living in impoverished areas and not having access to mental care. it is difficult to discuss about digital solutions in the context of isolated areas with low data availability and limited access to professional help. social stigma is another obstacle present regardless of age, gender or race, which makes early intervention difficult. persons facing difficulties often avoid discussing their issues from various reasons. however, researchers working with machine learning algorithms can draw plenty of expertise from the unstructured data roaming the world wide web. the advent of social media platforms brings up an influx of large quantities of various types of unstructured textual data. the continuous advancements made in the field of machine learning enable the possibility to analyse such volumes of data efficiently. experiments in this interdisciplinary domain managed to bring up useful input for mental health practitioners, sociolinguists, computer scientist and other researchers in the field. @xcite perform one of the most influential quantitative studies, which reveals the way patterns of parts of speech, as labelled by liwc founders, correlate with types of personalities and types of mental illnesses. the classes and the psychological dimensions mapped together served as a start for many projects including the prediction of dark triad personality traits by @xcite and the risk of selfharm by @xcite . research in the area is conducted mainly on texts from mental health support groups, on just a few illnesses and some groups of individuals. our main research questions for this article are if and to what extent it is possible to detect and classify mental illnesses from general texts, if there ## related work nlp researchers have shown an increased interest in the area at the intersection of machine learning and psychiatry in the last years. social media is an indispensable resource for research. yet, the particularities of the online setting rise a range of challenges. as there are not any standards established for using social data, practitioners from many fields pointed to the dangers of using such data without a clear framework. @xcite address the issue of "biases, methodological pitfalls, and ethical boundaries" -discussing the problems often left unaddressed by researchers working with this kind of data. @xcite analyse not only the ethical dilemma revolving around this type of studies, but also their feasibility and the integration of the social component into the compound of a socio-technical system. when it comes to detecting mental illnesses from social media data, we have many examples at hand, which often look at data coming from those reddit communities, which are support groups for people struggling with an illness or another. most articles look at a single illness in comparison to a control group: @xcite and @xcite schizophrenia. our goal is to detect a wide range of mental illnesses using deep learning techniques, which seem like the best candidates for this task. @xcite employ deep learning methods similar to ours, but we concentrate on obtaining better results by training the models on individual posts rather than posts grouped by users, which might not work as expected. for example, if a user produced few contributions or has a fresh account, they would probably have few posts available. on the other hand, some types of user are the observing type and rarely contribute to discussions. one aspect worth mentioning is the nature of the data used in many classification tasks. texts containing explicit content and linguistic cues pertaining to the properties of a certain illness are often used. @xcite al. (2020) and thorstad et al. (2019) perform automatic text classification by their author's mental illnesses, with good results, on texts that specifically discussed these conditions on dedicated forums. nevertheless, these classifications are of little help in finding risk population, when looking at general text, which does not include mental illness topics. among the few researchers who report using datasets containing general discussions coming from people who self-reported their diagnosis in one of the support communities are @xcite and @xcite . the results are favorable and leave room for improvement. we believe it is important to experiment further for a better understanding of the ways in which mental illnesses can be detected in earlier stages and how even general discussions contain traces of how mental illnesses manifest themselves in language. in addition, this is a direction worthy of exploration because the persons asking for guidance represent a very small and idiosyncratic part of the population battling with mental illnesses, thus early mental illness detection from general text might be of a real help. ## data we used the smhd dataset introduced by @xcite . this dataset contains non-explicit texts: a large-scale resource for exploring online language usage for multiple mental health conditions. they test some classification algorithms, but no deep learning. also, employed liwc categories for classification. these categories include standard linguistic dimensionspro-nouns, articles, present tense, future tense; psychological processes -positive emotions, negative emotions, anger, anxiety; personal concerns -work, achievements. the smhd dataset contains texts extracted from reddit's general discussion communities grouped on users and illnesses. individuals diagnosed with a mental illness were detected by searching for self-reports in the dedicated support groups. the dataset features multiple illnesses, which are present in the psychiatric taxonomy dsm-5 (american psychiatric association, 2013). as stated by the authors of the dataset, "six conditions are top-level dsm-5 disorders: schizophrenia spectrum disorders (schizophrenia), bipolar disorders (bipolar), depressive disorders (depression), anxiety disorders (anxiety), obsessive-compulsive disorders (ocd) and feeding and eating disorders (eating). the three other conditions are one rank lower: post-traumatic stress disorder (ptsd) is classified under trauma-and stress-related disorders, and autism spectrum disorders (autism) and attention-deficit/hyperactivity disorder (adhd) under neurodevelopmental disorders". the opposing group of users is the control one, whose members are selected based on having no posts in the support groups and at least 50 posts on reddit. the complete dataset contains 20,406 diagnosed users and 335,952 control users. the texts do not contain any terms related to mental health, neither the diagnosed groups, nor the control ones. our experiments will use just a selection of each group of illnesses to speed up the computation process. the models are not user centered and will learn from each individual post. selecting data based on a fixed number of users was not suitable for our tasks due to the imbalance at the user level when it comes to the number of comments and posts available. therefore, we selected randomly 50,000 posts for each group of users. the numbers shown in tables 1 and 2 might reflect certain particularities about an illness, how the diagnosed users communicate in the online environment. this variation depends also on how the users engage, whether they create posts or comment on somebody else's and on the format adopted by each community -if pictures are posted often, then the comments are on the shorter side, if story telling is the center of the community, people engage with the purpose of telling their opinion or a similar story, hence the lengthier texts. the authors of the dataset conducted a linguistic analysis based on liwc categories. several differences were observed between the diagnosed groups and the control users. @xcite and @xcite underline that pronounced usage of firstperson singular with most conditions is consistent with the theory that illness drives one towards selffocus. an interesting finding underlining the bias of the dataset towards the predominantly male demographic is the female references that point to discussions about relationships and love related issues with the bipolar, depression and anxiety groups. reddit does not impose a very strict post limit hence we have diverse lengths. however, the deep learning models we used impose a limit for training. the smhd has already undergone preprocessing, but we needed more cleaning. we remove any posts shorter than 4 tokens. very short texts are often noise like thankful comments or very short approval phrases, which would confuse the model and do not contain significant meaning the next section will look at another data related problem, namely the ethics and biases of working with social media data. ## ethics and biases reddit represents a social media application whose users are part of communities and engage in discussions. each social media network represents a cluster of people who are defined by certain characteristics. the hootsuite yearly report @xcite shows that more than 60% of the reddit users are males aged 18 to 34. accordingly, studies show that there is a tendency in males to display less emotionally charged input due to the social stigma in the offline world. @xcite find that men often avoid seeking professional help or talking about their problems. concealing their emotional state in real life is a strategy in order to avoid prejudices and is not something specific for the female population. ireland and mehl's (2014) research conducted in the psychology area proves that manifestations of negative emotions are muted across many settings and situations. alternatively, @xcite and @xcite demonstrate that people tend to discuss personal things in anonymous spaces and share unpopular opinions. in this situation, reddit represents a good source of data for a population, which is underrepresented in clinical studies. @xcite prove that some platforms might be more attractive for a demographic than others. behavioral biases imply that users of a platform display a particular behavior, observable in how they interact with each other or what type of content they create. one such bias is the way in which users seek and share information. de @xcite discovered that users diagnosed with one illness behave differently in this aspect from the others. nevertheless, we cannot claim that this is representative for all the individuals diagnosed with a mental illness. there are certain biases plaguing the studies based on social media, which should be at least mentioned for awareness. here, we consider the population bias a positive fact, which enables studies targeting the young adult and adult males. however, this bias does not affect much our dataset, because the data collected comes from neutral communities where a variety of topics is discussed. ## discriminative features we run a naïve bayes classifier in order to find out the most informative features from each category in our dataset. we used the classifier implemented in the scikit-learn library by @xcite to get a top of n most informative words by scores. our experiment includes the 9 illnesses as labels, and the control group. the top n words can be seen in ## classification methods identifying significant differences between our groups was the main drive for training classifiers. we trained 3 different models based on the transformers architecture to see how each performs binary classification between a diagnosed group and a control one. we obtained state-of-the-bertforsequenceclassification by @xcite . in order to setup this model, we experimented using different hyperparameters, loss functions, batch sizes and number of epochs. the authors of bert recommend using it with the following specifications: our machine needed smaller batch sizes to be able to train the model, so we used 3. we established a learning rate (adam 𝜀) of 1e-5 for the adamw loss function implemented by @xcite . we trained the model for 3 epochs only, because we noticed overfitting starting with the 4th epoch. the second method we used is xlnet, which is another method for pre-training language representations introduced by @xcite . xlnet was meant to overcome the limitations imposed by bert with its autoregressive model and does so by outperforming it on 20 tasks as shown by @xcite . for this method, we have a different formatted input and there is no limit for the length of the input texts. however, the input arrays need to be of the same size. this is addressed by padding the inputs that do not meet the size of the longest sequence. padding means simply adding 0s until the length is met. for this classifier we had to limit the length of sequences to 126 due to computational resources. the optimum batch size was 8. the loss function we used was adamw with the same hyperparameters as for bert. we trained this model for 4 epochs. with a training set of approximately 100000 texts, we get a number of 50000 training steps. the last model we used, roberta implemented by @xcite , is facebook ai's training method and it promises to improve on bert. the researchers involved in implementing roberta prove that bert was undertrained and there is still a long way to go in terms of design choices and the way in which the improvements are reported. we did not use the full size of our dataset due to its large size and subsequent long training times. finetuning roberta implies loading the weights of the pretrained model, in our case, the robertaforsequenceclassification model. we use a sequence length of 256 and a batch size of 8. the loss function used here is adamw with adam 𝜀 of 2e-5. ## results we obtained the results using 50,000 posts for each group alike. the compound of 100,000 posts for each binary classifier was split in 80,000 for training and 20,000 for testing. we trained our models with different hyperparameters until we reached the optimum ones detailed in no. of testing posts 20,000 20,000 20,000 we used a naïve bayes classifier to discover the most important features for each group of users. our results add to the group of articles showing good prospects for this field. an encouraging finding is the sufficiency of focusing on general text rather than mental health support groups and classification by posts rather than individuals or groups. another takeaway is the sufficiency of post-level classification and avenue to improve this approach in future work by paying attention to contextual cues such as time, events, entailment of posts or any other possible triggers that might help the earlier detection of a mental illness. further experimentation with different setups and data that are more diverse is also required. this would benefit our research and increase the possibility of future integration of automated tools, which could assist clinicians in the earlier detection of mental health issues.