D isor BERT : A Double Domain Adaptation Model for Detecting Signs of Mental Disorders in Social Media (2023)

Abstract: Mental disorders affect millions of people worldwide and cause interference with their thinking and behavior. Through the past years, awareness created by health campaigns and other sources motivated the study of these disorders using information extracted from social media platforms. In this work, we aim to contribute to the study of these disorders and to the understanding of how mental problems reflect on social media. To achieve this goal, we propose a double-domain adaptation of a language model. First, we adapted the model to social media language, and then, we adapted it to the mental health domain. In both steps, we incorporated a lexical resource to guide the masking process of the language model and, therefore, to help it in paying more attention to words related to mental disorders. We have evaluated our model in the detection of signs of three major mental disorders: Anorexia, Self-harm, and Depression. Results are encouraging as they show that the proposed adaptation enhances the classification performance and yields competitive results against state-of-the-art methods.

## introduction mental disorders are among the most common illnesses worldwide. some estimates foot_0 indicate that more than 50% of the population will be diagnosed with a mental disorder at some point in their lives. the prevalence of these disorders is highly concerning since they alter the way people think, feel, and take action, resulting in the incapacity of daily life routines. in addition, the recent covid-19 pandemic triggered a serious global social and economic disruption, which had a direct effect on people's lives, and brought many challenges that can be stressful and overwhelming @xcite . this situation was particularly difficult for people with mental health conditions and caused an increase in the prevalence of anxiety and depression @xcite . there is therefore an increasing need for developing new tools to monitor the presence of mental disorders and to respond to early signs of psychological concerns. nowadays, social media content is massive and provides an opportunity to do research on how people undergo difficulties. many people use online platforms to publicly share their daily routines and important events, while others take advantage of the anonymity of these spaces to explicitly discuss mental health issues and to seek help @xcite . in this work, we aim to contribute to the detection of signs of mental disorders by automatically analyzing social media posts. this type of analysis is expected to support new technologies able to warn about the onset of mental disorders and provide supporting evidence. as argued by @xcite , these new forms of screening should not be taken as "magic substitutes for the human expert" but, instead, as computational tools that can substantially reduce the workload of public health systems, e.g., by facilitating preventive measures. latest developments in natural language processing (nlp) encourage the fine-tuning of pretrained language models for a wide variety of tasks. this approach often yields good results, but it is problematic for tasks where the language is highly domain-specific @xcite , such as in the case of mental disorders. an alternative is to pre-train the model -e.g., bert @xcite -with data from the target domain. however, pre-training is expensive and complex, and collecting a sufficiently large training corpus can be difficult for certain domains, as exposed by the creators of mentalbert @xcite . instead of pre-training the model, we propose to perform a domain adaptation, similar to the one proposed in @xcite but refined in two stages @xcite that exploit a novel lexicon-driven learning. this process takes an already trained model and continues its training using a (relatively) small corpus focused on the target domain @xcite . in particular, we propose disorbert, a two-step domain adaptation model of bert for detecting signs of mental disorders in social media. first, we teach bert the general structure of the language used in social media texts (e.g., in reddit posts), then we specialize it in the kind of language used to express information about mental disorders. furthermore, we exploit lexical knowledge to guide the language model's masking process. instead of learning the occurrence of general words, this "guided masking" opts to bias the learning process towards words that are important to the target application, which in our case is the detection of anorexia, depression, and self-harm. we can summarize our contributions as follows: ## related work the detection of mental disorders is an interdisciplinary research area that has been fostered thanks to the current availability of a variety of data sources and computational models @xcite . in recent years, several works have explored social media platforms to study the manifestation of mental disorders @xcite . social media sources have been exploited to detect features that help to identify signs of need of medical or psychological support @xcite . for example, expressions of distress or negative feelings, particularly published by young people @xcite , abound in online media. a variety of methods have been applied to find relevant and discriminative patterns from user-generated text. for example, some studies employed words or word sequences as features @xcite @xcite . other groups of studies have applied sentiment analysis techniques to model emotional properties of users' posts @xcite , or exploited a set of psychological categories to capture social relationships, thinking styles as well as individual differences @xcite . in @xcite and @xcite , the authors explored the association between linguistic inquiry features and suicide risk factors, extracting patterns in different linguistic profiles. similarly, for the detection of signals related to suicide attempts, @xcite used word embeddings and a bidirectional long short-term memory (lstm) to capture contextual information. furthermore, for detecting suicidal ideation, @xcite explored the incorporation of images into text-based representations. these authors analyzed the combination and relationship of textual and visual information, identifying significant differences in the use of both. more recently, with the increasing popularity of transformers, bert-based classifiers have been fine-tuned for detecting different mental disorders @xcite . on the other hand, a number of studies have trained language models for specific domains @xcite . for example, @xcite exploited a large-scale annotated dataset with scientific data to adapt bert to the scientific domain, showing improvements over the bert base model in multiple classification tasks. similarly, in @xcite , bert was pre-trained on largescale biomedical corpora outperforming the original bert in a variety of biomedical text mining tasks. in recent work, nguyen et al. (2020) trained a bert model with approximately 850m tweets achieving outstanding results in several tweet analysis tasks. closer to our work, @xcite adapted bert to the mental health domain by collecting specific data from reddit. their model, named mental-bert, was trained using 13,671,785 sentences for around eight days using four tesla v100 gpus. the aforementioned pre-training approach is usually effective, but it is also expensive as collecting a corpus of a suitable size to pre-train bert can be a big challenge in many domains. to overcome this issue, we propose to perform a double-domain adaptation of bert, a less-expensive process that takes advantage of the already trained general model. we explain this strategy in detail in the following section. ## from bert to disorbert: a mental disorder detection model this section introduces disorbert, a language model specially suited for the detection of signs of mental disorders in social media. as mentioned above, the construction of disorbert consists of a two-stage domain adaptation of bert. in short, the idea is to first teach bert the general structure of the language used in a large social media platform (e.g., in reddit), and next, to specialize the model in the language of users with mental disorders. this whole process is depicted in figure 1 . for domain adaptation, we follow the procedure suggested by @xcite and @xcite , which consists of continuing bert's pre-training through the fine-tuning of a masked language model for more epochs. the process first uses the reddit corpus for fine-tuning, and then, in a second adaptation, uses a collection of documents related to mental health. the idea is to adapt the language model from the general data of wikipedia and books corpora (bert's sources for training) to the more specific language of reddit and mental health. for this process, we trained our model for three epochs, using a batch size of 128, a learning rate of 2e -5 , on a gpu nvidia tesla v100 32gb sxm2. it is particularly relevant to note that, in each of these steps, we employed a depression lexicon to guide the model's masking process and, therefore, to bias the learning towards words that are important to the target application domain. ## evaluation and discussion the first thing to notice is that bert performed well but mentalbert and disorbert are better choices, highlighting the importance of having domain-oriented models. going into detail, we can observe that most of our proposed models outperformed the baselines in terms of f 1 . our singledomain adaptations obtained slight improvement in comparison with baselines, while the doubledomain adaptation further increased performance, particularly with the incorporation of lexical knowledge. here it is important to highlight that the lexicon employed is not specific to the language of anorexia or self-harm but, still, it was also beneficial for these two target tasks. from another perspective, disorbert showed a good balance between precision and recall, whereas other variants (e.g., rnn-glove) improved precision at the expense of recall. disor-bert has, therefore, a solid retrieval behavior and it can effectively find multiple traces of psychological risks. this is an important outcome since high recall is essential for clinical screening tools. however, there might be some potential use cases where high recall is not the most preferable choice, e.g., a social network that wants to focus on the riskiest behavior. for these scenarios, it may be necessary to modify our model to prioritize precision. in figure 2 , we plot the precision and recall of disorbert and the baselines. our model tends to locate in the main diagonal region (indicating its good balance), while other methods have high precision or recall but score low in the other dimension. for a more detailed analysis, we applied mcnemar's statistical test @xcite to compare the best disorbert results with the best baseline results, comparison against erisk participants: figure 3 presents a boxplot of the f 1 scores of all participants for the anorexia and self-harm erisk shared tasks foot_4 . the red circles represent the best disorbert model. for both tasks disorbert gets to the highest quartile, and, especially, in the anorexia detection task, our result is above the highest-scoring participant. these results indicate that our approach is competitive in comparison with the participants. however, it is important to mention that erisk participants focused on obtaining early and accurate predictions of the users, while our approach focuses exclusively on determining accurate classifications. overall, we can highlight the following conclusions of the experimental results: bert is a language model trained from a general corpus, while disorbert is a model guided to the mental health domain. let us illustrate the behavior of the learned model, and the kind of textual segments it tends to pay more attention to. first, we analysed the most likely words the models generate when given a sentence with masked words. as sentences, we used examples from the beck depression inventory (bdi) @xcite . this clinical tool, which consists of 21 items, aims to identify and measure the severity of typical symptoms of depression in adults and adolescents aged 17 and older. for example, it measures mood, pessimism, sense of failure, self-dissatisfaction, and guilt, among others. this test gives several responses for each item. we selected one of the answers for each one, masked a keyword, and looked at the words predicted by bert and disorbert. in let us now look at the models in a different way. for each bdi sentence, we know the target masked word and we can extract the position of this word in the ranked list provided by each model. for example, in the third case of ## conclusion in this study, we explored a double domain adaptation approach for the tasks of detecting signs of anorexia, depression, and self-harm in social media. the first step of the domain adaptation focused on learning the writing style of social media users. the second step was oriented to learn about mental disorders and how users refer to psychological issues. in both steps, we incorporated lexical knowledge to guide the model toward words that are highly indicative of mental-related topics. results suggest that combining domain adaptation with lexical knowledge helps in detecting traces of mental disorders. this approach outperformed traditional and state-of-the-art baselines and is competitive with the performance of top early-risk algorithms. furthermore, the analysis of our method revealed that the context learned by the model is important in getting a better understanding of the concerns expressed by people. in future work, we want to explore the application of different lexical resources that are even more specialized for the target tasks, as well as the usage of clinical data to train more specialized language models, e.g., mimic @xcite . on the other hand, emojis are often important features for social media analysis, and we want to explore their incorporation into our training process. also, we are interested in expanding this study to different languages, since most of the work related to mental disorders has focused on english.