C hinese M ental BERT : Domain-Adaptive Pre-training on Social Media for C hinese Mental Health Text Analysis (2024)

Abstract: In the current environment, psychological issues are prevalent and widespread, with social media serving as a key outlet for individuals to share their feelings. This results in the generation of vast quantities of data daily, where negative emotions have the potential to precipitate crisis situations. There is a recognized need for models capable of efficient analysis. While pre-trained language models have demonstrated their effectiveness broadly, there’s a noticeable gap in pre-trained models tailored for specialized domains like psychology. To address this, we have collected a huge dataset from Chinese social media platforms and enriched it with publicly available datasets to create a comprehensive database encompassing 3.36 million text entries. To enhance the model’s applicability to psychological text analysis, we integrated psychological lexicons into the pre-training masking mechanism. Building on an existing Chinese language model, we performed adaptive training to develop a model specialized for the psychological domain. We evaluated our model’s performance across six public datasets, where it demonstrated improvements compared to eight other models. Additionally, in the qualitative comparison experiment, our model provided psychologically relevant predictions given the masked sentences. Due to concerns regarding data privacy, the dataset will not be made publicly available. However, we have made the pre-trained models and codes publicly accessible to the community via: https://github.com/zwzzzQAQ/Chinese-MentalBERT.

## introduction mental illnesses, particularly depression, impose a considerable strain on global societies. the world health organization reports that approximately 3.8% of the global population suffers from depression @xcite . notably, the incidence of depression in china accounts for as high * *corresponding author: guanghui.fu@inria.fr as 6.9% of the prevalence @xcite . individuals experiencing emotional distress often resort to passive coping mechanisms and seldom seek professional help @xcite . traditional channels for emotional crisis intervention, such as hotlines and psychological clinics, are not designed to proactively identify individuals facing emotional challenges @xcite . moreover, the resources for such interventions are frequently inadequate. the stigma associated with mental illness has led many to use social networks as a primary outlet for expressing their emotional struggles @xcite . platforms like x (twitter), and sina weibo in china serve as venues for individuals to share their feelings and opinions in real time, with posts often providing immediate insights into one's daily experiences and emotional states @xcite . within specific topics or hashtags on social media, there is a pronounced focus on the expression of negative emotions, with some users displaying evident suicidal tendencies @xcite . this situation underscores the critical necessity to develop tools aimed at enabling the early detection of such distress signals and implementing timely intervention strategies @xcite . pre-trained language models (plms), such as bert @xcite , have demonstrated remarkable success across a variety of language tasks and have seen extensive application @xcite . recently, the development of large language model (llm) technology has garnered global interest, notably within the psychology sector, where a plenty of exploratory applications have been initiated @xcite . however, according to research by @xcite , in comparison to supervised learning methods, llms are yet to fully addressed the complexity of psychological tasks. thus, the development of plms specifically targeting for supervised learning still crucial, particularly for the specific domains. to address the lack of large-scale plms tailored for specific applications in chinese community mental health, we introduce the chinese mental-bert, a pre-trained language model specifically designed for psychological tasks. to the best of our knowledge, it is the first plm developed specifically for the mental health field in chinese. in this study, we employ a domain-adaptive pre-training model @xcite , and introduce a novel lexicon guided masking mechanism strategy based on the chinese depression lexicon. we conduct four chinese mental health datasets from social media and public dataset, including over 3.36 million data items for domain-adaptive pre-training. this lexion guided masking mechanism strategically biases the learning process towards vocabulary crucial for the intended application, enhancing the model's relevance and effectiveness in its target domain. we evaluated the performance of our model on six mental health-related public datasets, including: two cognitive distortion classifications, one suicide risk classification, and three sentiment analysis tasks. the results demonstrate that our model outperforms eight comparison methods across six public datasets. due to privacy issues, we cannot open the data, we have made all the training code and models publicly available to support research in chinese mental health. ## related work social media has been used to identify signs of needing medical or psychological support @xcite . recent studies show that mental health analyses on social media are using nlp technologies to capture users' mental health states @xcite . with the advent of bert @xcite , studies have been carried out on using this technology to assess suicidal tendencies and identify depressive tendencies @xcite @xcite . yang et al. (2022) leveraged knowledge graph method to screen high-suicide risk comments within online forums and explored various attributes such as time, content, and suicidal behavior patterns by analyzing these comments. @xcite proposed a distant supervision method to develop an automated method capable of categorizing users into high or low suicide risk categories based on their social media comments. this model serves as an early warning system to aid volunteers in preventing potential suicides among social media users. on the other hand, many studies focus on domain-adaptive pretraining in specific fields to pursue better domain performance. @xcite systematically examined methods for adapting bert to the legal field. they gathered 12 gb of varied english legal text from public sources and achieved improved performance compared to the baseline on three end-tasks. @xcite pretrained the bert model with a domain-specific regimen on extensive biomedical corpora, leading to improved performance compared to the original bert across various text mining tasks in the biomedical field. the two most closely related works to our study are the research in the domain of mental health data analysis. @xcite implemented bert within the mental health sector by creating a targeted dataset from reddit, resulting in the development of a model known as mentalbert. subsequently, @xcite introduced a dualdomain adaptation process for language models pretraining, which involves firstly adapting the model to the social media text and then to the mental health domain. throughout both stages, the integration of lexical resources played a critical role in directing the language model's masking procedure, thereby ensuring a heightened focus on vocabulary associated with mental disorders. currently, there is no plm customized for the chinese mental health domain, and unique challenges exist within the chinese field. to bridge this gap, we collected around 4 million data from social media for domain adaptive pre-training in the chinese mental health domain. the guided masking mechanism based on the domain lexicon can help the model focus on the domain-related context. ## pretraining corpus ultimately, our corpus comprised over 4.04 million posts/comments from 434,207 users. after ## results we compared the performance of our proposed method, which includes two masking mechanisms (random masking and guided masking), with eight other models across six open datasets. the experimental results can be seen in tables 3 and 4 . for a detailed analysis, word2vec-cnn outperformed word2vec-bilstm in all tasks, although word2vec-based methods were generally have lower performance on all six datasets. es-pecially, on the "c2d2" data, their performance was 7.81% lower in f1-score compared to our model, highlighting their limitations. among the four pre-trained model-based methods (bert, chinese-bert-wwm-ext, dkplm-financial and dkplm-medical), performance differences across the six tasks were not great. in the best-performing "cognitive" task, the best model bert still had an 1.69% point lower f1-score than our model. also, in the "eswect-epidemic" task, the bestperforming bert model had an f1-score 3.65% lower than our model. for llms, chinese-alpaca-2 outperformed llama2-chinese-chat in all tasks. however, llms generally fell short compared to pre-trained model-based methods, except for the "eswect-usual" task, where chinese-alpaca-2 outperformed chinese-bert-wwm-ext by 1.19% points in f1-score. this indicates that traditional deep learning solutions still have advantages in domain-specific tasks. also fine-tuning llms can be challenging due to their large model sizes for model training. the superior performance of our models in all tasks is attributed to our domainadaptive training and guided masking mechanism in mental health domain. the performance of our model with random masking was lower than with the guided masking training approach in all tasks, especially in the waimai_10k task, where there was a 1.76% point f1-score gap. this shows that the good performance of our method is not only due to the collection of large datasets, but also benefits from the training mechanism based on lexicon guided masking. among the six tasks, we consider "cognitive" and "suicide" to be the best for validating model performance due to having the highest inter-method standard deviations (sd), which are 4.72 and 4.66, respectively. the high sds indicate that the differences between methods are meaningful, suggesting that performance on these tasks does not saturate too quickly and that the model's performance can effectively highlight the differences between methods @xcite . the best performance achieved by our model in these two tasks reconfirms the reliability of our approach. ## qualitative comparison we conducted a qualitative analysis to explore the behavior and tendencies of language models by predicting masked words, using questions from the symptom checklist-90 (scl-90 scale) @xcite as our experimental basis. the scl-90 scale, a widely recognized 90-item tool for evaluating mental health, assesses nine primary psychiatric symptoms and psychological distress. in this study, specific keywords in each question of the scl-90 scale were obscured, and the predictions made by chinese-bert-wwm-ext and chinese mentalbert-models trained through random or guided masking mechanisms were analyzed. the examples shown in the table reveal that the chinese-bert-wwm-ext model typically generates more general and less emotionally charged predictions compared to the proposed models. for instance, when faced with sentences expressing self-blame or difficulty breathing, the basic model opts for neutral words like "告诉" (tell) and "急性" (acute), which lack the emotional depth present in the context. in contrast, the proposed models, developed with a focus on psychological or emotional states, consistently select words that better capture the negative emotions or psychological nuances implied in the sentences, such as "折磨" (torture) and "困难" (trouble). comparing the two masking mechanisms (random and guided), highlights their different tendencies to word prediction. while both are inclined towards psychological and emotional expressions, the model trained with guided mechanism exhibits a clearer focus on accurately capturing the emotional context of the sentences. for example, in predicting masked words related to thinking about death, the guided model predicted words that directly relate to emotional states like "难过" (sadness) and "伤心" (grief), demonstrating its enhanced sensitivity towards psychological vocabularies. this suggests that the guided training mechanism employed in the model improves its ability to predict emotionally relevant context. ## conclusion in this paper, we present chinese mentalbert, the first adaptive pre-trained language model for the chinese mental health domain to the best of our knowledge. the model features a simple yet effective domain adaptive framework, and experiments have shown its strong performance in chinese psychology-related tasks. the domain lexiconguided masking mechanism used in this study can adjust the model's tendency to enhance the perfor- sentence masked words prediction chinese-bert-wwm-ext ours-random ours-guided chinese: 经常责怪自己 告诉(tell) 折磨(torture) 折磨(torture) [mask] chinese: 经常[mask][mask]自己 提问(question) 怀疑(doubt) 怀疑(doubt) english: often blame myself 反问(counter-question) 伤害(hurt) 压抑(depress) [mask] english: often [mask] myself 暗笑(chuckle) 提醒(warn) 伤残(disable) chinese: 呼吸有困难 急性(acute) 问题(question) 困难(trouble) [mask] chinese: 呼吸有[mask][mask] english: having trouble breathing 气性(temperament) 限制(limit) 压力(pressure) [mask] english: having [mask] breathing chinese: 想到死亡的事 以前(before) 以前(before) 难过(saddness) [mask] chinese: 想到[mask][mask]的事 english: thinking about death 过去(past) 好多(a lot) 伤心(grief) [mask] english: thinking about [mask] in future research, we plan to validate our model across a broader range of data types and tasks, including analyzing mental health interview data and summarizing psychology-related content. additionally, we aim to explore the use of diverse lexical resources tailored to specific tasks and employ clinical data to develop more specialized language models.