SMHD - GER : A Large-Scale Benchmark Dataset for Automatic Mental Health Detection from Social Media in G erman (2023)

Abstract: Mental health problems are a challenge to our modern society, and their prevalence is predicted to increase worldwide. Recently, a surge of research has demonstrated the potential of automated detection of mental health conditions (MHC) through social media posts, with the ultimate goal of enabling early intervention and monitoring population-level health outcomes in real-time. Progress in this area of research is highly dependent on the availability of high-quality datasets and benchmark corpora. However, the publicly available datasets for understanding and modelling MHC are largely confined to the English language. In this paper, we introduce SMHD-GER (Self-Reported Mental Health Diagnoses for German), a large-scale, carefully constructed dataset for MHC detection built on high-precision patterns and the approach proposed for English. We provide benchmark models for this dataset to facilitate further research and conduct extensive experiments. These models leverage engineered (psycho-)linguistic features as well as BERT-German. We also examine nuanced patterns of linguistic markers characteristics of specific MHC.

## introduction mental health is a major challenge in healthcare and in our modern societies at large @xcite . the world health organization estimates that 970 million people worldwide suffer from mental health conditions 12 , with the rate of undiagnosed mental disorders estimated to be as high as 45% @xcite . the enormous societal impact of mental health conditions (mhc) requires prevention and intervention strategies that focus primarily on screening and early detection. the last decade has seen a surge in digital mental health research, an interdisciplinary line of research that brings together insights from computational linguistics, cognitive psychology and computational social sciences to understand the relationship between patterns of language use and mental health conditions (d'alfonso, 2020; @xcite . natural language processing, in particular, is increasingly recognized as having transformative potential to support healthcare professionals in the diagnosis and treatment of mental disorders and enable people to lead healthy lives (see @xcite @xcite for recent overviews of this research). progress in this area of research is highly dependent on the availability of high-quality datasets and benchmark corpora. social media has emerged as an increasingly vital resource for obtaining such data, as it is now a central place for individuals to participate in discussions, share information, and seek advice. based on data drawn from platforms such as twitter and reddit, recent work has developed scalable methods for constructing mental health datasets based on self-reported diagnoses or grouping individuals based on activity patterns @xcite @xcite . however, recent reviews on the state of data used for mental health status on social media show that the vast majority of the publicly available datasets for understanding and modeling mhc are on the english language: for example, of the 102 datasets reviewed in @xcite 83% were on english with the remaining 17% distributing over five other languages: chinese (9.8%), japanese (3.9), korean (1.9), spanish and portuguese (@xmath0. @xcite report that 81% of all the datasets are in english, followed by datasets in chinese (10%), and arabic (1.5%). while an overwhelming focus on english data is a theme throughout the nlp community, it is a specific concern in this domain where culture often influences the presentation of mental health disorders @xcite . thus, there is an urgent need for publicly available, high-quality mental health datasets and benchmark models to support early detection of mhc in languages other than english. the main contributions of this work are threefold: (1) we introduce smhd-ger (self-reported mental health diagnosis for german), a new large dataset of social media posts for mental health detection in the german language, and make it publicly available; (2) we provide benchmark models for the detection of four mental health conditions based on a comprehensive set of text-based features that pertain to multiple levels of language use, the german bert-based model, and hybrid models that combine the two; and (3) we uncover nuanced patterns of linguistic markers characteristic of specific mental health conditions. the remainder of this paper is organized as follows: in section 2 we briefly review available social media datasets and nlp classification methods for mhc detection. section 3 details the construction of the smhd-ger dataset along with an ethics and privacy statement. section 4 presents the results of a analysis of linguistic markers of specific mhc. in section 5, we describe the modeling approach for our benchmark models, and in section 6, we present and discuss the results. finally, we conclude with directions for future work in section 6. ## related work in this section, we provide a concise overview of some of the most widely used self-disclosure social media datasets along with the classification methods used in the detection of mental health conditions. the self-disclosure approach to obtaining labeled data from social media was introduced in @xcite and further refined in consecutive work @xcite . in this approach, public self-reports of mental health diagnoses are identified through the use of carefully designed 'diagnosis patterns' combined with 'diagnosis keywords' mapped to particular mental health conditions: a user is included for a specific mhc if one of the condition keywords occurs within a certain distance of the diagnosis pattern. @xcite originally applied this approach to twitter data and identified approximately 1,200 users with four mhc (bipolar, depression, ptsd, sad) by matching diagnosis patterns in their tweets (e.g., "i was diagnosed with depression"). this dataset was employed in the shared task at the 2nd computational linguistics and clinical psychology workshop (clpsych 2015) that focused on identifying depression and ptsd users on twitter @xcite . submissions to the task used traditional (shallow) classification models trained on unigram vectors, character language models, closed-vocabulary approaches (e.g. @xcite and supervised topic models. the leading systems reached average precision rates over 85% for both mhc. however, the dataset had a balanced distribution between the classes, rather than one that accurately reflect the user population. this hampered the reliable estimation of actual false alarm rates, as the number of false alarms in the general population is estimated to be 7-15 times higher than in the clpsych 2015 test sample @xcite . the text content of a tweet can contain up to 280 characters or unicode glyphs. thus, this format presents a barrier to capturing mental health related language signals. recent work on compiling datasets for mental health is increasingly turning to reddit for long-form content that can provide additional linguistic insights foot_2 : @xcite applied the self-disclosure approach to create the reddit self-reported depression diagnosis (rsdd) dataset, which contains 9,210 users with depression and 107,274 control users. apart from increasing the dataset size by an order of magnitude -969 posts per user with mean post length of 148 words, the rsdd dataset displays a realistic number of control users matched with each diagnosed user. the main limitation of the rsdd dataset is its focus on a single mental health condition, depression. in what is to our our knowledge the most comprehensive, carefully constructed mental health dataset based on the self-disclosure approach, @xcite expand on rsdd by including for eight additional mhc: the self-reported mental health diagnoses (smhd) dataset, whose design underlies the current work, comprises 20,406 diagnosed users and 335,952 matched controls. diagnosed users were identified using a refined version of the high precision diagnosis patterns used in rsdd, which incorporated synonyms in matching patterns from two synonym mapping ontologies (medsyn, @xcite . control users were selected based on a similar reddit posting activity, i.e. each daignosed user was matched with an average of 9 control users with a similar number of posts and a similar range of subreddits they posted in. importantly, smhd does not contain any posts that contain any mental health terms or that have been posted in a mental health-related subreddits. the detection of mhc can thus not be based on terms associated with specific mental health conditions. along with the dataset itself, @xcite provided benchmarks for both binary (mhc vs. control) and multi-class classification settings. the classification methods included several traditional (shallow) machine learning models (logistic regression, xgboost @xcite , support vector machine with linear kernel) trained on tf-idf bag-of-words features, a shallow neural net model trained on character ngrams (supervised fasttext, @xcite , and a convolutional neural network trained on ngram sequences represented by the fasttext embeddings. subsequent work has improved mhc detection accuracy using hierarchical attention networks @xcite and attention-based model using bert representations @xcite . recently, @xcite leveraged transformer language models (bert @xcite in combination with attention-based blstm models trained on engineered language features for mhc detection. ## analysis of linguistic markers in this section, we address the exploration of nuanced patterns of linguistic markers that are indica-tive of specific mhc. we first obtained measurements of 117 engineered language features that can be roughly divided into five groups: (1) features related to morphological and syntactic structural complexity (@xmath1, (2) features related to lexical sophistication, variety, and richness (@xmath2, (3) word-level ngram features related to register-specific language use (@xmath3, (4) features covering the german version of the liwc (linguistic inquiry and word count) dictionary (@xmath4, and (5) word-level dictionary features from three lexicons related to emotion, affect and sentiment (@xmath5. an overview of these features can be found in the first group of includes surface features related to the length of production units, such as the average length of clauses and sentences, and the type and frequency of embedded structures, such as mean length of sentence or number of dependent clauses per sentence @xcite . this group also includes an information-theoretic feature based on the deflate algorithm @xcite . the second group of features probing lexical density features, such as the ratio of the number of lexical (as opposed to grammatical) words to the total number of words in a text, lexical variation, i.e. the range of vocabulary as manifested in language use as captured by text-size (corrected) type-token ratio @xcite . the third group comprises register-based n-gram frequency features that take into account both frequency rank and the number of word n-grams (@xmath6, 5]). the latter were derived from four corpora compiled as to represent language use in four language registers (academic, fiction, news, spoken; see the fourth feature group is based on the german version of the liwc dictionary (linguistic inquiry and word count) @xcite . the creators of the german version of the liwc validated this version and demonstrated that the german liwc categories have a high degree of equivalence to their english counterparts @xcite . building on the results of previous studies using liwc categories for mhc detection in english (e.g. @xcite , we expect that subcategories of particular interest for the mhc classification task will include words with positive or negative emotions, words related to social processes (family/friends/society), pronouns that can capture inclusive (we, us) or exclusionary (you, they, them) language use, and words related to how the person feels (sad, anxious). the fifth group includes features from three lexicons: meemolon @xcite ) is a lexicon comprising eight emotional variables with more than 100k lexical entries for eight emotional variables: valence, arousal, dominance, and joy, anger, sadness, fear, and disgust. angst is the german adaptation of the affective norms for english words @xcite . it comprises 1,003 german translations of the anew material that were were rated on a total of six dimensions: the three original scales for valence, arousal, and dominance plus three additional arousal ratings on an adapted scale. sentiws @xcite is a dictionary containing 3,468 sentiment bearing german words (1,650 negative and 1,818 positive) across four word classes (adjectives, adverbs, nouns and verbs) along with their weighted senti-ment scores. all measurements of these features were obtained using an automated text analytics system that employs a sliding window technique to compute measurements at the level of individual sentences. these measurements capture the within-text distributions of scores for a given feature (for recent applications, see e.g. wiechmann et al., 2022 or @xcite . tokenization, sentence splitting, part-of-speech tagging, lemmatization and syntactic pcfg parsing were performed using stanford corenlp @xcite . examples of these within-text distributions is shown in figure 1 . each of panels in figure 1 shows the distributions of four of the 117 textual features for one of six randomly selected texts representing different mhc groups. we note that the distribution of feature values is generally not uniform, but shows large fluctuations over the course of the text. the six texts are characterized by different patterns of spikes of specific features: for example, the bipolar text exhibits a large spike in the sentiws.pos feature, which refers to words with positive semantic orientation. the ocd text is characterized by regular peaks of the liwc.self feature, which captures self-focused language. the anxiety text displays frequent spikes of high values (>2 standard deviations from corpus average) for three of the four features. in comparison, the control text shows less fluctuation with features scores being closer to the corpus average values. the classification mod-els described in section 5.1 are designed to detect and exploit these fluctuations for the detection of specific mhcs. the average scores of all features across all groups are provided in to identify profiles of language use that are characteristic of particular mhc, we compare these feature scores across users in each mhc group using factorial analyses of variance (anova). we focus on those features that display significant differences across groups (@xmath7, for @xmath8.05). figure 2 presents a cluster heatmap visualizing the patterns in the data matrix with the mhc groups and the 16 most significant language features. the results of these analyses revealed some interesting patterns of differential language use: we find that the control group is situated at the margin of the clustering, indicating that the patterns of language use of diagnosed mhc are distinguishable from this baseline. the language use of anxiety is distinctly different from all other mhc. it is characterized by very high feature scores on five liwc dimensions related words referring to self-reference, death and sadness. they are further characterized by high scores on the top feature cluster, comprising words referring to anger, fear, disgust, sadness, arousal and negative emotions. the language use of schizophrenia, is similar to anxiety in that it too displays a larger proportion of words indicating negative emotions. however, it is characterized by low scores on liwc dimensions related words referring to self-reference. they are also characterized by low scores on the n-gram frequency features, indicating dependence on conventional phrases from specific speech registers. a striking feature of obsessive-compulsive disorder (ocd) is its heavy reliance on such terms. a characteristic feature of (unipolar) depression is a markedly increased use of words with positive semantic orientation, in stark contrast to bipolar depression, which has significantly lower scores on this dimension. this is intriguing in light of the fact that distinguishing between bipolar disorder and recurrent unipolar depression is a major clinical challenge (de almeida and phillips, 2013). in general, conditions of depression and bipolar disorder, attention-deficit/hyperactivity disorder (adhd) and post traumatic stress disorder (ptsd) display similar patterns of language use. these findings reflect evidence in the psychiatric literature indicating that there is considerable overlap in clinical symptoms and pathophysiological processes and that depressive symptoms may also occur in the context of another psychiatric disorder (e.g., bipolar disorder) @xcite . furthermore, psychiatric data suggest that depressive disorders (i.e., major depressive disorder and dysthymia) are highly comorbid with other common mental disorders @xcite . ## results and discussion ## conclusion and future work we introduced smhd-ger, a large dataset of reddit users with diverse mental health conditions and matched control users. the dataset was created using adaptations of the high-precision diagnostic patterns developed for the original english version @xcite . furthermore, we investigated the differences in language use between users with mental health conditions and control groups, as measured by a large set of linguistic and psychological cues. we provided strong benchmark models designed to identify diagnosed users for the four most frequently attested mhc. we found that blstm networks trained on within-text distributions of interpretable linguistic features consistently outperformed a transformer-based model based on gbert. a hybrid model combining the two approaches proved to be the most effective method for two of the four conditions. we make our dataset available to the community in the hope that it will encourage further research into these problems and improve the reproducibility of suggested approaches.