What to Fuse and How to Fuse: Exploring Emotion and Personality Fusion Strategies for Explainable Mental Disorder Detection (2023)

Abstract: Mental health disorders (MHD) are increasingly prevalent worldwide and constitute one of the greatest challenges facing our healthcare systems and modern societies in general. In response to this societal challenge, there has been a surge in digital mental health research geared towards the development of new techniques for unobtrusive and efficient automatic detection of MHD. Within this area of research, natural language processing techniques are playing an increasingly important role, showing promising detection results from a variety of textual data. Recently, there has been a growing interest in improving mental illness detection from textual data by way of leveraging emotions: ‘Emotion fusion’ refers to the process of integrating emotion information with general textual information to obtain enhanced information for decision-making. However, while the available research has shown that MHD prediction can be improved through a variety of different fusion strategies, previous works have been confined to a particular fusion strategy applied to a specific dataset, and so is limited by the lack of meaningful comparability. In this work, we integrate and extend this research by conducting extensive experiments with three types of deep learning-based fusion strategies: (i) feature-level fusion, where a pre-trained masked language model for mental health detection (MentalRoBERTa) was infused with a comprehensive set of engineered features, (ii) model fusion, where the MentalRoBERTa model was infused with hidden representations of other language models and (iii) task fusion, where a multi-task framework was leveraged to learn the features for auxiliary tasks. In addition to exploring the role of different fusion strategies, we expand on previous work by broadening the information infusion to include a second domain related to mental health, namely personality. We evaluate algorithm performance on data from two benchmark datasets, encompassing five mental health conditions: attention deficit hyperactivity disorder, anxiety, bipolar disorder, depression and psychological stress.

## introduction mental health disorders (mhd) are increasingly prevalent worldwide and constitute one of the greatest challenges facing our healthcare systems and modern societies in general. in response to this societal challenge, there has been a surge in digital mental health research geared towards the development of new techniques for unobtrusive and efficient automatic detection of mhd. within this area of research, natural language processing techniques are playing an increasingly important role, showing promising detection results from a variety of textual data. recently, there has been a growing interest in improving mental illness detection from textual data by way of leveraging emotions: 'emotion fusion' refers to the process of integrating emotion information with general textual information to obtain enhanced information for decision-making. however, while the available research has shown that mhd prediction can be improved through a variety of different fusion strategies, previous works have been confined to a particular fusion strategy applied to a specific dataset, and so is limited by the lack of meaningful comparability. as a result, the clinical community is increasingly seeking new approaches to the early detection and monitoring of mental health problems that can greatly improve the effectiveness of interventions, reduce their cost, and prevent them from becoming chronic. in this context, natural language processing (nlp) is recognized as having transformative potential to support healthcare professionals and stakeholders in the early detection, treatment and prevention of mental disorders (for comprehensive reviews, see @xcite @xcite . data from social media are particularly appealing to the nlp research community due to their scope and the deep embeddedness in contemporary culture @xcite . research utilizing nlp techniques in combination with social media has yielded new insights into population mental health and shown promise for incorporating datadriven analytics into the treatment of psychiatric disorders @xcite . recently, this line of research has developed a growing interest in improving nlp approaches to mental illness detection by leveraging information from related domains, in particular emotion (see @xcite . behavioral and psychological research has long established links between emotions and mental disorders: for example, individuals with depressive symptoms have difficulty regulating their emotions, resulting in lower emotional complexity @xcite . disrupted emotion regulation has also been implicated in anxiety @xcite . in the light of such links, information about emotions is useful in diagnosing mental disorders. 'emotion fusion' refers to the process of "integrating emotion information with general textual information to obtain enhanced information for decision-making" @xcite . by the same rationale, information fusion approaches are likely to benefit from the inclusion of additional individual characteristics known to be associated with mental disorders, such as personality traits. like emotion, personality has been linked to a diverse set of mental disorders based on genetic and behavioral evidence: for example, genomewide association studies have demonstrated that genetic risk factors for depression are largely shared with the neuroticism peronality trait @xcite . correlational studies comparing subjects diagnosed with major depressive disorder (mdd) and healthy control subjects found that vulnerability to depression was associated with several personality dimensions, such that mdd subjects were characterized by high neuroticism and low extraversion, accompanied by low scores on openness and conscientiousness @xcite . analyses language of use of twitter users with self-disclosed depression and ptsd revealed that text-derived personality played s an important role in predicting the mental disorders @xcite . in addition to the question of 'what to fuse', information fusion approaches also raise the algorithmic question of 'how to fuse' the auxiliary information effectively. the available research has shown that mhd prediction can be improved through a variety of different fusion strategies. however, previous work has typically focused on a specific fusion strategy applied to a specific dataset, limiting their comparability. in this work, we integrate and extend research on information fusion for mental disorder detection by conducting extensive experiments with three types of deep learning-based fusion strategies: (i) feature-level fusion, where a pre-trained masked language model for mental health detection (men-talroberta, htt) was infused with a comprehensive set of engineered features, (ii) model fusion, where the mentalroberta model was infused with hidden representations of other language models and (iii) task fusion, where a multi-task frame-work was leveraged to learn the features for auxiliary tasks. in addition to exploring the role of different fusion strategies, we expand on previous work by broadening the information infusion to include a second domain related to mental health, i.e. personality. we evaluate our model on data from two benchmark datasets, encompassing five mental health conditions: attention deficit hyperactivity disorder, anxiety, bipolar disorder, depression and psychological stress. 1 the remainder of the paper is structured as follows: section 2 presents a concise discussion of related work applying each of the three information fusion strategies. section 3 introduces the datasets used to perform the mental health detection experiments. in section 4, we describe our three mental status detection models that instantiate the three fusion strategies. section 5 details the experimental setup including the specification of the fine-tuned mentalroberta model baseline model. section 6 presents and discusses the results of our experiments. finally, we conclude with possible directions for future work in section 7. ## related work in this section we provide a concise discussion of selected works for each of the three fusion strategies. a comprehensive overview of work information fusion for mental illness detection from social media data has recently been provided by @xcite . one strand of recent work in the feature-level fusion approach is characterized by the integration of information from several groups of features extracted using nlp tools @xcite utilized a feature attention network (fan) to combine indicators of mental disorders from four groups: (1) word-level features related to depressive symptoms taken from the diagnostic and statistical manual of mental disorders (dsm-5, apa, 2013), (2) word-level sentiment scores of obtained from the sentiwordnet dictionary @xcite , (3) features related ruminative thinking, expressed as the amount of repetition of topics in a social media post @xcite and (4) writing style features, measured in terms of the sequencing of part-of-speech in a social media. the fan consists of four feature networks -one for each feature groups -fed into a post-level attention layer. the authors eval-uated the performance of their approach on the reddit self-reported depression diagnosis dataset @xcite al. (2017) ), a large scale general forum dataset contaning data from 9,210 users with an average of 969 posts for each user. their model was competitive with a convolutional neural network baseline model, despite using a much smaller number of posts in training data (only 500 posts per user). a second strand of feature-fusion approaches combines emotion features extracted using nlp tools with textual embeddings from pretrained language models, before feeding these into a cnn/lstm structure to construct the mhc classification model. for example, @xcite used a hierarchical attention network with lstm post-level and user-level encoders that combined multi-dimensional representations of texts. specifically, their approach combined (i) content features, captured through word sequences encoded as 300-dimenional embeddings based on pre-trained glove vectors @xcite , (ii) style features, expressed by numerical vectors representing stopword frequencies as bag-of-words, normalized by text lengths and usage of pronouns or other parts of speech, and (iii) emotion and sentiment features, represented by numerical vectors of word category ratios from two emotion-and sentimentrelated lexicons, liwc @xcite and nrc emotion @xcite . they evaluated the model on the erisk reddit datasets on depression, anorexia and self-harm @xcite , reaching competitive result across all three mental disorders, outperforming a strong roberta baseline model in the detection of two of them (self-harm and depression). turning to the model fusion approach, @xcite presented a time-aware transformer based model for the screening of suicidal risk on social media. their model, called statenet, uses a dual transformer-based architecture to learn the linguistic and emotional cues in tweets. statenet combines the 768-dimensional encoding obtained from sentence bert, capturing the language cues of the tweet to be assessed, with an aggregate representation of the emotional spectrum, obtained from a pre-trained bert model fine-tuned on the the emonet dataset @xcite . this second model, referred to as the plutchik transformer, tokenizes each post and adds the [cls] token at the beginning of each post. the authors then express the the aggregate represen-tation of the emotional spectrum as the the final hidden state corresponding to this [cls] token (768-dimensional encoding). they evaluated the statenet models on the task of tweet-level prediction of suicide idation on the twitter timeline dataset @xcite , which contained 32,558 tweets. statenet significantly outperforms competitive baselines models for suicidal risk assessment, demonstrating the utility of combining contextual linguistic and emotional cues for suicide risk assessment. recently, @xcite explored the use of multi-task learning and emotion-infused language model finetuning for psychological stress detection. in this work, the authors introduced an innovative task fusion approach that utilized a multi-task learning setup to perform stress detection and emotion detection at the same time on the same input data. as currently available datasets for stress detection are not labeled for emotion, they first separately trained bert models on different versions of the goemotions dataset @xcite and employed these to derive emotion labels for the stress detection dataset used in their experiments @xcite . the authors then used these emotion labels as 'silver data' to train on them alongside stress in a multi-task learning setting with hard parameter sharing @xcite . their models achieved comparable performance to a state-of-the-art fine-tuned bert baseline. importantly, based on analyses designed to probe their models and discover what information they learn to use, the authors demonstrated that their task fusion approach improved the explainabilty of deep learning-sbased mental health prediction models. specifically, by performing correlational analyses of the models predictions on each task, they were able to explore the usefulness of the emotion prediction layers in explaining stress classifications. as can be seen from this overview, with the exception of @xcite , previous studies have focused on specific fusion strategies applied to a variety of mental health conditions. by applying different fusion strategies to five mental disorders (ahdh, anxiety, bipolar disorder, depression) and related symptomatology (psychological stress), we aim to facilitate the evaluation of current approaches to information fusion. ## data four datasets were used in the present work: the data used in the task of mental health detection were obtained from two publicly available social media datasets: (1) the self-reported mental health diagnoses (smhd) dataset @xcite and ( 2 ) the dreaddit dataset @xcite . both smhd and dreaddit were constructed from data from reddit, a social media platform consisting of individual topic communities called subreddits, including those relevant to mhc detection. the statistics of these datasets are provided in dreaddit is a dataset of social media posts from subreddits in five domains that include stressful and non-stressful text. for a subset of 3.5k users employed in this paper, binary labels (+/-stressful) were obtained from crowdsourced annotations aggregated as the majority vote from five annotators for each data point. as the smhd and dreaddit datasets are la-beled only with mental health status, two additional datasets were used to provide auxiliary information about personality and emotion. following the approach used in @xcite , we first separately trained roberta models on the goemotions dataset @xcite and the kaggle mbti dataset @xcite and used these models to predict emotion and personality labels for smhd and dreaddit. a table with dataset statistics for these resources is provided in the appendix. goemotions is the largest available manually annotated dataset for emotion prediction. it consists of 58 thousand reddit comments, labeled by 80 human raters for 27 emotion categories plus a neutral category. the authors provided a mapping of these 27 categories to ekman's six basic emotions (anger, disgust, fear, joy, sadness, and surprise), which are assumed to be physiologically distinct @xcite . drawing on the results of experiments with different emotion mappings reported in @xcite , these six basic emotions are used in the present work. the kaggle mbti dataset was collected through the personalitycafe forum foot_1 and thus provides a diverse sample of people interacting in an informal online social environment. it consists of samples of social media interactions from 8675 users, all of whom indicated their myers-briggs type indicator (mbti) personality type @xcite . the mbti is a widely administered questionnaire that describes personality in terms of 16 types that result from combining binary categories from four dimensions: (a) extraversion/introversion (e/i) -preference for how people direct and receive their energy, based on the external or internal world, (b) sensing/intuition (s/n) -preference for how people take in information, through the five senses or through interpretation and meanings, (c) thinking/feeling (t/f) -preference for how people make decisions, relying on logic or emotion over people and particular circumstances, and (d) judgment/perception (j/p) -how people deal with the world, by ordering it or remaining open to new information. ## models we experiment with seven information-infusion models that differ (i) in the type of information to be infused (personality, emotion, both) and (ii) the fusion strategy applied to incorporate that information into the mental health detection models. the architectures of these models is shown in figure 1 . ## results and discussion building upon the approach described in @xcite , we go a step further to probe our full task fusion models and discover the exact nature of the information it learned to use, i.e. how the six basic emotion categories (anger, disgust, fear, joy, sadness, and surprise) and four personality dimensions (extraversion/introversion (e/i), sensing/intuition (s/n), thinking/feeling (t/f) and judgment/perception (j/p)) guided the prediction of mental health status. to this end, we calculated pearson correlation coefficients between the predicted probabilities for each of the five mental health conditions and the probabilities for the four personality and six emotion categories. in line with results from experimental and genome-wide association studies of mental health and personality @xcite , these results suggest that personality dimensions are important in understanding vulnerability to mental health disorders. ## conclusion in this work, we presented the first comprehensive experimental evaluation of current deep learningbased fusion strategies (feature-level fusion, model fusion, task fusion) for the detection of mental disorders. we go beyond previous work by applying these approaches to five mental health conditions. the results of our experiments showed that the task fusion strategy is most promising for the detection of three of the five conditions (adhd, anxiety, and bipolar disorder), while feature-level fusion is most advantageous for the detection of psychological distress and depression. we demonstrated that the prediction of mental health from textual data benefits from the infusion of two information sources related to mental disorders, i.e. emotion and personality. furthermore, we show that information fusion models can improve the classification accuracy of strong transformer-based prediction models while enhancing their explainability. in this paper, we focused on developing binary classifiers that aim to distinguish between individuals with a particular mental illness and control users. in future work, we intend to addresses the more complex problem of distinguishing between multiple mental health conditions, which is essential if we are to uncover the subtle differences among the statistical patterns of language use associated with particular disorders. we further intend to employ our approach to longitudinal data to gain valuable insights into the evolution of symptoms over time and extend it to languages beyond english, specifically german.