E mo M ent: An Emotion Annotated Mental Health Corpus from Two S outh A sian Countries (2022)

Abstract: People often utilise online media (e.g., Facebook, Reddit) as a platform to express their psychological distress and seek support. State-of-the-art NLP techniques demonstrate strong potential to automatically detect mental health issues from text. Research suggests that mental health issues are reflected in emotions (e.g., sadness) indicated in a person’s choice of language. Therefore, we developed a novel emotion-annotated mental health corpus (EmoMent),consisting of 2802 Facebook posts (14845 sentences) extracted from two South Asian countries - Sri Lanka and India. Three clinical psychology postgraduates were involved in annotating these posts into eight categories, including ‘mental illness’ (e.g., depression) and emotions (e.g., ‘sadness’, ‘anger’). EmoMent corpus achieved ‘very good’ inter-annotator agreement of 98.3% (i.e. % with two or more agreement) and Fleiss’ Kappa of 0.82. Our RoBERTa based models achieved an F1 score of 0.76 and a macro-averaged F1 score of 0.77 for the first task (i.e. predicting a mental health condition from a post) and the second task (i.e. extent of association of relevant posts with the categories defined in our taxonomy), respectively.

## introduction mental health issues remain a leading cause for poor well-being and suicide. the world health organisation (who) indicates that 400 million people are affected by mental disorders such as depression, resulting in a cost of us$ 1 trillion per year from the global economy allocated for depression and anxiety disorders alone @xcite . recent research using ai and nlp demonstrates strong potential to automatically detect mental health issues from digital footprints such that professionals could provide timely interventions and mental health resources to vulnerable persons. these data contain useful information to understand patients' distressed state of mind outside a traditional clinical environment. research suggests that mental health issues are reflected in the 'emotions' (e.g. sadness, anger) indicated in one's expression of language. despite the popularity of research studies in detecting mental disorders using online data such as twitter @xcite @xcite and emotion modeling @xcite @xcite , the automated identification of the association between emotions and mental disorders have largely being ignored, apart from a recent study (cease corpus @xcite ) that focused on the role of emotions on suicidal ideation. motivated by this, we introduce a novel, emotion-annotated mental health (emoment) corpus foot_0 using facebook posts extracted from two south asian countries -sri lanka and india. in south asia, due to the lack of awareness of symptoms of mental illnesses and its associated stigma, people often do not seek professional help, resulting in many instances of mental disorders being left undiagnosed @xcite . however, since recently, these countries have demonstrated a tendency to use social media, particularly facebook, to seek mental health help using private and public groups (e.g. psychology group in sri lanka, indian psychology association). depression and anxiety disorders are amongst the most common mental disorders worldwide @xcite . therefore, our dataset includes de-identifiable facebook posts from individuals who have indicated a diagnosis of depression or anxiety, the disorder-related issues they express including associated emotions, and their help-seeking behaviours from professionals and/or community. emoment consists of 2802 posts (14845 sentences) extracted from public facebook groups dedicated to discuss mental health concerns in sri lanka and india. three clinical psychology postgraduates were involved in the data annotation process. their task was to read the entire post and assign one or more labels from a given set of eight categories (e.g. mental illness, sadness, psychosomatic, irrelevant) ( we fine-tuned bert @xcite and roberta @xcite based deep learning models on the emoment corpus to predict the relevance of a post to a mental health condition (first task), and to associate relevant posts with the categories defined in our taxonomy (second task) (see section 3.3). our roberta-based models achieved a f1 score of 0.76 for the first task, and a macroaveraged f1 score of 0.77 for the second task. the novel contributions of our paper includes; 1) the development of the first emotion-annotated mental health corpus in english language 2) the development of the first taxonomy to annotate mental health conditions and emotions from facebook data, and 3) the development and evaluation of deep learning models (roberta) to predict the presence of mental conditions, emotions, and psychosomatic issues with 'good' performance. additionally, our research contributed to the integration of knowledge from two domains -mental health and emotion modelling through various quantitative and qualitative analyses, in particular, low-resource languages such as sinhala. currently, the diagnosis of a mental disorder is primarily based on the knowledge and experience of a professional, who arrives at a diagnosis subsequent to talking to a patient and/or care-givers. in this method, patients have to reflect on events that occurred in the past to help professionals diagnose their condition. real-time experiences of patients, which is an important element for diagnosis and treatment plan, is not usually considered. the majority of online self-reflective posts on the other hand generate real-time, reliable data to uncover distressed states of mind at the time of occurring. therefore, a corpus like emoment, developed from user-generated data allows practitioners to understand the mental states of patients beyond a traditional clinical interview. these automated identification of mental disorders or mental conditions,from user-generated content provides a useful tool for improving diagnosis and personalised treatment plans. ## related work people use language as a direct tool to express their feelings and emotions, providing a wealth of information to determine their emotional status and mental health conditions @xcite . motivated by this, many datasets have been developed to support research in the two fields of: emotion modelling and mental health modelling, using social media as one of the primary data sources. the existing datasets on emotion modelling are mostly based on two emotion taxonomies: ekman's basic emotions (fear, anger, joy, sadness, disgust, and surprise) @xcite , and plutchik's wheel of emotions (anger, anticipation, disgust, fear, joy, sadness, surprise and trust) @xcite . examples of emotion modelling datasets include @xcite @xcite @xcite . the existing mental health modelling datasets are based on the problem domains such as suicidal attempts, self-injury, loneliness, depression, anxiety and post traumatic stress disorder. the focus of most existing datasets are limited to one or two problem domains, hindering the diagnosis capabilities of the ai models that they are based on. @xcite @xcite . despite the availability of numerous emotion modelling and mental health modelling datasets, there are certain limitations in almost all of these datasets. the datasets from these two research fields are independent of one another (i.e., noninteractive). the complementary integration of emotion and mental health modelling provide enhanced insights on a person's emotional and mental well-being, which is useful in assisting professionals to diagnose and personalise treatment plans. additionally, almost all of the currently available datasets are based on resource-rich languages such as english @xcite , limiting the understanding of cultural aspects of language use in emotion and mental health modelling. despite the importance of jointly-modelling emotions and mental health, the availability of emotion-annotated mental health datasets are vastly limited. for example, the emotion-annotated mental health dataset of ghosh et al. @xcite , cease, is specific to suicide notes. motivated by this, we present a novel emotion-annotated mental health dataset based on facebook data to facilitate joint-modeling of emotions and mental health conditions. to propose baseline models from the constructed datasets, most previous studies in emotion, mental health, and emotion-annotated mental health domains have utilised recent advancements of deep learning techniques such as bert, lstms and rnns @xcite . for example, the cease dataset proposes an ensemble model using lstm, cnn, and gru @xcite . to adhere with this, we also leveraged the recent advancements in deep learning techniques using bert and roberta based models. ## emoment corpus this section describes the development of the emo-ment corpus -data collection, data cleaning, taxonomy development, and data annotation. ## corpus analysis we constructed the emoment corpus by selecting posts which had two or more annotators agreeing on a category. ## conclusion this paper presented the first emotion-annotated mental health corpus -emoment, which was developed using facebook posts from two south asian countries -sri lanka and india. we have provided a comprehensive research study, demonstrating the development of an empirically-sound emotion-annotated mental health taxonomy using the grounded theory approach. we also developed strong baselines using roberta-based models and achieved an f1 score of 0.76 for the first task (i.e., predicting the relevance of a post to a mental health condition) and a macro-averaged f1 score of 0.77 for the second task (i.e., predicting the relevant labels in our taxonomy). however, our results suggest that there is ample room for future improvements in emotion-annotated mental health modelling. the models presented in the paper consider the emotionannotated mental health modelling as two separate tasks, one binary classification to determine the relevancy, and multi-label classification to predict fine-grained labels of posts. an interesting next step would be to co-model these two tasks by leveraging multi-task learning (mtl).