Towards Identifying Fine-Grained Depression Symptoms from Memes (2023)

Abstract: The past decade has observed significant attention toward developing computational methods for classifying social media data based on the presence or absence of mental health conditions. In the context of mental health, for clinicians to make an accurate diagnosis or provide personalized intervention, it is crucial to identify fine-grained mental health symptoms. To this end, we conduct a focused study on depression disorder and introduce a new task of identifying fine-grained depressive symptoms from memes. Toward this, we create a high-quality dataset (RESTORE) annotated with 8 fine-grained depression symptoms based on the clinically adopted PHQ-9 questionnaire. We benchmark RESTORE on 20 strong monomodal and multimodal methods. Additionally, we show how imposing orthogonal constraints on textual and visual feature representations in a multimodal setting can enforce the model to learn non-redundant and de-correlated features leading to a better prediction of fine-grained depression symptoms. Further, we conduct an extensive human analysis and elaborate on the limitations of existing multimodal models that often overlook the implicit connection between visual and textual elements of a meme.

## introduction mental health disorders have a profound impact on society. almost 1 billion people worldwide suffer from mental health disorders, predominantly depression, anxiety, mood, and substance use disorders @xcite . two of the most common mental health disorder, depression and anxiety account for the us @xmath06 trillion by 2030 @xcite . apart from the economic burden, the social burden of mental health disorders is huge. suicide is now the fourth leading cause of death among those aged 15 to 29 years old @xcite figure 1 : example of a depressive meme. if we merely evaluate the textual content ("these would give me a peaceful scene in a land of trouble"), it is difficult to establish the author's true feelings. however, the memeimage can provide complementary information to help recognize the depression symptom (self-harm) correctly. @xcite . however, considering its preponderance and global burden, depression continues to be significantly undertreated in all practice settings, where fewer than one-third of adults with depression receive effective treatment. denial of illness and stigma are the two most common obstacles to appropriate diagnosis and treatment of depression @xcite . recently, social media data has emerged as a powerful "lens" for tracking and detecting depression @xcite . the vast majority of the existing works on depression have utilized the textual or multi-modal information available in the social media data to primarily classify the posts based on the perceived depressive behavior (depressive or non-depressive) @xcite @xcite . however, for healthcare professionals to intervene and provide effective treatment, it is crucial for them to understand the leading symptoms of that depressive behavior. motivated by this, we aim to develop a practical decision support system that swift through social media posts and can provide healthcare professionals deeper insights into one's depressive behaviors by capturing the fine-grained depressive symptoms. in the past, there have been few attempts @xcite to capture the depression symptoms; however, they are confined to only textual information. recently, a new form of communication has emerged in social media: 'meme'. a meme usually consists of an expressive image embedded with a short block of text. it is designed to convey a complex idea or emotional state of mind, which is far easier to understand than a textual description of thoughts. they acknowledge a shared experience between the creator and the viewer and therefore have become a fast way of communication on social media. outside the mental health domain, there are numerous studies on meme processing and understanding for emotion detection @xcite , cyberbullying @xcite , and hateful meme detection @xcite . however, to our knowledge, none of the existing studies have yet leveraged the visual information available in memes specifically to capture the finegrained depression symptoms. there are two main reasons to consider the visual information: (i) according to a recent survey foot_0 , images appear in more than 42% of tweet; and (ii) textual information alone cannot capture the overall semantic meaning. for example, in figure 1 , considering only the text, "these would give me a peaceful scene in a land of trouble", would not be sufficient to identify the depressive symptoms or even to distinguish whether the meme is depressive or not. however, it is evident from the image that the meme expresses suicidal thoughts/intent. therefore, in order to obtain a holistic view of a post and accurately determine the depression symptoms, it is necessary to take into account the visual information available in social media posts. to this end, we propose a new task -fine-grained depression symptom identification from memes and hypothesize that leveraging the multi-modal information available in the memes can more effectively help identify depression symptoms from social media posts. towards this, we utilize clinically established 9-scale pa-tient health questionnaire (phq-9) @xcite depression symptoms categories to classify the depressive memes that we collected from two popular social media forums -reddit and twitter. in particular, we make the following contributions: ## related works based on the data modalities, we categorize the existing works on depression detection as follows: language as highlighted in @xcite study, people's thoughts are frequently reflected in their language, and the linguistic cues, such as informal language usage, first-person referencing, and greater usage of negative emotion words, generally typify psychiatric disorders @xcite . numerous research in computational linguistics has modeled the language usage in mental health-related discourse to predict mental health states @xcite and infer risk to various mental disorders using social media data @xcite @xcite @xcite . most of the earlier works utilized a feature-driven approach @xcite to detect depression. recently with the availability of multiple benchmark datasets @xcite , existing methods are designed using neural models @xcite . while most of these existing work studies depression at a coarser level, there have been only a few efforts towards inferring depressive symptoms by analyzing the textual information in social media posts @xcite . vision the visual information available in shared images offers valuable psychological cues for understanding a user's depression status. previous studies @xcite @xcite conducted in a clinical setting have established that certain non-verbal behaviors such as downward gaze angle, dull smiles, and shorter average lengths of a smile characterize depressive behaviors. recently, with the popularity of photo and video-sharing social networking services such as instagram have piqued the interest of researchers in investigating people's depressive behavior from their visual narratives. reece and danforth (2017); manikonda and de choudhury (2017) investigated the role of public instagram profiles in identifying a depressive user. multimodal (language+vision+speech) in recent years, there has been growing attention towards exploiting multimodal information such as speech, vision and text for depression detection @xcite @xcite . existing studies have devised several neural approaches to effectively combine the information from various modalities. for instance, @xcite utilized the hierarchical bidirectional lstm network to extract and fuse the local video and audio features to predict the degree of depression. @xcite proposed a multi-agent reinforcement learning method for identifying depressive users. @xcite developed the topic-enriched multi-task learning framework that achieved stateof-the-art performance on multimodal depression detection tasks. in contrast to the above approaches, our study aims to find the fine-grained depression symptom from memes that have not yet been explored before. ## dataset creation in this section, we present a new benchmark dataset: restore 2 for identifying fine-grained depressive symptoms from memes, that was created following a clinically-guided approach and includes contributions from medical informatics experts and psychologist at each phase. 2 the dataset is available at https://forms.gle/ qe5fygesbzr1siud7 3.1 task structure dataset selection and curation. we collect posts from two popular social media platforms: twitter and reddit. we chose these platforms as a data source because of their rising popularity among depressive users to publicly express their thoughts, moods, emotions, and feelings. this, coupled with the greater degree of anonymity, facilitates self-disclosure and allows users to be more truthful and open in sharing sensitive issues and personal life without fear of being embarrassed or judged. thus these user-generated self-narratives provide low-cost, large-scale, non-intrusive data to understand depressive behavior patterns and outcomes outside the controlled clinical environment, both in real-time and longitudinally. to ensure that we capture a broader spectrum of depressive behaviors, we use a domain-specific depression lexicon @xcite . the lexicon contains depression-related terms from 8 symptom categories following the phq-9 foot_1 questionnaire. we use the depression lexicon to collect tweets from twitter public profiles that mention at least one of the words from the lexicon in their profile description. in a similar manner, we collect reddit posts; however, we restrict ourselves to the following subreddits: "mental health", "depression", "suicide watch", "depression memes", "eating disorder", and "sleeping disorder". objective. given a meme (containing image and an embedded text) and an 8 fine-grained phq-9 depression symptom categories, the goal is to identify all depression symptoms that are expressed in the meme. ## restore dataset analysis visual analysis. we conducted a visual analysis of the memes to study how depression symptoms are related to color features. we performed color analysis by computing the pixel-level average w.r.t hsv (hue, saturation, and value), in which lower hue scores imply more redness and higher hue scores suggest more blue. saturation describes an image's vibrancy. value relates to the brightness of an image, with lower scores indicating a darker image. we observe that most of the memes, irrespective of symptom categories, are less colorful and have lower saturation values, suggesting negative emotions. these cases were prominent in "low self esteem", "lack of interest", and "self harm", where users often share memes that were less vivid, darker (higher grayscale), and have a high hue. in contrast, the memes related to "eating disorder" are brighter and more colorful, mainly because of the presence of food in the memes. ## proposed approach existing multimodal approaches focus on generating text-image feature representations by detecting the objects in the image and learning an alignment between textual and visual tokens. however, a meme can convey multiple perspectives, and detecting the object alone may not be sufficient to generate a semantically-rich text-image representation. therefore, to capture the image's multiple views that could be beneficial in effectively distinguishing the depression symptoms, we introduce orthogonal feature generation in a multimodal setting. we begin by first encoding the meme image i and its embedded text t with the pre-trained resnet-152 model and bert model, respectively. we selected these models because of their simplicity and comparable performance to other languagevision models. to capture multiple perspectives of the image, we perform the 2-dimensional adaptive average pooling (adaptive-avg-pool) of output size s 0 × s 1 on resnet-152 model output f that results in image representation h @xmath1. with this approach, we obtained feature representations h 1 @xmath2. ## implementation details we utilized the pre-trained weights of bertbase foot_2 , roberta-large foot_3 , mentalbert foot_4 and xlnet-base foot_5 from huggingface @xcite . for the pre-trained vision models, we followed the torchvision api 9 and obtained the pre-trained weights of the vision models. in particularly, we use resnet152, resnext101_32x8d, densenet161, efficientnet_b4, regnet_y_800mf, vit_l_32, and convnext_large pre-trained weights to fine-tune on the phq-9 depression symptom identification task. we use the huggingface implementation 10 of visualbert to fine-tune the model on the phq-9 depression symptom identification task. for mmbt foot_8 and clip foot_9 also we follow the huggingface implementation and fine-tune the model on phq-9 depression symptom identification task. for the visual analysis of the restore dataset, we use the open-cv python library foot_10 . we fine-tuned each model on the restore training dataset for 10 epochs. the length of the maximum original text is set to 256 tokens. we normalized the images with pixel mean and standard deviation values before feeding them to the monomodal and multimodal networks. we evaluate the performance of each model on the restore validation dataset and use the best (maximum micro f1-score) model to evaluate the performance on the restore test dataset. to update the monomodal (vision) model parameters, we used adamw @xcite optimizer with the learning rate of 4e -5. for the monomodal (language) and multimodal approaches, we used the adamw optimization algorithm with a learning rate of 4e -5. we set the batch size 64 to train all the benchmark models. we train the proposed network with batch size 16 and adamw optimization algorithm (with the learning rate of 2e -5) for 10 epochs. the dimension (k) of feature map obtained from resnet-152 is 2048. for liwc and hsv experiments, we set the size of the hidden unit as 20. we performed all the experiments on a single nvidia tesla v100x gpu having 32gb memory. we observed the average runtime to train our framework is 11.55 minutes per epoch. ## conclusion this work presents the first study towards identifying fine-grained depression symptoms from memes. we created a high-quality dataset -restore, consisting of 9, 837 depressive memes annotated with phq-9 symptom categories and benchmark the dataset on 20 monomodal and multimodal models. further, we introduce a novel method to incorpo-rate various perspective in the meme that obtained best f1-score over other approaches. finally, our thorough human analysis of the model predictions indicates the model's limitation while dealing with memes, which will be considered in the future.