Your Model Is Not Predicting Depression Well And That Is Why: A Case Study of PRIMATE Dataset (2024)

Abstract: This paper addresses the quality of annotations in mental health datasets used for NLP-based depression level estimation from social media texts. While previous research relies on social media-based datasets annotated with binary categories, i.e. depressed or non-depressed, recent datasets such as D2S and PRIMATE aim for nuanced annotations using PHQ-9 symptoms. However, most of these datasets rely on crowd workers without the domain knowledge for annotation. Focusing on the PRIMATE dataset, our study reveals concerns regarding annotation validity, particularly for the lack of interest or pleasure symptom. Through reannotation by a mental health professional, we introduce finer labels and textual spans as evidence, identifying a notable number of false positives. Our refined annotations, to be released under a Data Use Agreement, offer a higher-quality test set for anhedonia detection. This study underscores the necessity of addressing annotation quality issues in mental health datasets, advocating for improved methodologies to enhance NLP model reliability in mental health assessments.

## introduction applying various nlp techniques to automatically estimate the depression level from social media texts has been a widely researched topic in the field of nlp applied for mental health. most of these datasets consist of online posts gathered from popular social media platforms, such as twitter or reddit. these posts are usually annotated by crowd workers who had only a brief training with a mental health professional (mhp) or sometimes only had access to the annotation instructions. while there exist multiple depression-related datasets based on social media texts, most of them only present binary annotation, i.e. whether the user is depressed or not. the most common sources of data are @xcite @xcite and x (former twitter) @xcite . most of the studies use automatic methods of annotations, such as regular expression matching of self-reported terms, like "i have been diagnosed with depression". some of them perform manual verification and annotation either via layman crowd workers @xcite or by the authors themselves @xcite . recently, the interest in more fine-grained depression annotation has emerged. in particular, the two recent datasets d2s @xcite and primate @xcite , identify depressed social media posts from x and reddit, respectively and annotate them with phq-9 symptoms @xcite . both datasets have been annotated with the help of crowd workers and later verified by mhps. however, the verification process was different. for d2s, conflicting annotations were resolved with the majority voting, and the psychiatrist resolved the ties. after that, 100 random samples were selected for quality control and verified by a psychiatrist. additionally, zirikly and dredze (2022) annotated a random sample of d2s with the explanations for each symptom with the help of two mhps foot_0 , increasing the validity of the data. in the case of primate, no information is given on the quality control procedure. this raises concerns about the validity of the annotations; thus, we selected primate for our case study. in this study, on the example of the primate dataset, we show that the validity of the annotations for the mental health data is a concern when performed by layman crowd workers. our mhp reannotated 170 posts from the primate dataset for the lack of interest or pleasure (anhedonia) symp-tom. the mhp is the second author of the paper, who is also a practising clinical psychology intern. our annotations include more fine-grained labels ("mentioned" vs "answerable", as well as an additional "writer's symptom" label) as well as spans of texts that serve as evidence of the labels. we observe a high number of false positives in the pri-mate labels, which can be related to the high difficulty of conceptualizing anhedonia @xcite . the annotations are to be released under a data use agreement (dua), and we believe that it can serve as a higher-quality test set for anhedonia detection. the annotation was performed by five crowd workers with additional quality control by an mhp. the information about the annotation procedure or crowd worker training, as well as how exactly the mhps were involved in the quality control, are not provided in the paper. the only metric on the annotation process is an annotator agreement using fleiss' kappa, which is reported to be 67% for initial annotation and 85% after involvement of the mhps. in total, the dataset consists of 2003 posts. figure 1 shows the label co-occurrence matrix of the training set. two symptoms, dep and lse, co-occur the most with all the other symptoms, which can be explained by their general prevalence in the dataset. the connection between the lack of interest or pleasure (loi) and lack of energy (ene) is also seen in the dataset, which reflects high comorbidity of these symptoms @xcite . ## experimental setup in our experiments, we aimed to test how well current pre-trained language models can model the depression symptom detection problem using the primate dataset. we first chose dis-tilbert @xcite as a baseline and bert-base @xcite , roberta-base, roberta-large @xcite , deberta-base, and deberta-large (he et al., 2020) as higherperforming models. in particular, deberta has shown constant improvements in various nlp tasks and replaced bert and roberta as the state-ofthe-art model for many of them foot_2 . for fine-tuning, we used the implementation from transformers library @xcite ## results and discussion roberta models perform better for sle and sui prediction. nevertheless, distilbert sets a strong baseline and performs on par with larger models overall. finally, loi shows a decrease in performance for all the models compared to the distil-bert. we investigate the diminished performance of the loi symptom since it is a core symptom of a major depressive disorder (association, 2013) and shows unstable results for our models. furthermore, loi is one of the symptoms of schizophrenia (association, 2013) and is associated with both anxiety and depression @xcite . thus, we selected a subset of 170 posts from the validation set based on the distilbert predictions: if at least one symptom was predicted incorrectly, the post was selected. next, an mhp read all the posts in the subset and labelled them for the presence of loss of interest or pleasure (loi). the mhp as-signed three labels to each post: a) "mentioned" if the symptom is talked about in the text, but it is not possible to infer its duration or intensity; b) "answerable" if there is clear evidence of anhedonia; c) "writer's symptoms" which shows whether the author of the post discusses themselves or a third person. additionally, the mhp selected the part of the text that supports the positive label. figure 2 shows examples for the reannotated posts 4 . the first example is labelled as "mentioned" since it contains evidence of a symptom but does not contain information about the loss of interest. the second example is labelled as "answerable" because it is possible to infer that the person used to have interest in what they were doing before but lost it at some point in time. finally, the last example shows the post without signs of loi that describes the condition of another person. considering the "writer's symptom" label, in 18 out of 170 selected posts, the author describes a symptom of another person rather than themselves. this raises the question of how these posts should be annotated and whether they should be included in the dataset at all. we suspect that the language of describing one's condition or feelings in the first person is different from the third person. we leave this question for future debate and assign "mentioned" and "answerable" labels to the posts describing a third person in the same manner as to the personal posts. our findings are consistent with the original results presented by @xcite . similar to our experiment, they also trained a classifier based on the bert-base model and reported low mcc for loi. however, we provided the evi4 all example posts are paraphrased for privacy. while we agree that our reannotated test set is also, to some extent, susceptible to errors, we believe that it serves as a more reliable benchmark for the anhedonia symptom. a more fine-grained, evidence-based labelling scheme reduces the risk of mislabelling and is more transparent for further verification. finally, it lays the foundation for future collaboration to produce a higher-quality redditbased dataset for depression symptom estimation. ## conclusion in conclusion, this study highlights the importance of evaluating and enhancing the quality of annotations in mental health datasets, particularly within the context of automated depression level estimation from social media texts. while recent datasets such as primate introduce commendable efforts toward nuanced annotations using phq-9 symptoms, our examination of the primate dataset reveals concerns about annotation validity, specifically regarding the lack of interest or pleasure symptom. through careful reannotation by a mental health professional, we discerned a considerable number of false positives among the original labels indicative of challenges in conceptualizing anhedonia. the findings presented here advocate for a more rigorous and standardized approach to mental health dataset annotation, emphasizing the need for greater involvement of domain experts in the annotation process. the release of our refined annotations under a data use agreement (dua) contributes a valuable resource for future research, offering a higher quality test set for anhedonia detection. moving forward, a concerted effort toward refining annotation methodologies and promoting collaboration between domain experts and nlp practitioners is imperative to foster advancements in this crucial intersection of technology and mental health research.