Qualitative Analysis of Depression Models by Demographics (2021)

Abstract: Models for identifying depression using social media text exhibit biases towards different gender and racial/ethnic groups. Factors like representation and balance of groups within the dataset are contributory factors, but difference in content and social media use may further explain these biases. We present an analysis of the content of social media posts from different demographic groups. Our analysis shows that there are content differences between depression and control subgroups across demographic groups, and that temporal topics and demographic-specific topics are correlated with downstream depression model error. We discuss the implications of our work on creating future datasets, as well as designing and training models for mental health.

## introduction models of mental health trained on social media data exhibit biases in downstream performance on different gender and racial/ethnic demographic groups @xcite . an important factor is that minority groups (people of color in general) are underrepresented in datasets and thus models under perform compared to majority groups. while size and balance of datasets contribute to the gap in performance, there may be differences in the manner in which depressive behavior is exhibited across demographic groups, creating problems in generalization. difference in depression prevalence across demographics have long been known @xcite , although there is no clear explanation for why this is the case @xcite . on social media, demographic-based mental health analyses have used matched control samples @xcite , which allow for comparison of behaviors across groups @xcite . these types of analyses have focused on downstream performance of trained models @xcite and how they show differences in depression rates, but there have been no qualitative studies investigating these demographic differences @xcite . others have used qualitative studies to analyze behaviors and performance of machine learning models in general @xcite . previous work has analyzed representative sentences (ettinger, 2020), hashtags @xcite , performed a thematic analysis by using the linguistic inquiry and word count dictionary @xcite or trained topic models @xcite @xcite . we propose a qualitative language analysis to reveal what differences occur, and how these differences can contribute to downstream performance. what language trends characterize depression and how do these vary across demographic groups? we use an analysis method similar to @xcite but instead of training an latent dirichelt allocation (lda) @xcite topic model and performing point-wise mutual inference to obtain topics related to demographics, we train a partially-labeled lda model @xcite which allows us to assign labels to demographic groups as well as depression and control groups to obtain label-specific topics to our user groups. we base our analysis on datasets from previous work using twitter. we train simple text-based models based on previous work on these datasets @xcite . we use a labeled topic model to characterize what content indicates depression and how this content varies by demographic group. our analysis shows variations in content between depression and control subgroups across demographic groups, however, most of these differences are due to non-clinical phenomena e.g. vi-ral content trends such as tv shows awards. further, model error analysis corroborates that temporal trends and nongeneralizable topics of demographic groups are correlated with downstream model error. our qualitative analysis approach can be utilized to analyze language differences across demographics on other datasets and mental health tasks. we discuss the implications of our work on creating new datasets, as well as designing and training language models for mental health. ## ethical considerations given the sensitive nature of mental health topics and demographics of individuals, additional precautions (based on depression diagnoses @xcite ; gender identity @xcite ; race/ethnicity identity (wood-doughty et al., 2020)) were taken during this study. data sourced from external research groups was retrieved according to each datasets respective data use policy. for gender labels, due to current limitations on datasets and methods, we consider the folk perception of gender, as described in @xcite , and for race/ethnicity labels we use the mutually-exclusive non-hispanic white, non-hispanic black,non-hispanic asian and hispanic/latinx, following wood-doughty et al. (2020). we acknowledge that both our gender and racial/ethnic categories do not fully capture many individuals' gender and/or race/ethnicity. additionally, we acknowledge the limitations of the demographic inference methods employed to obtain the demographic labels that have been raised in multiple previous studies @xcite . while we carefully consider these issues, we believe the urgency of understanding mental health models @xcite warrants our work and hope that our results provide sufficient evidence to justify further study in this area. this research was deemed exempt from review by our institutional review board (irb) under 45 cfr g 46.104. ## data we use two datasets for depression identification on twitter from previous studies: the clpsych 2015 shared task @xcite , and the multi-disorder multitask learning for mental health dataset @xcite . clpsych. the dataset contains publicly available tweets of individuals where the diagnosed multitask. this dataset combines subsets of several datasets @xcite . all methods used the same collection process: selfreport through regular expression matching, and control individuals by matching inferred age and gender with the same tool. additionally, the complete public history of tweets is collected for each individual as opposed to the latest 3000 tweets on clpsych resulting in a bigger dataset. we select the depression users (1400) and their matched control users resulting in 2800 individuals. while both dataset collection methods are nearly identical, the time period in which the tweets were collected, and the number of tweets and individuals are different for each dataset, likely leading to different types of depression indicators. note that while there is an overlap between multitask and clpsych of 110 individuals, it is a small percentage of both datasets (∼ 4% and ∼ 10% respectively). ## analysis the topic model identified label-specific topics for depression and control. appendix c shows the topics for both datasets as well as the top 10 most probable words per topic. some depression topics are reasonable e.g. mental-health (in both datasets) and social media stats (may be related to internet statistics and popularity). similarly, control topics like sports and beauty are active, positive and self-caring topics that are reasonable for being representative of our control group. however, some topics in both depression and control groups are not clearly tied to the groups e.g. for depression group, topics like one direction and 5 seconds of summer. these might be topics introduced by temporal phenomena impeding model generalization @xcite , rather than representative topics for those labels. ## conclusion we performed a qualitative analysis to find content differences related to mental health across demographic groups. we showed that there are content differences between depression and control subgroups, while most of these differences are due to non-clinical phenomena e.g. temporal topics. additionally, we find that dataset size might be a factor in these content differences. furthermore, model error analysis corroborates that temporal topics and demographic-specific topics are correlated with downstream model error. our findings support prior work on the importance of methods that seek to generalize temporal topics @xcite . we also find supporting evidence of the importance of dataset size as well as dataset balance in order to generalize to minority groups @xcite . though in our analysis we only consider one mental health disorder (depression), our methodology was able to generalize across two datasets. this suggests that it is a valid method for qualitative analysis on finding content differences in other mental health datasets. additionally, while we were limited in our demographic labels by current demographic models and dataset sizes, we showed that our approach is valid across two demographic axes and could be expanded to include other demographic axes (such as age and economic status), and include genders and racial/ethnic groups outside of the ones considered in this work. we hope our work warrants further studies of mental health language differences across more diverse demographic groups yielding more inclusive datasets and research.