On the State of Social Media Data for Mental Health Research (2021)

Abstract: Data-driven methods for mental health treatment and surveillance have become a major focus in computational science research in the last decade. However, progress in the domain remains bounded by the availability of adequate data. Prior systematic reviews have not necessarily made it possible to measure the degree to which data-related challenges have affected research progress. In this paper, we offer an analysis specifically on the state of social media data that exists for conducting mental health research. We do so by introducing an open-source directory of mental health datasets, annotated using a standardized schema to facilitate meta-analysis.

## introduction the last decade has seen exponential growth in computational research devoted to modeling mental health phenomena using non-clinical data @xcite . studies analyzing data from the web, such as social media platforms and peerto-peer messaging services, have been particularly appealing to the research community due to their scale and deep entrenchment within contemporary culture @xcite @xcite . such studies have yielded novel insights into population-level mental health @xcite and shown promising avenues for the incorporation of data-driven analyses in the treatment of psychiatric disorders @xcite . these research achievements have come despite complexities specific to the mental health space often making it difficult to obtain a sufficient sample size of high-quality data. for instance, behavioral disorders are known to display variable clinical presentations amongst different populations, rendering annotations of ground truth inher-ently noisy @xcite . scalable methods for capturing an individual's mental health status, such as using regular expressions to identify self-reported diagnoses or grouping individuals based on activity patterns, have provided opportunities to construct datasets aware of this heterogeneity @xcite . however, they typically rely on oversimplifications that lack the same clinical validation and robustness as something like a mental health battery @xcite . ethical considerations further complicate data acquisition, with the sensitive nature of mental health data requiring tremendous care when constructing, analyzing, and sharing datasets @xcite . privacy-preserving measures, such as de-identifying individuals and requiring irb approval to access data, have made it possible to share some data across research groups. however, these mechanisms can be technically cumbersome to implement and are subject to strict governance policies when clinical information is involved due to hipaa (price and cohen, 2019). moreover, many privacy-preserving practices require that signal relevant to modeling mental health, such as an individual's demographics or their social network, are discarded @xcite . this missingness has the potential to limit algorithmic fairness, statistical generalizability, and experimental reproducibility @xcite . although mental health researchers may anecdotally recall difficulties acquiring quality data or reproducing prior art due to data sharing constraints, no study to our knowledge has explicitly quantified this challenge. indeed, prior reviews of computational research for mental health have noted several of the aforementioned challenges, but have predominantly discussed technical methods (e.g. model architectures, feature engineering) developed to surmount existing constraints @xcite . recent work from @xcite , completed concurrently with our own, was the first review to focus specifically on the shortcomings of data for mental health research. our study affirms the findings of chancellor and de choudhury (2020), using an expanded pool of literature that more acutely focuses on language found in social media data. to this end, we construct a new open-source directory of mental health datasets, annotated using a standardized schema that not only enables researchers to identify relevant datasets, but also to identify accessible datasets. we draw upon this resource to offer nuanced recommendations regarding future dataset curation efforts. ## data to generate evidence-based recommendations regarding mental health dataset curation, we require knowledge of the extant data landscape. unlike some computational fields which have a surplus of well-defined and uniformly-adopted benchmark datasets, mental health researchers have thus far relied on a decentralized medley of resources. this fact, spurred in part by the variable presentations of psychiatric conditions and in part by the sensitive nature of mental health data, thus requires us to compile a new database of literature. in this section, we detail our literature search, establish inclusion/exclusion criteria, and define a list of dataset attributes to analyze. ## analysis our literature search yielded 139 articles referencing 111 nominally-unique datasets. application of exclusion criteria left us with 102 datasets. a majority of the datasets were released after 2012, with an average of 12.75 per year, a minimum of 1 (2012), and a maximum of @xcite . the 2015 clpsych shared task @xcite , reddit self-reported depression diagnosis @xcite , and "language of mental health" @xcite 3, and 3 additional publications respectively. all datasets known to be available for distribution are available with annotations in the appendix, while remaining datasets are found our digital directory. platforms. we identified 20 unique electronic media platforms across the 102 datasets. twitter (47 datasets) and reddit (22 datasets) were the most widely studied platforms. youtube, facebook, and instagram were relatively underutilized for mental health research -each found less than ten times in our analysis -despite being the three mostwidely adopted social media platforms globally @xcite . we expect our focus on nlp to moderate the presence of youtube and instagram based datasets, though not entirely given both platforms offer expansive text fields (i.e. comments, tags) in addition to their primary content of video and images @xcite . it is more likely that use of these platforms (and facebook) for research is hindered by increasingly stringent privacy policies and ethical concerns @xcite . tasks. we identified 36 unique mental health related modeling tasks across the 102 datasets. while the majority of tasks were examined less than twice, a few tasks were considered quite frequently. depression (42 datasets), suicidal ideation (26 datasets), and eating disorders (11 datasets) were the most common psychiatric conditions examined. anxiety, ptsd, self-harm, bipolar disorder, and schizophrenia were also prominently featured conditions, each found within at least four unique datasets. a handful of studies sought to characterize finer-grained attributes associated with higher-level psychiatric conditions (e.g. symptoms of depression, stress events and stressor subjects) @xcite . the dearth of anxiety-specific datasets was somewhat surprising given the condition's prevalence and the abundance of pyschometric batteries for assessing anxiety @xcite . that said, generalized anxiety disorder (gad) only accounts for a small proportion of the overall prevalence of anxiety disorders @xcite and many other types of anxiety disorders (e.g. social anxiety, ptsd, ocd, etc.) were typically treated as independent conditions @xcite . annotation. we identified 24 unique annotation mechanisms. it was common for several annotation mechanisms to be used jointly to increase precision of the defined task classes and/or evaluate the reliability of distantly supervised labeling processes. for example, some form of regular expression matching was used to construct 43 of datasets, with 23 of these including manual annotations as well. community participation/affiliation (24 datasets), clinical surveys (22 datasets), and platform activity (3 datasets) were also common annotation mechanisms. the majority of datasets contained annotations made on the individual level (63 datasets), with the rest containing annotations made on the document level (40 datasets). 3 size. of the 63 datasets with individual-level annotations, 23 associated articles described the amount of documents and 62 noted the amount of individuals available. of the 40 datasets with document-level annotations, 37 associated articles noted the amount of documents and 12 noted the number of unique individuals. the distribution of dataset sizes was primarily right-skewed. one concerning trend that emerged across the datasets was the presence of a relatively low number of unique individuals. indeed, these small sample sizes may further inhibit model generalization from platforms that are already demographicallyskewed @xcite . the largest datasets, which present the strongest opportunity to mitigate the issues presented by poorly representative online populations, tend to leverage the noisiest annotation mechanisms. for example, datasets that define a mainstream online community as a control 3 one dataset was annotated at both a document and individual level group may expect to find approximately 1 in 20 of the labeled individuals are actually living with mental health conditions such as depression @xcite , while regular expressions may fail to distinguish between true and non-genuine disclosures of a mental health disorder up to 10% of the time @xcite . primary language. six primary languages were found amongst the 102 datasets -english (85 datasets), chinese (10 datasets), japanese (4 datasets), korean (2 datasets), spanish (1 dataset), and portuguese (1 dataset). this is not to say that some of the datasets do not include other languages, but rather that the predominant language found in the datasets occurs with this distribution. while an overwhelming focus on english data is a theme throughout the nlp community, it is a specific concern in this domain where culture often influences the presentation of mental health disorders @xcite . availability. we were able to identify the availability of only 48 of the 102 unique datasets in our literature search. of these 48 datasets, 13 were known not to be available for distribution, generally due to limitations defined in the original collection agreement or removal from the public record @xcite . the remaining 35 datasets were available via the following distribution mechanisms: 18 may be reproduced using an api and instructions provided within the associated article, 12 require a signed data usage agreement and/or irb approval, 3 are available without restriction, and 2 may be retrieved directly from the author(s) with permission. of the 22 datasets that used clinically-derived annotations (e.g. mental health battery, medical history), 7 were unavailable for distribution due to terms of the original data collection process and 1 was removed from the public record. the remaining 14 had unknown availability. ## discussion in this study, we introduced and analyzed a standardized directory of social media datasets used by computational scientists to model mental health phenomena. in doing so, we have provided a valuable resource poised to help researchers quickly identify new datasets that support novel research. moreover, we have provided evidence that affirms conclusions from @xcite and may further encourage researchers to rectify existing gaps in the data landscape. based on this evidence, we will now discuss potential areas of improvement within the field. unifying task definitions. in just 102 datasets, we identified 24 unique annotation mechanisms used to label over 35 types of mental health phenomena. this total represents a conservative estimate given that nominally equivalent annotation procedures often varied non-trivially between datasets (e.g. phq-9 vs. ces-d assessments, affiliations based on twitter followers vs. engagement with a subreddit) @xcite . minor discrepancies in task definition reflect the heterogeneity of how several mental health conditions manifest, but also introduce difficulty contextualizing results between different studies. moreover, many of these definitions may still fall short of capturing the nuances of mental health disorders @xcite . as researchers look to transition computational models into the clinical setting, it is imperative they have access to standardized benchmarks that inform interpretation of predictive results in a consistent manner @xcite .