Then and Now: Quantifying the Longitudinal Validity of Self-Disclosed Depression Diagnoses (2022)

Abstract: Self-disclosed mental health diagnoses, which serve as ground truth annotations of mental health status in the absence of clinical measures, underpin the conclusions behind most computational studies of mental health language from the last decade. However, psychiatric conditions are dynamic; a prior depression diagnosis may no longer be indicative of an individual’s mental health, either due to treatment or other mitigating factors. We ask: to what extent are self-disclosures of mental health diagnoses actually relevant over time? We analyze recent activity from individuals who disclosed a depression diagnosis on social media over five years ago and, in turn, acquire a new understanding of how presentations of mental health status on social media manifest longitudinally. We also provide expanded evidence for the presence of personality-related biases in datasets curated using self-disclosed diagnoses. Our findings motivate three practical recommendations for improving mental health datasets curated using self-disclosed diagnoses:1) Annotate diagnosis dates and psychiatric comorbidities2) Sample control groups using propensity score matching3) Identify and remove spurious correlations introduced by selection bias

## introduction the ability to provide equitable access to psychiatric healthcare has become more difficult than ever, inhibited by an entanglement of lingering public policy effects @xcite , heightened levels of physician burnout @xcite , and infrastructural challenges arising from global crisis @xcite . meanwhile, social media platforms have become the predominant means of communication for much of the population, providing the opportunity to share personal experiences and seek support from others @xcite . noting these parallel timelines, computational scientists have devoted substantial effort to engineering statistical models capable of translating social media data into reliable insights regarding mental health. core objectives of this work include optimizing psychiatric treatment, identifying early stages of mental illness, and measuring the effect of public policy on a population's well-being @xcite . the most significant advances in computational mental health research have not come from improved modeling architectures @xcite , but from methods for curating largescale datasets which contain robust and clinicallyrelevant ground truth annotations of mental health status @xcite . use of regular expressions to identify genuine self-disclosures of a psychiatric diagnosis remains one of the most widely adopted annotation mechanisms by the research community @xcite , offering a relatively reliable proxy in place of clinical measures which are not only costly to collect, but also often unable to be shared beyond a single institution due to patient privacy policies @xcite . datasets leveraging self-disclosed diagnoses as annotations of mental health status have yielded a variety of insights that align with clinical knowledge and psychological theory @xcite . however, a growing body of work has raised questions about whether such datasets provide sufficient information to train statistical models that generalize to new populations @xcite . despite the prevalence of datasets dependent on self-disclosure, no analyses have considered how associating a single self-disclosed diagnosis label with data from a variable-length period of time may inhibit the learning of robust statistical relationships. if a user tweets a depression diagnosis in 2015, is their data from 2018 still representative of the condition? presentation of several mental health conditions change dynamically and (sometimes) precipitously over time @xcite ). yet, it remains common in the computational research community to treat mental health conditions as a static attribute with equal relevance at multiple time points @xcite . in reality, it is likely that only a small fraction of an individual's social media activity is appropriate for training optimal classifiers. moreover, that a mental health status label may be appropriate for only a subset of time suggests that evaluations of longitudinal model generalization as they are traditionally structured in the community may be insufficient @xcite . we ask: to what extent do mental health diagnosis self-disclosures remain valid over time? we focus specifically on extended durations (i.e., multiple years), a setting which has particular relevance to those who wish to estimate generalization strength of their statistical classifiers for use in longitudinal monitoring applications, as well as those interested in updating existing models with new data to mitigate the effects covariate shift @xcite . in reviewing recent online activity from individuals in the 2015 clpsych shared task dataset who disclosed a depression diagnosis on twitter over five years ago @xcite , we not only acquire a new understanding of how presentations of mental health status on social media present over time, but also find new evidence to support prior claims regarding the presence of personality-related confounds in datasets curated using self-disclosures @xcite . our analysis provides critical guidance to practitioners as they curate mental health social media datasets, while also elucidating factors which inhibit robustness in a dataset that remains one of the most widely adopted by the research community. ## background the majority of mental health research based on social media leverages the same experimental design-assume individuals have a fixed mental health status and attempt to infer this latent attribute using historical online activity traces (e.g., posts, follower network dynamics) @xcite . this training setting is convenient given the inherent complexities of acquiring temporally-granular psychiatric measures at scale @xcite . however, the setting implicitly relies on assumptions that are not supported by clinical knowledge regarding psychiatric dynamics @xcite . some work has been done to incorporate time-based priors into mental health models, which allow practitioners to train statistical classifiers using a static label while also explicitly accounting for longitudinal variation in label relevance @xcite . others have eschewed the use of a static label altogether and instead curated datasets that contain multiple points of ground truth mental health status, albeit still with some element of historical data aggregation @xcite . temporally-aware classifiers have achieved better performance benchmarks than their static counterparts in some cases @xcite , though these evaluations remain limited by the dearth of data with mental health status annotations at multiple time points. meanwhile, datasets which do support dynamic evaluation are curated almost exclusively using protected clinical measures @xcite , cost-intensive interviews @xcite , or non-trivial shifts in non-languagebased online behavior @xcite . computational studies that have focused on selfdisclosed diagnoses have not comprehensively reviewed how individual activity evolves over long periods of time @xcite . our study thus fulfills an important void in the research space by providing a new understanding of long term mental health dynamics in social media, and more particularly, within convenience samples curated using self-disclosed diagnoses. ## data we support our study using a newly updated version of the 2015 clpsych shared task dataset @xcite . the original twitter dataset was constructed in a two-stage process, with regular-expressions first being used to identify candidate self-disclosures of a depression diagnosis and experts manually verifying the authenticity of the match thereafter. individuals in the control group were sampled randomly from the 1% public twitter stream such that the joint distribution of inferred age and gender attributes @xcite was in alignment with the depression group. up to 3,000 tweets were acquired for each individual in the resulting sample using twitter's public api. the dataset has not only become one of the most widely adopted social media datasets for mental health @xcite , but also inspired the annotation procedures for numerous successors across various platforms and languages @xcite . in line with guidance from @xcite , individual identifiers in the official version of the clpsych dataset have been anonymized, with linkages between anonymized and de-anonymized identifiers erased in entirety. however, the original de-anonymized identifiers remain available under explicit permission from @xcite , who provided this information to reverse engineer the original anonymization mapping. to do so, we first query up to 3,200 of the most recent tweets from each de-anonymized user identifier using twitter's public api and further isolate all relevant tweets found in our institution's cache of twitter's 1% data stream. we identify candidate pairs of anonymized and de-anonymized accounts based on overlap of raw timestamps within the original dataset's collection window. normalized text (i.e., punctuation removal, case standardization) from candidate pairs is compared using exact matching to verify final linkages. statistics for the original dataset and its updated counterpart are provided in preprocessing. twitter's language tags and automatic language identification @xcite are used to isolate english text. retweets are excluded to most acutely highlight personal experiences with depression over time. unless specified otherwise, keyword-based tweet filtering is applied to preemptively mitigate sampling-induced biases which can artificially inflate estimates of predictive performance. some of these biases have been recognized and addressed by the research community (e.g., filtering tweets which include diagnosis disclosures and/or mental health related keywords/hashtags) @xcite , while others have been traditionally overlooked. a preliminary qualitative analysis of influential n-grams and their source tweets reveals a previously unrecognized surplus of "fan accounts" (e.g., supporters of harry styles and demi lovato) and tweets containing account statistics (e.g., new followers) within the depression cohort. meanwhile, daily horoscope tweets were identified with an anomalous frequency within the control group. the latter two sources of noise do not have a clear clinical explanation, while the former (i.e., fan accounts) arises in the context of discussion regarding the mental health of young celebrities. although some of these motifs represent genuine behavioral correlates of depression, their importance in prediction tends to be inflated due to context of the original collection time period. ## inference under latent dynamics enabling reliable use of statistical models to evaluate change in mental health status remains a core objective for computational researchers @xcite . our success in this task domain critically depends on access to ground truth at multiple time points, not only for evaluating generalization error @xcite , but also for mitigating the effects of covariate shift @xcite . as discussed above, it is often trivial to update activity traces for individuals with a prior mental health diagnosis disclosure. nonetheless, clinical knowledge suggests original disclosure-based labels may not be relevant over the course of time, either due to a condition's episodic presentations @xcite or the effects of psychiatric treatment @xcite . we ask whether the clpsych shared task dataset supports this theory. methods. a natural framework for answering this inquiry emerges from computational research regarding label noise @xcite . under such a perspective, we can view changes in mental health status as a stochastic process which blindly alters the correctness of class labels over time. the implications of this mechanism allow us to reason about predictive performance of a statistical classifier within and outside of the time period in which it is trained. differences in within-timeperiod performance for two different time periods may be caused by two factors-different levels of label noise and/or different signal-to-noise ratios. meanwhile, degradation in performance when transferring a classifier from one time period to another may be caused by three possible factorslabel noise in the source time period, label noise in the target time period, or distributional shift between the time periods. although isolated differences in predictive performance in a longitudinal setting do not implicate a single causal factor, multiple comparisons taken together may allow us to reason about underlying changes in the data. this logic guides our search for evidence in support of the hypothesis that mental health annotations cannot be treated as fixed attributes. we consider a standard longitudinal domain transfer setup @xcite , chunking the clpsych dataset into three discrete three-year periods 1 @xcite and evaluating within-and between-time-period predictive performance for all available pairs. we use monte carlo cross validation @xcite to obtain estimates of predictive generalization, chosen over alternative protocols that would be unreliable given the limited sample size of the updated clpsych dataset @xcite . each iteration of the cross validation procedure (1,000 total) begins by randomly splitting individuals into a 60/40 train/test split, with control and depression groups demographically aligned 2 using propensity scores @xcite . to control for differences in data availability between time periods, we not only constrain the sampling process such that splits have an equal class balance, but also that individual-level representations are constructed using an equal document history size (250 randomly-sampled posts from each time period). a single binary logistic regression classifier provided with document-term tf-idf representations @xcite is fit for each 1 time periods were chosen to maximize the number of discrete windows while ensuring enough posts were available to construct informative individual-level representations. ## interpreting model performance we attempt to better understand the variation in predictive performance estimated above by comparing language within the updated dataset to the original clpsych sample. in particular, we adopt a mixed methods approach that allows us to estimate changes in the proportion of depression labels which remain relevant in the updated dataset, and to qualitatively summarize drivers of model decision-making across time periods. we support our analysis by manually coding content-related motifs within a large sample of document histories in the updated dataset, focusing primarily on criteria for diagnosing depression as defined within the dsm-5 (apa, 2013). we draw inspiration from the growing literature on "train-set debugging" (koh and liang, 2017; han et al., 2020), which leverages instance attribution and other diagnostics to succinctly interpret the relationship between training data, learned model parameters, and downstream predictions. methods. an annotator is presented with up to 30 anonymized tweets made by a single individual during one of the time periods and asked to indicate whether the individual exhibits evidence of depression. the annotator must mark one of four options -uncertain, no evidence, some evidence (moderate confidence), strong evidence (high confidence). explicit disclosures of a depression diagnosis and references to living with depression are automatically assigned to the strong evidence category. otherwise, the annotator is instructed to indicate their confidence based on the nine dsm-5 criteria for diagnosing depression (apa, 2013) and their prior knowledge regarding the presentation of mental health conditions within social media. if at least some evidence of a depression diagnosis is indicated, the annotator is asked to identify whether the depression appears to be in remission (e.g., discussion of overcoming depression). they are also asked to indicate which dsm-5 criteria and/or prior knowledge was used to inform their decision, along with any other notable thematic content. our goal of this analysis is not to make diagnostic claims regarding the mental health status of individuals in our dataset, but rather to broadly understand what the statistical classifiers are learning. accordingly, tweets presented to the annotator are those which had the largest positive effect on the classifier's estimated probability of depression, as measured by their influence on user-level predictions within a given time period τ . formally, we define the influence of a tweet i(x) amongst a set of tweets @xmath0: note that we control for distributional shift over time by estimating influence using a model trained during the time period in which a tweet was posted. data. a total of 300 individuals (574 total instances) were selected randomly for annotation. one author, a doctoral student in computer science with multiple years experience working with the clpsych dataset, was responsible for all coding. they consulted one additional co-author, an expert in computational modeling of social media and mental health, to develop a common mental model for identifying dsm-5 criteria and other common linguistic motifs in the text. during a pilot round of coding, 16 thematic patterns were identified within the annotated instances to complement the original dsm-5 criteria. exemplary tweets (paraphrased non-trivially to preserve anonymity @xcite ) for each of the dsm-5 criteria and alternative thematic categories are provided in appendix a.3. a breakdown of annotation results is presented in the reliability. two non-authors with a background in computational psychology independently annotated a subsample of the coded instances to assess the primary coder's reliability. agreement regarding whether an individual exhibits evidence of depression was fair to moderate; we observe krippendorff's α measures of 0.438 and 0.499 for the four-class (uncertain, no evidence, some evidence, strong evidence) and three-class (uncertain, no evidence, some or strong evidence) scenarios, respectively @xcite . agreement regarding remission status varied significantly between pairs of annotators and was generally weaker than agreement regarding evidence of depression (α = 0.356). we include an analysis of the disagreements in appendix a.2 to better contextualize observations from the primary coder's annotations. succinctly, we identify two reasons for the variation: 1) each annotator's propensity to select the "uncertain" category, and 2) each annotator's sensitivity to displays of emotion as an indicator of depression. ## discussion demand for computational methods to quantify mental health dynamics within social media data is at an all time high @xcite . however, the potential impact of these methods remains bounded by the robustness of datasets used for their development. spanning nearly a decade of online activity, our study uniquely identifies evidence of these limitations as they currently manifest in non-clinically derived mental health social media datasets. this evidence leads us to offer three recommendations for enhancing data curation and model evaluation. ## ethical considerations ethical challenges emerging from use of public social media data to analyze an individual's mental health have been examined extensively by members of both computational and clinical/public health communities @xcite . privacy-related concerns are the most poignant for our study, which relies both on de-anonymizing records from a vulnerable population and manually reviewing/analyzing individual posts. indeed, many individuals who publicly discuss their mental health or disclose a psychiatric condition within social media admit that they worry about harmful repercussions of sharing such sensitive information with the public @xcite . primary fears include risking occupational stability, damaging interpersonal relationships, and being subjected to hostile communications. whether potential positive outcomes (e.g., development of systems for recommending mental health care, fiduciary aid to address population-level crises) offset these threats remains largely dependent on an individual's personal life experience. for example, psychiatric patients have expressed stronger approval toward analysis of their social media than members of the general public @xcite . the same holds true amongst younger individuals (naslund and aschbrenner, 2019). recognizing these viewpoints, we are careful to mitigate privacy-related risks to the greatest extent possible given our primary research aim. for example, account identifiers distributed within the 2015 clpsych shared task dataset are de-anonymized only temporarily to link updated records with existing post histories. we also redact account handles and urls from the text analyzed during our manual coding procedure ( §5). in line with protocols enumerated by @xcite , all data is stored on a remote server and secured using oslevel group permissions. we perform our analysis under the external guidance of clinical psychologists and psychiatrists. our study is also reviewed by our institutional review board (irb), obtaining exempt status under 45 cfr §46.104. critically, our intention is not to develop a publicfacing system for algorithmic analysis of mental health. rather, our goal is to evaluate the validity of an existing and widely-adopted data curation practice @xcite . failure to comprehensively understand biases that arise under this methodology can have severe detrimental effects in downstream systems. in the case of estimating populationlevel health trends, for instance, we have already seen machine learning classifiers produce outcomes that are inconsistent across computational studies @xcite @xcite and in conflict with traditional measurement techniques @xcite . continuing to pursue this line of research without questioning the validity of its underlying data has the potential to irreparably damage the public's trust in this domain, and worse, enable ill-informed decision making in highly-sensitive circumstances. we observe fair to moderate agreement for the evidence-of-depression task: α = 0.4376 and α = 0.4988 for the four-class and three-class versions, respectively. meanwhile, agreement on remission status is poor, reflected by a krippendorff's α of 0.3561. in isolation, these agreement measures would suggest the results of our analysis should be accepted tentatively at best @xcite . however, we argue these statistics are perhaps a bit conservative and skewed by the small sample size of annotations generated by b 2 . a review of the underlying distributions provides us an opportunity to understand axes of disagreement and, in turn, contextualize the results presented in §5. as shown in figure 1 , annotator b 2 exhibits a higher propensity to use the "uncertain" label in the evidence-of-depression tasks compared to annotators a 1 and b 1 . at the same time, while annotator b 2 is more inclined to indicate they are uncertain about an example than annotator a 1 , we note that annotator b 1 appears to have a higher baseline threshold of what constitutes evidence of depression than annotator a 1 . the latter is demonstrated by the fact that nearly all examples marked in the affirmative by b 1 were also marked as such by a 1 , but a large number of examples marked in the affirmative by a 1 were marked as not containing evidence of depression by b 1 . with respect to the remission status task (bottom subplot of figure 1 ), we note that annotator b 1 is more likely to mark an example as uncertain and more likely to mark an example as being inremission than annotators a 1 and b 2 . broadly, this distribution highlights the difficulty of distinguishing active cases of clinical depression from prior experiences and lingering effects. it also serves as support for our recommendation in §6 that researchers should attempt to include the time a diagnosis was received by an individual when curating new datasets. we acquire additional context for our results by examining the distribution of annotations as a function of the original clpsych labels. examining the results visualized in figure 2 , we first note that annotator a 1 classifies instances most accurately (under the assumption that ground truth is fixed over time). we believe this outcome to be a result of exposure bias; the annotation task was conducted after the completion of several modeling experiments, through which annotator a 1 was uniquely provided an opportunity to learn more about the presentation of depression by individuals in the 2015 clpsych shared task dataset. we also note the distribution of "uncertain" decisions from annotator b 2 concentrating within the original depression group. this seems to suggest annotator b 2 adopted a conservative coding approach when presented with instances that contained smaller degrees of evidence, whereas annotators a 1 and b 1 required a lower threshold of evidence to make a decision. to conclude our reliability analysis, we examine agreement regarding the manner in which each annotator made their decision (i.e., evidence identification). we find that annotators a 1 and b 1 generally identify diagnosis disclosures within the same instances. annotator b 2 often abstained from making a decision when presented with a disclosure due to uncertainty regarding the subject of the diagnosis. annotator a 1 also indicated the presence of a depressed and/or irritable mood at a significantly higher rate than the other annotators, seemingly more sensitive to extreme negative emotions than the other annotators. discussion. considering the difficulty of the annotation task, it is perhaps not surprising to have observed less than perfect annotator agreement. machine learning classifiers often require hundreds of posts to make an accurate estimate of an individual's mental health status, while our annotators were only provided at maximum of 30 posts and encouraged to rely on varying levels of prior knowledge regarding the presentation of depression in social media. critically, we emphasize that the goal of the analysis presented in §5 is not to curate ground truth labels of mental health status or act as clinical experts, but rather to understand biases that may exist in a depression dataset generated using self-disclosed diagnoses. the analysis of inter-rater reliability presented above provides an opportunity to further ground the results discussed in §5 and highlight areas that may benefit from future research.