Improving the Generalizability of Depression Detection by Leveraging Clinical Questionnaires (2022)

Abstract: Automated methods have been widely used to identify and analyze mental health conditions (e.g., depression) from various sources of information, including social media. Yet, deployment of such models in real-world healthcare applications faces challenges including poor out-of-domain generalization and lack of trust in black box models. In this work, we propose approaches for depression detection that are constrained to different degrees by the presence of symptoms described in PHQ9, a questionnaire used by clinicians in the depression screening process. In dataset-transfer experiments on three social media datasets, we find that grounding the model in PHQ9’s symptoms substantially improves its ability to generalize to out-of-distribution data compared to a standard BERT-based approach. Furthermore, this approach can still perform competitively on in-domain data. These results and our qualitative analyses suggest that grounding model predictions in clinically-relevant symptoms can improve generalizability while producing a model that is easier to inspect.

## introduction given the significance of mental health as a public health challenge @xcite , much work has investigated approaches for detecting mental health conditions using social media text @xcite @xcite . such approaches could be used by at-risk users and their clinicians to monitor behavioral changes (e.g., by monitoring changes in the presence of symptoms related to depression as treatment progresses. these approaches generally rely on datasets consisting of users with self-reported diagnoses (e.g., based on a statement like "i was just diagnosed with depression") for training and evaluation (e.g., @xcite . despite promising results on these tasks, related work argues that assessing depression and suicidal behavior is difficult in practical settings and even experienced clinicians frequently struggle to correctly interpret signals @xcite . furthermore, recent work has found that models trained on particular mental health datasets do not always generalize to others. @xcite ; @xcite find that systematic, spurious differences between diagnosed and control users can prevent trained models from generalizing to even other, similar social media data. similarly, outside the mental health domain, recent work reports that neural models often struggle to generalize to data outside their training distribution @xcite @xcite . in this work, we explore approaches for constraining the behavior of depression detection methods by the presence of symptoms known to be related to depression, like mood and sleep issues. to do so, we develop nine symptom detection models that correspond to questions present in phq9, a screening questionnaire that has been clinically validated and commonly used in practical setting @xcite . these questions ask how often the patient has experienced symptoms from nine symptom groups (e.g., how often have you had "little interest/pleasure in doing things?"). grounding depression detection in a trusted diagnostic tool produces several benefits. from the perspective of mental health professionals, the output of such model is inherently more reliable than a black-box model, because classification decisions are based on the presence of specific symptoms in specific posts that can be inspected in order to assess the quality of evidence for a diagnosis. further, we find this improves the model's ability to generalize, which may be due to limiting its ability to use spurious shortcuts. this strategy is complementary to strategies for reducing temporal and topical artifacts @xcite . our proposed approach consists of two sim-ple yet effective models: a questionnaire model that detects symptoms from phq9 and a depression detection model. we instantiate both with a range of methods that are progressively less constrained. at one end of the spectrum, the questionnaire model uses only manually-defined patterns and the depression model makes classification decisions by counting how many times these patterns appear in a user's posts. at the opposite end of the spectrum, there is no explicit questionnaire model and bert @xcite serves as an unconstrained depression detection model. in between, we relax the questionnaire model by training bert-based symptom classifiers using the manually-defined patterns, by considering symptom representations rather than counts, and by adding an extra trainable 'other' symptom. we find that our constrained models perform competitively compared to a standard unconstrained bert classifier when trained and evaluated on the same dataset, while additionally providing a model whose behavior can be understood in terms of relevant symptoms in specific posts. however, dataset-transfer evaluations demonstrate substantial degradation in bert's effectiveness. in this setting, our constrained models outperform the unconstrained bert and show improved generalizability, even across similar datasets. our contributions are: (1) comprehensive pattern sets for identifying the symptoms in phq9 and heuristics for using them to train weaklysupervised symptom classifiers, (2) a range of progressively less constrained methods for performing depression detection based on these symptoms, and (3) an extensive evaluation of depression detection methods. our implementation is available online foot_0 . ## related work natural language processing methods have been widely used for automatic mental health assessment. to support automated analyses of mental health related language, a variety of datasets have been proposed. @xcite al. (2014) rich bodies of work in this area focused on studying language use and linguistic styles in depressed users. liwc (tausczik and pennebaker, 2010) has been one of the most popular tools to characterize depression language @xcite . similarly to other nlp domains, the use of contextualized embeddings has improved the performance of classifiers @xcite . recent work shows that while such nlp models achieve promising results, they have poor generalization to new data platforms and user groups; for example, @xcite investigated various factors, including sample size, class imbalance, temporal misalignment (e.g., language dynamic, linguistic norms), deployment latency, and self-disclosure bias that may cause performance degradation when a model is transferred to a new dataset or domain. the issues can occur even when datasets appear to be similar, such as when redditbased datasets employ different rules for selecting diagnosed and control users. another problem is the black-box nature of model predictions which is a major hurdle in deploying ai models in clinical practice @xcite . in this work we aim at reducing this problem by proposing to ground depression assessment in a clinical questionnaire for measuring severity of depression. others have considered making predictions more explainable in the mental health domain. @xcite focused on leveraging a userlevel attention mechanism for detecting signs of anorexia in social media profiles. our method differs from theirs in that the explanations are the results of the analysis of the attention weights, while our approaches ground model predictions in a wellestablished clinical instrument. outside of our work, we are aware of two datasets that incorporate questionnaire information such as phq9 for identifying depression. the most recent erisk shared task @xcite relies on the beck depression inventory (bdi), a 21-item questionnaire that assesses level of depression based on the presence of feelings like sadness, pessimism, etc. models are built to estimate the user-level bdi score at given time frames. our approach differs in that we use phq9 and evaluate item scores at the post level, which grounds predictions in the presence of clinically-relevant symptoms. in erisk, a sum of bdi scores is the modeled outcome (corresponding to our baseline pattern-based (threshold) classifier). we use userlevel labels for evaluating depression status and evaluate how constraining on phq9 symptoms affects the user-level classification performance. delahunty et al. ( 2019 ) used a deep neural network to predict phq4 scores using clinical data that contains patients' phq4 scores @xcite . our work does not require access to phq labeled clinical data, which can be hard to obtain at scale. furthermore, delahunty's approach generalizes poorly to social media data. rinaldi et al. (2020) predict depression based on screening interviews that rely on phq9 categories. in their setting, phq9 is a channel to retrieve the depressed label, but is not used for explainability. yadav et al. (2020) propose a multitask learning framework that uses phq-9 and figurative language detection as auxiliary tasks. @xcite contemporaneously propose a micromodel architecture that they apply to mental health assessment tasks. our work shares several similarities with this approach, which uses micromodels that are similar to our symptom classifiers (questionnaire models). ## experiments we conduct experiments on three datasets; all consist of reddit social media data but follow different construction methodologies (e.g., identifying depressed users based on a self-report statement vs. based on starting a thread in a support subreddit). in addition to evaluating methods on each dataset, dataset-transfer evaluation allows us to evaluate how well methods generalize to similar datasets with different construction methodologies. ## case study in ## conclusion in this work, we propose a spectrum of methods for depression detection that are constrained by the presence of phq9 symptoms. in our experiments on the three datasets, we find these methods to perform well compared to strong baselines while generalizing better to similar datasets. this can be viewed as a proof-of-concept demonstrating that grounding depression predictions in phq9 can improve the generalizability of depression detection and the interpretability of the model. while this research focuses only on depression detection, the idea of constraining models to consider only relevant causes may be applied to a wider range of tasks, including detection of other mental health conditions with diagnostic questionnaires. for aspect-target sentiment classification. in proceedings of the 12th language resources and evaluation conference, pages 4933-4941. alex rinaldi, jean e fox tree, and snigdha chaturvedi. 2020. predicting depression in screening interviews from latent categorization of interview prompts. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 7-18. han-chin shing, suraj nair, ayah zirikly, meir friedenberg, hal daumé iii, and philip resnik. 2018. expert, crowdsourced, and machine assessment of suicide risk via online postings. in proceedings of the fifth workshop on computational linguistics and clinical psychology: from keyboard to clinic, pages 25-36, new orleans, la. association for computational linguistics. han-chin shing, philip resnik, and douglas w oard. 2020. a prioritization model for suicidality risk assessment. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 8124-8137. yla r tausczik and james w pennebaker. 2010. the psychological meaning of words: liwc and computerized text analysis methods. journal of language and social psychology, 29(1):24-54. jt wolohan, misato hiraga, atreyee mukherjee, z. sayyed, and matthew millard. 2018. detecting linguistic traces of depression in topic-restricted text: attending to self-stigmatized depression with nlp. shweta yadav, jainish chauhan, joy prakash sain, krishnaprasad thirunarayan, amit sheth, and jeremiah schumm. 2020. identifying depressive symptoms from tweets: figurative language enabled multitask learning framework. in proceedings of the 28th international conference on computational linguistics, pages 696-709, barcelona, spain (online). international committee on computational linguistics. andrew yates, arman cohan, and nazli goharian. 2017. depression and self-harm risk assessment in online forums. in proceedings of the 2017 conference on empirical methods in natural language processing, pages 2968-2978, copenhagen, denmark. association for computational linguistics. ayah zirikly, philip resnik, ozlem uzuner, and kristy hollingshead. 2019. clpsych 2019 shared task: predicting the degree of suicide risk in reddit posts. in proceedings of the sixth workshop on computational linguistics and clinical psychology, pages 24-33.