Explaining Models of Mental Health via Clinically Grounded Auxiliary Tasks (2022)

Abstract: Models of mental health based on natural language processing can uncover latent signals of mental health from language. Models that indicate whether an individual is depressed, or has other mental health conditions, can aid in diagnosis and treatment. A critical aspect of integration of these models into the clinical setting relies on explaining their behavior to domain experts. In the case of mental health diagnosis, clinicians already rely on an assessment framework to make these decisions; that framework can help a model generate meaningful explanations. In this work we propose to use PHQ-9 categories as an auxiliary task to explaining a social media based model of depression. We develop a multi-task learning framework that predicts both depression and PHQ-9 categories as auxiliary tasks. We compare the quality of explanations generated based on the depression task only, versus those that use the predicted PHQ-9 categories. We find that by relying on clinically meaningful auxiliary tasks, we produce more meaningful explanations.

## introduction mental illness has a huge impact on the health and well-being of the united states and world populations. in the us, 25% of the population suffered at some point from mental illness foot_0 . the urgency to address the mental health crisis became even more critical with the covid-19 pandemic and its negative impact on mental health, burdening kids and seniors especially @xcite . depression is among the most prevalent mental disorders. in the united states alone, 21 million adults had at least one major depressive episode foot_1 . computational linguistics and natural language processing (nlp) research on mental health has received increased attention in the last decade, with work on suicide risk assessment @xcite @xcite , anxiety prediction and classification @xcite , and @xcite , among many other tasks. although clinical data was used for some models @xcite , prior work also utilized other sources of data, such as social media to overcome challenges in data access and to better understand what influences mental health on a daily basis. the majority of the nlp research on depression classification is focused on improving performance to achieve state-of-the-art models. such models typically act like a black box, and predictions are therefore not explainable. this results in poor integration of these models into clinical settings, given that clinicians need to understand why a patient is identified as depressed, so that they can make informed decisions in regards to diagnosis and evidence-based treatment @xcite . additionally, it has been shown that blackbox models are not generalizable across different data genres or domains @xcite . this accentuates the need for explainable models, as they could help to troubleshoot and understand the transfer between datasets -e.g. within social media or from social media to electronic health records @xcite . recently, and with the proliferation of deep learning in particular, explainable ai (xai) has attracted significant attention, with the field publishing multiple techniques that provide explanations for machine learning models. techniques like lime @xcite and shap (lundberg and lee, 2017) have been widely adopted and proven to work in different domains, mental health being one of them @xcite @xcite . clinicians rely on ongoing assessment of pa-tient progress and well-being for therapeutic decisions. many assessment instruments exist, including questionnaires such as the patient health questionnaire (phq-9) for depression and the general anxiety disorder (gad-7) screener for anxiety. phq-9 @xcite is one of the most commonly used and validated depression assessment tools that mental health clinicians and primary care physicians use. the questionnaire addresses the presence and severity of nine symptoms or categories such as problems with sleep, eating, and self-harm to assess and monitor a patient's depression severity. in this work, we leverage the availability of phq-9, a clinically accepted and interpretable tool to measure depression severity, and integrate its items into depression classification models as auxiliary classification tasks. we claim and prove that lime explanations generated for models that use such clinically grounded auxiliary tasks are better and more informative than explanations on other blackbox models that do not use these auxiliary models in the decision process. we summarize our contributions as follows: ## related work depression classification has been an important area of focus in mental health nlp in social media data and electronic health records (ehr). to overcome the challenges of data access and to create community-based datasets, many initiatives started using twitter and reddit platforms to create depression annotated datasets. these datasets were collected using self-reported terms and regular expressions such as i was diagnosed with depression @xcite , or in the case of reddit, using mental health related subreddits (e.g. r/adhd) as a proxy to retrieve relevant posts @xcite @xcite . many common techniques related to linguistic features are used to perform the classification task such as using liwc in social media @xcite and ehr @xcite . researchers used a variety of machine learning techniques that range from conventional methods such as svm @xcite and lr @xcite , to deep learning techniques such as feedforward networks @xcite , cnn and lstm @xcite . many recent work also explored the use of recent pre-trained language models to improve the depression classification task performance, such as bert-cnn in (rodrigues @xcite and albert @xcite . there has been a line of research that focused on predicting the symptoms (phq categories). @xcite introduced a deep neural network model to predict phq-4 scores in reddit depression dataset @xcite and daic-woz transcribed clinical interviews @xcite . @xcite proposed identifying the presence of the depressive symptoms using the auxiliary task of figurative usage detection. in the area of explainable ai (xai), most the work that has been done focused on using explainable techniques to highlight the most important features in depression prediction. @xcite to highlight which features were most salient in the depression classification model. @xcite used lime to understand which features weighed the most in identifying college students at high risk of depressive disorder. in a recent work by @xcite , the authors showed the positive impact of using depression classifiers that are constrained by phq-9 symptoms, on their generalizability across different datasets. ## data in this work, we focus on social media data because public access to clinical datasets is limited. the publicly available social media datasets that address depression classification only contain labels for depression @xcite , and it is challenging to find publicly available data that has annotations for both depression and phq-9 categories. for our experiments, we use the depression to ( 2 ) symptoms (d2s) dataset @xcite . it is a collection of english only tweets that was crawled using depression-related terms that can be categorized into one of the phq-9 categories (symptoms): (s1) lack of interest, (s2) feeling down or depressed, (s3) trouble with sleeping, (s4) lack of energy, (s5) eating disorder, (s6) low self-esteem, (s7) concentration problems, (s8) hyper/lower activity, and (s9) self-harm. the dataset contains the list of annotated tweet ids, and a total of 3738 tweets labeled as depressive and 8417 as not depressive (control). the depressive tweets are further annotated with symptoms, where a label of 1 is assigned for s9 if the tweet has mentions of self-harm thoughts, 0 otherwise, and so forth for all 9 categories, where multiple categories can receive a 1 annotation. it is worth mentioning that the data, unlike phq-9 questionnaire, does not have scores for each category, but only a binary label. additionally, the original dataset has annotations for sarcasm and metaphor labels for the depressive tweets, since @xcite focused on the task of understanding how to classify phq-9 categories using the sarcasm and metaphor language labels. however, in our work we focus on the depressive and phq-9 symptoms annotations, with all annotations scoped at the tweet level, not at the user level as is the case in some other datasets. we collected the tweets corresponding to the tweet ids described in d2s using the twitter api. some tweets had become unavailable since the publication of d2s, resulting in a reduced dataset with 2132 depressive tweets and 5698 control tweets. notwithstanding the change in dataset size, we adopt the train, dev, and test splits of d2s to maintain consistency. ## depression classification models understanding the domain and the task should be the foundation in designing an nlp model, as opposed to simply applying nlp state-of-the-art models that are hard to interpret. this is especially true for clinical and mental health nlp, where a lack of explainability would result in poor integration in clinical settings. in our work, we aim to build models that mimic a clinical setting, where the clinician uses the scores from phq-9 questionnaires to screen if a patient is suffering from depression and to assess its severity. in this section we discuss the approaches we used to build models that predict if a tweet is depressive or not. we propose three models; the first two, similarly to previous literature, focus on the depression classification task as a standalone problem, without considering how symptoms, in our case the phq-9 categories, can help interpret and influence the model's performance. the last model aims to study how predicting symptoms can help in classifying depressive tweets. in the following subsections, we will describe our models and the balancing techniques we used to address the skewed distributions for the depressive and symptom labels. ## discussion when we look at the results of the depression and symptoms classification task in tables 2, 3, and 4, we note that mlp yields the worst results across almost all the labels (dep and symptoms), whereas lr provides the best results for dep. however, its performance on the symptoms is poor, especially for concentration (s7) and activity (s8), where the number of positive instances is very limited. the mtl model, meanwhile, performs slightly worse than lr for the dep class, but is able to perform much better for all the symptoms and is not susceptible to the imbalanced nature of the data. for instance, the f1-scores for s7 and s8 in the mtl setting improve drastically. this observation supports the claim that using the symptoms with the depression labels can provide more reliable performance where we can think of the symptoms predictions as the first layer of explanations we can provide to the clinicians. at certain times and without any trigger, i think i am probably not even mentally ill, but rather just an attention seeking sh ** mentally ill | attention seeking sh ** [gold annotation] even, mentally, ..., seeking, ill, attention [lr] even, time, trigger, ..., mentally, ill [mlp] mentally, seeking, sh ** , trigger, mentally,..., ill, attention [mtl] note that the recall of the lr explanations is the highest among the three models at 0.61 ( to further support our claim and to make sure that ensembling multiple models will not also produce better results than d, we implement bagging techniques. we create 9 random samples to mimic the 9 symptoms sample size. for instance, sample 1 will randomly select 237 depressive and 1378 control tweets to mimic the size of the s1 dataset; the same technique would apply for each of the samples. we report the f1-score, in ## ethics statement although tweets are publicly available, given the sensitivity of the task, we took the following extra measures, in light of what has been previously published by @xcite and @xcite . ## conclusion providing models that are explainable and adopt clinically grounded questionnaires is critical in building nlp solutions that can be integrated in clinical settings. in this work, we show that using auxiliary models, namely for phq-9 categories/symptoms, in combination with the depression classification models, allows us to generate explanations that are more meaningful and have higher recall when evaluated against a gold standard dataset of manually annotated explanations. this implies that we need to conduct more studies that can benefit from clinical practices and measures, and integrate it into the modeling design choices. to the best of our knowledge, we are the first to produce gold annotations of explanations for the depression classification task, and conduct a thorough analysis on how augmenting with the symptoms can improve the quality of the explanations.