Evaluating Lexicon Incorporation for Depression Symptom Estimation (2024)

Abstract: This paper explores the impact of incorporating sentiment, emotion, and domain-specific lexicons into a transformer-based model for depression symptom estimation. Lexicon information is added by marking the words in the input transcripts of patient-therapist conversations as well as in social media posts. Overall results show that the introduction of external knowledge within pre-trained language models can be beneficial for prediction performance, while different lexicons show distinct behaviours depending on the targeted task. Additionally, new state-of-the-art results are obtained for the estimation of depression level over patient-therapist interviews.

## introduction considerable interest has emerged in using natural language processing to unobtrusively infer one's mental health condition @xcite . a majority of studies have focused on predicting major depressive disorder (mdd) either as a symptom-based estimation @xcite or a binary classification problem @xcite . both clinically motivated research initiatives and social media studies have emerged. in the latter case, twitter @xcite , reddit @xcite and depressionrelated forums @xcite have fostered attention. in the former case, recorded patienttherapist conversations are transcribed and associated with self-assessment depression questionnaires, such as phq-8 @xcite or bdi @xcite . the daic-woz dataset @xcite has mostly been studied within the context of clinical research. different works have been proposed to automatically infer depression level on this dataset: multi-modal @xcite illustration of the lexicon-based input marking a) i'm pretty much good because see by me being a bus operator you run into circumstances and situations you gotta remain calm and still remain professional at the same time b) i'm @ pretty @ much @ good @ because see by me being a bus operator you run into circumstances and situations you gotta remain @ calm @ and still remain professional at the same time c) i'm @ pretty @ much @ good @ because see by me being a bus operator you run into circumstances and situations you gotta remain @ calm @ and still remain @ professional @ at the same @ time @ and text-based architectures @xcite . the primate dataset @xcite has also received recent attention within the context of early symptom prediction on social media posts. the most comprehensive work on this dataset is proposed by @xcite , which defines a context-and phq-aware transformer-based architecture. people with mdd have shown increased use of negative emotional words and decreased use of positive emotional words @xcite . in this line, xezonaki et al. (2020) and qureshi et al. ( 2020 ) used feature-level and task fusion of emotion and sentiment knowledge and showed improved performance for depression estimation. however, these works, along with other studies on social media mental health data @xcite , have used pre-transformer era neural architectures. recent state-of-the-art approaches that rely on transformer-based pre-trained language models (plms) have not explored external knowledge fusion @xcite . in this paper, we investigate whether pretrained language models could benefit from the introduction of emotional, sentimental, and domain-specific external knowledge from the lexicons: afinn (nielsen, 2011), nrc (mohammad and turney, 2013) and sdd @xcite . introducing this external knowledge into a transformer-based model is feature-level and is achieved by modifying the input with specific markers that highlight spans of text, as shown in results on the daic-woz dataset show that the performance of transformer-based models is impacted by the added lexicon information (especially sentiment), and new state-of-the-art values can be obtained from the combination of the three lexicons. however, such results are less expressive for the primate dataset, with slight improvements induced by the introduction of external information. overall, the improvement in predicting particular symptoms evidences that lexicon information can be helpful, provided that its content closely corresponds to the targeted task. ## methodology data. in this work, we use two depression datasets: daic-woz @xcite and primate @xcite . the daic-woz dataset contains 189 clinical interviews in a dialogue format. each interview has two actors: a human-controlled virtual therapist and a participant. the dataset is distributed in pre-determined splits, such that 107 interviews are used for training, 35 for validation, and 47 for testing. each interview utterance encoder word-level encoder pre-trained encoder mean [sep] pooling pre-trained encoder mean [sep] pooling pre-trained encoder mean [sep] pooling feed-forward layers symptom scores copy copy copy u 1 1 [sep] u 2 1 [sep] ... u 1 2 [sep] u 2 2 [sep] ... u 1 n [sep] u 2 n [sep] ... transformer encoder (4l, 12h) mean pooling ... ... ... in the dataset is accompanied with a phq-8 assessment, which consists of eight questions inquiring about symptoms. each question is scored from 0 to 3 on a likert scale, and the total phq score ranging from 0 to 24 is the sum of the eight symptom scores. according to a standard cutoff score of 10, the interviews can be divided into diagnostic classes, where subjects with phq-8 total @xmath0, and those with @xmath1. the eight listed symptoms are: loi (lack of interest), dep (feeling down), sle (sleeping disorder), ene (lack of energy), eat (eating disorder), lse (low self-esteem), con (concentration problem), mov (hyper/lower activity). the primate dataset is based on reddit posts from depression-related communities, or subreddits, in which people describe their health conditions. a total of 2003 posts were manually annotated with binary labels for each individual symptom from the phq-9 @xcite , each label signifying whether the corresponding symptom is discussed in the post or not. phq-9 has the same first eight symptoms as phq-8 and one additional sui (suicidal thoughts). the data was labeled by five crowd workers and verified by a mental health professional. the dataset is not presplit into the train, validation, and test sets, so we randomly take 1601, 201, and 201 posts for each split accordingly. model architecture. to encode the interview transcripts, we adopt the hierarchical model from @xcite . in their model, the interview is first split utterance-by-utterance, with each utterance processed by a word-level encoder. all utterance representations are then concatenated into one sequence, later processed by an utterancelevel encoder. in the end, the classification head produces a real number in the range from 0 to 3 for each symptom. several changes are made to the original architecture to gain training efficiency. first, the bilstm utterance-level encoder is replaced with a randomly initialized 4-layer 12-head transformer encoder. second, we change the way the input data is represented. in the original model, each utterance of the interview is encoded separately by a word-level encoder. this is far from optimal since most of the utterances are short (<10 tokens), thus, a lot of computation is wasted on padding tokens. instead, the utterances are concatenated into one input text separated by the [sep] special token. this way, the number of passes through the encoder is reduced from the number of utterances k to k, defined as in equation 1 , where |u i | is the number of tokens in an utterance and m is the maximum input length of the word-level encoder. in practice, it reduces the number of word-level encoder passes by ∼40 times for each input. after, we perform the mean [sep] pooling on the tokens representing each utterance to get the final utterance representation. the overview of the model architecture is presented in figure 1 . lexicons. to incorporate the external knowledge into the model, we use three lexicons: afinn (nielsen, 2011), nrc (mohammad and turney, 2013), and sdd @xcite . afinn is a sentiment lexicon that includes a list of 2,477 terms manually rated for the sentiment valence with a value between -5 (negative) and +5 (positive). nielsen (2011) used twitter postings together with different word lists as a source for the lexicon. nrc is a word-emotion association lexicon that is a list of 14,182 words and their associations with eight basic emotions (anger, fear, anticipation, trust, surprise, sadness, joy, and disgust) and two sentiments (negative and positive). mohammad and turney (2013) compiled terms from macquarie thesaurus @xcite , wordnet affect lexicon @xcite , and general inquirer @xcite and labeled them with the help of crowd-sourced workers. sdd is a part of the social-media depression detector and is a lexicon of more than 1,620 depression-related words and phrases created in collaboration with a psychologist clinician. input marking. in particular, we employ the technique proposed by @xcite to identify and annotate the lexicon words in the input text. it involves marking a lexicon word using the "@" token on either side (see ## results and discussion the combination of all lexicons is marginally beneficial overall, and results have deteriorated with the exclusive introduction of the sdd lexicon. on the other hand, for the mebert model, the combination of all the lexicons produces the best results overall, both symptom-wise and for the global phq-8 score. furthermore, both afinn and nrc lexicons improve the prediction for the mebert model, similar to the bert model. also, when only the sdd lexicon is used for input marking, the model shows worse performance than the baseline setting. figure 2 depicts a more detailed overview of the best-performing models: bert+nrc and mebert+all. additionally, we finetune the +rand version of both bert and mebert to verify if the improvement comes only from the input marking by randomly marking 8% of the words in each interview. from the results, the improvement for the bert+nrc model comes from the non-depressed population. mebert+all model, however, improves for both depressed and nondepressed populations and is less sensitive to the marking bias. interestingly, +rand models show some improvement for the non-depressed population, suggesting that input markings alone act as a regularizer. the results from the daic-woz show that plms can indeed benefit from the introduction of external knowledge about the sentiment and emotional value of the words. surprisingly, the introduction of the depression-specific lexicon had the opposite effect. we hypothesize that two reasons could cause it. first, as seen in ## conclusion this paper targets lexicon incorporation in transformer-based models for symptom-based depression estimation. the external information is supplied through a marking strategy, which avoids any modification to the model's architecture. the set of endeavoured experiments shows that introducing sentimental, emotional and/or domainspecific lexicons can correlate with overall performance improvement if adapted to the targeted task foot_1 .