KUCST @ LT - EDI - ACL 2022: Detecting Signs of Depression from Social Media Text (2022)

Abstract: In this paper we present our approach for detecting signs of depression from social media text. Our model relies on word unigrams, part-of-speech tags, readabilitiy measures and the use of first, second or third person and the number of words. Our best model obtained a macro F1-score of 0.439 and ranked 25th, out of 31 teams. We further take advantage of the interpretability of the Logistic Regression model and we make an attempt to interpret the model coefficients with the hope that these will be useful for further research on the topic.

## introduction depression foot_0 , foot_1 is a mental illness that affects to the 5% of adults. the world health organization states that depression is the leading cause of disability worldwide. human beings have varying mood, but depression is a condition that affects further than solely the mood. depending on the degree of intensity of its symptoms, it may become a serious health condition. in spite of the magnitude and risk, there is effective ways of treating mild, moderate and severe depression. in this paper we present our attempt for automatically classifying whether a social media post shows signs of moderate or severe depression. this is part of the shared task on detecting signs of depression from social media text at the lt-edi-2022 workshop (sampath et al., 2022). all our code is available in the following repository. @xcite the paper is structured as follows. first we introduce some related work on the topic. then, we introduce the data that we employed. we continue with the used features and the actual models that we trained. after that we present the results and briefly discuss the model coefficients, and finally we conclude the paper with some possible future directions. ## related work there have been several attempts to model the language of people with depression. in some works the focus is on detection of social media posts from users with different degrees of depression and in some other cases, the goal was to analyze the language style of people with depression. there is a large number of works that have attempted to detect depression from social media text. some works employ twitter (coppersmith et al., 2015; de choudhury et al., 2021; cavazos-rehg et al., 2016; mowery et al., 2016; pirina and çöltekin, 2018; tadesse et al., 2019) and they work with different degrees of granularity with depression or depression-related symptoms. other researchers have employed other social media that contain longer essays, such as reddit (ireland et al., 2020; iavarone and monreale, 2021) for the detection of depression, by employing posts of users that were self-reported to have depression. reddit posts have been further employed for, for instance, bipolar disorder detection (sekulic et al., 2018) or anxiety detection (shen and rudzicz, 2017). with regards to features, many works make use of the linguistic inquiry and word count (liwc) (pennebaker et al., 2007). as in other natural language processing related tasks, models based on contextual word embeddings have shown a good performance for depression detection, e.g. (martínez-castaño et al., 2020). as they report, the performance of the model is high but the interpretability of the model could be improved. besides, the use of personal pronouns have been analyzed by many researchers. for instance in rude et al. (2004), they analyzed the language use of currently-depressed, formerly-depressed and never-depressed college students. among other factors, they analyze the use of the first person pronoun "i" and they found that formerly-depressed and currently-depressed participants used the word "i" more often than the never-depressed participants. furthermore, tackman et al. (2019) claim that depressive symptomatology is manifested in a greater use of the first-person singular pronoun and find a small but reliable positive correlation between depression and i-talk. ## data we make use of the data provided by the organizers of the shared task, built from reddit posts (sampath and durairaj, 2022). @xcite some of these posts have no depression signs, others show moderate depression signs and finally, there are the ones that show severe depression signs. the dataset contains 8891 posts, from which the ones with no, moderate and severe depression signs are 1971, 6019 and 901, respectively. all posts are written in english. figure 1 shows a histogram with the length of the posts. ## features and models in this section we present the features that we employed. many of the features have been widely used for text classification and authorship analysis. words. bag of words as implemented by the countvectorizer package from the scikit-learn library (pedregosa et al., 2011). the expectation was that word usage might differ from depressed to non depressed users, and therefore, we expected that this feature would result beneficial. ## results and discussion our best model, the second one, obtained a macro f1-score of 0.4429 on the test data. the first model performed marginally worse with a macro f1-score of 0.439. when performing our own experiments based on the training data, using a balanced train/test split, we had observed a rather higher performance, from which we could say that our model does not generalize well enough. from the results, and by comparing to the rest of participants, we can say that our model has several aspects to be improved. in the team wise classification our model ranked 25 th , out of 31 teams. as the logistic regression model features are interpretable, we decided to analyze them more thoroughly, with the hope that this analysis is helpful for further research. for this analysis, we used the second model that makes use of the all the features and they were obtained after training the model @xcite all parameters are set to the default values. 8 https://scikit-learn.org/0.24/ modules/generated/sklearn.linear_model. logisticregression.html with all available training data. figures 3 , 4 , 5 and 6 show the same sorted ranking of the features. in each figure we mark the position of the top 5 features, for each output class and for each feature template. @xcite figure 4 shows that punctuation marks and nouns are can be good predictors. in figure 5 we can observe that the flesch reading ease metric seems to be a good predictor together with the type token ratio. from figure 6 we can observe that the first person ratio and the plural ratio seems to have a rather high effect in at least two classes of posts, meaning that they could be good predictors. finally, figure 3 shows the importance of the top 5 words. these last features seem to have more importance than other features. this is because the vectorizer for words foot_7 was used in the default configuration and no normalization was done afterwards (all other features had values between 0 and 1). this means that at the current stage we cannot compare the importance of specific features across feature templates based on the coefficients of the model. all the observations regarding feature importance should be taken with a grain of salt. a better approach would be to use a bootstrapping approach, training several models from subsets of the training corpus and analyzing the weight importance among several of those models. ## conclusion and future work in this paper we presented our attempt to classify whether a social media post from reddit shows signs of depression. we employed simple features and a linear model and we made an attempt to interpret the learned coefficients. as mentioned above, the model has several aspects that could be improved given its performance. below we outline some possibilities for further research. following recent advances in natural language processing, we think that including a pretrained word embedding model, such as bert (devlin et al., 2019) would have positively contributed to the performance. these features could be additional features to the ones that we currently use or we could even fine-tune a pretrained model for this specific task. another aspect we believe that could improve the model is to include further syntactic information. the use of dependency parsing is being currently tested, but besides, there is also an extension of the readability package 11 , where syntactic information is obtained. in addition to that, we expect that including the average sentiment of a post could be a relevant feature. furthermore, recent advances in structured sentiment analysis @xcite (barnes et al., 2022) could potentially reveal mood changes.