TEAM BIAS BUSTERS @ LT - EDI -2023: Detecting Signs of Depression with Generative Pretrained Transformers (2023)

Abstract: This paper describes our methodology adopted to participate in the multi-class classification task under the auspices of the Third Workshop on Language Technology for Equality, Diversity, Inclusion (LT-EDI) in the Recent Advances in Natural Language Processing (RANLP) 2023 conference. The overall objective was to employ ML algorithms to detect signs of depression in English social media content, classifying each post into one of three categories: no depression, moderate depression, and severe depression. To accomplish this we utilized generative pretrained transformers (GPTs), leveraging the full-scale OpenAI API. Our strategy incorporated prompt engineering for zero-shot and few-shot learning scenarios with ChatGPT and fine-tuning a GPT-3 model. The latter approach yielded the best results which allowed us to outperform our benchmark XGBoost classifier based on character-level features on the dev set and score a macro F1 score of 0.419 on the final blind test set.

## introduction and related works from a common-sense linguistic perspective, detecting signs of depression in text can be challenging for a number of reasons: it should be also noted that text analysis can provide only hints, but should never be used as a definitive diagnostic tool. only trained mental health professionals can diagnose depression. although discovering signs of depression in a written text is challenging because such text is not a direct indicator of someone's mental state, there are certain language patterns which might indicate a higher likelihood of depression. according to @xcite and @xcite the following is typical of texts written by people with depression in the order of increasing importance: similar ideas are repeated in newell et al. (2018) and @xcite . we attempted to quantify these characteristics in the form of counts of phrases that belong to each of these classes and use them as features for machine learning (ml) models. see more details in subsection 3.2 and subsection 4.2. in addition, @xcite states the importance of early detection of the signs of depression. the goal of their research was to study automatic analysis of texts to build predictive models that can identify individuals at risk of a mental disorder. the authors came up with four regression models to predict a higher emotional state of depression using such text features as the ratio of pronouns to nouns, ratio of verbs to nouns (readiness for action), ratio of finite verbs to number of sentences, and ratio of the number of punctuation marks to the number of sentences. ## dataset and task the dataset consists of social media posts in which people describe their emotions and feelings. the number of examples in each subset is as follows: training set -7201, development (dev) set -3245, test set -499. given these posts, our task was to classify the signs of depression into three categories: no depression, moderate depression, severe depression. as you can see from fig. 1 and fig. 2 , the distribution of categories is imbalanced with the majority category being "moderate depression" and the minority category -"severe depression". based on the common understanding that the same person cannot be both depressed and not depressed, and that two different people cannot write the same relatively long post in social media, we considered this a multi-class, but not multi-label classification i. e. each text can have only one label. in line with this, 232 complete duplicates were removed from the training set; complete here means that all the values in these rows in all columns were identical. in addition, there were 158 cases in the training set where the text of the post was the same, but the labels were different. since the same person cannot be depressed and not depressed at the same time, we decided to remove such cases because the true label was unknown (there were at least two different labels in each case), and we didn't feel to be qualified enough to decide which category each mislabeled text should belong to. there were only complete 23 duplicates in the dev set. as for the data leakage -there were only three posts that occurred both in the training and dev sets. the test set didn't have any overlap with the training or dev set. fig. 3 and fig. 4 demonstrate that the character length distribution across all datasets reveals a minority of abnormally lengthy texts. the majority, representing the 95th percentile, encompasses texts containing fewer than 2,500 characters. however, a substantial surge in text length is observed beyond this point, extending to and exceeding 20,000 characters. jumping ahead, we should say that training separate classifiers for different lengths of text didn't improve the aggregate results. a blind test set without labels was used for testing the model that had the best performance on the dev set. unlike the training and dev sets, the test set required heavy text cleaning as certain combinations of english characters and even single characters (mostly contractions at the sub-word level) figure 4: character length distribution -dev set ## conclusions our observations suggest that chatgpt exhibits a degree of unpredictability, complicating the task of identifying a consistently effective configuration due to its dynamic nature. hence, it is unsurprising that detecting depression using zero-shot and fewshot techniques proved challenging even for these cutting-edge models. in contrast, the largest finetunable openai model, davinci, which is older and smaller than chatgpt and lacks instruction following training, demonstrated superior efficiency for this task. the fine-tuning capability addressed both fewshot learning constraints which we discussed in subsection 4.1. the model, while being fine-tuned, sees all the training set examples, and during inference, you are only charged for the tokens in the single example to be classified. also, if our doubts about the annotation quality figure 8: confusion matrix -gpt-3 the exploration of non-textual features for depression detection warrants further study. enhanced methods of aggregating numerical information from text could also contribute to improved classification outcomes.