UMUT eam@ LT - EDI - ACL 2022: Detecting Signs of Depression from text (2022)

Abstract: Depression is a mental condition related to sadness and the lack of interest in common daily tasks. In this working-notes, we describe the proposal of the UMUTeam in the LT-EDI shared task (ACL 2022) concerning the identification of signs of depression in social network posts. This task is somehow related to other relevant Natural Language Processing tasks such as Emotion Analysis. In this shared task, the organisers challenged the participants to distinguish between moderate and severe signs of depression (or no signs of depression at all) in a set of social posts written in English. Our proposal is based on the combination of linguistic features and several sentence embeddings using a knowledge integration strategy. Our proposal achieved the 6th position, with a macro f1-score of 53.82 in the official leader board.

## introduction the automatic analysis of depression is a medium that allows people to support their mental health @xcite . the shared-task dep-sign lt-edi (acl-2022) @xcite aims to measure the ability of neural networks and natural language processing (nlp) tools to detect signs of depression from social media posts written in english. it is worth noting that this is not the first shared task concerning the identification of depression. in @xcite , the organisers of erisk 2017 develop a pilot project which main purpose is the identification of early risk detection of depression. in this shared task, the organisers proposed a multi-classification challenge that consists of identifying whether a moderate or severe sign of depression is observed in a short text or, on the contrary, no sign of depression is observed. for this, the performance of all participants is ranked using the macro averaged precision, recall and f1-score. the details of the dataset compilation can be found at @xcite . the dataset is distributed into three folds: training, validation, and testing. we decided to use this distribution and not to merge train and validation to make a custom training-validation split. ## methodology our pipeline can be summarised as follows. first, documents are pre-processed by removing punctuation symbols, spaces, emojis, and punctuation. second, four feature sets are extracted from the documents: linguistic features (lf), sentence embeddings from fasttext (se), bert (bf), and roberta (rf). third, several neural networks with different combinations of the feature sets are trained using hyperparameter tuning. forth, two additional ensembles are created to combine the features. finally, we use the best neural network to get the final submission with the official test. next, we describe the feature extraction stage. the linguistic features (lf) are extracted with the umutextstats tool @xcite . the linguistic features are related to stylometry (for instance, word and sentence length, or type-token ratio), part-of-speech, emojis and generic social network jargon. the main advantage of linguistic features versus state-of-the-art embeddings is that linguistic features are easy to interpret at the same time they achieve promising results, specially in author analysis tasks @xcite . the sentence embeddings from fasttext (se) are extracted with the fasttext tool @xcite . these sentence embeddings are not contextual. that is, the same word has the same representation, regardless of its context. finally, the sentence embeddings from bert (bf) and roberta (rf) are extracted from distilled models @xcite . we use the distilled versions because they require less computational resources. to obtain the sentence embeddings from bert or roberta, a hyperparameter selection stage of 10 models is conducted to obtain a good configuration of the models. next, the sentence embeddings from bert and roberta are obtained from the [cls] token (using the approach described at @xcite ). during the hyperparameter selection stage, we use tree of parzen estimators (tpe) @xcite for determining the best parameters (weight decay, batch size, warm-up speed, number of epochs, and learning rate). the next step is the training of several neural networks. we train a neural network for each feature set (lf, se, bf, rf), and a neural network that combines all feature sets (lf + se + bf + rf). all these neural networks are trained with hyperparameter selection. for this, we rely on ray tune @xcite . for each training, we evaluate different number of hidden layers, neurons, batch size, learning rate or regularisation mechanisms. we distinguish between (1) shallow neural networks, that are simple neural networks composed of one or two hidden layers with the same number of neurons in each layer; and (2) deep neural networks, that have 3, 4, 5, 6, 7 or 8 hidden layers. besides, the layers of deep neural networks are evaluated with different number of neurons disposed in several shapes (brick, triangle, diamond, rhombus, and funnel). for the rest of the parameters, we evaluate large batch sizes due to class imbalance, a dropout mechanism for regularisation (in different ratios), and small and large learning rates. the results for the hyperparameter optimisation stage are shown in ## results and discussion we report the results achieved with the validation split. from the results achieved with the feature sets separately, bf is the one that achieves better results (77.27% of f1-score). this result is similar to rf (76.91% of f1-score) and outperforms largely se and lf. with the knowledge integration strategy, the results outperform the ones achieved separately, with a f1-score of 77.90. besides, when the results are combined with ensembles, the results are larger with the average of the probabilities (mean) achieving a macro f1-score of 78.69. we decided to use for the final submission the predictions obtained with the knowledge integration strategy. this decision is taken because in past competitions we have achieved better results with this strategy with the official test (that is, we suspect this strategy generalises better than ensemble learning). accordingly, we show the classification report of the validation split in next, ## conclusions and promising research lines here we have described the participation of umuteam in the lt-edi-acl2022 shared task, concerning the identification of moderate and severe signs of depression in short texts. we achieved 6th position from a total of 31 participants with a system that combines linguistic features and three forms of sentence embeddings using knowledge integration. we are proud of our participation as it has allowed us to evaluate a subset of languageindependent linguistic features. accordingly, we will continue to adapt our methods to english. specifically, we will include linguistic features from figurative language, as the ones described at (del pilar salas-zárate et al., 2020).