F ilip N @ LT - EDI - ACL 2022-Detecting signs of Depression from Social Media: Examining the use of summarization methods as data augmentation for text classification (2022)

Abstract: Depression is a common mental disorder that severely affects the quality of life, and can lead to suicide. When diagnosed in time, mild, moderate, and even severe depression can be treated. This is why it is vital to detect signs of depression in time. One possibility for this is the use of text classification models on social media posts. Transformers have achieved state-of-the-art performance on a variety of similar text classification tasks. One drawback, however, is that when the dataset is imbalanced, the performance of these models may be negatively affected. Because of this, in this paper, we examine the effect of balancing a depression detection dataset using data augmentation. In particular, we use abstractive summarization techniques for data augmentation. We examine the effect of this method on the LT-EDI-ACL2022 task. Our results show that when increasing the multiplicity of the minority classes to the right degree, this data augmentation method can in fact improve classification scores on the task.

## introduction the number of people suffering from depression has been steadily increasing since the 1990s @xcite , therefore it is essential that we find an efficient method to identify this on the internet. over the past few years, transformers have taken over the field of natural language processing (nlp) and achieved state-of-the-art results on various problems @xcite . some classification problems in machine learning deal with the problem of class imbalance. in this paper, we examine the effect different degrees of data augmentation have on the performance of transformer models on a text classification task. the method of data augmentation is done using abstractive summarizations. our data augmentation is done by first generating summarizations for each of the training examples and then balance the dataset using these generated summarizations. for this, first we discuss the related literature in section 2. then in section 3 we briefly describe the dataset used. this will be followed by the description of our methods in section 4, after which we discuss our results in section 5, then end the paper with our conclusions and plans for future work in section 6. ## related work data augmentation is nothing new to the field of nlp, it is one of the standard approaches when improving the results of a model. there are many different approaches to data augmentation and the nlp survey @xcite ) puts these methods into three different categories rule-based, example interpolation and model-based techniques. the latter is the approach that we focus on in this paper in which we use the t5 @xcite model to summarize the original posts. there are multiple different tasks that data augmentation can aim to solve such as mitigating bias, fixing class imbalance and few-shot learning. yet in this paper we solely focus on fixing class imbalance. the area of data augmentation for fine-tuning transformers is limited and is still being explored @xcite . yet some research has been done such as genaug @xcite which describes methods to use data augmentation to finetune text generators. genaug focuses on characterlevel synthetic noise and keyword replacement as augmentation methods for fine-tuning. although this data augmentation is done for text generation with gpt-2 @xcite and not for a sentiment-analysis task. the @xcite paper shows that there are effective ways to use data augmentation methods to fine-tune transformers to achieve better results on abstractive summarization. using transformers as a data augmentation method has been done previously in papers such as @xcite where they used dialogpt @xcite to generate new training data and effectively double the training data. the augmented dataset was then used to fine-tune a t5 model where they showed a positive effect on the results. previous work using transformers to detect depression in social media posts has been done by @xcite . where they used the bert transformer to analyze the risk a user was of self-harming themselves by classifying social media posts from that user. ## data the dataset was provided by the organizers of lt-edi-acl2022 @xcite in the competition and it contains social media posts from different users, categorized as severe, moderate and not depression. the dataset was created and annotated by the methods described in @xcite . these posts differ greatly from berts training data, the dataset is not very large and very imbalanced (some classes are largely underrepresented). therefore a good dataset to study the effect of data augmentation. we were provided a training set, validation set and a test set. labels for the test set however, have not been published yet. hence this paper is based solely on the results from the training and validation set, therefore our validation set is used as a test set, further references in this paper to a test set is the validation set from the lt-edi-acl2022 competition. ## methodology the pipeline used in this paper consists of two major parts, the first part performs the data augmentation by summarizing the training dataset, the second part is the classification of the levels of depression. our approach to solving the problem of detecting depression in social media posts is to finetune a multiclass bert transformer on our dataset using different degrees of data augmentation. the following section will describe in-depth how this is done. to ensure repeatability, our code is shared in a github foot_0 repository. ## experiments and results the following subsection describes the experiments and result that were produced. bert ran four epochs of fine-tuning and the best macro f1-score 5 https://huggingface.co/bert-base-uncased was chosen to represent the result for that model. the models were trained on a shared dgx-1 cluster with 8 × 32gb nvidia v100 gpus. ## conclusions and future work we examined a method of using an abstractive summarization model, t5 to do data augmentation. this was done before fine-tuning a bert transformer on the dataset which was balanced to different degrees. we found that with the right degree of augmentation, the proposed method improved the performance of the bert model on the task of detecting signs of depression. one technique that can be examined for further improvement of the proposed model is splitting the posts longer than 512 tokens into several parts, summarizing the parts individually, and creating the final summary by concatenating the individual summaries. future work for data augmentation for fine-tuning transformers could be done by comparing the result using an extractive summarization method such as matchsum @xcite . our method was examined on one dataset, future work should use this technique of data augmentation to fine-tune for different domains on multiple datasets. there are also different methods of augmentation other than summarization that future work should examine and compare. in this paper we used bert for our classification, future work should use other transformer models to see how they perform using our augmentation method.