Curriculum-guided Abstractive Summarization for Mental Health Online Posts (2022)

Abstract: Automatically generating short summaries from users’ online mental health posts could save counselors’ reading time and reduce their fatigue so that they can provide timely responses to those seeking help for improving their mental state. Recent Transformers-based summarization models have presented a promising approach to abstractive summarization. They go beyond sentence selection and extractive strategies to deal with more complicated tasks such as novel word generation and sentence paraphrasing. Nonetheless, these models have a prominent shortcoming; their training strategy is not quite efficient, which restricts the model’s performance. In this paper, we include a curriculum learning approach to reweigh the training samples, bringing about an efficient learning procedure. We apply our model on extreme summarization dataset of MentSum posts —-a dataset of mental health related posts from Reddit social media. Compared to the state-of-the-art model, our proposed method makes substantial gains in terms of Rouge and Bertscore evaluation metrics, yielding 3.5% Rouge-1, 10.4% Rouge-2, and 4.7% Rouge-L, 1.5% Bertscore relative improvements.

## introduction summarization of mental health online posts is an emerging task that aims to summarize users' posts who are seeking help to enhance their mental state in online networks such as reddit foot_0 and reachout foot_1 . the post might address several issues of the user's concerns or simply be an elaboration on the user's mental and emotional situation. with user preference, each user-written post can be accompanied by a succinct summary (known as tl;dr foot_2 ), con-densing major points of the user post. this tl;dr summary is deemed to urge the counselors for a faster read of the user's posted content before reading the post in its entirety; hence, counsellors can provide responses promptly. herein, we aim to improve state-of-the-art results reported in @xcite for this task. large-scale deep neural models are often hard to train, leaning on intricate heuristic set-ups, which can be time-consuming and expensive to tune @xcite . this is especially the case for the transformers-based summarizers, which have been shown to consistently outperform the rnn networks when rigorously tuned @xcite , but also require heuristics such as specialized learning rates and large-batch training @xcite . in this paper, we attempt to overcome the mentioned problem on bart @xcite transformersbased summarizer by introducing a curriculum learning (cl) strategy @xcite for training the summarization model, leading to improved convergence time, and performance. inspired by humans' teaching style, curriculum learning suggests moving the teaching process from easier samples to more difficult ones and dates back to the nineties @xcite . the driving idea behind this approach is that networks can accomplish better task learning when the training instances are exposed to the network in a specific and certain order, from easier samples to more difficult ones @xcite . in the context of neural networks, this process can be thought of as a technique that makes the network robust to getting stuck at local optima, which is more likely in the early stages of the training process. given the mentioned challenge of summarization networks, we utilize the superloss @xcite function that falls into the family of confidence-aware curriculum learning techniques, introducing a new parameter called confidence (i.e., σ) to the network. we validate our model on mentsum @xcite dataset, containing over 24k instances mined from 43 mental health related communities on reddit social media. our experimental results show the efficacy of applying curriculum learning objectives on bart summarizer, achieving a new state-of-the-art performance. ## related work while majority of works in mental health research have focused on studying users' behavioral patterns through classification and prediction tasks @xcite @xcite @xcite @xcite , summarization of online mental health posts has been recently made viable. @xcite were the first to step on this direction via introducing mentsum dataset of online mental health posts, pinpointing the baseline results. curriculum learning (cl) has gained growing interest from the research communities during the last decade @xcite @xcite . @xcite were the first to apply this strategy in the context of sequence prediction through scheduled sampling approach, which gently changes the training process from ground truth tokens to model-generated ones during decoding. sample's difficulty is a key concept in this scheme as it is used to distinguish easy examples from difficult ones. researchers have used many textual features as the "difficulty measure" including ngram frequency @xcite , word rarity and sentence length @xcite . recent works @xcite have made use of confidence-aware approaches that learn the difficulty of training samples and dynamically reweight samples in the training process. ## our approach in this section, we describe the details of our proposed model, where a curriculum learning architecture is added to the bart's transformers-based framework, upweighting easier training samples; hence, increasing their contribution in learning stage. curricular learner for bart. considering the applicability of curriculum learning in training large-scale networks, we aim to use it in our summarization framework. before incorporating the curriculum learning strategy into our model's train- training with standard loss function training sample neural network (e.g., bart) training sample neural network (e.g., bart) loss (e.g., cross-entropy) training with superloss ground truth summary ground truth summary superloss loss (e.g., cross-entropy) optimizer optimizer minimizing minimizing . in which λ is the regularization parameter, and τ is the running or static average of task loss (i.e., l) during the training. while superloss provides a well-defined approach to curriculum learning strategy, learning σ parameter is not tractable for tasks with abundant training instances such as text summarization. to circumvent this issue and hinder imposing new learnable parameters, superloss suggests using the converged value of σ i at the limit, using this technique, the confidence parameters are not required to be learned during the training. @xcite found out that σ * λ (ℓ i ) has a closed-form solution, computed as follows, in which w is the lambert w function. with this in mind, superloss upweights easier samples dynamically during the training, providing a curriculum learning approach to summarization. we call this model currsum in our experiments. ## ethics and privacy while we use publicly available reddit data to train and validate our model, we recognize that special care should be taken when dealing with this type of data due to its sensitivity and users' privacy @xcite . thus, we have made no attempts to identify, contact the users, or explore user relations to trace their social media account. it has to be mentioned that the mentsum dataset is distributed through data usage agreement (dua) foot_5 document to further ensure that users' identity would never be disclosed. ## conclusion generating short summaries given the users' online posts can save counselors' reading time, and reduce their fatigue. on this basis, they can provide faster responses to community users. while neural transformers-based summarization models have shown to be promising, they suffer from inefficient training process that hinders their potentials for showing a promising performance. to compensate for this issue, in this paper, we incorporated a confidence-aware curriculum learning approach, which uses task-agnostic superloss, to the summarization framework in the hope of increasing model's generalization, and ultimately improving model performance. our automatic evaluations over mentsum dataset of mental health posts show the effectiveness of our model, yielding 3.5%, 10.4%, 4.7%, 1.5% relative improvements over bart summarizer on rouge-1, rouge-2, rouge-l, and bertscore, respectively. our model tailors the new state-of-the-art performance on mentsum dataset. we further showed various system-generated summaries to showcase the qualities and limitations of our proposed model.