Multi-Task Learning to Capture Changes in Mood Over Time (2022)

Abstract: This paper investigates the impact of using Multi-Task Learning (MTL) to predict mood changes over time for each individual (social media user). The presented models were developed as a part of the Computational Linguistics and Clinical Psychology (CLPsych) 2022 shared task. Given the limited number of Reddit social media users, as well as their posts, we decided to experiment with different multi-task learning architectures to identify to what extent knowledge can be shared among similar tasks. Due to class imbalance at both post and user levels and to accommodate task alignment, we randomly sampled an equal number of instances from the respective classes and performed ensemble learning to reduce prediction variance. Faced with several constraints, we managed to produce competitive results that could provide insights into the use of multi-task learning to identify mood changes over time and suicide ideation risk.

## introduction for many countries, suicide has been a formidable challenge, where 1.3% of world deaths in 2019 were due to suicides. of the committed suicides, most of them were by individuals before reaching their fifties and in countries with low to middle income @xcite . considering these factors, it is of utmost importance for any institution responsible for the mental health of the population to early detect users susceptible to suicide ideation and mental disorders. in recent years, social media has become an integral part of the everyday life of many. according to @xcite , more than 25% of users aged between 15 and 64 have shared their personal information (e.g., pictures, videos, text-based posts) publicly. this data rich with personal information opens the pathway for many research opportunities. the importance of using social media data to detect users susceptible to suicide ideation @xcite and mental disorders @xcite was demonstrated throughout the clpsych workshop series. when analyzing research, including publications in the clpsych workshop series, we could see that in comparison to traditional machine learning methods @xcite @xcite , recent research has focused more on using deep learning architectures @xcite @xcite that considerably reduce the time and effort required for feature engineering. however, researchers have continued using traditional machine learning methods to predict individuals susceptible to mental disorders and suicide ideation, which could be due to the lack of large sets of annotated data (e.g., @xcite or to the requirement of explainability (e.g., @xcite ). in this paper, we describe the experiments conducted using deep learning methods, specifically with multi-task learning, to predict a user's mood change over time (i.e., either a switch or an escalation in the mood) and also the suicide ideation risk level where a selected user can be categorized into one of the following risk categories: low, moderate, or severe. the main reason for selecting multi-task learning is to leverage its capabilities of sharing knowledge between related but different tasks that could potentially alleviate the negative impact of having a small number of training instances. for example, we identified the negative impact of having a limited number of data points during model training, specifically when using deep learning architectures where different regularization methods were used to reduce model overfitting and increase the model's generalizability. when predicting suicide ideation risk level, we used an additional dataset from @xcite , named the self-reported mental health diagnoses (smhd) dataset, which consists of users who have self-declared mental disorders. similar to gamaarachchige (2021), which demonstrates the impact different mental disorders have on suicide ideation detection (i.e., whether an individual is susceptible to suicide ideation or not), we investigate the impact mental disorders have on different suicide ideation risk levels (i.e., low, moderate, or severe). ## task and data the clpsych 2022 shared task consisted of two subtasks @xcite . the first task was to identify a user's mood change over time @xcite , and the second task was to predict the level of suicidality risk for an individual @xcite . then, when predicting the suicidality risk, the participants were encouraged to discover if there is any relationship between the mood change over time and the risk of suicidality. the dataset provided to the task participants consisted of users and their posts extracted from the reddit social media platform. apart from 3,089 posts distributed across 139 timelines posted by 83 users, the rest of the users were sampled from the university of maryland reddit suicidality dataset @xcite and the erisk dataset @xcite . the combined dataset statistics are shown in for both tasks, we combined the text fields "title" and "content", and after several preliminary preprocessing steps, we identified 5,143 posts where the majority of the posts were categorized as "none". the distribution of the classes in the training dataset is shown in ## methodology as mentioned before, we based our experiments on multi-task learning and specifically an architecture using a combination of soft and hard parameter sharing. multi-task learning allows related tasks to share representations @xcite , and based on how parameters are being shared, can be categorized into two types of architectures, which are hard parameter sharing and soft parameter sharing @xcite . each task will share model weights in hard parameter sharing, and features unique to individual tasks will be extracted through the taskspecific layers. even though model weights are not shared between layers in soft parameter sharing, the parameters are regularized between the layers to discover similarities. we used a custom loss function that combines "categorical cross-entropy", "mean squared error", and "cosine similarity" to regularize layer weights. when using mtl for "task a" (i.e., according to figure 1 ), the two tasks were to predict whether the post is a "switch" or "none" (i.e., "is" or "o") or whether it is an "escalation" or "none" (i.e., "ie" or "o"). to prepare the training and validation input for each task, we sampled an equal number of instances from each class where the number of instances to sample is based on the minority class. selecting an equal number of instances for each class made it possible to align the tasks so that similar tasks could potentially share a common feature space. we kept aside a sample with a class distribution to be the same as the original dataset for testing. for "task a", the task-specific layers consist of a multi-channel convolutional neural network (cnn) @xcite where each channel was responsible for filtering features constituting bigrams and trigrams. to reduce the number of learnable parameters, the output from the cnn layers was further transformed using global maximum pooling and then sent through a feedforward neural network. the output from each channel was then merged to form vectors that represent the taskspecific features. these vectors were submitted to a loss function to regularize the network weights further. the merged outputs from each task-specific layer were concatenated to form the shared representation where each task will learn from a common feature space. it was identified that the model started to overfit the training data within a few epochs and consequently generated poor results during inference. to overcome model overfitting, we used several regularization techniques such as dropout @xcite (i.e., a probability of 0.4 for "task a" and 0.2 for "task b")and l1 and l2 regularization to penalize larger weights in the multi-channel cnn. further experiments discovered that making the model more or less complex reduced prediction accuracies due to either overfitting or underfitting, respectively. we adopted an ensemble learning approach to reduce the variance in the results, which could be due to noise and random sampling. model training and evaluation were done on three stratified training and validation splits where the final output is generated using an ensemble strategy on the combined predictions. we used the model averaging ensemble @xcite strategy to generate the output. for "task b", we used the same methods as for "task a", except that we used an additional dataset to enhance the shared feature space between users susceptible to suicide ideation and mental disorders. therefore, we selected a random sample of users similar to the number of users in the suicide ideation detection dataset. for example, to extract shared hidden features between users with severe suicide ideation risk and ptsd, we randomly selected 61 users who have self-declared ptsd from the smhd dataset. the number of users is identified from the training dataset, where 61 users are categorized with severe suicide ideation risk. the output of the suicide ideation detection task predicts three classes, that is, whether the user has a "low", "moderate", or "severe" suicide ideation risk. for the second task, we conducted experiments using a different combination of mental disorders by predicting whether a given user has ptsd, anxiety, or bipolar disorder. the final predictions are based on a model where users with "moderate" and "severe" suicide risks were aligned (i.e., sharing a common feature space) with users who have self-declared ptsd, and users with "low" risk were aligned with users who have selfdeclared anxiety. we used randomly initialized and trainable embedding layers with a dimension of 300 units for both subtasks. for task-specific layers, we used rectified linear unit (relu) (2010) activation function and adam optimizer (2015) with a learning rate 0.001 to update network weights. ## experiments and results we trained our models for fifteen epochs and reduced the learning rate by a factor of 0.1 if the validation loss did not improve. if the validation loss did not continuously improve, we stopped training and returned the model weights that produced the minimum loss. for both tasks, we trained our models using a mini-batch of size 16. finally, we selected the label with the highest probability from the output generated using the model averaging ensemble. we submitted three results for "task a" and one for "task b". the difference between our two submissions, "uottawa-ai(2)" and "uottawa-ai(3)", is based on regularization, where with more optimized regularization hyperparameters (i.e., on the submission uottawa-ai(2)), we managed to train our model for more epochs and as a result produced a more generalized model. the "uottawa-ai(1)" submission results are from a model with fewer learnable parameters. our results, compared to a majority class baseline and two preliminary experiments conducted by the task organizers @xcite , are mentioned in tables 4, 5, 6, and 7. the results are macro averaged at the post level, window-based, and coverage-based (please refer @xcite for more details on the evaluation metrics). ## discussion when analyzing the results of "task a", we could see that our proposed architecture has produced competitive results when compared against the baseline and two of the preliminary experiments that use tf-idf features with logistic regression and the bert (bidirectional encoder representations from transformers) language model trained using the talklife dataset @xcite . we also identified that our submission "uottawa-ai(2)" has produced better coverage and windowbased (refer to even though the test results for the "task b" model have produced better outcomes than the majority class baseline and the preliminary models trained by the task organizers (refer to table to 7, our model has not performed well in comparison to the best results. one of the critical reasons for the low results is class imbalance. during training, there were only 11 instances for the "low" risk class compared to 55 and 61 for "moderate" and "severe" risk (refer to ## conclusion and future work we have investigated the applicability of multi-task learning to predict the change in mood of a social media user over time. with limited experiments, we managed to identify that mtl can be effectively applied to predict whether a post contains a mood shift, an escalation, or no change. using different mtl architectures, which adopted different forms of parameter sharing strategies, it was identified that a combination of both the parameter sharing strategies (i.e., hard and soft parameter sharing) managed to produce better results. the main drawbacks we faced when using deep learning methods for classification are the class imbalance and the limited number of data points. for both tasks, we adopted a sampling strategy that facilitates task alignment. for "task b", we introduced a complementary task intending to enrich the hidden features space so that we could, to a certain extent, eliminate the negative impact of having a smaller dataset with class imbalance. when analyzing the prediction outcomes, we could assume that features shared by certain mental disorders are not sufficient to define a decision boundary over suicide ideation risk levels. in future research, we will look into the possibilities of improving the prediction accuracies by making changes to the current architecture (e.g., by changing the constructs of task-specific and shared layers) and also by adding contextual (e.g., elmo foot_0 @xcite , bert @xcite ) and non-contextual embeddings (e.g., word2vec @xcite , fasttext @xcite ).