Exploring transformers and time lag features for predicting changes in mood over time (2022)

Abstract: This paper presents transformer-based models created for the CLPsych 2022 shared task. Using posts from Reddit users over a period of time, we aim to predict changes in mood from post to post. We test models that preserve timeline information through explicit ordering of posts as well as those that do not order posts but preserve features on the length of time between a user’s posts. We find that a model with temporal information may provide slight benefits over the same model without such information, although a RoBERTa transformer model provides enough information to make similar predictions without custom-encoded time information.

## introduction with the ubiquity of data online come opportunities for studying and providing support to individuals and communities. for example, a user's posts on reddit fora may reveal information about that user's emotional state over time @xcite . additionally, these tasks may seek to make early predictions about mental states, allowing for prompt intervention when needed @xcite . this work represents one such attempt as part of the 2022 clpsych shared task @xcite ), foot_0 using a transformer-based architecture to make predictions about changes in reddit user moods over time. we demonstrate how state-of-the-art transformer models like roberta @xcite provide predictions of changes in mood that are difficult to improve upon with custom features or sequential architectures. ## related work previous work has used social media to examine the ability of neural networks to make predictions about depression @xcite , suicidality @xcite , and related mental health disorders @xcite . @xcite introduce a task where participants attempt to make early identifications of depression from social media, finding that further improvements needed to be made before such models could successfully be used in a clinical setting. work on predicting temporal shifts in language use has frequently focused on lexical-semantic changes over time, with only recent research focusing on the impacts of temporally-aware language models on downstream tasks @xcite . for example, in a span prediction task, @xcite used a simple string representation of the year when texts were first created to finetune t5 language generation models. they found that adding the year as a prefix to the input aided learning of seen facts, improving performance on predictions of future events. tsakalidis et al. (2022b) identify individuals' changes in mental health over time. this temporal dimension can be helpful in monitoring clinical outcomes and it can also help online platform moderators prioritize interventions depending on an individual's vulnerability at a certain moment in time. they provide strong baseline models for this task, including both timeline-based models and timeline-agnostic models, finding that bert-based models outperform their remaining systems. thus, it is reasonable to assume that finetuning existing language models using the time information available in social media posts can help detect changes in mental health. ## approach we examine both timeline-agnostic models, which accept single data points in random order and timeline-preserving models, which require the order of posts in each timeline to be maintained. timeline-preserving models are expected to be most successful, as the dataset includes labels such as switch in mood (is) that require information from past data points to predict the label of the present data point. we incorporate such information both through sequence models such as lstms @xcite ) that encode and preserve information from previous data points to make predictions, as well as through explicit custom features representing the time between data points, which we refer to as time lag features. we choose roberta as a base for our models, as @xcite find bert-based models perform well on this task, and roberta models frequently outperform bert in practice @xcite . ## data the data used in this work are those selected for the clpsych 2022 shared task @xcite and drawn from the umd reddit suicidality dataset version 2 @xcite with queen mary university of london annotations, reddit-new, a new dataset created from posts by reddit users who posted on mental-health related subreddits and annotated for suicidality and moments of change @xcite , and the erisk dataset @xcite . these data consist of timelines of reddit posts by a series of users, selected based on individuals who participated in subreddit fora related to mental health. overall suicide risk; here, we focus solely on predictions of changes in mood over time. in order to access the data, each member of this team signed a data usage agreement and an nda due to the sensitive nature of this data. the data consists of a total of 4775 posts, broken down as shown in ## implementation details roberta @xcite models were based on hugging face's roberta-base 2 and were trained via the pytorch @xcite version of the robertaforsequenceclassification class using cross-entropy loss. roberta is not frozen for any of the architectures; linear layers, lstms, etc. were trained alongside the roberta weights. for timeline-agnostic models, we randomized the order of all posts in the training data. for timeline-preserving models, we randomized the order of the timelines in the training data but preserved the order of individual items within each timeline. for timeline-preserving models, when fewer than two previous posts were available (e.g., at the beginning of a timeline), padded masked posts were fed instead but were not used to update model parameters. ## model selection on the development set we used the development data to experiment with the various architectures we considered, with the 2 https://huggingface.co/roberta-base goal of selecting the best models to evaluate on the test set. each of the models described in section 3 was evaluated using the development partition. the timeline-preserving models (roberta-pre2-lin and roberta-pre2-lstm) showed much worse performance than the timeline-agnostic models, although the roberta-pre2-lin model that concatenated the three posts and fed them through linear layers did perform best for precision in the switch class and recall in the no-change class. still, its overall performance as measured by macro f1 was much worse than the timeline-agnostic models. the timeline-sensitive model using lstm layers performed even worse, making predictions only for the no-change majority class. based on these overall trends, two models were selected to make predictions on the test set: the roberta baseline model and roberta-lin-lag. we engaged in small-scale focused parameter tuning using the development set, selecting the best dropout and learning rate for each model from among a limited set of items. for the roberta baseline model, tuning selected a hidden dropout rate of 0.2, a learning rate of 3e-5, and a minibatch size of 8. for the roberta-lin-lag model, tuning selected a hidden dropout rate of 0.2, a learning rate of 5e-6, and a minibatch size of 8. other parameters used the default values from roberta-base. ## results on the test set the two selected models were used to make predictions on the held-out test set. the results in these models were compared to baseline models from @xcite whose results were provided to participants in the shared task. these models are majority, where only the majority (o) class is selected, logreg, where a logistic regression model is trained on tf-idf features, and bert (f), a bert model trained on focal loss. compared to the baseline models, our models show mixed results. both of our models outperform the baselines on recall and f1 for the is class, with our r-lin-lag also outperforming all baselines on precision for the is class. for the ie class, however, they are beaten by the logistic regression model. our roberta model outperforms the baseline for precision on the o class, though not recall or f1. overall, our models have the best macro average f1 at the post level. for coveragebased metrics, our models again perform best for the is class, although the logistic regression baseline again outperforms our models for the ie class, as well as for the o class and macro average recall. our model with time lag features performs the best for macro-average precision. ## qualitative error analysis to better understand the types of posts that prove problematic for our models, we examine a small subset of the prediction errors produced on the development partition of the dataset. we specifically focus on times when our model produced a nochange (o) label while the gold label was is or ie, as well as the reverse. due to the sensitive nature of this data, we do not provide specific examples, but rather describe trends in the data. the following are situations in which our models tends to predict a change in mood but no change should be predicted: our models tend to predict ie or is labels whenever a post discusses unhealthy or dangerous scenarios, such as traumatic experiences, or when someone expresses desperation. however, as seen in items 1 to 3, this does not always provide accurate results. this type of error accounted for the majority of incorrect predictions in the sample of the development set examined. additionally, our models occasionally predict that a post does not show a change in mood when it is an example of a is or ie. in these cases, errors are typically due to: 4. largely neutral texts containing one strong indicator of distress. with these items, errors are typically caused by posts where there are both positive and negative elements, or where there is one very negative element that is limited to a minority of the post. additionally, in cases where there is no content in the post, our models always make a prediction of no change; however, there are cases where the post title alone reveals that an is or ie label is more appropriate. ## conclusion we examined the ability of timeline-agnostic and timeline-preserving transformer-based models to make predictions about changes in mood over time, finding that more complex models do not necessarily improve predictions. we furthermore experiment with a custom feature representing the length of time between one post and another, demonstrating that this may provide some support to more complex models. overall, we see that this remains a difficult task, suggesting that further improvements need to be made to methods of longitudinal mood modeling.