Capturing Changes in Mood Over Time in Longitudinal Data Using Ensemble Methodologies (2022)

Abstract: This paper presents the system description of team BLUE for Task A of the CLPsych 2022 Shared Task on identifying changes in mood and behaviour in longitudinal textual data. These moments of change are signals that can be used to screen and prevent suicide attempts. To detect these changes, we experimented with several text representation methods, such as TF-IDF, sentence embeddings, emotion-informed embeddings and several classical machine learning classifiers. We chose to submit three runs of ensemble systems based on maximum voting on the predictions from the best performing models. Of the nine participating teams in Task A, our team ranked second in the Precision-oriented Coverage-based Evaluation, with a score of 0.499. Our best system was an ensemble of Support Vector Machine, Logistic Regression, and Adaptive Boosting classifiers using emotion-informed embeddings as input representation that can model both the linguistic and emotional information found in users? posts.

## introduction the changes in mood and behaviour in the social media discourse of users are markers that can be used for screening and prevention of future suicide attempts. the emotional signals expressed in language and switches to suicide ideation are used for assessing the suicide risk of online users. however, identifying a person's mood changes over time based on their linguistic content from the posting activity on online social media platforms is a challenging task. challenges come from different perspectives, including methodological challenges of noisy natural language understanding @xcite , ethical implications of research and deployment @xcite @xcite and challenges associated with longitudinal data analysis. despite different challenges, the potential role of artificial intelligence (ai) based language technologies in mental health is gaining increasing attention @xcite . for example, some social media domains started implementing auto-detection tools to prevent suicide @xcite . in this paper, we present the methodology and the results of the machine learning models developed using the 2022 clpsych shared task dataset @xcite . we experiment with machine learning algorithms for the classification task using as input text representations based on statistical tf-idf, pre-trained glove embeddings @xcite and embeddings extracted from pretrained transformer models. after that, we develop a majority voting scheme over the predictions to report the final labels for a user timeline. our best strategy is based on majority voting of logistic regression (lr), support vector machine (svm) and adaptive boosting (adaboost) classifiers using as input the embeddings extracted from the pre-trained transformer models fine-tuned for emotion detection. our team blue ranked second in terms of precision-oriented coverage-based evaluation (macro-avg) metric with an overall score of 0.499, whereas the top score in this evaluation metric is 0.506. ## related work with the rise in social media use, more people started discussing their mental health problems and seeking support online. this allowed natural language processing and psychology researchers to use social media data to search for cues of mental illnesses. the frequently used social media platforms for studying these issues are twitter @xcite and reddit @xcite . for suicide detection, there are two methodologies for screening the online content: at the user level or post level. for user-level classification, the aim is to detect from the whole history of the user if they are at risk of suicide or if they show suicide ideation prior to the attempt, for an intervention to be made and for trying to save their life @xcite @xcite . post-level classification is performed by screening one post at a time, searching for posts that are indicative of a user being at risk of suicide @xcite @xcite . @xcite collect suicide-related tweets and annotate them as strongly concerning, possibly concerning or safe to ignore. afterwards, the authors train machine learning classifiers (svm, lr) to distinguish the concern level for these tweets containing suicide-related words. coppersmith et al. (2016) explore the language of twitter users prior to a suicide attempt to find quantifiable signals that can be used for screening and prevention. their article reveals that users have more posts expressing anger and sadness before trying to commit suicide. however, these emotions get to the same level as control users after the attempt. furthermore, people who attempt suicide have a higher proportion of emotional posts, increasing after the incident. in line with these findings, several works are modelling the emotional information found in the online discourse of users for classifying the suicide risk @xcite @xcite . regarding longitudinal approaches for suicide detection, de choudhury et al. (2016) extract markers of shifts to suicide ideation from users engaged in the online discourse revolving around mental illnesses, such as hopelessness, high self-attention focus, anxiety, impulsiveness and others. using these markers, the authors can predict which individuals are more prone to express suicide ideation in future posts. through a time-aware approach, @xcite propose a framework that uses people's historical and emotional spectrum when assessing the risk of a specific post. @xcite propose to take the temporal information into account by identifying the changes in people's behaviour and mood on social media. the changes considered are switches (sudden mood changes) and escalation (gradual mood progression). these changes in mood or emotion found in the online discourse can be used for assessing the suicide risk of users. although the potential role of language technology in mental health using information from social media datasets is gaining increasing attention, continued progress on nlp for mental health is hampered by obstacles to shared, communitylevel access to relevant data. the 2021 clpsych shared task was introduced to address this problem by conducting a shared task using sensitive data in a secure environment @xcite and continued in the 2022 clpsych shared task @xcite . the goal of the tasks from the previous year was to assess the suicide risk of a user from posts 30 days or 6 months prior to a suicide attempt. the best-performing models used approaches such as weighted ensemble of different machine learning classifiers (lr, naive bayes classifiers, linear svm) @xcite , lstm architecture with topic modelling and dictionary-based features @xcite and bayesian modelling of features from linguistic inquiry and word count (liwc) @xcite , behavioural information or other features derived from already available or custom dictionaries @xcite . ## data and task a we participate in task a in the 2022 clpsych shared task, intending to capture the mood changes of individuals in a given time window based on their reddit posts. the dataset for this task was collected in @xcite . the posts from reddit's mental health-related subreddits in a given time window (timeline) @xcite @xcite were annotated by four annotators on the basis of three labels hinting at moments of change @xcite : none (o), escalation (ie), and switch (is). a total of 256 timelines and 6,205 posts are available for task a. thus, given a user's timeline, the aim is to classify each post as either a 'switch' (is), or an 'escalation' (ie) or 'none' (o). three metrics are used for evaluating the performance of the models in task a @xcite . post-level evaluation calculates the traditional precision, recall, and f1 scores per post and class, with the macro-average to get the final score. apart from the traditional post-level metric, timeline-based scores are also used for the evaluation, given the sequential nature of task a. in the window-based evaluation, precision and recall scores are calculated based on whether correct labels are in a certain time window. in the coverage-based evaluation, precision and recall scores are calculated based on the models' ability to capture regions of change. ## results and discussion at the time of writing the paper, we do not have access to the test data ground truth labels. therefore, we present the performance of our three ensemble systems on the validation data and the official results from the task organisers on the test data. in addition, we perform an error analysis by exploring in more detail at the predictions of the models on the validation data. ## conclusion in this paper, we presented the system description and results of team blue for the task of identifying moments of change from the clpsych 2022 shared task. we experimented with several text representation methods, such as tf-idf, sentence embeddings (from pre-trained transformer models, glove) and emotion-informed embeddings (extracted from the pre-trained transformer models fine-tuned for emotion detection). to identify the mood changes, we trained several classical machine learning classifiers. we chose to submit three ensemble systems based on maximum voting on the best performing models (svm, lr, ad-aboost) with different inputs. of the nine participating teams in task a, our team ranked second in the precision-oriented coverage-based evaluation, with a score of 0.499 (the top team had a score of 0.506). our best run was an ensemble method of svm, lr, and adaboost classifiers using as input emotion-informed embeddings that can model both the linguistic and emotional information found in users' posts. due to the enclave data system's technical difficulties, we have developed systems in 11 not actual examples from the dataset, but equivalent sentences in order to maintain anonymity three working days after getting the data in our local system. for future work, we plan to investigate the dataset in detail and develop improved models for identifying mood changes in longitudinal textual data and assess the suicide risk of social media users.