Exciting Mood Changes: A Time-aware Hierarchical Transformer for Change Detection Modelling (2024)

Abstract: Through the rise of social media platforms, longitudinal language modelling has received much attention over the latest years, especially in downstream tasks such as mental health monitoring of individuals where modelling linguistic content in a temporal fashion is crucial. A key limitation in existing work is how to effectively model temporal sequences within Transformer-based language models. In this work we address this challenge by introducing a novel approach for predicting ‘Moments of Change’ (MoC) in the mood of online users, by simultaneously considering user linguistic and time-aware context. A Hawkes process-inspired transformation layer is applied over the proposed architecture to model the influence of time on users’ posts – capturing both their immediate and historical dynamics. We perform experiments on the two existing datasets for the MoC task and showcase clear performance gains when leveraging the proposed layer. Our ablation study reveals the importance of considering temporal dynamics in detecting subtle and rare mood changes. Our results indicate that considering linguistic and temporal information in a hierarchical manner provide valuable insights into the temporal dynamics of modelling user generated content over time, with applications in mental health monitoring.

## introduction since the advent of the transformer model @xcite , much of the work in natural language processing (nlp) has focused on making improvements to attention mechanisms or leveraging different sub-modules of the transformer architecture among others, bringing significant gains in performance to multiple nlp tasks. however, less attention has been paid to the importance of longitudinal modelling of text, which is crucial for a wide range of downstream tasks such as those within the healthcare domain. work at the intersection of nlp and mental health has been focusing increasingly on temporally sensitive tasks, such as that of predicting changes in a mood ('moments of change' -'moc') of an online social media user on the basis of self disclosure @xcite . while transformer-based architectures have shown great potential for non-temporally sensitive tasks , the longitudinal modelling aspect of the majority of state-of-the-art on temporally sensitive tasks is based on rnn-based models @xcite @xcite . this has the drawback of (i) not utilising state-of-the-art (sota) models in nlp and (b) not studying the effect of the timing of the occurring events (e.g., social media posts) with respect to the task at hand @xcite . aiming at tackling the aforementioned challenges, this paper introduces a novel time-aware hierarchical transformer, to predict moc in online user posts. our model simultaneously analyzes linguistic patterns in textual content, via bert @xcite as a fine-tunable component, and integrates the temporal context of posts via a time-sensitive decay and self-excitation mechanism based on the hawkes process @xcite . our approach operates on sequences of temporally ordered user posts ('timelines'), recognizing that moments of emotional change show cascading effects, forming clusters of localized mood-changes due to self-excitation effects -that are crucial to understanding the trajectory and possible future of a user's emotional state. our approach is motivated by the two following guiding hypotheses: mental health and social media. early work from coppersmith et al. (2014) involved predicting mental health conditions from twitter posts at the user level. more recently, social media data has been used to aid the assessment of depression @xcite , suicidal ideation @xcite @xcite and anxiety @xcite , while shared tasks such as clpsych @xcite and clef erisk @xcite , have paved an avenue for the community to contribute towards the identification of a range of mental health conditions on social media. ## task definition identifying moments of change @xcite refers to the longitudinal task of detecting posts within a user's posting history which indicate that the user's mood has been changed compared to their recent past (on the basis of selfdisclosure) in one of the following two ways: (a) 'switch' (the post(s) indicate that the user's mood has switched from neutral/positive to negative, or from neutral/negative to positive); (b) 'escalation' (the post(s) indicate that the user's mood has escalated from negative to very negative, or from positive to very positive). switches (a) and escalations (b) are rare but important events as shown in existing annotated datasets @xcite -i.e., the user's mood stays constant in the vast majority of their posts -and as such the moc identification task is a challenging case of mental health monitoring, as indicated by sota results @xcite . the complexity of this task, and other longitudinal tasks in nlp, arises from the subtlety of linguistic cues and the importance of considering temporal context in predicting changes, which is often neglected. ## methodology our work aims to address the challenge of integrating temporal dynamics with textual content, an approach critical across many nlp tasks. here, we propose a novel hierarchical transformer architecture inspired by the hawkes process to simultaneously model linguistic and temporal contexts, specifically in social media posts. this approach allows us to longitudinally capture nuanced dynamics in emotional changes over time, a key factor in mental health monitoring and other related fields. in this section, we introduce our model ( §4.1), a time-aware hierarchical transformer (figure 1 ), inspired by the hawkes process, for modelling textual ( §4.1.2) and temporal ( §4.1.3) context in segmented timelines ( §4.1.1) of social media posts to predict mood changes of online users. ## results we present our main results in ## discussion we investigate the performance of each ablated model based on their precision (p), recall (r) and f1 scores for the rare moments of change classes "switch" (s), "escalation" (e), and "no change" (0), as well as their macro-average scores across all classes. ## conclusion from our ablation study, we have demonstrated the importance of our hawkes formulation, particularly the ability to capture event excitation and time-decay -to enhance our models to detect complex changes in mood. we have seen horobert consistently outperform other models in this study, across both datasets, illustrating the effectiveness of modelling changes in mood using a time-sensitive hierarchical transformer with an lstm component. our ablation study has helped validate our design choices and modifications made in our proposed model, and also help reveal important areas for further refinements in future work, by pinpointing the contribution of different model components in discerning the rare classes "switches" and "escalations".