Identifying Moments of Change from Longitudinal User Text (2022)

Abstract: Identifying changes in individuals’ behaviour and mood, as observed via content shared on online platforms, is increasingly gaining importance. Most research to-date on this topic focuses on either: (a) identifying individuals at risk or with a certain mental health condition given a batch of posts or (b) providing equivalent labels at the post level. A disadvantage of such work is the lack of a strong temporal component and the inability to make longitudinal assessments following an individual’s trajectory and allowing timely interventions. Here we define a new task, that of identifying moments of change in individuals on the basis of their shared content online. The changes we consider are sudden shifts in mood (switches) or gradual mood progression (escalations). We have created detailed guidelines for capturing moments of change and a corpus of 500 manually annotated user timelines (18.7K posts). We have developed a variety of baseline models drawing inspiration from related tasks and show that the best performance is obtained through context aware sequential modelling. We also introduce new metrics for capturing rare events in temporal windows.

## introduction linguistic and other content from social media data has been used in a number of different studies to obtain biomarkers for mental health. this is gaining importance given the global increase in mental health disorders, the limited access to support services and the prioritisation of mental health as an area by the world health @xcite . studies using linguistic data for mental health focus on recognising specific conditions related to mental health (e.g., depression, bipolar disorder) (husseini @xcite , or identifying self-harm ideation in user posts @xcite . however, none of these works, even when incorporating a notion of time @xcite figure 1 : example of an escalation (with a darker "peak") and a switch within a user's timeline. @xcite , identify how an individual's mental health changes over time. yet being able to make assessments on a longitudinal level from linguistic and other digital content is important for clinical outcomes, and especially in mental health @xcite . the ability to detect changes in individual's mental health over time is also important in enabling platform moderators to prioritise interventions for vulnerable individuals @xcite . users who currently engage with platforms and apps for mental health support (neary and schueller, 2018) would also benefit from being able to monitor their well-being in a longitudinal manner. motivated by the lack of longitudinal approaches we introduce the task of identifying 'moments of change' (moc) from individuals' shared online content. we focus in particular on two types of changes: switches -mood shifts from positive to negative, or vice versa -and escalations -gradual mood progression (see fig. 1 , detailed in § 3). specifically we make the following contributions: we have provided thorough annotation guidelines for moments of change that can aid mental health monitoring over time irrespective of the underlying condition. ## dataset creation we describe the creation of a dataset of individuals' timelines annotated with moments of change. a user's timeline p ## models & experiment design our aim is to detect and characterise the types of moc based on a user's posting activity. we therefore treat this problem as a supervised classification task (both at post level and in a sequential/timelinesensitive manner, as presented in §4.2) rather than an unsupervised task, even though we also consider effectively baselines with unsupervised components (fsd, scd in §4.2). contrary to traditional sentence or document-level nlp tasks, we incorporate timeline-sensitive evaluation metrics that account for the sequential nature of our model predictions ( §4.1). given a user's timeline, the aim is to classify each post within it as belonging to a "switch" (is), an "escalation" (ie), or "none" (o). at this point we don't distinguish between beginnings of switches/peaks of escalations and other posts in the respective ranges. while the task is sequential by definition, we train models operating both at the post level in isolation and sequential models at the timeline-level (i.e., accounting for user's posts over time), as detailed in §4.2. we contrast model performance using common post-level classification metrics as well as novel timeline-level evaluation approaches ( §4.1). this allows us to investigate the impact of (a) accounting for severe class imbalance and (b) longitudinal modelling. we have randomly divided the annotated dataset into 5 folds (each containing posts from 100 timelines) to allow reporting results on all of the data through cross-validation. ## ethics statement ethics institutional review board (irb) approval was obtained from the corresponding ethics board of the university of warwick prior to engaging in this research study. our work involves ethical considerations around the analysis of user generated content shared on a peer support network (talk-life). a license was obtained to work with the user data from talklife and a project proposal was submitted to them in order to embark on the project. the current paper focuses on the identification of moments of change (moc) on the basis of content shared by individuals. these changes involve recognising sudden shifts in mood (switches or es-calations). annotators were given contracts and paid fairly in line with university payscales. they were alerted about potentially encountering disturbing content and were advised to take breaks. the annotations are used to train and evaluate natural language processing models for recognising moments of change as described in our detailed guidelines. working with datasets such as talklife and data on online platforms where individuals disclose personal information involves ethical considerations @xcite . such considerations include careful analysis and data sharing policies to protect sensitive personal information. the data has been de-identified both at the time of sharing by talklife but also by the research team to make sure that no user handles and names are visible. any examples used in the paper are either paraphrased or artificial. potential risks from the application of our work in being able to identify moments of change in individuals' timelines are akin to those in earlier work on personal event identification from social media and the detection of suicidal ideation. potential mitigation strategies include restricting access to the code base and annotation labels used for evaluation. limitations our work in this paper considers moments of change as changes in an individual's mood judged on the basis of their self-disclosure of their well-being. this is faced by two limiting factors: (a) users may not be self-disclosing important aspects of their daily lives and (b) other types of changes related to their mental health (other than their mood/emotions, such as important life events, symptoms etc.) may be taking place. though our models could be tested in cases of nonself-disclosure (given the appropriate ground truth labels), the analysis and results presented in this work should not be used to infer any conclusion on such cases. the same also holds for other types of 'moments of change' mentioned in §2 (e.g., transition to suicidal thoughts), as well as other types of changes, such as changes in an individual in terms of discussing more about the future, studied in @xcite , or changes in their self-focus @xcite over time, which we do not examine in this current work.