Creation and evaluation of timelines for longitudinal user posts (2023)

Abstract: There is increasing interest to work with user generated content in social media, especially textual posts over time. Currently there is no consistent way of segmenting user posts into timelines in a meaningful way that improves the quality and cost of manual annotation. Here we propose a set of methods for segmenting longitudinal user posts into timelines likely to contain interesting moments of change in a user’s behaviour, based on their online posting activity. We also propose a novel framework for evaluating timelines and show its applicability in the context of two different social media datasets. Finally, we present a discussion of the linguistic content of highly ranked timelines.

## introduction an increasing body of work considers time-aware models trained on social media data for a number of different tasks, including personal event identification @xcite @xcite , suicidal ideation and suicide risk detection @xcite @xcite @xcite . for such tasks deriving meaningful timelines (i.e. sequences of posts by individuals), containing examples of the phenomenon under study from large-scale collections, together with associated annotations, is crucial. this is especially important for computational approaches in mental health (mh) given the surging numbers of those seeking help online @xcite . earlier work on personal life event detection considered selecting salient timelines through topic modelling @xcite or through a non-parametric generative approach @xcite . however, such approaches are unsuitable for identifying changes in mood or mh more generally. specifically, since timelines are selected based on linguistic content this introduces a sampling bias for downstream linguistic analysis and annotation @xcite . in recent work on suicidal ideation detection, timelines are chosen as the n most recent posts @xcite , which are not necessarily the most salient for annotation. ## related work since we aim to segment users' entire posting history into smaller sequences, manageable to annotate and salient in terms of containing moments of change in mental health, we consider work in the following areas: mental health monitoring (2.1); text segmentation (2.2); timeline summarization (2.3); change-point detection (2.4). ## evaluation of selected timelines while there is previous work in evaluating segments of posts in text segmentation ( §2.2) and timeline summarization ( §2.3), there is little to no prior work on frameworks for evaluating timeline selection methods for the purposes of efficiently annotating longitudinal datasets. as such we identify this as a nascent area of study -ripe for others to build upon, and propose a novel evaluation framework for selecting timelines for this task. we investigate several metrics for evaluating the methods from §3 in terms of their ability to select timelines that correspond to a high proportion of ground-truth moments of change (gtmoc), denoted hitherto as g. each cmoc generated by a method as a change point is denoted hitherto as c. since we do not have access to manual ground truth annotations outside of the span of our annotated timelines, we can only evaluate methods according to cmocs that fall within them. ## experiments we evaluate our timeline selection methods ( §3), using our evaluation framework ( §4) based on ground-truth human annotated data. ## conclusions & future work we have introduced methods and an evaluation framework for identifying timelines from users' social media posts, likely to contain a large amount of moments of change (moc). we use changes in posting behaviour as a proxy for changes in mood, to efficiently identify longitudinal user content worth annotating. our methods have been manually evaluated against ground truth mocs (gt-mocs) in two different datasets. bayesian online change point dection (bocpd) shows promise in detecting timelines rich in gtmocs. future work can explore the incorporation of textual content in the bocpd poisson-gamma model for the distinction between different types of gt-moc. we find that resulting timelines dense in gtmocs are characterised by a high deviation in sentiment from one post to the next, suggesting that such deviations may be a useful feature for distinguishing between different types of gtmoc. we expect that the methods proposed in our work will benefit researchers interested in creating longitudinally annotated textual datasets of user posts, particularly when annotating moments of change.