Sequential Path Signature Networks for Personalised Longitudinal Language Modeling (2023)

Abstract: Longitudinal user modeling can provide a strong signal for various downstream tasks. Despite the rapid progress in representation learning, dynamic aspects of modelling individuals’ language have only been sparsely addressed. We present a novel extension of neural sequential models using the notion of path signatures from rough path theory, which constitute graduated summaries of continuous paths and have the ability to capture non-linearities in trajectories. By combining path signatures of users’ history with contextual neural representations and recursive neural networks we can produce compact time-sensitive user representations. Given the magnitude of mental health conditions with symptoms manifesting in language, we show the applicability of our approach on the task of identifying changes in individuals’ mood by analysing their online textual content. By directly integrating signature transforms of users’ history in the model architecture we jointly address the two most important aspects of the task, namely sequentiality and temporality. Our approach achieves state-of-the-art performance on macro-average F1 score on the two available datasets for the task, outperforming or performing on-par with state-of-the-art models utilising only historical posts and even outperforming prior models which also have access to future posts of users.

## introduction representation learning has become a critical tool in natural language processing (nlp) applications, especially for user-specific tasks @xcite . despite its importance there is limited work on low-dimensional static user representations @xcite @xcite or more importantly on dynamic user representations @xcite @xcite . dynamically representing users through their textual data can be of paramount importance especially for addressing user-specific changes in their language over time, potentially indicative of underlying mental health conditions. current research on temporal user representations for mental health applications @xcite @xcite highlights the importance of sequential modeling but either relies heavily on emotion and network based features (which limit the generalisability of the representations) or models a user's entire available content as a whole, limiting its use to off-line rather than real-time applications. to address these, we propose an architecture that combines sequential modelling with path signatures @xcite . path signatures provide a pathwise definition to the solution of differential equations driven by rough signals and therefore a non-parametric way for sequential encoding. they are graduated summaries of continuous paths and have the ability to capture nonlinearities in trajectories. they have been proven effective in compressing sequential/temporal content @xcite for a range of applications including chinese character recognition @xcite , medical information extraction @xcite and emotion recognition through audio streams @xcite . we combine signature paths with contextual representations from a pre-trained bert @xcite and recurrent neural networks to obtain a novel sequential, temporally sensitive architecture. we apply this to a longitudinal task in mental health, that of identifying moments of change (moc) in individuals' mood @xcite . we make the following contributions: temporal representations. recent work has focused on expanding static representations in order to construct temporal user embeddings through user activity data @xcite @xcite . despite the importance of longitudinal online linguistic content, little work addresses dynamic temporally-sensitive user representations. for the task of semantic change detection, temporally sensitive word representations are obtained either over discrete time bins @xcite or jointly over time @xcite @xcite . such work addresses the change in words over long periods rather than changes in users, which may cover much shorter spans. @xcite tackled the problem of temporal user representations through the extension of dynamic word representations @xcite , through joint word and user temporal modeling in a probabilistic fashion, adopting a skip-gram model. this work precedes the advent of powerful pretrained language models (plms). dynamic topic models have been employed in social media for modeling the evolution of emotions and topics in subject-specific reviews and news corpora @xcite . although such work forms a strong foundation for dynamic representation modeling, temporal individual linguistic content spans across multiple unique topics unlike reviews and news documents that are heavily governed by aggregate topics. additionally, in longitudinal user modeling individuals' mood changes occur uniquely and at different speeds, rather than presenting a mass change of sentiment in topic-specific documents. lastly, since work on dynamic topic-emotion models precedes the plm era, there is need to further explore the effect of contextual word representations in capturing the dynamics of words governed by the post topics. ## conclusion and future work we present a novel sequential model architecture combining rnns with path signatures, applicable to longitudinal tasks which consider timelines of social media posts. our model achieves effective compression of a user's history through both signature transforms and sequential modeling via a bilstm. it does so through encoding the local progression of textual information in history through signatures in an integrated, robust and computationally efficient way. the use of signatures within our network allows for the incorporation of nonparametric higher order information in a learnable way and combines this benefit with the sequential modeling of local and long-term information through lstms. we evaluate our model on personalised longitudinal language modelling, on the task of identifying changes in a user's mood. our model well outperforms or performs on-par with all baselines for this task operating on historical data, for both of the two existing datasets, from the talk-life and reddit platforms. in the future we plan to investigate direct injection of signature transforms into transformer networks for time-sensitive modelling as well as explore other time-sensitive nlp tasks, such as rumour verification using social media threads @xcite .