Sig-Networks Toolkit: Signature Networks for Longitudinal Language Modelling (2024)

Abstract: We present an open-source, pip installable toolkit, Sig-Networks, the first of its kind for longitudinal language modelling. A central focus is the incorporation of Signature-based Neural Network models, which have recently shown success in temporal tasks. We apply and extend published research providing a full suite of signature-based models. Their components can be used as PyTorch building blocks in future architectures. Sig-Networks enables task-agnostic dataset plug-in, seamless preprocessing for sequential data, parameter flexibility, automated tuning across a range of models. We examine signature networks under three different NLP tasks of varying temporal granularity: counselling conversations, rumour stance switch and mood changes in social media threads, showing SOTA performance in all three, and provide guidance for future tasks. We release the Toolkit as a PyTorch package with an introductory video, Git repositories for preprocessing and modelling including sample notebooks on the modeled NLP tasks.

## introduction existing work on temporal and longitudinal modelling has largely focused on models that are taskoriented, including tracking mood changes in users' linguistic content @xcite , temporal clinical document classification @xcite , suicidal ideation detection on social media @xcite , real-time rumour detection @xcite . transformer-based models struggle to outperform more traditional rnns in such tasks, highlighting their limitations in temporal settings @xcite . inspired by the success of models with short-and long-term processing capabilities @xcite in producing compressed temporal representations, we develop a toolkit that applies signature network models @xcite to various longitudinal tasks. path signatures are capable of efficient and compressed encoding of sequential data, sequential pooling in neural models, enhancement of short-term dependencies in linguistic timelines and encoding agnostic to task and time irregularities. we make the following contributions: • we release an open-source pip installable toolkit for longitudinal nlp tasks, sig-networks, including examples on several tasks to facilitate usability and reproducibility. longitudinal nlp modelling has been sporadically explored in tasks like semantic change detection @xcite @xcite @xcite or dynamic topic modelling @xcite @xcite . such approaches have limited generalisability as they track the evolution of specific topics over long-periods of time. social media data have given rise to longitudinal tasks such as mental health monitoring @xcite , stance detection and rumour verification @xcite @xcite requiring more fine-grained temporal modelling . other tasks, like healthcare patient notes @xcite and dialogue act classification @xcite are also longitudinal in nature. path signature @xcite is a collection of iterated integrals studied in the context of solving differential equations driven by irregular signals. it provides a summary of complex un-parameterised data streams through an infinite graded sequence of important statistics. thus, it produces a collection of statistics efficiently summarising important information about the path. signatures are deemed invaluable in machine learning @xcite as sequential feature transformers @xcite @xcite @xcite , or integrated components of neural models @xcite @xcite . however, they have only been sparsely explored within nlp @xcite @xcite , addressing only sequentiality or temporality. motivated by the wide range of longitudinal nlp tasks and the work by @xcite we present a toolkit for neural sequential path signatures models achieving sota performance in a range of such tasks. libraries for computing path signatures include roughpy, esig, iisignature @xcite , signatory @xcite and signax (see links in appendix e). currently, only signatory and signax offer differentiable computations of the signature and log-signature transforms on gpu (with pytorch @xcite and jax @xcite , respectively). while the above libraries only perform signature computations, with signatory additionally allowing for data stream augmentation through convolutional neural networks, the sig-networks library provides users with a complete pipeline for the application of signature-based (signature network) models in longitudinal nlp tasks. in particular, sig-networks is a pip installable pytorch library using signatory for differentiable computations of signature transforms on gpu, providing a range of off-the shelf models for task-agnostic longitudinal modeling. furthermore, the pip installable nlpsig library simplifies the data preprocessing for signature network models, by forming streams of embeddings to be directly fed into the models. ## conclusion we present the sig-networks toolkit, which allows for flexible modeling of longitudinal nlp classification tasks using signature-based network models @xcite , proposing improvements and variants. we test our system on three nlp classification tasks of different domains and temporal granularity and show sota performance against competitive baselines, while also shedding light into temporal characteristics which affect optimal model selection. the toolkit is made available as a pytorch package with examples, making it easy to plug-in new datasets for future model extensions. in the future we are planning to provide further flexibility in the toolkit, by enabling the integration of signature libraries beyond signatory for signature computations. this will facilitate its extension to deep learning frameworks beyond pytorch. additionally, we would like to allow for the selection of a non-linguistic feature subset to serve as part of the stream and of a different subset to be concatenated with the sbert representation, rather than having a single processing option for the full set of such features. we also aim to enrich the examples corpus of our repository with additional longitudinal nlp tasks and collaborate with independent contributors on the integration of newly developed signature-based models.