Combining Hierachical VAE s with LLM s for clinically meaningful timeline summarisation in social media (2024)

Abstract: We introduce a hybrid abstractive summarisation approach combining hierarchical VAEs with LLMs to produce clinically meaningful summaries from social media user timelines, appropriate for mental health monitoring. The summaries combine two different narrative points of view: (a) clinical insights in third person, generated by feeding into an LLM clinical expert-guided prompts, and importantly, (b) a temporally sensitive abstractive summary of the user’s timeline in first person, generated by a novel hierarchical variational autoencoder, TH-VAE. We assess the generated summaries via automatic evaluation against expert summaries and via human evaluation with clinical experts, showing that timeline summarisation by TH-VAE results in more factual and logically coherent summaries rich in clinical utility and superior to LLM-only approaches in capturing changes over time.

## introduction social media users discuss different aspects of their lives, providing important clues about their mental health. previous work @xcite @xcite has studied users' social media posts to help identify depression, bipolar disorder @xcite or self-harm @xcite , with work on multi-task learning to capture user states at a particular moment in time @xcite . despite the importance of longitudinal assessments of linguistic and other digital content for mental health clinical outcomes @xcite , there is little work on considering the evolution of an individual's mental health over time through their social media. @xcite established the task of capturing changes (switches and escalations) in an individual's mood over time and showed how identifying * equal contribution. to the best of our knowledge we are the first to propose clinically meaningful summaries of social media user 'timelines' (sequences of chronologically ordered posts by a user). driven by the need to concisely summarise time-series language data which can span arbitrary lengths that exceed limits of many contemporary models and render purely extractive methods impractical, we propose a novel hybrid unsupervised abstractive method, timeline hierarchical vae (th-vae). our system makes use of a hierarchical variational autoencoder that compresses timeline information into compact representations and a large language model (llm), creating a two-layer summary that combines two different narrative points of view. specifically: (a) a high-level summary in third person useful for a clinician, generated by feeding into an llm expertguided prompts, and importantly (b) a temporally sensitive abstractive summary of the user's timeline in first person (evidence summary), generated by th-vae. the generation of the first person abstrac-tive evidence summary via th-vae is guided by mental health related key-phrases obtained through instruction prompting by an llm. the final resulting high level summary covers aspects considered to be crucial by clinicians from a wide range of therapeutic approaches, including individuals' diagnosis, intrapersonal and interpersonal patterns and extent of mental state changes over time @xcite . 1 we make the following contributions: our goal in user timeline summarisation is to capture important information and synthesise it. summaries in mental health. although summaries are clinically crucial for compiling information about individuals, there is limited literature on the subject, with the primary focus being on expertgenerated case study summarization (eells, 2022). only recently, researchers have started to use nlp capabilities to automatically generate summaries in the clinical domain. manas et al. (2021) demonstrated the usefulness of generating summarised diagnoses from a single-session interview. srivastava et al. (2022) summarised psychotherapy conversations at the level of single counseling sessions proposing that summaries should exploit domain knowledge and conversational elements. on social media, sotudeh et al. (2022) generated summaries of individual reddit posts, relying on formatting conventions (i.e. tldr) to extract short summaries provided by the users themselves without further content constraints. yang et al. (2023) instructiontuned llms to generate mental health analyses from static social media text. by contrast our work summarises user timelines and combines informa-tion from social media posts based on high-level expert domain knowledge, important for evaluating individuals' progression over time. summarising with llms. current work on llmbased summarisation focuses on news articles or instructional texts @xcite @xcite , using simple prompts (e.g. "summarize the following article:"). @xcite took a multi-step approach, extracting event information from news via curated guiding questions then summarising the prompted outputs. in our work, we summarise longitudinal user generated content and use clinically-informed prompts to generate high-level mental health observations. summary evaluation. existing mental health summarisation works utilised natural language generation metrics, for example rouge @xcite being used to measure n-gram overlap against reference documents @xcite @xcite . @xcite additionally applied bleurt @xcite , a learned metric trained on ratings, questeval @xcite , a metric based on question generation and answering, and mhic, a metric based on rouge that they defined to assess information captured in counselling summaries . contrary to prior work, our task involves twolayer mental health summaries combining firstperson social media content with high-level clinical concepts in third person, posing unique evaluation challenges. for example, data noisiness makes metrics learned on well-formed texts unsuitable, and evaluation must assess consistency both between summary layers and within the detailed high-level summary itself. to this end, we extend the line of work leveraging natural language inference (nli) in summary factuality and consistency evaluation @xcite . ## methodology task given a user's timeline (a series of posts between two dates @xcite ), the goal is to generate an abstractive summary that reflects the user's mental state and how it changes over time. this summary includes high-level information useful for clinicians in third person, and corresponding evidence from the timeline in first person. ## conclusions we present a novel method for hybrid abstractive summarisation using hierarchical vae and llms and the first approach to creating clinically meaningful mental health summaries from users' social media timelines. our approach results in summaries with a dual narrative perspective: high-level third person information useful for clinicians is combined with first person corresponding evidence from users' timelines. abstractive timeline summarisation is performed by three different systems (llm-, th-vae-and skeleton-based) whose generation is guided by key-phrases obtained by an llm through instruction prompting. high-level clinical summaries in third-person are generated by feeding the timeline summaries from all three systems into an llm. our proposed timeline summariser, th-vae, based on a hierarchical vae for long texts, can capture long dependencies between sub-timelines and while llm timeline summaries are the most fluent, they lag behind th-vae on logical coherence and factuality. from a clinical psychology viewpoint our work enables clinician access to consented clients' social media data allowing them to understand changes in their mental state over time. importantly it enables generation of automated summaries emphasizing essential clinical concepts which can aid mental health professionals to quickly grasp an individual's psychological condition and progression.