Large Language Models with Temporal Reasoning for Longitudinal Clinical Summarization and Prediction (2025)

Abstract: Recent advances in large language models (LLMs) have shown potential in clinical text summarization, but their ability to handle long patient trajectories with multi-modal data spread across time remains underexplored. This study systematically evaluates several state-of-the-art open-source LLMs, their Retrieval Augmented Generation (RAG) variants and chain-of-thought (CoT) prompting on long-context clinical summarization and prediction. We examine their ability to synthesize structured and unstructured Electronic Health Records (EHR) data while reasoning over temporal coherence, by re-engineering existing tasks, including discharge summarization and diagnosis prediction from two publicly available EHR datasets. Our results indicate that long context windows improve input integration but do not consistently enhance clinical reasoning, and LLMs are still struggling with temporal progression and rare disease prediction. While RAG shows improvements in hallucination in some cases, it does not fully address these limitations. Our work fills the gap in long clinical text summarization, establishing a foundation for evaluating LLMs with multi-modal data and temporal reasoning.

## introduction electronic health records (ehrs) encapsulate a wide range of multi-modal data, such as vital signs, laboratory results, radiology findings, and free text clinical notes @xcite @xcite . organized across various timestamps, they reflect the dynamic nature of patient care. in clinical settings, particularly for older patients in intensive care units (icus) with multiple encounters, chronic conditions, and complex treatment plans, ehrs become especially lengthy and 1 code is available at: https://github. com/lark-nlp-lab/longitudinal_clinical_ summarization . intricate. clinicians reviewing these long patient histories face significant cognitive burden, often leading to inaccurate decisions such as diagnostic errors @xcite . automated summarization can help improve care continuity and decision-making @xcite @xcite @xcite , but the volume and complexity of ehrs pose challenges, requiring models that handle diverse and evolving clinical data. as shown in figure 1 , summarizing longitudinal patient trajectories requires not only extracting temporally ordered events from multi-modal data, but also reasoning over how earlier findings trigger downstream actions. for instance, abnormal blood pressure and elevated white cell count raise suspicion of inflammation, prompting a chest ct; the ct then confirms an aortic dissection, which leads to intubation and ventilatory support, ultimately consolidated in the physician's diagnostic note. capturing these event-to-action links is what we mean by temporal reasoning, and it is critical for preserving clinical causality in the patient trajectory. our work evaluates large language models (llms) for summarizing complex, longitudinal ehrs, capturing full patient trajectories rather than isolated clinical snapshots. while llms have shown promise in clinical nlp tasks @xcite @xcite @xcite , existing evaluations primarily focus on short-context settings and task-specific finetuning. clinical summarization benchmarks, such as discharge summarization ("discharge me" @xcite ) and diagnosis generation from progress notes ("probsum" @xcite ), evaluate llms on isolated segments of patient records rather than full hospital trajectories. while retrieval-augmented generation (rag) has been applied to clinical tasks like diagnosis prediction and discharge summarization @xcite @xcite @xcite , its effectiveness for long, temporally rich clinical narratives remains unclear. this work addresses two key gaps: (1) current evaluations rarely test long-context llms in zeroshot settings, leaving their inherent capabilities and limitations in complex medical reasoning unknown; and (2) most clinical benchmarks focus on single time-point summaries, overlooking the demands of longitudinal patient care. we systematically evaluate llms on full patient trajectories to assess their ability to process temporally evolving, multi-modal clinical data. in this paper, we focus on two public ehr datasets: medical information mart for intensive care (mimic-iii) @xcite and ehrshot @xcite . we evaluate five state-of-the-art llms and their rag variants: mistral-7b-instruct-v0.1 @xcite , llama3-8b-instruct (ai@meta, 2024), qwen2.5-7b @xcite , deepseek-r1-distill-qwen-32b @xcite and llama2-13b-chat-hf @xcite . our work advances clinical llm summarization through the following contributions: • we reformulate discharge summarization, progress note summarization, and diagnosis classification into new long-context tasks requiring temporal reasoning, aligned with real clinical workflows. in the absence of benchmarks combining multimodal inputs, temporal structure, and clinical workflow alignment, our study provides a first step toward evaluating llms in real-world longitudinal summarization. the results highlight key limitations and suggest directions for more temporally grounded and clinically usable models. ## related work clinical text summarization existing work includes discharge summarization @xcite , diagnosis summarization @xcite @xcite , hospital course summarization @xcite . while these tasks inherently involve multidocument summarization by human physicians, nlp formulations typically treat them as singledocument summarization or multi-document summarization with fixed timestamps (e.g. at the time the patient is discharged). tabular reasoning llms face challenges when reasoning over structured tabular inputs, where tasks often require precise comparisons or aggregations. zhang et al. (2024) demonstrated the potential of llms for table manipulation in real usage scenarios, while ashury-tahan et al. (2025) introduced a benchmark showing that models remain sensitive to table formatting. these works highlight open problems in extending llm reasoning to structured clinical data. ## dataset and tasks we use two complementary datasets, mimic-iii and ehrshot, to evaluate llms' clinical reasoning. they differ in three key aspects: 1) modalities: mimic-iii includes both structured data and clinical notes, while ehrshot contains only structured data. 2) output type: mimic-iii tasks focus on generating short summaries of patient progress or discharge status, whereas ehrshot involves classification-based diagnosis prediction. 3) decision time span: mimic-iii targets immediate icu-related summaries, while ehrshot predicts diagnoses within a year of discharge, emphasizing long-term forecasting based on prior visits. despite these differences, both datasets address longitudinal patient information, requiring models to reason over time and synthesize complex clinical histories. ## methods we evaluate five llms with varying context capacities. llama2-13b and mistral-7b support up to 4k tokens, while llama3-8b allows for 8k. to test long-context capabilities, we include qwen2.5-7b and deepseek-r1 (32b), both of which support input lengths up to 128k tokens. while we did explore biomedical llms (pmc-llama, biogpt, biomistral), their performance was extremely poor, so we did not include them in this paper. ## results the results are organized by task: discharge summarization and assessment and plan generation figure 5 shows the effects of the four modalities (notes + tabular data, notes only, tabular only and shuffled tabular) and two context window sizes (24 and 48 hours) on performance. for the 24 hour window, the pure note modality dominates, performing best on mistral, qwen, deepseek and llama2. llama3 is the outlier and reports the best performance on the combined and shuffled tabular modalities. in the 48h window, trends are less clear, but most models still favor notes. temporal order appears unhelpful, as shuffled tabular data performs slightly better than chronological tabular data, likely because the data is near the discharge state, where patients' clinical conditions stabilize, leading to fewer changes over time. a& p generation. this task specifically requires clinical reasoning over past captured data (evidence) to summarize patient progression and plan for treatment, yet the relatively low rouge and bertscore values indicate that models still struggle with temporal reasoning and integrating historical context effectively. ## discussion another critical aspect to consider is running time and memory usage. all 7b llms require approximately 15gb of gpu ram for direct generation. rag demands significantly more memory, peaking at 32-33gb, nearly twice that of direct generation. for qwen, it takes 18-20 minutes to run rag on one progress note with prior context input, while direct generation takes 10 minutes on average. on ehrshot where input is shorter, diagnosis predic- ## conclusion we evaluated llms on long-context clinical summarization and prediction using two public ehr datasets. current models struggle with accurate summarization and temporal reasoning, making it hard to interpret medical event sequences. while rag offers some improvement, overall performance remains inadequate, highlighting the need for further progress.