P ro M ed TS : A Self-Supervised, Prompt-Guided Multimodal Approach for Integrating Medical Text and Time Series (2025)

Abstract: Large language models (LLMs) have shown remarkable performance in vision-language tasks, but their application in the medical field remains underexplored, particularly for integrating structured time series data with unstructured clinical notes. In clinical practice, dynamic time series data, such as lab test results, capture critical temporal patterns, while clinical notes provide rich semantic context. Merging these modalities is challenging due to the inherent differences between continuous signals and discrete text. To bridge this gap, we introduce ProMedTS, a novel self-supervised multimodal framework that employs prompt-guided learning to unify these heterogeneous data types. Our approach leverages lightweight anomaly detection to generate anomaly captions that serve as prompts, guiding the encoding of raw time series data into informative prompt embeddings. These prompt embeddings are aligned with textual representations in a shared latent space, preserving fine-grained temporal nuances alongside semantic insights. Furthermore, our framework incorporates tailored self-supervised objectives to enhance both intra- and inter-modal alignment. We evaluate ProMedTS on disease diagnosis tasks using real-world datasets, and the results demonstrate that our method consistently outperforms state-of-the-art approaches.

## introduction recent advancements in natural language processing (nlp) have revolutionized healthcare by enabling deeper insights into electronic health records (ehrs). ehrs combine structured data, such as time series laboratory (lab) test results, with unstructured data, including clinical notes and medical images. while large language models (llms) excel at processing unstructured text @xcite and vision transformers have driven progress in medical image analysis @xcite , integrating time series data with text remains a challenge. unlike text, which is composed of discrete tokens, time series data contains continuous signals with temporal dependencies as illustrated in figure 1(a) . current multimodal learning approaches, especially contrastive learning methods @xcite , have been effective in aligning vision data and text data. however, they are less suited to bridge the gap between time series data and text data. time series data require fine-grained temporal representations in a high-dimensional space and are often irregularly sampled, exhibit diverse frequencies, and include missing values @xcite . in addition, the lack of large-scale paired datasets that link raw time series with textual descriptions further hampers llms from incorporating structured information into clinical decision-making @xcite . without an effective fusion mechanism, llms cannot fully exploit the rich temporal patterns in structured ehr data. to address these challenges, we propose promedts, a self-supervised and prompt-guided framework designed to unify medical notes and time series lab tests for natural comprehension by llms. as illustrated in figure 1 (b), rather than only feeding raw time series data directly into llms, our framework introduces anomaly descriptions as a modality bridge to facilitate alignment between text and time series data and to aid in generating time series prompt embeddings. these descriptions are produced using anomaly detection technology @xcite , which converts continuous signals into human-readable summaries that encompass coarse-grained time series patterns. the process consists of two steps. first, anomaly descriptions establish a direct connection between time series lab tests and medical notes. second, the generated time series prompt embeddings contain both coarse-grained anomaly information and fine-grained time series variation patterns. these embeddings are then appended as prefix tokens to the input of the llm. this approach integrates structured time series information into the language modeling process without modifying the llm architecture, thereby unifying both modalities within the same encoding space and enhancing clinical decision-making. we optimize promedts with three self-supervised learning objectives for learning the time series prompt embeddings. a contrastive loss maps textual and time series modalities into a shared latent space to learn the coarse-grained lab test information. an anomaly-time series matching loss links numeric lab tests with their corresponding anomaly descriptions to reinforce consistency and learn the fine-grained lab test information. finally, an anomaly caption generation loss enhances the representation of multi-granularity time series information within the time series prompt embeddings. together, these objectives enable llms to process both structured and unstructured ehr data more effectively, addressing the gap between language and time series representations in healthcare applications. ## methodology in this section, we present the promedts framework for unifying heterogeneous ehr data through prompt-guided learning. we begin by defining the problem and describing the model inputs, then provide a high-level overview of the architecture. in subsequent sections, we detail each module and discuss how these components are applied to downstream tasks such as disease diagnosis. ## conclusion and future work in this paper, we introduce promedts, a lightweight and effective modality fusion framework that leverages self-supervised time series prompt learning for multimodal ehr integration. by bridging the modality gap between medical notes and lab test results, promedts enables llms to process structured and unstructured medical data more effectively. its three key modules and loss functions advance language-time series integration in healthcare, providing a scalable and adaptable approach for real-world clinical applications. evaluation on two real-world ehr datasets demon-