CTPD : Cross-Modal Temporal Pattern Discovery for Enhanced Multimodal Electronic Health Records Analysis (2025)

Abstract: Integrating multimodal clinical records—such as Electronic Health Records (EHR) and free-text clinical reports—has shown great potential in predicting clinical outcomes. However, prior work has primarily focused on capturing temporal interactions within individual samples and fusing multimodal information, overlooking critical temporal patterns across different patients. These patterns, such as trends in vital signs like abnormal heart rate or blood pressure, can indicate deteriorating health or an impending critical event of any individual in a given population. Similarly, clinical notes often contain textual descriptions that reflect these changes. Identifying corresponding temporal patterns across different modalities is crucial for improving the accuracy of clinical outcome predictions, yet it remains a challenging task. To address this gap, we introduce a Cross-modal Temporal Pattern Discovery (CTPD) framework, designed to efficiently extract meaningful cross-modal temporal patterns from multimodal EHR data. Our approach introduces shared initial temporal pattern representations and refines them using slot attention to generate temporal semantic embeddings. To ensure rich cross-modal temporal semantics in the learned patterns, we introduce a Temporal Pattern Noise Contrastive Estimation (TP-NCE) loss for cross-modal alignment, along with two reconstruction losses to retain core information of each modality. Evaluations on two clinically critical tasks—48 hour in-hospital mortality and 24-hour phenotype classification—using the MIMIC-III database demonstrate the superiority of our method over existing approaches. The code is anonymously available at https://github.com/HKU-MedAI/CTPD.

## introduction the increasing availability of electronic health records (ehr) presents significant opportunities for advancing predictive modeling in healthcare @xcite . ehr data is inherently multimodal and time-aware, encompassing structured data like vital signs, laboratory results, and medications, as well as unstructured data such as free-text clinical reports @xcite . integrating these diverse data types is crucial for comprehensive patient monitoring and accurate prediction of clinical outcomes @xcite @xcite . however, the irregularity and heterogeneity of multimodal data present significant challenges for precise outcome prediction. existing approaches primarily address either the irregularity of data @xcite @xcite or the fusion of multiple modalities @xcite @xcite , but they often neglect broader temporal trends that span across patient cases. these cross-modal temporal patterns, present in both structured and unstructured data, can provide high-level semantic insights into a patient's health trajectory and potential risks @xcite . as illustrated in fig. 1 , high-level semantic patterns related to medical conditions can emerge across multiple modalities. capturing these patterns in a cross-modal, temporal manner is essential for improving predictive performance. furthermore, critical temporal patterns in ehr data unfold at different time scales @xcite @xcite ), yet existing methods struggle to capture variations across these granularities. for instance, sudden changes in vital signs-such as a sharp drop in oxygen saturation or a rapid heart rate increase-may indicate an acute health crisis. in contrast, longer-term trends, such as persistently high blood pressure or a gradual decline in respiratory function, may signal deteriorating health or an impending critical event. effectively analyzing patient states across multiple time scales is crucial for comprehensive ehr modeling, yet remains an open challenge in current methodologies. figure 1 : motivation of our proposed ctpd: we visualized the time-series ehr with corresponding clinical notes in one icu stay of the mimic-iii dataset, and observed the temporal patterns across two modalities: blue text highlights respiratory status. oxygen requirements gradually decreased from 8l to 4l, and then to 2l nasal cannula, indicating steady respiratory improvement. note that this pattern is also reflected from the time series. green text captures cough progression and medication effects. symptom relief was observed after administering robitussin with codeine, demonstrating a delayed but positive response to treatment. yellow text represents infection monitoring. the detection of gram-positive cocci prompted blood culture collection (bld cx) for further evaluation, indicating active infection surveillance. to address these limitations, we propose the cross-modal temporal pattern discovery (ctpd) framework, designed to extract meaningful temporal patterns from multimodal ehr data to improve the accuracy of clinical outcome predictions. the core innovation of our approach is a novel temporal pattern discovery module, which identifies corresponding temporal patterns (i.e., temporal prototypes) with meaningful semantics across both modalities throughout the dataset. this approach ensures that the model captures essential temporal semantics relevant to patient outcomes, providing a more comprehensive understanding of the data. to further enhance the quality of the learned temporal patterns, we introduce a temporal pattern noise contrastive estimation (tp-nce) loss for aligning pattern embeddings across modalities, along with auxiliary reconstruction losses to ensure that the patterns retain core information of the data. moreover, our framework incorporates a transformer @xcite -based fusion mechanism to effectively fuse the discovered temporal patterns with timestamp-level representations from both modalities, leading to more accurate predictions. we evaluate ctpd on two critical clinical prediction tasks: 48-hour in-hospital mortality prediction and 24-hour phenotype classification, using the mimic-iii database. the results demonstrate the effectiveness of our approach, which significantly outperforms existing methods, and suggest a promising direction for improving multimodal ehr analysis for clinical prediction. ## conclusion temporal pattern discovery (ctpd) framework, which captures cross-modal temporal patterns and incorporates them with timestamp-level embeddings for more accurate clinical outcome predictions based on multimodal ehr data. to efficiently optimize the framework, we introduce a temporal pattern noise contrastive estimation (tp-nce) loss to enhance cross-modal alignment, along with two reconstruction objectives to retain core information from each modality. our experiments on two clinical prediction tasks using the mimic-iii dataset demonstrate the effectiveness of ctpd in multimodal ehr analysis.