Knowledge-enhanced Multimodal ECG Representation Learning with Arbitrary-Lead Inputs (2025)

Abstract: Recent advancements in multimodal representation learning for electrocardiogram (ECG) have moved onto learning representations by aligning ECG signals with their paired free-text reports. However, current methods often result in suboptimal alignment of ECG signals with their corresponding text reports, thereby limiting diagnostic accuracy. This is primarily due to the complexity and unstructured nature of medical language, which makes it challenging to effectively align ECG signals with the corresponding text reports. Additionally, these methods are unable to handle arbitrary combinations of ECG leads as inputs, which poses a challenge since 12-lead ECGs may not always be available in under-resourced clinical environments.In this work, we propose the **Knowledge-enhanced Multimodal ECG Representation Learning (K-MERL)** framework to address these challenges. K-MERL leverages large language models (LLMs) to extract structured knowledge from free-text reports, enhancing the effectiveness of ECG multimodal learning. Furthermore, we design a lead-aware ECG encoder to capture lead-specific spatial-temporal characteristics of 12-lead ECGs, with dynamic lead masking. This novel encoder allows our framework to handle arbitrary lead inputs, rather than being limited to a fixed set of full 12 leads, which existing methods necessitate.We evaluate K-MERL on six external ECG datasets and demonstrate its superior capability. K-MERL not only outperforms all existing methods in zero-shot classification and linear probing tasks using 12 leads, but also achieves state-of-the-art (SOTA) results in partial-lead settings, with an average improvement of **16%** in AUC score on zero-shot classification compared to previous SOTA multimodal methods. All data and code will be released upon acceptance.

## introduction recent advancements in deep learning have enabled automated classification of cardiovascular disease (cvd) using electrocardiograms (ecgs), one of the most crucial diagnostic tools. however, most methods are supervised, requiring large amounts of annotated data, which is costly and demands prohibitively extensive expert effort in annotation @xcite . to address this challenge, self-supervised multimodal learning has recently emerged as an effective approach for learning representative ecg features from accompanied free-text clinical reports @xcite @xcite . to this end, merl @xcite recently introduced the first comprehensive benchmark using the largest dataset mimic-ecg @xcite for pretraining, and six datasets @xcite ; liu 1 https://github.com/cheliu-computation/emnlp25kmerl @xcite @xcite for evaluating downstream task performance, including zero-shot classification and linear probing. despite outperforming signal-only selfsupervised approaches, multi-modal approaches, including merl @xcite , still have notable drawbacks: they directly align ecg signals with reports, introducing unnecessary noise due to the free-text nature of the reports, and failing to fully exploit the rich cardiac knowledge contained within the text. additionally, they encode ecg in a lead-agnostic manner, overlooking the unique spatial and temporal characteristics of the individual 12 ecg leads. moreover, they require all 12 leads to be available as input, limiting their ability to generalize across different lead combinations. this raises important practical concerns since full 12-lead ecg data is not always available in clinical environments due to factors such as patient mobility issues, the need for rapid assessments in emergencies, and limited resource in pre-hospital care environments @xcite @xcite @xcite @xcite . to overcome the challenges listed above, we make the following contributions: (1) we propose a framework dubbed knowledge-enhanced ecg multimodal representation learning (k-merl), which extracts cardiac-related entities from free-text ecg reports, converting unstructured reports into structured knowledge to enhance self-supervised ecg multimodal learning. to the best of our knowledge, this is the first work to leverage structured cardiac entities extracted from clinical reports to improve ecg multimodal learning. (2) to effectively capture and leverage the lead-specific spatial and temporal characteristics of 12-lead ecgs, we explore various tokenization and positional embedding techniques. in particular, we design lead-specific tokenization and lead-specific spatial positional embeddings, enabling the framework to capture the distinctiveness of each lead. (3) to enable our framework to handle arbitrary combinations of input leads, we introduce a dynamic lead masking strategy. in addition, we propose an independent segment masking strategy to further capture lead-specific temporal patterns. (4) our k-merl framework demonstrates superior performance in zero-shot classification and linear probing on multiple downstream datasets in various lead combinations, from a single lead to all 12 leads. ## analysis this section provides ablation studies on the key components of k-merl and reports zero-shot classification results for single-lead and 12-lead inputs across all downstream datasets. due to the page limit, we show more ablation studies in sec h. ## related work recent ecg self-supervised learning (essl) approaches have explored contrastive @xcite and generative @xcite @xcite objectives, but they remain unimodal and often lack clinical context. multimodal ecg methods @xcite @xcite typically align signals with text using simplistic prompts or assume fixed lead configurations, limiting generalizability. drawing inspiration from knowledge-enhanced radiograph-language models @xcite , we introduce structured cardiovascular entities extracted from ecg reports to provide fine-grained supervision without relying on radiology-specific ontologies, preserving clinical distinctions such as myocardial infarction subtypes @xcite . additionally, we address the practical challenge of partial-lead ecg inputs (alizadeh meghrazi et al., 2020) by designing dynamic masking and leadspecific processing, enabling our model to operate robustly under varying lead configurations. a more comprehensive review and comparison with prior work are provided in appendix a. ## conclusion we present k-merl, a knowledge-enhanced ecg multimodal learning framework capable of processing arbitrary lead inputs. first, we mine cardiacrelated entities as structured knowledge from ecg free-text reports using a general llm, without relying on external domain-specific resources. next, we align ecg features with these cardiac-related entities to integrate this knowledge into the ecg multimodal learning. additionally, we introduce lead-specific processing and lead&segment masking strategies to capture the spatial-temporal patterns unique to each ecg lead, enabling the model to handle varying lead inputs. our experiments on six downstream ecg classification tasks, along with extensive ablation studies, demonstrate k-merl's superior performance over existing ecg representation learning methods.