MEIT : Multimodal Electrocardiogram Instruction Tuning on Large Language Models for Report Generation (2025)

Abstract: Electrocardiogram (ECG) is the primary non-invasive diagnostic tool for monitoring cardiac conditions and is crucial in assisting clinicians. Recent studies have concentrated on classifying cardiac conditions using ECG data but have overlooked ECG report generation, which is time-consuming and requires clinical expertise. To automate ECG report generation and ensure its versatility, we propose the Multimodal ECG Instruction Tuning (MEIT) framework, the first attempt to tackle ECG report generation with LLMs and multimodal instructions. To facilitate future research, we establish a benchmark to evaluate MEIT with various LLMs backbones across two large-scale ECG datasets. Our approach uniquely aligns the representations of the ECG signal and the report, and we conduct extensive experiments to benchmark MEIT with nine open-source LLMs using more than 800,000 ECG reports. MEIT’s results underscore the superior performance of instruction-tuned LLMs, showcasing their proficiency in quality report generation, zero-shot capabilities, resilience to signal perturbation, and alignment with human expert evaluation. These findings emphasize the efficacy of our MEIT framework and its potential for real-world clinical application.

## introduction electrocardiogram (ecg) is the primary tool for heart disease diagnosis. in standard practice, cardiologists examine these ecg recordings and manually generate detailed diagnostic reports, which is a complex and time-consuming process. recently, ai models have been developed to streamline ecg data analysis for the classification task @xcite @xcite , yet the automatic generation of reports from ecg recordings remains relatively underexplored. unlike image-based medical report generation tasks (e.g., radiology reports), ecg report gener-ation faces unique challenges due to the concise, keyword-centric content of ecg reports, which contrasts with the more extensive anatomical descriptions in radiology. directly transferring methods from radiology is hindered by the distinct nature of ecg signals and the limited semantic overlap with imaging data. furthermore, there is still a lack of comprehensive benchmarks for evaluating the performance of ecg report generation. in this work, we introduce meit, a multimodal ecg instruction tuning framework that extends the capability of large language models (llms) for the task of ecg report generation. drawing on the versatility of llms and multimodal llms (mllms) @xcite @xcite @xcite @xcite in various language tasks, we develop a specialized instruction tuning strategy for ecg report generation. by aligning ecg recordings with human instructions, meit produces clinically relevant reports and demonstrates zero-shot performance when transferring across different continents and data collection devices. concretely, we construct a multimodal instruction dataset from publicly available ecg recordings and propose an efficient attention-based fusion method that incorporates ecg signals into the latent space of llms without adding new parameters. we also introduce a comprehensive benchmark to evaluate ecg report generation across two largescale datasets (with 20k and 800k ecg-report pairs), covering four tasks: (1) report generation quality, (2) zero-shot transfer across datasets, (3) robustness under signal perturbations, and (4) alignment with expert evaluation. we assess meit using ten open-source llms, demonstrating: (1) the superior performance of meit in ecg report generation and effective ecg representation learning; and (2) the strong transferability of instructiontuned llms across diverse clinical domains. in summary, our primary contribution is the meit framework, a novel method for automated ecg report generation and evaluation based on llms. it features a lightweight, attention-based fusion module for integrating ecg signals and text, along with a newly designed four-task benchmark for ecg report evaluation. empirical results highlight meit's ability to generate high-quality reports, perform robustly under data perturbations, and align closely with expert assessments, thereby paving the way for improved ecg interpretation and broader innovations in embedding biomedical signals into llms. instruction tuning. our work is also related to instruction tuning, which improves zero-shot learning in llms through task-specific instructions @xcite . models like instructgpt @xcite , @xcite , and alpaca @xcite leverage instruction data, including human feedback. multimodal variants, such as llava @xcite , minigpt-4 @xcite , and anymal @xcite , extend this to visual tasks. however, these methods focus on natural images and are not directly applicable to ecg signals. to bridge this gap, we introduce a specialized instruction-tuning framework for ecg report generation. ## limitations our meit framework first attempts to address automatic ecg report generation through ecg instruction tuning, establishing the first comprehensive benchmark for this process with mainstream llms. in this work, we mainly focus on generating reports from multimodal ecg instructions using llms. however, the generated results are not fully explainable or controllable, even though the generation procedure is transparent and trackable. this is because the underlying theory of llms remains largely unexplored, necessitating further investigation to ensure the quality and safety of the generated content. in the future, we aim to enable llms to utilize external, expert-verified knowledge databases, such as clinical protocols and medical textbooks, to enhance the explainability of the generated ecg reports. ## acknowledgment we thank the reviewers and acs for their helpful comments.