Unity in Diversity: Collaborative Pre-training Across Multimodal Medical Sources (2024)

Abstract: Although pre-training has become a prevalent approach for addressing various biomedical tasks, the current efficacy of pre-trained models is hindered by their reliance on a limited scope of medical sources. This limitation results in data scarcity during pre-training and restricts the range of applicable downstream tasks. In response to these challenges, we develop MedCSP, a new pre-training strategy designed to bridge the gap between multimodal medical sources. MedCSP employs modality-level aggregation to unify patient data within individual sources. Additionally, leveraging temporal information and diagnosis history, MedCSP effectively captures explicit and implicit correlations between patients across different sources. To evaluate the proposed strategy, we conduct comprehensive experiments, where the experiments are based on 6 modalities from 2 real-world medical data sources, and MedCSP is evaluated on 4 tasks against 19 baselines, marking an initial yet essential step towards cross-source modeling in the medical domain.

## introduction pre-training, a widely adopted technique with the primary objective of enhancing the performance of downstream tasks, is a practice extensively employed in natural language processing @xcite . in the medical domain, researchers have dedicated efforts in pretraining powerful models, including clinical-bert @xcite , clinicalt5 @xcite , and medhmp @xcite . while these pre-training techniques benefiting from unlabeled data have showcased superiority in diverse medical downstream tasks, they still suffer from the following challenges: data scarcity. training a robust pre-trained model typically requires a substantial corpus, particularly in a multimodal approach. however, obtaining a sizable training dataset in the medical domain poses challenges owing to concerns surrounding data privacy. hence, exploring innovative approaches for integrating more medical data into the pre-training process becomes imperative. limited downstream tasks. current pre-trained models in the medical domain are often trained using data from a single source, thus limiting the spectrum of applicable downstream tasks. for example, medhmp @xcite pretrained on electronic health records (ehrs) source is only suitable for predictive modeling tasks involving ehr data. in contrast, pre-trained models in the general domain are usually applicable to various tasks. for instance, flamingo @xcite , a visual-language model, achieves state-of-the-art performance on 16 few-shot learning tasks. therefore, considering the multimodal nature of medical data, an ideal pre-trained model should be equipped to address as many tasks as possible. to tackle these issues simultaneously, a promising approach involves training a model using diverse medical data sources from various datasets. this strategy not only augments the volume of training data but also broadens the spectrum of tasks. nevertheless, achieving this objective is inherently challenging due to the following reasons: firstly, the number of patients who have data across multiple data sources is significantly limited. for example, this number is 14,620 between the mimic-iv and mimic-cxr databases, representing only 22.36% and 45.19% of these two databases, respectively. the scarcity of patients with data spanning multiple sources further diminishes the limited connectivity between these sources, adding complexity to cross-source integration efforts. thus, designing an effective model that proficiently leverages the overlapped patients as a bridge to facilitate the training of other patients simultaneously is essential. secondly, modeling patients with data from multiple sources is challenging due to the intrinsic complexity of medical data. for example, a patient's chest x-ray images and reports may be housed in the mimic-cxr database, while diagnosis and treatment information are stored in the mimic-iv database. however, owing to the temporal nature of medical data, their recorded times of information across various sources may not align perfectly. therefore, exploring a reasonable approach to model these relationships is urgently needed. finally, implicit yet informative relationships often exist among patients across different sources, requiring appropriate handling. for instance, patients with analogous conditions may exhibit similar symptoms, despite being in separate databases. recognizing and leveraging these implicit connections is essential for facilitating cross-source training, as they hold significant potential for enhancing model performance through the aggregation of similar patient profiles. to address the aforementioned challenges inherent in multimodal medical records from diverse sources, we introduce a pioneering pre-training framework in this paper, named medical cross-source pre-training (medcsp), as shown in figure 1 . medcsp first encodes each modality from each source using modality-specific encoders in section 3.2. subsequently, it employs two distinct pre-training tasks. the first task explores modalitylevel relations among patients within individual sources (section 3.3), while the second task focuses on discovering relationships among patients across different sources (section 3.4). specifically, medcsp models relations for overlapped patients across sources by considering their record times in section 3.4.1 and establishes connections among patients in similar cohorts using their diagnosis similarities in section 3.4.2. through interactive modeling, medcsp acquires the capability to generate informative and representative medical embeddings for diverse downstream tasks. our exhaustive experiments across six modalities within two sources demonstrate the effectiveness of our pre-training strategy, providing an initial yet crucial solution for unifying diverse modalities across multiple medical sources. thus, research on pre-training with multimodal medical data excluding images remains relatively limited. some researchers have achieved success by aligning numerical clinical features with diagnosis codes @xcite , while others have explored the correlation between clinical language and codes. recent advancements include modeling complex interactions within ehr data, incorporating multiple modalities such as diagnosis codes, demographics, clinical notes, medication codes, and clinical monitoring data @xcite . nonetheless, these endeavors face challenges akin to those encountered in imagerelated pre-training, compounded by the issue of data scarcity within ehr data, which significantly restricts their broader applicability. multi-source multimodal pre-training. conventional pre-training techniques typically leverage diverse data sources to enhance the generalizability of representations, a principle that extends to multimodal settings. previous studies @xcite @xcite @xcite have demonstrated this by combining image-text pairs from multiple sources, thereby enhancing model performance across vari-ous domains. however, these models face limitations when confronted with new modalities, as they are built upon uniform data sources. recognizing the shortcomings of homogeneous multimodal pre-training approaches, recent endeavors have shifted focus towards harnessing more diverse and heterogeneous sources. recent studies @xcite have embraced data from varied modalities for joint pretraining, resulting in improved modality-specific encoders. despite these advancements, designed to cater to general fields, these approaches struggle to capture latent medical correlations within multimodal health data, thereby impeding the generation of domain-specific representations. ## experiments in this section, we first outline the configuration of our pretraining process in section 4.1 and then demonstrate evaluation with downstream tasks on ehr source (section 4.2) and medical image source (section 4.3), respectively. due to the space limitation, we put more experimental results in appendix d and e. ## conclusion this paper introduces a novel pre-training framework, medcsp, specifically designed for the complexities of diverse and highly heterogeneous medical sources. medcsp aggregates patient data within individual sources by aligning different modalities and subsequently captures patient relationships across multiple medical sources by leveraging temporal information and diagnosis history. our experiments across a range of medical tasks and sources demonstrate that medcsp achieves superior performance. the observations are further supported by ablation studies and case analyses, underscoring the potential of medcsp in advancing medical cross-source modeling.