FEDKIM : Adaptive Federated Knowledge Injection into Medical Foundation Models (2024)

Abstract: Foundation models have demonstrated remarkable capabilities in handling diverse modalities and tasks, outperforming conventional artificial intelligence (AI) approaches that are highly task-specific and modality-reliant. In the medical domain, however, the development of comprehensive foundation models is constrained by limited access to diverse modalities and stringent privacy regulations. To address these constraints, this study introduces a novel knowledge injection approach, FedKIM, designed to scale the medical foundation model within a federated learning framework. FedKIM leverages lightweight local models to extract healthcare knowledge from private data and integrates this knowledge into a centralized foundation model using a designed adaptive Multitask Multimodal Mixture Of Experts (M ^3 OE) module. This method not only preserves privacy but also enhances the model’s ability to handle complex medical tasks involving multiple modalities. Our extensive experiments across twelve tasks in seven modalities demonstrate the effectiveness of FedKIM in various settings, highlighting its potential to scale medical foundation models without direct access to sensitive data. Source codes are available at https://github.com/XiaochenWang-PSU/FedKIM.

## introduction similar to large language models @xcite and foundation models @xcite , medical foundation models @xcite have achieved superior performance of handling diverse modalities and tasks within the medical domain. these models have the potential to revolutionize medical diagnostics and treatment by leveraging data-driven insights from large volumes of multimodal healthcare data. ## performance evaluation we examine our proposed fedkim from the zeroshot evaluation (subsection 4.1) and fine-tuning evaluation (subsection 4.2) perspectives. ## conclusion this work introduces the concept of knowledge injection into medical foundation models, emphasizing its critical role and potential in the development of comprehensive medical models. we propose a novel approach, fedkim, designed to extract and inject healthcare knowledge into foundation models, thereby enhancing their ability to handle multiple tasks and modalities. fedkim leverages flexible federated learning techniques to extract knowledge from distributed medical data. the extracted knowledge is then injected into the foundation model using our proposed adaptive m 3 oe module. our exhaustive experimental results on 12 tasks and 7 modalities demonstrate the effectiveness of fedkim in diverse settings, showcasing its excellent capability in handling either encountered or unseen healthcare tasks. this study validates the potential of injecting knowledge into foundation models using federated learning, providing a crucial solution for developing a healthcare foundation model without accessing sensitive data.