LIFTED : Multimodal Clinical Trial Outcome Prediction via Large Language Models and Mixture-of-Experts (2025)

Abstract: Clinical trials are pivotal yet costly processes, often spanning multiple years and requiring substantial expenses, motivating predictive models to identify likely-to-fail drugs early and save resources. Recent approaches leverage deep learning to integrate multimodal data for clinical outcome prediction; however, they rely heavily on manually designed modality-specific encoders, limiting their adaptability to new modalities and ability to effectively share information across modalities. To address these challenges, we propose a multimodal mixture-of-experts (LIFTED) framework. Specifically, LIFTED transforms modality-specific data into natural language descriptions, encoded via unified, noise-resilient encoders. A sparse Mixture-of-Experts mechanism then identifies shared patterns across modalities, extracting consistent representations. Finally, another mixture-of-experts module dynamically integrates these modality representations, emphasizing critical information. Experiments show that LIFTED significantly outperforms baseline methods in predicting clinical trial outcomes across all phases, highlighting the effectiveness of our proposed approach.

## introduction the clinical trial is a crucial step in the development of new treatments to demonstrate the safety and efficacy of the drug. drugs must pass three trial phases involving human participants with target diseases before approval for manufacturing. however, the clinical trial is time-consuming and experiments expensive, taking multiple years and costing up to hundreds of millions of dollars @xcite . in addition, the success rate of clinical trials is exceedingly low and many drugs fail to pass these clinical trials @xcite . therefore, the ability to predict clinical trial outcomes beforehand, allowing the exclusion of drugs with a high likelihood of failure, holds the potential to yield significant cost savings. given the increasing accumulation of clinical trial data over the past decade (e.g., drug descriptions, and patient criteria), we can now leverage this wealth of data for the prediction of clinical trial outcomes. early attempts aim to improve the clinical trial outcome prediction results by modeling the components of the drugs (e.g., drug toxicity @xcite , modeled the pharmacokinetics @xcite ). recently, deep learning methods have been proposed for trial outcome predictions. for instance, @xcite predicted drug approvals for 15 different disease groups by incorporating drug and clinical trial features into machine learning models. @xcite proposed an interaction network leveraging multimodal data (e.g., molecule information, trial documents) to capture correlations for trial outcome predictions. however, this approach relies on modal-specific encoders to extract representations from different modal data, which require manually designed encoder structures and limit their extensibility when new modal data becomes available for use. to address these issues, we aim to design a unified encoder to extract representations from various modalities, but it poses the following challenges: to address those challenges, we propose an approach called multimodal clinical trial outcome prediction via large language models and mixture-of-experts (lifted), which extracts information from different modalities with a transformer based unified encoder, enhances the extracted features by a sparse mixture-of-experts (smoe) framework and integrates multimodal information with mixture-of-experts (moe). specifically, lifted unifies diverse multimodal features, even those in different formats, by converting them into natural language descriptions. subsequently, we build a unified transformer-based encoder to extract representations from these modal-specific language descriptions and refine the representations with an smoe framework. here, the representations from different modalities are dynamically routed by a noisy top-k gating network to a portion of shared expert models, facilitating the extraction of similar information patterns. in addition, we introduce representation augmentation to enhance the resilience of transform-based encoders and the smoe framework to potential data noise introduced during the data collection process. furthermore, lifted treats the extracted representations from various modalities as distinct experts and utilizes a mixture-of-experts module to dynamically combine these multimodal representations for each example. this dynamic combination allows for the automatic assignment of higher weights to more crucial modalities. finally, we evaluate lifted on the hint benchmark @xcite and the ctod benchmark @xcite to demonstrate the effectiveness of lifted and the effectiveness of our proposed components. ## multimodal clinical trial outcome prediction via large language models and mixture-of-experts this section introduces our proposed multimodal clinical trial outcome prediction via large language models and mixture-of-experts (lifted) approach. the objective of lifted is to integrate multimodal clinical trial data using natural language descriptions within a mixture-of-experts (moe) framework, as illustrated in figure 1 . our method involves two main steps: firstly, multimodal data is converted into natural language descriptions using a large language model (llm). secondly, these descriptions along with their corresponding labels are utilized to train our predictive model. specifically, we employ transformer-based encoders to derive meaningful representations from the textual data. to improve the robustness of these embeddings, we introduce perturbations and apply a consistency loss. finally, the representations from different modalities are integrated through an moe framework to produce the predictions. each of these components is explained comprehensively in the following sections. ## experiments in this section, we evaluate the performance of lifted aiming to answer the following questions: q1: compared to the existing methods with modalspecific encoders, can lifted achieve better performance with the unified transformer encoders? q2: do the key components, including the multimodal data integration component and the representation augmentation component, of lifted boost the performance? q3: does the sparse mixture-of-experts framework route similar information patterns in different modalities to the same expert models correctly? q4: does the mixture-of-experts approach precisely measure the importance of different modalities for each patient? ## conclusion we introduce lifted, an approach that unifies multimodal data using natural language descriptions and integrates this information within a mixture-of-experts (moe) framework for clinical trial outcome prediction. we employ noiseresilient encoders to extract representations from each modality, utilize a sparse moe framework to further dig the similar information patterns in different modalities, and introduce an auxiliary loss to improve the quality of modal-specific representations. empirically, our lifted method demonstrates superior performance compared to existing approaches across all three phases of clinical trials, underscoring the effectiveness and potent representational capacity of natural language and highlighting the potential for a unified text modality to supplant diverse information modalities.