S ci M ind: A Multimodal Mixture-of-Experts Model for Advancing Pharmaceutical Sciences (2024)

Abstract: Large language models (LLMs) have made substantial strides, but their use in reliably tackling issues within specialized domains, particularly in interdisciplinary areas like pharmaceutical sciences, is hindered by data heterogeneity, knowledge complexity, unique objectives, and a spectrum of constraint conditions. In this area, diverse modalities such as nucleic acids, proteins, molecular structures, and natural language are often involved. We designed a specialized token set and introduced a new Mixture-of-Experts (MoEs) pre-training and fine-tuning strategy to unify these modalities in one model. With this strategy, we’ve created a multi-modal mixture-of-experts foundational model for pharmaceutical sciences, named SciMind. This model has undergone extensive pre-training on publicly accessible datasets including nucleic acid sequences, protein sequences, molecular structure strings, and biomedical texts, and delivers good performance on biomedical text comprehension, promoter prediction, protein function prediction, molecular description, and molecular generation.

## introduction large language models (llms) have made substantial strides, providing a versatile, taskagnostic base for a variety of applications @xcite , @xcite , @xcite . however, their use in reliably tackling issues within specialized domains, particularly in interdisciplinary areas like pharmaceutical sciences, is hindered by several obstacles. these include data heterogeneity, knowledge complexity, unique objectives, and a spectrum of constraint conditions, which block the creation of groundbreaking applications @xcite , @xcite , @xcite . this research aims to lay the groundwork for a large-scale model within the pharmaceutical sciences. in this area, four diverse modalities including nucleic acids, proteins, molecular structures, and natural language are involved. of them, nucleic acids, proteins and molecular structures are the common modalities modeled by the pharmaceutical science community. predicting the properties of a molecule or a protein @xcite , @xcite , @xcite , designing and optimizing for new ones @xcite , @xcite , @xcite , and understanding how figure 1 : an overview of the four modalities in pharmaceutical sciences. the three traditional modalities, including nucleic acids (dna/rna), proteins, and small molecules, are typically modeled independently. recent advancements have been made in the realm of cross-modal modeling, as indicated by the solid lines. however, there is a gap domain. in recent times, the natural language modality has surfaced as a highly promising method to describe nucleotide sequences, small molecules, and proteins, and it is swiftly garnering attention. ## related works in this section, we will provide a concise overview of the related work on cross-modal models in the field of pharmaceutical sciences. ## scimind in this section, we will detail the design and training of our multi-modal mixture-of-experts model, scimind. the overview of the pre-training is illustrated in figure 2 . unlike existing models, our focus is on integrating all modalities into a single model. to this end, we have designed specialized token sets for each modality. however, each modality has a different level of complexity and requires a different number of parameters to avoid overfitting. to leverage the many opensource pretrained language models, we have chosen to construct a mixture-of-experts model by splitting the pretrained llama-2-7b model into 16 experts at each of the feedforward layers. ## conclusions and discussion in this paper, we introduced scimind, a unified pretraining framework designed to include all the modalities in pharmaceutical sciences. we've designed a specialized token set and introduce a new pre-training and fine-tuning strategy that leverages the advantages of large-parameter models while minimizing their expenses. this strategy, supported by a prior expert allocation and selection mechanism, allows data of different modalities to choose the most suitable processing path. this method not only leads to a sparser model architecture, thus cutting down on inference costs, but also avoids modal alignment and the potential performance decrease due to model size reduction. we've created a multi-modal mixture-of-expert foundational large model for pharmaceutical sciences, named scimind. this model has undergone extensive pre-training on publicly accessible datasets including nucleic acids, protein sequences, molecular structures strings, and biomedical texts, and could be fine-tuned for downstream tasks involving all modalities in pharmaceutical sciences. the experimental outcomes suggest that the scimind model not only delivers outstanding performance but also shows high flexibility and interpretability in response to prompt words, offering a sturdy base for its use in pharmaceutical sciences. due to the lack of well-aligned multimodal data, our model has not fully demonstrated its advantages. in addition to molecular captioning and generation by description, the inclusion of the protein modality will make the interaction between the language and small molecule modalities more explainable and useful. this approach helps accumulate more information and is a promising direction to explore.