A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery (2024)

Abstract: In many scientific fields, large language models (LLMs) have revolutionized the way text and other modalities of data (e.g., molecules and proteins) are handled, achieving superior performance in various applications and augmenting the scientific discovery process. Nevertheless, previous surveys on scientific LLMs often concentrate on one or two fields or a single modality. In this paper, we aim to provide a more holistic view of the research landscape by unveiling cross-field and cross-modal connections between scientific LLMs regarding their architectures and pre-training techniques. To this end, we comprehensively survey over 260 scientific LLMs, discuss their commonalities and differences, as well as summarize pre-training datasets and evaluation tasks for each field and modality. Moreover, we investigate how LLMs have been deployed to benefit scientific discovery. Resources related to this survey are available at https://github.com/yuzhimanhua/Awesome-Scientific-Language-Models.

## introduction the emergence of large language models (llms) @xcite brings a new paradigm to natural language processing (nlp) by replacing specialized models designed for each task with unified models that are reasonably effective for a wide spectrum of problems. in the scientific domain, such a paradigm not only reshapes people's strategies to handle tasks related to natural language (e.g., scientific papers, medical records, and climate reports) but also inspires analogous ideas to deal with other types of data (e.g., molecules, proteins, tables, and metadata). in addition to understanding existing scientific data, llms have shown their potential to accelerate scientific discovery @xcite @xcite through generation, planning, etc. ## llms in chemistry and materials early studies on chemistry llms mostly adopt a moderate-sized encoder-only architecture pre-trained with mlm (type 1.a, e.g., chembert @xcite , matscibert @xcite , and batterybert @xcite ). these models are usually evaluated on downstream tasks including reaction role labeling @xcite and abstract classification @xcite . recently, researchers have focused more on large-scale decoder-only llms trained with next token prediction and instruction tuning (type 2.a). examples include chemdfm @xcite , chemllm @xcite , and llasmol @xcite . given the desired generalization capability of such models, they are evaluated on a diverse set of tasks such as name conversion @xcite , reaction prediction @xcite , retrosynthesis @xcite ), text-based molecule design @xcite , and crystal generation @xcite @xcite . ## llms in biology and medicine the watershed moment in the evolution biomedical llms is still the emergence of billionparameter architectures and instruction tuning. before that, a wide variety of moderate-sized back-bones are explored, including both encoder-based (type 1.a, e.g., biobert @xcite , bio-electra (ozyurt, 2020), bioroberta @xcite , bioalbert @xcite , and clinical-longformer @xcite ) and (encoder-)decoder-based ones (type 2.a, e.g., sci-five @xcite , biobart @xcite , and biogpt @xcite ). evaluation tasks for these models range from biomedical ner, re, sentence similarity estimation, document classification, and qa (i.e., the blurb benchmark @xcite ) to natural language inference (nli) @xcite and entity linking @xcite . after the watershed, the trend becomes instruction-tuning billion-parameter llms (type 2.a, e.g., med-palm @xcite , medalpaca @xcite , and biomistral @xcite ). accordingly, evaluation tasks now include singleround qa @xcite and multi-round dialogue @xcite . meanwhile, there are studies proposing a bi-encoder architecture (type 3.a, e.g., @xcite and @xcite ) that specifically targets biomedical retrieval tasks, the benchmarks of which are nfcorpus @xcite , trec-covid @xcite , etc. ## challenges and future directions in this survey, we compile literature that elucidates the data, architectures, and tasks used for scientific llm pre-training, as well as how scientific llms have been applied to downstream applications in scientific discovery. in particular, we underscore analogous architectures, tasks, and trends observed during the evolution of scientific llms across different fields and modalities. beyond reviewing prior research, we present several challenges to inspire further exploration of this topic. diving into fine-grained themes. most existing scientific llms target a coarse-grained field (e.g., chemistry), while some tasks rely on highly specialized knowledge of a fine-grained theme (e.g., suzuki coupling). when llms are pre-trained on more general corpora, frequently appeared signals may dominate the model parameter space, and domain-specific tail knowledge may be wiped out. we believe that automatically curating indepth, theme-focused knowledge graphs @xcite to guide the generation process will be a promising direction to tackle this issue. generalizing to out-of-distribution scientific data. in the scientific domain, it is common that the testing distribution shifts from the training distribution @xcite : novel scientific concepts keep emerging in newly published papers; unseen molecules with different scaffolds and unseen proteins with different numbers of peptide chains may appear during testing. handling such out-of-distribution data remains a challenge for pretrained scientific llms. to our knowledge, invariant learning @xcite can serve as the theoretical foundation for out-of-distribution analyses, and how to integrate it into llm pre-training is worth exploring.