INDUS : Effective and Efficient Language Models for Scientific Applications (2024)

Abstract: Large language models (LLMs) trained on general domain corpora showed remarkable results on natural language processing (NLP) tasks. However, previous research demonstrated LLMs trained using domain-focused corpora perform better on specialized tasks. Inspired by this insight, we developed INDUS, a comprehensive suite of LLMs tailored for the closely-related domains of Earth science, biology, physics, heliophysics, planetary sciences and astrophysics, and trained using curated scientific corpora drawn from diverse data sources. The suite of models include: (1) an encoder model trained using domain-specific vocabulary and corpora to address NLP tasks, (2) a contrastive-learning based text embedding model trained using a diverse set of datasets to address information retrieval tasks and (3) smaller versions of these models created using knowledge distillation for applications which have latency or resource constraints. We also created three new scientific benchmark datasets, Climate-Change NER (entity-recognition), NASA-QA (extractive QA) and NASA-IR (IR) to accelerate research in these multi-disciplinary fields. We show that our models outperform both general-purpose (RoBERTa) and domain- specific (SciBERT) encoders on these new tasks as well as existing tasks in the domains of interest. Furthermore, we demonstrate the use of these models in two industrial settings- as a retrieval model for large-scale vector search applications and in automatic content tagging systems.

## introduction large language models (llms) trained on huge amounts of data have demonstrated impressive ca-* correspondence: bhatta@ibm.com, mr0051@uah.edu, aashka.trivedi@ibm.com, rahul.ramachandran@nasa.gov ## data sufficient high-quality in-domain corpora is essential to develop models that perform better than their counterparts trained on open-domain corpora. we meticulously identified corpora for each of the aforementioned domains, and created english-only models for containment. specifically, for each domain, we used open-source data which has a 3 methodology: encoder models ## methodology: sentence embedding models sentence embedding models represent text as lowdimensional vectors for efficient use in dense retrieval systems, such as retrieval augmented generation, where relevant passages for a query are identified by the similarity between their embeddings @xcite . embedding models are trained using a contrastive learning objective @xcite , which pushes the embeddings of a query closer to those of relevant passages and further away from those of non-relevant ones. we use the improved contrastive loss proposed in @xcite which introduces an additional bidirectional signal to expand negatives. base embedding model we created our sentence embedding model, indus-retriever base foot_8 , by fine-tuning indus base , following a bi-encoder framework @xcite . similar to prior work @xcite @xcite , we employed a stage-wise training approach. we first train on a large corpus of naturally occurring pairs collected from internet sources, and specifically include data from the science domain. furthermore, we created a domainspecific dataset from the ads data ( §2) by including title-abstract pairs. then, we finetune on high quality annotated datasets (e.g., question-answer pairs). appendix b contains comprehensive details about the datasets used in training. for both stages, we used large batch sizes and in-batch negatives to better approximate the contrastive objective. ## creating benchmarks benchmark datasets play a crucial role in assessing the language understanding capabilities of models. however, there is an absence of datasets tailored for the diverse and multidisciplinary fields under study. thus, to effectively benchmark the proposed nlp models, we introduced three new datasets for ner, qa and ir. appendix d compares the sizes of these datasets to popularly used benchmarks. ## experimental results baselines we compared indus models against open source models of similar sizes (all models obtained from huggingface): ## industrial applications of indus we show industrial applications of indus models for downstream tasks in the scientific domain. ## conclusion in this work, we presented indus, a constellation of models for use in the science domain and show their applications in industrial settings. we demonstrated the effectiveness of a custom tokenizer and in-domain data for training high-quality encoder models and sentence embedding models. further, we created smaller versions of the models suitable for applications with latency or resource constraints through state-of-the-art knowledge distillation techniques. for the benefit of the scientific community, we have released all models and benchmarks. a training details: encoder models indus base was trained with the masked language modeling objective, using the default hyperparameters recommended in