H oney B ee: Progressive Instruction Finetuning of Large Language Models for Materials Science (2023)

Abstract: We propose an instruction-based process for trustworthy data curation in materials science (MatSci-Instruct), which we then apply to finetune a LLaMa-based language model targeted for materials science (HoneyBee). MatSci-Instruct helps alleviate the scarcity of relevant, high-quality materials science textual data available in the open literature, and HoneyBee is the first billion-parameter language model specialized to materials science. In MatSci-Instruct we improve the trustworthiness of generated data by prompting multiple commercially available large language models for generation with an Instructor module (e.g. Chat-GPT) and verification from an independent Verifier module (e.g. Claude). Using MatSci-Instruct, we construct a dataset of multiple tasks and measure the quality of our dataset along multiple dimensions, including accuracy against known facts, relevance to materials science, as well as completeness and reasonableness of the data. Moreover, we iteratively generate more targeted instructions and instruction-data in a finetuning-evaluation-feedback loop leading to progressively better performance for our finetuned HoneyBee models. Our evaluation on the MatSci-NLP benchmark shows HoneyBee’s outperformance of existing language models on materials science tasks and iterative improvement in successive stages of instruction-data refinement. We study the quality of HoneyBee’s language modeling through automatic evaluation and analyze case studies to further understand the model’s capabilities and limitations. Our code and relevant datasets are publicly available at https://github.com/BangLab-UdeM-Mila/NLP4MatSci-HoneyBee.

## introduction natural language processing (nlp) holds considerable promise in expediting the discovery and un- * equal advising. † corresponding author. canada cifar ai chair. 1 https://github.com/banglab-udem-mila/nlp4matscihoneybee derstanding of novel material systems, which will be crucial for addressing contemporary societal challenges like climate change and drug discovery. the potential impact of nlp in materials science is chiefly underpinned by the vast reservoir of materials science knowledge contained in text-based resources, such as textbooks, scientific journals, and assorted reports. in spite of the prospective richness of materials science textual data available from diverse sources, a number of challenges continue to significantly hinder the effective digestion and comprehension of relevant materials science textual knowledge @xcite . some of the challenges relate to the general availability of data, while other relate to the ability to effectively process domain-specific information, such as chemical notation and data contained in figures and tables @xcite . this scarcity of readily accessible, high-quality text corpora suitable for efficient language model training has in turn slowed the development of comprehensive language models capable of spanning the extensive conceptual range within the highly interdisciplinary materials science field. while data availability remains an ongoing challenge in applying modern nlp tools for materials science, recent advancements have led to the emergence of large language models (llms) proficient in handling general language tasks that concurrently demonstrate substantial aptitude in areas like chemistry and materials science @xcite . such advancements provide the potential to harness the implicit knowledge encapsulated in these models, which have been trained on vast text corpora spanning a broad range of subjects, to generate accessible, instruction-based datasets for specialized domains like materials science. yet, while we can generate targeted instructionbased data to make applying nlp for materials science more accessible, the quality of these instructions requires rigorous evaluation before be-figure 1 : example instruction generated by the matsci-instruct process used to train the honeybee language model that contains general language knowledge and is specialized in materials science. the relevant text to correctly answer the instruction is highlighted. matsci-instruct follows a structured instruction generation template and ensures instruction quality through an iterative verification loop described in section 3.1. aside from data scarcity in scientific domains, another significant impediment to the application of nlp in materials science is the limited presence of specialized language models that incorporate both in-depth materials science knowledge and a robust understanding of general language. the bulk of today's available language models for materials science are built on the bert architecture @xcite @xcite , whose performance in general nlp tasks has been superseded by several more advanced language model architectures in recent years @xcite @xcite . this highlights the need for the development of more capable language models in materials science that can accommodate a broader knowledge base while effectively performing pertinent materials science language tasks. this paper seeks to concurrently address the previously outlined challenges of trustworthy instruction generation and capable, open-source language models for materials science. we propose matsci-instruct to generate reliable, instruction-based data from large language models. this data is then used to train honeybee, a billion-parameter specialized materials science language model based on the llama architecture @xcite . the key contributions of our research are as follows: • honeybee -a high-performance llama-based model progressively trained via matsci-instruct: utilizing the matsci-instruct two-step framework, we apply a progressive refinement-feedback strategy to finetune a llama model, culminating in the honeybee model. in our progressive finetuning strategy, the honeybee model's performance on matsci-instruct data guides subsequent instruction-data generation. this iterative process results in further refined instructions to generate higher-quality instructiondata for finetuning, ensuring the progressive acquisition of specialized knowledge by the model. we evaluate the performance of hon-eybee using a materials science language benchmark @xcite , and thoroughly analyze its strengths and limitations. nlp for materials science nlp applications within materials science are constrained by the dual shortage of openly accessible, high-quality data and high-performing language models. while strides have been made towards enhancing data availability @xcite @xcite , the primary focus has been on generating expertannotated data for finetuning bert-based models, which lack the advanced capabilities of contemporary llms. for a detailed review of the performance of various bert models on materials science language tasks, we refer the reader to @xcite . the prevailing scarcity of data and specialized llms in materials science motivates us to propose matsci-instruct, an instruction-based method for data creation, and honeybee, a specialized llm tailored for materials science. instruction finetuning llms llms consistently demonstrate substantial improvements when finetuned for specialized tasks, as seen with biomedical models like chatdoctor @xcite and huatuo @xcite . while the large model size of llms poses a challenge for effective finetuning, several efficient methods have been proposed @xcite , such as p-tuning @xcite , prefix tuning @xcite , prompt tuning @xcite , and lora @xcite . among these, lora utilizes low-rank matrix decomposition to limit the additional parameters required for fine-tuning. for data curation in specialized fields, instructionsbased fine-tuning extracts detailed data directly from llms @xcite , reducing human annotation effort and providing scalable solutions. for example, alpaca @xcite exploits llms to generate synthetic instructions for model finetuning. however, llmsynthesized data still suffer from data quality issues, which is especially critical for scientific domains. to address these concerns, we design a generationverification strategy for trustworthy data generation and a progressive refinement-feedback strategy for finetuning llms on specialized instructions. ## method our work consists of two interacting components: 1) matsci-instruct: a trustworthy instruction generation framework for obtaining scientific textual data from llms; 2) honeybee: a materials science llm progressively finetuned from llama @xcite using matsci-instruct generated data. we connect honeybee to matsci-instruct with a refinement-feedback loop to progressively generate new data and finetune honeybee based on its training status as shown in figure 2 . ## experiments our experiments focus on assessing the ability of matsci-instruct to create high-quality, trustworthy instruction-data relevant to materials science, as described in section 3.1, along with understanding the capabilities and limitations of honeybee. ## conclusion in this work, we introduce matsci-instruct, an iterative instruction generation method for materials science, and honeybee, a state-of-the-art large language model for materials science. to the best of our knowledge, honeybee is the first billionparameter scale language model that is specialized in materials science. honeybee outperforms current state-of-the-art general language models (llama, alpaca) and materials science bertbased language models (matbert, matscibert) in various materials science nlp tasks, and note honeybee's performance improvement with each successive matsci-instruct stage. matsci-instruct provides a valuable framework for generating instruction-data to progressively finetune llms where instruction-data from an instructor are verified by a verifier before being used for finetuning. additionally, poor instruction-data is refined based on feedback from an evaluator leading to higher quality instruction-data and model performance for the desired specialization as shown by the results in section 4. future work remains in augmenting materials science llms with external knowledge, such as known scientific facts, which can further improve an llm's reliability and interpretability. limitations while honeybee outperforms current state-of-theart methods in various materials science nlp tasks, it remains unclear how well honeybee would generalize the tasks outside of the matsci-nlp benchmark and matsci-instruct instruction-data to solve complex materials science challenges. such challenges may include creating a synthesis recipe for a new materials or explaining the behavior of a materials system based on fundamental scientific concepts. materials science remains a wide-ranging and complex field with many open questions remaining on the true capability of honeybee and other llms to understand important materials science concepts. matsci-instruct also relies on the availability of highly performant llms to serve as the instructor, verifier and evaluator which can be limited in their own capabilities. furthermore, our work focuses primarily on the materials science domain and further studies are needed to understand how applicable our methods would be to additional scientific domains.