H oney C omb: A Flexible LLM -Based Agent System for Materials Science (2024)

Abstract: The emergence of specialized large language models (LLMs) has shown promise in addressing complex tasks in materials science. Many LLMs, however, often struggle with the distinct complexities of materials science tasks, such as computational challenges, and rely heavily on outdated implicit knowledge, leading to inaccuracies and hallucinations. To address these challenges, we introduce HoneyComb, the first LLM-based agent system specifically designed for materials science. HoneyComb leverages a reliable, high-quality materials science knowledge base (MatSciKB) and a sophisticated tool hub (ToolHub) tailored specifically for materials science to enhance its reasoning and computational capabilities. MatSciKB is a curated, structured knowledge collection based on reliable literature, while ToolHub employs an Inductive Tool Construction method to generate, decompose, and refine API tools for materials science. Additionally, HoneyComb leverages a retriever module that adaptively selects the appropriate knowledge source or tools for specific tasks, thereby ensuring accuracy and relevance. Our results demonstrate that HoneyComb significantly outperforms baseline models across various tasks in materials science, effectively bridging the gap between current LLM capabilities and the specialized needs of this domain. Furthermore, our adaptable framework can be easily extended to other scientific domains, highlighting its potential for broad applicability in advancing scientific research and applications.

## introduction the emergence of large language models (llms) @xcite @xcite in recent years has brought about the application of llms across a wide range of fields related to science and engineering @xcite . this has resulted in a number of new benchmarks measuring the capabilities of language models to perform scientific tasks @xcite @xcite along with the development of custom llms and llm-based systems for scientific domains, including chemistry @xcite , biology @xcite and materials science @xcite @xcite . while much progress has been made in adapting llms to common tasks in natural language processing @xcite , many more challenges remain in having llms be effective agents for real-world materials science tasks @xcite . as highlighted by @xcite , llms often fail in performing advanced tasks for materials science. common mistakes by most llms include conceptual errors where models fail to retrieve correct concepts, equations, or facts relevant to the questions, and factual hallucinations where incorrect information is generated. an analysis by @xcite also revealed that llms by themselves struggle to generate relevant and correct information pertaining to specialized materials science tasks. while @xcite showed that instruction fine-tuning can help in improving performance, the high costs of continuous model training and fine-tuning make retraining-based approaches challenging to scale. moreover, this challenge is further compounded by the fact that relevant knowledge is continuously updated from diverse sources-including pre-print servers (e.g., arxiv and chemrxiv), peer-reviewed literature, open encyclopedias like wikipedia, and various relevant websites. furthermore, prior work has shown that utilizing external tools may be a more promising approach to solve complex scientific tasks rather instead of relying entirely on an llm's internal knowledge @xcite . to jointly address these challenges, we propose transforming llms into llm-based agents that access external knowledge and tools to boost their performance. this approach has already shown promise in adjacent domains, such as chemistry @xcite , by enabling models to access real-time data and utilize both general and domain-specific tools. altogether, the llm-based agents showcase greater capabilities and performance compared to their native llm counterparts. in this paper, we present honeycomb, the first, to the best of our knowledge, llm-based agent system specifically designed for materials science. while there has been emerging research in llms for scientific domains, few studies have focused on developing comprehensive agent systems for materials science. our work addresses two critical challenges: first, matscikb alleviates the challenge of obtaining reliable and relevant professional knowledge for materials science. as such, matscikb ensures the agent has access to the most current and accurate information, which is essential for effective performance. second, toolhub provides materials science-specific tools to augment the agent's capabilities. these tools enable the agent to perform specialized computational tasks and enhance its overall functionality. as detailed in section 4, we observe that with the aid of matscikb and toolhub, honeycomb outperforms its native llm counterparts in a more reliable way, given its ability to utilize up-to-date knowledge and tools. ## honeycomb in this work, we introduce honeycomb, a specialized agent system designed to advance materials science research, as shown in figure 1 . it integrates three key components: 1) matscikb, a comprehensive knowledge base; 2) toolhub, which includes general tools for broadly accessing up-to-date information and specialized tools developed through the inductive tool construction method for targeted materials science queries; and 3) retriever, which utilizes a hybrid approach for efficient and precise information retrieval. ## experiments we conduct experiments on two questionanswering datasets, namely mascqa @xcite and sciqa @xcite , to investigate the ability of honeycomb in materials science tasks. mascqa, derived from the graduate aptitude test in engineering (gate) in india, is tailored to reflect the real-world complexity and variety of issues encountered in materials science. this highly competitive examination assesses a comprehensive understanding of various undergraduate subjects (indian institute of technology kanpur, 2023; @xcite . with its 650 questions covering 14 domains, such as thermodynamics, atomic structure, and mechanical behavior, the dataset includes a wide range of question types: multiple choice questions (mcqs), numerical answer type (num), matching type (match), and mcqs with numerical options (mcqn). specifically designed for advanced problem-solving, this dataset is crucial for ensuring that our toolhub functions effectively in real-world materials science research and applications. it demonstrates the efficacy and adaptability of the honeycomb framework in tackling complex materials science issues within realistic scenarios. the second dataset, sciqa, comprises 11,679 multiple-choice questions spanning the core disciplines of fundamental sciences from various crowdsourced science exams @xcite . this compilation not only emphasizes the dataset's comprehensive and interdisciplinary nature but also focuses on fostering a nuanced conceptual understanding. sciqa serves as a critical testbed to ascertain whether the honeycomb framework can augment the llm's capabilities beyond its initial programming. by integrating supplementary information, it aids in addressing intricate queries and unraveling complex scientific concepts that may have been overlooked during the initial training phase of the llm. by bridging real-world complexities with rigorous academic standards, these datasets ensure that our matscikb and toolhub are not only versatile but also remain at the forefront of technological and scientific application. the choice of models for our experiments was driven by the need to evaluate the honeycomb framework's enhancement capabilities across a spectrum of large language models known for their robust performance in diverse applications. we selected gpt-3.5, gpt-4 (openai, 2024), llama-2 @xcite , and llama-3 (ai@meta, 2024) due to their widespread use and proven effectiveness in handling complex language tasks. these models, with llama-2 and llama-3 having parameter sizes of 7 billion and 8 billion, respectively, represent the current state-of-the-art in generalized language understanding and provide a solid baseline for benchmarking. additionally, we included honeybee @xcite , a specialized model with a parameter size of 7 billion, tailored specifically for materials science. the in-clusion of both general-purpose and specialized models allows us to showcase how domain-specific adaptations through honeycomb can elevate a model's functional scope beyond its original configuration, thus highlighting the adaptability and effectiveness of our framework. ## conclusion in this work, we introduced honeycomb, a pioneering llm-based agent system tailored for materials science, consisting of three key components: a curated materials science knowledge base (matscikb), a dual-layered toolhub encompassing both general and specialized computational tools, and a precision-focused retriever module. these components are integrated to deliver accurate, real-time information and ensure reliable performance in advanced materials science tasks. experimental results demonstrate that honey-comb outperforms contemporary general-purpose models (e.g., gpt and llama series) and special-