S ci A gent: Tool-augmented Language Models for Scientific Reasoning (2024)

Abstract: Scientific reasoning poses an excessive challenge for even the most advanced Large Language Models (LLMs). To make this task more practical and solvable for LLMs, we introduce a new task setting named tool-augmented scientific reasoning. This setting supplements LLMs with scalable toolsets, and shifts the focus from pursuing an omniscient problem solver to a proficient tool-user. To facilitate the research of such setting, we construct a tool-augmented training corpus named MathFunc which encompasses over 30,000 samples and roughly 6,000 tools. Building on MathFunc, we develop SciAgent to retrieve, understand and, if necessary, use tools for scientific problem solving. Additionally, we craft a benchmark, SciToolBench, spanning five scientific domains to evaluate LLMs’ abilities with tool assistance. Extensive experiments on SciToolBench confirm the effectiveness of SciAgent. Notably, SciAgent-Llama3-8B surpasses other LLMs with the comparable size by more than 8.0% in absolute accuracy. Furthermore, SciAgent-DeepMath-7B shows much superior performance than ChatGPT.

## introduction scientific reasoning @xcite aims to comprehend and make decisions regarding problems among stem (science, technology, engineering and mathematics) domains. it is a fundamental aspect of intelligence, a demanding capability of large language models (llms), and a notoriously challenging task. for instance, even @xcite achieves only 50% and 35% accuracy on theoremqa @xcite and @xcite , respectively. regarding open-source llms such as llama3 @xcite and mistral (jiang left: supervised fine-tuning setting. collecting annotations and fine-tuning llms domain by domain. right: our proposed tool-augmented setting. we decompose the scientific reasoning ability into three aspects: math reasoning ability, tool-use ability and domain-specific knowledge. llms are fine-tuned on math-related, toolaugmented samples (color in red). when adapting llms to a specific domain, a pluggable and domainspecific toolset is attached to provide the domain knowledge. no additional fine-tuning is further required. @xcite , the performances are only about 30% accuracy or even less. the challenge in scientific reasoning arises from the need for both mathematical (math) and domainspecific reasoning abilities. to address the physical problem in figure 3 , for example, it is necessary to both understand malus' law (domain knowledge) for analyzing the intensity of polarized light, and possess quantitative ability for calculating the light intensity ratios. a natural approach involves collecting annotations and fine-tuning llms to enhance their math and domain-specific reasoning abilities, as depicted in figure 1 (left). however, annotating scientific reasoning problems is extremely expensive. what is worse, adapting llms to a new domain demands a fresh round of annotation and fine-tuning, rendering this approach impractical. in this paper, we draw inspirations from tool learning @xcite to enhance llms' scientific reasoning capabilities. instead of solving scientific problem from scratch, humans have summarized and wrapped various points as generalized and well-documented functions in scientific computing softwares, such as matlab foot_0 , wolframalpha foot_1 , sympy foot_2 , etc. these functions foot_3 , which could be equivalently viewed as external tools, greatly facilitate math-adept users to solve difficult scientific problems. in analogy with humans, we do not pursue an omniscient solver across various scientific domains. instead, we assume the access to domainspecific toolsets and purse a unified, generalized llm-based tool-user as shown in the figure 1 @xcite . this approach tackles domain-specific reasoning challenges by enabling llms learn to use a reusable and scalable toolkit. it alleviates the reasoning challenges of llms by concentrating solely on enhancing their tool-use abilities. these abilities are not only easier to acquire but also applicable across a variety of scientific fields. by attaching domain-specific toolsets, our tool-users can be readily adapted to different fields without the need for additional in-domain fine-tuning. this work focuses on developing and benchmarking the ability of llms in scientific reasoning with the help of tools. we envision a scenario where llms have access to a domain-specific toolset, comprising various specialized functions. upon this scenario, we propose a complete framework of dataset construction, model training and evaluation. given a scientific question, llms are supposed to retrieve functions from the toolset and optionally incorporate functions into the formulated solution. we employ an automatic pipeline featuring gpt-4/-4o to compile a large-scale, mathrelated, tool-augmented training corpus named as mathfunc. this corpus is designed to enable llms to learn both essential math skills and how to retrieve, understand and use functions properly. as a result, mathfunc contains 31,375 samples and equipped with a toolset encompassing 5,981 generalized and well-documented functions. we detail this training corpus in section 3. we fine-tune open-source llms on mathfunc to develop tool-augmented agents named scia-gent detailed in section 4. as shown in figure 3 , sciagent firstly generate a high-level planning in response to a given question. the agents then use this plan, along with the question, to retrieve functions from the given toolset. leveraging these retrieved functions, the agents further complete the low-level action integrating natural language and python code. finally the agents execute the code to complete the problem at hand. to benchmark the tool-use abilities in scientific reasoning, we develop a new benchmark named scitoolbench as described in section 5. building upon theoremqa @xcite and @xcite , it has 4,250 questions covering five domains: mathematics, physical, chemistry, eecs, and finance. it also contains five domain-specific toolsets comprising a total of 2,285 functions. we evaluate sciagent on scitoolbench and another benchmark derived from creator-challenge @xcite . experimental results demonstrate that our agents present remarkable scientific reasoning capabilities. notably, sciagent-llama3-8b surpasses the best comparable open-source llms by an absolute 8.0% accuracy, and sciagent-deepmath-7b outperforms chatgpt by a large margin. we also conduct an extensive analysis of the benefits and limitations of sciagent series, providing valuable insights for future research. ## training corpus: mathfunc to our best knowledge, there are no readily available tool-augmented datasets in scientific reasoning domains. therefore, we construct a corpus named mathfunc teaching llms to better understand and use functions. mathfunc is composed of (1) a toolset f 5 including 5,981 generalized, well-documented, math-related functions and (2) a dataset d encompassing 31,375 samples in which solutions call the function from the toolset if necessary (e.g., 4 ⃝ in figure 2 ). we build this corpus based on math @xcite training set because we expect to teach llms both math skills and tool-use abilities. ## model: sciagent we develop sciagent for tool-augmented scientific reasoning task. it could make plan, retrieve functions, and leverage retrieved functions to facilitate the reasoning. we describe its inference procedure and training approach as below. ## benchmark: scitoolbench there currently exists no benchmark assessing the scientific reasoning capabilities of llms when aided by tools. to address this gap, we develop a benchmark called scitoolbench. our benchmark covers five domains: mathematics (math) foot_4 , physics, chemistry, finance, electrical engineering and computer science (eecs). each domain is composed of a set of questions and a domainspecific toolset. the toolset consists of abundant generalized, high-quality and well-documented functions. we expect llms to retrieve, understand and, if necessary, use functions in it for reasoning. ## experiments 6.1 setup we conduct experiments on scitoolbench to evaluate the tool-augmented scientific reasoning abilities of llms. we report results categorized by both question domains and construction methods for fine-grained analysis. we also employ cre-ation challenge @xcite as the second benchmark. it comprises 2,047 samples, with each sample consisting of a question and a ground-truth function. we re-purpose all functions to assemble a global toolset (thus including 2,047 functions). we report accuracy as the metric in all experiments. we also list the performance of chatgpt and gpt-4 for reference. for fair comparison, we provide all llms the same retriever in section 3.2 to retrieve functions from toolset (if attached). please see more details in appendix c. ## conclusion this work proposes tool-augmented scientific reasoning, a task aiming to solve challenging scientific problems aided by generalized and scalable tools. to facilitate and evaluate the scientific tooluse abilities of llms, we construct a math-related, tool-augmented training corpus mathfunc and a benchmark scitoolbench covering 5 scientific domains. additionally, we develop open-source agents, sciagent series, as competitive baselines. extensive experiments reveal that our agents exhibit tool-use abilities exceeding chatgpt in scientific reasoning tasks.