M onte C arlo Thought Search: Large Language Model Querying for Complex Scientific Reasoning in Catalyst Design (2023)

Abstract: Discovering novel catalysts requires complex reasoning involving multiple chemical properties and resultant trade-offs, leading to a combinatorial growth in the search space. While large language models (LLM) have demonstrated novel capabilities for chemistry through complex instruction following capabilities and high quality reasoning, a goal-driven combinatorial search using LLMs has not been explored in detail. In this work, we present a Monte Carlo Tree Search-based approach that improves beyond state-of-the-art chain-of-thought prompting variants to augment scientific reasoning. We introduce two new reasoning datasets: 1) a curation of computational chemistry simulations, and 2) diverse questions written by catalysis researchers for reasoning about novel chemical conversion processes. We improve over the best baseline by 25.8% and find that our approach can augment scientist’s reasoning and discovery process with novel insights.

## introduction scientific discovery thrives on uncovering the optimal combinations of factors that maximize a property of interest. for example, to discover new efficient fuels @xcite @xcite or chemical conversion processes requiring less energy, a scientist would need to consider the chemical reaction, the reactants that undergo the reaction, the catalysts that improve the rate of reaction, and find the optimal combination of operating conditions (fig. 2 ). mathematically, one could represent this as an optimization problem where we model a chemical process as a function and formulate the search problem as finding the optimal combination of all process parameters that minimizes a cost function modeled around energy efficiency. for highly empirical fields such as chemistry, these combinatorial searches require 1 our codebase and datasets are freely available at https: //github.com/pnnl/chemreasoner the search begins with a generic query at the root node. the answer from each node is passed to the child nodes and additional criterion are added to the prompt. for instance, low cost. information passed to children nodes is color coded to show the reasoning pathway. despite their promise, the brittleness of language models to their inputs and hallucination remain areas for concern @xcite . our initial investigation of llms revealed that basic prompting (such as "what is a good catalyst for reaction x?") leads to basic answers that could be found on a wikipedia page. to improve the quality of answers, one can incorporate desirable properties into the prompt which lead the llm to produce more specific answers (such as "what is a good catalyst with low cost for reaction x?"). additionally, llms often hallucinate, producing answers without grounding in scientific fact. achieving accurate answers with high speci-figure 2 : illustration of the combinatorial thinking used by human experts to reason about a catalyst (best viewed in color). they successively "think in terms of" different constraints and factors, each of which are related via scientific principles, and narrow down the set of possible candidates. our monte carlo reasoner emulates such cognitive thinking by prompting a language model with different combinations, yielding a tree-structured space of queries and potential candidates, and returns the optimal answer via efficient exploration of the possible space. in this work, we focus on the problem of prompting an llm to return the top-k catalysts for a given chemical reaction and generating the reasoning for each candidate. in collaboration with researchers from the catalysis community, we develop a new dataset, biofuels question reasoning (biofuelqr), consisting of complex reasoning questions and answers. we observe the reasoning pathways used by domain scientists and conclude that it is important for llms to progress from "thinking step-by-step" to "thinking step-by-step in terms of relevant properties". in this setting, we are given a question which has some relevant catalyst properties p, |p| = n (e.g. {"crystal planes", "toxicity"}) and we want to identify the best subset r ⊂ p, |r| = r of properties for the language model to "think" in terms of. considering that language models are sensitive to permutations of their inputs, there are p n @xmath0. this goal can be accomplished by learning to prompt the llm with the most relevant subset of properties @xcite or decomposing the set into a sequence of chained queries @xcite . in both cases, identification of the prompt-generating property subset becomes the limiting factor. to solve this problem, we propose the monte carlo reasoner (mcr), a generic heuristic search methodology that addresses the combinatorial chal-lenge of query decomposition. considering the practical challenges of learning prompts that are both human comprehensible (a key consideration for scientists) and provide the best performance @xcite , we pursue a stochastic, heuristic search-based approach that leverages llms trained on scientific literature with sophisticated instruction following capabilities @xcite . we formulate the task as a search problem in which an agent performs a query in an uncertain environment (represented by the llm) and determines a query variant to pursue based on the evaluated reward. given an initial query, we construct a tree structure of these unique query variants in order to progressively refine the original query (the root) into property-specific variations (the leaves). our methodology demonstrates improvement over basic querying of the llm without any additional training of the llm. instead, we use a monte carlo tree search algorithm (mcts) to perform a stochastic search over the existing knowledge space of an llm to achieve more scientifically valuable answers. our second major contribution is demonstrating the efficacy of using a scientific domain-specific reward function in llm-based computations for our top-k catalyst problem. estimation of the "adsorption energy" of a chemical structure is at the core of developing efficient chemical reactions (see appendix a.2 for details). finding catalysts that can enable chemical reactions with the least amount of external energy is key to developing environmen-tally friendly industrial processes. in this work, we implement such energy function specific considerations via a llm-derived reward function. our experiments (using questions detailed in in mcts, each prompt p is stored as a node in a tree t , where edges are prompt-action pairs (p i , a j ). the search tree decides at each prompt which action to take to obtain the best reward based on previous results. typically, prompt-action pairs are weighted by a policy, which determines a-priori the importance of each action for a prompt, given as prior probabilities. here, we assign equal weight to all possible actions. impossible actions are assigned weight of 0 (see appendix c.2). in mcts, each edge stores a count n (p, a), a weight representing a prior probability p(p, a), and the total downstream reward v (p, a) where here, γ is a discount factor and d is the (tree) distance of p ′ from p . if there are no discovered successors to p , then we set v (p, a) = 0. the search determines the next action to take with policy p(v, n, p): ## experiments experimental setup we conduct our experiments on two new chemistry-focused reasoning query benchmarks containing 130 queries ( for an example). we collected two answers from catalysis researchers for a subset of 51 queries to observe different reasoning patterns and human biases. see section c for details on dataset design. baselines we benchmark mcr's performance with three recent methods: 1) chain-of-thought (cot) prompting @xcite , 2) selfconsistency-based cot @xcite , 3) breadth-first-search (bfs) based tree-of-thoughts (tot) @xcite (a contemporary work to ours). experiments are based on gpt-3 textdavinci-003 foot_0 . key takeaways we find that mcr's use of stochastic search prunes the more uniform exploration of search space conducted by @xcite . to confirm if the increased reward over cot indeed translates into better reasoning quality, two catalysis experts compared the best answer generated by mcr with the gpt-3 cot implementation. overall, the experts preferred mcr to cot (figures 8, 9). figures 5 6 7 8 9 illustrates the evaluation for one such query. review from both experts (figures 8, 9) deemed the cot response in figure 6 incorrect and mcr correct (figure 8 ). the experts also evaluated how the prompts and llm answers evolve as mcr searches deeper in the prompt tree (figures 3 and 10 )-in many cases they found the llm answers to be logically coherent and in some cases even insightful enough for follow-up experimentation (see the second user feedback in figure 3 ). overall, both experts preferred mcr for having higher specificity over cot and reasoning in terms of correct properties (detailed in figures 8 , 9 ). ## conclusion and future work llms offer major promise to automate the cycle of scientific hypothesis generation and testing. our work tackles the challenge of identifying key prop-erties for augmenting a chemist's reasoning via use of a domain-specific reward function, enabling generation of relevant scientific explanations with high specificity. mcr is a zero-shot reasoning methodology that circumvents the need for large-scale, hard-to-obtain, domain-specific training data. we apply it to catalyst research: a highly empirical, reasoning-heavy scientific field dating back a century. future work can investigate large-scale evaluation of our benchmark, integration with atomistic prediction models trained on quantum chemistry datasets for more trustworthy reward functions, and finetuned language models.