LLM s for B ayesian Optimization in Scientific Domains: Are We There Yet? (2025)

Abstract: Large language models (LLMs) have recently been proposed as general-purpose agents for experimental design, with claims that they can perform in-context experimental design. We evaluate this hypothesis using open-source instruction-tuned LLMs applied to genetic perturbation and molecular property discovery tasks. We find that LLM-based agents show no sensitivity to experimental feedback: replacing true outcomes with randomly permuted labels has no impact on performance. Across benchmarks, classical methods such as linear bandits and Gaussian process optimization consistently outperform LLM agents. We further propose a simple hybrid method, LLM-guided Nearest Neighbour (LLMNN) sampling, that combines LLM prior knowledge with nearest-neighbor sampling to guide the design of experiments. LLMNN achieves competitive or superior performance across domains without requiring significant in-context adaptation. These results suggest that current open-source LLMs do not perform in-context experimental design in practice and highlight the need for hybrid frameworks that decouple prior-based reasoning from batch acquisition with updated posteriors.

## introduction experimental design enables data-efficient scientific discovery @xcite @xcite @xcite . in domains such as biology and chemistry each experiment can be costly or time-consuming so we need methods that select the most informative candidates under a constrained budget. classical approaches typically adopt a bayesian framework @xcite : they construct a surrogate model of the response function, update this model with new data, and select future experiments by optimizing an acquisition function such as expected improvement or information gain @xcite . recent work has proposed using llms to replace these components, offering a unified interface that can incorporate prior knowledge, reason over experimental history, and directly select candidates via prompting. examples include biodis-coveryagent (bda) @xcite for genetic perturbation design and llambo @xcite for hyperparameter optimization. these systems are built on proprietary llms (e.g., claude 3.5 sonnet @xcite or openai's gpt-3.5 @xcite and o1 @xcite ) and design experiments by iteratively prompting the llm with prior experimental outcomes, and rely on in-context learning to guide future experiment selection. in domains that are well-studied, the priors from the llms offer clear benefits over the traditional methods for the first round of experimentation because the llms can leverage information from the pretraining corpus to select actions. however, it is less clear whether in-context learning alone (without finetuning) leads to good action selection. this paper investigates whether such llm-based approaches, when implemented with open-source models and without external tool use, can perform effective experimental design. specifically, we ask, do off-the-shelf, instruction-tuned llms exhibit strong in-context experiment design abilities when prompted with experimental history? we address this question using the biodiscov-eryagent pipeline implemented with three publicly available llms-llama-3.1-8b-instruct @xcite , qwen-2-7b-instruct @xcite , and qwen-2.5-14b-instruct @xcite -and two closed-source llms-claude 4 sonnet @xcite and gpt 4o-mini (openai, 2024a)-applied to experimental design tasks in two domains: single-gene per-turbation and molecular property prediction. to evaluate in-context learning behavior, we conduct ablation studies that compare the standard biodis-coveryagent to a variant receiving randomly permuted feedback (bda-rand), removing any correspondence between past actions and outcomes. our experimental analysis reveals that: ## problem statement consider the following real-world experimental settings across two domains: single gene perturbation. let's say we have the ability to knock out a single gene at a time from the human cell to induce a certain desired phenotypic response. correspondingly, for every knockout, we are able to measure the phenotypic response value of perturbing the gene. however, exhaustive trials with the whole set of protein-coding genes (> 19000) are infeasible due to both time and budget constraints. thus, how do we design perturbation experiments to identify genes that produce the phenotype under tight budget constraints and a sequential experiment protocol? ## tasks and datasets we perform experiments on two scientific domains in this work. in particular, we work with single gene perturbations and chemical properties of molecules. we simulate running experiments on a set of candidates by retrieving the corresponding measurement value from a tabular dataset. however, in real-world settings, the corresponding experiments would be conducted in a laboratory or an in-silico simulator. in the subsequent subsections, we describe the datasets used in the two domains. ## llmnn: a hybrid experiment design method in light of the above observations, we propose llm guided nearest neighbour (llmnn) framework, a simple greedy approach that prompts an llm to guide the location of cluster centers and leverages nearest neighbour sampling to form the batch b i at any round i. the llmnn framework is designed to leverage: 1.) generalist llms' intrinsic domain knowledge to guide the search cluster centers + feedback human/lab oracle n rounds + experiment protocol + task instructions + dataset information + expected output format prompt language model selected candidates candidate memory nearest neighbour sampling cluster center nearest unexplored neighbours figure 2: schematic workflow of the llm-guided nearest neighbour (llmnn) method. the red rounded boxes are the inputs and outputs of the language model. for ease, only cluster centers have been shown in the output. please refer to section 6 for detailed output specification and information about the components of the method. refer to appendix a for the detailed prompt template and to appendix c for a full trace of the method. in the vast candidate space, and 2) the inductive bias that similar genes/molecules have similar functionalities/properties. the schematic workflow of the proposed llmnn framework is illustrated in figure 2 . below, we discuss the important components of the llmnn framework, followed by a detailed workflow of the method. candidate memory. stores the pool of candidates, each represented in the following format: {"name": <candidate name>, "score": <measurement value>, "explored": <@xmath0. for molecules, the candidate name is the smiles string and measurement is the property value, whereas for genes, the candidate name is the gene name in the hgnc nomenclature and measurement is the phenotypic response value. each candidate is also associated with an embedding. the explored key tracks whether a candidate has already been selected, preventing repetition in the optimization process. the memory interacts with the llmnn framework by receiving a query candidate name and returning the nearest unexplored neighbors based on distance in the embedding space. candidate embeddings. each candidate in the memory has an associated embedding. we experiment both with llm embeddings for the candidates and also domain-specific embeddings like achilles embeddings for genes @xcite and molformer @xcite embeddings for the smiles strings. the distance metric for achilles embeddings is the cosine distance, whereas for all other embeddings, the l2-squared distance is used. more details about the embeddings are included in appendix b.3 agent response format: we borrow the agent response format from biodiscoveryagent @xcite , and direct the llm to structure its responses into several parts: reflection, research plan, solution, similar to @xcite . through the reflection and research plan entries, the model is able to articulate its reasoning behind a particular prediction. solution contains a list of n c cluster centers to sample around next. for our experiments, we fix n @xmath1. workflow. the exact flow of the llmnn framework is defined as follows: ## does llmnn do any better on our experiment design domains? we observed biodiscoveryagent struggled to outperform the traditional baselines (when provided with the same priors via access to embeddings) on the genetic perturbation datasets, and that the performance was largely the result of having better priors. given this, one would expect better performance from llm-based approaches in settings with well-studied candidate spaces-e.g. genes in the human genome-compare with much larger spaces-e.g. all of chemical space. llmnn addresses the poor response to exper-imental feedback by taking advantage of the fact that llms encode candidates into a common embedding space. this allows us to greedily construct experimental batches by performing nearest neighbor sampling in the neighborhood of previously observed hits. llmnn requires minimal assumptions of llm generating valid gene names as per hgnc nomenclature and valid smiles strings, which is reasonable for modern-day llms that have been pretrained on an internet-scale of knowledge. further, llmnn is an attempt at exploiting the known inductive biases of the domain to achieve higher performance. method. we experiment with two variants of method ion. e. esol freesolv g. truth (| c gt |) 1156 112 64 random 77.8 29 14.4 coreset 151.6 57.6 20.4 linear ucb 283 76 39 gp 151.39 35 16.8 llama-3.1-8b backbone llmnn noexp 152.8 23.4 12.2 llmnn 103.2 22.2 10.2 qwen-2-7b backbone llmnn noexp 147.2 24 25.8 llmnn 134.8 36.2 23 qwen-2.5-14b backbone llmnn noexp 185.6 33.6 17.2 llmnn 136.2 32 12.6 claude 4 sonnet backbone llmnn noexp 173.3 60.6 43.6 llmnn 189.6 63.8 38 gpt 4o-mini backbone llmnn noexp 179.8 27.4 34.2 llmnn 119.8 31 29.8 results. given the strong performance of the llmnn method, we perform an ablation with llm being replaced by a random centroid selector in the method. our goal is to observe if llm guidance plays any role in the superior performance of our method or the gains are purely due to nearest neighbour sampling in a strong embedding space. tables 5 and 6 show the result comparing the ablation variant with the best-performing llmnn noexp variant, i.e., the one based on the claude 4 sonnet. we observe that the performance drops sharply, by over 50% in some cases, when removing llm guidance, underscoring its central role in deciding the centroids for nearest neighbour sampling. further, we also investigate if the gains from our method on molecular domains hold with a general-purpose llm embedder like openai's text-embedding-3-large model that isn't directly trained on smiles strings. as shown in ## conclusion in this work, we set out to critically examine this hypothesis using instruction-tuned open-and closedsource llms. we particularly focused on biodis-coveryagent and experimented on two domains: single gene perturbations (5 datasets) and molecular property optimization (3 datasets). when compared to classical methods like linear ucb and gaussian process that used the embedding from the same llm as bda, the open-source bda performed significantly worse. furthermore, on deeper experiments with random feedback to the llm's context, the open-source and claude 3.5 sonnet-based bda still retained a similar average performance, showcasing that the llms trained on next token prediction and rlhf do not leverage experimental feedback in the design of their experiments. finally, we proposed an llm-guided nearest neighbour framework (llmnn), a simple combination of llm and a classic nearest neighbour method. llmnn outperforms bda significantly on gene datasets and is applicable to molecular domains, and performs at par with the classical baselines, except on some molecular datasets. overall, this work suggests that more work is needed to effectively incorporate experimental feedback into llm-based experimental design pipelines. it introduces a plausible avenue for future research marked by the synergy of llms, classical methods, and domain-specific inductive biases.