Hypothesis Generation with Large Language Models (2024)

Abstract: Effective generation of novel hypotheses is instrumental to scientific progress. So far, researchers have been the main powerhouse behind hypothesis generation by painstaking data analysis and thinking (also known as the Eureka moment). In this paper, we examine the potential of large language models (LLMs) to generate hypotheses. We focus on hypothesis generation based on data (i.e., labeled examples). To enable LLMs to handle Long contexts, we generate initial hypotheses from a small number of examples and then update them iteratively to improve the quality of hypotheses. Inspired by multi-armed bandits, we design a reward function to inform the exploitation-exploration tradeoff in the update process. Our algorithm is able to generate hypotheses that enable much better predictive performance than few-shot prompting in classification tasks, improving accuracy by 31.7% on a synthetic dataset and by 13.9%, 3.3% and, 24.9% on three real-world datasets. We also outperform supervised learning by 12.1% and 11.6% on two challenging real-world datasets. Furthermore, we find that the generated hypotheses not only corroborate human-verified theories but also uncover new insights for the tasks.

## introduction hypothesis generation drives scientific progress. mendel's hypothesis on allele pairs lays the foundation for modern genetics; einstein's hypothesis in general theory of relativity led to the prediction and subsequent confirmation of gravitational waves. in the context of language modeling, the hypothesis on scaling law inspires recent progress in large language models (llms) @xcite . despite the importance of hypothesis generation, as @xcite point out, science has been curiously asymmetric. while many scientific publications present extensive formal and empirical evaluation of hypotheses, the generation of hypotheses happens off-stage by researchers. in order to generate novel hypotheses, researchers may read literature, analyze data, pick the brain of each other, and even "hallucinate" (see kekulé's discovery of the structure of the benzene molecule @xcite ). given the rise of large language models @xcite @xcite , we examine their potential of providing much needed assistance in hypothesis generation in this work. in particular, we focus on hypothesis generation based on data, a common approach in empirical sciences. our main question is how we can enable llms to generate hypotheses of high-quality. while one can easily prompt llms to generate hypotheses, llms may not be able to effectively leverage the input examples in a single long prompt. moreover, it is important to have measures of quality in the generation process so that we can filter bad hypotheses and come up with better ones. these two observations motivate us to start with a setup analogous to supervised learning. we can iteratively prompt an llm to generate hypotheses based on the training examples and use training accuracy as a measure of quality to guide the generation process. conveniently, we can also evaluate the quality of the final generated hypotheses with their performance on held-out examples, similar to supervised learning. to generate high-quality hypotheses with llms, we propose an algorithm inspired by the upper confidence bound algorithm in multi-armed bandits @xcite (hypogenic foot_0 , hypothesis generation in context; see figure 1 ). given initial hypotheses generated from a small number of examples, we need to assess their quality and propose new hypotheses to address their deficiencies. to navigate this exploration-exploitation tradeoff, we introduce a reward function and evaluate the top k hypotheses for each training example. we maintain a wrong example bank to capture the gap in knowledge of the hypotheses pool, and generate new hypotheses based on the wrong example bank to close the gap. the generated hypotheses naturally enable an interpretable hypothesis-based classifier. we propose a suite of inference strategies given a set of hypotheses. we apply our method to one synthetic task where there is a single known valid hypothesis and three real-world tasks (deceptive reviews, headline popular-ity, and tweet popularity). the real-world tasks focus on deception detection and message popularity prediction, which are known to be challenging even for humans @xcite . our al-new hypothesis: tweets with named entities like people, places, or organizations tend to get more retweets by being more specific. @xmath0.83 tweet 1: how do preseason stars fare during the regular season? [link] some scoring stats there. tweet 2: in the last decade, melo and kobe are the only two to lead the nba in scoring in both preseason and regular season. ## method we begin with a description of the problem formulation. given a set @xmath1, y 1 ), . . . , (x n , y n )} where x i is an example and y i is the corresponding label, the goal is to learn a set of hypotheses @xmath2, ..., h m } that describe theories of relationships between x and y. to this end, we prompt an llm to summarize demonstration examples into high-level hypotheses ( § 2.1). then, during inference, the llm makes inference based on the generated hypothesis ( § 2.2). ## experiment setup we introduce the experiment setup to evaluate hy-pogenic. ## results to demonstrate the effectiveness of our hypothesis generation approach, we present results via three evaluation methods. first, we show that in the standard supervised learning setup, our generated hypotheses enable more accurate predictions than baselines and even oracles when using a small set of examples. second, we evaluate the generated hypotheses by checking whether they can generalize across different inference llms and to out-of-distribution datasets. we find surprisingly consistent performance even when using a different llm to make inference from the generated hypotheses. so, we conduct a qualitative analysis to show that the generated hypotheses not only corroborate existing theories but also provide novel insights about the tasks at hand. ## conclusion & further discussion in this work, we propose hypogenic, a novel datadriven and automated method that leverages llms to generate hypotheses with the goal of discovering unknown knowledge. with hypogenic, we are not only are able to generate human-interpretable hypotheses but also achieve better predictive performance against competitive baselines and even oracles. furthermore, our method can generalize well with different models and datasets, including open models. notably, with our generated hypotheses, we uncover new insights in realworld tasks that are widely studied in social sciences. the key to success in hypogenic is not that llms remembers the correct hypotheses, but lies in their ability to "hallucinate" and combine potentially relevant concepts. the exploration-exploitation process then identifies the valuable hypotheses. hypogenic can be directly applied to complex social science tasks. we encourage future work to explore hypothesis generation that requires additional modalities and/or leverages existing literature along with past observations.