Large Language Models for Automated Open-domain Scientific Hypotheses Discovery (2024)

Abstract: Hypothetical induction is recognized as the main reasoning type when scientists make observations about the world and try to propose hypotheses to explain those observations. Past research on hypothetical induction is under a constrained setting: (1) the observation annotations in the dataset are carefully manually handpicked sentences (resulting in a close-domain setting); and (2) the ground truth hypotheses are mostly commonsense knowledge, making the task less challenging. In this work, we tackle these problems by proposing the first dataset for social science academic hypotheses discovery, with the final goal to create systems that automatically generate valid, novel, and helpful scientific hypotheses, given only a pile of raw web corpus. Unlike previous settings, the new dataset requires (1) using open-domain data (raw web corpus) as observations; and (2) proposing hypotheses even new to humanity. A multi-module framework is developed for the task, including three different feedback mechanisms to boost performance, which exhibits superior performance in terms of both GPT-4 based and expert-based evaluation.To the best of our knowledge, this is the first work showing that LLMs are able to generate novel (”not existing in literature”) and valid (”reflecting reality”) scientific hypotheses.

## introduction logical reasoning is central to human cognition @xcite . it is widely recognized as consisting of three components, which are deductive, inductive, and abductive reasoning @xcite . hypothetical induction is considered to be an important sub-type of inductive reasoning @xcite . it is recognized as the main reasoning type when scientists make observations about the world and try to propose hypotheses to explain the observations. 1 dataset, code, and generated hypotheses are available at https://github.com/senticnet/moose . for example, the proposal of geocentrism, heliocentrism, and newton's law of universal gravitation based on the observations of the motion of (celestial) objects can be seen as a result of hypothetical induction. hypothetical induction is a process of knowledge exploration from observations to hypotheses: it is challenging because it involves the exploration of knowledge that is even new to humanity. recent research on this has two main limitations @xcite . firstly, the observations in their dataset have already been manually selected from the raw web corpus, resulting in a close-domain setting. as a result, a developed system for this dataset relies on already manually selected observations, and cannot utilize the vast raw web corpus to propose hypotheses. secondly, the ground truth hypotheses are mostly commonsense knowledge (e.g., newton's law), making the task less challenging since llms might have already seen them during pretraining. to this end, we propose a new task setting of hypothetical induction, which is to generate novel and valid research hypotheses targeting being helpful to researchers while only given (vast) raw web corpus (figure 1 ). this hypothesis formation process is seen as the first step for scientific discovery @xcite . we call this task as "automated open-domain hypothetical induction (tomato)". it is "automated" since a method for this task should automatically propose hypotheses with few human efforts; it is open-domain since it is not restricted by any manually collected data. for the tomato task, we constructed a dataset consisting of 50 recent social science papers published after january 2023 in top social science journals. for each paper, social science experts collect its main hypothesis, identify its background and inspirations, find semantically similar contents for its background and inspirations from the web corpus, collect the full passage for each matched content, and use all collected web passages as raw web corpus. although the new dataset involves many manual selection processes, the manually selected contents are used more as benchmarking human performance for comparison. in the tomato task, a method is required to only utilize the raw web corpus in the dataset to propose hypotheses. in addition, the raw web corpus is mostly from common news, wikipedia, and business reviews, which means it can easily expand in scale without much human involvement. to tackle the tomato task, we develop a multimodule framework called moose based on large language model (llm) prompting (figure 4 ). to further improve the quality of the generated hypotheses, we also propose three different feedback mechanisms (present-feedback, past-feedback, and future-feedback) to use llms to retrospect and improve the llm-generated hypotheses for better quality. for present-feedback, the intuition is that, for some modules, their generation can be evaluated by other llms and be provided with feedback, which can be utilized by the modules to refine their generation by taking the feedback and previous generation as input and generating again. some modules can have feedback instantly after their generation to improve themselves. but just like the reward mechanism in reinforcement learning, some rewards (feedback) might be hard to obtain instantly, but need to wait for feedback for a future module. similarly, we develop past-feedback where a module can benefit from the feedback for a future module. the last one is future-feedback, where a current module can provide justifications for the current module's generation to help a future module's generation, or can provide some initial suggestions which a future module can build upon to further provide more in-depth generation. both @xcite evaluation and expert evaluation indicate that moose performs better than an llm @xcite based baseline, and each of the three feedback mechanisms can progressively improve the base framework. during expert evaluation, many hypotheses generated by moose are recognized by social science researchers to be both novel ("not existing in the literature") and valid ("reflecting reality"). to the best of our knowledge, this is the first work showing that llms can be leveraged to generate novel and valid research hypotheses, indicating the potential for llms to serve as a "copilot" for scientists. ## dataset collection in this section, we take one publication @xcite in our dataset as an example to illustrate the dataset collection process. in total, there are 50 papers published after january 2023. most social science publications highlight their hypotheses. figure 2 shows our selected main hypothesis in the example publication. the research backgrounds are given in the introduction section. in this example paper, the background is about facial recognition payment technology's usage in society. most social science publications also have a "hypothesis development" section (some may call it by other names, e.g., "theoretical development"). for example, the left part ("hypothesis development") in figure 3 shows the title of this section in the example paper. in this section, several theories used to develop the main hypothesis are separately introduced. usually, each theory takes one subsection. for example, the right part ("herding effect") in figure 3 shows the title of a subsection, which is a particular theory being used as an inspiration, which with the background can develop the hypothesis in figure 2 . for each publication in our dataset, we identify its main hypothesis, research background, and in-social science spirations, where the background and inspirations together provide enough information to be possible to develop the hypothesis. we also abstract the reasoning process from background and inspirations to hypothesis and note it down for each publication in our dataset. in this selected example, the reasoning process is easy, but it has medium difficulty for researchers to associate the inspiration (herding effect) to the background. for each publication, we include an expert-evaluated complexity for both the reasoning process and the association of the inspiration to the background (details in §a.3). instead of directly copying the background and inspirations from the paper to construct the dataset, we try to find semantically similar text contents from the web corpus as a substitution to avoid data contamination and fit the requirement of tomato task that a system should propose novel and valid research hypotheses only given raw web corpus. in the example paper, we find news sentences reporting the usage of facial recognition payment as ground truth background and a wikipedia description of the herding effect as ground truth inspiration. we also collect the web link and the full text of the manually selected web passages for backgrounds and inspirations to be used as raw web corpus. in addition, we collect the link and the publication date for all fifty papers. we also collected fourteen survey papers in related fields that might help check the novelty of the hypotheses. the dataset is fully constructed by a social science phd student. we illustrate why the dataset shouldn't be collected by automatic methods in §a.4. ## methodology in general, our method consists of a base multimodule framework and three feedback mechanisms (past-feedback, present-feedback, and futurefeedback). we call the full framework as multi-module framework with past present future feedback (moose). the base framework without any feedback is called moose-base. moose is described in figure 4 and algorithm 1. ## analysis 6.1 background and inspirations here we try to answer "is chatgpt necessary for background and inspiration selection?". base model is gpt-3.5. ## conclusion in this paper, we propose a new task, automated open-domain hypothetical induction (tomato), which is the first task in nlp to focus on social science research hypotheses discovery. along with the task, we construct a dataset consisting of 50 recent social science papers published in top academic journals. we also developed a multi-module framework moose for the tomato task, which contains a base framework and three novel feedback mechanisms. experiments indicate that moosebase outperforms an llm-based baseline, and the three feedback mechanisms can progressively fur-ther improve over moose-base. surprisingly, evaluated by phd students, moose is able to produce many novel ("not existing in the literature") and valid ("reflecting reality") research hypotheses. to the best of our knowledge, this is the first work showing that llms can be leveraged to generate novel and valid scientific hypotheses, indicating the potential of llms to serve as a "copilot" for scientists.