S ci MON : Scientific Inspiration Machines Optimized for Novelty (2024)

Abstract: We explore and enhance the ability of neural language models to generate novel scientific directions grounded in literature. Work on literature-based hypothesis generation has traditionally focused on binary link prediction—severely limiting the expressivity of hypotheses. This line of work also does not focus on optimizing novelty. We take a dramatic departure with a novel setting in which models use as input background contexts (e.g., problems, experimental settings, goals), and output natural language ideas grounded in literature. We present SciMON, a modeling framework that uses retrieval of “inspirations” from past scientific papers, and explicitly optimizes for novelty by iteratively comparing to prior papers and updating idea suggestions until sufficient novelty is achieved. Comprehensive evaluations reveal that GPT-4 tends to generate ideas with overall low technical depth and novelty, while our methods partially mitigate this issue. Our work represents a first step toward evaluating and developing language models that generate new ideas derived from the scientific literature. Code, data, and resources are publicly available for research purposes: https://github.com/eaglew/clbd.

## introduction can machines mine scientific papers and learn to suggest new directions? the idea that information from the literature can be used for automatically generating hypotheses has been around for decades @xcite . to date, the focus has been on a specific setting: hypothesizing links between pairs of concepts (often in drug discovery applications @xcite , e.g., new drug-disease links), where concepts are obtained from papers or knowledge bases previously derived from papers @xcite . this common setting has fundamental drawbacks. reducing the "language of scientific ideas" @xcite to this simplistic form limits the expressivity of the hypotheses we can hope to generate, and does not capture nuanced contexts that scientists consider: target application settings, requirements and constraints, motivations and challenges. in light of the strong progress recently made with large language models (llms), in this paper we explore a dramatically different setting: models that take descriptions of problem contextsand return natural language suggestions of novel scientific directions that are grounded in literature. we develop a framework named scimon (scientific inspiration machines with optimization for novelty), named after nobel laureate and ai pioneer herbert simon who authored early foundational work on automated scientific discovery @xcite . we first present an automated data collection methodology that collects examples of past problems and proposed ideas from scientific papers. we then use this data for both fine-tuning and in-context training of llms-training them to take problem descriptions and output proposed ideas to address them. we observe that state-of-art llms (e.g., @xcite ) struggle with generating novel scientific ideas, and contribute a new modeling framework for generating hypotheses that makes progress in improving the hypothesis generation ability of llms (figure 1 ). given a background problem description, models first dynamically retrieve inspirations from past literature in the form of related problems and their solutions along with contexts from a scientific knowledge graph. these retrieved inspirations serve to ground the generated ideas in existing literature. we then endow models with the ability to iteratively boost the novelty of generated ideas. given an idea i generated by the llm at step t, the model compares i with existing research in the literature; if it finds strongly overlapping research, the model is tasked with updating its idea to be more novel relative to prior work (much like a good researcher would do). we also introduce an in-context contrastive model which encourages novelty with respect to background context. we perform the first comprehensive evaluation of language models for generating scientific ideas in our new generative, contextual setting. we focus on ai/nlp ideas to facilitate analysis by ai researchers themselves, and also demonstrate generalization to the biomedical domain. we design extensive evaluation experiments using human annotators with domain expertise to assess relevance, utility, novelty, and technical depth. our methods substantially improve the ability of llms in our task; however, analyses show that ideas still fall far behind scientific papers in terms of novelty, depth and utility-raising fundamental challenges toward building models that generate scientific ideas. ## background and new setting we begin with a brief description of related work and background. we then present our novel setting. literature-based discovery nearly four decades have passed since don swanson pioneered literature-based discovery (lbd), based on the premise that the literature can be used for generating hypotheses @xcite . lbd has been focused on a very specific, narrow type of hypothesis: links between pairs of concepts (often drugs/diseases). the classic formalization of lbd goes back to @xcite who proposed the "abc" model where two concepts (terms) a and c are hypothesized as linked if they both co-occur with some intermediate concept b in papers. more recent work has used word vectors @xcite or link prediction models @xcite @xcite to discover scientific hypotheses as pairwise links between concepts. a tightly related body of research focuses on scientific knowledge graph link prediction @xcite , where predicted links may correspond to new hypotheses, and knowledge bases are reflections of existing scientific knowledge in specific domains, derived from literature. a fundamental gap in this line of work is in the lack of approaches for modeling nuanced contexts (sosa and altman, 2022) (e.g., the specific settings in which a drug may be relevant for a disease) for generating ideas in open-ended problem settings with unbounded hypothesis spaces, and for optimizing novelty. our setting can be viewed as a radical departure addressing the limitations in existing settings. llms for scientific innovation large language models (llms) have made remarkable progress in interpreting and producing natural language content and handling knowledge-intensive tasks such as in the medical domain @xcite . very recent work @xcite has explored the use of llms in a robotic chemistry lab setting, planning chemical syntheses of known compounds and executing experiments. robotic lab settings are inherently limited to narrow sub-areas where such experiments are possible and relevant. other very recent work @xcite used llms to produce code for machine learning tasks such as kaggle competitions, finding that a gpt-4 agent achieved 0% accuracy on research challenges such as babylm @xcite . gpt-4 has been anecdotally reported as having "strengths less like those of having a human co-author, and more like a mathematician working with a calculator" @xcite . our goal is to conduct a non-anecdotal evaluation and enhancement of strong llms' ability to generate novel open-ended scientific ideas. ## scimon models we present a new module to retrieve inspirations as contextual input ( §3.1). then, we describe another module to generate ideas given the con-text+inspiration ( §3.2). finally, we introduce a new iterative novelty optimization method to further improve idea quality ( §3.3). foot_4 ## conclusions and future directions we propose a new setting, model and comprehensive evaluation for scientific hypothesis generation with language models that are grounded in literature and optimized for novelty. we present a new framework named scimon in which mod-els take background problem contexts and provide suggestions that are novel while based on literature. models retrieve inspirations from semantic similarity graphs, knowledge graphs, and citation networks. we introduce a new iterative novelty boosting mechanism that helps large language models (llms) such as gpt-4 generate more novel ideas by explicitly comparing ideas to prior work and refining them. our experiments demonstrate that the task of generating natural language scientific hypotheses is highly challenging. while our methods improve upon baseline llms, generated ideas tend to be incremental and with insufficient detail. generating novel and meaningful scientific concepts and their compositions remains a fundamental problem @xcite . evaluation in this setting is also highly challenging, with a huge space of potentially plausible hypotheses formulated in natural language. one interesting direction is to expand scimon with a multimodal analysis of formulas, tables, and figures to provide a more comprehensive background context.