Chain of Ideas: Revolutionizing Research Via Novel Idea Development with LLM Agents (2025)

Abstract: Research ideation is crucial for scientific progress, but the exponential increase in scientific literature makes it challenging to stay updated and identify impactful directions. Recent developments in large language models(LLMs) offer a promising avenue to automate this process. However, existing methods for idea generation either trivially prompt LLMs or expose LLMs to extensive literature without indicating useful information. Inspired by human research processes, we propose a Chain-of-Ideas (CoI) agent, an LLM-based agent that organizes relevant literature in a chain structure to effectively mirror the progressive development in a research domain. This organization helps LLMs better grasp current advancements, thereby improving ideation capabilities. Further, we present Idea Arena, a protocol for evaluating idea-generation methods from different perspectives, which aligns closely with the preferences of human researchers. Experiments show that CoI agent consistently outperforms existing methods and matches human quality in idea generation. Moreover, CoI agent is budget-friendly, requiring only $0.50 to generate a candidate idea and its experimental design.

## introduction idea generation is a crucial aspect of scientific research for driving technological innovations and breakthroughs. traditionally, this process has been predominantly human-driven, necessitating experts to review extensive literature, identify limitations, and propose new research directions. however, the complexity and vastness of scientific literature and rapid technological advancements have made this task increasingly challenging for researchers. recent advancements in large language models (llms) @xcite @xcite have enabled these models to exceed human experts in various scientific tasks, including mathematics @xcite , theorem proving @xcite , and coding @xcite @xcite . building on this robust scientific foundation, one may hypothesize that llms could support a more abstract and creative research idea-generation task. notably, @xcite ; @xcite have validated this hypothesis, highlighting its substantial potential to expedite the discovery of uncharted research avenues. existing methods seek to address two key challenges in improving the quality of generated ideas: curating pertinent literature for llms to gain inspiration and ensuring the novelty of generated ideas. to address the first challenge, previous research improves retrieval augmented generation (rag) systems, which typically depend on textual similarity, with academic knowledge graphs @xcite . for the second challenge, existing approaches either apply predefined criteria such as novelty to guide the idea generation process @xcite or iteratively refine ideas until they demonstrate low embedding similarities with existing papers @xcite . however, existing approaches often expose llms to extensive literature for idea generation. this makes llms vulnerable to the influence of less relevant works, potentially resulting in ideas that lack logical coherence and technological innovation. as shown in the upper part of figure 1 , the llm borrows an idea from graphgpt @xcite and applies it into got framework @xcite to generate what they interpret as a "novel idea". however, this conflates two concepts: got is a prompting method, while graphgpt is a fine-tuning method. in contrast, human researchers systematically analyze a field's evolution from foundational to contemporary works, gaining insights that drive developments. such an understanding enables researchers to critically assess the limitations of earlier studies and identifying emerging trends. therefore, they are better grounded in devising innovative and impactful research ideas. motivated by human practices in conducting research, we propose a novel chain-of-ideas (coi) agent to address the previously identified logical inconsistencies in the ideation processes of llms. as shown in the bottom part of figure 1 , coi agent aims to provide a clear landscape of current research topics by systematically selecting and organizing the relevant papers and their ideas into a chain. coi agent offers distinctive advantages: firstly, it reduces interference from irrelevant literature by carefully selecting papers (e.g. cot @xcite ). second, llms are demonstrated with human practice to craft a novel idea. for example, llms are demonstrated how sc @xcite emerges as a novel idea from cot. third, coi exemplifies a global progression in research development. as a result, llms can gain a deep understanding of the motivations behind these developmental trends, facilitating the identification of promising future research directions. as shown in figure 2 , coi agent begins by retrieving an anchor paper for a given research topic. instead of indiscriminately aggregating all papers within the anchor's citation network, as done in @xcite , we build the coi by selecting relevant and significant literature from both the anchor's references and its subsequent works, thereby extending the chain both backward and forward from the anchor. the constructed coi is then used for idea generation and experiment design. during idea generation, we create multiple coi branches for a research topic, ensuring diverse perspectives and increasing the likelihood of novel and impactful idea discoveries. we also require the llm to predict future trends before finalizing the idea, allowing a gradual consolidation of the idea. note that although the names may sound similar, coi agent is fundamentally different from cot @xcite . cot is a general prompting strategy, while the coi agent is a specialized framework for idea generation. coi agent leverages the advantages of cot but incorporates additional optimizations for enhanced ideation. furthermore, cot serves as a reasoning strategy, whereas coi agent focuses on knowledge organization -structuring relevant research work into a coherent and logical format. therefore, coi and cot are orthogonal in nature, and can be combined to further enhance idea generation capabilities. we compare our coi agent against existing baselines on idea generation in the artificial intelligence (ai) field. to do this, we develop an arena-style evaluation framework called idea arena, where participant methods compete in pairs, demonstrating high agreement with human evaluation. in idea arena, coi agent consistently ranks first among all automated baselines, surpassing the second-best one by 65 elo scores in human evaluation. our analysis further shows that for llms to generate novel ideas, a clear developmental trend analysis is more pivotal than the quantity of related literature. our contributions are summarized as follows: 1) we propose coi agent to enhance llms' capability in idea generation. coi agent effectively mirrors the progressive nature of research development, allowing llms to better grasp the current research advancements. 2) we propose idea arena for a comprehensive evaluation of idea-generation methods, which shows high agreement with human researchers. 3) extensive experiments demonstrate the effectiveness of our coi agent in generating ideas that are comparable to human creativity. ## method in this section, we detail our coi agent, as illustrated in figure 2 , which consists of three stages: (1) coi construction, (2) idea generation, and (3) experiment design. first, given a research topic, the coi agent constructs multiple cois, reflecting different trends within the domain. then, for each coi, the llm predicts future research directions, and crafts ideas through step-by-step consolidation and iterative novelty checks. the best idea is then selected. lastly, the llm generates and refines an experiment design to implement the final idea. ## results idea generation. figure 3 present the results of idea generation by both gpt-4o and human researchers. concrete scores are in human-model agreements. to assess the reliability of our model-based evaluation within idea arena, we analyze the agreements between human and llm judges. we follow @xcite to compute the agreement, which is defined as the probability that two judges agree on the winner of one specific arena match. figure 3 presents pairwise agreements between humans and leading llms (gpt-4o, gemini-1.5-pro, claude-3.5-sonnet). gpt-4o achieves 74.9% agreement with humans, closely approaching human-to-human evaluation levels mentioned in @xcite . this finding indicates an acceptable alignment be- ## conclusions we introduce chain of ideas (coi) agent, a framework designed for generating novel research ideas. coi agent offers a promising and concise solution by organizing ideas into a chain structure, effectively mirroring the progressive development within a given research domain. it helps llms better understand current research to improve ideation. to comprehensively evaluate the capability of automated idea generation methods, we also propose idea arena, an evaluation system that requires participants to compete in pairs about their generated ideas for the research topics, which demonstrates high agreement with human evaluation. experimental results indicate that coi agent consistently outperforms other methods and is capable of generating ideas comparable to human creativity. limitations while the coi agent produces clear and technically sound ideas and experiment designs, they often lack feasibility compared to human ideas and experiments. this underscores feasibility as both a critical bottleneck in automated research innovation and a key area for future focus. additionally, our current methodology is confined to the design phase. a significant future research direction involves enabling the agent to autonomously conduct experiments based on its designs and refine its ideas based on the feedback from experimental results.