Exploring the Design of Multi-Agent LLM Dialogues for Research Ideation (2025)

Abstract: Large language models (LLMs) are increasingly used to support creative tasks such as research idea generation. While recent work has shown that structured dialogues between LLMs can improve the novelty and feasibility of generated ideas, the optimal design of such interactions remains unclear. In this study, we conduct a comprehensive analysis of multi-agent LLM dialogues for scientific ideation. We compare different configurations of agent roles, number of agents, and dialogue depth to understand how these factors influence the novelty and feasibility of generated ideas. Our experimental setup includes settings where one agent generates ideas and another critiques them, enabling iterative improvement. Our results show that enlarging the agent cohort, deepening the interaction depth, and broadening agent persona heterogeneity each enrich the diversity of generated ideas. Moreover, specifically increasing critic-side diversity within the ideation–critique–revision loop further boosts the feasibility of the final proposals. Our findings offer practical guidelines for building effective multi-agent LLM systems for scientific ideation.

## introduction large language models (llms) have recently demonstrated strong capabilities in supporting early-stage scientific ideation, such as proposing novel research directions or hypotheses based on seed topics @xcite . however, most prior work treats llms as singleshot generators, prompting a single model once to produce ideas without further refinement or critique. this overlooks the potential benefits of multiagent collaboration, which is central to human creativity. recent studies have begun exploring interaction among llms through debate, self-critique, and role-play @xcite @xcite , showing promising results in tasks such as factual question answering and strategic planning. despite these advances, it remains unclear how to best structure such interactions for open-ended creative tasks like research ideation. in this work, we conduct a controlled study to investigate how multi-agent llm dialogues should be designed to improve the quality of generated research ideas. we systematically explore the impact of three key design dimensions by varying the dialogue structure, corresponding to our main experimental manipulations: while these design aspects-diversity, parallelism/breadth, and depth-are frequently configured heuristically in prior multi-agent systems, their individual contributions and potential interactions in the context of open-ended creative generation like research ideation remain largely unexplored. our study aims to disentangle these effects through controlled comparisons across various configurations within a structured dialogue framework, to address this gap, we adopt a structured ideation-critique-revision framework and evaluate seven dialogue configurations across seven diverse research topics in ai and nlp. each configuration varies in agent count, persona specialization, and dialogue depth. outputs are evaluated automatically using diversity metrics and a gpt-4 preference tournament. our goal is not to propose a novel generation framework, but rather to provide empirical guidance for designing effective multi-agent llm ideation systems. our findings show that involving multiple agents with complementary domain expertise and allowing for multi-turn refinement significantly improves the quality of generated ideas. these results offer practical insights for the development of llm-based tools to support scientific creativity. multi-agent interaction with llms. classical mas surveys and logics emphasize coordination, commitment and diversity @xcite , echoing evidence that heterogeneous groups innovate better @xcite . for llms, debate @xcite , role-play societies (camel) @xcite , memory-rich generative agents @xcite , and critical discussions @xcite ) raise accuracy or behavioral realism, but mostly on factual or planning tasks-not open-ended ideation. multi-agent automation of research. fullpipeline systems chain specialized agents to write publishable papers (ai scientist @xcite , its successor ai scientist-v2 @xcite , and ai-researcher (jiabin tang, 2025)), iterate with an automated reviewer (cycleresearcher @xcite ), share results via a communal memory (agentrxiv @xcite )) or a review-experiment-write loop (agent laboratory @xcite ). community-level simulators model entire research graphs @xcite , while ai co-scientist validates biomedical hypotheses in vitro @xcite . virsci (many heads are better than one) @xcite focuses specifically on scientific idea generation, letting diverse "virtual scientists" iteratively refine proposals and surpass single-agent baselines. despite these advancesand promising single-agent creativity studies @xcite -no work has systematically measured how agent diversity, parallelism, and interaction depth trade off when generating research ideas. our experiments close this gap. ## results across all seven research topics, the three design axes introduced in section 3.2 exhibit clear and largely orthogonal effects on both diversity and quality. below we summarize the main findings; all numbers refer to averages over topics. overall trends. the baseline self-critique yields a solid improvement over the single setting, increasing the non-duplicate ratio by +0.08 ( effect of agent parallelism. increasing the number of parallel critics consistently raises diversity: moving from one to three critics boosts the non-duplicate ratio from 0.77 to 0.80, after which returns diminish ## conclusion we presented a controlled, factorial study on how to design multi-agent llm dialogues for scientific ideation. our results show that enlarging the agent cohort, deepening the interaction depth, and broadening persona heterogeneity each enrich the diversity of generated ideas. moreover, specifically increasing critic-side diversity within the ideation-critique-revision loop further boosts the feasibility of the final proposals. we provide empirical evidence that both agent diversity and dialogue depth contribute to more useful research ideas.