M otiv G raph- S o IQ : Integrating Motivational Knowledge Graphs and Socratic Dialogue for Enhanced LLM Ideation (2025)

Abstract: Large Language Models (LLMs) hold significant promise for accelerating academic ideation but face critical challenges in grounding ideas and mitigating confirmation bias during refinement. To address these limitations, we propose MotivGraph-SoIQ, a novel framework that enhances LLM ideation by integrating a Motivational Knowledge Graph (MotivGraph), which provides essential grounding from research literature, with a Q-Driven Socratic Ideator. The Ideator, a dual-agent system utilizing Socratic questioning, facilitates a rigorous refinement process that mitigates confirmation bias and significantly improves idea quality across dimensions of novelty, experimental feasibility, and motivation. Our experimental results demonstrate MotivGraph-SoIQ’s effectiveness. Comparative studies show significant quantitative improvements over SOTA methods across LLM-based scoring, ELO ranking, and human evaluation. Ablation studies further validate the crucial contributions of both the MotivGraph for enhancing idea novelty and practicality, and the Socratic dialogue with the teacher agent for substantial quality improvement. This work underscores the potential of combining structured knowledge with interactive, critique-based refinement for robust LLM ideation.

## introduction the potential of large language models (llms) in supporting academic research @xcite @xcite within the domain of scholarly research has garnered increasing attention @xcite @xcite . this includes the automated generation of literature reviews @xcite , assistance in experimental design, and the enhancement of academic writing @xcite . notably, leveraging the creativity of large language models to generate novel * these authors contribute equally to this work. motivation grounding human researchers establish academic motivation connections through an extensive literature review, which helps uncover their underlying motivations and problem-solving approaches. this process enables them to navigate complex knowledge domains, understand fundamental concepts, and promote innovation across different disciplines. for example, researchers may observe that ant colonies utilize pheromone trails to identify optimal paths to food sources. simultaneously, they recognize the challenge of optimizing data routing in large-scale wireless sensor networks. by linking these insights, they form an academic motivation connection between biological swarm intelligence and network optimization. such a connection can lead to the novel application of ant colony optimization algorithms to improve routing efficiency in sensor networks. the effectiveness of academic motivation connections lies in their ability to foster a comprehensive understanding of disparate fields and encourage combinatorial innovation, thereby generating novel and valuable ideas. however, the internal knowledge of large language models is probabilistic @xcite , unstable @xcite , and inherently biased due to training data distribution. relying solely on large language models to generate "academic motivation connections" can lead to unreliable innovation. concerns persist that llm-generated ideas may be primarily hallucinatory, superficial @xcite , or infeasible @xcite . although approaches have been proposed to ground llms with external academic resources for background information @xcite , their limited context windows hinder effective processing of extensive literature and the formation of deep connections. consequently, enabling llms to generate innovative and feasible ideas necessitates a motivation knowledge base capable of providing a profound grasp of academic research's underlying motivations and relationships, in a format compatible with llm processing characteristics. confirmation bias confirmation bias @xcite ) is a cognitive bias where individuals favor information that confirms pre-existing beliefs @xcite . human researchers are susceptible to favoring data that supports their hypotheses, sometimes overlooking contradictory evidence. discussions between the mentor and the researcher are crucial for its mitigation in scientific contexts. in these settings, researchers present their hypotheses and reasoning to their mentors, who challenge assumptions, question methodologies, and highlight overlooked counterexamples, helping to correct biased reasoning and flawed assumptions. llms also exhibit this bias, struggling with novel thought generation and self-correction once an initial stance is established @xcite . a key challenge in leveraging llms for academic ideation is enabling them to identify critical weaknesses in their generated ideas. while effective for superficial issues, current self-reflection methods fail to address fundamental shortcomings such as incorrect assumptions due to their vulnerability to confirmation bias @xcite . thus, developing strategies for llms to refine ideas while actively mitigating this bias remains a considerable challenge. in this paper, we propose socratic llm ideation with academic motivation graph (motivgraph-soiq) to address the challenges above. we propose the motivational knowledge graph (motivgraph) as a foundation for grounded idea generation. to build this graph, we develop science motivation miner, which automatically extracts (problem, challenge, method) triplets from published papers and organizes them into inter-connected nodes and edges. during ideation, our autonomous multi-tool framework guides llms to query and update the motivgraph at each step, ensuring that generated ideas reference specific graph nodes and include explicit source annotations. by grounding every concept in the underlying literature, this approach enhances traceability and strengthens the validity of the generated ideas. we propose the q-driven socratic ideator to enhance idea quality further and mitigate confirmation bias. inspired by the socratic method @xcite , an llm acts as a "mentor" to critically question a "researcher" agent. the mentor assesses logic, self-consistency, and rigor, while the researcher leverages structured domain knowledge to generate ideas. this dialogue prompts the researcher to rectify flaws, thereby avoiding confirmation bias inherent in self-reflection and reducing extra external knowledge requirements, simplifying the refinement process. we evaluate motivgraph-soiq on a topic set constructed from iclr25 papers and compare it against a strong baseline by having deepseek-v3 generate ideas under both approaches. using deepseek-v3-generated proposals, our method achieves 10.2 % higher novelty and 6 % higher motivational rationality in llm-based scoring, yielding an average llm elo score that is 38 points above the baseline. human evaluations of the same deepseek-v3-generated ideas confirm these gains, showing increases of 7.98 % in novelty and 5.56 % in motivational rationality. across all metrics, motivgraph-soiq consistently outperforms the baseline. our main contributions are summarized as follows: 1: to address the lack of motivational grounding and limited self-improvement in llm-based ideation, we propose motivgraph-soiq. this unified framework integrates a motivational knowledge graph with a socratic ideation loop to produce grounded, high-quality ideas. ## method in this section, we detail our llm-based ideation methodology, the motivgraph-soiq framework, which integrates two core components: (i) motiv-graph, a motivation-enhancing knowledge graph for structured motivation representation, and (ii) q-driven socratic ideator, an adversarial agentic system that refines ideas through "socratic questioning" and "maieutics". ## experiment to validate motivgraph-soiq's effectiveness, we conducted both comparative and ablation experiments. we constructed the motivgraph and an evaluation dataset using publicly available literature. the motivgraph provides motivational grounding, while the evaluation dataset, comprising 100 diverse iclr 2025 paper topics and their core ideas, serves as ground truth for assessing generated ideas. our comparisons against baselines demonstrate motivgraph-soiq's superiority, and ablation studies confirm the effectiveness of individual system components. see the appendix b.5 for details on the dataset. ## conclusion llms offer great promise for academic ideation but face challenges with idea grounding and confirmation bias. we introduce motivgraph-soiq, a novel framework that enhances llm ideation by integrating a motivational knowledge graph for grounding from literature and a q-driven socratic ideator. this dual-agent system uses socratic questioning to refine ideas, mitigating confirmation bias and improving novelty, experimental feasibility, and motivation. our results demonstrate motivgraph-soiq's effectiveness and superior performance across llm-based scoring, elo ranking, and human evaluation. ablation studies confirm the crucial contributions of both motivgraph and the socratic dialogue. this work highlights the power of combining structured knowledge with interactive, critique-based refinement for robust llm ideation.