S ci C ompanion: Graph-Grounded Reasoning for Structured Evaluation of Scientific Arguments (2025)

Abstract: The exponential growth of scientific publications has overwhelmed reviewers and researchers, with top conferences receiving thousands of submissions annually. Reviewers must assess feasibility, novelty, and impact under tight deadlines, often lacking tools to identify relevant prior work. Early-career researchers face similar challenges, with limited support to navigate fast-evolving fields. Existing LLM-based systems struggle with static retrieval, surface-level features, and lack multi-hop reasoning, leading to shallow or hallucinated assessments. Scientific evaluation requires a deep, relational understanding, which current retrieval-augmented generation (RAG) methods fail to achieve. We introduce SciCompanion, a graph-grounded reasoning framework for structured scientific evaluation. Given a paper or abstract-like input, SciCompanion builds a dynamic knowledge graph from recent publications, domain-specific databases, and curated metadata. It employs multi-hop reasoning to iteratively construct contextual graphs and generate structured critiques, enabling deeper exploration of scientific literature. Unlike sentiment-biased LLM evaluations, SciCompanion directly optimizes retrieval and graph refinement using Group Relative Policy Optimization (GRPO), producing reviews aligned with expert judgments. Experiments on ICLR and ACL datasets show that SciCompanion reduces evaluation error by over 30% compared to prompting-only baselines and allows smaller models to outperform larger ones. Evaluations across three datasets, using metrics for retrieval accuracy, semantic overlap, and multi-hop sensitivity, along with a case study, demonstrate SciCompanion’s robustness and versatility.

## introduction the exponential rise in scientific publications has immensely strained the peer review ecosystem. conferences in artificial intelligence and machine learning, such as neurips, icml, and iclr, have seen a significant increase in paper submissions, with neurips 2025 receiving over 10,000 submissions @xcite . similarly, acl conferences have experienced consistent year-over-year growth, with acl 2023 reporting 4,864 submissions, a marked increase from previous cycles @xcite . this surge creates unsustainable reviewer workloads due to high volume, tight deadlines, and unfamiliarity with subdomains @xcite . the "publish or perish" culture @xcite exacerbates this, encouraging quantity over rigor and leading to reviewer fatigue. early-career researchers and junior reviewers also struggle with the rapidly growing, fragmented literature @xcite . as prior work exceeds individual cognitive capacity, assessing novelty, identifying related work, and evaluating methodology becomes time-consuming and error-prone. this information overload compromises peer review quality and scientific judgment, highlighting the urgent need for intelligent, scalable, and trustworthy tools for transparently synthesizing, contextualizing, and evaluating contributions @xcite . large language models (llms) offer scalable language understanding but falter in the face of evolving, frontier scientific knowledge @xcite . in an attempt to resolve this, retrieval-augmented generation (rag) approaches @xcite incorporate external documents, but are typically static, nonadaptive, and unstructured @xcite . graph-based methods like graphrag @xcite offer structured retrieval, yet they typically focus on passive information linkage rather than critique-driven synthesis or task-conditioned reasoning. llm baselines lack alignment with expert review dimensions, often hallucinate unsupported claims @xcite , and fail to support multi-hop reasoning. for instance, standard llms may retrieve superficially relevant papers but fail at the multi-hop reasoning needed to uncover subtle connections that determine true novelty, as they are not inherently designed for deep, iterative exploration. recent specialized systems have sought to address these gaps: openreviewer @xcite uses large-scale fine-tuning on expert reviews to generate more critical and realistic feedback that matches human judgment distributions, while pa-persea @xcite introduces a multi-stage framework to standardize inconsistent review data before fine-tuning and apply a self-correction mechanism. however, these approaches are primarily grounded in supervised learning from existing text and do not incorporate dynamic, graph-based reasoning or reinforcement learning to guide the critique generation process. effective peer review demands systems for deep, context-sensitive evaluation that are: context-aware (interpreting domain nuance), critique-aligned (structured around feasibility, novelty, impact), and explainable (producing interpretable, trustworthy reasoning) @xcite @xcite . prior symbolic and graph-based tools @xcite @xcite offer structured exploration but are disconnected from modern llms' adaptive reasoning and lack reinforcement learning scaffolds for alignment with scientific critique @xcite , leaving a gap for structurally grounded, flexible systems. to address the limitations of static retrieval and shallow critique in scientific evaluation, we introduce scicompanion, a unified framework that integrates dynamic graph reasoning, reinforcement learning, and llm-driven critique generation (see overall, the summary of our contributions is: on three peerreview datasets, scicompanion outperforms rag and graphrag baselines by up to 11.2 points. • practical utility. we release an open-source implementation under the mit license to support reviewers and researchers in critique exploration and literature analysis at scicom-panion 1 . knowledge graph base query internal state graph t iterations update state graph inform query feasibility novelty impact overall score 1 overall score n-1 overall score n reward: r n-1 reward: r 1 reward: r n relative advantage calculation update model ## scicompanion: structure-aware reasoning for scientific paper evaluation scientific evaluation is a multifaceted task requiring reasoning over text and structured knowledge. scicompanion emulates this expert process using llms augmented with structured graph retrieval and rl. instead of static retrieval or shallow prompting, scicompanion builds a dynamic reasoning system that iteratively constructs context, formulates hypotheses, and aligns judgments with expert evaluations. ## experiments to assess the capabilities of scicompanion, we design a comprehensive evaluation protocol grounded in the core challenges outlined in the introduction: scalable critique generation, structured retrieval, and generalization across domains. our experiments aim to answer three central questions: (q1) how accurately can scicompanion emulate expert evaluations? (q2) what is the contribution of multi-step, graph-based retrieval to reasoning quality? (q3) how does reinforcement learning via grpo compare to standard prompting and fine-tuning strategies? datasets. we use three datasets for evaluation. iclr (5,482 ml papers with reviews) tests multi-dimensional critique (feasibility, novelty, impact) @xcite . acl soundness & overall (3,219 cl papers) provide labels for methodological rigor and overall recommendation @xcite . goodreads (5,000 book descriptions with user ratings) tests cross-domain adaptability with loosely structured text and non-expert preferences @xcite . for all experiments requiring training, we used a standard 80%/10%/10% random split for the training, validation, and test sets, respectively. evaluation metrics. for each dataset, the evaluation task involves predicting a continuous score or vector of scores ŝ approximating the expert or crowd-assigned ground truth y . we report results using three evaluation metrics. root mean square error (rmse) quantifies predictive accuracy against gold scores. point match rates measure the overlap between generated critiques and peer reviews in terms of strong and weak points. finally, retrieval accuracy is assessed by comparing the system's generated references to ground-truth citations in both full-text and abstract-only settings, providing insight into scicompanion's ability to surface contextually relevant evidence. experimental setting. models are tested in zero-shot (guidelines only), five-shot (exemplar reviews), and trained (finetuning and grpo-based rl) settings, reflecting increasing supervision. experiments use gpt-4o-mini, qwen2.5-7b, and qwen2.5-14b backbones (via vllm, fixed decoding). we used @xmath0, aiding resource-constrained deployment. ## conclusion we present scicompanion, a critique-aligned, graph-grounded reasoning framework for structured scientific evaluation. motivated by the rising scale and complexity of peer review, scicom-panion combines large language models with dynamic knowledge graphs and reinforcement learning to perform transparent, multi-hop assessments of scientific work. its architecture reflects how expert reviewers navigate literature, retrieving relevant prior work, reasoning over structured evidence, and grounding judgments in contextual understanding. our experiments across four diverse datasets demonstrate that scicompanion substantially improves evaluation quality, reducing rmse by up to 31.2% compared to prompting-only baselines. through structured graph construction and grpobased optimization, the framework enables smaller models (e.g., 7b) to match or exceed the performance of larger, unstructured counterparts, offering a practical, scalable solution for review assistance and domain exploration. by aligning llm behavior with scientific critique dimensions (feasibility, novelty, impact), scicompanion advances the frontier of trustworthy, interpretable ai for science. it offers a reproducible, extensible approach to enhance peer review and paves the way for future systems supporting hypothesis generation, literature synthesis, and human-ai discovery.