STORM - BORN : A Challenging Mathematical Derivations Dataset Curated via a Human-in-the-Loop Multi-Agent Framework (2025)

Abstract: High-quality math datasets are crucial for advancing the reasoning abilities of large language models (LLMs). However, existing datasets often suffer from three key issues: outdated and insufficient challenging content, neglecting human-like reasoning, and limited reliability due to single-LLM generation.To address these, we introduce STORM-BORN, an ultra-challenging dataset of mathematical derivations sourced from cutting-edge academic papers, which includes dense human-like approximations and heuristic cues.To ensure the reliability and quality, we propose a novel human-in-the-loop, multi-agent data generation framework, integrating reasoning-dense filters, multi-agent collaboration, and human mathematicians’ evaluations. We curated a set of 2,000 synthetic samples and deliberately selected the 100 most difficult problems.Even most advanced models like GPT-o1 solved fewer than 5% of them. Fine-tuning on STORM-BORN boosts accuracy by 7.84% (LLaMA3-8B) and 9.12% (Qwen2.5-7B).As AI approaches mathematician-level reasoning, STORM-BORN provides both a high-difficulty benchmark and a human-like reasoning training resource. Our code and dataset are publicly available at https://github.com/lwhere/STORM-BORN.

## introduction mathematical reasoning has emerged as a cornerstone for scaling large language models (llms) and probing their upper bounds of intelligence @xcite @xcite . recent advances stem from architectural innovations @xcite , enhanced pretraining data @xcite @xcite , supervised fine-tuning @xcite @xcite , reinforcement learning @xcite , and chain-ofthought prompting @xcite @xcite . current supervised mathematical datasets fall into two categories: numerical reasoning focuses on arithmetic computations yielding numbers @xcite @xcite , and theorem proving uses formal languages for computer-verifiable proofs @xcite . however, existing mathematical datasets face several challenges. first, they lack nuance and complexity. current datasets, like gsm8k @xcite and math @xcite , focus on grade-school or competition problems, offering limited complexity. as llms excel on these benchmarks, more advanced mathematical reasoning tasks are needed. second, human-like reasoning is limited. while formal languages @xcite ensure precise verification in recent datasets @xcite , they obscure interpretable and intuitive reasoning processes @xcite . finally, synthetic data lacks reliable annotations. while llms are used to generate large-scale math data @xcite @xcite , they struggle with curating and evaluating expert-level derivations, leading to unreliable step-by-step annotations. to address these limitations, we focus on a previously underexplored area: mathematical derivation. these tasks involve long chains of reasoning (cot), revisions, and iterative computations, posing significant challenges for current llms. more-(i) numerical reasoning (prm-800k) question: an unfair coin lands on heads with probability 3 /4 and tails with probability 1 /4. a heads flip gains @xmath08. what is the expected worth of a coin flip? express your answer as a decimal rounded to the nearest hundredth. ## overall pipeline in order to enhance llms' reasoning abilities for mathematical expressions found in research papers, we created storm-born, a dataset that involves advanced mathematical reasoning. this section describes in detail the construction process of storm-born. ## conclusion in conclusion, we present storm-born, a novel dataset designed to address the limitations of existing mathematical derivation datasets. curated from recent top-tier academic papers via the arxiv repository, storm-born is both nuanced and scalable, while avoiding data contamination. unlike isolated steps, we capture full derivations to preserve logical flow and encourage deep theoretical reasoning. using a human-in-loop and multiagent llm framework storm, we generate problems requiring at least three reasoning steps, ensuring complexity and creativity. expert evaluations ensure reliable annotations. empirical results highlight the dataset's challenge, with advanced llms like gpt-o1-pro solving fewer than 5% of the problems, compared to 95% accuracy on gsm8k. additionally, storm-born demonstrates strong generalization capabilities, offering a high-difficulty evaluation benchmark for ai's approach to mathematician-level reasoning. in this way, storm-born will be a pioneer in deriving expert evaluation and provide a human-evaluation and auto-evaluation template for future work.