REPRO -Bench: Can Agentic AI Systems Assess the Reproducibility of Social Science Research? (2025)

Abstract: Assessing the reproducibility of social science papers is essential for promoting rigor in research processes, but manual assessment is costly. With recent advances in agentic AI systems (i.e., AI agents), we seek to evaluate their capability to automate this process. However, existing benchmarks for reproducing research papers (1) focus solely on reproducing results using provided code and data without assessing their consistency with the paper, (2) oversimplify real-world scenarios, and (3) lack necessary diversity in data formats and programming languages. To address these issues, we introduce REPRO-Bench, a collection of 112 task instances, each representing a social science paper with a publicly available reproduction report. The agents are tasked with assessing the reproducibility of the paper based on the original paper PDF and the corresponding reproduction package. REPRO-Bench features end-to-end evaluation tasks on the reproducibility of social science papers with complexity comparable to real-world assessments. We evaluate three representative AI agents on REPRO-Bench, with the best-performing agent achieving an accuracy of only 21.4%. Building on our empirical analysis, we develop REPRO-Agent, which improves the highest accuracy achieved by existing agents by 71%. We conclude that more advanced AI agents should be developed to automate real-world reproducibility assessment. REPRO-Bench is publicly available at https://github.com/uiuc-kang-lab/REPRO-Bench.

## background and literature review since the onset of the reproducibility and replicability crisis (i.e., "replication crisis" @xcite ), the social science community has placed increasing emphasis on the validity of research results @xcite . the evaluation of research validity involves the assessment of two major dimensions: reproducibility and replicability (national academies of sciences, engineering, and medicine, 2019). reproducibility refers to the ability to obtain consistent results using the same data and methods as the original study, while replicability refers to achieving robust results using new data but the same methods @xcite . we study the capabilities of ai agents in assessing the computational reproducibility of social science papers. we focus on reproducibility using the original raw data, without re-executing the data collection process or introducing new data. following formal definitions and guidelines (national academies of sciences, engineering, and medicine, 2019; national science foundation, 2022), agents are required to assess the validity of research findings by checking the consistency between reproduced and reported results using provided data processing and analysis scripts. we review the importance of computational reproducibility in social science research in section 2.1 and highlight the limitations of existing benchmarks in evaluating ai agents' ability to assess reproducibility in section 2.2. ## repro-bench we propose repro-bench, a benchmark for evaluating ai agents' capability to assess the reproducibility of social science papers. we describe our data collection process in detail in section 3.1 and explain the methodology for determining groundtruth reproducibility scores in section 3.2. we present a detailed statistical analysis of repro-bench in section 3.3. based on these, we formally define the repro-bench tasks and describe their key features in section 3.4. ## experiment setup we introduce the agent environment, actions, and feedback in section 4.1, selected agents in section 4.2, and evaluation metrics in section 4.3. ## experiment results we present the quantitative results in section 5.1, analyze agent reasoning traces through case studies in section 5.2, and validate our findings by developing repro-agent with significant performance improvements in section 5.3. ## conclusion we introduce repro-bench, a benchmark designed to evaluate the capability of ai agents in assessing the reproducibility of social science papers. we evaluate three representative agents on repro-bench, with the highest accuracy reaching only 21.4%. building on our empirical findings, we develop repro-agent, which achieves a 71% relative improvement in accuracy, reaching 36.6%. however, this performance remains insufficient for practical applications, highlighting the need for developing more powerful ai agents with enhanced reasoning capabilities, better contextual understanding, and robust evaluation frameworks.