R esearch A rena: Benchmarking Large Language Models’ Ability to Collect and Organize Information as Research Agents (2025)

Abstract: Large language models (LLMs) excel across many natural language processing tasks but face challenges in domain-specific, analytical tasks such as conducting research surveys. This study introduces ResearchArena, a benchmark designed to evaluate LLMs’ capabilities in conducting academic surveys—a foundational step in academic research. ResearchArena models the process in three stages: (1) information discovery, identifying relevant literature; (2) information selection, evaluating papers’ relevance and impact; and (3) information organization, structuring knowledge into hierarchical frameworks such as mind-maps. Notably, mind-map construction is treated as a bonus task, reflecting its supplementary role in survey-writing. To support these evaluations, we construct an offline environment of 12M full-text academic papers and 7.9K survey papers. To ensure ethical compliance, we do not redistribute copyrighted materials; instead, we provide code to construct the environment from the Semantic Scholar Open Research Corpus (S2ORC). Preliminary evaluations reveal that LLM-based approaches underperform compared to simpler keyword-based retrieval methods, though recent reasoning models such as DeepSeek-R1 show slightly better zero-shot performance. These results underscore significant opportunities for advancing LLMs in autonomous research. We open-source the code to construct the ResearchArena benchmark at https://github.com/cxcscmu/ResearchArena.

## introduction large language models (llms) have demonstrated exceptional performance in natural language understanding, text generation, and a range of other tasks across domains @xcite @xcite . by integrating llms with external tools-such as code interpreters, vector databases, and search engines-their capabilities can be further enhanced, enabling the creation of autonomous agents that simulate human-like behavior through feedbackdriven task execution @xcite @xcite . however, the ability of llms to handle domainspecific expertise and advanced analytical tasks, such as conducting rigorous academic research, remains underexplored. the challenge of conducting domain-specific research is particularly relevant in an era characterized by rapid knowledge expansion across multiple fields. traditional methods for composing academic surveys are labor-intensive, often requiring months of effort by expert researchers to synthesize relevant findings. an llm capable of independently conducting research on topics outside its training data could bypass the need for continuous re-training and manual updates, offering a scalable and efficient solution for navigating the ever-growing body of scientific literature. while autonomous agents have shown success in executing relatively straightforward tasks-such as online shopping or playing card games @xcite -they face far greater challenges in complex tasks that demand extensive domain expertise and analytical depth. recent developments in agentic capabilities, such as the "deep research" features from both gemini and openai, highlight a growing focus on multi-step research planning and the synthesis of large-scale, diverse information sources @xcite . however, progress in systematically evaluating these agents' capacity for rigorous research remains limited, with few standardized benchmarks designed for advanced, domain-specific scenarios. to promote the development of research agents capable of conducting comprehensive surveys, we introduce the researcharena benchmark. this benchmark emphasizes academic papers due to their depth of research and structured format, qualities that are often more reliable than other sources like general web pages. the researcharena provides an offline environment where autonomous agents can collect and organize information for research across diverse topics. it comprises three sub-tasks for evaluation: information discovery, information selection, and information organization. these sub-tasks emulate general methodologies used by human researchers during literature surveys. conducting a literature survey involves defining the scope, establishing a search protocol, and iteratively analyzing and organizing findings into a cohesive structure. based on this process, re-searcharena introduces tasks to simulate and evaluate these stages, excluding text generation. this decision stems from the premise that the pre-writing research phase is foundational to successful article composition @xcite . moreover, evaluating complete articles is fraught with challenges due to variability in individual writing styles; hence, such assessments are reserved for future work. for information discovery, llms identify and retrieve academic papers relevant to a designated research topic by navigating vast scholarly corpora. information selection challenges llms to critically assess the relevance and impact of these papers, prioritizing significant contributions. as a bonus task, information organization requires llms to synthesize selected research into structured knowledge representations, such as mind maps, to highlight key insights and relationships within the topic. preliminary evaluations reveal that llms underperform compared to simpler keyword-based search methods in tasks requiring analytical depth. for example, using survey titles as retrieval queries consistently yields superior recall and precision compared to llm-driven information discovery and selection tasks. additionally, under the task of information organization, llms face challenges in constructing coherent structures without the oracle guidance, underscoring the need for improvements in organizational and analytical capabilities. nonetheless, we observe promising results from reasoning-oriented models such as @xcite , which show modest gains in zero-shot performance-suggesting that improved reasoning capabilities may help bridge some of these gaps. the dataset supporting researcharena comprises 12m full-text academic papers and 7.9k survey papers, curated from the semantic scholar open research corpus (s2orc) @xcite . this corpus ensures scholarly rigor and relevance, offering a robust foundation for benchmarking llm performance across diverse domains. ## related work previous research has employed diverse methodologies to compile datasets featuring academic survey papers. for instance, bigsurvey dataset @xcite aggregates over 7k survey papers from arxiv and includes approximately 434k references from microsoft academic service and semantic scholar. this dataset underwent extensive preprocessing by removing duplicates, unprocessable files, and normalizing text. on the other hand, surfer100 dataset @xcite includes 100 surveys emulating wikipedia page structures, compiled by eight annotators who summarized content from web pages. each survey contains predefined sections such as introduction, history, key ideas, variations, and applications, summarized concisely in 50 to 150 words. the bigsurvey dataset provides references in an abstract-only format, offering a concise overview of documents. surfer100 utilizes google search results to compile references for each survey topic, reflecting a broad spectrum of web-based information. in contrast, our dataset emphasizes full-text academic papers for a deeper understanding and leverages bibliographic references from original survey papers for enhanced authority and accuracy. a closely aligned task for llm agents in prior research involves generating wikipedia articles. @xcite proposed a method for generating english wikipedia articles by framing the task as a multi-document summarization challenge. their approach employs a combination of extractive and abstractive summarization techniques, identifying salient information using methods such as tf-idf and textrank @xcite . similarly, @xcite introduced the storm system, which tackles pre-writing challenges such as research and outline preparation. storm enhances the article generation process by simulating multi-perspective conversations, wherein an llm poses questions and aggregates responses from reliable sources to develop detailed outlines. additionally, recent work by gemini (google, 2024) and openai (openai, 2025) has explored "deep research" features that rely on multi-step planning, iterative web browsing, and extended context windows to generate comprehensive reports. these agentic systems aim to automate information discovery and summarization by dynamically adjusting their search strategies and synthesizing insights into structured outputs. ## collection methodology this section describes the multi-stage methodology for assembling the dataset of academic surveys, which includes survey selection, reference linking, and mind-map extraction. each stage has been designed to ensure the relevance and utility while addressing potential limitations in automation, domain variability, and data access. the final output is a structured dataset that facilitates benchmarking the autonomous research challenge. the details for the prompts used in each stage of the collection process can be found in appendix a. ## analysis this section details the makeup of our dataset in terms of disciplinary diversity, reference coverage, and the structural complexity of derived typologies, reflecting on how these factors contribute to the robustness and applicability across various domains. ## benchmark tasks this section presents a comprehensive overview of the benchmark tasks designed to evaluate the capabilities of research agents in discovering, selecting, and organizing information. each task targets a specific aspect of research proficiency, with rigorous constraints and evaluation metrics to ensure thorough and unbiased assessment. information discovery. provided a topic extracted from survey title, the task of information discovery requires research agents to identify a subset of documents r from a broader collection d. these documents in r should serve as supporting materials for the topic. ideally, r should encompass all references cited in the original survey s. however, within the collection d, there may exist another survey s ′ that delves into the same topic. if research agents were to use the references from s ′ directly, it would circumvent the need for a thorough discovery, defeating the purpose of this task. to prevent information leakage, we impose the additional constraint such that documents in d must be non-survey and published before s. to evaluate performance, we employ standard information retrieval metrics, recall and precision, to measure the proportion of relevant documents successfully retrieved and the proportion of retrieved documents that are relevant. together, these metrics determine the effectiveness and accuracy of the discovery process. for this task, the cutoff parameter k is set at 10 and 100. information selection. the task of information selection requires research agents to rank the discovered documents based on their importance to the topic. the labels are distinctions between influential and non-influential citations, as elaborated in section 3. normalized discounted cumulative gain (ndcg) @xcite and mean reciprocal rank (mrr) @xcite are used for evaluation. these measures are crucial because conducting research involves more than merely summarizing retrieved documents; it requires the presentation of key insights from the most significant sources. furthermore, both human researchers and autonomous agents are limited by their processing capacities. therefore, it is essential to prioritize and focus on the most critical information first. ## benchmarking in this section, we present preliminary evaluations of existing techniques, describing their configurations and performance metrics. these techniques encompass both naive keyword-based methods, such as title, and advanced llm-based methods, including storm. the exact wording of the prompts used in each baseline can be found in appendix b. we adopt the abstract as the indexing scheme, as it consistently yields the best performance. results for alternative indexing schemes are reported in appendix d. ## conclusion in conclusion, researcharena introduces a rigorous benchmark designed to evaluate llms in conducting research surveys on designated topics. by systematically decomposing the survey process into distinct tasks like information discovery, selection, and organization, this benchmark provides a detailed framework for evaluating autonomus research agents. our findings underscore the poten-tial of llms to revolutionize academic research, provided that future advancements can bridge the existing performance gaps. grounded in semantic scholar open research corpus, this work establishes a robust foundation for the future, aiming to improve the ability of llms to autonomously conduct expertise-level, domain-specific research.