S cholar B ench: A Bilingual Benchmark for Abstraction, Comprehension, and Reasoning Evaluation in Academic Contexts (2025)

Abstract: Prior benchmarks for evaluating the domain-specific knowledge of large language models (LLMs) lack the scalability to handle complex academic tasks. To address this, we introduce ScholarBench, a benchmark centered on deep expert knowledge and complex academic problem-solving, which evaluates the academic reasoning ability of LLMs and is constructed through a three-step process. ScholarBench targets more specialized and logically complex contexts derived from academic literature, encompassing five distinct problem types. Unlike prior benchmarks, ScholarBench evaluates the abstraction, comprehension, and reasoning capabilities of LLMs across eight distinct research domains. To ensure high-quality evaluation data, we define category-specific example attributes and design questions that are aligned with the characteristic research methodologies and discourse structures of each domain. Additionally, this benchmark operates as an English-Korean bilingual dataset, facilitating simultaneous evaluation for linguistic capabilities of LLMs in both languages. The benchmark comprises 5,031 examples in Korean and 5,309 in English, with even state-of-the-art models like o3-mini achieving an average evaluation score of only 0.543, demonstrating the challenging nature of this benchmark.

## introduction the emergence and application of large language models (llms) @xcite @xcite has significantly advanced performance across diverse natural language processing tasks and effectively addressed both conventional and complex challenges in the field. llms are trained on multilingual (tang et al., * equal contribution † corresponding authors. general-domain benchmarks predominantly utilize standardized examination questions, which typically focus on stem disciplines. consequently, they offer limited insight into specialized knowledge domains and inadequately capture llms' domain-specific problem-solving capabilities. to address these gaps, recent benchmark studies have focused on detailed evaluations of llm performance in specialized tasks requiring deep expert knowledge and practical application. for example, multimedqa @xcite ) and finben @xcite specifically assess llm within medical and financial domains, demonstrating their practical applicability in tasks such as information extraction and risk management. similarly, chemllmbench @xcite and datascibench @xcite provide comprehensive assessments within chemistry and in this paper, we introduce a new benchmark called scholarbench, designed to evaluate the problem-solving capabilities of llms in academic domains, with a focus on their parametric knowledge, and analyze their performance in advanced reasoning tasks within scholarly environments. academic-domain llm benchmarks help enhance the practical applicability of llms in academic research, education, and specialized fields. scholarbench offers three key features: • domain and attribute. to systematically evaluate performance across interdisciplinary academic domains, we define four primary domains natural sciences, applied sciences, social sciences, and humanities and further delineate a total of eight categories. in addition, we propose fine-grained attribute categories to capture the diversity of question types and enable nuanced assessments of generalization performance within and across academic domains. moving beyond simple item-level evaluation, the benchmark incorporates multidimensional assessment criteria including abstraction, reasoning, and comprehension to rigorously evaluate the practical problem-solving abilities and realworld applicability of llms. this approach offers a robust framework for holistic assessment of academic intelligence in llms. overall, scholarbench aims to assess llms' performance in academic domains, analyze question type-specific results from multiple perspectives to identify model strengths and weaknesses, and provide insights for improving llms in the academic domains. the contributions of this paper are as follows: ## open scholar benchmark we propose scholarbench, a benchmark that enables performance evaluation and analysis of llms in academic domains, and introduce its construction methodology. to define the multidimensional evaluation of llms, we present three key concepts: abstraction (c1) assesses the ability of llms to identify key information and summarize academic literature while maintaining the context. reasoning ability (c2) evaluates the llm's capacity for logical reasoning, demonstrated by its ability to infer answers based on parametric knowledge. the reasoning questions consist of short-answer, multiple-choice, multiple-selection, and true/false types, and are solved by the llms in a closed-book setting. comprehension (c3) is evaluated in an open-book setting, where the llm must identify and extract key information from a given academic paragraph to determine its ability to solve academic problems accurately. ## experiments in this chapter, we evaluate the performance of publicly available english-korean bilingual models to determine the validity of the scholarbench. ## analysis in this section, we evaluate and analyze the abstraction, comprehension, reasoning, and bilingual abilities. for more detailed experimental results, please refer to appendix g. ## conclusion we propose scholarbench, a new benchmark dataset for evaluating the capabilities of llms in academic domains, and analyze the experimental results. while previous benchmark's studies focus on specific tasks or general domains, scholarbench is designed to include various abilities and question types required in academic fields, such as abstraction, comprehension, reasoning, and bilingual ability. by defining eight academic domain categories and 63 corresponding attributes, we enhance the diversity of problems and provide a guide for more detailed and systematic evaluation of problem-solving abilities. as analysis results utilizing scholarbench, we confirmed the significant variations that llms show depending on the task types, academic fields, and model characteristics. for each task, we analyze model performance based on the problem characteristics such as reliance on structured information, context dependency, expression diversity, causal reasoning, relational understanding, and logical negation. this enables the evaluation of both topic generalization ability and topic-specific reasoning ability to handle the complex knowledge and reasoning within specific academic fields. for future work, we envision further expanding scholarbench to include more academic domains and new task types, such as rag and multimodal. additionally, we plan to specifically analyze model error types to identify the root causes of performance degradation and utilize scholarbench to develop llm learning and tuning strategies specialized for academic domains. finally, we expect that scholarbench will contribute to the development of more capable and reliable ai systems for the academic community.