Benchmarking for Domain-Specific LLM s: A Case Study on Academia and Beyond (2025)

Abstract: The increasing demand for domain-specific evaluation of large language models (LLMs) has led to the development of numerous benchmarks. These efforts often adhere to the principle of data scaling, relying on large corpora or extensive question-answer (QA) sets to ensure broad coverage. However, the impact of corpus and QA set design on the precision and recall of domain-specific LLM performance remains poorly understood. In this paper, we argue that data scaling is not always the optimal principle for domain-specific benchmark construction. Instead, we introduce Comp-Comp, an iterative benchmarking framework grounded in the principle of comprehensiveness and compactness. Comprehensiveness ensures semantic recall by covering the full breadth of the domain, while compactness improves precision by reducing redundancy and noise. To demonstrate the effectiveness of our approach, we present a case study conducted at a well-renowned university, resulting in the creation of PolyBench, a large-scale, high-quality academic benchmark. Although this study focuses on academia, the Comp-Comp framework is domain-agnostic and readily adaptable to a wide range of specialized fields. The source code and datasets can be accessed at https://github.com/Anya-RB-Chen/COMP-COMP.

## introduction large language models (llms) have demonstrated impressive capabilities across a wide range of tasks @xcite . while general-purpose llms excel in broad applications, domain-specific llms have emerged as critical tools for delivering precise and accurate responses tailored to specific user needs @xcite @xcite . these models are particularly valuable in fields where fixed-answer questions are prevalent, * corresponding author such as law, medicine, and academic, where precision is paramount @xcite . however, the development of domain-specific llms necessitates high-quality benchmarks that can effectively evaluate their performance. such benchmarks must not only ensure accurate responses within the domain, but also maintain broad coverage of the domain's semantic space to avoid catastrophic forgetting, where models lose proficiency in broader contexts @xcite . this dual requirement underscores the need to develop effective approaches for domain-specific benchmarking. previous studies on constructing domain-specific benchmarks have primarily relied on the principle of data scaling, utilizing extensive corpora or large question sets to ensure broad coverage @xcite . while this approach is widespread and forms the basis for many benchmark designs, it presents several limitations that can affect the efficacy of model evaluation. first, many benchmarks are derived from skill examinations, practice exercises, or expert-curated datasets, often overlooking a critical assessment of their effectiveness @xcite . although these benchmarks are diverse in question- in this paper, we propose comprehensiveness-compactness (comp-comp), a novel framework for constructing domain-specific benchmarks that dynamically balances the semantic distribution of corpora and benchmark questions. as illustrated in figure 1 , comp-comp iteratively expands the range of corpora and questions, continuously assessing their comprehensiveness (maximizing recall) and compactness (optimizing precision). the framework introduces three crucial mechanisms: (1) comprehensiveness and compactness monitoring: we encode the corpora and questions into a unified semantic space, using gaussian kernel density estimation (kde) to improve these metrics iteratively @xcite . (2) corpora-question interaction: we ensure comprehensive evaluation by generating new questions from underrepresented areas in the corpora. (3) superiority in closed domains: we prioritize achieving a balance between comprehensiveness and compactness, ensuring the framework excels in constructing closed-domain benchmarks that demand precise corpus evaluation and accurate answer verification. to validate our framework, we conducted a case study in a well-renowned university and applied comp-comp to develop polybench, one of the first and most extensive benchmarks for the academic domain (see in summary, our contributions are as follows: (1) we propose comp-comp, a principled approach that balances comprehensiveness and compactness in domain-specific benchmark construction. (2) we develop polybench, a large-scale, high-quality benchmark for the academic domain, which addresses the gap of lacking well-established academic llm benchmarks. (3) our method is extensible across different domains, offering valuable insights for future benchmarking. by addressing the limitations of existing methods and providing a systematic benchmark design, our work advances the development of domain-specific llms. benchmark construction contemporary benchmark construction faces three systemic limitations. first, reliance on scaling laws prioritizes quantity over semantic alignment, as seen in legal @xcite and medical @xcite benchmarks built from examination datasets that lack corpus representativeness. second, llm-generated evaluations @xcite risk training-evaluation distribution mismatch, as their metrics often diverge from actual domain usage patterns. third, concurrence studies like benchbench @xcite , while valuable for assessing general domain robustness, neglect corpus-benchmark integration, a fatal oversight for domain-specific evaluation where training data and test sets are intrinsically linked. these shortcomings collectively manifest in inefficient resource usage: benchmarks either over-collect questions (inflating costs) or under-cover critical domains (compromising validity). it remains underexplored how to maximize the evaluation effect and align the contents between training corpora and benchmarks of the domain-specific llms. to address this, we provide a viable approach and a case study to construct such a benchmark in a systematic scheme. ## the comp-comp framework in this section, we present the comp-comp, an automatic benchmark construction framework following the comprehensiveness-compactness principle, which guides both the corpora collection and qa generation stages. figure 2 shows the overview of comp-comp framework. ## polybench: an academic benchmark in this section, we employ the proposed framework to construct polybench, a comprehensive and compact benchmark in the academic domain. ## conclusion this work proposes comp-comp, which is a novel domain-specific llm benchmarking framework using comprehensiveness and compactness as the guiding principles. to validate its performance, we developed polybench, a large-scale benchmark in a closed academic domain, demonstrating the effectiveness of comp-comp. this work serves as a valuable reference for future studies, highlighting the importance of precise dataset curation in enhancing domain-specific llms.