Evaluating Large Language Models with Enterprise Benchmarks (2025)

Abstract: The advancement of large language models (LLMs) has led to a greater challenge of having a rigorous and systematic evaluation of complex tasks performed, especially in enterprise applications. Therefore, LLMs need to be benchmarked with enterprise datasets for a variety of NLP tasks. This work explores benchmarking strategies focused on LLM evaluation, with a specific emphasis on both English and Japanese. The proposed evaluation framework encompasses 25 publicly available domain-specific English benchmarks from diverse enterprise domains like financial services, legal, climate, cyber security, and 2 public Japanese finance benchmarks. The diverse performance of 8 models across different enterprise tasks highlights the importance of selecting the right model based on the specific requirements of each task. Code and prompts are available on GitHub.

## introduction large language models (llms) have garnered significant attention and adoption across various domains due to their remarkable capabilities in natural language understanding and generation. to align with the new era of llms, new benchmarks have been proposed recently to probe a diverse set of llm abilities. for example, big-bench (beyond the imitation game benchmark) @xcite and helm (holistic evaluation of language models) @xcite attempt to aggregate a wide range of natural language processing (nlp) tasks for holistic evaluation. towards the application of llms in real world, it is expected that llms are capable of processing enterprise text data, which is generated and accumulated through business operations of enterprises. an important characteristics of such data is that it often contain expressions used in specific domains such as finance, legal, climate, and cybersecurity. however, the existing benchmarks often lack domain-specific datasets, particularly for those enterprise domains. this gap poses challenges for practitioners seeking to assess llm performance tailored to their needs. enterprise datasets, though potentially useful as benchmarks, often face accessibility or regulatory issues. evaluating llms with these datasets can be difficult due to sophisticated concepts or techniques needed to convert use case-based inputs to the standard input format of evaluation harness (e.g., bigbench or helm), which indicates the need for standardized metrics and clear performance benchmarks. this highlights the necessity for robust evaluation frameworks that measure llm performance in specialized domains. emerging enterprise-focused or domainspecific llms, such as snowflake arctic foot_0 and bloomberggpt @xcite , are evaluated with limited enterprise application scope and volume. for textual inputs, snowflake arctic is assessed on world knowledge, common sense reasoning, and math. however, such non-domainspecific benchmarks often fail to address the complexities of enterprise applications, such as financial named entity recognition (ner), which requires precise domain language understanding. bloomberggpt is evaluated with several finance datasets, mostly proprietary, and does not include the summarization task. beyond the gaps in english llm enterprise benchmarking, there are additional challenges in the availability and development of such benchmarks in other languages, especially japanese. this gap includes a lack of comprehensive, high-quality datasets tailored specifically to japanese financial terminology, regulations, and market dynamics. additionally, there is limited benchmarking for tasks such as sentiment analysis, risk assessment, and financial forecasting in the japanese context. to narrow the gap between llm development and evaluation in enterprises, we present a framework in figure 1 by augmenting stanford's helm with emphasizes the use of enterprise benchmarks that cater specifically to domains such as finance, legal, climate, and cybersecurity. this framework aims to create and adopt standardized benchmarks reflecting real-world application requirements. this initiative not only addresses the current scarcity of domain-specific evaluation frameworks but also informs better decisions for deploying and optimizing llm technologies across diverse enterprise environments. together, our work makes the following key contributions: (i) developing a set of domain-specific benchmarks by curating datasets, enhancing metrics, and implementing prompts based on industry use cases and requirements. (ii) conducting extensive experiments to demonstrate that llms show different performance trends in domain-specific settings. (iii) enabling researchers and industry practitioners to assess and optimize llms tailored to specific domains by integrating the prompts and benchmark code into the widely adopted helm evaluation harness. this paper does not aim to provide an exhaustive evaluation of llm performance across all enterprise benchmarks; instead, it focuses on the evaluation process of llms in different domains. in the next section, we delve into the current state-of-the-art llm evaluation benchmarks. in section 3, we introduce 27 enterprise datasets in four enterprise domains. section 4 describes the key design considerations in the development of the benchmark. experiments and primary results are presented in section 5. the paper concludes in section 6. ## related work recently, researchers have developed several frameworks to assess the various capabilities of llms. examples include helm @xcite , mmlu @xcite , big-bench @xcite , eleutherai @xcite , and mmcu @xcite , which are widely used to evaluate llms on multiple nlp tasks. specifically, helm categorizes potential scenarios and metrics of interest for llms. however, these frameworks lack benchmarks and metrics for assessing llm performance in enterprise-focused problems. this work leverages the helm platform, extending its benchmark scenarios and metrics to include domain-specific llm evaluations. researchers are actively developing enterprisespecific llm benchmarks in domains like finance, legal, and cybersecurity. for example, finben @xcite introduces a finance-focused benchmark spanning 24 tasks, including information extraction, question answering, and risk management. however, its design is tailored to chinese language tasks, limiting its applicability to english texts and american market data. similarly, xu et al. @xcite provides an extensive analysis of finance-specific tasks, covering six domains and 25 specialized tasks in chinese. zhu et al. @xcite further propose cflue, the chinese financial language understanding evaluation benchmark, but its relevance to non-chinese languages remains constrained. in another effort, hirano @xcite ) makes an initial attempt to build a benchmark for japanese financial tasks, including performance evaluations for several models. while promising, this benchmark lacks the depth and task diversity seen in xu et al.'s comprehensive chinese evaluation, highlighting the need for further exploration of japanese-specific tasks for more robust assessments. enterprise benchmarks in legal are upcoming with works like legalbench @xcite , lawbench @xcite . lawbench is evaluated on multilingual and chinese-oriented llms while laiw is the chinese legal llms benchmark. legalbench provides a benchmark on reasoning while the others evaluate legal foundation inference and complex legal application tasks. lastly, in the cybersecurity domain, researchers have contributed to benchmarks like sevenllm @xcite , cyberbench @xcite , cyberseceval 2 @xcite and cybermetric @xcite . these benchmarks analyze tasks like cyber advisory and reasoning, question-answering, and cybersecurity incident analysis. compared to existing benchmarks, our enterprise benchmarks perform sentiment analysis and summarization tasks that have not been tackled in existing art. the benchmarks in our work are open-sourced and consolidated into a widely adopted evaluation framework to enable comprehensive evaluation across reasoning tasks. this work introduces benchmark datasets from four specific domains ( ## enterprise benchmarks although the collected tasks are mostly conventional, the combination of such tasks and domainspecific datasets are still rare and understudied in the field of llm applications. the focus of this paper is in catering a means for practitioners to evaluate the performance of processing domain-specific datasets. this is because it is known that a general domain llm might suffer from the degradation of performance when it processes domain-specific data due to the unique terminology and knowledge that are only used in a specific industry. as summarized in appendix a.1/ as summarized in appendix a.1/ ## benchmark development recent llms, primarily based on the decoder-only transformer architecture, have unique capabilities and limitations, such as in-context learning (fewshot learning) and input token length constraints. domain-specific benchmark datasets are often designed for different architectures (such as bert), necessitating adaptations in datasets and task implementations. in helm, a scenario represents an evaluation task with a specific dataset and corresponding metrics. these adaptations are incorporated into the development of the scenarios. the prompt for each scenario is included in the appendix a.3. the developed scenarios are adopted to a specific edition of helm, called helm classic, which collects the largest number of nlp scenarios among the helm editions. in this study, helm v0.4.0 is used. ## experiments and results this evaluation is conducted by augmenting helm's framework to encompass 27 publicly available datasets from multiple domains, namely financial, legal, climate and cybersecurity. for each benchmark, the evaluation is conducted on a specific configuration. the intention of this section is to demonstrate the usefulness for practitioners of our benchmarks in evaluating candidate models with their own settings. ## conclusion in summary, this work advances the evaluation of llms in domain-specific contexts by consolidating benchmark datasets and incorporating unique performance metrics into stanford's helm framework. this enables researchers and industry practitioners to assess and optimize llms for specific domains. this work demonstrated that one can get non-trivial evaluation results that are not expected from general-purpose nlp benchmarks. this was done on widely used 18 llms through extensive experiments on 27 publicly available benchmarks in financial, legal, climate, and cybersecurity domains, providing practical prompts for practitioners. our analysis offers valuable insights and highlights future needs for benchmarking llms in specialized applications. for the deployment of this work, we opensourced the code and prompts. in addition, a merge of the benchmark into the helm repository is ongoing to facilitate community adoption of this work. jie zhu, junhui li, yalong wen, and lifan guo. 2024. benchmarking large language models on cflue -a chinese financial language understanding evaluation dataset. preprint, arxiv:2405.10542.