U ltra E val: A Lightweight Platform for Flexible and Comprehensive Evaluation for LLM s (2024)

Abstract: Evaluation is pivotal for honing Large Language Models (LLMs), pinpointing their capabilities and guiding enhancements. The rapid development of LLMs calls for a lightweight and easy-to-use framework for swift evaluation deployment. However, due to the various implementation details to consider, developing a comprehensive evaluation platform is never easy. Existing platforms are often complex and poorly modularized, hindering seamless incorporation into researcher’s workflows. This paper introduces UltraEval, a user-friendly evaluation framework characterized by lightweight, comprehensiveness, modularity, and efficiency. We identify and reimplement three core components of model evaluation (models, data, and metrics). The resulting composability allows for the free combination of different models, tasks, prompts, and metrics within a unified evaluation workflow. Additionally, UltraEval supports diverse models owing to a unified HTTP service and provides sufficient inference acceleration.

## introduction llms have been deployed in diverse domains, such as finance @xcite , education @xcite , and law @xcite , demonstrating their versatility and efficacy @xcite . this advancement is significantly bridging the gap between the realization of the current state and artificial general intelligence (agi) @xcite . nevertheless, the expansion of model parameters and training datasets engenders increasing uncertainties and emergent capabilities, posing potential risks to humanity and challenges to stable training models @xcite @xcite . consequently, it is imperative to continuously and meticulously evaluate the evolving capabilities of llms throughout their development to ensure their responsible and beneficial applications. traditional benchmarks @xcite @xcite typically focus on evaluating model performance in a specific capability, making it challenging to assess the comprehensive abilities of a model. additionally, these benchmarks generally do not include model deployment. building pipelines from scratch for each combination of tasks and models for evaluation is highly inefficient and repetitive. therefore, an integrated evaluation framework is crucial. currently, some evaluation frameworks covering the entire pipeline from model deployment to model evaluation are proposed, and predominantly divided into two types: conversational websites, exemplified by platforms like chatbot arena foot_0 , and open-source evaluation tools, such as lm-evaluation-harness foot_1 . the former effectively assesses the conversational abilities of a model but lacks comprehensive task coverage and transparency in the evaluation process. the latter opensource frameworks face challenges such as incomplete task coverage, complex code structure, heavy implementations, high difficulty to use, and tightly coupled functionalities. these issues hinder both convenient and comprehensive assessments. in this paper, we identify three core components that form the evaluation process: models (or systems), task data, and metrics (i.e., evaluation methods), as illustrated in figure 1 . rethinking the implementations in these three aspects would benefit the construction of a lightweight and easy-to-use evaluation framework, which covers mainstream tasks and complete evaluation pipeline, and can be easily expanded according to user customization. to this end, we introduce ultraeval, a lightweight and user-friendly open-source framework for llms evaluation. it stands out for its modular and scalable design, enabling thorough assessment of model capabilities. as illustrated in figure 2 , we segment the evaluation pipeline into three main modules: data, model, and metrics, each operating independently and interacting through data exchange. specifically, ultraeval is characterized by the following features: evaluation is currently in a phase of rapid and exploratory growth. ultraeval will be continuously updated and provide detailed tutorials to help researchers to efficiently deploy evaluation pipeline. ## related work the advancement of llms has led to the emergence of various evaluation frameworks, each with its unique features. this section will provide a detailed overview of the current state of evaluation frameworks (also see chatbot arena @xcite offers a llm evaluation platform where users vote on model responses, using a crowdsourced, anonymous elo-rating system. although innovative, its reliance on human judgment limits its suitability for fast, routine assessments. alpacaeval foot_3 @xcite and fastchat foot_4 @xcite conduct evaluation by employing gpt-4 @xcite for automated judging. yet, in evaluating complex reasoning tasks, they tend to favor verbose responses and face issues with robustness. additionally, the scope of their evaluation capabilities is limited. helm foot_5 @xcite streamlines language model evaluation but is constrained by its support solely for automodelforcausallm foot_6 , excluding models without namespaces or stored locally. it lacks support for user models, demonstrates potential module coupling issues and absence of acceleration options. flageval's foot_7 "capability-task-indicator" framework is original but criticized for its closed-source approach and overly simplistic benchmark choices, raising data security and assessment depth concerns. despite their innovations, both platforms fall short of the adaptability and comprehensiveness seen in more versatile frameworks like ultraeval. llm harness @xcite , used by hug-gingface's open llm leaderboard, and open-compass foot_8 (contributors, 2023) have emerged as comprehensive solutions, offering extensive dataset support and rapid updates. these feature-rich environments, however, entail a trade-off: their complexity and dependency on specific software can complicate usage and customization. this underscores the importance of detailed documentation for those looking to adapt or extend these frameworks. similarly, instructeval foot_9 (chia et al., 2023), leveraging the llm harness infrastructure, caters specifically to models fine-tuned with instructions such as alpaca and flan-t5. despite its targeted approach, instructeval's limitations in model and task coverage hint at its niche application rather than widespread utility. the adoption of such frameworks reflects the evolving landscape of model evaluation, where finding a balance between comprehensiveness and usability poses an ongoing challenge. openai evals foot_10 and gpt-fathom foot_11 @xcite . openai evals offers a straightforward, open-source framework for appraising openai models, while gpt-fathom expands upon this by analyzing the progression from gpt-3 to gpt-4 using a wider dataset array. although it provides valuable insights into llm development, gpt-fathom shares openai evals' limitations in supporting a diverse range of models. ## ultraeval as illustrated in figure 1 , evaluation is a comprehensive process that integrates models, data, and metrics. with this in mind, the design philosophy considers both the independence and interconnectivity of each module. as shown in figure 2 , ultraeval encompasses the entire evaluation lifecycle @xcite and segments the evaluation workflow into three main modules. in this section, we delve into the design and implementation of each component within ultraeval in detail. ## evaluation ultraeval aims to provide a lightweight, comprehensive, and user-friendly evaluation framework to support research. as illustrated in figure 5 , ultraeval's modular design effectively combines various models, tasks and metrics for evaluation. using ultraeval, we evaluate models from the llama2 series @xcite and mistral(jiang et al., 2023) on these widely-used benchmarks. as indicated in the ## discussion and future work in this section, we discuss future work following this study. specifically, we focus on addressing data contamination and supporting a broader range of evaluation scenarios. data contamination refers to the phenomenon that examples from the evaluation set are also found in the training data @xcite , causing inaccuracies in model evaluation. common methods for contamination detection include n-gram over-lap and embedding similarity search. however, none of these methods are perfect, making research on contamination detection still crucial. we will integrate these methods. supporting more evaluation scenarios, such as multimodal, long-text, and retrieval-augmented generation (rag), is crucial for meeting a broader range of evaluation needs. this will be an important direction for our future work. ## conclusion we introduce ultraeval, a lightweight, userfriendly, and comprehensive framework for model evaluation. ultraeval establishes a unified structure with well-defined modules and flexible interactions, aiding researchers and developers in efficiently deploying evaluation workflows. moving forward, we plan to continuously integrate new technologies and features into ultraeval, extending beyond large language models to support the evaluation of multimodal models, retrieval-augmented generation (rag), agents, and more, to advance the research on agi. additionally, we aim to expand our collection of representative benchmarks and also develop our own, exploring the capabilities and limits of large models.