LLM e B ench: A Flexible Framework for Accelerating LLM s Benchmarking (2024)

Abstract: The recent development and success of Large Language Models (LLMs) necessitate an evaluation of their performance across diverse NLP tasks in different languages. Although several frameworks have been developed and made publicly available, their customization capabilities for specific tasks and datasets are often complex for different users. In this study, we introduce the LLMeBench framework, which can be seamlessly customized to evaluate LLMs for any NLP task, regardless of language. The framework features generic dataset loaders, several model providers, and pre-implements most standard evaluation metrics. It supports in-context learning with zero- and few-shot settings. A specific dataset and task can be evaluated for a given LLM in less than 20 lines of code while allowing full flexibility to extend the framework for custom datasets, models, or tasks. The framework has been tested on 31 unique NLP tasks using 53 publicly available datasets within 90 experimental setups, involving approximately 296K data points. We open-sourced LLMeBench for the community (https://github.com/qcri/LLMeBench/) and a video demonstrating the framework is available online (https://youtu.be/9cC2m_abk3A).

## introduction the rapid advancement of sophisticated large language models (llms), supported by in-context learning (icl) @xcite , has gained unprecedented popularity among both the research and development communities. the emergence of such large models facilitated diverse applications such as solving mathematical reasoning problems @xcite . given their success, a systematic evaluation and comparison against stateof-the-art is important to accurately gauge the potential of llms. a comprehensive evaluation allows understanding of the strengths and weaknesses of these models; guides us towards better human-llms interactions through prompting; and facilitates their broader applicability in different scenarios, especially in domains where safety and security are paramount concerns (e.g., healthcare, financial institutes) @xcite @xcite . numerous initiatives were launched to comprehensively assess the performance of llms on standard nlp tasks. the helm project @xcite conducted a thorough evaluation of llms for english, spanning various metrics and scenarios. additionally, the big-bench initiative @xcite introduced an extensive evaluation of 214 tasks, even encompassing languages with limited resources. notably, evaluations have been carried out on models like gpt2.5 @xcite , @xcite , and bloom @xcite within multitask, multilingual, and multimodal settings. these evaluations were further extended to low-resource languages @xcite @xcite . evaluating llms across diverse tasks often entails challenges related to costs, effort, and time due to complexities like handling api calls, task integration, dataset inclusion, evaluation measures, and potentially hosting datasets on public platforms (e.g., hugging face (hf)). to overcome these limitations, in this study, we introduce "llmebench", which facilitates a comprehensive evaluation of these llms through a seamless and flexible implementation. the proposed framework, as depicted in figure 1 , empowers users to assess various llms while simplifying the integration of custom tasks, datasets, and evaluation metrics. a few evaluation frameworks have emerged to facilitate extensive benchmarking of llms. among these are openai evals, foot_0 lm harness @xcite , and openicl @xcite . each framework offers functionalities tailored to specific requirements. for instance, openicl focuses on few-shot learning techniques. our contribution, llmebench, stands out by emphasizing a userfriendly, plug-and-play design, that can seamlessly integrate into existing experimental workflows, setting it apart from other alternatives. llmebench's uniqueness lies in the following features: ## llmebench in figure 1 , we provide the architecture of the llmebench framework. to ease the burden on users in implementing common elements across experimental setups, which are not specific to a task, the architecture was designed to support a uniform format for both input and intermediate outputs. this was achieved by employing a pipeline that utilizes key-value dictionaries to seamlessly pass data. the framework incorporates four fundamental modules, discussed below. the process starts with a dataset, where each input sample s i is routed to the asset module. within this module, a prompt is created and then passed to the model provider for processing. the model's response is then funneled back to the asset module for postprocessing. as the processing of all input samples and the generation of corresponding responses conclude, the evaluation module takes on the task of computing evaluation metrics. the whole process of intercommunication is overseen by a benchmark driver. throughout these processes, the inputs, processed data, and the intermediate outputs are cached for re-use and quick experimentation. ## features llmebench features a generic framework that serves a broad range of tasks and models in different learning settings (zero-/few-shot) to evaluate model performance. it enables scalable and rigorous evaluation across diverse tasks and languages while offering simplicity of implementation and flexibility in customization. ## evaluation of llmebench the framework has already been used across a variety of arabic nlp tasks and datasets @xcite . this involved extensive experimentation using zero-and few-shot learning with state-of-the-art large language models, including gpt-3.5-turbo, gpt-4, and the 8-bit version of the bloomz 176b model. in figure 2 , we provide a summary of the tasks, datasets, and models that have been implemented and evaluated. given that our assessment of the framework was based on the current state-of-the-art nlp tasks and datasets, we implemented task-and dataset-specific metrics reported in the literature. overall, it has been used to evaluate 31 nlp tasks, which were categorized based on acl tracks, 53 datasets, and different model providers within 2 learning setups. all the task recipes are available within the framework. ## related work efforts to assess the performance of llms on standard nlp tasks have been underway since the launch of chatgpt. notable studies, such as those by @xcite al. (2023), bang et al. (2023), ahuja et al. (2023), and hendy et al. (2023) , have conducted large-scale experiments considering multilinguality, multimodality, low-resource languages, and a wide range of datasets and tasks. such large-scale evaluations require off-theshelf and easy-to-use solutions to measure the performances of llms for a variety of nlp-related tasks. to evaluate openai's models, the company developed the evals foot_3 package, which requires the dataset to be prepared in json format using a predefined template. @xcite developed an evaluation framework as a part of their cross-lingual benchmarking effort. this comprehensive framework includes 15 distinct tasks set in a few-shot learning environment across 54 languages. however, this evaluation framework was not publicly available at the time of writing this paper. openicl @xcite is another framework designed specifically for zero-shot or few-shot learning setups. it incorporates various strategies, such as random selection, heuristic methods (including bm25 @xcite , topk @xcite ), and a model-based approach, to select fewshot examples. the openicl framework is implemented under the assumption that users will utilize hf datasets to load and evaluate models. a prompt is an important part that serves as a bridge between humans and llms. to explore the research in this direction @xcite developed the promptbench framework for prompt engineering. eleuther-ai developed lm evaluation harness @xcite , which includes the implementation of 200 tasks and supports multiple models from the hf hub. ## conclusions and future work in this paper, we introduce llmebench, an opensource framework designed to facilitate the llm benchmarking process. llmebench accelerates evaluation of llms using pre-implemented generic datasets, tasks and model providers. in addition, it features a modular design that empowers users to integrate (i) new tasks, (ii) datasets, and (iii) apis for models. the framework incorporates caching mechanisms that effectively reduce time, costs, and effort associated with task evaluations. currently, it includes predefined recipes covering 31 standard nlp tasks such as classification, translation, question-answering, and semantic parsing. these recipes can be readily extended to encompass novel nlp tasks, datasets, and llm models. in future, we aim to further enhance the framework by integrating a broader array of tasks and languages. by embracing an open-source approach and encouraging active community participation, we anticipate its sustained growth through the incorporation of diverse tasks, enriched datasets, and innovative models. additional enhancements under consideration encompass integrating crossvalidation datasets, and incorporating models featuring varied configurations (such as distinct iterations of bloom models). furthermore, we are actively developing more methods for few-shot selections. currently, our framework assumes seamless model access via apis. we are committed to enhancing accessibility by enabling users to effortlessly load and utilize both offline and online models for inference purposes.