Evalverse: Unified and Accessible Library for Large Language Model Evaluation (2024)

Abstract: This paper introduces Evalverse, a novel library that streamlines the evaluation of Large Language Models (LLMs) by unifying disparate evaluation tools into a single, user-friendly framework. Evalverse enables individuals with limited knowledge of artificial intelligence to easily request LLM evaluations and receive detailed reports, facilitated by an integration with communication platforms like Slack. Thus, Evalverse serves as a powerful tool for the comprehensive assessment of LLMs, offering both researchers and practitioners a centralized and easily accessible evaluation framework. Finally, we also provide a demo video for Evalverse, showcasing its capabilities and implementation in a two-minute format.

## introduction in recent years, the rapid advancements in large language models (llms) have significantly transformed the computational linguistics landscape, presenting novel opportunities and challenges @xcite . due to the vast scale and complexity of llms @xcite , they have demonstrated remarkable capabilities across numerous applications @xcite , ranging from natural language understanding and generation to more specialized tasks such as summarization @xcite , translation @xcite , and question-answering @xcite . however, the sheer pace of llm development has led to a fragmented ecosystem of evaluation tools and methodologies @xcite . this fragmentation not only hinders the comparative assessment of llms, but also places a considerable barrier to entry for both researchers and practitioners. to achieve the former, we integrate existing evaluation frameworks, such as lm-evaluationharness @xcite and fastchat @xcite , as submodules, allowing an easy extension of new benchmarks. these added submodules can reflect recent changes, allowing evalverse to remain up-to-date with relative ease. on the other hand, we also implement no-code evaluation features that utilize communication platforms such as slack foot_0 , making llm evaluation more accesible for individuals with less programming proficiency. this paper provides an in-depth examination of the architecture and functionality of evalverse, illustrating how it addresses the current challenges in llm evaluation. some of the key features of evalverse include no-code evaluation and a unified and expandable library for llm benchmarks, enhancing the efficiency and accessibility.