Libra-Leaderboard: Towards Responsible AI through a Balanced Leaderboard of Safety and Capability (2025)

Abstract: As large language models (LLMs) continue to evolve, leaderboards play a significant role in steering their development. Existing leaderboards often prioritize model capabilities while overlooking safety concerns, leaving a significant gap in responsible AI development. To address this gap, we introduce Libra-Leaderboard, a comprehensive framework designed to rank LLMs through a balanced evaluation of performance and safety. Combining a dynamic leaderboard with an interactive LLM arena, Libra-Leaderboard encourages the joint optimization of capability and safety. Unlike traditional approaches that average performance and safety metrics, Libra-Leaderboard uses a distance-to-optimal-score method to calculate the overall rankings. This approach incentivizes models to achieve a balance rather than excelling in one dimension at the expense of some other ones. In the first release, Libra-Leaderboard evaluates 26 mainstream LLMs from 14 leading organizations, identifying critical safety challenges even in state-of-the-art models.

## introduction llms such as llama, gpt, and claude have demonstrated remarkable capabilities in generating coherent and helpful responses @xcite @xcite . these models are increasingly vital in various domains, 1 leaderboard website: https://leaderboard.librai. tech/leaderboard 2 libra-eval: https://github.com/librairesearch/ libra-eval 3 demo video: https://youtu.be/xhn4py8twvg most existing leaderboards and evaluation frameworks focus heavily on capabilities such as knowledge, reasoning, and mathematics, pushing the boundaries of model performance @xcite @xcite . while these advancements are important, safety -a critical attribute in determining an llm's reliability and ethical alignment -has often received comparatively little attention. this imbalance has led to the development of models (open-source ones in particular) that excel in performance metrics, but fall short in addressing safety-critical issues such as bias, misinformation, and harmful responses. the risks posed by unsafe llms are profound. models trained on vast datasets may inadvertently propagate biases, amplify misinformation, or mishandle sensitive topics, leading to real-world consequences @xcite @xcite . in high-stakes domains like healthcare and legal advisory, the consequences of unsafe ai outputs can be severe, underscoring the urgent need for a paradigm shift in evaluation standards -one that prioritizes safety alongside capability. to help bridge this gap, we introduce libra-leaderboard, a balanced leaderboard backed by a safety-oriented evaluation framework with an interactive safety arena. libra-leaderboard aims to establish a comprehensive and balanced evaluation 268 paradigm for llms, emphasizing both safety and capability. its key features include: below, we describe the design and implementation of libra-leaderboard, including its underlying methodology and components. we also present preliminary assessment of 26 mainstream open-source and proprietary llms. our results reveal significant gaps in safety performance, even among stateof-the-art models, highlighting the need for frameworks like libra-leaderboard to drive progress in this critical area. ## related work the increasing adoption of llms has catalyzed the development of various leaderboards and evalua-tion frameworks to benchmark their capabilities. these platforms play a crucial role in measuring llm performance, guiding the research community, and informing model deployment strategies. ## libra-leaderboard the libra-leaderboard is a user-friendly interface powered by safety-oriented benchmark (section 3.1) and safety arena (section 3.2). by combining comprehensive benchmarks, a unified api, and advanced evaluation techniques, it offers an objective and scalable platform for comparing llms. its design integrates dynamic datasets, reproducible evaluation pipelines, and user feedback from the safety arena, ensuring a holistic and robust assessment of model capabilities. ## experiments as part of our initial study, we evaluated 26 llms from major organizations, including openai, anthropic, google, etc. the full list of models is introduced in appendix c. the main results are shown in figure 4 , with results categorized by type available in appendix d. based on the initial ex-periments, we made the following findings: