SLM -Bench: A Comprehensive Benchmark of Small Language Models on Environmental Impacts (2025)

Abstract: Small Language Models (SLMs) offer computational efficiency and accessibility, yet a systematic evaluation of their performance and environmental impact remains lacking. We introduce SLM-Bench, the first benchmark specifically designed to assess SLMs across multiple dimensions, including accuracy, computational efficiency, and sustainability metrics. SLM-Bench evaluates 15 SLMs on 9 NLP tasks using 23 datasets spanning 14 domains. The evaluation is conducted on 4 hardware configurations, providing a rigorous comparison of their effectiveness. Unlike prior benchmarks, SLM-Bench quantifies 11 metrics across correctness, computation, and consumption, enabling a holistic assessment of efficiency trade-offs. Our evaluation considers controlled hardware conditions, ensuring fair comparisons across models. We develop an open-source benchmarking pipeline with standardized evaluation protocols to facilitate reproducibility and further research. Our findings highlight the diverse trade-offs among SLMs, where some models excel in accuracy while others achieve superior energy efficiency. SLM-Bench sets a new standard for SLM evaluation, bridging the gap between resource efficiency and real-world applicability.

## introduction recent advancements in language models (lms) have profoundly influenced a wide range of domains, including finance @xcite , healthcare @xcite , and manufacturing @xcite . these models have demonstrated exceptional capabilities in retaining vast amounts of knowledge @xcite , solving highly complex tasks @xcite , and conducting intricate reasoning processes @xcite ) that closely align with human-level intentions. their ability to understand context, generate coherent text, and adapt to diverse applications has made them indispensable tools in both academic research and industry practices. however, the impressive performance of lms often comes at a significant cost. the large number of parameters that enable their functionality requires extensive computational resources @xcite , leading to prohibitively high operational expenses. additionally, these models demand intensive energy consumption, which not only drives up costs but also contributes to substantial environmental challenges. the carbon footprint associated with training and deploying large language models (llms) has raised concerns about their sustainability @xcite , as they emit significant amounts of co 2 . these issues underline the pressing need for more efficient and environmentally conscious alternatives to llms, particularly in an era of growing awareness around climate change and resource optimization. slms @xcite ) have emerged as a promising solution to mitigate the negative impacts associated with llms. by significantly reducing the number of parameters, slms aim to lower computational costs, minimize energy consumption, and decrease the associated carbon emissions, making them a more sustainable alternative. their potential has garnered increasing attention from both academic researchers and industry practitioners, positioning slms as a practical choice for applications requiring efficiency and scalability. despite their growing prominence, a notable gap exists in the systematic evaluation of slms. currently, there is no dedicated benchmark that comprehensively assesses the performance of slms across diverse tasks while also quantifying their environmental impacts. this lack of standardized evaluation hinders a deeper understanding of their practical implications, particularly in resourceconstrained environments where efficiency and sustainability are paramount. addressing this gap is essential to unlock the full potential of slms and guide their development and deployment to balance performance with environmental responsibility. in this paper, we aim to address the existing gap by introducing slm-bench (small language model-benchmark), a comprehensive benchmarking framework designed to evaluate slms across diverse settings with a focus on their environmental impacts. the insights gained from slm-bench will be instrumental in guiding the development and deployment of slm-powered applications, particularly in resource-constrained environments. slm-bench is characterized by three primary features: (i) focus on slms: unlike existing benchmarks that predominantly emphasize llms @xcite , slm-bench evaluates slms that are computationally efficient and accessible to a wider range of users and systems. (ii) measurement of environmental impacts: a unique aspect of slm-bench is its integration of metrics for energy consumption and co 2 emissions. this allows for a holistic assessment of the sustainability of slms, addressing a crucial aspect often overlooked in benchmarking efforts. (iii) evaluation across diverse settings: slm-bench rigorously evaluates 15 slms on 9 tasks using 23 datasets from 14 domains. the evaluation is conducted on 4 hardware configurations, providing a comprehensive analysis of their performance across varied scenarios. this extensive evaluation ensures a deeper understanding of slm capabilities and trade-offs. slm-bench is illustrated in figure 1 , where we provide an overview of its key components, including the selected slms, the datasets used, the selected domains, the tasks evaluated, and the metrics employed for assessment. to the best of our knowledge, slm-bench represents the first systematic benchmarking framework dedicated to the evaluation of slms with a specific focus on their environmental impacts. we hope that slm-bench will drive construction through evaluation, facilitating the development of slms in many more applications. aiming at reproducibility and transparency, the source code and homepage for slm-bench are provided at https: //github.com/hiveintel/slm-bench.git and 2 https://swarm.hiveintel.ai/leaderboard/ , respectively, which will be continuously updated to catch up with state-of-the-arts. in summary, our contributions are as follows. ## conclusion we introduce slm-bench, a comprehensive benchmark for evaluating the environmental impact of slms. it provides critical insights to guide the selection, optimization, and deployment of models, helping researchers and practitioners balance accuracy, efficiency, and sustainability-especially in resource-constrained settings. to support future research, slm-bench includes an open-source, extensible pipeline for continuous integration. in future work, we plan to incorporate additional slms and explore a wider range of hardware settings. additionally, we aim to enhance the accuracy of energy consumption and co 2 emission measurements by utilizing sensor-equipped devices. we also plan to incorporate instruction-following capabilities as a distinct task dimension, extending the benchmark's coverage of real-world use cases.