Towards Sustainable NLP : Insights from Benchmarking Inference Energy in Large Language Models (2025)

Abstract: Large language models (LLMs) are increasingly recognized for their exceptional generative capabilities and versatility across various tasks. However, the high inference costs associated with these models have not received adequate attention, particularly when compared to the focus on training costs in existing research. In response to this gap, our study conducts a comprehensive benchmarking of LLM inference energy across a wide range of NLP tasks, where we analyze the impact of different models, tasks, prompts, and system-related factors on inference energy. Specifically, our experiments reveal several interesting insights, including strong correlation of inference energy with output token length and response time. Also, we find that quantization and optimal batch sizes, along with targeted prompt phrases, can significantly reduce energy usage. This study is the first to thoroughly benchmark LLM inference across such a diverse range of aspects, providing insights and offering several recommendations for improving energy efficiency in model deployment.

## introduction recent discussions on the energy and carbon impact of machine learning (ml) algorithms have mainly concentrated on quantifying energy usage during the training phase of these models @xcite @xcite . studies on inference energy are much rarer because a single inference operation consumes considerably less energy and resources. however, under deployment, inference is performed many more times, making its energy impact significant and warranting separate investigation @xcite . for example, 90% of total cloud computing demand * equal contribution for aws, the largest global cloud provider, were for model inference purpose @xcite . moreover, a key motivation for training large language models is that a single model can achieve stateof-the-art performance across diverse nlp tasks due to its impressive zero-shot and few-shot capabilities, making it energy-efficient from a training perspective. however, when we consider the total carbon footprint of the entire lifetime of the model, the energy requirement for model inference plays a significant role, considering the number of inferences that are carried out during the model's lifetime. thus it is crucial to conduct a systematic study to quantify the energy requirements and carbon emissions for model inference across various models and tasks. ## experimental setup in this section, we describe the models, datasets and various settings we use for our experiments. ## factors affecting energy / accuracy in this section, we discuss how different task, model, and setup-related factors contribute to the inference energy and accuracy metrics. ## concluding discussions in this study, we benchmarked the power consumption of various large language models (llms) during inference across diverse nlp tasks.