Sprout: Green Generative AI with Carbon-Efficient LLM Inference (2024)

Abstract: The rapid advancement of generative AI has heightened environmental concerns, particularly regarding carbon emissions. Our framework, Sprout, addresses these challenges by reducing the carbon footprint of inference in large language models (LLMs). Sprout introduces “generation directives” to guide the autoregressive generation process, achieving a balance between ecological sustainability and high-quality outputs. By employing a strategic optimizer for directive assignment and a novel offline quality evaluator, Sprout reduces the carbon footprint of generative LLM inference by over 40% in real-world evaluations, using the Llama model and global electricity grid data. This work is crucial as the rising interest in inference time compute scaling laws amplifies environmental concerns, emphasizing the need for eco-friendly AI solutions.

## introduction the ai boom, driven by the demand for generative artificial intelligence (genai) @xcite @xcite , has prompted concerns over its environmental impact, particularly in terms of carbon emissions associated with the datacenters hosting these technologies. openai's reported pursuit of trillions in investment for ai chips (fortune, 2024), destined for their datacenter infrastructure, underscores the scale of resource expansion required to support genai's growth. generative large language models (llms) have gained a substantial user base across various scientific fields @xcite @xcite @xcite . this underscores a critical need for research focused on minimizing llms' environmental impact. although training these models requires extensive compute cycles and carbon footprint, it is the inference processes of these llms that are poised to become the predominant source of emissions, according to various prior studies @xcite @xcite . the carbon footprint of inference is expected to become even more significant as models like openai o1 @xcite tend to scale inference compute @xcite . unlike traditional natural language understanding models that predict a single masked word or sentiment, generative llms are even more carbon-demanding as they perform iterative predictions for each request until reaching a predefined token or iteration limit. despite the urgency of this issue, there lacks a solution for reducing carbon emissions specifically from the llm inference operations -which is natural given the field is in the early stages, but rapidly evolving. in this paper, we design sprout as the first work to address the sustainability challenges in running a generative llm inference service. various previous works have attempted to reduce the carbon footprint of machine learning (ml) applications @xcite @xcite , but none has designed optimizations tailored to llm inference which is becoming a dominant workload and requires intervention to reduce its carbon footprint. the following summarizes sprout's insights and contributions. introduction of generation directives to llm inference for carbon saving. previous works have identified the opportunity to manipulate the number of parameters in the model to save energy and cost @xcite , while sprout is the first work to identify that in generative language model inference, its autoregressive generation pattern presents a unique opportunity beyond previous works. sprout introduces the concept of "generation directives", a strategy to indirectly manipulate the number of autoregressive inference iterations while providing high-quality content generation. for example, a directive can guide the model to provide a concise response, saving significant carbon from generating a long sequence while still being accurate. identifying the variability in the carbon intensity of the electricity generation and the diverse requirements of different tasks, sprout can leverage different generation directives to minimize the carbon footprint of llm inference with a guarantee of generation quality. design, implementation and evaluation of carbon-friendly generation directive configuration for llm inference. we present sprout, a novel carbon-aware generative language model inference framework designed to reduce carbon footprint through the strategic use of token generation directives while maintaining high-quality outputs. from the selection of directive levels based on electricity grid carbon intensity and user behavior variability, sprout introduces a linear programming approach for system-level optimization, balancing carbon savings with generation quality. sprout identifies the difficulty in retrieving generation quality feedback, and implements an automatic offline and opportunistic quality assessment mechanism to ensure the framework's decisions are informed by up-to-date generation quality. we evaluate sprout using production software setup, state-of-the-art llm, representative corpus to synthesize user prompts, and real carbon intensity traces from global electricity grid operator regions. our evaluation confirms sprout's effectiveness in reducing carbon emissions by more than 40% while still achieving high generation quality. we open source sprout's artifact at https://doi.org/10.5281/zenodo.13879728 . ## background and motivation carbon footprint of an inference request. the carbon footprint is a metric for quantifying the amount of greenhouse gas emissions (gco 2 ) generated. when requesting a service from a datacenter server (e.g., http requests), its carbon footprint comprises the operational carbon and embodied carbon. the operational carbon comes from electricity generation to power the datacenter, which powers the hardware (e.g., gpus) that serves the request (carbon intensity × energy). the carbon intensity (denoted as co intensity 2 ## related work sustainable ai @xcite and sustainable hpc @xcite have explored various carbon trade-offs in ml infrastructure. various works have analyzed the ai development's impact on carbon emission @xcite @xcite @xcite . sprout is motivated by these works and takes the effort a step further to llm inference application. while sys-tems like carbon explorer @xcite , ecovisor @xcite , clover @xcite , and dodge et al. @xcite have been designed to adapt to varying carbon intensities, they have not been specifically optimized for generative llm inference workloads. previous works have explored pre-training and fine-tuning algorithms for controllable text generation, steering the generation towards specific lexical choices or sentiments @xcite @xcite . sprout proposes a promising new direction -controlling llms' generation toward carbon efficiency. @xcite demonstrate a reduction in the length of target text by omitting unedited tokens, which can be applied complementarily to sprout's various generation directives. jie et al. @xcite present a concurrent work that focuses on applying controlled-length summary generation for text summarization tasks. while their approach is relevant, it does not address adapting to changing carbon intensity or incorporating generation quality evaluator feedback for general language tasks to mitigate the environmental impact -which is sprout's main contribution. various works have focused on performance and memory optimization of llm inference, exploring strategies like sparsity and pruning @xcite , speculative decoding @xcite , gpu kernel tiling and fusion @xcite . these advancements are crucial for facilitating the deployment of larger llms to a broader audience. however, the environmental implications of these technologies are equally important. carburacy @xcite and llmcarbon @xcite offer carbon footprint evaluations to help researchers gauge the environmental impact of llm training, while sprout is the first work to tackle the carbon footprint challenge of generative llm inference. ## conclusion this paper introduced sprout, a framework to enhance the sustainability of generative language models. sprout can reduce the carbon footprint of llm inference by over 40%, indicating a greener future for natural language generation.