On the Concept of Resource-Efficiency in NLP (2023)

Abstract: Resource-efficiency is a growing concern in the NLP community. But what are the resources we care about and why? How do we measure efficiency in a way that is reliable and relevant? And how do we balance efficiency and other important concerns? Based on a review of the emerging literature on the subject, we discuss different ways of conceptualizing efficiency in terms of product and cost, using a simple case study on fine-tuning and knowledge distillation for illustration. We propose a novel metric of amortized efficiency that is better suited for life-cycle analysis than existing metrics.

## introduction resource-efficiency has recently become a more prominent concern in the nlp community. the association for computational linguistics (acl) has issued an efficient nlp policy document 1 and most conferences now have a special track devoted to efficient methods in nlp. the major reason for this increased attention to efficiency can be found in the perceived negative effects of scaling nlp models (and ai models more generally) to unprecedented sizes, which increases energy consumption and carbon footprint as well as raises barriers to participation in nlp research for economic reasons @xcite . these considerations are important and deserve serious attention, but they are not the only reasons to care about resource-efficiency. traditional concerns like guaranteeing that models can be executed with sufficient speed to enable real-time processing, or with sufficiently low memory footprint to fit on small devices, will continue to be important as well. resource-efficiency is however a complex and multifaceted problem. first, there are many relevant types of resources, which interact in complex (and sometimes antagonistic) ways. for example, adding more computational resources may improve time efficiency but increase energy consumption. for some of these resources, obtaining relevant and reliable measurements can also be a challenge, especially if the consumption depends on both software and hardware properties. furthermore, the life-cycle of a typical nlp model can be divided into different phases, like pre-training, fine-tuning and (long-term) inference, which often have very different resource requirements but nevertheless need to be related to each other in order to obtain a holistic view of total resource consumption. since one and the same (pre-trained) model can be finetuned and deployed in multiple instances, it may also be necessary to amortize the training cost in order to arrive at a fair overall assessment. to do justice to this complexity, we must resist the temptation to reduce the notion of resourceefficiency to a single metric or equation. instead, we need to develop a conceptual framework that supports reasoning about the interaction of different resources while taking the different phases of the life-cycle into account. the emerging literature on the subject shows a growing awareness of this need, and there are a number of promising proposals that address parts of the problem. in this paper, we review some of these proposals and discuss issues that arise when trying to define and measure efficiency in relation to nlp models. 2 we specifically address the need for a holistic assessment of efficiency over the entire life-cycle of a model and propose a novel notion of amortized efficiency. all notions and metrics are illustrated in a small case study on fine-tuning and knowledge distillation. their final recommendations include: (a) authors should report training time and sensitivity to hyperparameters. (b) academic researchers need equitable access to computation resources. (c) researchers should prioritize computationally efficient hardware and algorithms. schwartz et al. (2020) note that training costs in ai increased 300,000 times from 2012 to 2017, with costs doubling every few months, and argue that focusing only on the attainment of state-of-theart accuracy ignores the economic, environmental, or social cost of reaching the reported accuracy. they advocate research on green ai -ai research that is more environmentally friendly and inclusive than traditional research, which they call red ai. specifically, they propose making efficiency a more common evaluation criterion for ai papers alongside accuracy and related measures. hershcovich et al. (2022) focus specifically on environmental impact and propose a climate performance model card that can be used with only limited information about experiments and underlying computer hardware. at a minimum authors are asked to report (a) whether the model is publicly available, (b) how much time it takes to train the final model, (c) how much time was spent on all experiments (including hyperparameter search), (d) what the total energy consumption was, and (e) at which location the computations were performed. in addition, authors are encouraged to report on the energy mix at the location and the co 2 emission associated with different phases of model development and use. @xcite propose a new benchmark for efficient nlp models called elue (efficient language understanding evaluation) based on the concept of pareto state of the art, which a model is said to achieve if it achieves the best performance at a given cost level. the cost measures used in elue are number of model parameters and number of floating point operations (flops), while performance measures vary depending on the task (sentiment analysis, natural language inference, paraphrase and textual similarity). @xcite provide a survey of current research on efficient methods for nlp, using a taxonomy based on different aspects or phases of the model life-cycle: data collection and preprocessing, model design, training (including pre-training and fine-tuning), inference, and model selection. following @xcite , they define efficiency as the cost of a model in relation to the results it produces. they observe that cost can be measured along multiple dimensions, such as computational, time-wise or environmental cost, and that using a single cost indicator can be misleading. they also emphasize the importance of separately characterizing different stages of the model lifecycle and acknowledge that properly measuring efficiency remains a challenge. @xcite elaborate on the theme of potentially misleading efficiency characterizations by showing that some of the most commonly used cost indicators -number of model parameters, flops, and throughput (msec/example) -can easily contradict each other when used to compare models and are therefore insufficient as standalone metrics. they again stress the importance of distinguishing training cost from inference cost, and point out that their relative importance may vary depending on context and use case. for example, training efficiency is crucial if a model needs to be retrained often, while inference efficiency may be critical in embedded applications. ## the concept of efficiency in nlp efficiency is commonly defined as the ratio of useful output to total input: foot_1 ## case study to illustrate the different conceptualizations of resource-efficiency discussed in previous sections, we present a case study on developing and deploying a language model for a specific nlp task using different combinations of fine-tuning and knowledge distillation. the point of the study is not to advance the state of the art in resource-efficient nlp, but to show how different conceptualizations support the comparison of models of different sizes, at different performance levels, and with different development and deployment costs. ## conclusion in this paper, we have discussed the concept of resource-efficiency in nlp, arguing that it cannot be reduced to a single definition and that we need a richer conceptual framework to reason about different aspects of efficiency. as a complement to the established notion of pareto efficiency, which separates development and deployment under a product-as-performance model, we have proposed the notion of amortized efficiency, which enables a life-cycle analysis including both development and deployment under a product-as-output model. we have illustrated both notions in a simple case study, which we hope can serve as inspiration for further discussions of resource-efficiency in nlp. future work should investigate more sophisticated ways of incorporating performance level into the notion of amortized efficiency.