Hyperparameter Power Impact in Transformer Language Model Training (2021)

Abstract: Training large language models can consume a large amount of energy. We hypothesize that the language model’s configuration impacts its energy consumption, and that there is room for power consumption optimisation in modern large language models. To investigate these claims, we introduce a power consumption factor to the objective function, and explore the range of models and hyperparameter configurations that affect power. We identify multiple configuration factors that can reduce power consumption during language model training while retaining model quality.

## introduction large language models have pushed the boundaries of accuracy and performance in various nlp tasks, at the cost of energy efficiency. this is due to the increasing amount of compute time and power needed to train these models @xcite , thus increasing the amount of energy the computers training the models need to consume. the robustly optimized bert approach (roberta) @xcite achieved this by improving the bidirectional encoder representations from transformers (bert) @xcite in multiple ways, such as increasing the training time through more epochs and a larger amount of data, with bert already requiring 1507 kwh of electricity and emitting 652 kg of co 2 . other strategies, such as nas, an english to german machine translation model, consumed 656,347 kwh of electricity, corresponding to 248,019 kg of co 2 @xcite . while these models show great potential, it comes at the cost of high co 2 emissions. the core issue is that the electricity consumed is not guaranteed to be environmentally friendly, and often comes from sources such as coal or gas. according : these authors contributed to the paper equally. to @xcite , we must cut co 2 emissions by half to slow natural disasters. however, much research in the field ignores the perspective of energy efficiency. when looking at papers from three top ai conferences, namely acl, neurips, and cvpr, work tends to focus on accuracy rather than efficiency, or a mixture @xcite . an added benefit to developing more energyefficient models is a reduced barrier of entry to nlp research. researchers with good ideas may not be able to execute those ideas, given that state-of-theart results are locked behind large-scale compute @xcite . this study investigates how to reduce power consumption in training transformer language models. we seek to address the issue of high-power models by analysing the resulting models' hyperparameters, energy consumption, and perplexity and providing initial parameter guidelines for low-power, highperformance transformers, and an opening into the research of low-power transformers. our research question is: how can we reduce the energy consumption of models to both lower the barrier of entry and reduce co2 emissions, while still keeping an effective model? following @xcite , a possible approach to this problem could be the use of bayesian hyperparameter search. throughout our work, we managed to identify hyperparameter configurations that provide strong entries for both perplexity and energy consumption. these configurations were found through our methodology utilising bayesian optimisation, by combining the libraries hyperopt and pytorch. the optimal configurations collectively spanned the identified pareto frontier. ## related work there is a body of work on making language models and transformers efficient, for varying definitions of efficiency, but we're not aware of any that algo-rithmically integrate power consumption into the loss function or architecture search. the closest related work is that on task-specific network reduction and on low-power language processing. the former can be achieved through distillation, pruning, quantisation, or all of the above. for example, @xcite reduce trained bert models in size by orders of magnitude while retaining task performance. similarly, @xcite present highly efficient networks that as a result can process translations very quickly. however, all these techniques require the training of a large network first, thus only offering power savings at inference time. furthermore, @xcite presents scaling laws for neural language models, which can assist in more efficient training when applied. ## method our general approach is as follows. we specify a dataset, task, objective function, and hyperparameter space. we then explore hyperparameter space, repeatedly training models over the same data and evaluating them in terms of task performance and power consumption. this exploration optimises for good task performance and low power consumption, but is limited to a certain volume of model configurations. once complete, we analyse these model configurations further, investigating per-epoch performance, and common factors in high and low power consumption and task performance. ## energy the data here is presented in the same format as the previous section. the ## analysis for the analysis, we start with the most trivial optimisation steps and progress towards less trivial ones. ## conclusion we investigated how hyperparameters affected the power consumption and model quality of transformer-based language models. this paper has presented a method for low-power investigation of hyperparameter tuning, integrating power consumption measurement. we identified factors that increase power without giving quality and identified factors that increase quality without taking power. there are many possible extensions to this work and it is our hope that power consumption measurement will be improved and will be integrated with more architecture searches during model training.