HULK : An Energy Efficiency Benchmark Platform for Responsible Natural Language Processing (2021)

Abstract: Computation-intensive pretrained models have been taking the lead of many natural language processing benchmarks such as GLUE. However, energy efficiency in the process of model training and inference becomes a critical bottleneck. We introduce HULK, a multi-task energy efficiency benchmarking platform for responsible natural language processing. With HULK, we compare pretrained models’ energy efficiency from the perspectives of time and cost. Baseline benchmarking results are provided for further analysis. The fine-tuning efficiency of different pretrained models can differ significantly among different tasks, and fewer parameter number does not necessarily imply better efficiency. We analyzed such a phenomenon and demonstrated the method for comparing the multi-task efficiency of pretrained models. Our platform is available at https://hulkbenchmark.github.io/ .

## introduction environmental concerns of machine learning research have been rising as the carbon emission of specific tasks like neural architecture search reached an exceptional "ocean boiling" level @xcite . increased carbon emission has been one of the key factors to aggravate global warming 1 . research and development processes like parameter search further increase the environmental impact. when using cloud-based machines, the environmental impact is strongly correlated with the financial cost. the recent emergence of leaderboards such as squad @xcite , glue @xcite , and superglue @xcite has greatly boosted the development of advanced 1 source: https://climate.nasa.gov/causes/ models in the nlp community. pretrained models have proven to be the key ingredient for achieving state-of-the-art in conventional metrics. however, such models can be costly to train. for example, xlnet-large @xcite was trained on 512 tpu v3 chips for 500k steps, which costs around 61,440 dollars foot_0 , let alone staggeringly large carbon emission. moreover, despite impressive performance gain, the fine-tuning and inference efficiency of nlp models remain under-explored. as recently mentioned in a tweet foot_1 , the popular ai text adventure game ai dungeon has reached 100 million inferences. the energy efficiency of inference cost could be critical to both business planning and environmental impact. previous work @xcite on this topic proposed new metrics like fpo (floating-point operations) and other practices to report experimental results based on computing budget. other benchmarks like @xcite and @xcite compare the efficiency of models on the classic reading comprehension task squad and machine translation tasks. however, there has not been any concrete or practical reference for accurate estimation on nlp model pretraining, fine-tunning, and inference considering multi-task energy efficiency. energy efficiency can be reflected in many metrics, including carbon emission, electricity usage, time consumption, number of parameters, and fpo, as shown in @xcite . carbon emission and electricity are intuitive measures yet either hard to track or hardware-dependent. the number of model parameters does not reflect the actual cost for model training and inference. fpo model hardware time cost params bert base @xcite 4 tpu v2 pods 4 days @xmath06,912 334m xlnet base @xcite ---117m xlnet large @xcite 512 tpu v3 2.5 days @xmath175,203 125m roberta large @xcite 1024 v100 gpus 1 day @xmath265,536 223m distilbert* @xcite 8×16g v100 gpu 90 hours @xmath3294 14m electra base @xcite 16 tpu v3 96 hours @xmath48 per hour with 4 core tpu v3 chips and v100 gpu at $3.06 per hour. distilbert model was trained upon a pretrained bert model. parameter numbers are estimated using the pretrained models implemented in the transformers ( https://github.com/huggingface/ transformers ) library (wolf et al., 2019), shown in millions. the roberta model was trained for 100k steps. in most nlp pretrained model settings, there are three phases: pretraining, fine-tuning, and inference. if a model is trained from scratch, we consider such a model has no pretraining phase but is fine-tuned from scratch. typically pretraining takes several days and hundreds of dollars, according to our hulk benchmark, as shown in figure 1 , utilizes several classic datasets that have been widely adopted in the community as benchmarking tasks to benchmark energy efficiency. the benchmark compares pretrained models in a multi-task fashion. the tasks include natural language inference task mnli @xcite , sentiment analysis task sst-2 @xcite and named entity recognition task @xcite . such tasks are selected to provide a thorough comparison of end-to-end energy efficiency in pretraining, fine-tuning, and inference. with the hulk benchmark, we quantify the energy efficiency of model pretraining, fine-tuning, and inference phase by comparing the time and cost they require to reach a certain overall task-specific performance level on selected datasets. the design principle and benchmarking process are detailed in section 2. we also explore the relation between model parameters and fine-tuning efficiency and demonstrate energy efficiency consistency between different pretrained models' tasks. ## benchmark overview for the pretraining phase, the benchmark is designed to favor energy-efficient models in terms of time and cost that each model takes to reach specific multi-task performance pretrained from scratch. for example, we keep track of the time and cost of a bert model in the following way: after every thousand pretraining steps, we clone the model for fine-tuning and see if the final performance can reach our cut-off performance on different tasks. when the level is reached, time and cost for pretraining are used for comparison. modwe consider the time and cost each model takes to reach specific multi-task performance fine-tuned from given pretrained models for the fine-tuning phase. for each task with different difficulty and instance numbers, the fine-tuning characteristics may differ a lot. when pretrained models are used to deal with a non-standard downstream task, especially ad hoc application in industry, the task's fine-tuning time and cost cannot be estimated directly from any other standard task. therefore, it is essential to compare the multi-task efficiency for model choice. for the inference phase, each model's time and cost for making inference on a single instance on multiple tasks are compared similarly to the fine-tuning phase. specially, we estimate the time elapsed for each inference by averaging thousands of inference samples. ## baseline settings and analysis we adopt the reported resource requirements in the original papers as the pretraining phase baselines for computation-intensive tasks. for fine-tuning and inference phase, we conduct extensive experiments on given hardware (rtx 2080ti gpu) with different model settings as shown in in our fine-tuning setting, we are given a specific hardware and software configuration. we adjust the hyper-parameter using grid search to minimize the time fine-tuning towards cut-off performance. for example, we choose the proper batch size and learning rate for bert base to make sure the model converges and can reach expected performance as soon as possible with parameter searching. as shown in figure 2 , the fine-tuning performance curve differs a lot among pretrained models. the x-axis denoting time consumed is shown in log-scale for a better comparison of different models. none of the models take the lead in all tasks. however, if two pretrained models are in the same family, such as bert base and bert large , the model with a smaller number of parameters tend to converge a bit faster than the other in the ner and sst-2 task. in the mnli task, such a trend does not apply because of the increased difficulty level and the number of training instances, which favors a larger model capacity. even though albert model has a lot fewer parameters than bert, according to mlperf @xcite compares training and inference efficiency from a hardware perspective, providing helpful resources on hardware selection and model training. the benchmark focused on several typical applications, including image classification and machine translation. however, it does not separate different training phases, thus making it hard to find the reference for fine-tuning only applications. previous work @xcite on related topic working towards "green ai" proposes new metrics like fpo and new principle in efficiency evaluation. we further make more detailed and practical contributions to model energy efficiency benchmarking. other work like dawnbenchmark @xcite looks into the area of end-to-end model efficiency comparison for both computer vision and nlp task squad. the benchmark is very detailed and intuitive. however, it does not compare multitask efficiency performance and covered only one nlp task. similar to mlperf, it does not separate fine-tuning efficiency from training efficiency. the efficient nmt shared task of the 2nd workshop on neural machine translation and generation proposed an efficiency track to compare the inference time of neural machine translation models. our platform covers more phases and supports multi-task comparison. ## conclusion we developed the hulk platform focusing on the energy efficiency benchmarking of nlp models based on their end-to-end performance on selected nlp tasks. the hulk platform compares models in the pretraining, fine-tuning, and inference phase, making it clear to follow and propose more trainingefficient and inference-efficient models. we have compared the fine-tuning efficiency of given models during baseline testing and demonstrated more parameters lead to slower fine-tuning when using the same model design but do not hold when the model architecture changes.we expect more submissions in the future to flourish and enrich our benchmark.