Overview of the S ustai NLP 2020 Shared Task (2020)

Abstract: We describe the SustaiNLP 2020 shared task: efficient inference on the SuperGLUE benchmark (Wang et al., 2019). Participants are evaluated based on performance on the benchmark as well as energy consumed in making predictions on the test sets. We describe the task, its organization, and the submitted systems. Across the six submissions to the shared task, participants achieved efficiency gains of 20× over a standard BERT (Devlin et al., 2019) baseline, while losing less than an absolute point in performance.

## introduction while ever-larger pretrained language models have led to impressive gains across a variety of natural language processing (nlp) tasks, there is growing concern about the environmental impact of training and deploying these models @xcite . in response, there has been a growing body of research focusing on making these large models smaller and more efficient with minimal sacrifice to performance @xcite . the sustainlp 2020 shared task focuses on the development of computationally and energy efficient nlp systems. the task uses the superglue benchmark @xcite , a standard benchmark for natural language understanding. systems are evaluated on both the benchmark score as well as the energy consumed in evaluating the system on the benchmark. participants are therefore incentivized to develop models that are energy efficient while maintaining the high performance of recent models. the shared task received six submissions that employed a large variety of optimizations to improve system efficiency. overall, the submitted systems were on average 20× more efficient than a standard baseline using pretrained language models while nearly matching baseline performance. ## submissions we provided participants with a simple baseline that follows the standard paradigm of finetuning a pretrained language model to each task. for pretrained models, we use bert-base @xcite and roberta-large @xcite , as provided by the huggingface transformers library @xcite . there were six submissions to the shared task, four submissions to the gpu track and two submissions to the cpu track. all submissions were provided by @xcite . we provide a brief description of the six submissions below; see @xcite for in-depth descriptions. systems 1-* are submissions to the gpu track and systems 3-* are submissions to the cpu track. ## results energy and task results are respectively presented in tables 1 and among the systems, we find that the hybrid submissions (1-3, 1-4) consistently consume more power than the gpu-only counterparts (1-1, 1-2). all of the submissions that use a gpu (1-*) substantially outperform those that do not (3-*), which is in large part due to the large test set for record. we observe fairly high variance between similar systems (1-1 and 1-2; 1-3 and 1-4; 3-1 and 3-2). in the worst case, systems 3-1 and 3-2 only differ by the multirc model, but the energy consumption varies significantly. we attribute this variance to runtime differences in the environment. task performances are consistently around 2 absolute points lower in the submitted systems than the baseline, except for copa, where the submitted systems outperform the baseline. however, given the large efficiency improvements over the baseline, this tradeoff seems favorable. ## conclusion we describe the results of the sustainlp 2020 shared task. the six submissions were able to substantially improve over the baseline systems, obtaining improvements 20× in energy consumption while only losing a point in performance. to achieve these results, the submissions employed efficiency optimizations at numerous levels, including model architecture, storage, and runtime, which hints at the rich design space for efficient machine learning models.