Characterizing the Efficiency vs. Accuracy Trade-off for Long-Context NLP Models (2022)

Abstract: With many real-world applications of Natural Language Processing (NLP) comprising of long texts, there has been a rise in NLP benchmarks that measure the accuracy of models that can handle longer input sequences. However, these benchmarks do not consider the trade-offs between accuracy, speed, and power consumption as input sizes or model sizes are varied. In this work, we perform a systematic study of this accuracy vs. efficiency trade-off on two widely used long-sequence models - Longformer-Encoder-Decoder (LED) and Big Bird - during fine-tuning and inference on four datasets from the SCROLLS benchmark. To study how this trade-off differs across hyperparameter settings, we compare the models across four sequence lengths (1024, 2048, 3072, 4096) and two model sizes (base and large) under a fixed resource budget. We find that LED consistently achieves better accuracy at lower energy costs than Big Bird. For summarization, we find that increasing model size is more energy efficient than increasing sequence length for higher accuracy. However, this comes at the cost of a large drop in inference speed. For question answering, we find that smaller models are both more efficient and more accurate due to the larger training batch sizes possible under a fixed resource budget.

## introduction over the past few years, advances in sequence modeling have led to impressive results on several nlp benchmarks @xcite . a closer look at these results reveals that higher accuracies are typically achieved by increasingly larger and computationally intensive models, which have large carbon footprints that can have an adverse effect on the environment @xcite . this has led to the green ai initiative, which urges researchers to consider energy and computational efficiency when evaluating models in order to promote those which achieve high accuracies with smaller carbon footprints @xcite . however, although it has been a few years since green ai was introduced, efficiency metrics have still not been integrated into many recently proposed benchmarks such as the long range arena (lra) @xcite and scrolls @xcite . these benchmarks serve as a strong basis for comparison between transformer models in terms of accuracy. however, improved accuracy is often obtained by either increasing the input sequence length or the model size, and the energy cost of these improvements is not clear. moreover, previous characterizations of model efficiency in terms of speed (e.g., in lra) only focus on intermodel comparisons, keeping model sizes and input sequence lengths fixed. here, we argue that the accuracy-vs-efficiency trade-off also has implications for intra-model comparisons when selecting hyperparameters -e.g., increasing the sequence length might positively impact accuracy but may also negatively impact efficiency metrics. as a result, when faced with a fixed resource budget, it is not clear whether practitioners should opt for increasing the model size or increasing the input length for the most efficient use of resources. in this work, we perform a systematic study of the trade-off between efficiency and accuracy for two widely used long-context nlp models -big bird @xcite and longformer-encoder-decoder (led) @xcite ) -on four datasets from the scrolls benchmark. 1 we characterize efficiency using several metrics, including the total energy consumption during training, training speed, inference speed, and power efficiency. we compare the models across several different input lengths and two different model sizes (base and large). overall, for summarization, we find that, perhaps surprisingly, increasing model size is a more energy efficient way of increasing accu-racy as compared to increasing sequence length. however, if inference speed is the main efficiency metric of interest, then smaller models should be preferred. for question answering, on the other hand, we find that using smaller models is more efficient in terms of all metrics and more accurate due to the larger training batch sizes allowed under a fixed resource budget. ## methodology our main contribution is an analysis of how different sequence lengths affect the trade-off between accuracy, power, and speed in long-context transformer models during fine-tuning and inference. since our focus is on long-context nlp tasks, we investigated the following four input sequence lengths: 1024, 2048, 3072, and 4096.