Efficient Methods for Natural Language Processing: A Survey (2023)

Abstract: Recent work in natural language processing (NLP) has yielded appealing results from scaling model parameters and training data; however, using only scale to improve performance means that resource consumption also grows. Such resources include data, time, storage, or energy, all of which are naturally limited and unevenly distributed. This motivates research into efficient methods that require fewer resources to achieve similar results. This survey synthesizes and relates current methods and findings in efficient NLP. We aim to provide both guidance for conducting NLP under limited resources, and point towards promising research directions for developing more efficient methods.

## introduction scaling has become a key ingredient in achieving state-of-the-art performance in nlp (figure 1 ), as recent research suggests that some capabilities only emerge once models grow beyond a certain size @xcite . however, despite the merits of scaling, it poses key challenges to making these breakthroughs accessible in resourceconstrained environments (ahmed and wahed, 2020), in having a non-negligible environmental impact @xcite @xcite @xcite , and in complying with hardware constraints @xcite . to * equal contribution. marcos.treviso@tecnico.pt. definition efficiency is characterized by the relationship between resources going into a system and its output, with a more efficient system producing the same output with fewer resources. @xcite formalize efficiency as the cost of a model in relation to the results it produces: cost(r) ∝ e • d • h, i.e., the cost(•) of producing a certain nlp (r)esult as proportional to three (non-exhaustive) factors: (1) the cost of model execution on a single (e)xample, (2) the size of the (d)ataset, and (3) the number of training runs required for (h)yperparameter tuning. here we take a different approach, and consider the role that efficiency plays across the different steps in the nlp pipeline, by providing a detailed overview of efficiency methods specific to nlp (figure 2 ). ## model design efficient model design covers architectural changes and adding new modules to accelerate training. ## pre-training modern transfer learning approaches in nlp typically involve pre-training a model in a selfsupervised fashion on large amounts of text before fine-tuning it on specific tasks (section 5). improving the pre-training procedure of a model can significantly reduce the cost of hyperparameter tuning and increase data efficiency for fine-tuning @xcite @xcite . ## inference and compression inference involves computing a trained model's prediction for a given input. inference can be made more efficient by accelerating the process for time efficiency (latency), or by compressing the model to reduce memory requirements. ## hardware utilization many hardware-specific methods focus on reducing gpu memory consumption, a major bottleneck in transformer models. others leverage specialized hardware, co-design of hardware, and adaptations targeted to edge devices. many techniques can be combined and applied across different stages of training and inference (figure 2 ) for further efficiency. ## evaluating efficiency evaluating efficiency requires establishing which computational aspect one aims to minimize. we discuss the two most prominent aspects (flop/s and power consumption), and list open challenges. ## conclusion this survey provides a broad overview of considerations for increasing efficiency in modern nlp models, identifying both immediate successes and remaining challenges. most progress so far has been in model design, typically targeted at a specific computational budget and hard-ware paradigm. key challenges include better understanding and modeling trade-offs between end-task performance and resource consumption, and the dependency between hardware choices and software implementations. furthermore, we note that efficiency in nlp has many definitions and can be achieved in many different ways, but is also subject to various open challenges, and cannot be measured by a single metric. we outline several promising research directions aligned with overcoming these challenges, ranging from approaches that make better use of available data, strategies for reducing the cost of pre-training and fine-tuning large models, to prioritizing the importance of interactions between algorithms, software, and hardware. impressive advances in nlp enabled primarily by scaling computation have produced remarkable progress in a short span of time. however, in order to realize the full potential of this technology for a broader swath of society, we must reduce the amount of computation that is required to achieve these remarkable results. we hope that this survey can serve to accelerate advances in this important area of research with great potential for impact both within our field and for society as a whole. references chirag agarwal, daniel d'souza, and sara hooker. 2022. estimating example difficulty using variance of gradients. in proceedings of the ieee/cvf conference on computer vision and pattern recognition (cvpr), pages 10368-10378. https://doi.org /10.1109/cvpr52688.2022.01012 rishabh agarwal, max schwarzer, pablo samuel castro, aaron c. courville, and marc bellemare. 2021. deep reinforcement learning at the edge of the statistical precipice. in advances in neural information processing systems, volume 34, pages 29304-29320. curran associates, inc. armen aghajanyan, anchit gupta, akshat shrivastava, xilun chen, luke zettlemoyer, and sonal gupta. 2021a. muppet: massive multi-task representations with pre-finetuning. in proceedings of the 2021 conference on empirical methods in natural language processing, pages 5799-5811, online and punta cana, dominican republic. association for computational linguistics. https://doi.org/10 .18653/v1/2021.emnlp-main.468 armen aghajanyan, sonal gupta, and luke zettlemoyer. 2021b. intrinsic dimensionality explains the effectiveness of language model fine-tuning. in proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pages 7319-7328, online. association for computational linguistics. https://doi.org/10.18653/v1 /2021.acl-long.568 ameeta agrawal, suresh singh, lauren schneider, and michael samuels. 2021. on the role of corpus ordering in language modeling. in proceedings of the second workshop on simple and efficient natural language processing, pages 142-154, virtual. association for computational linguistics. orevaoghene ahia, julia kreutzer, and sara hooker. 2021. the low-resource double bind: an empirical study of pruning for low-resource machine translation. in findings of the association for computational linguistics: emnlp 2021, pages 3316-3333, punta cana, dominican republic. association for computational linguistics. nur ahmed and muntasir wahed. 2020. the de-democratization of ai: deep learning and the compute divide in artificial intelligence research. arxiv preprint arxiv:2010.15581v1. joshua ainslie, santiago ontanon, chris alberti, vaclav cvicek, zachary fisher, philip pham, anirudh ravula, sumit sanghai, qifan wang, and li yang. 2020. etc: encoding long and structured inputs in transformers. in proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pages 268-284, online. association for computational linguistics. https://doi .org/10.18653/v1/2020.emnlp-main.19 ahmed alajrami and nikolaos aletras. 2022. how does the pre-training objective affect what large language models learn about linguistic properties? in proceedings of the 60th annual meeting of the association for computational linguistics (volume 2: short papers), pages 131-147, dublin, ireland. association for computational linguistics. https://doi .org/10.18653/v1/2022.acl-short.16 uri alon, frank xu, junxian he, sudipta sengupta, dan roth, and graham neubig. 2022. neuro-symbolic language modeling with automaton-augmented retrieval. in proceedings of the 39th international conference on machine learning, volume 162 of proceedings of machine learning research, pages 468-485. pmlr. lasse f. wolff anthony, benjamin kanding, and raghavendra selvan. 2020. carbontracker: tracking and predicting the carbon footprint of training deep learning models. in proceedings of the workshop on challenges in deploying and monitoring machine learning systems, icml. vamsi aribandi, yi tay, tal schuster, jinfeng rao, huaixiu steven zheng, sanket vaibhav mehta, honglei zhuang, vinh q. tran, dara bahri, jianmo ni, jai gupta, kai hui, sebastian ruder, and donald metzler. 2022. ext5: towards extreme multi-task scaling for transfer learning. in international conference on learning representations. the fourth workshop on neural generation and translation. association for computational linguistics, online. yonatan bitton, michael elhadad, gabriel stanovsky, and roy schwartz. 2021. data efficient masked language modeling for vision and language. in findings of the association for computational linguistics: emnlp 2021, pages 3013-3028, punta cana, dominican republic. association for computational linguistics. https://doi.org/10.18653 /v1/2021.findings-emnlp.259 davis blalock, jose javier gonzalez ortiz, jonathan frankle, and john guttag. 2020. what is the state of neural network pruning? proceedings of machine learning and systems, 2:129-146. zalán bodó, zsolt minier, and lehel csató. 2011. active learning with clustering. in active learning and experimental design workshop in conjunction with aistats 2010, pages 127-139. jmlr workshop and conference proceedings. nikolay bogoychev, roman grundkiewicz, alham fikri aji, maximiliana behnke, kenneth heafield, sidharth kashyap, emmanouil-ioannis farsarakis, and mateusz chudyk. 2020. edinburgh's submissions to the 2020 machine translation efficiency task. in proceedings of the fourth workshop on neural generation and translation, pages 218-224, online. association for computational linguistics. young jin kim, marcin junczys-dowmunt, hany hassan, alham fikri aji, kenneth heafield, roman grundkiewicz, and nikolay bogoychev. 2019. from research to production and back: ludicrously fast neural machine translation. in proceedings of the 3rd workshop on neural generation and translation, pages 280-288, hong kong. association for computational linguistics. andreas kirsch, joost van amersfoort, and yarin gal. 2019. batchbald: efficient and diverse batch acquisition for deep bayesian active learning. in advances in neural information processing systems, volume 32. curran associates, inc. nikita kitaev, lukasz kaiser, and anselm levskaya. 2020. reformer: the efficient transformer. in international conference on learning representations.