The Framework Tax: Disparities Between Inference Efficiency in NLP Research and Deployment (2023)

Abstract: Increased focus on the computational efficiency of systems in natural language processing has motivated the design of efficient model architectures and improvements to underlying hardware accelerators. However, the resulting increases in computational throughput and reductions in floating point operations have not directly translated to improvements in wall-clock inference latency. We demonstrate that these discrepancies can be largely attributed to bottlenecks introduced by deep learning frameworks. We denote this phenomena as the framework tax, and observe that the disparity is growing as hardware speed increases over time. In this work, we examine this phenomena through a series of case studies analyzing the effects of model design decisions, framework paradigms, and hardware platforms on total model latency. Based on our findings, we provide actionable recommendations to researchers and practitioners aimed at narrowing the gap between efficient NLP model research and practice.

## introduction natural language processing systems have benefited from improvements in performance driven by scaling of training data and number of model parameters @xcite @xcite . however, the accompanying increases in computation raise concerns as to the efficiency of these systems due to associated environmental costs of development and deployment @xcite . in particular, efficiency is especially important in inference settings where models are used repeatedly and at scale, in contrast to training which poses a single upfront computational cost. for example, meta reports that inference workloads make up 70% of their ai power consumption, with the remaining 30% due to training and development @xcite , while google attributes 60% of their ml energy consumption to inference @xcite . inference is also estimated to make up 80 to 90% of ml cloud computing demand @xcite . in these settings, metrics of model speed such as latency and throughput are essential for inference workloads that are subject to real wall-clock time constraints such as real-time natural language generation and automated speech recognition @xcite . these concerns have motivated research in designing more efficient neural network model architectures and faster hardware accelerators. in the past five years alone, the number of papers that mention the terms efficient or efficiency in top machine learning venues has grown by over 2.5x and even more so at venues in natural language processing, increasing by 8.3x in the same span. 1 this has spurred innovations in the design of efficient neural network architectures for language aiming to reduce the number of trainable model parameters, floating point or multiply-accumulate operations (macs) @xcite @xcite . over the same period, gpu hardware accelerators have seen similar performance gains with the number of floating point operations per second (flops) growing by over 175%. 2 despite this progress, the supposed gains offered by higher performance hardware and more efficient models are often not realized in inference settings, where wall-clock model speed has not reliably improved, as seen in figure 1 . we show that this misalignment is primarily attributable to overhead incurred by deep learning frameworks used to implement and execute models. in the past, the execution of large neural networks common to natural language processing has been assumed to be compute-bounded by computationally intensive tensor operations @xcite . however, as the speed of hardware increases, the overhead introduced by the deep learning frameworks used to implement and deploy these models is no longer negligible and imposes bottlenecks on inference. in this work, we conduct a systematic investigation in which we show that improvements to neural network architectures and hardware have not translated into reductions in inference latency. we show that this breakdown is largely due to overhead introduced by deep learning frameworks. we refer to this phenomenon as the framework tax, and show that it exists across all deep learning framework paradigms (e.g. eager execution, just-in-time, and ahead-of-time compilation). at small batch sizes and sequence lengths, we show that fixed framework overhead dominates inference time leading proxy measurements of model efficiency, such as macs and parameter count, to breakdown as predictors of inference latency. furthermore, we note that existing efforts to improve model efficiency by designing lower flop model architectures and faster gpu kernels do not reduce latency, when total execution is bound by fixed-cost framework overhead. moreover, we note that as hardware performance increases, nlp systems will become increasingly framework-bound at larger batch sizes. an exhaustive comparison of the rapidly growing space of models and hardware platforms is out of scope, so we instead identify the most popular model components, inference environments, and hardware accelerators for testing (i.e. isolating the cases most likely to mislead a practitioner). we analyze the performance of transformer and convo-2 per theoretical flops of nvidia gpus @xcite @xcite @xcite . based on our findings, we provide a series of recommendations for nlp researchers and practitioners presented through a collection of case studies. among these, we recommend usage of static or ahead-of-time inference runtimes when batch sizes are small as they can substantially reduce framework overhead. alternatively, when using eager execution-based frameworks for inference, we recommend increasing model width or batch size at no cost to latency and take advantage of performance gains associated with increased model capacity @xcite . for example, hidden dimensions in self-attention and fully connected layers can be doubled to increase model capacity without affecting latency, implying that model designers have an extra degree of freedom often overlooked when designing around parameters or flops. we hope that our analysis and recommendations will help bridge the gap between efficient nlp research and practice. ## experiments we evaluate models frameworks from each of the major paradigms: eager execution pytorch, deferred execution torchscript, and statically compiled onnx runtime with a cuda backend. we use pytorch 1.12.1 with cuda 11.6 and python 3.8.13. we use onnx runtime 1.7.0 with cuda 11.1.1 and cudnn v8.0.4.3. baseline experiments are run on a compute node with an nvidia rtx-8000 gpu and an intel xeon e5-2630 cpu with 32 gb of ddram memory. we simulate text data by randomly generating token sequences of length 128, as commonly used in sentence classification tasks @xcite . we report averaged metrics from 100 forward passes after 10 warm-up passes to initialize model weights and data. gpu core and memory utilization are measured with the nvidia nsight compute and nvidia management library. we select bert-base @xcite and resnet-50 @xcite as representative models for encoder-only and convolutional neural network architectures commonly used for sentence and image classification tasks @xcite . we evaluate model architectures for natural language generation, speech, and vision and show that they all observe framework bound behavior in appendices c, d and e. ## discussion computational graph optimizations and compilation improve latency. removal of host language dependencies and graph optimizations provides substantial speedups over eager frameworks for inference at low batch sizes. however, feature completeness for operators and control flow varies across graph optimizers and compilers. for example, the fnet architecture @xcite relies on ffts as a deterministic swap-in for self-attention. fft operations are not currently supported by onnx or torchscript as expected, fnet executed in pytorch outperforms bert executed in onnx despite less framework overhead and numerous static optimizations -with a 10.31% speedup at batch size 1. for improvements in research to translate to deployment, additional investment can be directed towards support for complex control flow and operators in inference runtimes. dynamic computational graphs can be faster for input sentences with variable lengths. dynamic computational graphs can leverage input sparsity to reduce latency when processing variable length text. for example, pytorch with sparse tensor optimizations reduces the latency of static cuda graphs by 80.56% at batch size 128 when processing sparse inputs. at large input sizes, framework overhead from graph operations is negligible. for batch sizes larger than 16, we find that there is minimal latency difference across models, inference runtimes, and frameworks. in the compute-bound regime, number of flops is still a poor latency predictor due to variable execution time for different operations. for example, efficient mobile architectures that depend on inverted-residual layers are memory inefficient and are much slower per-flop than standard convolution and linear layers. for framework-bound models, model depth is a reasonable proxy for latency. number of floating point operations is a poor indicator of latency in a framework-bound setting, as total runtime is generally constant and tied to framework overheads and the size of the computational graph. in framework-bound models, the size of the computational graph is related to model depth. estimations of latency for models deployed in production settings must account for their target framework and hardware platform. model development frequently occurs using eager execution research frameworks. however, deployment often occurs in inference runtimes and on mobile devices or specialized hardware. this misalignment can mislead the development of "efficient" models and result in claimed gains that do not translate to realworld deployment settings. as such, researchers should be clear in specifying the setting of their "efficiency" gains, such as the target frameworks and hardware platform, when developing new methods. for example, techniques such as hardware-aware neural architecture search which leverage direct latency measures must also control for framework choices to account for this mismatch. throughput and input size can be increased at minimal cost for framework-bound models. for a given model, latency is constant regardless of batch size until compute kernels saturate and exceed cpu launch costs. if computation is bot- model width can be increased at no cost for framework-bound models. for a given batch size, individual layers of a framework bound model can be made wider by increasing hidden dimension or filter size without impacting latency. wider models are known to exhibit better task performance due to increased parameter count and expressivity @xcite . model designers can leverage wider architectures with their target inference framework and hardware setting in mind to achieve higher utilization. for example, models processing few examples during inference can leverage wider layers to avoid framework bottlenecks. see using higher-performing hardware does not necessarily improve end-to-end performance. framework overhead limits the impact of improved hardware as it limits utilization. this trend will continue as ml-specific hardware advances without efforts to address software bottlenecks. for example, single-example inference with both bert is slower using an a100 than using a v100 gpu despite a 2.75x increase in peak computational throughput. ## conclusion we conduct an extensive study of neural networks from the convolutional and transformer architecture paradigms across a variety of software and hardware platforms. we show that inference performed with these large neural networks, which was previously assumed to be compute bounded, is in fact limited by overhead incurred by deep learning frameworks. while wider transformer architectures (e.g. bert-base) exhibit less boundedness behaviors than narrower, deeper cnns (e.g. resnet-50), we show that all models exhibit framework boundedness. additionally, we observe that these inefficiencies are becoming more apparent as hardware accelerator speeds increase. we introduce the concept of the framework tax to describe when improvements in hardware speed and reductions in required computation fail to translate to speedups in model latency and throughput due to bottlenecks incurred by deep learning frameworks. we hope that these observations raise awareness of the impact and limitations created by choice of deep learning frameworks on model development and deployment.