Original: T HE E FFICIENCY M ISNOMER
Revised: T HE E FFICIENCY M ISNOMER

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Model efﬁciency is a critical aspect of developing and deploying machine learning models.
Revised: Model efﬁciency is a critical aspect of developing and deploying machine learning models.

Original: Inference time and latency directly affect the user experience, and some applications have hard requirements.
Revised: Inference time and latency directly affect the user experience, and some applications have hard requirements.

Original: In addition to inference costs, model training also have direct ﬁnancial and environmental impacts.
Revised: In addition to inference costs, model training also have direct ﬁnancial and environmental impacts.

Original: Although there are numerous well-established metrics (cost indicators) for measuring model efﬁciency, researchers and practitioners often assume that these metrics are correlated with each other and report only few of them.
Revised: Although there are numerous well-established metrics (cost indicators) for measuring model efﬁciency, researchers and practitioners often assume that these metrics are correlated with each other and report only few of them.

Original: In this paper, we thoroughly discuss common cost indicators, their advantages and disadvantages, and how they can contradict each other.
Revised: In this paper, we thoroughly discuss common cost indicators, their advantages and disadvantages, and how they can contradict each other.

Original: We demonstrate how incomplete reporting of cost indicators can lead to partial conclusions and a blurred or incomplete picture of the practical considerations of different models.
Revised: We demonstrate how incomplete reporting of cost indicators can lead to partial conclusions and a blurred or incomplete picture of the practical considerations of different models.

Original: We further present suggestions to improve reporting of efﬁciency metrics.
Revised: We further present suggestions to improve reporting of efﬁciency metrics.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Aside from model quality, the efﬁciency (Menghani, 2021) of a model is often an important aspect to consider and is commonly used to measure the relative utility of different methods.
Revised: Aside from model quality, the efﬁciency (Menghani, 2021) of a model is often an important aspect to consider and is commonly used to measure the relative utility of different methods.

Original: After all, training time spent on accelerators is directly linked to ﬁnancial costs and environmental impact.
Revised: After all, training time spent on accelerators is directly linked to ﬁnancial costs and environmental impact.

Original: Meanwhile, the speed of a model may be directly linked to user experience.
Revised: Meanwhile, the speed of a model may be directly linked to user experience.

Original: To this end, there have been well-established ways in the literature to assess and report the efﬁciency of a model such as number of trainable parameters, number of ﬂoating-point operations (FLOPs), and speed/throughput.
Revised: To this end, there have been well-established ways in the literature to assess and report the efﬁciency of a model such as number of trainable parameters, number of ﬂoating-point operations (FLOPs), and speed/throughput.

Original: While it is commonly assumed that these cost indicators are correlated (e.g., a lower number of parameters would translate to a higher throughput)
Revised: While it is commonly assumed that these cost indicators are correlated (e.g., a lower number of parameters would translate to a higher throughput)

Original: we show that this might not necessarily be the case.
Revised: we show that this might not necessarily be the case.

Original: Therefore, incomplete reporting across the spectrum of cost indicators may lead to an incomplete picture of the metrics, advantages and drawbacks of the proposed method.
Revised: Therefore, incomplete reporting across the spectrum of cost indicators may lead to an incomplete picture of the metrics, advantages and drawbacks of the proposed method.

Original: To this end, we show that it may also be possible to, perhaps unknowingly, misrepresent a model’s efﬁciency by only reporting favorable cost indicators.
Revised: To this end, we show that it may also be possible to, perhaps unknowingly, misrepresent a model’s efﬁciency by only reporting favorable cost indicators.

Original: Moreover, the choice of cost indicators may also result in unfair, incomplete, or partial conclusions pertaining to model comparisons.
Revised: Moreover, the choice of cost indicators may also result in unfair, incomplete, or partial conclusions pertaining to model comparisons.

Original: We refer to this phenomenon as the ‘ efﬁciency
Revised: We refer to this phenomenon as the ‘ efﬁciency

Original: misnomer
Revised: misnomer

Original: The overall gist of the efﬁciency misnomer is that no single cost indicator is sufﬁcient.
Revised: The overall gist of the efﬁciency misnomer is that no single cost indicator is sufﬁcient.

Original: Incomplete reporting (e.g., showing only FLOPs, or the number of trainable parameters) as a measure of efﬁciency can be misleading.
Revised: Incomplete reporting (e.g., showing only FLOPs, or the number of trainable parameters) as a measure of efﬁciency can be misleading.

Original: For example, a model with low FLOPs may not actually be fast, given that FLOPs does not take into account information such as degree of parallelism (e.g., depth, recurrence) or hardware-related details like the cost of a memory access.
Revised: For example, a model with low FLOPs may not actually be fast, given that FLOPs does not take into account information such as degree of parallelism (e.g., depth, recurrence) or hardware-related details like the cost of a memory access.

Original: Despite this, FLOPs has been used as the most common cost indicator in many research papers, especially in the recent computer vision literature, to quantify model efﬁciency (Szegedy et al., 2015;
Revised: Despite this, FLOPs has been used as the most common cost indicator in many research papers, especially in the recent computer vision literature, to quantify model efﬁciency (Szegedy et al., 2015;

Original: He et al., 2016; Tan and Le, 2019;
Revised: He et al., 2016; Tan and Le, 2019;

Original: Feichtenhofer et al., 2019; Fan et al., 2021).
Revised: Feichtenhofer et al., 2019; Fan et al., 2021).

Original: Likewise, the number of trainable parameters (size of the model) despite being commonly used as the de-facto cost indicator in the NLP community (Devlin et al., 2018; Liu et al., 2019; Lan et al., 2019) and previously the vision community (Krizhevsky et al., 2012; Simonyan and Zisserman, 2015;
Revised: Likewise, the number of trainable parameters (size of the model) despite being commonly used as the de-facto cost indicator in the NLP community (Devlin et al., 2018; Liu et al., 2019; Lan et al., 2019) and previously the vision community (Krizhevsky et al., 2012; Simonyan and Zisserman, 2015;

Original: Huang et al., 2017; Tan and Le, 2019), can also be misleading when used as a standalone measure of efﬁciency.
Revised: Huang et al., 2017; Tan and Le, 2019), can also be misleading when used as a standalone measure of efﬁciency.

Original: Intuitively, a model can have very few trainable parameters and still be very slow, for instance when the parameters are shared among many computational steps (Lan et al., 2019;
Revised: Intuitively, a model can have very few trainable parameters and still be very slow, for instance when the parameters are shared among many computational steps (Lan et al., 2019;

Original: Dehghani et al., 2018).
Revised: Dehghani et al., 2018).

Original: While the number of trainable parameters can often be insightful to decide if a model ﬁts in memory, it is unlikely to be useful as a standalone cost indicator.
Revised: While the number of trainable parameters can often be insightful to decide if a model ﬁts in memory, it is unlikely to be useful as a standalone cost indicator.

Original: That said, it is still common practice to parameter-match models to make ‘fair’ comparisons (Mehta et al., 2020;
Revised: That said, it is still common practice to parameter-match models to make ‘fair’ comparisons (Mehta et al., 2020;

Original: [Figure]
Revised: [Figure]

Original: Lee-Thorp et al., 2021; Tay et al., 2020a; Xue et al., 2021; Wightman et al., 2021), even if one model is in reality slower or faster than another.

Original: Meanwhile, using throughput/speed as the primary indicator of efﬁciency can also be problematicsince this tightly couples implementation details, hardware optimizations, and infrastructure details (e.g., input pipeline latency) into the picture.
Revised: Meanwhile, using throughput/speed as the primary indicator of efﬁciency can also be problematicsince this tightly couples implementation details, hardware optimizations, and infrastructure details (e.g., input pipeline latency) into the picture.

Original: Hence, this might not present an apples-to-apples comparison of certain methods or worse, across different infrastructures or hardware.
Revised: Hence, this might not present an apples-to-apples comparison of certain methods or worse, across different infrastructures or hardware.

Original: Given that the landscape of research on model architectures is diverse, the relationship between cost indicators may strongly deviate from the norm, and learning how to fairly compare models within the context of efﬁciency-based cost indicators is crucial.
Revised: Given that the landscape of research on model architectures is diverse, the relationship between cost indicators may strongly deviate from the norm, and learning how to fairly compare models within the context of efﬁciency-based cost indicators is crucial.

Original: For instance, there seems to be a rising trend towards sparse models (Fedus et al., 2021;
Revised: For instance, there seems to be a rising trend towards sparse models (Fedus et al., 2021;

Original: Riquelme et al., 2021) which usually have an incredibly large number of trainable parameters but maintain the FLOPs and speed of dense models.
Revised: Riquelme et al., 2021) which usually have an incredibly large number of trainable parameters but maintain the FLOPs and speed of dense models.

Original: On the contrary, there are models that are considered lightweight due to their small number of trainable parameters (Lan et al., 2019; Dehghani et al., 2018) but, in actual practice, consume similar amounts of compute.
Revised: On the contrary, there are models that are considered lightweight due to their small number of trainable parameters (Lan et al., 2019; Dehghani et al., 2018) but, in actual practice, consume similar amounts of compute.

Original: Figure 1 shows an example of how the scaling behavior of a model with respect to parameter count can look favorable, while taking the FLOPs or throughput as the cost indicator, different model scales much better.
Revised: Figure 1 shows an example of how the scaling behavior of a model with respect to parameter count can look favorable, while taking the FLOPs or throughput as the cost indicator, different model scales much better.

Original: This example shows how looking at one metric can be deceptive and cost a lot of time and resources, e.g. by choosing the wrong candidate for scaling up.
Revised: This example shows how looking at one metric can be deceptive and cost a lot of time and resources, e.g. by choosing the wrong candidate for scaling up.

Original: Besides the fact that different cost indicators capture different aspects, they can be chosen to reﬂect either the cost of training or the cost at inference time.
Revised: Besides the fact that different cost indicators capture different aspects, they can be chosen to reﬂect either the cost of training or the cost at inference time.

Original: Both training and inference costs can be crucial, depending on the context.
Revised: Both training and inference costs can be crucial, depending on the context.

Original: Also, a single cost indicator can favor a model over another during inference, but not training (or vice versa).
Revised: Also, a single cost indicator can favor a model over another during inference, but not training (or vice versa).

Original: For instance, a model that shares parameters in depth is particularly memory-efﬁcient during inference, but during training, the size of activation that need to be kept for the backward pass is just as large as for a similar model with no parameter sharing.
Revised: For instance, a model that shares parameters in depth is particularly memory-efﬁcient during inference, but during training, the size of activation that need to be kept for the backward pass is just as large as for a similar model with no parameter sharing.

Original: The overarching conundrum here is that ﬁrst of all, no single cost indicator captures a holistic view that is universally useful to all practitioners or researchers.
Revised: The overarching conundrum here is that ﬁrst of all, no single cost indicator captures a holistic view that is universally useful to all practitioners or researchers.

Original: We show that the trade-offs between cost indicators fall far from the standard assumptions and can be non-trivial to navigate.
Revised: We show that the trade-offs between cost indicators fall far from the standard assumptions and can be non-trivial to navigate.

Original: Moreover, we argue that cost indicators that one cares about strongly depend on the applications and setup in which the models are supposed to be used.
Revised: Moreover, we argue that cost indicators that one cares about strongly depend on the applications and setup in which the models are supposed to be used.

Original: For example, for an embedded application, inference speed is paramount, while for deployed recommendation systems, training cost can be extremely important as models are constantly being retrained.
Revised: For example, for an embedded application, inference speed is paramount, while for deployed recommendation systems, training cost can be extremely important as models are constantly being retrained.

Original: The overall contributions of this paper are as follows: We call out the intrinsic difﬁculty of measuring model efﬁciency within the context of deep neural networks.
Revised: The overall contributions of this paper are as follows: We call out the intrinsic difﬁculty of measuring model efﬁciency within the context of deep neural networks.

Original: We  present the advantages and disadvantages of eachcost indicator and discuss why they might be insufﬁcient as a standalone metric.
Revised: We review the most common cost indicators and present the advantages and disadvantages of each and discuss why they might be insufﬁcient as a standalone metric.

Original: While obvious, we show examples of how model efﬁciency might be misrepresented by incomplete reporting of these cost indicators.
Revised: While obvious, we show examples of how model efﬁciency might be misrepresented by incomplete reporting of these cost indicators.

Original: We characterize this problem, coin the term ‘efﬁciency misnomer‘ , and show that it is more prevalent than imagined.
Revised: We characterize this problem, coin the term ‘efﬁciency misnomer‘ , and show that it is more prevalent than imagined.

Original: We brieﬂy review some of the existing work that report comparisons of different models with respect to a cost indicator and discuss the current common practices for analyzing the efﬁciency of algorithms. Along with the discussion and analyses, we provide some suggestions and recommendations. Understanding and comparing the efﬁciency of models is becoming more important considering the success of scaling up neural network models (Kaplan et al., 2020; Hernandez et al., 2021; Tay et al., 2021b; Zhai et al., 2021). Investing in scaling a model up can be a costly process and choosing efﬁcient variants that present better cost-performance trade-offs has become a crucial step.
Revised: We present experiments where comparing model efﬁciency strongly depends on the choice of cost indicator, like scenarios where there is parameter sharing, sparsity, or parallelizable operations in the model.

Original: Moving forward, we argue that only reporting a single cost indicator can be misleading if it comes along with the claim of a model being generally more efﬁcient compared to others. In fact, it may not be possible to evaluate model efﬁciency from all possible points of view, for instance, due to lack of access to all possible accelerators. This ofcourse is no issue, unless the efﬁciency claims are overgeneralized. In this paper, we mainly call for a more clear analysis of the efﬁciency of  modelsthat includes not only various cost indicators but also the conditions and setups under which a model is considered efﬁcient.
Revised: Moreover, we brieﬂy review some of the current common practices in the literature and discuss how existing work report comparisons of different models and analyze the efﬁciency of algorithms.

Original: This will help researchersmake faster and more targeted progress in developing more efﬁcient algorithms and practitioners get a clearer picture of which models align with their needs.
Revised: Along with the discussion and analyses, we provide some concrete suggestions and recommendations that we believe would help researchers and practitioners draw more accurate conclusions about the efﬁciency of different models.

Original:  A P RIMER ON C OST INDICATORS
Revised:  A P RIMER ON C OST INDICATORS

Original: One of the main considerations in designing neural network architectures is quality-cost tradeoff (Paleyes et al., 2020).
Revised: One of the main considerations in designing neural network architectures is quality-cost tradeoff (Paleyes et al., 2020).

Original: In almost all cases, the more computational budget is given to a method, the better the quality of its outcome will be.
Revised: In almost all cases, the more computational budget is given to a method, the better the quality of its outcome will be.

Original: To account for such a trade-off, several cost indicators are used in the literature of machine learning and its applications to showcase the efﬁciency of different models.
Revised: To account for such a trade-off, several cost indicators are used in the literature of machine learning and its applications to showcase the efﬁciency of different models.

Original: These indicators take different points of view to the computational costs.
Revised: These indicators take different points of view to the computational costs.

Original: FLOPs: A widely used metric as the proxy for the computational cost of a model is the number of ﬂoating-point multiplication-and-addition operations (Johnson, 2018;
Revised: FLOPs: A widely used metric as the proxy for the computational cost of a model is the number of ﬂoating-point multiplication-and-addition operations (Johnson, 2018;

Original: Kim et al., 2021; Arnab et al., 2021; Tay et al., 2021a;b; Narayanan et al., 2021;
Revised: Kim et al., 2021; Arnab et al., 2021; Tay et al., 2021a;b; Narayanan et al., 2021;

Original: Liu et al., 2021).
Revised: Liu et al., 2021).

Original: Alternative to FLOPs, the number of multiply-accumulate (MAC 1 ) as a single unit of operation is also used in the literature (Johnson, 2018).
Revised: Alternative to FLOPs, the number of multiply-accumulate (MAC 1 ) as a single unit of operation is also used in the literature (Johnson, 2018).

Original: Reported FLOPs are usually calculated using theoretical values.
Revised: Reported FLOPs are usually calculated using theoretical values.

Original: Note that theoretical FLOPs ignores practical factors, like which parts of the model can be parallelized.
Revised: Note that theoretical FLOPs ignores practical factors, like which parts of the model can be parallelized.

Original: Number of Parameters: Number of trainable parameters is also used as an indirect indicator of computational complexity as well as memory usage (during inference)
Revised: Number of Parameters: Number of trainable parameters is also used as an indirect indicator of computational complexity as well as memory usage (during inference)

Original: (Kim et al., 2021; Arnab et al., 2021; Tan and Le, 2019; Liu et al., 2021;
Revised: (Kim et al., 2021; Arnab et al., 2021; Tan and Le, 2019; Liu et al., 2021;

Original: Guo et al., 2020; Mahabadi et al., 2021a;b; Houlsby et al., 2019).
Revised: Guo et al., 2020; Mahabadi et al., 2021a;b; Houlsby et al., 2019).

Original: Many research works that study the scaling law (Kaplan et al., 2020; Hernandez et al., 2021;
Revised: Many research works that study the scaling law (Kaplan et al., 2020; Hernandez et al., 2021;

Original: Tay et al., 2021b), especially in the NLP domain, use the number of parameters as the primary cost indicator (Devlin et al., 2018; Raffel et al., 2019; Liu et al., 2019; Xue et al., 2021).
Revised: Tay et al., 2021b), especially in the NLP domain, use the number of parameters as the primary cost indicator (Devlin et al., 2018; Raffel et al., 2019; Liu et al., 2019; Xue et al., 2021).

Original: Speed: Speed is one the most informative indicator for comparing the efﬁciency of different models (So et al., 2021;
Revised: Speed: Speed is one the most informative indicator for comparing the efﬁciency of different models (So et al., 2021;

Original: Dosovitskiy et al., 2020; Arnab et al., 2021; Tay et al., 2021b;
Revised: Dosovitskiy et al., 2020; Arnab et al., 2021; Tay et al., 2021b;

Original: Kim et al., 2021; He et al., 2021a; Narayanan et al., 2021;
Revised: Kim et al., 2021; He et al., 2021a; Narayanan et al., 2021;

Original: Lagunas et al., 2021;
Revised: Lagunas et al., 2021;

Original: Liu et al., 2021; Tay et al., 2020c;b).
Revised: Liu et al., 2021; Tay et al., 2020c;b).

Original: In some setups, when measuring speed, the cost of “pipeline” is also taken into account which better reﬂects the efﬁciency in a real-world scenario.
Revised: In some setups, when measuring speed, the cost of “pipeline” is also taken into account which better reﬂects the efﬁciency in a real-world scenario.

Original: Note that speed strongly depends on hardware and implementation, so keeping the hardware ﬁxed or normalizing based on the amount of resources used is the key for a fair comparison.
Revised: Note that speed strongly depends on hardware and implementation, so keeping the hardware ﬁxed or normalizing based on the amount of resources used is the key for a fair comparison.

Original: Speed is often reported in various forms:
Revised: Speed is often reported in various forms:

Original: Throughput refers to the number of examples (or tokens) that are processed within a speciﬁc period of time, e.g., “examples (or tokens) per second”.
Revised: Throughput refers to the number of examples (or tokens) that are processed within a speciﬁc period of time, e.g., “examples (or tokens) per second”.

Original: • Latency usually refers to the inference time (forward pass) of the model given an example or batch of examples, and is usually presented as “seconds per forward pass”.
Revised: • Latency usually refers to the inference time (forward pass) of the model given an example or batch of examples, and is usually presented as “seconds per forward pass”.

Original: The main point about latency is that compared to throughput, it ignores parallelism introduced by batching examples.
Revised: The main point about latency is that compared to throughput, it ignores parallelism introduced by batching examples.

Original: As an example, when processing a batch of 100 examples in 1 second, throughput is 100 examples per second, while latency is 1 second.
Revised: As an example, when processing a batch of 100 examples in 1 second, throughput is 100 examples per second, while latency is 1 second.

Original: Thus, latency is an important factor for real-time systems that require user input.
Revised: Thus, latency is an important factor for real-time systems that require user input.

Original: • Wall-clock time/runtime measures the time spent to process a ﬁxed set of examples by the model.
Revised: • Wall-clock time/runtime measures the time spent to process a ﬁxed set of examples by the model.

Original: Usually, this is  used to measure the training cost, for instance the total training time up to convergence.
Revised: This is often used to measure the training cost, e g., total training time up to convergence.

Original: • Pipeline bubble is the time that computing devices are idle at the start and end of every batch (Narayanan et al., 2021). This indirectly measures the speed of the non-pipeline parts of the process.
Revised: • Pipeline bubble is the time that computing devices are idle at the start and end of every batch (Narayanan et al., 2021), which indirectly measures the speed of the non-pipeline parts of the process.

Original: • Memory Access Cost (MAC) corresponds to the number of memory accesses.
Revised: • Memory Access Cost (MAC) corresponds to the number of memory accesses.

Original: It typically makes up a large portion of runtime and is the actual bottleneck when running on modern platforms with strong computational power such as GPUs and TPUs (Ma et al., 2018).
Revised: It typically makes up a large portion of runtime and is the actual bottleneck when running on modern platforms with strong computational power such as GPUs and TPUs (Ma et al., 2018).

Original: Other cost indicators Depending on the use case, there are other cost indicators that may play a key role in determining the efﬁciency of a model.

Original: For instance, sample efﬁciency that can be expressed as training data size, e.g. number of data points (or tokens for language models Kaplan et al.

Original: that a model needs in order to reach a reasonable performance.

Original: Being sample efﬁcient depends on many factors, like inductive biases of the model or the training curriculum.

Original: Another important cost indicator is the environmental impact and carbon footprint of a model, during training or inference.

Original: Quantifying this cost directly is not straightforward, but (Strubell et al., 2019) proposed the approximate environmental costs of an ML model as Footprint

Original: = ( ee train + queries × ee inference )

Original: × CO2e datacenter / KWh, where, ee indicate the electrical energy (Patterson et al., 2021).

Original: One can also consider the expenses for training or serving models as a cost indicator which is reported in the form of ﬁgures (Sharir et al., 2020).

Original: As an example, the ﬁgures for training can be computed as total-train-time × total number of chips × prices offered by cloud solutions 2 .

Original: This cost indicator is more often used in business documents, than in scientiﬁc articles.

Original: Doll ´ ar et al.

Original: proposed using the number of model activations as a complexity metric.

Original: Model activations refers to the number of elements in the output tensors from the building blocks of the model, e.g., output size of convolutional layers in ResNet.

Original: Doll ´ ar et al.

Original: (2021) argue that the number of activations is a reliable predictor for the model runtime and showed that compared to FLOPs, it is more strongly correlated with the runtime on memory-bandwidth limited hardware.

Original: Memory consumption is one of the main dimensions of efﬁciency and it has been used as a cost indicator in various research works (Bondarenko et al., 2021; Dosovitskiy et al., 2020; Kondratyuk et al., 2021).

Original: Reporting memory footprint often is done in form of “peak memory usage”, during training that takes into account the memory consumption by the model, optimizer, and the pipeline.

Original: Another form that is also common is comparing the maximum batch size that ﬁts on a speciﬁc device for a speciﬁc task.

Original: Unlike model activation size, memory footprint depends on the hardware and implementation.

Original: It has a direct implication of whether a model can ﬁt on a device, which is a hard constraint in some cases.

Original: The cost indicators we discussed above present different perspectives on efﬁciency.
Revised: The cost indicators we discussed above present different perspectives on efﬁciency.

Original: However, some of these cost indicators may depend on factors that are not inherent to the design of the model, but on the hardware, the model runs on (e.g., CPU, GPU, or TPU), the framework that the model is implemented in (e.g., JAX, PyTorch, or TensorFlow), or even programming skill.
Revised: However, some of these cost indicators may depend on factors that are not inherent to the design of the model, but on the hardware, the model runs on (e.g., CPU, GPU, or TPU), the framework that the model is implemented in (e.g., JAX, PyTorch, or TensorFlow), or even programming skill.

Original: These confounding factors add up to the difﬁculty of comparisons.
Revised: These confounding factors add up to the difﬁculty of comparisons.

Original: For instance, theoretical FLOPs provides a hardware-independent comparison, however, it does not necessarily translate to the speed of a model as it does not capture the sequential dependencies of operations and memory accesses in the model.
Revised: For instance, theoretical FLOPs provides a hardware-independent comparison, however, it does not necessarily translate to the speed of a model as it does not capture the sequential dependencies of operations and memory accesses in the model.

Original: On the other hand, throughput and peak memory usage, which could better reﬂect the model’s efﬁciency in a real-world scenario, strongly depend on the hardware and the implementation. 
Revised: On the other hand, throughput and peak memory usage, which could better reﬂect the model’s efﬁciency in a real-world scenario, strongly depend on the hardware and the implementation. Software support can also be a limiting factor in achieving the best possible hardware performance for a model. In (Barham and

Revised: Isard, 2019), the authors make an excellent case for improving the programmability of software stack for modern accelerators to enable a wider class of models.

Original: In the rest of the paper, we mainly focus on the number of parameters, FLOPs and speed since they capture most common use cases and are most commonly encountered in the literature.
Revised: In the rest of the paper, we mainly focus on the number of parameters, FLOPs and speed since they capture most common use cases and are most commonly encountered in the literature.

Revised: Appendix B presents other cost indicators that could be important depending on the use case.

Original: 2. T RAINING OR I NFERENCE COST ?
Revised: 2. T RAINING OR I NFERENCE COST ?

Original: When talking about costs, we can disentangle the cost of training and the cost of inference.
Revised: When talking about costs, we can disentangle the cost of training and the cost of inference.

Original: Based on the estimate from NVIDIA (Leopold, 2019) and Amazon (Barr, 2019) as major cloud service providers, 80–90% of the ML workload is inference processing.
Revised: Based on the estimate from NVIDIA (Leopold, 2019) and Amazon (Barr, 2019) as major cloud service providers, 80–90% of the ML workload is inference processing.

Original: Thus, more often, the cost of a model during inference is taken as the real cost with the argument that the amortized per-usage cost of training can be really small compared to the inference cost when a model is deployed to be used by many users.
Revised: Thus, more often, the cost of a model during inference is taken as the real cost with the argument that the amortized per-usage cost of training can be really small compared to the inference cost when a model is deployed to be used by many users.

Original: However, with the trend of improving the performance of models by scaling up their computational budget and/or training data, as well as the fast progress and frequency of the emergence of new models, the training cost can be seen as a relevant concern.
Revised: However, with the trend of improving the performance of models by scaling up their computational budget and/or training data, as well as the fast progress and frequency of the emergence of new models, the training cost can be seen as a relevant concern.

Original: Moreover, in some cases, we have to frequently retrain models due to, for instance, privacy requirements, or where new data is being continuously generated like in many recommender systems.
Revised: Moreover, in some cases, we have to frequently retrain models due to, for instance, privacy requirements, or where new data is being continuously generated like in many recommender systems.

Original: The importance of inference efﬁciency is already clear and here, we will discuss more the importance of training efﬁciency as well as potential issues with reporting training costs.
Revised: The importance of inference efﬁciency is already clear and here, we will discuss more the importance of training efﬁciency as well as potential issues with reporting training costs.

Original: Several argumentscan be considered in favor of the importance of reporting training costs and why we need  models that train efﬁciently.
Revised: The are several arguments in favor of the importance of reporting training costs and the need for models that train efﬁciently.

Original: For instance, from a research and development point of view, when a class of models ismore efﬁcient during training, it is more likely to see improvements in their performance, due to ease of iterating on ideas around them.
Revised: For instance, from a research and development point of view, when a class of models is efﬁcient during training, it is more likely to see improvements in their performance, due to ease of iterating on ideas around them.

Original: Moreover, when a model shows merit in terms of performance, usually some posthoc modiﬁcations can be applied to improve its inference efﬁciency.
Revised: Moreover, when a model shows merit in terms of performance, usually some posthoc modiﬁcations can be applied to improve its inference efﬁciency.

Original: Besides, the memory requirements of different models can be wildly different during training, although their inference memory consumption is comparable.
Revised: Besides, the memory requirements of different models can be wildly different during training, although their inference memory consumption is comparable.

Original: For instance, different optimizers use different amounts of memory on the device, for instance, SGD-Momentum (Qian, 1999) vs
Revised: For instance, different optimizers use different amounts of memory on the device, for instance, SGD-Momentum (Qian, 1999) vs

Original: SAM (Foret et al., 2020).
Revised: SAM (Foret et al., 2020).

Original: If the success of a model is strongly tied to using an optimizer with a high memory cost, it can become a bottleneck during training when we plan to scale that model up compared to the model that uses a more memory-efﬁcient optimizer.
Revised: If the success of a model is strongly tied to using an optimizer with a high memory cost, it can become a bottleneck during training when we plan to scale that model up compared to the model that uses a more memory-efﬁcient optimizer.

Original: Another example is a model that has a high degree of parameter sharing, which could be extremely efﬁcient in terms of inference memory usage, but during training, the size of activation we need to keep for the backward pass is as big as a similar model with no parameter sharing.
Revised: Another example is a model that has a high degree of parameter sharing, which could be extremely efﬁcient in terms of inference memory usage, but during training, the size of activation we need to keep for the backward pass is as big as a similar model with no parameter sharing.

Original: Activation and their statistics form a big portion of memory usage compared to parameter size, decreasing the “number of parameters” by parameter sharing may not lead to any signiﬁcant decrease in the memory usage during training (Dehghani et al., 2018). 
Revised: Activation and their statistics form a big portion of memory usage compared to parameter size, decreasing the “number of parameters” by parameter sharing may not lead to any signiﬁcant decrease in the memory usage during training (Dehghani et al., 2018). [Figure]

Original: Gaming with training time Whilst “training time” can be a great cost indicator, it is also prone to 
Revised: Gaming with training time Whilst “training time” can be a great cost indicator, it is also prone to Goodhart’s Law:

Original: Goodhart’s Law: When used as the main metric, it can and will be gamed and lose its meaning.
Revised: When used as the main metric, it can and will be gamed and lose its meaning.

Original: First of all, it is difﬁcult to compare architectures with respect to training time, since training time includes the whole “recipe” and different ingredients may be better ﬁts for different architectures.
Revised: First of all, it is difﬁcult to compare architectures with respect to training time, since training time includes the whole “recipe” and different ingredients may be better ﬁts for different architectures.

Original: For instance, MobileNet and EfﬁcientNets almost exclusively work well with the RMSProp optimizer (Pham, 2021).
Revised: For instance, MobileNet and EfﬁcientNets almost exclusively work well with the RMSProp optimizer (Pham, 2021).

Original: Second, because the full training recipe is inevitably involved, the only meaningful claim to be made is achieving higher accuracy with smaller total training cost; a good example of this is Table 1 in Wightman et al. 
Revised: Second, because the full training recipe is inevitably involved, the only meaningful claim to be made is achieving higher accuracy with smaller total training cost; a good example of this is Table 1 in Wightman et al. (2021).

Original: The training cost may be in terms ofwall-clock time, FLOPs, currency, examples, CO2 emission, or any other metric the authors care about.
Revised: The training cost may be in terms of any of the indicators described above.

Original: However, counting training cost in terms of steps can be problematic as step count in itself is not a meaningful metric, and can be stretched arbitrarily with optimizers introducing multi-step lookahead schemes (Zhang et al., 2019).
Revised: However, counting training cost in terms of steps can be problematic as step count in itself is not a meaningful metric, and can be stretched arbitrarily with optimizers introducing multi-step lookahead schemes (Zhang et al., 2019).

Original: Conversely, it may be tempting to claim that a method A performs almost as well as another method
Revised: Conversely, it may be tempting to claim that a method A performs almost as well as another method

Original: B with dramatically reduced training cost .
Revised: B with dramatically reduced training cost .

Original: Such a claim is not valid, as method A may not have been optimized for training cost, and may well just be re-tuned with that in mind and outperform method B .
Revised: Such a claim is not valid, as method A may not have been optimized for training cost, and may well just be re-tuned with that in mind and outperform method B .

Revised: For example, training hyper-parameters such as learning rate and weight decay, can be tuned such that they achieve good quality quickly, but then plateau to lower points than in “slower” settings that eventually reach higher quality.

Revised: This is illustrated by Figure 2, obtained from Kolesnikov et al.

Revised: (2020), where the “long” training schedule (which is not optimized for training cost) for ResNet-101x achieves the best performance, but the “short” schedule converges signiﬁcantly faster to a lower ﬁnal accuracy.

Original: An example of this happened with the Rainbow baseline in Kaiser et al.
Revised: Another example, in the context of reinforcement learning, is the Rainbow baseline of Kaiser et al.

Original: (2019) which was subsequently re-tuned in van Hasselt et al.
Revised: (2019) which was subsequently re-tuned in van Hasselt et al.

Original: Some hyper-parameters, such as learning-rate and weight-decay, can be tuned so as to reach good quality quickly but then plateau lower than in “slower” settings that eventually reach higher quality.
Revised: and shown to beneﬁt from signiﬁcantly longer training schedules too.

Original:  Some works, like the MLPerf benchmark (Mattson et al., 2019), aim at decreasing the training cost required to reach a ﬁxed quality X with a ﬁxed model M.
Revised: [Figure] Some works, like the MLPerf benchmark (Mattson et al., 2019), aim at decreasing the training cost required to reach a ﬁxed quality X with a ﬁxed model M.

Original: While this is a good way of ﬁxing the many moving pieces of training a deep learning model, it should be noted that due to Goodhart’s law, results do not mean more than “reaching quality X with model M”.
Revised: While this is a good way of ﬁxing the many moving pieces of training a deep learning model, it should be noted that due to Goodhart’s law, results do not mean more than “reaching quality X with model M”.

Original: Speciﬁcally,  a method A reaches quality X twice faster than a method B while both using model M, this can neither be used to imply anything about their comparison when using model N, nor to imply anything on their comparison with the target quality X+ (cid:15) : it could well be that method B gets to X+ (cid:15) faster than A , or worse, A may never reach X+ (cid:15) .
Revised: Speciﬁcally, if a method A reaches quality X twice as fast as a method B while both using model M, this can neither be used to imply anything about their comparison when using model N, nor to imply anything on their comparison with the target quality X+ (cid:15) : it could well be that method B gets to X+ (cid:15) faster than A , or worse, A may never reach X+ (cid:15) .

Revised: Figure 3 shows another concrete example, obtained from (Kolesnikov et al., 2020) where we see faster initial convergence of a ResNet model when trained with lower weight decay, which may trick the practitioner into selecting a sub-optimal value, while using a higher weight decay leads to a slower convergence, but a better ﬁnal performance.

Original: A ﬁnal concern is that, methods which seemingly train faster tend to be used more often and hence get optimized more over time, with the danger of getting stuck in a local minimum (Hooker, 2020).
Revised: A ﬁnal concern is that, methods which seemingly train faster tend to be used more often and hence get optimized more over time, with the danger of getting stuck in a local minimum (Hooker, 2020; Dehghani et al., 2021b).

Original: 2. P OTENTIAL D ISAGREEMENT BETWEEN COST INDICATORS
Revised: 2. P OTENTIAL D ISAGREEMENT BETWEEN COST INDICATORS

Original: In this section, we will discuss some of the cases in which there could be disagreement between some of the cost indicators we discussed before.
Revised: In this section, we will discuss some of the cases in which there could be disagreement between some of the cost indicators we discussed before.

Original: Sharing parameters When we introduce a form of parameter sharing, it is clear that compared to the same model with no parameter tying, we end up having fewer trainable parameters, while the number of FLOPs or speed stays the same.
Revised: Sharing parameters When we introduce a form of parameter sharing, it is clear that compared to the same model with no parameter sharing, we end up having fewer trainable parameters, while the number of FLOPs or speed stays the same.

Original: An example is the comparison of the Universal
Revised: An example is the comparison of the Universal

Original: Transformer (UT) (Dehghani et al., 2018) with vanilla Transformer (Vaswani et al., 2017) Figure 1.
Revised: Transformer (UT) (Dehghani et al., 2018) with vanilla Transformer (Vaswani et al., 2017) Figure 1.

Original: UT shares parameters of the model in depth, thus stays close to the frontiers of quality-number of parameters.
Revised: UT shares parameters of the model in depth, thus stays close to the frontiers of quality-number of parameters.

Original: However, when looking at the FLOPs, to maintain a similar capacity in terms of size, UT requires more computation, which makes it not so efﬁcient from the FLOPs point of view.
Revised: However, when looking at the FLOPs, to maintain a similar capacity in terms of parameter count, UT requires more computation, which makes it not so efﬁcient from the FLOPs point of view.

Original: Introducing Sparsity Sparsity is becoming one of the main ways of both scaling up and down deep neural networks.
Revised: Introducing Sparsity Sparsity is becoming one of the main ways of both scaling up and down deep neural networks.

Original: One needs to distinguish between at least two broad classes of sparse neural networks: structured and unstructured.
Revised: One needs to distinguish between at least two broad classes of sparse neural networks: structured and unstructured.

Original: Structured sparse models replace large, dense parts of a model by a collection of much smaller, still dense parts.
Revised: Structured sparse models replace large, dense parts of a model by a collection of much smaller, still dense parts.

Original: Examples include variants as simple as replacing dense convolutions by grouped (Xie et al., 2017) or separable (Howard et al., 2017) ones, or as complicated as replacing large blocks by many smaller “expert” blocks and routing examples through the best suited ones only (Fedus et al., 2021;
Revised: Examples include variants as simple as replacing dense convolutions by grouped (Xie et al., 2017) or separable (Howard et al., 2017) ones, or as complicated as replacing large blocks by many smaller “expert” blocks and routing examples through the best suited ones only (Fedus et al., 2021;

Original: Riquelme et al., 2021).
Revised: Riquelme et al., 2021).

Original: The latter, often called Mixture of Experts (MoE), allows growing the capacity in terms of parameter count, while keeping the computational cost small and constant.
Revised: The latter, often called Mixture of Experts (MoE), allows growing the capacity in terms of parameter count, while keeping the computational cost small and constant.

Original: Figure 1 compares the quality-cost of Switch Transformer (Fedus et al., 2021), a MoE, to that of vanilla Transformer (Vaswani et al., 2017).
Revised: Figure 1 compares the quality-cost of Switch Transformer (Fedus et al., 2021), a MoE, to that of vanilla Transformer (Vaswani et al., 2017).

Original: While Switch falls short in terms of quality vs parameter count, it offers a great trade-off with respect to quality vs FLOPs and speed.
Revised: While Switch falls short in terms of quality vs parameter count, it offers a great trade-off with respect to quality vs FLOPs and speed.

Original:  Unstructured sparse models are models where weights of dense operations are made to contain many (almost) zeros (Gale et al., 2019;
Revised: [Figure] Unstructured sparse models are models where weights of dense operations are made to contain many (almost) zeros (Gale et al., 2019;

Original: Evci et al., 2020), which do not contribute to the operation’s result and thus, in principle, can be skipped.
Revised: Evci et al., 2020), which do not contribute to the operation’s result and thus, in principle, can be skipped.

Original: This keeps the overall structure of the original model, while signiﬁcantly reducing the FLOPs of the most expensive operations.
Revised: This keeps the overall structure of the original model, while signiﬁcantly reducing the FLOPs of the most expensive operations.

Original: Both types of sparse models result in large reductions in theoretical FLOPs, often of several orders of magnitude.
Revised: Both types of sparse models result in large reductions in theoretical FLOPs, often of several orders of magnitude.

Original: However, these do not translate to equally large speed-ups.
Revised: However, these do not translate to equally large speed-ups.

Original: Difﬁculties for structured sparse models (especially for the MoE type) include the overhead of routing, and the inability to effectively use batched operations, while for unstructured sparsity, it is not possible for the corresponding low-level operations to reach the same efﬁciency as their dense counterparts on current hardware, where memory access is signiﬁcantly more expensive than compute (Gale et al., 2020).
Revised: Difﬁculties for structured sparse models (especially for the MoE type) include the overhead of routing, and the inability to effectively use batched operations, while for unstructured sparsity, it is not possible for the corresponding low-level operations to reach the same efﬁciency as their dense counterparts on current hardware, where memory access is signiﬁcantly more expensive than compute (Gale et al., 2020).

Original: Degree of parallelism: Scaling Depth (D) vs. Scaling Width (W) When scaling up the model size, different strategies can be used.
Revised: Degree of parallelism: Scaling Depth (D) vs. Scaling Width (W) When scaling up the model size, different strategies can be used.

Original: Although these different strategies may have a similar effect in terms of parameter count, and even quality, they can have different effects on the cost in terms of
Revised: Although these different strategies may have a similar effect in terms of parameter count, and even quality, they can have different effects on the cost in terms of

Original: FLOPs and throughput.
Revised: FLOPs and throughput.

Original: The most common knobs for scaling up models are changing depth (number of layers) and width (hidden dimension) of models (Tay et al., 2021b).
Revised: The most common knobs for scaling up models are changing depth (number of layers) and width (hidden dimension) of models (Tay et al., 2021b).

Original: To study such an effect, we ran a set of controlled experiments with Vision Transformers (Dosovitskiy et al., 2020), where we scale the width by increasing  the hidden dimensions ofself-attention heads  as well as that of the FFN, and we also scale up the depth, by only increasing the number of encoder blocks.
Revised: To study such an effect, we ran a set of controlled experiments with Vision Transformers (Dosovitskiy et al., 2020), where we scale the width by increasing number of heads (while maintaining the hidden dimensions of heads ﬁxed) as well as that of the FFN, and we also scale up the depth, by only increasing the number of encoder blocks.

Revised: Note that when changing depth or width of the model (see Table 2 in Appendix A for the exact conﬁgurations) all other hyper-parameters are kept ﬁxed based on the default values given by the referenced papers.

Original: Figure 2 shows the accuracy of these models with respect to different cost indicators.
Revised: Figure 4 shows the accuracy of these models with respect to different cost indicators.

Revised: In general, we can see that considering FLOPs or number parameters as the cost indicator, increasing width is beneﬁcial for smaller budget regions, while increasing depth gives better quality with lower cost when we scale up to higher budget regions.

Revised: However, this is not necessarily the case when considering speed (msec/img) as the cost indicator.

Original: Comparing D 48 (a ViT with 48 layers) to W 3072 , (a ViT with FFN dimension 3072 and QKV dimension 768 split across 8 heads) in terms of FLOPs or number of parameters, they have more or less similar cost, suggesting the D 48 as a clear Pareto efﬁcient model.
Revised: As an example, comparing D 48 (a ViT with 48 layers) to W 3072 , (a ViT with FFN dimensionand QKV dimension 768 split across 8 heads) in terms of FLOPs or number of parameters, they have more or less similar cost, suggesting the D 48 as a clear Pareto efﬁcient model 2 .

Original: However, when looking at the speed (msec/img), we observe that W 3072 is not necessarily worse than D 48 , since there are less sequential and more parallelizable operations.
Revised: However, when looking at the speed (msec/img), we observe that W 3072 is not necessarily worse than D 48 , since there are less sequential and more parallelizable operations.

Original: Target platform, and implementation
Revised: Target platform, and implementation

Original: In some cases, a certain design in the hardware may lead to discrepancies between the cost indicators.
Revised: In some cases, a certain design in the hardware may lead to discrepancies between the cost indicators.

Original: As an example, tensor factorization is used as one of the common techniques to accelerate the matrix multiplication and used for making neural network models more efﬁcient (Zhang et al., 2015b; Jaderberg et al., 2014).
Revised: As an example, tensor factorization is used as one of the common techniques to accelerate the matrix multiplication and used for making neural network models more efﬁcient (Zhang et al., 2015b; Jaderberg et al., 2014).

Original: Weight decomposition, regardless of its effect on the quality of the model, can reduce the number of FLOPs in neural networks by[Figure] 75% (Zhang et al., 2015a).
Revised: Weight decomposition, regardless of its effect on the quality of the model, can reduce the number of FLOPs in neural networks by 75% (Zhang et al., 2015a).

Original: However, it has been shown that using weight decomposition on GPUs can be slower as CUDNN that is optimized for 3 × 3 (He et al., 2017) convolutions
Revised: However, it has been shown that using weight decomposition on GPUs can be slower as CUDNN that is optimized for 3 × 3 (He et al., 2017) convolutions

Original: and they prefer single large matrix multiplication instead of several small ones.
Revised: and they prefer single large matrix multiplication instead of several small ones.

Original: FNet (Lee-Thorp et al., 2021) also shows how certain architectures can have signiﬁcantly different speed in different hardware (e.g., 
Revised: FNet (Lee-Thorp et al., 2021) also shows how certain architectures can have signiﬁcantly different speed in different hardware (e.g., [Figure]

Revised: Values for accuracies and cost indicators are obtained from (Steiner et al., 2021; Liu et al., 2021).

Revised: Cost indicators are based on PyTorch implementation of the included models and the throughput is measured on a V100 GPU, using timm (Wightman, 2019).

Original: GPUs and TPUs).
Revised: GPUs and TPUs).

Original: When using a speciﬁc hardware and compiler, some small design choices can signiﬁcantly affect the cost of a model.
Revised: When using a speciﬁc hardware and compiler, some small design choices can signiﬁcantly affect the cost of a model.

Original: As an example, Zhai et al.
Revised: As an example, Zhai et al.

Original: (2021) show that for ViT, using global average pooling instead of a CLS as the representation of the input can signiﬁcantly reduce the memory cost on TPU V3.
Revised: (2021) show that for ViT, using global average pooling instead of a CLS as the representation of the input can signiﬁcantly reduce the memory cost on TPU-V3.

Original: This is because the current TPU hardware pads the length dimension of the inputs to a multiple of 128, which may result in up to a 50% memory overhead.
Revised: This is because the current TPU hardware pads the length dimension of the inputs to a multiple of 128, which may result in up to a 50% memory overhead.

Original: Another example for how implementation details can affect the efﬁciency is presented in (Vaswani et al., 2021), where the author discusses how carefully designed implementation of the local neighborhood gathering function for local attention might reduce memory usage while avoiding unnecessary extra computation.
Revised: Another example for how implementation details can affect the efﬁciency is presented in (Vaswani et al., 2021), where the author discusses how carefully designed implementation of the local neighborhood gathering function for local attention might reduce memory usage while avoiding unnecessary extra computation.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: The most common use of cost indicators in the ML community is for (1) comparing different models in terms of efﬁciency using a cost indicator, (2) evaluating different models in terms of quality while ﬁxing the cost across models to have a fair comparison, and (3) choosing models with the right trade-off for the context at hand, e.g, in architecture search.
Revised: The most common use of cost indicators in the ML community is for (1) comparing different models in terms of efﬁciency using a cost indicator, (2) evaluating different models in terms of quality while ﬁxing the cost across models to have a fair comparison, and (3) choosing models with the right trade-off for the context at hand, e.g, in architecture search.

Original: While (1) and (2) are just two sides of the same coin, the focus in the ﬁrst case is on ﬁnding models with better quality, while the second case is concerned with comparing quality while keeping a certain efﬁciency metric constant.
Revised: While (1) and (2) are just two sides of the same coin, the focus in the ﬁrst case is on ﬁnding models with better quality, while the second case is concerned with comparing quality while keeping a certain efﬁciency metric constant.

Original: 3. O N PARTIAL CONCLUSIONS FROM INCOMPLETE COMPARISONS
Revised: 3. O N PARTIAL CONCLUSIONS FROM INCOMPLETE COMPARISONS

Original: As we showed in Section 2, claiming a model is more efﬁcient by reporting better scores on a subset of cost indicators can lead to partial, incomplete and possibly biased conclusions
Revised: As we showed in Section 2, claiming a model is more efﬁcient by reporting better scores on a subset of cost indicators can lead to partial, incomplete and possibly biased conclusions.

Original: Figure 3 compares the FLOPs, parameters and run time for several models used for image classiﬁcation task.
Revised: Figure 5a compares the FLOPs, parameters and run time for several models versus their accuracy on image classiﬁcation task.

Original: As seen previously in Fig. 1, the relative positions of different model architectures are not consistent as the performance metric is varied.
Revised: As seen previously in Figure 1, the relative positions of different model architectures are not consistent as the performance metric is varied.

Original: For example, on one hand, EfﬁcientNets (Tan and Le, 2019) are on the Pareto-frontier of and accuracy-parameter and accuracy-FLOPs trade-offs.
Revised: For example, on one hand, EfﬁcientNets (Tan and Le, 2019) are on the Pareto-frontier of and accuracy-parameter and accuracy-FLOPs trade-offs.

Original: On the other hand, the accuracy-throughput curve is dominated by SwinTransformers (Liu et al., 2021).
Revised: On the other hand, the accuracy-throughput curve is dominated by SwinTransformers (Liu et al., 2021).

Original: As such, there is no model that is clearly more efﬁcient here and conclusions may differ strongly depending on which efﬁciency metric is employed.
Revised: As such, there is no model that is clearly more efﬁcient here and conclusions may differ strongly depending on which efﬁciency metric is employed.

Original: Note that when comparing variants of a single model family, most of cost indicators correlate.
Revised: Note that when comparing variants of a single model family, most of cost indicators correlate.

Original: Hence, it could be sufﬁcient to consider one of these cost indicators 3 .
Revised: Hence, it could be sufﬁcient to consider one of these cost indicators 3 .

Original: However, comparisons using a single cost indicator between models with completely different architectures are more complicated.
Revised: However, comparisons using a single cost indicator between models with completely different architectures are more complicated.

Original: Note, [Figure]
Revised: Figure 5b shows the correlation between the cost indicators.

Original: how, for a similar number of FLOPs, EfﬁcientNet has slower throughput than other model families such as RegNet and SwinTransformer. Conversely, for a similar number of FLOPs, EfﬁcientNet has fewer parameters than RegNet and SwinTransformers.
Revised: For example, we can see that for a similar number of FLOPs, EfﬁcientNet has fewer parameters than other model families such as RegNet and SwinTransformer. Conversely, for a similar number of FLOPs, EfﬁcientNet has slower throughput (i.e., higher msec/examples) than RegNet and SwinTransformer.

Original: These differences are not surprising, given that Fig. is comparing transformer-based (Dosovitskiy et al., 2020; Liu et al., 2021) and convolution-based architectures (Tan and Le, 2019; Radosavovic et al., 2020), and including some optimized via architecture  for speciﬁc cost-indicators (EfﬁcientNets are optimized for FLOPs).
Revised: These differences are not surprising, given that we are comparing transformerbased (Dosovitskiy et al., 2020; Liu et al., 2021) and convolution-based architectures (Tan and Le, 2019; Radosavovic et al., 2020). Moreover, some of these models are found via architecture search when optimizing for speciﬁc cost-indicators (e.g., EfﬁcientNets are optimized for FLOPs).

Original: 3. M AKING FAIR COMPARISONS THROUGH AN EFFICIENCY LENS
Revised: 3. M AKING FAIR COMPARISONS THROUGH AN EFFICIENCY LENS

Original: Efﬁciency metrics and cost indicators are often used to ground comparisons between two or more models or methods.
Revised: Efﬁciency metrics and cost indicators are often used to ground comparisons between two or more models or methods (e.g., a proposed model and baselines).

Original: By keeping one or more cost indicators ﬁxed, one would often list models side by side in order to make comparisons fair.
Revised: By keeping one or more cost indicators ﬁxed, one would often list models side by side in order to make comparisons fair.

Original: There are two main strategies here, namely (1) parameter-matched comparisons and/or (2) ﬂop/compute matched comparisons.
Revised: There are two main strategies here, namely (1) parameter-matched comparisons, where conﬁguration of all models are chosen to have similar number of trainable parameters; and/or (2) ﬂop/compute matched comparisons, where all models have similar computational budget, e.g., conﬁgurations are chosen to have similar FLOPs.

Original: Whilst there is no straightforward answer to which strategy is the better one, we highlight two case studies and discuss implications and general recommendations.
Revised: Whilst there is no straightforward answer to which strategy is the better one, we highlight two case studies and discuss implications and general recommendations.

Original: 3.2. T HE ISSUES WITH PARAMETER MATCHED COMPARISONS
Revised: 3.2. T HE ISSUES WITH PARAMETER MATCHED COMPARISONS

Original: We delve into some of the potential issues with the parameter-matched comparisons.
Revised: We delve into some of the potential issues with the parameter-matched comparisons.

Original: The gist of many of these issues is that not every parameter is created equal , causing many intricate complexities that might complicate fair comparisons among models.
Revised: The gist of many of these issues is that not every parameter is created equal , causing many intricate complexities that might complicate fair comparisons among models.

Original: Here we present two example situations where parameter matched comparison could go wrong.
Revised: Here we present two example situations where parameter matched comparison could go wrong.

Original: Token-Free Models Token-free models (Xue et al., 2021; Tay et al., 2021c; Jaegle et al., 2021) get rid of the large subword vocabulary by modeling at the character or byte level.
Revised: Token-Free Models Token-free models Xue et al., 2021; Tay et al., 2021c; Jaegle et al., 2021) get rid of the large subword vocabulary by modeling at the character or byte level.

Original: A large number of parameters originating from the embedding matrix is therefore dropped when transiting to token-free models.
Revised: A large number of parameters originating from the embedding matrix is therefore dropped when transiting to token-free models.

Original: Hence, it remains an open question of how to fairly compare these class of models with their subword counterparts.
Revised: Hence, it remains an open question of how to fairly compare these class of models with their subword counterparts.

Original: ByT5 (Xue et al., 2021) proposed to up-scale the Transformer stack in order to compensate for lost parameters in the embedding layer.
Revised: ByT5 (Xue et al., 2021) proposed to up-scale the Transformer stack in order to compensate for lost parameters in the embedding layer.

Original: While this argument was made in the spirit of ‘fairness’ (e.g., comparing both methods at a parameter-matched setup), the up-scaling causes a substantial reduction in model speed.
Revised: While this argument was made in the spirit of ‘fairness’ (e.g., comparing both methods at a parameter-matched setup), the up-scaling causes a substantial reduction in model speed.

Original: This is largely because parameters in the embedding matrix typically do not incur much computation while increased parameters in other parts of the Transformer stack (more depth/width) lead to a relatively substantial compute cost.
Revised: This is largely because parameters in the embedding matrix typically do not incur much computation while increased parameters in other parts of the Transformer stack (more depth/width) lead to a relatively substantial compute cost.

Original: What is referred to as base or small size here refers to a model that is signiﬁcantly more computationally costly (in terms of speed, throughput and FLOPs) than their other counterparts and hence the naming alone can be very misleading to practitioners.
Revised: What is referred to as base or small size here refers to a model that is signiﬁcantly more computationally costly (in terms of speed, throughput and FLOPs) than their other counterparts and hence the naming alone can be very misleading to practitioners.

Original: To make this fair, ideally, the authors would have to present results that are both parameter matched and compute-matched to present a clearer picture.
Revised: To make this fair, ideally, the authors would have to present results that are both parameter matched and compute-matched to present a clearer picture.

Original: Sparse Models and Mixture-of-Experts A deﬁning feature of sparse models (Lample et al., 2019;
Revised: Sparse Models and Mixture-of-Experts A deﬁning feature of sparse models (Lample et al., 2019;

Original: Fedus et al., 2021) is that they remain compute-matched while enabling scaling to a large number of parameters.
Revised: Fedus et al., 2021) is that they remain compute-matched while enabling scaling to a large number of parameters.

Original: Hence, parameter matched comparisons do not make sense for sparse models and parameter-matching sparse models can be seen as an unfair method of unnecessarily downplaying the strengths of sparse models.
Revised: Hence, parameter matched comparisons do not make sense for sparse models and parameter-matching sparse models can be seen as an unfair method of unnecessarily downplaying the strengths of sparse models.

Revised: This has been also shown in Figure 1, where comparing a switch transformer with a vanilla transformer with the same number of parameters always goes in favor of vanilla transformer.

Original: It is good to note that many works in the literature employ computematched comparisons for comparing sparse models (Narang et al., 2021; Fedus et al., 2021; Lample et al., 2019).
Revised: Note that many works in the literature employ compute-matched comparisons for comparing sparse models (Narang et al., 2021; Fedus et al., 2021; Lample et al., 2019).

Original: Vision Transformers and Sequence Length Models with a ﬂexible sequence length, such as
Revised: Vision Transformers and Sequence Length Models with a ﬂexible sequence length, such as

Original: ViTs when varying patch-size, have the opposite property of sparse models: one can instantiate architectures with signiﬁcantly different capacity and speed while keeping the parameter count ﬁxed.
Revised: ViTs when varying patch-size, have the opposite property of sparse models: one can instantiate architectures with signiﬁcantly different computational cost (e.g., FLOPs) and speed while keeping the parameter count ﬁxed.

Original: Thus, this type of model should not be compared based on parameter count either.
Revised: Thus, this type of model should not be compared based on parameter count either.

Revised: Table 1 presents the parameter size, FLOPs, and speed of ViT using different patch sizes.

Revised: We can see inverse corelation between parameter size and FLOPs/speed.

Revised: Model with bigger patch sizes have less FLOPs and are faster, while having similar or slightly more parameters.

Revised: [Table]

Original: 3.2. T HE ISSUES WITH C OMPUTE M ATCHED C OMPARISONS
Revised: 3.2. T HE ISSUES WITH C OMPUTE M ATCHED C OMPARISONS

Original: We have previously discussed how parameter matched comparisons may be unfair.
Revised: We have previously discussed how parameter matched comparisons may be unfair.

Original: This section discusses a case where compute matched comparisons may raise concerns.
Revised: This section discusses a case where compute matched comparisons may raise concerns.

Original: Consider a scenario where the proposed method achieves compute saving by an architectural design that does not inﬂuence model parameters at all (e.g., downsampling sequences).
Revised: Consider a scenario where the proposed method achieves compute saving by an architectural design that does not inﬂuence model parameters at all (e.g., downsampling sequences).

Original: A seemingly fair comparison here would be to take a standard model and remove layers or hidden dimensions until both models are compute matched.
Revised: A seemingly fair comparison here would be to take a standard model and remove layers or hidden dimensions until both models are compute matched.

Original: However, this comparison runs the risk of a baseline model that is handicapped by signiﬁcantly insufﬁcient model capacity  and therefore, substantially underperform the proposed method.
Revised: However, this comparison runs the risk of a baseline model that is handicapped by signiﬁcantly insufﬁcient model capacity in terms of parameter count and therefore, substantially underperform the proposed method.

Original: This is evident in the Perceiever IO work (Jaegle et al., 2021) where the baseline BERT is shrunk to a mere 20M parameters (compared to 425M in the proposed approach) in a compute matched comparison setup.
Revised: This is evident in the Perceiver IO work (Jaegle et al., 2021) where the baseline BERT is shrunk to a mere 20M parameters (compared to 425M in the proposed approach) in a compute matched comparison setup.

Original: Moreover, the baseline BERT is also handicapped in terms of depth (6 layers vs 40 layers) which is shown in (Tay et al., 2021b) to be unfavorable.
Revised: Moreover, the baseline BERT is also handicapped in terms of depth (6 layers vs 40 layers) which is shown in (Tay et al., 2021b) to be unfavorable.

Original: Note that depth is not taken into account for FLOP-matched comparisons and if the authors were to account for speed-match, then the baseline here would be substantially faster than the proposed method.
Revised: Note that depth is not taken into account for FLOP-matched comparisons and if the authors were to account for speed-match, then the baseline here would be substantially faster than the proposed method.

Original: Overall, we do note that making fair compute matched comparisons is clearly nontrivial and a challenging problem.
Revised: Overall, we do note that making fair compute matched comparisons is clearly nontrivial and a challenging problem.

Original: However, our recommendation is that when there is just no easy way to make a fair comparison, we encourage authors to make the best effort in ﬁnding a best ‘matched’ setup and show multiple alternatives if possible.
Revised: However, our recommendation is that when there is just no easy way to make a fair comparison, we encourage authors to make the best effort in ﬁnding a best ‘matched’ setup and show multiple alternatives if possible.

Original: 3. C OST INDICATORS FOR ARCHITECTURE SEARCH
Revised: 3. C OST INDICATORS FOR ARCHITECTURE SEARCH

Original: Aside from being used for comparing models, many architecture search studies add a cost indicator to the loss function as a resource constraint (He et al., 2021b) and account for efﬁciency.
Revised: Aside from being used for comparing models, many architecture search studies add a cost indicator to the loss function as a resource constraint (He et al., 2021b) and account for efﬁciency.

Original: To this end, these studies mostly use the parameter size (Pham et al., 2018), FLOPs (Hsu et al., 2018), memory access cost (MAC)
Revised: To this end, these studies mostly use the parameter size (Pham et al., 2018), FLOPs (Hsu et al., 2018), memory access cost (MAC)

Original: (Ma et al., 2018), or real latency (Tan et al., 2019; Wu et al., 2019).
Revised: (Ma et al., 2018), or real latency (Tan et al., 2019; Wu et al., 2019).

Original: Given that we have shown earlier how cost indicators may disagree with one another or lead to impartial conclusions, we suggest that practitioners place extra caution in choosing cost indicators for architecture search algorithms especially given its computational cost and sensitivity to the selected cost indicator.
Revised: Given that we have shown earlier how cost indicators may disagree with one another or lead to impartial conclusions, we suggest that practitioners place extra caution in choosing cost indicators for architecture search algorithms especially given its computational cost and sensitivity to the selected cost indicator.

Original: Here, making assumptions that cost indicators are always interchangeable runs the risk of conducting a massive search for ﬁnding models that are impractical.
Revised: Here, making assumptions that cost indicators are always interchangeable runs the risk of conducting a massive search for ﬁnding models that are impractical.

Original:  S UGGESTIONS AND C ONCLUSION
Revised:  S UGGESTIONS AND C ONCLUSION

Original: A lot of recent work has focused on comparing different model architectures on the basis of a cost indicator like parameter count or FLOPs.
Revised: A lot of recent work has focused on comparing different model architectures on the basis of a cost indicator like parameter count or FLOPs.

Original: We have demonstrated in this paper that using any cost indicator alone can be misleading, with parameter count being the most problematic one.
Revised: We have demonstrated in this paper that using any cost indicator alone can be misleading, with parameter count being the most problematic one.

Original: Oftentimes, parameter count is used to imply “model capacity”; however, when varying model architecture in any nontrivial way, this is wrong.
Revised: Oftentimes, parameter count is used to imply “model capacity”; however, when varying model architecture in any nontrivial way, this is wrong.

Original: Correctly estimating and comparing capacity across model architectures is an open research problem.
Revised: Correctly estimating and comparing capacity across model architectures is an open research problem.

Original: Since each indicator stands for something different and comes with its own pros and cons, we suggest always reporting and plotting curves using all available cost indicators, and refraining from highlighting results using just a single one.
Revised: Since each indicator stands for something different and comes with its own pros and cons, we suggest always reporting and plotting curves using all available cost indicators, and refraining from highlighting results using just a single one.

Original: Moreover, given that it is basically impractical to provide a wholistic report of all cost metrics (for instance, runtime on all possible hardwares) narrowing down the efﬁciency claims to the exact setup that models are evaluated and avoiding overgeneralized conclusions in comparisons can already provide a much more clear picture to the community. 
Revised: Moreover, given that it is basically impractical to provide a holistic report of all cost metrics (for instance, runtime on all possible hardwares) narrowing down the efﬁciency claims to the exact setup that models are evaluated and avoiding overgeneralized conclusions in comparisons can already provide a much more clear picture to the community. Appendix A E XPERIMENTAL SETUP : S CALING D EPTH VS . S CALING W IDTH Detailed conﬁgurations and scores for the experiments on scaling Vision Transformer, by increasing the depth vs increasing the width of the model. [Table]

Revised: B A DDITIONAL C OST I NDICATORS

Revised: Depending on the use case, there are other cost indicators that may play a key role in determining the efﬁciency of a model:

Revised: Sample efﬁciency that can be expressed as training data size, e.g

Revised: number of data points (or tokens for language models Kaplan et al.

Revised: that a model needs in order to reach a reasonable performance.

Revised: Being sample efﬁcient depends on many factors, like inductive biases of the model or the training curriculum.

Revised: Carbon footprint of a model, during training or inference that is a proxy of the environmental impact.

Revised: Quantifying this cost directly is not straightforward, but (Strubell et al., 2019) proposed the approximate environmental costs of an ML model as Footprint

Revised: = ( ee train + queries × ee inference ) ×

Revised: CO2e datacenter / KWh, where, ee indicate the electrical energy (Patterson et al., 2021).

Revised: Monetary , i.e., the expenses for training or serving models, which is reported in the form of ﬁgures (Sharir et al., 2020).

Revised: As an example, the ﬁgures for training can be computed as total-train-time × total number of chips × prices offered by cloud solutions 4 .

Revised: This cost indicator is more often used in business documents, than in scientiﬁc articles.

Revised: number of model activations , proposed by Doll´ar et al.

Revised: (2021) as a complexity metric.

Revised: Model activations refers to the number of elements in the output tensors from the building blocks of the model, e.g., output size of convolutional layers in ResNet.

Revised: Doll´ar et al.

Revised: (2021) argue that the number of activations is a reliable predictor for the model runtime and showed that compared to FLOPs, it is more strongly correlated with the runtime on memory-bandwidth limited hardware.

Revised: Memory consumption is one of the main dimensions of efﬁciency and it has been used as a cost indicator in various research works (Bondarenko et al., 2021; Dosovitskiy et al., 2020; Kondratyuk et al., 2021).

Revised: Reporting memory footprint often is done in form of “peak memory usage”, during training that takes into account the memory consumption by the model, optimizer, and the pipeline.

Revised: Another form that is also common is comparing the maximum batch size that ﬁts on a speciﬁc device for a speciﬁc task.

Revised: Unlike model activation size, memory footprint depends on the hardware and implementation.

Revised: It has a direct implication of whether a model can ﬁt on a device, which is a hard constraint in some cases.