Original: Deep Compression of Pre-trained Transformer Models
Revised: Deep Compression of Pre-trained Transformer Models

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Pre-trained transformer models have achieved remarkable success in natural language processing (NLP) and have recently become competitive alternatives to
Revised: Pre-trained transformer models have achieved remarkable success in natural language processing (NLP) and have recently become competitive alternatives to

Original: Convolution Neural Networks (CNN) and Recurrent Neural Networks (RNN) invision and speech tasks, respectively.
Revised: Convolution Neural Networks (CNN) and Recurrent Neural Networks (RNN) in vision and speech tasks, respectively.

Original: Due to their excellent computational efﬁciency and scalability, transformer models can be trained on exceedingly largeamounts of data at the expense of tremendous growth in model size.
Revised: Due to their excellent computational efﬁciency and scalability, transformer models can be trained on exceedingly large amounts of data at the expense of tremendous growth in model size.

Original: As highperformance, large-scale, and pre-trained transformer models become increasinglyavailable for users to download and ﬁne-tune for customized downstream tasks,their deployment becomes challenging due to the vast amount of operations andlarge memory footprint.
Revised: As high performance, large-scale, and pre-trained transformer models become increasingly available for users to download and ﬁne-tune for customized downstream tasks, their deployment becomes challenging due to the vast amount of operations and large memory footprint.

Original: To address this challenge, we introduce methods to deeplycompress pre-trained transformer models across three major application domains:
Revised: To address this challenge, we introduce methods to deeply compress pre-trained transformer models across three major application domains:

Original: NLP, speech, and vision.
Revised: NLP, speech, and vision.

Original: Speciﬁcally, we quantize transformer backbones down to4-bit and further achieve 50% ﬁne-grained structural sparsity on pre-trained BERT,
Revised: Speciﬁcally, we quantize transformer backbones down to 4-bit and further achieve 50% ﬁne-grained structural sparsity on pre-trained BERT,

Original: Wav2vec2.0, and Vision Transformer (ViT) models to demonstrate 16x compression while maintaining model accuracy.
Revised: Wav2vec2.0, and Vision Transformer (ViT) models to demonstrate 16x compression while maintaining model accuracy.

Original: This is achieved by identifying criticalinitialization strategies for quantization- and sparsity- aware ﬁne-tuning as well asdeveloping novel techniques such as quantizers with a zero-preserving format andscheduled dropout.
Revised: This is achieved by identifying critical initialization strategies for quantization- and sparsity- aware ﬁne-tuning as well as developing novel techniques such as quantizers with a zero-preserving format and scheduled dropout.

Original: These hardware-friendly techniques need only to be applied inthe ﬁne-tuning phase for downstream tasks, which renders them especially suitablefor acceleration and deployment of pre-trained transformer models.
Revised: These hardware-friendly techniques need only to be applied in the ﬁne-tuning phase for downstream tasks, which renders them especially suitable for acceleration and deployment of pre-trained transformer models.

Original:  Introduction
Revised:  Introduction

Original: Since its inception in 2017 , the attention-based transformer architecture [1] has excelled in many
Revised: Since its inception in 2017 , the attention-based transformer architecture [1] has excelled in many

Original: Natural Language Processing (NLP) tasks and become the preferred building block for state-of-the-artlanguage models.
Revised: Natural Language Processing (NLP) tasks and become the preferred building block for state-of-the-art language models.

Original: With their excellent capacity for transfer learning, transformer models that are pretrained on large corpora such as BERT [2]) and GPT-3 [3] can be ﬁne-tuned for downstream tasksto achieve state-of-the-art performance.
Revised: With their excellent capacity for transfer learning, transformer models that are pretrained on large corpora, such as BERT [2]) and GPT-3 [3], can be ﬁne-tuned for downstream tasks to achieve state-of-the-art performance.

Original: Today, ﬁne-tuning a transformer-based pre-trained languagemodel has become the de facto standard for NLP tasks.
Revised: Today, ﬁne-tuning a transformer-based pre-trained language model has become the de facto standard for NLP tasks.

Original: Inspired by this success in NLP, transformermodels are also being actively explored in other application domains and making tremendous progress.
Revised: Inspired by this success in NLP, transformer models are also being actively explored in other application domains and making tremendous progress.

Original: For vision applications, transformer models have demonstrated excellent performance — matchingor exceeding CNN-based models on image classiﬁcation [4, 5] and object detection [6] tasks.
Revised: For vision applications, transformer models have demonstrated excellent performance — matching or exceeding CNN-based models on image classiﬁcation [4, 5] and object detection [6] tasks.

Original: Forspeech applications, models with a transformer backbone such as Wav2vec2.0 [7], HuBERT [8], and
Revised: For speech applications, models with a transformer backbone such as Wav2vec2.0 [7], HuBERT [8], and

Original: WavLM [9] show comparable performance to traditional RNN/LSTM-based models, but with muchimproved computation efﬁciency and scalability
Revised: WavLM [9] show comparable performance to traditional RNN/LSTM-based models, but with much improved computation efﬁciency and scalability.

Original: Pre-trained transformer models used as a ’Swiss Army knife’ for a wide range of machine learningtasks are becoming so-called "Foundation Models", which can potentially change the landscapeof deep learning research and development [10].
Revised: Pre-trained transformer models used as a ‘Swiss Army knife’ for a wide range of machine learning tasks are becoming so-called “Foundation Models”, which can potentially change the landscape of deep learning research and development [10].

Original: Today, transformer models are pre-trained onincreasingly large amounts of data, which has grown model sizes to unprecedented levels.
Revised: Today, transformer models are pre-trained on increasingly large amounts of data, which has grown model sizes to unprecedented levels.

Original: In the past few years, many high performance, large-scale models with millions to billions parametershave been released for users to download and ﬁne-tune for customized downstream tasks [11].
Revised: In the past few years, many high performance, large-scale models with millions to billions parameters have been released for users to download and ﬁne-tune for customized downstream tasks [11].

Original: However, the vast number of computation operations and large memory footprint are becoming keybarriers to their practical deployment and adoption in production settings.
Revised: However, the vast number of computation operations and large memory footprint are becoming key barriers to their practical deployment and adoption in production settings.

Original: To address this challenge,techniques such as knowledge distillation [12–16] have been used to pre-train compact transformerlike models.
Revised: To address this challenge, techniques such as knowledge distillation [12–16] have been used to pre-train compact transformerlike models.

Original: However, standard transformer topologies remain the preferred architecture to achievehigh performance on a wide range of tasks.
Revised: However, standard transformer topologies remain the preferred architecture to achieve high performance on a wide range of tasks.

Original: To compress deep neural networks, quantization has proven to be effective in many application domains and is a common technique to accelerate DNNs on GPU and other AI hardware platforms [17].
Revised: To compress deep neural networks, quantization has proven to be effective in many application domains and is a common technique to accelerate DNNs on GPU and other AI hardware platforms [17].

Original: Thus far, quantization of transformer models has primarily been studied for NLP applications.
Revised: Thus far, quantization of transformer models has primarily been studied for NLP applications.

Original: 8-bitquantization was ﬁrst applied to BERT in [18, 19] and showed negligible model accuracy degradation.
Revised: 8-bit quantization was ﬁrst applied to BERT in [18, 19] and showed negligible model accuracy degradation.

Original: Later, weight precision was pushed to 4-bit or lower [20, 21], but activations remained in 8-bit dueto sensitivity to quantization errors.
Revised: Later, weight precision was pushed to 4-bit or lower [20, 21], but activations remained in 8-bit due to sensitivity to quantization errors.

Original: Recently, knowledge distillation has been shown  effective inquantizing BERT models down to 4-bit for both weights and activations while maintaining baselineaccuracy.
Revised: Recently, knowledge distillation has been shown to be effective in quantizing BERT models down to 4-bit for both weights and activations while maintaining baseline accuracy.

Original: Vision transformers (ViT) have been successfully quantized down to 6-bit [22] usingpost-training quantization (PTQ) techniques.
Revised: Vision transformers (ViT) have been successfully quantized down to 6-bit [22] using post-training quantization (PTQ) techniques.

Original: For speech, the authors in [23] quantized a simpliﬁedtransformer down to 8-bit.
Revised: For speech, the authors in [23] quantized a simpliﬁed transformer down to 8-bit.

Original: To date, vision and speech transformer models have not yet been successfully quantized down to precisions as low as 4-bit while NLP transformer models have only reachedthis level with complex distillation techniques.
Revised: To date, vision and speech transformer models have not yet been successfully quantized down to precisions as low as 4-bit while NLP transformer models have only reached this level with complex distillation techniques.

Original: Pruning is another promising method to compress transformer models — motivated by the assumptionthat the rich features learned during the pre-training phase may be redundant when ﬁne-tuning fordownstream tasks and can thus be pruned.
Revised: Pruning is another promising method to compress transformer models — motivated by the assumption that the rich features learned during the pre-training phase may be redundant when ﬁne-tuning for downstream tasks and can thus be pruned.

Original: In the past few years, pruning has been extensivelyexplored on transformer models for NLP [24], ViT [25], and Speech [26] to achieve 40-50% sparsitywithout signiﬁcant accuracy loss.
Revised: In the past few years, pruning has been extensively explored on transformer models for NLP [24], ViT [25], and Speech [26] to achieve 40-50% sparsity without signiﬁcant accuracy loss.

Original: However, most such work focuses on unstructured sparsity, whichis challenging to harness in modern efﬁcient hardware using vector- and matrix-based instructions.
Revised: However, most such work focuses on unstructured sparsity, which is challenging to harness in modern efﬁcient hardware using vector- and matrix-based instructions.

Original: Hardware vendors are instead introducing techniques such as ﬁne-grained structured sparsity [27],which can achieve hardware efﬁciency improvements across a wide range of DNN models.
Revised: Hardware vendors are instead introducing techniques such as ﬁne-grained structured sparsity [27], which can achieve hardware efﬁciency improvements across a wide range of DNN models.

Original: Applying both quantization and pruning techniques to the same model is a particular challenge —due primarily to difﬁculties in coordinating two separate lossy mechanisms to optimize and recovermodel accuracy [28] [29].
Revised: Applying both quantization and pruning techniques to the same model is a particular challenge — due primarily to difﬁculties in coordinating two separate lossy mechanisms to optimize and recover model accuracy [28] [29].

Original: To our knowledge, [27] is the only work to explore the combinationof quantization and sparsity on pre-trained transformer models such as BERT.
Revised: To our knowledge, [27] is the only work to explore the combination of quantization and sparsity on pre-trained transformer models such as BERT.

Original: In those results, for50% sparsity, INT8 is the lowest precision that achieves reasonable accuracy.
Revised: In those results, for 50% sparsity, INT8 is the lowest precision that achieves reasonable accuracy.

Original: To recover modelaccuracy in the sparse INT8 model, pre-training must be repeated with sparsity before ﬁne-tuning fordownstream tasks — a requirement that becomes extremely challenging due to access limitations topre-training datasets and the signiﬁcant compute resources needed for today’s large-scale models [3].
Revised: To recover model accuracy in the sparse INT8 model, pre-training must be repeated with sparsity before ﬁne-tuning for downstream tasks — a requirement that becomes extremely challenging due to access limitations to pre-training datasets and the signiﬁcant compute resources needed for today’s large-scale models [3].

Original: In this paper, we combine 4-bit quantization and 50% pruning to deeply compress transformermodels across multiple application domains, i.e. BERT, Wav2vec2.0, and Vision Transformer (ViT)models.
Revised: In this paper, we combine 4-bit quantization and 50% pruning to deeply compress transformer models across multiple application domains, i.e. BERT, Wav2vec2.0, and Vision Transformer (ViT) models.

Original: Speciﬁcally, transformer backbones were quantized down to 4-bit while combined with 50%ﬁne-grained structural sparsity to achieve up to 16x model compression while maintaining modelaccuracy.
Revised: Speciﬁcally, transformer backbones are quantized down to 4-bit and combined with 50% ﬁne-grained structural sparsity to achieve up to 16x model compression while maintaining model accuracy.

Original: Our method does not require repeating pre-training nor accessing pre-training datasets.
Revised: Our method does not require repeating pre-training nor accessing pre-training datasets.

Original: Instead, quantization and pruning are performed during the ﬁne-tuning phase on downstream tasksusing downloaded pre-trained models.
Revised: Instead, quantization and pruning are performed during the ﬁne-tuning phase on downstream tasks using downloaded pre-trained models.

Original: Our techniques are straightforward to implement without theneed for complex methods such as knowledge distillation, which may require large memory andcompute resource.
Revised: Our techniques are straightforward to implement without the need for complex methods such as knowledge distillation, which may require large memory and compute resource.

Original: Our precision format and sparsity conﬁguration are hardware-friendly and can beefﬁciently implemented in modern deep learning accelerators.
Revised: Our precision format and sparsity conﬁguration are hardware-friendly and can be efﬁciently implemented in modern deep learning accelerators.

Revised: Our methodology generalizes across different application domains, which brings an opportunity to greatly simplify the software and hardware-based acceleration and deployment of pre-trained transformer models.

Original: In summary, we make the followingcontributions towards accurate, sparse, 4-bit transformer models across application domains:
Revised: In summary, we make the following contributions towards accurate, sparse, 4-bit transformer models across application domains:

Original: We report, for the ﬁrst time, transformer backbones quantized down to 4-bit precision with 50%ﬁne-grained structural sparsity on pre-trained BERT, Wav2vec2.0, and ViT models.
Revised: We report, for the ﬁrst time, transformer backbones quantized down to 4-bit precision with 50% ﬁne-grained structural sparsity on pre-trained BERT, Wav2vec2.0, and ViT models.

Original: We achieve1.62%, 0.45% and 0.52% degradation for the question-answering task on SQuAD1.1, the speechrecognition task on Librispeech, and the image classiﬁcation task on ImageNet1k, respectively.
Revised: We achieve 1.62%, 0.45% and 0.52% degradation for the question-answering task on SQuAD1.1, the speech recognition task on Librispeech, and the image classiﬁcation task on ImageNet1k, respectively.

Original: Without sparsity, all 4-bit models achieve < 1% accuracy loss.2.
Revised: Without sparsity, all 4-bit models achieve < 1% accuracy loss with respect to the baseline.

Original: We propose procedures for quantization-aware and sparsity-aware ﬁne-tuning to minimize theimpact of these two lossy processes.
Revised: We propose procedures for quantization-aware and sparsity-aware ﬁne-tuning to minimize the impact of these two lossy processes.

Original: In particular, we show that proper initialization of thequantized model, quantization scales, and sparse model are critical to minimize accuracy losses.3.
Revised: In particular, we show that proper initialization of the quantized model, quantization scales, and sparse model are critical to minimize accuracy losses.

Original: We introduce novel techniques, including a SAWB+ weight quantizer with a symmetric, zeropreserving quantization format and scheduled dropout, to enable the deep compression.
Revised: We introduce novel techniques, including a SAWB+ weight quantizer with a symmetric, zeropreserving quantization format and scheduled dropout, to enable the deep compression.

Original:  Quantization-aware and Sparsity-Aware Fine-Tuning
Revised:  Quantization-aware and Sparsity-Aware Fine-Tuning

Original: Performing quantization/sparsity-aware training during the pre-training phase on large amounts ofdata may offer the best opportunity to optimize and recover model accuracy for extremely compressedtransformer models [27].
Revised: Performing quantization/sparsity-aware training during the pre-training phase on large amounts of data may offer the best opportunity to optimize and recover model accuracy for extremely compressed transformer models [27].

Original: However, in practice, the pre-training phase is not generally transparentto end users as it is often difﬁcult to access pre-training data and repeat the pre-training phase,which may require hundreds of GPUs and very long training times [11].
Revised: However, in practice, the pre-training phase is not generally transparent to end users as it is often difﬁcult to access pre-training data and repeat the pre-training phase, which may require hundreds of GPUs and very long training times [11].

Original: On the other hand, theﬁne-tuning phase for downstream tasks is often lightweight, during which users can directly operateon their own datasets and tune models for only a few epochs [11].
Revised: On the other hand, the ﬁne-tuning phase for downstream tasks is often lightweight, during which users can directly operate on their own datasets and tune models for only a few epochs [11].

Original: In this work, we thus focuson quantization/sparsity-aware ﬁne-tuning and introduce techniques in this context to enable deepcompression of pre-trained transformer models.
Revised: In this work, we thus focus on quantization/sparsity-aware ﬁne-tuning and introduce techniques in this context to enable deep compression of pre-trained transformer models.

Revised: Figure 1: a) The transformer encoder block consisting of FC modules and BMM modules.

Revised: Gray modules are linear layers where pruning is applied as shown in b), which is a schematic of a ﬁne-grain structurally pruned weight matrix.

Original: Figure 1 shows operations that are quantized and sparsiﬁed in a transformer block.
Revised: Figure 1 shows operations that are quantized and sparsiﬁed in a transformer block.

Original: Six linear layers,including QKV layers, a projection layer and two feed-forward layers are quantized to 4-bit and theirweights are pruned by 50%.
Revised: Six linear layers, including QKV layers, a projection layer and two feed-forward layers are quantized to 4-bit and their weights are pruned by 50%.

Original: All input activations of two batch matrix-matrix product (BMM) layersare quantized.
Revised: All input activations of two batch matrix-matrix product (BMM) layers are quantized.

Original: To preserve the ﬁdelity of the probability distribution, we keep BMM2 in 8-bit.
Revised: To preserve the ﬁdelity of the probability distribution, we keep BMM2 in 8-bit.

Original: 2.1 Quantization2.1.1 Weight Quantization
Revised: 2.1 Quantization 2.1.1 Weight Quantization

Original: SAWB+ is an enhancement of the SAWB (statistics-aware weight binning) quantizer introducedby [30].
Revised: SAWB+ is an enhancement of the SAWB (statistics-aware weight binning) quantizer introduced by [30].

Original: SAWB exploits both the ﬁrst and second moments of the weight distribution to minimize thequantization error.
Revised: SAWB exploits both the ﬁrst and second moments of the weight distribution to minimize the quantization error.

Original: The optimal quantization scale, α w is calculated by the equation:
Revised: The optimal quantization scale, α w is calculated by the equation:

Original: [Equation]
Revised: [Equation]

Original: (1) where c1 and c2 are coefﬁcients empirically determined ofﬂine by ﬁtting six standard distributionsat a given precision.
Revised: where c1 and c2 are coefﬁcients empirically determined ofﬂine by ﬁtting six standard distributions at a given precision.

Original: The details of extracting c1 and c2 can be ﬁnd in [31] and the coefﬁcientsused in this paper are listed in Appendix-B. SAWB can effectively capture weight distributions toachieve excellent performance on a range of DNN models; however, in the backward path, SAWBcan cause instability. 
Revised: The details of extracting c1 and c2 can be found in [31] and the coefﬁcients used in this paper are listed in Appendix-B. SAWB can effectively capture weight distributions to achieve excellent performance on a range of DNN models; however, in the backward path, SAWB can cause instability. As shown in the left of equation 2, where W and W q represent weight and

Original: As shown in the left of equation 2, the clamped weights see zero gradientsduring optimization, which can cause two potential problems: 1) the variance of the weight gradientsis reduced by truncation, which slows learning in non-quantized optimization, and 2) larger weightsclamped early in training may never be updated, which reduces the representation power of theweight space during training.
Revised: [Table] quantized weight respectively, the clamped weights see zero gradients during optimization, which can cause two potential problems: 1) the variance of the weight gradients is reduced by truncation, which slows learning in non-quantized optimization, and 2) larger weights clamped early in training may never be updated, which reduces the representation power of the weight space during training.

Original: To overcome this problem, we introduce a simple modiﬁcation toallow the gradients of clipped weights to pass through during weight updates while keeping theforward pass the same.
Revised: To overcome this problem, we introduce a simple modiﬁcation to allow the gradients of clipped weights to pass through during weight updates while keeping the forward pass the same.

Original: This modiﬁed technique, SAWB+, captures the statistical characteristicsof the distribution of weights just as in SAWB, but also has the ability to search the entire weightspace during optimization with the same gradient variance per iteration step.
Revised: This modiﬁed technique, SAWB+, captures the statistical characteristics of the distribution of weights just as in SAWB, but also has the ability to search the entire weight space during optimization with the same gradient variance per iteration step.

Original: In this work, we use SAWB+ for the weight quantization for all transformer models and observe no degradation fromweight quantization.[Table]
Revised: In this work, we use SAWB+ for the weight quantization for all transformer models and observe no degradation from weight quantization.

Original: [Equation]
Revised: [Equation]

Original: Zero alignment is another critical technique to enable pruning at low precision.
Revised: Zero alignment is another critical technique to enable pruning at low precision.

Original: For integer weights,instead of utilizing the full range of precision levels, i.e. 2 k , we choose to lose one level whilekeeping the distribution symmetric about zero, resulting in 2 k − 1 levels.
Revised: For integer weights, instead of utilizing the full range of precision levels, i.e. 2 k , we choose to lose one level while keeping the distribution symmetric about zero, resulting in 2 k − 1 levels.

Original: For example, in INT4, thenumber of integer levels is 15, i.e. [-7, -6, -5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5, 6, 7].
Revised: For example, in INT4, the number of integer levels is 15, i.e. [-7, -6, -5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5, 6, 7].

Original: This way, ﬂoatingpoint zeros in input will be represented by integer zeros.
Revised: This way, ﬂoating point zeros in input will be represented by integer zeros.

Original: This zero alignment has signiﬁcant impacton tuning the sparse model.
Revised: This zero alignment has signiﬁcant impact on tuning the sparse model.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2, with zero-aligned weights, the BERT-base modelconverges to a much lower training loss and higher evaluation score in the ﬁne tuning process ascompared with the non-aligned case.
Revised: 2, with zero-aligned weights, the BERT-base model converges to a much lower training loss and higher evaluation score in the ﬁne tuning process as compared with the non-aligned case.

Original: The intuition is that with zeros in the weights as existing masks,the zero-aligned model requires less effort to learn the rest of the masks to achieve the same amount ofsparsity.
Revised: The intuition is that with zeros in the weights as existing masks, the zero-aligned model requires less effort to learn the rest of the masks to achieve the same amount of sparsity.

Original: The same improvement is also observed in Wav2vec2.0 and ViT models, as shown in Table1.
Revised: The same improvement is also observed in Wav2vec2.0 and ViT models, as shown in Table 1.

Original: Thus, just through zero alignment, the accuracy of three models can be boosted by 1.01%, 1.23%and 0.35%, respectively.
Revised: Thus, just through zero alignment, the accuracy of three models can be boosted by 1.01%, 1.23% and 0.35%, respectively.

Original: Furthermore, symmetric and uniformly distributed quantization bins withzero alignment are also hardware friendly - allowing for simpliﬁed multiplication and accumulation(MAC) design and efﬁcient data-ﬂow.
Revised: Furthermore, symmetric and uniformly distributed quantization bins with zero alignment are also hardware friendly - allowing for simpliﬁed multiplication and accumulation (MAC) design and efﬁcient data-ﬂow [32].

Original: 2.1.2 Activation Quantization
Revised: 2.1.2 Activation Quantization

Original: For activation quantization, we use two well-known quantizers: MinMax and PACT [31].
Revised: For activation quantization, we use two well-known quantizers: MinMax and PACT [31].

Original: We extendthe original 1-sided PACT to a 2-sided PACT to quantize activations with both positive and negativevalues.
Revised: We extend the original 1-sided PACT to a 2-sided PACT to quantize activations with both positive and negative values.

Original: α and α n are used to deﬁne the dynamic range and the clamped activation output is uniformlyquantized to k bits for the dot-product computation.
Revised: α and α n are used to deﬁne the dynamic range and the clamped activation output is uniformly quantized to k bits for the dot-product computation.

Original: For the MinMax quantizer, α and α n take the min and max of the tensors.
Revised: For the MinMax quantizer, α and α n take the min and max of the tensors.

Original: For PACT, both clippinglevels are parameterized and dynamically adjusted via gradient descent-based training.
Revised: For PACT, both clipping levels are parameterized and dynamically adjusted via gradient descent-based training.

Original: When using
Revised: When using

Original: PACT, we sample 10 batches of the downstream dataset and initialize α and α n of each layer basedon a given percentile (e.g. 99% and 1%) of the observed activation distribution [30].
Revised: PACT, we sample 10 batches of the downstream dataset and initialize α and α n of each layer based on a given percentile (e.g. 99% and 1%) of the observed activation distribution [30].

Original: Note that weuse the same quantizer for both symmetric and asymmetric activation functions such as GeLU.
Revised: Note that we use the same quantizer for both symmetric and asymmetric activation functions such as GeLU.

Original: Wealso preserve activations of zero value by aligning zero to an integer level.
Revised: We also preserve activations of zero value by aligning zero to an integer level.

Original: Unlike for weights, thefull 2 k levels are utilized for activation quantization.
Revised: Unlike for weights, the full 2 k levels are utilized for activation quantization.

Original: While experimenting with different quantizers, we observed that for all INT8 models, the MinMaxquantizer out-performed the PACT quantizer, as shown in Table 2.
Revised: While experimenting with different quantizers, we observed that for all INT8 models, the MinMax quantizer out-performed the PACT quantizer, as shown in Table 2.

Original: This observation comes as a surprise, since ideally, learned α s should match the performance of the special MinMax case.
Revised: This observation comes as a surprise, since ideally, learned α s should match the performance of the special MinMax case.

Original: Asshown in Fig.
Revised: As shown in Fig.

Original: 3, even when α is initialized to 99.9 or 99.95 percentile, training curves are noisierthan when initializing to the max of the sampled activations.
Revised: 3, even when α is initialized to 99.9 or 99.95 percentile, training curves are noisier than when initializing to the max of the sampled activations.

Original: Further investigation led us to concludethat a small amount of large-amplitude outliers play an important role in transformer training.
Revised: Further investigation led us to conclude that a small amount of large-amplitude outliers play an important role in transformer training compared to CNN-based models [33].

Original: INT4 models are very limited in the number of precision levels available and hence become moresensitive to quantization errors as compared to INT8.
Revised: INT4 models are very limited in the number of precision levels available and hence become more sensitive to quantization errors as compared to INT8.

Original: For this reason, we use the MinMax quantizerfor INT8 and sparse INT8 models while using PACT for INT4 models.
Revised: For this reason, we use the MinMax quantizer for INT8 and sparse INT8 models while using PACT for INT4 models.

Original: The details of α initializationconditions are listed in Appendix-B.[Figure]
Revised: The details of α initialization conditions are listed in Appendix-B.

Original: 2.2 Fine-Grained Structured Pruning
Revised: 2.2 Fine-Grained Structured Pruning

Original: [Figure] Figure 1b depicts the details of the ﬁne-grained structured pruning performed in this work, which isbased on the 2:4 sparsity proposed in [27].
Revised: Figure 1b depicts the details of the ﬁne-grained structured pruning performed in this work, which is based on the 2:4 sparsity proposed in [27].

Original: We form groups of 4 consecutive weights values alongthe "in_features" dimension of weights and zero out 2 values to achieve 50% sparsity.
Revised: We form groups of 4 consecutive weights values along the "in_features" dimension of weights and zero out 2 values to achieve 50% sparsity.

Original: The locationof the two zero values can vary across the different groups in a layer.
Revised: The location of the two zero values can vary across the different groups in a layer.

Original: The pruned weights can becompressed by storing only the non-zero weight values and their corresponding indices within eachgroup — 2 non-zero values and 2 indices per group.
Revised: The pruned weights can be compressed by storing only the non-zero weight values and their corresponding indices within each group — 2 non-zero values and 2 indices per group.

Original: Since indices can only take one of four possiblevalues for a group size of 4, 2 bits are sufﬁcient to represent each index within a group.
Revised: Since indices can only take one of four possible values for a group size of 4, 2 bits are sufﬁcient to represent each index within a group.

Original: We use a mask for weight pruning.
Revised: We use a mask for weight pruning.

Original: The weight pruning mask is extracted only once before the startof the training instead of, e.g. Iterative Magnitude Pruning (IMP) that can be otherwise adopted[28] [29].
Revised: The weight pruning mask is extracted only once before the start of the training instead of, e.g. Iterative Magnitude Pruning (IMP) that can be otherwise adopted [28] [29].

Original: At each iteration, the weights are quantized ﬁrst and the pruning mask is applied on thequantized weights and then fed into matrix computation.
Revised: At each iteration, the weights are quantized ﬁrst and the pruning mask is applied on the quantized weights and then fed into matrix computation.

Original: [Figure]

Original: 2.3 Quantization- and Sparsity-Aware Fine-Tuning
Revised: 2.3 Quantization- and Sparsity-Aware Fine-Tuning

Original: [Figure] A key challenge in combining quantization and sparsity is to determine the workﬂow by whichto apply these two lossy processes during ﬁne-tuning, e.g. tuning with quantization followed bypruning, or vice versa.
Revised: A key challenge in combining quantization and sparsity is to determine the workﬂow by which to apply these two lossy processes during ﬁne-tuning, e.g. tuning with quantization followed by pruning, or vice versa.

Original: We discover that quantization and sparsity can be jointly ﬁne-tuned in oneoptimization process provided that the process is properly initialized.
Revised: We discover that quantization and sparsity can be jointly ﬁne-tuned in one [Table] optimization process provided that the process is properly initialized.

Original: In this section, we explain thekey initialization methods for the compressed model to achieve high accuracy.
Revised: In this section, we explain the key initialization methods for the compressed model to achieve high accuracy.

Original: 2.3.1 Initialization for Quantization
Revised: 2.3.1 Initialization for Quantization

Original: Pre-trained models are generally trained on large datasets in full FP32 precision.
Revised: Pre-trained models are generally trained on large datasets in full FP32 precision.

Original: The rich featureslearned during pre-training should provide opportunity for models to accommodate quantizationerror during the ﬁne-tuning phase.
Revised: The rich features learned during pre-training should provide opportunity for models to accommodate quantization error during the ﬁne-tuning phase.

Original: However, when we use the pre-trained model as initialization forquantization-aware ﬁne-tuning, the resulting quantized models consistently show inferior performanceas shown in Table 3.
Revised: However, when we use the pre-trained model as initialization for quantization-aware ﬁne-tuning, the resulting quantized models consistently show inferior performance as shown in Table 3.

Original: For INT4 even without sparsity, the three models studied suffer signiﬁcantdegradation of 2.01%, 0.82%, 1.37%.
Revised: For INT4 even without sparsity, the three models studied suffer signiﬁcant degradation of 2.01%, 0.82%, 1.37%.

Original: Even in INT8, the ViT model incurs > 1% degradation.
Revised: Even in INT8, the ViT model incurs > 1% degradation.

Original: Instead, we can use full precision ﬁne-tuned models to initialize quantization-aware ﬁne-tuning.
Revised: Instead, we can use full precision ﬁne-tuned models to initialize quantization-aware ﬁne-tuning.

Original: Asshown in Table 3, this signiﬁcantly boosts the performance of the quantized model both in INT8and INT4 precisions.
Revised: As shown in Table 3, this signiﬁcantly boosts the performance of the quantized model both in INT and INT4 precisions.

Original: Particularly for INT4, all three models achieve accuracy within < 1% of the [Table]
Revised: Particularly for INT4, all three models achieve accuracy within < 1% of the baseline.

Original: [Table] baseline. To the best of our knowledge, this is the ﬁrst time that INT4 transformer models (bothweight and activation) achieve iso-accuracy without the teacher-student distillation technique, whichrequires computation of the full precision teacher models during tuning rather than just using them forinitialization.
Revised: To the best of our knowledge, this is the ﬁrst time that INT4 transformer models (both weight and activation) achieve iso-accuracy without the teacher-student distillation technique, which requires computation of the full precision teacher models during tuning rather than just using them for initialization.

Original: The results also indicate that initialization with pre-trained models is more vulnerable toquantization noise despite redundant features, whereas initialization with the full precision ﬁne-tunedmodels effectively learn the features most relevant to downstream tasks, which improves the resultantquantized models.
Revised: The results also indicate that initialization with pre-trained models is more vulnerable to quantization noise despite redundant features, whereas initialization with the full precision ﬁne-tuned models effectively learn the features most relevant to downstream tasks, which improves the resultant quantized models.

Original: When combined with the quantizers described in Section 2.1, quantized modelaccuracy is signiﬁcantly improved.
Revised: When combined with the quantizers described in Section 2.1, quantized model accuracy is signiﬁcantly improved.

Original: 2.3.2 Initialization for pruning
Revised: 2.3.2 Initialization for Pruning

Original: When adding sparsity, for INT8 precision models, the full precision ﬁne-tuned models are effective asinitialization, as shown in table 3.
Revised: When adding sparsity, for INT8 precision models, the full precision ﬁne-tuned models are effective as initialization, as shown in table 3.

Original: Both the Wav2vec2.0 and ViT models can reach baseline accuracywhile the Bert-base model is within 1% of the baseline.
Revised: Both the Wav2vec2.0 and ViT models can reach baseline accuracy while the Bert-base model is within 1% of the baseline.

Original: Note that we achieve the same accuracy on asparse INT8 BERT model as reported in [27], but without the need to repeat pre-training.
Revised: Note that we achieve the same accuracy on a sparse INT8 BERT model as reported in [27], but without the need to repeat pre-training.

Original: For sparse INT4 models, however, it becomes difﬁcult to maintain accuracy with this same initialization.
Revised: For sparse INT4 models, however, it becomes difﬁcult to maintain accuracy with this same initialization.

Original: As shown in Table 3, BERT and ViT models, in particular, suffer signiﬁcant degradation > 2% .
Revised: As shown in Table 3, BERT and ViT models, in particular, suffer signiﬁcant degradation > 2% .

Original: To overcome this, we propose a different initialization strategy, in which we use a pre-tuned sparse
Revised: To overcome this, we propose a different initialization strategy, in which we use a pre-tuned sparse

Original: INT8 model as initialization to ﬁne-tune the sparse INT4 model.
Revised: INT8 model as initialization to ﬁne-tune the sparse INT4 model.

Original: This is inspired by the observationthat INT8 models show no degradation when using MinMax quantizers.
Revised: This is inspired by the observation that INT8 models show no degradation when using MinMax quantizers.

Original: Intuitively, a pre-tunedsparse INT8 model is a sparse architecture that can already tolerate low precision, which makes ita strong starting point to further adjust to the larger quantization errors associated with INT4.
Revised: Intuitively, a pre-tuned sparse INT8 model is a sparse architecture that can already tolerate low precision, which makes it a strong starting point to further adjust to the larger quantization errors associated with INT4.

Original: Asshown in Table 3, this initialization method boosts the accuracy signiﬁcantly and brings Wav2vec2.0and ViT sparse INT4 models to within 1% of baseline accuracy.
Revised: As shown in Table 3, this initialization method boosts the accuracy signiﬁcantly and brings Wav2vec2. and ViT sparse INT4 models to within 1% of baseline accuracy.

Original: Fig.
Revised: Fig.

Original: 4 shows the convergence curvesfor the ﬁne-tuning of sparse INT4 ViT models initialized using FP32 and sparse INT8 ﬁne-tunedmodels.
Revised: 4 shows the convergence curves for the ﬁne-tuning of sparse INT4 ViT models initialized using FP32 and sparse INT8 ﬁne-tuned models.

Original: The training loss starts at a much lower level when using sparse INT8 initialization and thenetwork converges to lower loss and higher accuracy across the entire tuning process.
Revised: The training loss starts at a much lower level when using sparse INT8 initialization and the network converges to lower loss and higher accuracy across the entire tuning process.

Original: 2.3.3 Scheduled Dropout
Revised: 2.3.3 Scheduled Dropout

Original: For the BERT model, sparse INT8 initialization starts at lower training loss just as in Wav2vec2.0and ViT, but it does not show immediate improvement in the F1 score due to over-ﬁtting potentiallycaused by the small size of the SQuAD1.1 dataset.
Revised: For the BERT model, sparse INT8 initialization starts at lower training loss just as in Wav2vec2. and ViT, but it does not show immediate improvement in the F1 score due to over-ﬁtting potentially caused by the small size of the SQuAD1.1 dataset.

Original: To effectively take advantage of sparse INT8initialization and avoid over-ﬁtting in the early stages of training, we propose scheduled dropout.
Revised: To effectively take advantage of sparse INT initialization and avoid over-ﬁtting in the early stages of training, we propose scheduled dropout.

Original: Scheduled dropout initially applies a very high dropout rate, e.g. 0.35, and then linearly decreases therate until the 10k-th iteration, where it ﬁnally stays constant, e.g. at 0.2, until the end of the ﬁne-tuningphase.
Revised: Scheduled dropout initially applies a very high dropout rate, e.g. 0.35, and then linearly decreases the rate until the 10k-th iteration, where it ﬁnally stays constant, e.g. at 0.2, until the end of the ﬁne-tuning [Table] phase.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 5, scheduled dropout improves generalization from the beginning of thetraining, which enables 4-bit models to get closer to baseline F1 scores.
Revised: 5, scheduled dropout improves generalization from the beginning of the training, which enables 4-bit models to get closer to baseline F1 scores.

Original:  Implementation
Revised: 2.4 Hardware Implementation

Original: To evaluate the hardware beneﬁts of quantized and pruned transformers, we consider a systolicarray-based AI hardware accelerator similar to [32] [33].
Revised: To evaluate the hardware beneﬁts of quantized and pruned transformers, we consider a systolicarray-based AI hardware accelerator similar to [34] [35].

Original: We enhance the accelerator to furtherexploit ﬁne-grained structured weight sparsity by skipping computations on zero-valued weights.
Revised: We enhance the accelerator to further exploit ﬁne-grained structured weight sparsity by skipping computations on zero-valued weights.

Original: Speciﬁcally, the systolic array is enhanced to read only the non-zero weights in a group and performa multiply-accumulate operation with the activation value corresponding to the non-zero index.
Revised: Speciﬁcally, the systolic array is enhanced to read only the non-zero weights in a group and perform a multiply-accumulate operation with the activation value corresponding to the non-zero index.

Original: The primary hardware beneﬁt of sparse transformers is the memory capacity and bandwidth savingsobtained from storing their weights in a compressed manner (Section 2.2).
Revised: The primary hardware beneﬁt of sparse transformers is the memory capacity and bandwidth savings obtained from storing their weights in a compressed manner (Section 2.2).

Original: These savings are adirect function of the amount of weight sparsity and the precision of weights.
Revised: These savings are a direct function of the amount of weight sparsity and the precision of weights.

Original: Accordingly, for INT8precision with 50% sparsity, we achieve a memory capacity savings of 1.6 × with respect to a dense
Revised: Accordingly, for INT  precision with 50% sparsity, we achieve a memory capacity savings of 1.6 × with respect to a dense

Original: INT8 baseline and for INT4 precision with 50% sparsity, we achieve a memory capacity savings of1.33 × with respect to a dense INT4 baseline, across all benchmarks.
Revised: INT8 baseline and for INT4 precision with 50% sparsity, we achieve a memory capacity savings of 1.33 × with respect to a dense INT4 baseline, across all benchmarks.

Original: Overall, when compared to adense FP16 baseline, a sparse INT8 transformer and a sparse INT4 transformer achieves 6.4 × and10.67 × savings in memory capacity, respectively.
Revised: Overall, when compared to a dense FP16 baseline, a sparse INT8 transformer and a sparse INT4 transformer achieves 6.4 × and 10.67 × savings in memory capacity, respectively.

Original: In addition to the memory capacity savings, wealso get performance beneﬁts from only performing computations on the non-zero weight values.
Revised: In addition to the memory capacity savings, we also get performance beneﬁts from only performing computations on the non-zero weight values.

Original: The overall performance beneﬁts observed for a benchmark depends on the size of different layersin the transformer block and their execution time fraction.
Revised: The overall performance beneﬁts observed for a benchmark depends on the size of different layers in the transformer block and their execution time fraction.

Original: By pruning the 6 layers inside eachtransformer block shown in Figure 1, we affect their execution times while keeping the executiontime for all other layers unchanged.
Revised: By pruning the 6 layers inside each transformer block shown in Figure 1, we affect their execution times while keeping the execution time for all other layers unchanged.

Original: We observe that the 6 pruned layers account for 57.7%, 52.4%and 51.1% of the overall execution time for BERT-base, Wav2vec2.0, and ViT respectively in theirdense INT8 execution.
Revised: We observe that the 6 pruned layers account for 57.7%, 52.4% and 51.1% of the overall execution time for BERT-base, Wav2vec2.0, and ViT respectively in their dense INT8 execution.

Original: In dense INT4 execution, these 6 layers account for 40.5%, 35.5% and 34.3%of the overall execution time for BERT-base, Wav2vec2.0 and ViT respectively.
Revised: In dense INT4 execution, these 6 layers account for 40.5%, 35.5% and 34.3% of the overall execution time for BERT-base, Wav2vec2.0 and ViT respectively.

Original: Accordingly, weachieve a performance improvement of 1.4 × , 1.35 × and 1.34 × for BERT-base, Wav2vec2.0 and ViTrespectively in INT8 with respect to dense models and a performance improvement 1.25 × , 1.22 × and 1.21 × in INT4.
Revised: Accordingly, we achieve a performance improvement of 1.4 × , 1.35 × and 1.34 × for BERT-base, Wav2vec2.0 and ViT respectively in INT8 with respect to dense models and a performance improvement 1.25 × , 1.22 × and 1.21 × in INT4.

Original: When compared to a dense FP16 baseline, the performance improvements of sparse
Revised: When compared to a dense FP16 baseline, the performance improvements of sparse

Original: INT8 and INT4 further increase by a factor of 2.76 × and 3.78 × on an average across all benchmarks.
Revised: INT8 and INT4 further increase by a factor of 2.76 × and 3.78 × on an average across all benchmarks.

Original:  Experimental Setup and Results
Revised:  Experimental Setup and Results

Original: We evaluate the proposed methods on three representative pre-trained models and correspondingdownstream benchmarks in multiple application domains to demonstrate the effectiveness of ourmethods.
Revised: We evaluate the proposed methods on three representative pre-trained models and corresponding downstream benchmarks in multiple application domains to demonstrate the effectiveness of our methods.

Original: Speciﬁcally, we investigate the BERT-base model on the SQuAD1.1 benchmark, the
Revised: Speciﬁcally, we investigate the BERT-base model on the SQuAD1.1 benchmark, the

Original: Wav2vec2.0 model on the Librispeech dataset and the ViT-base model on the ImageNet1k benchmark.
Revised: Wav2vec2.0 model on the Librispeech dataset and the ViT-base model on the ImageNet1k benchmark.

Original: We implement quantization and pruning on top of Huggingface Transformer [34] for BERT and
Revised: We implement quantization and pruning on top of Huggingface Transformer [36] for BERT and

Original: Wav2vec2.0 models, and Timm packages [35] for the ViT model.
Revised: Wav2vec2.0 models, and Timm packages [37] for the ViT model.

Original: We use the recipe published in theoriginal papers to ﬁne-tune the downstream benchmarks and use the results as the baseline to evaluateour method.
Revised: We use the recipe published in the original papers to ﬁne-tune the downstream benchmarks and use the results as the baseline to evaluate our method.

Original: As shown in Table 4, our baseline matches the published results in the correspondingpaper or git repository.
Revised: As shown in Table 4, our baseline matches the published results in the corresponding paper or git repository.

Original: All experiments are performed on NVIDIA V100 GPUs.
Revised: All experiments are performed on NVIDIA V100 GPUs.

Original: Listed below isbasic information on the models and benchmarks — more details are presented in Appendix-A.
Revised: Listed below is basic information on the models and benchmarks — more details are presented in Appendix-A.

Original: Pre-trained BERT-base model on the SQuAD1.1 : The BERT-base model has 12 transformerblocks with 12 self-attention heads and a hidden dimension of 768.
Revised: Pre-trained BERT-base model on the SQuAD1.1 : The BERT-base model has 12 transformer blocks with 12 self-attention heads and a hidden dimension of 768.

Original: We use the pre-trained BERTmodel provided by Huggingface Transformer for ﬁne-tuning SQuAD 1.1, a standard questionanswering task for NLP.
Revised: We use the pre-trained BERT model provided by Huggingface Transformer for ﬁne-tuning SQuAD 1.1, a standard questionanswering task for NLP.

Original: For ﬁne-tuning, we use batch size of 12 and sequence length of 384.
Revised: For ﬁne-tuning, we use batch size of 12 and sequence length of 384.

Original: We usethe AdamW optimizer with a learning rate of 3e-5 with linear decay.
Revised: We use the AdamW optimizer with a learning rate of 3e-5 with linear decay.

Original: The model is ﬁne-tuned for2-4 epochs with a dropout probability of 0.1-0.2.
Revised: The model is ﬁne-tuned for 2-4 epochs with a dropout probability of 0.1-0.2.

Original: For the sparse INT4 model, scheduled dropout isapplied as introduced in section 2.3.3.
Revised: For the sparse INT4 model, scheduled dropout is applied as introduced in section 2.3.3.

Original: Wav2vec2.0 large model on the Librispeech : The Wav2vec2.0-large model contains 24 transformerblocks with 16 attention heads and a hidden dimension of 1024 [7].
Revised: Wav2vec2.0 large model on the Librispeech : The Wav2vec2.0-large model contains 24 transformer blocks with 16 attention heads and a hidden dimension of 1024 [7].

Original: We use the facebook/wav2vec2large-lv60 pretrained model on HuggingFace Transformer, which is pre-trained on the audio datafrom LibriVox.
Revised: We use the facebook/wav2vec2large-lv60 pretrained model on HuggingFace Transformer, which is pre-trained on the audio data from LibriVox.

Original: The pretrained model is ﬁne-tuned on Librispeech’s 100 hour clean subset usingstandard Connectionist Temporal Classiﬁcation (CTC) loss for the Automatic Speech Recognition(ASR) downstream task.
Revised: The pretrained model is ﬁne-tuned on Librispeech’s 100 hour clean subset using standard Connectionist Temporal Classiﬁcation (CTC) loss for the Automatic Speech Recognition (ASR) downstream task.

Original: The model has a varied token length averaging 600-800.
Revised: The model has a varied token length averaging 600-800.

Original: For ﬁne tuning, weuse the AdamW optimizer with a learning rate of 3e-4.
Revised: For ﬁne tuning, we use the AdamW optimizer with a learning rate of 3e-4.

Original: The learning rate decays linearly after 500warm-up steps.
Revised: The learning rate decays linearly afterwarm-up steps.

Original: We use a batch size of 32 and tune the model for 6 epochs.
Revised: We use a batch size of 32 and tune the model for 6 epochs.

Original: ViT-base model on ImageNet1k :
Revised: ViT-base model on ImageNet1k :

Original: The ViT-base model contains 12 transformer blocks with 12attention heads and a hidden dimension of 768 [4].
Revised: The ViT-base model contains 12 transformer blocks withattention heads and a hidden dimension of 768 [4].

Original: To clearly understand the impact of quantization/pruning on transfer learning, we use the pre-trained model that is only trained on ImageNet21kand then ﬁne-tune it on ImageNet1k for downstream image classiﬁcation.
Revised: To clearly understand the impact of quantization/pruning on transfer learning, we use the pre-trained model that is only trained on ImageNet21k and then ﬁne-tune it on ImageNet1k for downstream image classiﬁcation.

Original: For ﬁne-tuning, we use apatch size of 16x16 and ﬁne-tune resolution of 384x384, following the original settings of [4].
Revised: For ﬁne-tuning, we use a patch size of 16x16 and ﬁne-tune resolution of 384x384, following the original settings of [4].

Original: Theoptimizer is SGD with a learning rate of 0.01.
Revised: The optimizer is SGD with a learning rate of 0.01.

Original: We tune the model for 8 epochs using a cosine learningrate schedule, gradient clipping of 1.0, and batch size of 512.
Revised: We tune the model for 8 epochs using a cosine learning rate schedule, gradient clipping of 1.0, and batch size of 512.

Original: For sparse INT8/4 models, we ﬁnd thatit is beneﬁcial to stimulate training by increasing the learning rate to 0.05 and tune for 16 epochsuntil convergence settles.
Revised: For sparse INT8/4 models, we ﬁnd that it is beneﬁcial to stimulate training by increasing the learning rate to 0.05 and tune for 16 epochs until convergence settles.

Original: Quantization and pruning are applied only on transformer backbones as they contribute > 99%of the total compute operations.
Revised: Quantization and pruning are applied only on transformer backbones as they contribute > 99% of the total compute operations.

Original: We use a common setting for all three models and benchmarks.
Revised: We use a common setting for all three models and benchmarks.

Original:  Speciﬁcally, the SAWB+ quantizer is used for weight quantization.
Revised: [Table] Speciﬁcally, the SAWB+ quantizer is used for weight quantization.

Original: The MinMax quantizer is usedfor the activation quantization for INT8 and sparse INT8 models, and the PACT quantizer is usedfor INT4 and sparse INT4 models.
Revised: The MinMax quantizer is used for the activation quantization for INT8 and sparse INT8 models, and the PACT quantizer is used for INT4 and sparse INT4 models.

Original: Full precision ﬁne-tuned models are used for initialization of
Revised: Full precision ﬁne-tuned models are used for initialization of

Original: INT8, sparse INT8 and INT4 models.
Revised: INT8, sparse INT8 and INT4 models.

Original: For the sparse INT4 model, we use a sparse INT8 model forinitialization as discussed in section 2.3.2.
Revised: For the sparse INT4 model, we use a sparse INT8 model for initialization as discussed in section 2.3.2.

Original: More details of the training methodology are presented in
Revised: More details of the training methodology are presented in

Original: Appendix-B. Table 4 shows the ﬁnal results on the three benchmarks with different precision/sparsitysettings.
Revised: Appendix-B. Table 4 shows the ﬁnal results on the three benchmarks with different precision/sparsity settings.

Original:  As shown, aside from the sparse INT4 BERT-base model, which shows a small degradationof 1 . 6% , all other compressed models achieve iso-accuracy within < 1% of their respective baselines,making them good candidates for efﬁcient deployment.
Revised: The uncertainty analysis of the results is presented in Appendix-B, where table 7 As shown, aside from the sparse INT4 BERT-base model, which shows a small degradation of 1 . 6% , all other compressed models achieve iso-accuracy within < 1% of their respective baselines, making them good candidates for efﬁcient deployment.

Original:  Related Work
Revised:  Related Work

Original: Quantization has been extensively studied to compress CNN and RNN models.
Revised: Quantization has been extensively studied to compress CNN and RNN models.

Original: Over the years, manyapproaches, such as quantization-aware training [31], post-training-quantizaiton [22], knowledgedistillation [36], and quantizers, such as PACT [31], LSQ [37], LSQ+ [38], and SAWB [30], have[Table] been developed to improve the performance of quantized models.
Revised: Over the years, many approaches, such as quantization-aware training [31], post-training-quantizaiton [22], knowledge distillation [38], and quantizers, such as PACT [31], LSQ [39], LSQ+ [40], and SAWB [30], have been developed to improve the performance of quantized models.

Original: For NLP, these techniques havebeen applied to compress transformer models.
Revised: For NLP, these techniques have been applied to compress transformer models.

Original: FullyQT
Revised: FullyQT

Original: [18] and [39] developed an 8-bit transformerfor machine translation.
Revised: [18] and [41] developed an 8-bit transformer for machine translation.

Original: Q8BERT [19] and IBERT [40] quantize BERT and RoBERTa [41] modelsrespectively in 8-bit.
Revised: Q8BERT [19] and IBERT [42] quantize BERT and RoBERTa [43] models respectively in 8-bit.

Original: Later studies focus on lower precision weight quantization.
Revised: Later studies focus on lower precision weight quantization.

Original: QBERT [20]quantizes weights down to 4-bit without signiﬁcant degradation using Hession information.
Revised: QBERT [20] quantizes weights down to 4-bit without signiﬁcant degradation using Hession information.

Original: TernaryBERT, BinaryBERT and BiBERT [42–44] pushed weight precision to even lower than 4-bit, whilemaintaining activations in 8-bit.
Revised: TernaryBERT, BinaryBERT and BiBERT [44–46] pushed weight precision to even lower than 4-bit, while maintaining activations in 8-bit.

Original: Recently, KDLSQ-BERT [45] and MKQ-BERT [46] discoveredthat distillation techniques can be used to effectively quantize activations in the BERT model into4-bit.
Revised: Recently, KDLSQ-BERT [47] and MKQ-BERT [48] discovered that distillation techniques can be used to effectively quantize activations in the BERT model into 4-bit.

Revised: Our methods, however, are more straightforward, easier to implement, and can achieve good performance while avoiding some of the limitations of distillation such as the requirement of storing and computing the teacher models.

Revised: More importantly, our methods and distillation are mutual exclusive and can be combined to further improve the performance of compressed models.

Original: There has been less work in quantizing vision and speech transformers, however, Bie et.
Revised: There has been less work in quantizing vision and speech transformers, however, Bie et.

Original: al [23]quantize a transformer-based encoder-decoder architecture to 8-bit for the end-to-end ASR task, butthe transformer is simpliﬁed with only 6 encoder/decoder layers and quantization is only appliedto weights.
Revised: al [23] quantize a transformer-based encoder-decoder architecture to 8-bit for the end-to-end ASR task, but the transformer is simpliﬁed with only 6 encoder/decoder layers and quantization is only applied to weights.

Original: Liu et al.
Revised: Liu et al.

Original: [22] and FQ-ViT [22] quantize vision transformer models such as ViT [47],
Revised: [22] and FQ-ViT [22] quantize vision transformer models such as ViT [49],

Original: DeiT [36] and DETR [48] and achieve iso-accuracy at 8-bit precision, but noticeable degradation isobserved when the precision is reduced down to 6-bit.
Revised: DeiT [38] and DETR [50] and achieve iso-accuracy at 8-bit precision, but noticeable degradation is observed when the precision is reduced down to 6-bit.

Original: Pruning has been extensively applied to compress transformer models in NLP [49–51], ViT [52–54],and Speech [26].
Revised: Pruning has been extensively applied to compress transformer models in NLP [51–53], ViT [54–56], and Speech [26].

Original: However, most pruning approaches are unstructured, which can be challengingfor hardware-based acceleration.
Revised: However, most pruning approaches are unstructured, which can be challenging for hardware-based acceleration.

Original: Structured pruning in the form of directly removing attentionheads [55, 56] shows promising results, but these methods may not be applicable to transformers inother application domains [53].
Revised: Structured pruning in the form of directly removing attention heads [57, 58] shows promising results, but these methods may not be applicable to transformers in other application domains [55].

Original: To the best of our knowledge, there is no common structural pruningapproach that can be broadly applied.
Revised: To the best of our knowledge, there is no common structural pruning approach that can be broadly applied.

Original: The most relevant work is [27], which explores the combination of quantization and sparsity ontransformer models and is the ﬁrst to propose ﬁne-grained structural pruning.
Revised: The most relevant work is [27], which explores the combination of quantization and sparsity on transformer models and is the ﬁrst to propose ﬁne-grained structural pruning.

Original: However, the workis only veriﬁed on pre-trained BERT models, the lowest precision reached is 8-bit, and the sparsemodel must repeat pre-training to recover model accuracy.
Revised: However, the work is only veriﬁed on pre-trained BERT models, the lowest precision reached is 8-bit, and the sparse model must repeat pre-training to recover model accuracy.

Original: Using our approach, we achieve the sameaccuracy while only impacting the ﬁne-tuning phase.
Revised: Using our approach, we achieve the same accuracy while only impacting the ﬁne-tuning phase.

Original: We also successfully push precision down to4-bit and demonstrate our approach across three representative application domains.
Revised: We also successfully push precision down to 4-bit and demonstrate our approach across three representative application domains.

Original:  Conclusion
Revised:  Conclusion

Original: Pre-trained transformer models have demonstrated state-of-the-art performance in a wide rangeof applications, but tremendous growth in computational requirements and memory footprint render deployment in practical applications a signiﬁcant challenge.
Revised: Pre-trained transformer models have demonstrated state-of-the-art performance in a wide range of applications, but tremendous growth in computational requirements and memory footprint render deployment in practical applications a signiﬁcant challenge.

Original: To address this, we introduce anovel quantization/sparsity-aware ﬁne-tuning methodology to successfully compress pre-trainedtransformer models by carefully choosing suitable quantizers, data format, initialization, and regularization techniques.
Revised: To address this, we introduce a novel quantization/sparsity-aware ﬁne-tuning methodology to successfully compress pre-trained transformer models by carefully choosing suitable quantizers, data format, initialization, and regularization techniques.

Original: Our methods are demonstrated on popular pre-trained transformer models andbenchmarks across three application domains.
Revised: Our methods are demonstrated on popular pre-trained transformer models and benchmarks across three application domains.

Original: This work is critical as it lays the foundation to deploylarge pre-trained transformer models in practical end-user applications including data centers andedge scenarios.
Revised: This work is critical as it lays the foundation to deploy large pre-trained transformer models in practical end-user applications including data centers and edge scenarios.

Original: Our inference solutions can accelerate ML deployment and provide signiﬁcant costand energy savings for AI inference.
Revised: Our inference solutions can accelerate ML deployment and provide signiﬁcant cost and energy savings for AI inference.

Original: Our solutions could still be subject to unexpected instabilities,and going forward, will require task-speciﬁc robustness studies to prepare these models againstadversarial attacks.
Revised: Our solutions could still be subject to unexpected instabilities, and going forward, will require task-speciﬁc robustness studies to prepare these models against adversarial attacks.

Original: More details on the broader impact of our work are addressed in Appendix-C. 
Revised: More details on the broader impact of our work are addressed in Appendix-C. Acknowledgments and Disclosure of Funding

Revised: This work is fully funded by IBM Research.

Revised: The authors would like to thank I-Hsin Chung, MingHung Chen, Paul Crumley and James Norris for their support on computing infrastructure; Xiao

Revised: Sun, Kailash Gopalakrishnan, Yuhong Li, Xiaodong Cui, Andrea Fasoli, Jiamin Ni, Derrick Liu and

Revised: Mauricio Serrano for the technical discussions; Jeffrey Burns and Mukesh Khare for their executive support and feedback on the draft; IBM AI Hardware Center for providing computing resources.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.
Revised: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] or [N/A] .

Original: You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section ?? .
Revised: [Yes] See Section ?? .

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [No] The code and the data are proprietary.
Revised: [No] The code and the data are proprietary.

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes] See Section 2 and
Revised: [Yes] See Section 2 and(b)

Original: (b) Did you describe the limitations of your work?
Revised: Did you describe the limitations of your work?

Original: [Yes] (c) Did you discuss any potential negative societal impacts of your work?[Yes]
Revised: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Original:  (d) Have you read the ethics review guidelines and ensured that your paper conforms to[Table] them?
Revised: [Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]
Revised: [Yes]

Original: [Table]

Original: If you are including theoretical results...[Table] (a) Did you state the full set of assumptions of all theoretical results?[N/A]
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Table] (b) Did you include complete proofs of all theoretical results?[N/A] [Table]
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: If you ran experiments...

Original: [Table]
Revised: [N/A] 3.

Original:  (a) Did you include the code, data, and instructions needed to reproduce the main experi- [Table] mental results (either in the supplemental material or as a URL)?[Yes] Instructions
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: and code examples are in Appendix

Original: [Table]

Original:  (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?[Yes] in Appendix
Revised: [Yes] Instructions and code examples are in Appendix (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Table] (c) Did you report error bars (e.g., with respect to the random seed after running ex- [Table] periments multiple times)?
Revised: [Yes] in Appendix (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [No] but key results are averaged results over random

Original: [Table]

Original: seeds

Original: [Table] (d) Did you include the total amount of compute and the type of resources used (e.g., type[Table] of GPUs, internal cluster, or cloud provider)?
Revised: [No] but key results are averaged results over random seeds (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] 
Revised: [Yes] 4.

Original: [Table] If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...[Table] (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?[N/A]
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original:  (d) Did you discuss whether and how consent was obtained from people whose data you’re[Table] using/curating?
Revised: [N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable[Table] information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [N/A] 
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if[Table] applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A][Table] (b) Did you describe any potential participant risks, with links to Institutional Review 
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [Table] Board (IRB) approvals, if applicable?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A] 
Revised: [N/A] A

Original: [Table] (c) Did you include the estimated hourly wage paid to participants and  the total amountspent on participant compensation?
Revised: In this section, we provide details on the datasets, model architectures, and settings used for full precision ﬁne-tuning to replicate the baseline accuracy reported in prior publications.

Original: [N/A]
Revised: SQuAD1.1 is the Stanford Question Answering dataset [59].

Original: [Table]
Revised: It is a reading comprehension dataset, consisting of a collection of 100k question and answer pairs from reading passages, where the answer to every question is a segment of corresponding passage. The task is to predict the answer text span in a passage. BERT-base model [2] is a transformer model pre-trained on a large corpus of English data, BooksCorpus [60], with 800M words and English Wikipedia of 2500M words. The model was pre-trained in a self-supervised fashion using a masked language modeling (MLM) procedure.

Revised: The BERT-base model consists of an input embedding, 12 transformer blocks, and an output linear layer, with a total of 100M parameters.

Revised: The input embedding is a sum of token embeddings, segmentation embeddings and the position embeddings.

Revised: Each transformer block contains 12 self-attention heads and a hidden size of 768.

Revised: To ﬁne-tune BERT-base on the SQuAD1.1 downstream task, we use a batch size of 12 and a sequence length of 384.

Revised: The experiments are performed on 4 V100 GPUs with a per-gpu batch size of 3.

Revised: For the full precision ﬁne-tuning baseline, we follow the ﬁne-tuning strategy from [2] and use the AdamW optimizer with a learning rate of 3e-5 with linear decay.

Revised: The model is ﬁne-tuned for 2 epochs with a dropout probability of 0.1.

Revised: We obtain a baseline F1 score of 88.69, which closely matches the F score (88.50) published in [2].

Revised: Fig.

Revised: 6a shows the convergence curve of the full precision ﬁne-tuning baseline.

Revised: A.2 Wav2vec2.0/Librispeech

Revised: Librispeech is a corpus of English speech for the automatic speech recognition (ASR) task [61].

Revised: It contains 1000 hours of speech sampled at 16 kHz.

Revised: The training data is split into 3 partitions of 100hr, 360hr and 500hr sets with ’clean’ and ’other’ categories.

Revised: In this work, we use the 100hr-clean data for the downstream task and the clean validation subset for evaluation.

Revised: Wav2vec2.0-large is a speech model pre-trained on the audio data from LibriVox (LV-60k) [62] in a self-supervised manner [7].

Revised: In this work, we use the Wav2vec2.0-large model, which has a large transformer backbone with 24 transformer blocks.

Revised: The hidden dimension, inner dimension, and number of attention heads in each transformer block are 1024, 4096 and 16, respectively.

Revised: Prior to being fed into the transformer backbone, the raw waveform is encoded through multiple 1d-convolution layers followed by layer normalization and GeLU activations.

Revised: The pre-trained model is ﬁne-tuned on Librispeech’s 100 hour clean subset using standard Connectionist Temporal Classiﬁcation (CTC) loss.

Revised: We follow the implementation and settings from

Revised: HuggingFace Transformer [36] for the ﬁne-tuning.

Revised: Speciﬁcally, for the full precision ﬁne-tuning baseline, we use the AdamW optimizer with betas=(0.9,0.999) and a learning rate of 3e-4.

Revised: The learning rate decays linearly after 500 warm-up steps.

Revised: We use 8 V100 GPUs to tune the model for 3 epochs with a total batch size of 32.

Revised: We achieve a baseline Word Error Rate (WER) of 4.20 %, matching the result provided by HuggingFace Transformer (4.2 %).

Revised: Fig.

Revised: 6b shows the convergence curve of the full precision ﬁne-tuning baseline.

Revised: A.3 ViT/ImageNet1k

Revised: ImageNet1k [63] is an image classiﬁcation benchmark which consists of 1000-categories of objects with over 1.2M training and 50K validation images.

Revised: ViT-base model is a BERT-like transformer encoder model taking images as the input for image classiﬁcation tasks.

Revised: The input images are split into ﬁxed-sized patches of 16x16 and linearly embedded.

Revised: The ViT-base model has 12 transformer blocks with 12 attention heads and a hidden dimension of 768 [4].

Revised: In this paper, we use the pre-trained model that is pre-trained on ImageNet21k [63] and then ﬁne-tune it on ImageNet1k for downstream image classiﬁcation.

Revised: We use a resolution of 384x384 for ﬁne-tuning following the original settings of [4].

Revised: The optimizer is SGD with a learning rate of 0.01.

Revised: We tune the model for 8 epochs using a Cosine learning rate schedule, gradient clipping of 1.0, and batch size of 512 on 32 V100 GPUs.

Revised: With these settings, we achieve an accuracy of 84.12, which matches the accuracy (83.97) published in [4].

Revised: Fig.

Revised: 6c shows the convergence curve of full precision ﬁne-tuning baseline.

Revised: B Appendix-B: Quantization- and Sparsity-aware Fine-tuning Settings

Revised: In this section, we provide details on the implementation of quantization and pruning operations, as well as the hyper-parameters used for the ﬁne-tuning of deeply compressed models.

Revised: B.1 Quantization/Pruning Implementation

Revised: We implement quantization and pruning in the PyTorch framework.

Revised: For each linear module or batchmatrix-matrix multiplication (bmm) operation, we insert quantization operations to quantize both the activations and weights.

Revised: Fig.

Revised: 7 shows screenshot examples of quantized modules with inserted quantization operations from the graph of the quantized BERT-base model.

Revised: For linear modules, both input activations and weights are quantized, as shown in Fig.

Revised: 7(a) a query layer in self-attention and

Revised: Fig.

Revised: 7(c) an intermediate-dense layer in the feed-forward network (FFN); for bmm operations, both input activations are quantized as shown in Fig. 7(b).

Revised: Fig.

Revised: 8 presents a toy example showing a deeply compressed linear module, i.e. QLinear, running a forward pass with random input.

Revised: The linear layer is quantized in 4-bit for both weights and activations using SAWB+ and PACT quantizers, respectively.

Revised: Weights are further pruned with 50% sparsity using a ﬁne-grained group of 4 as discussed in section 2.2.

Revised: The printout shows the pruning mask tensor, pruned weight tensor, and quantized weight tensor computed during the forward pass.

Revised: B.2 Fine-tunine Setting

Revised: We use a common setting for all three models and benchmarks.

Revised: Full precision ﬁne-tuned models are used for the initialization of INT8, sparse INT8, and INT4 models.

Revised: For the sparse INT4 model, we use a sparse INT8 model for initialization as explained in section 2.3.

Revised: Fig.

Revised: 9 shows a schematic of the ﬁne-tuning procedures.

Revised: The SAWB+ quantizer is used for weight quantization for all models as discussed in section 2.1.1.

Revised: The coefﬁcients used in SAWB+ are listed in Table 5.

Revised: The MinMax or

Revised: PACT quantizer is used for the activation quantization.

Revised: For the PACT quantizer (discussed in section

Revised: [Table]

Revised: 2.1.2), three hyper-parameters are used to train α and α n parameters, i.e. initiation in percentile, learning rate ( α _ lr ) and L2 decay ( α _ decay ).

Revised: The detailed settings used for three benchmarks are as follows.

Revised: Table 6 lists the settings for BERT-base/SQuAD1.1 benchmark.

Revised: We use the same baseline optimization methods as described in Appendix 1.1, except that the compressed models are ﬁne-tuned for 4 epochs with a larger dropout (0.2) or a scheduled dropout as introduced in section 2.3.3.

Revised: Fig.

Revised: 10 shows the convergence curves of the deep compressed models.

Revised: Table 7 lists the settings for Wav2vec2.0-large/Librispeech benchmark.

Revised: We use the same baseline optimization methods as described in Appendix 1.2, except that the compressed models are ﬁne-tuned for 6 epochs.

Revised: Fig.

Revised: 11 shows the convergence curves of the deeply compressed models.

Revised: Table 8 lists the settings for ViT-base/ImageNet1k benchmark.

Revised: For INT8/4 models without pruning, we use the same baseline optimization methods as described in Appendix 1.3.

Revised: For sparse INT8/ models, we tune the model for 16 epochs with an initial learning rate of 0.05, keeping the rest

Revised: [Table]

Revised: of hyper-parameters the same as the baseline.

Revised: Fig.

Revised: 12 shows the convergence curves of the deep compressed models.

Revised: B.3 Error Bar

Revised: To understand the uncertainty of the results, we run ﬁve experiments with random initial seeds for all three sparse INT4 models.

Original: A Appendix
Revised: The means and standard deviations calculated using the ﬁve runs are shown in the Table 9. The results are consistent with Table 4. C Appendix-C: Broader Impact Dedicated hardware accelerators for DNN inference, including CPUs, GPUs, TPUs and other AI platforms, have powered the deployment of machine learning for real-life applications in both cloud and edge devices.

Revised: Reduced precision innovations (FP16, FP8 and INT8), together with sparsity,

Original: [Table]
Revised: [Table]

Revised: INT

Revised: [Equation]

Revised: INT

Revised: [Equation]

Original: Optionally include extra information (complete proofs, additional experiments and plots) in theappendix.
Revised: INT [Table] [Table] have recently improved the capability of these accelerators by 4-8 × and have dramatically improved energy cost and carbon emissions.

Original: This section will often be part of  the supplemental material.
Revised: Although pre-trained transformers have unlocked the power of transfer learning and are leading to breakthroughs in multiple application domains, the architecture is too complex for many production systems, such as those for edge-computing inference. There are many ongoing efforts to reduce the size of these models while retaining model performance and transferability. Deep compression of transformers, which is presented in this work, aims to push this front aggressively to enable faster and cheaper inference systems for a wide spectrum of deep learning models and domains. We believe that sparse 4-bit inference solutions can accelerate ML deployment and provide signiﬁcant cost and energy savings for corporations and research institutes — in addition to helping reduce the carbon / climate impact of AI inference.

Revised: By improving power efﬁciency by about 4 × over current transformers running in FP16 (and 8 × vs. default FP32 designs), the carbon footprint for predicting with large DNN models can be signiﬁcantly reduced [64].

Revised: The reduction in computational energy and memory footprint could also enable the inference of large transformer models to be carried out on edge devices (mobile platforms, health care devices,

Original: [Table]
Revised: [Table]

Revised: security cameras, consumer drones, etc.).

Revised: This, in turn, could alleviate security and privacy concerns of sending data back to the Cloud for prediction tasks.

Revised: We would also like to emphasize that although we have shown promising results and limited accuracy loss in comparison to FP32 downstream tasks, deeply compressed transformer models using our solutions could still be subject to unexpected instabilities.

Revised: This may necessitate a careful examination of these optimization techniques and numerical formats over a wider range of models and perfected alongside the development of ML model research.

Revised: The risk of using deeply compressed transformer models in real inference applications is most likely higher than full precision dense models and thus requires task-speciﬁc robustness studies to prepare these models against adversarial attacks.

Revised: In addition, More work is also needed to assess the impact of deeply compressed models in fairness and explainability.

Revised: In addition, for each application domain, we chose representative models, datasets, and tasks (SQuAD1.1, Librispeech, and ImageNet1k) that have been used widely to evaluate compression algorithms.

Revised: Going forward, it will be important to study different (simple and complex) downstream tasks across multiple datasets within each domain; this will be the focus of our future work.

Revised: We will study various datasets for each task to assess the generalization of our proposed techniques.