Original: HALP: H ARDWARE -A WARE L ATENCY P RUNING
Revised: HALP: H ARDWARE -A WARE L ATENCY P RUNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Structural pruning can simplify network architecture and improve inference speed.
Revised: Structural pruning can simplify network architecture and improve inference speed.

Original: We propose Hardware-Aware Latency Pruning (HALP) that formulates structural pruning as a global resource allocation optimization problem, aiming at maximizing the accuracy while constraining latency under a predeﬁned budget.
Revised: We propose Hardware-Aware Latency Pruning (HALP) that formulates structural pruning as a global resource allocation optimization problem, aiming at maximizing the accuracy while constraining latency under a predeﬁned budget.

Original: For ﬁlter importance ranking, HALP leverages latency lookup table to track latency reduction potential and global saliency score to gauge accuracy drop.
Revised: For ﬁlter importance ranking, HALP leverages latency lookup table to track latency reduction potential and global saliency score to gauge accuracy drop.

Original: Both metrics can be evaluated very efﬁciently during pruning, allowing us to reformulate global structural pruning under a reward maximization problem given target constraint.
Revised: Both metrics can be evaluated very efﬁciently during pruning, allowing us to reformulate global structural pruning under a reward maximization problem given target constraint.

Original: This makes the problem solvable via our augmented knapsack solver, enabling HALP to surpass prior work in pruning efﬁcacy and accuracy-efﬁciency trade-off.
Revised: This makes the problem solvable via our augmented knapsack solver, enabling HALP to surpass prior work in pruning efﬁcacy and accuracy-efﬁciency trade-off.

Original: We examine HALP on both classiﬁcation and detection tasks, over varying networks, on ImageNet and VOC datasets.
Revised: We examine HALP on both classiﬁcation and detection tasks, over varying networks, on ImageNet and VOC datasets.

Original: In particular, for ResNet-50/- pruning on ImageNet, HALP improves network throughput by 1 .
Revised: In particular, for ResNet-50/- pruning on ImageNet, HALP improves network throughput by 1 .

Original: 60 × / 1 . 90 × with +0 .
Revised: 60 × / 1 . 90 × with +0 .

Original: 3% / − 0 .
Revised: 3% / − 0 .

Original: 2% top-1 accuracy changes, respectively.
Revised: 2% top-1 accuracy changes, respectively.

Original: For SSD pruning on VOC,
Revised: For SSD pruning on VOC,

Original: HALP improves throughput by 1 .
Revised: HALP improves throughput by 1 .

Original: 94 × with only a 0 . 56 mAP drop.
Revised: 94 × with only a 0 . 56 mAP drop.

Original: HALP consistently outperforms prior art, sometimes by large margins.
Revised: HALP consistently outperforms prior art, sometimes by large margins.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Convolutional Neural Networks (CNNs) act as the central tenet behind the rapid development in computer vision tasks such as classiﬁcation, detection, segmentation, image synthesis, among others.
Revised: Convolutional Neural Networks (CNNs) act as the central tenet behind the rapid development in computer vision tasks such as classiﬁcation, detection, segmentation, image synthesis, among others.

Original: As performance boosts, so do model size, computation, and latency.
Revised: As performance boosts, so do model size, computation, and latency.

Original: With millions, sometimes billions of parameters ( e .
Revised: With millions, sometimes billions of parameters ( e .

Original: g .,
Revised: g .,

Original: GPT-3 Brown et al.
Revised: GPT-3 Brown et al.

Original: , modern neural networks face increasing challenges upon ubiquitous deployment, that mostly faces stringent constraints such as energy and latency Dai et al.
Revised: , modern neural networks face increasing challenges upon ubiquitous deployment, that mostly faces stringent constraints such as energy and latency Dai et al.

Original: ; Molchanov et al.
Revised: ; Molchanov et al.

Original: In certain cases like autonomous driving, a breach of real-time constraint not only undermines user experience, but also imposes critical safety concerns.
Revised: In certain cases like autonomous driving, a breach of real-time constraint not only undermines user experience, but also imposes critical safety concerns.

Original: Even for cloud service, speeding up the inference of neural networks directly translates into higher throughput, allowing more clients and users to beneﬁt from the service.
Revised: Even for cloud service, speeding up the inference of neural networks directly translates into higher throughput, allowing more clients and users to beneﬁt from the service.

Original: One effective and efﬁcient method to reduce model complexity is through network pruning.
Revised: One effective and efﬁcient method to reduce model complexity is through network pruning.

Original: The primary goal of pruning is to remove the parameters, along with their computation, that are deemed least important for inference Alvarez & Salzmann (2016); Han et al.
Revised: The primary goal of pruning is to remove the parameters, along with their computation, that are deemed least important for inference Alvarez & Salzmann (2016); Han et al.

Original: ; Liu et al. (2019b);
Revised: ; Liu et al. (2019b);

Original: Molchanov et al.
Revised: Molchanov et al.

Original: Compatible with other compression streams of work such as quantization Cai et al.
Revised: Compatible with other compression streams of work such as quantization Cai et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: (2020b)
Revised: (2020b)

Original: ; Zhu et al.
Revised: ; Zhu et al.

Original: and distillation Hinton et al.
Revised: and distillation Hinton et al.

Original: Mullapudi et al.
Revised: Mullapudi et al.

Original: ; Yin et al.
Revised: ; Yin et al.

Original: (2020), network pruning enables a ﬂexible tuning of model complexity towards varying constraints, while requiring much less design efforts by neural architecture search Tan et al.
Revised: (2020), network pruning enables a ﬂexible tuning of model complexity towards varying constraints, while requiring much less design efforts by neural architecture search Tan et al.

Original: ; Vahdat et al.
Revised: ; Vahdat et al.

Original: ; Wu et al.
Revised: ; Wu et al.

Original: and architecture re-designs Howard et al.
Revised: and architecture re-designs Howard et al.

Original: ; Ma et al.
Revised: ; Ma et al.

Original: ; Tan & Le (2019).
Revised: ; Tan & Le (2019).

Original: Thus, in this work, we study pruning, in particular structured pruning that removes ﬁlters (or neurons) to beneﬁt off-the-shelf platforms, e . g ., GPUs.
Revised: Thus, in this work, we study pruning, in particular structured pruning that removes ﬁlters (or neurons) to beneﬁt off-the-shelf platforms, e . g ., GPUs.

Original: As the pruning literature develops, the pruning criteria also evolves to better reﬂect ﬁnal efﬁciency.
Revised: As the pruning literature develops, the pruning criteria also evolves to better reﬂect ﬁnal efﬁciency.

Original: The early phase of the ﬁeld focuses on maximum parameter removal in seek for minimum representations of the pretrained model.
Revised: The early phase of the ﬁeld focuses on maximum parameter removal in seek for minimum representations of the pretrained model.

Original: This leads to a ﬂourish of approaches that rank neurons effectively to measure their importance Molchanov et al.
Revised: This leads to a ﬂourish of approaches that rank neurons effectively to measure their importance Molchanov et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: (2020a).
Revised: (2020a).

Original: As each neuron/ﬁlter possesses intrinsically different computation, following works explore proxy to enhance redundancy removal, FLOPs being one of the most widely adopted metrics Li et al.
Revised: As each neuron/ﬁlter possesses intrinsically different computation, following works explore proxy to enhance redundancy removal, FLOPs being one of the most widely adopted metrics Li et al.

Original: ; Wu et al.
Revised: ; Wu et al.

Original: ; Yu & Huang (2019) to reﬂect how many multiplication and addition computes needed for the model.
Revised: ; Yu & Huang (2019) to reﬂect how many multiplication and addition computes needed for the model.

Original: Hovewer, for models with very similar FLOPs, their latency can vary signiﬁcantly Tan et al.
Revised: Hovewer, for models with very similar FLOPs, their latency can vary signiﬁcantly Tan et al.

Original: Recently, more and more works start directly working on reducing latency Chen et al.
Revised: Recently, more and more works start directly working on reducing latency Chen et al.

Original: ; Yang et al.
Revised: ; Yang et al.

Original: However, not much was done in the ﬁeld of GPU friendly pruning methods due to nontrivial latency-architecture trade-off.
Revised: However, not much was done in the ﬁeld of GPU friendly pruning methods due to nontrivial latency-architecture trade-off.

Original: For example, as recently observed in Radu et al.
Revised: For example, as recently observed in Radu et al.

Original: (2019), GPU usually imposes staircase-shaped latency patterns for convolutional operators with varying channels, which inevitably occur per varying pruning rate, see the latency surface in Fig.
Revised: (2019), GPU usually imposes staircase-shaped latency patterns for convolutional operators with varying channels, which inevitably occur per varying pruning rate, see the latency surface in Fig.

Original: This imposes a constraint that pruning needs to be done in groups to achieve latency improvement.
Revised: This imposes a constraint that pruning needs to be done in groups to achieve latency improvement.

Original: Moreover, get- the exact look-up table of layers under different pruning conﬁguration will beneﬁt maximizing performance while minimizing latency
Revised: Moreover, get- the exact look-up table of layers under different pruning conﬁguration will beneﬁt maximizing performance while minimizing latency

Original: [Figure]
Revised: [Figure]

Original: different layers in the deep neural network will result in different accuracy-latency trade-off.
Revised: different layers in the deep neural network will result in different accuracy-latency trade-off.

Original: Typically, removing channels from the latter layers has smaller impact on accuracy and smaller impact on latency versus removing channels from the early layers.
Revised: Typically, removing channels from the latter layers has smaller impact on accuracy and smaller impact on latency versus removing channels from the early layers.

Original: We ask the question, if it is better to remove more neurons from latter layer or less from early layer to achieve the same accuracy-latency trade-off.
Revised: We ask the question, if it is better to remove more neurons from latter layer or less from early layer to achieve the same accuracy-latency trade-off.

Original: By nature, the problem is combinatorial and requires the appropriate solution.
Revised: By nature, the problem is combinatorial and requires the appropriate solution.

Original: In this paper, we propose hardware-aware latency pruning (HALP) that formulates pruning as a resource allocation optimization problem to maximize the accuracy while maintaining a latency budget.
Revised: In this paper, we propose hardware-aware latency pruning (HALP) that formulates pruning as a resource allocation optimization problem to maximize the accuracy while maintaining a latency budget.

Original: The overall workﬂow is shown in Fig.
Revised: The overall workﬂow is shown in Fig.

Original: For latency estimate per pruned architecture, we pre-analyze the operator-level latency values by creating a look-up table for every layer of the model on the target hardware.
Revised: For latency estimate per pruned architecture, we pre-analyze the operator-level latency values by creating a look-up table for every layer of the model on the target hardware.

Original: Then we introduce an additional score for each neuron group to reﬂect and encourage latency reduction.
Revised: Then we introduce an additional score for each neuron group to reﬂect and encourage latency reduction.

Original: To this end, we ﬁrst rank the neurons according to their importance estimates, and then dynamically adjust their latency contributions.
Revised: To this end, we ﬁrst rank the neurons according to their importance estimates, and then dynamically adjust their latency contributions.

Original: With neurons re-calibrated towards the hardware-aware latency curve, we now select remaining neurons to maximize the gradient-based importance estimates for accuracy, within the total latency constraint.
Revised: With neurons re-calibrated towards the hardware-aware latency curve, we now select remaining neurons to maximize the gradient-based importance estimates for accuracy, within the total latency constraint.

Original: This makes the entire neuron ranking solvable under the knapsack paradigm.
Revised: This makes the entire neuron ranking solvable under the knapsack paradigm.

Original: To enforce the neuron selection order in a layer to be from the most important to the least, we have enhanced the knapsack solver so that the calculated latency contributions of the remaining neurons would hold.
Revised: To enforce the neuron selection order in a layer to be from the most important to the least, we have enhanced the knapsack solver so that the calculated latency contributions of the remaining neurons would hold.

Original: HALP surpasses prior art in pruning efﬁcacy, see Fig.
Revised: HALP surpasses prior art in pruning efﬁcacy, see Fig.

Original: 2 and the more detailed analysis in Section 4.
Revised: 2 and the more detailed analysis in Section 4.

Original: Our main contributions are summarized as follows:
Revised: Our main contributions are summarized as follows:

Original: • We propose a latency-driven structured pruning algorithm that exploits hardware latency traits to yield direct inference speedups.
Revised: • We propose a latency-driven structured pruning algorithm that exploits hardware latency traits to yield direct inference speedups.

Original: • We orient the pruning process around a quick yet highly effective knapsack scheme that seeks for a combination of remaining neuron groups to maximize importance while constraining to the target latency.
Revised: • We orient the pruning process around a quick yet highly effective knapsack scheme that seeks for a combination of remaining neuron groups to maximize importance while constraining to the target latency.

Original: • We introduce a group size adjustment scheme for knapsack solver amid varying latency contributions across layers, hence allowing full exploitation of the latency landscape of the underlying hardware.
Revised: • We introduce a group size adjustment scheme for knapsack solver amid varying latency contributions across layers, hence allowing full exploitation of the latency landscape of the underlying hardware.

Original: • We compare to prior art when pruning ResNet, MobileNet, VGG architectures on ImageNet, PASCAL VOC and demonstrate that our method yields consistent latency and accuracy improvements over state-of-the-art methods.
Revised: • We compare to prior art when pruning ResNet, MobileNet, VGG architectures on ImageNet, PASCAL VOC and demonstrate that our method yields consistent latency and accuracy improvements over state-of-the-art methods.

Original: Our ImageNet pruning results present a viable 1 .
Revised: Our ImageNet pruning results present a viable 1 .

Original: 6 × to 1 .
Revised: 6 × to 1 .

Original: 9 × speedup while preserving very similar original accuracy of the ResNets.
Revised: 9 × speedup while preserving very similar original accuracy of the ResNets.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Pruning methods .
Revised: Pruning methods .

Original: Depending on when to perform pruning, current methods can generally be divided into three groups Frankle & Carbin (2019b): i)
Revised: Depending on when to perform pruning, current methods can generally be divided into three groups Frankle & Carbin (2019b): i)

Original: prune pretrained models Han et al.
Revised: prune pretrained models Han et al.

Original: Luo et al.
Revised: Luo et al.

Original: ; Li et al.
Revised: ; Li et al.

Original: ; He et al.
Revised: ; He et al.

Original: ; Molchanov et al.
Revised: ; Molchanov et al.

Original: (2016; 2019); Gale et al.
Revised: (2016; 2019); Gale et al.

Original: (2019), ii) prune at initialization Frankle & Carbin (2019a); Lee et al.
Revised: (2019), ii) prune at initialization Frankle & Carbin (2019a); Lee et al.

Original: ; de Jorge et al.
Revised: ; de Jorge et al.

Original: (2020), and iii) prune during training Alvarez & Salzmann (2016); Gao et al.
Revised: (2020), and iii) prune during training Alvarez & Salzmann (2016); Gao et al.

Original: Lym et al.
Revised: Lym et al.

Original: Despite notable progresses in the later two streams, pruning pretrained models remains as the most popular paradigm, with structural sparsity favored by off-the-shelf inference platforms such as GPU.
Revised: Despite notable progresses in the later two streams, pruning pretrained models remains as the most popular paradigm, with structural sparsity favored by off-the-shelf inference platforms such as GPU.

Original: To improve on inference efﬁciency, many previous pruning methods trim down the neural network aiming to achieve a high compression rate while maintaining an acceptable accuracy.
Revised: To improve on inference efﬁciency, many previous pruning methods trim down the neural network aiming to achieve a high compression rate while maintaining an acceptable accuracy.

Original: The estimation of neuron importance has been widely studied in literature Hu et al.
Revised: The estimation of neuron importance has been widely studied in literature Hu et al.

Original: ; Luo et al.
Revised: ; Luo et al.

Original: Molchanov et al. (2016).
Revised: Molchanov et al. (2016).

Original: For example, Molchanov et al.
Revised: For example, Molchanov et al.

Original: (2019) proposes to use Taylor expansion to measure the importance of neurons and prunes the least-ranked ones until a desired number of neurons are pruned.
Revised: (2019) proposes to use Taylor expansion to measure the importance of neurons and prunes the least-ranked ones until a desired number of neurons are pruned.

Original: However, a compression ratio does not directly translate into computation reduction ratio, amid the fact that each neuron/ﬁlter possesses different computation.
Revised: However, a compression ratio does not directly translate into computation reduction ratio, amid the fact that each neuron/ﬁlter possesses different computation.

Original: There are recent methods that focus primarily on reducing FLOPs.
Revised: There are recent methods that focus primarily on reducing FLOPs.

Original: Some of them take FLOPs into consideration when calculating the neuron importance to encourage penalizing neurons that induce high computations Wu et al.
Revised: Some of them take FLOPs into consideration when calculating the neuron importance to encourage penalizing neurons that induce high computations Wu et al.

Original: An alternative line of work propose to select the best pruned network from a set of candidates Li et al.
Revised: An alternative line of work propose to select the best pruned network from a set of candidates Li et al.

Original: ; Yang et al.
Revised: ; Yang et al.

Original: However, it would take a long time for candidate selection due to the large amount of candidates.
Revised: However, it would take a long time for candidate selection due to the large amount of candidates.

Original: In addition, these methods use
Revised: In addition, these methods use

Original: FLOPs as a proxy of latency, which is usually inaccurate as networks with similar FLOPs might have signiﬁcantly different latencies
Revised: FLOPs as a proxy of latency, which is usually inaccurate as networks with similar FLOPs might have signiﬁcantly different latencies

Original: Tan et al.
Revised: Tan et al.

Original: Latency-aware compression.
Revised: Latency-aware compression.

Original: Emerging compression techniques shift attention to directly prune to cut down on latency.
Revised: Emerging compression techniques shift attention to directly prune to cut down on latency.

Original: One popular stream is Neural Architecture Search (NAS) methods Dai et al.
Revised: One popular stream is Neural Architecture Search (NAS) methods Dai et al.

Original: (2019); Dong et al.
Revised: (2019); Dong et al.

Original: Tan et al.
Revised: Tan et al.

Original: ; Wu et al.
Revised: ; Wu et al.

Original: (2019) that adaptively adjusts the architecture of the network for a given latency requirement.
Revised: (2019) that adaptively adjusts the architecture of the network for a given latency requirement.

Original: They incorporate the platform constraints into the optimization process in both the architecture and parameter space to jointly optimize the model size and accuracy.
Revised: They incorporate the platform constraints into the optimization process in both the architecture and parameter space to jointly optimize the model size and accuracy.

Original: Despite remarkable insights, NAS methods remain computationally expensive in general compared to their pruning counterparts.
Revised: Despite remarkable insights, NAS methods remain computationally expensive in general compared to their pruning counterparts.

Original: Latency-oriented pruning has also gained a growing amount of attention.
Revised: Latency-oriented pruning has also gained a growing amount of attention.

Original: Chen et al.
Revised: Chen et al.

Original: (2018) presents a framework for network compression under operational constraints, using Bayesian optimization to iteratively obtain compression hyperparameters that satisﬁes the constraints.
Revised: (2018) presents a framework for network compression under operational constraints, using Bayesian optimization to iteratively obtain compression hyperparameters that satisﬁes the constraints.

Original: Along the same line,
Revised: Along the same line,

Original: NetAdapt Yang et al.
Revised: NetAdapt Yang et al.

Original: (2018) iteratively prunes neurons across layers under the guidance of the empirical latency measurements on the targeting platform.
Revised: (2018) iteratively prunes neurons across layers under the guidance of the empirical latency measurements on the targeting platform.

Original: While these methods push the frontier of latency constrained pruning, the hardware-incurred latency surface in fact offers much more potential under our enhanced pruning policy - as we show later, large rooms for improvements remain un-exploited and realizable.
Revised: While these methods push the frontier of latency constrained pruning, the hardware-incurred latency surface in fact offers much more potential under our enhanced pruning policy - as we show later, large rooms for improvements remain un-exploited and realizable.

Original:  M ETHOD
Revised:  M ETHOD

Original: In this section, we ﬁrst formulate the pruning process as an optimization process, before diving deep into the importance estimation for accuracy and latency.
Revised: In this section, we ﬁrst formulate the pruning process as an optimization process, before diving deep into the importance estimation for accuracy and latency.

Original: Then, we elaborate on how to solve the optimization via knapsack regime, augmented by dynamic grouping of neurons.
Revised: Then, we elaborate on how to solve the optimization via knapsack regime, augmented by dynamic grouping of neurons.

Original: We ﬁnalize the method by combining these key steps under one realm of HALP.
Revised: We ﬁnalize the method by combining these key steps under one realm of HALP.

Original: 3. O BJECTIVE F UNCTION
Revised: 3. O BJECTIVE F UNCTION

Original: Consider a neural network that consists of L layers performing linear operations on their inputs, together with non-linear activation layers and potentially pooling layers.
Revised: Consider a neural network that consists of L layers performing linear operations on their inputs, together with non-linear activation layers and potentially pooling layers.

Original: Suppose there are N l neurons (output channels) in the l th layer and each neuron is encoded by parameters W nl ∈ R C inl × K l × K l , where C in is the number of input channels and K is the kernel size.
Revised: Suppose there are N l neurons (output channels) in the l th layer and each neuron is encoded by parameters W nl ∈ R C inl × K l × K l , where C in is the number of input channels and K is the kernel size.

Original: By putting all the neurons across the network together, we get the neuron parameter set W = {{ W nl } N l n =1 } Ll =1 , where N = (cid:80)
Revised: By putting all the neurons across the network together, we get the neuron parameter set W = {{ W nl } N l n =1 } Ll =1 , where N = (cid:80)

Original: Ll =
Revised: Ll =

Original: 1 N l is the total number of neurons in the network.
Revised: 1 N l is the total number of neurons in the network.

Original: Given a training set D
Revised: Given a training set D

Original: = { ( x i , y i ) }
Revised: = { ( x i , y i ) }

Original: Mi =1 , the problem of network pruning with a given constraint C can be generally formulated as the following optimization problem:
Revised: Mi =1 , the problem of network pruning with a given constraint C can be generally formulated as the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original: where ˆ W ⊂ W is the set of remaining neurons after pruning and L is the loss of the task.
Revised: where ˆ W ⊂ W is the set of remaining neurons after pruning and L is the loss of the task.

Original: f ( · ) encodes the network function, and Φ( · ) maps the network to the constraint C , such as latency,
Revised: f ( · ) encodes the network function, and Φ( · ) maps the network to the constraint C , such as latency,

Original: FLOPs, or memory.
Revised: FLOPs, or memory.

Original: We primarily focus on latency for Φ( · ) in this work while the method easily scales to other constraints.
Revised: We primarily focus on latency for Φ( · ) in this work while the method easily scales to other constraints.

Original: The key to solving the aforementioned problem relies on identifying the portion of the network that satisﬁes the constraint while incurring minimum performance disruption:
Revised: The key to solving the aforementioned problem relies on identifying the portion of the network that satisﬁes the constraint while incurring minimum performance disruption:

Original: [Equation]
Revised: [Equation]

Original: where p l denotes the number of kept neurons at layer l , I l
Revised: where p l denotes the number of kept neurons at layer l , I l

Original: ( p l ) signals the maximum importance to the ﬁnal accuracy with p l neurons, and T l ( p l − 1 , p l ) checks on the associated latency contribution of layer l with p l − 1 input channels and p l output channels.
Revised: ( p l ) signals the maximum importance to the ﬁnal accuracy with p l neurons, and T l ( p l − 1 , p l ) checks on the associated latency contribution of layer l with p l − 1 input channels and p l output channels.

Original: p 0 denotes a ﬁxed input channel number for the ﬁrst convolutional block, e .
Revised: p 0 denotes a ﬁxed input channel number for the ﬁrst convolutional block, e .

Original: g ., 3 for RGB images.
Revised: g ., 3 for RGB images.

Original: We next elaborate on I ( · ) and T ( · ) in detail.
Revised: We next elaborate on I ( · ) and T ( · ) in detail.

Original: Importance score.
Revised: Importance score.

Original: To get the importance score of a layer to ﬁnal accuracy, namely I l ( p l ) in Eq.
Revised: To get the importance score of a layer to ﬁnal accuracy, namely I l ( p l ) in Eq.

Original: 2, we take it as the accumulated score from individual neurons (cid:80) p l j =

Revised: 2, we take it as the accumulated score from individual neurons (cid:80) p l j =

Original: I jl .
Revised: I jl .

Original: We ﬁrst approximate the importance of neurons using the Taylor expansion of the loss change Molchanov et al.
Revised: We ﬁrst approximate the importance of neurons using the Taylor expansion of the loss change Molchanov et al.

Original: Speciﬁcally, we prune on batch normalization layers and the importance of the n -th neuron in the l -th layer is calculated as
Revised: Speciﬁcally, we prune on batch normalization layers and the importance of the n -th neuron in the l -th layer is calculated as

Original: [Equation]
Revised: [Equation]

Original: where g denotes the gradient of the weight, γ nl and β nl are the corresponding weight and bias from the batch normalization layer, respectively.
Revised: where g denotes the gradient of the weight, γ nl and β nl are the corresponding weight and bias from the batch normalization layer, respectively.

Original: Unlike a squared loss in Molchanov et al.
Revised: Unlike a squared loss in Molchanov et al.

Original: (2019), we use absolute difference as we observe slight improvements.
Revised: (2019), we use absolute difference as we observe slight improvements.

Original: In order to maximize the total importance, we keep the most important neurons at a higher priority.
Revised: In order to maximize the total importance, we keep the most important neurons at a higher priority.

Original: To this end, we rank the neurons in the l th layer according to their importance score in a descending order and denote the importance score of the j th -ranked neuron as I jl , thus we have
Revised: To this end, we rank the neurons in the l th layer according to their importance score in a descending order and denote the importance score of the j th -ranked neuron as I jl , thus we have

Original: [Equation]
Revised: [Equation]

Original: Latency contribution.
Revised: Latency contribution.

Original: We empirically obtain the layer latency T l ( p l − 1 , p l ) in Eq. 2 by pre-building a layer-wise look-up table with pre-measured latencies.
Revised: We empirically obtain the layer latency T l ( p l − 1 , p l ) in Eq. 2 by pre-building a layer-wise look-up table with pre-measured latencies.

Original: This layer latency corresponds to the aggregation of the neuron latency contribution of each neuron in the layer, c jl :
Revised: This layer latency corresponds to the aggregation of the neuron latency contribution of each neuron in the layer, c jl :

Original: [Equation]
Revised: [Equation]

Original: The latency contribution of the j -th neuron in the l -th layer can also be computed using the entries in the look up table as:
Revised: The latency contribution of the j -th neuron in the l -th layer can also be computed using the entries in the look up table as:

Original: [Equation]
Revised: [Equation]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure] [Table] [Figure] [Table]
Revised: In practice

Original: [Figure]
Revised: [Figure]

Original: [Table] [Figure] [Table] [Figure]
Revised: we ﬁrst rank globally neurons by importance and then consider their latency contribution

Original: [Table]
Revised: [Figure]

Original: In practice, we ﬁrst rank globally neurons by importance and then consider their latency contribution.

Original: Thus, we can draw the following properties.
Revised: Thus, we can draw the following properties.

Original: If we remove the least important neuron in layer l , then the number of neurons will change from p l to p l − 1 , leading to a latency reduction 
Revised: If we remove the least important neuron in layer l , then the number of neurons will change from p l to p l − 1 , leading to a latency reduction c

Original: [Equation]

Original:  l as this neuron’s latency contribution score.
Revised: p l l as this neuron’s latency contribution score.

Original: We assign the potential latency reduction to neurons in the layer by the importance order.
Revised: We assign the potential latency reduction to neurons in the layer by the importance order.

Original: The most important neuron in that layer would always have a latency contribution c 1 l .
Revised: The most important neuron in that layer would always have a latency contribution c 1 l .

Original: At this stage, ﬁnding the right combination of neurons to keep imposes a combinatorial problem, and in the next section we tackle it via reward maximization considering latency and accuracy traits.
Revised: At this stage, ﬁnding the right combination of neurons to keep imposes a combinatorial problem, and in the next section we tackle it via reward maximization considering latency and accuracy traits.

Original: 3. A UGMENTED K NAPSACK S OLVER
Revised: 3. A UGMENTED K NAPSACK S OLVER

Original: Given both importance and latency estimates, we now aim at solving Eq. 2.
Revised: Given both importance and latency estimates, we now aim at solving Eq. 2.

Original: By plugging back in the layer importance Eq. 4 and layer latency Eq. 5, we come to
Revised: By plugging back in the layer importance Eq. 4 and layer latency Eq. 5, we come to

Original: [Equation]
Revised: [Equation]

Original: This simpliﬁes the overall pruning process into a knapsack problem only with additional preceding constraints.
Revised: This simpliﬁes the overall pruning process into a knapsack problem only with additional preceding constraints.

Original: The preceding enforcement originates from the fact that for a neuron with rank j in the l th layer, the neuron latency contribution only holds when all the neurons with rank r = 1 , . . .
Revised: The preceding enforcement originates from the fact that for a neuron with rank j in the l th layer, the neuron latency contribution only holds when all the neurons with rank r = 1 , . . .

Original: , j −are kept in the l th layer and the rest of the neurons with rank r = j + 1 , j + 2 , · · · , p l are removed.
Revised: , j −are kept in the l th layer and the rest of the neurons with rank r = j + 1 , j + 2 , · · · , p l are removed.

Original: Yet the problem is solvable by specifying each item with a list of preceding items that need to be selected before its inclusion.
Revised: Yet the problem is solvable by specifying each item with a list of preceding items that need to be selected before its inclusion.

Original: We augment the knapsack solver to consider the reordered neurons with descending importance score so that all the preceding neurons will removed before it.
Revised: We augment the knapsack solver to consider the reordered neurons with descending importance score so that all the preceding neurons will removed before it.

Original: A detailed description of the pseudo code of the augmented knapsack solver is provided in Algo.
Revised: A detailed description of the pseudo code of the augmented knapsack solver is provided in Algo.

Original: The augmented solver is required to make sure that the latency cost is correct.
Revised: The augmented solver is required to make sure that the latency cost is correct.

Original: 3. N EURON G ROUPING
Revised: N EURON G ROUPING

Original: Considering each neuron individually results in burdensome computation during pruning.
Revised: Considering each neuron individually results in burdensome computation during pruning.

Original: We next explore grouping neurons so that a number of them can be jointly considered and removed enabling faster pruning Yin et al.
Revised: We next explore grouping neurons so that a number of them can be jointly considered and removed enabling faster pruning Yin et al.

Original: This helps exploit hardware-incurred channel granularity guided by the latency, and speed up knapsack solving of Eq.
Revised: This helps exploit hardware-incurred channel granularity guided by the latency, and speed up knapsack solving of Eq.

Original: We refer to the difference of neurons counts between two latency cliffs of the staircase-patterned latency as the latency step size.
Revised: We refer to the difference of neurons counts between two latency cliffs of the staircase-patterned latency as the latency step size.

Original: In our method, we group s channels in a layer as an entirety, where the value of s is equal to the latency step size.
Revised: In our method, we group s channels in a layer as an entirety, where the value of s is equal to the latency step size.

Original: The neurons are grouped by the order of importance.
Revised: The neurons are grouped by the order of importance.

Original: Then we aggregate the importance score and latency contribution for the grouped entity.
Revised: Then we aggregate the importance score and latency contribution for the grouped entity.

Original: When dealing with skip connections in ResNet and group convolutions in MobielNet, we not only group neurons within a layer, we also group the neurons sharing the same channel index from the connected layers.
Revised: When dealing with skip connections in ResNet and group convolutions in MobielNet, we not only group neurons within a layer, we also group the neurons sharing the same channel index from the connected layers Ding et al. ; Luo & Wu (2020).

Original: Note that the latency step size for different layers might be different.
Revised: Note that the latency step size for different layers might be different.

Original: For crosslayer grouping, we use the largest group size among the layers.
Revised: For cross-layer grouping, we use the largest group size among the layers.

Original: Latency-aware grouping enables additional performance beneﬁts when compared to a heuristic universal grouping, as we will later show in the experiments.
Revised: Latencyaware grouping enables additional performance beneﬁts when compared to a heuristic universal grouping, as we will later show in the experiments.

Original: One noticeable beneﬁt of neuron grouping is the simpliﬁcation of knapsack problem that scales linearly with the number of candidates under consideration (see Line 2 of Alg. 1).
Revised: One noticeable beneﬁt of neuron grouping is the simpliﬁcation of knapsack problem that scales linearly with the number of candidates under consideration (see Line 2 of Alg. 1).

Original: As an example, by grouping neurons together for a ResNet50, the total number of (grouped) neurons can be greatly reduced from 26 , 560 to only 215 , this drastically speedups the solver.
Revised: As an example, by grouping neurons together for a ResNet50, the total number of (grouped) neurons can be greatly reduced from 26 , 560 to only 215 , this drastically speedups the solver.

Original: Such saving can be especially beneﬁcial as network size scales up, a very likely trend that literature foresees.
Revised: Such saving can be especially beneﬁcial as network size scales up, a very likely trend that literature foresees.

Original: 3. F INAL HALP R EGIME
Revised: 3. F INAL HALP R EGIME

Original: With all aforementioned steps, we formulate the ﬁnal HALP as follows.
Revised: With all aforementioned steps, we formulate the ﬁnal HALP as follows.

Original: The pruning process takes a trained network as input and prunes it iteratively to satisfy the requirement of a given latency budget C .
Revised: The pruning process takes a trained network as input and prunes it iteratively to satisfy the requirement of a given latency budget C .

Original: We perform one pruning every r minibatches and repeat it k pruning steps in total.
Revised: We perform one pruning every r minibatches and repeat it k pruning steps in total.

Original: In particular, we set k milestones gradually decreasing the total latency to reach the goal via exponential scheduler de Jorge et al.
Revised: In particular, we set k milestones gradually decreasing the total latency to reach the goal via exponential scheduler de Jorge et al.

Original: (2020), with C 1 > C 2 > · · · > C k , C k = C .
Revised: (2020), with C 1 > C 2 > · · · > C k , C k = C .

Original: The algorithm gradually trims down neurons using steps below:
Revised: The algorithm gradually trims down neurons using steps below:

Original: • Step 1 .
Revised: • Step 1 .

Original: For each minibatch, we get the gradients of the weights and update the weights as during the normal training.
Revised: For each minibatch, we get the gradients of the weights and update the weights as during the normal training.

Original: We also calculate each neuron’s importance score as Eq. 3.
Revised: We also calculate each neuron’s importance score as Eq. 3.

Original: • Step 2 .
Revised: • Step 2 .

Original: Over multiple minibatches we calculate the average importance score for each neuron and rank them accordingly.
Revised: Over multiple minibatches we calculate the average importance score for each neuron and rank them accordingly.

Original: Then we count the number of neurons remaining in each layer and dynamically adjust the latency contribution as in Eq.
Revised: Then we count the number of neurons remaining in each layer and dynamically adjust the latency contribution as in Eq.

Original: • Step 3 .
Revised: • Step 3 .

Original: We group neurons as described in Sec.
Revised: We group neurons as described in Sec.

Original: 3.3 and calculate group’s importance and latency reduction.
Revised: 3.3 and calculate group’s importance and latency reduction.

Original: Then we get the nearest preceding group for each layer.
Revised: Then we get the nearest preceding group for each layer.

Original: • Step 4 .
Revised: • Step 4 .

Original: We execute the Algo.
Revised: We execute the Algo.

Original: 1 to select the neurons being remained with current latency milestone.
Revised: 1 to select the neurons being remained with current latency milestone.

Original: Repeat starting from the Step 1 until k milestones are reached
Revised: Repeat starting from the Step 1 until k milestones are reached.

Original: Once pruning ﬁnishes we ﬁne-tune the network to recover accuracy.
Revised: Once pruning ﬁnishes we ﬁne-tune the network to recover accuracy.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We demonstrate the efﬁcacy and feasibility of the proposed HALP method in this section.
Revised: We demonstrate the efﬁcacy and feasibility of the proposed HALP method in this section.

Original: We use
Revised: We use

Original: ImageNet ILSVRC2021 Russakovsky et al.
Revised: ImageNet ILSVRC2021 Russakovsky et al.

Original: (2015) for classiﬁcation.
Revised: (2015) for classiﬁcation.

Original: We ﬁrst study the impact of grouping size s on the pruned top-1 accuracy and the inference time to show the effectiveness of our grouping scheme.
Revised: We ﬁrst study the impact of grouping size s on the pruned top-1 accuracy and the inference time to show the effectiveness of our grouping scheme.

Original: We then study four architectures (ResNet50, ResNet101, MobileNetV1 and VGG16) on different GPUs and compare our pruning results with the state-of-the-art methods on classiﬁcation task.
Revised: We then study four architectures (ResNet50, ResNet101, VGG16 and MobileNetV1/V2) on different GPUs and compare our pruning results with the state-of-the-art methods on classiﬁcation task.

Original: Finally, we further show the generalization ability of our algorithm by testing with object detection task.
Revised: Finally, we further show the generalization ability of our algorithm by testing with object detection task.

Original: We introduce the details of experimental setting in appendix Sec.
Revised: We introduce the details of experimental setting in appendix Sec.

Original: A.
Revised: A.

Original: Unless otherwise stated, we target a NVIDIA TITAN V GPU for main experiments and measure latency at batch size 256.
Revised: Unless otherwise stated, we target a NVIDIA TITAN V GPU for main experiments and measure latency at batch size 256 for ResNets and VGG, and 512 for MobileNets.

Original: We include experimental results for smaller batch size in the appendix, which also show the efﬁcacy of our algorithm.
Revised: We include experimental results for smaller batch size in the appendix, which also show the efﬁcacy of our algorithm.

Original: 4. R ESULTS ON I MAGE N ET
Revised: 4. R ESULTS ON I MAGE N ET

Original: To demonstrate the effectiveness of the proposed HALP method, we compare HALP with several state-of-the-art methods on the large scale dataset ImageNet.
Revised: To demonstrate the effectiveness of the proposed HALP method, we compare HALP with several state-of-the-art methods on the large scale dataset ImageNet.

Original: ResNets.
Revised: ResNets.

Original: We start by pruning ResNet50 and ResNet101 and compare our results with state-of-theart methods in Tab. 1 on TITAN V.
Revised: We start by pruning ResNet50 and ResNet101 and compare our results with state-of-theart methods in Tab. 1 on TITAN V.

Original: In order to have a fair comparison of the latency, for all the other methods, we recreate pruned networks according to the pruned structures they published and measure the latency.
Revised: In order to have a fair comparison of the latency, for all the other methods, we recreate pruned networks according to the pruned structures they published and measure the latency.

Original: Those methods showing ‘-’ in the table do not have pruned structures published so we are unable to measure the latency.
Revised: Those methods showing ‘-’ in the table do not have pruned structures published so we are unable to measure the latency.

Original: For our method, by setting the percentage of latency to remain after pruning to be X , we get the ﬁnal pruned model and refer to it as HALP-
Revised: For our method, by setting the percentage of latency to remain after pruning to be X , we get the ﬁnal pruned model and refer to it as HALP-

Original: X % .
Revised: X % .

Original: We report
Revised: We report

Original: FPS (frames per second) in the table and calculate the speedup of a pruned network as the ratio of FPS 
Revised: FPS (frames per second) in the table and calculate the speedup of a pruned network as the ratio of FPS between pruned and unpruned models.

Original: [Table]
Revised: [Table]

Original:  From the results comparison we can ﬁnd that for pruned networks with similar FLOPs using different methods, our method achieves the highest accuracy and also the fastest inference speed.
Revised: [Figure] From the results comparison we can ﬁnd that for pruned networks with similar FLOPs using different methods, our method achieves the highest accuracy and also the fastest inference speed.

Original: This also shows that FLOPs do not correspond 1 : 1 to the latency.
Revised: This also shows that FLOPs do not correspond 1 : 1 to the latency.

Original: Among these methods for ResNet comparison, EagleEye Li et al.
Revised: Among these methods for ResNet comparison, EagleEye Li et al.

Original: (2020) yields the closest accuracy to ours, but the speedup is lower than ours.
Revised: (2020) yields the closest accuracy to ours, but the speedup is lower than ours.

Original: In Table.
Revised: In Table.

Original: 1, for the pruned ResNet50 network with 3G FLOPs remaining, our method achieves a . 4% higher top1 accuracy and slightly ( . 04 × ) faster inference.
Revised: 1, for the pruned ResNet50 network with 3G FLOPs remaining, our method achieves a . 4% higher top1 accuracy and slightly ( . 04 × ) faster inference.

Original: It is expected that the advantage of our method for accelerating the inference is more obvious when it comes to a more compact pruned network, which is 14% (or . 20 × ) additionally faster for a 2G-FLOPs network while increasing accuracy by .
Revised: It is expected that the advantage of our method for accelerating the inference is more obvious when it comes to a more compact pruned network, which is 14% (or . 20 × ) additionally faster for a 2G-FLOPs network while increasing accuracy by .

Original: When we further prune the network to 1G, we get a pruned network that has more FLOPs but still faster inference speed ( . 17 × ) compared to EagleEye-1G, while we also get .
Revised: When we further prune the network to 1G, we get a pruned network that has more FLOPs but still faster inference speed ( . 17 × ) compared to EagleEye-1G, while we also get .

Original: 3% higher accuracy.
Revised: 3% higher accuracy.

Original: We analyze the pruned network structure in detail in the supplementary material.
Revised: We analyze the pruned network structure in detail in the supplementary material.

Original: We plot the results comparison in Fig.
Revised: We plot the results comparison in Fig.

Original: 2, where we also add the results of training our pruned network with a teacher model RegNetY-16GF (top1 82 .
Revised: 2, where we also add the results of training our pruned network with a teacher model RegNetY-16GF (top1 82 .

Original: Radosavovic et al.
Revised: Radosavovic et al.

Original: With knowledge distillation, our model is 2 .
Revised: With knowledge distillation, our model is 2 .

Original: 70 × faster than the original model at 1% accuracy drop.
Revised: 70 × faster than the original model at 1% accuracy drop.

Original: [Figure]
Revised: [Figure]

Original: Scalability to other networks.
Revised: Scalability to other networks.

Original: We next experiment with other models including bothMobileNetV1 Howard et al.
Revised: We next experiment with other models including both

Original: (2017) and VGG Simonyan & Zisserman (2015).
Revised: VGG Simonyan & Zisserman (2015), MobileNetV1 Howard et al.

Original:  Same as pruning on ResNets, we perform pruning with many different latency constraints and compare with prior art in Tab. 1.
Revised: and MobileNetV2 Sandler et al. Same as pruning on ResNets, we perform pruning with many different latency constraints and compare with prior art in Tab. 1.

Original: As shown, among these methods, the proposed HALP performs significantly better with higher top-1 accuracy and larger inference speedup.
Revised: As shown, among these methods, the proposed HALP performs significantly better with higher top-1 accuracy and larger inference speedup.

Original: Scalability to other hardware.
Revised: Scalability to other hardware.

Original: Our approach is not limited to a single platform.
Revised: Our approach is not limited to a single platform.

Original: We can compute the latency look-up table to apply HALP to a different platform.
Revised: We can compute the latency look-up table to apply HALP to a different platform.

Original: We repeat the ResNet50 experiments as described earlier on a Jetson TX2 and compare our results with other methods.
Revised: We repeat the ResNet50 experiments as described earlier on a Jetson TX2 and compare our results with other methods.

Original: The latency is measured with a batch size 32 .
Revised: The latency is measured with a batch size 32 .

Original: As shown in Fig.3, in a Jetson TX2, our approach presents faster inference time while maintaining similar FLOPs and better accuracy.
Revised: As shown in Fig.3, in a Jetson TX2, our approach presents faster inference time while maintaining similar FLOPs and better accuracy.

Original: 4. HALP A CCELERATION ON GPU S WITH T ENSOR
Revised: 4. HALP A CCELERATION ON GPU S WITH T ENSOR

Original: [Table] [Table]
Revised: [Table] [Table]

Original: To make it closer to the real application in production
Revised: To make it closer to the real application in production

Original: [Table]
Revised: [Table]

Original: we also export the models into onnx format and test the inference speed with TensorRT.
Revised: we also export the models into onnx format and test the inference speed with TensorRT.

Original: We run the inference of the model with
Revised: We run the inference of the model with

Original: FP32, FP16 and also INT8.
Revised: FP32, FP16 and also INT8.

Original: For INT8, we quantize the model using entropy calibration with 2560 randomly selected ImageNet training images.
Revised: For INT8, we quantize the model using entropy calibration with 2560 randomly selected ImageNet training images.

Original: Since the INT8 TensorCore speedup is not supported in TITAN V GPU, we only report the quantized results on RTX3080 GPU.
Revised: Since the INT8 TensorCore speedup is not supported in TITAN V GPU, we only report the quantized results on RTX3080 GPU.

Original: The accelerations and the corresponding top1 accuracy drop (compared to PyTorch baseline model) are listed in Tab. 2.
Revised: The accelerations and the corresponding top1 accuracy drop (compared to PyTorch baseline model) are listed in Tab. 2.

Original: We can further build a TensorRT
Revised: We can further build a TensorRT

Original: INT8 lookup table to achieve potentially larger speedup, which remains as our future work.
Revised: INT8 lookup table to achieve potentially larger speedup, which remains as our future work.

Original:  D ESIGN E FFORT FOR P RUNING
Revised: 4. D ESIGN E FFORT FOR P RUNING

Original: In addition to noticeable performance boosts, HALP in fact requires less design effort compared to prior art, as summarized in Tab. 3.
Revised: In addition to noticeable performance boosts, HALP in fact requires less design effort compared to prior art, as summarized in Tab. 3 (details in Appendix O).

Original: NetAdapt Yang et al. 
Revised: NetAdapt Yang et al. and

Original: and AutoSlim Yu & Huang (2019) generate many proposals during iterative pruning.
Revised: AutoSlim Yu & Huang (2019) generate many proposals during iterative pruning.

Original: Then evaluations of the proposals are needed to select the best candidate that will proceed to the next pruning iteration.
Revised: Then evaluations of the proposals are needed to select the best candidate that will proceed to the next pruning iteration.

Original: EagleEye Li et al.
Revised: EagleEye Li et al.

Original: (2020) preobtains 1000 candidates before pruning and evaluates all of them in order to get the best one.
Revised: pre-obtains 1000 candidates before pruning and evaluates all of them in order to get the best one.

Original: Such pruning candidate selection is intuitive but causes a lot of additional time costs.
Revised: Such pruning candidate selection is intuitive but causes a lot of additional time costs.

Original: The computation cost for MetaPruning Liu et al.
Revised: The computation cost for MetaPruning Liu et al.

Original: (2019a)
Revised: (2019a)

Original: and
Revised: and

Original: AMC He et al.
Revised: AMC He et al.

Original: (2018) can be even higher because they need to train auxiliary network to generate the pruned structure.
Revised: (2018) can be even higher because they need to train auxiliary network to generate the pruned structure.

Original: [Table] 
Revised: [Table] [Table]

Original: Compared to these methods, our method does not require auxiliary network training nor subnetwork evaluation.
Revised: Compared to these methods, our method does not require auxiliary network training nor subnetwork evaluation.

Original: The latency contribution in our method can be quickly obtained during pruning by the pre-generated latency lookup table.
Revised: The latency contribution in our method can be quickly obtained during pruning by the pre-generated latency lookup table.

Original: Although creating the table for the target platform might cost time, we only do it once for all pruning ratios.
Revised: Although creating the table for the target platform might cost time, we only do it once for all pruning ratios.

Original: Solving the augmented knapsack problem brings extra computation, however, after neuron grouping, it only takes around additional 30 minutes of CPU time in total for
Revised: Solving the augmented knapsack problem brings extra computation, however, after neuron grouping, it only takes around additional 30 minutes of CPU time in total for

Original: ResNet50 pruning and less than 1 minute for
Revised: ResNet50 pruning and less than 1 minute for

Original: MobileNetV1, which is negligible compared to the ﬁne-tuning process or training additional models.
Revised: MobileNetV1, which is negligible compared to the ﬁne-tuning process or training additional models.

Original: Moreover, this is signiﬁcantly lower than other methods, for example the fastest of them EagleEye Li et al.
Revised: Moreover, this is signiﬁcantly lower than other methods, for example the fastest of them EagleEye Li et al.

Original: (2020) requires 30 GPU hours.
Revised: (2020) requires 30 GPU hours.

Original: 4. E FFICACY OF N EURON G ROUPING
Revised: 4. E FFICACY OF N EURON G ROUPING

Original: We ﬁrst show the beneﬁts of latency-aware neuron grouping and compare the performance under different group size settings.
Revised: We ﬁrst show the beneﬁts of latency-aware neuron grouping and compare the performance under different group size settings.

Original: Performance comparison.
Revised: Performance comparison.

Original: As described in Sec. 3.3, we group s neurons in a layer as an entirety so that they are removed or kept together.
Revised: As described in Sec. 3.3, we group s neurons in a layer as an entirety so that they are removed or kept together.

Original: Choosing different group sizes can lead to different performances, and also different computation cost on the augmented knapsack problem solving.
Revised: Choosing different group sizes can lead to different performances, and also different computation cost on the augmented knapsack problem solving.

Original: In our method, we set an individual group size for each layer according to each layer’s latency step size in the look-up table.
Revised: In our method, we set an individual group size for each layer according to each layer’s latency step size in the look-up table.

Original: We name the grouping in our method as latency-aware grouping (LG).
Revised: We name the grouping in our method as latency-aware grouping (LG).

Original: For instance, for a ResNet50, using this approach we set the individual group size of 23 layers to 32, of 20 layers to 64, and 10 layers to 128.
Revised: For instance, for a ResNet50, using this approach we set the individual group size of 23 layers to 32, of 20 layers to 64, and 10 layers to 128.

Original: Layers closer to the input tend to use a smaller group size.
Revised: Layers closer to the input tend to use a smaller group size.

Original: Another option for neuron grouping is to heuristically set a ﬁxed group size for all layers as literature does Yin et
Revised: Another option for neuron grouping is to heuristically set a ﬁxed group size for all layers as literature does Yin et

Original: al.
Revised: al.

Original: Fig.
Revised: Fig.

Original: 4 shows the performance of our grouping approach compared to various ﬁxed group sizes for a ResNet50 pruned with different constraints.
Revised: 4 shows the performance of our grouping approach compared to various ﬁxed group sizes for a ResNet50 pruned with different constraints.

Original: As shown, using small group sizes yields the worst performance independently of the latency constraint.
Revised: As shown, using small group sizes yields the worst performance independently of the latency constraint.

Original: At the same time, a very large group such as
Revised: At the same time, a very large group such as

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 256 do also harms the ﬁnal performance.
Revised: 256 do also harms the ﬁnal performance.

Original: Intuitively, a large group size averages the contribution of many neurons and therefore is not discriminative enough to select the most important ones.
Revised: Intuitively, a large group size averages the contribution of many neurons and therefore is not discriminative enough to select the most important ones.

Original: Besides, large groups might promote pruning the entire layer in a single pruning iteration, leading to performance degradation.
Revised: Besides, large groups might promote pruning the entire layer in a single pruning iteration, leading to performance degradation.

Original: On the other hand, small group sizes such as 2 promote removing unimportant groups of neurons.
Revised: On the other hand, small group sizes such as 2 promote removing unimportant groups of neurons.

Original: These groups do not signiﬁcantly improve the latency, but they can contribute to the ﬁnal performance.
Revised: These groups do not signiﬁcantly improve the latency, but they can contribute to the ﬁnal performance.

Original: In contrast, our latency-aware grouping performs the best, showing the efﬁcacy of our neuron grouping scheme.
Revised: In contrast, our latency-aware grouping performs the best, showing the efﬁcacy of our neuron grouping scheme.

Original: Algorithm efﬁciency improvement.
Revised: Algorithm efﬁciency improvement.

Original: Setting the group size according to the latency step size not only improves the performance, but also reduces computation cost on knapsack problem solving for neuron selection since it reduces the total number of object N to a smaller value.
Revised: Setting the group size according to the latency step size not only improves the performance, but also reduces computation cost on knapsack problem solving for neuron selection since it reduces the total number of object N to a smaller value.

Original: In our ResNet50 experiment, except for the ﬁrst convolution layer, the group size of other layers varies from 32 to 128 .
Revised: In our ResNet50 experiment, except for the ﬁrst convolution layer, the group size of other layers varies from 32 to 128 .

Original: By neuron grouping, the value of N can be reduced to 215 , which takes around one minute on average at each pruning step to solve the knapsack problem on
Revised: By neuron grouping, the value of N can be reduced to 215 , which takes around one minute on average at each pruning step to solve the knapsack problem on

Original: CPU.
Revised: CPU.

Original: We have 30 pruning steps in total in our experiments, thus the time spent on neuron selection is around 30 minutes in total, which can be negligible compared to training time.
Revised: We have 30 pruning steps in total in our experiments, thus the time spent on neuron selection is around 30 minutes in total, which can be negligible compared to training time.

Original: 4. G ENERALIZATION TO O BJECT D ETECTION .
Revised: 4. G ENERALIZATION TO O BJECT D ETECTION .

Original: [Figure]
Revised: [Figure]

Original: To show the generalization ability of our proposed HALP algorithm, we also apply the algorithm to the object detection task.
Revised: To show the generalization ability of our proposed HALP algorithm, we also apply the algorithm to the object detection task.

Original: In this experiment we take the popular architecture Single Shot Detector (SSD)
Revised: In this experiment we take the popular architecture Single Shot Detector (SSD)

Original: Liu et al.
Revised: Liu et al.

Original: (2016) on the PASCAL VOC dataset Everingham et al.
Revised: (2016) on the PASCAL VOC dataset Everingham et al.

Original: Following the “07+12” setting in Liu et al.
Revised: Following the “07+12” setting in Liu et al.

Original: (2016), we use the union of VOC and VOC2012 trainval as our training set and use the VOC2007 test as test set.
Revised: (2016), we use the union of VOC and VOC2012 trainval as our training set and use the VOC2007 test as test set.

Original: We pretrain a
Revised: We pretrain a

Original: SSD512 detector with ResNet50 as the backbone.
Revised: SSD512 detector with ResNet50 as the backbone.

Original: The details of the SSD structure are elaborated in the appendix.
Revised: The details of the SSD structure are elaborated in the appendix.

Original: Same to classiﬁcation task, we prune the trained detector and ﬁnetune afterwards.
Revised: Same to classiﬁcation task, we prune the trained detector and ﬁnetune afterwards.

Original: We only prune the backbone network in the detector.
Revised: We only prune the backbone network in the detector.

Original: The results in Fig.
Revised: The results in Fig.

Original: 5 show that the pruned detectors maintain the similar ﬁnal mAP but reduce the
Revised: 5 show that the pruned detectors maintain the similar ﬁnal mAP but reduce the

Original: FLOPs and improve the inference speed greatly, with 77% FLOPs reduction and around 1 .
Revised: FLOPs and improve the inference speed greatly, with 77% FLOPs reduction and around 1 .

Original: 94 × speedup at the cost of only 0 .
Revised: 94 × speedup at the cost of only 0 .

Original: 56% mAP drop.
Revised: 56% mAP drop.

Original: We compare the pruned detector to some other commonly-used detectors in the table.
Revised: We compare the pruned detector to some other commonly-used detectors in the table.

Original: The results show that pruning a detector using HALP improves performance in almost all aspects.
Revised: The results show that pruning a detector using HALP improves performance in almost all aspects.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We proposed hardware-aware latency pruning (HALP) that focuses on structured pruning for underlying hardware towards latency budgets.
Revised: We proposed hardware-aware latency pruning (HALP) that focuses on structured pruning for underlying hardware towards latency budgets.

Original: We formulated pruning as a resource allocation optimization problem to achieve maximum accuracy within a given latency budget.
Revised: We formulated pruning as a resource allocation optimization problem to achieve maximum accuracy within a given latency budget.

Original: We further proposed a latency-aware neuron grouping scheme to further improve latency reduction.
Revised: We further proposed a latency-aware neuron grouping scheme to further improve latency reduction.

Original: Over multiple neural network architectures, classiﬁcation and detection tasks, and changing datasets, we have shown the efﬁciency and efﬁcacy of HALP by showing consistent improvements over state-of-the-art methods.
Revised: Over multiple neural network architectures, classiﬁcation and detection tasks, and changing datasets, we have shown the efﬁciency and efﬁcacy of HALP by showing consistent improvements over state-of-the-art methods.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: For reproducibility purpose, we have included all the experimental setup, details, and all hypermarameters in the paper and the Appendix.
Revised: For reproducibility purpose, we have included all the experimental setup, details, and all hypermarameters in the paper and the Appendix.

Original: The code will be released upon acceptance.
Revised: The code will be released upon acceptance

Original: A E XPERIMENTAL S ETTINGS
Revised: A E XPERIMENTAL S ETTINGS

Original: For image classiﬁcation, we focus on pruning networks on the large-scale ImageNet ILSVRC dataset Russakovsky et al.
Revised: For image classiﬁcation, we focus on pruning networks on the large-scale ImageNet ILSVRC dataset Russakovsky et al.

Original: (2015) ( 1 . 3 M images, 1000 classes).
Revised: (2015) ( 1 . 3 M images, 1000 classes).

Original: Each pruning process consumes a single node with eight NVIDIA Tesla V100 GPUs.
Revised: Each pruning process consumes a single node with eight NVIDIA Tesla V100 GPUs.

Original: We use PyTorch Paszke et al.
Revised: We use PyTorch Paszke et al.

Original: (2017) V1.4.
Revised: (2017) V1.4.

Original: model zoo for pretrained weights for our pruning for a fair comparison with literature.
Revised: model zoo for pretrained weights for our pruning for a fair comparison with literature.

Original: In our experiments we perform iterative pruning.
Revised: In our experiments we perform iterative pruning.

Original: Speciﬁcally, we prune every 320 minibatches after loading the pretrained model with k = 30 pruning steps in total to satisfy the constraint.
Revised: Speciﬁcally, we prune every 320 minibatches after loading the pretrained model with k = 30 pruning steps in total to satisfy the constraint.

Original: We ﬁnetune the network for 90 epochs in total with an individual batch size at 128 for each GPU.
Revised: We ﬁnetune the network for 90 epochs in total with an individual batch size at 128 for each GPU.

Original: For ﬁnetuning, we follow NVIDIA’s recipe Nvidia (2020) with mixed precision and Distributed Data
Revised: For ﬁnetuning, we follow NVIDIA’s recipe Nvidia (2020) with mixed precision and Distributed Data

Original: Parallel training.
Revised: Parallel training.

Original: The learning rate is warmed up linearly in the ﬁrst 8 epochs and reaches the highest learning rate, then follows a cosine decay over the remaining epochs Loshchilov & Hutter (2017).
Revised: The learning rate is warmed up linearly in the ﬁrst 8 epochs and reaches the highest learning rate, then follows a cosine decay over the remaining epochs Loshchilov & Hutter (2017).

Original: For latency lookup table construction, we target a NVIDIA TITAN V GPU with batch size 256 for latency measurement to allow for highest throughput for inference, and target a Jetson TX with inference batch size 32 .
Revised: For latency lookup table construction, we target a NVIDIA TITAN V GPU with batch size 256 for latency measurement to allow for highest throughput for inference, and target a Jetson TX with inference batch size 32 .

Original: We pre-generate a layer latency look-up table on the platform by iteratively reducing of the number of neurons in a layer to characterize the latency with NVIDIA cuDNN Chetlur et al.
Revised: We pre-generate a layer latency look-up table on the platform by iteratively reducing of the number of neurons in a layer to characterize the latency with NVIDIA cuDNN Chetlur et al.

Original: V7.6.5.
Revised: V7.6.5.

Original: We proﬁle each latency measurement 100 times and take the average to avoid randomness.
Revised: We proﬁle each latency measurement 100 times and take the average to avoid randomness.

Original: B E FFICACY OF N EURON G ROUPING ON M OBILE N ET
Revised: B E FFICACY OF N EURON G ROUPING ON M OBILE N ET

Original: In this section, we show the beneﬁts of latency-aware neuron grouping and the performance under different group size settings on MobileNetV1.
Revised: In this section, we show the beneﬁts of latency-aware neuron grouping and the performance under different group size settings on MobileNetV1.

Original: Since MobileNet has group convolutional layers to speedup the inference, we take the group convolutional layer with its preceding connected convolutional layer together as coupled cross-layers Gao et al.
Revised: Since MobileNet has group convolutional layers to speedup the inference, we take the group convolutional layer with its preceding connected convolutional layer together as coupled cross-layers Gao et al.

Original: (2019) to make sure the input channel number and output channel number of the group convolution remain the same.
Revised: (2019) to make sure the input channel number and output channel number of the group convolution remain the same.

Original: All the 27 convolutional layers can be divided into 14 coupled layers.
Revised: All the 27 convolutional layers can be divided into 14 coupled layers.

Original: In our method, with the neuron grouping, we set the individual group size of 1 coupled layer to 16 , of 3 coupled layers to 32 and 10 coupled layers to 64 .
Revised: In our method, with the neuron grouping, we set the individual group size of 1 coupled layer to 16 , of 3 coupled layers to 32 and 10 coupled layers to 64 .

Original: Also, for MobileNetV1 pruning, we add the additional constraint that each layer has at least one group of neurons remaining to make sure that the pruned network is trainable.
Revised: Also, for MobileNetV1 pruning, we add the additional constraint that each layer has at least one group of neurons remaining to make sure that the pruned network is trainable.

Original: We compare our latency-aware neuron grouping with an heuristic option by setting a ﬁxed group size for all layers.
Revised: We compare our latency-aware neuron grouping with an heuristic option by setting a ﬁxed group size for all layers.

Original: Fig.
Revised: Fig.

Original: 6 shows the comparison results between our neuron grouping method and various ﬁxed group sizes for a MobielNet pruned with different latency constraints on ImageNet.
Revised: 6 shows the comparison results between our neuron grouping method and various ﬁxed group sizes for a MobielNet pruned with different latency constraints on ImageNet.

Original: As shown, similar to ResNet50, using small group sizes such as 8 , 16 leads to worse performance; a large group size like 128 also harms the performance signiﬁcantly.
Revised: As shown, similar to ResNet50, using small group sizes such as 8 , 16 leads to worse performance; a large group size like 128 also harms the performance signiﬁcantly.

Original: Our observations on ResNet pruning also hold in MobileNetV1 setting, further emphasizing the efﬁcacy of our latency-aware neuron grouping.
Revised: Our observations on ResNet pruning also hold in MobileNetV1 setting, further emphasizing the efﬁcacy of our latency-aware neuron grouping.

Original: C C OMPARISON WITH E AGLE E YE ON I MAGE N ET
Revised: C C OMPARISON WITH E AGLE E YE ON I MAGE N ET

Original: We now use the same unpruned baseline model provided by EagleEye Li et al.
Revised: We now use the same unpruned baseline model provided by EagleEye Li et al.

Original: (2020) to compare our proposed HALP method with EagleEye Li et al.
Revised: (2020) to compare our proposed HALP method with EagleEye Li et al.

Original: (2020) varying the latency constraint.
Revised: (2020) varying the latency constraint.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 7, our approach dominates EagleEye by consistently delivering a higher top-1 accuracy with a signiﬁcantly faster inference time.
Revised: 7, our approach dominates EagleEye by consistently delivering a higher top-1 accuracy with a signiﬁcantly faster inference time.

Original: We then analyze the structure difference between our pruned model and the EagleEye model.
Revised: We then analyze the structure difference between our pruned model and the EagleEye model.

Original: As mentioned in the main text that the proposed HALP method tries to make the number of remaining neurons in each layer fall to the right side of a step if the latency on the targeting platform presenting a staircase pattern.
Revised: As mentioned in the main text that the proposed HALP method tries to make the number of remaining neurons in each layer fall to the right side of a step if the latency on the targeting platform presenting a staircase pattern.

Original: Fig.
Revised: Fig.

Original: 8 shows two examples of pruned layers after pruning from HALP- 45% and
Revised: 8 shows two examples of pruned layers after pruning from HALP- 45% and

Original: EagleEye-2G model.
Revised: EagleEye-2G model.

Original: In the left ﬁgure, we show that the layer in our pruned model has only 5 more neurons pruned than that in EagleEye model, the latency is reduced to a much lower level which is a 0 .
Revised: In the left ﬁgure, we show that the layer in our pruned model has only 5 more neurons pruned than that in EagleEye model, the latency is reduced to a much lower level which is a 0 .

Original: 76 ms drop while we have 31 more input channels.
Revised: 76 ms drop while we have 31 more input channels.

Original: In the right ﬁgure, we also show that sometimes we can remain a lot more neurons ( 30 neurons) in layer with only little latency ( 0 . 21 ms) increase.
Revised: In the right ﬁgure, we also show that sometimes we can remain a lot more neurons ( 30 neurons) in layer with only little latency ( 0 . 21 ms) increase.

Original: These two examples both show the ability of method to fully exploit the latency traits and beneﬁt the inference speed.
Revised: These two examples both show the ability of method to fully exploit the latency traits and beneﬁt the inference speed.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Our method beneﬁts a lot from the non-linear latency characteristic since we are trying to keep as many neurons as possible under the latency constraint.
Revised: Our method beneﬁts a lot from the non-linear latency characteristic since we are trying to keep as many neurons as possible under the latency constraint.

Original: If the latency of the layer on the targeting platform shows linear pattern, the advantage of our method becomes smaller.
Revised: If the latency of the layer on the targeting platform shows linear pattern, the advantage of our method becomes smaller.

Original: Fig.
Revised: Fig.

Original: 8 shows the latency behavior of the example layers on the targeting platform when reducing the number of input and output channels.
Revised: 8 shows the latency behavior of the example layers on the targeting platform when reducing the number of input and output channels.

Original: As we can see, the staircase pattern becomes less obvious as the number of input channel reduces and the GPU has sufﬁcient capacity for the reduced computation.
Revised: As we can see, the staircase pattern becomes less obvious as the number of input channel reduces and the GPU has sufﬁcient capacity for the reduced computation.

Original: This happens during pruning, especially for large prune ratios.
Revised: This happens during pruning, especially for large prune ratios.

Original: In such a case, the FLOP count reﬂects the latency more accurately, and the performance gap between reducing FLOPs and reducing latency can possibly become small.
Revised: In such a case, the FLOP count reﬂects the latency more accurately, and the performance gap between reducing FLOPs and reducing latency can possibly become small.

Original: Nevertheless, our method can help avoid some latency peaks as shown in Fig. 8, which could otherwise happen using other pruning methods.
Revised: Nevertheless, our method can help avoid some latency peaks as shown in Fig. 8, which could otherwise happen using other pruning methods.

Original: D P RUNING R ESULTS ON O BJECT D ETECTION
Revised: D P RUNING R ESULTS ON O BJECT D ETECTION

Original: In this section we show the detailed pruning results on objection detection task for Sec. 4.5.
Revised: In this section we show the detailed pruning results on objection detection task for Sec. 4.5.

Original: To prune the detector, we ﬁrst train a SSD512 with ResNet50 as backbone.
Revised: To prune the detector, we ﬁrst train a SSD512 with ResNet50 as backbone.

Original: We also train some other popular models for performance comparison.
Revised: We also train some other popular models for performance comparison.

Original: The detailed numbers of Fig.
Revised: The detailed numbers of Fig.

Original: 5 are shown in Tab. 4.
Revised: 5 are shown in Tab. 4.

Original: E R ESULTS WITH S MALL B ATCH S IZE
Revised: E R ESULTS WITH S MALL B ATCH S IZE

Original: In the main paper, we use a large batch 256 in the experiment to allow for highest throughput for inference, which also makes the latency of the convolution layers show apparent staircase pattern so that we can take full advantage of the latency characteristic.
Revised: In the main paper, we use a large batch 256 in the experiment to allow for highest throughput for inference, which also makes the latency of the convolution layers show apparent staircase pattern so that we can take full advantage of the latency characteristic.

Original: In this section, we show that with small batch size 1 that no obvious staircase pattern showing up in layer latency, our HALP algorithm still delivers better results compared to other methods.
Revised: In this section, we show that with small batch size 1 that no obvious staircase pattern showing up in layer latency, our HALP algorithm still delivers better results compared to other methods.

Original: When we use batch size 1 for inference, the layer latency of ResNet50 does not show obvious staircase pattern in most of the layers due to the insufﬁcient usage of GPU.
Revised: When we use batch size 1 for inference, the layer latency of ResNet50 does not show obvious staircase pattern in most of the layers due to the insufﬁcient usage of GPU.

Original: Therefore in this experiment, we use the latency lookup table granularity as a neuron grouping size, which in our case is 2 , to fully
Revised: Therefore in this experiment, we use the latency lookup table granularity as a neuron grouping size, which in our case is 2 , to fully

Original: [Table]
Revised: [Table]

Original: fall to exploit the hardware latency traits during pruning.
Revised: fall to exploit the hardware latency traits during pruning.

Original: We show our pruned results and the comparison with other methods in Tab. 5.
Revised: We show our pruned results and the comparison with other methods in Tab. 5.

Original: As shown in the table, while other methods reduce the total FLOPs of the network after pruning, they do not reduce the actual latency much, which is up to 1 . 

Revised: As shown in the table, while other methods reduce the total FLOPs of the network after pruning, they do not reduce the actual latency much, which is up to 1 .

Original: × faster than the original one at the cost of 2 . 8% top1 accuracy drop.
Revised: × faster than the original one at the cost of 2 . 8% top1 accuracy drop.

Original: Compared to these methods, although we get less FLOPs reduction using our proposed method, the pruned models are faster and get higher accuracy, which is 1 .
Revised: Compared to these methods, although we get less FLOPs reduction using our proposed method, the pruned models are faster and get higher accuracy, which is 1 .

Original: 22 × faster than the unpruned model while getting slightly higher accuracy and 1 . 37 × faster with only 0 .
Revised: 22 × faster than the unpruned model while getting slightly higher accuracy and 1 . 37 × faster with only 0 .

Original: 9% accuracy drop.
Revised: 9% accuracy drop.

Original: F I MPLEMENTATION D ETAILS
Revised: F I MPLEMENTATION D ETAILS

Original: Convert latency in ﬂoat to int.
Revised: Convert latency in ﬂoat to int.

Original: Solving the neuron selection problem using the proposed augmented knapsack solver (Algo. 1 in the main paper), requires the neuron latency contribution and the latency constraint to be integers as shown in line 4 of the algorithm.
Revised: Solving the neuron selection problem using the proposed augmented knapsack solver (Algo. 1 in the main paper), requires the neuron latency contribution and the latency constraint to be integers as shown in line 4 of the algorithm.

Original: To convert the measured latency from a full precision ﬂoating-point number to integer type, we multiply the latency by 1000 and perform rounding.
Revised: To convert the measured latency from a full precision ﬂoating-point number to integer type, we multiply the latency by 1000 and perform rounding.

Original: Accordingly, we also scale and round the latency constraint value.
Revised: Accordingly, we also scale and round the latency constraint value.

Original: Deal with negative latency contribution.
Revised: Deal with negative latency contribution.

Original: The neuron latency contribution in our augmented knapsack solver must be a non-negative value since we have dp array ∈ R C
Revised: The neuron latency contribution in our augmented knapsack solver must be a non-negative value since we have dp array ∈ R C

Original: and we need to visit dp array [ c − c n ] as in line 5 of Algo.
Revised: and we need to visit dp array [ c − c n ] as in line 5 of Algo.

Original: 1 in the main paper.
Revised: 1 in the main paper.

Original: However, by analyzing the layer latency from the look-up table we ﬁnd that for some layers the measured latency might even increase when reducing some number of neurons.
Revised: However, by analyzing the layer latency from the look-up table we ﬁnd that for some layers the measured latency might even increase when reducing some number of neurons.

Original: This means that the latency contribution could possibly be negative.
Revised: This means that the latency contribution could possibly be negative.

Original: The simplest way to deal with the negative values is to directly set the negative latency contributions to be 0 .
Revised: The simplest way to deal with the negative values is to directly set the negative latency contributions to be 0 .

Original: This leads to the problem that the summed latency contribution would be larger than the actual latency value, causing less neurons being selected.
Revised: This leads to the problem that the summed latency contribution would be larger than the actual latency value, causing less neurons being selected.

Original: Thus, during our implementation, we keep those negative latency values as they are, but update the vector size of dp array to R C − min(min( c ) , 0) where min( c ) is the minimum latency contribution.
Revised: Thus, during our implementation, we keep those negative latency values as they are, but update the vector size of dp array to R C − min(min( c ) , 0) where min( c ) is the minimum latency contribution.

Original: With such, the vector size of dp array would be extended when there is negative latency contribution.
Revised: With such, the vector size of dp array would be extended when there is negative latency contribution.

Original: This makes it possible to add one neuron with negative latency contribution to a subset of neurons whose summed latency is larger than the latency constraint.
Revised: This makes it possible to add one neuron with negative latency contribution to a subset of neurons whose summed latency is larger than the latency constraint.

Original: After the addition, the total latency will still remain under the constraint.
Revised: After the addition, the total latency will still remain under the constraint.

Original: Pruning of the ﬁrst layer.
Revised: Pruning of the ﬁrst layer.

Original: In our experiments, we leave the ﬁrst convolutional layer of ResNets unpruned to help maintain the top-1 classiﬁcation accuracy.
Revised: In our experiments, we leave the ﬁrst convolutional layer of ResNets unpruned to help maintain the top-1 classiﬁcation accuracy.

Original: For MobileNet, the ﬁrst convolutional layer is coupled with its following group convolutional layer.
Revised: For MobileNet, the ﬁrst convolutional layer is coupled with its following group convolutional layer.

Original: In our MobileNet experiments, we prune the ﬁrst coupled layers at most to the half of neurons.
Revised: In our MobileNet experiments, we prune the ﬁrst coupled layers at most to the half of neurons.

Original: SSD for object detection.
Revised: SSD for object detection.

Original: Our SSD model is based on Liu et al.
Revised: Our SSD model is based on Liu et al.

Original: When we train SSDVGG16, we use the exactly same structure as described in the paper.
Revised: When we train SSDVGG16, we use the exactly same structure as described in the paper.

Original: When we train a SSDResNet50, the main difference between our model and the model described in the original paper is in the backbone, where the VGG is replaced by the ResNet50.
Revised: When we train a SSDResNet50, the main difference between our model and the model described in the original paper is in the backbone, where the VGG is replaced by the ResNet50.

Original: Following Huang et al.
Revised: Following Huang et al.

Original: (2017), we apply the following enhancements in our backbone:
Revised: (2017), we apply the following enhancements in our backbone:

Original: • The last stage of convolution layers, last avgpool and fc layers are removed from the original ResNet50 classiﬁcation model.
Revised: • The last stage of convolution layers, last avgpool and fc layers are removed from the original ResNet50 classiﬁcation model.

Original: • All strides in the 3 rd stage of ResNet50 layers are set to 1 × 1 .
Revised: • All strides in the 3 rd stage of ResNet50 layers are set to 1 × 1 .

Original: The backbone is followed by 6 additional coupled convolution layers for input size 512 × 512 , orfor input size 300 × 300 .
Revised: The backbone is followed by 6 additional coupled convolution layers for input size 512 × 512 , orfor input size 300 × 300 .

Original: A BatchNorm layer is added after each convolution layer.
Revised: A BatchNorm layer is added after each convolution layer.

Original: The settings of these additional convolution layers are listed in Tab. 6, each layer is represented as (output channel, kernel size, stride, padding).
Revised: The settings of these additional convolution layers are listed in Tab. 6, each layer is represented as (output channel, kernel size, stride, padding).

Original: layer
Revised: layer

Original: SSD
Revised: SSD

Original: SSD512-slim
Revised: SSD512-slim

Original: SSD layer1-conv ( 512 , 3 , 1 , 1 ) ( 256 , 1 , 1 , 0 ) ( 512 , 3 , 1 , 1 ) layer
Revised: SSD layer1-conv ( 512 , 3 , 1 , 1 ) ( 256 , 1 , 1 , 0 ) ( 512 , 3 , 1 , 1 ) layer

Original: [Table]
Revised: [Table]

Original: The detector heads are similar to the ones in the original paper.
Revised: The detector heads are similar to the ones in the original paper.

Original: The ﬁrst detection head is attached to the last layer of the backbone.
Revised: The ﬁrst detection head is attached to the last layer of the backbone.

Original: The rest detection heads are attached to the corresponding additional layers.
Revised: The rest detection heads are attached to the corresponding additional layers.

Original: No additional BatchNorm layer in the detector heads.
Revised: No additional BatchNorm layer in the detector heads.

Original: G FLOP S - CONSTRAINED P RUNING
Revised: G FLOP S - CONSTRAINED P RUNING

Original: Our implementation of latency-constrained pruning can be easily converted to be a FLOPsconstrained.
Revised: Our implementation of latency-constrained pruning can be easily converted to be a FLOPsconstrained.

Original: When constraining on FLOPs, Φ( · ) in the objective function (Eq.1 in the main paper) becomes the FLOPs measurement function and C becomes the FLOPs constraint.
Revised: When constraining on FLOPs, Φ( · ) in the objective function (Eq.1 in the main paper) becomes the FLOPs measurement function and C becomes the FLOPs constraint.

Original: Since the FLOPs of a layer linearly decreases as the number of neurons decreases in the layer, we do not need to group neurons in a layer any more.
Revised: Since the FLOPs of a layer linearly decreases as the number of neurons decreases in the layer, we do not need to group neurons in a layer any more.

Original: The problem can also be solved by original knapsack solver since each neuron’s FLOPs contribution in a layer is exactly the same and no preceding constraint is required.
Revised: The problem can also be solved by original knapsack solver since each neuron’s FLOPs contribution in a layer is exactly the same and no preceding constraint is required.

Original: We conduct some experiments by constraining the FLOPs and compare the results with EagleEye Li et al.
Revised: We conduct some experiments by constraining the FLOPs and compare the results with EagleEye Li et al.

Original: We name the experiments using the same algorithm as HALP but targeting on optimizing the FLOPs as FLOP-T. As shown in Tab. 7, with our pruning framework applying the knapsack solver, our results show higher top-1 accuracy compared to the pruned networks of EagleEye with
Revised: We name the experiments using the same algorithm as HALP but targeting on optimizing the FLOPs as FLOP-T. As shown in Tab. 7, with our pruning framework applying the knapsack solver, our results show higher top-1 accuracy compared to the pruned networks of EagleEye with

Original: [Table]
Revised: [Table]

Original: similar FLOPs remaining.
Revised: similar FLOPs remaining.

Original: We also observe a larger gap between the methods when it comes to a more compact network.
Revised: We also observe a larger gap between the methods when it comes to a more compact network.

Original: H FLOP S VS . L ATENCY
Revised: H FLOP S VS . L ATENCY

Original: FLOPs can be regarded as a proxy of inference latency; however, they are not equivalent [4, 33, 35, 40, 42].
Revised: FLOPs can be regarded as a proxy of inference latency; however, they are not equivalent [4, 33, 35, 40, 42].

Original: We do global ﬁlter-wise pruning and have the same problem as NAS.
Revised: We do global ﬁlter-wise pruning and have the same problem as NAS.

Original: The latency on a GPU usually imposes staircase-shaped patterns for convolutional operators with varying channels and requires pruning in groups.
Revised: The latency on a GPU usually imposes staircase-shaped patterns for convolutional operators with varying channels and requires pruning in groups.

Original: In contrast, FLOPs will change linearly.
Revised: In contrast, FLOPs will change linearly.

Original: Depth-wise convolution, compared to dense counterparts, has signiﬁcantly fewer FLOPs but almost the same GPU latency due to execution being memory-bounded 1 .
Revised: Depth-wise convolution, compared to dense counterparts, has signiﬁcantly fewer FLOPs but almost the same GPU latency due to execution being memory-bounded 1 .

Original: The discrepancy also holds for ResNets where the same amount of FLOPs impose more latency in earlier layers than later ones as the number of channels increases and feature map dimension shrinks – both increase compute parallelism.
Revised: The discrepancy also holds for ResNets where the same amount of FLOPs impose more latency in earlier layers than later ones as the number of channels increases and feature map dimension shrinks – both increase compute parallelism.

Original: For example, the ﬁrst 7 × 7 conv layer and the ﬁrst bottleneck 3 × 3 conv in ResNet50 have nearly identical FLOPs but the former is 60% slower on-chip.
Revised: For example, the ﬁrst 7 × 7 conv layer and the ﬁrst bottleneck 3 × 3 conv in ResNet50 have nearly identical FLOPs but the former is 60% slower on-chip.

Original: We compare our results of FLOPs-targeted (FLOP-T) showed in Sec.
Revised: We compare our results of FLOPs-targeted (FLOP-T) showed in Sec.

Original: G and results using latencytargeted pruning (HALP) in Tab. 8.
Revised: G and results using latencytargeted pruning (HALP) in Tab. 8.

Original: As shown in the table, using different optimization targets leads to quite different FPS vs FLOPs curves.
Revised: As shown in the table, using different optimization targets leads to quite different FPS vs FLOPs curves.

Original: In overall, with similar FLOPs remaining, using our HALP algorithm targeting on reducing the actual latency can get more efﬁcient networks with more image being processed per second.
Revised: In overall, with similar FLOPs remaining, using our HALP algorithm targeting on reducing the actual latency can get more efﬁcient networks with more image being processed per second.

Original: [Table]
Revised: [Table]

Original: [Figure] 
Revised: [Figure] [Table] We also show a more straightforward relationship between the actual latency of a layer and its FLOPs in Fig. 9. We use the 2 nd convolution layer in the 1 st residual block of ResNet50 as an example.

Original: [Table]
Revised: We vary the number of neurons of the layer from 0 to 128 and measure the actual latency on GPU (TITAN V) as well as the FLOPs of the layer. We can see from the ﬁgure that the actual latency I A LGORITHM 1 E XPLANATION

Original: [Figure]
Revised: We follow the standard Knapsack problem dynamic programming solution to break down the original problem into sub-problems. Speciﬁcally, for each neuron j (or neuron group) in layer l , we can choose to either include or not include it under the latency constraint c . When it is kept, the total importance score increases I jl while the latency constraint for the other neurons becomes c − c jl ; If the neuron is removed, the latency constraint for the other neurons remains c . We choose to keep or remove the current neuron to maximize total importance.

Revised: At the same time, we check whether the more important neurons in the same layer are included to ensure the correctness of the latency.

Revised: The neuron selection from the remaining neurons is a sub-problem to solve.

Revised: Precisely, we use a vector maxV ∈ R ( C +1) to store the maximum importance that we can achieve under the latency constraint c , 0 ≤ c ≤ C and keep ∈ R L × ( C +1) , a 2D vector where keep [ l, c ] denotes the number of neuron groups we need to maintain in layer l to obtain the maximum importance maxV [ c ] .

Revised: We process the neurons according to their importance score in decreasing order.

Revised: In this way, all preceding neurons to the current one (i.e., neurons with a higher importance score in the same layer) will be always considered ﬁrst.

Revised: To decide if we keep or remove the current neuron, we check the total importance score and the inclusion status of its preceding neurons, so we can maximize the total importance and ensure the latency cost correctness.

Original: [Table]
Revised: [Figure] In this work, similar to many other prior methods Alvarez & Salzmann (2016); Molchanov et al. (2019); Yu & Huang (2019), we do iterative pruning with k pruning steps in total.

Revised: In this experiment, we analyze the the accuracy of the ﬁnal result as a function of k .

Revised: We set the value of k to 10 , 20 , 30 and 40 for iterative pruning, and also use k = 1 to perform a single-shot pruning.

Revised: The result of this experiment is shown in Fig.

Revised: As shown, we get similar results independent of k .

Revised: Imporantly, all these results outperform EagleEye Li

Revised: et al.

Revised: As expected, there is a drop in accuracy for single-shot pruning ( k = 1 ), especially for large pruning ratios.

Revised: The main reason is the neuron importance would change as we remove some other neurons and, in this setting, the value is not updated.

Revised: Iterative pruning does not have this limitation as the importance score and the latency cost of the remaining neurons is updated after each pruning step to reﬂect any changes.

Revised: In our experiments, we use k = 30 as it provides a good trade off between latency and accuracy.

Revised: K M ORE P RUNING R ESULTS ON M OBILE N ETS

Revised: [Figure]

Revised: Table.

Revised: 9 and Fig.

Revised: 11 provide additional pruning results for lightweight networks such as

Revised: MobileNet-V1 and MobileNet-V2.

Revised: For the unpruned models, we ﬁnd that even MobileNetV2 has signiﬁcantly lower FLOPs, the inference time is larger compared to MobileNetV1.In both cases, HALP yields inference speeds-ups of 1 .

Revised: 22 × and 1 .

Revised: 33 × for MobileNetV1 and MobileNet-V2 respectively, while maintaining the original top1 accuracy.

Revised: [Table]

Revised: L D ETAILED C ONFIGURATION OF P RUNED M ODELS

Revised: We provide the detailed conﬁguration of our pruned models of Tab. 1.

Revised: For each model, we list the number of neurons remaining in each convolution layer, starting from the input to the output.

Revised: For

Revised: ResNets we use [ · ] to denote a residual block and ( · ) to denote the neuron number of the residual bypass layer.

Revised: Note that for ResNets, the conﬁguration is the “raw” conﬁguration after the pruning.

Revised: For each residual block [ x 1 , x 2 , x 3 ] , because of the existing of bypass layer, we allow the entire layer pruning and it still leads to trainable networks.

Revised: Whenever there is entire layer pruning ( x 1 == 0 or x 2 ==or x 3 == 0 ), all the other layers in the block can be removed and this branch can be replaced by a constant value.

Revised: In such a case, the corresponding block in our ﬁnal pruned model is cleaned to be [0 , 0 , 0] .

Revised: We also provide the detailed structure plot of two pruned ResNet50 models in Fig.

Revised: 12 for a better visualization.

Revised: [Table]

Revised: M D IFFERENT C HOICE OF I MPORTANCE C ALCULATION

Revised: We use the ﬁrst Taylor expansion Molchanov et al.

Revised: (2019) to estimate the loss change induced by pruning as the importance score of the neurons.

Revised: It is a gradient-based importance calculation and is shown to be given promising results.

Revised: In this section, we use the L2 norm of the neuron weights as the importance measurement and apply HALP framework to ResNet50 ImageNet classiﬁcation

Revised: [Table]

Revised: l a y e r 3 . 2 .

Revised: c on vl a y e r 3 . 3 .

Revised: c on vl a y e r 3 . 3 .

Revised: c on vl a y e r 3 . 3 .

Revised: c on vl a y e r 3 . 4 .

Revised: c on vl a y e r 3 . 4 .

Revised: c on vl a y e r 3 . 4 .

Revised: c on vl a y e r 3 . 5 .

Revised: c on vl a y e r 3 . 5 .

Revised: c on vl a y e r 3 . 5 .

Revised: c on vl a y e r 4 . 0 .

Revised: c on vl a y e r 4 . 0 .

Revised: c on vl a y e r 4 . 0 .

Revised: c on vl a y e r 4 . 0 .

Revised: do w n s a m p l e l a y e r 4 .

Revised: c on vl a y e r 4 .

Revised: c on vl a y e r 4 .

Revised: c on vl a y e r 4 . 2 .

Revised: c on vl a y e r 4 . 2 .

Revised: c on vl a y e r 4 . 2 .

Original: {+c on v
+}
Revised: {+c on v
+}

Revised: N u m be r

Revised: o f N eu r on s remainprune

Revised: [Table]

Revised: c on v 1 l a y e r 1 . 0 .

Revised: c on vl a y e r 1 . 0 .

Revised: c on vl a y e r 1 . 0 .

Revised: c on vl a y e r 1 . 0 .

Revised: do w n s a m p l e l a y e r 1 . 1 .

Revised: c on vl a y e r 1 . 1 .

Revised: c on vl a y e r 1 . 1 .

Revised: c on vl a y e r 1 . 2 .

Revised: c on vl a y e r 1 . 2 .

Revised: c on vl a y e r 1 . 2 .

Revised: c on vl a y e r 2 . 0 .

Revised: c on vl a y e r 2 . 0 .

Revised: c on vl a y e r 2 . 0 .

Revised: c on vl a y e r 2 . 0 .

Revised: do w n s a m p l e l a y e r 2 .

Revised: c on vl a y e r 2 . 1 .

Revised: c on vl a y e r 2 . 1 .

Revised: c on vl a y e r 2 . 2 .

Revised: c on vl a y e r 2 . 2 .

Revised: c on vl a y e r 2 . 2 .

Revised: c on vl a y e r 2 . 3 .

Revised: c on vl a y e r 2 . 3 .

Revised: c on vl a y e r 2 . 3 .

Revised: c on vl a y e r 3 . 0 .

Revised: c on vl a y e r 3 . 0 .

Revised: c on vl a y e r 3 . 0 .

Revised: c on vl a y e r 3 . 0 .

Revised: do w n s a m p l e l a y e r 3 . 1 .

Revised: c on vl a y e r 3 . 1 .

Revised: c on vl a y e r 3 . 1 .

Revised: c on vl a y e r 3 . 2 .

Revised: c on vl a y e r 3 . 2 .

Revised: c on vl a y e r 3 . 2 .

Revised: c on vl a y e r 3 . 3 .

Revised: c on vl a y e r 3 . 3 .

Revised: c on vl a y e r 3 . 3 .

Revised: c on vl a y e r 3 . 4 .

Revised: c on vl a y e r 3 . 4 .

Revised: c on vl a y e r 3 . 4 .

Revised: c on vl a y e r 3 . 5 .

Revised: c on vl a y e r 3 . 5 .

Revised: c on vl a y e r 3 . 5 .

Revised: c on vl a y e r 4 . 0 .

Revised: c on vl a y e r 4 . 0 .

Revised: c on vl a y e r 4 . 0 .

Revised: c on vl a y e r 4 . 0 .

Revised: do w n s a m p l e l a y e r 4 .

Revised: c on vl a y e r 4 .

Revised: c on vl a y e r 4 .

Revised: c on vl a y e r 4 . 2 .

Revised: c on vl a y e r 4 . 2 .

Revised: c on vl a y e r 4 . 2 .

Original: {+c on v
+}
Revised: {+c on v
+}

Revised: N u m be r

Revised: o f N eu r on s

Revised: [Figure]

Revised: task.

Revised: As shown in Tab. 11, Our algorithm is generic applying to different importance measurements.

Revised: As shown, using L2 norm of weights as importance measurements leads to slightly lower accuracy.

Revised: [Table]

Revised: N

Revised: L

Revised: ATENCY

Revised: L

Revised: OOK - UP T ABLE

Revised: C REATION AND C ALIBRATION

Revised: [Figure]

Revised: c t

Revised: [Figure]

Revised: In this section, we provide additional details to build the latency look-up table used in HALP, computational cost and the correlation between the estimated and the real ones.

Revised: As mentioned in Appendix A, we pre-generate the layer latency look-up table on the platform with

Revised: NVIDIA cuDNN Chetlur et al.

Revised: V7.6.5.

Revised: For each layer, we iteratively reduce the number of neurons in the layer (each time reduce 8 neurons) and characterize the corresponding latency.

Revised: For each latency measurement, we use one proﬁle for GPU warm up and another 3 proﬁles and take the average to avoid randomness.

Revised: The average standard deviation of proﬁles for an operation is 8 .

Revised: 67 e − 3 .

Revised: On TITAN V GPU, it takes around 5 hours to build the look-up table for ResNets family and 1 hour for MobileNets family.

Revised: Note that the LUT can be shared by the network architectures within the same family since they usually have similar layer structures.

Revised: We only need to create the latency look-up table once for all the possible latency targets.

Revised: There are indeed some gaps between the predicted latency and the real latency of the model, because the latency look-up table is created layer-wise on convolution layers.

Revised: There are additional costs in real inference such as pooling, non-linear activation etc.

Revised: We plot the correlation between the expected latency reduction from look-up table and the real latency reduction ratio of our pruned models in Fig.

Revised: We also calculate the Pearson Correlation Coefﬁcient r for all the networks in the ﬁgure.

Revised: As shown, we can clearly see a linear correlation between the predicted and real latency reduction.

Revised: O B REAKDOWN OF THE A LGORITHM E XECUTION T IME

Revised: We provide additional details of the algorithm process of different methods in this section.

Revised: We estimate the time cost needed to get the pruned network structure for each method.

Revised: The time of following ﬁnetuning is not taken into consideration.

Revised: For a fair comparison, we set the number of pruning steps k for all iterative pruning methods to 30 .

Revised: All the values are approximated as all the methods are running on the same device (a NVIDIA V100 GPU) to get a pruned ResNet50.

Revised: For

Revised: AutoSlim Yu & Huang (2019), MetaPruning Liu et al. (2019a) and AMC He et al.

Revised: (2018), more

Revised: GPU time is needed for additional training of the network.

Revised: [Figure]

Revised: [Table]