Original: Latency-aware Spatial-wise Dynamic Networks
Revised: Latency-aware Spatial-wise Dynamic Networks

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Spatial-wise dynamic convolution has become a promising approach to improvingthe inference efficiency of deep networks.
Revised: Spatial-wise dynamic convolution has become a promising approach to improvingthe inference efficiency of deep networks.

Original: By allocating more computation to themost informative feature pixels, such an adaptive inference paradigm alleviatesthe spatial redundancy in image features and reduces a considerable amount ofunnecessary computation.
Revised: By allocating more computation to themost informative feature pixels, such an adaptive inference paradigm alleviatesthe spatial redundancy in image features and reduces a considerable amount ofunnecessary computation.

Original: However, the theoretical efficiency achieved by previousmethods can hardly translate into the realistic speedup, especially on the multicore processors (e.g. GPUs).
Revised: However, the theoretical efficiency achieved by previousmethods can hardly translate into the realistic speedup, especially on the multicore processors (e.g. GPUs).

Original: The key challenge is that the existing literature hasonly focused on designing algorithms with minimal computation , ignoring thefact that the practical latency can also be influenced by scheduling strategies andhardware properties .
Revised: The key challenge is that the existing literature hasonly focused on designing algorithms with minimal computation , ignoring thefact that the practical latency can also be influenced by scheduling strategies andhardware properties .

Original: To bridge the gap between the theoretical computation andthe practical efficiency, we propose a latency-aware spatial-wise dynamic network(LASNet), which performs coarse-grained spatially adaptive inference under theguidance of a novel latency prediction model .
Revised: To bridge the gap between the theoretical computation andthe practical efficiency, we propose a latency-aware spatial-wise dynamic network(LASNet), which performs coarse-grained spatially adaptive inference under theguidance of a novel latency prediction model .

Original: This latency prediction model canefficiently estimate the inference latency of dynamic networks by simultaneouslyconsidering the algorithms, the scheduling strategies, and the hardware properties.
Revised: This latency prediction model canefficiently estimate the inference latency of dynamic networks by simultaneouslyconsidering the algorithms, the scheduling strategies, and the hardware properties.

Original: We use the latency predictor to guide both the algorithm design and the schedulingoptimization on various hardware platforms.
Revised: We use the latency predictor to guide both the algorithm design and the schedulingoptimization on various hardware platforms.

Original: Experiments on image classificationdemonstrate that the proposed framework significantly improves the trade-offbetween the accuracy and the inference efficiency of deep networks.
Revised: Experiments on image classificationdemonstrate that the proposed framework significantly improves the trade-offbetween the accuracy and the inference efficiency of deep networks.

Original: For example,the average latency of a ResNet-101 on the ImageNet validation set could bereduced by 23% and 45% on a server GPU (Nvidia Tesla-V100) and an IoT device(Nvidia Jetson TX2 GPU) respectively without sacrificing the accuracy.
Revised: For example,the average latency of a ResNet-101 on the ImageNet validation set could bereduced by 23% and 45% on a server GPU (Nvidia Tesla-V100) and an IoT device(Nvidia Jetson TX2 GPU) respectively without sacrificing the accuracy.

Original:  Introduction
Revised:  Introduction

Original: Dynamic neural networks [6] have attracted great research interests in recent years.
Revised: Dynamic neural network [6] has attracted great research interests in recent years.

Original: Compared tostatic models [8, 14, 10, 20] which treat different inputs equally during inference, dynamic networkscan allocate the computation in a data-dependent manner.
Revised: Compared tostatic models [8, 13, 9, 19] which treat different inputs equally during inference, dynamic networkscan allocate the computation in a data-dependent manner.

Original: For example, they can conditionallyskip the computation of network layers [12, 29, 27] and convolutional channels [16, 1] or performspatially adaptive inference on the most informative image regions (e g. the foreground areas)[5, 4, 28, 32, 30, 7].
Revised: For example, they can conditionallyskipping the computation of network layers [11, 28, 26] and convolutional channels [15, 1] orperform spatially adaptive inference on the most informative image regions (e g. the foregroundareas)

Original: Spatial-wise dynamic networks, which typically decide whether to computeeach feature pixel with plug-in masker modules [4, 28, 32, 7] (see Figure 1 (a)), have shown verypromising results in improving the inference efficiency of convolution neural networks (CNNs).
Revised: Spatial-wise dynamic networks, which typically decide whether tocompute each feature pixel with plug-in masker modules [4, 27, 31, 7] (see Figure 1 (a)), have shownvery promising results in improving the inference efficiency of convolution neural networks (CNNs).

Original: Despite the remarkable theoretical efficiency achieved by spatial-wise dynamic networks [4, 28, 32],researchers have found it challenging to translate the theoretical results into realistic speedup,especially on some multi-core processors, e.g., GPUs [32, 2, 7].
Revised: Despite the remarkable theoretical efficiency achieved by spatial-wise dynamic networks [4, 27, 31],researchers have found it challenging to translate the theoretical results into realistic speedup,especially on some multi-core processors, e.g. GPUs [31, 2, 7].

Original: The challenges are two-fold: 1) mostprevious approaches [4, 28, 32] perform spatially adaptive inference at the finest granularity : everypixel is flexibly decided whether to be computed or not.
Revised: The challenges are two-fold: 1) mostprevious approaches [4, 27, 31] perform spatially adaptive inference at the finest granularity : everypixel is flexibly decided whether to be computed or not.

Original: Such flexibility induces non-contiguousmemory access [32] and requires specialized scheduling strategies (Figure 1 (b)); 2) the existingliterature has only adopted the hardware-agnostic FLOPs (floating-point operations) as an inaccurateproxy for the efficiency, lacking  latency-aware guidance on the algorithm design.
Revised: Such flexibility induces non-contiguousmemory access [31], and requires specialized scheduling strategies (Figure 1 (b)); 2) the existingliterature has only adopted the hardware-agnostic FLOPs (floating-point operations) as an inaccurateproxy for the efficiency, lacking a latency-aware guidance on the algorithm design.

Original: For dynamic
Revised: For dynamic

Original: [Figure]
Revised: [Figure]

Original: networks, the adaptive computation with sub-optimal scheduling strategies further enlarges thediscrepancybetween the theoretical FLOPsand the practical latency.
Revised: networks, the adaptive computation with sub-optimal scheduling strategies further enlarges thediscrepancy theoretical FLOPs

Original: Note that it has been validatedby previous works that the latency on CPUs has a strong correlation with FLOPs [7, 32].

Original: Therefore,we mainly focus on the GPU platform in this paper, which is more challenging and less explored.

Original: We address the above challenges by proposing a latency-aware spatial-wise dynamic network(LASNet).
Revised: We address the above challenges by proposing a latency-aware spatial-wise dynamic network(LASNet).

Original: Three key factors to the inference latency are considered: the algorithm , the schedulingstrategy , and the hardware properties .
Revised: Three key factors to the inference latency are considered: the algorithm , the schedulingstrategy , and the hardware properties .

Original: Given a target hardware device, we directly use the latency,rather than the FLOPs, to guide our algorithm design and scheduling optimization (see Figure 1 (c)).
Revised: Given a target hardware device, we directly use the latency,rather than the FLOPs, to guide our algorithm design and scheduling optimization (see Figure 1 (c)).

Original: Because the memory access pattern and the scheduling strategies in our dynamic operators differfrom those in static networks, the libraries developed for static models (e.g. cuDNN) are sub-optimalfor the acceleration of dynamic models.
Revised: Because the memory access pattern and the scheduling strategies in our dynamic operators differfrom those in static networks, the libraries developed for static models (e.g. cuDNN) are sub-optimalfor the acceleration of dynamic models.

Original: Without the support of libraries, each dynamic operatorrequires scheduling optimization, code optimization, compiling, and deployment for each device.
Revised: Without the support of libraries, each dynamic operatorrequires scheduling optimization, code optimization, compiling, and deployment for each device.

Original: Therefore, it is laborious to evaluate the network latency on different hardware platforms.
Revised: Therefore, it is laborious to evaluate the network latency on different hardware platforms.

Original: To this end,we propose a novel latency prediction model  to efficiently estimate the realistic latency of a networkby simultaneously considering the aforementioned three factors.
Revised: To this end,we propose a novel latency prediction model , to efficiently estimate the realistic latency of a networkby simultaneously considering the aforementioned three factors.

Original: Compared to the hardware-agnostic
Revised: Compared to the hardware-agnostic

Original: FLOPs, our predicted latency can better reflect the practical efficiency of dynamic models.
Revised: FLOPs, our predicted latency can better reflect the practical efficiency of dynamic models.

Original: Guided by this latency prediction model, we establish our latency-aware spatial-wise dynamicnetwork (LASNet), which adaptively decides whether to allocate computation on feature patchesinstead of pixels [4, 28, 32] (Figure 2 top).We name this behavior as spatially adaptive inference at acoarse granularity .
Revised: Guided by this latency prediction model, we establish our latency-aware spatial-wise dynamic network(LASNet), which performs spatially adaptive inference at a coarse granularity . Specifically, thedecisions on whether to allocate computation are made for feature patches instead of individual pixels[4, 27, 31] (Figure 2 top).

Original: While  less flexible than the pixel-level adaptive computation in previous works[4, 28, 32], it facilitates more contiguous memory access, benefiting the realistic speedup on hardware.
Revised: While spatially adaptive inference at such a patch level is less flexible thanthe previous pixel level, it facilitates more contiguous memory access.

Original: The scheduling strategy and the implementation are further ameliorated for faster inference.
Revised: The scheduling strategy andthe implementation are further ameliorated for faster inference on hardware devices.

Original: It is worth noting that LASNet is designed as a general framework in two aspects: 1) the coarsegrained spatially adaptive inference paradigm can be conveniently implemented in various CNNbackbones, e.g., RegNets [22] and ResNets [8]; and 2) the latency prediction model is an off-the-shelltool which can be directly used for various computing platforms (e.g. server GPUs and IoT devices).
Revised: It is worth noting that LASNet is designed as a general framework in two aspects: 1) the coarsegrained spatially adaptive inference paradigm can be conveniently implemented in various CNNbackbones, e.g. RegNets [21] and ResNets [8]; and 2) the latency prediction model is an off-the-shelltool which can be directly used for various computing platforms (e.g. server GPUs and IoT devices).

Original: We evaluate the performance of LASNet on multiple CNN architectures on image classification,object detection, and instance segmentation tasks.
Revised: We evaluate the performance of LASNet on multiple CNN architectures on image classification andobject detection tasks.

Original: Experiment results show that our LASNet improvesthe efficiency of deep CNNs both theoretically and practically.
Revised: Experiment results show that our LASNet improves the efficiency of deep CNNs both theoretically and practically.

Original: For example, the inference latency of ResNet-101 is reduced by 23% and 45% on an Nvidia Tesla V100 GPU and an Nvidia Jetson [-TX
 GPU, respectively, without sacrificing-] the accuracy on the ImageNet [3] validation set.
Revised: For example, the inference latency of ResNet-101 is reducedby 23% and 45% on a Nvidia Tesla V100 GPU and a Nvidia Jetson [-TX
 GPU, respectively, without sacrificing-]TX2 GPU respectively withoutsacrificing the accuracy on the ImageNet [3] validation set.

Original: Moreover, theproposed method outperforms various lightweight networks in a low-FLOPs regime.
Revised: Moreover, the proposed method alsooutperforms various lightweight networks in a low-FLOPs regime.

Original: Our main contributions are summarized as follows:
Revised: Our main contributions are summarized as follows:

Original: 1) We propose LASNet, which performs coarse-grained spatially adaptive infee.g. , nce guided by thepractical latency instead of the theoretical FLOPs.
Revised: 1) We propose LASNet, which performs coarse-grained spatially adaptive inference guided by thepractical latency instead of the theoretical FLOPs.

Original: To the best of our knowledge, LASNet is the firstframework that directly considers the real latency in the design phase of dynamic neural networks;2)
Revised: To the best of our knowledge, LASNet is the firstframework which directly considers the real latency in the design phase of dynamic neural networks;2)

Original: We propose a latency prediction model, which efficiently estimates the latency of dynamic operatorsby simultaneously considering the algorithm, the scheduling strategy 
Revised: We propose a latency prediction model, which efficiently estimates the latency of dynamic operatorsby simultaneously considering the algorithm, the scheduling strategy and the hardware properties;3)

Original: and the hardware properties;3) Experiments on image classification, object detection, and instance segmentation tasks verify thatour proposed LASNet can effectively improve the practical efficiency of different CNN architectures
Revised: Experiments on image classification and object detection tasks verify that our proposed LASNet can effectively improve the practical efficiency of different CNN architectures.

Original:  Related works
Revised:  Related works

Original: Spatial-wise dynamic network is a common type of dynamic neural networks [6].
Revised: Spatial-wise dynamic network is a common type of dynamic neural networks [6].

Original: Compared tostatic models which treat different feature locations evenly during inference, these networks performspatially adaptive inference on the most informative regions (e.g., foregrounds), and reduce theunnecessary computation on less important areas (e.g., backgrounds).
Revised: Compared tostatic models which treat different feature locations evenly during inference, these networks performspatially adaptive inference on the most informative regions (e.g. foregrounds), and reduce theunnecessary computation on less important areas (e.g. backgrounds).

Original: Existing works mainly includethree levels of dynamic computation: resolution level [33, 34], region level [30] and pixel level[4, 28, 32].
Revised: Existing works mainly includethree levels of dynamic computation: resolution level [32, 33], region level [29] and pixel level[4, 27, 31].

Original: The former two generally manipulate the network inputs [30, 34] or require specialarchitecture design [33].
Revised: The former two generally manipulate the network inputs [29, 33] or require specialarchitecture design [32].

Original: In contrast, pixel-level dynamic networks can flexibly skip the convolutionson certain feature pixels in arbitrary CNN backbones [4, 28, 32].
Revised: In contrast, pixel-level dynamic networks can flexibly skip the convolutionson certain feature pixels in arbitrary CNN backbones [4, 27, 31].

Original: Despite its remarkable theoreticalefficiency, the pixel-wise dynamic computation brings considerable difficulty to achieving realisticspeedup on multi-core processors, e.g., GPUs.
Revised: Despite its remarkable theoreticalefficiency, the pixel-wise dynamic computation brings considerable difficulty to achieving realisticspeedup on multi-core processors, e.g. GPUs.

Original: Compared to the previous approaches [4, 28, 32]which only focus on reducing the theoretical computation, we propose to directly use the latency toguide our algorithm design and scheduling optimization.
Revised: Compared to the previous approaches [4, 27, 31]which only focus on reducing the theoretical computation, we propose to directly use the latency toguide our algorithm design and scheduling optimization.

Original: Hardware-aware network design.
Revised: Hardware-aware network design.

Original: To bridge the gap between theoretical and practical efficiencyof deep models, researchers have started to consider the real latency in the network design phase.
Revised: To bridge the gap between theoretical and practical efficiencyof deep models, researchers have started to consider the real latency in the network design phase.

Original: There are two lines of works in this direction.
Revised: There are two lines of works in this direction.

Original: One directly performs speed tests on targeteddevices, and summarizes some guidelines to facilitate hand-designing lightweight models [20].
Revised: One directly performs speed tests on targeteddevices, and summarizes some guidelines to facilitate hand-designing lightweight models [19].

Original: Theother line of work searches for fast models using the neural architecture search (NAS) technique[26, 31].
Revised: Theother line of work searches for fast models using the neural architecture search (NAS) technique[25, 30].

Original: However, all existing works try to build static models, which have intrinsic computationalredundancy by treating different inputs in the same way.
Revised: However, all existing works try to build static models, which have intrinsic computationalredundancy by treating different inputs in the same way.

Original: However, speed tests for dynamic operatorson different hardware devices can be very laborious and impractical.
Revised: However, speed tests for dynamic operatorson different hardware devices can be very laborious and impractical.

Original: In contrast, our proposed latencyprediction model can efficiently estimate the inference latency on any given computing platforms bysimultaneously considering algorithm design, scheduling strategies and hardware properties.
Revised: In contrast, our proposed latencyprediction model can efficiently estimate the inference latency on any given computing platforms bysimultaneously considering algorithm design, scheduling strategies and hardware properties.

Original:  Methodology
Revised:  Methodology

Original: In this section, we first introduce the preliminaries of spatially adaptive inference, and then demonstrate the architecture design of our LASNet.
Revised: In this section, we first introduce the preliminaries of spatially adaptive inference, and then demonstrate the architecture design of our LASNet.

Original: The latency prediction model is then explained, whichguides the granularity settings and the scheduling optimization for LASNet.
Revised: The latency prediction model is then explained, whichguides the granularity settings and the scheduling optimization for LASNet.

Original: We further present theimplementation improvements for faster inference, followed by the training strategies.
Revised: We further present theimplementation improvements for faster inference, followed by the training strategies.

Original: 3.1 Preliminaries
Revised: 3.1 Preliminaries

Original: Spatially adaptive inference.
Revised: Spatially adaptive inference.

Original: The existing spatial-wise dynamic networks are usually established byattaching a masker M in each convolutional block of a CNN backbone (see Figure 1 (a)).
Revised: The existing spatial-wise dynamic networks are usually established byattaching a masker M in each convolutional block of a CNN backbone (see Figure 1 (a)).

Original: Specifically,let x ∈
Revised: Specifically,let x ∈

Original: R
Revised: R

Original: H
Revised: H

Original: × W × C denote the input of a block, where H and W are the feature height and width, and
Revised: × W × C denote the input of a block, where H and W are the feature height and width, and

Original: C is the channel number.
Revised: C is the channel number.

Original: The masker M takes x as input, and generates a binary-valued spatialmask M = M ( x ) ∈ { 0 , 1 } H × W .
Revised: The masker M takes x as input, and generates a binary-valued spatialmask M = M ( x ) ∈ { 0 , 1 } H × W .

Original: Each element of M determines whether to perform convolutionoperations on the corresponding location of the output feature.
Revised: Each element of M determines whether to perform convolutionoperations on the corresponding location of the output feature.

Original: The unselected regions will be filledwith the values from the input [4, 28] or obtained via interpolation [32].
Revised: The unselected regions will be filledwith the values from the input [4, 27] or obtained via interpolation [31].

Original: We define the activation rateof a block as r = (cid:80) i,j M i,j H × W , representing the ratio of the calculated pixels.
Revised: We define the activation rateof a block as r = (cid:80) i,j M i,j H × W , representing the ratio of the calculated pixels.

Original: Scheduling strategy.
Revised: Scheduling strategy.

Original: During inference, the current scheduling strategy for spatial-wise dynamicconvolutions generally involve three steps [23] (see Figure 1 (b)): 1) gathering , which re-organizesthe selected pixels (if the convolution kernel size is greater than 1 × 1 , the neighbors are also required)along the batch dimension; 2) computation , which performs convolution on the gathered input; and3) scattering , which fills the computed pixels on their corresponding locations of the output feature.
Revised: During inference, the current scheduling strategy for spatial-wise dynamicconvolutions generally involve three steps [22] (see Figure 1 (b)): 1) gathering , which re-organizesthe selected pixels (if the convolution kernel size is greater than 1 × 1 , the neighbors are also required)along the batch dimension; 2) computation , which performs convolution on the gathered input; and3) scattering , which fills the computed pixels on their corresponding locations of the output feature.

Original: Limitations.
Revised: Limitations.

Original: Compared to performing convolutions on the entire feature map, the aforementionedscheduling strategy reduces the computation while bringing considerable overhead to memory accessdue to the mask generation and the non-contiguous memory access.
Revised: Compared to performing convolutions on the entire feature map, the aforementionedscheduling strategy reduces the computation while bringing considerable overhead to memory accessdue to the mask generation and the non-contiguous memory access.

Original: Such overhead would increasethe overall latency, especially when the granularity of dynamic convolution is the finest pixel level.
Revised: Such overhead would increasethe overall latency, especially when the granularity of dynamic convolution is the finest pixel level.

Original: 3.2 Architecture design
Revised: 3.2 Architecture design

Original: Spatial granularity.
Revised: Spatial granularity.

Original: As mentioned above, pixel -level dynamic convolutions [4, 28, 32] raise substantial challenges to achieving realistic speedup on multi-core processors due to the non-contiguousmemory access.
Revised: As mentioned above, pixel -level dynamic convolutions [4, 27, 31] raise substantial challenges to achieving realistic speedup on multi-core processors due to the non-contiguousmemory access.

Original: To this end, we propose to optimize the granularity of spatially adaptive inference.
Revised: To this end, we propose to optimize the granularity of spatially adaptive inference.

Original: Specifically, take the commonly used bottleneck structure in [8] as an example, our coarse-grainedspatial-wise dynamic convolutional block is illustrated in Figure 2.
Revised: Specifically, take the commonly used bottleneck structure in [8] as an example, our coarse-grainedspatial-wise dynamic convolutional block is illustrated in Figure 2.

Original: Instead of directly producing
Revised: Instead of directly producing

Original: [Figure]
Revised: [Figure]

Original: a mask with the shape of H × W , we first generate a low-resolution mask M coarse ∈{ 0 ,
Revised: a mask with the shape of H × W , we first generate a low-resolution mask M coarse ∈{ 0 ,

Original: 1 } HS × WS ,where S is named as the spatial granularity .
Revised: 1 } HS × WS ,where S is named as the spatial granularity .

Original: Each element in M coarse determines the computationof an S × S -sized feature patch.
Revised: Each element in M coarse determines the computationof an S × S -sized feature patch.

Original: For instance, the feature size in the first ResNet stage 1 is 56 × 56 .
Revised: For instance, the feature size in the first ResNet stage 1 is 56 × 56 .

Original: Then the possible choices for S are { 1 , 2 , 4 , 7 , 8 , 14 , 28 , 56 } .
Revised: Then the possible choices for S are { 1 , 2 , 4 , 7 , 8 , 14 , 28 , 56 } .

Original: The mask M coarse is then upsampledto the size of H × W .
Revised: The mask M coarse is then upsampledto the size of H × W .

Original: Notably, S = 1 means that the granularity is still at the pixel level as previousmethods [4, 28, 32].
Revised: Notably, S = 1 means that the granularity is still at the pixel level as previousmethods [4, 27, 31].

Original: In this paper, the other extreme situation ( S = 56 ) is not considered, when themasker directly determines whether to skip the whole block (i.e. layer skipping [27, 29]).
Revised: In this paper, the other extreme situation ( S = 56 ) is not considered, when themasker directly determines whether to skip the whole block (i.e. layer skipping [26, 28]).

Original: The maskeris composed of a pooling layer followed by a 1 × 1 convolution.
Revised: The maskeris composed of a pooling layer followed by a 1 × 1 convolution.

Original: Differences to existing works.
Revised: Differences to existing works.

Original: Without using the interpolation operation [32] or the carefullydesigned two-branch structure [7], the proposed block architecture is simple and sufficiently generalto be plugged into most backbones with minimal modification.
Revised: Without using the interpolation operation [31] or the carefullydesigned two-branch structure [7], the proposed block architecture is simple and sufficiently generalto be plugged into most backbones with minimal modification.

Original: Our formulation is mostly similar tothat in [28], which could be viewed as a variant of our method with the spatial granularity S =1 for allblocks.
Revised: Our formulation is mostly similar tothat in [27], which could be viewed as a variant of our method with the spatial granularity S =1 for allblocks.

Original: Instead of performing spatially adaptive inference at the finest pixel level, our granularity S isoptimized under the guidance of our latency prediction model (details are presented in the following
Revised: Instead of performing spatially adaptive inference at the finest pixel level, our granularity S isoptimized under the guidance of our latency prediction model (details are presented in the following

Original: Sec. 4.2) to achieve realistic speedup on target computing platforms.
Revised: Sec. 4.2) to achieve realistic speedup on target computing platforms.

Original: 3.3 Latency prediction model
Revised: 3.3 Latency prediction model

Original: As stated before, it is laborious to evaluate the latency of dynamic operators on different hardwareplatforms.
Revised: As stated before, it is laborious to evaluate the latency of dynamic operators on different hardwareplatforms.

Original: To efficiently seek  preferable granularity settings on arbitrary hardware devices, wepropose a latency prediction model G , which can directly predict the delay of executing dynamicoperators on any target devices.
Revised: To efficiently seek for preferable granularity settings on arbitrary hardware devices, wepropose a latency prediction model G , which can directly predict the delay of executing dynamicoperators on any target devices.

Original: For a spatial-wise dynamic convolutional block, the latency predictor
Revised: For a spatial-wise dynamic convolutional block, the latency predictor

Original: G takes the hardware properties H , the layer parameters P , the spatial granularity S, and the activationrate r as input and predicts the latency ℓ of a dynamic convolutional block: ℓ = G ( H , P , S, r ) .
Revised: G takes the hardware properties H , the layer parameters P , the spatial granularity S and the activationrate r as input, and predicts the latency ℓ of a dynamic convolutional block: ℓ = G ( H , P , S, r ) .

Original: Hardware modeling. We model a hardware device as multiple processing engines (PEs), and parallelcomputation can be executed on these PEs.
Revised: We model a hardware device as multiple processing engines (PEs), and parallel computations can beexecuted on the PEs.

Revised: For example, the processor engine of Nvidia GPU is streaming multiprocessors(SMs), which executes multiple threads in parallel, and the processing engine of neural networkaccelerators executes the matrix multiplication with high parallelism.

Original: As shown in Figure 3, we model the memory system as athree-level structure [9]: 1) off-chip memory, 2) on-chip global memory, and 3) memory in PE.
Revised: As shown in Figure 3, wemodel the memory system as a three-level structure: 1) off-chip memory, 2) on-chip global memory,and 3) memory in PE.

Original: Sucha hardware model enables us to accurately predict thecost on both data movement and computation .
Revised: We simulate the data movement and considers the efficiency of non-contiguousmemory accesses.

Original: Latency prediction.
Revised: The on-chip memory is much faster than off-chip memory, but it has a limitedsize.

Original: When simulating the data movement procedure, the efficiency of non-contiguousmemory accesses under different granularity S settings is considered.
Revised: We cache the data on on-chip memory to improve performance. The computation of thedynamic operator should be distributed to different PEs.

Original: As for the computation latency,it is important toadopt a proper scheduling strategy to increase the parallelism of computation. Therefore, we search for the optimal scheduling (the configuration of tiling and in-PE parallelcomputing) of  dynamic operations to maximize the utilization of hardware resources.A moredetailed description of our latency prediction model is presented in the supplementary material.
Revised: The predictor performs tiling on the outputdimensions and assigns each tile to a PE. We searches for the optimal scheduling (the configurationof tiling and in-PE parallel computing) of each dynamic operation given the hardware parametersand layer configuration to maximize the utilization of hardware resources.

Original: Empirical validation. We take the first  block in ResNet-101 as an example and vary the activationrate r to evaluate the performance of our  prediction model.
Revised: We take the first dynamicconvolutional block in ResNet-101 as an example, and vary the activation rate r to evaluate theperformance of our latency prediction model.

Original: The comparison between our predictionsand the real testing latency on the Nvidia V100 GPU is illustrated in Figure 4, from which wecanobserve that our predictor can  accurately estimate the real latency in a wide range of activation rates.
Revised: The comparison between our predictions and the realtesting latency on the Nvidia V100 GPU is illustrated in Figure 4, from which we can observe thatour predictor is able to accurately estimate the real latency in a wide range of activation rates.

Original: O ff - c h i p
Revised: O ff - c h i p

Original: M e m o r y ( G DD R /
Revised: M e m o r y ( G DD R /

Original: DD R / H B M )
Revised: DD R / H B M )

Original: O n - c h i p
Revised: O n - c h i p

Original: M e m o r y ( LL C / G l o
Revised: M e m o r y ( LL C / G l o

Original: [Figure]
Revised: [Figure]

Original: a l B u ff e r )
Revised: a l B u ff e r )

Original: [Figure]
Revised: [Figure]

Original: 1.0 Activation rate ( r ) 

Revised: 1.0 Activation rate ( r )

Original: L a t e n c y ( s )
Revised: L a t e n c y ( s )

Original: PredictedReal
Revised: PredictedReal

Original: 3.4 Implementation details
Revised: 3.4 Implementation details

Original: We use general optimization methods like fusing activation functions and batch normalizationlayers into convolution layers.
Revised: We use general optimization methods like fusing activation functions and batch normalizationlayers into convolution layers.

Original: We also optimize the specific operators in our spatial-wise dynamicconvolutional blocks as follows (see also Figure 2 for an overview).
Revised: We also optimize the specific operators in our spatial-wise dynamicconvolutional blocks as follows (see also Figure 2 for an overview).

Original: Fusing the masker and the first convolution.
Revised: Fusing the masker and the first convolution.

Original: As mentioned in Sec.
Revised: As mentioned in Sec.

Original: 3.1, the masker in each blockconsumes very little computation, but it takes the whole feature map as input.
Revised: 3.1, the masker in each blockconsumes very little computation, but it takes the whole feature map as input.

Original: Therefore, it is amemory-bounded operation (the inference time is mainly spent on memory access).
Revised: Therefore, it is amemory-bounded operation (the inference time is mainly spent on memory access).

Original: Since the maskerand the first convolution in the block share the same input, there is an opportunity to fuse these twooperations to avoid the repeated access of the input data.
Revised: Since the maskerand the first convolution in the block share the same input, there is an opportunity to fuse these twooperations to avoid the repeated access of the input data.

Original: Note that a spatial-wise dynamic convolutionrequires the output of the masker.
Revised: Note that a spatial-wise dynamic convolutionrequires the output of the masker.

Original: If we fuse the two layers, the first convolution will be changed to astatic operation, which may increase the inference latency.
Revised: If we fuse the two layers, the first convolution will be changed to astatic operation, which may increase the inference latency.

Original: There exists a threshold of activation rater th , when r > r th , the overall latency can be reduced.
Revised: There exists a threshold of activation rater th , when r > r th , the overall latency can be reduced.

Original: We decide whether to fuse them according tothe average activation rate.
Revised: We decide whether to fuse them according tothe average activation rate.

Original: See more details in the supplementary material.
Revised: See more details in the supplementary material.

Original: Fusing the gather operation and the dynamic convolution.
Revised: Fusing the gather operation and the dynamic convolution.

Original: Traditional approaches first gather theinput pixels of the first dynamic convolution in a block (see Figure 1 (b)).
Revised: Traditional approaches first gather theinput pixels of the first dynamic convolution in a block (see Figure 1 (b)).

Original: The gather operation isalso a memory-bounded operation.
Revised: The gather operation isalso a memory-bounded operation.

Original: Furthermore, when the size of the convolution kernel exceeds1 × 1, the area of input patches may overlap, resulting in repeated memory load/store.
Revised: Furthermore, when the size of the convolution kernel exceeds1 × 1, the area of input patches may overlap, resulting in repeated memory load/store.

Original: We fuse thegather operation into the dynamic convolution to reduce the memory access.
Revised: We fuse thegather operation into the dynamic convolution to reduce the memory access.

Original: Fusing the scatter operation and the add operation.
Revised: Fusing the scatter operation and the add operation.

Original: Traditional approaches scatter the outputpixels of the last dynamic convolution, and then execute the element-wise addition (see Figure 1 (b)).
Revised: Traditional approaches scatter the outputpixels of the last dynamic convolution, and then execute the element-wise addition (see Figure 1 (b)).

Original: We fuse these two operators to reduce the memory access.
Revised: We fuse these two operators to reduce the memory access.

Original: The ablation study in Sec.
Revised: The ablation study in Sec.

Original: 4.4 validatesthe effectiveness of the proposed fusing methods.
Revised: 4.4 validatesthe effectiveness of the proposed fusing methods.

Original: 3.5 Training
Revised: 3.5 Training

Original: Optimization of non-differentiable maskers.
Revised: Optimization of non-differentiable maskers.

Original: The masker modules are required to produce binaryvalued spatial masks for making discrete decisions, and cannot be directly optimized with backpropagation.
Revised: The masker modules are required to produce binaryvalued spatial masks for making discrete decisions, and cannot be directly optimized with backpropagation.

Original: Following [32, 28, 7], we adopt straight-through Gumbel Softmax [15, 21] to train thenetwork in an end-to-end fashion.
Revised: Following [31, 27, 7], we adopt straight-through Gumbel Softmax [14, 20] to train thenetwork in an end-to-end fashion.

Original: Specifically, let
Revised: Specifically, let

Original: ˜ M ∈ R H
Revised: ˜ M ∈ R H

Original: × W × 2 denote the output of the maskgenerator.
Revised: × W × 2 denote the output of the maskgenerator.

Original: The decisions are obtained with the argmax function during inference.
Revised: The decisions are obtained with the argmax function during inference.

Original: In the trainingphase, a differentiable approximation is defined by replacing the argmax operation with a Softmax:
Revised: In the trainingphase, a differentiable approximation is defined by replacing the argmax operation with a Softmax:

Original: [Equation]
Revised: [Equation]

Original: where τ is the Softmax temperature.
Revised: where τ is the Softmax temperature.

Original: Following the common practice [28, 7], we let τ decreaseexponentially from 5.0 to 0.1 in training to facilitate the optimization of maskers.
Revised: Following the common practice [27, 7], we let τ decreaseexponentially from 5.0 to 0.1 in training to facilitate the optimization of maskers.

Original: Training objective.
Revised: Training objective.

Original: The FLOPs of each spatial-wise dynamic convolutional block can be calculatedbased on our defined activation rate r [28].
Revised: The FLOPs of each spatial-wise dynamic convolutional block can be calculatedbased on our defined activation rate r [27].

Original: Then we can obtain the FLOPs of the overall dynamicnetwork F dyn .
Revised: Then we can obtain the FLOPs of the overall dynamicnetwork F dyn .

Original: Let F stat denotes the FLOPs of its static counterpart.
Revised: Let F stat denotes the FLOPs of its static counterpart.

Original: We optimize their ratio toapproximate a target 0 < t < 1 : L FLOPs =
Revised: We optimize their ratio toapproximate a target 0 < t < 1 : L FLOPs =

Original: ( F dyn F stat − t ) 2 .
Revised: ( F dyn F stat − t ) 2 .

Original: In addition, we define loss item L bounds asin [28] to constrain the upper bound and the lower bound of activation rates in early training epochs.
Revised: In addition, we define loss item L bounds asin [27] to constrain the upper bound and the lower bound of activation rates in early training epochs.

Original: We further propose to leverage the static counterparts of our dynamic networks as “teachers” to guidethe optimization procedure.
Revised: We further propose to leverage the static counterparts of our dynamic networks as “teachers” to guidethe optimization procedure.

Original: Let y and y ′
Revised: Let y and y ′

Original: denote the output logits of a dynamic model (“student”)and its “teacher”, respectively.
Revised: denote the output logits of a dynamic model (“student”)and its “teacher”, respectively.

Original: Our final loss can be written as
Revised: Our final loss can be written as

Original: [Equation]
Revised: [Equation]

Original: where L task represents the task-related loss, e.g., cross-entropy loss in image classification.
Revised: where L task represents the task-related loss, e.g. cross-entropy loss in image classification.

Original: KL( ·||· )denotes the Kullback–Leibler divergence, and α, β are the coefficients balancing these items.
Revised: KL( ·||· )denotes the Kullback–Leibler divergence, and α, β are the coefficients balancing these items.

Original: We useσ to denote the log-Softmax function, and T is the temperature for computing KL-divergence.
Revised: We useσ to denote the log-Softmax function, and T is the temperature for computing KL-divergence.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 0.6 0.8 1.0 Activation rate ( r ) 0.
Revised: 0.6 0.8 1.0 Activation rate ( r ) 0.

Original: L a t e n c y r a t i o ( r )
Revised: L a t e n c y r a t i o ( r )

Original: S =1 S =2 S = 0.0 0.2 0.4 0.

Revised: S =1 S =2 S = 0.0 0.2 0.4 0.

Original: 0.8 1.0 Activation rate ( r ) 0.
Revised: 0.8 1.0 Activation rate ( r ) 0.

Original: L a t e n c y r a t i o ( r )
Revised: L a t e n c y r a t i o ( r )

Original: S =
Revised: S =

Original:  Experiments
Revised:  Experiments

Original: In this section, we first introduce the experiment settings in Sec. 4.1.
Revised: In this section, we first introduce the experiment settings in Sec. 4.1.

Original: Then the latency of differentgranularity settings are analyzed in Sec.
Revised: Then the latency of differentgranularity settings are analyzed in Sec.

Original: The performance of our LASNet on ImageNet is furtherevaluated in Sec. 4.3, followed by the ablation studies in Sec. 4.4.
Revised: The performance of our LASNet on ImageNet is furtherevaluated in Sec. 4.3, followed by the ablation studies in Sec. 4.4.

Original: Visualization results are illustratedin Sec. 4.5, and we finally validate our method on the object detection task (Sec. 4.6).
Revised: Visualization results are illustratedin Sec. 4.5, and we finally validate our method on the object detection task (Sec. 4.6).

Original: The results onthe instance segmentation task are presented in the supplementary material.

Original: For simplicity, we add“LAS-” as a prefix before model names to denote our LASNet, e.g., LAS-ResNet-50.
Revised: For simplicity,we add “LAS-” as a prefix before model names to denote our LASNet, e.g. LAS-ResNet-50.

Original: 4.1 Experiment settings
Revised: 4.1 Experiment settings

Original: Latency prediction.
Revised: Latency prediction.

Original: Various types of hardware platforms are tested, including a server GPU (Tesla
Revised: Various types of hardware platforms are tested, including a server GPU (Tesla

Original: V100), a desktop GPU (GTX1080) and IoT devices (e.g., Nvidia Nano and Jetson TX2).
Revised: V100), a desktop GPU (GTX1080) and IoT devices (e.g. Nvidia Nano and Jetson TX2).

Original: Themajor properties considered by our latency prediction model include the number of processingengines (#PE), the floating-point computation in a processing engine (#FP32), the frequency and thebandwidth.
Revised: Themajor properties considered by our latency prediction model include the number of processingengines (#PE), the floating-point computation in a processing engine (#FP32), the frequency and thebandwidth.

Original: It can be observed that the server GPUs generally have a larger #PE than the IoT devices.
Revised: It can be observed that the server GPUs generally have a larger #PE than the IoT devices.

Original: The batch size is set as 1 for all dynamic models and computing platforms.
Revised: The batch size is set as 1 for all dynamic models and computing platforms.

Original: Image classification.
Revised: Image classification.

Original: The image classification experiments are conducted on the ImageNet [3]dataset.
Revised: The image classification experiments are conducted on the ImageNet [3]dataset.

Original: Following [28], we initialize the backbone parameter from a pre-trained checkpoint 2 , andfinetune the whole network for 100 epochs with the loss function in Eq.
Revised: Following [27], we initialize the backbone parameter from a pre-trained checkpoint 2 , andfinetune the whole network for 100 epochs with the loss function in Eq.

Original: We fix α = 10 , β = 0 .
Revised: We fix α = 10 , β = 0 .

Original: 5and T = 4 . 

Revised: 5and T = 4 .

Original: for all dynamic models.
Revised: for all dynamic models.

Original: More details are provided in the supplementary material.
Revised: More details are provided in the supplementary material.

Original: 4.2 Latency prediction results
Revised: 4.2 Latency prediction results

Original: In this subsection, we present the latency prediction results of the spatial-wise dynamic convolutionalblocks in two different models: LAS-ResNet-101 [8] (on V100) and LAS-RegNetY-800MF [22] (on
Revised: In this subsection, we present the latency prediction results of the spatial-wise dynamic convolutionalblocks in two different models: LAS-ResNet-101 [8] (on V100) and LAS-RegNetY-800MF [21] (on

Original: TX2).
Revised: TX2).

Original: All the blocks have the bottleneck structure with different channel numbers and convolutiongroups, and the RegNetY is equipped with Squeeze-and-Excitation (SE)
Revised: All the blocks have the bottleneck structure with different channel numbers and convolutiongroups, and the RegNetY is equipped with Squeeze-and-Excitation (SE)

Original: [11] modules.
Revised: [10] modules.

Original: We first define ℓ dyn as the latency of a spatial-wise dynamic convolutional block, and ℓ stat as that ofa static block without a masker.
Revised: We first define ℓ dyn as the latency of a spatial-wise dynamic convolutional block, and ℓ stat as that ofa static block without a masker.

Original: Their ratio is denoted as r ℓ = ℓ dyn ℓ stat .
Revised: Their ratio is denoted as r ℓ = ℓ dyn ℓ stat .

Original: We investigate the relationshipbetween r ℓ and the activation rate r (cf.
Revised: We investigate the relationshipbetween r ℓ and the activation rate r (cf.

Original: Sec. 3.5) for different granularity settings.
Revised: Sec. 3.5) for different granularity settings.

Original: The results in
Revised: The results in

Original: Figure 5 demonstrate that: 1) pixel-level spatially adaptive inference ( S =1) cannot always improvethe practical efficiency.
Revised: Figure 5 demonstrate that: 1) pixel-level spatially adaptive inference ( S =1) cannot always improvethe practical efficiency.

Original: Such fine-grained adaptive inference is adopted by most previous works[28, 32], and our result can explain the reason why they can only achieve realistic speedup on lesspowerful CPUs [32] or specialized devices [2]; 2) a proper granularity S > 1 effectively alleviates thisproblem; 3) the advantage of coarse-grained spatially adaptive inference ( S > 1 ) is more significant
Revised: Such fine-grained adaptive inference is adopted by most previous works[27, 31], and our result can explain the reason why they can only achieve realistic speedup on lesspowerful CPUs [31] or specialized devices [2]; 2) a proper granularity S > 1 effectively alleviates thisproblem; 3) the advantage of coarse-grained spatially adaptive inference ( S > 1 ) is more significanton the server GPU with a larger #PE, as the finest granularity ( S = 1 ) brings considerable overheadon the memory access and is not friendly to parallel computation.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: on the server GPU with a larger #PE, as the finest granularity ( S = 1 ) brings considerable overheadon the memory access and is not friendly to parallel computation.

Original: The latency prediction results are further used to seek for a preferable granularity setting for the first3 stages (we fix S = 1 for the last stage, where the feature resolution is 7 × 7 ).
Revised: The latency prediction results are further used to seek for a preferable granularity setting for the first3 stages (we fix S = 1 for the last stage, where the feature resolution is 7 × 7 ).

Original: Therefore, we plotthe relationship between r ℓ and S in Figure 6.
Revised: Therefore, we plotthe relationship between r ℓ and S in Figure 6.

Original: It can be observed that: 1) r ℓ generally decreaseswith S increasing for a given r on V100; 2) an overly large S brings insignificant improvement on
Revised: It can be observed that: 1) r ℓ generally decreaseswith S increasing for a given r on V100; 2) an overly large S brings insignificant improvement on

Original: V100, and even harms the practical efficiency on the less powerful TX2.
Revised: V100, and even harms the practical efficiency on the less powerful TX2.

Original: Therefore, we can simply set
Revised: Therefore, we can simply set

Original: S net =4-4-2-1 for the 4 stages in a network on TX2.
Revised: S net =4-4-2-1 for the 4 stages in a network on TX2.

Original: As for the server GPU V100, S net =8-4-7-1 will beappropriate for realistic speedup.
Revised: As for the server GPU V100, S net =8-4-7-1 will beappropriate for realistic speedup.

Original: The accuracy-latency plots in Figure 7 also validate this observation.
Revised: The accuracy-latency plots in Figure 7 also validate this observation.

Original: More results of our latency prediction model are presented in the supplementary material.
Revised: More results of our latency prediction model are presented in the supplementary material.

Original: 4.3 ImageNet classification results
Revised: 4.3 ImageNet classification results

Original: We now empirically evaluate our proposed LASNet on the ImageNet dataset.
Revised: We now empirically evaluate our proposed LASNet on the ImageNet dataset.

Original: The network performance is measured in terms of the trade-off between classification accuracy and inference efficiency.
Revised: The network performance is measured in terms of the trade-off between classification accuracy and inference efficiency.

Original: Both theoretical (i.e. FLOPs) and practical efficiency (i.e. latency) are tested in our experiments.
Revised: Both theoretical (i.e. FLOPs) and practical efficiency (i.e. latency) are tested in our experiments.

Original: 4.3.1 Standard baseline comparison: ResNets
Revised: 4.3.1 Standard baseline comparison: ResNets

Original: We first establish our LASNet based on the standard ResNets [8].
Revised: We first establish our LASNet based on the standard ResNets [8].

Original: Specifically, we build LAS-ResNet50 and LAS-ResNet-101 by plugging our maskers in the two common ResNet structures.
Revised: Specifically, we build LAS-ResNet50 and LAS-ResNet-101 by plugging our maskers in the two common ResNet structures.

Original: Compared baselines include various types of dynamic inference approaches: 1) layer skipping(SkipNet [29] and Conv-AIG [27]); 2) channel skipping (BAS [1]); and 3) pixel-level spatial-wisedynamic network (DynConv [28]).
Revised: Compared baselines include various types of dynamic inference approaches: 1) layer skipping(SkipNet [28] and Conv-AIG [26]); 2) channel skipping (BAS [1]); and 3) pixel-level spatial-wisedynamic network (DynConv [27]).

Original: For our LASNet, we compare various settings of the spatialgranularity S net .
Revised: For our LASNet, we compare various settings of the spatialgranularity S net .

Original: We set training targets (cf.
Revised: We set training targets (cf.

Original: Sec. 3.5) t ∈{ 0 , 4 , 0 . 5 , 0 . 6 , 0 . 7 } for our dynamic modelsto evaluate their performance in different sparsity regimes.
Revised: Sec. 3.5) t ∈{ 0 , 4 , 0 . 5 , 0 . 6 , 0 . 7 } for our dynamic modelsto evaluate their performance in different sparsity regimes.

Original: We apply the same operator fusion(Sec. 3.4) for both our models and the compared baselines [27, 28] for fair comparison.
Revised: We apply the same operator fusion(Sec. 3.4) for both our models and the compared baselines [26, 27] for fair comparison.

Original: Results are presented in Figure 7a.
Revised: Results are presented in Figure 7a.

Original: On the left we plot the relationship of accuracy v.s. FLOPs.
Revised: On the left we plot the relationship of accuracy v.s. FLOPs.

Original: Itcan be observed that our LAS-ResNets with different granularity settings significantly outperformthe competing dynamic neural networks.
Revised: Itcan be observed that our LAS-ResNets with different granularity settings significantly outperformthe competing dynamic neural networks.

Original: Surprisingly, coarse-grained spatially adaptive inference( S net =4-4-2-1 and S net =8-4-7-1 for the 4 stages) can achieve even higher accuracy when consumingsimilar FLOPs on ResNets, despite the sacrificed flexibility compared to S net =1-1-1-1.
Revised: Surprisingly, coarse-grained spatially adaptive inference( S net =4-4-2-1 and S net =8-4-7-1 for the 4 stages) can achieve even higher accuracy when consumingsimilar FLOPs on ResNets, despite the sacrificed flexibility compared to S net =1-1-1-1.

Original: We compare the practical latency of three granularity settings in Figure 7a (middle on TX2 andright on V100) based on our latency prediction model.
Revised: We compare the practical latency of three granularity settings in Figure 7a (middle on TX2 andright on V100) based on our latency prediction model.

Original: We can witness that although they achievecomparable theoretical efficiency (Figure 7a left), larger S is more hardware-friendly comparedto the finest granularity.
Revised: We can witness that although they achievecomparable theoretical efficiency (Figure 7a left), larger S is more hardware-friendly comparedto the finest granularity.

Original: For example, the inference latency of LAS-ResNet-101 ( S net =1-1-1-1)is significantly higher than the ResNet-101 baseline on V100 (Figure 7a right), even though itstheoretical computation is much smaller than that of the static model.However, larger granularities [Figure] [Table]
Revised: For example, the inference latency of LAS-ResNet-101 ( S net =1-1-1-1)is significantly higher than the ResNet-101 baseline on V100 (Figure 7a right), even though itstheoretical computation is much smaller than that of the static model.

Original: [Figure] [Table] [Figure] [Table]
Revised: However, larger granularities( S net =4-4-2-1 and S net =8-4-7-1) can effectively improve the inference latency due to its lower burdenon the memory access.

Original: [Figure]

Original: [Table] [Figure] [Table] [Figure]
Revised: The superiority of coarse-grained spatially adaptive inference diminishes (but

Original: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table][Table] [Figure] [Table]
Revised: [Table]

Original: [Figure]

Original: [Table][Figure] [Table] [Figure]
Revised: [Table]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]
Revised: [Figure]

Original: ( S net =4-4-2-1 and S net =8-4-7-1) can effectively improve the inference latency due to its lower burdenon the memory access.

Original: The superiority of coarse-grained spatially adaptive inference diminishes (butstill exists) on the less powerful IoT device, Nvidia Jetson TX2 (Figure 7a right).
Revised: still exists) on the less powerful IoT device, Nvidia Jetson TX2 (Figure 7a right).

Original: The advantage ismore significant on the Tesla V100 GPU, because of its large number of processing engines (#PE).
Revised: The advantage ismore significant on the Tesla V100 GPU, because of its large number of processing engines (#PE).

Original: Moreover, the realistic speedup ratio r ℓ is more close to the theoretical FLOPs ratio target t on
Revised: Moreover, the realistic speedup ratio r ℓ is more close to the theoretical FLOPs ratio target t on

Original: TX2, because the latency is computation-bounded on such IoT devices.
Revised: TX2, because the latency is computation-bounded on such IoT devices.

Original: Interestingly, the optimalsettings for the two different hardware platforms are different ( S net =4-4-2-1 is superior on TX2,while S net =8-4-7-1 leads to higher efficiency on V100).
Revised: Interestingly, the optimalsettings for the two different hardware platforms are different ( S net =4-4-2-1 is superior on TX2,while S net =8-4-7-1 leads to higher efficiency on V100).

Original: Remarkably, the latency of ResNet-101could be reduced by 23% and 45% on V100 and TX2 respectively without sacrificing the accuracy.
Revised: Remarkably, the latency of ResNet-101could be reduced by 23% and 45% on V100 and TX2 respectively without sacrificing the accuracy.

Original: The classification accuracy is increased by 1.9% with similar inference efficiency.
Revised: The classification accuracy is increased by 1.9% with similar inference efficiency.

Original: 4.3.2 Lightweight baseline comparison: RegNets
Revised: 4.3.2 Lightweight baseline comparison: RegNets

Original: We further evaluate our LASNet in lightweight CNN architectures, i.e. RegNets-Y [22].
Revised: We further evaluate our LASNet in lightweight CNN architectures, i.e. RegNets-Y [21].

Original: Two differentsized models are tested: RegNetY-400MF and RegNetY-800MF.
Revised: Two differentsized models are tested: RegNetY-400MF and RegNetY-800MF.

Original: Compared baselines include othertypes of efficient models, e.g., MobileNets-v2 [25], ShuffletNets-v2 [20] and CondenseNets [13].
Revised: Compared baselines include othertypes of efficient models, e.g. MobileNets-v2 [24], ShuffletNets-v2 [19] and CondenseNets [12].

Original: The results are presented in Figure 7b.
Revised: The results are presented in Figure 7b.

Original: The x-axis for the three sub-figures are the FLOPs, the latencyon TX2, and the latency on the Nvidia nano GPU, respectively.
Revised: The x-axis for the three sub-figures are the FLOPs, the latencyon TX2, and the latency on the Nvidia nano GPU, respectively.

Original: We can observe that our methodoutperforms various types of static models in terms of the trade-off between accuracy and efficiency.
Revised: We can observe that our methodoutperforms various types of static models in terms of the trade-off between accuracy and efficiency.

Original: More results on image classification are provided in the supplementary material.
Revised: More results on image classification are provided in the supplementary material.

Original: 4.4 Ablation studies
Revised: 4.4 Ablation studies

Original: Operator fusion.
Revised: Operator fusion.

Original: We first conduct ablation studies to investigate the effect of our operator fusionintroduced in Sec.
Revised: We first conduct ablation studies to investigate the effect of our operator fusionintroduced in Sec.

Original: One convolutional block in the first stage of a LAS-ResNet-101 ( S =4, r =0.6)is tested.
Revised: One convolutional block in the first stage of a LAS-ResNet-101 ( S =4, r =0.6)is tested.

Original: We summarize the results in Table 1.
Revised: We summarize the results in Table 1.

Original: It can be observed that every step of operator fusionbenefits the practical latency of a block, as the overhead on memory access is effectively reduced.
Revised: It can be observed that every step of operator fusionbenefits the practical latency of a block, as the overhead on memory access is effectively reduced.

Original: [Table]
Revised: [Table]

Original: More granularities settings.
Revised: More granularities settings.

Original: We test various granularity settings on LAS-ResNet-101 to examinethe effects of S in different stages.
Revised: We test various granularity settings on LAS-ResNet-101 to examinethe effects of S in different stages.

Original: The results onthe Tesla-V100 GPU are presented in Figure 8.
Revised: The results onthe Tesla-V100 GPU are presented in Figure 8.

Original: Itcan be found that the finest granularity ( S net =11-1-1) leads to substantial inefficiency despite thereduced FLOPs (cf.
Revised: Itcan be found that the finest granularity ( S net =11-1-1) leads to substantial inefficiency despite thereduced FLOPs (cf.

Original: Figure 7a left).
Revised: Figure 7a left).

Original: Coarse-grainedspatially adaptive inference in the first two stages( S net =4-4-1-1) effectively reduces the inference latency.
Revised: Coarse-grainedspatially adaptive inference in the first two stages( S net =4-4-1-1) effectively reduces the inference latency.

Original: We further increase S in the third stage to 2 and 7, and this procedure consistently improvesthe realistic efficiency on the V100 GPU.
Revised: We further increase S in the third stage to 2 and 7, and this procedure consistently improvesthe realistic efficiency on the V100 GPU.

Original: It is worth noting that increasing the granularity S does [Figure] [Table] not always improve the inference efficiency on other computing devices.
Revised: It is worth noting that increasing the granularity S doesnot always improve the inference efficiency on other computing devices.

Original: For example, S net =4-4-2-1achieves a sweetspot on Nvidia Jetson TX2 (see Figure 7a middle). 
Revised: For example, S net =4-4-2-1achieves a sweetspot on Nvidia Jetson TX2 (see Figure 7a middle). [Figure] [Table]

Original: 4.5 Visualization
Revised: 4.5 Visualization

Original: We visualize the masks generated by our masker in the third block of a LAS-ResNet-101 ( S net =44-2-1) in Figure 9.
Revised: We visualize the masks generated by our masker in the 3rd block of a LAS-ResNet-101 ( S net =4-4-21) in Figure 9.

Original: The brilliant areas correspond to the locations of 1 elements in a mask, and thecomputation on the dimmed regions is skipped by our dynamic model.
Revised: The brilliant areas are corresponding to the locations of 1 elements in the mask, andthe computation on the dimmed regions are skipped by our dynamic model.

Original: It can be found that the maskeris trained to accurately locate the most task-related regions (even the tiny aircraft at the corner), whichhelps reduce the unnecessary computation on background areas.
Revised: It can be found that themasker is trained to accurately locate the most task-related regions (even the extremely small aircraftat the corner), which helps reduce the unnecessary computation on background areas.

Original: Moreover, these results suggest thatfor the first stage, the granularity S =4 is sufficiently flexible to recognize the important regions, anda win-wincan be achieved between  accuracy and  efficiency.
Revised: Moreover,these results suggest that for the first stage, the granularity S =4 is sufficiently flexible to recognize theimportant regions, and therefore achieves a win-win between the accuracy and the practical efficiency.

Original: Interestingly, the masker could selectsome objects that are not labeled for the sample, e.g., the flower beside the hummingbird and thehuman holding the camera.
Revised: Interestingly, the masker could select some objects that are not labeled for the sample, e.g. the flowerbeside the hummingbird and the human holding the camera.

Original: This suggests that our spatial-wise dynamic networks can automaticallyrecognize the regions with semantics, and their capability is not limited by the classification labels.
Revised: This suggests that our spatial-wisedynamic networks can automatically recognize the regions with semantics, and its capability is notlimited by the classification labels.

Original: This property is helpful in some downstream tasks, such as object detection (Sec. 4.6), which requiresdetecting multiple classes and objects in an image.
Revised: This property is helpful on some downstream tasks such as objectdetection (Sec. 4.6), which requires detecting multiple classes and objects in an image .

Original: 4.6 Object detection results
Revised: 4.6 Object detection results

Original: We further evaluate our LASNet on the COCO [19] object detection task.
Revised: We further evaluate our LASNet on the COCO [18] object detection task.

Original: The mean average precision(mAP), the average backbone FLOPs, and the average backbone latency on the validation set areused to measure the network performance.
Revised: The mean average precision(mAP), the average backbone FLOPs, and the average backbone latency on the validation set areused to measure the network performance.

Original: We test two commonly used detection frameworks: Faster
Revised: We test two commonly used detection frameworks: Faster

Original: R-CNN [24] with Feature Pyramid Network [17] and RetinaNet [18].
Revised: R-CNN [23] with Feature Pyramid Network [16] and RetinaNet [17].

Original: Thanks to the generality ofour method, we can conveniently replace the backbones with ours pre-trained on ImageNet, and thewhole models are finetuned on COCO with the standard setting for 12 epochs (see detailed setup inthe supplementary material).
Revised: Thanks to the generality ofour method, we can conveniently replace the backbones with ours pre-trained on ImageNet, and thewhole models are finetuned on COCO with the standard setting for 12 epochs (see detailed setup inthe supplementary material).

Original: The input images are resized to a short side of 800 and a long side notexceeding 1333.
Revised: The input images are resized to a short side of 800 and a long side notexceeding 1333.

Original: The results of our LAS-ResNet-101 with different S net settings are presented in
Revised: The results of our LAS-ResNet-101 with different S net settings are presented in

Original: Table 2.
Revised: Table 2.

Original: It could be observed that our LASNet can reduce the practical latency on GTX1080 and
Revised: It could be observed that our LASNet can reduce the practical latency on GTX1080 and

Original: TX2 by 27% and 45% respectively while improving the mAP of both detection frameworks.
Revised: TX2 by 27% and 45% respectively while improving the mAP of both detection frameworks.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we propose to build latency-aware spatial-wise dynamic networks (LASNet) under theguidance of a latency prediction model .
Revised: In this paper, we propose to build latency-aware spatial-wise dynamic networks (LASNet) under theguidance of a latency prediction model .

Original: By simultaneously considering the algorithm, the schedulingstrategy and the hardware properties, we can efficiently estimate the practical latency of spatialwise dynamic operators on arbitrary computing platforms.
Revised: By simultaneously considering the algorithm, the schedulingstrategy and the hardware properties, we can efficiently estimate the practical latency of spatialwise dynamic operators on arbitrary computing platforms.

Original: Based on the empirical analysis on therelationship between the latency and the granularity of spatially adaptive inference, we optimizeboth the algorithm and the scheduling strategies to achieve realistic speedup on many multi-coreprocessors, e.g., the Tesla V100 GPU and the Jetson TX2 GPU.
Revised: Based on the empirical analysis on therelationship between the latency and the granularity of spatially adaptive inference, we optimizeboth the algorithm and the scheduling strategies to achieve realistic speedup on many multi-coreprocessors, e.g. the Tesla V100 GPU and the Jetson TX2 GPU.

Original: Experiments on image classification,object detectionand instance segmentation tasks validate that the proposed method significantlyimproves the practical efficiency of deep CNNs, and outperforms various competing approaches.
Revised: Experiments on image classificationand object detection tasks validate that the proposed method significantly improves the practicalefficiency of deep CNNs, and outperforms various competing approaches.

Revised: The current limitationmight be that our latency-ware framework is only constructed for spatial-wise dynamic networks.

Revised: Support for more types of dynamic models (e.g. channel skipping) will be explored in the future.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.
Revised: Please read the checklist guidelines carefully for information onhow to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section 4.1 and Section 4.6,the public ImageNet and COCO datasets are cited.
Revised: [Yes] See Section 4.1 and Section 4.6,the public ImageNet and COCO datasets are cited.

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes](b) Did you describe the limitations of your work?

Original: [Yes] See Section 5.
Revised: [Yes] See Section 5.(c)

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See the
Revised: [Yes] See the

Original: supplementary material.
Revised: supplementary material.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A](b)
Revised: [N/A]

Original:  Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [No] The code will be released if the paper is accepted.
Revised: [No] The code willbe released if the paper is accepted.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes] Most important hyperparameters are given in the main paper, and
Revised: [Yes] Most important hyperparameters are given in the main paper, and

Original: more details are presented in the supplementary material.
Revised: more details are presented in the supplementary material.

Original: (c) Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experi-
Revised: g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [No] Since the experiment results on the large-scale dataset,
Revised: [No] Since the experiment results on the large-scale dataset,

Original: ImageNet and COCO, are generally stable, we perform several repeat experiments in
Revised: ImageNet and COCO, are generally stable, we perform several repeat experiments in

Original: the early exploring phase, and dismiss the error bars in our main results as in mostprevious works.
Revised: the early exploring phase, and dismiss the error bars in our main results as in most previous works.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] The details are presented in the
Revised: [Yes] The details are presented in the

Original: supplementary material.
Revised: supplementary material.

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] See Section 4.1.
Revised: [Yes] See Section 4.1.

Original: We use the torchvisionpre-trained models.
Revised: We use the torchvisionpre-trained models.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]