Original: Latency-aware Spatial-wise Dynamic Networks
Revised: Latency-aware Spatial-wise Dynamic Networks

Original: Abstract
Revised: Abstract

Original: Spatial-wise dynamic convolution has become a promising approach to improving the inference efﬁciency of deep networks.
Revised: Spatial-wise dynamic convolution has become a promising approach to improving the inference efficiency of deep networks.

Original: By allocating more computation to the most informative pixels, such an adaptive inference paradigm reduces the spatial redundancy in image features and saves a considerable amount of unnecessary computation.
Revised: By allocating more computation to the most informative pixels, such an adaptive inference paradigm reduces the spatial redundancy in image features and saves a considerable amount of unnecessary computation.

Original: However, the theoretical efﬁciency achieved by previous methods can hardly translate into a realistic speedup, especially on the multi-core processors ( e.g. GPUs).
Revised: However, the theoretical efficiency achieved by previous methods can hardly translate into a realistic speedup, especially on the multi-core processors ( e.g. GPUs).

Original: The key challenge is that the existing literature has only focused on designing algorithms with minimal computation , ignoring the fact that the practical latency can also be inﬂuenced by scheduling strategies and hardware properties .
Revised: The key challenge is that the existing literature has only focused on designing algorithms with minimal computation , ignoring the fact that the practical latency can also be influenced by scheduling strategies and hardware properties .

Original: To bridge the gap between theoretical computation and practical efﬁciency, we propose a latency-aware spatial-wise dynamic network (LASNet), which performs coarse-grained spatially adaptive inference under the guidance of a novel latency prediction model .
Revised: To bridge the gap between theoretical computation and practical efficiency, we propose a latency-aware spatial-wise dynamic network (LASNet), which performs coarse-grained spatially adaptive inference under the guidance of a novel latency prediction model .

Original: The latency prediction model can efﬁciently estimate the inference latency of dynamic networks by simultaneously considering algorithms, scheduling strategies, and hardware properties.
Revised: The latency prediction model can efficiently estimate the inference latency of dynamic networks by simultaneously considering algorithms, scheduling strategies, and hardware properties.

Original: We use the latency predictor to guide both the algorithm design and the scheduling optimization on various hardware platforms.
Revised: We use the latency predictor to guide both the algorithm design and the scheduling optimization on various hardware platforms.

Original: Experiments on image classiﬁcation, object detection and instance segmentation demonstrate that the proposed framework signiﬁcantly improves the practical inference efﬁciency of deep networks.
Revised: Experiments on image classification, object detection and instance segmentation demonstrate that the proposed framework significantly improves the practical inference efficiency of deep networks.

Original: For example, the average latency of a ResNet-101 on the ImageNet validation set could be reduced by 36% and 46% on a server GPU (Nvidia Tesla-V100) and an edge device (Nvidia
Revised: For example, the average latency of a ResNet-101 on the ImageNet validation set could be reduced by 36% and 46% on a server GPU (Nvidia Tesla-V100) and an edge device (Nvidia

Original: Jetson TX2 GPU) respectively without sacriﬁcing the accuracy.
Revised: Jetson TX2 GPU) respectively without sacrificing the accuracy.

Original: Code is available at https://github.com/LeapLabTHU/LASNet .
Revised: Code is available at https://github.com/LeapLabTHU/LASNet .

Original:  Introduction
Revised:  Introduction

Original: Dynamic neural networks [7] have attracted great research interests in recent years.
Revised: Dynamic neural networks [7] have attracted great research interests in recent years.

Original: Compared to static models [11, 17, 13, 23] which treat different inputs equally during inference, dynamic networks can allocate the computation in a data-dependent manner.
Revised: Compared to static models [11, 17, 13, 23] which treat different inputs equally during inference, dynamic networks can allocate the computation in a data-dependent manner.

Original: For example, they can conditionally skip the computation of network layers [15, 9, 32, 30] or convolutional channels [19, 1], or perform spatially adaptive inference on the most informative image regions ( e.g. the foreground areas)
Revised: For example, they can conditionally skip the computation of network layers [15, 9, 32, 30] or convolutional channels [19, 1], or perform spatially adaptive inference on the most informative image regions ( e.g. the foreground areas)

Original: Spatial-wise dynamic networks, which typically decide whether to compute each feature pixel with masker modules [5, 31, 35, 8] (Figure 1 (a)), have shown promising results in improving the inference efﬁciency of convolution neural networks (CNNs).
Revised: Spatial-wise dynamic networks, which typically decide whether to compute each feature pixel with masker modules [5, 31, 35, 8] (Figure 1 (a)), have shown promising results in improving the inference efficiency of convolution neural networks (CNNs).

Original: Despite the remarkable theoretical efﬁciency achieved by spatial-wise dynamic networks [5, 31, 35], researchers have found it challenging to translate the theoretical results into realistic speedup,
Revised: Despite the remarkable theoretical efficiency achieved by spatial-wise dynamic networks [5, 31, 35], researchers have found it challenging to translate the theoretical results into realistic speedup,

Original: [Figure]
Revised: [Figure]

Original: especially on some multi-core processors, e.g., GPUs [35, 3, 8].
Revised: especially on some multi-core processors, e.g., GPUs [35, 3, 8].

Original: The challenges are two-fold: 1) most previous approaches [5, 31, 35] perform spatially adaptive inference at the ﬁnest granularity : every pixel is ﬂexibly decided whether to be computed or not.
Revised: The challenges are two-fold: 1) most previous approaches [5, 31, 35] perform spatially adaptive inference at the finest granularity : every pixel is flexibly decided whether to be computed or not.

Original: Such ﬂexibility induces non-contiguous memory access [35] and requires specialized scheduling strategies (Figure 1 (b)); 2) the existing literature has only adopted the hardware-agnostic FLOPs (ﬂoating-point operations) as an inaccurate proxy for the efﬁciency, lacking latency-aware guidance on the algorithm design.
Revised: Such flexibility induces non-contiguous memory access [35] and requires specialized scheduling strategies (Figure 1 (b)); 2) the existing literature has only adopted the hardware-agnostic FLOPs (floating-point operations) as an inaccurate proxy for the efficiency, lacking latency-aware guidance on the algorithm design.

Original: For dynamic networks, the adaptive computation with sub-optimal scheduling strategies further enlarges the discrepancy between the theoretical FLOPs and the practical latency.
Revised: For dynamic networks, the adaptive computation with sub-optimal scheduling strategies further enlarges the discrepancy between the theoretical FLOPs and the practical latency.

Original: Note that it has been validated by previous works that the latency on CPUs has a strong correlation with FLOPs [8, 35].
Revised: Note that it has been validated by previous works that the latency on CPUs has a strong correlation with FLOPs [8, 35].

Original: Therefore, we mainly focus on the GPU platform in this paper, which is more challenging and less explored.
Revised: Therefore, we mainly focus on the GPU platform in this paper, which is more challenging and less explored.

Original: We address the above challenges by proposing a latency-aware spatial-wise dynamic network (LASNet).
Revised: We address the above challenges by proposing a latency-aware spatial-wise dynamic network (LASNet).

Original: Three key factors to the inference latency are considered: the algorithm , the scheduling strategy , and the hardware properties .
Revised: Three key factors to the inference latency are considered: the algorithm , the scheduling strategy , and the hardware properties .

Original: Given a target hardware device, we directly use the latency, rather than the FLOPs, to guide our algorithm design and scheduling optimization (Figure 1 (c)).
Revised: Given a target hardware device, we directly use the latency, rather than the FLOPs, to guide our algorithm design and scheduling optimization (Figure 1 (c)).

Original: Because the memory access pattern and the scheduling strategies in our dynamic operators differ from those in static networks, the libraries developed for static models ( e.g. cuDNN) are sub-optimal for the acceleration of dynamic models.
Revised: Because the memory access pattern and the scheduling strategies in our dynamic operators differ from those in static networks, the libraries developed for static models ( e.g. cuDNN) are sub-optimal for the acceleration of dynamic models.

Original: Without the support of libraries, each dynamic operator requires scheduling optimization, code optimization, compiling, and deployment for each device.
Revised: Without the support of libraries, each dynamic operator requires scheduling optimization, code optimization, compiling, and deployment for each device.

Original: Therefore, it is laborious to evaluate the network latency on different hardware platforms.
Revised: Therefore, it is laborious to evaluate the network latency on different hardware platforms.

Original: To this end, we propose a novel latency prediction model to efﬁciently estimate the realistic latency of a network by simultaneously considering the aforementioned three factors.
Revised: To this end, we propose a novel latency prediction model to efficiently estimate the realistic latency of a network by simultaneously considering the aforementioned three factors.

Original: Compared to the hardware-agnostic
Revised: Compared to the hardware-agnostic

Original: FLOPs, our predicted latency can better reﬂect the practical efﬁciency of dynamic models.
Revised: FLOPs, our predicted latency can better reflect the practical efficiency of dynamic models.

Original: Guided by this latency prediction model, we establish our latency-aware spatial-wise dynamic network (LASNet), which adaptively decides whether to allocate computation on feature patches instead of pixels [5, 31, 35] (Figure 2 top).
Revised: Guided by this latency prediction model, we establish our latency-aware spatial-wise dynamic network (LASNet), which adaptively decides whether to allocate computation on feature patches instead of pixels [5, 31, 35] (Figure 2 top).

Original: We name this paradigm as spatially adaptive inference at a coarse granularity .
Revised: We name this paradigm as spatially adaptive inference at a coarse granularity .

Original: While less ﬂexible than the pixel-level adaptive computation in previous works [5, 31, 35], it facilitates more contiguous memory access, beneﬁting the realistic speedup on hardware.
Revised: While less flexible than the pixel-level adaptive computation in previous works [5, 31, 35], it facilitates more contiguous memory access, benefiting the realistic speedup on hardware.

Original: The scheduling strategy and the implementation are further ameliorated for faster inference.
Revised: The scheduling strategy and the implementation are further ameliorated for faster inference.

Original: It is worth noting that LASNet is designed as a general framework in two aspects: 1) the coarsegrained spatially adaptive inference paradigm can be conveniently implemented in various CNN backbones, e.g., ResNets [11], DenseNets [17] and RegNets [25]; and 2) the latency predictor is an off-the-shell tool which can be directly used for various computing platforms ( e.g. server GPUs and edge devices).
Revised: It is worth noting that LASNet is designed as a general framework in two aspects: 1) the coarsegrained spatially adaptive inference paradigm can be conveniently implemented in various CNN backbones, e.g., ResNets [11], DenseNets [17] and RegNets [25]; and 2) the latency predictor is an off-the-shell tool which can be directly used for various computing platforms ( e.g. server GPUs and edge devices).

Original: We evaluate the performance of LASNet on multiple CNN architectures on image classiﬁcation, object detection, and instance segmentation tasks.
Revised: We evaluate the performance of LASNet on multiple CNN architectures on image classification, object detection, and instance segmentation tasks.

Original: Experiment results show that our LASNet improves the efﬁciency of deep CNNs both theoretically and practically.
Revised: Experiment results show that our LASNet improves the efficiency of deep CNNs both theoretically and practically.

Original: For example, the inference latency of ResNet-101 on ImageNet [4] is reduced by 36% and 46% on an Nvidia Tesla V100 GPU and an Nvidia Jetson TX2 GPU, respectively, without sacriﬁcing the accuracy.
Revised: For example, the inference latency of ResNet-101 on ImageNet [4] is reduced by 36% and 46% on an Nvidia Tesla V100 GPU and an Nvidia Jetson TX2 GPU, respectively, without sacrificing the accuracy.

Original: Moreover, the proposed method outperforms various lightweight networks in a low-FLOPs regime.
Revised: Moreover, the proposed method outperforms various lightweight networks in a low-FLOPs regime.

Original: Our main contributions are summarized as follows:
Revised: Our main contributions are summarized as follows:

Original: We propose LASNet, which performs coarse-grained spatially adaptive inference guided by the practical latency instead of the theoretical FLOPs.
Revised: We propose LASNet, which performs coarse-grained spatially adaptive inference guided by the practical latency instead of the theoretical FLOPs.

Original: To the best of our knowledge, LASNet is the ﬁrst framework that directly considers the real latency in the design phase of dynamic neural networks; 2.
Revised: To the best of our knowledge, LASNet is the first framework that directly considers the real latency in the design phase of dynamic neural networks; 2.

Original: We propose a latency prediction model, which can efﬁciently and accurately estimate the latency of dynamic operators by simultaneously considering the algorithm, the scheduling strategy, and the hardware properties; 3.
Revised: We propose a latency prediction model, which can efficiently and accurately estimate the latency of dynamic operators by simultaneously considering the algorithm, the scheduling strategy, and the hardware properties; 3.

Original: Experiments on image classiﬁcation and downstream tasks verify that our proposed LASNet can effectively improve the practical efﬁciency of different CNN architectures.
Revised: Experiments on image classification and downstream tasks verify that our proposed LASNet can effectively improve the practical efficiency of different CNN architectures.

Original:  Related works
Revised:  Related works

Original: Spatial-wise dynamic network is a common type of dynamic neural networks [7].
Revised: Spatial-wise dynamic network is a common type of dynamic neural networks [7].

Original: Compared to static models which treat different feature locations evenly during inference, these networks perform spatially adaptive inference on the most informative regions ( e.g. , foregrounds), and reduce the unnecessary computation on less important areas ( e.g. , backgrounds).
Revised: Compared to static models which treat different feature locations evenly during inference, these networks perform spatially adaptive inference on the most informative regions ( e.g. , foregrounds), and reduce the unnecessary computation on less important areas ( e.g. , backgrounds).

Original: Existing works mainly include three levels of dynamic computation: resolution level [36, 37], region level [33] and pixel level [5, 31, 35].
Revised: Existing works mainly include three levels of dynamic computation: resolution level [36, 37], region level [33] and pixel level [5, 31, 35].

Original: The former two generally manipulate the network inputs [33, 37] or require special architecture design [36].
Revised: The former two generally manipulate the network inputs [33, 37] or require special architecture design [36].

Original: In contrast, pixel-level dynamic networks can ﬂexibly skip the convolutions on certain feature pixels in arbitrary CNN backbones [5, 31, 35].
Revised: In contrast, pixel-level dynamic networks can flexibly skip the convolutions on certain feature pixels in arbitrary CNN backbones [5, 31, 35].

Original: Despite its remarkable theoretical efﬁciency, pixel-wise dynamic computation brings considerable difﬁculty to achieving realistic speedup on multi-core processors, e.g. , GPUs.
Revised: Despite its remarkable theoretical efficiency, pixel-wise dynamic computation brings considerable difficulty to achieving realistic speedup on multi-core processors, e.g. , GPUs.

Original: Compared to the previous approaches [5, 31, 35] which only focus on reducing the theoretical computation, we propose to directly use the latency to guide our algorithm design and scheduling optimization.
Revised: Compared to the previous approaches [5, 31, 35] which only focus on reducing the theoretical computation, we propose to directly use the latency to guide our algorithm design and scheduling optimization.

Original: Hardware-aware network design.
Revised: Hardware-aware network design.

Original: To bridge the gap between theoretical and practical efﬁciency of deep models, researchers have started to consider the real latency in the network design phase.
Revised: To bridge the gap between theoretical and practical efficiency of deep models, researchers have started to consider the real latency in the network design phase.

Original: There are two lines of works in this direction.
Revised: There are two lines of works in this direction.

Original: One directly performs speed tests on targeted devices, and summarizes some guidelines to facilitate hand-designing lightweight models [23].
Revised: One directly performs speed tests on targeted devices, and summarizes some guidelines to facilitate hand-designing lightweight models [23].

Original: The other line of work searches for fast models using the neural architecture search (NAS) technique [29, 34].
Revised: The other line of work searches for fast models using the neural architecture search (NAS) technique [29, 34].

Original: However, all existing works try to build static models, which have intrinsic computational redundancy by treating different inputs in the same way.
Revised: However, all existing works try to build static models, which have intrinsic computational redundancy by treating different inputs in the same way.

Original: However, speed tests for dynamic operators on different hardware devices can be very laborious and impractical.
Revised: However, speed tests for dynamic operators on different hardware devices can be very laborious and impractical.

Original: In contrast, our proposed latency prediction model can efﬁciently estimate the inference latency on any given computing platforms by simultaneously considering algorithm design, scheduling strategies and hardware properties.
Revised: In contrast, our proposed latency prediction model can efficiently estimate the inference latency on any given computing platforms by simultaneously considering algorithm design, scheduling strategies and hardware properties.

Original:  Methodology
Revised:  Methodology

Original: In this section, we ﬁrst introduce the preliminaries of spatially adaptive inference, and then demonstrate the architecture design of our LASNet.
Revised: In this section, we first introduce the preliminaries of spatially adaptive inference, and then demonstrate the architecture design of our LASNet.

Original: The latency prediction model is then explained, which guides the granularity settings and the scheduling optimization for LASNet.
Revised: The latency prediction model is then explained, which guides the granularity settings and the scheduling optimization for LASNet.

Original: We further present the implementation improvements for faster inference, followed by the training strategies.
Revised: We further present the implementation improvements for faster inference, followed by the training strategies.

Original: 3.1 Preliminaries
Revised: 3.1 Preliminaries

Original: Spatially adaptive inference.
Revised: Spatially adaptive inference.

Original: The existing spatial-wise dynamic networks are usually established by attaching a masker M in each convolutional block of a CNN backbone (Figure 1 (a)).
Revised: The existing spatial-wise dynamic networks are usually established by attaching a masker M in each convolutional block of a CNN backbone (Figure 1 (a)).

Original: Speciﬁcally, let x ∈
Revised: Specifically, let x ∈

Original: R
Revised: R

Original: H
Revised: H

Original: × W × C denote the input of a block, where H and W are the feature height and width, and
Revised: × W × C denote the input of a block, where H and W are the feature height and width, and

Original: C is the channel number.
Revised: C is the channel number.

Original: The masker M takes x as input, and generates a binary-valued spatial mask M = M ( x ) ∈ { 0 , 1 } H × W .
Revised: The masker M takes x as input, and generates a binary-valued spatial mask M = M ( x ) ∈ { 0 , 1 } H × W .

Original: Each element of M determines whether to perform convolution operations on the corresponding location of the output feature.
Revised: Each element of M determines whether to perform convolution operations on the corresponding location of the output feature.

Original: The unselected regions will be ﬁlled with the values from the input [5, 31] or obtained via interpolation [35].
Revised: The unselected regions will be filled with the values from the input [5, 31] or obtained via interpolation [35].

Original: We deﬁne the activation rate of a block as r = (cid:80) i,j M i,j H × W , representing the ratio of the calculated pixels.
Revised: We define the activation rate of a block as r = (cid:80) i,j M i,j H × W , representing the ratio of the calculated pixels.

Original: Scheduling strategy.
Revised: Scheduling strategy.

Original: During inference, the current scheduling strategy for spatial-wise dynamic convolutions generally involve three steps [26] (Figure 1 (b)): 1) gathering , which re-organizes the selected pixels (if the convolution kernel size is greater than 1 × 1 , the neighbors are also required) along the batch dimension; 2) computation , which performs convolution on the gathered input; and 3) scattering , which ﬁlls the computed pixels on their corresponding locations of the output feature.
Revised: During inference, the current scheduling strategy for spatial-wise dynamic convolutions generally involve three steps [26] (Figure 1 (b)): 1) gathering , which re-organizes the selected pixels (if the convolution kernel size is greater than 1 × 1 , the neighbors are also required) along the batch dimension; 2) computation , which performs convolution on the gathered input; and 3) scattering , which fills the computed pixels on their corresponding locations of the output feature.

Original: [Figure]
Revised: [Figure]

Original: Limitations.
Revised: Limitations.

Original: Compared to performing convolutions on the entire feature map, the aforementioned scheduling strategy reduces the computation while bringing considerable overhead to memory access due to the mask generation and the non-contiguous memory access.
Revised: Compared to performing convolutions on the entire feature map, the aforementioned scheduling strategy reduces the computation while bringing considerable overhead to memory access due to the mask generation and the non-contiguous memory access.

Original: Such overhead would increase the overall latency, especially when the granularity of dynamic convolution is at the ﬁnest pixel level.
Revised: Such overhead would increase the overall latency, especially when the granularity of dynamic convolution is at the finest pixel level.

Original: 3.2 Architecture design
Revised: 3.2 Architecture design

Original: Spatial granularity.
Revised: Spatial granularity.

Original: As mentioned above, pixel -level dynamic convolutions [5, 31, 35] raise substantial challenges to achieving realistic speedup on multi-core processors due to the non-contiguous memory access.
Revised: As mentioned above, pixel -level dynamic convolutions [5, 31, 35] raise substantial challenges to achieving realistic speedup on multi-core processors due to the non-contiguous memory access.

Original: To this end, we propose to optimize the granularity of spatially adaptive inference.
Revised: To this end, we propose to optimize the granularity of spatially adaptive inference.

Original: Speciﬁcally, take the commonly used bottleneck structure in [11] as an example, our coarse-grained spatial-wise dynamic convolutional block is illustrated in Figure 2.
Revised: Specifically, take the commonly used bottleneck structure in [11] as an example, our coarse-grained spatial-wise dynamic convolutional block is illustrated in Figure 2.

Original: Instead of directly producing a mask with the shape of H × W , we ﬁrst generate a low-resolution mask M coarse ∈ { 0 ,
Revised: Instead of directly producing a mask with the shape of H × W , we first generate a low-resolution mask M coarse ∈ { 0 ,

Original: 1 } HS × WS , where S is named as the spatial granularity .
Revised: 1 } HS × WS , where S is named as the spatial granularity .

Original: Each element in M coarse determines the computation of an S × S -sized feature patch.
Revised: Each element in M coarse determines the computation of an S × S -sized feature patch.

Original: For instance, the feature size in the ﬁrst ResNet stage 3 is 56 × 56 .
Revised: For instance, the feature size in the first ResNet stage 3 is 56 × 56 .

Original: Then the possible choices for S are { 1 , 2 , 4 , 7 , 8 , 14 , 28 , 56 } .
Revised: Then the possible choices for S are { 1 , 2 , 4 , 7 , 8 , 14 , 28 , 56 } .

Original: The mask M coarse is then upsampled to the size of H × W .
Revised: The mask M coarse is then upsampled to the size of H × W .

Original: Notably, S = 1 means that the granularity is still at the pixel level as previous methods [5, 31, 35].
Revised: Notably, S = 1 means that the granularity is still at the pixel level as previous methods [5, 31, 35].

Original: Note that the other extreme situation ( S = 56 ) is not considered in this paper, when the masker directly determines whether to skip the whole block ( i.e. layer skipping [30, 32]).
Revised: Note that the other extreme situation ( S = 56 ) is not considered in this paper, when the masker directly determines whether to skip the whole block ( i.e. layer skipping [30, 32]).

Original: Such an overly aggressive approach will lead to  considerable drop of accuracy, as we presented in
Revised: Such an overly aggressive approach will lead to a considerable drop of accuracy, as we presented in

Original: Appendix C.2.
Revised: Appendix C.2.

Original: The masker is composed of a pooling layer followed by a 1 × 1 convolution.
Revised: The masker is composed of a pooling layer followed by a 1 × 1 convolution.

Original: Differences to existing works.
Revised: Differences to existing works.

Original: Without using the interpolation operation [35] or the carefully designed two-branch structure [8], the proposed block architecture is simple and sufﬁciently general to be plugged into most backbones with minimal modiﬁcation.
Revised: Without using the interpolation operation [35] or the carefully designed two-branch structure [8], the proposed block architecture is simple and sufficiently general to be plugged into most backbones with minimal modification.

Original: Our formulation is mostly similar to that in [31], which could be viewed as a variant of our method with the spatial granularity S
Revised: Our formulation is mostly similar to that in [31], which could be viewed as a variant of our method with the spatial granularity S

Original: =1 for all blocks.
Revised: =1 for all blocks.

Original: Instead of performing spatially adaptive inference at the ﬁnest pixel level, our granularity S is optimized under the guidance of our latency prediction model (details are presented in the following
Revised: Instead of performing spatially adaptive inference at the finest pixel level, our granularity S is optimized under the guidance of our latency prediction model (details are presented in the following

Original: Sec. 4.2) to achieve realistic speedup on target computing platforms.
Revised: Sec. 4.2) to achieve realistic speedup on target computing platforms.

Original: 3.3 Latency prediction model
Revised: 3.3 Latency prediction model

Original: As stated before, it is laborious to evaluate the latency of dynamic operators on different hardware platforms.
Revised: As stated before, it is laborious to evaluate the latency of dynamic operators on different hardware platforms.

Original: To efﬁciently seek preferable granularity settings on arbitrary hardware devices, we propose a latency prediction model G , which can directly predict the delay of executing dynamic operators on any target devices.
Revised: To efficiently seek preferable granularity settings on arbitrary hardware devices, we propose a latency prediction model G , which can directly predict the delay of executing dynamic operators on any target devices.

Original: For a spatial-wise dynamic convolutional block, the latency predictor
Revised: For a spatial-wise dynamic convolutional block, the latency predictor

Original: G takes the hardware properties H , the layer parameters P , the spatial granularity S , and the activation rate r as input and predicts the latency (cid:96) of a dynamic convolutional block: (cid:96) = G ( H , P , S, r ) .
Revised: G takes the hardware properties H , the layer parameters P , the spatial granularity S , and the activation rate r as input and predicts the latency ℓ of a dynamic convolutional block: ℓ = G ( H , P , S, r ) .

Original: Hardware modeling.
Revised: Hardware modeling.

Original: We model a hardware device as multiple processing engines (PEs), and parallel computation can be executed on these PEs.
Revised: We model a hardware device as multiple processing engines (PEs), and parallel computation can be executed on these PEs.

Original: As shown in Figure 3, we model the memory system as a
Revised: As shown in Figure 3, we model the memory system as a

Original: O ff - c h i p
Revised: O ff - c h i p

Original: M e m o r y ( G DD R /
Revised: M e m o r y ( G DD R /

Original: DD R / H B M )
Revised: DD R / H B M )

Original: O n - c h i p
Revised: O n - c h i p

Original: M e m o r y ( LL C /G l o
Revised: M e m o r y ( LL C /

Original: [Figure]
Revised: [Figure]

Original: a l
Revised: l

Original: [Figure]
Revised: [Figure]

Original: u ff e r )
Revised: u ff e r )

Original: [Figure]
Revised: [Figure]

Original: 1.0 Activation rate ( r ) 

Revised: 1.0 Activation rate ( r )

Original: L a t e n c y ( s )
Revised: L a t e n c y ( s )

Original: PredictedReal three-level structure [12]: 1) off-chip memory, 2) on-chip global memory, and 3) memory in PE.
Revised: PredictedReal three-level structure [12]: 1) off-chip memory, 2) on-chip global memory, and 3) memory in PE.

Original: Such a hardware model enables us to accurately predict the cost on both data movement and computation .
Revised: Such a hardware model enables us to accurately predict the cost on both data movement and computation .

Original: Latency prediction.
Revised: Latency prediction.

Original: When simulating the data movement procedure, the efﬁciency of noncontiguous memory accesses under different granularity S settings is considered.
Revised: When simulating the data movement procedure, the efficiency of non-contiguous memory accesses under different granularity S settings is considered.

Original: As for the computation latency, it is important to adopt a proper scheduling strategy to increase the parallelism of computation.
Revised: As for the computation latency, it is important to adopt a proper scheduling strategy to increase the parallelism of computation.

Original: Therefore, we search for the optimal scheduling (the conﬁguration of tiling and in-PE parallel computing) of dynamic operations to maximize the utilization of hardware resources.
Revised: Therefore, we search for the optimal scheduling (the configuration of tiling and in-PE parallel computing) of dynamic operations to maximize the utilization of hardware resources.

Original: A more detailed description of our latency prediction model is presented in Appendix A.
Revised: A more detailed description of our latency prediction model is presented in Appendix A.

Original: Empirical validation.
Revised: Empirical validation.

Original: We take the ﬁrst block in ResNet-101 as an example and vary the activation rate r to evaluate the performance of our prediction model.
Revised: We take the first block in ResNet-101 as an example and vary the activation rate r to evaluate the performance of our prediction model.

Original: The comparison between our predictions and the real testing latency on the Nvidia V100 GPU is illustrated in Figure 4, from which we can observe that our predictor can accurately estimate the real latency in a wide range of activation rates.
Revised: The comparison between our predictions and the real testing latency on the Nvidia V100 GPU is illustrated in Figure 4, from which we can observe that our predictor can accurately estimate the real latency in a wide range of activation rates.

Original: 3.4 Implementation details
Revised: 3.4 Implementation details

Original: We use general optimization methods like fusing activation functions and batch normalization layers into convolution layers.
Revised: We use general optimization methods like fusing activation functions and batch normalization layers into convolution layers.

Original: We also optimize the speciﬁc operators in our spatial-wise dynamic convolutional blocks as follows (see also Figure 2 for an overview).
Revised: We also optimize the specific operators in our spatial-wise dynamic convolutional blocks as follows (see also Figure 2 for an overview).

Original: Fusing the masker and the ﬁrst convolution.
Revised: Fusing the masker and the first convolution.

Original: As mentioned in Sec.
Revised: As mentioned in Sec.

Original: 3.1, the masker in each block consumes very little computation, but it takes the whole feature map as input.
Revised: 3.1, the masker in each block consumes very little computation, but it takes the whole feature map as input.

Original: Therefore, it is a memory-bounded operation (the inference time is mainly spent on memory access).
Revised: Therefore, it is a memory-bounded operation (the inference time is mainly spent on memory access).

Original: Since the masker and the ﬁrst convolution in the block share the same input, there is an opportunity to fuse these two operations to avoid the repeated access of the input data.
Revised: Since the masker and the first convolution in the block share the same input, there is an opportunity to fuse these two operations to avoid the repeated access of the input data.

Original: Note that a spatial-wise dynamic convolution requires the output of the masker.
Revised: Note that a spatial-wise dynamic convolution requires the output of the masker.

Original: If we fuse the two layers, the ﬁrst convolution will be changed to a static operation, which may increase the inference latency.
Revised: If we fuse the two layers, the first convolution will be changed to a static operation, which may increase the inference latency.

Original: There exists a threshold of activation rate r th , when r > r th , the overall latency can be reduced.
Revised: There exists a threshold of activation rate r th , when r > r th , the overall latency can be reduced.

Original: We decide whether to fuse them according to the average activation rate.
Revised: We decide whether to fuse them according to the average activation rate.

Original: See more details in Appendix B.1.
Revised: See more details in Appendix B.

Original: Fusing the gather operation and the dynamic convolution.
Revised: Fusing the gather operation and the dynamic convolution.

Original: Traditional approaches ﬁrst gather the input pixels of the ﬁrst dynamic convolution in a block (Figure 1 (b)).
Revised: Traditional approaches first gather the input pixels of the first dynamic convolution in a block (Figure 1 (b)).

Original: The gather operation is also a memory-bounded operation.
Revised: The gather operation is also a memory-bounded operation.

Original: Furthermore, when the size of the convolution kernel exceeds 1 × 1, the area of input patches may overlap, resulting in repeated memory load/store.
Revised: Furthermore, when the size of the convolution kernel exceeds 1 × 1, the area of input patches may overlap, resulting in repeated memory load/store.

Original: We fuse the gather operation into the dynamic convolution to reduce the memory access.
Revised: We fuse the gather operation into the dynamic convolution to reduce the memory access.

Original: Fusing the scatter operation and the add operation.
Revised: Fusing the scatter operation and the add operation.

Original: Traditional approaches scatter the output pixels of the last dynamic convolution, and then execute the element-wise addition (Figure 1 (b)).
Revised: Traditional approaches scatter the output pixels of the last dynamic convolution, and then execute the element-wise addition (Figure 1 (b)).

Original: We fuse these two operators to reduce the memory access.
Revised: We fuse these two operators to reduce the memory access.

Original: The ablation study in Sec.
Revised: The ablation study in Sec.

Original: 4.4 validates the effectiveness of the proposed fusing methods.
Revised: 4.4 validates the effectiveness of the proposed fusing methods.

Original: 3.5 Training
Revised: 3.5 Training

Original: Optimization of non-differentiable maskers.
Revised: Optimization of non-differentiable maskers.

Original: The masker modules are required to produce binaryvalued spatial masks for making discrete decisions, and cannot be directly optimized with back propagation.
Revised: The masker modules are required to produce binaryvalued spatial masks for making discrete decisions, and cannot be directly optimized with back propagation.

Original: Following [35, 31, 8], we adopt straight-through Gumbel Softmax [18, 24] to train the network in an end-to-end fashion.
Revised: Following [35, 31, 8], we adopt straight-through Gumbel Softmax [18, 24] to train the network in an end-to-end fashion.

Original: Speciﬁcally, let
Revised: Specifically, let

Original: ˜ M ∈ R H
Revised: ˜ M ∈ R H

Original: × W × 2 denote the output of the mask generator.
Revised: × W × 2 denote the output of the mask generator.

Original: The decisions are obtained with the argmax function during inference.
Revised: The decisions are obtained with the argmax function during inference.

Original: In the training phase, a differentiable approximation is deﬁned by replacing the argmax operation with a Softmax:
Revised: In the training phase, a differentiable approximation is defined by replacing the argmax operation with a Softmax:

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table] [Figure]

Original:  [Figure]
Revised: [Table] [Figure] [Table] [Figure]

Original: [Table]
Revised: [Table]

Original: L
Revised: [Figure] [Table] [Figure]

Original: [Table]
Revised: [Table]

Original:  t 
Revised: L a t e n c y r a t i o ( r ) [Figure] [Figure] =

Revised: [Figure]

Revised: 4 S

Revised: [Figure]

Revised: [Table]

Revised: S

Original: [Table]
Revised: [Table]

Revised: S

Original: [Table]
Revised: [Table]

Revised: 14 S

Original: L a t e n
Revised: [Table] t [Table] n

Original: [Table]
Revised: [Table]

Revised: t

Revised: i

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: L a t e n c y r a t i o ( r )
Revised: L a t e n c y r a t i o ( r )

Original: S =1 S =2 S =4 S =7 S = 0.
Revised: S =1 S =2 S =4 S =7 S = 0.

Original: 1.0 Activation rate ( r ) 0.2 0.
Revised: 1.0 Activation rate ( r ) 0.2 0.

Original: L a t e n c y r a t i o ( r )
Revised: L a t e n c y r a t i o ( r )

Original: S =1 S =2 S
Revised: S =1 S =2 S

Original: = 0.
Revised: = 0.

Original: 1.0 Activation rate ( r ) 0.2 0.
Revised: 1.0 Activation rate ( r ) 0.2 0.

Original: L a t e n c y r a t i o ( r )
Revised: L a t e n c y r a t i o ( r )

Original: S = where τ is the Softmax temperature.
Revised: S = where τ is the Softmax temperature.

Original: Following the common practice [31, 8] we let τ decrease exponentially from 5.0 to 0.1 in training to facilitate the optimization of maskers.
Revised: Following the common practice [31, 8], we let τ decrease exponentially from 5.0 to 0.1 in training to facilitate the optimization of maskers.

Original: Training objective.
Revised: Training objective.

Original: The FLOPs of each spatial-wise dynamic convolutional block can be calculated based on our deﬁned activation rate r [31].
Revised: The FLOPs of each spatial-wise dynamic convolutional block can be calculated based on our defined activation rate r [31].

Original: Then we can obtain the FLOPs of the overall dynamic network F dyn .
Revised: Then we can obtain the FLOPs of the overall dynamic network F dyn .

Original: Let F stat denotes the FLOPs of its static counterpart.
Revised: Let F stat denotes the FLOPs of its static counterpart.

Original: We optimize their ratio to approximate a target 0 < t < 1 : L FLOPs =
Revised: We optimize their ratio to approximate a target 0 < t < 1 : L FLOPs =

Original: ( F dyn F stat − t ) 2 .
Revised: ( F dyn F stat − t ) 2 .

Original: In addition, we deﬁne loss item L bounds as in [31] to constrain the upper bound and the lower bound of activation rates in early training epochs.
Revised: In addition, we define loss item L bounds as in [31] to constrain the upper bound and the lower bound of activation rates in early training epochs.

Original: We further propose to leverage the static counterparts of our dynamic networks as “teachers” to guide the optimization procedure.
Revised: We further propose to leverage the static counterparts of our dynamic networks as “teachers” to guide the optimization procedure.

Original: Let y and y (cid:48) denote the output logits of a dynamic model (“student”) and its “teacher”, respectively.
Revised: Let y and y ′ denote the output logits of a dynamic model (“student”) and its “teacher”, respectively.

Original: Our ﬁnal loss can be written as
Revised: Our final loss can be written as

Original: [Equation]
Revised: [Equation]

Original: where L task represents the task-related loss, e.g. , cross-entropy loss in image classiﬁcation.
Revised: where L task represents the task-related loss, e.g. , cross-entropy loss in image classification.

Original: KL( ·||· ) denotes the Kullback–Leibler divergence, and α, β are the coefﬁcients balancing these items.
Revised: KL( ·||· ) denotes the Kullback–Leibler divergence, and α, β are the coefficients balancing these items.

Original: We use σ to denote the log-Softmax function, and T is the temperature for computing KL-divergence.
Revised: We use σ to denote the log-Softmax function, and T is the temperature for computing KL-divergence.

Original:  Experiments
Revised:  Experiments

Original: In this section, we ﬁrst introduce the experiment settings in Sec. 4.1.
Revised: In this section, we first introduce the experiment settings in Sec. 4.1.

Original: Then the latency of different granularity settings are analyzed in Sec.
Revised: Then the latency of different granularity settings are analyzed in Sec.

Original: The performance of our LASNet on ImageNet is further evaluated in Sec. 4.3, followed by the ablation studies in Sec. 4.4.
Revised: The performance of our LASNet on ImageNet is further evaluated in Sec. 4.3, followed by the ablation studies in Sec. 4.4.

Original: Visualization results are illustrated in Sec. 4.5, and we ﬁnally validate our method on the object detection task (Sec. 4.6).
Revised: Visualization results are illustrated in Sec. 4.5, and we finally validate our method on the object detection task (Sec. 4.6).

Original: The results on the instance segmentation task are presented in .
Revised: The results on the instance segmentation task are presented in .

Original: For simplicity, we add “LAS-” as a preﬁx before model names to denote our LASNet, e.g. , LAS-ResNet-50.
Revised: For simplicity, we add “LAS-” as a prefix before model names to denote our LASNet, e.g. , LAS-ResNet-50.

Original: 4.1 Experiment settings
Revised: 4.1 Experiment settings

Original: Latency prediction.
Revised: Latency prediction.

Original: Various types of hardware platforms are tested, including a server GPU (Tesla
Revised: Various types of hardware platforms are tested, including a server GPU (Tesla

Original: V100), a desktop GPU (GTX1080) and edge devices ( e.g. , Nvidia Nano and Jetson TX2).
Revised: V100), a desktop GPU (GTX1080) and edge devices ( e.g. , Nvidia Nano and Jetson TX2).

Original: The major properties considered by our latency prediction model include the number of processing engines (#PE), the ﬂoating-point computation in a processing engine (#FP32), the frequency and the bandwidth.
Revised: The major properties considered by our latency prediction model include the number of processing engines (#PE), the floating-point computation in a processing engine (#FP32), the frequency and the bandwidth.

Original: It can be observed that the server GPUs generally have a larger #PE than the IoT devices.
Revised: It can be observed that the server GPUs generally have a larger #PE than the IoT devices.

Original: The batch size is set as 1 for all dynamic models and computing platforms.
Revised: The batch size is set as 1 for all dynamic models and computing platforms.

Original: Image classiﬁcation.
Revised: Image classification.

Original: The image classiﬁcation experiments are conducted on the ImageNet [4] dataset.
Revised: The image classification experiments are conducted on the ImageNet [4] dataset.

Original: Following [31], we initialize the backbone parameter from a pre-trained checkpoint 4 , and
Revised: Following [31], we initialize the backbone parameter from a pre-trained checkpoint 4 , and

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: ﬁnetune the whole network for 100 epochs with the loss function in Eq.
Revised: finetune the whole network for 100 epochs with the loss function in Eq.

Original: We ﬁx α = 10 , β = 0 .and
Revised: We fix α = 10 , β = 0 .and

Original: T = 4 . 

Revised: T = 4 .

Original: for all dynamic models.
Revised: for all dynamic models.

Original: More details are provided in Appendix B.
Revised: More details are provided in Appendix B.

Original: 4.2 Latency prediction results
Revised: 4.2 Latency prediction results

Original: In this subsection, we present the latency prediction results of the spatial-wise dynamic convolutional blocks in two different models: LAS-ResNet-101 [11] (on V100) and LAS-RegNetY-800MF [25] (on
Revised: In this subsection, we present the latency prediction results of the spatial-wise dynamic convolutional blocks in two different models: LAS-ResNet-101 [11] (on V100) and LAS-RegNetY-800MF [25] (on

Original: TX2).
Revised: TX2).

Original: All the blocks have the bottleneck structure with different channel numbers and convolution groups, and the RegNetY is equipped with Squeeze-and-Excitation (SE)
Revised: All the blocks have the bottleneck structure with different channel numbers and convolution groups, and the RegNetY is equipped with Squeeze-and-Excitation (SE)

Original: [14] modules.
Revised: [14] modules.

Original: We ﬁrst deﬁne (cid:96) dyn as the latency of a spatial-wise dynamic convolutional block, and (cid:96) stat as that of a static block without a masker.
Revised: We first define ℓ dyn as the latency of a spatial-wise dynamic convolutional block, and ℓ stat as that of a static block without a masker.

Original: Their ratio is denoted as r (cid:96) = (cid:96) dyn (cid:96) stat .
Revised: Their ratio is denoted as r ℓ = ℓ dyn ℓ stat .

Original: We investigate the relationship between r (cid:96) and the activation rate r (cf.
Revised: We investigate the relationship between r ℓ and the activation rate r (cf.

Original: Sec. 3.5) for different granularity settings.
Revised: Sec. 3.5) for different granularity settings.

Original: The results in
Revised: The results in

Original: Figure 5 demonstrate that: 1) even equipped with our special optimization on the scheduling strategies, pixel-level spatially adaptive inference ( S =1) cannot always improve the practical efﬁciency.
Revised: Figure 5 demonstrate that: 1) even equipped with our special optimization on the scheduling strategies, pixel-level spatially adaptive inference ( S =1) cannot always improve the practical efficiency.

Original: Such ﬁne-grained adaptive inference is adopted by most previous works [31, 35], and our result can explain the reason why they can only achieve realistic speedup on less powerful CPUs [35] or specialized devices [3]; 2) a proper granularity S > 1 effectively alleviates this problem on both hardware devices.
Revised: Such fine-grained adaptive inference is adopted by most previous works [31, 35], and our result can explain the reason why they can only achieve realistic speedup on less powerful CPUs [35] or specialized devices [3]; 2) a proper granularity S > 1 effectively alleviates this problem on both hardware devices.

Original: By setting S > 1 , realistic speedup could be achieved with larger activation rates.
Revised: By setting S > 1 , realistic speedup could be achieved with larger activation rates.

Original: The latency prediction results are further used to seek for a preferable granularity setting for the ﬁrst 3 stages (we ﬁx S = 1 for the last stage, where the feature resolution is 7 × 7 ).
Revised: The latency prediction results are further used to seek for a preferable granularity setting for the first 3 stages (we fix S = 1 for the last stage, where the feature resolution is 7 × 7 ).

Original: Therefore, we plot the relationship between r (cid:96) and S in Figure 10.
Revised: Therefore, we plot the relationship between r ℓ and S in Figure 10.

Original: It can be observed that: 1) r (cid:96) generally decreases with S increasing for a given r ; 2) an overly large S (less ﬂexible adaptive inference) brings insigniﬁcant improvement on both devices.
Revised: It can be observed that: 1) r ℓ generally decreases with S increasing for a given r ; 2) an overly large S (less flexible adaptive inference) brings insignificant improvement on both devices.

Original: Especially, enlarging S from 8 to 28 in the ﬁrst stage of a LAS-ResNet brings very little improvement on V100.
Revised: Especially, enlarging S from 8 to 28 in the first stage of a LAS-ResNet brings very little improvement on V100.

Original: Based on the results in Figure 10, we can trade off between ﬂexibility and efﬁciency by selecting appropriate S for different models and hardware devices.
Revised: Based on the results in Figure 10, we can trade off between flexibility and efficiency by selecting appropriate S for different models and hardware devices.

Original: For example, we can simply set S net =8-4-7-1 5 in a LAS-ResNet-101 to achive realistic speedup.
Revised: For example, we can simply set S net =8-4-7-1 5 in a LAS-ResNet-101 to achive realistic speedup.

Original: The accuracy-latency plots in Figure 7 also validate this observation.
Revised: The accuracy-latency plots in Figure 7 also validate this observation.

Original: More results of our latency prediction model on the desktop-level GPU, GTX1080, are presented in Appendix C.1.
Revised: More results of our latency prediction model on the desktop-level GPU, GTX1080, are presented in Appendix C.1.

Original: 4.3 ImageNet classiﬁcation results
Revised: 4.3 ImageNet classification results

Original: We now empirically evaluate our proposed LASNet on the ImageNet dataset.
Revised: We now empirically evaluate our proposed LASNet on the ImageNet dataset.

Original: The network performance is measured in terms of the trade-off between classiﬁcation accuracy and inference efﬁciency.
Revised: The network performance is measured in terms of the trade-off between classification accuracy and inference efficiency.

Original: Both theoretical ( i.e. FLOPs) and practical efﬁciency ( i.e. latency) are tested in our experiments.
Revised: Both theoretical ( i.e. FLOPs) and practical efficiency ( i.e. latency) are tested in our experiments.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 4.3.1 Standard baseline comparison: ResNets
Revised: 4.3.1 Standard baseline comparison: ResNets

Original: We ﬁrst establish our LASNet based on the standard ResNets [11].
Revised: We first establish our LASNet based on the standard ResNets [11].

Original: Speciﬁcally, we build LASResNet-50 and LAS-ResNet-101 by plugging our maskers in the two common ResNet structures.
Revised: Specifically, we build LASResNet-50 and LAS-ResNet-101 by plugging our maskers in the two common ResNet structures.

Original: The baselines include various types of dynamic inference approaches: 1) layer skipping (SkipNet [32] and Conv-AIG [30]); 2) channel skipping (BAS [1]); and 3) pixel-level spatial-wise dynamic network (DynConv [31]).
Revised: The baselines include various types of dynamic inference approaches: 1) layer skipping (SkipNet [32] and Conv-AIG [30]); 2) channel skipping (BAS [1]); and 3) pixel-level spatial-wise dynamic network (DynConv [31]).

Original: For our LASNet, we compare various settings of the spatial granularity
Revised: For our LASNet, we compare various settings of the spatial granularity

Original: S net .
Revised: S net .

Original: We set training targets (cf.
Revised: We set training targets (cf.

Original: Sec. 3.5) t ∈{ 0 , 4 , 0 . 5 , 0 . 6 , 0 . 7 } for our dynamic models to evaluate their performance in different sparsity regimes.
Revised: Sec. 3.5) t ∈{ 0 , 4 , 0 . 5 , 0 . 6 , 0 . 7 } for our dynamic models to evaluate their performance in different sparsity regimes.

Original: We apply the same operator fusion (Sec. 3.4) for both our models and the compared baselines [30, 31] for fair comparison.
Revised: We apply the same operator fusion (Sec. 3.4) for both our models and the compared baselines [30, 31] for fair comparison.

Original: Results are presented in Figure 7 (a).
Revised: Results are presented in Figure 7 (a).

Original: On the left we plot the relationship of accuracy v.s. FLOPs.
Revised: On the left we plot the relationship of accuracy v.s. FLOPs.

Original: It can be observed that our LAS-ResNets with different granularity settings signiﬁcantly outperform the competing dynamic neural networks.
Revised: It can be observed that our LAS-ResNets with different granularity settings significantly outperform the competing dynamic neural networks.

Original: Surprisingly, coarse-grained spatially adaptive inference ( S net =4-4-2-1 and S net =8-4-7-1 for the 4 stages) can achieve even higher accuracy when consuming similar FLOPs on ResNets, despite the sacriﬁced ﬂexibility compared to S net =1-1-1-1.
Revised: Surprisingly, coarse-grained spatially adaptive inference ( S net =4-4-2-1 and S net =8-4-7-1 for the 4 stages) can achieve even higher accuracy when consuming similar FLOPs on ResNets, despite the sacrificed flexibility compared to S net =1-1-1-1.

Original: We conjecture that a larger S is also beneﬁcial to the optimization of maskers.
Revised: We conjecture that a larger S is also beneficial to the optimization of maskers.

Original: We compare the practical latency of three granularity settings in Figure 7 (a) predicted by our latency prediction model (middle on TX2 and right on V100).
Revised: We compare the practical latency of three granularity settings in Figure 7 (a) predicted by our latency prediction model (middle on TX2 and right on V100).

Original: We can witness that although they achieve comparable theoretical efﬁciency (Figure 7 (a) left), larger S is more hardware-friendly compared to the ﬁnest granularity.
Revised: We can witness that although they achieve comparable theoretical efficiency (Figure 7 (a) left), larger S is more hardware-friendly compared to the finest granularity.

Original: For example, the inference latency of LAS-ResNet-101 ( S net =1-1-1-1) is signiﬁcantly higher than the ResNet-101 baseline on V100 (Figure 7 (a) right), even though its theoretical computation is much smaller than that of the static model.
Revised: For example, the inference latency of LAS-ResNet-101 ( S net =1-1-1-1) is significantly higher than the ResNet-101 baseline on V100 (Figure 7 (a) right), even though its theoretical computation is much smaller than that of the static model.

Original: However, larger granularities ( S net =4-4-2-1 and S net =8-4-7-1) can effectively improve the inference latency due to its lower burden on the memory access.
Revised: However, larger granularities ( S net =4-4-2-1 and S net =8-4-7-1) can effectively improve the inference latency due to its lower burden on the memory access.

Original: Remarkably, the latency of ResNet-101 could be reduced by 36% and 46% on
Revised: Remarkably, the latency of ResNet-101 could be reduced by 36% and 46% on

Original: V100 and TX2 respectively without sacriﬁcing the accuracy when t =0.4.
Revised: V100 and TX2 respectively without sacrificing the accuracy when t =0.4.

Original: The classiﬁcation accuracy is increased by 1.9% with similar inference efﬁciency.
Revised: The classification accuracy is increased by 1.9% with similar inference efficiency.

Original: It can be observed that the realistic speedup ratio r (cid:96) is more close to the theoretical FLOPs ratio target t on the less powerful TX2, because the latency is computation-bounded ( i.e. the latency is mainly spent on computation) on such IoT devices.
Revised: It can be observed that the realistic speedup ratio r ℓ is more close to the theoretical FLOPs ratio target t on the less powerful TX2, because the latency is computation-bounded ( i.e. the latency is mainly spent on computation) on such IoT devices.

Original: In contrast, there is a larger gap between practical and theoretical efﬁciency on the more powerful
Revised: In contrast, there is a larger gap between practical and theoretical efficiency on the more powerful

Original: V100, as the latency is bounded by the memory access cost.
Revised: V100, as the latency is bounded by the memory access cost.

Original: 4.3.2 Lightweight baseline comparison: RegNets
Revised: 4.3.2 Lightweight baseline comparison: RegNets

Original: We further evaluate our LASNet in lightweight CNN architectures, i.e. RegNets-Y [25].
Revised: We further evaluate our LASNet in lightweight CNN architectures, i.e. RegNets-Y [25].

Original: Two different sized models are tested: RegNetY-400MF and RegNetY-800MF.
Revised: Two different sized models are tested: RegNetY-400MF and RegNetY-800MF.

Original: Compared baselines include other types of efﬁcient models, e.g. , MobileNets-v2 [28], ShufﬂetNets-v2 [23] and CondenseNets [16].
Revised: Compared baselines include other types of efficient models, e.g. , MobileNets-v2 [28], ShuffletNets-v2 [23] and CondenseNets [16].

Original: The results are presented in Figure 7 (b).
Revised: The results are presented in Figure 7 (b).

Original: The x-axis for the three sub-ﬁgures are the FLOPs, the latency on TX2, and the latency on the Nvidia nano GPU, respectively.
Revised: The x-axis for the three sub-figures are the FLOPs, the latency on TX2, and the latency on the Nvidia nano GPU, respectively.

Original: We can observe that our
Revised: We can observe that our

Original: 3.2 Latency on Nvidia Tesla V100 GPU (ms) 76.
Revised: 3.2 Latency on Nvidia Tesla V100 GPU (ms) 76.

Original: A cc u r a c y ( % ) static ResNets LAS-ResNet-101 ( S net =1-1-1-1) LAS-ResNet-101 ( S net =4-4-1-1) LAS-ResNet-101 ( S net =4-4-2-1) LAS-ResNet-101 ( S net =4-4-7-1)
Revised: A cc u r a c y ( % ) static ResNets LAS-ResNet-101 ( S net =1-1-1-1) LAS-ResNet-101 ( S net =4-4-1-1) LAS-ResNet-101 ( S net =4-4-2-1) LAS-ResNet-101 ( S net =4-4-7-1)

Original: LAS-ResNet-101 ( S net =8-4-7-1) method outperforms various types of static models in terms of the trade-off between accuracy and efﬁciency.
Revised: LAS-ResNet-101 ( S net =8-4-7-1) method outperforms various types of static models in terms of the trade-off between accuracy and efficiency.

Original: More results on image classiﬁcation are provided in Appendix C.2.
Revised: More results on image classification are provided in Appendix C.2.

Original: 4.4 Ablation studies
Revised: 4.4 Ablation studies

Original: We conduct ablation studies to validate the effectiveness of our coarse-grained spatially adaptive inference (Sec. 3.2) and operator fusion operations (Sec. 3.4).
Revised: We conduct ablation studies to validate the effectiveness of our coarse-grained spatially adaptive inference (Sec. 3.2) and operator fusion operations (Sec. 3.4).

Original: More granularities settings.
Revised: More granularities settings.

Original: We test various granularity settings on LAS-ResNet-101 to examine the effects of S in different stages.
Revised: We test various granularity settings on LAS-ResNet-101 to examine the effects of S in different stages.

Original: The results on the Tesla-V100 GPU are presented in Figure 8.
Revised: The results on the Tesla-V100 GPU are presented in Figure 8.

Original: It can be found that the ﬁnest granularity ( S net =1-1-1-1) leads to substantial inefﬁciency despite the reduced FLOPs (cf.
Revised: It can be found that the finest granularity ( S net =1-1-1-1) leads to substantial inefficiency despite the reduced FLOPs (cf.

Original: Figure 7 (a) left).
Revised: Figure 7 (a) left).

Original: Coarse-grained spatially adaptive inference in the ﬁrst two stages ( S net =4-4-1-1) effectively reduces the inference latency.
Revised: Coarse-grained spatially adaptive inference in the first two stages ( S net =4-4-1-1) effectively reduces the inference latency.

Original: We further increase S in the third stage to 2 and 7, and this procedure consistently improves the realistic efﬁciency on the V100 GPU.
Revised: We further increase S in the third stage to 2 and 7, and this procedure consistently improves the realistic efficiency on the V100 GPU.

Original: This trend also holds on the GTX 

Revised: This trend also holds on the GTX

Original: GPU (see the results in Appendix C.2.[Table]
Revised: GPU (see the results in Appendix C.2.

Original: Add (µs) (cid:55) (cid:55) (cid:55) 163.

Original: [Table]
Revised: [Table]

Original: (cid:51) (cid:55) (cid:55) 90. [Table] (cid:51) (cid:51) (cid:55) 86.7 (cid:51) (cid:51) (cid:51) 71.
Revised: ✗ ✗ ✗ 163.2 ✓ ✗ ✗ 90.1 ✓ ✓ ✗ 86.7 ✓ ✓ ✓ 71.

Original: Operator fusion.
Revised: Operator fusion.

Original: We investigate the effect of our operator fusion introduced in Sec. 3.4.
Revised: We investigate the effect of our operator fusion introduced in Sec. 3.4.

Original: One convolutional block in the ﬁrst stage of a LAS-ResNet- ( S =4, r =0.6) is tested.
Revised: One convolutional block in the first stage of a LAS-ResNet- ( S =4, r =0.6) is tested.

Original: The results in Table 1 validate that every step of operator fusion beneﬁts the practical latency of a block, as the overhead on memory access is effectively reduced.
Revised: The results in Table 1 validate that every step of operator fusion benefits the practical latency of a block, as the overhead on memory access is effectively reduced.

Original: Especially, the fusion of the masker operation and the ﬁrst convolution is crucial to reducing the latency.
Revised: Especially, the fusion of the masker operation and the first convolution is crucial to reducing the latency.

Original: 4.5 Visualization
Revised: 4.5 Visualization

Original: We visualize the masks generated by our masker in the third block of a LAS-ResNet-101 ( S net =44-2-1) in Figure 12.
Revised: We visualize the masks generated by our masker in the third block of a LAS-ResNet-101 ( S net =44-2-1) in Figure 12.

Original: The brilliant areas correspond to the locations of 1 elements in a mask, and the computation on the dimmed regions is skipped by our dynamic model.
Revised: The brilliant areas correspond to the locations of 1 elements in a mask, and the computation on the dimmed regions is skipped by our dynamic model.

Original: We can witness that the masker accurately locate the most task-related regions (even the tiny aircraft at the corner), which helps reduce the unnecessary computation on background areas.
Revised: We can witness that the masker accurately locate the most task-related regions (even the tiny aircraft at the corner), which helps reduce the unnecessary computation on background areas.

Original: These resultsalso suggest that for the ﬁrst stage, the granularity S =4 is sufﬁciently ﬂexible to recognize the important regions, and a win-win can be achieved between accuracy and efﬁciency.
Revised: These resultsalso suggest that for the first stage, the granularity S =4 is sufficiently flexible to recognize the important regions, and a win-win can be achieved between accuracy and efficiency.

Original: Interestingly, the masker could select some objects that are not labeled for the sample, e.g. , the ﬂower beside the hummingbird and the human holding the camera.
Revised: Interestingly, the masker could select some objects that are not labeled for the sample, e.g. , the flower beside the hummingbird and the human holding the camera.

Original: This indicates that our spatial-wise dynamic networks can automatically recognize the regions with semantics, and their capability is not limited by the classiﬁcation labels.
Revised: This indicates that our spatial-wise dynamic networks can automatically recognize the regions with semantics, and their capability is not limited by the classification labels.

Original: This property is helpful in some downstream tasks, such as object detection (Sec. 4.6) and instance segmentation (Sec. 4.7), which require detecting multiple classes and objects in an image.
Revised: This property is helpful in some downstream tasks, such as object detection (Sec. 4.6) and instance segmentation (Sec. 4.7), which require detecting multiple classes and objects in an image.

Original: More visualization results could be found in Appendix C.3.
Revised: More visualization results could be found in Appendix C.3.

Original: 4.6 COCO Object detection
Revised: 4.6 COCO Object detection

Original: We further evaluate our LASNet on the COCO [22] object detection task.
Revised: We further evaluate our LASNet on the COCO [22] object detection task.

Original: The mean average precision (mAP), the average backbone FLOPs, and the average backbone latency on the validation set are used to measure the network performance.
Revised: The mean average precision (mAP), the average backbone FLOPs, and the average backbone latency on the validation set are used to measure the network performance.

Original: We test two commonly used detection frameworks: Faster
Revised: We test two commonly used detection frameworks: Faster

Original: R-CNN [27] with Feature Pyramid Network [20] and RetinaNet [21].
Revised: R-CNN [27] with Feature Pyramid Network [20] and RetinaNet [21].

Original: Thanks to the generality of our method, we can conveniently replace the backbones with ours pre-trained on ImageNet, and the whole models are ﬁnetuned on COCO with the standard setting for 12 epochs (see detailed
Revised: Thanks to the generality of our method, we can conveniently replace the backbones with ours pre-trained on ImageNet, and the whole models are finetuned on COCO with the standard setting for 12 epochs (see detailed

Original: [Table] [Table]
Revised: [Table] [Table]

Original: setup in Appendix B.3).
Revised: setup in Appendix B.3).

Original: The input images are resized to a short side of 800 and a long side not exceeding 1333.
Revised: The input images are resized to a short side of 800 and a long side not exceeding 1333.

Original: The results of our LAS-ResNet-101 with different S net settings are presented in
Revised: The results of our LAS-ResNet-101 with different S net settings are presented in

Original: Table 2.
Revised: Table 2.

Original: We can observe from the results that when setting the training target as 0.4, the latency of our LAS-ResNet-101 with S net =4-4-7-1 is signiﬁcantly lower than the static baseline on all devices without sacriﬁcing mAP in both detection frameworks.
Revised: We can observe from the results that when setting the training target as 0.4, the latency of our LAS-ResNet-101 with S net =4-4-7-1 is significantly lower than the static baseline on all devices without sacrificing mAP in both detection frameworks.

Original: With a larger training targets, our LASNet can increase the mAP by 0.9% and 0.8% in Faster R-CNN and RetinaNet respectively, while still being faster than the baseline method.
Revised: With a larger training targets, our LASNet can increase the mAP by 0.9% and 0.8% in Faster R-CNN and RetinaNet respectively, while still being faster than the baseline method.

Original: 4.7 COCO instance segmentation
Revised: 4.7 COCO instance segmentation

Original: We also present the results of instance segmentation on COCO, which demonstrate the effectiveness of our LASNet on the dense prediction task.
Revised: We also present the results of instance segmentation on COCO, which demonstrate the effectiveness of our LASNet on the dense prediction task.

Original: From the results in Table 3, we can observe that when setting the training target as 0.4, the Mask R-CNN [10] models ( S net =4-4-2-1 and S net =4-4-7-1) runs faster on all tested hardware devices without sacriﬁcing the performance.
Revised: From the results in Table 3, we can observe that when setting the training target as 0.4, the Mask R-CNN [10] models ( S net =4-4-2-1 and S net =4-4-7-1) runs faster on all tested hardware devices without sacrificing the performance.

Original: With a training target of 0.5, the AP mask and AP box of the Mask R-CNN model could be increased by 0.9% and 1.0% respectively while still running faster than the baseline method.
Revised: With a training target of 0.5, the AP mask and AP box of the Mask R-CNN model could be increased by 0.9% and 1.0% respectively while still running faster than the baseline method.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we propose to build latency-aware spatial-wise dynamic networks (LASNet) under the guidance of a latency prediction model .
Revised: In this paper, we propose to build latency-aware spatial-wise dynamic networks (LASNet) under the guidance of a latency prediction model .

Original: By simultaneously considering the algorithm, the scheduling strategy and the hardware properties, we can efﬁciently estimate the practical latency of spatialwise dynamic operators on arbitrary computing platforms.
Revised: By simultaneously considering the algorithm, the scheduling strategy and the hardware properties, we can efficiently estimate the practical latency of spatialwise dynamic operators on arbitrary computing platforms.

Original: Based on the empirical analysis on the relationship between the latency and the granularity of spatially adaptive inference, we optimize both the algorithm and the scheduling strategies to achieve realistic speedup on many multi-core processors, e.g. , the Tesla V100 GPU and the Jetson TX2 GPU.
Revised: Based on the empirical analysis on the relationship between the latency and the granularity of spatially adaptive inference, we optimize both the algorithm and the scheduling strategies to achieve realistic speedup on many multi-core processors, e.g. , the Tesla V100 GPU and the Jetson TX2 GPU.

Original: Experiments on image classiﬁcation, object detection and instance segmentation tasks validate that the proposed method signiﬁcantly improves the practical efﬁciency of deep CNNs, and outperforms various competing approaches.
Revised: Experiments on image classification, object detection and instance segmentation tasks validate that the proposed method significantly improves the practical efficiency of deep CNNs, and outperforms various competing approaches.

Original: Acknowledgement
Revised: Acknowledgement

Original: This work is supported in part by the National Key R&D Program of China under Grant 2020AAA0105200, the National Natural Science Foundation of China under Grantsand the Tsinghua University-China Mobile Communications Group Co.,Ltd.
Revised: This work is supported in part by the National Key R&D Program of China under Grant 2020AAA0105200, the National Natural Science Foundation of China under Grantsand the Tsinghua University-China Mobile Communications Group Co.,Ltd.

Original: Joint Institute..
Revised: Joint Institute..

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information on how to answer these questions.
Revised: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] or [N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] or [N/A] .

Original: You are strongly encouraged to include a justiﬁcation to your answer  either by referencing the appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section 4.1 and Section 4.6, the public ImageNet and COCO datasets are cited.
Revised: [Yes] See Section 4.1 and Section 4.6, the public ImageNet and COCO datasets are cited.

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] See Appendix D. (c) Did you discuss any potential negative societal impacts of your work?
Revised: [Yes] See Appendix D. (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See Appendix D. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [Yes] See Appendix D. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 3.
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] See the abstract.
Revised: [Yes] See the abstract.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] Most important hyperparameters are given in the main paper, and more details are presented in the supplementary material (Appendix B).
Revised: [Yes] Most important hyperparameters are given in the main paper, and more details are presented in the supplementary material (Appendix B).

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [No] Since the experiment results on the large-scale dataset, ImageNet and COCO, are generally stable, we perform several repeat experiments in the early exploring phase, and dismiss the error bars in our main results as in most previous works.
Revised: [No] Since the experiment results on the large-scale dataset, ImageNet and COCO, are generally stable, we perform several repeat experiments in the early exploring phase, and dismiss the error bars in our main results as in most previous works.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] The details are presented in the supplementary material (Appendix B).
Revised: [Yes] The details are presented in the supplementary material (Appendix B).

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] See Section 4.1.
Revised: [Yes] See Section 4.1.

Original: We use the torchvision pre-trained models.
Revised: We use the torchvision pre-trained models.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No] (d) Did you discuss whether
Revised: [No] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?[N/A] Appendix A Latency prediction model.
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: As the dynamic operators in our method have not been supported by current deep learning libraries, we propose a latency prediction model to efﬁciently estimate the real latency of these operators on hardware device.

Original: The inputs of the latency prediction model include: 1) the structural conﬁguration of a convolutional block, 2) its activation rate r which decides the computation amount, 3) the spatial granularity S , and 4) the hardware properties mentioned in Table 4.

Original: The latency of a dynamic block is predicted as follows.

Original: Input/output shape deﬁnition .

Original: The ﬁrst step of predicting the latency of an operation is to calculate the shape of input and output.

Original: Taking the gather-conv2 operation as an example, the input of this operation is the activation with the shape of C in × H × W , where C in is the number of input channels, and H and W are the resolution of the feature map.

Original: The shape of the output tensor is

Original: P × C out × S × S , where P is the number of output patches, C out is the number of output channels and S is the spatial granularity.

Original: Note that P is obtained based on the output of our maskers.

Original: Operation-to-hardware mapping.

Original: Next, we map the operations to hardware.

Original: As is mentioned in the paper, we model a hardware device as multiple processing engines (PEs).

Original: We assign the computation of each element in the output feature map to a PE.

Original: Speciﬁcally, we consecutively split the output feature map into multiple tiles .

Original: The shape of each tile is T P × T C × T S 1 × T S 2 .

Original: These split tiles are assigned to multiple PEs.

Original: The computation of the elements in each tile is executed in a PE.

Original: We can conﬁgure different shapes of tiles.

Original: In order to determine the optimal shape of the tile, we make a search space of different tile shapes.

Original: The tile shape has 4 dimensions.

Original: The candidates of each dimension are power-of-2 and do not exceed the corresponding dimension of the feature map.

Original: Latency estimation.

Original: Then, we evaluate the latency of each tile shape in the search space and select the optimal tile shape with the lowest latency.

Original: The latency includes the data movement latency and the computation latency: (cid:96) =

Original: (cid:96) data + (cid:96) computation .

Original: 1) Data movement latency (cid:96) data .

Original: The estimation of the latency for data movement requires us to model the memory system of a hardware device.

Original: We model the memory system of hardware as a three-level architecture [12]: off-chip memory, on-chip global memory, and local memory in PE.

Original: The input data and weight data are ﬁrst transferred from the off-chip memory to the on-chip global memory.

Original: We assume the hardware can make full use of the off-chip memory bandwidth to simplify the latency prediction model.

Original: After that, the data used to compute the output tiles is transferred from on-chip global memory to the local memory of each PE.

Original: The latency of data movement to local memory is estimated by its bandwidth and efﬁciency .

Original: We assume each PE only moves the corresponding input feature maps and weights once to compute a output tile so as to simplify the prediction model.

Original: The input data movement latency (cid:96) in is calculated by adding the time from off-chip memory to on-chip global memory and the time from on-chip global memory to local-memory together: (cid:96) in = (cid:96) oﬀ2on + (cid:96) global2local .

Original: Contrary to the input data, the output data (cid:96) out are moved from local memory to on-chip global memory and then to off-chip memory: (cid:96) out = (cid:96) local2global

Original: + (cid:96) on2oﬀ .

Original: We calculate the total data movement latency by adding the input and output data movement latency together: (cid:96) data = (cid:96) in + (cid:96) out .

Original: The latency of data movement is affected by the granularity S : when the granularity S is small, the same input data has a higher probability of being sent to multiple PEs to compute different output patches, which signiﬁcantly increases the number of on-chip memory movement.

Original: And due to the small amount of data transmitted each time and the data is randomly distributed, the efﬁciency of data movement will be low.

Original: This accounts for our experiment results in the paper that a larger S will effectively improve the practical efﬁciency

Original: . 2) Computation latency (cid:96) computation .

Original: The computation latency of each tile is estimated using the PE’s maximum throughput of FP32 computation and the FLOPs of computing an output tile.

Original: The total computation latency can be obtained according to the number of tiles and the number of PEs

Original: To summarize, our latency prediction model can predict the real latency of dynamic operators by considering both the data movement cost and the computation cost.

Original: Guided by the latency prediction model, we propose our LASNets with coarse-grained spatially adaptive inference ( S > 1 ).

Original: It is validated in our paper that LASNets achieve better efﬁciency than previous approaches [35, 31]

Original: ( S = 1 ), as it effectively reduces the data movement latency, which is rarely considered by other researchers.

Original: B Detailed experimental settings

Original: In this section, we present the detailed experiment settings which are not provided in the main paper due to the page limit.

Original: B.1 Latency prediction

Original: Hardware properties considered by our latency prediction model include the number of processing engines (#PE), the ﬂoating-point computation in a processing engine (#FP32), the frequency and the bandwidth.

Original: We test four types of hardware devices, and their properties are listed in Table 4.

Original: [Table]
Revised: [N/A]

Original: It could be found that the server GPU V100 is the most powerful hardware device, especially with the most number of processing engines (#PE).

Original: Therefore, spatially adaptive inference could easily fall into a memory-bounded operation on V100 due to its high parallelism.

Original: Our experiment results in Figure 7 (a) and Figure 8 in the paper can reﬂect this phenomenon: the more ﬂexibility the computation is, the harder to improve the practical efﬁciency.

Original: In contrast, on the less powerful computing devices such as the IoT devices, the real acceleration is close to the theoretical effect (compare Figure 7 (a) left with Figure 7 (a) middle).

Original: Operator fusion.

Original: 1) Fusing the masker and the ﬁrst convolution.

Original: We mentioned in Sec.

Original: 3.4 of the paper that the masker operation is fused with the ﬁrst 1 × 1 convolution in a block to reduce the cost on memory access.

Original: This is feasible because the two operators share the same input feature, and their convolutional kernel sizes are both 1 × 1.

Original: Note that during the inference stage, we only need to perform arg max along the channel dimension of a mask M ∈ R 2 × H × W to obtain the positions of the gathered pixels.

Original: Therefore, we can reduce the output channel number of our maskers from 2 to 1 since the convolution is a linear operation:

Original: [Equation]

Original: Afterwards, we fuse the masker with the ﬁrst convolution layer by performing once convolution whose output channel number is C + 1 , where C is the original output width of the ﬁrst convolution.

Original: The output of this step is split into a feature map (for further computation) and a mask (for obtaining the index for gathering).

Original: Such operator fusion avoids the repeated reading the input feature, and helps reduce the inference latency (see Table 1 in the paper).

Original: 2) Fusing the gather operation and the dynamic convolution.

Original: To facilitate the scheduling on hardware devices with multiple PEs, the masker generates the indices of activated patches instead of sparse mask at inference time.

Original: In this way, it is easy to evenly distribute the computation of output patches to different PEs, thus avoiding unbalanced computation of PEs.

Original: Each element in the indices represents the index of an activated patch.

Original: PE fetches the input data from the corresponding positions on the feature map according to the index.

Original: The output patches could be densely stored in memory.

Original: Such operator fusion beneﬁts the contiguous memory access and parallel computation on multiple PEs.

Original: 3) Fusing the scatter operation and the add operation

Original: Similar to the previous operation, each PE fetches a tile of data from the residual feature map according to the index, adds them with the corresponding feature map from previous dynamic

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: L a t e n c y r a t i o ( r )

Original: S =1 S =2 S =4 S =7 S =

Original: S =14 S = 0.

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: L a t e n c y r a t i o ( r )

Original: S =

Original: [Figure]

Original: =2 S

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: ( r

Original: [Figure]

Original: [Figure]

Original: L a t e n c y r a t i o ( r )

Original: S =1 S =2 S

Original: = 0.

Original: [Figure]

Original: ( r ) 0.

Original: L a t e n c y r a t i o ( r )

Original: S =

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: convolution, and then stores the results to the corresponding position on the residual feature map according to the index.

Original: This optimization can signiﬁcantly reduce the costs on memory access.

Original: Speed test.

Original: We test the latency on real hardware devices to evaluate the accuracy of our latency prediction model.

Original: On GPUs, we use TVM [2] and CUDA (version 11.6) for code generation and compilation respectively.

Original: The results in Fig.

Original: 4 of the paper validate the effectiveness of our model.

Original: B.2 ImageNet classiﬁcation

Original: As mentioned in the paper, we use pre-trained CNN models in the ofﬁcial torchvision website to initialize our backbone parameters, and ﬁnetune the overall models for 100 epochs.

Original: The initial learning rate is set as 0.01 × batch size/128, and decays with a cosine shape.

Original: The training batch size is determined on the model size and the GPU memory.

Original: For example, we train our LAS-ResNet-101 on 8 RTX 3090 GPUs with the batch size of 512, and the batch size for LAS-ResNet-50 is doubled.

Original: We use the same weight decay and the standard data augmentation as in the RegNet paper [25].

Original: For our own hyper-parameter τ in Eq.

Original: (1) of the paper, this Gumbel temperature τ exponentially decreases from 5 to 0.1 in the training procedure.

Original: For the training hyper-parameter in Eq.

Original: (2), we simply ﬁx α = 10 , β = 0 .

Original: [-5 and T = 4 . 
-]
Revised: [-5 and T = 4 . 
-]

Original: for all dynamic models.

Original: We conduct a very simple grid search with a

Original: RegNet for β ∈ { 0 . 3 , 0 . 5 } and T ∈ { 1 . 0 , 4 . 0 } to determine their values.

Original: B.3 COCO object detection & instance segmentation

Original: We use the standard setting suggested in [20, 21, 10], except that we decrease the learning rate for our pre-trained backbone network.

Original: We simply set a learning rate multiplier 0.5 for Faster R-CNN [27], 0.2 for RetinaNet [21] and 0.5 for Mask R-CNN [10].

Original: As for the additional loss items, the hyper-parameters are kept the same as training our classiﬁcation models, except that the temperature is ﬁxed as 0.1 in the 12 training epochs.

Original: C More experimental results

Original: In this section, we report more experimental results which are not presented in the main paper.

Original: C.1 Latency prediction

Original: In Figure 5 and Figure 6 of the paper, we report the latency prediction results of LAS-ResNet on V and TX2.

Original: Here we present the results on GTX1080 (Figure 10).

Original: It can be found that S net =

Original: 8-4-7- (which is the same to the optimal setting on V100 in the paper) will lead to faster inference on

Original: GTX1080.

Original: This is reasonable since GTX1080 has a large #PE than those IoT devices, and requires more contiguous memory access (a larger S for coarse-grained spatially adaptive inference) to achieve realistic speedup ( r (cid:96) < 1 ).

Original: The accuracy-latency curves in Figure 11 further validate this observation.

Original: C.2 ImageNet classiﬁcation

Original: Results on GTX1080 GPU.

Original: In Figure 7 of the paper, we report the ImageNet classiﬁcation results of LAS-ResNets on V100 and TX2.

Original: Here we present the results of LAS-ResNet on GTX (Figure 11).

Original: From the results in Figure 11, we can get the same conclusion as in the paper, that a proper large S is more parallel-friendly than the ﬁnest granularity.

Original: Remarkably, the latency of

Original: ResNet-101 could be reduced by 41% without sacriﬁcing the accuracy on GTX1080 under the spatial granularity setting of S net =8-4-7-1.

Original: With a similar inference latency, the accuracy of a static

Original: ResNet-101 could be increased by 1.0% by our LAS-ResNet-101 ( S net =4-4-2-1).

Original: Results of the extreme situation of S net =56-28-14-7.

Original: We mentioned in the paper that when we set S net =56-28-14-7, the spatially adaptive inference paradigm will reduce into layer skipping.

Original: We experiment on ResNet-50, and ﬁnd that although being slightly faster (13ms) than our LAS-ResNet- ( S net =4-4-2-1, 16ms), the accuracy of LAS-ResNet-50 with S net =56-28-14-7 could be signiﬁcantly degraded from 76.6% (ours, S net =4-4-2-1) to 76.1%.

Original: Therefore, we mainly focus on the discussion of spatially adaptive inference in this paper.

Original: C.3 More visualization results

Original: In addition to Figure 9 in the paper, here we present more visualization results of the regions selected by our masker in the 3-rd block of a LAS-ResNet-101 ( S net =4-4-2-1) in Figure 12, which demonstrate that our spatially adaptive inference paradigm can effectively locate the most task-related areas in image features, and reduce the unnecessary computation on those background areas.

Original: D Limitations

Original: The current limitations of our work include the following aspects:

Original: 1) the latency-ware co-designing framework is only constructed for spatial-wise dynamic networks.

Original: Support for more types of dynamic models ( e.g. channel skipping) will be explored in the future; 2) To achieve faster inference, the spatial masks are deﬁned the same for all input/output channels, which might limit the ﬂexibility of adaptive inference.

Original: Future work may explore more ﬂexible forms of dynamic computation; 3)

Original: The combination with other acceleration techniques such as Winograd, and the implementation on more CNN backbones may be worth studying in the future.

Original: Social impact.

Original: Our work can help reduce the inference cost of deep CNNs, but the training of our models might potentially increase the carbon emissions.