Original: D YNA T UNE : D YNAMIC T ENSOR P ROGRAM O PTIMIZATION IN D EEP N EURAL N ETWORK C OMPILATION
Revised: D YNA T UNE : D YNAMIC T ENSOR P ROGRAM O PTIMIZATION IN D EEP N EURAL N ETWORK C OMPILATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recently, the DL compiler, together with Learning to Compile has proven to be a powerful technique for optimizing deep learning models.
Revised: Recently, the DL compiler, together with Learning to Compile has proven to be a powerful technique for optimizing deep learning models.

Original: However, existing methods focus on accelerating the convergence speed of the individual tensor operator rather than the convergence speed of the entire model, which results in long optimization time to obtain a desired latency.
Revised: However, existing methods focus on accelerating the convergence speed of the individual tensor operator rather than the convergence speed of the entire model, which results in long optimization time to obtain a desired latency.

Original: In this paper, we present a new method called DynaTune, which provides signiﬁcantly faster convergence speed to optimize a DNN model.
Revised: In this paper, we present a new method called DynaTune, which provides signiﬁcantly faster convergence speed to optimize a DNN model.

Original: In particular, we consider a Multi-Armed Bandit (MAB) model for the tensor program optimization problem.
Revised: In particular, we consider a Multi-Armed Bandit (MAB) model for the tensor program optimization problem.

Original: We use UCB to handle the decision-making of time-slot-based optimization, and we devise a Bayesian belief model that allows predicting the potential performance gain of each operator with uncertainty quantiﬁcation, which guides the optimization process.
Revised: We use UCB to handle the decision-making of time-slot-based optimization, and we devise a Bayesian belief model that allows predicting the potential performance gain of each operator with uncertainty quantiﬁcation, which guides the optimization process.

Original: We evaluate and compare DynaTune with the state-of-the-art DL compiler.
Revised: We evaluate and compare DynaTune with the state-of-the-art DL compiler.

Original: The experiment results show that DynaTune is 1.2–2.4 times faster to achieve the same optimization quality for a range of models across different hardware architectures.
Revised: The experiment results show that DynaTune is 1.2–2.4 times faster to achieve the same optimization quality for a range of models across different hardware architectures.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The enormous computational intensity of Deep Neural Network (DNN) models has attracted great interest in optimizing their performance.
Revised: The enormous computational intensity of Deep Neural Network (DNN) models has attracted great interest in optimizing their performance.

Original: Popular deep learning (DL) frameworks such as PyTorch (Paszke et al., 2019) and TensorFlow (Abadi et al., 2016) adopt custom optimized kernels such as Intel MKL-DNN or Nvidia cuDNN (Chetlur et al., 2014) as back-end.
Revised: Popular deep learning (DL) frameworks such as PyTorch (Paszke et al., 2019) and TensorFlow (Abadi et al., 2016) adopt custom optimized kernels such as Intel MKL-DNN or Nvidia cuDNN (Chetlur et al., 2014) as back-end.

Original: However, given the increasing complexity of tensor operations in DNNs and the volatility of DL algorithms, it calls for developing fast and automated compilation frameworks to handle the unprecedented amount of innovations.
Revised: However, given the increasing complexity of tensor operations in DNNs and the volatility of DL algorithms, it calls for developing fast and automated compilation frameworks to handle the unprecedented amount of innovations.

Original: To imitate or even exceed the success of hand-optimized libraries, recent research has developed neural network compilers, such as XLA (Leary & Wang, 2017), Glow (Rotem et al., 2018), Tensor Comprehension (Vasilache et al., 2018), and TVM (Chen et al., 2018a).
Revised: To imitate or even exceed the success of hand-optimized libraries, recent research has developed neural network compilers, such as XLA (Leary & Wang, 2017), Glow (Rotem et al., 2018), Tensor Comprehension (Vasilache et al., 2018), and TVM (Chen et al., 2018a).

Original: Among them,
Revised: Among them,

Original: TVM has shown superior performance improvements using a technique called Learning to Compile (AutoTVM)
Revised: TVM has shown superior performance improvements using a technique called Learning to Compile (AutoTVM)

Original: (Chen et al., 2018b).
Revised: (Chen et al., 2018b).

Original: AutoTVM optimizes the code by generating many versions of a tensor operator and chooses the best through a learned cost model and search over a large space of code transformation choices.
Revised: AutoTVM optimizes the code by generating many versions of a tensor operator and chooses the best through a learned cost model and search over a large space of code transformation choices.

Original: While the Learning to Compile approach produces highly optimized code of DNN models, it suffers from excessively long optimization time.
Revised: While the Learning to Compile approach produces highly optimized code of DNN models, it suffers from excessively long optimization time.

Original: As an example, although AutoTVM is able to demonstrate close to 2 × performance improvement over TensorFlow on ResNet-18, the optimization time can take several hours or even tens of hours (Chen et al., 2018b).
Revised: As an example, although AutoTVM is able to demonstrate close to 2 × performance improvement over TensorFlow on ResNet-18, the optimization time can take several hours or even tens of hours (Chen et al., 2018b).

Original: The long optimization time hinders the turnaround time and even puts the practical utility of the current compiler-based solutions into question.
Revised: The long optimization time hinders the turnaround time and even puts the practical utility of the current compiler-based solutions into question.

Original: Recent works strive to reduce the optimization time by improving the search strategy for the code transformation plan and lowering the hardware measurement cost (Ahn et al., 2020;
Revised: Recent works strive to reduce the optimization time by improving the search strategy for the code transformation plan and lowering the hardware measurement cost (Ahn et al., 2020;

Original: Adams et al., 2019).
Revised: Adams et al., 2019).

Original: However, these approaches mostly focus on accelerating the convergence speed of optimization at the individual tensor operator level (e.g., Conv2D, batched GEMM), which may result in slow convergence and long optimization time of the entire model, which often contains tens of tensor operators.
Revised: However, these approaches mostly focus on accelerating the convergence speed of optimization at the individual tensor operator level (e.g., Conv2D, batched GEMM), which may result in slow convergence and long optimization time of the entire model, which often contains tens of tensor operators.

Original: Different from existing methods, we introduce DynaTune, a DL code optimization algorithm that minimizes the sum of the execution time of all operators in a model as much as possible and as quickly as possible.
Revised: Different from existing methods, we introduce DynaTune, a DL code optimization algorithm that minimizes the sum of the execution time of all operators in a model as much as possible and as quickly as possible.

Original: Speciﬁcally, the contributions of our paper consist of (1) a preliminary analysis that reveals the challenges and opportunities from existing DL code optimization strategies, (2) a time-slot-based optimization scheme, which simultaneously explores different operators and learns in an online manner that allows to dynamically switch to optimizing more promising tensors operators.
Revised: Speciﬁcally, the contributions of our paper consist of (1) a preliminary analysis that reveals the challenges and opportunities from existing DL code optimization strategies, (2) a time-slot-based optimization scheme, which simultaneously explores different operators and learns in an online manner that allows to dynamically switch to optimizing more promising tensors operators.

Original: (3) a Bayesian belief model that predicts future performance gains of operators, which helps make better decisions and expedites the convergence speed.
Revised: (3) a Bayesian belief model that predicts future performance gains of operators, which helps make better decisions and expedites the convergence speed.

Original: (4) a detailed evaluation of the proposed algorithm with modern DNNs (ResNet-18, VGG, SqueezeNet, Transformer) on both CPU and GPU.
Revised: (4) a detailed evaluation of the proposed algorithm with modern DNNs (ResNet-18, VGG, SqueezeNet, Transformer) on both CPU and GPU.

Original: Compared with the leading framework, AutoTVM, DynaTune is 1.2–2.4 × times faster to obtain the same levels of optimization.
Revised: Compared with the leading framework, AutoTVM, DynaTune is 1.2–2.4 × times faster to obtain the same levels of optimization.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: DL compilation pipeline.
Revised: DL compilation pipeline.

Original: A typical DL compiler contains multiple passes to optimize a model trained by popular DL frameworks such as TensorFlow (Abadi et al., 2016), PyTorch (Paszke et al., 2019), or MXNET (Chen et al., 2015), as shown in Fig.
Revised: A typical DL compiler contains multiple passes to optimize a model trained by popular DL frameworks such as TensorFlow (Abadi et al., 2016), PyTorch (Paszke et al., 2019), or MXNET (Chen et al., 2015), as shown in Fig.

Original: In the ﬁrst pass (box with dotted line), the compiler frontend applies target-independent and white-box target-dependent optimizations that do not include a measure of actual execution time.
Revised: In the ﬁrst pass (box with dotted line), the compiler frontend applies target-independent and white-box target-dependent optimizations that do not include a measure of actual execution time.

Original: The target-independent passes perform optimizations such as operator fusion and data layout transformation, and the white-box target-dependent optimizations apply heuristic rules for code transformation based on domain knowledge.
Revised: The target-independent passes perform optimizations such as operator fusion and data layout transformation, and the white-box target-dependent optimizations apply heuristic rules for code transformation based on domain knowledge.

Original: Recent work such as AutoTVM (Chen et al., 2018b) extends the pipeline with another pass, a black-box target-dependent pass, which uses learning machinery to perform optimizations.
Revised: Recent work such as AutoTVM (Chen et al., 2018b) extends the pipeline with another pass, a black-box target-dependent pass, which uses learning machinery to perform optimizations.

Original: Black-box target-dependent pass.
Revised: Black-box target-dependent pass.

Original: In this pass, the compiler converts code transformation decisions as code templates.
Revised: In this pass, the compiler converts code transformation decisions as code templates.

Original: A template contains knobs that control various aspects of the optimization (e.g., memory tiling, loop transformations, vectorization) and determines whether the code (1) fully utilizes the internal parallelism within processors, (2) uses the shared memory wisely, and (3) maximizes data locality.
Revised: A template contains knobs that control various aspects of the optimization (e.g., memory tiling, loop transformations, vectorization) and determines whether the code (1) fully utilizes the internal parallelism within processors, (2) uses the shared memory wisely, and (3) maximizes data locality.

Original: Due to the large transformation space, the compiler makes use of an auto-tuner (with an optimization algorithm) and real hardware measurements to ﬁnd the best transformation on target hardware (e.g., CPU, GPU, ARM, or IoT devices) (Chen et al., 2018b).
Revised: Due to the large transformation space, the compiler makes use of an auto-tuner (with an optimization algorithm) and real hardware measurements to ﬁnd the best transformation on target hardware (e.g., CPU, GPU, ARM, or IoT devices) (Chen et al., 2018b).

Original:  C HALLENGES AND M OTIVATIONS
Revised:  C HALLENGES AND M OTIVATIONS

Original: This section presents several studies that reveal the challenges of existing DL compilation that guided our design in Sec. 4.
Revised: This section presents several studies that reveal the challenges of existing DL compilation that guided our design in Section 4.

Original: Challenge 1.
Revised: Challenge 1.

Original: Existing DL compilation focuses on accelerating the convergence speed of individual tensor operator instead of the entire model, resulting in slow convergence and long optimization time.
Revised: Existing DL compilation focuses on accelerating the convergence speed of individual tensor operator instead of the entire model, resulting in slow convergence and long optimization time.

Original: Prior work (Chen et al., 2018a;b; Vasilache et al., 2018; Ahn et al., 2020) optimizes one tensor operator at a time in a predeﬁned order (e.g., in declaration order).
Revised: Prior work (Chen et al., 2018a;b; Vasilache et al., 2018; Ahn et al., 2020) optimizes one tensor operator at a time in a predeﬁned order (e.g., in declaration order).

Original: However, such an optimization strategy is not always appropriate in practice.
Revised: However, such an optimization strategy is not always appropriate in practice.

Original: For example, there is often an extreme performance difference (e.g., an order of magnitude) between optimized and unoptimized operators.
Revised: For example, there is often an extreme performance difference (e.g., an order of magnitude) between optimized and unoptimized operators.

Original: If optimizing operators sequentially, the overall model inference time stays high as long as there are still unoptimized operators.
Revised: If optimizing operators sequentially, the overall model inference time stays high as long as there are still unoptimized operators.

Original: As a result, practitioners may need to wait until all tensor operators have ﬁnished optimization to get the desired latency, which results in long optimization time.
Revised: As a result, practitioners may need to wait until all tensor operators have ﬁnished optimization to get the desired latency, which results in long optimization time.

Original: With the active research that has been pushing the model size to millions or even billion-scale parameters with a training time of only a few hours or less than one hour (Yamazaki et al., 2019; Goyal et al., 2017; You et al., 2017; Lin et al., 2019; Shoeybi et al., 2019; You et al., 2019), it becomes even more prominent to reduce the inference optimization cost of the current solution.
Revised: With the active research that has been pushing the model size to millions or even billion-scale parameters with a training time of only a few hours or less than one hour (Yamazaki et al., 2019; Goyal et al., 2017; You et al., 2017; Lin et al., 2019; Shoeybi et al., 2019; You et al., 2019), it becomes even more prominent to reduce the inference optimization cost of the current solution.

Original: Furthermore, since major players in the industry have adopted many of these DL compilers (Wu et al., 2019a;b; Lattner et al., 2020; Liu et al., 2019), fast convergence is desirable for many users of these pipelines to have a better control of the optimization cost and good performance.
Revised: Furthermore, since ma- jor players in the industry have adopted many of these DL compilers (Wu et al., 2019a;b; Lattner et al., 2020; Liu et al., 2019), fast convergence is desirable for many users of these pipelines to have a better control of the optimization cost and good performance.

Original: For example, deployment engineers may want to obtain an optimized model sooner or quickly get a latency upper-bound estimate of a model in development.
Revised: For example, deployment engineers may want to obtain an optimized model sooner or quickly get a latency upper-bound estimate of a model in development.

Original: Challenge 2.
Revised: Challenge 2.

Original: Static scheduling has only a limited view of the tensor program and has difﬁculty taking advantage of the actual optimization behavior.
Revised: Static scheduling has only a limited view of the tensor program and has difﬁculty taking advantage of the actual optimization behavior.

Original: We note that from an execution point of view, the optimization of tensor operators is independent of each other, so that we may optimize them in any order and even non-consecutively.
Revised: We note that from an execution point of view, the optimization of tensor operators is independent of each other, so that we may optimize them in any order and even non-consecutively.

Original: As a result, dynamic optimization has a big advantage for iterative DL compilation: We can intelligently order the optimization sequence of operators (i.e., scheduling) to accelerate the convergence of the optimization signiﬁcantly.
Revised: As a result, dynamic optimization has a big advantage for iterative DL compilation: We can intelligently order the optimization sequence of operators (i.e., scheduling) to accelerate the convergence of the optimization signiﬁcantly.

Original: For example, it would be better to switch to optimizing another operator if we convincingly identify that the other operator has a higher potential.
Revised: For example, it would be better to switch to optimizing another operator if we convincingly identify that the other operator has a higher potential.

Original: That being said, is it realistic to assume that all the information concerning optimizing the operators is available before the optimization even starts so that we can decide the schedule from the very beginning?
Revised: That being said, is it realistic to assume that all the information concerning optimizing the operators is available before the optimization even starts so that we can decide the schedule from the very beginning?

Revised: Our preliminary analysis indicates that the amount of computation of an operator (known a priori) has a very disproportionate impact on the optimization time and latency reduction.

Original: Fig.
Revised: Fig.

Original: 2 shows that the amount of computation of an operator(known a priori) has a very disproportionate impact on the optimization time and latency reduction, e.g., operators 17  takes the longest time to optimize. However, it yields the least amount of latency reduction.
Revised: 2 shows that although operator 17 of VGG (Simonyan & Zisserman, 2015) takes the longest time to optimize, it yields the least amount of latency reduction 1 .

Original: Our further investigation shows that the underlying code transformation space is nonlinear (Fig. 3).
Revised: Our further investigation shows that the underlying code transformation space is non-linear, as shown in Fig.

Original: As a result, the optimization behavior tends to change over time, which is hard to recognize and predict with static knowledge only.
Revised: As a result, the optimization behavior tends to change over time, which is hard to recognize and predict with static knowledge only.

Original: Challenge 3.
Revised: Challenge 3.

Original: Even with dynamic information, it is not clear how to best extrapolate estimated performance.
Revised: Even with dynamic information, it is not clear how to best extrapolate estimated performance.

Original: Given the optimization results, there is an incentive to adopt a ” predictthen-optimize ” paradigm that builds a model to learn the correlation between the optimization cost and the observed optimization performance the model can be used to make predictions for potential performance gains.
Revised: Given the optimization results, there is an incentive to adopt a ” predictthen-optimize ” paradigm that builds a model to learn the correlation between the optimization cost and the observed optimization performance the model can be used to make predictions for potential performance gains.

Original: To identify the characteristics of the optimization behavior, we plot 16 optimization curves of best-found GFLOPS (Giga Floating Point Operations Per Second) in Fig.
Revised: To identify the characteristics of the optimization behavior, we plot 16 optimization curves of best-found GFLOPS (Giga Floating Point Operations Per Second) in Fig.

Original: 4 to ﬁnd patterns that can be used for designing a prediction model.
Revised: 4 to ﬁnd patterns that can be used for designing a prediction model.

Original: We ﬁnd that most curves (1) roughly follow an increasing curve with a diminishing return, (2) saturate towards an unknown ﬁnal value, and (3) occasionally exhibit sudden jumps.
Revised: We ﬁnd that most curves (1) roughly follow an increasing curve with a diminishing return, (2) saturate towards an unknown ﬁnal value, and (3) occasionally exhibit sudden jumps.

Original: The curve saturates to an unknown value because the performance cannot exceed the hardware peak GFLOPS, which is 9.7-TFLOPS in our case.
Revised: The curve saturates to an unknown value because the performance cannot exceed the hardware peak GFLOPS, which is 9.7-TFLOPS in our case.

Original: The curve has sudden jumps because the code transformation space has change points, as shown in Fig. 3.
Revised: The curve has sudden jumps because the code transformation space has change points, as shown in Fig. 3.

Original: By taking into account the curve information, we believe it has more opportunity to dynamically optimize operators that likely lead to greater performance improvements.
Revised: By taking into account the curve information, we believe it has more opportunity to dynamically optimize operators that likely lead to greater performance improvements.

Original:  M ETHOD
Revised:  M ETHOD

Original: In this section, we present our design for DynaTune.
Revised: In this section, we present our design for DynaTune.

Original: We illustrate the difference between the existing
Revised: We illustrate the difference between the existing

Original: DL optimization (Fig. 5) and the high-level design of DynaTune (Fig. 6).
Revised: DL optimization (Fig. 5) and the high-level design of DynaTune (Fig. 6).

Original: 4. D YNAMIC M ULTI -T ENSOR -O PERATOR O PTIMIZATION P ROBLEM
Revised: 4. D YNAMIC M ULTI -T ENSOR -O PERATOR O PTIMIZATION P ROBLEM

Original: In this paper, we take a view of accelerating the convergence speed of multi-tensor-operator optimization by ﬁrst considering a Multi-Armed Bandits model for the problem.
Revised: In this paper, we take a view of accelerating the convergence speed of multi-tensor-operator optimization by ﬁrst considering a Multi-Armed Bandits model for the problem.

Original: In particular, we partition time into some ﬁxed-length time slots { 1 , 2 , ..., T } .
Revised: In particular, we partition time into some ﬁxed-length time slots { 1 , 2 , ..., T } .

Original: Similar to MAB, we deﬁne an operator scheduler that operates in discrete time steps.
Revised: Similar to MAB, we deﬁne an operator scheduler that operates in discrete time steps.

Original: At the beginning of any time slot t ∈ { 1 , 2 , ..., T } , the scheduler needs to choose an operator k t ∈
Revised: At the beginning of any time slot t ∈ { 1 , 2 , ..., T } , the scheduler needs to choose an operator k t ∈

Original: [ K ] for tuning in that slot.
Revised: [ K ] for tuning in that slot.

Original: The scheduler then obtains a list of observations (i.e., best-found performance measured in GFLOPS)
Revised: The scheduler then obtains a list of observations (i.e., best-found performance measured in GFLOPS)

Original: Perf k ( t · L k : ( t +1) · L k ) of k , where L k is the number of iterations an operator has been optimized.
Revised: Perf k ( t · L k : ( t +1) · L k ) of k , where L k is the number of iterations an operator has been optimized.

Original: The remaining unchosen operators stay the same unless it is selected (i.e., rested arms).
Revised: The remaining unchosen operators stay the same unless it is selected (i.e., rested arms).

Original: The latency reduction for k at time slot t is r t ( k t ) =
Revised: The latency reduction for k at time slot t is r t ( k t ) =

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation] [Table] [Equation] ] , where op ( k ) represents the number of ﬂoating point operations of k .
Revised: k ] , where op ( k ) represents the number of ﬂoating point operations of k .

Original: We further deﬁne an optimal schedule,i.e., π ∗ =
Revised: We further deﬁne an optimal schedule,i.e., π ∗ =

Original: { k ∗ 1 , ..., k ∗ t , ...
Revised: { k ∗ 1 , ..., k ∗ t , ...

Original: k ∗ T } , where k ∗ t is the operator at step t one could have taken in hindsight (after seeing all performance realizations) that would yield the maximum latency reduction.
Revised: k ∗ T } , where k ∗ t is the operator at step t one could have taken in hindsight (after seeing all performance realizations) that would yield the maximum latency reduction.

Original: The cumulative regret is then deﬁned as:R(T) =
Revised: The cumulative regret is then deﬁned as:

Original: (cid:80) Tt =1 ( r t ( k ∗ t ) − r t ( k t )) , where k ∗ t is the operator at step t one could have taken in hindsight (after seeing all performance realizations) that would yield the maximum latency reduction.
Revised: [Equation] =1 ( r t ( k ∗ t ) − r t ( k t )) , where k ∗ t is the operator at step t one could have taken in hindsight (after seeing all performance realizations) that would yield the maximum latency reduction.

Original: The goal is therefore to design the scheduler, such that the cumulative regret is minimized.
Revised: The goal is therefore to design the scheduler, such that the cumulative regret is minimized.

Original: We call this a dynamic multi-tensor-operator optimization problem .
Revised: We call this a dynamic multi-tensor-operator optimization problem .

Original: 4. T IME -S LOT -B ASED S CHEDULING
Revised: 4. T IME -S LOT -B ASED S CHEDULING

Original: To select which operator to optimize in a time slot, we consider deterministic selection policies that belong to the class of index-based MAB policies (Gittins et al., 2011).
Revised: To select which operator to optimize in a time slot, we consider deterministic selection policies that belong to the class of index-based MAB policies (Gittins et al., 2011).

Original: Among many options, we choose upper conﬁdence bound (UCB) as our action function to represent the exploration and exploitation tradeoff (Auer et al., 2002).
Revised: Among many options, we choose upper conﬁdence bound (UCB) as our action function to represent the exploration and exploitation tradeoff (Auer et al., 2002).

Original: In particular, assume c t [ k ] be the number of times that an operator k ∈
Revised: In particular, assume c t [ k ] be the number of times that an operator k ∈

Original: [ K ] has been selected for optimization up to slot t .
Revised: [ K ] has been selected for optimization up to slot t .

Original: If c t − 1 [ k ] > 0 , we denote y k ( t ) to be the estimated performance of operator k at the end of t .
Revised: If c t − 1 [ k ] > 0 , we denote y k ( t ) to be the estimated performance of operator k at the end of t .

Original: The UCB for operator k at t is deﬁned as u k ( t ) =
Revised: The UCB for operator k at t is deﬁned as u k ( t ) =

Original: r k ( t ) + (cid:113) C × logt c t
Revised: r k ( t ) + (cid:113) C × log

Original: [Equation]
Revised: [Equation]

Original: 1 [ k ] =
Revised: 1 [ k ] =

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ×
Revised: [ k ]

Original: [Equation]
Revised: [Equation]

Original: [ k ] for c t − 1 [ k ] >and u k ( t ) = γ (constant) for c t − 1 [ k ] = 0 .
Revised: and u k ( t ) = γ (constant) for c t − 1 [ k ] = 0 .

Original: The scheduler then computes UCB for each operator and selects the next one that maximizes UCB.
Revised: The scheduler then computes UCB for each operator and selects the next one that maximizes UCB.

Original: Our deﬁnition of UCB measures the potential latency reduction from an operator k compared to other operators’ expected performances.
Revised: Our deﬁnition of UCB measures the potential latency reduction from an operator k compared to other operators’ expected performances.

Original: The ﬁrst term is a point estimate of the future latency reduction, converted from GFLOPS improvement.
Revised: The ﬁrst term is a point estimate of the future latency reduction 3 , converted from GFLOPS improvement.

Original: The second term is related to the size (according to Chernoff-Hoeffding bounds (Hoeffding, 1994)) of the one-sided conﬁdence interval, which allows the true expected performance falls within with overwhelming probability.
Revised: The second term is related to the size (according to Chernoff-Hoeffding bounds (Hoeffding, 1994)) of the one-sided conﬁdence interval, which allows the true expected performance falls within with overwhelming probability.

Original: In the evaluation section, we evaluate several other action functions, including (cid:15) -greedy and softmax sampling.
Revised: In the evaluation section, we evaluate several other action functions, including (cid:15) -greedy and softmax sampling.

Original: We observe that these methods, in general, offer very similar performance.
Revised: We observe that these methods, in general, offer very similar performance.

Original: We choose UCB since some recent theoretical studies show that the growth rate of the regret from UCB is sublinear and long-run average optimal in the non-stationary bandit setting (Cortes et al., 2017).
Revised: We choose UCB since some recent theoretical studies show that the growth rate of the regret from UCB is sublinear and long-run average optimal in the non-stationary bandit setting (Cortes et al., 2017).

Original: 4. B AYESIAN B ELIEF M ODELS FOR E XPECTED P ERFORMANCE I MPROVEMENTS
Revised: 4. B AYESIAN B ELIEF M ODELS FOR E XPECTED P ERFORMANCE I MPROVEMENTS

Original: To obtain an optimal solution, k ∗ t requires perfect information, hence infeasible to achieve in practice.
Revised: To obtain an optimal solution, k ∗ t requires perfect information, hence infeasible to achieve in practice.

Original: Since we identify some patterns (in Sec. 3) that indicate that there is a functional relationship between the expected performance and the already observed performance, we propose a Bayesian belief model f k ( t ) , enabled by Markov Chain Monte Carlo (MCMC), to capture our current knowledge of the optimization results to predict future performance and get updated as new observations become available.
Revised: Since we identify some patterns (in Sec. 3) that indicate that there is a functional relationship between the expected performance and the already observed performance, we propose a Bayesian belief model f k ( t ) , enabled by Markov Chain Monte Carlo (MCMC), to capture our current knowledge of the optimization results to predict future performance and get updated as new observations become available.

Original: In particular, we choose parametric curve models whose shape coincides with our knowledge about the form of optimization curves: increasing, saturating functions such as those from the power-law ( pow2 , pow4 , log power ) or the logarithmic family ( loglinear , logloglinear ).
Revised: In particular, we choose parametric curve models whose shape coincides with our knowledge about the form of optimization curves: increasing, saturating functions such as those from the power-law ( pow2 , pow4 , log power ) or the logarithmic family ( loglinear , logloglinear ).

Original: Among these functions, we choose log power function as it works well in our case.
Revised: Among these functions, we choose log power function as it works well in our case.

Original: To handle breakpoints as mentioned in Sec. 3, we employ a piece-wise parameterization to improve the approximation to the shape of the underlying relationship: since the abrupt jumps often happen just a few times, we cut observation x into segments if a jump leads to more than a relative ∆
Revised: To handle breakpoints as mentioned in Sec. 3, we employ a piece-wise parameterization to improve the approximation to the shape of the underlying relationship: since the abrupt jumps often happen just a few times, we cut observation x into segments if a jump leads to more than a relative ∆

Original: Algorithm
Revised: Algorithm

Revised: A

Original: 1: Input: A model Φ with a list of [K] = { 1,...,K } operators 2: Output: An optimized model Φ ∗3:
Revised: with a list of K] { 1,...,K } operators : Output: An optimized model Φ ∗

Original: Init: c = (0, 0,...,0) 4: for t = 1,...,T do 5: fork = 1,...,K do 6: 
Revised: Init do : for do 6: Observe history

Revised: performance

Revised: of

Original: Observe history performance of operator k and predict a future performance using thebelief model in Section 4.
Revised: operator k and predict a future performance using the

Revised: belief model

Original:  Update the UCB value u k ( t ) using the equation in Section 4.
Revised: in Section 4. Update the UCB value u k ( t ) using the equation in Section 4.

Original: 8: k t ← arg max u k ( t ) 9:
Revised: 8: k t ← arg max u k ( t ) 9:

Original: Allocate time slot L to k t for actual optimization 10: c [ k ] ← c [ k ] +11: if k.ﬁnished then 12:
Revised: Allocate time slot L to k t for actual optimization 10: c [ k ] ← c [ k ] +11: if k.ﬁnished then 12:

Original: Exclude k from [K] 13:
Revised: Exclude k from [K] 13:

Original: Collect new data of observed GFLOPS from optimizing k and update the belief model
Revised: Collect new data of observed GFLOPS from optimizing k and update the belief model

Original: Data normalization.
Revised: Data normalization.

Original: Before feeding the observations to the model, since the observed GFLOPS of different operators has different ranges, we normalize the GFLOPS of all operators to a common scale.
Revised: Before feeding the observations to the model, since the observed GFLOPS of different operators has different ranges, we normalize the GFLOPS of all operators to a common scale.

Original: Given that the maximum GFLOPS of any operator is bounded by the theoretical peak performance on the target deployment hardware (e.g., 9.3-TeraFLOPS on a Nvidia P
Revised: Given that the maximum GFLOPS of any operator is bounded by the theoretical peak performance on the target deployment hardware (e.g., 9.3-TeraFLOPS on a Nvidia P

Original: GPU), we apply the following formula to normalize observation x as follows:
Revised: GPU), we apply the following formula to normalize observation x as follows:

Original: Normalized x = x/Peak ( target hardware ) which transfers the observation’s values to a new range 0 to 1.
Revised: Normalized x = x/Peak ( target hardware ) which transfers the observation’s values to a new range 0 to 1.

Original: We convert the normalized x back to GFLOPS when calculating UCB.
Revised: We convert the normalized x back to GFLOPS when calculating UCB.

Original: Modeling uncertainty.
Revised: Modeling uncertainty.

Original: Since our goal aims at allocating time to optimize tensor operators that are highly likely to bring performance improvement, we need to model uncertainty as truthfully as possible.
Revised: Since our goal aims at allocating time to optimize tensor operators that are highly likely to bring performance improvement, we need to model uncertainty as truthfully as possible.

Original: To model uncertainty, we perform MCMC sampling from the posterior p ( θ k | Perf k [1 : t · L k ])
Revised: To model uncertainty, we perform MCMC sampling from the posterior p ( θ k | Perf k [1 : t · L k ])

Original: ∝ p ( Perf k [1 : t · L k ] | θ k ) p ( θ k ) of model parameters θ k of the curve function f k ( t ) given the observed performance Perf k [1 : t · L k ] .
Revised: ∝ p ( Perf k [1 : t · L k ] | θ k ) p ( θ k ) of model parameters θ k of the curve function f k ( t ) given the observed performance Perf k [1 : t · L k ] .

Original: A sample approximation for Perf k [ t (cid:48) ] with t (cid:48) > t can then be formed as 
Revised: A sample approximation for Perf k [ t (cid:48) ] with t (cid:48) > t can then be formed as [Equation]

Original: [Equation]
Revised: [Equation]

Original: Among many options to do MCMC sampling, we choose Goodman & Weare’s Afﬁne Invariant
Revised: Among many options to do MCMC sampling, we choose Goodman & Weare’s Afﬁne Invariant

Original: MCMC Ensemble sampler (Goodman & Weare, 2010), which signiﬁcantly outperforms standard
Revised: MCMC Ensemble sampler (Goodman & Weare, 2010), which signiﬁcantly outperforms standard

Original: M-H methods and produces independent samples (which takes O(N) likelihood evaluations) with a much shorter autocorrelation time.
Revised: M-H methods and produces independent samples (which takes O(N) likelihood evaluations) with a much shorter autocorrelation time.

Original: We initialize the ensemble samplers in a tight N-dimensional
Revised: We initialize the ensemble samplers in a tight N-dimensional

Original: Gaussian ball in parameter space around the maximum likelihood result, which is obtained through non-linear least-squares ﬁt.
Revised: Gaussian ball in parameter space around the maximum likelihood result, which is obtained through non-linear least-squares ﬁt.

Original: We set a uniform prior for θ and make sure the prediction is nondecreasing, i.e., the predicted performance is not worse than the more recently observed best-found performance, by also explicitly encoding this knowledge into the prior.
Revised: We set a uniform prior for θ and make sure the prediction is nondecreasing, i.e., the predicted performance is not worse than the more recently observed best-found performance, by also explicitly encoding this knowledge into the prior.

Original: We obtain the predictive mean µ and standard deviation σ of the posterior parameter distribution using 200 MCMC samples.
Revised: We obtain the predictive mean µ and standard deviation σ of the posterior parameter distribution using 200 MCMC samples.

Original: We then compute the expected positive improvement EI (Brochu et al., 2010) over the best known measured performance, while taking into account the possible uncertainty in that prediction.
Revised: We then compute the expected positive improvement EI (Brochu et al., 2010) over the best known measured performance, while taking into account the possible uncertainty in that prediction.

Original:  E VALUATION
Revised:  E VALUATION

Original: In this section, we evaluate DynaTune experimentally, seeking answers to how DynaTune helps accelerate the convergence of optimizing all operators in a model.
Revised: In this section, we evaluate DynaTune experimentally, seeking answers to how DynaTune helps accelerate the convergence of optimizing all operators in a model.

Original: We integrate DynaTune with
Revised: We integrate DynaTune with

Original: AutoTVM (Chen et al., 2018b) and use as AutoTVM our baseline for comparison.
Revised: AutoTVM (Chen et al., 2018b) and use as AutoTVM our baseline for comparison.

Original: We implement
Revised: We implement

Original: DynaTune in Python, and we leverage emcee (Foreman-Mackey et al., 2013) to implement the
Revised: DynaTune in Python, and we leverage emcee (Foreman-Mackey et al., 2013) to implement the

Original: MCMC sampling.
Revised: MCMC sampling.

Original: We do a warmup to ﬁnd the ﬁrst non-zero GFLOPS as a starting point for all operators.
Revised: We do a warmup to ﬁnd the ﬁrst non-zero GFLOPS as a starting point for all operators.

Original: We use the default hyperparameters provided by AutoTVM for the underlying code optimization.
Revised: We use the default hyperparameters provided by AutoTVM for the underlying code optimization.

Original: To obtain the parameter posterior, we run the ensemble MCMC with 10 walkers and 500 sampling steps.
Revised: To obtain the parameter posterior, we run the ensemble MCMC with 10 walkers and 500 sampling steps.

Original: A convergence diagnostics of MCMC is presented in Appendix B.
Revised: A convergence diagnostics of MCMC is presented in Appendix B.

Original: All the free parameters in the curve model are taken care of by MCMC sampling.
Revised: All the free parameters in the curve model are taken care of by MCMC sampling.

Original: For UCB, we use C = 0 .if
Revised: For UCB, we use C = 0 .if

Original: the initial latency of an operator is < 1ms, and C = 2 for all the other cases.
Revised: the initial latency of an operator is < 1ms, and C = 2 for all the other cases.

Original: We performindependent runs of each conﬁguration with different random seeds and report the median together with a 95% conﬁdence interval.
Revised: We performindependent runs of each conﬁguration with different random seeds and report the median together with a 95% conﬁdence interval.

Original: 5. C OMPARISON OF A UTO TVM AND D YNA T UNE FOR O PTIMIZING M ODELS WITH M ULTIPLE T ENSOR O PERATORS
Revised: 5. C OMPARISON OF A UTO TVM AND D YNA T UNE FOR O PTIMIZING M ODELS WITH M ULTIPLE T ENSOR O PERATORS

Original: In the previous approach (Chen et al., 2018b), authors optimize one operator at a time until all operators in a model have been optimized.
Revised: In the previous approach (Chen et al., 2018b), authors optimize one operator at a time until all operators in a model have been optimized.

Original: We compare the performance of AutoTVM and DynaTune on how much optimization speedup we obtain as a function of the wall-clock time.
Revised: We compare the performance of AutoTVM and DynaTune on how much optimization speedup we obtain as a function of the wall-clock time.

Original: Due to space limitations, we include four tasks, covering both CPU and GPU hardware: ResNet-18 (He et al., 2016) and SqueezeNet (Iandola et al., 2016) on CPU (Intel Xeon CPU E5-2690 v3 @ 2.60GHz 2600 MHz), VGG (Simonyan & Zisserman, 2015)
Revised: Due to space limitations, we include four tasks, covering both CPU and GPU hardware: ResNet-18 (He et al., 2016) and SqueezeNet (Iandola et al., 2016) on CPU (Intel Xeon CPU E5-2690 v3 @ 2.60GHz 2600 MHz), VGG (Simonyan & Zisserman, 2015)

Original: Transformer Encoder (Iandola et al., 2016) on
Revised: Transformer Encoder (Iandola et al., 2016) on

Original: GPUs (Nvidia Tesla P100), which have K = 12, 18, 18, and 6 tunable operators, respectively.
Revised: GPUs (Nvidia Tesla P100), which have K = 12, 18, 18, and 6 tunable operators, respectively.

Original: Fig.
Revised: Fig.

Original: 7 visualizes the results.
Revised: 7 visualizes the results.

Original: The x-axis denotes the wall-clock time of optimizing the model.
Revised: The x-axis denotes the wall-clock time of optimizing the model.

Original: The y-axis denotes the lowest model latency obtained as time moves on.
Revised: The y-axis denotes the lowest model latency obtained as time moves on.

Original: Overall, the observation is that
Revised: Overall, the observation is that

Original: DynaTune’s dynamic optimization converges signiﬁcantly faster than the baseline most of the time and is 1.2–2.4 times faster than the baseline to achieve the lowest latency.
Revised: DynaTune’s dynamic optimization converges signiﬁcantly faster than the baseline most of the time and is 1.2–2.4 times faster than the baseline to achieve the lowest latency.

Original: The baseline has a much slower convergence, because it tries to ﬁnd optimal transformation plan for one operator before starting optimizing another one.
Revised: The baseline has a much slower convergence, because it tries to ﬁnd optimal transformation plan for one operator before starting optimizing another one.

Original: Since there can be an order of magnitude difference between the optimized and unoptimized code, the model latency remains high until the last operator has been optimized.
Revised: Since there can be an order of magnitude difference between the optimized and unoptimized code, the model latency remains high until the last operator has been optimized.

Original: In contrast, DynaTune is able to expedite the optimization signiﬁcantly by reordering the optimization sequence of operators and dynamically pick promising operators to optimize.
Revised: In contrast, DynaTune is able to expedite the optimization signiﬁcantly by reordering the optimization sequence of operators and dynamically pick promising operators to optimize.

Original: As a result, DynaTune obtains the same optimization quality as the baseline but in a much faster speed.
Revised: As a result, DynaTune obtains the same optimization quality as the baseline but in a much faster speed.

Original: Furthermore, we also plot the optimization by assuming having access to the oracle information.
Revised: Furthermore, we also plot the optimization by assuming having access to the oracle information.

Original: For
Revised: For

Original: ResNet-18 and VGG, DynaTune gets optimization results close to the oracle.
Revised: ResNet-18 and VGG, DynaTune gets optimization results close to the oracle.

Original: For SqueezeNet and
Revised: For SqueezeNet and

Original: Transformer, DynaTune converges slower than the oracle in the beginning but quickly catches up at around one third of the optimization time, presumably because it is more difﬁcult for the Bayesian model to predict performance in the beginning then later, indicating room for improvement.
Revised: Transformer, DynaTune converges slower than the oracle in the beginning but quickly catches up at around one third of the optimization time, presumably because it is more difﬁcult for the Bayesian model to predict performance in the beginning then later, indicating room for improvement.

Original: These results conﬁrm that a dynamic approach like DynaTune is capable of performing model-level code optimization in a much more efﬁcient way than the existing approach.
Revised: These results conﬁrm that a dynamic approach like DynaTune is capable of performing model-level code optimization in a much more efﬁcient way than the existing approach.

Original: 5. C OMPARISON OF D YNA T UNE WITH S TATIC S CHEDULES
Revised: 5. C OMPARISON OF D YNA T UNE WITH S TATIC S CHEDULES

Original: In this section, we compare the effectiveness of DynaTune with static allocation schemes.
Revised: In this section, we compare the effectiveness of DynaTune with static allocation schemes.

Original: In particular, we compare with three mechanisms: (1) Random, which randomly assign time slots to operators, (2) Round-robin, which assigns time slots in circular order, and (3) Linear, which allocate time linearly with respect to the number of ﬂoating-point operations each operator has.
Revised: In particular, we compare with three mechanisms: (1) Random, which randomly assign time slots to operators, (2) Round-robin, which assigns time slots in circular order, and (3) Linear, which allocate time linearly with respect to the number of ﬂoating-point operations each operator has.

Original: Fig.
Revised: Fig.

Original: 8 shows that DynaTune consistently outperforms other static schemes and achieves 1.1–2.4 times speedup to obtain the lowest latency.
Revised: 8 shows that DynaTune consistently outperforms other static schemes and achieves 1.1–2.4 times speedup to obtain the lowest latency.

Original: As mentioned in Sec. 3, static knowledge alone is insufﬁcient for making well-suited schedule decisions.
Revised: As mentioned in Sec. 3, static knowledge alone is insufﬁcient for making well-suited schedule decisions.

Original: In contrast, the improvement in DynaTune comes from constantly making decisions and replaning the allocation strategy based on new observations.
Revised: In contrast, the improvement in DynaTune comes from constantly making decisions and replaning the allocation strategy based on new observations.

Original: 5. C OMPARISON OF D YNA T UNE WITH A LTERNATIVE D YNAMIC S CHEMES
Revised: 5. C OMPARISON OF D YNA T UNE WITH A LTERNATIVE D YNAMIC S CHEMES

Original: We also compare the effectiveness of our approach by comparing the following dynamic schemes: (1) Dynamic allocation (DA) + Random selection (Rand): Randomly assigns time slots to operators and reallocates a time slot if the chosen operator has been early terminated.
Revised: We also compare the effectiveness of our approach by comparing the following dynamic schemes: (1) Dynamic allocation (DA) + Random selection (Rand): Randomly assigns time slots to operators and reallocates a time slot if the chosen operator has been early terminated.

Original: (2) Dynamic allocation (DA) + Round-robin selection (RR): Assigns time slots in circular order, the other is the same as the ”DA + Rand” conﬁguration.
Revised: (2) Dynamic allocation (DA) + Round-robin selection (RR): Assigns time slots in circular order, the other is the same as the ”DA + Rand” conﬁguration.

Original: For all schemes, we use the same early termination condition, e.g., an operator is considered as ﬁnished if it has not seen any GFLOPS improvement for a given number of iterations (e.g., 200).
Revised: For all schemes, we use the same early termination condition, e.g., an operator is considered as ﬁnished if it has not seen any GFLOPS improvement for a given number of iterations (e.g., 200).

Original: Fig.
Revised: Fig.

Original: 9 shows that, when not equipped with the Bayesian model (e.g., Rand and RR), the optimization converges relatively slower than with the model to obtain a lower latency.
Revised: 9 shows that, when not equipped with the Bayesian model (e.g., Rand and RR), the optimization converges relatively slower than with the model to obtain a lower latency.

Original: In contrast, DynaTune achieves 1–1.4 times speedup to obtain the lowest latency, presumably because the prediction helps to identify operators that potentially bring high latency reduction.
Revised: In contrast, DynaTune achieves 1–1.4 times speedup to obtain the lowest latency, presumably because the prediction helps to identify operators that potentially bring high latency reduction.

Original: We also evaluated DynaTune with two alternative action functions: (cid:15) -greedy and softmax sampling.
Revised: We also evaluated DynaTune with two alternative action functions: (cid:15) -greedy and softmax sampling.

Original: As Fig. 13 shown, while UCB offers marginal gains in some cases, the performance of all three action functions are very similarly (more results can be found in Appendix A), indicating that potentially all three action functions can be used for operator selection. 
Revised: As Fig. 14 shown, while UCB offers marginal gains in some cases, the performance of all three action functions are very similarly (more results can be found in Appendix A), indicating that potentially all three action functions can be used for operator selection. 5. C OMPARISON WITH C HAMELEON

Original: [Figure]
Revised: Chameleon (Ahn et al., 2020) is a recent work that uses reinforcement learning and adaptive sampling to efﬁciently explore the code transformation space of a tensor operator. In this part, we compare DynaTune with the open-sourced version of Chameleon 4 . We follow the provided instructions and evaluate Chameleon on the same set of models: ResNet-18 and SqueezeNet on CPU, VGG and Transformer on GPU. Figure 10c and Figure 10d show that although Chameleon is faster than the baseline AutoTVM on VGG and Transformer on GPU, it is slower than DynaTune to converge to the lowest latency.

Revised: This is because although the optimization speed of each operator has been improved by Chameleon, it still sequentially optimizes one operator at a time.

Revised: Therefore, the overall convergence speed is still bounded by the least optimized tensor operators.

Revised: In contrast, DynaTune focuses on improving the convergence speed of multi-tensor-operators and dynamically allocates time to improve the overall convergence speed.

Revised: We also evaluate Chameleon on CPU, given that model inference is not uncommon on CPU (Wu et al., 2019a; Zhang et al., 2019; 2018).

Revised: When running on CPU, a bit surprisingly, we observe that Chameleon is slower than the baseline AutoTVM on ResNet-18 (Figure 10a) and SqueezeNet (Figure 10b).

Revised: We analyze the performance and ﬁnd that the RL optimizer in Chameleon adds a non-trivial amount of overhead than the default optimizer (SA + XGBoost) used in AutoTVM on

Revised: CPU.

Revised: As a result, although Chameleon reduces the hardware measurement cost and the number of iterations to ﬁnd the best conﬁguration, its optimization time on CPU is longer than the baseline

Revised: AutoTVM because of this extra overhead and is therefore also slower than DynaTune on CPU.

Revised: Overall, DynaTune is 1.4–4.7 times faster than Chameleon to reach the same latency.

Revised: Although

Revised: DynaTune is faster than Chameleon, we want to point out that DynaTune can be combined with

Revised: Chameleon to achieve better performance, at least on GPU.

Revised: [Figure]

Revised: [Figure]

Original: Inference time comparison Fig.
Revised: Inference time comparison Fig.

Original: 10 compares the ﬁnal inference time optimized by TVM, AutoTVM, and DynaTune respectively.
Revised: 11 compares the ﬁnal inference time optimized by TVM, AutoTVM, and DynaTune respectively.

Original: Overall, DynaTune achieves up to 26.7% faster inference speed over TVM and similar code performance as AutoTVM.
Revised: Overall, DynaTune achieves up to 26.7% faster inference speed over TVM and similar code performance as AutoTVM.

Original: AutoTVM and DynaTune achieve much higher speedup on ResNet-18 and Transformer, presumably because the heuristic-based optimizations in TVM are sub-optimal.
Revised: AutoTVM and DynaTune achieve much higher speedup on ResNet-18 and Transformer, presumably because the heuristic-based optimizations in TVM are sub-optimal.

Original: While achieving comparable optimization quality as AutoTVM,
Revised: While achieving comparable optimization quality as AutoTVM,

Original: DynaTune signiﬁcantly reduces the lengthy optimization time.
Revised: DynaTune signiﬁcantly reduces the lengthy optimization time.

Original: Segment curve ﬁtting.
Revised: Segment curve ﬁtting.

Original: Fig.
Revised: Fig.

Original: 12 gives an example on the curve ﬁtting.
Revised: 13 gives an example on the curve ﬁtting.

Original: It is easy to see that the data between 1 and 80 could be approximated well by one segment, and the data between 80 andcould be approximated by another segment.
Revised: It is easy to see that the data between 1 and 80 could be approximated well by one segment, and the data between 80 andcould be approximated by another segment.

Original: Cost breakdown.
Revised: Cost breakdown.

Original: Fig.
Revised: Fig.

Original: 11 shows the breakdown in the average time required scheduling a time slot (averaged across four models), which involves the optimization time (e.g., 100s) and the scheduling cost.
Revised: 12 shows the breakdown in the average time required scheduling a time slot (averaged across four models), which involves the optimization time (e.g., 100s) and the scheduling cost.

Original: Overall, baseline (sequential), random, and round-robin incur almost no overhead in scheduling.
Revised: Overall, baseline (sequential), random, and round-robin incur almost no overhead in scheduling.

Original: Our Bayesian belief model takes time to do ﬁtting (MLE and MCMC) and inference.
Revised: Our Bayesian belief model takes time to do ﬁtting (MLE and MCMC) and inference.

Original: However, it only adds 1.44s, which adds 1.4% overhead and can be easily compensated by the time saved from picking a promising operator to optimize.
Revised: However, it only adds 1.44s, which adds 1.4% overhead and can be easily compensated by the time saved from picking a promising operator to optimize.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: DynaTune uniquely offers a solution that exclusively enables (i) Multi-Armed Bandits based dynamic multi-tensor-operator optimization and (ii) extrapolation of optimization performance by
Revised: DynaTune uniquely offers a solution that exclusively enables (i) Multi-Armed Bandits based dynamic multi-tensor-operator optimization and (ii) extrapolation of optimization performance by

Original: Bayesian inference in the context of (iii) optimizing DL compilers.
Revised: Bayesian inference in the context of (iii) optimizing DL compilers.

Original: As such, we discuss the related work from each of the three independent research directions.
Revised: As such, we discuss the related work from each of the three independent research directions.

Original: Optimizing compilers.
Revised: Optimizing compilers.

Original: TVM (Chen et al., 2018a) and TensorComprehensions (Vasilache et al., 2018) use simulated annealing and genetic algorithm to search the best code transformation plan for neural networks.
Revised: TVM (Chen et al., 2018a) and TensorComprehensions (Vasilache et al., 2018) use simulated annealing and genetic algorithm to search the best code transformation plan for neural networks.

Original: Subsequently, AutoTVM (Chen et al., 2018b) incorporates boosted decision trees as a surrogate model to reduce the number of real hardware measurements.
Revised: Subsequently, AutoTVM (Chen et al., 2018b) incorporates boosted decision trees as a surrogate model to reduce the number of real hardware measurements.

Original: CHAMELEON (Ahn et al., 2020) proposes to use reinforcement learning for efﬁcient search space exploration.
Revised: CHAMELEON (Ahn et al., 2020) proposes to use reinforcement learning for efﬁcient search space exploration.

Original: However, while existing approaches study how to accelerate the convergence of individual operators,
Revised: However, while existing approaches study how to accelerate the convergence of individual operators,

Original: DynaTune focuses on improving the convergence speed of the entire model using Multi-Armed
Revised: DynaTune focuses on improving the convergence speed of the entire model using Multi-Armed

Original: Bandit learning for dynamic optimization cross all tensor operators.
Revised: Bandit learning for dynamic optimization cross all tensor operators.

Original: DynaTune can be combined with other approaches that speed up the convergence of individual operators to maximize gains.
Revised: DynaTune can be combined with other approaches that speed up the convergence of individual operators to maximize gains.

Original: Program autotuning has also been studied in the context of generic programming, where black box optimization (Ansel et al., 2014) and runtime information (Novillo, 2014) are utilized to generate optimized code.
Revised: Program autotuning has also been studied in the context of generic programming, where black box optimization (Ansel et al., 2014) and runtime information (Novillo, 2014) are utilized to generate optimized code.

Original: Rather than advancing the generic problem of program auto-tuning, our technique is designed and validated speciﬁcally for the DL model compilation problem.
Revised: Rather than advancing the generic problem of program auto-tuning, our technique is designed and validated speciﬁcally for the DL model compilation problem.

Original: Multi-Armed Bandits.
Revised: Multi-Armed Bandits.

Original: The Multi-Armed Bandits (MAB), ﬁrst introduced by Robbins (Robbins, 1952), with various modiﬁcations have been used to model a plethora of dynamic optimization problems under uncertainty (Zelen, 1969; Bergemann & V ¨ alim ¨ aki, 1996; Kleinberg & Leighton, 2003; Awerbuch & Kleinberg, 2004; Bergemann & Hege, 2005; Caro & Gallien, 2007; Pandey et al., 2007).
Revised: The Multi-Armed Bandits (MAB), ﬁrst introduced by Robbins (Robbins, 1952), with various modiﬁcations have been used to model a plethora of dynamic optimization problems under uncertainty (Zelen, 1969; Bergemann & V ¨ alim ¨ aki, 1996; Kleinberg & Leighton, 2003; Awerbuch & Kleinberg, 2004; Bergemann & Hege, 2005; Caro & Gallien, 2007; Pandey et al., 2007).

Original: To the best of our knowledge, our work exclusively explores a different problem, which is optimizing DL compilers using MAB.
Revised: To the best of our knowledge, our work exclusively explores a different problem, which is optimizing DL compilers using MAB.

Original: Furthermore, our problem can be viewed as an instance of a rested, non-stationary MAB, which is complicated to solve since the evolution of the stochastic process depends on the choices made by the algorithm.
Revised: Furthermore, our problem can be viewed as an instance of a rested, non-stationary MAB, which is complicated to solve since the evolution of the stochastic process depends on the choices made by the algorithm.

Original: We are able to design new UCB-style algorithms that improve the DL compilation in this speciﬁc non-stationary rested bandit scenario.
Revised: We are able to design new UCB-style algorithms that improve the DL compilation in this speciﬁc non-stationary rested bandit scenario.

Original: Bayesian Inference.
Revised: Bayesian Inference.

Original: Bayesian inference is a broad ﬁeld and has been widely used in statistical analysis (Box & Tiao, 2011).
Revised: Bayesian inference is a broad ﬁeld and has been widely used in statistical analysis (Box & Tiao, 2011).

Original: For example, Bayesian inference has been used in hyperparameter tuning (Domhan et al., 2015; Lu et al., 2019).
Revised: For example, Bayesian inference has been used in hyperparameter tuning (Domhan et al., 2015; Lu et al., 2019).

Original: DynaTune shares similarities with these methods in building a Bayesian model for prediction, but it differs in its context and has its unique challenges.
Revised: DynaTune shares similarities with these methods in building a Bayesian model for prediction, but it differs in its context and has its unique challenges.

Original: The Bayesian belief model we design is to predict operators that may lead to greater latency reduction whilst performing an optimization to accelerate the process.
Revised: The Bayesian belief model we design is to predict operators that may lead to greater latency reduction whilst performing an optimization to accelerate the process.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Although highly optimized code can be achieved through existing DL compilers, an obvious drawback is that they optimize operators one at a time, leading to slow convergence of optimization speed when the model has multiple operators.
Revised: Although highly optimized code can be achieved through existing DL compilers, an obvious drawback is that they optimize operators one at a time, leading to slow convergence of optimization speed when the model has multiple operators.

Original: In this paper we have introduced a method, called
Revised: In this paper we have introduced a method, called

Original: DynaTune, which treats the optimization of multiple operators in a model as a whole and dynamically optimizes all operators to expedite convergence.
Revised: DynaTune, which treats the optimization of multiple operators in a model as a whole and dynamically optimizes all operators to expedite convergence.

Original: Combined with a Bayesian belief model, the dynamic optimization prioritizes operators that have larger latency reduction.
Revised: Combined with a Bayesian belief model, the dynamic optimization prioritizes operators that have larger latency reduction.

Original: As a result, DynaTune achieves much faster convergence speed in getting optimized models, outperforming the state-of-the-art approach.
Revised: As a result, DynaTune achieves much faster convergence speed in getting optimized models, outperforming the state-of-the-art approaches.

Revised: A CKNOWLEDGEMENT

Revised: The authors appreciate the anonymous ICLR reviewers for providing constructive feedback for improving the quality of this paper.

Revised: All authors are not funded by any other agency.

Original: A C OMPARISON WITH (cid:15) - GREEDY AND SOFTMAX SAMPLING
Revised: A C OMPARISON WITH (cid:15) - GREEDY AND SOFTMAX SAMPLING

Original: We compare DynaTune with two alternative operator selection methods: (1) (cid:15) -greedy:
Revised: We compare DynaTune with two alternative operator selection methods: (1) (cid:15) -greedy:

Original: Like our approach, but UCB is replaced with (cid:15) -greedy, which select the operator with the highest expected latency reduction based on EI with probability 1 − (cid:15) (e.g., (cid:15) =0.05) and randomly select an operator with (cid:15) to ensure exploration.
Revised: Like our approach, but UCB is replaced with (cid:15) -greedy, which select the operator with the highest expected latency reduction based on EI with probability 1 − (cid:15) (e.g., (cid:15) =0.05) and randomly select an operator with (cid:15) to ensure exploration.

Original: (2) Softmax: Like our approach but UCB is replaced with softmax sampling, which stochastically samples operators to optimize based on expected latency reduction at each step.
Revised: (2) Softmax: Like our approach but UCB is replaced with softmax sampling, which stochastically samples operators to optimize based on expected latency reduction at each step.

Original: Fig.
Revised: Fig.

Original: 14–Fig.
Revised: 15–Fig.

Original: 17 show the comparison results.
Revised: 18 show the comparison results.

Original: With the Bayesian model enabled in cases, we observe that softmax sampling, (cid:15) -greedy, and UCB work very similarly across multiple models.
Revised: With the Bayesian model enabled in cases, we observe that softmax sampling, (cid:15) -greedy, and UCB work very similarly across multiple models.

Original: We choose UCB in our design because it has been theoretically studied that UCB has a sublinear regret growth rate.
Revised: We choose UCB in our design because it has been theoretically studied that UCB has a sublinear regret growth rate.

Original: B MCMC A NALYSIS
Revised: B MCMC A NALYSIS

Original: Fig.
Revised: Fig.

Original: 18 plots the time series (i.e., trace) of the parameters in the Markov chains to assess the behavior of our MCMC sampling.
Revised: 19 plots the time series (i.e., trace) of the parameters in the Markov chains to assess the behavior of our MCMC sampling.

Original: It shows the positions of each walker (10) as a function of the number of steps in the chain.
Revised: It shows the positions of each walker (10) as a function of the number of steps in the chain.

Original: The walkers start in small distributions around the maximum likelihood values and then quickly wander and start exploring the full posterior distribution.
Revised: The walkers start in small distributions around the maximum likelihood values and then quickly wander and start exploring the full posterior distribution.

Original: In fact, after aboutsteps, the MCMC appear to have converged to stationarity (i.e., self-replicating).
Revised: In fact, after aboutsteps, the MCMC appear to have converged to stationarity (i.e., self-replicating).

Original: We also examine the mean acceptance fraction of the ensemble, which is 0.54 by average, and it indicates that the
Revised: We also examine the mean acceptance fraction of the ensemble, which is 0.54 by average, and it indicates that the

Original: MCMC sampler has generated a sufﬁcient number of parameter samples that have also gained good information about the underlying distribution.
Revised: MCMC sampler has generated a sufﬁcient number of parameter samples that have also gained good information about the underlying distribution.

Original: C C OMPARISON WITH C HAMELEON

Original: Chameleon (Ahn et al., 2020) is a recent work that uses reinforcement learning and adaptive sampling to efﬁciently explore the code transformation space of a tensor operator.

Original: In this part, we compare DynaTune with the open-sourced version of Chameleon 1 .

Original: We follow the provided instructions and evaluate Chameleon on the same set of models: ResNet-18 and SqueezeNet on CPU, VGG and Transformer on GPU.

Original: Figure 19c and Figure 19d show that although Chameleon is faster than the baseline AutoTVM on VGG and Transformer on GPU, it is slower than DynaTune to converge to the lowest latency.

Original: This is because although the optimization speed of each operator has been improved by Chameleon, it still sequentially optimizes one operator at a time.

Original: Therefore, the overall convergence speed is still bounded by the least optimized tensor operators.

Original: In contrast, DynaTune focuses on improving the convergence speed of multi-tensor-operators and dynamically allocates time to improve the overall convergence speed.

Original: The original Chameleon paper seems to only evaluate its approach on GPU, but model inference is not uncommon on CPU.

Original: When running on CPU, a bit surprisingly, we observe that Chameleon is slower than the baseline AutoTVM on ResNet-18 (Figure 19a) and SqueezeNet (Figure 19b).

Original: We analyze the performance and ﬁnd that the RL optimizer in Chameleon adds a non-trivial amount of overhead than the default optimizer (simulated annealing + XGBoost cost model) in AutoTVM on

Original: CPU.

Original: As a result, although Chameleon reduces the hardware measurement cost and the number of iterations to ﬁnd the best conﬁguration, its optimization time on CPU is longer than the baseline

Original: AutoTVM because of this extra overhead and is much slower than DynaTune on CPU.

Original: Overall, DynaTune is 1.4–4.7 times faster than Chameleon to reach the same latency.

Original: Although

Original: DynaTune is faster than Chameleon, we want to point out that DynaTune can be combined with

Original: Chameleon to achieve better performance, at least on GPU.

Original: [Figure]

Original: [Figure]

Original: C  OMPARISON WITH A NSOR SCHEDULER
Revised: C C OMPARISON WITH A NSOR SCHEDULER

Original: We also compare our approach with a concurrent work  (Zheng et al., 2020).
Revised: In this section, we compare our approach with a concurrent work called Ansor (Zheng et al., 2020).

Original: We use therecently open-sourced implementation 2 for the evaluation. The original Ansor paper does not seem to describe how to set the hyperparameters for its scheduler, so we use the default hyperparameters α = 0 .
Revised: We use the default hyperparameters α = 0 .

Original: 2 , β = 2 , backward window size = 3 in the code.
Revised: 2 , β = 2 , backward window size=3 in the open-sourced implementation 5 for the evaluation.

Original: We use the same evaluation methodology as ours by performing ﬁve independent runs of each conﬁguration with different random seeds and reporting the median together with a 95% conﬁdence interval.
Revised: We use the same evaluation methodology as ours by performing ﬁve independent runs of each conﬁguration with different random seeds and reporting the median together with a 95% conﬁdence interval.

Original: Figure 20a–20d show that both DynaTune and the Ansor scheduler outperform the baseline AutoTVM by a large margin.
Revised: Figure 20a–20d show that both DynaTune and the Ansor scheduler outperform the baseline AutoTVM by a large margin.

Original: This is expected, because both approaches accelerate the convergence of model optimization through dynamic optimization.
Revised: This is expected, because both approaches accelerate the convergence of model optimization through dynamic optimization.

Original: Figure 21a–21d show a more detailed comparison between DynaTune and Ansor.
Revised: Figure 21a–21d show a more detailed comparison between DynaTune and Ansor.

Original: Overall, Ansor seems to reduce the latency faster in the beginning, but DynaTunecan always catch up and is often quicker to reach the lowest latency.
Revised: Overall, Ansor seems to reduce the latency faster in the beginning, but DynaTune always catches up and is often quicker to reach the lowest latency.

Original: For example, DynaTune achieves a faster convergence to reach the lowest latency than Ansor on ResNet-18, VGG, and Transformer.
Revised: For example, DynaTune achieves a faster convergence to reach the lowest latency than Ansor on ResNet-18, VGG, and Transformer.

Original: Although Zheng et al.
Revised: Although Zheng et al.

Original: and our approach share a similar high-level idea of dynamically allocating time resources to different tasks , the exact mechanisms on how to allocate resources are very different.
Revised: and our approach share a similar high-level idea of dynamically allocating time resources to different tasks, the exact mechanisms on how to allocate resources are very different.

Original: The tasks scheduler from Zheng et al.
Revised: The tasks scheduler from Zheng et al.

Original: (2020) decides which task to optimize based on a heuristic score, which is a weighted sum between the latency reduction rate of a task in a recent small time window and an expected latency reduction in the future based on task similarity information.
Revised: (2020) decides which task to optimize based on a heuristic score, which is a weighted sum between the latency reduction rate of a task in a recent small time window and an expected latency reduction in the future based on task similarity information.

Original: The estimation of the score not only requires deﬁning and heuristically adjusting similarity groups but also requires two hyperparameters α and β to control the weight to decide which estimations to trust more.
Revised: The estimation of the score not only requires deﬁning and heuristically adjusting similarity groups but also requires two hyperparameters α and β to control the weight to decide which estimations to trust more.

Original: However, it is not immediately clear how such weights should be set and how to adapt them to different models.
Revised: However, it is not immediately clear how such weights should be set and how to adapt them to different models.

Original: For example, the default hyperparameter settings may cause the scheduling to be overly greedy (i.e., getting stuck at a local optimum), which may explain why
Revised: For example, the default hyperparameter settings may cause the scheduling to be overly greedy (i.e., getting stuck at a local optimum), which may explain why

Original: Ansor converges faster in the beginning but is slower to reach the best latency towards the end.
Revised: Ansor converges faster in the beginning but is slower to reach the best latency towards the end.

Original: In contrast, we use a Bayesian belief model to predict how much expected latency reduction from each task in the next time slot, and all the free parameters in our belief model are taken care of by
Revised: In contrast, we use a Bayesian belief model to predict how much expected latency reduction from each task in the next time slot, and all the free parameters in our belief model are taken care of by

Original: MCMC sampling, which gives our method some plausible advantage in terms of simplicity.
Revised: MCMC sampling.

Original: Furthermore, our approach takes uncertainty quantiﬁcation into account, which presumably helps the search escape from local optimum.
Revised: Furthermore, our approach takes uncertainty quantiﬁcation into account, which presumably helps the search escape from local optimum.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]