Original: L EARNED H ARDWARE /S OFTWARE C O -D ESIGN OF N EURAL A CCELERATORS
Revised: L EARNED H ARDWARE /S OFTWARE C O -D ESIGN OF N EURAL A CCELERATORS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The use of deep learning has grown at an exponential rate, giving rise to numerous specialized hardware and software systems for deep learning.
Revised: The use of deep learning has grown at an exponential rate, giving rise to numerous specialized hardware and software systems for deep learning.

Original: Because the design space of deep learning software stacks and hardware accelerators is diverse and vast, prior work considers software optimizations separately from hardware architectures, effectively reducing the search space.
Revised: Because the design space of deep learning software stacks and hardware accelerators is diverse and vast, prior work considers software optimizations separately from hardware architectures, effectively reducing the search space.

Original: Unfortunately, this bifurcated approach means that many proﬁtable design points are never explored.
Revised: Unfortunately, this bifurcated approach means that many proﬁtable design points are never explored.

Original: This paper instead casts the problem as hardware/software co-design, with the goal of automatically identifying desirable points in the joint design space.
Revised: This paper instead casts the problem as hardware/software co-design, with the goal of automatically identifying desirable points in the joint design space.

Original: The key to our solution is a new constrained Bayesian optimization framework that avoids invalid solutions by exploiting the highly constrained features of this design space, which are semicontinuous/semi-discrete.
Revised: The key to our solution is a new constrained Bayesian optimization framework that avoids invalid solutions by exploiting the highly constrained features of this design space, which are semicontinuous/semi-discrete.

Original: We evaluate our optimization framework by applying it to a variety of neural models, improving the energy-delay product by 18% (ResNet) and 40% (DQN) over hand-tuned state-of-the-art systems, as well as demonstrating strong results on other neural network architectures, such as MLPs and Transformers.
Revised: We evaluate our optimization framework by applying it to a variety of neural models, improving the energy-delay product by 18% (ResNet) and 40% (DQN) over hand-tuned state-of-the-art systems, as well as demonstrating strong results on other neural network architectures, such as MLPs and Transformers.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The compute requirements of deep learning are growing at a double exponential rate (Hernandez & Brown, 2020), with more powerful models requiring exponentially more compute to train.
Revised: The compute requirements of deep learning are growing at a double exponential rate (Hernandez & Brown, 2020), with more powerful models requiring exponentially more compute to train.

Original: This growth has been enabled by large systems of hardware accelerators, like GPUs and TPUs (NVIDIA, 2017; Jouppi et al., 2017).
Revised: This growth has been enabled by large systems of hardware accelerators, like GPUs and TPUs (NVIDIA, 2017; Jouppi et al., 2017).

Original: However, the continued scaling of these systems is limited by issues of power density, cooling, and memory, so we need to improve computational efﬁciency.
Revised: However, the continued scaling of these systems is limited by issues of power density, cooling, and memory, so we need to improve computational efﬁciency.

Original: Efﬁciency improvements can be sought at each layer of the deep learning stack, from better learning algorithms (Kingma & Ba, 2014), to improved neural network architectures (Tan & Le, 2019), to deep learning compilers (Chen et al., 2018), to specialized DNN accelerators that increase hardware efﬁciency (Chen et al., 2014a; 2016).
Revised: Efﬁciency improvements can be sought at each layer of the deep learning stack, from better learning algorithms (Kingma & Ba, 2014), to improved neural network architectures (Tan & Le, 2019), to deep learning compilers (Chen et al., 2018), to specialized DNN accelerators that increase hardware efﬁciency (Chen et al., 2014a; 2016).

Original: In this paper, we focus on the low-level software and hardware portions of this stack, with the goal of automatically optimizing the energy × delay product of executing a particular model on a hardware accelerator.
Revised: In this paper, we focus on the low-level software and hardware portions of this stack, with the goal of automatically optimizing the energy × delay product of executing a particular model on a hardware accelerator.

Original: We consider two components from the deep learning stack: the hardware accelerator and the software compiler that maps a model onto that hardware.
Revised: We consider two components from the deep learning stack: the hardware accelerator and the software compiler that maps a model onto that hardware.

Original: This area is commonly referred to as hardware/software co-design, and since it requires human expertise from multiple disciplines (software engineers, compiler writers, hardware architects), it is typically driven by manual heuristics or heuristic-based search (Yang et al., 2020).
Revised: This area is commonly referred to as hardware/software co-design, and since it requires human expertise from multiple disciplines (software engineers, compiler writers, hardware architects), it is typically driven by manual heuristics or heuristic-based search (Yang et al., 2020b).

Original: We propose a different approach, recognizing that for a given DNN model, this hardware/software co-design can be framed as a joint search of the space of all of the valid mappings and hardware architectures that can correctly execute the model.
Revised: We propose a different approach, recognizing that for a given DNN model, this hardware/software co-design can be framed as a joint search of the space of all of the valid mappings and hardware architectures that can correctly execute the model.

Original: We formally parameterize this space based on prior work (Parashar et al., 2019), and we ﬁnd that standard optimization techniques, including off-the-shelf Bayesian optimization, perform poorly because the design space is semi-discrete and the vast majority of the points in the space are infeasible.
Revised: We formally parameterize this space based on prior work (Parashar et al., 2019), and we ﬁnd that standard optimization techniques, including off-the-shelf Bayesian optimization, perform poorly because the design space is semi-discrete and the vast majority of the points in the space are infeasible.

Original: Prior work (Nardi et al., 2019) makes a similar observation, noting that (1) complex constraints such as hardware area and energy budget limit the feasible parameter values (i.e. small feasibility set), (2) some constraints are unknown until after a sample point has been evaluated (i.e. unknown feasibility).
Revised: Prior work (Nardi et al., 2019) makes a similar observation, noting that (1) complex constraints such as hardware area and energy budget limit the feasible parameter values (i.e. small feasibility set), (2) some constraints are unknown until after a sample point has been evaluated (i.e. unknown feasibility).

Original: Our solution casts the search as a bilevel optimization problem, as shown in Figure 1.
Revised: Our solution casts the search as a bilevel optimization problem, as shown in Figure 1.

Original: The outer loop optimizes over hardware architectures, while the inner loop optimizes over software mappings for a given architecture.
Revised: The outer loop optimizes over hardware architectures, while the inner loop optimizes over software mappings for a given architecture.

Original: Both of these are heavily constrained black-box global optimization problems
Revised: Both of these are heavily constrained black-box global optimization problems

Original: [Figure]
Revised: [Figure]

Original: that require expensive simulations to obtain performance estimates.
Revised: that require expensive simulations to obtain performance estimates.

Original: We therefore propose a nested, constrained Bayesian optimization (BO) formulation that uses Bayesian models of hardware and software performance to guide the search towards promising regions of the design space.
Revised: We therefore propose a nested, constrained Bayesian optimization (BO) formulation that uses Bayesian models of hardware and software performance to guide the search towards promising regions of the design space.

Original: Our approach is extensible to a variety of different neural network architectures, and we make the code publicly available.
Revised: Our approach is extensible to a variety of different neural network architectures, and we make the code publicly available.

Original: We ﬁnd that when compared against the state-of-the-art manually-designed hardware accelerators that use heuristic software mappings, our BO-based approach provides signiﬁcant improvements in the speed and energy efﬁciency of the resulting system, improving the energy-delay product (EDP) by 16.0% to 40.2% on a series of neural networks.
Revised: We ﬁnd that when compared against the state-of-the-art manually-designed hardware accelerators that use heuristic software mappings, our BO-based approach provides signiﬁcant improvements in the speed and energy efﬁciency of the resulting system, improving the energy-delay product (EDP) by 16.0% to 40.2% on a series of neural networks.

Original: The key to our solution is our robust BO software optimizer, whose consistent efﬁciency allows our approach to scale to this huge search space.
Revised: The key to our solution is our robust BO software optimizer, whose consistent efﬁciency allows our approach to scale to this huge search space.

Original: This paper makes the following contributions:
Revised: This paper makes the following contributions:

Original: • We present the ﬁrst system that automatically co-optimizes both the hardware  and software portions of DNN accelerator design using a principled and systematic search algorithm.
Revised: • We present the ﬁrst system that automatically co-optimizes both the hardware architecture and software mapping phases of DNN accelerator design using a principled and systematic search algorithm.

Original: • We present a constrained formulation of hardware and software design for BO, a challenging problem given the high ratio (90%) of invalid hardware and software designs.
Revised: • We present a constrained formulation of hardware and software design for BO, a challenging problem given the high ratio (90%) of invalid hardware and software designs.

Original: • We present a nested hardware/software formulation of BO that is extensible to other hardware accelerator designs.
Revised: • We present a nested hardware/software formulation of BO that is extensible to other hardware accelerator designs.

Original: • We provide model-speciﬁc hardware and state-of-the-art results on multiple models.
Revised: • We provide model-speciﬁc hardware and state-of-the-art results on multiple models.

Original:  A F ORMAL R EPRESENTATION OF S OFTWARE AND H ARDWARE
Revised:  A F ORMAL R EPRESENTATION OF S OFTWARE AND H ARDWARE

Original: Hardware/software co
Revised: Hardware/software co

Original: -design typically performed manually, but we believe that this vast design space is best navigated by an intelligent search process.
Revised: -design typically performed manually, but we believe that this vast design space is best navigated by an intelligent search process.

Original: To facilitate this automation, this section formally deﬁnes the hardware and software design spaces.
Revised: To facilitate this automation, this section formally deﬁnes the hardware and software design spaces.

Original: 2. P ARAMETERIZING THE D ESIGN S PACE
Revised: 2. P ARAMETERIZING THE D ESIGN S PACE

Original: Software design points can be parameterized by the loop ordering, loop tiling, and computational parallelism of the seven-level loop nest used to compute a convolutional layer (see appendix), as has been noted by recent work (Parashar et al., 2019; Yang et al., 2020).
Revised: Software design points can be parameterized by the loop ordering, loop tiling, and computational parallelism of the seven-level loop nest used to compute a convolutional layer (see appendix), as has been noted by recent work (Parashar et al., 2019; Yang et al., 2020b).

Original: These software parameters are subject to hardware constraints, such as the quantity and layout of processing elements (PEs) and the size of storage elements.
Revised: These software parameters are subject to hardware constraints, such as the quantity and layout of processing elements (PEs) and the size of storage elements.

Original: Hardware parameters can be broken down into a two broad categories:
Revised: Hardware parameters can be broken down into a two broad categories:

Original: Resource conﬁgurations represent the physical aspects of hardware, such as buffer sizes, tile sizes, and the cluster size of global buffers, as well as the layouts of the PE array and of the global buffer.
Revised: Resource conﬁgurations represent the physical aspects of hardware, such as buffer sizes, tile sizes, and the cluster size of global buffers, as well as the layouts of the PE array and of the global buffer.

Original: Dataﬂow conﬁgurations represent the usage of the PE array that are implemented in hardware, such as the blocking factors and degree of parallelism at the PE level, which also determines the communication patterns among PEs.
Revised: Dataﬂow conﬁgurations represent the usage of the PE array that are implemented in hardware, such as the blocking factors and degree of parallelism at the PE level, which also determines the communication patterns among PEs.

Original: Figure 2 shows two possible design points for a 1D convolution.
Revised: Figure 2 shows two possible design points for a 1D convolution.

Original: Both design points tile and parallelize the channel (C) dimension.
Revised: Both design points tile and parallelize the channel (C) dimension.

Original: To the right of each component in the architecture is a set of loops that speciﬁes the control logic for the component, which can be broken down into temporal streaming ( for loops) and spatial distribution ( parallel_for loops).
Revised: To the right of each component in the architecture is a set of loops that speciﬁes the control logic for the component, which can be broken down into temporal streaming ( for loops) and spatial distribution ( parallel_for loops).

Original: For example, in the architecture on the left, the global buffer distributes across the PEs 1 weight from 4 separate channels ( c2 ), and the
Revised: For example, in the architecture on the left, the global buffer distributes across the PEs 1 weight from 4 separate channels ( c2 ), and the

Original: PEs perform all operations that the weight participates in.
Revised: PEs perform all operations that the weight participates in.

Original: In this design, all data reuse is captured within the PE, so the global buffer need not store anything.
Revised: In this design, all data reuse is captured within the PE, so the global buffer need not store anything.

Original: By contrast, the architecture on the right distributes a single output element across the PEs to compute partial sums, which are stored in the global buffer across iterations.
Revised: By contrast, the architecture on the right distributes a single output element across the PEs to compute partial sums, which are stored in the global buffer across iterations.

Original: Both these design points consist of the same architectural components, but the dataﬂows vary, imposing different constraints on the software.
Revised: Both these design points consist of the same architectural components, but the dataﬂows vary, imposing different constraints on the software.

Original: The appendix shows the details of the parameterization of a more practical 2D convolution
Revised: The appendix shows the details of the parameterization of a more practical 2D convolution

Original: [Figure]
Revised: [Figure]

Original: 2. C ONSTRAINTS IN THE D ESIGN S PACE
Revised: 2. C ONSTRAINTS IN THE D ESIGN S PACE

Original: There are several reasons why the vast space of hardware and software parameters is ﬁlled with impractical or invalid design points.
Revised: There are several reasons why the vast space of hardware and software parameters is ﬁlled with impractical or invalid design points.

Original: First, hardware designs are fundamentally constrained by area (the total amount of compute and storage resources) and factors such as available memory bandwidth.
Revised: First, hardware designs are fundamentally constrained by area (the total amount of compute and storage resources) and factors such as available memory bandwidth.

Original: Second, the design cost and latency of additional area grow super-linearly (Shao et al., 2019), which leads to many impractical design points.
Revised: Second, the design cost and latency of additional area grow super-linearly (Shao et al., 2019), which leads to many impractical design points.

Original: Software constraints are generally governed by feasibility instead of practicality and predominantly depend on the hardware conﬁguration and the speciﬁc neural network workload.
Revised: Software constraints are generally governed by feasibility instead of practicality and predominantly depend on the hardware conﬁguration and the speciﬁc neural network workload.

Original: For a speciﬁc hardware accelerator, there is a limited number of available resources, so the software optimization problem can be viewed as a search for the most efﬁcient use of hardware PEs and buffers.
Revised: For a speciﬁc hardware accelerator, there is a limited number of available resources, so the software optimization problem can be viewed as a search for the most efﬁcient use of hardware PEs and buffers.

Original: For example, the loop blocking optimization factors a neural network across multiple hardware storage buffers—and the feasible factorizations are constrained by the size of the hardware buffers.
Revised: For example, the loop blocking optimization factors a neural network across multiple hardware storage buffers—and the feasible factorizations are constrained by the size of the hardware buffers.

Original:  B AYESIAN O PTIMIZATION 3. O VERVIEW
Revised:  B AYESIAN O PTIMIZATION 3. O VERVIEW

Original: Bayesian optimization (Jones et al., 1998; Brochu et al., 2010; Shahriari et al., 2015) is an effective approach for the optimization of expensive, possibly noisy black-box functions.
Revised: Bayesian optimization (Jones et al., 1998; Brochu et al., 2010; Shahriari et al., 2015) is an effective approach for the optimization of expensive, possibly noisy black-box functions.

Original: BO has been used to optimize hyperparameters (Snoek et al., 2012), conﬁgure algorithms (Hutter et al., 2011), optimize
Revised: BO has been used to optimize hyperparameters (Snoek et al., 2012), conﬁgure algorithms (Hutter et al., 2011), optimize

Original: A/B experiments (Letham et al., 2019), and more.
Revised: A/B experiments (Letham et al., 2019), and more.

Original: For our problem, we have a parameterized representation and access to a simulator.
Revised: For our problem, we have a parameterized representation and access to a simulator.

Original: Since one of our main concerns is sample efﬁciency,
Revised: Since one of our main concerns is sample efﬁciency,

Original: Bayesian optimization is particularly suitable.
Revised: Bayesian optimization is particularly suitable.

Original: The actual cost of evaluation depends on the experimental infrastructure, but in general, it is much more expensive to evaluate a hardware design choice than to evaluate software optimizations, because hardware design can take hours (to produce a hardware simulator or an FPGA) to days or even months (to produce an ASIC).
Revised: The actual cost of evaluation depends on the experimental infrastructure, but in general, it is much more expensive to evaluate a hardware design choice than to evaluate software optimizations, because hardware design can take hours (to produce a hardware simulator or an FPGA) to days or even months (to produce an ASIC).

Original: Bayesian optimization has two major components: (1) a surrogate model provides a Bayesian posterior probability distribution that predicts potential values of the objective function.
Revised: Bayesian optimization has two major components: (1) a surrogate model provides a Bayesian posterior probability distribution that predicts potential values of the objective function.

Original: (2) an acquisition function uses the model to identify the next point to evaluate.
Revised: (2) an acquisition function uses the model to identify the next point to evaluate.

Original: 3. G AUSSIAN PROCESSES
Revised: 3. G AUSSIAN PROCESSES

Original: A common surrogate model is a Gaussian process (GP) (Rasmussen & Williams, 2006) due to its simplicity and ﬂexibility.
Revised: A common surrogate model is a Gaussian process (GP) (Rasmussen & Williams, 2006) due to its simplicity and ﬂexibility.

Original: A GP is prior distribution over the space of functions comprised of a mean function m ( x ) and a covariance, or kernel function k
Revised: A GP is prior distribution over the space of functions comprised of a mean function m ( x ) and a covariance, or kernel function k

Original: ( x , x (cid:48) ) .
Revised: ( x , x (cid:48) ) .

Original: Suppose we are given a dataset of N input/output pairs over a bounded domain Ω with D input dimensions and scalar outputs.
Revised: Suppose we are given a dataset of N input/output pairs over a bounded domain Ω with D input dimensions and scalar outputs.

Original: For brevity, we write this as ( X, y ) , where X ∈ Ω N × D and y ∈ R N .
Revised: For brevity, we write this as ( X, y ) , where X ∈ Ω N × D and y ∈ R N .

Original: The posterior predictive distribution over function values f for a new input x is given by P ( f | x , X, y ) =
Revised: The posterior predictive distribution over function values f for a new input x is given by P ( f | x , X, y ) =

Original: N ( µ ( x ) , σ 2 ( x [Equation] ) , where
Revised: N ( µ ( x ) , σ 2 ( x )) , where

Original: [Equation]
Revised: [Equation]

Original: Where K XX is a matrix formed by evaluating the kernel on X , K x X is the vector of kernel evaluations between x and X , and m X is the vector of mean function evaluations on the input dataset.
Revised: Where K XX is a matrix formed by evaluating the kernel on X , K x X is the vector of kernel evaluations between x and X , and m X is the vector of mean function evaluations on the input dataset.

Original: A common choice for the kernel is squared exponential.
Revised: A common choice for the kernel is squared exponential.

Original: Given two input vectors x i and x j , this is deﬁned as k ( x i , x j ) =
Revised: Given two input vectors x i and x j , this is deﬁned as k ( x i , x j ) =

Original: α 2 exp (cid:16) − (cid:107) x i − x j (cid:107) 2 (cid:96)(cid:17) .
Revised: α 2 exp (cid:16) − (cid:107) x i − x j (cid:107) 2 (cid:96)(cid:17) .

Original: α and (cid:96) are kernel hyperparameters.
Revised: α and (cid:96) are kernel hyperparameters.

Original: Another kernel that we ﬁnd particularly useful is a linear kernel on top of explicit features.
Revised: Another kernel that we ﬁnd particularly useful is a linear kernel on top of explicit features.

Original: Given a feature mapping φ ( x ) : R D → R K , the linear kernel can be written as k ( x i , x j ) =
Revised: Given a feature mapping φ ( x ) : R D → R K , the linear kernel can be written as k ( x i , x j ) =

Original: φ ( x i ) (cid:62) φ ( x j ) .
Revised: φ ( x i ) (cid:62) φ ( x j ) .

Original: When we have strong prior information about the relevant feature interactions that govern the black-box function, this kernel allows us to encode these interactions directly and produces a more sample-efﬁcient posterior estimate.
Revised: When we have strong prior information about the relevant feature interactions that govern the black-box function, this kernel allows us to encode these interactions directly and produces a more sample-efﬁcient posterior estimate.

Original: In cases where the observations from the black-box function are noisy, we can add a noise kernel
Revised: In cases where the observations from the black-box function are noisy, we can add a noise kernel

Original: K noise =
Revised: K noise =

Original: τ 2 I to K XX , where τ 2 is a hyperparameter.
Revised: τ 2 I to K XX , where τ 2 is a hyperparameter.

Original: This implies a Gaussian observation likelihood.
Revised: This implies a Gaussian observation likelihood.

Original: Following common practice, we use the constant mean m ( x ) =
Revised: Following common practice, we use the constant mean m ( x ) =

Original: c ∀ x .
Revised: c ∀ x .

Original: All kernel and mean hyperparameters are learned by maximizing the marginal likelihood of the GP on the current dataset.
Revised: All kernel and mean hyperparameters are learned by maximizing the marginal likelihood of the GP on the current dataset.

Original: 3. A CQUISITION FUNCTIONS
Revised: 3. A CQUISITION FUNCTIONS

Original: A critical component in the BO framework is the choice of acquisition function a ( · ) that assigns each design point a value that represents the utility of testing this point.
Revised: A critical component in the BO framework is the choice of acquisition function a ( · ) that assigns each design point a value that represents the utility of testing this point.

Original: Two commonly used acquisition functions are expected improvement (EI) and lower conﬁdence bound (LCB).
Revised: Two commonly used acquisition functions are expected improvement (EI) and lower conﬁdence bound (LCB).

Original: EI computes the amount we expect to improve upon the current best observed objective value y ∗ ≡ max { y i } Ni =1 by evaluating a design point x .
Revised: EI computes the amount we expect to improve upon the current best observed objective value y ∗ ≡ max { y i } Ni =1 by evaluating a design point x .

Original: Formally, it can be written as
Revised: Formally, it can be written as

Original: [Equation]
Revised: [Equation]

Original: where f is the latent function from the surrogate model, and y ∗ is the best value observed.
Revised: where f is the latent function from the surrogate model, and y ∗ is the best value observed.

Original: LCB (Srinivas et al., 2009) provides an explicit tradeoff between the predictive mean and variance and is deﬁned as
Revised: LCB (Srinivas et al., 2009) provides an explicit tradeoff between the predictive mean and variance and is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Where λ represents a tradeoff parameter.
Revised: Where λ represents a tradeoff parameter.

Original: A small λ promotes greater exploitation, and a large λ promotes greater exploration.
Revised: A small λ promotes greater exploitation, and a large λ promotes greater exploration.

Original: We found λ = 1 to work well in our experiments.
Revised: We found λ = 1 to work well in our experiments.

Original: Beyond these, there are many other possible acquisition functions that could be used in future exploration (Thompson, 1933; Hennig & Schuler, 2012; Hernández-Lobato et al., 2014; Frazier, 2009).
Revised: Beyond these, there are many other possible acquisition functions that could be used in future exploration (Thompson, 1933; Hennig & Schuler, 2012; Hernández-Lobato et al., 2014; Frazier, 2009).

Original: 3. C ONSTRAINTS
Revised: 3. C ONSTRAINTS

Original: In our problem, the vast majority of the design space will produce invalid solutions.
Revised: In our problem, the vast majority of the design space will produce invalid solutions.

Original: When the constraints are a known function of the input features, we can directly account for them as input constraints.
Revised: When the constraints are a known function of the input features, we can directly account for them as input constraints.

Original: Otherwise, we must run the simulation and treat invalid points using an output constraint.
Revised: Otherwise, we must run the simulation and treat invalid points using an output constraint.

Original: Here, we will describe these constraint types, and how they are incorporated into BO.
Revised: Here, we will describe these constraint types, and how they are incorporated into BO.

Original: Input constraints are explicit constraints that are used when optimizing the acquisition function.
Revised: Input constraints are explicit constraints that are used when optimizing the acquisition function.

Original: They directly prevent the search from suggesting points that will violate the constraints.
Revised: They directly prevent the search from suggesting points that will violate the constraints.

Original: As some constraints are non-linear, this optimization is itself very challenging, as it is a global optimization problem with non-convex constraints.
Revised: As some constraints are non-linear, this optimization is itself very challenging, as it is a global optimization problem with non-convex constraints.

Original: In the unconstrained case, maximizing the acquisition function often takes a hybrid approach: generating a random initial set of points and reﬁning them by gradient ascent.
Revised: In the unconstrained case, maximizing the acquisition function often takes a hybrid approach: generating a random initial set of points and reﬁning them by gradient ascent.

Original: Maintaining feasibility with non-convex constraints is far more challenging, however.
Revised: Maintaining feasibility with non-convex constraints is far more challenging, however.

Original: We therefore optimize the acquisition function in a simple way by performing rejection sampling on the design space: we randomly sample parameters until we obtain 150 feasible points, and then choose the one the maximizes the acquisition function.
Revised: We therefore optimize the acquisition function in a simple way by performing rejection sampling on the design space: we randomly sample parameters until we obtain 150 feasible points, and then choose the one the maximizes the acquisition function.

Original: On average the sampling takes 22K random samples to get a pool of 150 feasible points.
Revised: On average the sampling takes 22K random samples to get a pool of 150 feasible points.

Original: We have found that practically this is a simple yet effective strategy for our problems, we leave more involved optimization schemes for future work.
Revised: We have found that practically this is a simple yet effective strategy for our problems, we leave more involved optimization schemes for future work.

Original: Output constraints are used when we do not know the form of the constraint a-priori and must run the simulator to test feasibility.
Revised: Output constraints are used when we do not know the form of the constraint a-priori and must run the simulator to test feasibility.

Original: This is also referred to as an “unknown” constraint, and BO has been adapted to incorporate a constraint model in addition to the regression model (Gelbart et al., 2014).
Revised: This is also referred to as an “unknown” constraint, and BO has been adapted to incorporate a constraint model in addition to the regression model (Gelbart et al., 2014).

Original: These simultaneously learn about the constraint boundaries while modeling the objective.
Revised: These simultaneously learn about the constraint boundaries while modeling the objective.

Original: Let C ( x ) denote the event that x satisﬁes constraint C .
Revised: Let C ( x ) denote the event that x satisﬁes constraint C .

Original: Constrained BO uses a Bayesian classiﬁer to model P ( C ( x )) .
Revised: Constrained BO uses a Bayesian classiﬁer to model P ( C ( x )) .

Original: It is relatively straightforward to adapt a GP regressor to classiﬁcation (Rasmussen & Williams, 2006).
Revised: It is relatively straightforward to adapt a GP regressor to classiﬁcation (Rasmussen & Williams, 2006).

Original: Under a Bayesian classiﬁer, the acquisition function a ( x ) is modiﬁed to account for the probability that the constraint is satisﬁed, with 0 utility if it is not satisﬁed.
Revised: Under a Bayesian classiﬁer, the acquisition function a ( x ) is modiﬁed to account for the probability that the constraint is satisﬁed, with 0 utility if it is not satisﬁed.

Original: [Equation]
Revised: [Equation]

Original: Where I[ C ( x )] is the indicator function that evaluates to 1 if the constraint is satisﬁed and 0 otherwise.
Revised: Where I[ C ( x )] is the indicator function that evaluates to 1 if the constraint is satisﬁed and 0 otherwise.

Original: We therefore maintain two models: one regression model to capture the objective and one classiﬁer to model the constraint in order to avoid evaluations in infeasible regions.
Revised: We therefore maintain two models: one regression model to capture the objective and one classiﬁer to model the constraint in order to avoid evaluations in infeasible regions.

Original:  B AYESIAN O PTIMIZATION FOR H ARDWARE /S OFTWARE C O - DESIGN 4. O VERVIEW OF N ESTED H ARDWARE /S OFTWARE O PTIMIZATION
Revised:  B AYESIAN O PTIMIZATION FOR H ARDWARE /S OFTWARE C O - DESIGN 4. O VERVIEW OF N ESTED H ARDWARE /S OFTWARE O PTIMIZATION

Original: Provided the constraints discussed in Section 2 and the BO formulation from Section 3, we propose a nested approach for co-optimizing hardware/software parameters.
Revised: Provided the constraints discussed in Section 2 and the BO formulation from Section 3, we propose a nested approach for co-optimizing hardware/software parameters.

Original: The overall approach is outlined in
Revised: The overall approach is outlined in

Original: Figure 1.
Revised: Figure 1.

Original: The goal is to ﬁnd the optimal hardware parameters for a neural model and the optimal set of software parameters for each layer in the neural model.
Revised: The goal is to ﬁnd the optimal hardware parameters for a neural model and the optimal set of software parameters for each layer in the neural model.

Original: Since software constraints depend on a feasible hardware design, we ﬁrst propose the hardware parameters, then for that hardware co-optimize the software mapping.
Revised: Since software constraints depend on a feasible hardware design, we ﬁrst propose the hardware parameters, then for that hardware co-optimize the software mapping.

Original: Speciﬁcally, let x h and x s denote the set of hardware and software parameters in the parameter space to be optimized.
Revised: Speciﬁcally, let x h and x s denote the set of hardware and software parameters in the parameter space to be optimized.

Original: In the nested search process, we ﬁrst use the hardware optimizer to generate a design of hardware.
Revised: In the nested search process, we ﬁrst use the hardware optimizer to generate a design of hardware.

Original: In particular, we perform the hardware search in the space of possible hardware S h to optimize all hardware parameters, where the objective is to minimize f ( x h | NN ) which we deﬁne as the energy-delay product (EDP) of running the neural network (NN) model on the given hardware, assuming the optimal software mapping for each individual layer.
Revised: In particular, we perform the hardware search in the space of possible hardware S h to optimize all hardware parameters, where the objective is to minimize f ( x h | NN ) which we deﬁne as the energy-delay product (EDP) of running the neural network (NN) model on the given hardware, assuming the optimal software mapping for each individual layer.

Original: This step produces a hardware speciﬁcation and can be formalized as argmin h ∈S h f
Revised: This step produces a hardware speciﬁcation and can be formalized as argmin h ∈S h f

Original: ( x h | NN ) .
Revised: ( x h | NN ) .

Original: For the chosen hardware design, our framework performs the software search for each individual neural layer in its constrained software mapping space S s | h, NN j to optimize the mapping parameters, where NN j denotes the j th layer in the neural network model, and the objective becomes f ( x s | x h , NN j ) , which is deﬁned as the EDP of running the layer j on the ﬁxed hardware.
Revised: For the chosen hardware design, our framework performs the software search for each individual neural layer in its constrained software mapping space S s | h, NN j to optimize the mapping parameters, where NN j denotes the j th layer in the neural network model, and the objective becomes f ( x s | x h , NN j ) , which is deﬁned as the EDP of running the layer j on the ﬁxed hardware.

Original: This step produces a design point that represents the best set of software mappings for each layers on the given hardware structure, and can be formalized as argmin s ∈S s |
Revised: This step produces a design point that represents the best set of software mappings for each layers on the given hardware structure, and can be formalized as argmin s ∈S s |

Original: h f ( x s | x h ) .
Revised: h f ( x s | x h ) .

Original: The layerwise EDPs are then summed up as the EDP of the neural model, which is fed back to the hardware optimizer to generate the next hardware setting.
Revised: The layerwise EDPs are then summed up as the EDP of the neural model, which is fed back to the hardware optimizer to generate the next hardware setting.

Original: The iterative search between hardware and software will repeat for a user-deﬁned number of trials.
Revised: The iterative search between hardware and software will repeat for a user-deﬁned number of trials.

Original: In this work, we set 50 for hardware search and 250 for software search.
Revised: In this work, we set 50 for hardware search and 250 for software search.

Original: The combination of hardware and software that achieves the best EDP during the optimization process becomes the ﬁnal model-speciﬁc hardware structure and layer-speciﬁc software mappings.
Revised: The combination of hardware and software that achieves the best EDP during the optimization process becomes the ﬁnal model-speciﬁc hardware structure and layer-speciﬁc software mappings.

Original: A random sample is used in the ﬁrst iteration of both the hardware and software search.
Revised: A random sample is used in the ﬁrst iteration of both the hardware and software search.

Original: In our Bayesian optimization (BO) framework, we use separate BO models to search in the hardware and software space.
Revised: In our Bayesian optimization (BO) framework, we use separate BO models to search in the hardware and software space.

Original: We now describe their design considerations, particularly the choice of kernel and feature transformation.
Revised: We now describe their design considerations, particularly the choice of kernel and feature transformation.

Original: 4. BO FOR O PTIMIZING H ARDWARE A RCHITECTURES
Revised: 4. BO FOR O PTIMIZING H ARDWARE A RCHITECTURES

Original: Kernel design.
Revised: Kernel design.

Original: The main design choice for BO is the GP kernel to use.
Revised: The main design choice for BO is the GP kernel to use.

Original: For the hardware search, we choose a linear kernel on top of feature transformations that represent the relationship between the different parameters.
Revised: For the hardware search, we choose a linear kernel on top of feature transformations that represent the relationship between the different parameters.

Original: This feature transformation allows us to explicitly encode domain knowledge.
Revised: This feature transformation allows us to explicitly encode domain knowledge.

Original: For example, by comparing the layout parameters of the 2D PE array and global buffer we can obtain the ratio between these adjacent storage layers, which correlates to the maximal degree of parallel buffer accesses in each dimension.
Revised: For example, by comparing the layout parameters of the 2D PE array and global buffer we can obtain the ratio between these adjacent storage layers, which correlates to the maximal degree of parallel buffer accesses in each dimension.

Original: The details of the features are given Figure 13 in the appendix.
Revised: The details of the features are given Figure 13 in the appendix.

Original: We also add a noise kernel to deal with noise in the hardware evaluation.
Revised: We also add a noise kernel to deal with noise in the hardware evaluation.

Original: This is because the software optimizer is not guaranteed to ﬁnd the best software mapping for each layer.
Revised: This is because the software optimizer is not guaranteed to ﬁnd the best software mapping for each layer.

Original: There is some randomness in the software search, and therefore independent runs of software optimization for a ﬁxed hardware design may yield different results.
Revised: There is some randomness in the software search, and therefore independent runs of software optimization for a ﬁxed hardware design may yield different results.

Original: Constraints.
Revised: Constraints.

Original: There are both known and unknown constraints in the hardware search.
Revised: There are both known and unknown constraints in the hardware search.

Original: The known constraints, such as the compute and storage budget, are treated as input constraints that reject invalid samples.
Revised: The known constraints, such as the compute and storage budget, are treated as input constraints that reject invalid samples.

Original: The unknown constraints have to do with feasibility (if there exist valid software mappings of neural layers onto the hardware, and if the valid mappings can be sampled during the software optimization).
Revised: The unknown constraints have to do with feasibility (if there exist valid software mappings of neural layers onto the hardware, and if the valid mappings can be sampled during the software optimization).

Original: Following Section 3, these constraints are treated as output constraints and are modeled by a GP with a squared exponential kernel.
Revised: Following Section 3, these constraints are treated as output constraints and are modeled by a GP with a squared exponential kernel.

Original: 4. BO FOR O PTIMIZING S OFTWARE M APPINGS
Revised: 4. BO FOR O PTIMIZING S OFTWARE M APPINGS

Original: Kernel design.
Revised: Kernel design.

Original: Similar to hardware optimization, we use a linear kernel and transform the parameters to features that encode relational information.
Revised: Similar to hardware optimization, we use a linear kernel and transform the parameters to features that encode relational information.

Original: As the hardware is ﬁxed in the search of software mappings, we are able to compute features such as buffer usage, which potentially help make the predictions more accurate.
Revised: As the hardware is ﬁxed in the search of software mappings, we are able to compute features such as buffer usage, which potentially help make the predictions more accurate.

Original: The evaluation of a mapping on a given hardware is deterministic in our infrastructure, thus there is no need for a noise kernel in the GPs.
Revised: The evaluation of a mapping on a given hardware is deterministic in our infrastructure, thus there is no need for a noise kernel in the GPs.

Original: Constraints.
Revised: Constraints.

Original: As both the hardware and neural model are known during software optimization, all constraints are known and are treated as input constraints that automatically reject invalid samples.
Revised: As both the hardware and neural model are known during software optimization, all constraints are known and are treated as input constraints that automatically reject invalid samples.

Original:  E VALUATION 5. M ETHODOLOGY
Revised:  E VALUATION 5. M ETHODOLOGY

Original: Infrastructure.
Revised: Infrastructure.

Original: We conduct our evaluation on Timeloop (Parashar et al., 2019), which is an open-source infrastructure for evaluating the hardware design and software optimization of DNN accelerators.
Revised: We conduct our evaluation on Timeloop (Parashar et al., 2019), which is an open-source infrastructure for evaluating the hardware design and software optimization of DNN accelerators.

Original: Timeloop represents the key architecture attributes of DNN accelerators that realize a broad space of hardware structure and topology, which generate an accurate projection of performance and energy efﬁciency for DNN workloads.
Revised: Timeloop represents the key architecture attributes of DNN accelerators that realize a broad space of hardware structure and topology, which generate an accurate projection of performance and energy efﬁciency for DNN workloads.

Original: In the evaluation, Timeloop takes two inputs: 1) the hardware conﬁguration, which consists of the hardware-related parameters, and 2) the software mapping, which consists of the software parameters that describe the mapping.
Revised: In the evaluation, Timeloop takes two inputs: 1) the hardware conﬁguration, which consists of the hardware-related parameters, and 2) the software mapping, which consists of the software parameters that describe the mapping.

Original: As most accelerators are designed for neural network inference, we limit the use case to inference in this work and leave training for future work.
Revised: As most accelerators are designed for neural network inference, we limit the use case to inference in this work and leave training for future work.

Original: Workloads.
Revised: Workloads.

Original: To show that our solution automatically produces efﬁcient hardware for a variety of neural networks, we use our BO framework to optimize critical layers from CNNs (ResNet (He et al., 2016) and DQN (Mnih et al., 2013)), as well as an MLP and Transformer (Vaswani et al., 2017).
Revised: To show that our solution automatically produces efﬁcient hardware for a variety of neural networks, we use our BO framework to optimize critical layers from CNNs (ResNet (He et al., 2016) and DQN (Mnih et al., 2013)), as well as an MLP and Transformer (Vaswani et al., 2017).

Original: Experimental Setup.
Revised: Experimental Setup.

Original: We use Eyeriss (Chen et al., 2016), a state-of-the-art DNN accelerator, as our baseline.
Revised: We use Eyeriss (Chen et al., 2016), a state-of-the-art DNN accelerator, as our baseline.

Original: All workloads are evaluated on the Eyeriss implementation with 168 PEs (Chen et al., 2016) except for the Transformer model, which runs on the larger version of Eyeriss with 256 PEs (Parashar et al., 2019).
Revised: All workloads are evaluated on the Eyeriss implementation with 168 PEs (Chen et al., 2016) except for the Transformer model, which runs on the larger version of Eyeriss with 256 PEs (Parashar et al., 2019).

Original: In the software mapping search, we use Eyeriss’s hardware speciﬁcations and search for the best software mapping for each neural layer.
Revised: In the software mapping search, we use Eyeriss’s hardware speciﬁcations and search for the best software mapping for each neural layer.

Original: In the hardware search, we perform the search under the same compute and storage resource constraints as Eyeriss for each neural model.
Revised: In the hardware search, we perform the search under the same compute and storage resource constraints as Eyeriss for each neural model.

Original: Metrics.
Revised: Metrics.

Original: Hardware accelerators are designed to achieve both speed and energy efﬁciency, so we adopt the widely used energy-delay product (EDP) as the objective.
Revised: Hardware accelerators are designed to achieve both speed and energy efﬁciency, so we adopt the widely used energy-delay product (EDP) as the objective.

Original: Since EDP values can vary by an order of magnitude, we normalize by dividing by the best (minimal) EDP value and take the reciprocal for optimization curves.
Revised: Since EDP values can vary by an order of magnitude, we normalize by dividing by the best (minimal) EDP value and take the reciprocal for optimization curves.

Original: For the hardware/software co-design, we report the EDP improvements of each neural model, which is averaged across all layers (see Figure 11 and 12 in the appendix).
Revised: For the hardware/software co-design, we report the EDP improvements of each neural model, which is averaged across all layers (see Figure 11 and 12 in the appendix).

Original: For software mapping optimizations, we report the layer-wise EDP improvements.
Revised: For software mapping optimizations, we report the layer-wise EDP improvements.

Original: Baselines.
Revised: Baselines.

Original: In hardware search, we compare against constrained random search that repeatedly takes the ﬁrst random sample in the design space that satisﬁes the constraints.
Revised: In hardware search, we compare against constrained random search that repeatedly takes the ﬁrst random sample in the design space that satisﬁes the constraints.

Original: In software search, we useboth constrained random search,  and out-of-the-box BO that optimizes in a continuous parameter space and rounds to the nearest valid parameters.
Revised: In software search, we use constrained random search, TVM with XGBoost and TreeGRU (Chen et al., 2018), and out-of-the-box BO that optimizes in a continuous parameter space and rounds to the nearest valid parameters.

Original: 5. S OFTWARE M APPING O PTIMIZATION
Revised: 5. S OFTWARE M APPING O PTIMIZATION

Original: We show the results of software mapping optimization ﬁrst, as the capability of ﬁnding a good mapping is the base of evaluating a hardware design.
Revised: We show the results of software mapping optimization ﬁrst, as the capability of ﬁnding a good mapping is the base of evaluating a hardware design.

Original: Figure 3 shows the improvements of BO over our constrained random search formulation.
Revised: Figure 3 shows the improvements of BO over our constrained random search formulation.

Original: Our BO formulation outperforms random search as well as a standard BO formulation that optimizes discrete parameters using a relax-and-round approach. 
Revised: Our BO formulation outperforms random search, both variants of TVM as well as a standard BO formulation that optimizes discrete parameters using a relax-and-round approach. [Figure]

Original: 5. H ARDWARE C ONFIGURATION O PTIMIZATION
Revised: 5. H ARDWARE C ONFIGURATION O PTIMIZATION

Original: Figure 4 shows the optimization curves for hardware/software co-design.
Revised: Figure 4 shows the optimization curves for hardware/software co-design.

Original: The comparison of hardware search algorithms shows that BO provides consistently better performance than the constrained[Figure] random search, and the comparison of software search algorithms shows the importance of mapping optimization in the co-design process.
Revised: The comparison of hardware search algorithms shows that BO provides consistently better performance than the constrained random search, and the comparison of software search algorithms shows the importance of mapping optimization in the co-design process.

Original: As shown in Figure 5a, we ﬁnd that the designs searched by
Revised: As shown in Figure 5a, we ﬁnd that the designs searched by

Original: BO achieve signiﬁcantly better EDP on all neural models compared to the state-of-the-art manually designed accelerator (18.3%, 40.2%, 21.8% and 16.0% for ResNet, DQN, MLP and Transformer respectively).[Figure] r [Figure]
Revised: BO achieve signiﬁcantly better EDP on all neural models compared to the state-of-the-art manually designed accelerator (18.3%, 40.2%, 21.8% and 16.0% for ResNet, DQN, MLP and Transformer respectively).

Original: a li z

Original: [Figure]

Original: P ( r

Original: [Figure]

Original: c i p r o c a l )

Original: [Figure]
Revised: [Figure]

Original: for hardware search, and 250 attempts are made to ﬁnd the optimal software mapping for each layer in the model on the hardware speciﬁcation.

Original: Best viewed in color.

Original: 5. A BLATION S TUDIES
Revised: 5. A BLATION S TUDIES

Original: Surrogate models and acquisition functions.
Revised: Surrogate models and acquisition functions.

Original: There exist popular variants for both the surrogate models and acquisition functions.
Revised: There exist popular variants for both the surrogate models and acquisition functions.

Original: In Figure 5b, we compare the surrogate models of Gaussian process (GP) with random forest (RF) and the acquisition functions of expected improvement (EI) and lower conﬁdence bound (LCB).
Revised: In Figure 5b, we compare the surrogate models of Gaussian process (GP) with random forest (RF) and the acquisition functions of expected improvement (EI) and lower conﬁdence bound (LCB).

Original: As shown, in the transformed feature space, GP generally performs better than RF, and LCB generally outperforms EI.
Revised: As shown, in the transformed feature space, GP generally performs better than RF [Figure] and LCB generally outperforms EI.

Original: Exploration vs. Exploitation.
Revised: Exploration vs. Exploitation.

Original: The LCB acquisition function explicitly balances exploration and exploitation with a hyperparameter λ .
Revised: The LCB acquisition function explicitly balances exploration and exploitation with a hyperparameter λ .

Original: To further study the impact of exploration vs. exploitation in the use of LCB, we test LCB with different λ values in Figure 5c.
Revised: To further study the impact of exploration vs. exploitation in the use of LCB [Figure] we test LCB with different λ values in Figure 5c.

Original: We ﬁnd that LCBs with λ values that are greater than 0.5 provide stable performance in the optimization curves, while LCB with λ = 0 .
Revised: We ﬁnd that LCBs with λ values that are greater than 0.5 provide stable performance in the optimization curves [Figure] while LCB with λ = 0 .

Original: 1 suffers from insufﬁcient exploration.
Revised: 1 suffers from insufﬁcient exploration

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: 5. A RCHITECTURAL I NSIGHTS
Revised: 5. A RCHITECTURAL I NSIGHTS

Original: To show that our automated design can produce new architectural insights, we provide a qualitative comparison of Eyeriss with our solution for DQN.
Revised: To show that our automated design can produce new architectural insights, we provide a qualitative comparison of Eyeriss with our solution for DQN.

Original: Our design predominantly differs in the shape of the PE array, as well as in the number of memory buffers used.
Revised: Our design predominantly differs in the shape of the PE array, as well as in the number of memory buffers used.

Original: Eyeriss allocates the majority of its local buffer storage for ﬁlter weights, which are poorly utilized.
Revised: Eyeriss allocates the majority of its local buffer storage for ﬁlter weights, which are poorly utilized.

Original: Our design increases buffer utilization by storing multiple inputs and output elements.
Revised: Our design increases buffer utilization by storing multiple inputs and output elements.

Original: We can also plug our hardware conﬁguration into the heuristic-based optimizer from prior work (Parashar et al., 2019) and attempt to ﬁnd a software mapping.
Revised: We can also plug our hardware conﬁguration into the heuristic-based optimizer from prior work (Parashar et al., 2019) and attempt to ﬁnd a software mapping.

Original: We do this using the 12 × 14 PE array from DQN.
Revised: We do this using the 12 × 14 PE array from DQN.

Original: Timeloop’s software optimizers are unable to ﬁnd the same mapping that we do, with the best result being 52% worse than the baseline.
Revised: Timeloop’s software optimizers are unable to ﬁnd the same mapping that we do, with the best result being 52% worse than the baseline.

Original: This demonstrates the utility of a learned co-design approach that enables the software optimizer to be robust across different hardware architectures.
Revised: This demonstrates the utility of a learned co-design approach that enables the software optimizer to be robust across different hardware architectures.

Original:  R ELATED W ORK 6. H ARDWARE TO O PTIMIZE DNN S
Revised:  R ELATED W ORK 6. H ARDWARE TO O PTIMIZE DNN S

Original: Accelerators are specialized processors that provide signiﬁcant performance and efﬁciency improvements by targeting speciﬁc workloads; they also typically require signiﬁcant manual design.
Revised: Accelerators are specialized processors that provide signiﬁcant performance and efﬁciency improvements by targeting speciﬁc workloads; they also typically require signiﬁcant manual design.

Original: For deep learning, these primitives are often basic linear algebra subprograms (BLAS).
Revised: For deep learning, these primitives are often basic linear algebra subprograms (BLAS).

Original: Prior work has designed specialized hardware to execute BLAS kernels.
Revised: Prior work has designed specialized hardware to execute BLAS kernels.

Original: Google’s TPU (Jouppi et al., 2017) uses large hardware structures called systolic arrays (Kung & Leiserson, 1979), and NVIDIA’s
Revised: Google’s TPU (Jouppi et al., 2017) uses large hardware structures called systolic arrays (Kung & Leiserson, 1979), and NVIDIA’s

Original: GPUs have tensor cores (NVIDIA, 2017).
Revised: GPUs have tensor cores (NVIDIA, 2017).

Original: DianNao (Chen et al., 2014a) computes fully connected layers of a neural network using multiply-accumulate trees, and its successor, DaDianNao
Revised: DianNao (Chen et al., 2014a) computes fully connected layers of a neural network using multiply-accumulate trees, and its successor, DaDianNao

Original: (Chen et al., 2014b), improves data locality for large neural networks by tiling processing elements (PEs) around on-chip eDRAM banks.
Revised: (Chen et al., 2014b), improves data locality for large neural networks by tiling processing elements (PEs) around on-chip eDRAM banks.

Original: A more specialized version, ShiDianNao (Du et al., 2015), focuses on dense convolutional neural networks (CNNs), which exhibit regular computational behavior with high data reuse.
Revised: A more specialized version, ShiDianNao (Du et al., 2015), focuses on dense convolutional neural networks (CNNs), which exhibit regular computational behavior with high data reuse.

Original: Eyeriss (Chen et al., 2016) and Eyeriss v2 (Chen et al., 2019) also focus on CNNs, introducing a speciﬁc dataﬂow that exploits a reuse pattern exhibited by 2D convolutions.
Revised: Eyeriss (Chen et al., 2016) and Eyeriss v2 (Chen et al., 2019) also focus on CNNs, introducing a speciﬁc dataﬂow that exploits a reuse pattern exhibited by 2D convolutions.

Original: To improve performance scaling, Eyeriss v2 uses a more sophisticated interconnect than its predecessor, and it also introduces a variant that targets sparse CNNs.
Revised: To improve performance scaling, Eyeriss v2 uses a more sophisticated interconnect than its predecessor, and it also introduces a variant that targets sparse CNNs.

Original: Prior work (Parashar et al., 2017; Zhang et al., 2016) has dealt with sparsity by suppressing zero-valued activations and storing and operating on compressed data.
Revised: Prior work (Parashar et al., 2017; Zhang et al., 2016) has dealt with sparsity by suppressing zero-valued activations and storing and operating on compressed data.

Original: Many other domain speciﬁc architectures have been proposed to take advantage of local communication patterns (Farabet et al., 2011), 3D-stacked bandwidth memory (Kim et al., 2016; Gao et al., 2017), or multi-chip modules (Shao et al., 2019).
Revised: Many other domain speciﬁc architectures have been proposed to take advantage of local communication patterns (Farabet et al., 2011), 3D-stacked bandwidth memory (Kim et al., 2016; Gao et al., 2017), or multi-chip modules (Shao et al., 2019).

Original: Recent work (Yang et al., 2020) presents heuristics for automatically synthesizing hardware using a domain-speciﬁc language.
Revised: Recent work (Yang et al., 2020b) presents heuristics for automatically synthesizing hardware using a domain-speciﬁc language.

Original: 6. S OFTWARE TO O PTIMIZE DNN S
Revised: 6. S OFTWARE TO O PTIMIZE DNN S

Original: Software  optimizations for reducing the compute and storage requirements of neural networks include loop blocking (tiling), loop reordering, and loop unrolling (Whaley & Dongarra, 1998;
Revised: Software compiler optimizations for reducing the compute and storage requirements of neural networks include loop blocking (tiling), loop reordering, and loop unrolling (Whaley & Dongarra, 1998;

Original: Mullapudi et al., 2016;
Revised: Mullapudi et al., 2016;

Original: Bondhugula et al., 2008).
Revised: Bondhugula et al., 2008).

Original: Compilers such as TVM (Chen et al., 2018) have used learned cost models to optimize execution efﬁciency.
Revised: Compilers such as TVM (Chen et al., 2018) have used learned cost models to optimize execution efﬁciency.

Original: Similarly, Timeloop uses a grid or random search to optimize software mappings on a user-speciﬁed hardware architecture (Parashar et al., 2019).
Revised: Similarly, Timeloop uses a grid or random search to optimize software mappings on a user-speciﬁed hardware architecture (Parashar et al., 2019).

Original: However, all previous software optimizers treat hardware as a black box and ignore interactions between hardware and software.
Revised: However, all previous software optimizers treat hardware as a black box and ignore interactions between hardware and software.

Revised: 6. N EURAL A RCHITECTURE S EARCH

Original: Ours is the ﬁrst work that systematically explores the space of both hardware and software optimizations; this larger search space requires a more principled search method, which motivates our constrained Bayesian optimization framework.
Revised: Neural architecture search (NAS) is a research area that seeks to automatically select the optimal neural network topology or architecture to select for each problem, often agnostic to the underlying hardware or software (Zoph & Le, 2016). NAS is disjoint to our work, as we seek to optimize lower-level hardware/software primitives, but an optimization approach that considers neural network architecture, software mappings, and hardware architecture could be future work. 6. B I -L EVEL O PTIMIZATION A PPROACHES Recent research has started to optimize pairs of these categories jointly.

Revised: For example, EDD (Li et al., 2020) co-optimizes NAS and software compiler optimizations on embedded systems, but does not consider hardware architecture.

Revised: Several prior papers have co-optimized NAS with hardware architecture (Jiang et al., 2020; Yang et al., 2020a; Lin et al.;

Revised: Abdelfattah et al., 2020).

Revised: These works either ﬁx the software compiler optimizations (Jiang et al., 2020; Lin et al.;

Revised: Abdelfattah et al., 2020)

Revised: or tie the compiler optimizations to the hardware search

Revised: (Yang et al., 2020a), which simpliﬁes the search problem, but the compiler optimizations cannot be customized to different workloads once the hardware is synthesized.

Revised: Ours is the ﬁrst work that systematically co-optimizes the space of both hardware and software compiler optimizations.

Revised: This larger search space requires a principled search method, which motivates our constrained Bayesian optimization framework.

Revised: Our nested search minimizes expensive hardware generation and allows us to optimize the hardware for the entire model, but still optimize software mappings for each layer of the neural network.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we have cast hardware/software co-design as a Bayesian optimization problem.
Revised: In this paper, we have cast hardware/software co-design as a Bayesian optimization problem.

Original: We have shown that standard mechanisms have difﬁculty navigating the complex, highly constrained design space, so we have presented a novel constrained formulation that allows the optimizer to efﬁciently identify desirable points in this design space.
Revised: We have shown that standard mechanisms have difﬁculty navigating the complex, highly constrained design space, so we have presented a novel constrained formulation that allows the optimizer to efﬁciently identify desirable points in this design space.

Original: The use of machine learning to automate hardware/software co-design opens many opportunities for future work.
Revised: The use of machine learning to automate hardware/software co-design opens many opportunities for future work.

Original: For example, transfer learning could dramatically reduce design time across designs and models.
Revised: For example, transfer learning could dramatically reduce design time across designs and models.

Original: The techniques described here are not limited to DNN architectures, which is signiﬁcant because as we enter the golden age of computer architecture (Hennessy & Patterson, 2019), it is essential that we develop automatic mechanisms for architectural exploration that quickly produce custom hardware accelerators.
Revised: The techniques described here are not limited to DNN architectures, which is signiﬁcant because as we enter the golden age of computer architecture (Hennessy & Patterson, 2019), it is essential that we develop automatic mechanisms for architectural exploration that quickly produce custom hardware accelerators.

Original: A A PPENDIX A. P ARAMETERS AND CONSTRAINTS
Revised: A A PPENDIX A. P ARAMETERS AND CONSTRAINTS

Original: [Figure]
Revised: [Figure]

Original: the arrangement of the
Revised: the arrangement of the

Original: [Figure]
Revised: [Figure]

Original: Decide the partition of local
Revised: Decide the partition of local

Original: [Figure]
Revised: [Figure]

Original: The partition leads to sub-buffers with inflexible sizes
Revised: The partition leads to sub-buffers with inflexible sizes

Original: [Figure]
Revised: [Figure]

Original: This is useful as the latency to access each smaller sub-
Revised: This is useful as the latency to access each smaller sub-

Original: [Figure]
Revised: [Figure]

Original: decreases
Revised: decreases

Original: [Figure]
Revised: [Figure]

Original: the arrangement of global buffer
Revised: the arrangement of global buffer

Original: [Figure]
Revised: [Figure]

Original: and its connection between global buffer and per PE’s
Revised: and its connection between global buffer and per PE’s

Original: [Figure]
Revised: [Figure]

Original: (Local buffer of PEs along the X-axis shares the instances of global buffer along the
Revised: (Local buffer of PEs along the X-axis shares the instances of global buffer along the

Original: X
Revised: X

Original: [Figure]
Revised: [Figure]

Original: axis)
Revised: axis)

Original: [Figure]
Revised: [Figure]

Original: PE
Revised: PE

Original: [Figure]
Revised: [Figure]

Original: Factors
Revised: Factors

Original: [Figure]
Revised: [Figure]

Original: PE
Revised: PE

Original: [Figure]
Revised: [Figure]

Original: mesh
Revised: mesh

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: width
Revised: width

Original: [Figure]
Revised: [Figure]

Original: a global buffer
Revised: a global buffer

Original: [Figure]
Revised: [Figure]

Original: Determines of the number of wider structures
Revised: Determines of the number of wider structures

Original: [Figure]
Revised: [Figure]

Original: multiple entries are ganged
Revised: multiple entries are ganged

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: B H YPERPARAMTERS FOR BO
Revised: B H YPERPARAMTERS FOR BO

Original: In Figure 10 we report the hyperparamters for BO.
Revised: In Figure 10 we report the hyperparamters for BO.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: C N EURAL M ODEL S PECIFICATIONS .
Revised: C N EURAL M ODEL S PECIFICATIONS .

Original: In Figure 11 and Figure 12 we report the speciﬁcations of neural models benchmarked in this paper.
Revised: In Figure 11 and Figure 12 we report the speciﬁcations of neural models benchmarked in this paper.

Original: D P ARAMTERIZATION OF 2D C ONVOLUTION
Revised: D P ARAMTERIZATION OF 2D C ONVOLUTION

Original: Listing 14 gives the seven-level nested loop that comprises a 2D convolution.
Revised: Listing 14 gives the seven-level nested loop that comprises a 2D convolution.

Original: Figure 15 shows a design point for the CONV4 layer of ResNet.
Revised: Figure 17 shows a design point for the CONV4 layer of ResNet.

Original: The architecture components are again the same as in the 1D example, but since the memory footprint is signiﬁcantly larger, the PE can no longer capture all data reuse, so the Global Buffer must store large portions of the inputs and outputs. 
Revised: The architecture components are again the same as in the 1D example, but since the memory footprint is signiﬁcantly larger, the PE can no longer capture all data reuse, so the Global Buffer must store large portions of the inputs and outputs. E E XAMPLE P ARAMETER V ECTOR

Revised: Below are example vectors of hardware and software parameters our BO optimizes.

Original: E A DDITIONAL RESULTS E. S OFTWARE OPTIMIZATION
Revised: F A DDITIONAL RESULTS F. S OFTWARE OPTIMIZATION

Original: In Figure 16 we show more examples of the software optimization over multiple layers of the different architectures.
Revised: In Figure 18 we show more examples of the software optimization over multiple layers of the different architectures.

Original: Our Bayesian optimization formulation consistently outperforms the baselines (Chen et al., 2018).
Revised: Our Bayesian optimization formulation consistently outperforms the baselines (Chen et al., 2018).

Original: E. A BLATIONS
Revised: F. A BLATIONS

Original: In Figure 17 we compare different surrogate models and acquisition functions for Bayesian optimization of the software mapping.
Revised: In Figure 19 we compare different surrogate models and acquisition functions for Bayesian optimization of the software mapping.

Original: We found Gaussian processes with LCB to consistently outperform other alternatives.
Revised: We found Gaussian processes with LCB to consistently outperform other alternatives.

Original: In Figure 18 we investigate the robustness of LCB for software optimization using different values of λ .

Original: We found that λ = 0 .

Original: 1 tends to be too greedy, but that above λ = 0 .

Original: 5 , LCB tends to be fairly robust.

Original: Model
Revised: Model

Original: Layers
Revised: Layers

Original: Specifications
Revised: Specifications

Original: ResNet
Revised: ResNet

Original: ResNet-K
Revised: ResNet-K

Original: Filter size: 3 ×
Revised: Filter size: 3 ×

Original: Output size: 56 ×# input channel:# output channel:
Revised: Output size: 56 ×# input channel:# output channel:

Original: Stride:
Revised: Stride:

Original: ResNet-K
Revised: ResNet-K

Original: Filter size: 3 ×
Revised: Filter size: 3 ×

Original: Output size: 28 ×# input channel:# output channel:
Revised: Output size: 28 ×# input channel:# output channel:

Original: Stride:
Revised: Stride:

Original: ResNet-K
Revised: ResNet-K

Original: Filter size: 3 ×
Revised: Filter size: 3 ×

Original: Output size: 14 ×# input channel:# output channel:
Revised: Output size: 14 ×# input channel:# output channel:

Original: Stride:
Revised: Stride:

Original: ResNet-K
Revised: ResNet-K

Original: Filter size: 3 ×
Revised: Filter size: 3 ×

Original: Output size: 7 ×# input channel:# output channel:
Revised: Output size: 7 ×# input channel:# output channel:

Original: Stride:
Revised: Stride:

Original: DQN
Revised: DQN

Original: DQN-K
Revised: DQN-K

Original: Filter size: 8 ×
Revised: Filter size: 8 ×

Original: Output size: 20 ×# input channel:# output channel:
Revised: Output size: 20 ×# input channel:# output channel:

Original: Stride:
Revised: Stride:

Original: DQN-K
Revised: DQN-K

Original: Filter size: 4 ×
Revised: Filter size: 4 ×

Original: Output size: 9 ×# input channel:# output channel:
Revised: Output size: 9 ×# input channel:# output channel:

Original: Stride:
Revised: Stride:

Revised: In Figure 20 we investigate the robustness of LCB for software optimization using different values of λ .

Revised: We found that λ = 0 .

Revised: 1 tends to be too greedy, but that above λ = 0 .

Revised: 5 , LCB tends to be fairly robust.

Original: Model
Revised: Model

Original: Layers
Revised: Layers

Original: Specifications
Revised: Specifications

Original: MLP
Revised: MLP

Original: MLP-K d in : 512 d out :
Revised: MLP-K d in : 512 d out :

Original: MLP-K d in : 64 d out :
Revised: MLP-K d in : 64 d out :

Original: [Figure]
Revised: [Figure]

Original: Transformer-K d model = 

Revised: Transformer-K d model =

Original: d v =d k =h =
Revised: d v =d k =h =

Original: Transformer-K d model = 

Revised: Transformer-K d model =

Original: d v =d k =h =
Revised: d v =d k =h =

Original: Transformer-K d model = 

Revised: Transformer-K d model =

Original: d v =d k =h =
Revised: d v =d k =h =

Original: Transformer-K d model = 

Revised: Transformer-K d model =

Original: d v =d k =h =
Revised: d v =d k =h =

Original: [Figure]
Revised: [Figure]

Original: f o r n in [ 0 :N) f
Revised: f o r n in [ 0 :N) f

Original: o r k in [ 0 :K) f
Revised: o r k in [ 0 :K) f

Original: o r r in [ 0 :R) f
Revised: o r r in [ 0 :R) f

Original: o r s in [ 0 : S ) f
Revised: o r s in [ 0 : S ) f

Original: o r p in [ 0 : P ) f
Revised: o r p in [ 0 : P ) f

Original: o r q in [ 0 :Q) f
Revised: o r q in [ 0 :Q) f

Original: o r c in [ 0 :C) o
Revised: o r c in [ 0 :C) o

Original: [Equation]
Revised: [Equation]

Original: weights
Revised: weights

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: ] ∗
Revised: ] ∗

Original: [Equation]
Revised: [Equation]

Original: loop.
Revised: loop.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Revised: P

Original: [Figure]
Revised: [Figure]

Original: ( r
Revised: r

Original: [Figure]
Revised: [Figure]

Original: c i pr
Revised: c i p

Original: [Figure]
Revised: [Figure]

Original: c a l )
Revised: [Figure] models and acquisition functions.

Original: [Figure]
Revised: [Figure]