D ATA -D RIVEN O FFLINE O PTIMIZATION FOR A RCHITECTING H ARDWARE A CCELERATORS
Anonymous authors
Paper under double-blind review
A BSTRACT
To attain higher efﬁciency, the industry has gradually reformed towards applicationspeciﬁc hardware accelerators.
While such a paradigm shift is already starting to show promising results, designers need to spend considerable manual effort and perform large number of time-consuming simulations to ﬁnd accelerators that can accelerate multiple target applications while obeying design constraints.
Moreover, such a “simulation-driven” approach must be re-run from scratch every time the set of target applications or design constraints change.
An alternative paradigm is to use a “data-driven”, ofﬂine approach that utilizes logged simulation data, to architect hardware accelerators, without needing any form of simulations.
Such an approach not only alleviates the need to run time-consuming simulation, but also enables data reuse and applies even when set of target applications changes.
In this paper, we develop such a data-driven ofﬂine optimization method for designing hardware accelerators, dubbed P RIME , that enjoys all of these properties.
Our approach learns a conservative, robust estimate of the desired cost function, utilizes infeasible points and optimizes the design against this estimate without any additional simulator queries during optimization.
P RIME architects accelerators—tailored towards both single- and multi-applications—improving performance upon stat-of-theart simulation-driven methods by about 1.54 × and 1.20 × , while considerably reducing the required total simulation time by 93% and 99%, respectively.
In addition, P RIME also architects effective accelerators for unseen applications in a zero-shot setting, outperforming simulation-based methods by 1.26 × .
I NTRODUCTION
The death of Moore’s Law (Esmaeilzadeh et al., 2011) and its spiraling effect on the semiconductor industry have driven the growth of specialized hardware accelerators.
These specialized accelerators are tailored to speciﬁc applications (Yazdanbakhsh et al., 2021a; Reagen et al., 2017; Nardi et al., 2019; Shi et al., 2020).
To design specialized accelerators, designers ﬁrst spend considerable amounts of time developing simulators that closely model the real accelerator performance, and then optimize the accelerator using the simulator.
While such simulators can automate accelerator design, this requires a large number of simulator queries for each new design, both in terms of simulation time and compute requirements, and this cost increases with the size of the design space (Yazdanbakhsh et al., 2021b; Shi et al., 2020; Hegde et al., 2021).
Moreover, most of the accelerators in the design space are typically infeasible (Hegde et al., 2021;
Yazdanbakhsh et al., 2021a) because of build errors in silicon or compilation/mapping failures.
When the target applications change or a new application is added, the complete simulation-driven procedure is generally repeated.
To make such approaches efﬁcient and practically viable, designers typically “bake-in” constraints or otherwise narrow the search space, but such constraints can leave out high-performing solutions (Dave et al., 2019; Parashar et al., 2019; Chatarasi et al., 2020).
An alternate approach, proposed in this work, is to devise a data-driven optimization method that only utilizes a database of previously tested accelerator designs, annotated with measured performance metrics, to produce new optimized designs without additional active queries to an explicit silicon or a cycle-accurate simulator.
Such a data-driven approach provides three key beneﬁts: (1) it signiﬁcantly shortens the recurring cost of running large-scale simulation sweeps, (2) it alleviates the need to explicitly bake in domain knowledge or search space pruning, and (3) it enables data re-use by empowering the designer to optimize accelerators for new unseen applications, by the virtue of effective generalization.
While data-driven approaches have shown promising results in biology (Fu
Levine, 2021;
Brookes et al., 2019; Trabucco et al., 2021a), using ofﬂine optimization methods to design accelerators has been challenging primarily due to the abundance of infeasible design points (Yazdanbakhsh et al., 2021a; Hegde et al., 2021) (See Figure 3 and Appendix Figure 13).
The key contribution of this paper is a data-driven approach, P RIME , to automatically architect high-performing application-speciﬁc accelerators by using only previously collected ofﬂine data.
P RIME learns a robust surrogate model of the task objective function from an existing ofﬂine dataset, and ﬁnds high-performing application-speciﬁc accelerators by optimizing the architectural parameters against this learned surrogate function, as shown in Figure 1.
While naïvely learned
[Figure]
surrogate functions usually produces poor-performing, outof-distribution designs that appear quite optimistic under the learned surrogate (Kumar & Levine, 2019; Brookes et al., 2019; Trabucco et al., 2021a).
The robust surrogate in P RIME is explicitly trained to prevent overestimation on “adversarial” designs that would be found during optimization.
Furthermore, in contrast to prior works that discard infeasible points (Hegde et al., 2021; Trabucco et al., 2021a), our proposed method instead incorporates infeasible points when learning the conservative surrogate by treating them as additional negative samples.
During evaluation, P RIME optimizes the learned conservative surrogate using a discrete optimizer.
Our results show that P RIME architects hardware accelerators that improve over the best design in the training dataset, on average, by 2.46 × (up to 6.7 × ) when specializing for a single application.
In this case, P RIME also improves over the best conventional simulator-driven optimization methods by 1.54 × (up to 6.6 × ).
These performance improvements are obtained while reducing the total simulation time to merely 7% and 1% of that of the simulator-driven methods for single-task and multi-task optimization, respectively.
More importantly, a contextual version of P RIME can design accelerators that are jointly optimal for a set of nine applications without requiring any additional domain information.
In this challenging setting, P RIME improves over simulator-driven methods, which tend to scale poorly as more applications are added, by 1.38 × .
Finally, we show that the surrogates trained with P RIME on a set of training applications can be readily used to obtain accelerators for unseen target applications, without any retraining on the new application.
Even in this zero-shot optimization scenario, P RIME outperforms simulator-based methods that require re-training and active simulation queries by up to 1.67 × .
In summary, P RIME allows us to effectively address the shortcomings of simulation-driven approaches, (1) signiﬁcantly reduces the simulation time, (2) enables data reuse and enjoys generalization properties, and (3) does not require domain-speciﬁc engineering or search space pruning.
To facilitate further research in leveraging machine learning in architecting hardware accelerators, we also release the dataset in our experiments (Appendix B.3), consisting of many accelerator design points (both infeasible and feasible).
B ACKGROUND ON H ARDWARE A CCELERATORS
The goal of specialized hardware accelerators—Google TPUs (Jouppi et al., 2017; Gupta & Akin, 2020), Nvidia GPUs (Nvidia, 2021b), GraphCore (GraphCore, 2021)—is to improve the performance of speciﬁc applications, such as machine learning models.
To design such accelerators, architects typically create a parameterized design and sweep over parameters using simulation.
Target hardware accelerators.
Our primary evaluation uses an industry-grade and highly parameterized template-based accelerator following prior work (Yazdanbakhsh et al., 2021b).
This template enables architects to determine the organization of various components, such as compute units, memory cells, memory, etc., by searching for these conﬁgurations in a discrete design space.
Some
ML applications may have large memory requirements (e.g., large language models (Brown et al., 2020)) demanding sufﬁcient on-chip memory resources, while others may beneﬁt from more compute blocks.
The hardware design workﬂow directly selects the values of these parameters.
In addition to this accelerator and to further show the generality of our method to other accelerator design problems, we evaluate two distinct dataﬂow accelerators with different search spaces, namely
NVDLA-style (Nvidia, 2021a) and ShiDianNao-style (Du et al., 2015) from Kao et al.
(2020) (See
Section 6 and Appendix C for a detailed discussion; See Table 6 for results).
[Table]
[Figure]
[Table]
How does an accelerator work?
We brieﬂy explain the computation ﬂow on our template-based accelerators (Figure 2) and refer the readers to Appendix C for details on other accelerators.
This template-based accelerator is a 2D array of processing elements (PEs).
Each PE is capable of performing matrix multiplications in a single instruction multiple data (SIMD) paradigm (Gove et al., 1993).
A controller orchestrates the data transfer (both activations and model parameters) between off-chip DRAM memory and the on-chip buffers and also reads in and manages the instructions (e.g. convolution, pooling, etc.) for execution.
The computation stages on such accelerators start by sending a set of activations to the compute lanes, executing them in SIMD manner, and either storing the partial computation results or ofﬂoading them back into off-chip memory.
Compared to prior works (Hegde et al., 2021; Du et al., 2015; Kao et al., 2020), this parameterization is unique—it includes multiple compute lanes per each PE and enables
SIMD execution model within each compute lane—and yields a distinct accelerator search space accompanied by an end-to-end simulation framework.
More details in Appendix C.
P ROBLEM S TATEMENT , T RAINING D ATA AND E VALUATION P ROTOCOL
Our template-based parameterization maps the accelerator, denoted as x , to a discrete design space, x = [ x 1 , x 2 , · · · , x K ] , and each x i is a discrete-valued variable representing one component of the microarchitectural template, as shown in Table 1 (See Appendix C for the description of other accelerator search spaces studied in our work).
A design maybe be infeasible due to various reasons, such as a compilation failure or the limitations of physical implementation, and we denote the set of all such feasibility criterion as Feasible( x ) .
The feasibility criterion depends on both the target software and the underlying hardware, and it is not easy to identify if a given x is infeasible without explicit simulation.
We will require our optimization procedure to not only learn the value of the objective function but also to learn to navigate through a sea of infeasible solutions to high-performing feasible solutions x ∗ satisfying Feasible( x ∗ )
= 1 .
Our training dataset D consists of a modest set of accelerators x i that are randomly sampled from the design space and evaluated by the hardware simulator.
We partition the dataset D into two subsets, D feasible and D infeasible .
Let f ( x ) denote the desired objective (e.g., latency, power, etc.)
we intend to optimize over the space of accelerators x .
We do not possess functional access to f ( x ) , and the optimizer can only access f
( x ) values for accelerators x in the feasible partition of the data, D feasible .
For all infeasible accelerators, the simulator does not provide any value of f ( x ) .
In addition to satisfying feasibility, the optimizer must handle explicit constraints on parameters such as area and power (Flynn & Luk, 2011).
In our applications, we impose an explicit area constraint,
Area( x ) ≤ α 0 , though additional explicit constraints are also possible.
To account for different constraints, we formulate this task as a constrained optimization problem.
Formally:
[Equation]
While Equation 1 may appear similar to other standard black-box optimization problems, solving it over the space of accelerator designs is challenging due to the large number of infeasible points, the need to handle explicit design constraints, and the difﬁculty in navigating the non-smooth landscape (See Figure 3 and Figure 12 in the Appendix) of the objective function.
What makes optimization over accelerators challenging?
Compared to other domains where model-based optimization methods have been applied (Brookes et al., 2019;
Trabucco et al., 2021a),
[Table]
optimizing accelerators introduces a number of practical challenges.
First, accelerator design spaces typically feature a narrow manifold of feasible accelerators within a sea of infeasible points (Nardi et al., 2019; Shi et al., 2020; Gelbart et al., 2014), as visualized in Figure 3 and Appendix (Figure 13)
[Table]
While some of these infeasible points can be identiﬁed via simple rules (e.g. estimating chip area usage), most infeasible points correspond to failures during compilation or hardware simulation.
These infeasible points are generally not straightforward to formulate into the optimization problem and requires simulation (Shi et al., 2020;
Parashar et al., 2019; Yazdanbakhsh et al., 2021a).
Second, the optimization objective can exhibit high sensitivity to small variations in some architecture parameters (Figure 12b) in some regions of the design space, but remain relatively insensitive in other parts, resulting in a complex optimization landscape.
This suggests that optimization algorithms based on local parameter updates (e.g., gradient ascent, evolutionary schemes, etc.) may have a challenging task traversing the nearly ﬂat landscape of the objective, which can lead to poor performance.
Training dataset.
We used an ofﬂine dataset D of (accelerator parameters, latency) via random sampling from the space of 452M possible accelerator conﬁgurations.
Our method is only provided with a relatively modest set of feasible points ( ≤ 8000 points) for training, and these points are the worst-performing feasible points across the pool of randomly sampled data.
This dataset is meant to reﬂect an easily obtainable and an application-agnostic dataset of accelerators that could have been generated once and stored to disk, or might come from real physical experiments.
We emphasize that no assumptions or domain knowledge about the application use case was made during dataset collection.
Table 2 depicts the list of target applications, evaluated in this work, includes three variations of
MobileNet (Gupta & Akin, 2020; Sandler et al., 2018; Howard et al., 2019), three in-house industry-level models for object detection (M4,
M5, M6; names redacted to prevent anonymity violation), a U-net model (Ronneberger et al., 2015), and two RNN-based encoderdecoder language models (Graves, 2012;
He et al., 2019; Sainath et al., 2020; Li et al., 2021).
These applications span the gamut from small models, such as M6, with only 0.4 MB model parameters that demands less on-chip memory, to the medium-sized models ( ≥ 5 MB), such as MobileNetV3 and M4 models, and large models ( ≥ 19 MB), such as t-RNNs, hence requiring larger on-chip memory.
Evaluation protocol.
To compare state-of-the-art simulator-driven methods and our data-driven method, we limit the number of feasible points (costly to evaluate) that can be used by any algorithm to equal amounts.
We still provide infeasible points to any method and leave it up to the optimization method to use it or not.
This ensures our comparisons are fair in terms of the amount of data available to each method.
However, it is worthwhile to note that in contrast to our method where worse-quality data points from small ofﬂine dataset are used, the simulator-driven methods have an inherent advantage because they can steer the query process towards the points that are more likely to be better in terms of performance.
Following prior work (Brookes et al., 2019; Trabucco et al., 2021a;b), we evaluate each run of a method by ﬁrst sampling the top n = 256 design candidates according to the algorithm’s predictions, evaluating all of these under the ground truth objective function and recording the performance of the best accelerator design.
The ﬁnal reported results is the median of ground truth objective values across ﬁve independent runs.
P RIME : A RCHITECTING A CCELERATORS VIA C ONSERVATIVE S URROGATES
As shown in Figure 4, our method ﬁrst learns a conservative surrogate model of the optimization objective using the ofﬂine dataset.
Then, it optimizes the learned surrogate using a discrete optimizer.
The optimization process does not require access to a simulator, nor to real-world experiments beyond the initial dataset, except when evaluating the ﬁnal top-performing n = 256 designs (Section 3).
Learning conservative surrogates using logged ofﬂine data.
Our goal is to utilize a logged dataset of feasible accelerator designs labeled with the desired performance metric (e.g., latency),
D feasible , and infeasible designs, D infeasible to learn a mapping f θ : X → R , that maps the ac- celerator conﬁguration x to its corresponding metric y .
This learned surrogate can then be optimized by the optimizer.
While a straightforward approach for learning such a mapping is to train it via supervised regression, by minimizing the mean-squared error E x i ,y i
∼D [( f θ ( x i ) − y i ) 2 ] , prior work (Kumar & Levine, 2019; Kumar et al., 2020; Trabucco et al., 2021a) has shown that such predictive models can arbitrarily overestimate the value of an unseen input x i .
This can cause the optimizer to ﬁnd a solution x ∗ that performs poorly in the simulator but looks promising under the learned model.
We empirically validate this overestimation hypothesis and ﬁnd it to confound the optimizer in on our problem domain as well (See Figure 14 in Appendix).
[Figure]
L 2 =
ˆ f θ ( x j )
L 1 = (ˆ f θ ( x i ) − y i ))
Transformer Layer
Transformer Layer xx c ··· ···
Attention
Accelerator Conﬁguration f θ ( x ) f mθ ( x ) f 1 θ ( x )
To prevent overestimated values at unseen inputs from confounding the optimizer, we build on COMs (Trabucco et al., 2021a) and train f θ ( x ) with an additional term that explicitly maximizes the function value f θ ( x ) at unseen x values.
Such unseen designs x , where the learned function f θ ( x ) is likely to be overestimated, are “negative mined” by running a few iterations of an approximate stochastic optimization procedure that aims to maximize f θ in the inner loop.
This procedure is analogous to adversarial training (Goodfellow et al., 2014).
Equation 2 formalizes this objective:
[Equation]
x − i denotes the negative samples produced from an optimizer Opt( · ) that attempts to maximize the current learned model, f θ .
We will discuss our choice of Opt in the Appendix Section B.
Incorporating design constraints via infeasible points.
While prior work (Trabucco et al., 2021a) simply optimizes Equation 2 to learn a surrogate, this is not enough when optimizing over accelerators, as we will also show empirically (Appendix A.1).
This is because explicit negative mining does not provide any information about accelerator design constraints.
Fortunately, this information is provided by infeasible points, D infeasible .
The training procedure in Equation 2 provides a simple way to do incorporate such infeasible points: we simply incorporate x (cid:48)
i ∼ D infeasible as additional negative samples and maximize the prediction at these points.
This gives rise to our ﬁnal objective:
[Equation]
Multi-model optimization and zero-shot generalization.
One of the central beneﬁts of a datadriven approach is that it enables learning powerful surrogates that generalize over the space of applications, potentially being effective for new unseen application domains.
In our experiments, we evaluate P RIME on designing accelerators for multiple applications denoted as k = 1 , · · · , K , jointly or for a novel unseen application.
In this case, we utilized a dataset
D =
{D 1 , · · · , D K } , where each
D k consists of a set of accelerator designs, annotated with the latency value and the feasibility criterion for a given application k .
While there are a few overlapping designs in different parts of the dataset annotated for different applications, most of the designs only appear in one part.
To train a single conservative surrogate f θ ( x ) for multiple applications, we extend the training procedure in Equation 3 to incorporate context vectors c k ∈ R d for various applications driven by a list of application properties in Table 2.
The learned function in this setting is now conditioned on the context f θ ( x , c k ) .
We train f θ via the objective in Equation 3, but in expectation over all the contexts and their corresponding datasets: min
θ E k ∼ [ K ]
[Equation]
L
[Equation]
θ
[Equation]
Once such a contextual surrogate is learned, we can either optimize the average surrogate across a set of contexts { c i , c 2 , · · · , c n } to obtain an accelerator that is optimal for multiple applications simultaneously on an average (“multi-model” optimization), or optimize this contextual surrogate for a novel context vector, corresponding to an unseen application (“zero-shot” generalization).
In this case, P RIME is not allowed to train on any data corresponding to this new unseen application.
While such zero-shot generalization might appear surprising at ﬁrst, note that the context vectors are not simply one-hot vectors, but consist of parameters with semantic information, which the surrogate can generalize over.
Optimizing the learned conservative surrogate.
Prior work (Yazdanbakhsh et al., 2021a) has shown that the most effective optimizers for accelerator design are meta-heuristic/evolutionary optimizers.
We therefore choose to utilize, ﬁreﬂy (Yang, 2010; Yang & Deb, 2010; Liu et al., 2013) to optimize our conservative surrogate.
This algorithm maintains a set of optimization candidates (a.k.a.
“ﬁreﬂies”) and jointly update them towards regions of low objective value, while adjusting their relative distances appropriately to ensure multiple high-performing, but diverse solutions.
We discuss additional details in Appendix B.1.
Cross validation: which model and checkpoint should we evaluate?
Similarly to supervised learning, models trained via Equation 3 can overﬁt, leading to poor solutions.
Thus, we require a procedure to select which hyperparameters and checkpoints should actually be used for the design.
This is crucial, because we cannot arbitrarily evaluate as many models as we want against the simulator.
While effective methods for model selection have been hard to develop in ofﬂine optimization (Trabucco et al., 2021a;b), we devised a simple scheme using a validation set for choosing the values of α and β (Equation 3), as well as which checkpoint to utilize for generating the design.
For each training run, we hold out the best 20% of the points out of the training set and use them only for cross-validation as follows.
Typical cross-validation strategies in supervised learning involve tracking validation error (or risk), but since our model is trained conservatively, its predictions may not match the ground truth, making such validation risk values unsuitable for our use case.
Instead, we track
Kendall’s ranking correlation between the predictions of the learned model f θ ( x i ) and the ground truth values y i (Appendix B) for the held-out points for each run.
We pick values of α , β and the checkpoint that attain the highest validation ranking correlation.
We present the pseudo-code for
P RIME (Algorithm 1) and implementation details in Appendix B.1.
R ELATED W ORK
Optimizing hardware accelerators has become more important recently.
Prior works (Parsa et al., 2020;
Iqbal et al., 2020; Sun et al., 2020; Nardi et al., 2019;
Cong et al., 2018;
Koeplinger et al., 2018; Balaprakash et al., 2016; Ansel et al., 2014; Hegde et al., 2021; Venkatesan et al., 2019; Xu et al., 2020) mainly rely on expensive-to-query hardware simulators to navigate the search space and/or target single-application accelerators.
For example, HyperMapper (Nardi et al., 2019) targets compiler optimization for FPGAs by continuously interacting with the simulator in a design space with relatively few infeasible points.
Mind Mappings (Hegde et al., 2021), optimizes software mappings to a ﬁxed hardware provided access to millions of feasible points and throws away infeasible points during learning.
MAGNet (Venkatesan et al., 2019) uses a combination of pruning heuristics and online Bayesian optimization to generate accelerators for image classiﬁcation models in a singleapplication setting.
AutoDNNChip (Xu et al., 2020) uses two-level online optimization to generate customized accelerators for ASIC and FPAG platforms.
In contrast, P RIME , does not only learn a surrogate using ofﬂine data but can also leverage information from infeasible points and can work with just a few thousand feasible points.
In addition, we devise a contextual version of P RIME that is effective in designing accelerators that are jointly optimized for multiple applications, different from prior work.
Finally, to our knowledge, our work, is the ﬁrst to demonstrate generalization to unseen applications for accelerator design, outperforming state-of-the-art online methods.
A popular approach for solving black-box optimization problems is model-based optimization (MBO) (Snoek et al., 2015; Shahriari et al., 2016; Snoek et al., 2012).
Most of these methods fail to scale to high-dimensions, and have been extended with neural networks (Snoek et al., 2015; 2012;
Kim et al., 2019; Garnelo et al., 2018b;a; Angermueller et al., 2020a; 2019; Mirhoseini et al., 2020).
While these methods work well in the active setting, they are susceptible to out-of-distribution inputs (Trabucco et al., 2021b) in the ofﬂine, data-driven setting.
To prevent this, ofﬂine MBO methods that constrain the optimizer to the manifold of valid, in-distribution inputs have been developed Brookes et al.
; Fannjiang & Listgarten (2020); Kumar & Levine (2019).
However, modeling the manifold of valid inputs can be challenging for accelerators.
P RIME dispenses with the need for generative modeling, while still avoiding out-of-distribution inputs.
P RIME builds on
“conservative” ofﬂine RL and ofﬂine MBO methods that train robust surrogates (Kumar et al., 2020;
Trabucco et al., 2021a).
However, unlike these approaches, P RIME can handle constraints by learning from infeasible data and utilizes a better optimizer (See Appendix Table 7 for a comparison).
In addition, while prior works area mostly restricted to a single application, we show that P RIME is effective in multi-task optimization and zero-shot generalization.
E XPERIMENTAL E VALUATION
Our evaluations aim to answer the following questions: Q(1) Can P RIME design accelerators tailored for a given application that are better than the best observed conﬁguration in the training dataset, and comparable to or better than state-of-the-art simulation-driven methods under a given simulator-query budget? Q(2)
Does P RIME reduce the total simulation time compared to other methods?
Q(3)
Can P RIME produce hardware accelerators for a family of different applications?
Q(4)
Can P RIME trained for a family of applications extrapolate to designing a high-performing accelerator for a new, unseen application, thereby enabling data reuse?
Additionally, we ablate various properties of P RIME (Appendix A.6) and evaluate its efﬁcacy in designing accelerators with distinct dataﬂow architectures, with a larger search space (up to 2.5 × 10 114 possible candidates).
Baselines and comparisons.
We compare P RIME against three online optimization methods that actively query the simulator: (1) evolutionary search with the ﬁreﬂy optimizer (Yazdanbakhsh et al., 2021a) (“Evolutionary”), which is the shown to outperform other online methods for accelerator design; (2) Bayesian Optimization (“Bayes Opt”)
(Golovin et al., 2017), (3) MBO (Angermueller et al., 2019).
In all the experiments, we grant all the methods the same number of feasible points.
Note that our method do not get to select these points, and use the same exact ofﬂine points across all the runs, while the online methods can actively select which points to query, and therefore require new queries for every run.
“ D (Best in Training)” denotes the best latency value in the training dataset used in P RIME .
We also present ablation results with different components of our method removed in Appendix A.6, where we observe that utilizing both infeasible points and negative sampling are generally important for attaining good results.
Appendix A.1 presents additional comparisons to COMs Trabucco et al.
(2021a)—which only obtains negative samples via gradient ascent on the learned surrogate and does not utilize infeasible points—and P3BO Angermueller et al.
(2020b)—an state-of-the-art online method in biology.
Architecting application-speciﬁc accelerators.
We ﬁrst evaluate
P RIME in designing specialized accelerators for each of the applications in Table 2.
We train a conservative surrogate using the method in Section 4 on the logged dataset for each application separately.
The area constraint α (Equation 1) is set to α = 29 mm 2 , a realistic budget for accelerators (Yazdanbakhsh et al., 2021a).
Table 3 summarizes the results.
On average, the best accelerators designed by
P RIME outperforms the best accelerator conﬁguration in the training dataset (last row Table 3), by 2.46 × .
P RIME also outperforms the accelerators in the best online method by 1.54 × (up to 5.80 × and 6.62 × in t-RNN Dec and t-RNN Enc, respectively).
Moreover, perhaps surprisingly, P RIME generates accelerators that are better than all the online optimization methods in 7/9 domains, and performs on par in several other scenarios (on average only 6.8 % slowdown compared to the best accelerator with online methods in M5 and M6).
These results indicates that ofﬂine optimization of accelerators using P RIME can be more data-efﬁcient compared to online methods with active simulation queries.
To answer Q(2) , we compare the total simulation time of P RIME and the best evolutionary approach from Table 3 on the MobileNetEdge domain.
On average, not only that P RIME outperforms the best online method that we evaluate, but also considerably reduces the total simulation time by 93%, as shown in Figure 5.
Even the total simulation time to the ﬁrst occurrence of the ﬁnal design that is eventually returned by the online methods is about 11 × what P RIME requires to ﬁne a better design.
This indicates that data-driven P RIME is much more preferred in terms of the performance-time trade-off.
The fact that our ofﬂine approach P RIME outperforms the online evolutionary method (and also other state-of-the-art online MBO methods; see Table 8) is surprising, and we suspect this is because online methods get stuck early-on during optimization, while utilizing ofﬂine data allows us
P RIME to ﬁnd better solutions via generalization (see Appendix B.1.1).
[Table] [Table]
Architecting accelerators for multiple applications.
To answer Q(3) , we evaluate the efﬁcacy of the contextual version of P RIME in designing an accelerator that attains the lowest latency averaged over a set of application domains.
As discussed previously, the training data used does not label a given accelerator with latency values corresponding to each application, and thus, P RIME must extrapolate accurately to estimate the latency of an accelerator for a context it is not paired with in the training dataset.
This also means that P RIME cannot simply return the accelerator with the best average latency and must run non-trivial optimization.
We evaluate our method in seven different scenarios (Table 4), comprising various combinations of models from Table 2 and under different area constraints, where the smallest set consists of the three MobileNet variants and the largest set consists of nine models from image classiﬁcation, object detection, image segmentation, and speech recognition.
This scenario is also especially challenging for online methods since the number of jointly feasible designs is expected to drop signiﬁcantly as more applications are added.
For instance, for the case of the MobileNet variants, the training dataset only consists of a few (20-30) accelerator conﬁgurations that are jointly feasible and high-performing (Appendix B.2—Figure 10).
Table 4 shows that, on average, P RIME ﬁnds accelerators that outperform the best online method by 1.2 × (up to 41%).
While P RIME performs similar to online methods in the smallest three-model scenario (ﬁrst row), it outperforms online methods as the number of applications increases and the set of applications become more diverse.
In addition, comparing with the best jointly feasible design point across the target applications, P RIME ﬁnds signiﬁcantly better accelerators (3.95 × ).
Finally, as the number of model increases the total simulation time difference between online methods and P RIME further widens (Figure 6).
These results indicate that P RIME is effective in designing accelerators jointly optimized across multiple applications while reusing the same dataset as for the single-task, and scales more favorably than its simulation-driven counterparts.
Appendix A.4 expounds the details of the designed accelerators for nine applications, comparing our method and the best online method.
[Table] [Table]
[Table] [Table]
Accelerating previously unseen applications (“zero-shot” optimization).
Finally, we answer
Q(4) by demonstrating that our data-driven ofﬂine method, P RIME enables effective data reuse by using logged accelerator data from a set of applications to design an accelerator for an unseen new application, without requiring any training on data from the new unseen application(s).
We train a contextual version of P RIME using a set of “training applications” and then optimize an accelerator using the learned surrogate with different contexts corresponding to “test applications
[Table]
” without any additional query to the test application dataset.
Table 5 shows, on average, P RIME outperforms the best online method by 1.26 × (up to 66 % ) and only 2 % slowdown in 1/4 cases.
Note that the difference in performance increases as the number of training applications increases.
These results show the effectiveness of P RIME in the zero-shot setting (more results in Appendix A.5).
Applying P RIME on other accelerator architectures and dataﬂows.
Finally, to assess the the generalizability of P RIME to other accelerator architectures Kao et al.
(2020), we evaluate P RIME to optimize latency of two style of dataﬂow accelerators—NVDLA-style and ShiDianNao-style—across three applications (Appendix C details the methodology).
As shown in Table 6, P RIME outperforms the online evolutionary method by 6% and improves over the best point in the training dataset by 3.75 × .
This demonstrates the efﬁcacy of P RIME with different dataﬂows and large design spaces.
D ISCUSSION
In this work, we present a data-driven ofﬂine optimization method, P RIME to automatically architect hardware accelerators.
Our method learns a conservative surrogate of the objective function by leveraging infeasible data points to better model the desired objective function of the accelerator using a one-time collected dataset of accelerators, thereby alleviating the need for time-consuming simulation.
Our results show that, on average, our method outperforms the best designs observed in the logged data by 2.46 × and improves over the best simulator-driven approach by about 1.54 × .
In the more challenging setting of designing accelerators jointly optimal for multiple applications or for new, unseen applications, zero-shot, P RIME outperforms simulator-driven methods by 1.2 × , while reducing the total simulation time by 99%.
The efﬁcacy of P RIME highlights the potential for utilizing the logged ofﬂine data in an accelerator design pipeline.
While P RIME outperforms the online methods we utilize, in principle, a strong online method can be devised by running P RIME in the inner loop.
Our goal is to not advocate that ofﬂine methods must replace online methods, but that training a strong ofﬂine optimization algorithm on ofﬂine datasets of low-performing designs can be a highly effective ingredient in hardware accelerator design.
R EPRODUCIBILITY S TATEMENT
We provide complete experimental details regarding the application domains, accelerators considered, hyperparameters for our method and our cross-validation strategy in Appendix B. Additionally, we also open-source our dataset, in anonymous folder with this submission and provide python notebooks to parse and utilize the dataset.
More information is provided in Appendix B.3.
We will also open source the code to ensure the reproducibility of our results.
Appendices A A DDITIONAL E XPERIMENTS
In this section, we present additional experiments compared to the method of Trabucco et al.
(2021a))
, present some additional results obtained by jointly optimizing multiple applications (Appendix A.2), provide an analysis of the designed accelerators (Appendix A.4) and ﬁnally, discuss how our trained conservative surrogate can be used with a different evaluation time constraint (Appendix A.3).
A. C OMPARISON TO O THER B ASELINE M ETHODS
Comparison to COMs.
In this section, we perform a comparative evaluation of P RIME to the COMs method Trabucco et al.
(2021a).
Like several ofﬂine reinforcement learning algorithms (Kumar et al., 2020), our method, P RIME and COMs are based on the key idea of learning a conservative surrogate of the desired objective function, such that it does not overestimate the value of unseen data points, which prevents the optimizer from ﬁnding accelerators that appear promising under the learned model but are not actually promising under the actual objective.
The key differences between our method and COMs are: (1) P RIME uses an evolutionary optimizer ( Opt( · ) ) for negative sampling compared to gradient ascent of COMs, which can be vastly beneﬁcial in discrete design spaces as our results show empirically, (2) P RIME can explicitly learn from infeasible data points provided to the algorithm, while COMs does not have a mechanism to incorporate the infeasible points into the learning of surrogate.
To further assess the importance of these differences in practice, we run COMs on three tasks from Table 3, and present a comparison our method, COMs, and Standard method in
Table 7.
The “Standard” method represents a surrogate model without utilizing any infeasible points.
On average, P RIME outperforms COMs by 1.17 × (up to 1.24 × in M6).
[Table] [Table]
[Table]
Comparison to generative ofﬂine MBO methods.
We provide a comparison between P RIME and prior ofﬂine MBO methods based on generative models (Kumar & Levine, 2019).
We evaluate model inversion networks (MINs) (Kumar & Levine, 2019) on our accelerator data.
However, we were unable unable to train a discrete objectiveconditioned GAN model to 0.5 discriminator accuracy on our ofﬂine dataset, and often observed a collapse of the discriminator.
As a result, we trained a δ − VAE (Razavi et al., 2019), conditioned on the objective function (i.e., latency).
A standard VAE (Kingma &
Welling, 2013) suffered from posterior collapse and thus informed our choice of utilizing a δ − VAE.
The latent space of a trained objective-conditioned δ − VAE corresponding to accelerators on a held-out validation dataset (not used for training) is visualized in the t-SNE plot in the ﬁgure on the right.
This is a 2D t-SNE of the accelerators conﬁgurations (§Table 1).
The color of a point denotes the latency value of the corresponding accelerator conﬁguration, partitioned into three bins.
Observe that while we would expect these objective conditioned models to disentangle accelerators with different objective values in the latent space, the models we trained did not exhibit such a structure, which will hamper optimization.
While our method P RIME could also beneﬁt from a generative optimizer (i.e., by using a generative optimizer in place of Opt( · ) with a conservative surrogate), we leave it for future work to design effective generative optimizers on the accelerator manifold.
Comparison to P3BO.
We perform a comparison against P3BO method, the state-of-the-arts online method in biology (Angermueller et al., 2020b).
On average, P RIME outperforms the P3BO method by 2.5 × (up to 8.7 × in U-Net).
In addition, we present the comparison between the total simulation runtime of the P3BO and Evolutionary methods in Figure 7.
Note that, not only the total simulation time of P3BO is around 3.1 × higher than the Evolutionary method
[Table]
but also the latency of ﬁnal optimized accelerator is around 18% for MobileNetEdge.
On the other hand, the total simulation time of P RIME for the task of accelerator design for MobileNetEdge is lower than both methods (only 7% of the Evolutionary method as shown in Figure 5).
[Table]
A.
[Table]
FOR A CCELERATOR D ESIGN
Extending our results in Table 4, we present another variant of optimizing accelerators jointly for multiple applications.
In that scenario, the learned surrogate model is reused to architect an accelerator for a subset of applications used for training.
We train a contextual conservative surrogate on the variants of MobileNet (Table 2) as discussed in Section 4, but generated optimized designs by only optimizing the average surrogate on only two variants of MobileNet (MobileNetEdge and MobileNetV2).
This tests the ability of our approach P RIME to provide a general contextual conservative surrogates, that can be trained only once and optimized multiple times with respect to different subsets of applications.
Observe in Table 9, P RIME architects high-performing accelerator conﬁgurations (better than the best point in the dataset by 3.29 × – last column) while outperforming the online optimization methods by 7%.
A. L EARNED S URROGATE M ODEL R EUSE UNDER D IFFERENT D ESIGN C ONSTRAINT
We also test the robustness of our approach in handling variable constraints at test-time such as different chip area budget.
We evaluate the learned conservative surrogate trained via P RIME under a reduced value of the area threshold, α , in Equation 1.
To do so, we utilize a variant of rejection sampling – we take the learned model trained for a default area constraint α = 29 mm 2 and then reject
[Table]
[Table] [Table]
×
[Table]
all optimized accelerator conﬁgurations which do not satisfy a reduces area constraint: Area( x ) ≤ α
=
mm 2 .
Table 10 summarizes the results for this scenario for the MobileNetEdge (Gupta & Akin, 2020) application under the new area constraint ( α = 18 mm 2 ).
A method that produces diverse designs which are both high-performing and are spread across diverse values of the area constraint are expected to perform better.
As shown in Table 10, P RIME provides better accelerator than the best online optimization from scratch with the new constraint value by 4.4%, even when
P RIME does not train its conservative surrogate with this unseen test-time design constraint.
Note that, when the design constraint changes, online methods generally need to restart the optimization process from scratch and undergo costly queries to the simulator.
This would impose additional overhead in terms of total simulation time (§ Figure 5 and Figure 6).
However, the results in Table 10 shows that our learned surrogate model can be reused under different test-time design constraint eliminating additional queries to the simulator.
A. A NALYSIS OF D ESIGNED A CCELERATORS
In this section, we overview the best accelerator conﬁgurations that P RIME and the Evolutionary method identiﬁed for multi-task accelerator design (See Table 4), when the number of target applications are nine and the area constraint is set to 100 mm 2 .
The average latencies of the best accelerators found by P RIME and the Evolutionary method across nine target applications are 383.57 ms and 518.58 ms , respectively.
In this setting, our method outperforms the best online method by 1.35 × .
[Table]
Table 11 shows per application latencies for the accelerator suggested by our method and the Evolutionary method.
The last column shows the latency improvement of P RIME over the Evolutionary method.
Interestingly, while the latency of the accelerator found by our method for MobileNetEdge,
MobileNetV2, MobileNetV3, M4, t-RNN Dec, and t-RNN Enc are better, the accelerator identiﬁed by the online method yields lower latency in M5, M6, and U-Net.
To better understand the trade-off in design of each accelerators designed by our method and the
Evolutionary method, we present all the accelerator parameters (See Table 1) in Table 12.
The accelerator parameters that are different between each of the designed accelerator are shaded in gray (e.g. # of PEs-Y, # of Cores, # of Compute Lanes, PE Memory, Instruction Memory, and
Activation Memory).
Last row of Table 12 depicts the overall chip area usage in mm 2 .
P RIME not only outperforms the Evolutionary algorithm in reducing the average latency across the set of target applications, but also reduces the overall chip area usage by 1.97 × .
Studying the identiﬁed accelerator conﬁguration, we observe that P RIME trade-offs compute ( 64 cores vs. 128 cores ) for larger PE memory size ( 2,097,152 vs. 1,048,576 ).
These results show that P RIME favors PE memory size to accommodate for the larger memory requirements in t-RNN Dec and t-RNN Enc (See Table 2 Model
Parameters) where large gains lie.
Favoring larger on-chip memory comes at the expense of lower compute power in the accelerator.
This reduction in the accelerator’s compute power leads to higher latency for the models with large number of compute operations, namely M5, M6, and U-Net (See last row in Table 2).
M4 is an interesting case where both compute power and on-chip memory is favored by the model (6.23 MB model parameters and 3,471,920,128 number of compute operations).
This is the reason that the latency of this model on both accelerators, designed by our method and the
Evolutionary method, are comparable (400.88 ms in P RIME vs. 406.28 ms in the online method).
A. C OMPARISON WITH O NLINE M ETHODS IN Z ERO -S HOT S ETTING
We evaluated the Evolutionary (online) method under two protocols for the last two rows of Table 5: ﬁrst, we picked the best designs (top-performing 256 designs similar to the P RIME setting in Section 4) found by the evolutionary algorithm on the training set of applications and evaluated them on the target applications and second, we let the evolutionary algorithm continue simulator-driven optimization on the target applications.
The latter is unfair, in that the online approach is allowed access to querying more designs in the simulator.
Nevertheless, we found that in either conﬁguration, the evolutionary approach performed worse than P RIME which does not access training data from the target application domain.
For the area constraint 29 mm 2 and 100 mm 2 , the Evolutionary algorithm reduces the latency from 1127.64 → 820.11 and 861.69 → 552.64, respectively, although still worse than P RIME .
In the second experiment in which we unfairly allow the evolutionary algorithm to continue optimizing on the target application, the Evolutionary algorithm suggests worse designs than Table 5 (e.g. 29 mm 2 : 1127.64 → 1181.66 and 100 mm 2 : 861.69 → 861.66).
[Table] [Table]
[Table]
A. P RIME A BLATION S TUDY
Here we ablate over variants of our method: (1) Opt was not used for negative sampling (“P RIMEOpt ” in Table 13) (2) infeasible points were not used (“P RIME -Infeasible” in Table 13).
As shown in Table 13, the variants of our method generally performs worse compared to the case when both negative sampling and infeasible data points are utilized in training the surrogate model.
A. C OMPARISON WITH H UMAN -E NGINEERED A CCELERATORS
In this section, we compare the optimized accelerator design found by P RIME that is targeted towards single applications to the manually optimized EdgeTPU design (Yazdanbakhsh et al., 2021b; Gupta & Akin, 2020).
EdgeTPU accelerators are primarily optimized towards running applications in image classiﬁcation, particularly, MobileNetV2, MobileNetV3 and MobileNetEdge.
The goal of this comparison is to present the potential beneﬁt of P RIME for a dedicated application when compared to human designs.
For this comparison, we utilize an area constraint of 27 mm 2 and a DRAM bandwidth of 25 Gbps, to match the speciﬁcations of the EdgeTPU accelerator.
Table 14 shows the summary of results in two sections, namely “Latency” and “Chip Area”.
The ﬁrst and second under each section show the results for P RIME and EdgeTPU, respectively.
The ﬁnal column for each section shows the improvement of the design suggested by P RIME over EdgeTPU.
On average (as shown in the last row), P RIME ﬁnds accelerator designs that are 2.69 × (up to 11.84 × in t-RNN Enc) better than EdgeTPU in terms of latency.
Our method achieves this improvement while, on average, reducing the chip area usage by 1.50 × (up to 2.28 × in MobileNetV3).
Even on the MobileNet image-classiﬁcation domains, we attain an average improvement of 1.85 × .
[Table]
A. Z ERO -S HOT R ESULTS ON A LL A PPLICATIONS
In this section, we present the results of zero-shot optimization from Table 5 on all the nine applications we study in the paper (i
[Table]
e., test applications = all nine models: MobileNet (Edge, V2,
V3), M6, M5, M4, t-RNN (Enc and Dec)
[Table]
and U-Net).
We investigate this for two sets of training applications and two different area budgets.
As shown in Table 15, we ﬁnd that P RIME does perform well compared to the online evolutionary method
[Table]
A. D IFFERENT T RAIN AND VALIDATION S PLITS
In the main paper, we used the worst 80% of the feasible points in the training dataset for training and used the remaining 20% of the points for cross-validation using our strategy based on Kendall’s rank correlation.
In this section, we explore some alternative training-validation split strategies to see how they impact the results.
To do so, we consider two alternative strategies: (1) training on 95% of the worst designs, validation on top 5% of the designs, and (2) training on the top 80% of the designs and validation on the worst 20% of the designs.
We apply these strategies to MobileNetEdge, M and t-RNN Enc models from Table 3, and present a comparative evaluation in Table 16 below.
Results.
As shown in Table 16, we ﬁnd that cross-validating using the best 5% of the points in the dataset led to a reduced latency (298.50 → 273.30) on MobileNetEdge, and retained the same performance on M6.
However, it increased the latency on t-RNN Enc (130.67 → 137.45).
This indicates at the possibility that while top 5% of the datapoints can provide a better signal for crossvalidation in some cases, this might also hurt performance if the size of the 5% dataset becomes extremely small (as in the case of t-RNN Enc, the total dataset size is much smaller than either
MobileNetEdge or M6).
The strategy of cross-validating using the worst 20% of the points hurt performance on M6 and t-RNN
Enc, which is perhaps as expected, since the worst 20% of the points may not be indicative of the best points found during optimization.
However, while it improves performance on the MobileNetEdge application compared to the split used in the main paper but it is still worse than using the top 5% of the points for validation.
[Table]
A. U NDERFITTING AND O VERFITTING A NALYSIS OF P RIME W . R . T . D ATASET S IZE
In this section, we study the overﬁtting and underﬁtting characteristics of P RIME when more or less datapoints are used.
To understand these characteristics, we ran some preliminary experiments with
P RIME and a non-conservative surrogate model which does not penalize values on infeasible points or performs negative sampling on datasets of three sizes (2k, 4k and 8k datapoints) on MobileNetV2.
We plot the training approximation error (= relative L1 error in the prediction of latency) of the learned surrogate for both P RIME and the standard surrogate, and ﬁnd that these values increases when the dataset size is 8k compared to when the dataset size is 2k.
An example of this trend is shown in Figure 8.
Additionally, as shown in the plot, the value of the validation rank correlation metric ﬁrst increases and then begins to decrease for limited data, resembling a classic overﬁtting trend for both P RIME and the standard surrogate.
On the other hand, the value of this validation metric generally continuously increases and still exhibits a non-decreasing trend when the dataset size is 8k, indicating that the model has not started overﬁtting yet, and is underﬁtting.
We suspect that this happens because we utilize models of the same capacity, but the size of the dataset changes, and likely warrants higher capacity models.
This can lead to potentially worse performance in certain cases.
We hypothesize that a simple way to address this limitation is to increase network capacity (e.g., by adding more transformer layers) or by changing learning rate schedules.
B D ETAILS OF P RIME
In this section, we provide training details of our method P RIME including hyperparameters and compute requirements and details of different tasks.
B. H YPERPARAMETER AND T RAINING D ETAILS
Algorithm 1 outlines our overall system for accelerator design.
P RIME parameterizes the function f θ ( x ) as a deep neural network as shown in Figure 4.
The architecture of f θ ( x ) ﬁrst embeds the discrete-valued accelerator conﬁguration x into a continuous-valued 640-dimensional embedding via two layers of a self-attention transformer (Vaswani et al., 2017).
Rather than directly converting this 640-dimensional embedding into a scalar output via a simple feed-forward network, which we found a bit unstable to train with Equation 3, possibly due to the presence of competing objectives for a comparison), we pass the 640-dimensional embedding into M different networks that map it to M different scalar predictions ( f iθ ( x ))
Mi
=1 .
Finally, akin to attention (Vaswani et al., 2017) and mixture of experts (Shazeer et al., 2017), we train an additional head to predict weights ( w i )
Mi
=1 ≥ 0 of a linear combination of the predictions at different heads that would be equal to the ﬁnal prediction: f θ ( x ) =
[Equation]
Ki =1 w i f iθ ( x ) .
Such an architecture allows the model to use different predictions f iθ ( x ) , depending upon the input, which allows for more stable training.
To train f θ ( x ) , we utilize the
Adam (Kingma & Ba, 2015) optimizer.
Equation 3 utilizes a procedure Opt that maximizes the learned function approximately.
We utilize the same technique as Section 4 (“optimizing the learned surrogate”) to obtain these negative samples.
We periodically refresh Opt , once in every 20K gradient steps on f θ ( x ) over training.
1: Initialize a neural model f θ 0 ( x ) and a set M = 23 of negative particles to be updated by the ﬁreﬂy optimizer { x − 1 (0) , · · · , x − i (0) , x − M (0) } to random conﬁgurations from the design space.
2: for iteration i = 0 , 1 , 2 , 3 , . . .
until convergence do 3: for ﬁreﬂy update step t = 0 , 1 , . .
,(cid:46) Inner loop do 4:
Update the M ﬁreﬂies according to the ﬁreﬂy update rule in Equation 5, 5: towards maximizing f θ i ( x ) according to: (cid:46) Negative mining 6: x i ( ti + 1) =
x i ( ti ) + β ( x i ( ti ) − x j ( ti ))
+ η(cid:15) ti 7: end for 8:
Find the best ﬁreﬂy found in these steps to be used as the negative sample: 9: x − i =
arg min { f θ i ( x − 1 ( ti )) , · · · , f θ i ( x − M ( ti )) }
(cid:46) Find negative sample 10:
Run one gradient step on θ i using Equation 3 with x − i as the negative sample 11: if i % p == 0 , (p = 20000), then: (cid:46) Periodically reinitialize the optimizer 12:
Reinitialize ﬁreﬂy particles { x − 1 (0) , · · · , x − i (0) , x − M (0) } to random designs.
13: end for 14: Return the ﬁnal model f θ ∗ ( x )
The hyperparameters for training the conservative surrogate in Equations 3 and its contextual version are as follows:
• Architecture of f θ ( x ) .
As indicated in Figure 4, our architecture takes in list of categorical (one-hot) values of different accelerator parameters (listed in Table 1), converts each parameter into 64 -dimensional embedding, thus obtaining a 10 × 64 sized matrix for each accelerator, and then runs two layers of self-attention (Vaswani et al., 2017) on it.
The resulting 10 × 64 output is ﬂattened to a vector in R 640 and fed into M = 7 different prediction networks that give rise to f 1 θ ( x ) , · · · , f Mθ ( x ) , and an additional attention 2-layer feed-forward network (layer sizes =
[256 , 256] ) that determines weights w 1 , · · · , w M , such that w i ≥ 0 and (cid:80)
[Equation]
i = 1 .
Finally the output is simply
[Equation]
• Optimizer/learning rate for training f θ
e −
β β
[Equation]
• Validation set split.
Top 20% high scoring points in the training dataset are used to provide a validation set for deciding coefﬁcients α , β and the checkpoint to evaluate.
• Ranges of α , β .
We trained several f θ ( x ) models with α ∈ [0 . 0 , 0 . 01 , 0 . 1 , 0 . 5 , 1 . 0 , 5 . 0] and β ∈ [0 . 0 , 0 . 01 , 5 . 0 , 0 .
Then we selected the best values of α and β based on the highest Kendall’s ranking correlation on the validation set.
Kendall’s ranking correlation between two sets of objective values: S = { y 1 , y 2 , · · · , y N } corresponding to ground truth latency values on the validation set and S (cid:48)
= { y
[Equation]
1 , y (cid:48) 2 , · · · , y (cid:48) N } corresponding to the predicted latency values on the validation set is given by τ equal to:
[Equation]
• Clipping f θ ( x ) during training .
Equation 3 increases the value of the learned function f θ ( x ) at x = x 0 ∈ D infeasible and x − ∼ Opt( f θ ) .
We found that with the small dataset, these linear objectives can run into numerical instability, and produce + ∞ predictions.
To avoid this, we clip the predicted function value both above and below by ± 10000 .
0 , where the valid range of ground-truth values is O (1000) .
• Negative sampling with Opt( · ) .
As discussed in Section 4, we utilize the ﬁreﬂy optimizer for both the negative sampling step and the ﬁnal optimization of the learned conservative surrogate.
When used during negative sampling, we refresh (i.e., reinitialize) the ﬁreﬂy parameters after every p = 20000 gradient steps of training the conservative surrogate, and run t = 5 steps of ﬁreﬂy optimization per gradient step taken on the conservative surrogate. • Details of ﬁreﬂy:
The initial population of ﬁreﬂies depends on the number of accelerator conﬁgurations ( C ) following the formula 10 + (cid:98) ( C 1 . 2 + C ) × 0 .
5 (cid:101) .
In our setting with ten accelerator parameters (See Table 1), the initial population of ﬁreﬂies is 23.
We use the same hyperparameters: γ = 1 . 0 , β 0 = 1 . 0 , for the optimizer in all the experiments and
[Table]
never modify it.
The update to a particular optimization particle (i.e., a ﬁreﬂy) x i , at the t -th step of optimization is given by:
[Equation]
where x j ( t ) , j (cid:54)
=
i is a different ﬁreﬂy that achieves a better objective value compared to x i and the function β is given by: β
[Equation]
• Training set details: The training dataset sizes for the studied applications are shown in Table 17.
To recap, to generate the dataset, we ﬁrst randomly sampled accelerators from the deign space, and evaluated them for the target application, and constituted the training set from the worst-performing feasible accelerators for the given application.
Since different applications admit different feasibility criteria (differences in compilation, hardware realization, and etc.), the dataset sizes for each application are different, as the number of feasible points is different.
Note however that as mentioned in the main text, these datasets all contain ≤ 8000 feasible points.
Discussion on
data quality:
In the cases of t
RNN Dec t-RNN Enc
, and
U-Net we ﬁnd
that the
number
of feasible points is much smaller compared to
other
applications, and we suspect
this is because
our
random sampling procedure does not ﬁnd enough feasible points.
This is a limitation
of our
data collection strategy and
we
intentionally chose this naïve
strategy
to keep data collection simple.
Other techniques for improving data collection and making sure that the data does not consist of only infeasible points includes strategies
such as utilizing logged data from
past runs of online evolutionary methods, mixed with some data collected via random sampling to improve coverage of the design space.
B.1. D ETAILS OF F IREFLY U SED FOR O UR O NLINE E VOLUTIONARY M ETHOD
In this section, we discuss some details for ﬁreﬂy optimization used in the online evolutionary method.
Stopping criterion: We stopped the ﬁreﬂy optimization when the latency of the best design found did not improve over the previous 1000 iterations, but we also made sure to run ﬁreﬂy optimization for at least 8000 iterations, to make sure that both the online and ofﬂine methods match in terms of the data budget.
We also provide the convergence curves for ﬁreﬂy optimization on various single-application problems from Table 3 in Figure 9.
What happens if we run ﬁreﬂy optimization for longer?
We also experimented with running the evolutionary methods for longer (i.e., 32k simulator accesses compared to 8k), to check if this improves the performance of the evolutionary approach.
As shown in Table 18, we ﬁnd that while this procedure does improve performance in some cases, the performance does not improve much beyond 8k steps.
This indicates that there is a possibility that online methods can perform better than
P RIME if they are run for many more optimization iterations against the simulator, but they may not be as data-efﬁcient as P RIME .
Hyperparameter tuning for ﬁreﬂy: Since the online optimization algorithms we run have access to querying the simulator over the course of training, we can simply utilize the value of the latest proposed design as a way to perform early stopping and hyperparameter tuning.
A naïve way to perform hyperparameter tuning for such evolutionary methods is to run the algorithm for multiple rounds with multiple hyperparameters, however this is compute and time intensive.
Therefore, we
[Table]
adopted a dynamic hyperparameter tuning strategy.
Our implementation of the ﬁreﬂy optimizer tunes hyperparameters by scoring a set of hyperparameters based on its best performance over a sliding window of T data points.
This allows us to adapt to the best hyperparameters on the ﬂy, within the course of optimization, effectively balancing the number of runs that need to be run in the simulator and hyperparameter tuning.
This dynamic hyperparameter tuning strategy requires some initial coverage of the hyperparameter space before hyperparameter tuning begins, and therefore, this tuning begins only after 750 datapoints.
After this initial phase, every T = 50 iterations, the parameters γ and β 0 are updated via an evolutionary scoring strategy towards their best value.
Discussion of t-RNN Enc and t-RNN Dec .
Finally, we discuss the results of the evolutionary approach on the t-RNN Enc and t-RNN Dec tasks, for which the convergence plots are shown in
Figures 9h and 9i.
Observe that the best solution found by this optimization procedure converges quite quickly in this case (with about 1000 iterations) and the evolutionary method, despite the dynamic hyperparameter tuning is unable to ﬁnd a better solution.
We hypothesize that this is because the performance of a local optimization method may suffer heavily due to the poor landscape of the objective function, and it may get stuck if it continuously observes only infeasible points over the course of optimization.
B.1. E XACT H YPERPARAMETERS F OUND B Y O UR C ROSS -V ALIDATION S TRATEGY
In this section, we present the exact hyperparameters found by our cross-validation strategy discussed in Section 4.
To recap, our ofﬂine cross-validation strategy ﬁnds the early stopping checkpoint and selects the values of α and β in Equation 3 that attain the highest rank correlation on a held-out validation set consisting of top 20% of the dataset feasible samples.
The values of α , β and checkpoint selected for the experiments in Table 3, Table 4, Table 5 and 6 are shown in Table 19.
B. D ETAILS OF A RCHITECTING A CCELERATORS FOR M ULTIPLE A PPLICATIONS S IMULTANEOUSLY
Now we will provide details of the tasks from Table 4 where the goal is to architect an accelerator which is jointly optimized for multiple application models.
For such tasks, we augment data-points for each model with the context vector c k from Table 2 that summarizes certain parameters for each application.
For entries in this context vector that have extremely high magnitudes (e.g., model parameters and number of compute operations), we normalize the values by the sum of values across the applications considered to only encode the relative scale, and not the absolute value which is not required.
To better visualize the number of feasible accelerators for joint optimization, Figureshow the tSNE plot (raw architecture conﬁgurations are used as input) of high-performing accelerator conﬁgurations.
The blue-colored dots are the jointly feasible accelerators in the combined dataset, and note that these data points are no more than 20-30 in total.
The highlighted red star presents the best design suggested by P RIME with average latency of 334.70 (Table 4).
This indicates that this contextual, multi-application problem poses a challenge for data-driven methods: these methods need to produce optimized designs even though very few accelerators are jointly feasible in the combined dataset.
Despite this limitation, P RIME successfully ﬁnds more efﬁcient accelerator conﬁgurations that attain low latency values on each of the applications jointly, as shown in Table 4.
B. D ATASET R ELEASE
To reproduce the results presented in this paper and to further facilitate research in model-based optimization for accelerator design, we release a much larger dataset across various accelerator conﬁgurations.
We collected these data points using a detailed industrial simulator Yazdanbakhsh et al.
(2021b).
Along with the dataset, we release a script to parse the dataset.
The dataset can be found at the
[Figure]
, where we observe that online optimization plateaus the fastest for these RNN applications.
following anonymized link (clickable): P RIME Dataset.
[The link is: https://drive.google.
com/drive/folders/1DUfA_GfZPPu7esS7-c34G8N8uCTWg7N8?usp=sharing ]
B. D ATASET S ENSITIVITY TO A CCELERATOR P ARAMETERS
We visualize the sensitivity of the objective function (e.g. latency) with respect to the changes in certain accelerator parameters, such as memory size (Table 1), in Figure 12b, illustrating this sensitivity.
As shown in the Figure, the latency objective that we seek to optimize can exhibit high sensitivity to small variations in the architecture parameters, making the optimization landscape particularly ill-behaved.
Thus, a small change in one of the discrete parameters, can induce a large change in the optimization objective.
This characteristic of the dataset further makes the optimization task challenging.
C O VERVIEW OF A CCELERATORS AND S EARCH S PACE
This section brieﬂy discuss the additional accelerators (similar to (Kao et al., 2020)) that we evaluate in this work, namely NVDLA (Nvidia, 2021b) and ShiDianNao (Du et al., 2015), and their corresponding search spaces.
NVDLA:
Nvidia
Deep Learning Accelerator NVDLA (Nvidia, 2021a) is an open architecture inference accelerator designed and maintained by Nvidia.
In compared to other inference accelerators,
NVDLA is a weight stationary accelerator.
That is, it retains the model parameters on each processing
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
elements and parallelizes the computations across input and output channels.
NVDLA-style dataﬂow accelerators generally yield better performance for the computations of layers at the later processing stages.
This is because these layers generally have larger model parameters that could beneﬁt from less data movement associated to the model parameters.
ShiDianNao: Vision Accelerator
Figure 11 shows the high-level schematic of ShiDianNao accelerator (Du et al., 2015).
ShiDianNao-style dataﬂow accelerator is an output-stationary accelerator.
That
[Figure] [Figure]
is, it keeps the partial results inside each PE and instead move the model parameters and input channel data.
As such, in compared to NVDLA-style accelerators, ShiDianNao provides better performance for the computations of the layers with large output channels (generally ﬁrst few layers of a model).
Search space of dataﬂow accelerators.
We follow a similar methodology as (Kao et al., 2020) to evaluate additional hardware accelerators, discussed in the previous paragraphs.
We use MAESTRO (Kwon et al., 2019), an analytical cost model, that supports the performance modeling of various dataﬂow accelerators.
In this joint accelerator design and dataﬂow optimization problem, the total number of parameters to be optimized is up to 106—the tuple of (# of PEs, Buffers) per per model layer—with each parameter taking one of 12 discrete values.
This makes the hardware search space consist of ≈ 2.5 × 10 114 accelerator conﬁgurations.
We also note that while the method proposed in (Kao et al., 2020) treats the accelerator design problem as a sequential decision making problem, and uses reinforcement learning techniques, P RIME simply designs the whole accelerator in a single step, treating it as a model-based optimization problem. D S UBSET OF A PPLICATIONS FOR G OOD Z ERO -S HOT P ERFORMANCE
In this section, we present the results of an ablation study with the goal to identify a subset of applications such that training on data from only these applications yields good zero-shot performance across all the nine applications studied in this work.
Since we cannot train P RIME for every subset of applications because the space of subsets of all applications is exponentially large, we utilized some heuristics in devising the subset of applications we would train on, with the goal to make interesting observations that allow us to devise rough guidelines for performing application selection.
Our heuristic for devising subsets of applications: Building on the intuition that applications with very different compute to memory ratios (shown in Table 21) may require different accelerator designs – for example, if our goal is to run a compute-intensive application, we likely need an accelerator design with more compute units – we study two subsets of training applications: (1) MobileNetV2, MobileNetV3, M6, M5, and (2) MobileNetV2, MobilenetV3, M5, t-RNN Enc. Note that, these two combinations only differ in whether some RNN application was used in training or not. As shown in Table
, the t-RNN applications admit a very different compute to memory ratio, for instance, while
[Figure] [Figure]
this ratio is 5 .
68 e − 4 for t-RNN Enc and t-RNN Dec, it is much different ∼ 0 .
01 − 0 .
2 for other models MobileNetEdge, MobileNetV2, MobileNetV3, M and M6.
This means that likely t-RNN
Enc and Dec will require different kinds of accelerators for good performance compared to the other applications.
Results: We present the performance of zero-shot evaluating the designed accelerator obtained by training on combinations (1) and (2) , and also the accelerator obtained by training on (3) seven applications from Table 5 in Table 20 as reference.
We make some key takeaways from the results:
• The performance of both conﬁguration (1) and training with seven applications ( (3) , last row of Table 20) are similar.
In case (2) , when the training applications consist of four applications in which one application is t-RNN Enc, with drastically different compute to memory ratio (Table 21), the performance on an average across all applications becomes slightly worse (compare the performance in (2) vs (3) ).
[Figure]
i c t e d L a t e n c y
PRIME does not overestimate
[Figure]
[Figure] [Table] [Table] Conclusion and guidance on selecting good applications: The above results indicate that only a few applications (e
[Table]
g., four applications in case (1) ) can be enough for good performance on all nine applications.
While this set may not be not minimal, it is certainly a much smaller set compared to the nine applications considered. Adding an RNN application in case (2) increases latency in compared to case (1) , because t-RNN Enc likely admits a very different optimal accelerator compared to the other applications due to a very different compute/memory ratio, which in turn skews the generalization of the surrogate learned by P RIME when trained only on this limited set of four applications. However, when seven applications are provided in case (3) , even when the set of training applications includes t-RNN, its contribution on the P RIME surrogate is reduced since many other compute intensive applications are also provided in the training set and the resulting accelerator performs well. Practitioner guidance: The primary practitioner guidance we can conclude here is that the models used for training must be representative of the overall distribution of the target models that we want to zero-shot generalize to .
Since a number of our applications are compute intensive, we were able to simply utilize set (1) to attain good performance on all the applications. On the other hand, in case (2) , when the t-RNN Enc application was over-represented – while seven of nine applications we considered were primarily compute intensive, one out of four applications we used for training in case (2) were memory intensive – this hurt performance on the overall set. Therefore, we believe that ensuring that the training subset of applications is adequately aligned with the overall set of applications in terms of compute/memory ratio statistic is imperative for favorable zero-shot performance. For a practitioner deciding between zero-shot generalization and additional data collection , it may make sense to test if the target application admits a similar value of the compute to memory ratio as an already existing application.
If it does, then the practitioner might be able to utilize the zero-shot generalization, as is indicated with the good performance of case (1) , whereas if the compute/memory ratio is heavily different from any seen application, zero-shot generalization to the target application may be worse. Finally, making sure that the training applications adequately reﬂect the compute/memory ratio statistic for the overall target set is important.