H YBRID C
LOUD -E DGE N ETWORKS FOR E FFICIENT I NFERENCE
Anonymous authors
Paper under double-blind review
A BSTRACT
Although deep neural networks (DNNs) achieve state-of-the-art accuracy on largescale and ﬁne-grained prediction tasks, they are high capacity models and often cannot be deployed on edge devices.
As such, two distinct paradigms have emerged in parallel: 1) edge device inference for low-level tasks, 2) cloud-based inference for large-scale tasks.
We propose a novel hybrid option, which marries these extremes and seeks to bring the latency and computational cost beneﬁts of edge device inference to tasks currently deployed in the cloud.
Our proposed method is an end-to-end approach, and involves architecting and training two networks in tandem.
The ﬁrst network is a low-capacity network that can be deployed on an edge device, whereas the second is a high-capacity network deployed in the cloud.
When the edge device encounters challenging inputs, these inputs are transmitted and processed on the cloud.
Empirically, on the ImageNet classiﬁcation dataset, our proposed method leads to substantial decrease in the number of ﬂoating point operations (FLOPs) used compared to a well-designed high-capacity network, while suffering no excess classiﬁcation loss.
A novel aspect of our method is that, by allowing abstentions on a small fraction of examples ( < 20% ), we can increase accuracy without increasing the edge device memory and FLOPs substantially (up to 7 % higher accuracy and 3 X fewer FLOPs on ImageNet with 80 % coverage), relative to MobileNetV3 architectures.
I NTRODUCTION
Deep Neural Networks (DNNs) achieve state-of-the-art (SOTA) performance on challenging tasks such as image recognition (Tan & Le, 2019; Howard et al., 2019), language modelling (Devlin et al., 2018), and machine translation (Wu et al., 2016b).
High accuracy on such tasks often comes at a high memory and compute cost, making DNN deployment on low resource edge hardware like microcontroller units (MCUs) very challenging (Banbury et al., 2021; Fedorov et al., 2019;
Lin et al., 2020; Fedorov et al., 2020; Gural & Murmann, 2019).
Edge-Device and Cloud ML
As such, two paradigms currently co-exist.
Edge-device inference utilizes lightweight architectures and focuses on low-level tasks such as smart messaging and face recognition (Google LLC, 2021).
In parallel, more complex and nuanced tasks are deployed in the cloud, which is a term we use to refer to over-provisioned hardware platforms like a GPU server (Alemi, 2016).
The fundamental drawback of cloud ML, however, is the increased latency and energy consumption arising from communication, which can be prohibitive for many applications.
Indeed, the meager cost, size, and power requirements of MCUs make them the platform of choice for a large number of applications, so that MCU shipments outnumber GPU shipments by roughly 50 to(Fedorov
et al., 2019; Lin et al., 2020).
While methods like pruning (Molchanov et al., 2017; Han et al., 2015), quantization (Jacob et al., 2018), knowledge distillation (Hinton et al., 2015), and adaptive computation (Bejnordi et al., 2019) could be leveraged to reduce the size of cloud-based models, these strategies fundamentally limit the resulting achievable accuracy due to the reduced model capacities (Fedorov et al., 2019; Lin et al., 2020).
For example, SOTA accuracy on ImageNet is 84.3% (Tan & Le, 2019) with 37B FLOPs; whereas the best deployable model on an STM32F746 MCU under 5 frames per second constraint achieves 51.1% accuracy with 12.8M FLOPs (Lin et al., 2020) .
Hybrid Edge-Cloud Inference.
Motivated by these emerging trends, we propose a best-of-both hybrid solution, which allows for deploying cloud-based AI tasks on edge devices like MCUs, while lowering the average total latency (the sum of communication and computational latency).
[Figure]
Our end-to-end hybrid approach involves architecting and training three distinct networks in concert, individually named ‘base’, ‘global’, and ‘routing’ (Fig. 1).
The base model is compact and designed for devices with low-resource hardware constraints like latency and memory usage.
In contrast, the global model has a large capacity and is deployed in the resource-rich cloud environment.
Finally, the routing model, which is very compact, is used to decide whether a query should be communicated to the global model, or handled entirely by the base model, thus enabling the two to work in tandem to maximise performance while controlling usage of computational or communication resources (see
Fig. 3).
Ideally, the global and base models are ﬁnetuned to be most accurate on the queries routed speciﬁcally to them, while the routing model in turn only issues queries to the global model when they are too hard to be processed by the base model.
Technical Contributions.
Training Methodology.
Learning an efﬁcient hybrid model requires us to solve the challenging problem of discovering regions of ‘easy’ queries, where classiﬁcation can reliably be performed by a simple model.
In addition, training is challenging since a choice of base and global model affects the optimal routing, which cyclically affects the former.
We propose an alternating-optimisation scheme that can be modularly executed, and design an efﬁcient proxy supervision for the router, which together allow for simpliﬁed training.
The result is a ﬂexible scheme that can be used either to train all three models or only a subset.
The routing model is further designed with a ﬂexible assignment criterion that allows efﬁciently trading-off between a range of accuracy and resource usages, thus yielding a variety of operating points with a single training round.
Neural Architecture Search (NAS).
Hybrid design also raises a novel architectural issue - while present DNN architectures are aligned towards a single model that performs end-to-end inference, a hybrid scheme may require a coupled design for the base and global models to best exploit the available resources.
To discover efﬁcient joint designs, we propose a NAS method that utilises an efﬁcient proxy score to quickly determine ﬁtness of a pair of base and global architectures, and performs an evolutionary search to optimise the accuracy at any given combined resource usage.
Our approach is ﬂexible and can also be used to, e.g., adapt a base architecture to a given global model.
Empirical Validation.
Proposed above is a novel end-to-end methodology to design and train resourceefﬁcient hybrid architectures and models that can perform at SOTA accuracy whilst satisfying hardware constraints.
Extensive experimentation on the ImageNet dataset shows that the resulting scheme pareto dominates methods that learn a single efﬁcient architecture, demonstrating 2 − 3 .
5 % accuracy gains at any FLOP count when compared to prior efﬁcient architecture designs such as MobileNetV and OFA ( Fig. 3, 4).
Further, in settings where the global device has much higher compute capacity (hence negligible inference cost), we show that whilst processing 70% of queries at the base, our design can match the accuracy of these designs with up to 4 . 5 × improvement in base FLOPs (Fig. 2).
R ELATED W ORK
Efﬁcient Architectures.
Previous works have designed low complexity DNNs for mobile applications (Iandola et al., 2016;
Gholami et al., 2018; Sandler et al., 2018; Howard et al., 2019) using low-rank decomposition, separable convolutions, and hand-crafted feature blocks. Tan & Le (2019) study the effect of width, depth, channels, and input resolution on DNN memory and FLOP costs.
In parallel, neural architectures searches under constraints such as FLOP (Liu et al., 2017; Zoph & Le, 2016; Dong & Yang, 2019; Elsken et al., 2019), latency (Cai et al., 2020), memory (Fedorov et al., 2019), etc have been carried out.
These efforts are complementary to our hybrid scheme since we can leverage these improved architectures as base models and achieve similar gains in performance.
Low Compute Transformations.
Researchers have explored methods to obtain low-capacity models from SOTA DNNs, including compression and pruning (Han et al., 2016), quantization (Wu et al., 2016a), hashing (Chen et al., 2015), and knowledge distillation(Hinton et al., 2015).
Since these transformations are orthogonal to our proposal and can be leveraged post-hoc, we do not pursue these techniques in order to simplify our exposition.
Adaptive Neural Networks.
Han et al.
(2021) present a comprehensive survey on designing dynamic neural networks that budget more computational resources for harder examples.
These include (a) cascade-based early exit networks (Park et al., 2015; Bolukbasi et al., 2017; Nan & Saligrama, 2017;
Wang et al., 2018), where the constituent networks are independently designed and do not share intermediate features; (b) early exit networks (Teerapittayanon et al., 2017; Dai et al., 2020; Li et al., 2019) where classiﬁers are introduced at intermediate layers; and (c) multi-scale networks with early exits (Huang et al., 2018; Yang et al., 2020), which are allowed to operate at different input resolution, width or depth.
Kang et al. (2017) splits network execution between device and cloud, resulting in higher communication as the features require more storage than input. Further, a high-performing model can neither be stored nor executed on a constrained MCU due to low RAM and Flash.
In the context of our problem, much of the focus in these works is on scaling-up capacity without a proportional increase in inference time.
Although our proposed hybrid approach bears resemblance to these works, we are focused on the opposite scenario, namely, how to overwhelmingly reduce resource usage (FLOPs and communication latency) to allow for deployment on edge devices without degrading accuracy achievable by a large model.
As such, our perspective necessitates posing an end-to-end system-wide hybrid objective and requires systematic integration and optimization of all of the degrees of freedom (architectures, routing & coverage, base and global networks).
In contrast, prior works optimize these aspects in a decoupled and isolated manner.
As a case in point, works focusing on architectures and early exit networks utilize simple entropy thresholding for routing.
As we will show in our results, carefully designed routing schemes, which are jointly optimized along with base and global models, can result in substantial gains over entropy thresholding.
Li et al.
(2021) model a hybrid system with a similar design but missing crucial details, namely, (a) no coverage penalty in the train loss, (b) router is entangled with the base and global network, while we decouple it using the routing oracle, and (c) no evaluations on Imagenet (see Appendix §B.7 for details).
Learning with Abstention.
Many researchers (Liu et al., 2019; Gangrade et al., 2021; Geifman &
El-Yaniv, 2019) have studied the problem of learning with a reject option, where a model can abstain prediction on some examples with the goal of minimizing the number of errors and abstentions.
Although we get an abstaining classiﬁer from the hybrid model by simply ignoring the global model, our main objective is to improve the performance of the hybrid system that includes the global model.
M ETHOD
Let X be a feature space and Y a set of labels.
A hybrid design is composed of three models:
• A base model b : X → Y , that can be deployed on an edge device.
• A global model
g : X → Y , that is deployed in the cloud and typically has high accuracy.
• A routing model r : X → { 0 , 1 } , that is a very low resource model deployed alongside the base model, and routes hard queries to the global model.
We will treat these models as soft classiﬁers, outputting |Y| -dimensional scores { b y } and { g y } , and two scores r 0 ( x ) and r 1 ( x ) for the routing model.
In this paper, r is realized by a 2-layer DNN with input b y ( x ) .
The default hard output for the base is the top entry b ( x ) =
arg max y b y ( x ) , and similarly for g .
By default r assigns x to the global model if r 1 ( x ) > r 0 ( x ) , i.e., r ( x ) = 1 { r 1 ( x ) > r 0 ( x ) } , but this can be relaxed to r ( x ; t ) :
= 1 { r 1 ( x ) > t + r 0 ( x ) } , where the hyper-parameter t allows a routing model to trade-off accuracy and resource usage in order to avoid separately training for each desired level.
The decision produced by the system for an instance x ∈ X is
[Equation]
The coverage of the hybrid system is the fraction of instances that are processed by the base only, i.e.
[Equation]
where P denotes the joint law over ( X, Y ) .
The hybrid accuracy is
[Equation]
Architectures and Costs.
The resources required to evaluate a DNN are mainly a function of its architecture - the number and arrangement of its layers and weights.
We use α to denote a generic architecture, and say that a model f ∈ α if it is realizable by this architecture.
To quantify the resource consumption, let R ( α ) denote the cost per inference for a model with architecture α .
In our design, the base is always executed, with the output fed into the routing model.
Therefore, the hybrid
FLOP count of a hybrid model ( r, b, g ) , such that b ∈ α b and g ∈ α g , is
[Equation]
where R r is a ﬁxed, small quantity required to execute r .
We can model many resources, including the FLOPs required to execute the model, and the latency on edge devices like MCUs (Banbury et al., 2021).
Additionally, for settings where the global model is on the cloud with a compute-rich environment, the resource costs are dominated by communication latency, modeled as R ( α g )
=
τ, R ( α b ) {+= 
 (where+} τ is the mean communication delay).
We use coverage as a proxy metric for communication latency as it measures the data split between the base and global.
In the following, we will focus on FLOP and coverage metrics, although we investigate inference latency in §3.4..
Overall Formulation Let A b and A g be sets of base and global architectures, which may incorporate implementation restrictions, and (cid:37) a target resource usage level.
Our objective is
[Equation]
The outer maximisation over ( α b , α g ) in (3) amounts to an architecture search, while the maximisation over ( r, b, g ) in a ﬁxed architecture corresponds to learning a hybrid model.
The following sections describe our method for solving (3).
Brieﬂy, we propose to decouple the inner and outer optimization problems in (3) for the sake of efﬁciency - hybrid models are trained by an empirical risk minimisation (ERM) strategy, whilst the architecture search is carried out using fast proxies for the accuracy attainable by a given pair of architectures without directly training hybrid models.
2. L EARNING H YBRID M ODELS
This section focuses on training hybrid models for ﬁxed architectures α b , α g , i.e., the inner problem
[Equation]
Since architectures are ﬁxed in (4), the FLOP constraint amounts to a constraint on the hybrid coverage.
As is standard, we will approach (4) via an ERM over a Lagrangian of relaxed losses.
However, a number of design considerations and issues need to be addressed before such an approach is viable, as discussed below.
The overall scheme is summarised in Algorithm 3 in §A
Alternating optimisation .
Problem (4) has a cyclical non-convexity. A given r affects the optimal b and g (since these must adapt to the regions assigned by r ), and vice-versa.
We approach this issue by alternating optimisation.
First, we train global and base models according to standard methods.
Then, we learn a routing network r under a coverage penalty.
The resulting r feeds back into the loss functions of b and g , and these models get retrained.
This cycle may be repeated many times.
Modularity of training .
Resulting scheme allows training r with a ﬁxed ( b, g ) , as it helps learn a cheap routing model with pre-trained cloud and mobile models.
Similarly, by dropping the optimisation over g , we can hybridise too expensive to re-train global models.
Additionally, one can initially train the global and freeze it after a few cycles to save compute.
Finally, we can learn each component to different degrees, e.g., we may take many more gradient steps on g than r or b in any training cycle.
Learning Routers via Proxy Supervision.
Given a ﬁxed pair of base and global model ( b, g ) , the problem (4) reduces to the following, where C (cid:37) is the coverage needed to ensure R ≤ (cid:37) .
[Equation]
While a na¨ıve approach is to relax r and pursue ERM, we instead reformulate the problem.
Observe that (5) demands that r ( X )
=
if b ( X ) =
Y, g ( X ) (cid:54) =
Y , and that r ( X )
= 1 if b ( X ) (cid:54) =
Y, g ( X ) =
Y .
Further, while case b ( X ) =
g
( X ) =
Y is not differentiated, the coverage constraint promotes r ( X ) = 0 for such points.
Thus, the program can be viewed as a supervised learning problem of ﬁtting the routing oracle , i.e.
[Equation]
b [Equation] [Equation] Indeed, o is the ideal routing without the FLOP constraint.
It can be evaluated on training datafor any given ( b, g ) and training dataset D
= { ( x i , y i ) } , we produce the oracle dataset D o ;( b,g ) :
= { ( x i , o ( x i ; b, g )) } .
We use this dataset as supervision for the routing model r , which allows us to utilise the entire gamut of tools of machine learning that are essential for practically learning good binary functions, thus gaining over approaches that directly try to relax the objective of (5) .
Note that the oracle o does not respect the FLOP constraint.
We can satisfy such a constraint by randomly assigning some points from g to b while incurring an error. The oracle is indifferent to such an arrangement.
From a learnability perspective, we would like the points ﬂipped to the base to promote regularity in the dataset. Although, such a goal is ill-speciﬁed (and unlikely to be captured well by simple rules such as ordering points by a soft loss of g ).
Instead, we handle this issue indirectly by imposing a coverage penalty whilst training the routing model and leave it to the optimisation to discover the appropriate regularity by minimising error w.r.t.
o under this penalty.
Focusing competency and loss functions .
To improve accuracy whilst controlling coverage, we focus the capacity of each of the models on the regions relevant to it - so, b is biased towards being more accurate on the region r − 1 ( { 0 } ) , and similarly g on r − 1 ( { 1 } ) .
Similarly, for the routing network r , it is more important to match o ( x ) on the regions where it is 1 , since these regions are not captured accurately by the base and thus need the global capacity.
We realise this inductive bias by introducing model-dependent weights in each loss function to emphasise the appropriate regions.
The Routing Loss consists of two terms, traded off by a hyperparameter
λ r - the ﬁrst penalises deviation of coverage from a given target ( cov ), and the second to promote alignment with o and is biased by the weight W r ( x ) = 1 + 2 o ( x ) to preferentially ﬁt o − 1 ( { 1 } ) .
The symbol (cid:96) denotes a surrogate loss (such as cross entropy), while ( · ) + is the ReLU function, i.e., ( z ) + = max( z, 0) .
All sums below are over a dataset D
= { ( x i , y i ) } of size N .
Empirically, we ﬁnd that λ r = 1 produces effective results.
[Equation]
The Base Loss and the Global Loss are each a weighted variant of the standard classiﬁcation loss, which are biased by the appropriate weights to emphasise the regions assigned to either model by the routing network - W b ( x ) =
2 − r ( x ) and W g ( x ) = 1 + r ( x ) .
[Equation]
and
[Equation]
2. E VOLUTIONARY A RCHITECTURE S EARCH
This section describes the joint architecture search implicit in (3).
We use an evolutionary search(Elsken et al., 2019; Liu et al., 2021) due to its simplicity and effectiveness.
It requires a fast way to evaluate the ﬁtness of a base-global pair ( α b , α g ) , i.e. the value of the program (4) for a given pair of architectures.
While this can be obtained by carrying out hybrid training as previously described it is impractical due to the time cost.
The following describes a cheaper proxy for the same.
To quickly approximate the result of training over r , we use the agreement oracle routing ˆ o .
This assigns all inputs for which the base and global agree to the base, and the remainder to the global model.
We choose the agreement oracle as opposed to the oracle in (6), as it is more realistic in that it does not assume knowledge of the true classiﬁcation label, while still being easy to compute.
What remains is the optimisation over b and g .
We adopt a different solution for these, rooted in the space of architectures itself.
Recently, Cai et al.
(2020) showed that it is possible to design spaces of architectures such that each architecture α is associated with a canonical set of parameters θ α that are near-optimal in an accuracy objective, in the sense that a slight ﬁne-tuning of these models yields a good solution.
This is realised via the use of a ‘super-net,’ the components of which can be individually changed, leading to a combinatorially large set of architectures.
Importantly, training a super-net with such a property has comparable costs to training a standard network, and this cost is further amortized over a large number of resulting architectures.
We use OFA space of Cai et al.
(2020) as architectural search space, and incorporate the oracle proxy to avoid training r , yielding the score A (ˆ o, θ α b , θ α g ) for the ﬁtness of ( α b , α g ) .
The resulting architecture search scheme is summarised in Alg. 4 in §A.
E XPERIMENTS
In this section, we will ﬁrst evaluate hybrid models with off-the-shelf base and global architectures.
Next, we will perform evolutionary search to ﬁnd architectures under various resource constraints and evaluate hybrid models with these newly found architectures.
Finally, we infer that accuracy of the hybrid model consistently increases with increasing differences between base and global FLOPs.
For a proof-of-concept, we limited the cloud model to 600 M FLOPS, which has a SOTA accuracy of ≈ 80% (Cai et al., 2020).
While there are other models such as EfﬁcientNet achieving higher accuracy ( 84 .
3% ), these require substantially more FLOPs ( 37 B).
Although our hybrid training could leverage such models, our limited computing resources made this infeasible.
Highlights.
We list a few salient observations from our empirical results.
• Pareto Dominance and Latency Reduction.
Hybrid models consistently outperform high-capacity models at lower Hybrid FLOPs (Fig. 3) and at a latency reduction of 70% (3 in 10 examples pass to the cloud) our FLOP gains are substantial ( Fig. 2).
Similarly, we improve accuracy at the same FLOP level both in terms of base FLOPs and Hybrid FLOPs.
For instance, 80% is SOTA accuracy for a stand-alone 600M model.
We achieve 79% accuracy in a Hybrid scheme with a base model of 143M (see Table. 1) and 78.5% with 350M hybrid FLOPs.
• Rapid Customization.
Proposed approach allows for optimizing accuracy level to match any intermediate FLOP count with little training, saving computation for training intermediate models.
• SOTA performance on Resource Constrained Base.
Hybrid scheme allows the base model to be deployable on a low-resource hardware.
With 12M base FLOPs and only 3 in 10 examples passed to a larger model we gain about 16% improvement in accuracy (see 3).
• Evolutionary Search yields better Hybrid Models.
Given any single model, we can obtain a better hybrid model using evolutionary architecture search with similar FLOP count.
• Routing outperforms Entropy Metric.
Our routing method exploits base and global models characteristics and dominates entropy-thresholding used in many adaptive neural networks.
Experimental Setup.
For simple exposition, we focus on the classiﬁcation task on the Imagenet (Russakovsky et al., 2015) dataset, consisting of 1 .
28 M train and 50 K validation images.
We follow standard data augmentation (mirroring, resize and crop to shape 224 × 224 ) for training and single crop for testing.
Similar to previous works, we report results on the validation set.
We borrow the pre-trained baselines from their public implementations as described in the appendix sec.
B.2.
For evolutionary architecture search, we utilize the supernet from the OFA search space (Cai et al., 2020).
We describe our hyper-parameter settings in the appendix sec.
B.1.
Depending on the computational budget, one can create hybrid models in three ways (a) ’Hybrid-(r)’ - only training routing while using pre-trained base and global, (b) ’Hybrid-(rb)’ - training routing and base while using pre-trained global, and (c) ’Hybrid-(rbg)’ - training all three components.
We trained the hybrid model to achieve a similar coverage level as the oracle.
Post training, we vary the coverage level by adjusting the threshold hyper-parameter in the routing to generate the performance at different hybrid FLOPs.
3. H YBRID M ODELS USING O FF - THE -S HELF C LASSIFIERS 3.1. N O R ESOURCE C ONSTRAINTS
In this setting, we assume no constraints on the model deployment.
We pick up an architecture family and create a hybrid model using the smallest and largest architecture.
For convenience, we perform this experiment for two known families, namely MobileNetV3(Howard et al., 2019) and OFA(Cai et al., 2020).
From MobileNetV3, we pick the smallest model ( 48 M FLOPs, 67 . 6% accuracy) as base and largest model ( 215 M FLOPs, 75 .
7% accuracy) as global to create Hybrid-MobileNetV3 model.
Similarly, from OFA, we pick the smallest model ( 67 M FLOPs, 70 .
4% accuracy) as base and largest model ( 595 M FLOPs, 80% accuracy) as global to create Hybrid-OFA model.
Figure 3 (a) and (b) plot the FLOPs vs top1 accuracy, and compare the hybrid models with best-known baseline models in the architecture space.
We show hybrid FLOPs for the hybrid models (see Eq. 2).
These experiments provide evidence for the following properties of hybrid models:
• Hybrid models dominate any standalone model between base and global model.
Hybrid models outperform off-the-shelf classiﬁers at every intermediate FLOP count.
For ex., a pre-trained model in MobileNetV3 with 155 M FLOP achieves 73 .
3% accuracy while our hybrid model ’Hybrid-(rbg)’ [Figure] [Table] achieves 75 .
5% accuracy.
Similarly, a model in OFA with 230 M FLOP achieves
4% accuracy while the hybrid model achieves
6% accuracy.
• Hybrid achieves SOTA w.r.t a global model at ≈ 20% lower FLOP count.
Global model in MobileNetV3 achieves 75 .
7% accuracy at 215 M FLOPs, while hybrid model achieves same accuracy at 177 M FLOPs.
Similarly, global model in OFA achieves 79 .
9% accuracy at 595 M FLOPs, while hybrid model achieves same accuracy at 483 M FLOPs.
• Training a Hybrid model for intermediate FLOPs is inexpensive.
To achieve a single model at any FLOPs, we ﬁnd an architecture with the FLOP constraint and train it to achieve non-trivial performance.
Hybrid model with smallest and largest model allows us to trade-off FLOPs for accuracy and save compute for training models for an intermediate FLOP constraint.
• Outperform the entropy thresholding baseline used in dynamic neural networks.
• End-to-end training of all components lead to increasing gains.
Hybrid models improve in performance as additional components are trained in the alternative minimization (Algorithm 3), i.e. Hybrid-rbg is the best performing model followed by Hybrid-rb and Hybrid-r.
3.1. R ESOURCE C ONSTRAINED B ASE
In this setting, our base model operates at a ﬁxed computational budget on an edge device and one cannot deploy the global model on this device.
For simplicity, we assume the latency between the base and global models to be negligible.
We create hybrid models using base models from the
MobileNetV3 family.
Since in this setting, the goal is to save compute / battery on the device and achieve near SOTA performance, we use a high performing OFA model as the global model and operate the base model at a ﬁxed coverage level.
We report two metrics: (a) base accuracy achieved by the base by predicting only on the coverage portion of the routing, (b) hybrid accuracy - accuracy achieved by the hybrid model, where the examples abstained by the base are sent to the global model.
Table 1 shows the base and hybrid accuracy at three coverage levels, 90% , 80% and 70% .
Hybrid models operating at a ﬁxed coverage level provide the following beneﬁts:
• Hybrid models achieve near SoTA accuracy with ≈ 3 x less FLOPs.
Using base with 48 M FLOPs and 67 .
61% accuracy, the hybrid model achieves 71 .
59% accuracy at 90% coverage, improving to 76 .
77% accuracy at 70% coverage.
To achieve 76 .
77% accuracy with a single model would require > 400 M FLOPs, which is too large to be deployed on an MCU. Similarly, using a base
[Table]
with 215 M FLOPs and
% accuracy, the
hybrid model achieves [Table]
accuracy with coverage .
Global with ≈ 600 M FLOPs has 80% accuracy
• Abstaining base model achieves signiﬁcantly better performance than the base at full coverage.
Hybrid scheme allows the system to operate without a global model.
In this case.
the result is an abstaining classiﬁer operating on the device, i [Table] e. it rejects few input examples and provides predictions on the rest.
For ex., a base model with 48 M FLOP achieves an accuracy of 83 .
41% when it only covers 70% examples.
Similarly, a base model of 215 M achieves 90 .
11% accuracy with 70% coverage.
In both cases, there is a gain of at least 15 points in the accuracy.
3. H YBRID M ODELS WITH E VOLUTIONARY A RCHITECTURE S EARCH
So far we have been generating hybrid models using off-the-shelf classiﬁers that are not tuned to maximize hybrid performance.
In this experiment, we search for base and global pairs using evolutionary search in the OFA space.
We constrain the search to operate at ﬁxed hybrid FLOPs.
After ﬁnding base and global pairs from the evolutionary search
[Table]
we create hybrid models with the newly found architectures.
We perform this experiment for three hybrid FLOP constraints: 150 M, 250 M, and 350 M. We draw baseline architecture samples from the OFA space using their optimized architecture search.
For fair comparison, we do not ﬁne tune models found by the architecture searches for both OFA and hybrid models.
Figure 4 plots the operating curves for the hybrid models found using different FLOP constraints.
Table 2 shows the hybrid model performance at the constraint points used in the search.
Evolutionary search based hybrid models provide the following beneﬁts
• Hybrid Models with evolutionary search yields higher accuracy at any target FLOP.
As illustrated in Table 2, evolutionary search ﬁnds hybrid models that outperform the models found from the optimized OFA search.
For ex., at target 350 M FLOP, OFA ﬁnds an architecture with 74 .
93% accuracy while evolutionary search ﬁnds a hybrid model that achieves 77 .
91% accuracy.
• Hybrid models pareto dominate single models at any target FLOP.
In ﬁgure 4(a), hybrid model for 150 M FLOP outperforms the OFA search baseline beyond the target 150 M FLOP.
Similar observation can be made for 250 M and 350 M FLOPs.
• Hybrid model achieves near SoTA accuracy with ≈ 3 X less FLOPs.
For ex., using a base with 146 M FLOPs and 71 .
68% accuracy, the hybrid model achieves 76 .
1% accuracy at 80% coverage.
To achieve 76 . 1% accuracy a single model requires > 450 M FLOPs. BaseMACs R / RB
Cov.=
Acc.(%)
Cov.=
Acc.(%) Cov.= Acc.(%) [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
• Abstaining base model achieves signiﬁcantly better performance than the base at full coverage.
For ex., a base model with 225 M FLOPs achieves an accuracy of 79 .
86% when it only covers 90% examples.
This performance increases to 88 .
94% accuracy with 70% coverage.
3. H YBRID M ODELS UNDER O VERWHELMING R ESOURCE C ONSTRAINTS
In this experiment, we explore hybrid models where base is severely resource constrained.
Concretely, we consider ImageNet classiﬁcation on a tiny MCU (STM32F746 MCU with 320KB SRAM and 1MB
Flash), the same seting as MCUNets Lin et al.
Using MCUNet TFLite model ( 12 .
79 M FLOPs, 51 .
51% accuracy) as base, we create a hybrid model by adding various OFA global models.
The resulting hybrid model performance is given in Table 3, with the energy comparison Appendix.
B.4.
Below we summarize the beneﬁts of deploying hybrid models:
• SOTA Accuracy and Pareto Dominance.
Hybrid model with deployable base achieves near SoTA accuracy with ≈ 3 X fewer FLOPs.
In addition, hybrid models consistently dominate stand-alone designs across different target accuracy levels.
• Micro-controller Implementation.
Successfully deployed base and routing models on a microcontroller with negligible ( ∼ 2% ) slowdown.
[Table]
3. H YBRID M ODELS UNDER L ATENCY M ETRIC
For simplicity, we used the coverage (i.e., data kept on the base) as a proxy for the communication latency since it is the major contributor to inference cost in a hybrid system.
In this section, we explicitly deﬁne the communication device and measure the three latency components in this hybrid system: (a) inference latency spent on-device, (b) communication latency for examples sent to cloud, (c) inference latency on the cloud.
Table 4 benchmarks the latency ( a+b+c ) of the hybrid approach against various baselines.
It shows that the hybrid model operates at nearly half the latency and power consumption compared to the global only solution and provides 8% improvements over the best available model on the device. We provide details and other conﬁgurations in the Appendix §B.6. [Table]  C ONCLUSION
We proposed a novel hybrid edge-cloud network to handle the majority of the workload of large-scale prediction on edge devices.
Currently, large-scale prediction tasks are exclusively handled at the cloud with high-capacity DNNs, and although recent works propose methods for compression of high-capacity models, the resulting models when required to achieve SOTA accuracy are still too large.
Our proposed solution is based on leveraging a low-capacity network that can be deployed on an edge device, along with a high-capacity network deployed in the cloud.
When the edge device encounters challenging inputs, these inputs are transmitted and processed on the cloud.
We proposed a novel end-to-end framework for optimizing network architectures, network models, as well as the routing protocol in a systematic manner.
Our proposed method demonstrates substantial decrease in the number of overall ﬂoating point operations (FLOPs) on ImageNet dataset compared to a well-designed high-capacity network, while suffering no excess classiﬁcation loss.
Furthermore, when communication latency to the cloud is the dominant issue, we show that across different target accuracy regimes, we realize 4 × FLOP gains on the low-capacity model with 70% coverage.
R
A PPENDIX A A LGORITHMS
We summarise the methodological proposals as algorithms.
The overall method is to begin with training a super-net in the sense of Cai et al.
(2020), for which the methods of their paper can be utilised.
This produces a set of architectures A , with associated canonical models for each α ∈ A .
The overall procedure then is summarised as Algorithm 1.
This uses the two main procedures of architecture search (Algorithm 4) and hybrid training (Algorithm 3) as subroutines, which in turn may be executed in a modular way as discussed at length in the main text.
In addition, we frequently tune a given router r and base and global models to locally trade-off resource usage levels and accuracy (which saves on retraining on each different value of (cid:37) that one may be interested in.
This is realised by ﬁnding a value t adjusted to the constraint, and using the routing function r ( x ; t )
= 1 { r o ( x ) ≥ r 1 ( x ) + t } .
Such a t may be found as in Algorithm 2.
Algorithm 1 End-to-end Hybrid Procedure
1: Input: Training data B = { ( x i , y i ) } Ni =1 , Validation data V = { ( x j , y j ) } Mj
=1 , resource constraint (cid:37) .
2: Train supernet using the method of Cai et al.
(Architecture Search)
3: A ← resulting set of algorithms.
4: ( α b , α g ) ← output of Algorithm 4 with V, (cid:37), A . 5: Train initial models b 0 ∈ α b , g 0 ∈ α g using B
6: ( r, b, g ) ← output of Algorithm 3 instantiated with B, b 0 , g 0 , and with appropriate hyperparameters.
: Return:
r,
Algorithm 2 Tuning Routing Model
1: Input
resource
, Hybrid
do 5: Ω
← highest
6: Ω child ← ∅ 7: for n
1 to N pop do 8:
Randomly pick α ib α par 9: α α
← Mutate
Compute α mg .
11: if mb θ mg
> ϕ then 12:
GOTO 9.
Add ( α mb , α mg ) to Ω child 14: Ω pop = Ω par (cid:83) Ω child 15: Return : Ω pop
B I MPLEMENTATION D ETAILS B. H YPER - PARAMETER S ETTINGS .
We use SGD with momentum as the default optimizer in all our experiments.
We initialize our hybrid models from the corresponding pre-trained models and use a learning rate of 1 e − 4 for learning base and global models.
We use a learning rate of 1 e − 2 for learning the routing network.
We decay the learning rate using a cosine learning rate scheduler.
As recommended in the earlier works, we use a weight decay of 1 e − 5 .
We set the number of epochs to be 50 .
We use a batch size of 256 in our experiments.
B. M ODEL D ETAILS
Entropy Thresholding Baseline.
As per recommendation in the literature (Teerapittayanon et al., 2017; Gangrade et al., 2021)
we compute the entropy H of the base prediction probability
distribution b y ( x ) .
This baseline allows access to a tunable threshold t .
Predictions with entropy below this threshold are kept with the base model while the predictions with entropy above this threshold are sent to the cloud model.
We use similar tuning as Algorithm 2 to trade-off resource usage.
Routing Model.
Our routing model uses predictions from the base model and creates a 2-layer neural network from these predictions.
We create meta features from these predictions to reduce the complexity of the network, by (a) adding entropy as a feature, (b) and adding correlations between top 10 predictions, resulting in a 101 dimensional input feature vector.
The feed-forward network has 256 neurons in the ﬁrst and 64 neurons in the second layer.
The ﬁnal layer outputs a two dimensional score leading to binary prediction for the routing r .
Note that the routing network described in this manner contributes to less than 2% compute budget of the base model and hence its compute cost is negligible in comparison to the base and global models.
MobileNetV3.
We have used the small and large conﬁgurations as base and global models in our experiments (see Sec. 3.1 ).
We borrowed pre-trained models from publicly available implementation.
Table 5 lists the performance and compute characteristics of these borrowed models.
Once-for-All.
We borrowed the pre-trained OFA models from the ofﬁcial public repository 2 .
Tablelists the accuracy, number of parameters and FLOPs for these models.
We note that these models have been specialized by the authors with ﬁne-tuning to achieve the reported performance.
[Table] [Table]
B. O NCE - FOR -A LL S EARCH E XPERIMENTS .
In our evolutionary search experiments (see Sec. 3.2)
[Table]
we have used the OFA search to create the baseline models that eliminate the effect of ﬁne-tuning available in the pre-trained models from their ofﬁcial repository.
We created the baseline by using their optimized search to ﬁnd models at different FLOPs, namely { 70 M, 100 M, 150 M, 200 M, 250 M, 300 M, 400 M, 500 M } .
We report the performance of these models in the Table 7.
Note that as per recommendation, we tune the batch norm statistics of these models to get the correct accuracy.
For our evolutionary search experiments, we used the OFA search space.
In OFA codebase, there are two search spaces with MobileNetV3 backbone: (a) with width multiplier 1 and (b) with width multipler 1 . 2 .
For our joint architecture search with target FLOPs 150 M, we used the smaller backbone with width = 1 as this space allows smaller base models in the < 100 M FLOPs region.
While we used the backbone with width = 1 . 2 for our search for target ﬂops 250 M and 350 M, as these hybrid FLOPs allow larger base models.
OFA space allows searching over expansion factor options [3,4,6], kernel sizes [3,5,7], block depths [2,3,4], and resolutions [144, 160, 176, 192, 208, 224].
To perform a mutation, each optimization variable is modiﬁed with probability 0 .
1 , where modiﬁcation entails re-sampling the variable from a uniform distribution over all of the options.
The population size is set to 100, and the parent set size is set to 25.
Table 2 shows the characteristics of the base and global models found using this search.
[Table]
B. MCUN ET E XPERIMENTS
We deploy both MCUNet and our base with routing model on the MCU using the TensorFlow Lite for Microcontrollers (TFLM) runtime.
Due to lack of operator support for reductions and sorting in TFLM, we replace the relevant operators with supported operations whose compute and memory complexity upperbounds the un-supported operations.
Table 9 compares the performance energy proﬁle of the hybrid model and the baseline when deployed on the micro-controller (STM32F746) with 320KB SRAM & 1MB Flash.
It clearly shows that there is a negligible cost of deploying the proposed routing scheme and only results in < 2% slowdown.
Table 8 shows the performance of the hybrid model against the baseline model at various hybrid ﬂops.
It can be seen that the hybrid model dominates the baseline model at intermediate FLOPs.
[Table] [Table]
Fig 2 collates the performance of the hybrid and baseline models from the Experiments section (see
Tables 1 and 3, 70% coverage column).
Baseline corresponds to the best baseline models at various
MACs.
Hybrid numbers correspond to the hybrid model where base operates at 70% coverage level.
We list the baseline and hybrid performance metrics in Table 10 for completeness.
[Table]
L ATENCY E XPERIMENTS
When we use MCU as the base device, we use the LoRAWAN communication protocol that enables such a low capacity device to operate at a transmission rate of 1 kbps.
Thus, it takes nearly 1200 ms to transfer an image of size 150 KB (typical image in the Imagenet dataset).
Similarly, for base devices capable of operating with 3G, LTE or Wi-Fi communication devices, we borrow the transmission numbers from Kang et al.
Speciﬁcally, to transfer an image with 152 KB size, (a) 3G network takes 870 ms, (b) LTE takes 180 ms, and (c) WiFi takes 95ms.
Note that for a micro-controller even a 3G network would not be available, instead a much slower communication device is used.
We use the on-device and on-cloud inference latency from the MCUNet and OFA repositories.
For any method, we compute the latency as the inference time taken for an example, i.e., inference time on device + communication time + inference time on cloud.
To compute energy usage, we use the active-mode operating characteristics of the base device and voltage supply to compute the power and multiply it by the amount of time the device spends doing these operations.
Table 11 benchmarks the inference latency ( inference cost on device + communication cost + inference cost on the cloud ) of the hybrid approach against various baselines.
[Table]
B. D IFFERENCE BETWEEN A PPEAL N ET AND OUR H YBRID DESIGN .
Below we highlight main difference between AppealNet (Li et al.
and our proposal.
• AppealNet formulation does not explicitly model any coverage constraint that enables the base model to operate at a tunable coverage level.
In contrast, we explicitly model a coverage penalty.
• Jointly learning the routing without any supervision is a hard problem.
Instead, we relax this formulation by introducing the routing oracle that specializes in a routing network for a given base and global pair.
With this oracle, the task of learning routing reduces to a binary classiﬁcation problem with the routing labels obtained from the oracle.
This also decouples the routing task from the base and global entanglement.
• In addition, we propose a neural architecture search that ﬁnds a pair of base and global architectures that optimise the hybrid accuracy at any given combined resource usage.
• Empirically, AppealNet does not have any evaluations for the Imagenet scale dataset.
The closest comparison we can ﬁnd is with the Tiny-Imagenet dataset (one-tenth of the size of the Imagenet).
While we cannot compare the two directly, since we solve a much harder problem than Tiny-Imagenet, we can make the following observations.
At 70% coverage level, for AppealNet, the minimum performance difference between the hybrid model and the global model is ≈ 1 .
2% (see AppealNet, Fig. 5(d)), while our closest to the global in case of the MobileNet baseline is 0 .
3% (see our paper Table 1, row 3).
Note that AppealNet performance will go down on Imagenet in comparison to Tiny-Imagenet due to the hardness of the problem.