O PTIMAL C ONTROL V IA N EURAL N ETWORKS : A C ONVEX A PPROACH
A BSTRACT
Control of complex systems involves both system identiﬁcation and controller design.
Deep neural networks have proven to be successful in many identiﬁcation tasks, however, from model-based control perspective, these networks are difﬁcult to work with because they are typically nonlinear and nonconvex.
Therefore many systems are still identiﬁed and controlled based on simple linear models despite their poor representation capability.
In this paper we bridge the gap between model accuracy and control tractability faced by neural networks, by explicitly constructing networks that are convex with respect to their inputs.
We show that these input convex networks can be trained to obtain accurate models of complex physical systems.
In particular, we design input convex recurrent neural networks to capture temporal behavior of dynamical systems.
Then optimal controllers can be achieved via solving a convex model predictive control problem.
Experiment results demonstrate the good potential of the proposed input convex neural network based approach in a variety of control applications.
In particular we show that in the
MuJoCo locomotion tasks, we could achieve over 10% higher performance using 5 × less time compared with state-of-the-art model-based reinforcement learning method; and in the building HVAC control example, our method achieved up to 20% energy reduction compared with classic linear models.
I NTRODUCTION
Decisions on how to best operate and control complex physical systems such as the power grid, commercial and industrial buildings, transportation networks and robotic systems are of critical societal importance.
These systems are often challenging to control because they tend to have complicated and poorly understood dynamics, sometimes with legacy components are built over a long period of time (Wolf, 2009).
Therefore detailed models for these systems may not be available or may be intractable to construct.
For instance, since buildings account for 40% of the global energy consumption (Cheng et al., 2008), many approaches have been proposed to operate buildings more efﬁciently by controlling their heating, ventilation, and air conditioning (HVAC) systems (Zhang et al., 2017).
Most of these methods, however, suffer from two drawbacks.
On one hand, a detailed physics model of a building can be used to accurately describe its behavior, but this model can take years to develop.
On the other hand, simple control algorithms have been developed by using linear (RC circuit) models (Ma et al., 2012) to represent buildings, but the performance of these models may be poor since the building dynamics can be far from linear (Shaikh et al., 2014).
In this paper, we leverage the availability of data to strike a balance between requiring painstaking manual construction of physics based models and the risk of not capturing rich and complex system dynamics through models that are too simplistic.
In recent years—with the growing deployment of sensors in physical and robotics systems—large amount of operational data have been collected, such as in smart buildings (Suryadevara et al., 2015), legged robotics (Meger et al., 2015) and manipulators (Deisenroth et al., 2011).
Using these data, the system dynamics can be learned directly and then automatically updated at periodic intervals.
One popular method is to parameterize these complex system dynamics using deep neural networks to capturing complex relationships (He et al., 2016; Vaswani et al., 2017), yet few research investigated how to integrate deep learning models into real-time closed-loop control of physical systems.
A key reason that deep neural networks have not been directly applied in control is that even though they provide good performances in learning system behaviors, optimization on top of these networks is challenging (Kawaguchi, 2016).
Neural networks, because of their structures, are generally not convex from input to output.
Therefore, many control applications (e.g., where real-time decisions need to be made) choose to favor the computational tractability offered by linear models despite their poor ﬁtting performances.
In this paper we tackle the modeling accuracy and control tractability tradeoff by building on the input convex neural networks (ICNN) in (Amos et al., 2017) to both represent system dynamics and to ﬁnd optimal control policies.
By making the neural network convex from input to output, we are able to obtain both good predictive accuracies and tractable computational optimization problems.
The overall methodology is shown in Fig.
Our proposed method (shown in Fig.(b)) ﬁrstly utilizes an input convex network model to learn the system dynamics and then computes the best control decisions via solving a convex model predictive control (MPC) problem, which is tractable and has optimality guarantees.
This is different from existing methods that uses model-free end-to-end controller which directly maps input to output (shown in Fig.
1 (a)).
Another major contribution of our work is that we explicitly prove that ICNN can represent all convex functions and systems dynamics, and is exponentially more efﬁcient than widely used convex piecewise linear approximations (Magnani & Boyd, 2009).
[Figure]
method, (a) an input convex neural network is ﬁrst trained to learn the system dynamics, then (b) we solve a convex predictive control problem to ﬁnd the optimal actions which are input convex neural networks’ inputs.
The optimization steps are also based on objectives and dynamics constraints represented by the trained networks.
1. R ELATED W ORK
The work in (Amos et al., 2017) was an impetus for this paper.
The key differences are that the goal in (Amos et al., 2017) is to show that ICNN can achieve similar classiﬁcation performances as conventional neural networks and how the former can be used in inference and prediction problems.
Our goal is to use these networks for optimization and closed-loop control, and in a sense that we are more interested in the overall system performances and not directly the performance of the networks.
We also extend the class of networks to include RNNs to capture dynamical systems.
Control and decision-making have used deep learning mainly in model-free end-to-end controller settings (shown in Fig.
1 (a)), such as sequential decision making in game (Mnih et al., 2013), robotics manipulation (Levine & Koltun, 2014; Levine et al., 2016), and control of cyber-physical systems (Wei et al., 2017; O’Neill et al., 2010).
However, much of the success relies heavily on a reinforcement learning setup where the optimal state-action relationship can be learned via a large number of samples.
However, many physical systems do not ﬁt into the reinforcement learning process, where both the sample collection is limited by real-time operations, and there are physical model constraints hard to represent efﬁciently.
To address the above sample efﬁciency, safety and model constraints incompatibility concerns faced by model-free reinforcement learning algorithms in physical system control, we consider a model-based control approach in this work.
Model-based control algorithms often involve two stages – system identiﬁcation and controller design.
For the system identiﬁcation stage, the goal is to learn a ﬁxed form of system model to minimize some prediction error (Ljung, 1998).
Most efﬁcient model-based control algorithms have used a relatively simple function estimator for the system dynamics identiﬁcation (Nagabandi et al., 2018), such as linear model (Ma et al., 2012) and Gaussian processes (Meger et al., 2015; Deisenroth et al., 2011).
These simpliﬁed models are sample-efﬁcient to learn, and can be nicely incorporated in the sub-sequent optimal control problems.
However, such simple models may not have enough representation capacity in modeling large-scale or high-dimension systems with nonlinear dynamics.
Deep neural networks (DNNs) feature powerful representation capability, while the main challenge of using DNNs for system identiﬁcation is that such models are typically highly non-linear and non-convex (Kawaguchi, 2016), which causes great difﬁculty for following decision making.
A recent work from (Nagabandi et al., 2018) is close in spirit as our proposed method.
Similarly, the authors use a model-based approach for robotics control, where they ﬁrst ﬁt a neural network for the system dynamics and then use the ﬁtted network in an MPC loop.
However, since (Nagabandi et al., 2018) use conventional NN for system identiﬁcation, they cannot solve the MPC problem to global optimality.
Our work shows how the proposed ICNN control algorithm achieves the beneﬁts from both sides of the world.
The optimization with respect to inputs can be implemented using off-the-shelf deep learning optimizers, while we are able to obtain good identiﬁcation accuracies and tractable computational optimization problems by using proposed method at the same time.
C LOSED - LOOP CONTROL WITH INPUT CONVEX NEURAL NETWORKS
In this paper, we consider the settings where a neural network is used in a closed-loop system.
The fundamental goal is to optimize system performance which is beyond the learning performance of network on its own.
In this section we describe how input convex neural networks (ICNN) can be extremely useful in these systems by considering two related problems.
First, we show how
ICNN perform in single-shot optimization problems.
Then we extend the results to an input convex recurrent neural networks (ICRNN), which allows us to both capture systems’ complex dynamics and make time-series decisions.
t- ...
[Figure]
(b)
Input
Output
Hidden
ModelWeights 1 D 2 D 3 D
W
U
V 2 D i D k D 1 z 2 z
y 1 W 2 W k W u y z
-u = ˆ u
Activations + +
K-1 z + + + u ˆ t + + t+ + + t+ + + tt t+ t+
2. S INGLE - SHOT PROBLEM
The following proposition states a simple sufﬁcient condition for a neural network to be input convex:
Proposition 1.
The feedforward neural network in Fig. 2(a) is convex from input to output given that all weights between layers W 1: k and weights in the “passthrough” layers D 2: k are non-negative, and all of the activation functions are convex and nondecreasing (e.g. ReLU).
The structure of the input convex neural network (ICNN) structure in Proposition 1 is motivated by the structure in (Amos et al., 2017) but modiﬁed to be more suitable to control of dynamical systems.
In (Amos et al., 2017) it only requires W 2: k to be non-negative while having no restrictions on weights W 1 and D 2: k .
Our construction achieves the exact representation by expanding the inputs to include both u ( ∈ R d ) and − u .
Then any negative weights in W 1 and D 2: k in (Amos et al., 2017)’s ICNN structure is set to zero and its negation (which is positive) is added as the weight for corresponding − u .
The reason for our construction is to allow the network to be “rolled out in time” when we are dealing with dynamical systems and multiple networks need to be composed together.
An simple example that demonstrates how the proposed ICNN can be used to ﬁt a convex function comes form ﬁtting the | u | function.
This function is convex and both decreasing and increasing.
Let the activation function be ReLU ( · ) = max ( · , 0 ) .
We can write | u | = − u + 2 ReLU ( u ) (Amos et al., 2017).
However, in this representation, we need a negative weight, the − 1 in front of u , and this would be troublesome if we compose several networks together.
In our proposed ICNN structure with all positive weights and input negation duplicates, we can write | u | = v + 2 ReLU ( u ) , where we impose a constraint v = − u .
Such doubline on the number of input variables may potentially make the network harder to train.
Yet during control, having all of the weights positive maintains the convexity between inputs and outputs even if multiple steps are considered which will be discussed in
Section 2.2.
The constraint v = − u is linear and can be easily included in any convex optimization.
This proposition follows directly from composition of convex functions (Boyd & Vandenberghe, 2004).
Although it allows for any increasing convex activation functions, in this paper we work with the popular ReLU activation function.
Two notable additions in ICNN compared with conventional feedforward neural networks are: 1) Addition of the direct “passthrough” layers connecting inputs to hidden layers and conventional feedforward layers connecting hidden layers for better representation power.
2) the expanded inputs that include both u and − u .
The proposed ICNN structure is shown in Fig. 2(a).
Note that such construction guarantees that the network is convex and non-decreasing with respect to the expanded inputs ˆ u = (cid:20) u − u (cid:21) , while the output can achieve either decreasing or non-decreasing functions over u .
Fundamentally, ICNN allows us to use neural networks in decision making processes by guaranteeing the solution is unique and globally optimal.
Since many complex input and output relationships can be learned through deep neural networks, it is natural to consider using the learned network in an optimization problem in the form of
[Equation]
where U is a convex feasible space.
Then if f is an ICNN, optimizing over u is a convex problem, which can be solved efﬁciently to global optimality.
Note that we will always duplicate the variables by introducing v = − u , but again this does not change the convexity of the problem.
Of course, since the weights of the network are restricted to be nonnegative, the performance of the network (e.g., classiﬁcation) may be worse.
A common thread we observe in this paper is that trading off classiﬁcation performance with tractability can be preferable.
2. C LOSED - LOOP CONTROL AND RECURRENT NEURAL NETWORKS
In addition to the single-shot optimization problem in (1), we are interested in optimally controlling a dynamical system.
To model the temporal dependency of the system dynamics, we propose to use recurrent neural networks (instead of feed-forward neural networks).
Recurrent networks carry an internal state of the system, which introduces coupling with previous inputs to the system.
Fig.
2(b) shows the proposed input convex recurrent neural networks (ICRNN) structure.
This network maps from input ˆ u to output y with memory unit z
according to the following Eq.
[Equation]
where ˆ u = (cid:20) u − u (cid:21) , and D 1 , D 2 , D 3 are added direct “passthrough” layers for augmenting representation power.
If we unroll the dynamics with respect to time, we have y t =
f
( ˆ u 1 , ˆ u 2 ,..., ˆ u t ; θ ) where θ =
[ U , V , W , D 1 , D 2 , D 3 ] are network parameters, and σ 1 , σ 2 denote the nonlinear activation functions.
The next proposition states a sufﬁcient condition for the network to be input convex.
Proposition 2.
The network shown in Fig. 2(b) is a convex function from inputs to output if all weights
U , V , W , D 1 , D 2 , D 3 are non-negative, and all activation functions are convex and nondecreasing (e.g.
ReLU).
The proof of this proposition again follows directly from the composition rule of convex functions.
Similarly to the ICNN case, by expanding the inputs vector to include both u and − u and restricting all weights to be non-negative, the resulted ICRNN structure is a convex and non-decreasing mapping from inputs to output.
The proposed ICRNN structure can be leveraged to represent system dynamics for close-loop control.
Consider a physical system with discrete-time dynamics, at time step t , let’s deﬁne s t as the system states , u t as the control actions , and y t as the system output .
For example, for the real-time control of a building system, s t includes the room temperature, humidity, etc; u t denotes the building appliance scheduling, room temperature set-points, etc; and output y t is the building energy consumption.
In addition, there maybe exogenous variables that impact the output of the system, for example, outside temperature will impact the energy consumption of the building.
However, since the exogenous variables are not impacted by any of the control actions we take, we suppress them in the formulation below.
The time evolution of a system is described by
[Equation]
where (4b) describes the coupling between the current inputs to the future system states.
Physical systems described by (4) may have signiﬁcant inertia in the sense that the outcome of any control actions is delayed in time and there are signiﬁcant couplings across time periods.
Since we use ICRNNs to represent both the system dynamics g ( · ) and the output f ( · ) , the control variable u expands as ˆ u .
The optimal receding horizon control problem at time t can be written as, minimize u
[Equation]
subject to y
[Equation]
where a new variable ˆ x = [ s t , ˆ u t ] is introduced for notational simplicity, which called system inputs .
It is the collection of system states s t and duplicated control actions u t and − u t , therefore ensuring the mapping from u t to any future states and outputs remains convex.
J ( ˆ x τ , y τ ) is the control system cost incurs at time τ , that is a function of both the system inputs ˆ x τ and output y τ .
The functions f ( · ) and g ( · ) in Eq.
(5b)-(5c) are parameterized as ICRNNs, which represent the system dynamics from sequence of inputs ( ˆ x τ − n w , ˆ x τ − n w + 1 ,..., ˆ x τ ) to the system output y τ , and the dynamics from control actions to system states, respectively.
n w is the memory window length of the recurrent neural network.
The equations (5d) and (5e) duplicate the input variables u and enforce the consistency condition between u and its negation v .
Lastly, (5f) and (5g) are the constraints on feasible system states and control actions respectively.
Note that as a general formulation, we do not include the duplication tricks on state variables, so the dynamics ﬁtted by (5b) and (5c) are non-decreasing over state space, which are not equivalent to those dynamics represented by linear systems.
However, since we are not restricting the control space, and we have explicitly included multiple previous states in the system transition dynamics, so the non-decreasing constraint over state space should not restrict the representation capacity by much.
In Section.3 we theoretically prove the representability of proposed networks.
Optimization problem in (5) is a convex optimization with respect to inputs u
= [ u t ,..., u t + T ] , provided the cost function J ( ˆ x τ , y τ ) =
J ( s τ , ˆ u τ , y τ ) is convex with respect to ˆ u τ , and convex, nondecreasing w.r.t.
s τ and y τ .
A problem is convex iff both the objective function and constraints are convex.
In the above problem, since f and g are parameterized as ICRNNs, that are convex and non-decreasing functions from input to output.
Therefore, it is straightforward to show that both the objective function (5a) and constraints (5b)-(5c) are convex following the composition rule of convex functions.
The convexity of the problem in (5) guarantees that it can be solved efﬁciently and optimally using gradient descend method.
Since both the objective function (5a) and the constraints (5b)-(5c) are parameterized as neural networks, and their gradients can be calculated via back-propagation with the modiﬁcation where cost is propagated to the input rather than the weights of the network.
For implementation, the gradients can be convinently calculated via existing modules such as Tensorﬂow viaback-propagation.
Let u ∗ = { u ∗ t , u ∗ t + 1 ,..., u ∗ t + T } be the optimal solution of the optimization problem at time t .
Then the ﬁrst element of u ∗ is implemented to the real-time system control, that is u ∗ t .
The optimization problem is repeated at time t + 1, based on the updated state prediction using u ∗ t , yielding a model predictive control strategy.
E FFICIENCY AND REPRESENTATION POWER OF ICNN
Besides the computational traceability of the input convex networks, as an system identiﬁcation model, we are also interested its predictive accuracies and capacity.
This section provides theoretical analysis on the representation ability and efﬁciency of input convex neural networks.
3. R EPRESENTATION POWER OF INPUT CONVEX NEURAL NETWORK
Deﬁnition 1.
Given a function f : R d → R , we say that the function ˆ f approximate f within ε if | f ( x ) − ˆ f
( x ) | ≤ ε for all x in the domain of f.
Theorem 1.
[Representation power of ICNN]
For any Lipschiz convex function over a compact domain, there exists a neural network with nonnegative weights and ReLU activation functions that approximates it within ε .
Lemma 1.
Given a continuous Lipschiz convex function f : R d → R with compact domain and ε > 0 , it can be approximated within ε by maximum of a ﬁnite number of afﬁne functions.
That is, there exists ˆ f ( x ) = max i = 1 ,..., N { µ i T x + b i } such that | f ( x ) − ˆ f
( x ) | ≤ ε for all x ∈ domf.
Sketch of proof for Theorem 1.
Supposing Lemma 1 is true, the proof of Theorem 1 boils down to showing that neural network with nonnegative weights and ReLU activation functions can exactly represent a maximum of afﬁne functions.
The proof is constructive.
We ﬁrst construct a neural network with ReLU activation functions and both positive and negative weights, then we show that the weights between different layers of the network can be restricted to be nonnegative by a simple duplication trick.
Speciﬁcally, since the weights in the input layer and passthrough layers in the ICNN can be negative, we simply add a negation of each input variable (e.g. both x and − x are given as inputs) to the network.
These variables need satisfy a consistency constraint since one is the negation of the other.
Since this constraint is linear, it preserves the convexity of optimization problems.
The details of the proofs are given in the Appendix B.
This proof is similar in spirit to theorems in (Hanin, 2017; Arora et al., 2016).
The key new result is a simpler construction than the one used in (Hanin, 2017) and the restriction to nonnegative weights between the layers.
Similar to Theorem 1, an analogous result about the representation power of ICRNN can be shown for systems with convex dynamics.
Given a dynamical system described by rolled out system dynamics y t
= f
( x 1 ,..., x t ) is convex, then there exists a recurrent neural network with nonnegative weights and ReLU activation functions that approximates it within ε .
A broad range of systems can be captured by this model.
For example, the linear quadratic (Gaussian) regulator problem can be described using a ICRNN if we identify y as the cost of the regulator (Skogestad & Postlethwaite, 2007; Boyd et al., 1994).
An example of a nonlinear system is the control of electrochemical batteries.
It can be shown from ﬁrst principles that the degradation of these types of batteries is convex in their charge and discharge actions (Shi et al., 2018) and our framework offers a powerful data-driven way to control batteries found in electric vehicles, cell phones, and power systems.
3. ICNN VS . CONVEX PIECEWISE LINEAR FITTING
In the proof of Theorem 1, we ﬁrst approximate a convex function by a maximum of afﬁne functions then construct a neural network according to this maximum.
Then a natural question is why learn a neural network and not directly the afﬁne functions in the maximum?
This approach was taken in (Magnani & Boyd, 2009), where a convex piecewise-linear function (max of afﬁne functions) are directly learned from data through a regression problem.
A key reason that we propose to use ICNN (or ICRNN) to ﬁt a function rather than directly ﬁnding a maximum of afﬁne functions is that the former is a much more efﬁcient parameterization than the latter.
As stated in Theorem 2, a maximum of K afﬁne functions can be represented by an ICNN with K layers, where each layer only requires a single ReLU activation function.
However, given a single layer ICNN with K ReLU activation functions, it may take a maximum of 2 K afﬁne functions to represent it exactly.
Therefore in practice, it would be much easier to train a good ICNN than ﬁnding a good set of afﬁne functions.
Theorem 2.
[Efﬁciency of Representation]
Let f ICNN : R d → R be an input convex neural network with K ReLU activation functions.
Then Ω ( 2 K ) functions are required to represent f ICNN using a max of afﬁne functions.
Let f CPL : R d
→ R be a max of K afﬁne functions.
Then O ( K ) activation functions are sufﬁcient to represent f CPL exactly with an ICNN.
The proof of this theorem is given in Appendix C.
E XPERIMENTS
In this section, we verify the effectiveness of ICNN and ICRNN by presenting experimental results on two decision-making problems: continuous control benchmarks on MuJoco locomotion tasks (Todorov et al., 2012) and energy management of reference large-scale commercial building (Crawley et al., 2001), respectively.
The proposed method can be used as a ﬂexible building block in decision making problems, where we use ICNN to represent system dynamics for MuJoco simulators, and we use ICRNN in an end-to-end fashion to ﬁnd the optimal control inputs.
Both examples demonstrate that proposed method: 1) discovers the connection between controllable variables and the system dynamics or cost objectives; 2) is lightweight and sample-efﬁcient; 3) achieves generalizable and more stable control performances compared with previous model-based reinforcement learning and simpliﬁed linear control approaches.
4. M U J O C O L OCOMOTION T ASKS Experimental Setup
We consider four simulated robotic locomotion tasks: swimmer, half-cheetah, hopper, ant implemented in MuJoCo under the OpenAI rllab framework (Duan et al., 2016).
We train and represent the locomotion state transition dynamics s t + 1 = g ( s t , u t ) 2 using a 2-layer ICNN with ReLU activations, which could be integrated into the following ﬁnite-horizon control problem to ﬁnd the optimal action sequence u t ,..., u t + T for ﬁxed looking ahead horizon T : [Equation]
6a
[Equation]
subject
[Equation] 6b) [Equation] where the objective (6a) is convex because r ( s τ , u τ ) is a concave reward function related to system states such as velocity and control actions (the detailed forms of r ( s τ , u τ ) for different locomotion tasks are listed in Appendix D).
To achieve better model generalization on locomotion dynamics, we also followed (Nagabandi et al., 2018), and applied DAGGER
(Ross et al., 2011) to iteratively collect labeled robotic rollouts and train the supervised dyamics model (6b) using on-policy locomotion samples.
See Appendix D for furthur simulation hyperparameters and experimental details.
For each aggregated iterations of collecting rollouts data and training ICNN model, we validate the controller performance on standalone validation rollouts by optimally solving (6).
[Figure]
Baselines We compare our system modeling and continuous control method with state-of-the-art model-based RL algorithm (Nagabandi et al., 2018), where the authors used a normal multi-layer perceptrons (MLP) model to parameterize the system dynamics (6b).
We refer to their method as random-shooting algorithm, since they can not solve (6) to optimality, and they used pre-deﬁned number of random-shooting control sequences (denoted as K ) to query the trained MLP and ﬁnd a best sequence as the rollout policy.
Such a method is able to ﬁnd good control policies in the degree of 10 4 timesteps, which are much more sample-efﬁcient than model-free RL methods (Duan et al., 2016; Mnih et al., 2015).
To make fair comparisons with baseline method, we keep the same setup on the rollouts number and initial random action training.
Our framework makes the neural networks convex w.r.t input by adding passthrough links to the 2-layer model and keeping all the layer weights nonnegative.
We evaluate the performance of both algorithms on three randomly selected ﬁxed random seeds for four tasks.
Similar to the ﬁne tuning steps in (Nagabandi et al., 2018), control policies found by ICNN can also be plugged in as initialized policies for subsequent model-free reinforcement learning algorithms.
Continuous Control Performance
During training, we found both ICNN and MLP are able to predict robotic states quite accurately based on (6b).
This provides a good system dynamics model which is beneﬁcial to solve control policies.
The control performances are shown in Fig.
3, where we compare the average reward of proposed method and random-shooting method with K = 100 overvalidation rollouts during each aggregated iteration (see Fig. 8 in Appendix D.4 for random shooting performance with varying K ).
The policy found by ICNN outperforms the random-shooting method in all settings with varying horizon T for all of the four locomotion tasks.
Intuitively, ICNN should perform better when the action space is larger, since random-shooting method can not search through the action space efﬁciently with a ﬁxed K .
This is illustrated in the example of ant , where with more training samples aggregated and MLP model representing more accurate dynamics, random-shooting gets stuck to ﬁnd better control policies and there is little improvement reﬂected in the control performance.
Moreover, since we are skipping the expensive process on calculating rewards of each random shooting trajectory and ﬁnding the best one, our method only implements ICNN inference step based on (6) and is much faster than random shooting methods in most settings, especially when K is large (see Table.
2 for wall-clock time in Appendix
D.3).
For instance, in the case of Swimmer , our proposed method only uses 1 5 of time compared to (Nagabandi et al., 2018).
This also indicates that our method is even much more sample-efﬁcient than off-the-shelf model-free RL methods, where we use two orders of magnitude less training data to reach similar validation rewards (Duan et al., 2016; Mnih et al., 2015) (see Fig.
9 in Appendix D.4).
4. B UILDING E NERGY M ANAGEMENT
Experimental Setup We now move on to optimally control a dynamical system with signiﬁcant inertia.
We consider the real-time control problem of building’s HVAC (heating, ventilation, and air conditioning) system to reduce its energy consumption.
Building energy management remains to be a hard problem in control area.
The exact system dynamics are unknown and hard to model due to the complex heating transfer dynamics, time-varying environments and the scale of the system in terms of states and actions (Kouro et al., 2009).
At time t , we assume the building’s running proﬁle x t : =
[ s t , u t ] is available, where s t denotes building system states, including outside temperature, room temperature measurements, zone occupancies and etc.
u t denotes a collection of control actions such as room temperature set points and appliance schedule.
Output is the electricity consumption P t .
This is a model predictive control problem in the sense that we want to ﬁnd the best control inputs that minimize the overall energy consumption of building by looking ahead several time steps.
To achieve this goal, we ﬁrstly learn an ICRNN model f ( · ) of the building dynamics, which is trained to minimize the error between P t and f ( x t − n w ,..., x t ) , while n w denotes the memory window of recurrent neural networks. Then we solve: minimize [Equation]
[Equation] [Equation] (7a) subject to s t + τ = g ( x t + τ − n w ,..., x t + τ − 1 , u t +
[Equation]
∀ τ (7b) [Equation] [Equation] where the objective (7a) is minimizing the total energy consumption in future T steps ( T is the model predictive control horizon), and (7b) is used for modeling building states, in which g ( · ) are parameterized as ICRNNs.
Note that the formulation (7) is also ﬂexible with different loss functions.
For instance, in practice, we could reuse trained dynamics model (7b), and integrate electricity prices into the overall objective so that we could directly learn real-time actions to minimize electricity bills (please refer to Appendix E for more results).
The constraints on control actions u t and system states s t are given in (7c) and (7d).
For instance, the temperature set points as well as real measurements should not exceed user-deﬁned comfort regions.
(b) (c)
E ne r g y
C on s u m p t
i on [ J ]
Mon
Tue
Wed
Thu
Fri
Ground Truth
ICRNN
Normal RNN
Mon Tue Wed Thu Fri
T e m pe r a t u r e
[ C ] 182022
Basement
Top Middle      8 ×
Mon
Tue
Wed
Thu
Fri (a)
T e m pe r a t u r e
[ C ]
Mon Tue Wed Thu Fri
RC Model Dynamics Fitting
{+Control Performance × 
+}
To test the performance of the proposed method, we set up a 12-story large ofﬁce building, which is a reference EnergyPlus commercial building model from US Department of Energy (DoE) 3 , with a total ﬂoor area of 498 , 584 square feet which is divided into 16 separate zones.
By using the whole year’s weather proﬁle, we simulate the building running through the year and record ( x t , P t ) with a resolution of 10 minutes.
We use 10 months’ data to train the ICRNN and subsequent 2 months’ data for testing.
We use 39 building system state variables s t (uncontrollable), along with 16 control variables u t . Output is a single value of building energy consumption at each time step. We set the model predictive control horizon T = 36 (six hours). We employ an ICRNN with recurrent layer of dimension 200 to ﬁt the building input-output dynamics f ( · ) .
The model is trained to minimize the
MSE between its predictions and the actual building energy consumption using stochastic gradient descent.
We use the same network structure and training scheme to ﬁt state transition dynamics g ( · ) .
Baseline We set the model-based forecasting and optimization benchmark using an linear resistorcircuit (RC) circuit model to represent the heat transfer in building systems, and solve for the optimal control actions via MPC (Ma et al., 2012).
At each step, MPC algorithm takes into account the forecasted states of the building based on the ﬁtted RC model and implements the current step control actions.
We also compare the performance of ICRNN against the conventionally trained RNN in terms of building dynamics ﬁtting performance and control performance.
To solve the MPC problem with conventional RNN models, we also use gradient-based method with respect to controls.
However, since conventional RNN models are generally not convex from input to output, there is no guarantee to reach a global optimum (or even a local one).
Results In terms of the ﬁtting performance, ICRNN provides a competitive result compared to conventional RNN model.
The overall test root mean square error (RMSE) is 0 .
054 for ICRNN and 0 .
051 for conventional RNN, both of which are much smaller than the error made by RC model (0.240).
Fig. 4(a) shows the ﬁtting performance on 5 working days in test data.
This illustrates the good performance of ICRNN in modeling building HVAC system dynamics.
Then by using the learned ICRNN model of building dynamics, we obtain the suggested room control actions u ∗ t by solving the optimal building control problem (7).
As shown in Fig. 4(b), with the same constraints on building temperature interval of [ 19 ◦ C , 24 ◦ C ] , the building energy consumption is reduced by 23 .
25% after implementing the new temperature set points calculated by ICRNN.
On the contrary, since there is no guarantee for ﬁnding optimal control actions by optimizing over conventional RNN’s input, the control solutions given by conventional RNN could only reduce 11 .
73% of electricity.
Solutions given by RC model only saves 4 .
07% of electricity.
More importantly, in Fig. 4(c)
we demonstrate the control actions outputted by our method against MPC with conventional RNN in two randomly selected building zones, the building basement and top ﬂoor central area.
It shows that our proposed approach is able to ﬁnd a group of stable control actions for the building system control.
While in the conventional RNN case, it generates control set points which have undesirable, drastic variations.
S UMMARY AND D ISCUSSION
In this work we proposed a novel optimal control framework that uses deep neural networks engineered to be convex from the input to the output.
This framework bridges machine learning and control by representing system dynamics using input convex (recurrent) neural networks.
We show that many interesting data-driven control problems can be cast as convex optimization problems using the proposed network architecture.
Experiments on both benchmark MuJoCo locomotion tasks and building energy management demonstrate our methodology’s potential in a variety of control and optimization problems
A PPENDIX A. T OY E XAMPLE
Consider a synthetic example which contains two circles of noisy input data u ∈ R 2 , along with discrete data label y ∈ { 0 , 1 } which is based on input coming from inner loop ( y = 0) or outer loop ( y = 1).
Suppose a decision maker is interested in ﬁnding the u that maximizes the probability of y being 0.
This optimization problem can be solved by ﬁrstly learning a neural network classiﬁer from u to y , and then to ﬁnd the u point which minimizes the output of the neural network.
More speciﬁcally, let f NN be a conventional neural network and f ICNN be an ICNN.
Then the objective becomes minimizing f NN ( u ) or f ICNN ( u ) .
Figure 5 shows the decision boundaries for f NN and f ICNN , respectively.
These networks are composed of 2 hidden layers, with 200 neurons in each layer, and are trained using the same random seed, same number of samples (100) until loss convergence.
The decision boundaries of a conventional network have many “zigzags”, which makes solving (1) challenging, especially if u is constrained.
In contrast, the ICNN has convex level sets (by construction) as decision boundaries, which leads to a convex optimization problem.
[Figure]
A PPENDIX B. P ROOF OF T HEOREM
Proof.
Lemma 1 follows from well established facts in function analysis stating that piecewise linear functions are dense in the space of all continuous functions over compact sets (Royden & Fitzpatrick, 2010) and convex piecewise linear functions are dense in the space of all convex continuous functions (Cox, 1971;
Gavrilovi ´ c, 1975).
Using the fact that convex piecewise linear functions can be represented as a maximum of afﬁne functions (Magnani & Boyd, 2009; Wang, 2004) gives the desired result in the lemma.
Lemma 1 shows that all continuous Lipschitz convex functions f ( x ) :
R d → R over convex compact sets can be approximated using maximum of afﬁne functions.
Then it sufﬁces to show that an ICNN can exactly represent a maximum of afﬁne functions.
To do this, we ﬁrst construct a neural network with ReLU activation function with both positive and negative weights that can represent a maximum of afﬁne functions.
Then we show how to restrict all weights to be nonnegative.
As a starting example, consider a maximum of two afﬁne functions
[Equation]
To obtain the exact same function using a neural network, we ﬁrst rewrite it as
[Equation]
Now deﬁne a two-layer neural network with layers z 1 and z 2 as shown in Fig. 6:
[Equation]
where σ is the ReLU activation function and the second layer is linear.
By construction, this neural network is the same function as f CPL given in (8). [Equation] [Figure]
[Equation]
The above argument extends directly to a maximum of K linear functions.
Suppose
[Equation]
Again the trick is to rewrite f CPL ( x ) as a nested maximum of afﬁne functions.
For notational convenience, let L i =
a Ti x + b i , L (cid:48)
i
=
L i − L i + 1 .
Then
[Equation]
The last equation describes a K layer neural network, where the layers are:
[Equation]
Each layer of of this neural network uses only a single activation function.
Although the above neural network exactly represent a maximum of linear functions, it is not convex since the coefﬁcients between layers could be negative.
In particular, each layer involves an inner product of the form ( a i − a i + 1 ) T x
and the coefﬁcients are not necessarily nonnegative.
To overcome this, we simply expand the input to include x and − x .
Namely, deﬁne a new input ˆ x ∈ R 2 d as
[Equation]
Then any inner product of the form h T x can be written as h T x = d ∑ j =h
i x i
=
∑ i : h i ≥h i
x i + ∑ i : h
i <h i x i =
∑ i : h i ≥h i
x i + ∑ i : h
i <( − h i )( − x i )
=
∑ i : h i ≥h i
ˆ x i + ∑
i : h i <( − h i )( ˆ x i + d ) , where all coefﬁcients are nonnegative in the above sum.
Therefore any inner product between a coefﬁcient vector and the input x can be written as an inner product between a nonnegative coefﬁcient vector and the expanded input ˆ x .
Therefore, without loss of generality, we can limit all of the weights between layers to be nonnegative, and thus the neural network to be input convex.
Note that in optimization problems, we need to enforce consistency in ˆ x be including (12) as a constraint.
However, this is a linear equality constraint, which maintains the convexity of the optimization problem.
A PPENDIX C. P ROOF OF T HEOREM
Proof.
The second statement of Theorem 2 directly follows the construction in the proof of Theorem 1, which shows that a maximum of K afﬁne functions can be represent by a K -layer ICNN (with a single ReLU function in each layer).
So it remains to show the ﬁrst statement of Theorem 2.
To show that a maximum of afﬁne functions can require exponential number of pieces to approximate a function speciﬁed by an ICNN with K activation functions, consider a network with 1 hidden layer of K nodes and the weights of direct “passthrough” layers are set to 0:
[Equation]
It contains 3 K parameters: w 0 i , w 1 i and b i , where w 0 i ∈ R d and w 1 i , b i ∈ R .
In order to represent the same function by a maximum of afﬁne functions, we need to assess the value of every activation unit σ ( w T 0 i x + b i ) .
If w T 0 i x + b i ≥ 0, σ ( w T
i x + b i ) =
w T 0 i x
+ b i ; otherwise, σ ( w T 0 i x + b i ) = 0.
In total, we have 2 K potential combinations of piecewise-linear function, including
L
[Equation]
, if all
[Equation]
=
[Equation]
1 i [Equation] i , if w T 01 x + b 1 < 0 and all other w T
i x + b i
[Equation]
[Equation]
i
[Equation]
i , if w T 02 x + b 2 < 0 and other w T [-
-]0 i x + b i ≥······
[Equation]
L
[Equation]
= 0 , if all w T 0 i x + b i < 0 .
So the following maximum over 2 K pieces is required to represent the single linear ICNN: max { L 1 , L 2 ,..., L 2 K } . [Table] Cheetah Hopper
Ant
[Table]
[Table]
Rollout Samples
To train the neural network dynamics model (both ICNN and MLP), we ﬁrst collect initial rollout data using fully random action sequences u t ∼ Uniform [ -1 , 1 ] with a random chosen initial state.
During the data collection process in aggregated iterations, to improve model generalization and explore larger state spaces, we add Gaussian noise to the optimal control policies u t = u t + N ( 0 , 0 . 001 ) .
Neural Networks Training We represent the MuJoCo dynamics with a 2-hidden-layer neural networks with hidden sizes
512 − 512.
The passthrough links of ICNN are of same size of corresponding added layers.
We train both models using Adam optimizer with a learning rate 0.001 and a mini-batch size of 512.
Due to the different complexity of MuJoCo tasks, we vary training epochs and summarize the training details in Table.
D.2 E NVIRONMENT D ETAILS
In all of the MuJoCo locomotion tasks, s includes state variables such as robot positions, velocity along each axis; u includes action efforts for the agent.
We use standard reward functions r ( s t , u t ) for moving tasks, which could be also promptly calculated in (6a) as the control objective.
For the ease of neural network training and action sampling, we normalize all the action and states in the range of [ − 1 , 1 ] .
We use DAGGER (Ross et al., 2011) for 6 aggregated iterations for all cases, and during aggregated iteration, we use a split of 10% random rollouts collected as described in 5, and other 90% coming from past iterations’ control policies (on-policy rollouts).
Note that we use 10 random control sequences in our method to initialize the policy ﬁnding approach and avoid the long computation time for taking gradients on ﬁnding optimal u t .
Other environment parameters are described in Table.
D.3 W ALL -C LOCK T IME
In Table.2, we show the average run time for the total of 6 aggregation iterations over 3 runs.
Finding control policies via ICNN is using less or equal training time compared to random-shooting method with K = 100, while achieving better task rewards than K = 1000 for different control horizons.
All the experiments are running on a computer with 8 cores Intel I7 6700 CPU.
Note that we do not use GPU for accelerating ICNN optimization step (6), which could furthur improve our method’s efﬁciency.
D.4 D ETAILS OF S IMULATION R ESULTS MuJoCo Dynamics Modeling
In Fig.
7, we compare the ICNN and normal MLP ﬁtting performance of the MuJoCo dynamics modeling (6b), which illustrates that both MLP and ICNN are able to ﬁnd a data-driven dynamics model for ant MuJoCo agent, which is of the most complex dynamics we considered for locomotion tasks.
The multi-step prediction errors of ICNN is comparable to normal MLP used in (Nagabandi et al., 2018) for different length of rollout steps.
Swimmer
K = K =
K =
ICNN
H =18.
H =21.
H =40.
Half-Cheetah
K = K =
K =
ICNN
H =34.
H =48.
H =113.
Hopper
K = K =
K =
ICNN
H =5. H =5.
H =10.
Ant
K = K =
K =
ICNN
H =399.
H =480.
H =979.
ICNN + .
MLP
[Figure]
More Simulation Results In Fig.
8, we compare our control method with random-shooting approach with varying settings on shooting number K , which shows that our approach is more efﬁcient in ﬁnding control policies.
In Fig.
9, we compare our control method with the rllab implementation of trust region policy optimization (TRPO) (Schulman et al., 2015), an end-to-end deep reinforcement learning approach for mujoco locomotion tasks.
More speciﬁcally, we compare the algorithms’ performances with relatively few available rollout samples.
While our approach quickly learns the dynamics and then ﬁnd control actions via optimization steps, TRPO is hard to learn the actions directly with few provided rollouts.
Similarly to the model-based and model-free (Mb-Mf) approach described
Swimmer
Half-Cheetah
C u m u l a ti v e
R e w a r d   - -                                   -
[Figure]
K=
K=1000 K=
ICNN + + + .
Horizon=
Horizon= Horizon=
C u m u l a ti v e
R e w a r d
[Figure]
[Figure]
Hopper
[Figure]
Ant
[Figure]
u l a ti v e
R e w a r
[Figure]
[Figure]
m u l a ti v e
R e w a r d
Swimmer
Half-Cheetah
Hopper
[Figure]
A v e r a g e R e t u r n A v e r a g e R e t u r n
A v e r a g e R e t u r n A v e r a g e R e t u r n
ICNN
Random Shooting
TRPO
A PPENDIX E. D ETAILS ON B UILDING E NERGY M ANAGEMENT E.1 M INIMIZING E LECTRICITY C OSTS
To further demonstrate the potential of our proposed control framework in dealing with different real world tasks, we modify the setting of the building control example in Section 4.2 to a more complicated case.
Instead of directly minimize the total energy consumption of building, we aim to minimize the total energy cost of building which subject to a varying time-of-use electrical price λ .
The optimization problem in (7) should be re-written as, minimize u t ,..., u t + T
T ∑
[Equation]
[Equation]
[Equation]
τ ) (14a) subject to s t + τ = g
( x t + τ − n w ,..., x t + τ − 1 , u t + τ ) , ∀ τ (14b)
[Equation]
(14c)
[Equation]
(14d) where the objective (14a) is minimizing the total energy cost of building in future T steps ( T is the model predictive control horizon) subject to time-of-use electricity price λ τ , and (14b) is used for modeling building states, in which g ( · ) are parameterized as ICRNNs.
Same as the previous building control case, we have constraints on both control actions u t and system states s t are given in (14c) and (14d).
For instance, the temperature set points as well as real measurements should not exceed user-deﬁned comfort regions.
In Fig.
10 we visualize our model ﬂexibility by using Seattle’s
Time-of-Use (TOU) price from Seattle City Light 4 , and minimizing one week’s electricity bills.
We could see ICRNN capture the long term relationships between control variables and ﬁnal costs, and raise the energy consumption during off-peak price a little, but reduce the energy consumption during peak hours.
Mon
Tue
Wed
Thu
Fri ×
Ground Truth
ICRNN
E ne r g y C on s u m p t
i on [ J ] 0.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23(a)
(b)
E l e c t r i c i t y
P r i c e [ $ / K wH ]
Hours
E.2 C ONTROL C ONSTRAINTS E FFECTS
In Fig.
11 we add one more comparison on the control constraints effects on the ﬁnal control performance by using ICRNN.
Interestingly, with different set point constraints, the ICRNN ﬁnds similar solutions for off-peak electricity usage, which may correspond to necessary energy consumptions, such as lightning and ventilation.
Moreover, when we set no constraints on the system, it would cut down more than 80% of total energy during peak hours.
[Figure]
Wed
[Figure]
Fri