Original: Lyapunov Design for Robust and Efﬁcient Robotic Reinforcement Learning
Revised: Lyapunov Design for Robust and Efﬁcient Robotic Reinforcement Learning

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract: Recent advances in the reinforcement learning (RL) literature haveenabled roboticists to automatically train complex policies in simulated environments.
Revised: Abstract: Recent advances in the reinforcement learning (RL) literature have enabled roboticists to automatically train complex policies in simulated environments.

Original: However, due to the poor sample complexity of these methods, solving reinforcement learning problems using real-world data remains a challenging problem.
Revised: However, due to the poor sample complexity of these methods, solving RL problems using real-world data remains a challenging problem.

Original: This paper introduces a novel cost-shaping method which aims to reducethe number of samples needed to learn a stabilizing controller.
Revised: This paper introduces a novel cost-shaping method which aims to reduce the number of samples needed to learn a stabilizing controller.

Original: The method addsa term involving a control Lyapunov function (CLF) – an ‘energy-like’ functionfrom the model-based control literature – to typical cost formulations.
Revised: The method adds a term involving a Control Lyapunov Function (CLF) – an ‘energy-like’ function from the model-based control literature – to typical cost formulations.

Original: Theoretical results demonstrate the new costs lead to stabilizing controllers when smallerdiscount factors are used, which is well-known to reduce sample complexity.
Revised: Theoretical results demonstrate the new costs lead to stabilizing controllers when smaller discount factors are used, which is well-known to reduce sample complexity.

Original: Moreover, the addition of the CLF term ‘robustiﬁes’ the search for a stabilizingcontroller in the low-data regime.
Revised: Moreover, the addition of the CLF term ‘robustiﬁes’ the search for a stabilizing controller by ensuring that even highly sub-optimal polices will stabilize the system.

Original: We demonstrate our approach with two hardware examples where we learn stabilizing controllers for a cartpole and an A1quadruped with only seconds and a few minutes of ﬁne-tuning data, respectively.
Revised: We demonstrate our approach with two hardware examples where we learn stabilizing controllers for a cartpole and an A1 quadruped with only seconds and a few minutes of ﬁnetuning data, respectively.

Revised: Furthermore, simulation benchmark studies show that obtaining stabilizing policies by optimizing our proposed costs requires orders of magnitude less data compared to standard cost designs.

Original:  Introduction
Revised:  Introduction

Original: Recent years have witnessed many success stories for reinforcement learning (RL)-based controllerswhich are trained using massive amounts of simulation data.
Revised: A key challenge in robotics is reasoning about the long-horizon behavior induced by a control policy.

Original: The primary obstacle for controller design methods which are based on a nominal dynamics model (including traditional ‘model-based’methods) isthe ‘reality gap’ between the simulation environment and the real-world system. Thisdiscrepancy occurs because it is generally impractical to model every facet of a real robotic system. This has led to the use of techniques such as domain randomization [3, 4], which attempt to robustifythe trained controller by adding random perturbations to the simulated dynamics.
Revised: This is because important system properties such as stability are inherently long-horizon phenomena.

Original: Another approachis domain adaptation [5], which trains controllers over a distribution of unknown dynamics param- eters and then attempts to identify the ‘true’ parameters online. In  the last few years, these methods have been key ingredients in a number of impressive hardware results [6, 7, 8, 3, 9, 10]. However, despite these successes, there has been a renewed effort to incorporate data from hard- ware experiments into RL pipelines [11]. These two approaches are not at odds as
Revised: In reinforcement learning (RL), the discount factor implicitly controls how far into the future policy optimization algorithms plan when optimizing the objective speciﬁed by the user.

Original: -tuning’ [12, 13, 14, 15] approachescan be used to improve a policy based on a simulation model. This isan ideal middle ground which enables engineers to incorporate known features of the system intothe model while still allowing model-free methods to improve performance by leveraging complexunknown nonlinearities in the environment.
Revised: Standard approaches to designing objective functions for robotic RL, such as penalizing the distance to a reference trajectory, inherently require a large discount factor to learn control policies which stabilize the system [3, 4].

Original: However, the poor sample complexity of RL methodsstill presents an obstacle, as learning optimal polices can require signiﬁcant amounts of data. One of the key parameters in RL problems is the discount factor , which controls the effect thatlong-horizon behaviors have on the overall cost function. Theoretical and empirical results [16] havedemonstrated that problems with large discount factors are signiﬁcantly more challenging to solve.
Revised: Unfortunately, problems with large discount factors can be extremely difﬁcult to solve, often requiring vast data sets and careful tuning of hyper-parameters [5]. As a number of recent success stories have demonstrated [6, 7, 8, 9, 10, 11], ever-increasing computational resources can be used to solve these problems in simulation and deploy the resulting controllers directly on the real-world system.

Original: The standard approach to synthesizing stabilizing control policies with RL (and also model-basedoptimal control approaches [17]) is  to simply penalize the distance of  the state from the desiredoperating point. Unfortunately, costs of this form typically require using a high discount factorto learn a stabilizing policy [18, 17]. This is because stabilization is an inherently long-horizonbehavior and, when costs of this form are used, the agent must plan over long horizons to see thebeneﬁts of stabilizing the system (rather than simply minimizing control effort).
Revised: However, because it is impractical to model every detail of complex hardware platforms, achieving the best performance will require learning from real-world data.

Original: This paper introduces a cost-shaping framework which aims to reduce the sample complexityof learning stabilizing control policies. Our method adds a term involving an (approximate) control
Revised: This paper introduces a cost-shaping framework which enables users to reliably learn stabilizing control policies with small amounts of real-world data by solving problems with small discount factors.

Original:  Lyapunov function (CLF) for the system to the traditional cost formulations discussed above. CLFsare tools from themodel-based control  literature which are generalized ‘energy functions’ for thesystem.
Revised: Our approach uses Control Lyapunov Functions (CLFs), a standard design tool from the control theory literature [12, 13, 14, 15]. CLFs are ‘energy-like’ functions for the system which reduce the search for a stabilizing controller to a myopic one-step criterion.

Original: Any control law which decreases the value of the CLF at each time-step can be guaranteedto stabilize the system.
Revised: In particular, any controller which decreases the energy of the CLF at each instance of time will stabilize the system.

Original: Thus, CLFs reduce the long-horizon objective of stabilizing the systemto a simple one-step condition. 
Revised: Thus, CLFs reduce the long-horizon objective of stabilizing the system to a simple one-step condition. When a

Original: The CLF term we add to the cost function rewards actions whichdecrease the CLF over time, andthus incentivizes policies which stabilize the system. We providea formal stability analysis which demonstrates that the new cost functions lead to  stabilizing controlpolicies, even when small discount factors are used and the learned policy is highly suboptimal.
Revised: CLF is available and the dynamics are known, constructive techniques from the control literature can be used to synthesize a stabilizing controller. However, when there is uncertainty in the dynamics, it is difﬁcult to guarantee that a controller will always decrease the value of the CLF, or that we have even designed a true CLF for the system.

Original: We illustrate the effectiveness of our cost-shaping technique in three distinct contexts. In ourhardware experiments for the cartpole and A1, we ﬁne-tune control policies which are based on anominal simulation model for the real world system.
Revised: Our approach is to 1) design an approximate CLF for the real- world system using an approximate dynamics model and 2) modify the ‘standard’ choice of cost functions mentioned above by adding a term which incentivizes controllers which decrease the approximate CLF over time.

Original: For the A1 experiment, the nominal controlleris a model-based control architecture [19] and we use a  CLF which is hand-desired around thedesired trotting gait.
Revised: This technique effectively uses the approximate CLF as supervision for reinforcement learning, enabling the user to embed known system structures into the learning process while retaining the ﬂexibility of RL to overcome unknown dynamics. Indeed, as our analysis demonstrates, when our approach is used reinforcement learning algorithms implicitly learn to ‘correct’ the approximate CLF provided by the user. When the candidate CLF is close to being a true CLF for the system (in a sense we make precise below), a stabilizing controller can be efﬁciently learned by solving a problem with a small discount factor. Moreover, the addition of the approximate CLF ‘robustiﬁes’ the search for a stabilizing controller by ensuring that even highly suboptimal policies will stabilize the system.

Original: For the cartpole swing-up task, it is much more difﬁcult to hand-design a CLFfor the desired task. Thus, we ﬁrst train a RL policy for the task using  the soft actor-critic algorithm[20] in simulation.
Revised: Finally, in situations where it is too difﬁcult to design a nominal CLF by hand, we demonstrate how one can be learned using a simulation model and the standard style of RL objective discussed above.

Original: We then use the value function learned during the initial training as a CLF forour ﬁne tuning experiments. We discuss why the learned value function is a reasonable CLF for thesystem in Section 2.2.
Revised: Speciﬁcally, we use the value function learned by the RL algorithm as an approximate CLF for the real-world system. Altogether, beyond accelerating and robustifying RL, our approach also expands the applicability of CLF-based design techniques.

Original: For both of our hardware experiments we rapidly learn stabilizing controllersby solving problems with small discount factors.
Revised: We apply this technique to develop data-efﬁcient ﬁne-tuning strategies, wherein a nominal controller developed using a simulation model is reﬁned with small amounts of real-world data. For the A1 experiment, the nominal controller is a model-based control architecture [16], and we handdesign a CLF using a highly simpliﬁed linearized reduced-order model for the system. Even though this model is very crude, we are nonetheless able to learn a precise tracking controller for this DOF system with only 5 minutes of real-world data.

Original: Finally, with a toy simulation example of aninverted pendulum we demonstrate how our method can still accelerate learning by using moderatevalues of discount factors even when the proposed CLF used in our formulation is very crude.
Revised: For the cartpole swing-up task we used the value function from a simulation-based RL problem as the candidate CLF for the real-world system, using the learning process described above. Our ﬁne-tuning approach then learned a robust swing-up controller after observing only one 10 second trajectory from the real-world system.

Original: 1.1 Related Work
Revised: 1.1 Related Work

Original: We outline how our approach departs from related work; Appendix A contains further discussion.
Revised: We outline how our approach departs from related work; Appendix A contains further discussion.

Original: Discount Factors, Sample Complexity and Reward Shaping:
Revised: Discount Factors, Sample Complexity and Reward Shaping:

Original: It is well-understood that the discount factor has a signiﬁcant effect on the size of the data set that RL algorithms need to achieve adesired level of performance.
Revised: It is well-understood that the discount factor has a signiﬁcant effect on the size of the data set that RL algorithms need to achieve a desired level of performance.

Original: Speciﬁcally, it has been shown in numerous contexts [21, 22, 23, 24]that smaller discount factors lead to problems which can be solved more efﬁciently.
Revised: Speciﬁcally, it has been shown in numerous contexts [17, 18, 19, 20] that smaller discount factors lead to problems which can be solved more efﬁciently.

Original: This has led toa number of works which explicitly treat the discount factor as a parameter which can be used tocontrol the complexity of the problem alongside reward shaping techniques [25, 26, 16, 27, 28, 29].
Revised: This has led to a number of works which explicitly treat the discount factor as a parameter which can be used to control the complexity of the problem alongside reward shaping techniques [21, 22, 5, 23, 24, 25].

Original: Compared to these works, our primary contribution is to demonstrate how CLFs can be combinedwith model-free algorithms to rapidly learn stabilizing controllers for robotic systems.
Revised: Compared to these works, our primary contribution is to demonstrate how CLFs can be combined with model-free algorithms to rapidly learn stabilizing controllers for robotic systems.

Original: Fine-tuning with Real World Data: Recently, there has been much interest in using RL to ﬁnetune policies which have been pre-trained in simulation [12, 13, 14, 15].
Revised: Fine-tuning with Real World Data: Recently, there has been much interest in using RL to ﬁnetune policies which have been pre-trained in simulation [26, 27, 28, 29].

Original: These methods typicallyoptimize the same cost function with a large discount factor in both simulation and on the real robot.
Revised: These methods typically optimize the same cost function with a large discount factor in both simulation and on the real robot.

Original: In contrast, using our cost reshaping techniques, we solve a different problem with a smaller discountfactor on hardware which can be solved more efﬁciently.
Revised: In contrast, using our cost reshaping techniques, we solve a different problem with a smaller discount factor on hardware which can be solved more efﬁciently.

Original: In Appendix D, we show that our methodoutperforms typical ﬁne-tuning approaches under moderate perturbations to the dynamics model.
Revised: In Appendix D, we show that our method outperforms typical ﬁne-tuning approaches under moderate perturbations to the dynamics model.

Original: Learning with Control Lyapunov Functions: A number of recent works have also tried to overcome the reality gapby using data-driven methods to improve CLF-based controllers [30, 31, 32,33, 34, 35].
Revised: Learning with Control Lyapunov Functions: A number of recent works have also tried to overcome the reality gap using data-driven methods to improve CLF-based controllers [30, 31, 32, 33, 34, 35].

Original: While these methods workvery well when a true CLF for the real world system isavailable, our method is more general as we can still efﬁciently learn stabilizing controllers whenonly an approximate CLF is available by modulating the discount factor used to optimize our cost.
Revised: While these methods work well when a true CLF for the real-world system is available, our method is more general as we can still efﬁciently learn stabilizing controllers when only an approximate CLF is available by modulating the discount factor used to optimize our cost.

Original:  Background and Problem Setting
Revised:  Background and Problem Setting

Original: Throughout the paper we will consider deterministic discrete-time systems of the form:
Revised: Throughout the paper we will consider deterministic discrete-time systems of the form:

Original: [Equation]
Revised: [Equation]

Original: [Equation] ) where x k ∈ X ⊂ R n is the state at time k , u k ∈ U ⊂ R m is the input applied to the system at thattime, and F : R n × U → R n is the transition function for the system.
Revised: where x k ∈ X ⊂ R n is the state at time k , u k ∈ U ⊂ X is the input applied to the system at that time, and F : X ×U → R n is the transition function for the system.

Original: This general nonlinear modelis broad enough to cover many important continuous control tasks for robotics.
Revised: This general nonlinear model is broad enough to cover many important continuous control tasks for robotics.

Original: We will let Π denotethe space of all control polices π : X → U for the system.
Revised: We will let Π denote the space of all control polices π : X → U for the system.

Revised: To ease exposition, for our theoretical analysis we will focus on the case where the goal is to stabilize the system to a single point, namely the origin.

Revised: Through our examples we will demonstrate how our cost-shaping technique can be leveraged to achieve more complicated tasks, and in Section 5 we outline a path for extending our theoretical results to these settings in future work.

Original: 2.1 Control Lyapunov Functions
Revised: 2.1 Control Lyapunov Functions

Original: Control Lyapunov functions [36, 37, 38, 39] are ‘energy-like’ functions for the dynamics (1).
Revised: Control Lyapunov Functions [12, 13, 14, 15] are ‘energy-like’ functions for the dynamics (1):

Original: Whena CLF is available it greatly simpliﬁes the search for an asymptotically stabilizing policy for thesystem.

Original: This is because, while asymptotic stability is an inﬁnite-horizon property, CLFs reduce thesearch for a stabilizing policy to a set of greedy one-step criteria.

Original: We review the notion of asymptoticstability in Appendix B for unfamiliar readers.

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: We say that a positive deﬁnite function W : R n → R is a control Lyapunov function(CLF) for (1) if the following condition holds for each x ∈ X\{ 0 } :
Revised: We say that a positive deﬁnite function W : R n → R is a Control Lyapunov Function (CLF) for (1) if the following condition holds for each x ∈ X\{ 0 } :

Original: [Equation]
Revised: [Equation]

Original: The condition (2) ensures that for each x ∈ X there exists a choice of input which decreases the‘energy’ W ( x ) .
Revised: The condition (2) ensures that for each x ∈ X there exists a choice of input which decreases the ‘energy’ W ( x ) .

Original: Any policy which satisﬁes the one-step condition W ( F ( x, π ( x )))
Revised: Any policy which satisﬁes the one-step condition W ( F ( x, π ( x )))

Original: − W ( x ) < 0 canbe guaranteed to asymptotically stabilize the system [40].
Revised: − W ( x ) <can be guaranteed to asymptotically stabilize the system [36] (see Appendix B for background on stability theory).

Original: Given a CLF for the system, model-basedmethods constructively synthesize a controller which satisﬁes this property using either closed-formequations [37] or by solving an online (convex) optimization problem [41, 39] to satisfy (2) .
Revised: Given a CLF for the system, model-based methods constructively synthesize a controller which satisﬁes this property using either closed-form equations [13] or by solving an online (convex) optimization problem [37, 15] to satisfy (2).

Revised: However, when the dynamics are unknown it is difﬁcult to ensure that we have synthesized a ‘true’ CLF for the system.

Original: Remark 1.
Revised: Remark 1.

Original: (Designing Control Lyapunov Functions)
Revised: (Designing Control Lyapunov Functions)

Original: While there is no general procedure for designing CLFs by hand for general nonlinear systems, there do exist constructive procedures fordesigning CLFs for many important classes of robotic systems, such as manipulator arms [38] androbotic walkers [39] using structural properties of the system.
Revised: While there is no general procedure for designing CLFs by hand for general nonlinear systems, there do exist constructive procedures for designing CLFs for many important classes of robotic systems, such as manipulator arms [14] and robotic walkers [15] using structural properties of the system.

Original: However, even in these cases, the

Original: CLF is typically designed ignoring constraints on the inputs and thus is only a local CLF (such as inthe inverted pendulum example we consider in Section 4).

Original: In cases where it is too difﬁcult to designa CLF by hand for the desired task, such as in the cartpole swing-up example we consider in Section4, we demonstrate howto learn a CLF using a simulationenvironment. Remark 2. (Effects of Model Uncertainty) An additional complication arises when there is a discrepancy between the dynamics of the real robot and the model  used to design the CLF.
Revised: Moreover, in our examples we will investigate how a CLF can be learned from a simulation model and how very coarse CLF candidates can be used to accelerate learning a stabilizing controller.

Original: Fortunately,

Original: CLFs are known to be robust under moderate dynamics perturbations [42, 30].

Original: 2.2 Stability of Dynamic Programming and Reinforcement Learning
Revised: 2.2 Stability of Dynamic Programming and Reinforcement Learning

Original: Here we review how ‘typical’ choices of cost functions found in the literature can be used to learnstabilizing controllers.
Revised: Here we investigate how a common class of cost functions found in the literature can be used to learn stabilizing controllers.

Original: These results are from the dynamic-programming literature, and will serveas a point of comparison for the stability and robustness properties of our method.

Original: In particular, weﬁrst consider a running cost (cid:96) :
Revised: In particular, we consider a running cost (cid:96) :

Original: X ×U → R of the form (cid:96) ( x, u ) =
Revised: X × U → R of the form (cid:96) ( x, u ) =

Original: Q ( x )+ R ( u ) , where Q : R n → Ris the state cost and R : U → R is the input.
Revised: Q ( x ) + R ( u ) , where Q : X → R is the state cost and R : U → R is the input cost.

Original: Both Q and R are assumed to be positive deﬁnitefunctions (in practice, both are usually quadratic functions).
Revised: Both Q and R are assumed to be positive deﬁnite (in practice, both are usually quadratic).

Original: Given a policy π ∈ Π , discount factorγ ∈ [0 , 1] , and initial condition x 0 ∈ X  the associated long-run cost is:
Revised: Given a policy π ∈ Π , discount factor γ ∈ [0 , 1] , and initial condition x 0 ∈ X , the associated long-run cost is:

Original: [Equation]
Revised: [Equation]

Original: ) , where V π γ : R n → R ∪ {∞} is the value function associated to π .
Revised: where V πγ : X → R ∪ {∞} is the value function associated to π .

Original: Small discount factors incentivizepolicies which greedily optimize a small number of time steps into the future, while larger discountfactors promote policies which reduce the cost in the long-run.
Revised: Small discount factors incentivize policies which greedily optimize a small number of time-steps into the future, while larger discount factors promote policies which reduce the cost in the long-run.

Original: We say that a policy π ∗ γ ∈ Π isoptimal if it achieves the smallest cost from each x ∈ X :
Revised: We say that a policy π ∗ γ ∈ Π is optimal if it achieves the smallest cost from each x ∈ X :

Original: [Equation]
Revised: [Equation]

Original: where V ∗ γ : X → R ∪ {∞} is the optimal value function .
Revised: where V ∗ γ : X → R ∪ {∞} is the optimal value function .

Original: Together V ∗ γ and π ∗ γ capture the ‘ideal’behavior induced by the cost function (3).
Revised: Together V ∗ γ and π ∗ γ capture the ‘ideal’ behavior induced by the cost function (3).

Revised: It is well-known [17] that the optimal value function will satisfy the Bellman equation:

Revised: [Equation]

Original: Unfortunately, it is impractical to search over the spaceof all policies Π .
Revised: and an optimal policy π ∗ γ will satisfy π ∗ γ [Equation] x ) ∈ [Equation]

Revised: ∈U

Revised: [Equation]

Revised: γV

Revised: [Equation]

Revised: F

Revised: [Equation]

Revised: x, u ))

Revised: + (cid:96) ( x, u ) (cid:3) , ∀ x ∈ X .

Revised: Unfortunately, it is impractical to directly search over Π to ﬁnd a policy which meets these conditions.

Original: This necessitates the use of function approximation schemes (e.g. feed-forwardneural networks) to instead represent a subset of policies ˆΠ ⊂ Π to search over.
Revised: This necessitates the use of function approximation schemes (e.g. feed-forward neural networks) to instead represent a subset of policies ˆΠ ⊂ Π to search over.

Original: Indeed, modern RLapproaches for robotics randomly sample the space of trajectories to optimize problems of the from:
Revised: Indeed, modern RL approaches for robotics randomly sample the space of trajectories to optimize problems of the form:

Original: [Equation]
Revised: [Equation]

Original:  where X 0 is a distribution over initial conditions.
Revised: 5) where X 0 is a distribution over initial conditions.

Original: While this approach enables these methods tooptimize high-dimensional policies, they are data-hungry, can display high-variance and thus frequently return highly sub-optimal policies when data is limited.
Revised: While this approach enables these methods to optimize high-dimensional policies, they are data-hungry, can display high-variance and thus frequently return highly sub-optimal policies when data is limited.

Original: To better understand the effect thatthis has on the stability of learned policies, for each π ∈ ˆΠ  deﬁne the optimality gap :
Revised: To better understand the effect that this has on the stability of learned policies, for each π ∈ ˆΠ and γ ∈ [0 , 1] deﬁne the optimality gap :

Original: [Equation]
Revised: [Equation]

Original: and recall thatfrom (3), for each x ∈ X the policy satiﬁes the difference equation:
Revised: The temporal difference equation [17] dictates that for each x ∈ X the policy satisﬁes:

Original: [Equation]
Revised: [Equation]

Original: From these equations we can obtain:
Revised: From these equations we can obtain:

Original: [Equation]
Revised: [Equation]

Original: (10) where we have ﬁrst rearranged (7), then used V πγ ( x ) =
Revised: where we have ﬁrst rearranged (6), then used V πγ ( x ) =

Original: V ∗ γ ( x ) + (cid:15) πγ
Revised: V ∗ γ ( x ) + (cid:15) πγ

Original: ( x ) , and ﬁnally we have used(cid:96) ( x, π ( x )) ≤ Q ( x ) .
Revised: ( x ) , and ﬁnally we have used (cid:96) ( x, π ( x )) ≥ Q ( x ) .

Original: Inequalities of this sort are the building block for proving the stability ofsuboptimal polices in the dynamic programming literature [17, 18].
Revised: Inequalities of this sort are the building block for proving the stability of suboptimal polices in the dynamic programming literature [4, 3].

Original: Remark 3.
Revised: Remark 2.

Original: (Value Functions as CLFs)
Revised: (Value Functions as CLFs)

Original: By inspecting the cost (3) we see that V πγ is positive deﬁnite(since Q is positive deﬁnite).
Revised: By inspecting the cost (3) we see that V πγ is positive deﬁnite (since Q is positive deﬁnite).

Original: Thus, if the right-hand side of (10) is negative for each x ∈ X \ { 0 } ,this inequality shows that V πγ is a CLF for (1) , and that π is an asymptotically stabilizing controlpolicy.
Revised: Thus, if the right-hand side of (9) is negative for each x ∈ X \ { 0 } , this inequality shows that V πγ is a CLF for (1) , and that π is an asymptotically stabilizing control policy.

Original: In other words, V πγ is a CLF which is implicitly learned during the training process.
Revised: In other words, V πγ is a CLF which is implicitly learned during the training process.

Original: Indeed,many RL algorithms directly learn an estimate of the value function, a fact which we later exploit tolearn a CLF for the cartpole swing up-task in Section 4 using the nominal simulation environment,which is then used for ﬁne-tuning a learned controller with real-world data.
Revised: Indeed, many RL algorithms directly learn an estimate of the value function, a fact which we later exploit to learn a CLF for the cartpole swing up-task in Section 4 using the nominal simulation environment.

Original: Note that the RHS of (10) will only be negative if V ∗ γ ( x ) + (cid:15)
Revised: Note that the right hand side of (9) will only be negative if V ∗ γ ( x ) + (cid:15)

Original: πγ ( x ) < 11 − γ Q ( x )for each x ∈ X .
Revised: πγ ( x ) < 11 − γ Q ( x ) .

Original: Since from (3) we know that V ∗ γ ( x ) > Q ( x ) for each x ∈ X , even the optimal policy (which has nooptimality gap) will only be stabilizing if γ is large enough.
Revised: Since from (3) we know that V ∗ γ ( x ) > Q ( x ) for each x ∈ X , even the optimal policy (which has no optimality gap) will only be stabilizing if γ is large enough.

Original: On the other hand, for a ﬁxed γ ∈ (0 , 1] ,this inequality also explains how sub-optimal a policy can be while maintaining stability.
Revised: On the other hand, for a ﬁxed γ ∈ (0 , 1] , this inequality also quantiﬁes how sub-optimal a policy can be while maintaining stability.

Original: To make these observations more quantitative we make the following assumptions:
Revised: To make these observations more quantitative we make the following assumption:

Original: Assumption 1.
Revised: Assumption 1.

Original: There exists C  ≥ 1 such that V ∗ γ ( x ) ≤ CQ ( x ) for eachγ ∈ [0 , 1] and x ∈ X .
Revised: For each γ ∈ [0 , 1] there exists C γ ≥ 1 such that V ∗ γ ( x ) ≤ C γ Q ( x ) for each x ∈ X .

Original: Assumption 2.

Original: There exists δ > 0 such that the optimality gap for the policy π ∈ ˆΠ satisﬁes thebound (cid:15) πγ ( x ) ≤ δQ ( x ) for each x ∈ X and a desired discount factor γ ∈ (0 , 1] .

Original: Growth conditions of this form are standard in the literature on the stability of approximatedynamic programming [43, 18, 17, 44].
Revised: Growth conditions of this form are standard in the literature on the stability of approximate dynamic programming [38, 3, 4, 39].

Original: Noting that V ∗ γ ( x ) ≤ V ∗ γ [-=
-]
Revised: Note that, because the running cost (cid:96) is non-negative, we have C γ (cid:48) ≤ C γ [-=
-](cid:48)(cid:48) if γ (cid:48) ≤ γ (cid:48)(cid:48) .

Original: ( x ) for each γ ∈ [0 , 1) , theconstant C≥ 1 upper-bounds the ratio between the one-step cost and the optimal undiscountedvalue function.
Revised: In particular, the constant C 1 upper-bounds the ratio between the one-step cost and the optimal undiscounted value function.

Original: This bound is related to how contractive theassociated optimal  policy is [43], as if C is smaller then the optimal state trajectory must converge  to  the origin more rapidly. Assumption2 provides a measure of how suboptimal the chosen policy is when compared to one-step of thestate-cost.
Revised: When C 1 is smaller, the optimal undiscounted policy is more ‘contractive’ and approximate dynamic programming methods converge more rapidly to an optimal solution [38]. Thus, intuitively the constants C γ ≥ 1 will be smaller when the system is easier to stabilize.

Original: The following result is essentially a specialization of the main result from [44]:
Revised: The following result is essentially a specialization of the main result from [39]:

Original: Proposition 1.
Revised: Proposition 1.

Original: Let Assumption 1 hold. Let π ∈ ˆΠ be a policy which satisﬁes Assumption 2 fordiscount factor γ ∈ (0 , 1] .
Revised: Let Assumption 1 hold and let γ ∈ [0 , 1] and π ∈ ˆΠ be ﬁxed. Further assume that there exists δ > 0 such that for each x ∈ X we have i ) (cid:15) πγ ( x ) ≤ δQ ( x ) and ii ) C γ + δ < 11 − γ .

Original: Then, π asymptotically stabilizes (1)if C + δ < 11 − γ .
Revised: Then, π asymptotically stabilizes (1) .

Original: Proof.
Revised: Proof.

Original: Combining Assumptions 1 and 2 with equation (10) yields:
Revised: Combining conditions i ) and ii ) with equation (9) yields:

Original: [Equation]
Revised: [Equation]

Original: Note that if C  + δ < 11 − γ then the right hand side of (11) will be negative deﬁnite, which establishesthat π asymptotically stabilizes the system.
Revised: Thus the RHS of the preceding equation will be negative-deﬁnite if C γ + δ < 11 − γ , which demonstrates the desired result.

Original: Remark 4.
Revised: Remark 3.

Original: (Stability Properties of  Cost Function)
Revised: (Stability Properties of the Cost Function)

Original: In the following section we will derive an analogous result to Proposition 1 for the novel reshaped cost function we propose below.
Revised: In the following section we will derive an analogous result to Proposition 1 for the novel reshaped cost function we propose below.

Original: When comparingthese results we will primarily focus on the effect of the constant C  ≥ 1 (and the equivalent constant for the new setting).
Revised: When comparing these results we will primarily focus on the effect of the constants C γ ≥ 1 (and the equivalent constants for the new setting).

Original: This constant provides an estimate for the size of  discount factor thatis needed to stabilize(1) when the cost (3) is used. In particular,  the discount factor must satisfy11 − γ >  C to guarantee stability.
Revised: The C γ constants can be used to bound how large of a discount factor is need to stabilize the system. In particular, Proposition 1 implies that the optimal policy will stabilize the system for each γ which satisﬁes γ > 1 − 1 C γ .

Original: Theconstant C  also characterizes how ‘robust’ the cost function isto suboptimal policies.
Revised: The C γ constants also characterizes how ‘robust’ the cost function is to suboptimal policies.

Original: In particular, for a ﬁxed discount factor, the policy will stabilize the systemif δ < 11 − γ + C , thus smaller values of  C  permit more suboptimal policies.
Revised: In particular, for a ﬁxed discount factor, the policy will stabilize the system if δ < 11 − γ − C γ . Thus smaller values of the C γ constants permit more suboptimal policies.

Original:  Lyapunov Design for Inﬁnite Horizon Reinforcement Learning
Revised:  Lyapunov Design for Inﬁnite Horizon Reinforcement Learning

Original: Our method uses a  candidate control Lyapunov function W : R n → R for the nonlinear dynamics(1), and reshapes (3) to our proposed new long horizon cost ˜ V π γ : X → R ∪ {∞} :[Equation]
Revised: Our method uses a positive deﬁnite candidate Control Lyapunov Function W : R n → R for the nonlinear dynamics (1), and reshapes (3) to our proposed new long horizon cost ˜ V πγ : X → R ∪{∞} :

Original: [Equation]
Revised: [Equation]

Original: Here the term ‘candidate’ means that we guess that W is a CLF for the system.

Original: As we shall seebelow, our method works best when W is in fact a CLF for the system, but still provides beneﬁtswhen it is only an ‘approximate’ CLF for the system (in a sense we will make precise later).
Revised: As we shall see below, our method works best when W is in fact a CLF for the system, but still provides beneﬁts when it is only an ‘approximate’ CLF for the system (in a sense we will make precise later).

Original: Given the reshaped cost, we then solve the following policy optimization problem:
Revised: For each γ ∈ [0 , 1] the new optimal value function is given by:

Original: [Equation]
Revised: [Equation]

Original: The new cost function (12) includes the amount that W changes at each time step, and thus encourages choices of inputs which decrease W over time. 
Revised: The new cost (10) includes the amount that W changes at each time step, and thus encourages choices of inputs which decrease W over time. In this case, the Bellman equation [17] dictates: [Equation]

Original: For very small discount factors, the new costbiases the policy search in (13) towards policies which myopically decrease the value of W at eachstep.
Revised: [Equation] where ∆ W ( x, u ) : = W ( F ( x, u )) − W ( x ) .

Original: To see this, note that in theextreme case where γ = [-
-]
Revised: To gain some intuition for the approach let us consider the two extremes where γ = 0 and γ = 1 . In the case where γ = [-
-]1 , by inspection we see that ˜ V ∗ 1 = V ∗ 1 − W solves the Bellman equation. Plugging in this solution demonstrates that any optimal policy ˜ π ∗ 1 must satisfy ˜ π ∗ 1 ( x ) ∈ arg min u ∈U [ V ∗ 1 ( F ( x, u )) + (cid:96) ( x, u )] .

Original: we can simplify the cost to:
Revised: This is precisely the optimality condition for the original cost (3) when γ = 1 , and thus the set of optimal policies for the two problems coincide. Thus, in this case, by embedding the CLF in the cost we are effectively using W as a warm-start initial guess for the optimal value function. In the other extreme where γ = 0 , from (12) we see that an optimal policy must satisfy ˜ π

Original: [Equation]
Revised: [Equation]

Original: Since the policy cannot affect the initial condition x 0 ∈ R , the only terms in (14) which affect thepolicy optimization (13) are W ( F ( x, π ( x ))) and R ( π ( x )) .
Revised: ( x )

Original: Thus, optimal policies will disregard theoriginal state cost Q , and myopically balance decreasing W against the input cost R . On the otherhand, when γ is large, optimal polices can increase the value of W in the short run, if it means that
Revised: [Equation]

Original:  W and Q can both be decreased in the long-run. The new optimal value function is given by:
Revised: min u ∈U (cid:2) ∆ W ( x, u ) + (cid:96) ( x

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: u ) (cid:3) . Thus, when γ = 0 the optimal policy attempts to greedily decrease the value of the candidate CLF and the one-step cost on the input. As we shall see below, when intermediate discount factors are used, optimal policies may instead decrease the value of W over the course of several steps.

Original: and each policy mustnow satisfy the new difference equation:
Revised: Using the new cost function (10), each policy must satisfy the new difference equation:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: In our stability analysis we will use the following composite function as a candidate CLF for (1):
Revised: In our stability analysis, we will use the following composite function as a candidate CLF for (1):

Original: [Equation]
Revised: [Equation]

Revised: We provide an interpretation of this curious candidate CLF in Remark 4 below, but ﬁrst perform an initial analysis similar to the one presented in the previous section.

Original: Deﬁning for each π ∈ ˆΠ  and x ∈ X the new optimality gap:
Revised: Deﬁning for each π ∈ ˆΠ , γ ∈ [0 , 1] and x ∈ X the new optimality gap:

Original: [Equation]
Revised: [Equation]

Original: and following steps analogous to those taken in (8)-(10), we can obtain the following:
Revised: and following steps analogous to those taken in (7)-(9), we can obtain the following:

Original: [Equation]
Revised: [Equation]

Original: Similar to the analysis in the previous section, we will aim to understand when the right hand side of(21) is negative, as this will characterize when π stabilizes the system.
Revised: Similar to the analysis in the previous section, we will aim to understand when the right-hand side of (18) is negative, as this will characterize when π stabilizes the system.

Original: One key difference betweenthe inequality (10) and (21) is that, while the original value function V ∗ γ is necessarily positivedeﬁnite,
Revised: One key difference between the inequalities (9) and (18) is that, while the original value function V ∗ γ is necessarily positive deﬁnite, ˜

Original: ˜ V ∗ γ can actually take on negative values since the addition of the CLF term allows thenew running cost in (12) to be negative.
Revised: V ∗ γ can actually take on negative values since the addition of the CLF term allows the new running cost in (10) to be negative.

Original: As we shall see, this forms the basis for the stability androbustness properties our cost formulation enjoys when W is designed properly.
Revised: As we shall see, this forms the basis for the stability and robustness properties our cost formulation enjoys when W is designed properly.

Original: Remark 5.
Revised: Remark 4.

Original: (Learning Corrections to W)
Revised: (Learning Corrections to W)

Original: Even though ˜ V πγ can take on negative values, in Lemma2 in the appendix we demonstrate that ˜ V π γ = W + ˜

Original: V πγ is a positive deﬁnite function.

Original: Thus, whenthe right hand side of (21) is negative for each x ∈ X \ { 0 } , inequality (21) demonstrates that  V πγis in fact a CLF for (1) and that π stabilizes the system.
Revised: When the right hand side of (18) is negative for each x ∈ X \ { 0 } , inequality (18) demonstrates that ˜ V π γ is in fact a CLF for (1) and that π stabilizes the system (see Theorem 1).

Original: We can think of W as an ‘initial guess’for a CLF for the system, while γV πγ is a ‘correction’ to W that is implicitly made by a learnedpolicy π .
Revised: We can think of W as an ‘initial guess’ for a CLF for the system, while γ ˜ V πγ is a ‘correction’ to W that is implicitly made by a learned policy π .

Original: Roughly speaking, the larger the discount factor, the larger this correction.
Revised: Roughly speaking, the larger the discount factor, the larger this correction.

Original: Thus theuser can trade-off how much the learned policy is able to correct the candidate CLF W against theadditional complexity of solving a problem with a higher discount factor, depending on how good ofa CLF candidateW is believed to be for the real robot.
Revised: Thus, the user can trade-off how much the learned policy is able to correct the candidate CLF W against the additional complexity of solving a problem with a higher discount factor, depending on how ‘good’ they believe the CLF candidate to be.

Original: We ﬁrst state a general stability result for suboptimal policies associated to the new cost, andthen discuss how the choice of W affects the stability of suboptimal control policies:
Revised: We ﬁrst state a general stability result for suboptimal policies associated to the new cost, and then discuss how the choice of W affects the stability of suboptimal control policies:

Original: Assumption 3.
Revised: Assumption 2.

Original: There exists ˜ C  ∈ R such that  V ∗ γ ( x ) ≤ ˜ CQ ( x ) for eachγ ∈ [0 , 1] and x ∈ X .
Revised: For each γ ∈ [0 , 1] there exists ˜ C γ ∈ R such that ˜ V ∗ γ ( x ) ≤ ˜ C γ Q ( x ) for each x ∈ X .

Original: Assumption 4.
Revised: Because the reshaped one-step cost W ( F ( x, u )) − W ( x ) + (cid:96) ( x, u ) can take on negative values, so can the ˜ C γ constants. Moreover, in this case it is possibe to have ˜ C γ (cid:48) ≥ ˜ C γ

Original: There exists ˜ δ > 0 such that the optimality gap for the policy π ∈ ˆΠ satisﬁes thebound ˜ (cid:15) πγ ( x ) ≤˜ δQ ( x ) for each x ∈ X and a desired discount factor γ ∈ (0 , 1] . 
Revised: (cid:48)(cid:48) when γ (cid:48) ≤ γ (cid:48)(cid:48) . This is because when larger discount factors are used, the optimal policy can beneﬁt from decreasing W further into the future.

Original: These Assumptions are entirely analogous to Assumptions 1 and 2, except in Assumption 3 theupper bound on the value function is allowed to take on negative values.
Revised: The following stability result is analogous to Proposition 1:

Original: Theorem 1.
Revised: Theorem 1.

Original: Let Assumption 3 hold. Let π ∈ ˆΠ be a policy which satisﬁes Assumption 4 for discountfactor γ ∈ (0 , 1] . Then π asymptotically stabilizes (1) if ˜C
Revised: Let Assumption 2 hold and let γ ∈ [0 , 1] and π ∈ ˆΠ be ﬁxed. Further assume that there exists ˜ δ > 0 such that for each x ∈ X we have i ) ˜ (cid:15) πγ ( x ) ≤ δQ ( x ) and ii ) ˜

Original:  + ˜ δ < 11 − γ .
Revised: C γ + ˜ δ < 11 − γ .

Original:  The proof is conceptually similar to the proof of Proposition 1, and follows from the inequality(21) and Assumptions 3 and 4; we delegate the proof to Appendix C for brevity.
Revised: Then, π asymptotically stabilizes (1) . The proof is conceptually similar to the proof of Proposition 1; we delegate the proof to Appendix C for brevity.

Original: Indeed, note thatthe conditions for stability under the new cost are essentially identical to those for the previous costin Proposition 1.
Revised: Indeed, note that the conditions for stability under the new cost are essentially identical to those for the previous cost in Proposition 1.

Original: As alluded to in Remark 4, we will primarily focus on comparing how large theconstants C ≥ 1 and ˜ C  ∈ R are for the two problems, as they control the discount factor requiredto learn a stabilizing policy and also the ‘robustness’ of the cost.
Revised: As alluded to in Remark 3, we will primarily focus on comparing how large the constants C γ ≥and ˜ C γ ∈ R are for the two problems, as they control the discount factor required to learn a stabilizing policy and also the ‘robustness’ of the cost to suboptimal controllers.

Original: The following result provides acondition on W which ensures that ˜ C ≤ 0 .
Revised: We provide two characterizations which ensure that ˜ C γ < C γ .

Original: This condition is  from the model-predictive controlliterature [45, 46], which use CLFs  as terminal costs.
Revised: The ﬁrst condition is taken from the model-predictive control literature [40, 41], where CLFs are used as terminal costs for ﬁnite-horizon prediction problems.

Original: Proof  can be found in Appendix C.
Revised: Proof of the following result can be found in Appendix C:

Original: Lemma 1.
Revised: Lemma 1.

Original: Supposethere exists M ≥ 0 such that for each x ∈ X the following condition holds:
Revised: Suppose that for each x ∈ X the following condition holds:

Original: [Equation]
Revised: [Equation]

Original: Then Assumption 3 is satisﬁed with constant ˜ C = − M .
Revised: Then Assumption 2 is satisﬁed with constant ˜ C γ ≤ 0 .

Revised: The hypothesis of Lemma 1 implies that i ) W is a true CLF for the system and ii ) W dominates the running cost (cid:96) , in the sense that W can be decreased more rapidly than (cid:96) accumulates.

Revised: Effectively, this condition implies that it is advantageous for polices to myopically decrease W at each time step.

Revised: Consequently, when this condition holds optimal polcies associated to the reshaped costs (10) will stabilize the system for any choice of discount factor.

Original: Remark 6.
Revised: The following deﬁnition generalizes this condition to cases where W may not be a true CLF for the system but can be decreased over several time-steps: Deﬁnition 2. We say that the candidate CLF W ¯ γ -dominates the running cost (cid:96) if for each discount factor ¯ γ ≤ γ ≤ 1 and x ∈ X

Original: (Robustness of reshaped cost)
Revised: we have ˜ V ∗ γ ( x ) ≤ V ∗ γ ( x ) . The condition in (2) effectively provides a way of characterizing how ‘close’ W is to being a true CLF for the real-world system. In particular, the larger ¯ γ the further into the future RL algorithms must look to see the beneﬁts of decreasing W .

Original: The condition in Lemma 1 implies that  Wis a CLF forthe system which ‘dominates’ the original running cost (cid:96) , in the sense that at each time step W canbe decreased faster than the running cost (cid:96) accumulates.
Revised: Our previous discussion, which showed that ˜ V ∗ 1 = V ∗ 1 − W , demonstrates that every candidate CLF 1 -dominates the cost. Moreover, clearly W can only 0 -dominate the original cost if it is a CLF for the system.

Original: In this case, we will have ˜C <
Revised: While this condition is more difﬁcult to verify for intermediate values of ¯ γ , it provides qualitative insight into how even approximate CLFs for the system can still make it easier to obtain stabilizing controllers. Remark 5. (Robustness of reshaped cost) When the condition of Lemma 1 is satisﬁed we will have ˜

Original: C implyingthe new cost enjoys the desirable  properties discussed above.Note that the constant M > 0 in
Revised: C γ ≤ 0 < C γ , implying the new cost enjoys the desirable robustness properties discussed above.

Original: Lemma 1 can be increased by putting more weight on W in the cost (12) .

Original: In cases where W isonly an ‘approximate’ CLF for the system, meaning that W cannot be decreased at each time step,but can instead be decreased over a window of a small number of time steps, V ∗ γ may still take onnegative values and contribute to the right hand side of (21) being negative. However, in this casemoderate values of the discount factor may be needed to see these beneﬁts, as policy optimizationalgorithms must implicitly plan further into the future  to see the beneﬁts of ‘going down’ W . Theinverted pendulum example in Section 4 illustrates this point.
Revised: When W satisﬁes the ‘approximate CLF’ condition in Deﬁnition (2) , it will only enjoy these beneﬁts when the discount factor is large enough. We leave it as a matter for future work to provide quantitative estimates for the ˜ C γ constants in these regimes, and to provide sufﬁcient conditions which ensure W ¯ γ -dominates the running cost.

Original:  Examples and Practical Implementations
Revised:  Examples and Practical Implementations

Original: We highlight the main results for each of our examples, but leave certain details  to Appendix D. 
Revised: We summarize the main results for each of our examples, but leave most details and plots to Appendix D. In every experiment we report, the soft actor-critic algorithm (SAC) [42] is used as the learning algorithm to optimize the various reward structures we investigate. Velocity Tracking for A1 Quadruped: We apply our approach to train a neural network controller which augments and improves a nominal model-based controller [16] for a quadruped robot using real-world data. As illustrated by the pink curve in Fig. 2 (left), the nominal controller fails to accurately track desired velocities speciﬁed by the user.

Original: Fine-tuning a Learned Policy for Cartpole Swing-Up: We ﬁrst ﬁne-tune a swing-up controller forthe Quanser cartpole system [1] using real world data and  an initial policy which was pre-trained in simulation but that does not translate well to the real system.
Revised: We design a CLF around the desired gait using a linearized reduced-order model for the system. We then collect rollouts of 10 s on the robot hardware with randomly chosen desired velocity proﬁles, and solve an RL problem using our cost and a discount factor γ = 0 . Our approach is able to learn a policy which signiﬁcantly improves the tracking performance of the nominal controller within 5 minutes (30 episodes) of hardware data, as shown in Fig. 2 (left).

Revised: A video of these results can be found in https://youtu.be/ l7kBfitE5n8 , and more details are provided in Appendix D. Furthermore, in Fig.

Revised: 2 (right) we benchmark our approach in simulation against an RL agent trained with a ‘standard’ cost which penalizes the squared error with respect to the desired velocity.

Revised: As this ﬁgure demonstrates, our method is able to rapidly decrease the average tracking error in only around 2 thousand steps from the environment.

Revised: In contrast, the benchmark approach is only able to reach this level of performance for the ﬁrst time after around 24 thousand steps.

Revised: A1 Quadruped Walking with an Unknown Load: We attach an un-modeled load to the A quadruped, that is equivalent to one-third the mass of the robot.

Revised: Fine-tuning on hardware the same base controller from the previous set-up where the CLF is designed to stabilize to the target gait, our approach is able to signiﬁcantly decrease the tracking error to about one-third its nominal value with only one minute of data collected on the robot hardware as illustrated in Fig.

Revised: 3 in Appendix D.

Revised: Additionally, in Appendix D, we run a simulated benchmark comparison and verify that our method clearly out-performs the ‘standard’ cost baseline for this task.

Revised: Fine-tuning a Learned Policy for Cartpole Swing-Up: We ﬁne-tune a swing-up controller for the

Revised: Quanser cartpole system [1] using real-world data and an initial policy which was pre-trained in simulation but that does not translate well to the real system.

Original: Due to the underactuated natureof the system, synthesizing a CLF, even when input constraints are ignored, is challenging.
Revised: Due to the underactuated nature of the system, synthesizing a CLF by hand is challenging.

Original: Thus,as alluded to previously, we use a ‘typical’ cost function of the form (3) and a discount factor ofγ = 0 .
Revised: Thus, as alluded to previously, we use a ‘typical’ cost function of the form (3) and a discount factor of γ = 0 .

Original: 999 to learn a stabilizing neural network policy π φ for a simulation model of the system usingthe SAC algorithm [47].
Revised: 999 to learn a stabilizing neural network policy π φ for a simulation model of the system.

Original: Given the discussion in Section 2.2, the value function V θ associated withthe learned policy will be a CLF for the simulation environment, and we use this as the candidate CLF ( W = V θ ) in our reward reshaping formulation (12) for ﬁne-tuning on hardware.
Revised: Given the discussion in Remark 2, we use the value function V θ associated with the simulation-based policy as the candidate CLF ( W = V θ ) for our reward reshaping formulation (10).

Original: In particular,when improving the simulation-based policy π φ with real world data, we keep the parameters ofthis network ﬁxed and learn an additional smaller policy π ψ (so that the overall control action isproduced by π φ + π ψ ) using our proposed CLF-based cost formulation.
Revised: When improving the simulation-based policy π φ with real-world data, we keep the parameters of this network ﬁxed and learn an additional smaller policy π ψ (so that the overall control action is produced by π φ + π ψ ) using our proposed CLF-based cost formulation.

Original: We solve the reshapedproblem with a discount factor γ = 

Revised: We solve the reshaped problem with a discount factor γ =

Original: and collect rollouts of 10 s on hardware.
Revised: and collect rollouts of 10 s on hardware.

Original: Our CLF-basedﬁne-tuning approach is able to successfully complete the swing-up task after collecting data fromjust one rollout, however the behavior of the policy near the top is not smooth and sometimes fails.
Revised: Our CLF-based ﬁne-tuning approach is able to successfully complete the swing-up task after collecting data from just one rollout.

Original: After collecting data from an additional rollout, the controller ismuch more reliable and robust torecover from several pushes, as demonstrated in the supplementary video.
Revised: After collecting data from an additional rollout, the controller is reliable and robust enough to recover from several pushes.

Original: A successful swing-upmotion is shown in Figure 2. Finally, in Appendix D, we compare our method with a typical ﬁnetuning approach which uses the same cost in both phases. We show that our method outperformsthe baseline under moderate perturbations to  the dynamics model.
Revised: A video of these experiments can be found in https://youtu.be/l7kBfitE5n8 , and more details and plots of the results are provided in Appendix D. Furthermore, in Appendix D we provide a simulation study comparing a standard ﬁne-tuning approach to our method, showing that our approach is able to more rapidly learn a reliable swing-up policy than the baseline and also achieves a higher reward.

Original: Improving Velocity Tracking for a Quadruped Locomotion Controller: Next, we ﬁne-tune amodel-based trotting controller [19] for a quadruped robot walking at different speeds using a CLFwhich is designed around the desired gait.
Revised: Fine-tuning a Bipedal Walking Controller in Simulation: We also apply our design methodology to ﬁne-tune a model-based walking controller [15] for a bipedal robot with large amounts of dynamics uncertainty.

Original: The tracking performance of this controller dependsheavily on the chosen feedback gains. As illustrated by  the pink curve in Fig.
Revised: Model uncertainty is introduced by doubling the mass of each link of the robot.

Original: 3 (left), large steady- state errors persist when the controller is not well tuned. Furthermore, when the space of controlparameters is large, tuning them by hand may not be straightforward. To address these limitations,we augment a policy π ψ , to our nominal controller. To learn the parameters ψ ,
Revised: The nominal controller fails to stabilize the gait and falls within a few steps.

Original:  we ﬁrst designa candidate CLF ( W ) based on a nominal linearized model of the robot, which is used to deﬁneour reward function. We then collect rollouts of 10 s on the robot hardware withrandomly chosendesired velocity proﬁles, and a discount factor γ = 0 .
Revised: To apply our method, we design a CLF around the target gait as in [15] to be used in our reward formulation. As a benchmark comparison, we also train policies with a reward which penalizes the distance to the target motion (no CLF term), as is most commonly done in RL approaches for bipedal locomotion which use target gaits in the reward [10].

Original: Our approach is able tolearn a policywhich signiﬁcantly improves the  tracking performance of the nominal controller within 5 minutes(30 episodes) of hardware data.
Revised: Our approach is able to signiﬁcantly reduce the average tracking error per episode after only 40000 steps of the environment (corresponding to 40 seconds of data), while the baseline does not reach a similar level of performance even after 1.2 million steps, as illustrated in Fig.

Original: More details ofthis experiment are provided in Appendix D.
Revised: 7 of Appendix D.

Original: Inverted Pendulum with Input Constraints:
Revised: Inverted Pendulum with Input Constraints:

Original: Our ﬁnal example demonstrates the utility of ourmethod even when W is a crude guess for a CLF for the system, through the use of moderatediscount factors.
Revised: Our ﬁnal example demonstrates the utility of our method even when W is a crude guess for a CLF for the system, through the use of moderate discount factors.

Original: We illustrate this for a simple inverted pendulum  by varying the magnitude of theinput constraints for the system.
Revised: We illustrate this for a simple inverted pendulum simulator by varying the magnitude of the input constraints for the system.

Original: We use the procedure from [39] to design a candidate CLF for thesystem.
Revised: We use the procedure from [15] to design a candidate CLF for the system.

Original: Like many CLF design techniques, this approach assumes there are no input constraints,and guides the pendulum to swing directly to the top.
Revised: Like many CLF design techniques, this approach assumes there are no input constraints and encourages the pendulum to swing directly up.

Original: However, when the input constraints are toostrict, this motion is infeasible and multiple pumps of the arm are required to swing upright. Thus, W is only a local CLF for the system, meaning it satisﬁes the CLF condition (2) only near theupright position. In the following simulations, we consider inputs bounds of the form | u | ≤ M for
Revised: As the input constraints are tightened, W becomes a poorer candidate CLF, as there is not enough actuation authority to decrease W at each time step.

Original: M ∈ { 20 , 7 , 4 } . For M = 20 the system has the control authority to swing directly to the top,while for M = { 7 , 4 } multiple pumping motions are required to gain enough momentum. Using thecandidate CLF, we ﬁrst train a stabilizing controller using a ‘typical’ cost function of the form (3),and then train a controller using the reshaped cost (12). For both forms of cost function, we sweepacross different values of discount factor to approximately determine the smallest discount factorthat can learn a stabilizing controller for each cost.
Revised: Even in this case, in line with the discussion of Remark 5, if a proper discount factor is used, the addition of the candidate CLF in the reward enables our method to rapidly learn a stabilizing controller for each setting of the input bound.

Original: In Figure 4, we display the training curves forthe smallest discount factor which lead to a stabilizing controller (see Appendix D for details on howwe determined if a controller is stabilizing). On each of the training curves the black dot denotes theﬁrst training epoch at which a stabilizing controller was obtained. As the plots clearly demonstrate,the addition of the CLF enables our method to more rapidly learn a stabilizing controller, but theeffect is more pronounced when W is a better candidate CLF, namely when the input bound islarger. Moreover, in line with the presented theory, these results show that our method is able tomore efﬁciently learn stabilizing policies with smaller discount factors.
Revised: These results are presented in Appendix D.

Original:  Discussion and Limitations
Revised:  Discussion and Limitations

Original: As we have mentioned previously, our approach has several limitations.
Revised: As we have mentioned previously, our approach has several limitations.

Original: The cost shaping technique we introduce in Section 3 only provides beneﬁts when W is in-fact a reasonable guess fora CLF for the true system.
Revised: The cost-shaping technique we introduce in Section 3 only provides beneﬁts when W is in-fact a reasonable guess for a CLF for the true system.

Original: This requires that the user have a dynamics model which captures theprimary features of the environments which affect the structure of CLFs for the system.
Revised: This requires that the user has a dynamics model which captures the primary features of the environments which affect the structure of CLFs for the system.

Original: While thecartpole simulations we provide in the Appendix D provide some intuition for when this will bethe case, further research is needed to better understand in what scenarios we can see signiﬁcantbeneﬁts from our method.
Revised: While the cart-pole simulations we provide in the Appendix D provide some intuition for when this will be the case, further research is needed to better understand in what scenarios we can see signiﬁcant beneﬁts from our method.

Original: Nonetheless, our two hardware experiments provide encouraging initialresults which indicate that our method can rapidly learn stabilizing controllers using CLFs whichare constructed using a nominal dynamics model.
Revised: Nonetheless, our two hardware experiments provide encouraging initial results which indicate that our method can rapidly learn stabilizing controllers using CLFs which are constructed using a nominal dynamics model.

Original: More broadly, there are many exciting avenuesfor further incorporating Lyapunov design techniques with RL, especially ofﬂine learning [11].
Revised: More broadly, there are many exciting avenues for further incorporating Lyapunov design techniques with RL, especially ofﬂine learning [43].