Fast Convergence for Unstable Reinforcement Learning Problems by Logarithmic Mapping
Abstract
For many of the reinforcement learning applications, the system is assumed to be inherently stable and with bounded reward, state and action space.
These are key requirements for the optimization convergence of classical reinforcement learning reward function with discount factors.
Unfortunately, these assumptions do not hold true for many real world problems such as an unstable linear–quadratic regulator (LQR) 1 .
In this work, we propose new methods to stabilize and speed up the convergence of unstable reinforcement learning problems with the policy gradient methods.
We provide theoretical insights on the efficiency of our methods.
In practice, we achieve good experimental results over multiple examples where the vanilla methods mostly fail to converge due to system instability.
1. Introduction
One of the mainstream methods to solve an RL problem is policy optimization via gradient descent.
However, the convergence of the policy optimization algorithm heavily relies on an unapparent yet critical assumption of the system dynamics itself: stability 2 .
In addition, to ensure the The intrinsic instability brings challenges for optimization and this paper aims to address it. For formal unstable RL definition, please refer to Appendix A convergence of policy optimization, we also require the
Lipschitz property of the cost function and its gradient.
In fact, in many of the existing RL benchmark examples such as OpenAI’s classical control environments, the state space/actions/costs are clipped to ensure that the policy would never move to extreme conditions and costs/states are bounded, in order to reduce the error derivatives (Mnih et al., 2015).
Unfortunately, similar formulations are not directly applicable to unstable systems, such as a LQR where the spectral radius of the state matrix is outside the unit circle, as the standard policy gradient based methods are likely to fail.
To address this issue and with the aim to enable and speed up the convergence of policy gradient methods for unstable RL problems, in this work we propose a logarithmic mapping method on loss functions supported by rigorous theoretical proof and experimental results.
2. Background and Related Work 2.1. Model-free Reinforcement Learning
Distinguished by directly modelling the system dynamics/environments or not, reinforcement learning methods could be categorized into “model-based” and “model-free” approach (Arulkumaran et al., 2017).
In this work, we mainly focus on the latter where the agent learns the policy by directly interacting with the system output such as rewards/costs, bypassing the inference of the underlying dynamics.
By using a model-free approach, we unify the stability sources and only consider small policy variance that may show dramatically different rewards under the unstable system.
2.2. Value Function Mapping for Reinforcement Learning
In reinforcement learning, there are a few previous work mapping the value function to another space for various purposes.
Van Seijen et al.
(2019) used Logarithmic Qlearning to revise the action gaps due to low discount factors.
(Fatemi & Tavakoli, 2022) decomposes the value function to a linear combination of a class of mapping functions to facilitate the learning process.
2.3. LQR Problem
For a discrete-time linear system, its state equation is represented by:
[Equation]
where x t ∈ R n and u t ∈ R m denote the system state and control action at time step t , A ∈ R n × n and B ∈ R n × m are the system transition matrices.
The feedback gain is parameterized by matrix K ∈ R m × n s.t.
u t = − Kx t .
The intermediate cost function is in the quadratic form of state x t and control u t , where Q ∈ R n × n and R ∈ R m × m are given positive definite matrices to parameterize the quadratic cost.
The optimal control problem can be formulated as minimize K
E x 0 ∼ P {+
+}
(cid:104)(cid:80)
Tt =0 x ⊤
[Equation]
Qx
[Equation]
+ u ⊤ t Ru t (cid:105) , where, x t =
( A − BK ) t x 0 , u t = − Kx t and P 0 is the distribution of initial condition x 0 .
Note that if T → ∞ , then the problem is called infinite horizon LQR, otherwise it is called finite horizon LQR.
According to (Abbeel, 2012), a T -time-step system is controllable if we can reach any target state x ∗ from any initial state x 0 .
In this paper, we assume the LQR is controllable.
2.4. Convergence of Policy Gradient Methods for LQR Problems
Fazel et al.
(2018) was the first to achieve the global convergence of policy gradient methods for infinite-horizon LQR problems.
Bhandari & Russo (2019) extends the LQR setup to a more general class of control policies.
Perdomo et al.
(2021) proposes to stabilize a dynamical system with a discounted annealing algorithm by gradually increasing discount factor γ dependent on the system and current policy.
In the scenario of finite horizon LQR and stochastic noise, Hambly et al.
(2021) provides a global linear convergence guarantee.
Tu & Recht (2018) studied Least-Squares
Temporal Difference (LSTD) method on LQR and number of samples needed for LSTD estimator of value function.
Nevertheless, all the above work require the assumption of the system to be stable under the policy throughout optimization, or equivalently, A − BK has a spectral radius less than 1.
Unfortunately, with an unstable A and random initialization of policy K , this assumption is mostly invalid.
In contrast, our proposed algorithm target at a finite horizon setup and could still perform well without this assumption at all.
3. Proposed Methods 3.1. Finite Horizon LQR
Let C K,T =
E x 0 ∼ D (cid:104)(cid:80) Tt =
x
⊤ t
Qx t + u ⊤ t
Ru t (cid:105) be the expected cost of trajectory for T time steps.
For unstable LQR in the infinite horizon, the cost function is not traceable since C K,T → ∞ when T → ∞ .
To bypass this barrier, we focus on a finite horizon case considering the cost for first T steps.
The problem formulation is: min K C K,T s.t. x t +1 =
Ax t + Bu t , u t = − Kx t .
In practice, infinite horizon cases are also approximated by finite step trajectories (Fazel et al., 2018) so that the implementations are identical.
3.2. Finite Horizon Unstable RL
Road- map for convergence rate analysis:
In this section, we provide a theoretical view of convergence rate bound for unstable RL problems under the vanilla setting and our proposed method.
First, we formulate a Markov decision process (MDP) problem with cost formulation in
Assumption 3.1, where the cost function is allowed to exponentially grow against time t with some base number ϕ .
We also assume the Lipschitz property (Assumption 3.4) and local strong convexity (Assumption 3.5) of such ϕ against model parameter θ .
For the optimization process, we view it as a dynamical system, translating the necessary condition for monotonic decreasing as bounding the updating step by the inverse of the spectral norm of the optimization Hessian matrix.
In order to satisfy such condition for convergence, the learning rate should be bounded and therefore the convergence rate is limited.
The final results are summarized in Theorem 3.8 for vanilla policy gradient and in Theorem 3.10 for our proposed logarithmic mapping.
We defer the proofs and other few supportive theorems and lemmas to Appendix G.
3.2.1. P ROBLEM F ORMULATION AND A SSUMPTIONS
To formulate the problem, consider a discrete-time continuous MDP ⟨S , A , P , C⟩ , where S is the continuous state space,
A is the continuous action space, P ( s t +1 | s t , a ) is the transition probability,
c t ( s, a ) is the immediate cost at time step t and s 0 is the initial condition.
Assume that the cost is upper bounded by a polynomial of time step, s.t., | c t ( s, a ) | ≤ DC t , with positive constants D > 0 and C > 0 .
The target is to find an optimal policy to decrease the accumulated cost.
When C ≤ 1 , the cost is bounded by D independent of t and the system is I/O stable with bounded output being the common setup for RL problems.
In this work, we consider a more general setting with possibility that C > 1 .
We formulate the problem into finite time horizon of step
T ∈ Z + , with accumulated cost v T ( s, θ ) as a function of initial state s ∈ R n and policy parameter θ ∈ R d .
[Equation]
where V T ( θ ) is the expected cost over the initial state distribution.
For a vanilla policy gradient method, we have the following update step:
[Equation]
with η > 0 .
The Hessian matrix is denoted as:
J T ( θ ) = ∇ 2 θ V T ( θ ) .
Denote ρ max ( A ) = max {| λ | : λ is an eigenvalue of A } as the spectral radius of the state matrix A .
Assumption 3.1.
Assume the step cost can be parameterized by basis function E [
[Equation] [Equation] ϕ
[Equation]
θ ) t , where d k > 0 and 0 ≤ C < ϕ k ( θ ) ≤ C .
Remark 3.2 .
The motivation of such a basis function is inspired by the departing output trajectories of unstable systems.
Taking the finite horizon LQR example in Section 3.1, the step cost {+x ⊤ t Qx t + u ⊤ t Ru t can be rewritten as x ⊤ 
+}
( A − BK ) t ⊤ ( Q + K ⊤ RK )( A − BK ) t x 0 , which is bounded by ∥ x 0 ∥ 2 ∥ Q + K ⊤ RK ∥∥ A − BK ∥ 2 t .
The ∥ A − BK ∥ 2 term corresponds to ϕ in Assumption 3.1 with exponential growth with time step t and other terms remain positive constant.
Therefore, the finite horizon LQR cost can be formulated into Assumption 3.1 regardless of stability.
The formulation is also valid for optimal control problems with polynomial cost functions or other unstable RL problems with exponentially growing cost as a function of time.
Remark 3.3 .
Despite the basis function ϕ k ( θ ) plays an important role in optimization and determines the maximum step size (will be shown in Lemma G.1), their actual value and gradient properties might not be calculated or even learnable from practice.
The introduction of such decomposition is used for theoretical analysis.
In experiments, we select learning rates as hyperparameter for adaptation to different ϕ k ( θ ) .
Assumption 3.4.
Assume ϕ k
( θ ) is twice differentiable with
Lipschitz constant L 2 and the gradient of ϕ k ( θ ) is also
Lipschitz continuous with L 1 , s.t., ∥∇ ϕ k ( θ
[Equation]
, ∥
[Equation]
k
[Equation]
θ 2 ∥ .
for L 1 , L 2 ∈ R + , θ 1 , θ 2 ∈ R d .
Assumption 3.5.
Assume local α -strong convexity of ϕ k ( θ ) :
ϕ k ( θ 1 + θ 2 ) ≥ ϕ k ( θ 1 ) + θ ⊤ 2 ∇ θ ϕ k ( θ 1 ) + α 2 ∥ θ 2 ∥ 2 .
for all the θ 1 , θ 2 ∈ A and θ ∗ = arg min θ ∈A ϕ k
[Equation]
Remark 3.6 .
The strong convexity assumption is indeed a “strong” one for many of the unstable RL examples and policy basis functions.
The purpose of such assumption is to pave the path for convergence rate analysis.
Like many other popular optimization methods in ML field such as gradient descent, we do not guarantee the algorithm’s convergence to the global optimal.
In practice, the stochastic gradient method could find a near-optimal result.
Besides, we also assume that all the ϕ k ( θ ) reach local optimal with the same θ ∗ , since the actual cost function can be decomposed into more basis functions without the loss of generality.
3.2.2. C ONVERGENCE R ATE D ERIVATION
Theorem 3.7.
Suppose
V T ( θ ) satisfies
Assumption 3.1 and Assumption 3.4, using the vanilla gradient descent algorithm from Equation (2) , if
[Equation]
( θ ) t − 2 ] , then η < 2 /ρ max ( J T ( θ )) is satisfied for monotonic decrease of value function.
By Theorem 3.7, we claim that to stabilize the convergence for the finite horizon unstable problem, the learning rate η needs to be smaller than the inverse of polynomial term of max ( ϕ k ( θ )) , otherwise the optimization is likely to diverge.
Theorem 3.8.
Assume ϕ
k
( θ ) is local strong convex as stated in Assumption 3.5 and the fixed learning rate η satisfies the conditions in Theorem 3.7, then if we run gradient descent for V T ( θ ) , it yields a solution:
[Equation]
where √ q denotes the convergence rate and its square q is lower bounded, s.t.,
[Equation]
Our Proposed Logarithmic Mapping of Finite Horizon Value Function:
In the vanilla setup, the value function of gradient descent is V T ( s, θ ) .
We propose a logarithmic mapping, (cid:101)
[Equation]
to regularize the spectral radius of gradient Hessian and gradient variance.
The sampled gradient approximation has the form of (cid:98)(cid:101) V T ( s, θ ) := 1 b (cid:80) bj =1 log ( v T ( s j , θ )) .
Theorem 3.9.
Consider the parameterization of V T ( θ ) and Lipschitz condition in Assumption 3.1 and Assumption 3.4, if we run gradient descent for logarithm mapped
V T ( θ ) , then η < V T ( θ ) / (cid:80) mk =1 d k
[ L 1 ( (cid:80) Tt =
tϕ k ( θ ) t − 1 )
+
L 22 ( [Equation] Tt {+=
+} t ( t − 1) ϕ
k ( θ ) t − 2 ] is satisfied for monotonic decrease of the value function.
Theorem 3.10.
Assume ϕ
k
( θ ) is local strong convex as stated in Assumption 3.5 and the fixed learning rate η l < C
L 1 T + L 22 T ( T − 1) , then running the gradient descent for logarithm mapped (cid:101) V T ( θ ) with Equation (4) yields a solution
[Equation]
where the square of the step convergence rate q l has a varying lower bound s.t.
q l ≥ (1 −
[Equation]
Remark 3.11 .
Compared with Equation (8), the
[Equation]
term is proportional to the inverse of current spectral radius.
Considering an unstable system initialized with random policy, the initial ρ max ( J T ( θ 0 )) could be much larger than ρ max ( J T ( θ l )) for θ l in the later part of the optimization.
Practically, using the logarithmic mapping achieves a much faster convergence rate.
3.3. Regulating Spectral Radius as Fast Pre-processing
When an unstable system is controlled by a random policy and initialized by an arbitrary condition, the states/costs will most likely grow rapidly due to the diverging nature of the system.
It is computationally costly to optimize the value function from such initial policy.
To speed up the optimization, we propose a fast pre-processing method in
Algorithm 1 by finding a policy close to the stable zone.
Algorithm 1 is deferred to Appendix.
In the pre-process, we neglect the value function and only regulate the spectral radius of the system dynamics estimated by power iteration.
4. Experiments
In this section, we use LQR as illustrative examples to demonstrate the efficacy of our methods.
We also apply these methods to general unstable RL applications in continuous control and defer the results to Appendix F. For each experiment, we run 3 random seeds for reproducibility.
Figure 1 shows the vulnerability of optimization for unstable systems with large learning rates.
For larger ρ max , the optimization is more likely to fail with larger learning rates.
However, a small learning rate could result in slow convergence.
Therefore, the major challenge for vanilla policy gradient method on unstable RL problem is how to find an optimal learning rate without crashing the optimization.
In our experiments, we test different learning rates by log intervals such as { 1 e - 1 , 1 e - 2 , ... } and select the largest one without breaking the optimization.
In Figure 2 we compare the vanilla gradient method and its variants with our proposed logarithmic mapping.
The subplots are normalized cost difference towards optimal, normalized policy difference to the optimal and estimated spectral norm by power iteration, respectively.
Note that the y-axis of the cost figure is log scale.
The optimal policy K ⋆ is the analytical optimal solution for infinite horizon LQR.
When optimizing the model with vanilla sum loss function from random initialization without any pre-processing, the system starts from an extremely unstable condition with large cost.
From Lemma G.1, a small learning rate is needed when the system is unstable with control policy parameters.
In practice, 1 e - 14 is the maximum learning rate to accommodate the instability.
If we run an efficient policy pre-process with a total of 300 episodes, we could use 1 e - 8 as learning rate.
However, the optimization is impractical because the controller gradually stabilizes and the fixed learning rate is then too small for further parameter update.
We also tested other discount factor such as γ = 0 .
5 to shrink the cost and allow a larger learning rate 1 e - 7 , but this method also fails because the small γ neglect the long term effect of the dynamics with spectral radius reaching 2.
Combining our proposed pre-process and log-mapped loss function, we are able to use 1 e - 2 as learning rate and both cost and policy parameters approach optimal quickly.
5. Conclusion
In this paper, we focus on the gradient-based optimization for a special branch of RL problems.
Due to the unstable nature of the system, small deviation leads to exponentially growing effects on the state evolving trajectory and the reward/cost function, which raised issues for gradient-based optimizations.
We proposed two methods to alleviate the effect of instability and their effectiveness is validated from both theoretical and experimental points of view.
A. Formulating Instability
In the dynamical system literature, stability usually denotes the input-to-state (ISS) (Sontag & Wang, 1995) stability in system dynamics, where a small deviation of system state or control action perturbation will not lead to dramatic change of future states.
Formally, consider a general continuous dynamical system ˙ x = f ( x, u ) with continuously differentiable f ( · ) and x ( t, x 0 , u ) denote the trajectory of x given initial condition x 0 and control feedback u .
Then the system is ISS stable if there exist K function γ : R + → R + and KL function β : R + × R + → R + , s.t.
ISS stable:
[Equation]
(5) where ∥ u ∥ ∞ =
sup {∥ u ( t ) ∥} < ∞ for t ≥ 0 .
Function γ ( · ) is called K function if γ ( · ) is continuously increasing and γ (0) = 0 , β ( · , · ) is called KL function if β ( · , t ) is K function for all the t ≥ 0 .
In this paper, we refer to stability as the input-to-output (I/O) stability (Sontag & Wang, 1999).
Let the output y = h
( x ( t )) be a function of x , for instance, y can be the quadratic function to regulate the error or other system properties we wish to stabilize.
The system is input-to-output (I/O) stable if there exist K function γ ( · ) : R + → R + and KL function β ( · , · ) : R + × R + → R + , s.t.
I/O stable:
[Equation]
[Equation]
Both ISS and I/
O stability indicate bounded inputs leading to bounded system behavior, while there does not exist any causal relationship between the two with arbitrary choice of output function y ( · ) .
For further clarification, we provide a linear system example in Appendix B.
In reinforcement learning setting, the agent always returns an observable ”cost-to-go” but not necessarily the whole trajectory.
This is because the returned cost serves as the evaluation metric for RL algorithm.
Thus, I/O stability is a more suitable stability concept for analyzing RL problems and that’s why it is selected for this work.
B. Example on stability definition
We use the illustrative example from (Sontag, 2008).
Consider a n dimension linear system ˙ x
= Ax + Bu where
A ∈ R n × n being full rank matrix, x ∈ R n with initial condition x (0) = x 0 , B ∈ R n × m and u = u ( t ) ∈
R m .
By solving inhomogeneous ODE, the solution is
[Equation]
Lemma B.1.
The system is ISS stable if all the eigenvalues of A are strictly negative.
Proof.
let β ( x 0 , t ) be ∥ e At ∥∥ x 0 ∥ and γ ( x 0 ) be ∥ B ∥ (cid:82) ∞
∥
e Aτ ∥ d τ .
With all the eigenvalues of A being strictly negative, both ∥ e At ∥ and ∥ B ∥ (cid:82) ∞
∥
e Aτ ∥
d τ are bounded.
∥ x ( t, x 0 , u ) ∥ ≤ ∥ e At ∥∥ x 0 ∥ + ∥ B ∥ (cid:82) ∞
∥
e Aτ ∥
d τ ∥ u ∥ ∞ , therefore satisfies 5.
Consider y ( x ) :
= x itself, the system is I/O stable.
While if we take y ( x ) := 1 ∥ x ∥ , then the system is ISS but not I/O stable with y → ∞ with x 0 , u → 0 .
Now suppose A has non-negative eigenvalues and AB is not empty matrix, then (cid:82) t 0 e A ( t − τ )
Bu ( τ ) d τ is not bounded by γ function since ∥ B ∥ (cid:82) ∞
∥
e Aτ ∥
d τ when t → ∞ , which also means the effect of previous action u will grow or at least not vanish along the time trajectory.
Then the system is not ISS.
But the system could be I/
O stable if we take trivial output like y (
· ) := 0 .
In this paper, we consider I/O stability, regardless the problem being ISS or not ISS.
While in many of the real-world RL applications such as target tracking, the output function y ( · ) is correlated to the norm of x such as using distance to target as cost function.
In this case, I/O stability is dependent on ISS.
Specifically, for LQR problems, the eigenvalues of system matrix A determines the system ISS and also I/O stability(discrete LQR requires the eigenvalue within the unit circle and continuous LQR requires eigenvalues to left half-plane).
Therefore, in the discrete LQR experiments, we use matrix A to manipulate I/O stability.
Since we are dealing with I/O stability, RL scenarios with ISS but not I/O stable system is beyond the scope of this paper, for instance, a unstable invert pendulum problem with cost clipped to [0 , 1] .
C. Algorithm for Pre-processing
Input: system state transition function f and policy θ , finite time step T , batch size b ,
Initialize θ for l = 1 to N do
Sample { x 10 ...x i 0 ...
x b 0 }
Loss ←for t = 0 to T do for i = 1 to b parallel do x i +1 t ← f ( x it , θ )
Loss ← Loss + ReLU ( ∥ x iT ∥ ∥
x iT − 1 ∥ − 1) end for end for
Loss ← Loss/b
Update parameters: θ ← θ − η ∇ θ Loss end for
D. Comparing logarithmic Mapping with Gradient Normalization
Normalizing gradient is a classical approach to speed up the convergence, where we have the follow update step:
[Equation]
Compared with logarithmic mapping, the gradient normalization has similar theoretical performance in deterministic case by controlling the spectral radius of the optimization step.
In the stochastic case, the updating step consists of a summation of gradient over the mini-batch followed by a normalization process.
In logarithmic mapping, the log function is applied on individual examples ahead of summation.
Therefore, the outliers with relatively large noise can be “normalized” to prevent them from dominating sampling summation.
Besides, the portion of unstable examples with large loss are expected to drop during optimization, it is necessary to map the exponentially growing effect of these unstable cases into linear forms.
In practice, our logarithmic mapping outperforms the gradient normalization in the convergence speed, as shown in the experiment section (Section 4).
Figure 3 are the comparisons between logarithmic mapping and normalizing gradient, where learning rate 1 e - 1 and 1 e 0 will crash the optimization respectively.
The plots of η = 1 e - 2 for logarithmic mapping and η = 1 e - 1 for normalizing gradient effectively show similar convergence rate with minor fluctuation at the beginning.
The logarithmic mapping eventually reaches a slightly better performance due to normalizing gradient’s trapping in the local minimum.
Noticeably, if both methods are coupled, the initial fluttering disappears and plots are smoother.
E. More experiments of unstable LQR with different spectral radius
We include additional results for unstable LQR in Figure 4 both with pre-process enabled.
ρ max ( A ) = 2 is a relatively moderate case, the vanilla method could use a learning rate of η = 1 e − 4 and slowly converge to optimal.
In ρ max ( A ) =case, the vanilla method crashes for η > 1e − 11 and the optimization stagnates for η = 1e − 12 .
The logarithmic mapping has similar performance in ρ max ( A ) = 2 case and converges faster than the latter in ρ max ( A ) = 10 case.
F. General Unstable RL
Figure 5 shows 3 customized unstable environments: unstable cart-pole, unstable mountain car and Planar Vertical Take-off and Landing (PVTOL) aircraft.
We use a single hidden layer neural network with 64 hidden neurons and ReLU activation functions.
The input layer and output layer has the same dimension of environment state and action space respectively.
Similar to LQR experiments, we search a largest learning rate without crashing the optimization.
Each experiment is performed under 3 random seeds.
The lower half of variance is omitted for visualization in log-scale plots.
F.1. Modified cart-pole
Compared with standard cart-pole problem from OpenAI Gym package, we use a continuous force input and enlarged its force magnitude to introduce more instability to the input-output system (a small amount of control feedback could dramatically change the system behavior).
Besides, we allow the agent to simulate fixed 20 time steps instead of terminating the episode if the agent runs into an undesired zone.
The cost function is defined in the quadratic form of the distance between current state towards target position, instead of using the 0/1 reward depending on whether the episode is done or not.
Figure 6 shows the cost against epochs for cart-pole problem with and without pre-process.
For vanilla sum loss without pre-process, η = 1 e - 8 is the maximum allowed learning rate and there is a significant difference in convergence speed compared with other two.
The logarithmic mapped cost is higher but close to sum loss with normalizing gradient.
With a pre-processed policy, the system is more stable at the beginning and therefore larger learning rates are allowed.
All three methods could reach the optimal.
To remark on the cart-pole problem, the instability mostly comes from the large force magnitude instead of the unbounded state space because there exists local equilibrium when the pole sticks downward.
Therefore, compared to the following 2 environments, it is less challenging and could be addressed with vanilla sum loss with a simple pre-process.
F.2. Modified mountain car
Similar to the cart-pole treatment, we remove the terminal conditions and re-define the cost function in the quadratic form.
The control target is to drive the car to a certain location and stabilize it.
We manipulate a steep slope by adding an acceleration term proportional to the cube of horizontal displacement from the peak and there does not exist any local equilibrium point.
The results are shown in Figure 7.
Both vanilla sum loss and normalizing gradient require small learning rate, the logarithmic mapping outperforms the other two methods.
When pre-processing is engaged, the vanilla sum loss still converges slowly, the other two methods share similar performance and achieve a smaller cost compared with the optimal results without the pre-process.
F.3. PVTOL Aircraft
The Planar Vertical Take-off and Landing (PVTOL) aircraft (Lin et al., 1999) is a simplified 2D model of realistic aircraft maneuver.
The aircraft state includes the lateral/vertical displacement of the gravity center and roll angle.
The control feedback U t and U m are longitudinal thrust and lateral rolling force.
Notice U m provides both force and rolling moment to the airplane.
The target is to control and airplane to certain state and cost function is also in the quadratic form.
Similar to the unstable mountain car example, both vanilla sum loss and normalizing gradient show slow convergence when pre-process is not engaged.
The logarithmic mapping is capable to achieve optimal results regardless of the pre-treatment.
G. More theoretical results on unstable RL
Lemma G.1.
Update the value function V T ( θ ) by policy gradient method with θ ← θ − η ∇ θ V T ( θ ) , choose step size η < 2 / max ξ ρ max ( J T ( θ + ξη ∇ J T ( θ ))) for ξ ∈ [0 , 1] , then V T ( θ ) is monotonically decreasing.
Proof.
Let ξ ∈ [0 , 1] be a scalar, denote s = − η ∇ V T ( θ ) , g ( ξ )
=
V T ( θ + ξs ) , we have
[Equation]
substitute s = η ∇ V T ( θ ) into the equation, we have
[Equation]
when (1 − η 2 max ξ ρ max ( J T ( θ + η ∇ J T ( θ )))) term is negative.
Theorem G.2.
If V T ( θ ) satisfies Assumption 3.1 and Assumption 3.4, using the vanilla gradient descent algorithm from
Equation (2) , then ρ max ( J T ( θ )) <
(cid:80) mk =
{+1 d k [ L 1 ( (cid:80) Tt =
+}
tϕ k ( θ )
[Equation]
− 1 )
+
[Equation]
[Equation]
Tt
[Equation]
[Equation]
ϕ k
[Equation]
θ
[Equation]
Proof.
[Equation]
Hessian
[Equation]
where
J
[Equation]
=
[Equation]
[Equation]
Denote the eigenvalues of ∂ 2 ϕ k
( θ )
∂θ 2 , ∂ϕ k ( θ ) ∂θ ∂ϕ k ( θ ) ∂θ ⊤ , J kT ( θ ) , as µ k 1 > ... > µ kn , υ k 1 > ... >
υ kn , ν k 1 > ... > ν kn respectively.
Notice ∂ϕ k ( θ ) ∂θ ∂ϕ k ( θ ) ∂θ ⊤ is positive semi-definite and has same non-zero eigenvalue with ∂ϕ k ( θ )
∂θ ⊤ ∂ϕ k ( θ )
∂θ , then υ k 1 ≤ L 22 by
Lipschitz condition.
To bound the eigenvalues of ∂ 2 ϕ k ( θ ) ∂θ 2 ,
[Equation]
Notice ∂ 2 ϕ k ( θ ) ∂θand ∂ϕ k ( θ ) ∂θ ∂ϕ k ( θ ) ∂θ ⊤ are Hermitian, by Weyl’s inequality to bound:
[Equation]
≤ d k [ L 1 (
T (cid:88) t =
tϕ k ( θ ) t − 1 ) + L 22 (
T (cid:88) t
= t ( t − 1)
ϕ k ( θ ) t − 2 )] . ρ max ( J T ( θ )) ≤ m (cid:88) k = d k [ L 1 (
T (cid:88) t =
tϕ k ( θ ) t − 1 )
+ L 22 ( T (cid:88) t
= t ( t − 1) ϕ k ( θ ) t − 2 )] .
Lemma G.3.
If function f ( x ) : R m → R + is L 1 smooth and L 2 Lipschitz and non-negative for x ∈ S ⊂ R m , then its polynomial f ( x ) n is ( nf S n − 1 L 1 + n ( n − 1) f S n − 2 L 22 ) smooth on S , where f S = max x ∈ S
[ f ( x )]
Proof.
With function f ( x ) being L 1 smooth, equivalently
[Equation]
With function f ( x ) being L 2 Lipschitz, ∥∇ f ( x ) ∥ ≤ L 2 , for polynomial f ( x ) n ,
[Equation]
where the L 22 term comes from the fact that ∇ f ( x ) ∇ f ( x ) ⊤ has same non-zero eigenvalue with ∇ f ( x ) ⊤ ∇ f ( x ) .
Therefore, f ( x ) n is locally ( nf S n − 1 L 1 + n [Equation] n − 1) f S n − [Equation]
L 22 ) smooth on the support S .
Lemma G.4.
[Equation]
where
[Equation]
tϕ k
( θ )
[Equation]
1 L
[Equation] t − 1) ϕ [Equation] ( θ
[Equation]
t
[Equation]
where
[Equation]
Proof.
With Assumption 3.4 on ϕ k ( θ ) , apply Lemma G.3 on the straight line from θ to θ ∗ , V T ( θ ) ∼ (cid:80) mk =1 d k (cid:80) Tt {+=
+}
ϕ k ( θ ) t is L ′ smooth on the straight line.
[Equation]
where second inequality comes from the L ′ smoothness on the straight line from θ to θ ∗ .
therefore,
[Equation]
Remark G.5 .
If ϕ k ( θ ∗ + ξ ( θ − θ ∗ )) is monotonically increasing on ξ , then
[Equation]
then the smoothness along the updated step is bounded by the spectral radius of the Hessian on θ , as ρ max ( J T ( θ )) in
Theorem G.2.
Proposition G.6.
For twice-differentiable f , f is α -strong convexity function if and only if ∇ 2 f ( x ) ≽ αI for some α >and x ∈ R d .
Proof.
For a twice-differentiable function, α -strong convexity is equivalent to the smallest eigenvalue of Hessian of f being lower bounded by α .
Theorem 3.7 (Restated) .
Suppose V T ( θ ) satisfies Assumption 3.1 and Assumption 3.4, using the vanilla gradient descent algorithm from Equation (2) , if η < 1 / (cid:80) mk =1 d k [
[Equation] tϕ [Equation] [Equation]
( θ ) t − 2 ] , then η < 2 /ρ max ( J T ( θ )) is satisfied for monotonic decrease of value function.
The proof is completed by substituting Theorem G.2 into Lemma G.1 and taking ξ = 0 .
Theorem 3.8 (Restated) .
Assume ϕ
k
( θ ) is local strong convex as stated in Assumption 3.5 and the fixed learning rate η satisfies the conditions in Theorem 3.7, then if we run gradient descent for V T ( θ ) , it yields a solution:
[Equation]
8) where ω ∗ = min θ
[Equation]
tϕ
[Equation]
θ
[Equation]
1 )] , √ q denotes the convergence rate and its square q is lower bounded s.t. q
[Equation]
Proof.
by Proposition G.6, ∂ 2 ϕ k ( θ ) ∂θ≽ αI .
From (7),
[Equation]
by Proposition G.6, V T ( θ ) is ω ∗ α strong convex:
[Equation]
where the inequality condition is satisfied with our analysis in Lemma G.4 and Remark G.5 when η l < 1 /
[Equation]
in Theorem [Equation]
[Equation]
Since we have a non-increasing step size, η l is upper bounded by
[Equation]
α ) is greater than (1 − 2 ω ∗ α ρ max ( J T ( θ 0 )) ) .
Theorem 3.10 (Restated) .
Assume ϕ
k
( θ ) is local strong convex as stated in Assumption 3.5 and the fixed learning rate η l <
[Equation] T − 1) , then if we run gradient descent for logarithmic mapped V T ( θ ) with Equation (4) , it yields a solution:
[Equation]
where the the square of the step convergence rate q l has a varying lower bound s.t.
q l ≥ (
[Equation]
Proof.
[Equation]
where the inequality condition is satisfied with our analysis in Lemma G.4 and Remark G.5 when η l
V T ( θ l )
[Equation]
The latter is valid because
[Equation]
is lower bounded by a constant
[Equation]
Because
[Equation]
we have (1 −
[Equation]