Robot Reinforcement Learning on the Constraint Manifold
Anonymous Author(s)
AfﬁliationAddress email
Abstract:
Reinforcement learning in robotics is extremely challenging due tomany practical issues, including safety, mechanical constraints, and wear and tear.
Typically, these issues are not considered in the machine learning literature.
Onecrucial problem in applying reinforcement learning in the real world is Safe Exploration, which requires physical and safety constraints satisfaction throughoutthe learning process.
To explore in such a safety-critical environment, leveragingknown information such as robot models and constraints is beneﬁcial to providemore robust safety guarantees.
Exploiting this knowledge, we propose a novelmethod to learn robotics tasks in simulation efﬁciently while satisfying the constraints during the learning process.
Introduction
[Equation] [Equation]
Despite the notable success of Deep Reinforcement Learning (RL) in solving complex tasks in the discrete world,video games, as well as continuous control problems insimulation [1, 2, 3, 4], applying RL in the real world remains a challenging task.
One important factor that cannotbe neglected in real-world applications is the necessity ofsatisfying constraints.
Many practical considerations canbe formulated in the form of constraints, such as safety andmechanical viability.
For example, in the robot manipulation task, the robot should not take actions that damage theenvironment and can not take actions that exceed its feasible range.
However, typical RL algorithms, which maximize the cumulative reward by continuous trial and error, donot take into account the satisfaction of constraints duringthe exploration process.
Exploring the environment whilemeeting the constraints is a challenging problem.
Safe exploration is an signiﬁcant ﬁeld of RL which requiresto comply with the constraints during the whole learningprocess [5].
There are several safe exploration frameworksin the literature: a possible direction is proposed in [6, 7]that relies on prior knowledge (policies, value functions)to initialize the system in a safe region and gradually increase the area of exploration using new information obtained from the environment.
Other approaches rely on thedeﬁnition of a safe policy [8, 9], which tries to pull the agentback to a safe state.
However, these choices require excessive work in deﬁning such policy, and safepolicies could conﬂict against each other when multiple constraints are violated.
Finally, other methods incorporate model information of constraints with model-free RL algorithms and not require the deﬁnition of a manual policy [10, 11, 12].
In these approaches, the agent tries to ﬁnd the feasibleaction using constrained optimization techniques at each time step.
In this paper, we propose a novel method, Acting on the TAngent Space of the Constraint Manifold(ATACOM), which the agent explores in the tangent space of the constraint manifold, as shown in
Figure 1.
The proposed method convert the constrained RL problem to a typical unconstrained RLproblem.
This method allows us to utilize any model-free RL algorithms while maintaining theconstraints below the tolerance.
Furthermore, ATACOM can handle both equality and inequalityconstraints.
For example, in the task of a robot wiping a table, the end-effector should move onthe surface of the table (equality constraints) while the joint positions and velocities are withinits joint limits (inequality constraints).
In addition, for tasks with equality constraints, our methodexplores the lower-dimensional manifold embedded in the original action space.
To test our method,we demonstrate three different tasks, CircleMoving, PlanarAirHockey , and IiwaAirHockey , withdifferent combinations of equality and inequality constraints.
We test ﬁve state-of-the-art modelfree RL algorithms (PPO, TRPO, DDPG, TD3, SAC) in each environment.
The result shows that allalgorithms can learn the policy efﬁciently while maintaining the constraints below the tolerance.
The advantage of ATACOM can be summarized as follows: (i) can deal both with equality andinequality constraints .
All of the constraints at each time step are maintained below the toleranceduring the whole learning process.
(ii) does not require an initial feasible policy, the agent can learnfrom scratch .
(iii) requires no manual safe backup policy to move the system back into the saferegion.
(iv) can be applied to any model-free RL algorithm, using both deterministic and stochasticpolicies .
(v) can focus the exploration on the lower-dimensional manifold instead of exploring inthe original action space for equality constrained problem.
(vi) have better learning performanceas the inequality constraints restrict to a smaller feasible state-action space.
As a downside, ourmethod requires
Related Work.
In the last decades, Constrained Markov Decision Processes (CMDP) [13] hasattracted a lot of interest from RL researchers, to solve constrained control problems.
Under thisframework, several different forms of constraints have been studied.
One important form of constraint is the expected cost below a threshold.
Many works maximize the expected return whilemaintaining the expected cost below a threshold [11, 14, 15, 16, 17, 18, 19, 20].
Different types ofconstrained optimization techniques are applied in the policy update process.
Achiam et al. proposed a trust-region method Constrained Policy Optimization (CPO) inspired from Trust Region
Policy Optimization (TRPO)
Liu et al. proposed the interior point method for policy optimization [15].
Another type of approach is to adapt the Lagrangian relaxation method for the constrained
RL setting [13, 16, 14, 17].
Lastly, Chow et al.
proposed a method to generate the Lyapunov function that guarantees constraints satisfaction [11, 21].
These approaches focus on the constraint ofthe cumulative cost and require an initial feasible policy.
However, this cumulative cost criterioncannot ensure safety for tasks where avoiding catastrophic failures is crucial, e.g., car crashing.
Other approaches focus on the state dependant constraints, which should be fulﬁlled at every timestep.
To meet this requirement, safe exploration methods can be employed.
Garcia, et al., proposeda method based on a risk function and a baseline agent, where the control action is sampled basedon the evaluation of the risk [6].
The shielding [8] and backup policy [9] frameworks interfere withthe control action to pull the system back to the safe states.
These approaches require a manualdeﬁned safe policy.
Berkenkamp, et al.
[7], Wachi, et al.
[22], Koller, et al.
[23], and Hewing, etal.[24] proposed model-based approaches to ensure the safety.
These approaches start from an initialfeasible policy and progressively increase the safe region based on the learned dynamics model.
Recent methods also try to incorporate the model and the constraint information with the modelfree RL algorithms.
Dalal, et al., added a safe layer which analytically ﬁnds the closest action w.r.tthe policy derived one [10].
Cheng, et al., proposed a barrier function method to guarantee safetyduring the exploration [12].
Finally, other approaches has also address the safety issue from differentperspectives, such as the policy composition [25, 26] and reachability-based approach [27, 28, 29].
Our approach considers the second group of constraints.
However, different from other comparablemethods, ATACOM does not require an initial policy, it can learn from scratch.
In addition, ourmethod does not require a backup policy either, as the constraint violations are forecasted and corrected at each step.
Furthermore, our method is not speciﬁcally restricted to any learning algorithm.
Learning on the Constraint Manifold
In this section, we discuss ATACOM in detail.
We ﬁrst introduce the mathematical notation used inthis paper.
Then, to demonstrate the core concept, we start with a simple scenario that the constrainton the subset of the state variable q and the action can be formulated as a function of the state velocitya = Λ ( ˙ q ) .
Next, considering the continuity of velocity (sampling over the velocity does not ensurethe continuity), we convert the original state constraint to a viability constraint that incorporates thevelocity of the constraint.
The action is chosen as a function of the acceleration a = Λ (¨ q ) .
From arobotics point of view, this a can be the torque applied to each joint, and Λ is the inverse dynamicsmodel.
Then, to cope with the velocity limit, we add the viability condition to the acceleration.
Lastly, we discuss some important practical issues of ATACOM, such as the error correction, thetangent space convention that determines the null space bases.
Deﬁnitions We consider the CMDP with continuous state-action space.
A CMDP is a tuple( S , A , P, R, γ, C ) , where S is a state space, A is an action space, P : S ×
A × S → [0 , 1] is a 109 transition kernel, γ is a discount factor, and
C : { c i : S → R | i ∈ 1 , ..., k } is a set of immediate 110 state-constraint functions.
Assumption In this paper, we decompose the state variable s ∈ S into the directly controllablestate q ∈ Q and uncontrollable state x ∈ X , i.e., s =
[ q x ] T .
We assume that the constraints c ( q ) ≤ 113 0 are known and depend purely on the controllable state.
In addition, we assume that the action a canbe determined based on the i -th order time derivative of the controllable state, i.e., a = Λ ( q ( i ) ) , i ∈{ 1 , 2 , ... } .
For example, we can determine the joint torque using an inverse dynamics model or send 116 the desired positions/velocities obtained via integration to a tracking controller (e.g., PID controller).
The general form of the constrained reinforcement learning problem can be formulated as
[Equation]
2.1 State Constraints
The state constraints are deﬁned as
[Equation]
where f : R Q → R F , g : R Q → R G are two C 2 mappings for F equality and G inequalityconstraints, and F < Q .
We add the slack variables µ ∈ R G in inequality constraints to convert the 122 original constraints (1) into equality constraints
[Equation]
The constraint set (2) is a ( F + G ) dimensional manifold embedded in ( Q + G ) dimensional space.
124 We calculate the time derivative of (2)
[Equation]
with the Jacobians J f ∈ R F × Q and J g ∈ R G × Q of f ( q ) and g ( q ) , respectively.
Both Jacobiansare combined into the Jacobian Matrix J c ( q , µ ) ∈ R ( F + G ) ×
( Q + G ) of the complete constraint set.
We can ﬁnd the null space matrix N c ( q , µ ) =
Null[ J c ( q , µ )]
∈ R ( Q + G ) × ( Q − F ) via SVD [30] or
QR [31] decomposition, such that J c ( q , µ ) N c ( q , µ ) = 0 .
Each column of the orthogonal matrix 129 N c ( q , µ ) represents a basis vector of the null space of J c ( q , µ ) .
These null space bases can also 130 be viewed as the tangent space bases of the constraint manifold as illustrated in Figure 1.
We canconstruct a tangent space velocity of the constraint manifold by a coordinate α as
[Equation]
Substituting [ ˙ q ˙ µ ] T of (3) by [ ˙ q T ˙ µ T ] T of (4) , we have the constraint velocity
[Equation]
Equation (5) implies that the constraints do not change regardless of the choice of α .
Based onthis concept, the ATACOM method can be summarized as follows:
Starting from a feasible point
[Figure]
straints
[Figure]
Velocity
[Figure]
a max
Acceleration
¨ q ˙ µ ¨ q min ¨ q max -
( q (0) , µ (0)) ∈ { ( q , µ )
| c ( q , µ ) =
0 } , we choose the tangent space velocity [ ˙ q T ( t ) , ˙ µ T ( t )]
T =
N c ( q ( t ) , µ ( t ))
α ( t ) and the corresponding action as a ( t ) =
Λ ( ˙ q T ( t )) .
Thus, the constrained RLproblem is converted into an unconstrained RL problem.
The resulting trajectory q ( t ) satisﬁes theconstraints c ( q ( t ) , µ ( t ))
= 0 .
2.2 Viability Constraints
For a physical system, it is often required a continuous velocity command.
However, directly sam- 141 pling velocities ˙
q does not ensure this continuity.
A simple solution is to sample accelerations,apply force to the system or determine the velocity via integration.
Furthermore, when considering 143 inequality constraints, it is also desirable that ˙ g ( q , ˙ q ) ≤ 0 when g ( q ) = 0 to avoid overshooting.
144 We convert the original state constraints (1) to viability constraints inspired by the linear viabilitycondition [32]
[Equation]
(6) with diagonal matrices K f ∈ R F × F ,
K g ∈ R G × G having all positive entries.
The matrices K f and
K g determine the maximum velocities of the constraints ˙ f and ˙ g w.r.t to the value of the constraints.
The viability constraint of the inequality constraint is illustrated in Figure 2.
When g ( q ) < 0 , theupper bound of the constraint velocity is ˙ g max > 0 which means that it is still possible to get close tothe constraint boundary.
However, if g ( q ) > 0 , the upper bound of constraint velocity ˙ g max shouldbe smaller than zero to pull the violations back.
Analogous to the derivations from equation (2) and (3), we have
[Equation]
and
[Equation]
where b f ( q , ˙ q ) = ˙
q T H f ( q ) ˙ q , b g ( q , ˙ q ) = ˙
q T H g ( q ) ˙
q and H f ∈ R F × Q × Q , H g (
q ) ∈
R G × Q × Qare Hessians of f ( q ) , g ( q ) , respectively.
We can construct the joint acceleration as
[Equation]
[Equation]
with the pseudo-inverse J † c ( q , µ ) and the null space matrix N c ( q , µ ) of the Jacobian J c ( q , µ ) , respectively.
The ﬁrst term in equation (9) is the necessary acceleration that maintains the curvature ofthe constraints manifold (7) and the second term is the tangent space acceleration of the constraints.
When starting from the point [ q (0) , ˙ q (0) , µ (0)]
∈ { ( q , ˙ q , µ ) | c
q , ˙ q , µ ) = 0 } and sampling over α ,the joint acceleration ¨ q and the corresponding action a satisfy the constraints.
2.3 Viability Acceleration Bound
In robotics as well as other mechanical systems, it is important to consider the velocity constraints 163 of the actuator.
Also, the acceleration should be bounded properly to avoid overshooting.
We againuse the concept of viability to determine the upper and lower bound of the acceleration
[Equation]
with the minimum and the maximum joint velocity limits v min , max and the acceleration limitsa min , max , K a > 0 is a constant.
The feasible acceleration region is illustrated in Figure 3.
Analogous to the viability constraints, the feasible region of the acceleration is modiﬁed depending on thestate of joint velocities.
This technique effectively prevents overshooting.
2.4 Error Correction and Control Action Selection
[Figure]
For time-continuous systems, the state is obtained at a certain 171 sampling rate and the action is applied for a certain period.
Thistime discretization results in constraint violations at each time 173 step.
Therefore, we add an error correction term.
We constructa P-controller with a diagonal matrix K c for the constraints
[Equation]
Combining (9) with (10), we get the joint acceleration applied 176 to the system(cid:20)
[Equation]
q
[Equation]
The ﬁrst term on the RHS is the necessary accelerations/velocities to maintain the constraints and the second termon the RHS is the tangent acceleration that can be exploredfreely.
Figure 5 illustrates the vector ﬁeld of error correctionterm and null space term of the circle constraint.
The graycurves show the sampled trajectories converging to the constraint manifold due to the error correction.
The control action can be determined by a = Λ(¨ q ) at differentlevels.
For example, we can use the inverse dynamics modelto calculate the joint torque when the robot is controlled viatorque command.
We can also apply the integration method todetermine the desired positions/velocities, then use a sufﬁcient accurate tracking controller (e.g.,
PID controller + Feedforward Term) to track the desired trajectory.
However, the tracking errorscould potentially cause hazardous constraint violations.
In this paper, we control the joint torquecalculated by a perfect dynamic model in simulation to simplify the analysis and to exclude theconstraint violations caused by the tracking error of the controller.
We present the block diagram ofthe controlling framework in Appendix A.
2.5 Null Space Convention
The orthogonal null space matrix N c can be determined through SVD or QR decomposition.
However, the representation of the null space bases is not unique.
It is difﬁcult to preserve the consistency
Algorithm
J
J a
+ and r from the environment
[Equation]
[Figure]
of the null space bases computed by the numerical decomposition method
To solve thisissue, we propose a convention to ensure the uniqueness of the null space bases.
Each column of the null space matrix N c is a unit vector indicating a direction of [¨ q ˙ µ ] T .
However,this unit vector could sometimes contribute majorly to the part of the slack variable and the entriesfor the joint accelerations could be very small.
As a result, the joint acceleration obtained fromα ∈ [ α min , α max ] can only cover a small region of the acceleration.
As illustrated in Figure 4,the red arrow is a unit basis vector of the tangent space, and the reachable joint acceleration by auniversal scaling factor could only cover part of the feasible joint acceleration, as the red area shownin Figure 4.
To alleviate the previously mentioned issue, we compute the Reduced Column Echlon
Form (RCEF) of the null space matrix N Rc = RCEF ( N c ) .
Given that the RCEF of a matrix isunique, we obtain unique bases of the null space.
In addition, for RCEF, each row containing aleading 1 has zeros in all its other entries.
Generally speaking, there exist N independent jointswhose acceleration can be solely determined by α , where N is the dimensions of the null space.
Also, we can deﬁne the feasible range of α as α i ∈
[¨ q i, min , ¨ q i, max ] .
Through this convention, thejoint acceleration is able to cover the full feasible range.
Null space bases and feasible region areshown as the blue vector and the blue area in Figure 4.
Experiments and Evaluation
To illustrate the properties of our approach, we demonstrate three different experiments in this section.
We ﬁrst demonstrate a toy task, CircularMotion , shown in Figure 6a.
In this task, we considerstate equality, inequality, and velocity constraints.
Secondly, we show a robotic environment withonly inequality constraints, PlanarAirHockey shown in Figure 6b.
A 3 DoF planar robot playing thehitting task in the air hockey scenario while keeping the end-effector inside the table boundary andthe robot’s joint positions and velocities within its limits.
Finally, we demonstrate, IiwaAirHockeyin Figure 6c, a 7 DoF KUKA IIWA robot learning the hitting task in the simulator.
In additionto the constraints of the 3-dimensional task, we add an equality constraint to ensure that the robotend-effector stays on the table surface.
More details about experiments can be found in the appendix.
CircularMotion .
In this task, shown in Figure 6a, the red point tries to move along a unit circle in2D space while keeping the velocity of each direction below the velocity limits and maintaining theposition above a certain height.
The objective is to reach the target point (green square) located in(1 , 0) .
The control action is the acceleration a =
[¨ x ¨ y ] T .
We compare ATACOM with two other approaches for the task.
(i) TerminatedCircularMotion wherethe episode terminates when the maximum constraint violations up to a threshold.
(ii) ErrorCorrectionCircularMotion , where the error correction term in (10) is added before the action is appliedto the environment.
We test ﬁve model-free RL algorithms (SAC, DDPG, TD3, TRPO, and PPOimplemented in Mushroom-RL [35]) for each approach.
Figure 7 shows the learning curve and constraint violations of all test RL algorithms for ATACOM.
Every algorithm is able to improve the learning performance and SAC outperforms the others methods, which matches our expectations.
Figure 7b and 7c show the maximum constraint function andmaximum joint velocity constraints at each time step.
It can be shown that the maximum constraintviolations during the whole learning process remain small.
The velocity limit violations are zeroafter 30 epochs which means the learned policies try to fully exploit the constraints.
[Figure] [Figure]
In Figure 8 we compare ATACOM, TerminatedCircularMotion
[Figure]
and ErrorCorrectionCircularMotion .
We select the best learning algorithms for each approach (SAC for all cases).
Compared to the
ErrorCorrection and TerminatedCircularMotion , our method focus on a lower-dimensional exploration space that avoid the constraint violations while the others do not.
We can conclude from the
Figures that ATACOM has not only lower constraint violations but also better learning performanceamong the three approaches.
PlanarAirHockey .
In this experiment, we apply a 3-joints planar robot for the air-hockey hittingtask, as illustrated in Figure 6b.
The end-effector of the robot is kept on the table surface and theobjective is to hit the puck to the opponent’s goal.
In this environment, we only consider inequalityconstraints, i.e., the robot end-effector should stay inside the table’s region, and the joint positionsand velocities should not exceed its limits.
The control action is the joint torque obtained by theinverse dynamics model.
In this experiment, we assume the dynamics model is perfectly known toeliminate the constraint violation due to the tracking error or the model mismatch.
In this task, we compared ATACOM with the SafeLayer method proposed by Dalal et al.
,[10] andthe Unconstrained air-hockey environments.
Since the SafeLayer method at the beginning requires afree exploration process to learn the constraint function, we only compare the learning performanceand the constraint violations after this process.
For the unconstrained environment, the robot iscompletely free to explore, and the episodes only terminate when the maximum episode step isreached.
In this experiment, we only compare the best DDPG result after the parameter sweep, asthe available implementation of SafeLayer only supports DDPG.
The result is shown in Figure 9.
We can see that ATACOM have the best learning performance andthe minimum constraint violations among the three methods.
SafeLayer did not learn the constraintfunction of joint velocities properly.
Furthermore, the learned constraints appear to be too restrictiveto learn a good policy.
Compared to the method of Unconstrained approach, although ATACOMhas the same dimension as the Unconstrained , ATACOM explores only in the feasible region whilethe Unconstrained approach explores the whole state-action space.
This consideration explains why
ATACOM outperforms the baselines in terms of learning performances.
[Figure] [Figure]
IiwaAirHockey .
In the third experiment, wedemonstrate the same air-hockey hitting task with a
KUKA LBR IIWA14 Robot in the Pybullet simulator.
In this task, we add an equality constraint to ensure the end-effector stays on the table surface.
Wealso add inequality constraints to avoid collision ofthe end-effector and joint limit constraints as mention in the PlanarAirHockey task.
In addition, wealso add inequality constraints to avoid the collisionbetween the 4/6-th link and the table.
We enforcethe joint velocity limits in the simulation as the realworld’s KUKA controller does.
We compare the impact of different simulation step sizes.
The step sizerefers to the sampling frequency in the real world.
At each simulation step, the torque is computed by the previous agent’s action until the new controlaction is received.
The error correction term is added at each time step.
We choose the simulationstep size as 0 .
02 s, 0 .
004 s, 0 .
002 s, and 0 .
001 s and keep agent control frequency to be 50Hz.
Figure 10 demonstrates the discounted return at the ﬁnal epoch, the maximum constraint violationc max , and the average constraint violations c avg throughout the learning process.
For a sufﬁcientlysmall step size, such as 0 .
004 s, 0 .
002 s, 0 .
001 s, the learning agent is able to learn the hitting policy.
When the step size is too big, e.g., 0 .
02 s, the error correction term dominates the control actionand the agent has a poor learning performance.
From the result in this experiment, we demonstratethat ATACOM is able to solve high dimensional tasks.
The simulation result also provides us theguidance for the real-world application:
The higher frequency of sampling and error correction, thesmaller constraint violations will occur.
Conclusion
In this article, we present ATACOM, a safe exploration method for Constrained RL based on theknowledge of the model and the mathematical formulations of constraints.
ATACOM explores thetangent space of the constraint manifold.
This exploration technique allows us to utilize any type ofmodel-free RL method while maintaining the constraint violations below a small threshold.
Fromthe experiments, we have shown that ATACOM not only has small constraints violations but alsobetter learning performance w.r.t.
the other baselines.
These performance gains occur because
ATACOM only focuses on the safe region (from inequality) and subspace (from equality constraint)of the whole state-action space.
However, our method still has some limitations.
Our method requires a sufﬁciently accurate modelor a good tracking controller.
This assumption does not hold in most real-world applications sincemodel errors, disturbances, and sensor noise could potentially cause unexpected constraint violations.
To deploy this method in real-world robots, we will focus on the model mismatch problemand may require a backup policy to avoid too stringent constraint design.
Furthermore, our currentapproach only focuses on the constraint with only controllable state c ( q ) = 0 , even if preliminaryresults (Appendix E) suggests an extension into constraints with the uncontrollable state.