Redundancy Resolution as Action Bias in Policy Search for Robotic Manipulation
Anonymous Author(s)
AfﬁliationAddress email
Abstract: We propose a novel approach that biases actions during policy search by lifting the concept of redundancy resolution from multi-DoF robot kinematicsto the level of the reward in deep reinforcement learning and evolution strategies.
The key idea is to bias the distribution of executed actions in the sense that the immediate reward remains unchanged.
The resulting biased actions favor secondaryobjectives yielding policies that are safer to apply on the real robot.
We demon- strate the feasibility of our method, considered as policy search with redundantaction bias (PSRAB), in a reaching and a pick-and-lift task with a 7-DoF Frankarobot arm trained in RLBench – a recently introduced benchmark for robotic manipulation – using state-of-the-art TD3 deep reinforcement learning and OpenAI’sevolutionary strategy.
We show that it is a ﬂexible approach without the need ofsigniﬁcant ﬁne-tuning and interference with the main objective even across different policy search methods and tasks of different complexity.
We evaluate ourapproach in simulation and on the real robot.
Introduction
Deep reinforcement learning has achieved remarkable success in recent years ranging from Atarigames [1] to complex locomotion [2, 3] and dexterous manipulation tasks [4, 5].
Likewise, evolutionstrategies provide a highly scalable alternative, which is capable of reducing the effective wall-clocktime to a fraction at the cost of sample efﬁciency [6, 7].
Despite their successes, both approachesare challenging to train on real-world systems due to their high sample complexity.
Further, theystill suffer signiﬁcant instabilities [8] making them very prone to changes in hyperparameters andrewards.
To improve the efﬁciency of learning and the quality of learned policies, reward shapingcan be used, which however often needs signiﬁcant ﬁne-tuning.
Another popular means for improvement is imitation learning, which enhances exploration and pushes the agent’s policy towardsdemonstrated expert policies by inserting those into the replay buffer [9, 10, 11].
We interpret thelatter as adding an expert bias on the training data.
We propose an alternative approach to add a bias by lifting the concept of redundancy resolutionfrom multi-DoF robot kinematics to the level of the reward in deep RL and evolution strategies.
The key idea is to bias to the distribution of executed actions while keeping the immediate rewardunchanged.
This redundant action bias (RAB) favors secondary objectives, e.g., for pushing theagent towards safer state-space regions or avoiding obstacles.
We investigate RAB for reaching andmanipulation policies exploiting the kinematic positioning redundancy of 7-DOF arms.
We showthat for both, deep reinforcement learning and evolutionary strategies, RAB is a feasible means toconsider secondary objectives without much interference with the main learning task.
Our approachyields safer and more natural-looking policies while keeping the reward-function as simple as possible.
In contrast to reward shaping, we also demonstrate that our approach does not require anyﬁne-tuning when used in different policy search methods and across tasks of different complexity.
Further, we show that our approach can handle interference between the main and secondary objective, and can enhance the performance of agents, which might get trapped in a local optimum.
Wedenote this approach policy search with redundant action bias (PSRAB).
In our experiments, we train reinforcement learning agents using the state-of-the-art twin-delayeddeep deterministic policy gradient (TD3) algorithm [3] as well as agents using OpenAI’s evolutionstrategy [6] (OpenAI-ES).
Training is conducted in RLBench [12], a benchmarking environment forrobotic manipulation, on a simulated Franka Panda robot arm.
All agents are trained in joint-spaceon a reaching and a pick-and-lift task with varying kinematic redundancy resolution.
Finally, allagents are evaluated on the real robot using zero-shot transfer.
Related Work
In recent years, a lot of work in robotics considered reinforcement learning in task-space as noforward kinematics need to be learned by the agent while redundancy resolution of complex robotsis often left to the controller [9, 10, 13, 14].
For example, Kaspar et al.
[13] trained a peg-inhole task using a soft actor-critic (SAC) [15] agent to generate high-level commands in task-space.
These commands were then translated to joint-space using the operational space control (OSC)framework [16], which handles the redundancy resolution.
Note that this nevertheless requires thedeveloper to deﬁne a redundancy resolution criterion as generally no degrees of freedom shouldremain uncontrolled.
Besides this, handling the agent’s actions and redundancy resolution in thesame action-space allows to take advantage of all DOFs of the robot, which is convenient in caseof interference between the main and the secondary objective.
Our results shows that training andresolving redundancy together in the joint-space allows a more ﬂexible adaption by the agent.
Regarding learning in joint-space, Peng et al.
[17] presented an approach that learned an objectpushing task using a simulator with randomized dynamics.
They used a recurrent version of thedeep deterministic policy gradient (DDPG) [18] algorithm with a long short-term memory (LSTM)network [19].
They used hindsight experience replay (HER) [14] to allow efﬁcient training withsparse rewards.
Despite the redundant robot arm, they did not explicitly handle redundancy.
Ourresults, however, indicate that not handling redundancy results in policies with high variety, whicheven exploit idiosyncrasies of a simulator yielding unnatural behavior that is infeasible in reality.
Kumar et al.
[20] used a proximal policy optimization (PPO) agent to solve a reaching task on environments with surrounding obstacles.
Additionally, they used curriculum learning to incrementallyincrease the state-space cardinality.
They carefully shaped the reward function to utilize redundancyand avoid obstacle contact.
In contrast, our approach directly learns on the full state-space and handles redundancy without reward shaping.
Huang et al.
[21] presented a problem of reward shapingfor in-hand manipulation, where a penalty reward for non-gentleness led to policies that may avoidcontact altogether.
We argue that it is beneﬁcial to keep the reward function as simple as possibleand use conventional approaches, such as redundancy resolution, to take secondary objectives intoaccount.
We use redundancy resolution to push the agent towards desired, e.g., safer, state-space regions.
Hence, our approach can be associated with so-called safe reinforcement learning [22], which coversapproaches that consider safety constraints during the learning and/or deployment process.
Theauthors in [9, 10, 11] combined reinforcement learning with learning from demonstrations (LfD) toimpose an expert bias on the training set.
For example, Vecerik et al.
[11] trained a DDPG agent ondifferent insertion tasks, where expert demonstrations were collected beforehand and inserted intothe replay buffer.
They showed that using expert demonstrations with a prioritized experience replay(PER) buffer [23] allows efﬁcient training in a sparse reward setting.
Algorithmic Background
We consider the standard reinforcement learning setup.
At each time step t , the agent observesa state s t ∈ S and uses its policy π : S → A to choose an action a t ∈ A , which results in asubsequent state s t +1 ∈ S .
After each taken action a t , the reward function r : S ×
A → R returnsa scalar signal r t , which evaluates the last state-action transition.
The objective is to maximizethe sum of discounted rewards R t = (cid:80)
Ti = t γ i − t r
( s i , a i ) , where γ ∈ [0 , 1] is a discount factorand T is the length of an episode.
We assume that a state s t is fully observable for simplicity.
We optimize a differentiable policy π θ parametrized by θ with respect to the performance measure
J ( θ ) =
E s ∼ p
π
[ R t | π θ ] , where p π is the likelihood of a trajectory under policy π .
The parameters θcan be updated based on the positive gradient of the expected return such that θ t +1 ← θ + η ∇ θ J ( θ ) ,where ∇ θ J ( θ ) is the policy gradient, i.e., the gradient of the policy’s performance.
Policy gradientmethods sample rollouts following their policy π and update the parameters based on the sampledpolicy gradient of an episode, which generally has high variance [24].
Actor-Critic Methods: In contrast, actor-critic methods learn either a state or action-value representation and train the policy on that instead.
Within this work, we consider an action-value function
Q π ( s t , a t ) =
E s i>t ∼ p π ,a i>t ∼ π [ R t | s t , a t ] , which describes the estimated sum of discounted rewards 100 when starting in state s t , taking action a t , and following the policy π thereafter.
The policy is considered as the actor while the action-value function is considered as the critic.
For a deterministicpolicy, Silver et al.
[25] provided the deterministic policy gradient:
[Equation]
Action-value functions are expressed using the recursive bellman equation and can be solved usingthe bootstrapping principle such as in Q-Learning.
As we are interested in continuous state andaction-spaces, we use a differentiable function Q φ ( s t , a t ) for the critic as well, where φ denotesthe parameter vector.
We drop the dependency on π in Q as we optimize the critic off-policy byminimizing the loss:
[Equation]
where µ is an arbitrary behavior policy, e.g., encouraging exploration, and Υ t is the target used forupdating the critic:
[Equation]
Deep Deterministic Policy Gradient (DDPG): Approximate reinforcement learning is known tobe relatively unstable and sample inefﬁcient.
To tackle this, DDPG [2] builds on the success of deep
Q-learning [26] and adapts the two main mechanisms of the latter – target networks and experiencereplay – to the actor-critic case.
DDPG uses two target networks – one for the actor and one for thecritic – to generate a consistent target Υ .
Both target networks slowly track their online counterparts.
The replay buffer stores every transition during training in a tuple of ( s t , a t , r t , s t +1 ) .
The actor andthe critic are then updated on randomly sampled experience from the replay buffer.
This breaks thecorrelation between samples of an episode and allows sample reuse.
Twin Delayed DDPG (TD3): Fujimoto et al.
[3] introduced three modiﬁcation to DDPG to furtherenhance performance and stabilize training.
Firstly, they delayed the actor updates to enhance thequality of action-values.
That is, while the critic is updated every d Q steps, the actor is updated everyd π steps, where d q < d π .
Secondly, they tackled the problem of narrow peaks in the action-valueestimate of deterministic policies by introducing target policy smoothing.
During target generation(c.f.
, Equation (3)), noise is added to the actions chosen by the policy.
Thirdly, they tackled theoverestimation bias of actor-critic architectures by introducing a second critic (and target critic),which are both used for target generation.
That is, Equation (3) is adapted to:
[Equation]
with ˜
[Equation]
∼ clip(
[Equation]
(4) where Q (cid:48) φ (cid:48) ,i is the target network of the i -th critic, π (cid:48) θ (cid:48) is the target network of the actor, and ˜ ε is anoise vector drawn from a Gaussian with zero mean and variance ˜ σ clipped between − c and c .
Evolution Strategies Instead of optimizing the policy parameters, evolution strategies optimize theparameters of a search distribution p ψ from which the policy parameters can be sampled, whereψ denotes the parameters of the search distribution.
Therefore, we reformulate the reinforcementlearning problem to a black-box optimization (BBO) problem.
The black-box function we want tooptimize is the ﬁtness F of an offspring ˜ θ sampled from p ψ on a complete episode under a parameterized policy π ˜ θ .
That is, the performance measure used to optimize the search distribution’s parameters is deﬁned as J ( ψ ) =
E ˜ θ ∼ p ψ ,s ∼ p
π [ F (˜ θ )]
=
E ˜ θ ∼ p ψ ,s ∼ p
π [ ˆ R
| π ˜ θ ] , where ˆ R 0 = (cid:80)
Ti =
r
( s i , a i ) isthe sum of undiscounted rewards when starting in an initial state s 0 and running a complete episodeunder policy π ˜ θ .
The parameters ψ can be updated using the gradient of J ( ψ ) known as searchgradient [27]:
[Equation]
OpenAI-ES OpenAI-ES [6] is a special type of evolution strategy, which uses an isotropic Gaussianwith a mean vector θ and a ﬁxed standard deviation as its search distribution p ψ ∼ N ( θ, σ 2 I ) .
Asthe standard deviation is ﬁxed, the search distribution’s parameters ψ only include the mean vector θ .
Inserting the isotropic Gaussian into Equation (5) and approximating the expectation by samplingyields:
[Equation]
where λ is the number of offsprings drawn from p ψ , N o is the number of episodes on which an offspring is evaluated on, and (cid:15) is sampled from N (0 , I ) as ˜ θ ∼ N ( θ, σ 2 I ) is equivalent to ˜ θ
= θ + σ(cid:15) .
OpenAI-ES uses mirrored sampling [28] and rank-based ﬁtness shaping [27].
The latter substitutesthe ﬁtness F j in Equation (6) with rank-based utilities u j , to render the algorithm invariant to rankpreserving (i.e, strictly monotonic) transformation of the ﬁtness function [29].
Policy Search with Redundant Action Bias
[Figure]
This section presents our method to exploit redundancy for biasing theactions in order to push the agent towards more desired state-space regions, i.e., regions satisfying secondary objectives.
Task Redundancy and Redundancy Constraint: To avoid interference with the main objective, we consider redundancy with respect tothe immediate reward of a state-action transition.
It is noted that thisis a simpliﬁed redundancy deﬁnition, as the actual main objective isdeﬁned by the cumulated reward.
A biased action ˆ a t = a t + ˆ b is redundant with respect to a t in s t , if the rewards of the biased and unbiased state-action transition are equal, i.e., the redundancy constraintis r ( s t , ˆ a t ) =
r ( s t , a t ) .
As the reward remains unchanged, we denotethis as task redundancy.
When the bias ˆ b does not affect the pose or theposition of the end-effector in a robotic task, it can be simpliﬁed to kinematic redundancy as it is considered here.
Figure 1 illustrates kinematicand task redundancy in a reaching task.
All rewards used are functionsof the end-effector position p ( e ) (c.f., Equation (13) and (15)) allowingus to express the redundancy constraint using the position of the end-effector after a state-actiontransition such that p ( e ) t +1 ( s t , ˆ a t ) =
p ( e ) t +1 ( s t , a t ) .
Problem setting – State-Space and Action-Space, and Redundancy resolution: Our task spacecomprises proprioceptive information of the 7-DOF Panda robot arm and a task-speciﬁc sub-state:joint positions q ∈ R 7 , joint velocities ˙ q ∈ R 7 , joint torques τ ∈ R 7 , joint position of the gripperﬁngers q g ∈ R 2 , a binary value κ g ∈ { 0 , 1 } indicating whether the gripper is open or not, anda task-speciﬁc sub-state ξ ∈ R N ξ (c.f., Equation (14)).
Relying on velocity control in order toapply standard robotic redundancy resolution methods, an action consists of desired joint velocities˙ q d ∈ R 7 and a binary value κ d ∈ { 0 , 1 } indicating whether the gripper should be closed or not:
[Equation]
In our tasks, we have redundancy in positioning of the M = 3 Cartesian DoF of the end-effector bymeans of the N = 7 DoF of the robot.
To resolve this redundancy, we generate motion ˙ q 0 in thenull-space of the Jacobian J v , which maps joint velocities ˙ q to linear velocities v of the end-effector:
[Equation]
) where J + v = J (cid:62) v (J v J (cid:62) ) − 1 is the right pseudoinverse of J v
and (I − J + v J v ) is a nullspace projector,which maps an arbitrary vector g ∈ R M into the Jacobian’s nullspace.
Secondary Objective and Action Bias: Through the choice g , a secondary objective can be expressed.
The only constraint is that the secondary objective is a function of q .
Our ﬁrst objective isto stay close to a desired reference joint conﬁguration ˜
q such that the loss can be deﬁned as:
[Equation] [Equation]
where W ∈ R N × N is a diagonal matrix, which weights the error in each dimension.
In doing so, wecan provide an upright joint conﬁguration ˜ q as a reference to avoid skewed conﬁgurations, which are more likely to cause contact with the table and look less natural.
We also deﬁne a loss for collisionavoidance in the sense that we want to maximize the distance between robot links and obstacles:
[Equation]
where d ij is the distance between the link of the i -th joint and the j -th obstacle, w ii are the diagonalelements of W , and K is the number of obstacles.
Regardless of the choice of loss L , g can bedeﬁned as a step into the direction of the negative gradient of L .
That is, ˙ q 0 can be computed with:
[Equation]
[Equation]
where α denotes a step-size parameter.
Now the velocities ˙ q 0 can be used to bias the policy withoutinterfering with the main task, i.e., changing the reward of a state-action transition:
[Equation]
with ˆ b = ( ˙ q 0 , [Equation] (cid:62) . To allow a comparison to evolution strategies, we add only the unbiased action a t to the replaybuffer, even though TD3 is able to learn on off-policy actions.
When using RAB, the secondary objective is solved in the space of redundant actions, which limits the optimization scope of secondaryobjectives and thus limits the interference with the main objective.
In contrast, when doing rewardshaping, both objectives are embedded in the reward and thus have a similar optimization scope;both objectives are optimized in the same action-space.
This inherently causes interference of theobjectives making reward shaping tedious to tune, as shown in [21].
Experiments
This section presents the results of this work.
We train TD3 and OpenAI-ES with and withoutredundancy resolution on a reaching task, and on a pick-and-lift task.
We evaluate the trained agentsboth in simulation and on the real robot.
Videos and further results can be found in the supplements.
5.1 Simulation Setup
RLBench is based on PyRep [30] and CoppeliaSim [31], and provides the reaching and pick-and-lifttasks.
However, some adaptions had to be made to the tasks and RLBench itself, as shown in thesupplements.
Further, our strategy for distributed training in TD3 can be found in the supplements.
Reaching Task
The goal in the reaching task is to put the robot’s tip at a target position – indicatedby a red sphere (c.f., Figure 3).
The target position is randomly sampled within a boundary infront of the robot.
The task-speciﬁc sub-state (c.f. Equation (7)) consists of the target position:
[Equation] ∈ R 3 .
RLBench only comes with sparse rewards yet allows reward shaping.
Weuse a distance-based reward r ( d ) for the reaching task:
[Equation]
[Equation] where d ( . ) is a distance function, d min is a minimum distance to limit the maximum reward, and βis a scaling parameter.
Pick-and-Lift Task:
The goal in this task is to pick an object – a cube with an edge length of 5 cm –and lift it to a target position.
The position of the object and the target position are sampled randomlywithin a boundary in front of the robot.
The task-speciﬁc sub-state consists of the position of theobject p ( obj ) t ∈ R 3 , the orientation of the latter encoded as a quaternion q ( obj ) t ∈ R 4 , the position ofthe target p ( tar ) t ∈ R 3 , and a binary value ι t indicating whether the object is grasped or not:
[Equation]
As for the reaching task, we use distance-based rewards, which are extended with sparse rewards for(partial) success:
[Equation]
) where ζ t is a binary value that is 1 . 0 for success and else 0 . 0 , r ( succ ) is the reward for success, ι t isbinary value that is 1 .
if the object is grasped and else 0 . 0 , and r ( grasp ) is the reward for graspingthe object.
¯ ζ t is the negation of ζ t .
[Figure] Training: All critics and actors are trained with an identical network architecture as in [2]: twohidden layers with 400 and 300 neurons.
For TD3, the Adam optimizer [32] was used with alearning rate of 1 e − 3 for the actor and the critic.
At each time step, the critic is updated on abatch of 100 samples drawn randomly from the replay buffer.
At the beginning of training, thereplay buffer is ﬁlled to half with experience from a pure random policy.
Then, the proportionof pure random actions is continuously decreased during training until its minimum.
In addition,uncorrelated exploration noise is added to the actions.
For the reaching task, we used an actiondelay of d π = 2 and, for the pick-and-lift, we used d π = 20 , which was needed to stabilize training.
For OpenAI-ES, we use a momentum optimizer with a learning rate of 1 e − 2 and used a batchsizeof 128 off-springs, which is signiﬁcantly less than in [6], to limit the computational costs.
Further,we found that the trained policy generalizes too much and lacks sensitivity to small changes in thestate.
This resulted in a policy that always puts the gripper in the center of the boundary.
We foundthat evaluating each off-spring on N episodes (c.f., Equation (6)) – instead of one as done by theoriginal authors – solved this problem.
5.2 Results
Figure 2 presents the validation results of TD3 and OpenAI-ES agents trained on the reaching andthe pick-and-lift task with and without redundancy resolution using a reference conﬁguration, whichis shown on the left side of Figure 3.
Note that the reward is shown per step for TD3 and per episodefor OpenAI-ES, as the latter demands ﬁxed episode lengths. Our intent is to show the differencein performance between an agent with and without RAB, and not between TD3 and OpenAI-ES.
Further, note that the RAB was not adapted across the different agents and tasks, i.e., the weights W and the step-size α (c.f., Equation (9) and (11)) to control the strength of the bias and thus theﬁnal orientation of the end-effector are kept ﬁxed. For each task, the main objective – consisting ofthe proportion of successful episodes and the reward – and the secondary objective – consisting ofthe loss L 1 (c.f., Equation (9)) – are shown. For agents not using redundancy resolution, the loss
L 1 is calculated for comparison, but is not optimized. For agents using redundancy resolution, theloss L 1 is optimized using RAB.
For those agents, the plots of the secondary objective show the bestsolution found under the current policy π θ . It is emphasized that the main objective is optimized overmultiple episodes – using TD3 or OpenAI-ES – while the secondary objective is optimized within anepisode using RAB. Starting with the TD3 agent in a reaching task (upper left plot), it can be seenthat our approach does not have a noteworthy effect on the main objective while minimizing the lossfor the secondary objective. Both agents solve the reaching tasks at the end of training.
However,
[Figure]
W i t h R e dund a n c y
R e s o l u t
i o n
W i t h o u t
R e dund a n c y
R e s o l u t
i o n
[Figure]
the one with redundancy resolution has signiﬁcantly less self-motion as shown in the supplementaryvideo.
As can be seen in the upper right plot, both OpenAI-ES agents were not able to fully solvethe reaching task.
However, it can be seen that RAB has not only a positive effect on the secondaryobjective but also on the main objective.
This is due to the fact that the agent without RAB learnedin two stages; at the beginning of training, it updated towards policies, which quickly brought theend-effector in the vicinity of the target position’s spawn boundary, and later it learned to reach thetarget position within the boundary.
This resulted in a similar two-stage behavior in the ﬁnal policy,where the agent quickly moves into the spawn boundary and then ﬂips the gripper to reach the target.
This ﬂipping behavior caused the high L 1 loss of the agent without RAB.
As RAB pushes the agenttowards an upright conﬁguration, the agent learned to smoothly reach the target without gettingtrapped in a two-stage behavior, which avoided the local optimum and increased ﬁnal performance.
Figure 3 presents an exemplary episode.
This result shows that the interference between the mainand secondary objective – due to our simpliﬁed redundancy constraint – can have a positive effectas well.
We hypothesize that such a positive effect arises when agents learn an unnatural behavior –here ﬂipping the gripper – and when both objectives are aligned well – it is natural to grasp from anupright conﬁguration.
TD3 was also able to solve the pick-and-lift task, as shown in the lower-leftplot.
As can be seen, RAB slowed down training yet reaches a similar performance at the end.
Wehypothesize that this is due to the reduction of one DoF of the end-effector when putting the gripperon the table, which might simplify learning to grasp at the beginning of training.
Further, it can beseen that the agent without RAB has a signiﬁcantly higher variance in performance throughout thetraining, which can be traced back to the greater variance of the state distribution.
Figure 4 presentsan exemplary episode of a TD3 agent with and without RAB on a simulated and a real pick-and-lifttask.
As can be seen, RAB again signiﬁcantly reduces the loss of the secondary objective and hasa positive inﬂuence on the orientation of the end-effector allowing it to grasp the object withoutphysical contact with the table.
This allows the straightforward transfer of the trained policy fromsimulation to the real robot without compromising safety.
Finally, the lower right plot shows the
[Figure]
performance of an OpenAI-ES agent on a pick-and-lift task.
As can be seen, neither of the agentswas able to solve the task as they get stuck in a local optimum, which is approaching the object.
Asfor TD3, RAB reduced the variance in performance and reduced the loss of the secondary objectiveyet led to slightly worse performance of the main objective.
RAB can be used for different desired end-effector orientations as well.
As an example, in Figure 5we provided a reference position where the end-effector is inclined to the right.
Passing the latter toa TD3 with RAB on a pick-and-lift task yielded a policy that grasps from the right, which was veryrarely experienced without RAB.
Note that such a reference position causes more interference withthe main objective, as the resulting policy generates more complex trajectories.
Interestingly, theagent learned to overcome the bias for some object positions resulting in policies that evenly graspthe object, similarly to Figure 4.
The video in supplements presents such a case.
This shows thattraining and resolving redundancy together in the same action-space allows a more ﬂexible adaptionby the agent in case of interference.
Finally, we have used the loss L 1 shown in Equation (10) to maximize the distance between therobot’s links and an obstacle.
Figure 6 presents the results of a TD3 agent on a reaching task.
Ascan be seen, RAB resulted in a policy that ducks under the obstacle.
It is emphasized that the agentcan not “see” the obstacle, as it is not part of its state-space nor is it included in the reward.
Notethat RAB does not guarantee collision-free policies when considering kinematic redundancy, as itdoes not control end-effector movement.
We provide more results and insights, especially on thereal robot, in the supplements.
Conclusion
We demonstrated the use of an action bias derived from the concept of redundancy resolution knownfrom multi-DoF robot kinematics to push the agent towards desired state-space regions in order tosatisfy secondary objectives.
This allows us to keep the reward function as simple as possible, whichstreamlines the setup of reinforcement learning environments.
We showed that our approach can beapplied across different tasks and policy search methods without much ﬁne-tuning and performanceloss while allowing ﬂexible adaption by the agent in case of interference with the main objective.
We argued that the limited optimization scope of RAB makes it more straightforward to apply, whencompared to reward shaping. Even though we have considered a simpliﬁed redundancy constraint, itis possible to extend this approach to action-spaces that are truly redundant to the main objective. Asan example, in the case of actor-critic algorithms, we propose to search in the critic’s nullspace foractions that have redundant action-values.
But future work has to investigate the actual size of thisnullspace and perhaps introduce approaches to deliberately expand the latter. Further, experimentshave to show the feasibility of this approach as the critic only provides an estimate of the actualaction-values.