SCAPE: Learning Stiffness Control from Augmented Position Control Experiences
Anonymous Author(s)
AfﬁliationAddress email
Abstract: We introduce a sample-efﬁcient method for learning state-dependentstiffness control policies for dexterous manipulation.
The ability to control stiffness facilitates safe and reliable manipulation by providing compliance and robustness to uncertainties.
Most current reinforcement learning approaches toachieve robotic manipulation have exclusively focused on position control, often due to the difﬁculty of learning high-dimensional stiffness control policies.
This difﬁculty can be partially mitigated via policy guidance such as imitationlearning.
However, expert stiffness control demonstrations are often expensiveor infeasible to record.
Therefore, we present an approach to learn Stiffness
Control from Augmented Position control Experiences (SCAPE) that bypassesthis difﬁculty by transforming position control demonstrations into approximate,suboptimal stiffness control demonstrations.
Then, the suboptimality of the augmented demonstrations is addressed by using complementary techniques that helpthe agent safely learn from both the demonstrations and reinforcement learning.
By using simulation tools and experiments on a robotic testbed, we show that theproposed approach efﬁciently learns safe manipulation policies and outperformslearned position control policies and several other baseline learning algorithms.
Introduction
In recent years, deep reinforcement learning has been successfully used to improve object manipulation with robotic hands [1, 2].
However, one of the primary limitations of these works is that in mostrobotic hands, the robot joint poses are explicitly controlled through position control and forces areimplicitly decided.
Lack of explicit control over the forces leads to limited safety and inability tohandle uncertainties [3].
These might be critical factors when a robot is operating in unstructuredenvironments and during the exploratory phase of the learning process [4, 5].
Modulation of stiffness in concert with position control has been shown to address robustness, safety,and performance under uncertainties [6, 7, 8] and has gained much attention in the learning community as well [9].
However, stiffness control imposes additional action dimensions, which affectssample-efﬁciency of policy learning.
This hindrance can be partially mitigated through guidancevia imitation learning [10].
Expert demonstrations have been successfully collected and used in policy learning for position control-based robotic hands [1, 11, 12].
However, in the case of stiffnesscontrol, such expert demonstrations are expensive and difﬁcult to acquire. Typical demonstrationscan directly be recorded via various sensors, but stiffness is not a measurable quantity, but rather arelationship.
In prior literature, admittance control has been used to capture equilibrium position trajectories [13,14].
Subtle and quick impact perturbations are used to measure the compensatory forces and torquesemployed by the demonstrator while performing the trajectory, which are then used to implicitlycalculate the stiffness at certain positions.
These stiffness estimates are then used to further estimateand model the stiffness proﬁles, thereby compounding potential errors.
This estimation process ismore ambiguous for object manipulation due to the required precision and accuracy, and thereforeposes a major challenge to learning stiffness control from demonstrations.
In this paper, we present a novel learning strategy—Stiffness Control from Augmented Position control Experiences (SCAPE)—for learning state-dependent stiffness control policies in highdimensional problems such as dexterous manipulation.
Imitation learning is used in conjunctionwith reinforcement learning to provide policy guidance, and we propose a way to bypass the needfor stiffness demonstrations through an augmentation process.
This process leverages the knowledge of the robot model such that we do not require expert stiffness control demonstrations.
Instead,position control demonstrations are augmented to infer approximate, suboptimal stiffness controldemonstrations.
To address this suboptimality, we use a Q-ﬁlter [11] to prevent the agent frommimicking dangerous choices that may appear in the inferred stiffness demonstrations.
We also introduce the concept of an imitation regulator that controls the mode of imitation depending on theassessment of the current policy.
Ablation studies show that these techniques play meaningful rolesin both safety and stability of learning.
Through simulation and experiments, we show that SCAPEproduces a successful policy that is robust to different types of realistic uncertainties, and safe interms of force interaction.
Background and Related Works
Some of the notable past works in learning stiffness control rely on a reference trajectory, which werefer to as trajectory-dependent approaches.
However, the lack of robustness to uncertainties andvariability in the task dynamics renders these approaches inapplicable to dexterous manipulation.
Other recent research aims to learn a stiffness controller that dynamically adapts to the environment,which we refer to as state-dependent approaches.
In this section, we brieﬂy explain some of thenotable attempts to learn stiffness control, and discuss possible shortcomings.
2.1 Trajectory-dependent Stiffness Controllers
One possible approach is to learn time-indexed gain scheduling through PI 2 [15], which is a stochastic optimization method that results in a time-indexed reference trajectory that can be tracked by therobot without modeling the inverse kinematics or dynamics.
This approach adds additional parameters to control compliance so that the resulting controller takes environment dynamics into accountand modulates the gains accordingly [16, 17, 18].
However, a solution from PI 2 can only optimize about a pre-deﬁned cost function and cannot be used for object-centered manipulation, whichrequires highly divergent position and stiffness trajectories depending on the goal and observations.
Another approach uses an Incremental Gaussian Mixture Model (IGMM) and Gaussian Mixture Regression (GMR) to predict the interaction force for the next time-step, and feed-forward appropriatecontrol effort [19, 20].
The goal of this approach is to learn a feed-forward model such that the feedback stiffness control effort can be minimized.
However, this approach makes a critical assumptionthat expert demonstrations with force trajectories are available, which renders it inapplicable withoutsuch demonstrations.
Trajectory planners combined with reinforcement learning can also be used.
Once the trajectory isdeﬁned, a residual control policy can be learned to adjust the gains according to the current observation.
This method is mostly used in simple tasks where a trajectory planner is readily available,such as in peg-in-hole assembly tasks [21, 22, 23].
While this is a suitable approach for closed environments, it is less effective for dexterous manipulation where desired trajectories can change basedon dynamic observations such as dropping the object or unexpected interaction forces.
In addition,due to the lack of policy guidance, the learning process requires a complex reward function as wellas an extensive amount of training time even with a trajectory planner aiding the policy search.
2.2 State-dependent Stiffness Controllers
Due to the speciﬁcity of the solution, relying on a ﬁxed reference trajectory or scheduled gain isbound to result in catastrophic failure in dynamically changing environments.
To account for a largedegree of variability in the environment, state-dependent stiffness control policies have recently beenproposed.
In this paper, we compare our work with these state-dependent methods, as the existingtrajectory-dependent approaches are inapplicable in dynamic settings.
In one related approach, during hopping and wiping motions [24], the robots successfully producestiffness control policies that outperform direct torque control and position control policies.
How-
(s, g)
Desired stiffness Desired robot pose
Current robot pose
Error
Environment
Robot
Controller
Commanded robot pose Interaction
Self-Imitation Learning
Imitation Learning
Augmented Demonstration
Experience Buffer
Imitation regulator
Policy Update
Experience Buffer (s, π(s, g), r, g)
SR
Policy Output π(s, g) (s, g)
SR
SR < SR ref ?
SR ≥ SR ref ?
(s, g)
Desired stiffness Desired robot pose
Current robot pose
Error
Environment
Robot
Controller
Commanded robot pose Interaction
Self-Imitation Learning
Imitation Learning
Augmented Demonstration
Experience Buffer
Imitation regulator
Policy Update
Experience Buffer (s, π(s, g), r, g)
SR
Policy Output π(s, g) (s, g)
SR
SR < SR ref ?
SR ≥ SR ref ?
(a)
Augmented Demonstrations (s', a') pairs (s, a) pairs
Robot Model
Inherent Stiffness (k passive )
Position Control Demonstrations k passive Augmented Demonstrations (s', a') pairs (s, a) pairs
Robot Model
Inherent Stiffness (k passive )
Position Control Demonstrations k passive (b) ever, this approach is strictly limited to simple and repetitive low-dimensional movements.
For instance, the robot is allowed to move only in one direction in the hopping task, or along a pre-deﬁnedcircular path in the wiping task.
A similar work demonstrates the performance difference based on the form of the control policy [25],where variable impedance control outperforms all other controller types in simple tasks.
Despite theeffort to simplify the problem into a lower-dimensional manifold by keeping the gripper closed inthe door opening task, the variable impedance control (VIC) fails to outperform the ﬁxed impedancecontrol.
A possible culprit can be the high dimensionality induced by VIC. A similar work [26]observes stability-guaranteed learning for VIC during peg-in-hole tasks.
Other work depicts a task-space impedance controller’s performance in quadruped locomotion [27].
Variable impedance control policies outperform the direct torque control policy in terms of cumulative rewards and robustness to disturbances.
Interestingly, impedance controllers resulted in moreenergy-efﬁcient policies although the reward function did not consider energy.
It is worth notingthat an extensively tuned gait planner was provided to the agent to learn a successful policy.
While these state-dependent approaches show promising results in terms of robustness to uncertainties, these existing approaches focus on simple single-stage, repetitive tasks and require extensivereward shaping due to the lack of policy guidance.
In the next section, we explain how SCAPEaddresses this issue and produces successful state-dependent stiffness control policies, which can beused in multi-stage tasks such as grasping and manipulation.
Methods
In this section, we present our approach, SCAPE, to learn state-dependent stiffness control withoutrequiring stiffness control demonstrations.
Use of imitation learning within reinforcement learningimproves policy guidance, thereby enabling high-dimensional dexterous manipulation.
For SCAPEand all the baselines, we employ Deep Deterministic Policy Gradient with Hindsight Experience Replay (DDPG + HER, [11]).
The overall learning scheme is depicted in Fig.
1a, where the stiffness control policy takes in observed states s and the goal g , and produces an action π ( s, g ) whichcontains the desired stiffness in addition to the desired pose.
The controller block is the high-levelcontroller, which employs task-space stiffness control.
SR refers to the overall success rate of thecurrent policy (i.e., how often does the object reach the goal states while staying intact?).
A commonly used action representation in dexterous manipulation describes only the desired position of the actuator in the inner control loop, hence the name position control.
On the other hand, astiffness control policy outputs actions that describe the desired position of the actuators as well asthe desired stiffness in the corresponding joint, as shown in Fig.
1a.
However, as explained earlier,it is difﬁcult to obtain expert stiffness control demonstrations for imitation learning.
Therefore, weuse augmented position control demonstrations as shown in Fig
1b.
These demonstrations are typically generated from teleoperation or kinesthetic teaching, and containdesired position trajectories. In our study, we use 25 demonstrations that accomplish task-relatedkinematic goals without consideration for the object fragility (e.g., commanded to fully close thegrippers).
We leverage the fact that the stiffness of the robot is known either from the simulationmodel or the hardware speciﬁcations, and that position control works by moving to the desired position with the inherent stiffness or position gain of the actuator.
In this paper, we refer to thisinherent stiffness as k passive .
Therefore, state-action pairs, ( s, a ) , in common position controldemonstrations can be augmented to that of stiffness control demonstrations, ( s (cid:48) , a (cid:48) ) , where thedesired stiffness is k passive .
Consequently, these augmented demonstrations become suboptimalstiffness control demonstrations since the commanded stiffness of the robot is ﬁxed to k passive .
Wecan then infer the reward function of the task from the augmented demonstrations without manualreward shaping, and at the same time use it to learn improved stiffness control.
Note that k passiveis n-dimensional stiffness, from which we can choose any dimension of interest and modulate thestiffness.
In this paper, we modulate the stiffness in the grasping dimension (i.e., k passive ∈ R ).
3.1 Outperforming the Demonstration
Simple imitation learning in the form of behavioral cloning does not allow the agent to improve beyond the performance of the demonstrations due to the cloning loss [11].
The weight of the cloningloss can be decreased iteratively, assuming that the agent is able to learn a policy equivalent to thedemonstrator early in the iteration [1].
But it is unclear how to determine the amount of dependencyon the demonstrations with respect to the iteration.
Also, simply reducing the dependency does notprevent the agent from cloning the undesirable behaviors seen in the suboptimal demonstrations weuse.
Therefore, we adopt additional techniques to encourage the policy to outperform the augmenteddemonstrations and address its suboptimality.
3.1.1 Q-Filter
We use a Q-ﬁlter [11] to choose which replay transitions to clone from.
The fundamental motivation behind learning from demonstrations is the assumption that the demonstrations provide anear-optimal action.
However, this is not true for the augmented demonstrations.
A Q-ﬁlter allowsthe agent to compare the Q values produced by the transitions from demonstrations, ( s i , a i , g ) , andthe current policy, ( s i , π θ ( s i , g ) , g ) .
By comparing their values, the agent does not clone the behavior if its current policy provides a better action for the given demonstration state.
More formally, thecloning loss L bc can be deﬁned as:
[Equation]
However, it is often difﬁcult to infer the subtle difference in the qualities of the policies solely fromexamining the resulting Q estimates due to the overestimation issue of Q values [28].
Although theusage of the Q-ﬁlter improves the safety of learning, it tends to produce oscillatory gradients thatprevent convergence of the policy due to its Boolean property [11].
3.1.2 Imitation Regulator
To improve convergence of our method, we introduce an imitation regulator that observes the overallsuccess rates of the current policy and determines the appropriate source of imitation from: 1) theaugmented demonstrations and 2) the agent’s own past experience.
The latter is sometimes referredto as self-imitation learning [29].
The regulator controls the replay buffer D IR used for samplingtransition batches to imitate as follows:
[Equation]
[Equation]
where D SIL and D demo refer to the buffers that store the actual experience replay and the augmented demonstrations, respectively.
SR ∈ [0 , 1] is the overall success rate of the current policy.
Itis considered an overall success if the object reaches the goal states and also stays intact throughoutthe episode.
SR ref is the reference success rate that is empirically found.
Put simply, the regulatoractively switches the source of demonstrations from D demo to D SIL once the success rate reaches
SR ref .
This brings three primary beneﬁts.
First, the agent no longer references the suboptimaldemonstrations which contain undesirable behaviors.
Second, the policy converges faster from theminimized oscillation of gradients.
Third, by cloning the previously generated actions, the agentleverages exploration, thereby improving upon its current policy.
[Figure]
Environments
We use three robotic manipulation environments shown in Fig.
2 for simulation and experiments.
Details for each environment can be found in Appendix A. For simulation, we use robots provided by OpenAI Gym [30].
In all environments, we do not use the ground-truth force measurements duringtraining.
Instead, we use quasi-static force measurements from the series elasticity of the robot,which do not require force sensors (i.e., F ≈ k passive ( x desired − x current ) ).
Note that x is deﬁnedin the same coordinate frame as k passive .
For safety evaluation however, unless stated otherwise,we use ground-truth force measurements.
A successful policy must meet both task-related (e.g., didthe object reach the goal states?) and safety-related (e.g., is the object intact?) goals.
For instance,even if the object reaches the goal states, the task is considered a failure if the applied force exceedsthe breaking force.
The task-related kinematic reward is r task ( s ) : s → {− 1 , 0 } , and the safetyrelated reward is r safety ( s ) : s → ( −∞ , 0] .
Combining these reward functions naturally leads theagent to accomplish the kinematic goal while minimizing the estimated interaction force.
For allenvironments, the immediate reward R for the observation s is deﬁned as:
[Equation]
if kinematic goal is met
[Equation]
otherwise
[Equation]
where F is the estimated force, ˙q is the joint velocity, and α and β are normalization coefﬁcientsthat depend on the environment.
Furthermore, we implement a low-pass ﬁlter with a time constant of 0 .
05 s for all force measurements in the simulation.
To validate the robustness under realistic conditions, we include three typesof uncertainties during training: 1) random perturbation to the object within grasp, 2) measurementnoise in the object’s position, and 3) random control failure, that repeats the previous action.
4.1 Quasi-static Pick-and-place Environment (Block)
The Block environment entails a pick-and-place task and is used to verify whether SCAPE is capableof learning a safe and robust manipulation policy under quasi-static assumptions.
In this environmentwe assume that the ground-truth force matches the estimated force, which is reasonable to assumewhen the object is in grasp and the involved masses are small enough.
The observation includesrelative positions between the object, gripper, and the goal, as well as the gripper conﬁgurations,estimated force, and stiffness. The action includes Cartesian movement of the gripper, change in thegripper conﬁgurations, the changes in stiffness and its limit.
4.2 Dynamic Pick-and-place Environment (Chip)
Chip environment is a dynamic version of the pick-and-place environment, designed to demonstratethat SCAPE produces a successful manipulation policy even in dynamic situations where the agentdoes not have access to ground-truth force measurements.
The robot is required to slide the objectup the wall using friction.
Thus, the ground-truth force comes from not only the ﬁnger, but also fromfriction, which depends on velocity and normal force.
The observation and action spaces are similarto the Block environment, but the observation also includes the ﬁngertip velocity in Cartesian space,which is part of the kinematic goal in this case. If only position is considered for the kinematic goal,we have found that the agent constantly moves the object around the goal location. We postulatethat this phenomenon arises from kinetic friction being smaller than static friction.
4.3 In-hand Manipulation Environment (NuFingers)
To demonstrate the applicability for in-hand manipulation, we use the NuFingers testbed [7].
Weﬁrst train the agent with domain randomization [2] in a representative environment on Gym, anddirectly transfer the policy to the robot without any ﬁne-tuning to validate its transferability androbustness under uncertainties.
The ground-truth force measurements are only used for validation.
The observation includes polar coordinates and relative orientation from the object of each ﬁnger,forces, joint velocities, and stiffness.
The action includes deﬁne radial and tangential movementsof the ﬁngers.
Stiffness modulation is applied in the radial direction, which dominates the graspingforce.
The task-related kinematic goal is to rotate the object to the desired orientation.
Results
In this section, we demonstrate the performance of SCAPE in various environments.
Withoutthe proposed augmentation process, baseline algorithms must learn a stiffness control policy fromscratch [24, 25, 27], since expert demonstrations are not available.
We compare SCAPE with theseapproaches for the ablation study in Sec.
5.2, as learning from scratch fails catastrophically.
Forthe main experiments in Sec.
5.1, we compare our results with position control (existing approach),so that the difference only lies in the policy parametrization.
This comparison demonstrates theimportance of using state-dependent stiffness controllers when force-sensitive tasks are involved, asopposed to existing position counterparts that are widely used in dexterous manipulation [1, 2].
5.1 Experimental Results
Figure 3 depicts the success rates over epochs for the Block, Chip, and NuFingers environments.
For in-depth assessment, we break down the plots to also show success rates for the safety andkinematic goals.
Safety-related success rates refer to the fraction of the time when the experiencedforce is smaller than the breaking threshold.
Note that exceeding it at any time results in an overallfailure in the corresponding episode.
Kinematic pick-and-place tasks without uncertainties have been easily solved by the position controlpolicies in previous works [31].
However, Fig.
3a shows that for position control policies, the heavyforce penalty from large forces discourages exploration and prevents the agent from learning to evengrasp the object.
SCAPE on the other hand, uses stiffness control policies to successfully minimizethe interaction force and reaches a 100% overall success rate. [Figure] cce ss R a t e s
[Figure]
Q - v a l u e
F o r ce
S ti ff n e ss
F o r ce
E rr o r fr o m G o a l (after training) (evaluation)
Time
Epoch
Q - v a l u e
F o r ce
S ti ff n e ss
F o r ce
E rr o r fr o m G o a l (after training) (evaluation)
Time
Epoch
[Figure]
Q - v a l u e
F o r ce
S ti ff n e ss
F o r ce
E rr o r fr o m G o a l (evaluation) (after training)
Time
Epoch
Q - v a l u e
F o r ce
S ti ff n e ss
F o r ce
E rr o r fr o m G o a l (evaluation) (after training)
Time
Epoch
[Figure]
Q - v a l u e
F o r ce
S ti ff n e ss
F o r ce
E rr o r fr o m G o a l (after training) (evaluation)
Time
Epoch
Q - v a l u e
F o r ce
S ti ff n e ss
F o r ce
E rr o r fr o m G o a l (after training) (evaluation)
Time
Epoch
[Figure]
For the Chip environment where the action space is smaller, the position-controlled agent learns tocomplete the kinematic task entailed in the demonstrations to some degree as shown in Fig.
3b.
However, it fails to address the safety issues, thereby breaking the object in almost all of the evaluation episodes.
Note that unlike in the Block environment, the ground-truth force measurements areused for evaluation.
It is notable that SCAPE still produced a successful policy by referencing onlythe quasi-static force measurements from the series elasticity, which do not include friction.
For the NuFingers environment in Fig.
3c, we ﬁnd a similar trend as in the Block environment.
Theposition control approach fails to learn a policy that completes the kinematic task.
SCAPE however,reaps the beneﬁts of stiffness control and ﬁnds a successful policy.
Most importantly, in spite of themodel discrepancies between the simulation and the actual robot, the resulting policy proves to besuccessful after the sim-to-real transfer without additional training.
To summarize, it is evident thata state-dependent stiffness control policy outperforms the position control policy in terms of safetyand robustness under uncertainties and that the SCAPE is capable of producing successful policies.
Figure 4 depicts various data during and after training.
It is evident that the proposed approachquickly learns the necessary stiffness for the completion of the kinematic task.
The stiffness andinteraction force of the system are strongly related to one another as can be seen from the similartrends of the two curves.
The existing approach which uses position control on the other hand, doesnot adjust the stiffness and therefore fails to reduce the interaction force.
Also, notice that the objects in the Chip and NuFingers environments experience much more forcethan what the robot can exert.
This is because for evaluation, we use the ground-truth forces whichcome from various sources, such as the friction that depends on the normal force and the velocity ofthe object.
The proposed approach successfully minimizes the ground-truth force without the actualmeasurement. While actual measurements will improve the stability, we expect the improvement tobe marginal in the tested environments since large masses or explosive movements are not involved.
5.2 Ablation Study
To examine the effects of each technique used in this paper, we examine the Block environmentunder ﬁve different conditions:
• Condition 1: Reinforcement learning from scratch [24, 25, 27], without a Q-ﬁlter, withoutan imitation regulator
• Condition 2: Reinforcement learning + imitation learning from augmented demonstrations,without a Q-ﬁlter, without an imitation regulator
• Condition 3:
Reinforcement learning + imitation learning from augmented demonstrations,without a Q-ﬁlter, with an imitation regulator
[Figure]
• Condition 4:
Reinforcement learning + imitation learning from augmented demonstrations,
with a Q-ﬁlter, without an imitation regulator
• Condition 5:
Reinforcement learning + imitation learning from augmented demonstrations,
with a Q-ﬁlter, with an imitation regulator (SCAPE)
Condition 1 represents the approach taken by the researchers in the past [24, 25, 27], where imitationlearning was not used due to the absence of demonstration data.
Conditions 2-5 use the augmenteddemonstrations introduced in this paper, with different combinations of complementary techniques.
Note that condition 5 is used to produce the results in Fig.
Overall success rates generated under the different conditions are shown in Fig.
5a.
From theseresults, we conﬁrm that existing approaches [24, 25, 27] do not produce any meaningful results formulti-stage tasks (e.g., approaching an object, grabbing the object, and relocating the object).
Forsuch tasks, the augmented demonstrations play a crucial role in providing guidance to the policythrough imitation learning.
Moreover, in conditions 2 and 3, the agent is unable to ﬁlter out undesirable behaviors, therebyconsistently breaking the object during exploration as shown in Fig.
5b.
Therefore, we conﬁrm thatthe Q-ﬁlter allows the agent to make safe decisions.
Also, applying the imitation regulator withoutthe Q-ﬁlter fails from satisfying safety goals.
Lastly, without the imitation regulator, the agentkeeps referencing the suboptimal demonstrations, which causes oscillations and delays convergence.
Switching to self-imitation learning using the imitation regulator helps reinforce some of the pastgood behaviors preventing the policy from diverging, which is shown by the higher mean and smallervariance of the condition 5 compared to those of condition 4.
Conclusions
We conclude that our approach, SCAPE, is capable of producing a successful state-dependent stiffness control policy, which plays a crucial role in ensuring safety and performance in dexterousmanipulation.
Most existing approaches for robotics manipulation rely on position control and donot consider safety as part of the goal.
Therefore, they yield only kinematically satisfactory results.
However, in more realistic situations where safety and robustness under uncertainties are also critical, the state-dependent stiffness control policies signiﬁcantly outperform position control.
Existingattempts to learn state-dependent stiffness control policies are challenged by a multitude of practicalfactors whereas SCAPE produces competent manipulation skills by inferring important informationfrom position control demonstrations.
The suboptimality of the augmented demonstrations is alleviated by a combination of the Q-ﬁlter and the imitation regulator, which results in faster and morestable convergence to a successful policy.
These techniques prevent the agent from blindly imitatingthe suboptimal demonstrations, and help focus on the past desirable experience.
Through variousexperiments, we validate that SCAPE outperforms the existing approaches in learning successfulstiffness control policies.
Therefore, SCAPE ensures both safety and performance such that robustdexterous manipulation can be conveniently learned without expert demonstrations.