Training in Task Space to Speed Up and Guide Reinforcement Learning Recent breakthroughs in the reinforcement learning (RL) community have made significant advances towards learning and deploying policies on real world robotic systems. However, even with current stateoftheart algorithms and computational resources, these methods are still plagued with high sample complexity, and thus long training times, especially for high degree of freedom (DOF) systems. There are also concerns arising from lack of perceived stability or robustness from emerging policies. This paper aims at mitigating these drawbacks by: (1) modeling a complex, high DOF system with a representative simple one, (2) making explicit use of forward and inverse kinematics without forcing the RL algorithm to xe2x80x9clearnxe2x80x9d them on its own, and (3) learning locomotion policies in Cartesian space instead of joint space. In this paper, these methods are applied to JPLxe2x80x99s Robosimian, but they can be readily used on any system with a base and end effector(s). These locomotion policies can be produced in just a few minutes, trained on a single laptop. We compare the robustness of the resulting learned policies to those of other control methods.

A Comparison of Action Spaces for Learning Manipulation Tasks Designing reinforcement learning (RL) problems that can produce delicate and precise manipulation policies requires careful choice of the reward function, state, and action spaces. Much prior work on applying RL to manipulation tasks has defined the action space in terms of direct joint torques or reference positions for a jointspace proportional derivative (PD) controller. In practice, it is often possible to add additional structure by taking advantage of modelbased controllers that support both accurate positioning and control of the dynamic response of the manipulator. In this paper, we evaluate how the choice of action space for dynamic manipulation tasks affects the sample complexity as well as the final quality of learned policies. We compare learning performance across three tasks (peg insertion, hammering, and pushing), four action spaces (torque, joint PD, inverse dynamics, and impedance control), and using two modern reinforcement learning algorithms (Proximal Policy optimization and Soft ActorCritic). Our results lend support to the hypothesis that learning references for a taskspace impedance controller significantly reduces the number of samples needed to achieve good performance across all tasks and algorithms.