SimtoReal Transfer for Biped Locomotion We present a new approach for transfer of dynamic robot control policies such as biped locomotion from simulation to real hardware. Key to our approach is to perform system identification of the model parameters mu of the hardware (e.g. friction, centerofmass) in two distinct stages, before policy learning (presysID) and after policy learning (postsysID). PresysID begins by collecting trajectories from the physical hardware based on a set of generic motion sequences. Because the trajectories may not be related to the task of interest, presysID does not attempt to accurately identify the true value of mu, but only to approximate the range of mu to guide the policy learning. Next, a Projected Universal Policy (PUP) is created by simultaneously training a network that projects mu to a lowdimensional latent variable eta and a family of policies that are conditioned on eta. The second round of system identification (postsysID) is then carried out by deploying the PUP on the robot hardware using taskrelevant trajectories. We use Bayesian Optimization to determine the values for eta that optimize the performance of PUP on the real hardware. We have used this approach to create three successful biped locomotion controllers (walk forward, walk backwards, walk sideways) on the Darwin OP2 robot.

Fast and Safe Policy Adaptation via Alignmentbased Transfer Applying deep reinforcement learning to physical systems, as opposed to learning in simulation, presents additional challenges in terms of sample efficiency and safety. Collecting large amounts of hardware demonstration data is timeconsuming and the exploratory behavior of reinforcement learning algorithms may lead the system into dangerous states, especially during the early stages of training. To address these challenges, we apply transfer learning to reuse a previously learned policy instead of learning from scratch. In this paper, we propose a method where given a source policy, policy adaptation is performed via transfer learning to produce a target policy suitable for realworld deployment. For policy adaptation, alignmentbased transfer learning is applied to trajectories generated by the source policy and their corresponding safe target trajectories. We apply this method to manipulators and show that the proposed method is applicable to both intertask and interrobot transfer whilst considering safety. We also show that the resulting target policy is robust and can be further improved with reinforcement learning.