  Humans can master a new task within a few trials by drawing upon skills
acquired through prior experience. To mimic this capability, hierarchical
models combining primitive policies learned from prior tasks have been
proposed. However, these methods fall short comparing to the human's range of
transferability. We propose a method, which leverages the hierarchical
structure to train the combination function and adapt the set of diverse
primitive polices alternatively, to efficiently produce a range of complex
behaviors on challenging new tasks. We also design two regularization terms to
improve the diversity and utilization rate of the primitives in the
pre-training phase. We demonstrate that our method outperforms other recent
policy transfer methods by combining and adapting these reusable primitives in
tasks with continuous action space. The experiment results further show that
our approach provides a broader transferring range. The ablation study also
shows the regularization terms are critical for long range policy transfer.
Finally, we show that our method consistently outperforms other methods when
the quality of the primitives varies.
