Designing agents that acquire knowledge autonomously and use it to solve new tasks efficiently is an important challenge in reinforcement learning , and unsupervised learning provides a useful paradigm for autonomous acquisition of task-agnostic knowledge. In supervised settings, representations discovered through unsupervised pre-training offer important benefits when transferred to downstream tasks . Given the nature of the reinforcement learning problem, we argue that representation alone is not enough for efficient transfer in challenging domains and explore how to transfer knowledge through behavior. The behavior of pre-trained policies may be used for solving the task at hand (exploitation), as well as for collecting useful data to solve the problem (exploration ). We argue that policies pre-trained to maximize coverage will produce behavior thatis useful for both strategies. When using these policies for both exploitation and exploration, our agents discover better solutions  . The largest gains are generally observed in domains requiring structured exploration, including settings where the behavior of the pre-trained policies is misaligned with the downstream task.
Designing agents that acquire knowledge autonomously and use it to solve new tasks efficiently is an important challenge in reinforcement learning . Knowledge acquired during an unsupervised pre-training phase is often transferred by fine-tuning neural network weights once rewards are exposed, as is common practice in supervised domains . Given the nature of the reinforcement learning problem, we argue that standard fine-tuning strategies alone are not enough for efficient transfer in challenging domains . We introduce Behavior Transfer (BT), a technique that leverages pre-trained policies for exploration and that is complementary to transferring neural network weights. Our experiments show that, when combined with large-scale pre-training in the absence of rewards, existing intrinsic motivation objectives can lead to the emergence of complex behaviors. These pre-trained policies can then be leveraged by BT to discover better solutions than without pre-training, and combining BT with standard fine-tuning strategies results in additional benefits . The largest gains are generally observed in domains requiring structured exploration, including settings where the behavior of the pre-trained policies is misaligned with the downstream task.
R
, and unsupervised learning provides a useful paradigm for autonomous acquisition of task-agnostic knowledge. In supervised settings, representations discovered through
. Knowledge acquired during an
R
offer important benefits when transferred to downstream tasks
phase is often transferred by fine-tuning neural network weights once rewards are exposed, as is common practice in supervised domains
R
representation alone is
standard fine-tuning strategies alone are
R
and explore how to transfer knowledge through behavior. The behavior of
. We introduce Behavior Transfer (BT), a technique that leverages
R
policies may be used for solving the task at hand (exploitation), as well as for collecting useful data to solve the problem (exploration ). We argue that policies pre-trained to maximize coverage will produce behavior thatis useful for both strategies. When using these policies for both exploitation and exploration, our agents
policies for exploration and that is complementary to transferring neural network weights. Our experiments show that, when combined with large-scale pre-training in the absence of rewards, existing intrinsic motivation objectives can lead to the emergence of complex behaviors. These pre-trained policies can then be leveraged by BT to
A
than without pre-training, and combining BT with standard fine-tuning strategies results in additional benefits