Accelerating the learning processes for complex tasks by leveraging previously learned tasks has been one of the most challenging problems in reinforcement learning, especially when the similarity between source and target tasks is low or unknown. In this work , we propose a REPresentation-And-INstance Transfer algorithm (REPAINT) for deep actor-critic reinforcement learningparadigm. In representation transfer, we adopt a kickstarted training method using a pre-trained teacher policy by introducing an auxiliary cross-entropy loss. In instance transfer, we develop a sampling approach, i.e., advantage-based experience replay, on transitions collected following the teacher policy , where only the samples with high advantage estimates are retained for policy update. We consider both learningan unseen target task by transferring from previously learned teacher tasks and learning a partially unseen task composed of multiple sub-tasks by transferring from a pre-learned teacher sub-task. In several benchmark experiments, REPAINT significantly reduces the total training time and improves the asymptotic performance compared totraining with no prior knowledge and other baselines .
Accelerating learning processes for complex tasks by leveraging previously learned tasks has been one of the most challenging problems in reinforcement learning, especially when the similarity between source and target tasks is low . This work proposes REPresentation And INstance Transfer (REPAINT) algorithm for knowledge transfer in deep reinforcement learning. REPAINT not only transfers the representation of a pre-trained teacher policy in the on-policy learning, but also uses an advantage-based experience selection approach to transfer useful samples collected following the teacher policy in the off-policy learning. Our experimental results on several benchmark tasks show that REPAINT significantly reduces the total training time in generic cases of task similarity. In particular, when the source tasks are dissimilar to, or sub-tasks of, the target tasks, REPAINT outperforms other baselines in both training-time reduction and asymptotic performance of return scores .
D
the
R
or unknown. In this work , we propose a REPresentation-And-INstance Transfer algorithm
. This work proposes REPresentation And INstance Transfer
R
for deep actor-critic reinforcement learningparadigm. In representation transfer, we adopt a kickstarted training method using a
algorithm for knowledge transfer in deep reinforcement learning. REPAINT not only transfers the representation of a
R
by introducing an auxiliary cross-entropy loss. In instance transfer, we develop a sampling approach, i.e.,
in the on-policy learning, but also uses an
R
replay, on transitions
selection approach to transfer useful samples
R
, where only the samples with high advantage estimates are retained for policy update. We consider both learningan unseen target task by transferring from previously learned teacher tasks and learning a partially unseen task composed of multiple sub-tasks by transferring from a pre-learned teacher sub-task. In several benchmark experiments,
in the off-policy learning. Our experimental results on several benchmark tasks show that
R
and improves the asymptotic performance compared totraining with no prior knowledge and other baselines
in generic cases of task similarity. In particular, when the source tasks are dissimilar to, or sub-tasks of, the target tasks, REPAINT outperforms other baselines in both training-time reduction and asymptotic performance of return scores