We propose a novel reinforcement learning framework that performs self-supervised online reward shaping, yielding faster, sample efficient performance in sparse-reward environments  . The proposed framework alternates between updating a policy and inferring a reward function. While the policy update is performed with the inferred, potentially dense reward function, the original sparse reward is used to provide a self-supervisory signal for the reward update by serving as an ordering over the observed trajectories . The proposed framework is based on the theory that altering the reward function does not affect the optimal policy of the original MDP as long as certain relations between the altered and the original reward are maintained. We name the proposed framework ClAssification-based Reward Shaping (CaReS), since the altered reward is learned in a self-supervised manner using classifier-based reward inference . Experimental results on several sparse-reward environments demonstrate that  the proposed algorithm is not only significantly more sample efficient than the state-of-the-art reinforcement learning baseline but also achieves a similar sample efficiency to a baseline that uses hand-designed dense reward functions  .
We introduce Self-supervised Online Reward Shaping (SORS), which aims to improve the sample efficiency of any RL algorithm in sparse-reward environments by automatically densifying rewards . The proposed framework alternates between classification-based reward inference and policy update steps -- the original sparse reward provides a self-supervisory signal for reward inference by ranking trajectories that the agent observes, while the policy update is performed with the newly inferred, typically dense reward function. We introduce theory that shows that, under certain conditions, this alteration of the reward function will not change the optimal policy of the original MDP , while potentially increasing learning speed significantly . Experimental results on several sparse-reward environments demonstrate that , across multiple domains, the proposed algorithm is not only significantly more sample efficient than a standard RL baseline using sparse rewards, but, at times, also achieves similar sample efficiency compared to when hand-designed dense reward functions are used .
R
propose a novel reinforcement learning framework that performs self-supervised online reward shaping, yielding faster, sample efficient performance
introduce Self-supervised Online Reward Shaping (SORS), which aims to improve the sample efficiency of any RL algorithm
A
by automatically densifying rewards
R
updating a policy and inferring a reward function. While the policy update is performed with the inferred, potentially dense reward function, the
classification-based reward inference and policy update steps -- the
R
is used to provide
provides
R
the reward update by serving as an ordering over the observed trajectories . The proposed framework is based on the theory that altering
reward inference by ranking trajectories that the agent observes, while the policy update is performed with the newly inferred, typically dense reward function. We introduce theory that shows that, under certain conditions, this alteration of
R
does not affect
will not change
R
as long as certain relations between the altered and the original reward are maintained. We name the proposed framework ClAssification-based Reward Shaping (CaReS), since the altered reward is learned in a self-supervised manner using classifier-based reward inference
, while potentially increasing learning speed significantly
A
, across multiple domains,
R
the state-of-the-art reinforcement learning baseline but also achieves a
a standard RL baseline using sparse rewards, but, at times, also achieves
R
to a baseline that uses
compared to when
A
are used