Reward shaping (RS) is a powerful method in reinforcement learning (RL) for overcoming the problem of sparse and uninformative rewards. However, RS  relies on manually engineered shaping-reward functions whose construction is typically time-consuming and error-prone. It also requires domain knowledge which runs contrary to the goal of autonomous learning. In this paper, we introduce an automated RS framework in which the shaping-reward function is constructed in a novel stochastic game between two agents. One agent learns both which states to add shaping rewards and their optimal magnitudes and the other agent  learns the optimal policy for the task using the shaped rewards. We prove theoretically that our framework , which easily adopts existing RL algorithms, learns to construct a shaping-reward function that is tailored to the task and ensures convergence to higher performing policiesfor the given task . We demonstrate the superior performance of our method against state-of-the-art RS algorithms in Cartpole and the challenging console games Gravitar, Solaris and Super Mario .
Reward shaping (RS) is a powerful method in reinforcement learning (RL) for overcoming the problem of sparse or uninformative rewards. However, RS typically relies on manually engineered shaping-reward functions whose construction is time-consuming and error-prone. It also requires domain knowledge which runs contrary to the goal of autonomous learning. We introduce Reinforcement Learning Optimal Shaping Algorithm (ROSA), an automated RS framework in which the shaping-reward function is constructed in a novel Markov game between two agents. A reward-shaping agent (Shaper) uses switching controls to determine which states to add shaping rewards and their optimal values while the other agent (Controller) learns the optimal policy for the task using these shaped rewards. We prove that ROSA , which easily adopts existing RL algorithms, learns to construct a shaping-reward function that is tailored to the task thus ensuring efficient convergence to high performance policies . We demonstrate ROSA's congenial properties in three carefully designed experiments and show its superior performance against state-of-the-art RS algorithms in challenging sparse reward environments .
R
and
or
A
typically
D
typically
R
In this paper, we introduce
We introduce Reinforcement Learning Optimal Shaping Algorithm (ROSA),
R
stochastic
Markov
R
One agent learns both
A reward-shaping agent (Shaper) uses switching controls to determine
R
magnitudes and
values while
A
(Controller)
R
the
these
R
theoretically that our framework
that ROSA
R
and ensures convergence to higher performing policiesfor the given task
thus ensuring efficient convergence to high performance policies
R
the superior performance of our method
ROSA's congenial properties in three carefully designed experiments and show its superior performance
R
Cartpole and the challenging console games Gravitar, Solaris and Super Mario
challenging sparse reward environments