Building on the breakthrough of reinforcement learning, this paper introduces a unified framework of model-free reinforcement learning, CASA-B , Critic AS an Actor with Bandits Vote Algorithm. CASA-B is an actor-critic framework that estimates state-value, state-action-value and policy . An expectation-correct Doubly Robust Trace is introduced to learn state-value and state-action-value, whose convergence properties are guaranteed. We prove that CASA-B integrates a consistent path for the policy evaluation and the policy improvement  . The policy evaluation is equivalent to a compensational policy improvement, which alleviates the function approximation error, and is also equivalent to an entropy-regularized policy improvement, which prevents the policy from collapsing to a suboptimal solution. Building on this design, we find the entropy of the behavior policies' and the target policy's are disentangled. Based on this observation, we propose a progressive closed-form entropy control mechanism, which explicitly controls the behavior policies' entropy to arbitrary range . Our experiments show that CASAB is super sample efficient and achieves State-Of-The-Art on Arcade Learning Environment . Our mean Human Normalized Score is 6456.63\% and our median Human Normalized Score is 477.17\%, under 200M training scale .
This paper introduces a novel design of model-free reinforcement learning, CASA , Critic AS an Actor . CASA follows the actor-critic framework that estimates state-value, state-action-value and policy simultaneously. We prove that CASA integrates a consistent path for the policy evaluation and the policy improvement , which completely eliminates the gradient conflict between the policy improvement and the policy evaluation . The policy evaluation is equivalent to a compensational policy improvement, which alleviates the function approximation error, and is also equivalent to an entropy-regularized policy improvement, which prevents the policy from being trapped into a suboptimal solution. Building on this design, an expectation-correct Doubly Robust Trace is introduced to learn state-value and state-action-value, and the convergence is guaranteed . Our experiments show that the design achieves State-Of-The-Art on Arcade Learning Environment .
R
Building on the breakthrough of reinforcement learning, this
This
R
unified framework
novel design
R
CASA-B
CASA
R
with Bandits Vote Algorithm. CASA-B is an
. CASA follows the
R
. An expectation-correct Doubly Robust Trace is introduced to learn state-value and state-action-value, whose convergence properties are guaranteed.
simultaneously.
R
CASA-B
CASA
A
, which completely eliminates the gradient conflict between the policy improvement and the policy evaluation
R
collapsing to
being trapped into
R
we find the entropy of the behavior policies' and the target policy's are disentangled. Based on this observation, we propose a progressive closed-form entropy control mechanism, which explicitly controls the behavior policies' entropy to arbitrary range
an expectation-correct Doubly Robust Trace is introduced to learn state-value and state-action-value, and the convergence is guaranteed
R
CASAB is super sample efficient and
the design
D
. Our mean Human Normalized Score is 6456.63\% and our median Human Normalized Score is 477.17\%, under 200M training scale