We propose Learning Off-Policy with Online Planning (LOOP), combining the techniques from model-based and model-free reinforcement learning algorithms. The agent learns a model of the environment, and then uses trajectory optimization with the learned model to select actions. To sidestep the myopic effect of fixed horizon trajectory optimization, a value function  is attached to the end of the planning horizon. This value function is learned through off-policy reinforcement learning, using trajectory optimization as its behavior policy. Furthermore, we introduce "actor-guided" trajectory optimization to mitigate the actor-divergence issue in the proposed method. We benchmark our methods on continuous control tasks and demonstrate that it offers a significant improvement over the underlying model-based and model-free algorithms  .
We propose Learning Off-Policy with Online Planning (LOOP), an efficient reinforcement learning framework, combining the benefits of model-based local trajectory optimization and off-policy algorithms. The agent learns a dynamics model and then uses trajectory optimization with the learned model to select actions. To sidestep the myopic effect of fixed-horizon trajectory optimization, a value function learned through an off-policy algorithm is attached to the end of the planning horizon. We investigate various instantiations of this framework and demonstrate its benefit in three settings: online reinforcement learning, offline reinforcement learning, and safe learning. We show that this method significantly improves the underlying model-based and model-free algorithms and achieves state-of-the-art performance in a variety of settings .
R
combining the techniques from
an efficient reinforcement learning framework, combining the benefits of
R
and model-free reinforcement learning
local trajectory optimization and off-policy
R
model of the environment,
dynamics model
R
fixed horizon
fixed-horizon
A
learned through an off-policy algorithm
R
This value function is learned through off-policy
We investigate various instantiations of this framework and demonstrate its benefit in three settings: online
R
using trajectory optimization as its behavior policy. Furthermore, we introduce "actor-guided" trajectory optimization to mitigate the actor-divergence issue in the proposed method. We benchmark our methods on continuous control tasks and demonstrate that it offers a significant improvement over
offline reinforcement learning, and safe learning. We show that this method significantly improves
A
and achieves state-of-the-art performance in a variety of settings