Experience replay is widely used in various deep off-policy reinforcement learning(RL) algorithms. It stores previously collected samples for further reuse. To better utilize these samples, prioritized sampling is a promising technique to improve the performance of RL agents. Previous prioritization methods based on temporal-difference (TD ) errorare highly heuristic and divergent from the objective of RL . In this work, we analyze the optimal prioritization strategy that can minimize the regret of RL policytheoretically. Our theory suggests that the data with higher  TD error, better on-policiness and more corrective feedback should be assigned with higher weights during sampling. Based on this theory, we propose two practical algorithms, RM-DisCor and RM-TCE. RM-DisCor is a general algorithm and RM-TCE is a more efficient variant relying on the temporal ordering of states. Both algorithms improve the performance of off-policy RL algorithms in challenging RL benchmarks, including MuJoCo, Atari and Meta-World.
In reinforcement learning, experience replay stores past samples for further reuse. Prioritized sampling is a promising technique to better utilize these samples. Previous criteria of prioritization include TD error, recentness and corrective feedback, which are mostly heuristically designed . In this work, we start from the regret minimization objective, and obtain an optimal prioritization strategy for Bellman update that can directly maximize the return of the policy. The theory suggests that data with higher hindsight TD error, better on-policiness and more accurate Q value should be assigned with higher weights during sampling. Thus most previous criteria only consider this strategy partially. We not only provide theoretical justifications for previous criteria, but also propose two new methods to compute the prioritization weight, namely ReMERN and ReMERT. ReMERN learns an error network, while ReMERT exploits the temporal ordering of states. Both methods outperform previous prioritized sampling algorithms in challenging RL benchmarks, including MuJoCo, Atari and Meta-World.
R
Experience replay is widely used in various deep off-policy reinforcement learning(RL) algorithms. It stores previously collected
In reinforcement learning, experience replay stores past
R
To better utilize these samples, prioritized
Prioritized
R
improve the performance of RL agents. Previous prioritization methods based on temporal-difference (TD ) errorare highly heuristic and divergent from the objective of RL
better utilize these samples. Previous criteria of prioritization include TD error, recentness and corrective feedback, which are mostly heuristically designed
R
analyze the
start from the regret minimization objective, and obtain an
R
that can minimize the regret of RL policytheoretically. Our
for Bellman update that can directly maximize the return of the policy. The
D
the
A
hindsight
R
corrective feedback
accurate Q value
R
Based on this theory, we propose two practical algorithms, RM-DisCor and RM-TCE. RM-DisCor is a general algorithm and RM-TCE is a more efficient variant relying on
Thus most previous criteria only consider this strategy partially. We not only provide theoretical justifications for previous criteria, but also propose two new methods to compute the prioritization weight, namely ReMERN and ReMERT. ReMERN learns an error network, while ReMERT exploits
R
algorithms improve the performance of off-policy RL
methods outperform previous prioritized sampling