Importance Resampling for Off-policy Policy Evaluation
Authors: Matthew Schlegel, Wesley Chung, Daniel Graves, Martha White
Keywords: Reinforcement Learning, Off-policy policy evaluation, importance resampling, importance sampling
TL;DR: A resampling approach for off-policy policy evaluation in reinforcement learning.
Abstract: Importance sampling is a common approach to off-policy learning in reinforcement learning.  While it is consistent and unbiased, it can result in high variance updates to the parameters for the value function. Weighted importance sampling (WIS) has been explored to reduce variance for off-policy policy evaluation, but only for linear value function approximation. In this work, we explore a resampling strategy to reduce variance, rather than a reweighting strategy. We propose Importance Resampling (IR) for off-policy learning, that resamples experience from the replay buffer and applies a standard on-policy update. The approach avoids using importance sampling ratios directly in the update, instead correcting the distribution over transitions before the update. We characterize the bias and consistency of the our estimator, particularly compared to WIS. We then demonstrate in several toy domains that IR has improved sample efficiency and parameter sensitivity, as compared to several baseline WIS estimators and to IS. We conclude with a demonstration showing IR improves over IS for learning a value function from images in a racing car simulator.