  Off-policy reinforcement learning for control has made great strides in terms
of performance and sample efficiency. We suggest that for many tasks the sample
efficiency of modern methods is now limited by the richness of the data
collected rather than the difficulty of policy fitting. We examine the reasons
that directed exploration methods in the bonus-based exploration (BBE) family
have not been more influential in the sample efficient control problem. Three
issues have limited the applicability of BBE: bias with finite samples, slow
adaptation to decaying bonuses, and lack of optimism on unseen transitions. We
propose modifications to the bonus-based exploration recipe to address each of
these limitations. The resulting algorithm, which we call UFO, produces
policies that are Unbiased with finite samples, Fast-adapting as the
exploration bonus changes, and Optimistic with respect to new transitions. We
include experiments showing that rapid directed exploration is a promising
direction to improve sample efficiency for control.
