  Hippocampal reverse replay is thought to contribute to learning, and
particularly reinforcement learning, in animals. We present a computational
model of learning in the hippocampus that builds on a previous model of the
hippocampal-striatal network viewed as implementing a three-factor
reinforcement learning rule. To augment this model with hippocampal reverse
replay, a novel policy gradient learning rule is derived that associates place
cell activity with responses in cells representing actions. This new model is
evaluated using a simulated robot spatial navigation task inspired by the
Morris water maze. Results show that reverse replay can accelerate learning
from reinforcement, whilst improving stability and robustness over multiple
trials. As implied by the neurobiological data, our study implies that reverse
replay can make a significant positive contribution to reinforcement learning,
although learning that is less efficient and less stable is possible in its
absence. We conclude that reverse replay may enhance reinforcement learning in
the mammalian hippocampal-striatal system rather than provide its core
mechanism.
