  In this work, we study auxiliary prediction tasks defined by
temporal-difference networks (TD networks); these networks are a language for
expressing a rich space of general value function (GVF) prediction targets that
may be learned efficiently with TD. Through analysis in an illustrative domain
we show the benefits to learning state representations of exploiting the full
richness of TD networks, including both action-conditional predictions and
temporally deep predictions. Our main (and perhaps surprising) result is that
deep action-conditional TD networks with random structures that create random
prediction-questions about random features yield state representations that are
competitive with state-of-the-art hand-crafted value prediction and pixel
control auxiliary tasks in both Atari games and DeepMind Lab tasks. We also
show through stop-gradient experiments that learning the state representations
solely via these unsupervised random TD network prediction tasks yield agents
that outperform the end-to-end-trained actor-critic baseline.
