  Temporal information is essential to learning effective policies with
Reinforcement Learning (RL). However, current state-of-the-art RL algorithms
either assume that such information is given as part of the state space or,
when learning from pixels, use the simple heuristic of frame-stacking to
implicitly capture temporal information present in the image observations. This
heuristic is in contrast to the current paradigm in video classification
architectures, which utilize explicit encodings of temporal information through
methods such as optical flow and two-stream architectures to achieve
state-of-the-art performance. Inspired by leading video classification
architectures, we introduce the Flow of Latents for Reinforcement Learning
(Flare), a network architecture for RL that explicitly encodes temporal
information through latent vector differences. We show that Flare (i) recovers
optimal performance in state-based RL without explicit access to the state
velocity, solely with positional state information, (ii) achieves
state-of-the-art performance on pixel-based challenging continuous control
tasks within the DeepMind control benchmark suite, namely quadruped walk,
hopper hop, finger turn hard, pendulum swing, and walker run, and is the most
sample efficient model-free pixel-based RL algorithm, outperforming the prior
model-free state-of-the-art by 1.9X and 1.5X on the 500k and 1M step
benchmarks, respectively, and (iv), when augmented over rainbow DQN,
outperforms this state-of-the-art level baseline on 5 of 8 challenging Atari
games at 100M time step benchmark.
