  Learning to flexibly follow task instructions in dynamic environments poses
interesting challenges for reinforcement learning agents. We focus here on the
problem of learning control flow that deviates from a strict step-by-step
execution of instructions -- that is, control flow that may skip forward over
parts of the instructions or return backward to previously completed or skipped
steps. Demand for such flexible control arises in two fundamental ways:
explicitly when control is specified in the instructions themselves (such as
conditional branching and looping) and implicitly when stochastic environment
dynamics require re-completion of instructions whose effects have been
perturbed, or opportunistic skipping of instructions whose effects are already
present. We formulate an attention-based architecture that meets these
challenges by learning, from task reward only, to flexibly attend to and
condition behavior on an internal encoding of the instructions. We test the
architecture's ability to learn both explicit and implicit control in two
illustrative domains -- one inspired by Minecraft and the other by StarCraft --
and show that the architecture exhibits zero-shot generalization to novel
instructions of length greater than those in a training set, at a performance
level unmatched by two baseline recurrent architectures and one ablation
architecture.
