Constraining Action Sequences with Formal Languages for Deep Reinforcement Learning
Authors: Dong Xu, Eleanor Quint, Zeynep Hakguder, Haluk Dogan, Stephen Scott, Matthew Dwyer
Keywords: reinforcement learning, constraints, finite state machines
TL;DR: We constrain an agent's actions during reinforcement learning, for safety or to enhance exploration.
Abstract: We study the problem of deep reinforcement learning where the agent's action sequences are constrained, e.g., prohibition of dithering or overactuating action sequences that might damage a robot, drone, or other physical device.  Our model focuses on constraints that can be described by automata such as DFAs or PDAs. We then propose multiple approaches to augment the state descriptions of the Markov decision process (MDP) with summaries of recent action histories.  We empirically evaluate these methods applying DQN to three Atari games, training with reward shaping.  We found that our approaches are effective in significantly reducing, and even eliminating, constraint violations while maintaining high reward.  We  also observed that the total reward achieved by an agent can be highly sensitive to how much the constraints encourage or discourage exploration of potentially effective actions during training, and, in addition to helping ensure safe policies, the use of constraints can enhance exploration during training.