  Reinforcement learning (RL) has been demonstrated suitable to develop agents
that play complex games with human-level performance. However, it is not
understood how to effectively use RL to perform cybersecurity tasks. To develop
such understanding, it is necessary to develop RL agents using simulation and
emulation systems allowing researchers to model a broad class of realistic
threats and network conditions. Demonstrating that a specific RL algorithm can
be effective for defending a network under certain conditions may not
necessarily give insight about the performance of the algorithm when the
threats, network conditions, and security goals change. This paper introduces a
novel approach for network environment design and a software framework to
address the fundamental problem that network defense cannot be defined as a
single game with a simple set of fixed rules. We show how our approach is
necessary to facilitate the development of RL network defenders that are robust
against attacks aimed at the agent's learning. Our framework enables the
development and simulation of adversaries with sophisticated behavior that
includes poisoning and evasion attacks on RL network defenders.
