  Many defensive measures in cyber security are still dominated by heuristics,
catalogs of standard procedures, and best practices. Considering the case of
data backup strategies, we aim towards mathematically modeling the underlying
threat models and decision problems. By formulating backup strategies in the
language of stochastic processes, we can translate the challenge of finding
optimal defenses into a reinforcement learning problem. This enables us to
train autonomous agents that learn to optimally support planning of defense
processes. In particular, we tackle the problem of finding an optimal backup
scheme in the following adversarial setting: Given $k$ backup devices, the goal
is to defend against an attacker who can infect data at one time but chooses to
destroy or encrypt it at a later time, potentially also corrupting multiple
backups made in between. In this setting, the usual round-robin scheme, which
always replaces the oldest backup, is no longer optimal with respect to
avoidable exposure. Thus, to find a defense strategy, we model the problem as a
hybrid discrete-continuous action space Markov decision process and
subsequently solve it using deep deterministic policy gradients. We show that
the proposed algorithm can find storage device update schemes which match or
exceed existing schemes with respect to various exposure metrics.
