Revised: C AN

Revised: D EEP

Original: C AN D EEP RL SOLVE E RDOS -S ELFRIDGE -S PENCER G AMES ?
Revised: R EINFORCEMENT L EARNING S OLVE E RDOS -S ELFRIDGE -S PENCER G AMES ?

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep reinforcement learning has achieved many recent successes, but our understanding of its strengths and limitations is hampered by the lack of rich environments in which we can fully characterize optimal behavior, and correspondingly diagnose individual actions against such a characterization.
Revised: Deep reinforcement learning has achieved many recent successes, but our understanding of its strengths and limitations is hampered by the lack of rich environments in which we can fully characterize optimal behavior, and correspondingly diagnose individual actions against such a characterization.

Original: Here we consider a family of combinatorial games, arising from work of Erdos, Selfridge, and Spencer, and we propose their use as environments for evaluating and comparing different approaches to reinforcement learning.
Revised: Here we consider a family of combinatorial games, arising from work of Erdos, Selfridge, and Spencer, and we propose their use as environments for evaluating and comparing different approaches to reinforcement learning.

Original: These games have a number of appealing features: they are challenging for current learning approaches, but they form (i) a low-dimensional, simply parametrized environment where (ii) there is a linear closed form solution for optimal behavior from any state, and (iii) the difﬁculty of the game can be tuned by changing environment parameters in an interpretable way.
Revised: These games have a number of appealing features: they are challenging for current learning approaches, but they form (i) a low-dimensional, simply parametrized environment where (ii) there is a linear closed form solution for optimal behavior from any state, and (iii) the difﬁculty of the game can be tuned by changing environment parameters in an interpretable way.

Original: We use these Erdos-Selfridge-Spencer games not only to compare different algorithms, but also to compare approaches based on supervised and reinforcement learning, to analyze the power of multi-agent approaches in improving performance, and to evaluate generalization to environments outside the training set.
Revised: We use these Erdos-Selfridge-Spencer games not only to compare different algorithms, but also to compare approaches based on supervised and reinforcement learning, to analyze the power of multi-agent approaches in improving performance, and to evaluate generalization to environments outside the training set.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep reinforcement learning has seen many remarkable successes over the past few years But developing learning algorithms that are robust across tasks and policy representations remains a challenge.
Revised: Deep reinforcement learning has seen many remarkable successes over the past few years But developing learning algorithms that are robust across tasks and policy representations remains a challenge.

Original: Standard benchmarks like Mujoco and Atari provide rich settings for experimentation, but the speciﬁcs of the underlying environments differ from each other in many different ways, and hence determining the principles underlying any particular form of sub-optimal behavior is difﬁcult.
Revised: Standard benchmarks like Mujoco and Atari provide rich settings for experimentation, but the speciﬁcs of the underlying environments differ from each other in many different ways, and hence determining the principles underlying any particular form of sub-optimal behavior is difﬁcult.

Original: Optimal behavior in these environments is generally complex and not fully characterized, so algorithmic success is generally associated with high scores, making it hard to analyze where errors are occurring in any sort of ﬁne-grained sense.
Revised: Optimal behavior in these environments is generally complex and not fully characterized, so algorithmic success is generally associated with high scores, making it hard to analyze where errors are occurring in any sort of ﬁne-grained sense.

Original: An ideal setting for studying the strengths and limitations of reinforcement learning algorithms would be (i) a simply parametrized family of environments where (ii) optimal behavior can be completely characterized, (iii) the inherent difﬁculty of computing optimal behavior is tightly controlled by the underlying parameters, and (iv) at least some portions of the parameter space produce environments that are hard for current algorithms.
Revised: An ideal setting for studying the strengths and limitations of reinforcement learning algorithms would be (i) a simply parametrized family of environments where (ii) optimal behavior can be completely characterized, (iii) the inherent difﬁculty of computing optimal behavior is tightly controlled by the underlying parameters, and (iv) at least some portions of the parameter space produce environments that are hard for current algorithms.

Original: To produce such a family of environments, we look in a novel direction – to a set of two-player combinatorial games with their roots in work of Erdos and Selfridge (Erdos & L., 1973), and placed on a general footing by Spencer (1994).
Revised: To produce such a family of environments, we look in a novel direction – to a set of two-player combinatorial games with their roots in work of Erdos and Selfridge (Erdos & L., 1973), and placed on a general footing by Spencer (1994).

Original: Roughly speaking, these
Revised: Roughly speaking, these

Original: Erdos-Selfridge-Spencer (ESS) games are games in which two players take turns selecting objects from some combinatorial structure, with the feature that optimal strategies can be deﬁned by potential functions derived from conditional expectations over random future play.
Revised: Erdos-Selfridge-Spencer (ESS) games are games in which two players take turns selecting objects from some combinatorial structure, with the feature that optimal strategies can be deﬁned by potential functions derived from conditional expectations over random future play.

Original: These ESS games thus provide an opportunity to capture the general desiderata noted above, with a clean characterization of optimal behavior and a set of instances that range from easy to very hard as we sweep over a simple set of tunable parameters.
Revised: These ESS games thus provide an opportunity to capture the general desiderata noted above, with a clean characterization of optimal behavior and a set of instances that range from easy to very hard as we sweep over a simple set of tunable parameters.

Original: We focus in particular on one of the best-known games in this genre, Spencer’s attacker-defender game (also known as the “tenure game”; Spencer, 1994), in which — roughly speaking — an attacker advances a set of pieces up the levels of a board, while a defender destroys subsets of these pieces to try prevent any of them from reaching the ﬁnal level (Figure 1).
Revised: We focus in particular on one of the best-known games in this genre, Spencer’s attacker-defender game (also known as the “tenure game”; Spencer, 1994), in which — roughly speaking — an attacker advances a set of pieces up the levels of a board, while a defender destroys subsets of these pieces to try prevent any of them from reaching the ﬁnal level (Figure 1).

Original: An instance of the game can be parametrized by two key quantities.
Revised: An instance of the game can be parametrized by two key quantities.

Original: The ﬁrst is the number of levels K , which determines both the size of the state space and the approximate length of the game; the latter is directly related to the sparsity of win/loss signals as rewards.
Revised: The ﬁrst is the number of levels K , which determines both the size of the state space and the approximate length of the game; the latter is directly related to the sparsity of win/loss signals as rewards.

Original: The second quantity is a potential function φ , whose magnitude characterizes whether the instance favors the defender or attacker, and how much “margin of error” there is in optimal play.
Revised: The second quantity is a potential function φ , whose magnitude characterizes whether the instance favors the defender or attacker, and how much “margin of error” there is in optimal play.

Original: The environment therefore allows us to study learning by the defender, or by the attacker, or in a multi-agent formulation where the defender and attacker are learning concurrently.
Revised: The environment therefore allows us to study learning by the defender, or by the attacker, or in a multi-agent formulation where the defender and attacker are learning concurrently.

Original: Because we have a move-by-move characterization of optimal play, we can go beyond simple measures of reward based purely on win/loss outcomes and use supervised learning techniques to pinpoint the exact location of the errors in a trajectory of play.
Revised: Because we have a move-by-move characterization of optimal play, we can go beyond simple measures of reward based purely on win/loss outcomes and use supervised learning techniques to pinpoint the exact location of the errors in a trajectory of play.

Original: In the process, we are able to develop insights about the robustness of solutions to changes in the environment.
Revised: In the process, we are able to develop insights about the robustness of solutions to changes in the environment.

Original: These types of analyses have been long-standing goals, but they have generally been approached much more abstractly, given the difﬁculty in characterizing step-by-step optimally in non-trivial environments such as this one.
Revised: These types of analyses have been long-standing goals, but they have generally been approached much more abstractly, given the difﬁculty in characterizing step-by-step optimally in non-trivial environments such as this one.

Original: The main contributions of this work are thus the following:
Revised: The main contributions of this work are thus the following:

Original: The development of these combinatorial games as environments for studying the behavior of reinforcement learning algorithms, with sensitive control over the difﬁculty of individual instances using a small set of natural parameters.
Revised: The development of these combinatorial games as environments for studying the behavior of reinforcement learning algorithms, with sensitive control over the difﬁculty of individual instances using a small set of natural parameters.

Original: A comparison of the performance of an agent trained using deep RL to the performance of an agent trained using supervised learning on move-by-move decisions.
Revised: A comparison of the performance of an agent trained using deep RL to the performance of an agent trained using supervised learning on move-by-move decisions.

Original: Exploiting the fact that we can characterize optimal play at the level of individual moves, we ﬁnd an intriguing phenomenon: while the supervised learning agent is, not surprisingly, more accurate on individual move decisions than the deep RL agent, the deep RL agent generates a higher rate of reward than the supervised learning agent.
Revised: Exploiting the fact that we can characterize optimal play at the level of individual moves, we ﬁnd an intriguing phenomenon: while the supervised learning agent is, not surprisingly, more accurate on individual move decisions than the deep RL agent, the deep RL agent generates a higher rate of reward than the supervised learning agent.

Original: An investigation of the way in which the success of one of the two players (defender or attacker) in training turns out to depend crucially on the algorithm being used to implement the other player.
Revised: An investigation of the way in which the success of one of the two players (defender or attacker) in training turns out to depend crucially on the algorithm being used to implement the other player.

Original: We explore properties of this other player’s algorithm, and also properties of mulitagent learning, that lead to more robust policies.
Revised: We explore properties of this other player’s algorithm, and also properties of mulitagent learning, that lead to more robust policies.

Original: This is a largely empirical paper, building on a theoretically grounded environment derived from a combinatorial game.
Revised: This is a largely empirical paper, building on a theoretically grounded environment derived from a combinatorial game.

Original: We present learning and generalization experiments for a variety of commonly used model architectures and learning algorithms.
Revised: We present learning and generalization experiments for a variety of commonly used model architectures and learning algorithms.

Original: We aim to show that despite the simple structure of the game, it provides both signiﬁcant challenges for standard reinforcement learning approaches and a number of tools for precisely understanding those challenges.
Revised: We aim to show that despite the simple structure of the game, it provides both signiﬁcant challenges for standard reinforcement learning approaches and a number of tools for precisely understanding those challenges.

Original:  E RDOS -S ELFRIDGE -S PENCER G AME
Revised:  E RDOS -S ELFRIDGE -S PENCER G AME

Original: We ﬁrst introduce  Erdos-Selfridge-Spencer (ESS) Games, a family of games that has two particularly attractive properties for being a test bed for deep reinforcement learning: the ability to continuously vary the difﬁculty of the environment through two parameters, and the existence of a closed form solution that is expressible as a linear model .
Revised: We ﬁrst introduce the Erdos-Selfridge-Spencer Attacker Defender Games, a family of games that has two particularly attractive properties for being a test bed for deep reinforcement learning: the ability to continuously vary the difﬁculty of the environment through two parameters, and the existence of a closed form solution that is expressible as a linear model .

Original: 2. E RDOS -S ELFRIDGE -S PENCER (ESS) A TTACKER D EFENDER G AME
Revised: 2. E RDOS -S ELFRIDGE -S PENCER (ESS) A TTACKER D EFENDER G AME

Original: An ESS game involves two players: an attacker who moves pieces, and a defender who destroys pieces.
Revised: An Attacker-Defender game involves two players: an attacker who moves pieces, and a defender who destroys pieces.

Original: An instance of the game has K + 1 levels, (from 0 to K ) and N pieces that are initialized across these levels.
Revised: An instance of the game has K + 1 levels, (from 0 to K ) and N pieces that are initialized across these levels.

Original: The attacker’s goal is to get at least one of their pieces to level K , and the defender’s is to destroy all N pieces before this can happen.
Revised: The attacker’s goal is to get at least one of their pieces to level K , and the defender’s is to destroy all N pieces before this can happen.

Original: Each turn, the attacker proposes a partition A, B of the pieces still in play.
Revised: Each turn, the attacker proposes a partition A, B of the pieces still in play.

Original: The defender then chooses one of the sets to destroy and remove from play.
Revised: The defender then chooses one of the sets to destroy and remove from play.

Original: All pieces in the other set are moved up a level.
Revised: All pieces in the other set are moved up a level.

Original: The game ends when either one or more pieces reach level K , or when all pieces are destroyed.
Revised: The game ends when either one or more pieces reach level K , or when all pieces are destroyed.

Original: Figure 1 shows one turn of play.
Revised: Figure 1 shows one turn of play.

Original: With this setup, varying the number of levels K or the number of pieces N changes the difﬁculty for the attacker or the defender.
Revised: With this setup, varying the number of levels K or the number of pieces N changes the difﬁculty for the attacker or the defender.

Original: One of the most striking aspects of the ESS game is that we can make this tradeoff precise, and en route to doing so, also identify a linear optimal policy.
Revised: One of the most striking aspects of the Attacker-Defender game is that we can make this tradeoff precise, and en route to doing so, also identify a linear optimal policy.

Original: We start with a special case, where we can directly think of the game difﬁculty in terms of the number of levels K and the number of pieces N .
Revised: We start with a special case, where we can directly think of the game difﬁculty in terms of the number of levels K and the number of pieces N .

Original: Theorem 1.
Revised: Theorem 1.

Original: Consider an instance of the ESS game with K levels and N pieces, with all N pieces starting at level 0 .
Revised: Consider an instance of the Attacker-Defender game with K levels and N pieces, with all N pieces starting at level 0 .

Original: Then if N < 2 K , the defender can always win.
Revised: Then if N < 2 K , the defender can always win.

Original: The proof (Spencer, 1994) uses Erdos’s probabilistic method and is as follows: for any attacker strategy, assume the defender plays randomly.
Revised: The proof (Spencer, 1994) uses Erdos’s probabilistic method and is as follows: for any attacker strategy, assume the defender plays randomly.

Original: Let T be a random variable for the number of pieces that reach level K .
Revised: Let T be a random variable for the number of pieces that reach level K .

Original: Then T = (cid:80) T i where T i is the indicator that piece i reaches level K .
Revised: Then T = (cid:80) T i where T i is the indicator that piece i reaches level K .

Original: But then E [ T ] = (cid:80) E
Revised: But then E [ T ] = (cid:80) E

Original: [ T i ] =
Revised: [ T i ] =

Original: (cid:80)
Revised: (cid:80)

Original: i 2 − K : as the defender is playing randomly, any piece has probability 1 / 2 of advancing a level and 1 / 2 of being destroyed.
Revised: i 2 − K : as the defender is playing randomly, any piece has probability 1 / 2 of advancing a level and 1 / 2 of being destroyed.

Original: As all the pieces start at level 0 , they must advance K levels to reach the top, which happens with probability 2 − K .
Revised: As all the pieces start at level 0 , they must advance K levels to reach the top, which happens with probability 2 − K .

Original: But now, by choice of
Revised: But now, by choice of

Original: N , we have that (cid:80) i 2 − K = N 2 − K < 1 .
Revised: N , we have that (cid:80) i 2 − K = N 2 − K < 1 .

Original: Since T is an integer random variable, E [ T ] < 1 implies that the distribution of T has nonzero mass at 0 - in other words there is some set of choices for the defender that guarantees destroying all pieces.
Revised: Since T is an integer random variable, E [ T ] < 1 implies that the distribution of T has nonzero mass at 0 - in other words there is some set of choices for the defender that guarantees destroying all pieces.

Original: (cid:3)
Revised: (cid:3)

Original: This proof as given is an ‘existential’ one, but we can turn it into a concrete optimal strategy.
Revised: This proof as given is an ‘existential’ one, but we can turn it into a concrete optimal strategy.

Original: Extending the argument above, we note that a piece at level l has a 2 − ( K − l ) chance of survival.
Revised: Extending the argument above, we note that a piece at level l has a 2 − ( K − l ) chance of survival.

Original: We can thus deﬁne a potential function on states:
Revised: We can thus deﬁne a potential function on states:

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: Potential Function :
Revised: Potential Function :

Original: Given a game state as a K dimensional vector S =
Revised: Given a game state as a K dimensional vector S =

Original: ( n 1 , ..., n K ) , with n i the number of pieces at level i , we deﬁne the potential of the state as φ ( S ) =
Revised: ( n 1 , ..., n K ) , with n i the number of pieces at level i , we deﬁne the potential of the state as φ ( S ) =

Original: (cid:80) Ki =1 n i 2 − ( K − i ) .
Revised: (cid:80) Ki =1 n i 2 − ( K − i ) .

Original: Note that this is a linear function on the input state, expressible as φ ( S )
Revised: Note that this is a linear function on the input state, expressible as φ ( S )

Original: =
Revised: =

Original: w T · S for w a vector with w l = 2 − ( K − l ) .
Revised: w T · S for w a vector with w l = 2 − ( K − l ) .

Original: We can then in fact recast Theorem 1 as follows:
Revised: We can then in fact recast Theorem 1 as follows:

Original: Theorem 2.
Revised: Theorem 2.

Original: Consider an instantiation of the ESS game with K levels and N pieces, with pieces placed anywhere on the board, and let the initial state be S 0 .
Revised: Consider an instantiation of the Attacker-Defender game with K levels and N pieces, with pieces placed anywhere on the board, and let the initial state be S 0 .

Original: Then
Revised: Then

Original: (a) If φ ( S 0 ) < 1 , the defender can always win (b) If φ ( S 0 ) ≥ 1 , the attacker can always win.
Revised: (a) If φ ( S 0 ) < 1 , the defender can always win (b) If φ ( S 0 ) ≥ 1 , the attacker can always win.

Original: One way to prove this is by directly extending the proof of Theorem 1, with E
Revised: One way to prove this is by directly extending the proof of Theorem 1, with E

Original: [ T ] = (cid:80) E
Revised: [ T ] = (cid:80) E

Original: [ T i ] =
Revised: [ T i ] =

Original: (cid:80)
Revised: (cid:80)

Original: i 2 − ( K − i l ) where i l is the level of piece i .
Revised: i 2 − ( K − i l ) where i l is the level of piece i .

Original: After noting
Revised: After noting

Original: that (cid:80) i 2 − ( K − i l ) =
Revised: that (cid:80) i 2 − ( K − i l ) =

Original: φ ( S 0 ) < 1 by our deﬁnition of the potential function and choice of S 0 , we ﬁnish off as in Theorem 1.
Revised: φ ( S 0 ) < 1 by our deﬁnition of the potential function and choice of S 0 , we ﬁnish off as in Theorem 1.

Original: This deﬁnition of the potential function gives a natural, concrete strategy for the defender: the defender simply destroys whichever of A, B has higher potential.
Revised: This deﬁnition of the potential function gives a natural, concrete strategy for the defender: the defender simply destroys whichever of A, B has higher potential.

Original: If φ ( S 0 ) < 1 , then this strategy guarantees that any subsequent state S will also have φ ( S ) < 1 .
Revised: If φ ( S 0 ) < 1 , then this strategy guarantees that any subsequent state S will also have φ ( S ) < 1 .

Original: Assume without loss of generality that φ ( B ) ≤ φ ( A ) .
Revised: Assume without loss of generality that φ ( B ) ≤ φ ( A ) .

Original: The defender will spare set B .
Revised: The defender will spare set B .

Original: Since φ ( B ) ≤ φ ( A ) and φ ( A ) + φ ( B ) =
Revised: Since φ ( B ) ≤ φ ( A ) and φ ( A ) + φ ( B ) =

Original: φ ( S ) < 1 , the next state has potential 2 φ ( B ) (double the potential of B as all pieces move up a level) which is also less than 1 .
Revised: φ ( S ) < 1 , the next state has potential 2 φ ( B ) (double the potential of B as all pieces move up a level) which is also less than 1 .

Original: If φ ( S 0 ) ≥ 1 , we can devise a similar optimal strategy for the attacker.
Revised: If φ ( S 0 ) ≥ 1 , we can devise a similar optimal strategy for the attacker.

Original: The attacker picks two sets
Revised: The attacker picks two sets

Original: A, B such that both has potential ≥ 1 / 2 .
Revised: A, B such that both has potential ≥ 1 / 2 .

Original: (The fact that this can be done is shown in Theorem 3, and in (Spencer, 1994).)
Revised: The fact that this can be done is shown in Theorem 3, and in Spencer (1994).

Original: Then regardless of which of A, B is destroyed, the other, whose pieces all move up a level, doubles its potential, and thus all subsequent states S maintain φ ( S ) ≥ 1 , resulting in an eventual win for the attacker.
Revised: Then regardless of which of A, B is destroyed, the other, whose pieces all move up a level, doubles its potential, and thus all subsequent states S maintain φ ( S ) ≥ 1 , resulting in an eventual win for the attacker.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The Atari benchmark (Mnih et al., 2015) is a well known set of tasks, ranging from easy to solve (Breakout, Pong) to very difﬁcult (Montezuma’s Revenge).
Revised: The Atari benchmark (Mnih et al., 2015) is a well known set of tasks, ranging from easy to solve (Breakout, Pong) to very difﬁcult (Montezuma’s Revenge).

Original: Duan et al.
Revised: Duan et al.

Original: (2016) proposed a set of continuous environments, implemented in the MuJoCo simulator Todorov et al.
Revised: (2016) proposed a set of continuous environments, implemented in the MuJoCo simulator Todorov et al.

Original: An advantage of physics based environments is that they can be varied continuously by changing physics parameters (Rajeswaran et al., 2016), or by randomizing rendering (Tobin et al., 2017).
Revised: An advantage of physics based environments is that they can be varied continuously by changing physics parameters (Rajeswaran et al., 2016), or by randomizing rendering (Tobin et al., 2017).

Original: Deepmind Lab (Beattie et al., 2016) is a set of 3D navigation based environments.
Revised: Deepmind Lab (Beattie et al., 2016) is a set of 3D navigation based environments.

Original: OpenAI Gym (Brockman et al., 2016) contains both the Atari and MuJoCo benchmarks, as well as classic control environments like Cartpole (Stephenson, 1909) and algorithmic tasks like copying an input sequence.
Revised: OpenAI Gym (Brockman et al., 2016) contains both the Atari and MuJoCo benchmarks, as well as classic control environments like Cartpole (Stephenson, 1909) and algorithmic tasks like copying an input sequence.

Original: The difﬁculty of algorithmic tasks can be easily increased by increasing the length of the input.
Revised: The difﬁculty of algorithmic tasks can be easily increased by increasing the length of the input.

Original: Our proposed benchmark merges properties of both the algorithmic tasks and physics-based tasks, letting us increase difﬁculty by discrete changes in length or continuous changes in potential.
Revised: Our proposed benchmark merges properties of both the algorithmic tasks and physics-based tasks, letting us increase difﬁculty by discrete changes in length or continuous changes in potential.

Original:  D EEP R EINFORCEMENT L EARNING ON ESS G AMES
Revised:  D EEP R EINFORCEMENT L EARNING ON THE A TTACKER -D EFENDER G AME

Original: From the previous section, we see that the ESS games area family of environments with a difﬁculty knob that can be continuously adjusted through the start state potential, φ ( S 0 ), and the number of levels K .
Revised: From Section 2, we see that the Attacker-Defender games are family of environments with a difﬁculty knob that can be continuously adjusted through the start state potential φ ( S 0 ) and the number of levels K .

Original: In this section, we describe a set of baseline results on ESS games that motivate the exploration in the remainder of this paper.
Revised: In this section, we describe a set of baseline results on Attacker-Defender games that motivate the exploration in the remainder of this paper.

Original: Remarkably, while the optimal policy can be expressed with a linear model, we ﬁnd that in practice deep networks improve performance signiﬁcantly (Figure 15 in the Appendix).
Revised: Remarkably, while the optimal policy can be expressed with a linear model, we ﬁnd that in practice deep networks improve performance signiﬁcantly (Figure 15 in the Appendix).

Original: We thus focus our attention on neural network models.
Revised: We thus focus our attention on neural network models.

Original: We set up the ESS environment for deep RL as follows: the game state is represented by a K +dimensional vector for levels 0 to K , with coordinate l representing the number of pieces at level l .
Revised: We set up the Attacker-Defender environment for deep RL as follows: the game state is represented by a K + 1 dimensional vector for levels 0 to K , with coordinate l representing the number of pieces at level l .

Original: For the defender agent, the input is the concatenation of the partition A, B , giving a 2( K + 1) dimensional vector.
Revised: For the defender agent, the input is the concatenation of the partition A, B , giving a 2( K + 1) dimensional vector.

Original: The game start state S 0 is initialized randomly from a distribution over start states of a certain potential.
Revised: The game start state S 0 is initialized randomly from a distribution over start states of a certain potential.

Original: The policy is a fully connected neural network with two hidden layers of width 300 .
Revised: The policy is a fully connected neural network with two hidden layers of width 300 .

Original: 4. D EFENDER A GENT P ERFORMANCE WITH V ARYING D IFFICULTY
Revised: 4. D EFENDER A GENT P ERFORMANCE WITH V ARYING D IFFICULTY

Original: We ﬁrst look at training a defender agent against a procedural attacker, that randomly chooses between (mostly) playing optimally, and (occasionally) playing suboptimally, with the Disjoint
Revised: We ﬁrst look at training a defender agent against a procedural attacker, that randomly chooses between (mostly) playing optimally, and (occasionally) playing suboptimally, with the Disjoint

Original: Support Strategy .
Revised: Support Strategy .

Original: This strategy unevenly partitions the occupied levels between A, B so that one set has higher potential than the other, with the proportional difference between the two sets being sampled randomly.
Revised: This strategy unevenly partitions the occupied levels between A, B so that one set has higher potential than the other, with the proportional difference between the two sets being sampled randomly.

Original: Note that this strategy gives rise to very different states A, B (uneven potential, disjoint occupied levels) than the optimal strategy, and we ﬁnd that the model learns a much more generalizable policy when mixing between the two (Section 6).
Revised: Note that this strategy gives rise to very different states A, B (uneven potential, disjoint occupied levels) than the optimal strategy, and we ﬁnd that the model learns a much more generalizable policy when mixing between the two (Section 6).

Original: When testing out RL algorithms, we have two choices of difﬁculty parameters.
Revised: When testing out RL algorithms, we have two choices of difﬁculty parameters.

Original: The potential of the start state φ ( S 0 ) , changes how optimally the defender has to play, with values close to 1 giving much less leeway for mistakes in valuing the two sets.
Revised: The potential of the start state φ ( S 0 ) , changes how optimally the defender has to play, with values close to 1 giving much less leeway for mistakes in valuing the two sets.

Original: Changing K , the number of levels, directly affects the sparsity of the reward, with higher K resulting in longer games and less feedback.
Revised: Changing K , the number of levels, directly affects the sparsity of the reward, with higher K resulting in longer games and less feedback.

Original: Additionally,
Revised: Additionally,

Original: K also greatly increases the number of possible states and game trajectories (Appendix Theorem 4).
Revised: K also greatly increases the number of possible states and game trajectories (Appendix Theorem 4).

Original: We evaluate Proximal Policy Optimization (PPO)
Revised: We evaluate Proximal Policy Optimization (PPO)

Original: (Schulman et al., 2017), Advantage Actor Critic (A2C) (Mnih et al., 2016), and Deep Q Networks (DQN) (Mnih et al., 2015), using the OpenAI
Revised: (Schulman et al., 2017), Advantage Actor Critic (A2C) (Mnih et al., 2016), and Deep Q Networks (DQN) (Mnih et al., 2015), using the OpenAI

Original: Baselines implementations (Hesse et al., 2017).
Revised: Baselines implementations (Hesse et al., 2017).

Original: Algorithms are evaluated on varying start state potential and K .
Revised: Algorithms are evaluated on varying start state potential and K .

Original: Each algorithm is run with 3 random seeds, and we plot the minimum, mean, and maximum performance.
Revised: Each algorithm is run with 3 random seeds, and we plot the minimum, mean, and maximum performance.

Original: Results are shown in Figures 2, 3.
Revised: Results are shown in Figures 2, 3.

Original: Note that all algorithms show variation in performance across different settings of potentials and K , and show noticeable drops in performance with harder difﬁculty settings.
Revised: Note that all algorithms show variation in performance across different settings of potentials and K , and show noticeable drops in performance with harder difﬁculty settings.

Original: When varying potential in Figure 2 both PPO and A2C show larger variance than DQN, though PPO mostly matches or beats DQN in performance.
Revised: When varying potential in Figure 2 both PPO and A2C show larger variance than DQN, though PPO mostly matches or beats DQN in performance.

Original: When varying K , PPO shows less variance than DQN.
Revised: When varying K , PPO shows less variance than DQN.

Original: A2C shows the greatest variance and worst performance out of all three methods.
Revised: A2C shows the greatest variance and worst performance out of all three methods.

Original:  S UPERVISED L EARNING
Revised:  S UPERVISED L EARNING

Original: One remarkable aspect of the ESS game is that not only do we have an easily expressible optimal policy, but we know the ground truth on a per move basis.
Revised: One remarkable aspect of the Attacker-Defender game is that not only do we have an easily expressible optimal policy, but we know the ground truth on a per move basis.

Original: We can thus compare RL to aSupervised
Revised: We can thus compare RL to a

Original:  Learning setup, where we classify the correct action on a large set of sampled states.
Revised: Supervised Learning setup, where we classify the correct action on a large set of sampled states.

Original: Tocarry out
Revised: To

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: a g e R e
Revised: a g e R e

Original: [Figure]
Revised: [Figure]

Original: r d
Revised: r d

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  this test in practice, we ﬁrst train a defender policy network with reinforcement learning, saving all observations seen to a dataset.
Revised: carry out this test in practice, we ﬁrst train a defender policy network with reinforcement learning, saving all observations seen to a dataset.

Original: We then train a supervised network (with the same architecture as the defender policy network) to classify the optimal action.
Revised: We then train a supervised network (with the same architecture as the defender policy network) to classify the optimal action.

Original: This ensures both methods see the same number of unique data points, keeping the comparison fair.
Revised: This ensures both methods see the same number of unique data points, keeping the comparison fair.

Original: We then test the supervised network on how well it can play.
Revised: We then test the supervised network on how well it can play.

Original: The results, shown in Figure 4 are surprising.
Revised: The results, shown in Figure 4 are surprising.

Original: Reinforcement learning is better at playing the game, but does worse at predicting optimal moves.
Revised: Reinforcement learning is better at playing the game, but does worse at predicting optimal moves.

Original: This contrast forms an interesting counterpart to recent ﬁndings of Silver et al.
Revised: This contrast forms an interesting counterpart to recent ﬁndings of Silver et al.

Original: (2017), who in the context of Go also compared reinforcement learning to supervised approaches.
Revised: (2017), who in the context of Go also compared reinforcement learning to supervised approaches.

Original: A key distinction is that their supervised work was relative to a heuristic objective, whereas in our domain we are able to compare to provably optimal moves.
Revised: A key distinction is that their supervised work was relative to a heuristic objective, whereas in our domain we are able to compare to provably optimal moves.

Original: We conjecture that reinforcement learning is learning to focus most on moves that matter for winning.
Revised: We conjecture that reinforcement learning is learning to focus most on moves that matter for winning.

Original: To investigate this phenomenon, we compare the per move accuracy of reinforcement learning and supervised learning based on distance of the move from the end of the game.
Revised: To investigate this phenomenon, we compare the per move accuracy of reinforcement learning and supervised learning based on distance of the move from the end of the game.

Original: We ﬁnd that RL is better at the ﬁnal couple of moves, and then consistently better in most of the earlier parts of the game.
Revised: We ﬁnd that RL is better at the ﬁnal couple of moves, and then consistently better in most of the earlier parts of the game.

Original: Supervised learning beats RL for about one third of the latter
Revised: Supervised learning beats RL for about one third of the latter

Original: [Figure]
Revised: [Figure]

Original: a
Revised: a

Original: [Figure]
Revised: [Figure]

Original: w a
Revised: w a

Original: [Figure]
Revised: [Figure]

Original: d
Revised: d

Original: [Figure]
Revised: [Figure]

Original: Agent
Revised: Agent

Original: [Figure]
Revised: [Figure]

Original: Optimal
Revised: Optimal

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: part of the game, likely where potentials are easier to differentiate, but at the cost of better play earlier in the game.
Revised: part of the game, likely where potentials are easier to differentiate, but at the cost of better play earlier in the game.

Original:  G ENERALIZATION AND M ULTIAGENT L EARNING
Revised:  G ENERALIZATION AND M ULTIAGENT L EARNING

Original: Returning to our RL Defender Agent, we would like to know how robust its learned policy is.
Revised: Returning to our RL Defender Agent, we would like to know how robust its learned policy is.

Original: In particular, as we have so far been training our agent with a randomized but hard coded attacker, we would like to test how sensitive a defender agent is to the particular attacker strategy.
Revised: In particular, as we have so far been training our agent with a randomized but hard coded attacker, we would like to test how sensitive a defender agent is to the particular attacker strategy.

Original: We investigate this in Figure 6 where we ﬁrst train a defender agent on the optimal attacker and test on the disjoint support attacker.
Revised: We investigate this in Figure 6 where we ﬁrst train a defender agent on the optimal attacker and test on the disjoint support attacker.

Original: We notice a large drop in performance when switching from the optimal attacker to the disjoint support attacker.
Revised: We notice a large drop in performance when switching from the optimal attacker to the disjoint support attacker.

Original: As we know there exists an optimal policy which would generalize perfectly across all attacker strategies, this result suggests that the defender is overﬁtting to the particular attacker strategy.
Revised: As we know there exists an optimal policy which would generalize perfectly across all attacker strategies, this result suggests that the defender is overﬁtting to the particular attacker strategy.

Original: 6. T RAINING AN A TTACKER A GENT
Revised: 6. T RAINING AN A TTACKER A GENT

Original: One way to try and mitigate this overﬁtting issue is to set up a method of also training the attacker, with the goal of training the defender against a ﬁxed learned attacker, or even better, in the multiagent setting.
Revised: One way to try and mitigate this overﬁtting issue is to set up a method of also training the attacker, with the goal of training the defender against a ﬁxed learned attacker, or even better, in the multiagent setting.

Original: However, determining the correct setup to train the attacker agent ﬁrst requires devising a tractable parametrization of the action space.
Revised: However, determining the correct setup to train the attacker agent ﬁrst requires devising a tractable parametrization of the action space.

Original: A naive implementation of the attacker would be to have the policy output how many pieces should be allocated to A for each of the K + 1 levels.
Revised: A naive implementation of the attacker would be to have the policy output how many pieces should be allocated to A for each of the K + 1 levels.

Original: (as described in (Spencer, 1994)).
Revised: (as described in (Spencer, 1994)).

Original: This can grow exponentially in K , which is clearly impractical.
Revised: This can grow exponentially in K , which is clearly impractical.

Original: To address this, we ﬁrst prove a theorem that enables us to show that we can parametrize an optimal attacker with a much smaller action space.
Revised: To address this, we ﬁrst prove a theorem that enables us to show that we can parametrize an optimal attacker with a much smaller action space.

Original: Theorem 3.
Revised: Theorem 3.

Original: For any ESS game with K levels, start state S 

Revised: For any Attacker-Defender game with K levels, start state S

Original: and φ ( S 0 ) ≥ 1 , there exists a partition A, B such that φ ( A ) ≥ 0 .
Revised: and φ ( S 0 ) ≥ 1 , there exists a partition A, B such that φ ( A ) ≥ 0 .

Original: 5 , φ ( B ) ≥ 0 .
Revised: 5 , φ ( B ) ≥ 0 .

Original: 5 , and for some l , A contains pieces of level i > l , and B contains all pieces of level i < l .
Revised: 5 , and for some l , A contains pieces of level i > l , and B contains all pieces of level i < l .

Original: Proof.
Revised: Proof.

Original: For each l ∈ { 1 , 2 , . . .
Revised: For each l ∈ { 1 , 2 , . . .

Original: , K + 1 } , let A l be the set of all pieces from levels K to l , with
Revised: , K + 1 } , let A l be the set of all pieces from levels K to l , with

Original: A K +1 = ∅ .
Revised: A K +1 = ∅ .

Original: We have φ ( A i +1 ) ≤ φ ( A i ) , φ ( A K +1 )
Revised: We have φ ( A i +1 ) ≤ φ ( A i ) , φ ( A K +1 )

Original: = 0 , and φ ( A 1 )
Revised: = 0 , and φ ( A 1 )

Original: =
Revised: =

Original: φ ( S 0 ) ≥ 1 .
Revised: φ ( S 0 ) ≥ 1 .

Original: Thus, there exists an l such that φ ( A l +1 ) < 0 .
Revised: Thus, there exists an l such that φ ( A l +1 ) < 0 .

Original: 5 and φ ( A l ) > 0 .
Revised: 5 and φ ( A l ) > 0 .

Original: [Figure]
Revised: [Figure]

Original: Since A l only contains pieces from levels K to l + 1 , potentials φ ( A l +1 ) and φ ( A l ) are both integer multiples of 2 − ( K − l +1) , the value of a piece in level l .
Revised: Since A l only contains pieces from levels K to l + 1 , potentials φ ( A l +1 ) and φ ( A l ) are both integer multiples of 2 − ( K − l +1) , the value of a piece in level l .

Original: Letting φ ( A l +1 ) = n · 2 − ( K − l +1) and φ ( A l )
Revised: Letting φ ( A l +1 ) = n · 2 − ( K − l +1) and φ ( A l )

Original: = m · 2 − ( K − l +1) , we are guaranteed that level l has m − n pieces, and that we can move k < m − n pieces from A l +1 to A l such that the potential of the new set equals 0 . 5 .
Revised: = m · 2 − ( K − l +1) , we are guaranteed that level l has m − n pieces, and that we can move k < m − n pieces from A l +1 to A l such that the potential of the new set equals 0 . 5 .

Original: This theorem gives a different attacker parametrization.
Revised: This theorem gives a different attacker parametrization.

Original: The attacker outputs a level l .
Revised: The attacker outputs a level l .

Original: The environment assigns all pieces before level l to A , all pieces after level l to B , and splits level l among A and
Revised: The environment assigns all pieces before level l to A , all pieces after level l to B , and splits level l among A and

Original: B to keep the potentials of A and B as close as possible.
Revised: B to keep the potentials of A and B as close as possible.

Original: Theorem 3 guarantees the optimal policy is representable, and the action space linear in K instead of exponential in K .
Revised: Theorem 3 guarantees the optimal policy is representable, and the action space linear in K instead of exponential in K .

Original: With this setup, we train an attacker agent against the optimal defender with PPO, A2C, and DQN.
Revised: With this setup, we train an attacker agent against the optimal defender with PPO, A2C, and DQN.

Original: The DQN results were very poor, and so we show results for just PPO and A2C.
Revised: The DQN results were very poor, and so we show results for just PPO and A2C.

Original: In both algorithms we found there was a large variation in performance when changing K , which now affects both reward sparsity and action space size.
Revised: In both algorithms we found there was a large variation in performance when changing K , which now affects both reward sparsity and action space size.

Original: We observe less outright performance variability with changes in potential for small K but see an increase in the variance (Figure 7).
Revised: We observe less outright performance variability with changes in potential for small K but see an increase in the variance (Figure 7).

Original: 6. L EARNING THROUGH M ULTIAGENT P LAY
Revised: 6. L EARNING THROUGH M ULTIAGENT P LAY

Original: With this attacker training, we can now look at learning in a multiagent setting.
Revised: With this attacker training, we can now look at learning in a multiagent setting.

Original: We ﬁrst explore the effects of varying the potential and K as shown in Figure 8.
Revised: We ﬁrst explore the effects of varying the potential and K as shown in Figure 8.

Original: Overall, we ﬁnd that the attacker fares worse in multiagent play than in the single agent setting.
Revised: Overall, we ﬁnd that the attacker fares worse in multiagent play than in the single agent setting.

Original: In particular, note that in the top left pane of
Revised: In particular, note that in the top left pane of

Original: Figure 8, we see that the attacker loses to the defender even with φ ( S 0 )
Revised: Figure 8, we see that the attacker loses to the defender even with φ ( S 0 )

Original: = 1 . 1 for K = 15 .
Revised: = 1 . 1 for K = 15 .

Original: We can compare this to Figure 7 where with PPO, we see that with K = 15 , and potential 1 . 1 , the single agent attacker succeeds in winning against the optimal defender.
Revised: We can compare this to Figure 7 where with PPO, we see that with K = 15 , and potential 1 . 1 , the single agent attacker succeeds in winning against the optimal defender.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: r a g e  w
Revised: r a g e R w

Original: [Figure]
Revised: [Figure]

Original: defender.
Revised: defender.

Original: The rewards are reﬂected around 0 as the ESS game is a zero sum game.
Revised: The rewards are reﬂected around 0 as the Attacker-Defender game is a zero sum game.

Original: The sharp changes in performance correspond to the times we switch which agent is training.
Revised: The sharp changes in performance correspond to the times we switch which agent is training.

Original: We note that the defender performs much better in the multiagent setting: comparing the top and bottom left panes, we see far more variance and lower performance of the attacker compared to the defender performance below.
Revised: We note that the defender performs much better in the multiagent setting: comparing the top and bottom left panes, we see far more variance and lower performance of the attacker compared to the defender performance below.

Original: Furthermore, the attacker loses to the defender for potential 1 . 1 at K = 15 , despite winning against the optimal defender in Figure 7.
Revised: Furthermore, the attacker loses to the defender for potential 1 . 1 at K = 15 , despite winning against the optimal defender in Figure 7.

Original: We also see (right panes) that the attacker has higher variance and sharper changes in its performance even under conditions when it is guaranteed to win.
Revised: We also see (right panes) that the attacker has higher variance and sharper changes in its performance even under conditions when it is guaranteed to win.

Original: [Figure]
Revised: [Figure]

Original: A
Revised: A

Original: [Figure]
Revised: [Figure]

Original: e
Revised: e

Original: [Figure]
Revised: [Figure]

Original: a g e
Revised: a g e

Original: [Figure]
Revised: [Figure]

Original: e w a r d
Revised: e w a r d

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 6. S INGLE A GENT AND M ULTIAGENT G ENERALIZATION A CROSS O PPONENT S TRATEGIES
Revised: 6. S INGLE A GENT AND M ULTIAGENT G ENERALIZATION A CROSS O PPONENT S TRATEGIES

Original: Finally, we return again to our defender agent, and test generalization between the single and multiagent settings.
Revised: Finally, we return again to our defender agent, and test generalization between the single and multiagent settings.

Original: We train a defender agent in the single agent setting against the optimal attacker, and test on a an attacker that only uses the Disjoint Support strategy.
Revised: We train a defender agent in the single agent setting against the optimal attacker, and test on a an attacker that only uses the Disjoint Support strategy.

Original: We also test a defender trained in the multiagent setting (which has never seen any hardcoded strategy of this form) on the Disjoint
Revised: We also test a defender trained in the multiagent setting (which has never seen any hardcoded strategy of this form) on the Disjoint

Original: Support attacker.
Revised: Support attacker.

Original: The results are shown in Figure 9.
Revised: The results are shown in Figure 9.

Original: We ﬁnd that the defender trained as part of a multiagent setting generalizes noticeably better than the single agent defender.
Revised: We ﬁnd that the defender trained as part of a multiagent setting generalizes noticeably better than the single agent defender.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper we have proposed Erdos-Selfridge-Spencer games as rich environments for investigating reinforcement learning, exhibiting continuously tunable difﬁculty and an exact combinatorial characterization of optimal behavior.
Revised: In this paper we have proposed Erdos-Selfridge-Spencer games as rich environments for investigating reinforcement learning, exhibiting continuously tunable difﬁculty and an exact combinatorial characterization of optimal behavior.

Original: We have demonstrated that algorithms can exhibit wide variation in performance as we tune the game’s difﬁculty, and we use the characterization of optimal behavior to expose intriguing contrasts between performance in supervised learning and reinforcement learning approaches.
Revised: We have demonstrated that algorithms can exhibit wide variation in performance as we tune the game’s difﬁculty, and we use the characterization of optimal behavior to expose intriguing contrasts between performance in supervised learning and reinforcement learning approaches.

Original: Having reformulated the results to enable a trainable attacker, we have also been able to explore insights on overﬁtting, generalization, and multiagent learning.
Revised: Having reformulated the results to enable a trainable attacker, we have also been able to explore insights on overﬁtting, generalization, and multiagent learning.

Original: We also develop further results in the Appendix, including an analysis of catastrophic forgetting, generalization across different values of the game’s parameters, and a method for investigating measures of the model’s conﬁdence.
Revised: We also develop further results in the Appendix, including an analysis of catastrophic forgetting, generalization across different values of the game’s parameters, and a method for investigating measures of the model’s conﬁdence.

Original: We believe that this family of combinatorial games can be used as a rich environment for gaining further insights into deep reinforcement learning
Revised: We believe that this family of combinatorial games can be used as a rich environment for gaining further insights into deep reinforcement learning

Original: Appendix O THER G ENERALIZATION P HENOMENA G ENERALIZING O VER D IFFERENT P OTENTIALS AND K
Revised: Appendix O THER G ENERALIZATION P HENOMENA G ENERALIZING O VER D IFFERENT P OTENTIALS AND K

Original: In the main text we examined how our RL defender agent performance varies as we change the difﬁculty settings of the game, either the potential or K .
Revised: In the main text we examined how our RL defender agent performance varies as we change the difﬁculty settings of the game, either the potential or K .

Original: Returning again to the fact that the ESS game has an expressible optimal that generalizes across all difﬁculty settings, we might wonder how training on one difﬁculty setting and testing on a different setting perform.
Revised: Returning again to the fact that the AttackerDefender game has an expressible optimal that generalizes across all difﬁculty settings, we might wonder how training on one difﬁculty setting and testing on a different setting perform.

Original: Testing on different potentials in this way is straightforwards, but testing on different K requires a slight reformulation.
Revised: Testing on different potentials in this way is straightforwards, but testing on different K requires a slight reformulation.

Original: our input size to the defender neural network policy is 2( K + 1) , and so naively changing to a different number of levels will not work.
Revised: our input size to the defender neural network policy is 2( K + 1) , and so naively changing to a different number of levels will not work.

Original: Furthermore, training on a smaller K and testing on larger K is not a fair test – the model cannot be expected to learn how to weight the lower levels.
Revised: Furthermore, training on a smaller K and testing on larger K is not a fair test – the model cannot be expected to learn how to weight the lower levels.

Original: However, testing the converse (training on larger K and testing on smaller K ) is both easily implementable and offers a legitimate test of generalization.
Revised: However, testing the converse (training on larger K and testing on smaller K ) is both easily implementable and offers a legitimate test of generalization.

Original: We ﬁnd (a subset of plots shown in Figure 10) that when varying potential, training on harder games results in better generalization.
Revised: We ﬁnd (a subset of plots shown in Figure 10) that when varying potential, training on harder games results in better generalization.

Original: When testing on a smaller K than the one used in training, performance is inverse to the difference between train K and test K .
Revised: When testing on a smaller K than the one used in training, performance is inverse to the difference between train K and test K .

Original: C ATASTROPHIC F ORGETTING AND C URRICULUM L EARNING
Revised: C ATASTROPHIC F ORGETTING AND C URRICULUM L EARNING

Original: Recently, several papers have identiﬁed the issue of catastrophic forgetting in Deep Reinforcement
Revised: Recently, several papers have identiﬁed the issue of catastrophic forgetting in Deep Reinforcement

Original: Learning, where switching between different tasks results in destructive interference and lower performance instead of positive transfer.
Revised: Learning, where switching between different tasks results in destructive interference and lower performance instead of positive transfer.

Original: We witness effects of this form in the Attacker-Defender games.
Revised: We witness effects of this form in the Attacker-Defender games.

Original: As in Section 7, our two environments differ in the K that we use – we ﬁrst try training on a small K , and then train on larger K .
Revised: As in Section 7, our two environments differ in the K that we use – we ﬁrst try training on a small K , and then train on larger K .

Original: For lower difﬁculty (potential) settings, we see that this curriculum learning improves play, but for higher potential settings, the learning interferes catastrophically, Figure
Revised: For lower difﬁculty (potential) settings, we see that this curriculum learning improves play, but for higher potential settings, the learning interferes catastrophically, Figure

Original: [Figure]
Revised: [Figure]

Original: g
Revised: g

Original: [Figure]
Revised: [Figure]

Original: w a r
Revised: w a r

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: U NDERSTANDING M ODEL F AILURES V ALUE OF THE N ULL S ET
Revised: U NDERSTANDING M ODEL F AILURES V ALUE OF THE N ULL S ET

Original: The signiﬁcant performance drop we see in Figure 6 motivates investigating whether there are simple rules of thumb that the model has successfully learned.
Revised: The signiﬁcant performance drop we see in Figure 6 motivates investigating whether there are simple rules of thumb that the model has successfully learned.

Original: Perhaps the simplest rule of thumb is learning the value of the null set: if one of A, B (say A ) consists of only zeros and the other ( B ) has some pieces, the defender agent should reliably choose to destroy B .
Revised: Perhaps the simplest rule of thumb is learning the value of the null set: if one of A, B (say A ) consists of only zeros and the other ( B ) has some pieces, the defender agent should reliably choose to destroy B .

Original: Surprisingly, even this simple rule of thumb is violated, and even more frequently for larger K , Figure 12.
Revised: Surprisingly, even this simple rule of thumb is violated, and even more frequently for larger K , Figure 12.

Original: M ODEL C ONFIDENCE
Revised: M ODEL C ONFIDENCE

Original: We can also test to see if the model outputs are well calibrated to the potential values: is the model more conﬁdent in cases where there is a large discrepancy between potential values, and ﬁfty-ﬁfty where the potential is evenly split?
Revised: We can also test to see if the model outputs are well calibrated to the potential values: is the model more conﬁdent in cases where there is a large discrepancy between potential values, and ﬁfty-ﬁfty where the potential is evenly split?

Original: The results are shown in Figure 13.
Revised: The results are shown in Figure 13.

Original: 7. G ENERALIZING ACROSS S TART S TATES AND O PPONENT S TRATEGIES
Revised: 7. G ENERALIZING ACROSS S TART S TATES AND O PPONENT S TRATEGIES

Original: In the main paper, we mixed between different start state distributions to ensure a wide variety of states seen.
Revised: In the main paper, we mixed between different start state distributions to ensure a wide variety of states seen.

Original: This begets the natural question of how well we can generalize across start state distribution if we train on purely one distribution.
Revised: This begets the natural question of how well we can generalize across start state distribution if we train on purely one distribution.

Original: The results in Figure 14 show that training naively
Revised: The results in Figure 14 show that training naively

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: A v e r a g e
Revised: A v e r a g e

Original: R e w a r d
Revised: R e w a r d

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: In fact, the amount of possible starting states for a given K and potential φ ( S 0 ) = 1 grows super exponentially in the number of levels K .
Revised: In fact, the amount of possible starting states for a given K and potential φ ( S 0 ) = 1 grows super exponentially in the number of levels K .

Original: We can state the following theorem:
Revised: We can state the following theorem:

Original: Theorem 4.
Revised: Theorem 4.

Original: The number of states with potential 1 for a game with K levels grows like 2 Θ( K 2 ) (where 0 .
Revised: The number of states with potential 1 for a game with K levels grows like 2 Θ( K 2 ) (where 0 .

Original: 25 K 2 ≤ Θ( K 2 ) ≤ 0 .
Revised: 25 K 2 ≤ Θ( K 2 ) ≤ 0 .

Original: 5 K 2 )
Revised: 5 K 2 )

Original: We give a sketch proof.
Revised: We give a sketch proof.

Original: Proof.
Revised: Proof.

Original: Let such a state be denoted S .
Revised: Let such a state be denoted S .

Original: Then a trivial upper bound can be computed by noting that each s i can take a value up to 2 ( K − i ) , and producting all of these together gives roughly 2 K/ 2 .
Revised: Then a trivial upper bound can be computed by noting that each s i can take a value up to 2 ( K − i ) , and producting all of these together gives roughly 2 K/ 2 .

Original: For the lower bound, we assume for convenience that K is a power of 2 (this assumption can be avoided).
Revised: For the lower bound, we assume for convenience that K is a power of 2 (this assumption can be avoided).

Original: Then look at the set of non-negative integer solutions of the system of simultaneous equations
Revised: Then look at the set of non-negative integer solutions of the system of simultaneous equations

Original: [Equation]
Revised: [Equation]

Original: where j ranges over all even numbers between log( K ) + 1 and K .
Revised: where j ranges over all even numbers between log( K ) + 1 and K .

Original: The equations don’t share any variables, so the solution set is just a product set, and the number of solutions is just the product
Revised: The equations don’t share any variables, so the solution set is just a product set, and the number of solutions is just the product

Original: [Figure]
Revised: [Figure]

Original: (cid:81) j (2 j − 1 /K )
Revised: (cid:81) j (2 j − 1 /K )

Original: where, again, j ranges over even numbers between log( K ) + 1 and K .
Revised: where, again, j ranges over even numbers between log( K ) + 1 and K .

Original: This product is roughly 2 K 2 / 4 .
Revised: This product is roughly 2 K 2 / 4 .

Original: V ARYING M ODEL D EPTH
Revised: V ARYING M ODEL D EPTH

Original: As the optimal defender policy is expressible as a linear model, we empirically investigate whether depth is helpful.
Revised: As the optimal defender policy is expressible as a linear model, we empirically investigate whether depth is helpful.

Original: We ﬁnd that even with a temperature included, linear models perform worse than models with one or two hidden layers.
Revised: We ﬁnd that even with a temperature included, linear models perform worse than models with one or two hidden layers.