Original: A DVERSARIAL P OLICIES B EAT P ROFESSIONAL -
Revised: A DVERSARIAL P OLICIES B EAT P ROFESSIONAL -

Original: L EVEL G O AI S
Revised: L EVEL G O AI S

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We attack the state-of-the-art Go-playing AI system, KataGo, by training an adversarial policy that plays against a frozen KataGo victim.
Revised: We attack the state-of-the-art Go-playing AI system, KataGo, by training an adversarial policy that plays against a frozen KataGo victim.

Original: Our attack achieves a >99% win-rate against KataGo without search and a >80% win-rate when KataGo uses enough search to be near-superhuman.
Revised: Our attack achieves a >99% win-rate against KataGo without search, and a >50% win-rate when KataGo uses enough search to be near-superhuman.

Original: To the best of our knowledge, this is the first successful end-to-end attack against a Go AI playing at the level of a top human professional.
Revised: To the best of our knowledge, this is the ﬁrst successful end-to-end attack against a Go AI playing at the level of a top human professional.

Original: Notably, the adversary does not win by learning to play Go better than KataGo—in fact, the adversary is easily beaten by human amateurs.
Revised: Notably, the adversary does not win by learning to play Go better than KataGo—in fact, the adversary is easily beaten by human amateurs.

Original: Instead, the adversary wins by tricking KataGo into ending the game prematurely at a point that is favorable to the adversary.
Revised: Instead, the adversary wins by tricking KataGo into ending the game prematurely at a point that is favorable to the adversary.

Original: Our results demonstrate that even professional-level AI systems may harbor surprising failure modes.
Revised: Our results demonstrate that even professional-level AI systems may harbor surprising failure modes.

Revised: Example games are available at https://go-attack-iclr.netlify.app/ .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement learning from self-play has achieved superhuman performance in a range of games including Go (Silver et al., 2016), chess and shogi (Silver et al., 2016), and Dota (OpenAI et al., 2019).
Revised: Reinforcement learning from self-play has achieved superhuman performance in a range of games including Go (Silver et al., 2016), chess and shogi (Silver et al., 2016), and Dota (OpenAI et al., 2019).

Original: Moreover, idealized versions of self-play provably converge to Nash equilibria (Brown, 1951; Heinrich et al., 2015).
Revised: Moreover, idealized versions of self-play provably converge to Nash equilibria (Brown, 1951; Heinrich et al., 2015).

Original: Although realistic versions of self-play may not always converge, the strong empirical performance of self-play seems to suggest this is rarely an issue in practice.
Revised: Although realistic versions of self-play may not always converge, the strong empirical performance of self-play seems to suggest this is rarely an issue in practice.

Original: Nonetheless, prior work has found that seemingly highly capable continuous control policies trained via self-play can be exploited by adversarial policies (Gleave et al., 2020; Wu et al., 2021).
Revised: Nonetheless, prior work has found that seemingly highly capable continuous control policies trained via self-play can be exploited by adversarial policies (Gleave et al., 2020; Wu et al., 2021).

Original: This suggests that self-play may not be as robust as previously thought.
Revised: This suggests that self-play may not be as robust as previously thought.

Original: However, although the victim agents are state-of-the-art for continuous control, they are still well below human performance.
Revised: However, although the victim agents are state-of-the-art for continuous control, they are still well below human performance.

Original: This raises the question: are adversarial policies a vulnerability of self-play policies in general , or simply an artifact of insufficiently capable policies?
Revised: This raises the question: are adversarial policies a vulnerability of self-play policies in general , or simply an artifact of insufﬁciently capable policies?

Original: To answer this, we study a domain where self-play has achieved very strong performance: Go.
Revised: To answer this, we study a domain where self-play has achieved very strong performance: Go.

Original: Specifically, we attack KataGo (Wu, 2019), the strongest publicly available Go-playing AI system.
Revised: Speciﬁcally, we attack KataGo (Wu, 2019), the strongest publicly available Go-playing AI system.

Original: We train an adversarial policy end-to-end against a fixed victim policy network.
Revised: We train an adversarial policy end-to-end against a ﬁxed victim policy network.

Original: Using only 0 .
Revised: Using only 0 .

Original: 3% the compute used to train KataGo, we obtain an adversarial policy that wins > 99% of the time against
Revised: 3% the compute used to train KataGo, we obtain an adversarial policy that wins > 99% of the time against

Original: KataGo with no search, and > 80% against KataGo with enough search to be near-superhuman.
Revised: KataGo with no search, and > 50% against KataGo with enough search to be near-superhuman.

Original: Critically, our adversary does not win by learning a generally capable Go policy.
Revised: Critically, our adversary does not win by learning a generally capable Go policy.

Original: Instead, the adversary has learned a simple strategy that stakes out a small corner territory, then places some easily capturable stones in KataGo’s larger complementary territory.
Revised: Instead, the adversary has learned a simple strategy that stakes out a small corner territory, then places some easily capturable stones in KataGo’s larger complementary territory.

Original: This strategy, illustrated and explained in Figure 1, would lose against even amateur Go players, so the victim is in this instance less robust than human amateurs, despite having professional-level capabilities .
Revised: This strategy, illustrated and explained in Figure 1, loses against even amateur Go players (see Section E.2), so the victim is in this instance less robust than human amateurs, despite having professional-level capabilities .

Original: This is a striking example of non-transitivity, illustrated in Figure 2.
Revised: This is a striking example of non-transitivity, illustrated in Figure 2.

Original: Our adversary does not have any special powers: it can only place stones, or pass, like a regular player.
Revised: Our adversary has no special powers: it can only place stones, or pass, like a regular player.

Original: We do, however, give the adversaryblack-box access to a fixed KataGo victim during training.
Revised: We do, however, give the adversary access to a ﬁxed KataGo victim.

Original: In particular, we train the adversary using an AlphaZero-style training process (Silver et al., 2018), similar to that of KataGo.
Revised: In particular, we train the adversary using an AlphaZero-style training process (Silver et al., 2018), similar to that of KataGo.

Original: The key difference is that we collect games with the adversary playing the separate (fixed) victim network.
Revised: The key difference is that we collect games with the adversary playing the separate (ﬁxed) victim network.

Original: Additionally,during the adversary ’s tree search, we use the victim network to select victim moves.
Revised: Additionally, we use the victim network to select victim moves during the adversary’s tree search.

Original: KataGo is the strongest publicly available Go AI system at the time of writing.
Revised: KataGo is the strongest publicly available Go AI system at the time of writing.

Original: With search, KataGo is strongly superhuman, winning (Wu, 2019, section 5.1) against ELF OpenGo
Revised: With search, KataGo is strongly superhuman, winning (Wu, 2019, section 5.1) against ELF OpenGo

Original: (Tian et al., 2019)
Revised: (Tian et al., 2019)

Original: [Figure]
Revised: [Figure]

Original: and Leela Zero (Pascutto, 2019) that are themselves superhuman.
Revised: and Leela Zero (Pascutto, 2019) that are themselves superhuman.

Original: In Section A.5, we estimate that
Revised: In Section D, we estimate that

Original: KataGo without search plays at the level of a top 100 European player, and KataGo with a small amount of search is at the level of the best Go players on earth.
Revised: KataGo without search plays at the level of a top 100 European player, and KataGo with a small amount of search is at the level of the best Go players on earth.

Original: Our paper makes three contributions.
Revised: Our paper makes three contributions.

Original: First, we propose a novel attack method, hybridizing the attack of Gleave et al.
Revised: First, we propose a novel attack method, hybridizing the attack of Gleave et al.

Original: (2020) and AlphaZero-style training (Silver et al., 2018).
Revised: (2020) and AlphaZero-style training (Silver et al., 2018).

Original: Second, we demonstrate the existence of adversarial policies against the state-of-the-art Go AI system, KataGo.
Revised: Second, we demonstrate the existence of adversarial policies against the state-of-the-art Go AI system, KataGo.

Original: Finally, we find the adversary pursues a simple strategy that fools the victim into predicting victory causing it to pass prematurely.
Revised: Finally, we ﬁnd the adversary pursues a simple strategy that fools the victim into predicting victory causing it to pass prematurely.

Original: Our open-source implementation is available at —anonymized—.
Revised: Our open-source implementation is available at —anonymized—.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Our work is inspired by the presence of adversarial examples in a wide variety of models (Szegedy et al., 2014).
Revised: Our work is inspired by the presence of adversarial examples in a wide variety of models (Szegedy et al., 2014).

Original: Notably, many image classifiers reach or surpass human performance (Ho-Phuoc, 2018;
Revised: Notably, many image classiﬁers reach or surpass human performance (Ho-Phuoc, 2018;

Original: Russakovsky et al., 2015; Shankar et al., 2020; Pham et al., 2021).
Revised: Russakovsky et al., 2015; Shankar et al., 2020; Pham et al., 2021).

Original: Yet even these stateof-the-art image classifiers are vulnerable to adversarial examples (Carlini et al., 2019;
Revised: Yet even these stateof-the-art image classiﬁers are vulnerable to adversarial examples (Carlini et al., 2019;

Original: Ren et al., 2020).
Revised: Ren et al., 2020).

Original: This raises the question: could highly capable deep RL policies be similarly vulnerable?
Revised: This raises the question: could highly capable deep RL policies be similarly vulnerable?

Original: One might hope that the adversarial nature of self-play training would naturally lead to robustness.
Revised: One might hope that the adversarial nature of self-play training would naturally lead to robustness.

Original: This strategy works for image classifiers, where adversarial training is an effective if computationally expensive defense (Madry et al., 2018;
Revised: This strategy works for image classiﬁers, where adversarial training is an effective if computationally expensive defense (Madry et al., 2018;

Original: Ren et al., 2020).
Revised: Ren et al., 2020).

Original: This view is further bolstered by the fact that idealized versions of self-play provably converge to a Nash equilibrium, which is unexploitable (Brown, 1951; Heinrich et al., 2015).
Revised: This view is further bolstered by the fact that idealized versions of self-play provably converge to a Nash equilibrium, which is unexploitable (Brown, 1951; Heinrich et al., 2015).

Original: However, our work finds that in practice even state-of-the-art and professional-level deep RL policies are still vulnerable to exploitation.
Revised: However, our work ﬁnds that in practice even state-of-the-art and professional-level deep RL policies are still vulnerable to exploitation.

Original: It is known that self-play may not converge in non-transitive games (Balduzzi et al., 2019).
Revised: It is known that self-play may not converge in non-transitive games (Balduzzi et al., 2019).

Original: However, recent work has argued that real-world games like Go become increasingly transitive at higher levels of capabilities (Czarnecki et al., 2020).
Revised: However, Czarnecki et al. (2020) has argued that real-world games like Go grow increasingly transitive as skill increases.

Original: This would imply that while self-play may struggle with non- transitivity early on during training, comparisons involving highly capable policies such as KataGo should be largely transitive.
Revised: This would imply that while self-play may struggle with non-transitivity early on during training, comparisons involving highly capable policies such as KataGo should be mostly transitive.

Original: By contrast, we find a striking non-transitivity: our adversarial policy can exploit KataGo agents thatcan reliably beat human professionals, yet even an amateur Go player can beat our adversarial policy.
Revised: By contrast, we ﬁnd a striking non-transitivity: our adversary exploits KataGo agents that beat human professionals, yet even an amateur Go player can beat our adversary (Appendix E.2).

Original: Most prior work attacking deep RL has focused on perturbations to observations (Huang et al., 2017; Ilahi et al., 2022). 
Revised: Most prior work attacking deep RL has focused on perturbations to observations (Huang et al., 2017; Ilahi et al., 2022). Concurrent work by Lan et al.

Revised: (2022) shows that KataGo with ≤ 50 visits can be induced to play poorly after adding two adversarially chosen moves to the history.

Revised: Critically, these moves are “meaningless” in the sense of not signiﬁcantly changing the win rate estimated by KataGo with 800 visits.

Revised: These results show that KataGo’s network can seriously mis-evaluate certain states.

Original: However, such attacks are often not physically realizable, so we instead follow the threat model introduced by Gleave et al.
Revised: However, the threat model of Lan et al. unrealistically assumes the adversary can force the opponent to play a speciﬁc move. An attacker with this power has easier ways to win: it could simply make the opponent resign, or play a maximally bad move. We instead follow the threat model introduced by Gleave et al.

Original: (2020) of an adversarial agent acting in a shared environment.
Revised: (2020) of an adversarial agent acting in a shared environment.

Original: Prior work on such adversarial policies has focused on attacking subhuman policies in simulated robotics environments (Gleave et al., 2020; Wu et al., 2021).
Revised: Prior work on such adversarial policies has focused on attacking subhuman policies in simulated robotics environments (Gleave et al., 2020; Wu et al., 2021).

Original: In these environments, the adversary can often win just by causing small changes in the victim’s actions.
Revised: In these environments, the adversary can often win just by causing small changes in the victim’s actions.

Original: By contrast, our work focuses on exploiting professional-level Go policies that have a discrete action space.
Revised: By contrast, our work focuses on exploiting professional-level Go policies that have a discrete action space.

Original: Despite the more challenging setting, we find these policies are not only vulnerable to attack, but also fail in surprising ways that are quite different to human-like mistakes.
Revised: Despite the more challenging setting, we ﬁnd these policies are not only vulnerable to attack, but also fail in surprising ways that are quite different to human-like mistakes.

Original: Adversarial policies give a lower bound on the exploitability of an agent: how much expected utility a best-response policy achieves above the minimax value of the game.
Revised: Adversarial policies give a lower bound on the exploitability of an agent: how much expected utility a best-response policy achieves above the minimax value of the game.

Original: Exactly computing a policy’s exploitability is feasible in some low-dimensional games (Johanson et al., 2011), but not in larger games such as Go which has approximately 10 172 possible states (Allis, 1994, section 6.3.12).
Revised: Exactly computing a policy’s exploitability is feasible in some low-dimensional games (Johanson et al., 2011), but not in larger games such as Go which has approximately 10 172 possible states (Allis, 1994, section 6.3.12).

Original: Prior work has lower bounded the exploitability in some poker variants using search (Lisý & Bowling, 2017), but the method relies on domain-specific heuristics that are not applicable to Go.
Revised: Prior work has lower bounded the exploitability in some poker variants using search (Lisý & Bowling, 2017), but the method relies on domain-speciﬁc heuristics that are not applicable to Go.

Original: The most closely related workto ours is by Timbers et al.
Revised: In concurrent work Timbers et al.

Original: (2020), who develop the approximate best response (ABR) method to estimating exploitability.
Revised: (2022) developed the approximate best response (ABR) method to estimating exploitability.

Original: Both our and their method use the same key idea: an AlphaZero-style training procedure that is adapted to use the opponent ’s policy during search.
Revised: Whereas we exploit the open-source KataGo agent, they exploit a proprietary replica of AlphaZero from Schmid et al. They obtain a 90% win rate against no-search

Original: However, they differ in some details.
Revised: AlphaZero and 65% with 800 visits (Timbers et al., 2022, Figure 3).

Original: For example, we attempt to model the victim’s search process inside our adversary via (A-MCTS-R)
Revised: In section D.3 we estimate that their AlphaZero victim with 800 visits plays at least at the level of a top-200 professional, and may be superhuman.

Original: Additionally, our curriculum uses checkpoints, not just search. Nonetheless, our method does not represent any major conceptual breakthrough: indeed, our results are striking in large part because the attack is so simple.
Revised: That we were both able to exploit unrelated codebases conﬁrms the vulnerability is in AlphaZero-style training as a whole, not merely an implementation bug.

Original: Our main contribution relative to Timbers et al.
Revised: Our attack methodology is similar to Timbers et al.

Original: (2020) lies in our empirical results. The victims that we manage to exploit range in strength from a weak 1p/2p human professional to  the strongest humans on earth (see Section A.5).
Revised: : we both use an AlphaZero-style training procedure that is adapted to use the opponent’s policy during search.

Original: By contrast, Timbers et al.
Revised: However, unlike Timbers et al.

Original: only establish that their victim is at least that of a strong 3d/4d amateur, as they report winning 100% against Pachi at 100, iterations (Baudiš & Gailly, 2012).
Revised: we attempt to model the victim’s search process inside our adversary via A-MCTS-R and A-MCTS-VM (see Section 4).

Original: We can infer Timbers et al. ’s victim must be significantly weaker than AlphaZero, as their fully-trained Chess agent was unable to beat the Chess engine Stockfish beyond level 5, despite being trained for longer than their Go agent. Unfortunately the exact strength is impossible to determine as there is no direct comparison to state-of-the-art Go systems or human players.
Revised: Additionally, our curriculum uses checkpoints as well as search.

Original: An updated version of their paper (Timbers et al., 2022) does not provide any strength evaluation of their victim agent, but it is presumably weaker as it was trained with fewer timesteps than the original version.
Revised: Finally, we provide a detailed empirical investigation into how the attack works, including examples of games played by the adversary, the degree to which the adversary transfers, and an investigation of possible defences.

Original:  B ACKGROUND 3. T HREAT M ODEL
Revised:  B ACKGROUND 3. T HREAT M ODEL

Original: Following Gleave et al.
Revised: Following Gleave et al.

Original: (2020), we consider the setting of a two-player zero-sum Markov game (Shapley, 1953).
Revised: (2020), we consider the setting of a two-player zero-sum Markov game (Shapley, 1953).

Original: Our threat model assumes the attacker plays as one of the agents, which we will call the adversary , and seeks to win against some victim agent.
Revised: Our threat model assumes the attacker plays as one of the agents, which we will call the adversary , and seeks to win against some victim agent.

Original: Critically, the attacker does not have any special powers—they can only take the same actions available to a regular player.
Revised: Critically, the attacker does not have any special powers—they can only take the same actions available to a regular player.

Original: The key capability we grant to the attacker is gray-box access to the victim agent.
Revised: The key capability we grant to the attacker is gray-box access to the victim agent.

Original: That is, the attacker can evaluate theoutput of the victim’s neural network at arbitrary inputs.
Revised: That is, the attacker can evaluate the victim’s neural network on arbitrary inputs.

Original: However, the attacker does not have direct access to the network weights.
Revised: However, the attacker does not have direct access to the network weights.

Original: We furthermore assume the victim agent follows a fixed policy, corresponding to the common case of a pre-trained model deployed with static weights.
Revised: We furthermore assume the victim agent follows a ﬁxed policy, corresponding to the common case of a pre-trained model deployed with static weights.

Original: Gray-box access to a fixed victim naturally arises whenever the attacker can run a copy of the victim agent, such as a commercially available or open-source Go AI system.
Revised: Gray-box access to a ﬁxed victim naturally arises whenever the attacker can run a copy of the victim agent, such as a commercially available or open-source Go AI system.

Original: This is a challenging setting even with gray-box access.
Revised: This is a challenging setting even with gray-box access.

Original: Although finding an exact Nash equilibrium in a game as complex as Go is intractable, a priori it seems plausible that a professional-level Go system might have reached a near-Nash or ϵ -equilibrium .
Revised: Although ﬁnding an exact Nash equilibrium in a game as complex as Go is intractable, a priori it seems plausible that a professional-level Go system might have reached a near-Nash or (cid:15) -equilibrium .

Original: In this case, the victim could only be exploited by an ϵ margin.
Revised: In this case, the victim could only be exploited by an (cid:15) margin.

Original: Moreover, even if there exists a policy that can exploit the victim, it might be computationally expensive to find given that self-play training did not discover the vulnerability.
Revised: Moreover, even if there exists a policy that can exploit the victim, it might be computationally expensive to ﬁnd given that self-play training did not discover the vulnerability.

Original: Consequently, our two primary success metrics are the win rate of the adversarial policy against the victim and the adversary’s training time .
Revised: Consequently, our two primary success metrics are the win rate of the adversarial policy against the victim and the adversary’s training time .

Original: We also track the mean score difference between the adversary and victim, but this is not explicitly optimized for by the attack.
Revised: We also track the mean score difference between the adversary and victim, but this is not explicitly optimized for by the attack.

Original: Crucially, tracking training time rules out the degenerate “attack” of simply training KataGo for longer than the victim.
Revised: Crucially, tracking training time rules out the degenerate “attack” of simply training KataGo for longer than the victim.

Original: In principle, it is possible that a more sample-efficient training regime could produce a stronger agent than KataGo in a fraction of the training time.
Revised: In principle, it is possible that a more sample-efﬁcient training regime could produce a stronger agent than KataGo in a fraction of the training time.

Original: While this might be an important advance in computer
Revised: While this might be an important advance in computer

Original: Go, we would hesitate to classify it as an attack.
Revised: Go, we would hesitate to classify it as an attack.

Original: Rather, we are looking for the adversarial policy to demonstrate non-transitivity , as this suggests the adversary is winning by exploiting a specific weakness in the opponent.
Revised: Rather, we are looking for the adversarial policy to demonstrate non-transitivity , as this suggests the adversary is winning by exploiting a speciﬁc weakness in the opponent.

Original: That is, as depicted in Figure 2, the adversary beats the victim, the victim beats some baseline opponent, and that baseline opponent can in turn beat the adversary.
Revised: That is, as depicted in Figure 2, the adversary beats the victim, the victim beats some baseline opponent, and that baseline opponent can in turn beat the adversary.

Original: 3. K ATA G O
Revised: 3. K ATA G O

Original: We chose to attack KataGo as it is the strongest publicly available Go AI system.
Revised: We chose to attack KataGo as it is the strongest publicly available Go AI system.

Original: KataGo won against ELF OpenGo
Revised: KataGo won against ELF OpenGo

Original: (Tian et al., 2019) and Leela Zero (Pascutto, 2019) after training for only
Revised: (Tian et al., 2019) and Leela Zero (Pascutto, 2019) after training for only

Original: V100 GPU days (Wu, 2019, section 5.1).
Revised: V100 GPU days (Wu, 2019, section 5.1).

Original: ELF OpenGo is itself superhuman, having won allgames played against four top-30 professional players.
Revised: ELF OpenGo is itself superhuman, having won allgames played against four top-30 professional players.

Original: The latest networks of KataGo are even stronger than the original, having been trained for over 10,000 V100-equivalent GPU days.
Revised: The latest networks of KataGo are even stronger than the original, having been trained for over 10,000 V100-equivalent GPU days.

Original: Indeed, even the policy network with no search is competitive with top professionals (see Section A.5).
Revised: Indeed, even the policy network with no search is competitive with top professionals (see Section D.1).

Original: KataGo learns via self-play, using an AlphaZero-style training procedure (Silver et al., 2018).
Revised: KataGo learns via self-play, using an AlphaZero-style training procedure (Silver et al., 2018).

Original: The agent contains a neural network with a policy head , outputting a probability distribution over the next move, and a value head , estimating the win rate from the current state.
Revised: The agent contains a neural network with a policy head , outputting a probability distribution over the next move, and a value head , estimating the win rate from the current state.

Original: It then conducts MonteCarlo Tree Search (MCTS) using these heads to select self-play moves.
Revised: It then conducts MonteCarlo Tree Search (MCTS) using these heads to select self-play moves, described in Appendix B.1.

Original: KataGo trains the policy head to predict the outcome of this tree search, a policy improvement operator, and  the value head to predict whether the agent wins the self-play game.
Revised: KataGo trains the policy head to predict the outcome of this tree search, a policy improvement operator, and trains the value head to predict whether the agent wins the self-play game.

Original: In contrast to AlphaZero, KataGo also has a number of additional heads predicting auxiliary targets such as the opponent’s move on the following turn and which player “owns” a square on the board.
Revised: In contrast to AlphaZero, KataGo also has a number of additional heads predicting auxiliary targets such as the opponent’s move on the following turn and which player “owns” a square on the board.

Original: These heads’ output are not used for actual game play—they serve only to speed up training via the addition of auxiliary losses.
Revised: These heads’ output are not used for actual game play—they serve only to speed up training via the addition of auxiliary losses.

Original: KataGo additionally introduces architectural improvements such as global pooling, and improvements to the training process such as playout cap randomization.
Revised: KataGo additionally introduces architectural improvements such as global pooling, and improvements to the training process such as playout cap randomization.

Original: These modifications to KataGo improve its sample and compute efficiency by several orders of magnitude relative to prior work such as ELF OpenGo.
Revised: These modiﬁcations to KataGo improve its sample and compute efﬁciency by several orders of magnitude relative to prior work such as ELF OpenGo.

Original: For this reason, we choose to build our attack on top of KataGo, although in principle the same attack could be implemented on top of any
Revised: For this reason, we choose to build our attack on top of KataGo, although in principle the same attack could be implemented on top of any

Original: AlphaZero-style training pipeline.
Revised: AlphaZero-style training pipeline.

Original: We describe our extensions to KataGo in the following section.
Revised: We describe our extensions to KataGo in the following section.

Original:  A TTACK M ETHODOLOGY
Revised:  A TTACK M ETHODOLOGY

Original: Prior works, such as KataGo and AlphaZero, train on self-play games where the agent plays many games against itself.
Revised: Prior works, such as KataGo and AlphaZero, train on self-play games where the agent plays many games against itself.

Original: We instead train on games between our adversary and a fixed victim agent, and only train the adversary on data from the turns where it is the adversary’s move, since we wish to train the adversary to exploit the victim, not mimic it.
Revised: We instead train on games between our adversary and a ﬁxed victim agent, and only train the adversary on data from the turns where it is the adversary’s move, since we wish to train the adversary to exploit the victim, not mimic it.

Original: We dub this procedure victim-play .
Revised: We dub this procedure victim-play .

Original: In regular self-play, the agent models its opponent’s moves by sampling from its own policy network.
Revised: In regular self-play, the agent models its opponent’s moves by sampling from its own policy network.

Original: This makes sense in self-play, as the policy is playing itself.
Revised: This makes sense in self-play, as the policy is playing itself.

Original: But in victim-play, it would be a mistake to model the victim as playing from the adversary’s policy network.
Revised: But in victim-play, it would be a mistake to model the victim as playing from the adversary’s policy network.

Original: We introduce three distinct families of Adversarial MCTS (A-MCTS) to address this problem.
Revised: We introduce three distinct families of Adversarial MCTS (A-MCTS) to address this problem.

Original: See Appendix A.4 for the hyperparameter settings we used in experiments.
Revised: See Appendix C for the hyperparameter settings we used in experiments.

Original: Adversarial MCTS: Sample (A-MCTS-S) .
Revised: Adversarial MCTS: Sample (A-MCTS-S) .

Original: In A-MCTS-S, we modify the adversary’s search procedure to sample from the victim’s policy network at victim-nodes in the Monte Carlo tree when it is the victim’s move, and from the adversary’s network at adversary-nodes where it is the adversary’s turn.
Revised: In A-MCTS-S (Appendix B.2), we modify the adversary’s search procedure to sample from the victim’s policy network at victim-nodes in the Monte Carlo tree when it is the victim’s move, and from the adversary’s network at adversary-nodes where it is the adversary’s turn.

Original: Note the search procedure for the victim agent is unmodified, as the victim is frozen and certainly does not have access to our adversary.

Original: We also disable some optimizations added in KataGo, such as adding noise to the policy prior at the root.
Revised: We also disable some optimizations added in KataGo, such as adding noise to the policy prior at the root.

Original: Finally, we introduce a variant A-MCTS-S++ that averages the  policy prediction over board symmetries, to match the default behavior of KataGo.
Revised: Finally, we introduce a variant A-MCTS-S++ that averages the victim policy network’s predictions over board symmetries, to match the default behavior of KataGo.

Original: See

Original: Appendix A.2 for a full description, and Appendix A.1 for a review of the base MCTS algorithm.

Original: Adversarial MCTS: Recursive (A-MCTS-R) .
Revised: Adversarial MCTS: Recursive (A-MCTS-R) .

Original: When the victim we are attacking uses search, sampling from the victim network directly as in A-MCTS will systematically underestimate the strength of  the victim.
Revised: A-MCTS-S systematically underestimates the strength of victims that use search since it models the victim as sampling directly from the policy head.

Original: To resolve this, A-MCTS-R runs MCTS for the victim at each victim node in the A-MCTS-R tree.
Revised: To resolve this, A-MCTS-R runs MCTS for the victim at each victim node in the A-MCTS-R tree.

Original: Unfortunately, this change increases the computational complexity of both adversary training and inference by a factor equal to the victim search budget.
Revised: Unfortunately, this change increases the computational complexity of both adversary training and inference by a factor equal to the victim search budget.

Original: We include A-MCTS-R primarily as an upper bound to establish how much benefit can be gained by resolving this misspecification.
Revised: We include A-MCTS-R primarily as an upper bound to establish how much beneﬁt can be gained by resolving this misspeciﬁcation. Adversarial MCTS: Victim Model (A-MCTS-VM) .

Original: Initialization .
Revised: In A-MCTS-VM, we ﬁne-tune a copy of the victim network to predict the moves played by the victim in games played against the adversarial policy. This is similar to how the victim network itself was trained, but may be a better predictor as it is trained on-distribution. The adversary follows the same search procedure as in A-MCTS-S but samples from this predictive model instead of the victim. This therefore has the same inference complexity as A-MCTS-S, with slightly greater training complexity due to the need to train an additional network.

Revised: However, it does require white-box access to the victim.

Revised: Initialization .

Original: We randomly initialize the adversary’s network.
Revised: We randomly initialize the adversary’s network.

Original: Note that we cannot initialize the adversary’s weights to those of the victim as our threat model does not allow white-box access.
Revised: Note that we cannot initialize the adversary’s weights to those of the victim as our threat model does not allow white-box access.

Original: Additionally, a random initialization encourages exploration to find weaknesses in the victim, rather than simply a stronger Go player.
Revised: Additionally, a random initialization encourages exploration to ﬁnd weaknesses in the victim, rather than simply a stronger Go player.

Original: However, a randomly initialized network will almost always lose against a highly capable network, leading to a challenging initial learning problem.
Revised: However, a randomly initialized network will almost always lose against a highly capable network, leading to a challenging initial learning problem.

Original: We use
Revised: We use

Original: KataGo’s auxiliary targets to partially alleviate this problem: the adversary’s network can learn something useful about the game even from lost matches.
Revised: KataGo’s auxiliary targets to partially alleviate this problem: the adversary’s network can learn something useful about the game even from lost matches.

Original: Curriculum .
Revised: Curriculum .

Original: To help overcome the challenging random initialization, we introduce a curriculum that trains against successively stronger versions of the victim.
Revised: To help overcome the challenging random initialization, we introduce a curriculum that trains against successively stronger versions of the victim.

Original: We switch to a more challenging victim agent once the adversary’s win rate exceeds 50%.
Revised: We switch to a more challenging victim agent once the adversary’s win rate exceeds 50%.

Original: In particular, as KataGo releases the entire training history, we start with an early checkpoint and then move to later victim checkpoints.
Revised: In particular, as KataGo releases the entire training history, we start with an early checkpoint and then move to later victim checkpoints.

Original: Baselines .
Revised: Baselines .

Original: We also test hard-coded baseline adversarial policies written in Python.
Revised: We also test hard-coded baseline adversarial policies. These baselines were inspired by the behavior of our trained adversaries.

Original: The Edge plays random legal moves in the outermost ℓ ∞ -box available on the board.
Revised: The Edge plays random legal moves in the outermost (cid:96) ∞ - box available on the board.

Original: The Spiral attack is similar to the Edge attack, except that it plays moves in a deterministic counterclockwise order, forming a spiral pattern.
Revised: The Spiral attack is similar to the Edge attack, except that it plays moves in a deterministic counterclockwise order, forming a spiral pattern.

Original: Finally, we also implement Mirror Go , a classic novice strategy which plays the opponent’s last move reflected about the y = x diagonal.
Revised: Finally, we also implement Mirror Go , a classic novice strategy which plays the opponent’s last move reﬂected about the y = x diagonal.

Original: If the opponent plays on y = x , Mirror Go plays that move reflected along the y = − xdiagonal.
Revised: If the opponent plays on y = x , Mirror Go plays that move reﬂected along the y = − x

Original: Finally, if the mirrored vertex is taken, Mirror Go plays the closest legal move by ℓ 1 distance.
Revised: [Table]

Original:  E VALUATION
Revised:  E VALUATION

Original: We evaluate our attack method against KataGo (Wu, 2019).
Revised: We evaluate our attack method against KataGo (Wu, 2019).

Original: We begin in Section 5.1 by attacking the strongest policy network Latest directly (without search) using our A-MCTS-S algorithm, achieving a greater than 99% win rate against Latest .
Revised: In Section 5.1 we ﬁnd our A-MCTS-S algorithm achieves a greater than 99% win rate against Latest playing without search. Notably Latest is very strong even without search: we ﬁnd in Section D.1 that it is comparable to a top [Figure]

Original: Notably Latestis very strong even without search: we find in Section A.5.1 that it is comparable to a top 100 European player.
Revised: e r s [Figure] Latest European player.

Original: Our attack manages to transfer to the low-search regime as well, our best result inSection 5.
Revised: Our attack manages to transfer to the low-search regime as well, our best result in

Original: an 80% win rate against Latest with 64 playouts.
Revised: Section 5.2 being a 54% win rate against Latest with 64 playouts.

Original: In Section A.5.2, we estimate that Latest with 64 playouts is comparable to the best human Go players.
Revised: In Section D.2, we estimate that Latest with 64 playouts is comparable to the best human Go players.

Original: However, our adversary performs poorly against Latest with  morethan 64 playouts.
Revised: However, our adversary performs poorly against Latest with 128 or more playouts.

Original: Additionally, we find in Section 5.3 that the adversarial policy is specialized to the victim it is attacking, and has limited transfer to other victims.
Revised: Additionally, we ﬁnd in Section 5.3 that the adversarial policy is specialized to the victim it is attacking, and has limited transfer to other victims.

Original: Finally, in Section 5.4we study how the attack works, including investigating the value prediction of the victim and evaluating a hard-coded defense.
Revised: Finally, in Section 5.4 we study how the attack works, including investigating the value prediction of the victim and evaluating a hard-coded defense.

Original: 5. A TTACKING  V ICTIM P OLICY N ETWORK
Revised: 5. A TTACKING THE V ICTIM P OLICY N ETWORK

Original: We train an adversarial policy using A-MCTS-S and a curriculum, as described in Section 4.
Revised: We train an adversarial policy using A-MCTS-S and a curriculum, as described in Section 4.

Original: We start from a checkpoint Initial around a quarter of the way through training, until reaching the Latest checkpoint corresponding to the strongest KataGo network (see Appendix A.4.1 for details).
Revised: We start from a checkpoint Initial around a quarter of the way through training, until reaching the Latest checkpoint corresponding to the strongest KataGo network (see Appendix C.1 for details).

Original: We evaluate our best adversarial policy timesteps against Latest , achieving a greater than 99% win rate.

Original: Notably, this high win rate is achieved despite our adversarial policy being trained for only 3 . 4 × 10 7 time steps – just 0.3% as many time steps as the victim it is exploiting.

Original: Critically, this adversarial policy does not win by playing a stronger game of Go than the victim.

Original: Instead, it follows a bizarre strategy illustrated in Figure 1 that would lose against even human amateurs.

Original: In Figure 3 we evaluate our adversarial policy against the policy networks of both Initial and
Revised: In Figure 3 we evaluate our adversarial policy against the policy networks of both Initial and

Original: Latest .
Revised: Latest .

Original: We find our adversary attains a large (> 90% ) win rate against both victims throughout much of training.
Revised: We ﬁnd our adversary attains a large (> 90% ) win rate against both victims throughout much of training.

Original: However, over time the adversary overfits to Latest , with the win rate against
Revised: However, over time the adversary overﬁts to Latest , with the win rate against

Original: Initial falling to around 20%.
Revised: Initial falling to around 20%.

Revised: We evaluate our best adversarial policy checkpoint against Latest , achieving a greater than 99% win rate.

Revised: Notably, this high win rate is achieved despite our adversarial policy being trained for only 3 . 4 × 10 7 time steps – just 0.3% as many time steps as the victim it is exploiting.

Revised: Critically, this adversarial policy does not win by playing a stronger game of Go than the victim.

Revised: Instead, it follows a bizarre strategy illustrated in Figure 1 that loses even against human amateurs (see Section E.2).

Original: 5. T RANSFERING TO A V ICTIM W ITH S EARCH
Revised: 5. T RANSFERING TO A V ICTIM W ITH S EARCH

Original: We evaluate the ability of the adversarial policy trained in the previous section to exploit Latest playing with search.
Revised: We evaluate the ability of the adversarial policy trained in the previous section to exploit Latest playing with search.

Original: Although this adversarial policy achieves a win rate greater than 99% against
Revised: Although this adversarial policy achieves a win rate greater than 99% against

Original: Latest without search, in Figure 4a we find the win rate of A-MCTS-S drops to 80% at 32  visits.
Revised: Latest without search, in Figure 4a we ﬁnd the win rate of A-MCTS-S drops to 80% at 32 victim visits.

Revised: However, A-MCTS-S models the victim as having no search at both training and inference

Original: [Figure]
Revised: [Figure]

Original: However, A-MCTS-S models the victim as having no search at both training and inference time.
Revised: time.

Original: We also test A-MCTS-R, which correctly models the victim at inference by performing an MCTS search at each victim node in the adversary’s tree.
Revised: We also test A-MCTS-R, which correctly models the victim at inference by performing an MCTS search at each victim node in the adversary’s tree.

Original: We find that A-MCTS-R performs somewhat better, obtaining a greater than 99% win rate against Latest with 32 visits, but performance drops below 10% at 128 visits.
Revised: We ﬁnd that A-MCTS-R performs somewhat better, obtaining a greater than 99% win rate against Latest with 32 visits, but performance drops below 10% at 128 visits.

Original: Modeling the victim more accurately by performing search in A-MCTS-Rrather than sampling in AMCTS-S is one way to spend our inference-time compute budget. Alternatively, we can increase the amount of search the adversary performs.
Revised: Of course, A-MCTS-R is more computationally expensive than A-MCTS-S. An alternative way to spend our inference-time compute budget is to perform A-MCTS-S with a greater adversary visit count.

Original: In Figure 4b we show that we obtain up to an 80% win rate against Latest with 64 visits when the adversary has 1 , 024 visits.
Revised: In Figure 4b we show that we obtain up to a 54% win rate against Latest with 64 visits when the adversary has 4 , 096 visits.

Original: Note this uses fewer total visits than A-MCTS-R against a victim of 32 visits with 200 adversary visits ( 100 × 32 + 100 = 3300 ). So, at least for modest victim visit count, a deeper adversary search seems more effective than more accurately modeling the victim.
Revised: This is very similar to the performance of A-MCTS-R withvisits, which has a 49% win rate against the same victim. Interestingly, the inference cost of these attacks is also similar, with 4 , 096 neural network forward passes (NNFPs) per move for A-MCTS-S (one per visit) versus 6 , 500 NNFPs / move for A-MCTS-R.

Original: Note these experiments only attempt to transfer our adversarial policy.
Revised: Note these experiments only attempt to transfer our adversarial policy.

Original: It would also be possible to repeat the attack from scratch against a victim with search, producing a fresh adversarial policy.
Revised: It would also be possible to repeat the attack from scratch against a victim with search, producing a new adversarial policy.

Original: We leave thisas a direction for future work, asdue to computational constraints we are currentlyunable to run this attack.
Revised: We leave this for future work as we cannot currently run this attack due to computational constraints.

Original: 5. T RANSFERING TO O THER C HECKPOINTS
Revised: 5. T RANSFERING TO O THER C HECKPOINTS

Original: From Figure 8 in the appendix, we see that an adversary trained against Latest does better against
Revised: From Figure 9 in the appendix, we see that an adversary trained against Latest does better against

Original: Latest than Initial , despite Latest being a stronger agent.
Revised: Latest than Initial , despite Latest being a stronger agent.

Original: The converse also holds: an agent trained against Initial does better against Initial than Latest .
Revised: The converse also holds: an agent trained against Initial does better against Initial than Latest .

Original: This pattern holds for most visit counts where the adversary wins consistently, although in the case of the adversary for
Revised: This pattern holds for most visit counts where the adversary wins consistently, although in the case of the adversary for

Original: Latest the gap is fairly narrow ( 99% vs 80% win rate) at low visit counts.
Revised: Latest the gap is fairly narrow ( 99% vs. 80% win rate) at low visit counts.

Original: These results suggest that different checkpoints have unique vulnerabilities.
Revised: These results suggest that different checkpoints have unique vulnerabilities.

Original: 5. U NDERSTANDING THE A TTACK
Revised: 5. U NDERSTANDING THE A TTACK

Original: We observed in Figure 1 that the adversary appears to win by tricking the victim into passing prematurely, at a time favorable to the adversary.
Revised: We observed in Figure 1 that the adversary appears to win by tricking the victim into passing prematurely, at a time favorable to the adversary.

Original: In this section, we seek to answer three key questions.
Revised: In this section, we seek to answer three key questions.

Original: First, why does the adversary pass even when it leads to a guaranteed loss?
Revised: First, why does the adversary pass even when it leads to a guaranteed loss?

Original: Second, is passing causally responsible for the victim losing, or would it lose anyway for a different reason?
Revised: Second, is passing causally responsible for the victim losing, or would it lose anyway for a different reason?

Original: Third, is the adversary performing a simple strategy, or does it contain some hidden complexity?
Revised: Third, is the adversary performing a simple strategy, or does it contain some hidden complexity?

Original: Evaluating the Latest victim without search against the adversary from section 5.1 over n =games, we find that Latest passes (and loses) in 247 games and does not pass (and wins) in the 
Revised: Evaluating the Latest victim without search against the adversary from section 5.1 over n =games, we ﬁnd that Latest passes (and loses) in 247 games and does not pass (and wins) in the [Figure] Edge attack Mirror attack

Revised: Spiral

Original: [Figure]
Revised: [Figure]

Original: remaining 3 games.
Revised: remaining 3 games.

Original: In all cases, Latest ’s value head estimates a win probability  greater than 99.5%, although its true win percentage is only 1.2%.
Revised: In all cases, Latest ’s value head estimates a win probability of greater than 99.5% after the ﬁnal move it makes, although its true win percentage is only 1.2%.

Original: Latest predicts it will win by µ = 134 .points ( σ = 27 . 9 ), and passing would be reasonable if it weretruly so far ahead.
Revised: Latest predicts it will win by µ = 134 . 5 points ( σ = 27 . 9 ) after its ﬁnal move, and passing would be reasonable if it were so far ahead.

Original: But in reality it is just one move away from losing by an average of 86 .
Revised: But in fact it is just one move away from losing by an average of 86 .

Original: 26 points.
Revised: 26 points.

Original: We conjecture the victim’s prediction is so mistaken as the games induced by playing against the adversarial policy are very different to those seen during the victim’s self-play training.
Revised: We conjecture the victim’s prediction is so mistaken as the games induced by playing against the adversarial policy are very different to those seen during the victim’s self-play training.

Original: Certainly, there is no fundamental inability for neural networks to predict the outcome correctly.
Revised: Certainly, there is no fundamental inability for neural networks to predict the outcome correctly.

Original: The adversary’s value head achieves a mean-squared error of only 1 . 22 (compared to 49 , 742 for the victim), and predicts it will win 98 .
Revised: The adversary’s value head achieves a mean-squared error of only 3 . 18 (compared to 49 , 742 for the victim) on the adversary’s penultimate move. The adversary predicts it will win 98 .

Original: 7% of the time—extremely close to the true 98 . 8% win rate in this sample.
Revised: 6% of the time—extremely close to the true 98 . 8% win rate in this sample.

Original: To verify whether this pathological passing behavior is the reason the adversarial policy wins, we design a hard-coded defense for the victim: only passing when it cannot change the outcome of the game.
Revised: To verify whether this pathological passing behavior is the reason the adversarial policy wins, we design a hard-coded defense for the victim: only passing when it cannot change the outcome of the game.

Original: Concretely, we only allow the victim to pass when its only legal moves are in its own passalive territory , a concept described in the official KataGo rules which extends the traditional Go notion of a pass-alive group (Wu, 2021) (see Appendix A.3 for a full description of the algorithm).
Revised: Concretely, we only allow the victim to pass when its only legal moves are in its own passalive territory , a concept described in the ofﬁcial KataGo rules which extends the traditional Go notion of a pass-alive group (Wu, 2021) (see Appendix B.3 for a full description of the algorithm).

Original: We apply this defense to the Latest policy network.
Revised: We apply this defense to the Latest policy network.

Original: Whereas the adversarial policy in Section 5. won greater than 99% of games against vanilla Latest , we find that it loses all 1600 evaluation games against Latest with this defense.
Revised: Whereas the adversarial policy in Section 5. won greater than 99% of games against vanilla Latest , we ﬁnd that it loses all 1600 evaluation games against Latest with this defense.

Original: This confirms the adversarial policy wins via passing.
Revised: This conﬁrms the adversarial policy wins via passing.

Original: Unfortunately, this “defense” has several undesirable properties.
Revised: Unfortunately, this “defense” has two undesirable properties.

Original: First, it causes KataGo to continue to play even when a game is clearly won or lost, which is frustrating for human opponents.
Revised: First, it causes KataGo to continue to play even when a game is clearly won or lost, which is frustrating for human opponents.

Original: In fact, the average game length when playing with pass-hardening against an adversarial policy increases from 95 to 421 moves—over a factor of four!
Revised: In fact, the average game length when playing with pass-hardening against an adversarial policy increases from 95 to 421 moves—over a factor of four!

Original: This is also almost twice that of the 211 moves typical for Go games between professional players (Brouwer, 2014).
Revised: This is also almost twice that of the 211 moves typical for Go games between professional players (Brouwer, 2014).

Original: Second, the defense relies on hard-coded knowledge about Go, using a search algorithm to compute the pass-alive territories.
Revised: Second, the defense relies on hard-coded knowledge about Go, using a search algorithm to compute the pass-alive territories.

Original: Ideally, we would be able to apply AI techniques to systems where humans do not have such domain expertise: indeed, this was the key contribution of AlphaZero (Silver et al., 2018) over AlphaGo (Silver et al., 2016).
Revised: Ideally, we would be able to apply AI techniques to systems where humans do not have such domain expertise: indeed, this was the key contribution of AlphaZero (Silver et al., 2018) over AlphaGo (Silver et al., 2016).

Original: Moreover, there may be games where AI systems are vulnerable but where there is no simple algorithm to address the vulnerabilities.
Revised: Moreover, there may be games where AI systems are vulnerable but where there is no simple algorithm to address the vulnerabilities.

Original: Finally, we seek to determine if the adversarial policy is winning by pursuing a simple high-level strategy against an easily exploitable victim, or via a more subtle exploit such as forming an adversarial example by the pattern of stones it plays.
Revised: Finally, we seek to determine if the adversarial policy is winning by pursuing a simple high-level strategy, or via a more subtle exploit such as forming an adversarial example by the pattern of stones it plays.

Original: We start by evaluating the hard-coded baseline adversarial policies described in Section 4 that were in part inspired by the adversarial policy’s game play.
Revised: We start by evaluating the hard-coded baseline adversarial policies described in Section 4.

Original: In Figure 5, we find that the Edge attack achieves a 60% win rate playing as white against Latest with 1 visit, but quickly drops to around 10% at 32 visits. The Spiral and Mirror Go attacks fare worse than Edge at all visit counts.
Revised: In Figure 5, we see that all of our baseline attacks perform substantially worse than our trained adversarial policy (Figure 4a).

Original: Note that the baseline attacks only win by komi, and so never winwhen playing as black.
Revised: Moreover, all our baseline attacks only win by komi, and so never win as black.

Original: By contrast, our adversarial policy in Section 5.1 wins playing as either color, and often by a large margin (in excess of 50 points).
Revised: By contrast, our adversarial policy in Section 5.1 wins playing as either color, and often by a large margin (in excess of 50 points).

Original: We also attempted to manually mimic the adversary’s game play.
Revised: We also attempted to manually mimic the adversary’s game play with limited success.

Original: One of the authors, who has little Go experience, won a game against the 8-dan KGS ranked bot NeuralZ06 by pursuing a similar strategy to the adversary (see Figure 7 and Appendix A.6.1).

Original: However, this bot had the friendlyPassOk flag set to true, making the bot more “polite” but also easier to exploit.

Original: We were unable to manually win in our standard evaluation setting with friendlyPassOk set to false.

Original: Consequently, although the basics of our adversarial policy seem readily mimicable, matching its performance is challenging, suggesting it may be performing a more subtle exploit.
Revised: Although the basics of our adversarial policy seem readily mimicable, matching its performance is challenging, suggesting it may be performing a more subtle exploit.

Original:  L IMITATIONS AND F UTURE W ORK
Revised:  L IMITATIONS AND F UTURE W ORK

Original: This paper has demonstrated that even agents at the level of top human professionals can be vulnerable to adversarial policies.
Revised: This paper has demonstrated that even agents at the level of top human professionals can be vulnerable to adversarial policies.

Original: However, our results do not establish how common such vulnerabilities are. It is possible that KataGo is unusually vulnerable, although this seems unlikely as it is the strongest publicly-available Go AI system and uses a similar training process to other systems. Alternatively, perhaps it is easier to exploit AI systems in Go than in other games. Notably, Go games typically end by both players passing (or one player resigning), and our attack succeeds by tricking KataGo into ending the game prematurely.
Revised: However, our results do not establish how common such vulnerabilities are: it is possible Go-playing AI systems are unusually vulnerable.

Original: This failure mode would not occur in a game like chess with a clear termination condition, checkmate.

Original: A promising direction for future work is to evaluate our attack against strong AI systems in other games.
Revised: A promising direction for future work is to evaluate our attack against strong AI systems in other games.

Original: Now that we know even professional-level policies can be vulnerable to attack, it is natural to ask how we can defend against this possibility.
Revised: If professional-level policies can be vulnerable to attack, it is natural to ask how we can defend against this possibility.

Original: A simple approach is adversarial training: fine-tuning the victim agent against the trained adversary.
Revised: Fortunately, there are a number of promising multi-agent RL techniques.

Original: This approach was attempted in Gleave et al. but with  limited success—repeating the attack against the fine-tuned victim found a new adversarial policy that wins against the fine-tuned and original victim.
Revised: An important direction for future work is to evaluate policies trained with these approaches to determine if they are also exploitable, or if this is an issue limited to self-play. One promising technique is counterfactual regret minimization (Zinkevich et al., 2007, CFR) that can beat professional human poker players (Brown & Sandholm, 2018).

Original: However, it is possible adversarial training will work better for policies that are already professional-level, and so have fewer distinct failure modes. Alternatively, Czempin & Gleave (2022) find population-based training (PBT) leads to increased robustness against adversarial policies, albeit with  a substantial increase in training time.
Revised: CFR has difﬁculty scaling to high-dimensional state spaces, but regularization methods (Perolat et al., 2021) can natively scale to games such as Stratego with a game tree 10 175 times larger than Go (Perolat et al., 2022). Alternatively, methods using populations of agents such as policy-space response oracles (Lanctot et al., 2017), AlphaStar’s Nash league (Vinyals et al., 2019) or vanilla PBT (Czempin & Gleave, 2022) may be more robust than self-play, although are substantially more computationally expensive. Finally, we found it much harder to exploit agents that use search, with our attacks achieving a lower win rate and requiring more computational resources.

Original: Finally, another fruitful direction for future work is to develop effective and compute-efficient methods for attacking agents that use search. 
Revised: An interesting direction for future work is to see if there exist more effective and compute-efﬁcient methods for attacking agents that use search. If such methods do not exist, then search may be a viable defense against adversaries.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have developed the first adversarial policy exploiting an AI agent that performs at the level of a top human professional.
Revised: We have developed the ﬁrst adversarial policy exploiting an AI agent that performs at the level of a top human professional.

Original: Our attack method achieves a 99% win rate in less than 0.3% as much training time as the Latest victim network.

Original: Notably, the adversarial policy does not win by playing a strong game of Go—in fact, the adversarial policy can be easily beaten by a human amateur.
Revised: Notably, the adversarial policy does not win by playing a strong game of Go—in fact, the adversarial policy can be easily beaten by a human amateur.

Original: Instead, the adversary wins by exploiting a particular blindspot in the victim agent.
Revised: Instead, the adversary wins by exploiting a particular blindspot in the victim agent.

Original: This result suggests that even highly capable agents can harbor serious vulnerabilities.
Revised: This result suggests that even highly capable agents can harbor serious vulnerabilities.

Original: Moreover, these vulnerabilities can remain undiscovered for many years if not explicitly probed.

Original: AlphaGo was published in 2016, with replications quick to follow in the form of Leela (Pascutto, 2017), ELF OpenGo (Tian et al., 2019) and KataGo (Wu, 2019).
Revised: The original KataGo paper (Wu, 2019) was published in 2019 and KataGo has since been used by many Go enthusiasts and professional players as a playing partner and analysis engine.

Original: Both Go enthusiasts and professional players regularly pit themselves against these AI bots, yet have not been able to reliably win.
Revised: However, despite the large amount of attention placed on KataGo, the vulnerability in this paper was to our knowledge unknown.

Original: The inability of human players to win against these AI systems is especially surprising as our adversarial policy follows a qualitatively simple strategy that human players can mimic.
Revised: This suggests that learning-based attacks like the one developed in this paper may be an important tool for uncovering hard-to-spot vulnerabilities in AI systems.

Original: Our results underscore that improvements in capabilities do not always translate into adequate robustness.
Revised: Our results underscore that improvements in capabilities do not always translate into adequate robustness.

Original: These failures in Go AI systems are entertaining, but a similar failure in safety-critical systems such as automated financial trading or autonomous vehicles could have dire consequences.
Revised: These failures in Go AI systems are entertaining, but a similar failure in safety-critical systems such as automated ﬁnancial trading or autonomous vehicles could have dire consequences.

Original: Consequently, we believe the ML research community will need to invest considerable effort into improving robust training and adversarial defense techniques in order to produce models with the high levels of reliability needed for safety-critical systems.
Revised: We believe the ML research community should invest considerable effort into improving robust training and adversarial defense techniques in order to produce models with the high levels of reliability needed for safety-critical systems.

Original: A UTHOR C ONTRIBUTIONS
Revised: A UTHOR C ONTRIBUTIONS

Original: Removed for double blind submission.
Revised: Removed for double blind submission.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: Removed for double blind submission.
Revised: Removed for double blind submission.

Original: AA PPENDIX A. A R EVIEW OF M ONTE -C ARLO T REE S EARCH (MCTS)
Revised: A R ULES OF G O U SED F OR E VALUATION

Revised: We evaluate all games with Tromp-Taylor rules (Tromp, 2014), after clearing opposite-color stones within pass-alive groups computed by Benson’s algorithm (Benson, 1976). KataGo was conﬁgured to play using these rules in all our matches against it.

Revised: Indeed, these rules simply consist of KataGo’s version of Tromp-Taylor rules with SelfPlayOpts enabled (Wu, 2021).

Revised: We use a ﬁxed Komi of 6.5.

Revised: We chose these modiﬁed Tromp-Taylor rules because they are simple, and KataGo was trained on (variants) of these rules so should be strongest playing with them.

Revised: Although the exact rules used were randomized during KataGo’s training, modiﬁed Tromp-Taylor made up a plurality of the training data.

Revised: That is, modiﬁed Tromp-Taylor is at least as likely as any other conﬁguration seen during training, and is more common than some other options.

Revised: In particular, KataGo training randomized between area vs. territory scoring as well as ko, suicide, taxation and button rules from the options described in Wu (2021).

Revised: These conﬁguration settings are provided as input to the neural network (Wu, 2019, Table 4), so the network should learn to play appropriately under a range of rule sets.

Revised: Additionally, during training Komi was sampled randomly from a normal distribution with mean 7 and standard deviation 1 (Wu, 2019, Appendix D).

Revised: A. D IFFERENCE F ROM T YPICAL H UMAN P LAY

Revised: Although KataGo supports a variety of rules, all of them involve automatically scoring the board at the end of the game.

Revised: By contrast, when a match between humans end, the players typically confer and agree which stones are dead, removing them from the board prior to scoring.

Revised: If no agreement can be reached then the players either continue playing the game until the situation is clariﬁed, or a referee arbitrates the outcome of the game.

Revised: KataGo has a variety of optional features to help it play well under human scoring rules.

Revised: For example, KataGo includes an auxiliary prediction head for whether stones are dead or alive.

Revised: This enables it to propose which stones it believes are dead when playing on online Go servers.

Revised: Additionally, it include hard-coded features that can be enabled to make it play in a more human-like way, such as friendlyPassOk to promote passing when heuristics suggest the game is nearly over.

Revised: These features have led some to speculate that the victim passes prematurely in games such as those in Figure 1 because it has learned or is conﬁgured to play in a more human-like way.

Revised: Prima facie , this view seems credible: a human player certainly might pass in a similar situation to our victim, viewing the game as already won under human rules.

Revised: Although tempting, this explanation is not correct: the optional features described above were disabled in our evaluation.

Revised: Therefore KataGo loses under the rules it was both trained and conﬁgured to use.

Revised: In fact, the majority of our evaluation used the match command to run KataGo vs. KataGo agents which naturally does not support these human-like game play features.

Original:  In this section, we review the basic Monte-Carlo Tree Search (MCTS) algorithm.
Revised: We did use the gtp command, implementing the Go Text Protocol (GTP), for a minority of our experiments, such as evaluating KataGo against other AI systems or human players. In those experiments, we conﬁgured gtp to follow the same Tromp-Taylor rules described above, with any human-like extensions disabled. B S EARCH A LGORITHMS B. A R EVIEW OF M ONTE -C ARLO T REE S EARCH (MCTS) In this section, we review the basic Monte-Carlo Tree Search (MCTS) algorithm as used in AlphaGostyle agents (Silver et al., 2016).

Original: This formulation is heavily inspired by the description of MCTS given in (Wu, 2019).
Revised: This formulation is heavily inspired by the description of MCTS given in Wu (2019).

Original: MCTS is an algorithm for growing a game tree one node at a time.
Revised: MCTS is an algorithm for growing a game tree one node at a time.

Original: It starts from a tree T 1 with a single root node s 1 .
Revised: It starts from a tree T 0 with a single root node x 0 .

Original: It then goes through R playouts , where every playout adds a leaf node to the tree.
Revised: It then goes through N playouts , where every playout adds a leaf node to the tree.

Original: We will use T i to denote thestate of the game tree after i playouts, and will use s i to denote the node that was added to T i − 1 to get T i .
Revised: We will use T i to denote the game tree after i playouts, and will use x i to denote the node that was added to T i − 1 to get T i .

Original: After MCTS finishes, we have a tree T R with R nodes.
Revised: After MCTS ﬁnishes, we have a tree T N with N + 1 nodes.

Original: We then use simple statistics of T R to derive a sampling distribution for the next move.
Revised: We then use simple statistics of T N to derive a sampling distribution for the next move.

Original: A.1. MCTS PLAYOUTS
Revised: B.1. MCTS P LAYOUTS

Original: The playouts are governed by two auxiliary functions:
Revised: MCTS playouts are governed by two learned functions:

Original: a. A value function estimator ˆ V : T × S
Revised: a. A value function estimator ˆ V : T × X

Original: → R , which returns a real number ˆ V T ( s ) given a tree T and a state s .
Revised: → R , which returns a real number ˆ V T ( x ) given a tree T and a node x in T .

Original: The value function estimator is meant to estimate how good a state is for the root player.
Revised: The value function estimator is meant to estimate how good it is to be at x from the perspective of the player to move at the root of the tree.

Original: b. A policy estimator ˆ π : T × S → P ( A ) , which returns a probability distribution over actions ˆ π T ( s ) given a tree T and a state s .
Revised: b. A policy estimator ˆ π : T × X → P ( X ) , which returns a probability distribution over possible next states ˆ π T ( x ) given a tree T and a node x in T .

Original: The policy estimator is meant to approximate the  optimal policy from  the perspective of the player playing from s .
Revised: The policy estimator is meant to approximate the result of playing the optimal policy from x (from the perspective of the player to move at x ).

Original: For both KataGo and AlphaGo, the value function estimator and policy estimator are defined by two deep neural network heads with a shared backbone.
Revised: For both KataGo and AlphaGo, the value function estimator and policy estimator are deﬁned by two deep neural network heads with a shared backbone.

Original: The reason that ˆ V and ˆ π also take a tree T as an argument is because the estimators factor in the sequence of moves leading up to a state in the game.
Revised: The reason that ˆ V and ˆ π also take a tree T as an argument is because the estimators factor in the sequence of moves leading up to a node in the tree.

Original: A playout is performed by taking a walk in the current game tree T .
Revised: A playout is performed by taking a walk in the current game tree T .

Original: The walk goes down the tree until it attempts to walk to a node s ′ that does not exist in the tree.
Revised: The walk goes down the tree until it attempts to walk to a node x (cid:48) that either doesn’t exist in the tree or is a terminal node.

Original: This node isthen added  to the tree, after which the playout is complete.
Revised: At this point the playout ends and x (cid:48) is added as a new node to the tree (we allow duplicate terminal nodes in the tree).

Original: We startthe walk at the root of the tree.
Revised: Walks start at the root of the tree.

Original: Let s be where we are currently in the walk.
Revised: Let x be where we are currently in the walk.

Original: The child c we walk to  is given by 
Revised: The child c we walk to (which may not exist in the tree) is given by [Equation] opponent

Original: [Equation]
Revised: [Equation]

Revised: where the argmin and argmax are taken over all children reachable in a single legal move from x .

Original: There are some new pieces of notation in Eq 1.
Revised: There are some new pieces of notation in Eq 1.

Original: Here is what they mean:
Revised: Here are what they mean:

Original: ¯ V T : S → R takes a node x and returns the average value of ˆ V T across all the nodes in the subtree of T rooted at x (which includes x ).
Revised: ¯ V T : X → R takes a node x and returns the average value of ˆ V T across all the nodes in the subtree of T rooted at x (which includes x ).

Revised: In the special case that x is a terminal node, ¯ V T ( x ) is the result of the ﬁnished game as given by the game-simulator.

Original: When x does not exist in T, we instead use the more complicated formula
Revised: When x does not exist in T we instead use the more complicated formula

Original: [Equation]
Revised: [Equation]

Original: where par( x ) is the parent of x in T and β is a constant that controls how much we deprioritize exploration after we have already done some exploration.
Revised: where par T ( x ) is the parent of x in T and β is a constant that controls how much we de-prioritize exploration after we have already done some exploration.

Original: α  is a constant to trade off between exploration and exploitation.
Revised: α ≥ 0 is a constant to trade off between exploration and exploitation.

Original: N T : S → Z ≥ 0 takes a node x and returns the number ofnodes in the subtree of T rooted at x . 
Revised: S T : X → Z ≥ 0 takes a node x and returns the size of the subtree of T rooted at x . Duplicate terminal nodes are counted multiple times.

Original: If x is not in T , then N T ( x ) = 0 .
Revised: If x is not in T , then S T ( x ) = 0 .

Original: The first term in Eq 1 can be thought of as the exploitation term, and the second term can be thought of as the exploration term and is inspired by UCB algorithms.
Revised: The ﬁrst term in Eq 1 can be thought of as the exploitation term, and the second term can be thought of as the exploration term and is inspired by UCB algorithms.

Original: A.1. MCTS FINAL MOVE SELECTION
Revised: B.1. MCTS F INAL M OVE S ELECTION

Original: The final move to be selected by MCTS isusually sampled from a distribution proportional to
Revised: The ﬁnal move to be selected by MCTS is sampled from a distribution proportional to

Original: [Equation]
Revised: [Equation]

Original: where c in this case is a child of the root node.
Revised: where c in this case is a child of the root node.

Original: The temperature parameter τ trades off between exploration and exploitation.
Revised: The temperature parameter τ trades off between exploration and exploitation.

Original: A.1. U PDATELESS FORMULATION VS . STANDARD FORMULATION
Revised: B.1. E FFICIENTLY I MPLEMENTING MCTS

Original: To efficiently implement the playout procedure one should keep running values of ¯ V T and N T for every node in the tree.
Revised: To efﬁciently implement the playout procedure one should keep running values of ¯ V T and S T for every node in the tree.

Original: These values should be updated whenever a new node is added.
Revised: These values should be updated whenever a new node is added.

Original: The standard formulation of MCTS bakes these updates into the algorithm specification.
Revised: The standard formulation of MCTS bakes these updates into the algorithm speciﬁcation.

Original: The updateless formulation hides the procedure for computing ¯ V T and N T to simplify exposition.
Revised: Our formulation hides the procedure for computing ¯ V T and S T to simplify exposition.

Original: A. A DVERSARIAL MCTS: S AMPLE (A-MCTS-S)
Revised: B. A DVERSARIAL MCTS: S AMPLE (A-MCTS-S)

Original: In this section, we describe in detail how our Adversarial MCTS: Sample (A-MCTS-S) attack is implemented.
Revised: In this section, we describe in detail how our Adversarial MCTS: Sample (A-MCTS-S) attack is implemented.

Original: Similar to the standard MCTS procedure described in Appendix A.1, we build a game tree, starting from  tree T 1 with a single root node, and adding nodes to thegame treeT i via a series of R playouts.
Revised: We build off of the framework for vanilla MCTS as described in Appendix B.1. A-MCTS-S, just like MCTS, starts from a tree T 0 with a single root node and adds nodes to the tree via a series of N playouts.

Original: Finally, we derive the next move distribution from the final game tree T R by sampling from a distribution proportional toN A-MCTS
Revised: We derive the next move distribution from the ﬁnal game tree T N by sampling from the distribution proportional to

Original: [Equation]
Revised: [Equation]

Original: where c is a child of the root node, τ is a temperature parameter, and N A-MCTS T ( c ) is the number of visits to c other than those ending in victim-nodes where the game has not completed. Formally, it is the sum of the weights ofnodes in T given by
Revised: where c is a child of the root node of T N .

Original: [Equation]
Revised: [Equation]

Original: where a “terminal” node is one where the game isfinished (whether by the  turn limit being reached, one player resigning, or by two players passing consecutively). We grow the tree by A-MCTS playouts. At victim-nodes, we sample directly from the victim’s policy π v : walk
Revised: Here, S A-MCTS T is a modiﬁed version of S T that measures the size of a subtree while ignoring nonterminal victim-nodes (at victim-nodes it is the victim’s turn to move, and at self-nodes it is the adversary’s turn to move). Formally, S A-MCTS T ( x ) is the sum of the weights of nodes in the subtree of T rooted at x , with weight function

Original: [Equation]
Revised: [Equation]

Original: -MCTS
Revised: MCTS

Original: [Equation]
Revised: [Equation]

Original:  sample  from [Equation]
Revised: We grow the tree by A-MCTS playouts. At victim-nodes, we sample directly from the victim’s policy π v :

Original: [Equation]
Revised: [Equation]

Original: This is a perfect model of the victim without search.
Revised: This is a perfect model of the victim without search.

Original: However, it will tend to underestimate the strength of the victim when the victim plays with search.
Revised: However, it will tend to underestimate the strength of the victim when the victim plays with search.

Original: At self-nodes, we instead take the move with the best upper confidence bound.
Revised: At self-nodes, we instead take the move with the best upper conﬁdence bound just like in regular

Original: Specifically, we evaluate the adversary’s policy network ˆ π T ( s ) which returns a probability distribution over actions, and the value function estimator ˆ V T ( s ) which estimates the utility of a state s which is some linear combination of estimated win-probability and estimated final score-differential. These functions are parameterized by the tree T , not just the state s , as the networks factor in the sequence of moves leading up to a state in the game. We then compute an upper confidence bound equal to the sum of a weighted average of the value of a child ¯ V A-MCTS T ( c ) , and an exploration bonus equal to the policy prior ˆ π T ( s ) weighted by how few moves we have taken starting from c . Putting this together, we arrive at the expression:
Revised: MCTS:

Original: [Equation]
Revised: [Equation]

Original: Note this is similar to Eq 1 from the previous section.
Revised: Note this is similar to Eq 1 from the previous section.

Original: The key difference is thatrather than counting nodes, we count the weights :
Revised: The key difference is that we use S A-MCTS T [Equation]

Original: N A-MCTS T ( s ) is the total weight in the subtree rooted at s .
Revised: x ) (a weighted version of S T ( x ) ) and ¯ V A-MCTS T ( c ) (a weighted version of ¯ V T ( c ) ). Formally, ¯ V A-MCTS T

Original: ¯ V A-MCTS T ( c ) is the weighted average of the value function estimator ˆ V T ( x ) across all nodes x in the subtree of T rooted at c , weighted by w A-MCTS T ( s ) .
Revised: [Equation] c [Equation] is the weighted average of the value function estimator ˆ V T ( x ) across all nodes x in the subtree of T rooted at c , weighted by w A-MCTS T ( x ) .

Original: If c does not exist in T , then we instead use the more complicated formula
Revised: If c does not exist in T or is a terminal node, we fall back to the behavior of ¯ V T ( c ) .

Original: [Equation]
Revised: [Table]

Original: where par( x ) is the parent of x in T and β is a constant that controls how much we deprioritize exploration after having already done some exploration. A. P ASS -A LIVE DEFENSE
Revised: [Table]

Original: Our hard-coded defense modifies KataGo’s
Revised: Our hard-coded defense modiﬁes KataGo’s

Original: C++ code to directly remove passing moves from consideration after MCTS, setting their probability to zero.
Revised: C++ code to directly remove passing moves from consideration after MCTS, setting their probability to zero.

Original: Since the victim must eventually pass in order for the game to end, we allow passing to be assigned nonzero probability when there are no legal moves, or when the only legal moves are inside the victim’s own pass-alive territory.
Revised: Since the victim must eventually pass in order for the game to end, we allow passing to be assigned nonzero probability when there are no legal moves, or when the only legal moves are inside the victim’s own pass-alive territory.

Original: We use a pre-existing function inside the KataGo codebase, Board::calculateArea , to determine which moves are in pass-alive territory.
Revised: We use a pre-existing function inside the KataGo codebase, Board::calculateArea , to determine which moves are in pass-alive territory.

Original: The term “pass-alive territory” is defined in the KataGo rules as follows:
Revised: The term “pass-alive territory” is deﬁned in the KataGo rules as follows (Wu, 2021):

Original: A {maximal-non-black, maximal-non-white} region R is pass-alive-territory for {Black, White} if all {black, white} regions bordering it are pass-alive-groups, and all or all but one point in R is adjacent to a {black, white} pass-alive-group, respectively (Wu, 2021).
Revised: A {maximal-non-black, maximal-non-white} region R is pass-alive-territory for {Black, White} if all {black, white} regions bordering it are pass-alive-groups, and all or all but one point in R is adjacent to a {black, white} pass-alive-group, respectively.

Original: The notion “pass-alive group” is a standard concept in Go 
Revised: The notion “pass-alive group” is a standard concept in Go (Wu, 2021):

Original: : A black or white region R is a pass-alive-group if there does not exist any sequence of consecutive pseudolegal moves of the opposing color that results in emptying R (Wu, 2021).
Revised: A black or white region R is a pass-alive-group if there does not exist any sequence of consecutive pseudolegal moves of the opposing color that results in emptying R.

Original: KataGo uses an algorithm introduced by (Benson, 1976) to efficiently compute the pass-alive status of each group.
Revised: KataGo uses an algorithm introduced by Benson (1976) to efﬁciently compute the pass-alive status of each group.

Original: For more implementation details, we encourage the reader to consult the official
Revised: For more implementation details, we encourage the reader to consult the ofﬁcial

Original: KataGo rules and the KataGo codebase on GitHub.
Revised: KataGo rules and the KataGo codebase on GitHub.

Original: A. H YPERPARAMETER S ETTINGS
Revised: C H YPERPARAMETER S ETTINGS

Original: We enumerate the key hyperparameters used in our training run in Table 1.
Revised: We enumerate the key hyperparameters used in our training run in Table 1.

Original: For brevity, we omit hyperparameters that are the same as KataGo defaults andwhich have only a minor effect on performance.
Revised: For brevity, we omit hyperparameters that are the same as KataGo defaults and have only a minor effect on performance.

Original: The key difference from standard KataGo training is that our adversarial policy uses a b6c96 network architecture, consisting of 6 blocks and 96 channels.
Revised: The key difference from standard KataGo training is that our adversarial policy uses a b6c96 network architecture, consisting of 6 blocks and 96 channels.

Original: This is much smaller than the victim, which uses a b20c256 or b40c256 architecture.
Revised: This is much smaller than the victim, which uses a b20c256 or b40c256 architecture.

Original: We additionally disable a variety of game rule randomizations that help make KataGo a useful AI teacher in a variety of settings but are unimportant for our attack.
Revised: We additionally disable a variety of game rule randomizations that help make KataGo a useful AI teacher in a variety of settings but are unimportant for our attack.

Original: We also disable gatekeeping, designed to stabilize training performance, as our training has proved sufficiently stable without it.
Revised: We also disable gatekeeping, designed to stabilize training performance, as our training has proved sufﬁciently stable without it.

Original: We train at most 4 times on each data row before blocking for fresh data.
Revised: We train at most 4 times on each data row before blocking for fresh data.

Original: This is comparable to the original KataGo training run, although the ratio during that run varied as the number of asynchronous[Table] selfplay workers fluctuated over time.
Revised: This is comparable to the original KataGo training run, although the ratio during that run varied as the number of asynchronous selfplay workers ﬂuctuated over time.

Original: We use an adversary visit count of 600 , which is comparable to KataGo, though the exact visit count has varied between their training runs. 
Revised: We use an adversary visit count of 600 , which is comparable to KataGo, though the exact visit count has varied between their training runs. KGS handle Is KataGo? KGS rank

Original: A.4. C ONFIGURATION FOR C URRICULUM A GAINST V ICTIM W ITHOUT S EARCH
Revised: EGF rank EGD Proﬁle Fredda Fredrik Blomback cheater

Revised: Pavol Lisy

Revised: TeacherD

Revised: Dominik Boviz

Revised: NeuralZ (cid:88)

Revised: NeuralZ (cid:88)

Revised: NeuralZ (cid:88) ben

Revised: Benjamin Drean-Guenaizia sai

Revised: Alexandr Muromcev

Revised: Tichu

Revised: Matias Pankoke

Revised: Lukan

Revised: Lukas Podpera

Revised: HappyLook

Revised: Igor Burnaevskij

Revised: C. C ONFIGURATION FOR C URRICULUM A GAINST V ICTIM W ITHOUT S EARCH

Original: In Section 5.1, we train using a curriculum with an adversary win-rate threshold of 50% over the following checkpoints, all with no search:
Revised: In Section 5.1, we train using a curriculum over checkpoints, moving on to the next checkpoint when the adversary’s win-rate exceeds 50%. We ran the curriculum over the following checkpoints, all without search:

Original: Checkpoint 127: b20c256x2-s5303129600-d1228401921 ( Initial ).
Revised: Checkpoint 127: b20c256x2-s5303129600-d1228401921 ( Initial ).

Original: Checkpoint 200: b40c256-s5867950848-d1413392747 . 3.
Revised: Checkpoint 200: b40c256-s5867950848-d1413392747 . 3.

Original: Checkpoint 300: b40c256-s7455877888-d1808582493 . 4.
Revised: Checkpoint 300: b40c256-s7455877888-d1808582493 . 4.

Original: Checkpoint 400: b40c256-s9738904320-d2372933741 . 5.
Revised: Checkpoint 400: b40c256-s9738904320-d2372933741 . 5.

Original: Checkpoint 469: b40c256-s11101799168-d2715431527 . 6.
Revised: Checkpoint 469: b40c256-s11101799168-d2715431527 . 6.

Original: Checkpoint 505: b40c256-s11840935168-d2898845681 ( Latest ).
Revised: Checkpoint 505: b40c256-s11840935168-d2898845681 ( Latest ).

Original: These checkpoints can all be obtained from Wu 
Revised: These checkpoints can all be obtained from Wu (2022).

Original: We start with checkpoint 127 for computational efficiency: it is the strongest KataGo network of its size ( b20 = 20 blocks).
Revised: We start with checkpoint 127 for computational efﬁciency: it is the strongest KataGo network of its size, 20 blocks or b20 .

Original: After that the checkpoints are  approximately equally spaced in terms of training timesteps.
Revised: The subsequent checkpoints are all 40 block networks, and are approximately equally spaced in terms of training time steps.

Original: We include checkpoint 469 in between 400 and 505 for historical reasons: we ran some earlier experiments against checkpoint 469, so it is helpful to include checkpoint 469 in the curriculum to check performance is comparable to prior experiments.
Revised: We include checkpoint 469 in betweenand 505 for historical reasons: we ran some earlier experiments against checkpoint 469, so it is helpful to include checkpoint 469 in the curriculum to check performance is comparable to prior experiments.

Original: Checkpoint 505 is the latest confidently rated network.
Revised: Checkpoint 505 is the latest conﬁdently rated network.

Original: There are some more recent, larger networks ( b60 = 60 blocks) that may have an improvement of up to 150 Elo.
Revised: There are some more recent, larger networks ( b60 = 60 blocks) that may have an improvement of up to 150 Elo.

Original: However, they have had too few rating games to be confidently evaluated.
Revised: However, they have had too few rating games to be conﬁdently evaluated.

Original: A. S TRENGTH OFK ATA G O 
Revised: D S TRENGTH OF G O AI SYSTEMS

Original: In this section, we estimate the strength of  Latest  with and without search.
Revised: In this section, we estimate the strength of KataGo’s Latest network with and without search nd the AlphaZero agent from (Schmid et al., 2021) playing with 800 visits.

Original: A.5. S TRENGTH OF K ATA G O WITHOUT SEARCH
Revised: D. S TRENGTH OF K ATA G O W ITHOUT S EARCH

Revised: First, we estimate the strength of KataGo’s

Original:  We estimate strength using two independent methodologies and conclude that Latest without search is at the level of a weak professional.
Revised: Latest agent playing without search. We use two independent methodologies and conclude that Latest without search is at the level of a weak professional.

Original: One way to gauge the performance of Latest without search is to see how it fares against humans on online Go platforms.
Revised: One way to gauge the performance of Latest without search is to see how it fares against humans on online Go platforms.

Original: Per Table 2, on the online Go platform KGS, a slightly earlier (and weaker) checkpoint than Latest playing without search is roughly at the level of a top-100 European pro.
Revised: Per Table 2, on the online Go platform KGS, a slightly earlier (and weaker) checkpoint than Latest playing without search is roughly at the level of a top-100 European player.

Original: However, some caution is needed in relying on KGS rankings:
Revised: However, some caution is needed in relying on KGS rankings:

Original: Players on KGS compete under less focused conditions than in a tournament, so they may underperform.[Table] Is [Table]
Revised: Players on KGS compete under less focused conditions than in a tournament, so they may underperform.

Original: Fredda

Original: Fredrik Blomback

Original: [Table]

Original: Pavol Lisy

Original: [Table]

Original: Dominik Boviz

Original: [Table]

Original: Benjamin Drean-Guenaizia

Original: [Table]

Original: Alexandr [Table] [Table] KGS is a less serious setting than official tournaments, which makes cheating (e.g., using an AI) more likely.
Revised: KGS is a less serious setting than ofﬁcial tournaments, which makes cheating (e.g., using an AI) more likely.

Original: Thus human ratings may be inflated.
Revised: Thus human ratings may be inﬂated.

Original: Humans can play bots multiple times and adjust their strategies, while bots remain static.
Revised: Humans can play bots multiple times and adjust their strategies, while bots remain static.

Original: In a sense, humans are able to run adversarial attacks on the bots, and are even able to do so in a white-box manner since the source-code and network weights of a bot like KataGo are public
Revised: In a sense, humans are able to run adversarial attacks on the bots, and are even able to do so in a white-box manner since the source-code and network weights of a bot like KataGo are public.

Original: Another way to estimate the strength of Latest without search is to compare it to other AIs with known strengths and extrapolate performance across different amounts of search.
Revised: Another way to estimate the strength of Latest without search is to compare it to other AIs with known strengths and extrapolate performance across different amounts of search.

Original: Our analysis critically assumes the transitivity of Elo at high levels of play.
Revised: Our analysis critically assumes the transitivity of Elo at high levels of play.

Original: We walk through our estimation procedure below:
Revised: We walk through our estimation procedure below:

Original: Our anchor is ELF OpenGo at 80,000 visits per move, which won all 20 games played against four top-30 professional players (including five games against the now world number one) (Tian et al., 2019).
Revised: Our anchor is ELF OpenGo at 80,000 visits per move, which won all 20 games played against four top-30 professional players, including ﬁve games against the now world number one (Tian et al., 2019).

Original: We assume that ELF OpenGo
Revised: We assume that ELF OpenGo

Original: at 80,000 visits is strongly superhuman, meaning it has a 90%+ winrate over the strongest current human.
Revised: at 80,000 visits is strongly superhuman, meaning it has a 90%+ winrate over the strongest current human.

Original: At the time of writing,  top ranked player on Earth has an Elo of 3845 on goratings.org (Coulom, 2022).
Revised: At the time of writing, the top ranked player on Earth has an Elo of 3845 on goratings.org (Coulom, 2022).

Original: Under our assumption, ELF OpenGo at 80,000 visits per move would have an Elo of 4245+ on goratings.org. 2.
Revised: Under our assumption, ELF OpenGo at 80,000 visits per move would have an Elo of 4245+ on goratings.org. 2.

Original: The strongest network in the original KataGo paper was shown to be slightly stronger than ELF OpenGo
Revised: The strongest network in the original KataGo paper was shown to be slightly stronger than ELF OpenGo

Original: (Wu, 2019, Table 1) when both bots were run at 1600 visits per move.
Revised: (Wu, 2019, Table 1) when both bots were run at 1600 visits per move.

Original: From Figure 6, we see that the relative strengths of KataGo networks is maintained across different amounts of search.
Revised: From Figure 6, we see that the relative strengths of KataGo networks is maintained across different amounts of search

Original: We thus extrapolate that KataGo at 80,000 visits would also have an Elo of 3800+ on goratings.org.
Revised: We thus extrapolate that KataGo at 80,000 visits would also have an Elo of 4245+ on goratings.org.

Original: The strongest network in the original KataGo paper is comparable to the b15c192-s1503689216-d402723070 checkpoint on katagotraining.org (Wu, 2022).
Revised: The strongest network in the original KataGo paper is comparable to the b15c192-s1503689216-d402723070 checkpoint on katagotraining.org (Wu, 2022).

Original: We dub this checkpoint Original .
Revised: We dub this checkpoint Original .

Original: In a series of benchmark games, we found that Latest without search won 29/3200 games against Original with 1600 visits.
Revised: In a series of benchmark games, we found that Latest without search won 29/3200 games against Original with 1600 visits.

Original: This puts Original with 1600 visits ~800 Elo points ahead of Latest without search.
Revised: This puts Original with 1600 visits ~800 Elo points ahead of Latest without search.

Original: Finally, log-linearly extrapolating the performance of Original from 1600 to 80, visits using Figure 6 yields an Elo difference of ~900 between the two visit counts.
Revised: Finally, log-linearly extrapolating the performance of Original from 1600 to 80, visits using Figure 6 yields an Elo difference of ~900 between the two visit counts.

Original: Combining our work, we get that Latest without search is roughly 800 + 900 =
Revised: Combining our work, we get that Latest without search is roughly 800 + 900 =

Original: ~
Revised: ~

Original: Elo points weaker than ELF OpenGo with 80,000 visits.
Revised: Elo points weaker than ELF OpenGo with 80,000 visits.

Original: This would give Latest with-
Revised: This would give Latest without search an Elo rating of 4245 − 1700 =

Original: out search an Elo rating of ~2500 on goratings.org, putting it at the skill level of a weak professional. {+As a ﬁnal sanity check on these calculations, the raw AlphaGo Zero neural network was reported to have an Elo rating of 3,055, comparable to AlphaGo Fan’s 3,144 Elo. 
 Since AlphaGo Fan beat Fan Hui, a 2-dan professional player (Silver et al., 2017), this conﬁrms that well-trained neural networks can play at the level of human professionals.+}
Revised: ~2500 on goratings.org, putting it at the skill level of a weak professional. {+As a ﬁnal sanity check on these calculations, the raw AlphaGo Zero neural network was reported to have an Elo rating of 3,055, comparable to AlphaGo Fan’s 3,144 Elo. 
 Since AlphaGo Fan beat Fan Hui, a 2-dan professional player (Silver et al., 2017), this conﬁrms that well-trained neural networks can play at the level of human professionals.+}

Revised: Although there has been no direct comparison between

Original: A.5. S TRENGTH OF K ATA G O WITH SEARCH{+KataGo and AlphaGo Zero, we would expect them to be not wildly dissimilar. Indeed, if anything the latest versions of KataGo are likely stronger, beneﬁting from both a large distributed training run (amounting to over 10,000 V100 GPU days of training) and four years of algorithmic progress. [Figure] Network cp505 ( Latest ) cp
+}
Revised: {+KataGo and AlphaGo Zero, we would expect them to be not wildly dissimilar. Indeed, if anything the latest versions of KataGo are likely stronger, beneﬁting from both a large distributed training run (amounting to over 10,000 V100 GPU days of training) and four years of algorithmic progress. [Figure] Network cp505 ( Latest ) cp
+}

Revised: ( Initial ) cp103 ( Original )

Revised: cp cp cp cp

Revised: Agent

Revised: Victim?

Revised: Elo (rel GnuGo)

Revised: Elo (rel victim)

Revised: AlphaZero(s=16k, t=800k) + +

Revised: AG0 3-day(s=16k)

Revised: + +

Revised: AlphaGo Lee(time=1s)

Revised: + +

Revised: AlphaZero(s=800,t=800k)

Revised: (cid:88)

Revised: +

Revised: Pachi(s=100k)

Revised: + -

Revised: Pachi(s=10k) + -

Original:  In the previous section, we established that Latest without search is at the level of a weak professional with rating around ~2500 on goratings.org.
Revised: GnuGo(l=10) + - D. S TRENGTH OF K ATA G O W ITH S EARCH In the previous section, we established that Latest without search is at the level of a weak professional with rating around ~2500 on goratings.org.

Original: Assuming Elo transitivity, we can estimate the strength of Latest by utilizing Figure 6.
Revised: Assuming Elo transitivity, we can estimate the strength of Latest by utilizing Figure 6.

Original: In particular, our evaluation results tell us that Latest with 64 playouts/move is roughly 1063 Elo stronger than Latest with no search.
Revised: In particular, our evaluation results tell us that Latest with 64 playouts/move is roughly 1063 Elo stronger than Latest with no search.

Original: This puts Latest with 64 playouts/move at an Elo of ~3563 on goratings.org — within the top 20 in the world. 
Revised: This puts Latest with 64 playouts/move at an Elo of ~3563 on goratings.org—within the top 20 in the world. D. S TRENGTH OF A LPHA Z ERO

Revised: Prior work from Timbers et al.

Revised: (2022) described in Section 2 exploited the AlphaZero replica from

Revised: Schmid et al.

Revised: (2021) playing with 800 visits.

Original: A. E XPERIMENTAL R ESULTS A.6. M IMICKING THE A DVERSARIAL P OLICY
Revised: Unfortunately, this agent has never been evaluated against KataGo or against any human player, making it difﬁcult to directly compare its strength to those of our victims. Moreover, since it is a proprietary model, we cannot perform this evaluation ourselves. Accordingly, in this section we seek to estimate the strength of these AlphaZero agents using three anchors: GnuGo, Pachi and Lee Sedol. Our estimates suggest AlphaZero with 800 visits ranges in strength from the top 200 of human players, to being slightly superhuman.

Revised: We reproduce relevant Elo comparisons from prior work in Table 3.

Revised: In particular, Table 4 of Schmid et al.

Revised: (2021) compares the victim used in Timbers et al.

Revised: (2022), AlphaZero(s=800,t=800k), to two open-source AI systems, GnuGo and Pachi.

Revised: It also compares it to a higher visit count version

Revised: AlphaZero(s=16k, t=800k), from which we can compare using Silver et al.

Revised: (2018) to AG0 3-day and from there using Silver et al.

Revised: (2017) to AlphaGo Lee which played Lee Sedol.

Revised: Our ﬁrst strength evaluation uses the open-source anchor point provided by Pachi(s=10k).

Revised: The authors of Pachi (Baudiš & Gailly, 2012) report it achieves a 2-dan ranking on KGS (Baudiš & Gailly, 2020) when playing with 5000 playouts and using up to 15,000 when needed.

Revised: We conservatively assume this corresponds to a 2-dan EGF player (KGS rankings tend to be slightly inﬂated), giving an EGF Elo of 2200.

Revised: Assuming transitivity, the victim AlphaZero(s=800,t=800k) would then have an EGF Elo of 4299.

Revised: The top EGF professional Ilya Shiskin has an EGF Elo of 2830 (Federeation, 2022) at the time of writing, and 2979 on goratings.org (Coulom, 2022).

Revised: Using Ilya as an anchor, this would give AlphaZero(s=800,t=800k) a goratings.org Elo of 4299+2979-2830=4448.

Revised: This is superhuman, as the top player at the time of writing has an Elo of 3845.

Revised: However, some caution is needed here—the Elo gap between Pachi(s=10k) and AlphaZero(s=800,t=800k) is huge, making the exact value unreliable.

Revised: The gap from Pachi(s=100k) is smaller, however unfortunately to the best of our knowledge there is no public evaluation of Pachi at this strength.

Revised: However, the results in Baudiš & Gailly (2020) strongly suggest it would perform at no more than a 4-dan KGS level, or at most an EGF Elo of 2400.

Revised: Repeating the analysis above then gives a goratings.org Elo of 2400+(2308-869)+(2979-2830)=3988 Elo.

Revised: This still suggests the victim is superhuman, but only barely.

Revised: However, if we instead take GnuGo level 10 as our anchor, we get a quite different result.

Revised: It is known to play between 10 and 11kyu KGS (KGS, 2022a), or an EGF Elo of around 1050.

Revised: This gives an implied EGF Elo for AlphaZero(s=800,t=800k) of 3149, or a goratings.org Elo of 3298 Elo.

Revised: This is still strong, in the top 200 world players, but is far from superhuman.

Revised: The large discrepancy between these results led us to seek a third anchor point: how AlphaZero performed relative to previous AlphaGo models that played against humans.

Revised: A complication is that the version of AlphaZero that Timbers et al.

Revised: use differs from that originally reported in Silver et al.

Revised: (2018), however based on private communication with Timbers et al.

Revised: we are conﬁdent the performance is comparable:

Revised: These agents were trained identically to the original AlphaZero paper, and were trained for the full 800k steps.

Revised: We

Revised: actually

Revised: used the original code, and did

Revised: a lot of validation work with Julian Schrittweiser & Thomas Hubert (

Revised: two

Revised: of

Revised: the authors of the original AlphaZero

Revised: paper

Revised: and authors of the ABR

Revised: paper)

Revised: to

Revised: verify

Revised: that the reproduction

Revised: was

Revised: exact.

Revised: We

Revised: ran internal strength

Revised: comparisons that

Revised: match the original training runs.

Revised: Table 1 of Silver et al.

Revised: (2018) shows that AlphaZero is slightly stronger than AG0 3-day (AlphaGo

Revised: Zero, after 3 days of training), winning 60 out of 100 games giving an Elo difference of +70.

Revised: This tournament evaluation was conducted with both agents having a thinking time of 1 minute.

Revised: Table S from Silver et al.

Revised: (2018) reports that 16k visits are performed per second, so the tournament evaluation used a massive 960 k visits–signiﬁcantly more than reported on in Table 3.

Revised: However, from

Revised: Figure 6 we would expect the relative Elo to be comparable between the two systems at different visit counts, so we extrapolate AG0 3-day at 16k visits as being an Elo of 3139 − 70 = 3069 relative to AlphaZero(s=16k, t=800k).

Revised: Figure 3a from Silver et al.

Revised: (2017) report that AG0 3-day achieves an Elo of around 4500.

Revised: This compares to an Elo of 3,739 for AlphaGo Lee.

Revised: To the best of our knowledge, the number of visits achieved per second of AlphaGo Lee has not been reported.

Revised: However, we know that AG0 3-day and

Revised: AlphaGo Lee were given the same amount of thinking time, so we can infer that AlphaGo Lee has an Elo of − 761 relative to AG0 3-day.

Revised: Consequently, AlphaGo Lee(time=1s) thinking for 1 second has an Elo relative to GnuGo of 3069 − 761 = 2308 .

Revised: Finally, we know that AlphaGo Lee beat Lee Sedol in four out of ﬁve matches, giving AlphaGo Lee a +240 Elo difference relative to Lee Sedol, or that Lee Sedol has an Elo of 2068 relative to Gnu Go level 10.

Revised: This would imply that the victim is slightly stronger than Lee Sedol.

Revised: However, this result should be taken with some caution.

Revised: First, it relies on transitivity through many different versions of

Revised: AlphaGo.

Revised: Second, the match between AlphaGo Lee and Lee Sedol was played under two hours of thinking time with 3 byoyomi periods of 60 seconds per move Silver et al.

Revised: (2018, page 30).

Revised: We are extrapolating from this to some hypothetical match between AlphaGo Lee and Lee Sedol with only 1 second of thinking time per player.

Revised: Although the Elo rating of Go AI systems seems to improve log-linearly with thinking time, it is unlikely this result holds for humans.

Revised: E E XPERIMENTAL R ESULTS E. M IMICKING THE A DVERSARIAL P OLICY

Original: Our adversarial policies appear to follow a very simple strategy.
Revised: Our adversarial policies appear to follow a very simple strategy.

Original: They play in the corners and edges, staking out a small region of territory while allowing the victim to amass a larger territory.
Revised: They play in the corners and edges, staking out a small region of territory while allowing the victim to amass a larger territory.

Original: However, the adversary ensures that it is ahead in raw points prior to the victim securing its territory.
Revised: However, the adversary ensures that it is ahead in raw points prior to the victim securing its territory.

Original: If the victim then passes prematurely, the adversary wins.
Revised: If the victim then passes prematurely, the adversary wins.

Original: However, it is possible that this seemingly simple policy hides a more nuanced exploit.
Revised: However, it is possible that this seemingly simple policy hides a more nuanced exploit.

Original: For example, perhaps the pattern of stones it plays form an adversarial example for the victim’s network.
Revised: For example, perhaps the pattern of stones it plays form an adversarial example for the victim’s network.

Original: To test this, one of the authors attempted to mimic the adversarial policy after observing some of its games.
Revised: To test this, one of the authors attempted to mimic the adversarial policy after observing some of its games.

Revised: The author was unable replicate this attack when KataGo was conﬁgured in the same manner as for the training and evaluation runs in this paper.

Revised: However, when the friendlyPassOk ﬂag in

Original: The author was ableto successfully replicate this attack against the NeuralZ06 bot on KGS, as illustrated in Figure 7.
Revised: KataGo was turned on, the author was able successfully replicate this attack against the NeuralZ bot on KGS, as illustrated in Figure 7.

Original: This bot uses checkpoint 469 (see Appendix A.4.1) with no search.
Revised: This bot uses checkpoint 469 (see Appendix C.1) with no search.

Original: The author has little experience in Go and is certainly weaker than 20 kyu, so they did not win due to any skill in Go.
Revised: The author has limited experience in Go and is certainly weaker than 20 kyu, so they did not win due to any skill in Go.

Original: [Figure] 
Revised: [Figure] [Figure]

Revised: E. H UMANS VS . A DVERSARIAL P OLICY

Revised: The same author from Section E.1 (strength weaker than 20kyu) played two manual games against the strongest adversary from Figure 3.

Original: [Figure]
Revised: In both games the author was able to achieve an overwhelming victory. See Figure 8 for details. This evaluation is imperfect in one signiﬁcant way: the adversary was not playing with an accurate model of the author (rather it modeled the author as Latest with 1 visit). However, given our understanding of how the adversary works and the fact that the author in question knows not to prematurely pass, we predict that the adversary would probably not win even if it had access to an accurate model of the author.

Revised: E. T RANSFERRING TO O THER C HECKPOINTS

Revised: In Figure

Revised: [Figure]

Revised: we train adversaries against the Latest and Initial checkpoints respectively and evaluate against both checkpoints.

Revised: We ﬁnd adversaries do substantially better against the victim they were trained to target, although they do transfer to a limited extent.

Revised: [Figure]

Revised: Victim visits

Revised: [Figure]

Original: A v e r a g e
Revised: A v e r a g e

Original: K a t a
Revised: K a t a

Original: G o w i n m a r g i n
Revised: G o w i n m a r g i n

Original: Edge attack
Revised: Edge attack

Original: Mirror attack
Revised: Mirror attack

Original: Spiral attack
Revised: Spiral attack

Original: A.6. T RANSFERRING TO O THER C HECKPOINTS

Original: In Figure 8, we train adversaries against the Latest and Initial checkpoints.

Original: We find adversaries do substantially better against the victim they were trained to target, although they do transfer to a limited extent.

Original: A.6. B ASELINE A TTACKS
Revised: E. B ASELINE A TTACKS

Original: In Figure 9, we plot the win margin of the KataGo victim
Revised: In Figure 10, we plot the win margin of the KataGo victim

Original: Latest playing against baselines.
Revised: Latest playing against baselines.

Original: A. A DVERSARIAL B OARD S TATE
Revised: E. A DVERSARIAL B OARD S TATE

Original: This paper focuses on training an agent that can exploit Go-playing AI systems.
Revised: This paper focuses on training an agent that can exploit Go-playing AI systems.

Original: A related problem is to find an adversarial board state which could be easily won by a human, but which Go-playing AI systems will lose from.
Revised: A related problem is to ﬁnd an adversarial board state which could be easily won by a human, but which Go-playing AI systems will lose from.

Original: In many ways this is a simpler problem, as the adversarial board state need not be a state that the victim agent would allow us to reach in normal play.
Revised: In many ways this is a simpler problem, as an adversarial board state need not be a state that the victim agent would allow us to reach in normal play.

Original: Nonetheless, adversarial board states can be a useful tool to probe the blindspots that Go AI systems may have.
Revised: Nonetheless, adversarial board states can be a useful tool to probe the blindspots that Go AI systems may have.

Original: In Figure 10 we present a manually constructed adversarial board state.
Revised: In Figure 11 we present a manually constructed adversarial board state.

Original: Although quite unlike what would occur in a real game, it represents an interesting if trivial (for a human) problem.
Revised: Although quite unlike what would occur in a real game, it represents an interesting if trivial (for a human) problem.

Original: The black player can always win by executing a simple strategy.
Revised: The black player can always win by executing a simple strategy.

Original: If white plays in between two of black’s disconnected groups, then black should immediately respond by connecting those groups together.
Revised: If white plays in between two of black’s disconnected groups, then black should immediately respond by connecting those groups together.

Original: Otherwise, the black player can connect any two of its other disconnected groups together.
Revised: Otherwise, the black player can connect any two of its other disconnected groups together.

Original: Whatever the white player does, this strategy ensures that blacks’ groups will eventually all be connected together.
Revised: Whatever the white player does, this strategy ensures that blacks’ groups will eventually all be connected together.

Original: At this point, black has surrounded the large white group on the white and can capture it, gaining substantial territory and winning.
Revised: At this point, black has surrounded the large white group on the right and can capture it, gaining substantial territory and winning.

Original: Although this problem is simple for human players to solve, it proves quite challenging for otherwise sophisticated Go AI systems such as KataGo.
Revised: Although this problem is simple for human players to solve, it proves quite challenging for otherwise sophisticated Go AI systems such as KataGo.

Original: In fact, KataGo playing against a copy of itself loses as black 48% of the time.
Revised: In fact, KataGo playing against a copy of itself loses as black 48% of the time.

Original: Even its wins are far less decisive than they should be—18% of the time it wins by only 0.5 points!
Revised: Even its wins are far less decisive than they should be—18% of the time it wins by only 0.5 points!

Original: We conjecture this is because black’s winning strategy, although simple, must be executed flawlessly and over a long horizon.
Revised: We conjecture this is because black’s winning strategy, although simple, must be executed ﬂawlessly and over a long horizon.

Original: Black will lose if at any point it fails to respond to white’s challenge, allowing white to fill in both empty spaces between black’s groups.
Revised: Black will lose if at any point it fails to respond to white’s challenge, allowing white to ﬁll in both empty spaces between black’s groups.

Original: This problem is analogous to the classical cliff walking reinforcement learning task (Sutton & Barto, 2018, Example 6.6).
Revised: This problem is analogous to the classical cliff walking reinforcement learning task (Sutton & Barto, 2018, Example 6.6).