Are AlphaZero-like Agents Robust to Adversarial Perturbations?
Anonymous Author(s)
AffiliationAddress email
Abstract
AlphaZero (AZ) has demonstrated that neural network-based Go AIs can surpasshuman game performance by a large margin.
Even without any Monte Carlo treesearch (MCTS), the Policy-Value neural network (PV-NN), which served as aheuristic in AZ, achieves comparable performance to professional players thatmost humans cannot reach.
However, the robustness of those AZ agents and their
PV-NNs has not been studied in the literature.
Although it is well known thatconvolutional networks used in computer vision are not robust, existing adversarialattacks cannot be directly applied due to the difficulty of defining semanticallyinvariant perturbations.
Further, the discrete nature of Go prevents the use ofefficient gradient-based attacks.
In this paper, we develop the first adversarialattack on AZ agents of Go.
We show that both PV-NNs and AZ agents with fewsimulations can be fooled by adding one or two irrelevant stones.
For example, on58% of the AlphaGo Zero self-play games, our method can make the widely used
KataGo agent with 50 simulations play a losing action by adding two meaninglessstones on the board.
Moreover, these mistakes are so obvious that even normalhumans can independently interpret them.
In the experiments, we use the proposedmethod to examine the robustness of four publicly available Go AZ agents andone NoGo AZ agent.
The results show that those agents with few simulations arevulnerable and will make mistakes way below their level.
Introduction
AlphaZero (AZ) ([1]) like algorithms have achieved the state-of-the-art in Go – one of the mostchallenging games for artificial intelligence.
One of the main reasons is that they well train their
Policy-Value Neural Network (PV-NN) with reinforcement learning (RL) and then use it to guide the
Monte Carlo tree search (MCTS) [2, 3].
According to [4], PV-NN can achieve human professionallevel (Elo 3055) even without any lookahead.
Nowadays, even professional players trust AI agentsand endeavor to memorize their strategies.
However, are these AIs always better than humans?
Despite its significant impact, the robustness of AZ agents and its PV-NN network is under-explored.
Recently, it has been observed that many deep neural network models can be easily fooled by smalland human-imperceptible perturbations to the input [5, 6], but it is still unknown whether there exist“adversarial examples” for AZ agents and how easy we can find them.
Although adversarial attackalgorithms have been developed for computer vision [6], NLP [7, 8], and reinforcement learning[9, 10, 11, 12], none of the existing methods can be directly applied to attack Go AIs due to thedifficulties of defining “semantically invariant perturbation” in discrete games like Go.
In this paper, we proposed examining the robustness of agents for games like Go with “actionperturbation” that satisfies the following constraints.
First, the perturbed state should be a legal statethat is “close to” one of the given nature states.
If an example is not similar to a natural state, then it
A A
B B
C C
D D
E E
F F
G G
H H
J J
K K
L L
M M
N N
O O
P P
Q Q
R R
S S
T T
B
A will not be surprised that an agent is malfunctioning.
However, unlike continuous states, discrete states cannot simply add a small
-bation to a state like changing the x-axis position from 1 to 1.01.
Therefore, we use actionsas the perturbation and the number of actionswe added as distance to define which states are“close”.
Second, the perturbed state should alsobe semantically identical to the original state,and the equivalence should be verifiable to humans.
This constraint is to mimic the adversarialattacks on image classification, which demandsthe perturbation should be small enough and human imperceptible.
We thus constrain the addedactions to be “meaningless” (which will be formally defined in the paper).
By adding a limitednumber of “action perturbations”, we aim to findstates that the target agent will make mistakes(e.g.
, predicting a wrong move) way below itsnormal playing strength.
Take Fig.
1 as an example, B19 and P1 (marked as 1 and 2) are thetwo meaningless actions we added.
They aremeaningless because both actions do not help itsplayer gain any territory.
However, those actionsare able to let KataGo [13], a well-known Go AIwith 50 simulations, “forget” to kill the whitestones marked with squares (action A), and instead want to save the black stones marked with triangles (action B), even if doing so will lose thegame.
The contributions of this paper can be summarized below:
• For the first time, we reveal the vulnerability of both PV-NN and AZ agents.
With the proposed“action perturbation”, the adversarial examples we found (e
g., Fig. 1) are semantically equivalentto a natural state while leading to catastrophic behavior of the target agent.
Such examples may beexplored during large-scale MCTS and end up misleading the search.
• We design an efficient method to speedup the searching.
Our algorithm is usually more than 100times faster than brute force search in the experiments.• We conduct a comprehensive study on four state-of-the-art public AZ agents with six differentdatasets to reveal their robustness in the game of Go.
The results demonstrate that all these agentswith a small number of simulations are vulnerable.
In particular, by adding two meaningless moves,our attack can consistently achieve ≥ 90% success rates on the PV-NNs and achieve ≥ 58% onMCTS Agents with 50 simulations, which is the exact simulation count used by MuZero [14].Moreover, our method can also find bugs of AZ agents in other games since every game’s statecan be modified by actions.
For example, in our NoGo [15] experiment, adversarial examples are
found on 50% of data.
Background and Related Works
Terminology Games like Go can be described as a two-player zero-sum deterministic game [16]and can be defined as a tuple ⟨S , A , T , R⟩ , where S is the state space,
A is the action space,
T : S × A (cid:55)→ S is the transition function and R : S (cid:55)→ { 1 , 0 , − 1 } is the reward.
Each game startsfrom an initial state s 0 ∈ S (empty board) at time 0 .
At the current state s t , the turn-player, defined ass t .c , can play an action a t ∈ A ( s t ) .
In Go, players take turns placing a stone of his color on the board.
Therefore, all actions are composed of a color c and a position p indicating placing a c stone on p .
Theonly exception is the pass move a pass , which means the turn-player does not place any stone (gives upon his turn).
After playing action a t on state s t , we can get the next state by s t
+1 =
T ( s t , a t ) .
Thegame ends when reaching a terminal state s T , where we can determine who wins the game and eachplayer receives a reward.
Since the games are zero-sum, the reward of opponent player is −R ( s ) .
For the non terminal states, there will be no reward R ( s ) =
for any player.
Policy Value Neural Network (PV-NN) and Policy Value MCTS (Pv-MCTS) PV-NN is proposedin AlphaGo Zero paper [4].
PV-NN takes a state s as input and outputs a value v ( s ) and a policy p ( s ) .
Value 0 ≤ v ( s ) ≤ 1 is a scalar from 0 to 1 that indicates the estimated win-rate of current state.
Forexample, v ( s ) < 0 .
5 means that PV-NN believes the turn-player of state s is losing.
Note that if thevalue output range is − 1 ≤ v ( s ) ≤ 1 , the win-rate can be calculated by ( v ( s ) + 1) / 2 .
Policy p ( s ) isa vector that indicates each action a ’s probability 0 ≤ p ( a | s ) ≤ 1 .
A high probability for an action,e.g., p ( a | s ) > 0 .
9 , indicates that the the PV-NN highly recommends action a at state s .
Besides
PV-NN, AlphaGo Zero [4] also adopted Policy Value MCTS (PV-MCTS) to choose the best action.
Different from MCTS, PV-MCTS uses the value output of PV-NN instead of Monte Carlo simulationto evaluate the selected state values.
Moreover, PV-MCTS utilized the policy of PV-NN to narrowdown the search space with the PUCT algorithm [17].
The search efficiency of PUCT highly dependson the prior probability P ( s, a ∗ ) of best action a ∗ .
Therefore, if the root’s prior policy is incorrect,the search efficiency will drop dramatically.
Take Fig.
1 as an example, even after 50 simulations,
KataGo still couldn’t explore action A because the prior probability of action A is too small.
As morestates have been evaluated by PV-NN, PV-MCTS can provide stronger policy π ( s ) , value V ( s ) , andaction values Q ( s, a ) of a given root state s .
Adversarial Example It has been observed that many neural networks used in computer vision,
NLP and reinforcement learning are vulnerable to adversarial examples [5, 18, 9].
Traditionally, anadversarial example, created by minimally modifying a natural example, is semantically equivalentto the original natural example for humans but can make the target model produce a totally differentoutput.
Note that, the perturbed state should also look natural [19].
To define an adversarial examplefor a particular task, one has to define a reasonable perturbation set around the original example, anddesign an algorithm to find an example within the set that leads to incorrect behavior of the model(e.g.
, mis-classification).
In computer vision, the perturbation set is usually defined as a small ℓ p normball, as small perturbations to images are usually imperceptible to human.
And due to the continuoussearch space, existing image attacks often rely on gradient-based optimization to find adversarialexamples [20, 21, 22, 23].
On the other hand, for text models, there exists multiple definitions forthe perturbation set such as synonym substitution [24, 25], edit distance [26], or language modelbased scores [27].
And due to the discrete natural of text domain, finding an adversarial exampleleads to a combinatorial search problem, and several attacks have been proposed for this [28, 29, 24].
However, none of these attack methods can be directly applied to attack AZ agents for the followingreason: 1) Unlike image or NLP domain, the semantically invariant perturbation in Go is difficultto define (see Sec. 3.1).
2) Unlike other applications, the AI’s ability is much stronger than humanin Go, so it is non-obvious how to find adversarial examples that can be understood by human. 3)
To find an adversarial example in a game, one needs to go through all the states in the trajectorywhich introduces much higher complexity than attacking a classification model.
Although somerecent works have studied perturbations to states [12], actions [30], observations [9, 10, 31, 32] orrewards [33] in the reinforcement learning setting, none of them consider discrete input domains witha combinatorial large search space like Go or against planning based agent like AZ agents.
Blind Spots To improve Go AIs, some researchers [13] try to find their “blind spots”.
For example,from famous Tesuji (brilliant moves) problems, they can find some states such that the prior probabilityof the best action is almost 0 .
However, these blind spots are not adversarial examples since thelosing policy of the agent is still reasonable for humans even when knowing the best actions, andhence they are hard states for both humans and AI.
Further, there is no automatic and systematic wayto find those scarce blind spots.
On the other hand, we show that ≥ 58% of the games we selectedexist adversarial examples that normal humans can be better than state-of-the-art AIs.
By finding“bugs” efficiently, researchers may have enough training data to improve their AIs.
Method
Given a state s , a target agent (e.g., PV-NN or AZ agents), and a much weaker verifier (e.g., humans),our goal is to find an adversarial example s ′ that satisfies the following conditions:
C
The perturbed state s ′ is very close to the original state s in terms of ℓ 0 distance.
C
The perturbed state s ′ is semantically equivalent to s , and the verifier can verify that.
C
The target agent performs correctly on s but wrongly on s ′ , and the verifier can identify that.
Both C 1 and C 2 define a set of perturbed states, B ( s ) , for a given natural state s .
The problemof finding adversarial examples is then equivalent to finding a s ′ ∈ B ( s ) that satisfies the successcriterion (C 3 ).
In previous image attacks, B ( s ) is a small ℓ p ball [6, 20] and in text attacks B ( s ) isusually defined by word substitutions with synonyms [24].
However, in our case the definition of
B ( s ) is much more sophisticated and is computational expensive to check.
The existence of a muchweaker verifier ensures that the target network’s mistake is so simple that even the verifier can verifyit without knowing the examiner’s output (but knows about target agent’s outputs of s and s ′ ).
Hence,the playing strength of the verifier determines the difficulty of finding adversarial examples.
Theweaker the verifier is, the harder it is to find an example.
In addition, the algorithms are supposed tofind the s ′ without using the verifier since verifiers like humans are hard to include in an automaticprocess.
Therefore, we need to carefully design the B ( s ) and the success criterion.
In the following,we will discuss how to define B ( s ) and the success criterion in Subsections 3.1 and 3.2, respectively.
A search algorithm will then be introduced in Subsection 3.3 to speedup the search.
3.1 Defining the Perturbation Set
We define the perturbation set B ( s ) of a given state s based on conditions C 1 and C 2 .
For C 1 , we useactions as perturbation and the number of actions we added as distance.
In our experiments, we setdistance ℓ 0 as 2.
Since each state in Go can be represented by its trajectory (a list of actions), theperturbed state s ′ can be presented as
[Equation]
where a 0 , . .
. , a t − 1 is the trajectory of s and b 0 , b 1 are the extra actions.
For 1STEP attack, one ofthe b 0 , b 1 has to be the pass action (denoted as a pass ).
This means we place an additional stone on theboard without changing the turn player.
On the other hand, for 2STEP attack, both b 0 , b 1 are not thepass action, which means we add one black and one white stone on the board.
We do not includestates created by replacement like word substitution in text attacks, since there are no similar actionsin Games like Go.
For condition C 2 , we aim to maintain the semantic meaning after perturbation.
Since there is noground truth in Go, we resort to an “examiner” agent to verify the equivalence of states.
The examinershould be able to provide an accurate value V ( s ) , the policy π ( s ) , and the action values Q ( s, a ) for agiven state s .
For example, the examiner we used is the strongest PV-MCTS agent that runs with800 simulations.
To distinguish the output of the examiner and the target agent, we use v ( s ) andp ( a | s ) as the output of the target agent.
With the examiner, we define two states s, s ′ are semanticallyequivalent if the two states have the same turn players and similar win rate according to the examiner,which is shown in the following equation.
[Equation]
where s.c denotes the turn player ( c for color) of state s and η eq is the threshold used to define thatthe win rate is close.
However, we might miss some of the equivalent states since the examiner is notperfect and may be incorrect when evaluating V ( s ′ ) in Eq.
After adding perturbed actions, theexaminer’s prior policy might “forget” to consider the best action a ∗ s of s and thought s ′ is losing.
Therefore, we improved the performance of examiner on s ′ by providing a ∗ s as hint.
To be precise,we evaluate the value of s ′ by calculating max( V ( s ′ ) , 1 − V ( T ( s ′ , a ∗ s )) .
This is identical to forcingthe examiner to consider a ∗ s on s ′ .
This method makes it possible to find adversarial examples even ifthe target agent is the same as the examiner.
Besides ensuring s, s ′ to be semantically equivalent, such equivalence should be verifiable to themuch weaker verifier like humans.
Unlike standard image or text applications where human istreated as an oracle, the AI agents for Go are much more powerful than an human.
When the gameis too complicated, humans cannot even tell who has the advantage.
Therefore, even though bothstates s and s ′ are semantically equivalent (Eq. 1) to examiner, the verifier may not be able to tell.
Fortunately, we observe that, in the 1STEP attack where one of b 0 , b 1 is the pass action, the action weadded is easy for humans to understand that it does not affect the win rate.
In other words, humanplayers can identify “meaningless” actions which do not benefit the turn-player of the action.
Onthe other hand, in the 2STEP attack, it is harder for humans to verify whether the benefit of b 0 forthe turn player is the same as the benefit of b 1 for the next player in high-level games.
Therefore, in2STEP attack, we enforce the action we add to be meaningless: | V ( s ) − V ( F ( s, a ))
| ≤ η eq , where
F ( s, a ) : S × A (cid:55)→ S return the state after playing action a on state s without changing the turn player by skipping the opponent’s turn (Appendix F).
In addition, in the game of Go, we foundthat most meaningless actions are in one of the player’s territories (Appendix E).
Also, humanscan usually verify such actions faster.
Therefore, in the experiments of Go, we further restrict theperturbation action’s position a.p within one of the territories a.p ∈ Terr W ∪ Terr B , where Terr Wand Terr B is the territory of each color.
The results show that even with this stronger constraint, wecan find adversarial examples.
See Appendix G for the comparison without the territory constraint.
3.2 Success Criteria of Adversarial Attack
Next we define the success criteria (C 3 ) for our attack.
We consider two types of attacks: value attackand policy attack.
For value attack , the goal is to change the prediction of the target agent’s valuenetwork.
Therefore, we define the attack to be successful if
[Equation]
The first criterion ensures that the target agent produces the correct value on the original state s , andthe second criterion ensures that the target value network becomes incorrect after the perturbation.
Note that we already enforces V ( s )
≈ V ( s ′ ) in Subsection 3.1, so (2) implies that the perturbationwill change the target agent’s value but not the examiner’s value.
Constants η correct and η adv are thethresholds to define “correct” and “wrong” in C 3 .
For example, if we want to find an example that
PV-NN misclassified the winner of a state, we can set η adv = 0 .
We can further increase η adv if wewant the target’s output to be wrong by a larger margin.
In addition, the verifier can easily tell thattarget is wrong since s , s ′ are suppose to have the same winrate (C 2 ) but v ( s ) and v ( s ′ ) are different.
For policy attack , we aim to fool the policy output.
However, unlike normal classification, thereis more than one best action for a state.
Therefore, even when the perturbation can significantlychange the output policy, it doesn’t mean the new policy is incorrect.
We thus have to define thesuccessfulness of the policy attack by checking whether the value (computed by the examiner) willbe changed after playing the predicted action.
With these in mind, we say the policy attack leads to a“wrong” move if
[Equation]
where a ∗ s = arg max a ( p ( a | s )) and a ∗ s ′ =
arg max a ( p ( a |
s ′ )) are the target agent’s recommendedactions on s and s ′ .
The first criterion in (3) ensures that the target agent can predict a properaction for the original state s , and the second criterion checks whether the target agent will output asignificantly worse action in the perturbed state s ′ .
With Eq (3), although the examiner can verifythat the target agent plays a losing action on the perturbed state; the verifier may be uncertain thatthe action a ∗ s ′ is truly bad.
Luckily, in our qualitative study, by providing a ∗ s to humans as a hint,humans can immediately tell that a ∗ s is much more important to play than a ∗ s ′ in the state s ′ and certifythat the target agent is wrong on most pairs ( s, s ′ )
we found.
Hence, we add two more
restrictions:| Q ( s ′ , a ∗ s ) − V ( s ′ )
| ≤ η eq , and | Q ( s, a ∗ s ′ )
− V ( s ))
| ≥ η adv .
These restrictions ensure that a ∗ s is agood action and a ∗ s ′ is a bad action for both s and s ′ , so that the verifier can use them as an anchor.
3.3 An Efficient Attack Algorithm
Like other adversarial attacks on reinforcement learning [9, 10], we apply our attack to perturb a statein a given game, since it only takes one mistake for an agent to lose a game.
Formally, given a game
G = { s 0 , s 1 , . . .
, s T } , if we can find one pair of ( s i , s ′ i ) , where s ′
i is an adversarial example of s i ,then we have successfully attacked the agent on that game.
We will first apply the 1STEP attack to agame, and if it fails, we then conduct the 2STEP attack.
Since the search space is countable, a naive way to find an adversarial example is to conduct a bruteforce search to check all states in the search space.
However, this will lead to very high complexityand the search cannot be finished in practice.
Taking the 2STEP attack as an example, let T ≈ 300denote the game length, N ≈ 150 denote the average number of actions for a state, and M ≥ 800denote the simulation count of the examiner.
Then, we need O ( TNM ) time to obtain the searchspace for all B ( s i ) , i ∈ { 0 , 1 , . . .
, T } by checking which actions are meaningless.
Furthermore,assume that there are averagely ¯ N meaningless moves for each state.
Then, we need O ( T ¯ N 2 M )time to check the success criteria of each perturbation.
Note that in both stages the running time ofexaminer will dominate, as we need to run the examiner with much more MCTS steps.
Hence, wepropose the following two approaches to reduce the search complexity.
1: Input: a game s 1 , s 2 , . .
, s T , target agent t , examiner e 2: for i
= T to 0 do 3: if | e.V ( s i ) − t.v
( s i )
| > η correct then 4: continue 5: cands =
getMeaninglessActions ( e , s i ) 6: for b 0 in cands [0] do 7: for b 1 in cands [1] do 8: s ′ =
T ( T ( s i , b 0 ) , b 1 ) 9: if | t.v ( s ′ ) − e.V ( s i ) | ≥ | η adv − η eq | then 10: if | t.v ( s ′ ) − e.V ( s ′ )
| ≥ η adv and | e.V ( s ) − e.V ( s ′ )
| ≤ η eq then 11: return s ′ 12: return NULL
First, we reduce the time complexity of findingmeaningless actions by the following observation.
If an action a is a meaningful action (notmeaningless) of state s t , then it is likely that ais also a meaningful action of state s t − 1 .
Intuitively, if an action a is a meaningful action at s t ,it means that a can occupy some extra territorythat is not occupied at s t .
Hence, if a position isnot occupied in s t , it is likely that the positionis not occupied in s t − 1 and can be occupied byaction a too.
So action a is also a meaningfulaction for s t − 1 .
We formally prove this property under some assumptions in Appendix A.
Based on this observation, we run the searchin the backward manner from the final state s Tto the initial state s 1 .
Once an action is identified as meaningful at state s t , we do not needto check it again on any state { s i : i < t } .
Thiswill significantly save the computational time toenumerate B ( s ) .
The details of getting meaningless actions are shown in Appendix C.
Second, for the part of checking the attack success criterion, we derive a bound to quickly filter outunsuccessful perturbations.
Taking the value attack as an example, checking (1) and (2) requiresrunning the examiner on the perturbed state s ′ , which is the bottleneck of the algorithm.
To reducethis cost, we show that the condition of (1) and (2) implies
[Equation]
Since (4) only requires running examiner on the original state s instead of s ′ , we can check (4) firstbefore checking (1) and (2).
We observe this step can filter out more than 99% of the s ′ .
The overall attack algorithm is presented in Algorithm 1.
The input includes a game s 1 , s 2 , . . .
, s T , atarget agent t , and an examiner agent e .
As mentioned before, the search is conducted in a backwardmanner from s T back to s 1 (line 2).
For each s i , we first check whether s i is too hard for the target(line 3).
If so, we will skip this state (line 4).
We then compute all the meaningless actions of state s iusing the examiner with the efficient implementation mentioned above and store the meaninglessactions separately in cands [0] and cands [1] according to their color (line 5) as the candidates ofperturbation.
We then check whether each 2STEP perturbation will lead to a successful attack (line9-11).
Note that line 9 is based on (4) and examiner does not need to compute e.V ( s i ) again since ithas evaluated it on line 3.
For policy attack, we can skip all the states that are losing V ( s ) <
η advsince for those states there is no correct answer to attack.
(More details are in Appendix B).
[Table] [Table]
[Figure]
Experimental Results
Experiment Settings We evaluate our method on 19x19 Go with four GPUs (GTX 1080Ti).
Thefour open-source programs we used are KataGo
[13], Leela Zero [34], ELF OpenGo
[35], and CGI[36].
The strengths of these AI agents are KataGo
≫ Leela > ELF = CGI, where KataGo hasmore than 99% winrate against Leela when both using 800 MCTS simulations.
Since KataGo is thestrongest agent, we use its 40 blocks PV-NN with 800 simulations as our examiner.
For the thresholds,we set η eq = 0 .
1 , η correct = 0 .
15 , since after testing several different η eq and η correct values, thispair lead to more human-understandable results.
For the datasets, we selected 99 games from fivedifferent sources, which are AlphaGo Zero 40 blocks training self-play record (ZZ), AlphaGo Zerovs AlphaGo Master (ZM), AlphaGo Master vs Human champions (MH), the final games of LG
Cup World Go Championship (2001-2020) (LG), and the final games of Asian TV Cup (2001-2020)(ATV).
ZZ and ZM present the games played by AIs.
MH presents the games of AI vs Humans.
LGand ATV present the games of humans.
Note that the thinking time for ATV Cup is much shorter than
LG Cup, so we expect them to reflect human games with different strengths.
All the datasets have 20games, except ZZ has 19 games since the first game is played by two random agents.
After removingstates that are hard for examiner’s PV-NN to verify ( | e.v ( s ) − e.V ( s ) | > η eq ) and the meaningfulactions, the average search space of each game contains about 600 , 000 states.
Results on Different PV-NNs We first evaluate the robustness of the four agents’ PV-NNs since
PV-MCTS is much slower.
The results are shown in Table 1, where we attack each agent’s PV-NNwith both 1STEP and 2STEP attacks with various η adv .
We also present the average number ofevaluations required for the target agent and the examiner agent.
The first two groups in Table 1demonstrate the robustness of these agents against the value attack.
Group one ( η adv = 0 . 5 ) showsthat even the 1STEP attack can achieve above 90 % success rate on all agents and mostly achieve100 % on the 2STEP attack.
For those that has the same attack success rate, we can still compare theattack difficulties by the number of target evaluations.
For example, although the 2STEP successrates of KataGo, ELF, and CGI are all 100 %, the number of states that they have visited is totallydifferent.
For CGI, we are able to find an adversarial example after visiting 40 states, while ELFrequires visiting 11 , 108 states.
Hence, we conclude that Leela > ELF > KataGo >
CGI in termsof their robustness against value attack when η adv = 0 .
5 and η adv = 0 .
Interestingly, this rankingdoes not match the playing strength of each agent.
The third and fourth groups of Table 1 shows theresults of policy attack.
In general, we observe that it is harder to attack the policy than the value.
Interestingly, the ranking of four agents in terms of their policy’s robustness is KataGo
≫ Leela> ELF ≈ CGI, which aligns with the playing strengths of those agents.
In addition to Table 1, inorder to check whether it is possible to change the PV-NN’s output catastrophically, we conduct anextreme experiment on KataGo with η adv = 0 .
The result shows that the 2STEP success rates ofvalue and policy attacks still have 62% and 48%, respectively.
In Table 1, the column of SPEEDUPdemonstrates the speedup of our algorithm compared to the brute force algorithm.
The resultsshow that the proposed efficient search method is usually more than a hundred times faster than abrute-force search, especially for harder problems which require more 2STEP attacks.
Results on Different Datasets Besides the robustness of each agent, does the robustness varybetween different types of game records?
To answer this question, we consider the 5 datasets usedin the previous subsection, plus an additional FOX dataset, which consists of games randomlyselected from one of the most popular online Go playing platforms named FOX 1 .
We add this datasetto represent amateur players and see if the games played by weaker players are harder to attacksince those game are easier for human to understand.
The results in Table 2 show that PV-NNs arevulnerable to all levels of games, regardless of AI agents, professional players or non-professionalplayers.
Fig.
3 shows two of the examples.
We observe that the AlphaGo Master vs Human champions(MH) dataset has the lowest success rate on both policy and value attack compared to other datasets.
Our hypothesis is that AlphaGo Master is much stronger than human champions (AlphaGo Master
[Figure]
VALUE1STEPVALUE2STEPPOLICY1STEPPOLICY2STEP
[Figure]
[Table]
[Figure]
had played 60 games against human champions without losing any of them.)
When one playeris much stronger than the other, the winner of the game may be too obvious and easy for agentsto judge.
Also, it is harder for the policy to be weak enough to lose the game.
To support thishypothesis, we generate games of KataGo 40 blocks (K40) with 800 simulations vs KataGo 20 blocks(K20) with different simulations (20 games for each simulation) to simulate games played by playerswith different strengths.
Note that K20 is much weaker than K40 even with the same amount ofsimulations.
Therefore, when K20 uses fewer simulations, the gap between two players is even larger.
Fig.
2 shows the results of attacking KataGo’s
PV-NN with those games and η adv = 0 .
For gamesthat is generated by K20 only used 5 simulations, both policy and value output of KataGo’s
PV-NNare hard to attack.
The 1STEP policy attack even fails for all of the 20 games.
As we increase thesimulation count of K20, the success rates of all the attacks increase.
This suggests that the playingstrength between players of a game will likely affect the hardness of finding adversarial examples.
Robustness of PV-MCTS In this paragraph, we investigate the robustness of PV-MCTS withdifferent simulations.
Since PV-MCTS is much slower than PV-NN, we only test KataGo on the ZZdataset with η adv = 0 .
The results are shown on Table 3.
The first column is the simulation countof KataGo.
When simulation equal to one, it is the same as using PV-NN directly.
The first groupshows the results of value attack.
Even with 25 simulations, the 2STEP success rate is still 89%.
For group two, we observe that the policy of small simulations are even less robust than PV-NN’spolicy.
For example, when the simulations of KataGo is 10, the 1STEP success rate is 95%, while thepolicy of PV-NN is 84%.
Therefore, we conclude that with a small number of MCTS simulations,the agent will not be able to recover from the bad PV-NN outputs so will still be fooled by adversarialexamples.
However, when using more MCTS simulations, the attack success rates will still dropsince the agent has more chances to discover correct actions.
Quality of adversarial examples To examine the quality of the adversarial examples we found, werandomly selected 100 examples from all the experiments and conduct the following two studies toevaluate the quality of adversarial examples found by our algorithms.
Fig.
3 4 show four examples,and more can be found in Appendix H.
In the first study, we aim to examine whether the perturbations(one or two additional stones) obtained by our attack method are meaningless to human.
For eachadversarial example, we present both the original board and the additional stones to 3 humans (withlevel 2k, 3d, 5d) who served as verifiers, and for 90% of those perturbations they can certify that theperturbations are meaningless with a short thinking time.
The failure cases are the moves identifiedas meaningless by AI agents but it is not obvious to human.
Fig.
4 (right) shows one of the states,where the meaningless action J3 seems meaningful to human since it can capture the black stone
J2.
However, with a longer thinking time and discussion with each other, verifiers can understandthat the rest 10% of actions are meaningless.
In the second study, we aim to examine whether the“wrong” actions resulted by our policy attack can be verified by verifiers.
Similar to the previousexperiment, we present the original board, original action, perturbed board and the action afterperturbation to the examiners.
When the target agent is a PV-NN, human can identify 100% that therecommended actions after perturbation will change the result from winning to losing.
However, forthe adversarial examples where the target is PV-MCTS agents, only 70% of the recommended actions
A A
B B
C C
D D
E E
F F
G G
H H
J J
K K
L L
M M
N N
O O
P P
Q Q
R R
S S
T
T 1 B
A (a)
A A
B B
C C
D D
E E
F F
G G
H H
J J
K K
L L
M M
N N
O O
P P
Q Q
R R
S S
T
[Figure] [Figure]
A A
B B
C C
D D
E E
F F
G G
H H
J J
K K
L L
M M
N N
O O
P P
Q Q
R R
S S
T T
A
B (a)
[Figure]
A
[Figure]
B
[Figure]
C C
[Figure]
E E
F F
G G
H H
J
[Figure]
K K
[Figure]
M
[Figure]
O O
P P
Q Q
R
[Figure]
after perturbation can be identified as wrong by human.
The main reason is that as PV-MCTS agentsare able to lookahead, they tend to avoid actions that are clearly wrong, so the errors become moresubtle to human.
Fig.
4 (left) shows one of the states, where the white should play at A to save thestones marked with square, but with two meaningless moves added the PV-MCTS agent with 100simulations will play at B.
Although after playing B all the stones marked as square will be dead, it ishard for human to verify the outcome of this play.
Agents are sensitive to the ordering of actions in the trajectory Finally, after viewing thoseadversarial examples, we find that the policy attack succeeds often because the target agents aretoo reliant on the information of the last action.
For example, if KataGo knows the last actionof Fig. 1 besides the actions we add is white playing E3, even PV-NN knows that it needs toplay E1.
Based on this observation, we can simply improve the robustness of PV-NNs by thefollowing method.
Given a state s with trajectory a 0 , a 1 , . . .
, a t , we define an augmented state ¯ sas a 0 , a 1 , a 2 , . .
. , a t − 1 , a t , a t − 2 , a t − 3 .
If ¯ s is a legal state, we can define a robust policy p r ( a | s ) =( p ( a | s ) + p ( a | ¯ s )) / 2 .
With this method, PV-NNs can evaluated state s with different actions being thelast one.
We evaluated this method on the same setting as Table 3 (attacking KataGo’s PV-NN with
ZZ dataset and η adv = 0 . 5 ).
The success rate of 1STEP and 2STEP policy attacks dropped to 58%and 89%, even better than the original PV-NN and PV-MCTS with 25 simulations.
Experiments on the game of NoGo Our “action perturbation” method can also be used to findsevere bugs automatically on other games by removing the territory constrain.
We use the following9x9 NoGo [15] experiments to demonstrate it.
We use an AZ agent [37] with 800 simulations as theexaminer and use its PV-NN as the target agent.
Since NoGo is a game without any human experts,we use a traditional MCTS agent[38] with 1000 simulations as the weaker verifier.
Note that thetarget agent has a 100% winrate against the verifier in 1000 games.
Intuitively, the target networkwill not make a critical mistake that the verifier will not make.
However, on 20 self-play games ofthe target network (the average search space is about 10000), 50% of the games exist adversarialpolicy examples with η adv = 0 .
5 that the verifier can verify.
Moreover, our proposed efficient searchmethod based on Eq.
4 is 307x faster than the brute force search.
Conclusion
In this paper, we first properly define the perturbation set B ( s ) and the success criteria with the helpof an examiner, which become even more reliable after giving it some hints.
The adversarial exampleswe find are understandable to a much weaker verifier like normal human player.
We also proposed anew efficient search algorithm by reducing the usage of the examiner.
Our experiments found thateven the strongest AZ agent with small amount of simulations is vulnerable to our adversarial attack.
We hope our work can raise the attention that even for AI agents that surpass humans with a largemargin, they still can make simple mistakes that humans will not.
A limitation of this paper is that weare only able to identify adversarial states but haven’t been able to systematically guide the AZ agentsto those states [12].
However, we provide an efficient tool for finding those adversarial exampleswhich are "blind spots" of strong AZ agents, and the tool can be used for debugging and potentiallyimprove the performance of those agents, which may be more beneficial in practice.
Reproducibility & Ethics Statements
Reproducibility We have provided all the parameters of our method in the experiment settings.
The agents we used are download from GitHub and we just use the default settings of those agents.
We also provide our codes in the supplemental material.
We will later release it on GitHub.
Ourcode can be used on different games and agents as long as their actions are countable and the agentsupported GTP protocol.
Our code communicate with each agent with local port, hence each agentcan run on different docker.
We also provide dockers that can run those agents.
Ethics Our work is currently only use on games like Go and NoGo, therefore, there is no ethicalproblem at this stage.
However, if someone can lead some real world agent into states that found byour method, then it might be dangerous.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
In the Conclusion section
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] In Appendix A
(b) Did you include complete proofs of all theoretical results?
[Yes] In Appendix A
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] As supplemental
meterial and will opensource later
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] In Experiment settings
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[No] No random seeds are used.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] In Experiment settings
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] In Experiment settings
(b) Did you mention the license of the assets?
[Yes] In Experiment settings
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
In Experiment settings
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] The agents we used are opensource.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes]
Yes
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
A Proof of Meaningful Action
In this section, our goal is to prove that a meaningful action b 0 of the state s t will also be themeaningful action of the state s t − 1 under following two assumptions.a 1 Changing the order of the trajectory a 0 , a 1 , . .
. , a t − 1 of state s t will not change the state s t .a
The action a t − 1 is the best action of both s t − 1 and s ′ t − 1a 2 Adding an extra action to the board will not reduce the winrate of the action’s color.
Both assumptions are true in the most states on Go and NoGo.
For convenience, we assume the turncolor is of s t is black and b 0 is one of its meaningful action.
We also define V B ( s ) as the state valueof black for state s .
Now our goal is to prove that b 0 is still meaningful action to s t − 1 .
Since adding an extra action will only benefit to the action’s player and the turn player of b 0 is black.
According to Fig.
5 We just need to prove that
Given V B ( s t ) <
V B ( s ′ t ) , prove that V B ( s t − 1 ) <
V B ( s ′ t − 1 ) .
First since a t − 1 is the best action of s t − 1 , we have V B ( s t )
=
V B ( s t − 1 ) .
Also, since a t − 1 is the bestaction of s ′ t − 1 , we have V B ( s ′ t − 1 )
=
V B ( s ′′ ) .
Next, due to the a 1 assumption, we have s ′ t = s ′′ .
Finally, we have V B ( s ′ t − 1 ) =
V B ( s ′′ ) =
V B ( s ′ t ) > V B ( s t ) =
V B ( s t − 1 )
B Algorithm of the 2STEP Policy Attack
The algorithm is shown in algorithm 2.
1: Input: a seq states s i , target agent t , examiner e 2: for i
= T to 0 do 3: a s = arg max a ( t.p ( a | s ))
4: if | e.Q ( s i , a s ) − e.V ( s i ) | > η correct or e.V ( s i ) <
η adv then 5: continue 6: actions =
getMeaninglessActions ( e , s i ) 7: for b 0 in actions [0] do 8: for b 1 in actions [1] do 9: s ′ =
T ( T ( s i , b 0 ) , b 1 ) 10: a s ′ =
arg max a ( t.p ( a | s ′ ))
11: if | e.Q ( s i , a s ′ )
− e.V ( s i )
| ≥ η adv then 12: if | e.Q ( s ′ , a s ′ )
− e.V ( s ′ )
| ≥ η adv and | e.V ( s ) − e.V ( s ′ )
| ≤ η eq and | e.Q ( s ′ , a s ) − e.V ( s ′ )
| ≤ η correct then 13: return s ′ 14: return NULL
C Algorithm of Getting Meaningless Action
The algorithm is shown in algorithm 3.
do 8:
9: continue 10: if a.c == s i .c
then 11: s ′′ =
T ( T ( T ( s, a ) , a pass ) , a s ) 12: else 13: s ′′ =
T ( T ( T ( s, a pass ) , a ) , a s ) 14: v s ′′ =
e .get_quick_value( s ′′ ) 15: if a in meaningful _ actions and isEq( v s ′′ , V s ′ )
then 16: if isEq( e .get_value( s ′′ ), V s ′ ) then 17: meaningful _ actions .remove
( a ) 18: else if a not in meaningful _ actions and not isEq( v s ′′ , V s ′ )
then 19: if not isEq( e .get_value( s ′′ ), V s ′ ) then 20: meaningful _ actions .add
( a ) 21: if a not in meaningful _ actions then 22: if a.c == s i .c
then 23: ret [0].add(a) 24: else 25: ret [1].add(a) 26: return ret
D The Input of PV-NN
The inputs of PV-NN are not independent.
Take AlphaGo Zero as an example, given a state s t at stept , it will generate 17 feature planes as the input of PV-NN.
Each feature plane is a 19 × 19 binary 2Darray that includes the information the latest eight states.
For example, there are eight feature planes{ X t , X t − 1 , . . .
, X t − 7 } indicates the presence of the current player’s stones of s t , s t − 1 , . . .
, s t − 7 .
Ifposition p
=
( i, j ) has current player’s stone at time t , then X t [ i ][ j ] = 1 else X t [ i ][ j ] = 0 .
Sinceeach player can only place on stone at a time, X i will mostly be the same.
Hence, the legal inputsfeature are not independent.
Additionally, most AIs have more complex feature planes as input,including the domain knowledge of Go.
For example, a common feature is "liberty," which meanshow many additional enemy stones are needed to capture the position.
This kind of input is also notindependent to other feature planes.
E Territory
Starting from an empty board, one player places a stone on a vacant part of the board to surroundmore territory or defend our territory from being “captured" by the opponent’s stones.
It is critical toselect actions that can gain more territory.
Normally, once a position belongs to a color, it is hard tochange it.
Hence, putting a stone in any color’s territory is usually wasting a turn.
Since territory is so important in Go, many agents’ PV-NNs ([13]) has an additional output topredict the territory.
Given a state s , the territory output terr ( s ) is a vector of scalar.
Each elementof the vector − 1 ≤ terr ( s )
[ i ] ≤ 1 shows how a position i is likely to belong to.
For example,terr ( s )[ i ] < = − 0 .
8 means that position i is likely belongs to the color white and terr ( s )
[ i ] > = 0 .
8means position i is likely belongs to the color black.
In our experiment, we define the positions ofmeaningless actions should be one of the players’ territory.
That is, we will not consider an action aas meaningless action if its position p is not one of the player’s territory − 0 .
8 ≤ terr ( s )[ i ] ≤ 0 . 8 .
F Formal Definition of the Skip Function
Function F ( s, a ) : S × A (cid:55)→ S will play the action a on state s without changing the turn player byskipping the opponent’s turn.
Since action’s color a.c might not be the turn color of s , in that case,we need to play action pass a pass first.
Finally, F is formulated as follow:
[Equation]
where s.c, a.c are the color of the state and the action, a pass is the pass action.
G Go Experiments without Territory
In this section
[Table]
we attack KataGo’s
PV-NN without using the territory constraint.
The results areshown in Table 4 and can be compared with Table 5 which is the normal setting with territory.
Without the territory constraint, it is easier for our method to attack the target model.
For example, the2STEP success rate on policy attack become 100% after removing the territory constraint.
Anotherobservation is that without the territory constraint, it is more likely to call the examiner.
This mightbecause the meaningless actions under territory constraint are more stable.
Hence, if v ( s ′ ) is differentform V ( s ) , it is more likely that it is and adversarial example, instead of s ′ is semantically differentform s .
[Table]
=
[Table] [Table]
POLICY η adv =
[Table] [Table]
H Visualized Examples
Fig.
6 provides some examples we found on different agents and different types of games.
Theperturbation actions are marked as 1 and 2.
If it is a policy attack, the best action of both s and s ′ ismark as A and the bad action that the target model want to play is marked as B .
The subcaption ofeach examples first shows the which dataset it is form.
For example, "FOX_12k" means that the gamefrom dataset FOX and was generate from level 12k player.
For another example, "ZZ_8" means thenumber 8 game of AlphaGo Zero self-play.
Second, the subcaption describe the target model.
If it isjust PV-NN, we will present it using the first letter of the agent.
For example, ’K’ for KataGo, ’C’ for
CGI.
On the other hand, if the target model is MCTS, we will add a ’M’ and simulation count afterthe first letter of the model.
For example, "KM25" means KataGo MCTS agent with 25 simulation.
Finally, the third component of the subcaption show the type of attack and the threshold η adv .
Forexample, "P0.7" means policy attack and η adv = 0 .
A A
B B
C C
D D
E E
F F
G G
H H
J J
K K
L L
M M
N N
O O
P P
Q Q
R R
S S
T T
B A(a) FOX_12k K P0.
A A
B B
C C
D D
E E
F F
G G
H H
J J
K K
L L
M M
N N
O O
P P
Q Q
R R
S S
T T
B
A (b) MH_Dang C P0.
A A
B B
C C
D D
E E
F F
G G
H H
J J
K K
L L
M M
N N
O O
P P
Q Q
R R
S S
T
T (c) ZM_
E V0.
A A
B B
C C
D D
E E
F F
G G
H H
J J
K K
L L
M M
N N
O O
P P
Q Q
R R
S S
T
T (d) FOX_15k K V0.
A A
B B
C C
D D
E E
F F
G G
H H
J J
K K
L L
M M
N N
O O
P P
Q Q
R R
S S
T T
AB(e)
FOX_17K K P0.
A A
B B
C C
D D
E E
F F
G G
H H
J J
K K
L L
M M
N N
O O
P P
Q Q
R R
S S
T
T (f) FOX_18k K V0.
A A
B B
C C
D D
E E
F F
G G
H H
J J
K K
L L
M M
N N
O O
P P
Q Q
R R
S S
T
T(g) ZM_3 E V0.
A A
B B
C C
D D
E E
F F
G G
H H
J J
K K
L L
M M
N N
O O
P P
Q Q
R R
S S
T T
A
B(h) ATV_2 K P0.
A A
B B
C C
D D
E E
F F
G G
H H
J J
K K
L L
M M
N N
O O
P P
Q Q
R R
S S
T
T (i) ATV_3 K V0.
A A
B B
C C
D D
E E
F F
G G
H H
J J
K K
L L
M M
N N
O O
P P
Q Q
R R
S S
T T
B
A (j) ZZ_
KM25 P0.
A A
B B
C C
D D
E E
F F
G G
H H
J J
K K
L L
M M
N N
O O
P P
Q Q
R R
S S
T T
A
B 1(k) LG_12 K P0.
A A
B B
C C
D D
E E
F F
G G
H H
J J
K K
L L
M M
N N
O O
P P
Q Q
R R
S S
T T
A 2 B (l) ZZ_10 LM25 P0.
A A
B B
C C
D D
E E
F F
G G
H H
J J
K K
L L
M M
N N
O O
P P
Q Q
R R
S S
T T
B
A (m) LG_15 K P0.
A A
B B
C C
D D
E E
F F
G G
H H
J J
K K
L L
M M
N N
O O
P P
Q Q
R R
S S
T
T (n) LG_16 K V0.
A A
B B
C C
D D
E E
F F
G G
H H
J J
K K
L L
M M
N N
O O
P P
Q Q
R R
S S
T
T 2 B
A (o) ZZ_17 LM50 P0.
A A
B B
C C
D D
E E
F F
G G
H H
J J
K K
L L
M M
N N
O O
P P
Q Q
R R
S S
T
T 2 A
B (p) MH_KE E P0.
A A
B B
C C
D D
E E
F F
G G
H H
J J
K K
L L
M M
N N
O O
P P
Q Q
R R
S S
T T
B
A (q) MH_LIU C P0.
A A
B B
C C
D D
E E
F F
G G
H H
J J
K K
L L
M M
N N
O O
P P
Q Q
R R
S S
T T
B
A (r) LG_3 K P0.
A A
B B
C C
D D
E E
F F
G G
H H
J J
K K
L L
M M
N N
O O
P P
Q Q
R R
S S
T
T (s) MH_MEN P V0.
A A
B B
C C
D D
E E
F F
G G
H H
J J
K K
L L
M M
N N
O O
P P
Q Q
R R
S S
T
T (t) MH_CHEN
E V0.