Original: E XPLAIN Y OUR M OVE : U NDERSTANDING A GENT A CTIONS U SING F OCUSED F EATURE S ALIENCY
Revised: SARFA: S ALIENT AND R ELEVANT F EATURE A TTRIBUTION

Original: A BSTRACT
Revised: A BSTRACT

Original: As deep reinforcement learning (RL) is applied to more tasks, there is a need to visualize and understand the behavior of learned agents.
Revised: As deep reinforcement learning (RL) is applied to more tasks, there is a need to visualize and understand the behavior of learned agents.

Original: Saliency maps explain agent behavior by highlighting the features of the input state that are most relevant for the agent in taking an action.
Revised: Saliency maps explain agent behavior by highlighting the features of the input state that are most relevant for the agent in taking an action.

Original: Existing perturbation-based approaches to compute saliency often highlight regions of the input that are not relevant to the action taken by the agent.
Revised: Existing perturbation-based approaches to compute saliency often highlight regions of the input that are not relevant to the action taken by the agent.

Original: Our approach generates more focused saliency maps by balancing two aspects (speciﬁcity and relevance) that capture different desiderata of saliency.
Revised: Our approach, SARFA generates more focused saliency maps by balancing two aspects (speciﬁcity and relevance) that capture different desiderata of saliency.

Original: The ﬁrst captures the impact of perturbation on the relative expected reward of the action to be explained.
Revised: The ﬁrst captures the impact of perturbation on the relative expected reward of the action to be explained.

Original: The second downweights irrelevant features that alter the relative expected rewards of actions other than the action to be explained.
Revised: The second downweighs irrelevant features that alter the relative expected rewards of actions other than the action to be explained.

Original: We compare our approach with existing approaches on agents trained to play board games (Chess and Go) and Atari games (Breakout, Pong and Space Invaders).
Revised: We compare SARFA with existing approaches on agents trained to play board games (Chess and Go) and Atari games (Breakout, Pong and Space Invaders).

Original: We show through illustrative examples (Chess, Atari, Go), human studies (Chess), and automated evaluation methods (Chess) that our approach generates saliency maps that are more interpretable for humans than existing approaches.
Revised: We show through illustrative examples (Chess, Atari, Go), human studies (Chess), and automated evaluation methods (Chess) that SARFA generates saliency maps that are more interpretable for humans than existing approaches.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep learning has achieved success in various domains such as image classiﬁcation (He et al., 2016;
Revised: Deep learning has achieved success in various domains such as image classiﬁcation (He et al., 2016;

Original: Krizhevsky et al., 2012), machine translation (Mikolov et al., 2010), image captioning (Karpathy et al., 2015), and deep Reinforcement Learning (RL) (Mnih et al., 2015; Silver et al., 2017).
Revised: Krizhevsky et al., 2012), machine translation (Mikolov et al., 2010), image captioning (Karpathy et al., 2015), and deep Reinforcement Learning (RL) (Mnih et al., 2015; Silver et al., 2017).

Original: To explain and interpret the predictions made by these complex, ”black-box”-like systems, various gradient and perturbation techniques have been introduced for image classiﬁcation (Simonyan et al., 2013;
Revised: To explain and interpret the predictions made by these complex, “black-box”-like systems, various gradient and perturbation techniques have been introduced for image classiﬁcation (Simonyan et al., 2013;

Original: Zeiler & Fergus, 2014; Fong & Vedaldi, 2017) and deep sequential models (Karpathy et al., 2015).
Revised: Zeiler & Fergus, 2014; Fong & Vedaldi, 2017) and deep sequential models (Karpathy et al., 2015).

Original: However, interpretability for RL-based agents has received signiﬁcantly less attention.
Revised: However, interpretability for RL-based agents has received signiﬁcantly less attention.

Original: Interpreting the strategies learned by RL agents can help users better understand the problem that the agent is trained to solve.
Revised: Interpreting the strategies learned by RL agents can help users better understand the problem that the agent is trained to solve.

Original: For instance, interpreting the actions of a chess-playing agent in a position could provide useful information about aspects of the position.
Revised: For instance, interpreting the actions of a chess-playing agent in a position could provide useful information about aspects of the position.

Original: Interpretation of RL agents is also an important step before deploying such models to solve real-world problems.
Revised: Interpretation of RL agents is also an important step before deploying such models to solve real-world problems.

Original: Inspired by the popularity and use of saliency maps to interpret in computer vision, a number of existing approaches have proposed similar methods for reinforcement learning-based agents.
Revised: Inspired by the popularity and use of saliency maps to interpret in computer vision, a number of existing approaches have proposed similar methods for reinforcement learning-based agents.

Original: Greydanus et al.
Revised: Greydanus et al.

Original: (2018) derive saliency maps that explain RL agent behavior by applying a Gaussian blur to different parts of the input image.
Revised: (2018) derive saliency maps that explain RL agent behavior by applying a Gaussian blur to different parts of the input image.

Original: They generate saliency maps using differences in the value function and policy vector between the original and perturbed state.
Revised: They generate saliency maps using differences in the value function and policy vector between the original and perturbed state.

Original: They achieve promising results on agents trained to play Atari games.
Revised: They achieve promising results on agents trained to play Atari games.

Original: Iyer et al.
Revised: Iyer et al.

Original: (2018) compute saliency maps using a difference in the action-value ( Q ( s, a ) ) between the original and perturbed state. 
Revised: (2018) compute saliency maps using a difference in the action-value ( Q ( s, a ) ) between the original and perturbed state. [Figure]

Original: There are two primary limitations to these approaches.
Revised: There are two primary limitations to these approaches.

Original: The ﬁrst is that they highlight features whose perturbation affects actions apart from the one we are explaining.
Revised: The ﬁrst is that they highlight features whose perturbation affects actions apart from the one we are explaining.

Original: This is illustrated in Figure 1, which shows a chess position (it is white’s turn).
Revised: This is illustrated in Figure 1, which shows a chess position (it is white’s turn).

Original: Stockﬁsh 1 plays the move Bb6 in this position, which traps the black rook (a5) and queen (c7) 2 .
Revised: Stockﬁsh 1 plays the move Bb6 in this position, which traps the black rook (a5) and queen (c7) 2 .

Original: The knight protects the white bishop on a4, and hence the move works.
Revised: The knight protects the white bishop on a4, and hence the move works.

Original: In this position, if we consider the saliency of the white queen (square d1), then it is apparent that the queen is not involved in the tactic and hence the saliency should be low.
Revised: In this position, if we consider the saliency of the white queen (square d1), then it is apparent that the queen is not involved in the tactic and hence the saliency should be low.

Original: However, perturbing the state (by removing the queen) leads to a state with substantially different values for Q ( s, a ) and
Revised: However, perturbing the state (by removing the queen) leads to a state with substantially different values for Q ( s, a ) and

Original: V ( s ) .
Revised: V ( s ) .

Original: Therefore, existing approaches (Greydanus et al., 2018; Iyer et al., 2018) mark the queen as salient.
Revised: Therefore, existing approaches (Greydanus et al., 2018; Iyer et al., 2018) mark the queen as salient.

Original: The second limitation is that they highlight features that are not relevant to the action to be explained.
Revised: The second limitation is that they highlight features that are not relevant to the action to be explained.

Original: In Figure 1c, perturbing the state by removing the black pawn on c6 alters the expected reward for actions other than the one to be explained.
Revised: In Figure 1c, perturbing the state by removing the black pawn on c6 alters the expected reward for actions other than the one to be explained.

Original: Therefore, it alters the policy vector and is marked salient.
Revised: Therefore, it alters the policy vector and is marked salient.

Original: However, the pawn is not relevant to explain the move played in the position (Bb6).
Revised: However, the pawn is not relevant to explain the move played in the position (Bb6).

Original: In this work, we propose  a perturbation based approach for generating saliency maps for black-box agents that builds on two desired properties of action-focused saliency.
Revised: In this work we propose SARFA, a perturbation based approach for generating saliency maps for black-box agents that builds on two desired properties of action-focused saliency.

Original: The ﬁrst, speciﬁcity , captures the impact of perturbation only on the Q-value of the action to be explained.
Revised: The ﬁrst, speciﬁcity , captures the impact of perturbation only on the Q-value of the action to be explained.

Original: In the above example, this term downweights features such as the white queen that impact the expected reward of all actions equally.
Revised: In the above example, this term downweighs features such as the white queen that impact the expected reward of all actions equally.

Original: The second, relevance , downweights irrelevant features that alter the expected rewards of actions other than the action to be explained.
Revised: The second, relevance , downweighs irrelevant features that alter the expected rewards of actions other than the action to be explained.

Original: It removes features such as the black pawn on c6 that increase the expected reward of other actions (in this case, Bb4).
Revised: It removes features such as the black pawn on c6 that increase the expected reward of other actions (in this case, Bb4).

Original: By combining these aspects, we generate a saliency map that highlights features of the input state that are relevant for the action to be explained.
Revised: By combining these aspects, we generate a saliency map that highlights features of the input state that are relevant for the action to be explained.

Original: Figure 1 illustrates how the saliency map generated by our approach only highlights pieces relevant to the move, unlike existing approaches.
Revised: Figure 1 illustrates how the saliency map generated by SARFA only highlights pieces relevant to the move, unlike existing approaches.

Original: We use our approach to explain the actions taken by agents for board games (Chess and Go), and for Atari games (Breakout, Pong and Space Invaders).
Revised: We use our approach, SARFA to explain the actions taken by agents for board games (Chess and Go), and for Atari games (Breakout, Pong and Space Invaders).

Original: Using a number of illustrative examples, we show that our proposed approach obtains more focused and accurate interpretations for all of these setups when compared to Greydanus et al.
Revised: Using a number of illustrative examples, we show that SARFA obtains more focused and accurate interpretations for all of these setups when compared to Greydanus et al.

Original: (2018) and Iyer et al.
Revised: (2018) and Iyer et al.

Original: We also demonstrate that our approach is more effective in identifying important pieces in chess puzzles, and further, in aiding skilled chess players to solve chess puzzles (improves accuracy of solving them by nearly 25% and reduces the time taken by 31% over existing approaches).
Revised: We also demonstrate that SARFA is more effective in identifying important pieces in chess puzzles, and further, in aiding skilled chess players to solve chess puzzles (improves accuracy of solving them by nearly 25% and reduces the time taken by 31% over existing approaches).

Original:  A PPROACH
Revised:  A PPROACH

Original: We are given an agent M , operating on a state space S , with the set of actions
Revised: We are given an agent M , operating on a state space S , with the set of actions

Original: A s for s ∈ S , and a
Revised: A s for s ∈ S , and a

Original: Q -value function denoted as Q ( s, a ) for s ∈ S , a ∈ A s .
Revised: Q -value function denoted as Q ( s, a ) for s ∈ S , a ∈ A s .

Original: Following a greedy policy, let the action that was selected by the agent at state s be ˆ a , i.e. ˆ a = arg max a Q ( s, a ) .
Revised: Following a greedy policy, let the action that was selected by the agent at state s be ˆ a , i.e. ˆ a = arg max a Q ( s, a ) .

Original: The states are parameterized in terms of state-features F .
Revised: The states are parameterized in terms of state-features F .

Original: For instance, in a board game such as chess, the features are the 64 squares.
Revised: For instance, in a board game such as chess, the features are the 64 squares.

Original: For Atari games, the features are pixels.
Revised: For Atari games, the features are pixels.

Original: We are interested in identifying which features of the state s are important for the agent in taking action ˆ a .
Revised: We are interested in identifying which features of the state s are important for the agent in taking action ˆ a .

Original: We assume that the agent is in the exploitation phase[Figure] and therefore plays the action with the highest expected reward.
Revised: We assume that the agent is in the exploitation phase and therefore plays the action with the highest expected reward.

Original: This feature importance is described by an importance-score or saliency for each feature f , denoted by S , where S [ f ] ∈ (0 , 1) denotes the saliency of the f th feature of s for the agent taking action ˆ a .
Revised: This feature importance is described by an importance-score or saliency for each feature f , denoted by S , where S [ f ] ∈ (0 , 1) denotes the saliency of the f th feature of s for the agent taking action ˆ a .

Original: A higher value indicates that the f th feature of s is more important for the agent when taking action ˆ a .
Revised: A higher value indicates that the f th feature of s is more important for the agent when taking action ˆ a .

Original: Perturbation-based Saliency Maps The general outline of perturbation based saliency approaches is as follows.
Revised: Perturbation-based Saliency Maps The general outline of perturbation based saliency approaches is as follows.

Original: For each feature f , ﬁrst perturb s to get s (cid:48) .
Revised: For each feature f , ﬁrst perturb s to get s (cid:48) .

Original: For instance, in chess, we can perturb the board position by removing the piece in the f th square.
Revised: For instance, in chess, we can perturb the board position by removing the piece in the f th square.

Original: In Atari, Greydanus et al.
Revised: In Atari, Greydanus et al.

Original: (2018) perturb the input image by adding a Gaussian blur centered on the f th pixel.
Revised: (2018) perturb the input image by adding a Gaussian blur centered on the f th pixel.

Original: Second, query M to get Q ( s (cid:48) , a ) ∀ a ∈
Revised: Second, query M to get Q ( s (cid:48) , a ) ∀ a ∈

Original: A s ∩ A s’ .
Revised: A s ∩ A s’ .

Original: We take the intersection of A s and A s (cid:48) to represent the case where some actions may be legal in s but not in s (cid:48) and vice versa.
Revised: We take the intersection of A s and A s (cid:48) to represent the case where some actions may be legal in s but not in s (cid:48) and vice versa.

Original: For instance, when we remove a piece in chess, actions that were legal earlier may not be legal anymore.
Revised: For instance, when we remove a piece in chess, actions that were legal earlier may not be legal anymore.

Original: In the rest of this section, when we use “all actions” we mean all actions that are legal in both the states s and s (cid:48) .
Revised: In the rest of this section, when we use “all actions” we mean all actions that are legal in both the states s and s (cid:48) .

Original: Finally, compute S [ f ] based on how different
Revised: Finally, compute S [ f ] based on how different

Original: Q ( s, a ) and Q ( s (cid:48) , a )) are, i.e. intuitively, S
Revised: Q ( s, a ) and Q ( s (cid:48) , a )) are, i.e. intuitively, S

Original: [ f ] should be higher if Q ( s (cid:48) , a ) is signiﬁcantly different from Q ( s, a ) .
Revised: [ f ] should be higher if Q ( s (cid:48) , a ) is signiﬁcantly different from Q ( s, a ) .

Original: Greydanus et al.
Revised: Greydanus et al.

Original: (2018) compute the saliency map using S 1 [ f ] = 

Revised: (2018) compute the saliency map using S 1 [ f ] =

Original: | π s − π s (cid:48) | 2 , and
Revised: | π s − π s (cid:48) | 2 , and

Original: S 2 [ f ] = 12 ( V ( s ) − V
Revised: S 2 [ f ] = 12 ( V ( s ) − V

Original: s (cid:48) ))
Revised: s (cid:48) ))

Original: 2 , while Iyer et al.
Revised: 2 , while Iyer et al.

Original: (2018) use S [ f ] =
Revised: (2018) use S [ f ] =

Original: Q ( s, ˆ a ) − Q ( s (cid:48) , ˆ a ) .
Revised: Q ( s, ˆ a ) − Q ( s (cid:48) , ˆ a ) .

Original: In this work, we will propose an alternative approach to compute S [ f ] .
Revised: In this work, we will propose an alternative approach to compute S [ f ] .

Original: Properties We deﬁne two desired properties of an accurate saliency map for policy-based agents:
Revised: Properties We deﬁne two desired properties of an accurate saliency map for policy-based agents:

Original: Speciﬁcity: Saliency S [ f ] should focus on the effect of the perturbation speciﬁcally on the action being explained, ˆ a , i e. it should be high if perturbing the f th feature of the state reduces the relative expected reward of the selected action.
Revised: Speciﬁcity: Saliency S [ f ] should focus on the effect of the perturbation speciﬁcally on the action being explained, ˆ a , i.e. it should be high if perturbing the f th feature of the state reduces the relative expected reward of the selected action.

Original: Stated another way, S [ f ] should be high if Q ( s, ˆ a ) − Q ( s (cid:48) , ˆ a ) is substantially higher than Q ( s, a ) − Q ( s (cid:48) , a ) , a (cid:54)
Revised: Stated another way, S [ f ] should be high if Q ( s, ˆ a ) − Q ( s (cid:48) , ˆ a ) is substantially higher than Q ( s, a ) − Q ( s (cid:48) , a ) , a (cid:54)

Original: =
Revised: =

Original: ˆ a .
Revised: ˆ a .

Original: For instance, in ﬁgure 1, removing pieces such as the white queen impact all actions uniformly ( Q ( s, a ) − Q ( s (cid:48) , a ) is roughly equal for all actions).
Revised: For instance, in ﬁgure 1, removing pieces such as the white queen impact all actions uniformly ( Q ( s, a ) − Q ( s (cid:48) , a ) is roughly equal for all actions).

Original: Therefore, such pieces should not be salient for explaining ˆ a .
Revised: Therefore, such pieces should not be salient for explaining ˆ a .

Original: On the other hand, removing pieces such as the white knight on a4 speciﬁcally impacts the move ( ˆ a = Bb6) we are trying to explain ( Q ( s, Bb 6) − Q ( s (cid:48) , Bb 6) (cid:29) Q ( s, a ) − Q ( s (cid:48) , a ) for other actions a ).
Revised: On the other hand, removing pieces such as the white knight on a4 speciﬁcally impacts the move ( ˆ a = Bb6) we are trying to explain ( Q ( s, Bb 6) − Q ( s (cid:48) , Bb 6) (cid:29) Q ( s, a ) − Q ( s (cid:48) , a ) for other actions a ).

Original: Therefore, such pieces should be salient for ˆ a . 2.
Revised: Therefore, such pieces should be salient for ˆ a . 2.

Original: Relevance: Since the Q -values represent the expected returns, two states s and s (cid:48) can have substantially different Q -values for all actions, i e. may be higher for s (cid:48) for all actions if s (cid:48) is a better state.
Revised: Relevance: Since the Q -values represent the expected returns, two states s and s (cid:48) can have substantially different Q -values for all actions, i.e. may be higher for s (cid:48) for all actions if s (cid:48) is a better state.

Original: Saliency map for a speciﬁc action ˆ a in s should thus ignore such differences, i e. s (cid:48) should contribute to the saliency only if its effects are relevant to ˆ a .
Revised: Saliency map for a speciﬁc action ˆ a in s should thus ignore such differences, i.e. s (cid:48) should contribute to the saliency only if its effects are relevant to ˆ a .

Original: In other words, S [ f ] should be low if perturbing the f th feature of the state alters the expected rewards of actions other than ˆ a . 
Revised: In other words, S [ f ] should be low if perturbing the f th feature of the state alters the expected rewards of actions other than ˆ a . For instance , in

Original: For instance, in Figure 1, removing the black pawn on c6 increases the expected rewardof other actions (in this case, Bb4).
Revised: Figure , removing the black pawn on c increases the expected reward

Revised: of

Revised: other actions (in this case, Bb

Original: However, it does not effect the expected reward of the action to be explained (Bb6).
Revised: However, it does not effect the expected reward of the action to be explained (Bb6).

Original: Therefore, the pawn is not salient for explaining the move.
Revised: Therefore, the pawn is not salient for explaining the move.

Original: In general, such features that are irrelevant to ˆ a should not be salient.
Revised: In general, such features that are irrelevant to ˆ a should not be salient.

Original: Existing approaches to saliency maps do not capture these properties in how they compute the saliency.
Revised: Existing approaches to saliency maps do not capture these properties in how they compute the saliency.

Original: Both the saliency approaches used in Greydanus et al.
Revised: Both the saliency approaches used in Greydanus et al.

Original: (2018), i.e. S 1 [ f ] = 12 ( V ( s ) − V ( s (cid:48) )) 2 and
Revised: (2018), i.e. S 1 [ f ] = 12 ( V ( s ) − V ( s (cid:48) )) 2 and

Original: S 2 [ f ] = 12 | π s − π s (cid:48) | 2 , are not focusing on the action-speciﬁc effects since they aggregate the change over all actions.
Revised: S 2 [ f ] = 12 | π s − π s (cid:48) | 2 , are not focusing on the action-speciﬁc effects since they aggregate the change over all actions.

Original: Although the saliency computation in Iyer et al.
Revised: Although the saliency computation in Iyer et al.

Original: (2018) is somewhat more speciﬁc to the action, i.e. S [ f ] = Q ( s, ˆ a ) − Q ( s (cid:48) , ˆ a ) , it is ignoring whether the effects on Q are relevant only to ˆ a , or effect all the other actions as well.
Revised: (2018) is somewhat more speciﬁc to the action, i.e. S [ f ] = Q ( s, ˆ a ) − Q ( s (cid:48) , ˆ a ) , it is ignoring whether the effects on Q are relevant only to ˆ a , or effect all the other actions as well.

Original: This is illustrated in Figure 1.
Revised: This is illustrated in Figure 1.

Original: Identifying Speciﬁc Changes To focus on the effect of the change on the action, we are interested in whether the relative returns of ˆ a change with the perturbation.
Revised: Identifying Speciﬁc Changes To focus on the effect of the change on the action, we are interested in whether the relative returns of ˆ a change with the perturbation.

Original: Using Q ( s, ˆ a ) directly, as in Iyer et al.
Revised: Using Q ( s, ˆ a ) directly, as in Iyer et al.

Original: (2018), does not capture the relative changes to Q ( s, a ) for other actions.
Revised: (2018), does not capture the relative changes to Q ( s, a ) for other actions.

Original: To support speciﬁcity, we use the softmax over Q-values to normalize the values (as is also used in softmax action selection):
Revised: To support speciﬁcity, we use the softmax over Q-values to normalize the values (as is also used in softmax action selection):

Original: [Equation]
Revised: [Equation]

Original: and compute ∆ p
Revised: and compute ∆ p

Original: = P ( s, ˆ a ) − P ( s (cid:48) , ˆ a ) , the difference in the relative expected reward of the action to be explained between the original and the perturbed state.
Revised: = P ( s, ˆ a ) − P ( s (cid:48) , ˆ a ) , the difference in the relative expected reward of the action to be explained between the original and the perturbed state.

Original: A high value of ∆ p thus implies that the feature is important for the speciﬁc choice of action ˆ a by the agent, while a low value indicates that the effect is not speciﬁc to the action.
Revised: A high value of ∆ p thus implies that the feature is important for the speciﬁc choice of action ˆ a by the agent, while a low value indicates that the effect is not speciﬁc to the action.

Original: Identifying Relevant Changes Apart from focusing on the change in Q ( s, ˆ a ) , we also want to ensure that the perturbation leads to minimal effect on the relative expected returns for other actions.
Revised: Identifying Relevant Changes Apart from focusing on the change in Q ( s, ˆ a ) , we also want to ensure that the perturbation leads to minimal effect on the relative expected returns for other actions.

Original: To capture this intuition, we will compute the relative returns of all other actions, and compute saliency in proportion to their similarity.
Revised: To capture this intuition, we will compute the relative returns of all other actions, and compute saliency in proportion to their similarity.

Original: Speciﬁcally, we normalize the Q-values using a softmax apart from the selected action ˆ a .
Revised: Speciﬁcally, we normalize the Q-values using a softmax apart from the selected action ˆ a .

Original: [Equation]
Revised: [Equation]

Original: We use the KL-Divergence D KL =
Revised: We use the KL-Divergence D KL =

Original: P rem ( s (cid:48) , a ) ||
Revised: P rem ( s (cid:48) , a ) ||

Original: P rem ( s, a ) to measure the difference between
Revised: P rem ( s, a ) to measure the difference between

Original: P rem ( s (cid:48) , a ) and P rem ( s, a ) .
Revised: P rem ( s (cid:48) , a ) and P rem ( s, a ) .

Original: A high D KL indicates that the relative expected reward of taking some actions (other than the original action) changes signiﬁcantly between s and s (cid:48) .
Revised: A high D KL indicates that the relative expected reward of taking some actions (other than the original action) changes signiﬁcantly between s and s (cid:48) .

Original: In other words, a high
Revised: In other words, a high

Original: D KL indicates that the effect of the feature is spread over other actions, i.e. the feature may not be relevant for the selected action ˆ a .
Revised: D KL indicates that the effect of the feature is spread over other actions, i.e. the feature may not be relevant for the selected action ˆ a .

Original: Computing the Saliency To compute salience S [ f ] , we need to combine ∆ p and D KL .
Revised: Computing the Saliency To compute salience S [ f ] , we need to combine ∆ p and D KL .

Original: If D KL is high, S [ f ] should be low, regardless of whether ∆ p is high; the perturbation is affecting many other actions.
Revised: If D KL is high, S [ f ] should be low, regardless of whether ∆ p is high; the perturbation is affecting many other actions.

Original: Conversely, when D KL is low, S [ f ] should depend on ∆ p .
Revised: Conversely, when D KL is low, S [ f ] should depend on ∆ p .

Original: To be able to compare these properties on a similar scale, we deﬁne a normalized measure of distribution similarity K using
Revised: To be able to compare these properties on a similar scale, we deﬁne a normalized measure of distribution similarity K using

Original: D KL :
Revised: D KL :

Original: [Equation]
Revised: [Equation]

Original: As D KL goes from 0 to ∞ , K goes from 1 to 0 .
Revised: As D KL goes from 0 to ∞ , K goes from 1 to 0 .

Original: Thus, S [ f ] should be low if either ∆ p is low or K is low.
Revised: Thus, S [ f ] should be low if either ∆ p is low or K is low.

Original: Harmonic mean provides this desired effect in a robust, smooth manner, and therefore we deﬁne S [ f ] to be the harmonic mean of ∆ p and K :
Revised: Harmonic mean provides this desired effect in a robust, smooth manner, and therefore we deﬁne S [ f ] to be the harmonic mean of ∆ p and K :

Original: [Equation]
Revised: [Equation]

Original: Equation 4 captures our desired properties of saliency maps.
Revised: Equation 4 captures our desired properties of saliency maps.

Original: If perturbing the f th feature affects the expected rewards of all actions uniformly, then ∆ p is low and subsequently S [ f ] is low.
Revised: If perturbing the f th feature affects the expected rewards of all actions uniformly, then ∆ p is low and subsequently S [ f ] is low.

Original: This low value of ∆ p captures the property of speciﬁcity deﬁned above.
Revised: This low value of ∆ p captures the property of speciﬁcity deﬁned above.

Original: If perturbing the f th feature of the state affects the rewards of some actions other than the action to be explained, then D KL is high, K is low, and S [ f ] is low.
Revised: If perturbing the f th feature of the state affects the rewards of some actions other than the action to be explained, then D KL is high, K is low, and S [ f ] is low.

Original: This low value of K captures the property of relevance deﬁned above.
Revised: This low value of K captures the property of relevance deﬁned above.

Original:  R ESULTS
Revised:  R ESULTS

Original: To show that our approach produces more meaningful saliency maps than existing approaches, we use sample positions from Chess, Atari (Breakout, Pong and Space Invaders) and Go (Section 3.1).
Revised: To show that SARFA produces more meaningful saliency maps than existing approaches, we use sample positions from Chess, Atari (Breakout, Pong and Space Invaders) and Go (Section 3.1).

Original: To show that our approach generates saliency maps that provide useful information to humans, we conduct human studies on problem-solving for chess puzzles (Section 3.2).
Revised: To show that SARFA generates saliency maps that provide useful information to humans, we conduct human studies on problem-solving for chess puzzles (Section 3.2).

Original: To automatically compare the saliency maps generated by different perturbation-based approaches, we introduce a Chess saliency dataset (Section 3.3).
Revised: To automatically compare the saliency maps generated by different perturbation-based approaches, we introduce a Chess saliency dataset (Section 3.3).

Original: We use the dataset to show how our approach is better than existing approaches in identifying chess pieces that humans deem relevant in several positions.
Revised: We use the dataset to show how SARFA is better than existing approaches in identifying chess pieces that humans deem relevant in several positions.

Original: In Section 3.4, we show how our approach can be used to understand common tactical ideas in chess by interpreting the action of a trained agent.[Figure]
Revised: In Section 3.4, we show how SARFA can be used to understand common tactical ideas in chess by interpreting the action of a trained agent.

Original: To show that our approach works for black-box agents, regardless of whether they are trained using reinforcement learning, we use a variety of agents.
Revised: To show that SARFA works for black-box agents, regardless of whether they are trained using reinforcement learning, we use a variety of agents.

Original: We only assume access to the agent’s Q ( s, a ) function for all experiments.
Revised: We only assume access to the agent’s Q ( s, a ) function for all experiments.

Original: For experiments on chess, we use the Stockﬁsh agent 3 .
Revised: For experiments on chess, we use the Stockﬁsh agent 3 .

Original: For experiments on Go, we use the pre-trained MiniGo RL agent 4 .
Revised: For experiments on Go, we use the pre-trained MiniGo RL agent 4 .

Original: For experiments on Atari agents and for generating saliency maps for Greydanus et al.
Revised: For experiments on Atari agents and for generating saliency maps for Greydanus et al.

Original: (2018), we use their code and pre-trained RL agents 5 .
Revised: (2018), we use their code and pre-trained RL agents 5 .

Original: For generating saliency maps using Iyer et al.
Revised: For generating saliency maps using Iyer et al.

Original: (2018), we use our own implementation 6 .
Revised: (2018), we use our own implementation 6 .

Original: All of our code and more detailed results are available in our Github repository 6 . 
Revised: All of our code and more detailed results are available in our Github repository 6 . [Figure] [Figure]

Original: 3. I LLUSTRATIVE E XAMPLES
Revised: 3. I LLUSTRATIVE E XAMPLES

Original: In this section, we provide examples of generated saliency maps to highlight the qualitative differences between our approach that is action-focused and existing approaches that are not.
Revised: In this section, we provide examples of generated saliency maps to highlight the qualitative differences between SARFA that is action-focused and existing approaches that are not.

Original: Chess Figure 1 shows sample positions where our approach produces more meaningful saliency maps than existing approaches for a chess-playing agent (Stockﬁsh).
Revised: Chess Figure 1 shows sample positions where SARFA produces more meaningful saliency maps than existing approaches for a chess-playing agent (Stockﬁsh).

Original: Greydanus et al.
Revised: Greydanus et al.

Original:  and Iyer et al.
Revised: (2018) and Iyer et al.

Original: (2018) generate saliency maps that highlight pieces that are not relevant to the move played by the agent.
Revised: (2018) generate saliency maps that highlight pieces that are not relevant to the move played by the agent.

Original: This is because they use differences in Q ( s, a ) , V ( s ) or the the L 2 norm of the policy vector between the original and perturbed state to calculate the saliency maps.
Revised: This is because they use differences in Q ( s, a ) , V ( s ) or the the L 2 norm of the policy vector between the original and perturbed state to calculate the saliency maps.

Original: Therefore, pieces such as the white queen that affect the value estimate of the state are marked salient.
Revised: Therefore, pieces such as the white queen that affect the value estimate of the state are marked salient.

Original: In contrast, the saliency map generated by our approach only highlights pieces relevant to the move.
Revised: In contrast, the saliency map generated by SARFA only highlights pieces relevant to the move.

Original: Atari To show that our approach generates saliency maps that are more focused than those generated by Greydanus et al.
Revised: Atari To show that SARFA generates saliency maps that are more focused than those generated by Greydanus et al.

Original: (2018), we compare the approaches on three Atari games: Breakout, Pong, and
Revised: (2018), we compare the approaches on three Atari games: Breakout, Pong, and

Original: Space Invaders.
Revised: Space Invaders.

Original: Figures 2, 3, and 4 shows the results.
Revised: Figures 2, 3, and 4 shows the results.

Original: Our approach highlights regions of the input image more precisely, while the Greydanus et al.
Revised: SARFA highlights regions of the input image more precisely, while the Greydanus et al.

Original: (2018) approach highlights several regions of the input image that are not relevant to explain the action taken by the agent.
Revised: (2018) approach highlights several regions of the input image that are not relevant to explain the action taken by the agent.

Original: Go Figure 5 shows a board position in Go.
Revised: Go Figure 5 shows a board position in Go.

Original: It is black’s turn.
Revised: It is black’s turn.

Original: The four white stones threaten the three black stones that are in one row at the top left corner of the board.
Revised: The four white stones threaten the three black stones that are in one row at the top left corner of the board.

Original: To save those three black stones, black looks at the three white stones that are directly below the three black ones.
Revised: To save those three black stones, black looks at the three white stones that are directly below the three black ones.

Original: Due to another white stone below the three white stones, the continuous row of three white stones cannot be captured easily.
Revised: Due to another white stone below the three white stones, the continuous row of three white stones cannot be captured easily.

Original: Therefore black moves to place a black stone below that single white stone in an attempt to start capturing the four white stones.
Revised: Therefore black moves to place a black stone below that single white stone in an attempt to start capturing the four white stones.

Original: It takes the next few turns to surround the structure of four white stones with black ones, thereby saving its pieces.
Revised: It takes the next few turns to surround the structure of four white stones with black ones, thereby saving its pieces.

Original: The method described in Greydanus et al.
Revised: The method described in Greydanus et al.

Original: (2018) generates a saliency map that highlights almost all the pieces on the board.
Revised: (2018) generates a saliency map that highlights almost all the pieces on the board.

Original: Therefore, it reveals little about the pieces that the agent thinks are important.
Revised: Therefore, it reveals little about the pieces that the agent thinks are important.

Original: On the other hand, the map produced by Iyer et al.
Revised: On the other hand, the map produced by Iyer et al.

Original: (2018) highlights only a few pieces.
Revised: (2018) highlights only a few pieces.

Original: The saliency map generated by our approach correctly highlights the structure of four white stones and the black stones already present around them that may be involved in capturing them.
Revised: The saliency map generated by SARFA correctly highlights the structure of four white stones and the black stones already present around them that may be involved in capturing them.

Original: [Figure]
Revised: [Figure]

Original: [Figure][Figure] [Table] [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: To show that our approach generates saliency maps that provide useful information to humans, we conduct human studies on problem-solving for chess puzzles.
Revised: To show that SARFA generates saliency maps that provide useful information to humans, we conduct human studies on problem-solving for chess puzzles.

Original: We show ﬁfteen chess players (ELO 1600-2000) ten chess puzzles from https://www.chess.com (average difﬁculty ELO 1800).
Revised: We show forty chess players (ELO 1600-2000) ﬁfteen chess puzzles from https://www.chess.com (average difﬁculty ELO 1800).

Original: For each puzzle, we show either the puzzle without a saliency map, or the puzzle with a saliency map generated by our approach, Greydanus et al.
Revised: For each puzzle, we show either the puzzle without a saliency map, or the puzzle with a saliency map generated by SARFA, Greydanus et al.

Original: (2018), or Iyer et al.
Revised: (2018), or Iyer et al.

Original: The player is then asked to solve the puzzle.
Revised: The player is then asked to solve the puzzle.

Original: We measure the accuracy (number of puzzles correctly solved) and the average time taken to solve the puzzle, shown in Table 1.
Revised: We measure the accuracy (number of puzzles correctly solved) and the average time taken to solve the puzzle, shown in Table 1.

Original: The saliency maps generated by our approach are more helpful for humans when solving puzzles than those generated by other approaches.
Revised: The saliency maps generated by SARFA are more helpful for humans when solving puzzles than those generated by other approaches.

Original: We observed that the saliency maps generated by Greydanus et al.
Revised: We observed that the saliency maps generated by Greydanus et al.

Original: (2018) often confuse humans, because they highlight several pieces unrelated to the tactic.
Revised: (2018) often confuse humans, because they highlight several pieces unrelated to the tactic.

Original: The maps generated by Iyer et al.
Revised: The maps generated by Iyer et al.

Original: (2018) highlight few pieces and therefore are marginally better than showing no saliency maps for solving puzzles.
Revised: (2018) highlight few pieces and therefore are marginally better than showing no saliency maps for solving puzzles.

Original: 3. C HESS S ALIENCY D ATASET
Revised: 3. C HESS S ALIENCY D ATASET

Original: To automatically compare the saliency maps generated by different perturbation-based approaches, we introduce a Chess saliency dataset.
Revised: To automatically compare the saliency maps generated by different perturbation-based approaches, we introduce a Chess saliency dataset.

Original: The dataset consists of 100 chess puzzles 7 .
Revised: The dataset consists of 100 chess puzzles 7 .

Original: Each puzzle has a single correct move.
Revised: Each puzzle has a single correct move.

Original: For each puzzle, we ask three human experts (ELO > 2200) to mark the pieces that are important for playing the correct move.
Revised: For each puzzle, we ask three human experts (ELO > 2200) to mark the pieces that are important for playing the correct move.

Original: We take a majority vote of the three experts to obtain a list of pieces that are important for the move played in the position.
Revised: We take a majority vote of the three experts to obtain a list of pieces that are important for the move played in the position.

Original: The complete dataset is available in our Github repository 7 .
Revised: The complete dataset is available in our Github repository 7 .

Original: We use this dataset to compare our approach to existing approaches (Greydanus et al., 2018; Iyer et al., 2018).
Revised: We use this dataset to compare SARFA to existing approaches [Table]

Original: Each approach generates a list of squares and a score that indicates how salient the piece on the square is for a particular move. We scale the scores between 0 and 1 to generate ROC curves. Figure 6a shows the results. Our approach generates saliency maps that are better than existing approaches at identifying chess pieces that humans deem relevant in certain positions.
Revised: [Figure]

Original: To evaluate the relative importance of the two components in our saliency computation ( S [ f ] ;
Revised: To evaluate the relative importance of the two components in our saliency computation ( S [ f ] ;

Original: Equation 4), we compute saliency maps and ROC curves using each component individually, i.e.
Revised: Equation 4), we compute saliency maps and ROC curves using each component individually, i.e.

Original: S [ f ] = ∆ p or S [ f ] = K , and compare harmonic mean to other ways to combine them, i.e. using the average, geometric mean, and minimum of ∆ p and K .
Revised: S [ f ] = ∆ p or S [ f ] = K , and compare harmonic mean to other ways to combine them, i.e. using the average, geometric mean, and minimum of ∆ p and K .

Original: Figure 6b shows the results.
Revised: Figure 6b shows the results.

Original: Combination [Figure]
Revised: Combination of the two properties via harmonic mean leads to more accurate saliency maps than alternative approaches.

Original: 3. E XPLAINING T ACTICAL M OTIFS IN C HESS
Revised: 3. E XPLAINING T ACTICAL M OTIFS IN C HESS

Original: In this section, we show how our approach can be used to understand common tactical ideas in chess by interpreting the action of a trained agent.
Revised: In this section, we show how SARFA can be used to understand common tactical ideas in chess by interpreting the action of a trained agent.

Original: Figure 7 illustrates common tactical positions in chess.
Revised: Figure 7 illustrates common tactical positions in chess.

Original: The corresponding saliency maps are generated by interpreting the moves played by the Stockﬁsh agent in these positions.
Revised: The corresponding saliency maps are generated by interpreting the moves played by the Stockﬁsh agent in these positions.

Original: In Figure 7a, it is white to move.
Revised: In Figure 7a, it is white to move.

Original: The surprising Rook x d6 is the move played by Stockﬁsh.
Revised: The surprising Rook x d6 is the move played by Stockﬁsh.

Original: Figure 7d shows the saliency map generated by our approach.
Revised: Figure 7d shows the saliency map generated by SARFA.

Original: The map illustrates the key idea in the position.
Revised: The map illustrates the key idea in the position.

Original: Once black’s rook recaptures white’s rook, white’s bishop pins it to the black king.
Revised: Once black’s rook recaptures white’s rook, white’s bishop pins it to the black king.

Original: Therefore, white can increase the number of attackers on the rook.
Revised: Therefore, white can increase the number of attackers on the rook.

Original: The additional attacker is the pawn on e highlighted by the saliency map.
Revised: The additional attacker is the pawn on e4 highlighted by the saliency map.

Original: In Figure 7b, it is white to move.
Revised: In Figure 7b, it is white to move.

Original: Stockﬁsh plays Queen x h7.
Revised: Stockﬁsh plays Queen x h7.

Original: A queen sacriﬁce!
Revised: A queen sacriﬁce!

Original: Figure 7e shows the saliency map.
Revised: Figure 7e shows the saliency map.

Original: The map highlights the white rook and bishop, along with the queen.
Revised: The map highlights the white rook and bishop, along with the queen.

Original: The key idea is that once black captures the queen with his king (a forced move), then the white rook moves to h with checkmate.
Revised: The key idea is that once black captures the queen with his king (a forced move), then the white rook moves to h with checkmate.

Original: This checkmate is possible because the white bishop guards the important escape square on g6.
Revised: This checkmate is possible because the white bishop guards the important escape square on g6.

Original: The saliency map highlights both pieces.
Revised: The saliency map highlights both pieces.

Original: In Figure 7c, it is black to move.
Revised: In Figure 7c, it is black to move.

Original: Stockﬁsh plays the sacriﬁce rook x d4.
Revised: Stockﬁsh plays the sacriﬁce rook x d4.

Original: The saliency map in
Revised: The saliency map in

Original: Figure 7f illustrates several key aspects of the position.
Revised: Figure 7f illustrates several key aspects of the position.

Original: The black queen and light-colored bishop are threatening mate on g2.
Revised: The black queen and light-colored bishop are threatening mate on g2.

Original: The white queen protects g2.
Revised: The white queen protects g2.

Original: The white rook on a5 is unguarded.
Revised: The white rook on a5 is unguarded.

Original: Therefore, once white recaptures the sacriﬁced rook with the pawn on c3, black can attack both the white rook and queen with the move bishop to b4.
Revised: Therefore, once white recaptures the sacriﬁced rook with the pawn on c3, black can attack both the white rook and queen with the move bishop to b4.

Original: The idea is that the white queen is “overworked” or “overloaded” on d2, having to guard both the g2-pawn and the a5-Rook against black’s double attack. 
Revised: The idea is that the white queen is “overworked” or “overloaded” on d2, having to guard both the g2-pawn and the a5-Rook against black’s double attack. [Figure]

Original: 3. R OBUSTNESS TO P ERTURBATIONS
Revised: R OBUSTNESS TO P ERTURBATIONS

Original: We are also interested in evaluating the robustness of the generated saliency maps: is the saliency different if non-salient changes are made to the state?
Revised: We are also interested in evaluating the robustness of the generated saliency maps: is the saliency different if non-salient changes are made to the state?

Original: To evaluate the robustness of our approach, we perform two irrelevant perturbations to the positions in the chess saliency dataset.
Revised: To evaluate the robustness of SARFA, we perform two irrelevant perturbations to the positions in the chess saliency dataset.

Original: First, we pick a random piece amongst the ones labeled non-salient by human experts in a particular position, and remove it from the board.
Revised: First, we pick a random piece amongst the ones labeled non-salient by human experts in a particular position, and remove it from the board.

Original: We repeat this for each puzzle in the dataset to generate a new perturbed saliency dataset.
Revised: We repeat this for each puzzle in the dataset to generate a new perturbed saliency dataset.

Original: Second, we remove a random piece amongst ones labeled non-salient by our approach for each puzzle, creating another perturbed saliency dataset.
Revised: Second, we remove a random piece amongst ones labeled non-salient by SARFA for each puzzle, creating another perturbed saliency dataset.

Original: In order to evaluate the effect of non-salient perturbations on our generated saliency maps, we compute the AUC values for the generated saliency maps, as above, for these perturbed datasets.
Revised: In order to evaluate the effect of non-salient perturbations on our generated saliency maps, we compute the AUC values for the generated saliency maps, as above, for these perturbed datasets.

Original: Since we remove non-salient pieces, we expect the saliency maps and subsequently AUC value to be similar to the value on the original dataset.
Revised: Since we remove non-salient pieces, we expect the saliency maps and subsequently AUC value to be similar to the value on the original dataset.

Original: For both these perturbations, we get an AUC value of 0.92 , same as the value on the non-perturbed dataset, conﬁrming the robustness of our saliency maps to these non-relevant perturbations.
Revised: For both these perturbations, we get an AUC value of 0.92 , same as the value on the non-perturbed dataset, conﬁrming the robustness of our saliency maps to these non-relevant perturbations.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Since understanding RL agents is important both for deploying RL agents to the real world and for gaining insights about the tasks, a number of different kinds of interpretations have been introduced.
Revised: Since understanding RL agents is important both for deploying RL agents to the real world and for gaining insights about the tasks, a number of different kinds of interpretations have been introduced.

Original: A number of approaches generate natural language explanations to explain RL agents (Dodson et al., 2011; Elizalde et al., 2008; Khan et al., 2009).
Revised: A number of approaches generate natural language explanations to explain RL agents (Dodson et al., 2011; Elizalde et al., 2008; Khan et al., 2009).

Original: They assume access to an exact MDP model and that the policies map from interpretable, high-level state features to actions.
Revised: They assume access to an exact MDP model and that the policies map from interpretable, high-level state features to actions.

Original: More recently, Hayes & Shah (2017) analyze execution traces of an agent to extract explanations.
Revised: More recently, Hayes & Shah (2017) analyze execution traces of an agent to extract explanations.

Original: A shortcoming of this approach is that it explains policies in terms of hand-crafted state representations that are semantically meaningful to humans.
Revised: A shortcoming of this approach is that it explains policies in terms of hand-crafted state representations that are semantically meaningful to humans.

Original: This is often not practical for board games or Atari games where the agents learn from raw board/visual input.
Revised: This is often not practical for board games or Atari games where the agents learn from raw board/visual input.

Original: Zahavy et al.
Revised: Zahavy et al.

Original: (2016) apply t-SNE (Maaten & Hinton, 2008) on the last layer of a deep Q-network (DQN) to cluster states of behavior of the agent.
Revised: (2016) apply t-SNE (Maaten & Hinton, 2008) on the last layer of a deep Q-network (DQN) to cluster states of behavior of the agent.

Original: They use Semi-Aggregated
Revised: They use Semi-Aggregated

Original: Markov Decision Processes (SAMDPs) to approximate the black box RL policies.
Revised: Markov Decision Processes (SAMDPs) to approximate the black box RL policies.

Original: They use the more interpretable SAMDPs to gain insight into the agent’s policy.
Revised: They use the more interpretable SAMDPs to gain insight into the agent’s policy.

Original: An issue with the explanations is that they emphasize t-SNE clusters that are difﬁcult to understand for non-experts.
Revised: An issue with the explanations is that they emphasize t-SNE clusters that are difﬁcult to understand for non-experts.

Original: To build user trust and increase adoption, it is important that the insight into agent behavior should be in a form that is interpretable to the untrained eye and obtained from the original policy instead of a distilled one.
Revised: To build user trust and increase adoption, it is important that the insight into agent behavior should be in a form that is interpretable to the untrained eye and obtained from the original policy instead of a distilled one.

Original: Most relevant to our approach are the visual interpretable explanations of deep networks using saliency maps.
Revised: Most relevant to SARFA are the visual interpretable explanations of deep networks using saliency maps.

Original: Methods for computing saliency can be classiﬁed broadly into two categories.
Revised: Methods for computing saliency can be classiﬁed broadly into two categories.

Original: Gradient-based methods identify input features that are most salient to the trained DNN by using the gradient to estimate their inﬂuence on the output.
Revised: Gradient-based methods identify input features that are most salient to the trained DNN by using the gradient to estimate their inﬂuence on the output.

Original: Simonyan et al.
Revised: Simonyan et al.

Original: (2013) use gradient magnitude heatmaps, which was expanded upon by more sophisticated methods to address their shortcoming, such as guided backpropagation (Springenberg et al., 2014), excitation backpropagation (Zhang et al., 2018), DeepLIFT (Shrikumar et al., 2017),
Revised: (2013) use gradient magnitude heatmaps, which was expanded upon by more sophisticated methods to address their shortcoming, such as guided backpropagation (Springenberg et al., 2014), excitation backpropagation (Zhang et al., 2018), DeepLIFT (Shrikumar et al., 2017),

Original: GradCAM (Selvaraju et al., 2017), and GradCAM++ (Chattopadhay et al., 2018).
Revised: GradCAM (Selvaraju et al., 2017), and GradCAM++ (Chattopadhay et al., 2018).

Original: Integrate gradients (Sundararajan et al., 2017) provide two axioms to further deﬁne the shortcomings of these approaches: sensitivity (relative to a baseline) and implementation invariance, and use them to derive an approach.
Revised: Integrate gradients (Sundararajan et al., 2017) provide two axioms to further deﬁne the shortcomings of these approaches: sensitivity (relative to a baseline) and implementation invariance, and use them to derive an approach.

Original: Nonetheless, all gradient-based approaches still depend on the shape in the immediate neighborhood of a few points, and conceptually, use perturbations that lack physical meaning, making them difﬁcult to use and vulnerable to adversarial attacks in form of imperceivable noise (Ghorbani et al., 2019).
Revised: Nonetheless, all gradient-based approaches still depend on the shape in the immediate neighborhood of a few points, and conceptually, use perturbations that lack physical meaning, making them difﬁcult to use and vulnerable to adversarial attacks in form of imperceivable noise (Ghorbani et al., 2019).

Original: Further, they are not applicable to scenarios with black-box access to the agent, and even with white-box access to model internals, they are not applicable when agents are not fully differentiable, such as Stockﬁsh for chess.
Revised: Further, they are not applicable to scenarios with black-box access to the agent, and even with white-box access to model internals, they are not applicable when agents are not fully differentiable, such as Stockﬁsh for chess.

Original: We are more interested in perturbation-based methods for black-box agents: methods that compute the importance of an input feature by removing, altering, or masking the feature in a domain-aware manner and observing the change in output.
Revised: We are more interested in perturbation-based methods for black-box agents: methods that compute the importance of an input feature by removing, altering, or masking the feature in a domain-aware manner and observing the change in output.

Original: It is important to choose a perturbation that removes information without introducing any new information.
Revised: It is important to choose a perturbation that removes information without introducing any new information.

Original: As a simple example, Fong & Vedaldi (2017) consider a classiﬁer that predicts ’True’ if a certain input image contains a bird and ‘False’ otherwise.
Revised: As a simple example, Fong & Vedaldi (2017) consider a classiﬁer that predicts ’True’ if a certain input image contains a bird and ‘False’ otherwise.

Original: Removing information from the part of the image which contains the bird should change the classiﬁer’s prediction, whereas removing information from other areas should not.
Revised: Removing information from the part of the image which contains the bird should change the classiﬁer’s prediction, whereas removing information from other areas should not.

Original: Several kinds of perturbations have been explored, e.g. Zeiler & Fergus (2014); Ribeiro et al.
Revised: Several kinds of perturbations have been explored, e.g. Zeiler & Fergus (2014); Ribeiro et al.

Original: (2016) remove information by replacing a part of the input with a gray square.
Revised: (2016) remove information by replacing a part of the input with a gray square.

Original: Note that these approaches are implementation invariant by deﬁnition, and are sensitive with respect to the perturbation function.
Revised: Note that these approaches are implementation invariant by deﬁnition, and are sensitive with respect to the perturbation function.

Original: Existing perturbation-based approaches for RL (Greydanus et al., 2018; Iyer et al., 2018), however, by focusing on the complete Q (or V ), tend to produce saliency maps that are not speciﬁc to the action of interest.
Revised: Existing perturbation-based approaches for RL (Greydanus et al., 2018; Iyer et al., 2018), however, by focusing on the complete Q (or V ), tend to produce saliency maps that are not speciﬁc to the action of interest.

Original: Our proposed approach addresses this by measuring the impact only on the action being selected, resulting in more focused and useful saliency maps, as we show in our experiments.
Revised: SARFA addresses this by measuring the impact only on the action being selected, resulting in more focused and useful saliency maps, as we show in our experiments.

Original:  L IMITATIONS AND F UTURE W ORK
Revised:  L IMITATIONS AND F UTURE W ORK

Original: Saliency maps focus on visualizing the dependence between the input and output to the model, essentially identifying the situation-speciﬁc explanation for the decision.
Revised: Saliency maps focus on visualizing the dependence between the input and output to the model, essentially identifying the situation-speciﬁc explanation for the decision.

Original: Although such local explanations have applications in understanding, debugging, and developing trust with machine learning systems, they do not provide any direct insights regarding the general behavior of the model, or guarantee that the explanation is applicable to a different scenario.
Revised: Although such local explanations have applications in understanding, debugging, and developing trust with machine learning systems, they do not provide any direct insights regarding the general behavior of the model, or guarantee that the explanation is applicable to a different scenario.

Original: Attempts to provide a more general understanding of the model include carefully selecting prototype explanations to show to the user (van der Linden et al., 2019) and crafting explanations that are precise and actionable (Ribeiro et al., 2018).
Revised: Attempts to provide a more general understanding of the model include carefully selecting prototype explanations to show to the user (van der Linden et al., 2019) and crafting explanations that are precise and actionable (Ribeiro et al., 2018).

Original: We will explore such ideas for the RL setting in future, to provide explanations that accurately characterize the behavior of the policy function, in a precise, testable, and intuive manner.
Revised: We will explore such ideas for the RL setting in future, to provide explanations that accurately characterize the behavior of the policy function, in a precise, testable, and intuive manner.

Original: There are a number of limitations of our approach to generating saliency maps in our current implementation.
Revised: There are a number of limitations of SARFA to generating saliency maps in our current implementation.

Original: First, we perturb the state by removing information (removing pieces in Chess/Go, blurring pixels in Atari).
Revised: First, we perturb the state by removing information (removing pieces in Chess/Go, blurring pixels in Atari).

Original: Therefore, our approach cannot highlight the importance of absence of certain attributes, i.e. saliency of certain positions being empty.
Revised: Therefore, SARFA cannot highlight the importance of absence of certain attributes, i.e. saliency of certain positions being empty.

Original: In games such as Chess and Go, an empty square or ﬁle (collection of empty squares) can often be important for a particular move.
Revised: In games such as Chess and Go, an empty square or ﬁle (collection of empty squares) can often be important for a particular move.

Original: Future work will explore perturbation functions that add information to the state (perhaps in the form of adding pieces in Chess/Go).
Revised: Future work will explore perturbation functions that add information to the state (perhaps in the form of adding pieces in Chess/Go).

Original: Such functions, along with our approach, can be used to calculate the importance of empty squares.
Revised: Such functions, along with SARFA, can be used to calculate the importance of empty squares.

Original: Second, it is possible that perturbations may explore states that lie outside the manifold, i.e. they lead to invalid states.
Revised: Second, it is possible that perturbations may explore states that lie outside the manifold, i.e. they lead to invalid states.

Original: For example, unless explicitly prohibited like we do, our approach will compute the saliency of the king pieces by removing them, which is not allowed in the game, or remove the paddle from Pong.
Revised: For example, unless explicitly prohibited like we do, SARFA will compute the saliency of the king pieces by removing them, which is not allowed in the game, or remove the paddle from Pong.

Original: In future, we will explore strategies that take the valid state space into account when computing the saliency.
Revised: In future, we will explore strategies that take the valid state space into account when computing the saliency.

Original: Last we are estimating the saliency of each feature independently, ignoring feature dependencies and correlations, which may lead to incorrect saliency maps.
Revised: Last we are estimating the saliency of each feature independently, ignoring feature dependencies and correlations, which may lead to incorrect saliency maps.

Original: We will investigate approaches that perturb multiple features to quantify the importance of each feature (Ribeiro et al., 2016; Lundberg & Lee, 2017), and combine them with our approach to explaining black-box policy-based agents.
Revised: We will investigate approaches that perturb multiple features to quantify the importance of each feature (Ribeiro et al., 2016; Lundberg & Lee, 2017), and combine them with SARFA to explaining black-box policy-based agents.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We presented a perturbation-based approach that generates more focused saliency maps than existing approaches by balancing two aspects (speciﬁcity and relevance) that capture different desired characteristics of saliency.
Revised: We presented a perturbation-based approach that generates more focused saliency maps than existing approaches by balancing two aspects (speciﬁcity and relevance) that capture different desired characteristics of saliency.

Original: We showed through illustrative examples (Chess, Atari, Go), human studies (Chess) and automated evaluation methods (Chess) that our approach generates saliency maps that are more interpretable for humans than existing approaches.
Revised: We showed through illustrative examples (Chess, Atari, Go), human studies (Chess) and automated evaluation methods (Chess) that SARFA generates saliency maps that are more interpretable for humans than existing approaches.

Original: The results of our technique show that saliency can provide meaningful insights into a black-box RL agent’s behavior
Revised: The results of our technique show that saliency can provide meaningful insights into a black-box RL agent’s behavior

Revised: A CKNOWLEDGEMENTS

Revised: We would like to thank the anonymous reviewers for their helpful comments and suggestions.

Revised: This work is supported in part by the NSF Award No.

Revised: IIS-1756023 and in part by a gift from the Allen

Revised: Institute of Artiﬁcial Intelligence (AI2)

Original: A E XPERIMENTAL D ETAILS
Revised: A E XPERIMENTAL D ETAILS

Original: For experiments on chess, we use the latest version of the Stockﬁsh agent: https:// stockfishchess.org/ .
Revised: For experiments on chess, we use the latest version of the Stockﬁsh agent: https:// stockfishchess.org/ .

Original: Stockﬁsh works using a heuristic-based measure for each state along with Alpha-Beta Pruning to search over the state-space.
Revised: Stockﬁsh works using a heuristic-based measure for each state along with Alpha-Beta Pruning to search over the state-space.

Original: For experiments on Go, we use the pre-trained MiniGo RL agent: https://github.com/ tensorflow/minigo .
Revised: For experiments on Go, we use the pre-trained MiniGo RL agent: https://github.com/ tensorflow/minigo .

Original: This agent was trained using the AlphaGo Algorithm (Silver et al., 2016).
Revised: This agent was trained using the AlphaGo Algorithm (Silver et al., 2016).

Original: It also adds features and architecture changes from the AlphaZero Algorithm Silver et al.
Revised: It also adds features and architecture changes from the AlphaZero Algorithm Silver et al.

Original: For experiments on Atari agents and for generating saliency maps for Greydanus et al.
Revised: For experiments on Atari agents and for generating saliency maps for Greydanus et al.

Original: (2018), we use their code and pre-trained RL agents available at https://github.com/greydanus/ visualize_atari .
Revised: (2018), we use their code and pre-trained RL agents available at https://github.com/greydanus/ visualize_atari .

Original: These agents are trained using the Asynchronous Advantage Actor-Critic
Revised: These agents are trained using the Asynchronous Advantage Actor-Critic

Original: Algorithm (A3C) (Mnih et al., 2016).
Revised: Algorithm (A3C) (Mnih et al., 2016).

Original: For generating saliency maps using Iyer et al.
Revised: For generating saliency maps using Iyer et al.

Original: (2018), we use our implementation.
Revised: (2018), we use our implementation.

Original: All of our code and more detailed results are available in our Github repository: https://github.com/ rl-interpretation/understandingRL .
Revised: All of our code and more detailed results are available in our Github repository: https://github.com/ rl-interpretation/understandingRL .

Original: For chess and Go, we perturb the board position by removing one piece at a time.
Revised: For chess and Go, we perturb the board position by removing one piece at a time.

Original: We do not remove a piece if the resulting position is illegal.
Revised: We do not remove a piece if the resulting position is illegal.

Original: For instance, in chess, we do not remove the king.
Revised: For instance, in chess, we do not remove the king.

Original: For Atari, we use the perturbation technique described in Greydanus et al.
Revised: For Atari, we use the perturbation technique described in Greydanus et al.

Original: The technique perturbs the input image by adding a Gaussian blur localized around a pixel.
Revised: The technique perturbs the input image by adding a Gaussian blur localized around a pixel.

Original: The blur is constructed using the
Revised: The blur is constructed using the

Original: Hadamard product to interpolate between the original input image and a Gaussian blur.
Revised: Hadamard product to interpolate between the original input image and a Gaussian blur.

Original: The saliency maps for Atari agents have been computed on the frames provided by Greydanus et al.
Revised: The saliency maps for Atari agents have been computed on the frames provided by Greydanus et al.

Original: (2018) in their code repository.
Revised: (2018) in their code repository.

Original: The puzzles for conducting the Chess human studies, creating the Chess Saliency Dataset, and providing illustrative examples have been taken from Lichess: https://database.lichess.
Revised: The puzzles for conducting the Chess human studies, creating the Chess Saliency Dataset, and providing illustrative examples have been taken from Lichess: https://database.lichess.

Original: org/ .
Revised: org/ .

Original: The puzzles for illustrative examples on Go have been taken from OnlineGo: https: //online-go.com/puzzles .
Revised: The puzzles for illustrative examples on Go have been taken from OnlineGo: https: //online-go.com/puzzles .

Original: B S ALIENCY M APS FOR TOP 3 MOVES
Revised: B S ALIENCY M APS FOR TOP 3 MOVES

Original: Figure 8 shows the saliency maps generated by our approach for the top 3 moves in a chess position.
Revised: Figure 8 shows the saliency maps generated by SARFA for the top 3 moves in a chess position.

Original: The maps highlight the different pieces that are salient for each move.
Revised: The maps highlight the different pieces that are salient for each move.

Original: For instance, Figure 8a shows that for the move Qd4, the pawn on g7 is important.
Revised: For instance, Figure 8a shows that for the move Qd4, the pawn on g7 is important.

Original: This is because the queen move protects the pawn.
Revised: This is because the queen move protects the pawn.

Original: For the saliency maps in Figures 8b and 8c, the pawn on g7 is not highlighted.
Revised: For the saliency maps in Figures 8b and 8c, the pawn on g7 is not highlighted.

Original: C S ALIENCY M APS FOR L EELA Z ERO
Revised: C S ALIENCY M APS FOR L EELA Z ERO

Original: To show that our approach generates meaningful saliency maps in Chess for RL agents, we interpret the LeelaZero Deep RL agent https://github.com/leela-zero/leela-zero .
Revised: To show that SARFA generates meaningful saliency maps in Chess for RL agents, we interpret the LeelaZero Deep RL agent https://github.com/leela-zero/leela-zero .

Original: Figureshows the results.
Revised: Figure 9 shows the results.

Original: As discussed in Section 1, the saliency maps generated by (Greydanus et al.,2018;
Revised: As discussed in Section 1, the saliency maps generated by (Greydanus et al.,

Original: [Figure]
Revised: [Figure]

Original:  Iyer et al., 2018) highlight several pieces that are not relevant to the move being explained.
Revised: 2018; Iyer et al., 2018) highlight several pieces that are not relevant to the move being explained.

Original: On the other hand, the saliency maps generated by our approach highlight the pieces relevant to the move.
Revised: On the other hand, the saliency maps generated by SARFA highlight the pieces relevant to the move.