Original: V ISUAL E XPLANATION USING A TTENTION M ECHANISM IN A CTOR -C RITIC - BASED D EEP R EINFORCEMENT L
Revised: V ISUAL E XPLANATION USING A TTENTION M ECHANISM IN A CTOR -C RITIC - BASED D EEP R EINFORCEMENT L

Original: EARNING
Revised: EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep reinforcement learning (DRL) has great potential for acquiring the optimal action in complex environments such as games and robot control.
Revised: Deep reinforcement learning (DRL) has great potential for acquiring the optimal action in complex environments such as games and robot control.

Original: However, it is difﬁcult to analyze the decision-making of the agent, i.e., the reasons it selects the action acquired by learning.
Revised: However, it is difﬁcult to analyze the decision-making of the agent, i.e., the reasons it selects the action acquired by learning.

Original: In this work, we propose Mask-Attention
Revised: In this work, we propose Mask-Attention

Original: A3C (Mask A3C) that introduced an attention mechanism into Asynchronous Advantage Actor-Critic (A3C) which is an actor-critic-based DRL method, and can analyze decision making of agent in DRL.
Revised: A3C (Mask A3C) that introduced an attention mechanism into Asynchronous Advantage Actor-Critic (A3C) which is an actor-critic-based DRL method, and can analyze decision making of agent in DRL.

Original: A3C consists of a feature extractor that extracts features from an image, a policy branch that outputs the policy, value branch that outputs the state value.
Revised: A3C consists of a feature extractor that extracts features from an image, a policy branch that outputs the policy, value branch that outputs the state value.

Original: In our method, we focus on the policy and value branches and introduce an attention mechanism into them.
Revised: In our method, we focus on the policy and value branches and introduce an attention mechanism into them.

Original: The attention mechanism applies a mask processing on the feature maps of each branch using mask-attention that expresses the judgment reason for the policy and state value with a heat map.
Revised: The attention mechanism applies a mask processing on the feature maps of each branch using mask-attention that expresses the judgment reason for the policy and state value with a heat map.

Original: We visualized mask-attention maps for games on the Atariand found we could easily analyze the reasons behind an agent’s decision-making in various game tasks.
Revised: We visualized mask-attention maps for games on the Atariand found we could easily analyze the reasons behind an agent’s decision-making in various game tasks.

Original: Furthermore, experimental results showed that the agent’s higher performance could be achieved by introducing the attention mechanism.
Revised: Furthermore, experimental results showed that the agent’s higher performance could be achieved by introducing the attention mechanism.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement learning (RL) problems seek optimal actions to maximize cumulative rewards.
Revised: Reinforcement learning (RL) problems seek optimal actions to maximize cumulative rewards.

Original: Unlike supervised learning problems, RL problems collect training data by exploring the environment.
Revised: Unlike supervised learning problems, RL problems collect training data by exploring the environment.

Original: Therefore, it is achieving high performance in speciﬁc tasks (e.g., controlling autonomous systems (Kober et al., 2013; Gu et al., 2017; Rajeswaran et al., 2018) and video games (Tessler et al., 2017;
Revised: Therefore, it is achieving high performance in speciﬁc tasks (e.g., controlling autonomous systems (Kober et al., 2013; Gu et al., 2017; Rajeswaran et al., 2018) and video games (Tessler et al., 2017;

Original: Justesen et al., 2017; Shao et al., 2019))
Revised: Justesen et al., 2017; Shao et al., 2019))

Original: that difﬁcult to create training data.
Revised: that difﬁcult to create training data.

Original: In Go, AlphaGo is defeated a professional Go player (Silver et al., 2016).
Revised: In Go, AlphaGo is defeated a professional Go player (Silver et al., 2016).

Original: In 2015, the deep Q-network (DQN), a method that combines Q-learning (Watkins & Dayan, 1992) and deep neural network (DNN), achieved a score higher than human players on the Atari 2600 (Mnih et al., 2015).
Revised: In 2015, the deep Q-network (DQN), a method that combines Q-learning (Watkins & Dayan, 1992) and deep neural network (DNN), achieved a score higher than human players on the Atari 2600 (Mnih et al., 2015).

Original: Since the advent of DQN, deep RL (DRL), a method that combines deep learning and RL, has become mainstream, and it is now possible to solve problems featuring a huge number of states, such as images.
Revised: Since the advent of DQN, deep RL (DRL), a method that combines deep learning and RL, has become mainstream, and it is now possible to solve problems featuring a huge number of states, such as images.

Original: In general, deep learning can solve complex tasks by training using a large number of network parameters.
Revised: In general, deep learning can solve complex tasks by training using a large number of network parameters.

Original: However, it is difﬁcult to understand the reasoning behind the decision-making of the trained network because the network parameters used to make the decision are enormous.
Revised: However, it is difﬁcult to understand the reasoning behind the decision-making of the trained network because the network parameters used to make the decision are enormous.

Original: This problem occurs in DRL as well.
Revised: This problem occurs in DRL as well.

Original: The reason for judging the acquired action is unclear, since agents collect training data by searching the environment and the calculation inside the network is complicated.
Revised: The reason for judging the acquired action is unclear, since agents collect training data by searching the environment and the calculation inside the network is complicated.

Original: Therefore, in order to prove that the trained network is sufﬁciently reliable, it is important to analyze the reason for the judgment of the action that it outputs.
Revised: Therefore, in order to prove that the trained network is sufﬁciently reliable, it is important to analyze the reason for the judgment of the action that it outputs.

Original: One approach to interpreting the decision-making of a network, visual explanation, has been studied in the ﬁeld of computer vision (Zhou et al., 2016;
Revised: One approach to interpreting the decision-making of a network, visual explanation, has been studied in the ﬁeld of computer vision (Zhou et al., 2016;

Original: Selvaraju et al., 2017; Fukui et al., 2019).
Revised: Selvaraju et al., 2017; Fukui et al., 2019).

Original: Visual explanations analyze the factors of the network output by using an attention map that highlights the important regions in an input image.
Revised: Visual explanations analyze the factors of the network output by using an attention map that highlights the important regions in an input image.

Original: Visual explanation methods have also been applied to DRL models to help with understanding the decision-making of an agent (Sorokin et al., 2015;
Revised: Visual explanation methods have also been applied to DRL models to help with understanding the decision-making of an agent (Sorokin et al., 2015;

Original: Greydanus et al., 2018).
Revised: Greydanus et al., 2018).

Original: These methods can be categorized into two approaches: bottom-up and top-down.
Revised: These methods can be categorized into two approaches: bottom-up and top-down.

Original: Bottom-up visual explanations compute attention maps by using the gradient informa- tion of a network.
Revised: Bottom-up visual explanations compute attention maps by using the gradient informa- tion of a network.

Original: Because the bottom-up approach does not need to re-train a network, it can be applied to any trained network and is commonly used in computer vision and DRL.
Revised: Because the bottom-up approach does not need to re-train a network, it can be applied to any trained network and is commonly used in computer vision and DRL.

Original: The attention maps obtained by bottom-up approach are based on the input data and response values calculated from each layers.
Revised: The attention maps obtained by bottom-up approach are based on the input data and response values calculated from each layers.

Original: The bottom-up approach highlights local textural context.
Revised: The bottom-up approach highlights local textural context.

Original: Top-down visual explanations generates attention maps by using the response values in a network.
Revised: Top-down visual explanations generates attention maps by using the response values in a network.

Original: Unlike the bottom-up approach, the attention maps of the top-down approach are output for current network output.
Revised: Unlike the bottom-up approach, the attention maps of the top-down approach are output for current network output.

Original: In this paper, we propose Mask-Attention A3C (Mask A3C) which introduce an attention mechanism into Asynchronous Advantage Actor-Critic (A3C) of an actor-critic-based DRL method.
Revised: In this paper, we propose Mask-Attention A3C (Mask A3C) which introduce an attention mechanism into Asynchronous Advantage Actor-Critic (A3C) of an actor-critic-based DRL method.

Original: Mask
Revised: Mask

Original: A3C calculates an mask-attention that an attention map of policy and state value, and then a visual explanation for these values is achieved by visualizing the acquired mask-attention.
Revised: A3C calculates an mask-attention that an attention map of policy and state value, and then a visual explanation for these values is achieved by visualizing the acquired mask-attention.

Original: Our method also learns the policy and state value while considering mask-attention by implementing the attention mechanism, thereby improving the performance of the agent.
Revised: Our method also learns the policy and state value while considering mask-attention by implementing the attention mechanism, thereby improving the performance of the agent.

Original: Contributions
Revised: Contributions

Original: The main contributions of this paper are as follows.
Revised: The main contributions of this paper are as follows.

Original: • We propose a top-down visual explanation method that implements an attention mechanism in the DRL model.
Revised: • We propose a top-down visual explanation method that implements an attention mechanism in the DRL model.

Original: In the proposed method, mask-attention, which is an attention map for the outputs, can be obtained simply by forward pass.
Revised: In the proposed method, mask-attention, which is an attention map for the outputs, can be obtained simply by forward pass.

Original: In the proposed method, the decision-making of the agent after learning can be analyzed by visualizing the acquired mask-attention.
Revised: In the proposed method, the decision-making of the agent after learning can be analyzed by visualizing the acquired mask-attention.

Original: We conducted an experiment with games on the Atari 2600 and analyzed which information inﬂuences the agent’s decision-making.
Revised: We conducted an experiment with games on the Atari 2600 and analyzed which information inﬂuences the agent’s decision-making.

Original: • By implementing the attention mechanism in the policy branch and value branch of the actor-critic method, a different mask-attention can be obtained depending on the policy and state value.
Revised: • By implementing the attention mechanism in the policy branch and value branch of the actor-critic method, a different mask-attention can be obtained depending on the policy and state value.

Original: In this way, it is possible to analyze an agent’s decision-making from the two viewpoints of policy value and state value.
Revised: In this way, it is possible to analyze an agent’s decision-making from the two viewpoints of policy value and state value.

Original: • The proposed method outputs the control value of the agent while considering maskattention by implementing the attention mechanism.
Revised: • The proposed method outputs the control value of the agent while considering maskattention by implementing the attention mechanism.

Original: Therefore, the performance of the agent can be improved by emphasizing the area related to the control value.
Revised: Therefore, the performance of the agent can be improved by emphasizing the area related to the control value.

Original:  R ELATED WORKS 2. D EEP R EINFORCEMENT L EARNING
Revised:  R ELATED WORKS 2. D EEP R EINFORCEMENT L EARNING

Original: The deep Q-network (DQN) (Mnih et al., 2015), which is a typical method of DRL, expresses the action value function Q ( a | s ; θ ) by using a neural network and acquires the optimum action by training the network parameters θ .
Revised: The deep Q-network (DQN) (Mnih et al., 2015), which is a typical method of DRL, expresses the action value function Q ( a | s ; θ ) by using a neural network and acquires the optimum action by training the network parameters θ .

Original: DRL methods that learns the optimal action by a value function such as DQN is called value-based DRL, and have been studied extensively (Van Hasselt et al., 2016;
Revised: DRL methods that learns the optimal action by a value function such as DQN is called value-based DRL, and have been studied extensively (Van Hasselt et al., 2016;

Original: Wang et al., 2016; Bellemare et al., 2017; Hessel et al., 2018).
Revised: Wang et al., 2016; Bellemare et al., 2017; Hessel et al., 2018).

Original: There is also a policy-based DRL that directly learns the policy by expressing the policy π ( a | s ; θ ) with a neural network (Lillicrap et al., 2016; Schulman et al., 2015; 2017; Haarnoja et al., 2018).
Revised: There is also a policy-based DRL that directly learns the policy by expressing the policy π ( a | s ; θ ) with a neural network (Lillicrap et al., 2016; Schulman et al., 2015; 2017; Haarnoja et al., 2018).

Original: The actor-critic method (Konda &
Revised: The actor-critic method (Konda &

Original: Tsitsiklis, 2000), which is a policy-based method, consists of an actor that output the policy π ( a | s ; θ ) and a critic that output the state value V ( s ; θ ) .
Revised: Tsitsiklis, 2000), which is a policy-based method, consists of an actor that output the policy π ( a | s ; θ ) and a critic that output the state value V ( s ; θ ) .

Original: Here, the state value V ( s ; θ ) numerically expresses how the current state s contributes to the reward.
Revised: Here, the state value V ( s ; θ ) numerically expresses how the current state s contributes to the reward.

Original: The actor selects and performs an action according to a policy π ( a | s ; θ ) that is a probability distribution from a state s to an action a .
Revised: The actor selects and performs an action according to a policy π ( a | s ; θ ) that is a probability distribution from a state s to an action a .

Original: The critic estimate the state value V ( s ; θ ) as the evaluation value of the policy π ( a | s ; θ ) that is output by the actor.
Revised: The critic estimate the state value V ( s ; θ ) as the evaluation value of the policy π ( a | s ; θ ) that is output by the actor.

Original: To update the network parameters in the actor-critic method, the actor parameter update by the policy gradient method and the critic parameter update by the TD error are performed in parallel.
Revised: To update the network parameters in the actor-critic method, the actor parameter update by the policy gradient method and the critic parameter update by the TD error are performed in parallel.

Original: Other approaches include distributed DRL, which improves learning efﬁciency by constructing multiple environments and agents (Nair et al., 2015; Jaderberg et al., 2017; Kapturowski et al., 2019).
Revised: Other approaches include distributed DRL, which improves learning efﬁciency by constructing multiple environments and agents (Nair et al., 2015; Jaderberg et al., 2017; Kapturowski et al., 2019).

Original: Asynchronous Advantage Actor-Critic (A3C) (Mnih et al., 2016) is a distributed DRL method that utilizes the actor-critic method.
Revised: Asynchronous Advantage Actor-Critic (A3C) (Mnih et al., 2016) is a distributed DRL method that utilizes the actor-critic method.

Original: A3C introduces Asynchronous that is an asynchronous parameter update in distributed learning and Advantage that learns while considering rewards several steps ahead.
Revised: A3C introduces Asynchronous that is an asynchronous parameter update in distributed learning and Advantage that learns while considering rewards several steps ahead.

Original: Experiments with the Atari 2600 showed that A3C could achieved a high score in a short training time by executing the generation of experiences used for learning in parallel.
Revised: Experiments with the Atari 2600 showed that A3C could achieved a high score in a short training time by executing the generation of experiences used for learning in parallel.

Original: In this study, we acquire mask-attention, attention maps for policy and state value, by implementing an attention mechanism in A3C. By visualizing mask-attention at inference, the decision-making of the agent acquired by learning is analyzed from the visual explanation of the policy and state value.
Revised: In this study, we acquire mask-attention, attention maps for policy and state value, by implementing an attention mechanism in A3C. By visualizing mask-attention at inference, the decision-making of the agent acquired by learning is analyzed from the visual explanation of the policy and state value.

Original: 2. V ISUAL EXPLANATIONS Visual Explanations in Image Recognition
Revised: 2. V ISUAL EXPLANATIONS Visual Explanations in Image Recognition

Original: In the ﬁeld of image recognition, several methods utilizing an attention map have been proposed for analyzing the reason for judgments on the inference result of the network.
Revised: In the ﬁeld of image recognition, several methods utilizing an attention map have been proposed for analyzing the reason for judgments on the inference result of the network.

Original: Attention map is a map that visualized the network gazed area at the time of inference as a heat map.
Revised: Attention map is a map that visualized the network gazed area at the time of inference as a heat map.

Original: Zhou et al.
Revised: Zhou et al.

Original: (2016) proposed a class activation mapping (CAM), which acquires the attention map of a speciﬁc class from the response value of the convolutional layer and the weight of the fully connected layer.
Revised: (2016) proposed a class activation mapping (CAM), which acquires the attention map of a speciﬁc class from the response value of the convolutional layer and the weight of the fully connected layer.

Original: However, the recognition performance of CAM deteriorates because it is necessary to change a part of the network structure, such as by introducing global average pooling (GAP) between the convolution and fully connected layers.
Revised: However, the recognition performance of CAM deteriorates because it is necessary to change a part of the network structure, such as by introducing global average pooling (GAP) between the convolution and fully connected layers.

Original: For that problem, Selvaraju et al.
Revised: For that problem, Selvaraju et al.

Original: (2017) proposed gradient-weighted
Revised: (2017) proposed gradient-weighted

Original: CAM (Grad-CAM), which acquires an attention map by using the response value of the convolutional layer during forward pass and the gradient during back-propagation.
Revised: CAM (Grad-CAM), which acquires an attention map by using the response value of the convolutional layer during forward pass and the gradient during back-propagation.

Original: Grad-CAM avoids the problem that the recognition performance deteriorates by generating an attention map from the gradient information.
Revised: Grad-CAM avoids the problem that the recognition performance deteriorates by generating an attention map from the gradient information.

Original: In image recognition, the recognition accuracy is known to improve by using an attention map during learning.
Revised: In image recognition, the recognition accuracy is known to improve by using an attention map during learning.

Original: Fukui et al.
Revised: Fukui et al.

Original: (2019) proposed an attention branch network (ABN) that applies the attention map to the attention mechanism.
Revised: (2019) proposed an attention branch network (ABN) that applies the attention map to the attention mechanism.

Original: This method provides a visual explanation of the reason for judgment by the attention map simultaneously improves the recognition accuracy.
Revised: This method provides a visual explanation of the reason for judgment by the attention map simultaneously improves the recognition accuracy.

Original: Visual Explanations in Deep Reinforcement Learning
Revised: Visual Explanations in Deep Reinforcement Learning

Original: In DRL, several works have examined how to analyze the reason for judging an action that output of the network.
Revised: In DRL, several works have examined how to analyze the reason for judging an action that output of the network.

Original: Sorokin et al.
Revised: Sorokin et al.

Original: (2015) proposed the deep attention recurrent Q-network (DARQN).
Revised: (2015) proposed the deep attention recurrent Q-network (DARQN).

Original: They implements an attention mechanism in DQN, a representative value-based method.
Revised: They implements an attention mechanism in DQN, a representative value-based method.

Original: Manchin et al.
Revised: Manchin et al.

Original: (2019) introduces self-attention to a policy-based DRL method, in order to improve the score along with policy analysis.
Revised: (2019) introduces self-attention to a policy-based DRL method, in order to improve the score along with policy analysis.

Original: This method analyzes the agent’s decision making using an attention map for policy.
Revised: This method analyzes the agent’s decision making using an attention map for policy.

Original: On the other hand, our method can improve the interpretability of the agent’s decision making in the actor-critic based DRL method by simultaneously acquiring different attention maps for policies and state values.
Revised: On the other hand, our method can improve the interpretability of the agent’s decision making in the actor-critic based DRL method by simultaneously acquiring different attention maps for policies and state values.

Original: Greydanus et al.
Revised: Greydanus et al.

Original: (2018) has acquired a saliency map in A3C by calculating a perturbation image utilizing an applied Gaussian ﬁlter from the gradient at during back-propagation.
Revised: (2018) has acquired a saliency map in A3C by calculating a perturbation image utilizing an applied Gaussian ﬁlter from the gradient at during back-propagation.

Original: This method takes a bottom-up approach is similar to Grad-CAM, and therefore it is necessary to perform backpropagation to acquire the saliency map.
Revised: This method takes a bottom-up approach is similar to Grad-CAM, and therefore it is necessary to perform backpropagation to acquire the saliency map.

Original: In contrast, our method takes a top-down approach that implement an attention mechanism in the network structure of A3C.
Revised: In contrast, our method takes a top-down approach that implement an attention mechanism in the network structure of A3C.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2018) proposed attention guided imitation learning (AGIL), which guides the focus area of a network on the basis of human gaze information.
Revised: (2018) proposed attention guided imitation learning (AGIL), which guides the focus area of a network on the basis of human gaze information.

Original: They trained a model to replicate human attention with supervised gaze heat maps.
Revised: They trained a model to replicate human attention with supervised gaze heat maps.

Original: The input state was then augmented with this additional information.
Revised: The input state was then augmented with this additional information.

Original: This style of attention fundamentally differs from that used in our work as it incorporates hand crafted features as input.
Revised: This style of attention fundamentally differs from that used in our work as it incorporates hand crafted features as input.

Original: The most similar work with us is conducted by Mott et al.
Revised: The most similar work with us is conducted by Mott et al.

Original: They proposed a model for acquiring two attention (“what” and “where”) by using query-based attention in an actor-critic based
Revised: They proposed a model for acquiring two attention (“what” and “where”) by using query-based attention in an actor-critic based

Original: DRL method.
Revised: DRL method.

Original: This method requires an attention query to be generated, which requires signiﬁcant changes to the network architecture (e.g., Keys, Values).
Revised: This method requires an attention query to be generated, which requires signiﬁcant changes to the network architecture (e.g., Keys, Values).

Original: In contrast, our method has a simple structure in which an attention mechanism is introduced in the policy and value branches and does not require signiﬁcant changes to the network architecture.
Revised: In contrast, our method has a simple structure in which an attention mechanism is introduced in the policy and value branches and does not require signiﬁcant changes to the network architecture.

Original: Also, they obtained different attentions related to “what” and “where” by generating an attention query.
Revised: Also, they obtained different attentions related to “what” and “where” by generating an attention query.

Original: In contrast, our method improves interpretability by acquiring different attentions toward policy and state value, which are the outputs of actor-critic-based DRL methods.
Revised: In contrast, our method improves interpretability by acquiring different attentions toward policy and state value, which are the outputs of actor-critic-based DRL methods.

Original: The above visual explanations for DRL generates attention maps from the low-level feature maps extracted from early or middle convolutional layers, or output for action (e.g., policy output or Q value output on DQN models).
Revised: The above visual explanations for DRL generates attention maps from the low-level feature maps extracted from early or middle convolutional layers, or output for action (e.g., policy output or Q value output on DQN models).

Original: In DRL, a strategy is an important clue to solve given task and environment.
Revised: In DRL, a strategy is an important clue to solve given task and environment.

Original: From the viewpoint of the strategy, the state value of actor-critic based model plays a
Revised: From the viewpoint of the strategy, the state value of actor-critic based model plays a

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: C on v .
Revised: C on v .

Original: S i g m o i d
Revised: S i g m o i d

Original: C on v .
Revised: C on v .

Original: R eL U
Revised: R eL U

Original: C on v .
Revised: C on v .

Original: C on v .
Revised: C on v .

Original: M a x poo li ng + R eL U
Revised: M a x poo li ng + R eL U

Original: C on v .
Revised: C on v .

Original: M a x poo li ng + R eL U
Revised: M a x poo li ng + R eL U

Original: C on v .
Revised: C on v .

Original: M a x poo li ng +
Revised: M a x poo li ng +

Original: R eL U
Revised: R eL U

Original: Policy branch
Revised: Policy branch

Original: F C .
Revised: F C .

Original: Policy C on v L
Revised: Policy C on v L

Original: S T M
Revised: S T M

Original: C on v .
Revised: C on v .

Original: R eL U
Revised: R eL U

Original: F C .
Revised: F C .

Original: Value
Revised: Value

Original: Value branch
Revised: Value branch

Original: S i g m o i d
Revised: S i g m o i d

Original: C on v .
Revised: C on v .

Original: S i g m o i d
Revised: S i g m o i d

Original: C on v .
Revised: C on v .

Original: R eL U
Revised: R eL U

Original: C on v .
Revised: C on v .

Original: R eL U C on v .
Revised: R eL U C on v .

Original: F C .
Revised: F C .

Original: C on v .
Revised: C on v .

Original: M a x poo li ng + R eL U
Revised: M a x poo li ng + R eL U

Original: C on v .
Revised: C on v .

Original: M a x poo li ng + R eL U
Revised: M a x poo li ng + R eL U

Original: C on v .
Revised: C on v .

Original: M a x poo li ng +
Revised: M a x poo li ng +

Original: R eL U
Revised: R eL U

Original: Value branch
Revised: Value branch

Original: Policy branch
Revised: Policy branch

Original: F C .
Revised: F C .

Original: Policy
Revised: Policy

Original: Value
Revised: Value

Original: C on v L S T
Revised: C on v L S T

Original: M
Revised: M

Original: G AP Feature extractor
Revised: G AP Feature extractor

Original: Mask-Attention
Revised: Mask-Attention

Original: Input
Revised: Input

Original: AttentionMechanism
Revised: AttentionMechanism

Original: C on v .
Revised: C on v .

Original: S i g m o i d
Revised: S i g m o i d

Original: C on v .
Revised: C on v .

Original: R eL U
Revised: R eL U

Original: C on v .
Revised: C on v .

Original: R eL U C on v .
Revised: R eL U C on v .

Original: F C .
Revised: F C .

Original: C on v .
Revised: C on v .

Original: M a x poo li ng + R eL U
Revised: M a x poo li ng + R eL U

Original: C on v .
Revised: C on v .

Original: M a x poo li ng + R eL U
Revised: M a x poo li ng + R eL U

Original: C on v .
Revised: C on v .

Original: M a x poo li ng +
Revised: M a x poo li ng +

Original: R eL U
Revised: R eL U

Original: Value branch
Revised: Value branch

Original: Policy branch
Revised: Policy branch

Original: F C .
Revised: F C .

Original: Policy
Revised: Policy

Original: Value
Revised: Value

Original: C on v L S T
Revised: C on v L S T

Original: M
Revised: M

Original: G AP Feature extractor
Revised: G AP Feature extractor

Original: Mask-Attention
Revised: Mask-Attention

Original: Input
Revised: Input

Original: AttentionMechanism
Revised: AttentionMechanism

Original: Mask-Attention
Revised: Mask-Attention

Original: Mask-Attention Mask-Attention
Revised: Mask-Attention Mask-Attention

Original: C on v .
Revised: C on v .

Original: S i g m o i d
Revised: S i g m o i d

Original: C on v .
Revised: C on v .

Original: S i g m o i d
Revised: S i g m o i d

Original: C on v .
Revised: C on v .

Original: R eL U
Revised: R eL U

Original: C on v .
Revised: C on v .

Original: C on v .
Revised: C on v .

Original: M a x poo li ng + R eL U
Revised: M a x poo li ng + R eL U

Original: C on v .
Revised: C on v .

Original: M a x poo li ng + R eL U
Revised: M a x poo li ng + R eL U

Original: C on v .
Revised: C on v .

Original: M a x poo li ng +
Revised: M a x poo li ng +

Original: R eL U
Revised: R eL U

Original: Policy branch
Revised: Policy branch

Original: F C .
Revised: F C .

Original: Policy C on v
Revised: Policy C on v

Original: L S T M
Revised: L S T M

Original: C on v .
Revised: C on v .

Original: R eL U
Revised: R eL U

Original: F C .
Revised: F C .

Original: Value
Revised: Value

Original: Value branch
Revised: Value branch

Original: S i g m o i d
Revised: S i g m o i d

Original: C on v .
Revised: C on v .

Original: S i g m o i d
Revised: S i g m o i d

Original: C on v .
Revised: C on v .

Original: R eL U
Revised: R eL U

Original: C on v .
Revised: C on v .

Original: R eL U C on v .
Revised: R eL U C on v .

Original: F C .
Revised: F C .

Original: C on v .
Revised: C on v .

Original: M a x poo li ng + R eL U
Revised: M a x poo li ng + R eL U

Original: C on v .
Revised: C on v .

Original: M a x poo li ng + R eL U
Revised: M a x poo li ng + R eL U

Original: C on v .
Revised: C on v .

Original: M a x poo li ng +
Revised: M a x poo li ng +

Original: R eL U
Revised: R eL U

Original: Value branch
Revised: Value branch

Original: Policy branch
Revised: Policy branch

Original: F C .
Revised: F C .

Original: Policy
Revised: Policy

Original: Value
Revised: Value

Original: C on v L S T
Revised: C on v L S T

Original: M
Revised: M

Original: G AP Feature extractor
Revised: G AP Feature extractor

Original: Mask-Attention
Revised: Mask-Attention

Original: [Figure]
Revised: [Figure]

Original: AttentionMechanism
Revised: AttentionMechanism

Original: C on v .
Revised: C on v .

Original: S i g m o i d
Revised: S i g m o i d

Original: C on v .
Revised: C on v .

Original: R eL U
Revised: R eL U

Original: C on v .
Revised: C on v .

Original: R eL U C on v .
Revised: R eL U C on v .

Original: F C .
Revised: F C .

Original: C on v .
Revised: C on v .

Original: M a x poo li ng + R eL U
Revised: M a x poo li ng + R eL U

Original: C on v .
Revised: C on v .

Original: M a x poo li ng + R eL U
Revised: M a x poo li ng + R eL U

Original: C on v .
Revised: C on v .

Original: M a x poo li ng +
Revised: M a x poo li ng +

Original: R eL U
Revised: R eL U

Original: Value branch
Revised: Value branch

Original: Policy branch
Revised: Policy branch

Original: F C .
Revised: F C .

Original: Policy
Revised: Policy

Original: Value
Revised: Value

Original: C on v L S T
Revised: C on v L S T

Original: M
Revised: M

Original: G AP Feature extractor
Revised: G AP Feature extractor

Original: Mask-Attention
Revised: Mask-Attention

Original: Input
Revised: Input

Original: AttentionMechanism
Revised: AttentionMechanism

Original: Mask-Attention Mask-Attention
Revised: Mask-Attention Mask-Attention

Original: Mask-Attention
Revised: Mask-Attention

Original: Value
Revised: Value

Original: Policy
Revised: Policy

Original: [Figure]
Revised: [Figure]

Original: extractor
Revised: extractor

Original: Attentionmechanism Attentionmechanism
Revised: Attentionmechanism Attentionmechanism

Original: [Figure]
Revised: [Figure]

Original: crucial role because the state value is the expected value from current to future states and affects for the future action selections.
Revised: crucial role because the state value is the expected value from current to future states and affects for the future action selections.

Original: However, the existing methods outputs attention maps with respect to the instantaneous action selection.
Revised: However, the existing methods outputs attention maps with respect to the instantaneous action selection.

Original: Our method based on an actor-critic outputs two attention maps from both policy and value branches.
Revised: Our method based on an actor-critic outputs two attention maps from both policy and value branches.

Original: Considering both attention maps, we can understand the basis of an agent’s decision-making in more detail.
Revised: Considering both attention maps, we can understand the basis of an agent’s decision-making in more detail.

Original:  M ASK -A TTENTION A3C
Revised:  M ASK -A TTENTION A3C

Original: We propose Mask-Attention A3C (Mask A3C), that introduces an attention mechanism into A3C, which is an actor-critic-based distributed DRL method.
Revised: We propose Mask-Attention A3C (Mask A3C), that introduces an attention mechanism into A3C, which is an actor-critic-based distributed DRL method.

Original: In Mask A3C, by implementing an attention mechanism for the policy branch and value branch, we can acquire an attention map that expresses the focus area of the network for the output of each branch.
Revised: In Mask A3C, by implementing an attention mechanism for the policy branch and value branch, we can acquire an attention map that expresses the focus area of the network for the output of each branch.

Original: In our study, this attention map is called mask-attention.
Revised: In our study, this attention map is called mask-attention.

Original: By visualizing the mask-attention of each branch, we obtain a visual explanation of the reason for the judgment on the policy and state value.
Revised: By visualizing the mask-attention of each branch, we obtain a visual explanation of the reason for the judgment on the policy and state value.

Original: In addition, our method learns while considering the mask-attention by implementing the attention mechanism, which improves the performance of the agent.
Revised: In addition, our method learns while considering the mask-attention by implementing the attention mechanism, which improves the performance of the agent.

Original: 3. O VERVIEW OF M ASK A3C STRUCTURE
Revised: 3. O VERVIEW OF M ASK A3C STRUCTURE

Original: Figure 1 shows the network structure of the proposed Mask A3C. It consists of a feature extractor, policy branch, value branch, and attention mechanism.
Revised: Figure 1 shows the network structure of the proposed Mask A3C. It consists of a feature extractor, policy branch, value branch, and attention mechanism.

Original: The feature extractor calculates the feature map for the input image using the convolutional layer, the policy branch outputs policies, and the value branch outputs the state value function.
Revised: The feature extractor calculates the feature map for the input image using the convolutional layer, the policy branch outputs policies, and the value branch outputs the state value function.

Original: In A3C, temporal information can be considered by utilizing LSTM, which greatly improves the performance of the agent.
Revised: In A3C, temporal information can be considered by utilizing LSTM, which greatly improves the performance of the agent.

Original: However, LSTM cannot consider the spatial information for the input image, so if it is used in Mask A3C, mask-attention cannot be calculated.
Revised: However, LSTM cannot consider the spatial information for the input image, so if it is used in Mask A3C, mask-attention cannot be calculated.

Original: We therefore employ convolutional LSTM (ConvLSTM)
Revised: We therefore employ convolutional LSTM (ConvLSTM)

Original: (Xingjian et al., 2015), which can considers spatiotemporal information.
Revised: (Xingjian et al., 2015), which can considers spatiotemporal information.

Original: Mask-attention is calculated by applying the sigmoid function and one convolutional layer of 1 × 1 × # of channels to the feature map calculated by the feature extractor.
Revised: Mask-attention is calculated by applying the sigmoid function and one convolutional layer of 1 × 1 × # of channels to the feature map calculated by the feature extractor.

Original: 3. A TTENTION MECHANISM
Revised: 3. A TTENTION MECHANISM

Original: Mask A3C implements an attention mechanism in the policy branch and value branch so that the policy and state value function are learned in consideration of the acquired mask-attention.
Revised: Mask A3C implements an attention mechanism in the policy branch and value branch so that the policy and state value function are learned in consideration of the acquired mask-attention.

Original: The attention mechanism performs mask processing to the feature map of the middle layer in each branch using mask-attention.
Revised: The attention mechanism performs mask processing to the feature map of the middle layer in each branch using mask-attention.

Original: With this mask processing, the area that contributes to the optimum action and state value can be emphasized.
Revised: With this mask processing, the area that contributes to the optimum action and state value can be emphasized.

Original: This mask processing using mask-attention for the feature map is calculated as
Revised: This mask processing using mask-attention for the feature map is calculated as

Original: [Equation]
Revised: [Equation]

Original: where, s t is the state, F ( s t ) is the feature map of the middle layer in each branch, M ( s t ) is maskattention, and F ′ ( s t ) is the feature map after mask processing.
Revised: where, s t is the state, F ( s t ) is the feature map of the middle layer in each branch, M ( s t ) is maskattention, and F ′ ( s t ) is the feature map after mask processing.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: To evaluate the effectiveness of Mask A3C, we conducted an experiment using the game task of
Revised: To evaluate the effectiveness of Mask A3C, we conducted an experiment using the game task of

Original: OpenAI gym (Brockman et al., 2016).
Revised: OpenAI gym (Brockman et al., 2016).

Original: Three games were used: “Breakout”, “Ms.Pac-Man”, and “Space Invaders”.
Revised: Three games were used: “Breakout”, “Ms.Pac-Man”, and “Space Invaders”.

Original: The comparison methods were A3C, Policy Mask A3C , Value Mask A3C, and
Revised: The comparison methods were A3C, Policy Mask A3C , Value Mask A3C, and

Original: Mask A3C, for a total of four patterns.
Revised: Mask A3C, for a total of four patterns.

Original: Policy Mask A3C and Value Mask A3C refer to a Mask
Revised: Policy Mask A3C and Value Mask A3C refer to a Mask

Original: A3C in which the attention mechanism implemented in one side of branch (i.e., policy branch or value branch).
Revised: A3C in which the attention mechanism implemented in one side of branch (i.e., policy branch or value branch).

Original: The learning conditions were 35 for the number of workers, 0 . 

Revised: The learning conditions were 35 for the number of workers, 0 .

Original: for the learning coefﬁcient, and 0 .
Revised: for the learning coefﬁcient, and 0 .

Original: 99 for the discount rate.
Revised: 99 for the discount rate.

Original: The termination condition of learning was when the global steps reached 1 . 

Revised: The termination condition of learning was when the global steps reached 1 .

Original: × 10 8 .
Revised: × 10 8 .

Original: The termination condition of an episode was the end of one play in the game and the case where the number of steps reached 1 . 

Revised: The termination condition of an episode was the end of one play in the game and the case where the number of steps reached 1 .

Original: × 10 4 .
Revised: × 10 4 .

Original: We used the following three evaluation methods.
Revised: We used the following three evaluation methods.

Original: • Visual explanations using mask-attention • Score comparison on the Atari• Score comparison by inverting the gaze area of mask-attention
Revised: • Visual explanations using mask-attention • Score comparison on the Atari• Score comparison by inverting the gaze area of mask-attention

Original: 4. I MPLEMENTATION DETAILS
Revised: 4. I MPLEMENTATION DETAILS

Original: The input was a grayscale image of the game screen and the output was the action in each game.
Revised: The input was a grayscale image of the game screen and the output was the action in each game.

Original: The image used as input was resized to 80 × 80 .
Revised: The image used as input was resized to 80 × 80 .

Original: The output dimension of feature extractor was 32-dimensional for the ﬁrst two convolutional layers and 64-dimensional for the remaining one convolutional layer.
Revised: The output dimension of feature extractor was 32-dimensional for the ﬁrst two convolutional layers and 64-dimensional for the remaining one convolutional layer.

Original: For the hidden state in ConvLSTM, the output dimension was 64-dimensional.
Revised: For the hidden state in ConvLSTM, the output dimension was 64-dimensional.

Original: The policy branch consisted of one convolutional layer, one fully connected layer, and a softmax function.
Revised: The policy branch consisted of one convolutional layer, one fully connected layer, and a softmax function.

Original: The output dimension of the convolution layer were 32 dimensional and the number of output units of the fully connected layer was the number of actions in each game.
Revised: The output dimension of the convolution layer were 32 dimensional and the number of output units of the fully connected layer was the number of actions in each game.

Original: The value branch consisted of one convolutional layer with 32-dimensional output dimension and one fully connected layer with the number of output units of 1.
Revised: The value branch consisted of one convolutional layer with 32-dimensional output dimension and one fully connected layer with the number of output units of 1.

Original: The network structure of A3C in this experiments was the structure excluded the attention mechanism from Mask A3C.
Revised: The network structure of A3C in this experiments was the structure excluded the attention mechanism from Mask A3C.

Original: 4. V ISUAL E XPLANATIONS USING M ASK -A TTENTION
Revised: 4. V ISUAL E XPLANATIONS USING M ASK -A TTENTION

Original: Figures 2 and 3 show visualization examples of mask-attention in Atari 2600.
Revised: Figures 2 and 3 show visualization examples of mask-attention in Atari 2600.

Original: (Examples of other frames and other environment are available in Appendix A.)
Revised: (Examples of other frames and other environment are available in Appendix A.)

Original: Hereafter, we discuss the obtained mask-attentions shown in Figs.
Revised: Hereafter, we discuss the obtained mask-attentions shown in Figs.

Original: 2 and 3.
Revised: 2 and 3.

Original: Breakout
Revised: Breakout

Original: Breakout is a game in which the player hits the ball with a paddle to destroy various block.
Revised: Breakout is a game in which the player hits the ball with a paddle to destroy various block.

Original: The actions of the agent that is the paddle are “Noop”, “Left”, and “Right”.
Revised: The actions of the agent that is the paddle are “Noop”, “Left”, and “Right”.

Original: As shown in ﬁgure 2(a), the agent in Frame 1 attended to the traveling direction of the ball.
Revised: As shown in ﬁgure 2(a), the agent in Frame 1 attended to the traveling direction of the ball.

Original: In Frame 2, the agent attended to the right side of the paddle, and the action was “Right”.
Revised: In Frame 2, the agent attended to the right side of the paddle, and the action was “Right”.

Original: Frame 3 after the ball was hit back shows no gaze area.
Revised: Frame 3 after the ball was hit back shows no gaze area.

Original: From these results, we can see that the agent controlled the paddle according to the traveling direction of the ball.
Revised: From these results, we can see that the agent controlled the paddle according to the traveling direction of the ball.

Original: In ﬁgure 3(a), the agent of Point 1 was attending to the entire block, and we can see from the graph that the state value increased with the destruction of the blocks.
Revised: In ﬁgure 3(a), the agent of Point 1 was attending to the entire block, and we can see from the graph that the state value increased with the destruction of the blocks.

Original: In Point 2, the ball reached the top of the block and destroyed many blocks.
Revised: In Point 2, the ball reached the top of the block and destroyed many blocks.

Original: At that time, the gaze area of mask-attention was signiﬁcantly reduced according to the blocks.
Revised: At that time, the gaze area of mask-attention was signiﬁcantly reduced according to the blocks.

Original: After Point 2, the state value decreased according to the number of blocks that were destroyed.
Revised: After Point 2, the state value decreased according to the number of blocks that were destroyed.

Original: These results demonstrate that the agent recognized the blocks as the score source.
Revised: These results demonstrate that the agent recognized the blocks as the score source.

Original: Ms.Pac-Man
Revised: Ms.Pac-Man

Original: Ms.Pac-Man is a game in which the player collects scattered cookies while avoiding enemies.
Revised: Ms.Pac-Man is a game in which the player collects scattered cookies while avoiding enemies.

Original: The actions of the agent that is the Pac-Man are “Noop”,“Up”,“Down”,“Left”,“Right”,“Up
Revised: The actions of the agent that is the Pac-Man are “Noop”,“Up”,“Down”,“Left”,“Right”,“Up

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: + Left”,“Up + Right”,“Down + Left”, and “Down + Right”.
Revised: + Left”,“Up + Right”,“Down + Left”, and “Down + Right”.

Original: In ﬁgure 2(b), the agent of Framewas attending around Pac-Man.
Revised: In ﬁgure 2(b), the agent of Framewas attending around Pac-Man.

Original: The agent of Frame 2 was attending to the cookies remaining on the screen.
Revised: The agent of Frame 2 was attending to the cookies remaining on the screen.

Original: In Frame 3, Pac-Man moved to the gazed area of Frame 2 and acquired the cookie.
Revised: In Frame 3, Pac-Man moved to the gazed area of Frame 2 and acquired the cookie.

Original: These
Revised: These

Original: [Figure]
Revised: [Figure]

Original: results demonstrate that the agent controlled Pac-Man toward the cookies while simultaneously attending to the surroundings of Pac-Man.
Revised: results demonstrate that the agent controlled Pac-Man toward the cookies while simultaneously attending to the surroundings of Pac-Man.

Original: In ﬁgure 3(b), Point 1, which was the beginning of the game, was attending at the entire screen.
Revised: In ﬁgure 3(b), Point 1, which was the beginning of the game, was attending at the entire screen.

Original: In contrast, Point 2 is reduced the gaze area according to the decreases of cookies.
Revised: In contrast, Point 2 is reduced the gaze area according to the decreases of cookies.

Original: Also, from Point 1 to Point 2, the state value decreased as the cookies on the screen decreased.
Revised: Also, from Point 1 to Point 2, the state value decreased as the cookies on the screen decreased.

Original: These results demonstrate that the agent recognized the cookies as the score source.
Revised: These results demonstrate that the agent recognized the cookies as the score source.

Original: Space Invaders
Revised: Space Invaders

Original: Space Invaders is a shooting game in which the player repels the enemy invaders.
Revised: Space Invaders is a shooting game in which the player repels the enemy invaders.

Original: The actions of the agent that is the cannon are “Noop”,“Left”,“Right”,“Attack”,“Left + Attack”, and “Right + Attack”.
Revised: The actions of the agent that is the cannon are “Noop”,“Left”,“Right”,“Attack”,“Left + Attack”, and “Right + Attack”.

Original: In ﬁgure 2(c), the agent in Frame 1 is attended to the invaders and the action was ”Attack”.
Revised: In ﬁgure 2(c), the agent in Frame 1 is attended to the invaders and the action was ”Attack”.

Original: In Frame 2, we can see that the beam in Frame 1 was heading toward the invader, and in Frame 3, the beam was repelling the invaders that were attended at in Frame 1.
Revised: In Frame 2, we can see that the beam in Frame 1 was heading toward the invader, and in Frame 3, the beam was repelling the invaders that were attended at in Frame 1.

Original: In all frames, the agent attended around itself while avoiding the defensive walls.
Revised: In all frames, the agent attended around itself while avoiding the defensive walls.

Original: From these results, we can see that the agent repelled the invaders while simultaneously avoiding the defensive wall.
Revised: From these results, we can see that the agent repelled the invaders while simultaneously avoiding the defensive wall.

Original: In ﬁgure 3(c), the agent in Point 1 was attending to the all of the invaders and the agent in Point 2 was shrinking the gaze area according to the decreasing number of invaders.
Revised: In ﬁgure 3(c), the agent in Point 1 was attending to the all of the invaders and the agent in Point 2 was shrinking the gaze area according to the decreasing number of invaders.

Original: In addition, from Point 1 to Point 2, the state value is decreased according to the number of invaders.
Revised: In addition, from Point 1 to Point 2, the state value is decreased according to the number of invaders.

Original: These results demonstrate that the agent recognized the invaders as the score source.
Revised: These results demonstrate that the agent recognized the invaders as the score source.

Original: Discussion
Revised: Discussion

Original: We obtained different mask-attentions in the policy and state value by implementing an attention mechanism for each branch.
Revised: We obtained different mask-attentions in the policy and state value by implementing an attention mechanism for each branch.

Original: The policy represents the probability distribution of the action selection in the current state.
Revised: The policy represents the probability distribution of the action selection in the current state.

Original: Therefore, the mask-attention of the policy indicates the area
Revised: Therefore, the mask-attention of the policy indicates the area

Original: [Table]
Revised: [Table]

Original: that contributes to the action of the agent.
Revised: that contributes to the action of the agent.

Original: The state value represents the expected value of the return in the current state.
Revised: The state value represents the expected value of the return in the current state.

Original: Here, return is the sum of rewards during one episode.
Revised: Here, return is the sum of rewards during one episode.

Original: Therefore, the mask-attention of the state value indicates the area that represents the property of the game.
Revised: Therefore, the mask-attention of the state value indicates the area that represents the property of the game.

Original: 4. S CORE COMPARISON ON THE A TARI
Revised: 4. S CORE COMPARISON ON THE A TARI

Original: Table 1 shows the max and mean scores over 100 episodes on the Atari 2600.
Revised: Table 1 shows the max and mean scores over 100 episodes on the Atari 2600.

Original: As we can see, the mean score in Breakout was lower Policy Mask A3C, Value Mask A3C, and Mask A3C than the
Revised: As we can see, the mean score in Breakout was lower Policy Mask A3C, Value Mask A3C, and Mask A3C than the

Original: A3C.
Revised: A3C.

Original: In contrast, the max score in Breakout was 864 for all methods.
Revised: In contrast, the max score in Breakout was 864 for all methods.

Original: This score is the best score that can be obtained in Breakout.
Revised: This score is the best score that can be obtained in Breakout.

Original: Breakout is a simple game with no external factors: it simply consists of the player hitting the ball back with a paddle.
Revised: Breakout is a simple game with no external factors: it simply consists of the player hitting the ball back with a paddle.

Original: Therefore, we are consider that A3C
Revised: Therefore, we are consider that A3C

Original: and
Revised: and

Original: Mask A3C got the same score.
Revised: Mask A3C got the same score.

Original: In Ms.Pac-Man, the max and mean scores of Policy Mask A3C and
Revised: In Ms.Pac-Man, the max and mean scores of Policy Mask A3C and

Original: Mask A3C improved compared to those of A3C.
Revised: Mask A3C improved compared to those of A3C.

Original: The control of agent in Ms.Pac-Man is complex because it needs to select an action while considering external factors(e.g., the enemy).
Revised: The control of agent in Ms.Pac-Man is complex because it needs to select an action while considering external factors(e.g., the enemy).

Original: Policy
Revised: Policy

Original: Mask A3C and Mask A3C, which implements an attention mechanism on the policy branch, can emphasize the areas that contribute to the action (e.g., cookies and enemies), which is why Policy
Revised: Mask A3C and Mask A3C, which implements an attention mechanism on the policy branch, can emphasize the areas that contribute to the action (e.g., cookies and enemies), which is why Policy

Original: Mask A3C and Mask A3C obtained a higher score than A3C.
Revised: Mask A3C and Mask A3C obtained a higher score than A3C.

Original: In Space Invaders, Policy Mask A3C and Mask A3C improved the max and mean scores compared to A3C. Also, the max and mean scores of Value Mask A3C is almost the same as the A3C.
Revised: In Space Invaders, Policy Mask A3C and Mask A3C improved the max and mean scores compared to A3C. Also, the max and mean scores of Value Mask A3C is almost the same as the A3C.

Original: The agent in Space Invaders needs to select actions in consideration of external factors (e.g, enemies), the same as in Ms.Pac-Man.
Revised: The agent in Space Invaders needs to select actions in consideration of external factors (e.g, enemies), the same as in Ms.Pac-Man.

Original: Policy Mask A3C and Mask A3C, which implements an attention mechanism in the policy branch, can emphasize the areas that contribute to the action (e.g, defensive walls and invaders), which is why Policy Mask A3C and Mask A3C obtained a higher score than A3C. There is no signiﬁcant difference between the scores of Beamrider.
Revised: Policy Mask A3C and Mask A3C, which implements an attention mechanism in the policy branch, can emphasize the areas that contribute to the action (e.g, defensive walls and invaders), which is why Policy Mask A3C and Mask A3C obtained a higher score than A3C. There is no signiﬁcant difference between the scores of Beamrider.

Original: In Beamrider, two kinds of enemies are exists from the an agent’s point of view.
Revised: In Beamrider, two kinds of enemies are exists from the an agent’s point of view.

Original: One is an enemy that the agent should defeat, and the other is that the agent should avoid collisions.
Revised: One is an enemy that the agent should defeat, and the other is that the agent should avoid collisions.

Original: These enemies look like similar appearance.
Revised: These enemies look like similar appearance.

Original: Because our attention-mask highlighting the enemies and an attention mechanism are insufﬁcient to distinct these enemies, these do not contribute the score improvement.
Revised: Because our attention-mask highlighting the enemies and an attention mechanism are insufﬁcient to distinct these enemies, these do not contribute the score improvement.

Original: In Fishing Derby, Policy Mask A3C, Value Mask A3C, and Mask A3C improved the mean scores compared to A3C. In Fishing Derby, there are many ﬁsh that are score source, and getting the closest ﬁsh from the player is the fastest way to earn score.
Revised: In Fishing Derby, Policy Mask A3C, Value Mask A3C, and Mask A3C improved the mean scores compared to A3C. In Fishing Derby, there are many ﬁsh that are score source, and getting the closest ﬁsh from the player is the fastest way to earn score.

Original: Policy Mask A3C, Value Mask A3C, and Mask A3C, which implements an attention mechanism, can emphasize the ﬁsh closest to the player.
Revised: Policy Mask A3C, Value Mask A3C, and Mask A3C, which implements an attention mechanism, can emphasize the ﬁsh closest to the player.

Original: This is the reason why the Mask-attention methods obtained a higher score than that of A3C. In Seaquest, Mask A3C improves both the maximum and average scores compared to the other methods.
Revised: This is the reason why the Mask-attention methods obtained a higher score than that of A3C. In Seaquest, Mask A3C improves both the maximum and average scores compared to the other methods.

Original: This result is because only the Mask A3C agent could learn the action of replenishing oxygen.
Revised: This result is because only the Mask A3C agent could learn the action of replenishing oxygen.

Original: Seaquest is an oxygen gauge at the bottom of the screen, and the game ends when the oxygen is gone.
Revised: Seaquest is an oxygen gauge at the bottom of the screen, and the game ends when the oxygen is gone.

Original: Mask A3C, which implements an attention mechanism in policy and value branch, can emphasize the oxygen gauge, which is why Mask A3C obtained a higher score than other methods.
Revised: Mask A3C, which implements an attention mechanism in policy and value branch, can emphasize the oxygen gauge, which is why Mask A3C obtained a higher score than other methods.

Original: 4. C OMPARISON OF SCORES BY INVERTING GAZE AREA IN MASK - ATTENTION
Revised: 4. C OMPARISON OF SCORES BY INVERTING GAZE AREA IN MASK - ATTENTION

Original: In visually explaining the decision making of an agent using Mask A3C, we want to verify whether mask-attention represents an effective gaze area for the optimum action.
Revised: In visually explaining the decision making of an agent using Mask A3C, we want to verify whether mask-attention represents an effective gaze area for the optimum action.

Original: In case that the obtained game score by inverting mask-attention does not change, the mask-attention does not contribute for acquired agent’s action.
Revised: In case that the obtained game score by inverting mask-attention does not change, the mask-attention does not contribute for acquired agent’s action.

Original: In contrast, if the game score signiﬁcantly decreases, the mask-attention largely contributes for the actions acquiring the game score.
Revised: In contrast, if the game score signiﬁcantly decreases, the mask-attention largely contributes for the actions acquiring the game score.

Original: For this veriﬁcation method, we create a map in which the gaze area of the mask-attention in the policy branch is inverted and then calculate the score on Atari 2600 when the created map is used for the attention mechanism.
Revised: For this veriﬁcation method, we create a map in which the gaze area of the mask-attention in the policy branch is inverted and then calculate the score on Atari 2600 when the created map is used for the attention mechanism.

Original: We determine whether mask-attention is effective for the visual explanation of an action by comparing the scores
Revised: We determine whether mask-attention is effective for the visual explanation of an action by comparing the scores

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: the gaze area is inverted and when it is not inverted.
Revised: the gaze area is inverted and when it is not inverted.

Original: The map in which the gaze area of the mask-attention is inverted is created by
Revised: The map in which the gaze area of the mask-attention is inverted is created by

Original: [Equation]
Revised: [Equation]

Original: where, s t is the state (grayscale image in the experiment), M ( s t ) is mask-attention, and M inverse ( s t ) is the map after inverting the gaze area of mask-attention.
Revised: where, s t is the state (grayscale image in the experiment), M ( s t ) is mask-attention, and M inverse ( s t ) is the map after inverting the gaze area of mask-attention.

Original: Table 2 shows the score comparison by inverting the gaze area of mask-attention.
Revised: Table 2 shows the score comparison by inverting the gaze area of mask-attention.

Original: (Also, the tableshows for the decrease ratio from the normal score to the inverse score.)
Revised: (Also, the tableshows for the decrease ratio from the normal score to the inverse score.)

Original: As shown in table 2, the inverse score was signiﬁcantly lower than the normal score in all games.
Revised: As shown in table 2, the inverse score was signiﬁcantly lower than the normal score in all games.

Original: In addition, the inverse score in Breakout was equivalent to a random score, and the inverse score of Mask A3C in Ms.Pac-Man was 53.4 lower than random.
Revised: In addition, the inverse score in Breakout was equivalent to a random score, and the inverse score of Mask A3C in Ms.Pac-Man was 53.4 lower than random.

Original: In contrast, we can see that the inverse score in Space Invaders, Beamrider, Fishing Derby and Seaquest was higher than a random score.
Revised: In contrast, we can see that the inverse score in Space Invaders, Beamrider, Fishing Derby and Seaquest was higher than a random score.

Original: However, from Table 3, we can see that the decrease ratio in mean of Space Invaders, Beamrider,
Revised: However, from Table 3, we can see that the decrease ratio in mean of Space Invaders, Beamrider,

Original: Fishing Derby, and Seaquest was more than 90%, as with the other games.
Revised: Fishing Derby, and Seaquest was more than 90%, as with the other games.

Original: Therefore, we conclude that the gaze area of mask-attention in the policy branch can represents a useful area for action to obtain a high score.
Revised: Therefore, we conclude that the gaze area of mask-attention in the policy branch can represents a useful area for action to obtain a high score.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we proposed Mask-Attention A3C (Mask A3C) that introduces an attention mechanism into Asynchronous Advantage Actor-Critic (A3C).
Revised: In this paper, we proposed Mask-Attention A3C (Mask A3C) that introduces an attention mechanism into Asynchronous Advantage Actor-Critic (A3C).

Original: In Mask A3C, we acquired a mask-attention that expresses the important area for the policy and state value by implementing an attention mechanism in the policy and value branches.
Revised: In Mask A3C, we acquired a mask-attention that expresses the important area for the policy and state value by implementing an attention mechanism in the policy and value branches.

Original: This enables a visual explanation of the judgment reason in the decision-making of the agent, from the two viewpoints of policy and state value, by visualizing mask-attention.
Revised: This enables a visual explanation of the judgment reason in the decision-making of the agent, from the two viewpoints of policy and state value, by visualizing mask-attention.

Original: We also emphasize which areas contribute to the optimal action and state value by implementing an attention mechanism, and improves the performance of the agent.
Revised: We also emphasize which areas contribute to the optimal action and state value by implementing an attention mechanism, and improves the performance of the agent.

Original: Experiments with the Atari 2600 conﬁrmed the acquisition of different mask-attentions in the policy and value branches.
Revised: Experiments with the Atari 2600 conﬁrmed the acquisition of different mask-attentions in the policy and value branches.

Original: The results demonstrate that the the mask-attention of the policy branch indicates the area that contributes to action while that of the value branch indicates the area that expresses the property of the game.
Revised: The results demonstrate that the the mask-attention of the policy branch indicates the area that contributes to action while that of the value branch indicates the area that expresses the property of the game.

Original: We provided a useful analysis for the decision-making of agents in game tasks from the two viewpoints of policy and state value by visualizing these mask-attentions.
Revised: We provided a useful analysis for the decision-making of agents in game tasks from the two viewpoints of policy and state value by visualizing these mask-attentions.

Original: A comparison of game scores showed that the score improved when the attention mechanism was implemented in the policy branch.
Revised: A comparison of game scores showed that the score improved when the attention mechanism was implemented in the policy branch.

Original: However, our experiments were conducted with game tasks that are easy to analyze visually.
Revised: However, our experiments were conducted with game tasks that are easy to analyze visually.

Original: Our future work will entail the visual analysis of agents using maskattention is a future work for complex tasks (e.g., robot control and autonomous driving).
Revised: Our future work will entail the visual analysis of agents using maskattention is a future work for complex tasks (e.g., robot control and autonomous driving).

Original: A A PPENDIX : A DDITIONAL MASK - ATTENTION EXAMPLES Mask-attentions in policy
Revised: A A PPENDIX : A DDITIONAL MASK - ATTENTION EXAMPLES Mask-attentions in policy

Original: Figures 4, 5, 6, 7, 8, and 9 show mask-attentions in policy of each environment.
Revised: Figures 4, 5, 6, 7, 8, and 9 show mask-attentions in policy of each environment.

Original: [Figure]
Revised: [Figure]

Original: Figures 10, 11, and 12 show mask-attentions in state value.
Revised: Figures 10, 11, and 12 show mask-attentions in state value.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Transitions of Mask-attentions in policy and state value
Revised: Transitions of Mask-attentions in policy and state value

Original: Figures 13, 14, 15, 16, 17, and 18 show the in
Revised: Figures 13, 14, 15, 16, 17, and 18 show the in

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: PolicyAtt
Revised: PolicyAtt

Original: [Figure]
Revised: [Figure]

Original: Frame Frame
Revised: Frame Frame

Original: Frame Frame
Revised: Frame Frame

Original: [Figure]
Revised: [Figure]

Original: Frame Frame
Revised: Frame Frame

Original: Frame Frame
Revised: Frame Frame

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: Figures 19 and 20 show the Mask-attention in policy and value for the new state. In this experiment, we conﬁrm the reaction of the attention maps by artiﬁcially added new states (i

Original: [Figure]
Revised: [Figure]

Revised: e., ﬁsh and oxygen) that affect the agent’s policy and state

Original: [Figure] [Figure]
Revised: [Figure] [Figure]