Original: L ATENT FORWARD MODEL FOR R EAL - TIME S TRATEGY GAME PLANNING WITH INCOMPLETE INFORMATION
Revised: L ATENT FORWARD MODEL FOR R EAL - TIME S TRATEGY GAME PLANNING WITH INCOMPLETE INFORMATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Model-free deep reinforcement learning approaches have shown superhuman performance in simulated environments (e g., Atari games, Go, etc).
Revised: Model-free deep reinforcement learning approaches have shown superhuman performance in simulated environments (e g., Atari games, Go, etc).

Original: During training, these approaches often implicitly construct a latent space that contains key information for decision making.
Revised: During training, these approaches often implicitly construct a latent space that contains key information for decision making.

Original: In this paper, we learn a forward model on this latent space and apply it to model-based planning in a miniature Real-time Strategy game with incomplete information (MiniRTS [Tian et al.
Revised: In this paper, we learn a forward model on this latent space and apply it to model-based planning in a miniature Real-time Strategy game with incomplete information (MiniRTS [Tian et al.

Original: We ﬁrst show that the latent space constructed from existing actor-critic models contains relevant information of the game, and design training procedure to learn forward models.
Revised: We ﬁrst show that the latent space constructed from existing actor-critic models contains relevant information of the game, and design training procedure to learn forward models.

Original: We also show that our learned forward model can predict meaningful future state and is usable for latent space Monte-Carlo Tree Search (MCTS), in terms of win rates against rule-based agents.
Revised: We also show that our learned forward model can predict meaningful future state and is usable for latent space Monte-Carlo Tree Search (MCTS), in terms of win rates against rule-based agents.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Model-free deep reinforcement learning (DRL) approaches (e.g., deep Q-learning [Mnih et al.
Revised: Model-free deep reinforcement learning (DRL) approaches (e.g., deep Q-learning [Mnih et al.

Original: (2013)], DDPG [Lillicrap et al.
Revised: (2013)], DDPG [Lillicrap et al.

Original: A3C
Revised: A3C

Original: [Mnih et al.
Revised: [Mnih et al.

Original: (2016)], etc) have been applied extensively in many simulated environments with complete information and relatively simple game dynamics (e.g., Atari games, Go [Silver et al.
Revised: (2016)], etc) have been applied extensively in many simulated environments with complete information and relatively simple game dynamics (e.g., Atari games, Go [Silver et al.

Original: (2016; 2017)], Doom, etc).
Revised: (2016; 2017)], Doom, etc).

Original: The learned agent, which acts reactively based on the current game situation, can even achieve superhuman performance.
Revised: The learned agent, which acts reactively based on the current game situation, can even achieve superhuman performance.

Original: However, for complicated environments, planning ahead (or “predicting the future”) before making an actual decision is important.
Revised: However, for complicated environments, planning ahead (or “predicting the future”) before making an actual decision is important.

Original: Such a planning procedure requires a forward model that estimates the next state s t +1 given the current state s t and action a t , which is in general non-trivial to construct and estimate from the high-dimensional raw input.
Revised: Such a planning procedure requires a forward model that estimates the next state s t +1 given the current state s t and action a t , which is in general non-trivial to construct and estimate from the high-dimensional raw input.

Original: For partially observable environments (e.g.,
Revised: For partially observable environments (e.g.,

Original: Real-time Strategy Games like StarCraft), constructing a forward model is more difﬁcult even with a perfect domain knowledge of the game, due to the deliberate concealing of information and the additional requirement to capture the belief of the unknown for the agent.
Revised: Real-time Strategy Games like StarCraft), constructing a forward model is more difﬁcult even with a perfect domain knowledge of the game, due to the deliberate concealing of information and the additional requirement to capture the belief of the unknown for the agent.

Original: A natural question now arises.
Revised: A natural question now arises.

Original: Could we borrow the success of model-free approach to learn a forward model?
Revised: Could we borrow the success of model-free approach to learn a forward model?

Original: Note that in model-free approaches, a single shared network (called “trunk“) is often used to extract features from the input game situation to obtain a latent representation.
Revised: Note that in model-free approaches, a single shared network (called “trunk“) is often used to extract features from the input game situation to obtain a latent representation.

Original: From the latent space, multiple reinforcement learning quantities ( Q -function, value function V , advantage function A , etc) are predicted via simple linear transformations and used for decision making.
Revised: From the latent space, multiple reinforcement learning quantities ( Q -function, value function V , advantage function A , etc) are predicted via simple linear transformations and used for decision making.

Original: Strong performance of these approaches indicates that the learned latent space must have captured key ingredients of the input situation and remains low-dimensional.
Revised: Strong performance of these approaches indicates that the learned latent space must have captured key ingredients of the input situation and remains low-dimensional.

Original: Therefore, it is an excellent candidate for the state representation of a forward model.
Revised: Therefore, it is an excellent candidate for the state representation of a forward model.

Original: In this paper, we study whether it is possible to use the latent space learned by model-free approaches to construct forward models.
Revised: In this paper, we study whether it is possible to use the latent space learned by model-free approaches to construct forward models.

Original: We use MiniRTS [Tian et al.
Revised: We use MiniRTS [Tian et al.

Original: (2017)], an efﬁcient and simple twoplayer Real-time Strategy (RTS) game.
Revised: (2017)], an efﬁcient and simple twoplayer Real-time Strategy (RTS) game.

Original: MiniRTS captures the basic dynamics of its kind: the agent builds units (workers and troops) that consume resources, gathers resources, explores regions out of sights (“fog of war”), defends enemy’s attack, and invades enemy’s base.
Revised: MiniRTS captures the basic dynamics of its kind: the agent builds units (workers and troops) that consume resources, gathers resources, explores regions out of sights (“fog of war”), defends enemy’s attack, and invades enemy’s base.

Original: This game is incomplete information, because the agent can only see within its sight, and does not know the action of its opponent by default.
Revised: This game is incomplete information, because the agent can only see within its sight, and does not know the action of its opponent by default.

Original: Rather than unit based control as in [Synnaeve et al.
Revised: Rather than unit based control as in [Synnaeve et al.

Original: ; Usunier et al.
Revised: ; Usunier et al.

Original: (2016); Vinyals et al.
Revised: (2016); Vinyals et al.

Original: (2017)], the agent uses 9 discrete actions to control the overall strategy (e.g., build a particular kind of troops, attack or defend).
Revised: (2017)], the agent uses 9 discrete actions to control the overall strategy (e.g., build a particular kind of troops, attack or defend).

Original: Our contributions are three-fold: First, we propose to study the relationship between the latent space learned by model-free approaches and the state representation of forward models.
Revised: Our contributions are three-fold: First, we propose to study the relationship between the latent space learned by model-free approaches and the state representation of forward models.

Original: Very few works (e.g, DARLA [Higgins et al.
Revised: Very few works (e.g, DARLA [Higgins et al.

Original: (2017)], DQN [Mnih et al.
Revised: (2017)], DQN [Mnih et al.

Original: in model-free RL study these properties in depth, let alone using the latent state in model-based approaches for incomplete information game.
Revised: in model-free RL study these properties in depth, let alone using the latent state in model-based approaches for incomplete information game.

Original: To our knowledge, we are one of the ﬁrst works to explore such directions.
Revised: To our knowledge, we are one of the ﬁrst works to explore such directions.

Original: Second, we improve the performance of model-based agent in MiniRTS by input feature design and show that the latent space learned from actor-critic models [Mnih et al.
Revised: Second, we improve the performance of model-based agent in MiniRTS by input feature design and show that the latent space learned from actor-critic models [Mnih et al.

Original: (2016)] can reconstruct critical information of the game, e.g., Hit Point of the base and available resources.
Revised: (2016)] can reconstruct critical information of the game, e.g., Hit Point of the base and available resources.

Original: Finally, we propose novel algorithms that learn a forward model that maps a latent state h t to its future counterpart h t (cid:48)
Revised: Finally, we propose novel algorithms that learn a forward model that maps a latent state h t to its future counterpart h t (cid:48)

Original: ( t (cid:48) > t ) with reduced drifting.
Revised: ( t (cid:48) > t ) with reduced drifting.

Original: Such a forward model enables us to use model-based planning such as Monte-Carlo
Revised: Such a forward model enables us to use model-based planning such as Monte-Carlo

Original: Tree Search (MCTS) in incomplete information games.
Revised: Tree Search (MCTS) in incomplete information games.

Original: We show positive performance (8% higher than random planning) in terms of win rates against rule-based agents.
Revised: We show positive performance (8% higher than random planning) in terms of win rates against rule-based agents.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Forward modeling Model-based approaches are one of the standard ways to model complicated yet physically calibrated robots [Grizzle et al.
Revised: Forward modeling Model-based approaches are one of the standard ways to model complicated yet physically calibrated robots [Grizzle et al.

Original: In these cases, it is assumed, or known by design, that a forward model can be obtained easily.
Revised: In these cases, it is assumed, or known by design, that a forward model can be obtained easily.

Original: Furthermore, the state representation of the forward model is usually the internal state of the system (e.g., position and velocity of each robot joint), and is thus low-dimensional.
Revised: Furthermore, the state representation of the forward model is usually the internal state of the system (e.g., position and velocity of each robot joint), and is thus low-dimensional.

Original: On the other hand, learning a forward model from a complicated and high-dimensional observation is in general difﬁcult, even with the current deep learning architecture.
Revised: On the other hand, learning a forward model from a complicated and high-dimensional observation is in general difﬁcult, even with the current deep learning architecture.

Original: Computer vision researchers try to predict the next video frame given the last few frames [Mathieu et al.
Revised: Computer vision researchers try to predict the next video frame given the last few frames [Mathieu et al.

Original: (2015); van Amersfoort et al.
Revised: (2015); van Amersfoort et al.

Original: To reduce the complexity, one natural idea is to project the high-dimensional input to low-dimensional state that captures the important aspect of the situation, on which a forward model might be easier to learn.
Revised: To reduce the complexity, one natural idea is to project the high-dimensional input to low-dimensional state that captures the important aspect of the situation, on which a forward model might be easier to learn.

Original: [Agrawal et al.
Revised: [Agrawal et al.

Original: ] learns a forward model directly from visual input and uses that for manipulation.
Revised: ] learns a forward model directly from visual input and uses that for manipulation.

Original: To make their learned latent state nontrivial, a regularization term is applied to ensure the learned representation can reconstruct the input as much as possible.
Revised: To make their learned latent state nontrivial, a regularization term is applied to ensure the learned representation can reconstruct the input as much as possible.

Original: In comparison, we do not need a regularization since the latent space is pre-trained by model-free approaches.
Revised: In comparison, we do not need a regularization since the latent space is pre-trained by model-free approaches.

Original: One issue in forward modeling is that its accumulative prediction over many steps might drift and is not usable for planning.
Revised: One issue in forward modeling is that its accumulative prediction over many steps might drift and is not usable for planning.

Original: [Weber et al.
Revised: [Weber et al.

Original: (2017)] uses forward model prediction as another feature for a model-free approach, while Bansal et al.
Revised: (2017)] uses forward model prediction as another feature for a model-free approach, while Bansal et al.

Original: (2017) uses forward model as a way to guide exploration for model-free procedure.
Revised: (2017) uses forward model as a way to guide exploration for model-free procedure.

Original: In this paper, we use multi-step long-term prediction to stabilize the forward model.
Revised: In this paper, we use multi-step long-term prediction to stabilize the forward model.

Original: Real-time strategy games.
Revised: Real-time strategy games.

Original: Most works on Real-time Strategy (RTS) games assume that the agents have access to complete information, in which the behaviors of both the player and the opponents are observed.
Revised: Most works on Real-time Strategy (RTS) games assume that the agents have access to complete information, in which the behaviors of both the player and the opponents are observed.

Original: Micro-management is often viewed as a standard RL task, addressed by model-free approaches [Usunier et al.
Revised: Micro-management is often viewed as a standard RL task, addressed by model-free approaches [Usunier et al.

Original: ; Peng et al.
Revised: ; Peng et al.

Original: On the other hand, Monte-Carlo Tree Search (MCTS) has been applied (e.g, ABCD [Boutilier et al.
Revised: On the other hand, Monte-Carlo Tree Search (MCTS) has been applied (e.g, ABCD [Boutilier et al.

Original: (1997)], MCTSCD [Soemers (2014)]), which focus on concurrent planning with perfect information and perfect forward model.
Revised: (1997)], MCTSCD [Soemers (2014)]), which focus on concurrent planning with perfect information and perfect forward model.

Original: [Uriarte & Ontan´on (2015)] learns a short-term shallow forward model for local combat, but again assuming full access to the game.
Revised: [Uriarte & Ontan´on (2015)] learns a short-term shallow forward model for local combat, but again assuming full access to the game.

Original: Finally, [Butler & Demiris (2010)] deals with partial observability, but focuses particularly on scouting, rather than global strategies as we do.
Revised: Finally, [Butler & Demiris (2010)] deals with partial observability, but focuses particularly on scouting, rather than global strategies as we do.

Original: Grouping unit-based actions into macro (or strategic) actions is an open and challenging topic in RTS games.
Revised: Grouping unit-based actions into macro (or strategic) actions is an open and challenging topic in RTS games.

Original: Following [Tian et al.
Revised: Following [Tian et al.

Original: , we use 9 pre-deﬁned discrete strategic actions to facilitate training.
Revised: , we use 9 pre-deﬁned discrete strategic actions to facilitate training.

Original: Other recent works also model macro strategies from replays [Justesen & Risi (2017)] or from action scripts [Barriga et al.
Revised: Other recent works also model macro strategies from replays [Justesen & Risi (2017)] or from action scripts [Barriga et al.

Original: (2017)], both with deep models.
Revised: (2017)], both with deep models.

Original:  M ETHODS
Revised:  M ETHODS

Original: In model-free reinforcement learning, an agent learns to maximize its long-term reward, either from previous experience in the same environment (e.g. replay buffer), or from its own interaction with the environment with its behavior.
Revised: In model-free reinforcement learning, an agent learns to maximize its long-term reward, either from previous experience in the same environment (e.g. replay buffer), or from its own interaction with the environment with its behavior.

Original: In both cases, the agent does not need to have knowledge of the environment, nor know how the world changes after an action is taken, hence called model-free approach.
Revised: In both cases, the agent does not need to have knowledge of the environment, nor know how the world changes after an action is taken, hence called model-free approach.

Original: On the other hand, if an agent learns to build a model to predict how the state (its own state and/or environment state
Revised: On the other hand, if an agent learns to build a model to predict how the state (its own state and/or environment state

Original: ) changes after its action, then the agent is considered to take a model based approach
Revised: ) changes after its action, then the agent is considered to take a model based approach

Original: In this paper, we ﬁrst use an off-policy version of Batch A3C
Revised: In this paper, we ﬁrst use an off-policy version of Batch A3C

Original: [Wu & Tian (2017); Babaeizadeh et al.
Revised: [Wu & Tian (2017); Babaeizadeh et al.

Original: (2017)] to train a model-free agent.
Revised: (2017)] to train a model-free agent.

Original: During training, the parameters of the agent follows the gradient directions of the two objectives:
Revised: During training, the parameters of the agent follows the gradient directions of the two objectives:

Original: [Equation]
Revised: [Equation]

Original: where π ( a | s ; θ π ) and V ( s ; θ V ) are the policy and value functions of the agent.
Revised: where π ( a | s ; θ π ) and V ( s ; θ V ) are the policy and value functions of the agent.

Original: η t = π ( a t | s t ; θ π ) π ( a t | s t ; θ oldπ ) is an importance factor that captures the discrepancy between the policy distribution π ( a t
Revised: η t = π ( a t | s t ; θ π ) π ( a t | s t ; θ oldπ ) is an importance factor that captures the discrepancy between the policy distribution π ( a t

Original: | s t ; θ oldπ ) evaluated when we sample the action a t at state s t , and the policy we use now with the current parameter θ π .
Revised: | s t ; θ oldπ ) evaluated when we sample the action a t at state s t , and the policy we use now with the current parameter θ π .

Original: We apply the importance factor η t since Batch A3C does not start updating model parameters until it collects a full batch.
Revised: We apply the importance factor η t since Batch A3C does not start updating model parameters until it collects a full batch.

Original: During that period, the data that are ﬁrst collected is considered to be off-policy.
Revised: During that period, the data that are ﬁrst collected is considered to be off-policy.

Original: In many works that use shared feature extraction and multi-head predictions [Wang et al. (2015);
Revised: In many works that use shared feature extraction and multi-head predictions [Wang et al. (2015);

Original: Wu & Tian (2017); Silver et al.
Revised: Wu & Tian (2017); Silver et al.

Original: (2017)], the two set of the parameters θ V and θ π are mostly shared with a common trunk h with parameter θ U , and the policy/value function can be written as:
Revised: (2017)], the two set of the parameters θ V and θ π are mostly shared with a common trunk h with parameter θ U , and the policy/value function can be written as:

Original: [Equation]
Revised: [Equation]

Original: h compactly encodes the information used for both decision (policy π ) and evaluation (value V ) of the model.
Revised: h compactly encodes the information used for both decision (policy π ) and evaluation (value V ) of the model.

Original: Compared to the original high-dimensional input that might contain a lot of irrelevant information, h typically contains a rich but low-dimensional representation of the current situation that is critical for its assigned task.
Revised: Compared to the original high-dimensional input that might contain a lot of irrelevant information, h typically contains a rich but low-dimensional representation of the current situation that is critical for its assigned task.

Original: That might include world states directly related to short-term/longterm reward, and belief of the unobserved part of the environment.
Revised: That might include world states directly related to short-term/longterm reward, and belief of the unobserved part of the environment.

Original: From a model-based point of view, such a latent space learned from a successful model-free training is a good candidate for the state representation of a forward model.
Revised: From a model-based point of view, such a latent space learned from a successful model-free training is a good candidate for the state representation of a forward model.

Original: 3. B UILD A STRONG LATENT SPACE
Revised: 3. B UILD A STRONG LATENT SPACE

Original: Following the reasoning, in order to build a strong forward model, a natural approach is to start with a strong pre-trained model-free agent that performs well in the environment.
Revised: Following the reasoning, in order to build a strong forward model, a natural approach is to start with a strong pre-trained model-free agent that performs well in the environment.

Original: In this paper, we use
Revised: In this paper, we use

Original: MiniRTS [Tian et al.
Revised: MiniRTS [Tian et al.

Original: as the environment, which is a fast incomplete information Real-time
Revised: as the environment, which is a fast incomplete information Real-time

Original: Strategy Game with two players.
Revised: Strategy Game with two players.

Original: An agent which performs well in MiniRTS must build its own troops, gather resources efﬁciently, explore unseen territory, defend the attack from the enemy, and ﬁnd the right timing to attack back.
Revised: An agent which performs well in MiniRTS must build its own troops, gather resources efﬁciently, explore unseen territory, defend the attack from the enemy, and ﬁnd the right timing to attack back.

Original: The game ends when one player’s base is destroyed by the opponent.
Revised: The game ends when one player’s base is destroyed by the opponent.

Original: MiniRTS runs at 40K FPS on a laptop with quite complicated dynamics (e.g., collision detection, unit path planning, etc) and is suitable for our task.
Revised: MiniRTS runs at 40K FPS on a laptop with quite complicated dynamics (e.g., collision detection, unit path planning, etc) and is suitable for our task.

Original: To strengthen the model-free agent, we improve upon [Tian et al.
Revised: To strengthen the model-free agent, we improve upon [Tian et al.

Original: by appending more input feature channels.
Revised: by appending more input feature channels.

Original: We deﬁne three models Vanilla , BuildHistory and PrevSeen , each with a different set of features (Tbl. 1).
Revised: We deﬁne three models Vanilla , BuildHistory and PrevSeen , each with a different set of features (Tbl. 1).

Original: • Vanilla Similar to [Tian et al.
Revised: • Vanilla Similar to [Tian et al.

Original: (2017)], we use basic features that include Unit Type, Hit Point ratio, and available resources to build units.
Revised: (2017)], we use basic features that include Unit Type, Hit Point ratio, and available resources to build units.

Original: • BuildHistory
Revised: • BuildHistory

Original: In an incomplete information game, the decision a player made two minutes ago is not the same as the decision the player is making now, even if the perceived situation is identical.
Revised: In an incomplete information game, the decision a player made two minutes ago is not the same as the decision the player is making now, even if the perceived situation is identical.

Original: This is because during the period, the opponent (which the player cannot see) might have done a lot of work.
Revised: This is because during the period, the opponent (which the player cannot see) might have done a lot of work.

Original: To make the agent aware of the situation, we record the build since tick of each unit, and attach related information in the feature plane.
Revised: To make the agent aware of the situation, we record the build since tick of each unit, and attach related information in the feature plane.

Original: • PrevSeen
Revised: • PrevSeen

Original: Ideally the agent should remember all information since the game starts (e
Revised: Ideally the agent should remember all information since the game starts (e

Original: .g.
Revised: .g.

Original: , the opponent’s base location it saw at tick 100 , even if the current tick is 5000 ).
Revised: , the opponent’s base location it saw at tick 100 , even if the current tick is 5000 ).

Original: However, as shown in our experiment, Recurrent Network (RNN) is not able to carry such a longterm memory.
Revised: However, as shown in our experiment, Recurrent Network (RNN) is not able to carry such a longterm memory.

Original: Therefore, we modify the game engine so that the most recent information that the agent used to have access to, is also sent into the input.
Revised: Therefore, we modify the game engine so that the most recent information that the agent used to have access to, is also sent into the input.

Original: As shown next, it gives
Revised: As shown next, it gives

Original: [Table]
Revised: [Table]

Original: much faster convergence, and fast map exploration and in strategies.
Revised: much faster convergence, and fast map exploration and in strategies.

Original: As a result, the bot learns more aggressive strategies like rush by itself.
Revised: As a result, the bot learns more aggressive strategies like rush by itself.

Original: In Sec. 4.1, we show that PrevSeen has the strongest performance and its latent representation is a good candidate for forward models.
Revised: In Sec. 4.1, we show that PrevSeen has the strongest performance and its latent representation is a good candidate for forward models.

Original: To check whether the learned latent space is useful for forward modeling and understand what kind of game statistics it has captured, we further learn an interpretation network to predict the input from the hidden state.
Revised: To check whether the learned latent space is useful for forward modeling and understand what kind of game statistics it has captured, we further learn an interpretation network to predict the input from the hidden state.

Original: The interpretation network is designed to mirror the network that maps input to latent space, only replacing pooling layers with upsampling layers.
Revised: The interpretation network is designed to mirror the network that maps input to latent space, only replacing pooling layers with upsampling layers.

Original: Again, the latent space of PrevSeen can predict highly relevant input features, as shown in
Revised: Again, the latent space of PrevSeen can predict highly relevant input features, as shown in

Original: Sec. 4.2.
Revised: Sec. 4.2.

Original: 3. L EARN A FORWARD MODEL IN THE LATENT SPACE
Revised: 3. L EARN A FORWARD MODEL IN THE LATENT SPACE

Original: Once we have a rich latent space that contain important information for decision making, we thus train a forward model that carries h t to the near future, which can be used for prediction and modelbased planning.
Revised: Once we have a rich latent space that contain important information for decision making, we thus train a forward model that carries h t to the near future, which can be used for prediction and modelbased planning.

Original: One naive approach is to learn a forward model f to predict the next state h t
Revised: One naive approach is to learn a forward model f to predict the next state h t

Original: +

Revised: +

Original: given h t and a t .
Revised: given h t and a t .

Original: However, a simple prediction might suffer from drifting, in which if we apply f multiple times to predict far future, then the predict error will quickly accumulate and makes the long-term prediction unusable.
Revised: However, a simple prediction might suffer from drifting, in which if we apply f multiple times to predict far future, then the predict error will quickly accumulate and makes the long-term prediction unusable.

Original: This leads to detrimental performance for planning methods (e.g. MCTS) which require accurate long-term predictions.
Revised: This leads to detrimental performance for planning methods (e.g. MCTS) which require accurate long-term predictions.

Original: To solve this issue, during training, we not only require f ( h t , a t ) to be close to h t +1 , but also require f ( t (cid:48) )
Revised: To solve this issue, during training, we not only require f ( h t , a t ) to be close to h t +1 , but also require f ( t (cid:48) )

Original: ( h t , a t , a t +1 , . . .
Revised: ( h t , a t , a t +1 , . . .

Original: , a t + t (cid:48) − 1 ) to be close to h t
Revised: , a t + t (cid:48) − 1 ) to be close to h t

Original: + t (cid:48) , where f ( n ) is to apply the same forward model n times.
Revised: + t (cid:48) , where f ( n ) is to apply the same forward model n times.

Original: While a naive implementation requires O ( T 3 ) gradient update, such a training procedure can be implemented efﬁciently with the current autograd approach with only O ( T 2 ) gradient update.
Revised: While a naive implementation requires O ( T 3 ) gradient update, such a training procedure can be implemented efﬁciently with the current autograd approach with only O ( T 2 ) gradient update.

Original: The trained forward model enables us to apply existing planning method to complicated incomplete information games and give a sensible performance boost over baselines.
Revised: The trained forward model enables us to apply existing planning method to complicated incomplete information games and give a sensible performance boost over baselines.

Original:  E XPERIMENTS 4. T RAIN A STRONG MODEL - FREE AGENT
Revised:  E XPERIMENTS 4. T RAIN A STRONG MODEL - FREE AGENT

Original: We ﬁrst learn a model-free agent against rule-based AI ( AI SIMPLE in MiniRTS).
Revised: We ﬁrst learn a model-free agent against rule-based AI ( AI SIMPLE in MiniRTS).

Original: To maximize the performance of trained AI, we use T = 20 for all the experiments, as mentioned by [Hessel et al.
Revised: To maximize the performance of trained AI, we use T = 20 for all the experiments, as mentioned by [Hessel et al.

Original: (2017); Firoiu et al.
Revised: (2017); Firoiu et al.

Original: that multi-step training with large T is useful.
Revised: that multi-step training with large T is useful.

Original: We also set the frame skip (how frequent the action is made) as 50 for AI, and as 20 for AI SIMPLE .
Revised: We also set the frame skip (how frequent the action is made) as 50 for AI, and as 20 for AI SIMPLE .

Original: This gives the trained agent a slight disadvantage.
Revised: This gives the trained agent a slight disadvantage.

Original: For network structure, we use a convolutional networks with 2 conv + 1 pooling + 2 conv + 1 pooling, each convolutional layer has 64 channels, except for the ﬁrst and the last layer.
Revised: For network structure, we use a convolutional networks with 2 conv + 1 pooling + 2 conv + 1 pooling, each convolutional layer has 64 channels, except for the ﬁrst and the last layer.

Original: The input features
Revised: The input features

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: is 20-by-20 images with 35 channels (Tbl. 1).
Revised: is 20-by-20 images with 35 channels (Tbl. 1).

Original: For all approaches, the overall number of parameters is kept constant as in [Weber et al.
Revised: For all approaches, the overall number of parameters is kept constant as in [Weber et al.

Original: The resulting latent representation is 35 × 5 × 5 =dimensional, which is much lower than the original input space ( 14000 ).
Revised: The resulting latent representation is 35 × 5 × 5 =dimensional, which is much lower than the original input space ( 14000 ).

Original: After convolution, the latent representation is then used to predict discrete action distribution (policy π ) and value function
Revised: After convolution, the latent representation is then used to predict discrete action distribution (policy π ) and value function

Original: V .
Revised: V .

Original: For RNN, the next latent state h t +1 is computed by ﬁrst concatenating h t with a learnable embedding of a t , and then compressing them into a vector in the latent state by MLP.
Revised: For RNN, the next latent state h t +1 is computed by ﬁrst concatenating h t with a learnable embedding of a t , and then compressing them into a vector in the latent state by MLP.

Original: We also use frame-stacking (of size 4), which is an empirical trick to mimic short-term memory during training.
Revised: We also use frame-stacking (of size 4), which is an empirical trick to mimic short-term memory during training.

Original: Both RNN and frame-stacking use the feature sets of Vanilla .
Revised: Both RNN and frame-stacking use the feature sets of Vanilla .

Original: As in [Tian et al.
Revised: As in [Tian et al.

Original: (2017)], we use 9 discrete actions to drive the environments.
Revised: (2017)], we use 9 discrete actions to drive the environments.

Original: All these actions are globally strategic actions (e.g., attack, defend, which unit to build) and agents do not need to worry about their details (e.g., where to build the unit).
Revised: All these actions are globally strategic actions (e.g., attack, defend, which unit to build) and agents do not need to worry about their details (e.g., where to build the unit).

Original: History Information.
Revised: History Information.

Original: Adding historic data can help with the performance of RL approach.
Revised: Adding historic data can help with the performance of RL approach.

Original: However, similar to previous works [Wu & Tian (2017)], the performance of frame-stacking is already comparable to RNN approach, showing that the latent space really captures the short-term memory.
Revised: However, similar to previous works [Wu & Tian (2017)], the performance of frame-stacking is already comparable to RNN approach, showing that the latent space really captures the short-term memory.

Original: On the other hand, PrevSeen outperforms both RNN and BuildHistory .
Revised: On the other hand, PrevSeen outperforms both RNN and BuildHistory .

Original: In summary, while the short-term information can be encoded into the hidden space (either by RNN or frame-stacking), long-term information cannot be efﬁciently learned in the latent space and has to be encoded manually (as in PrevSeen ).
Revised: In summary, while the short-term information can be encoded into the hidden space (either by RNN or frame-stacking), long-term information cannot be efﬁciently learned in the latent space and has to be encoded manually (as in PrevSeen ).

Original: Complete/Incomplete information It leads to difference in training curves (Fig. 2) and ﬁnal performance (Tbl. 2) whether the agent can see the opponent’s behavior and whether the agent has immediate access to the information that encodes what it saw long-time before (e.g., sense the base of the opponent long before).
Revised: Complete/Incomplete information . This leads to difference in training curves (Fig. 2) and ﬁnal performance (Tbl. 2) whether the agent can see the opponent’s behavior and whether the agent has immediate access to the information that encodes what it saw long-time before (e.g., sense the base of the opponent long before).

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  l u e V a l u e
Revised: V a l u e V a l u e

Original: [Figure]
Revised: [Figure]

Original:  Value function For incomplete information setting, we see many sudden drops in the value function when game progresses, due to the fact that the agent does not estimate well about the opponent’s actions and thus thinks it is ina good shape.
Revised: [Figure] Value function . In incomplete information setting, we see many sudden drops in the value function when game progresses. This is because the agent does not estimate well about the opponent’s actions and thus thinks it is in good shape.

Original: To quantify it, we deﬁne the surprise metric as the difference between nearby estimated values:
Revised: To quantify it, we deﬁne the surprise metric as the difference between nearby estimated values:

Original: [Equation]
Revised: [Equation]

Original: Fig. 3 shows that AI trained with complete information indeed knows more about the situations than those trained with incomplete information, in particular at the beginning of the game.
Revised: Fig. 3(b) shows that AI trained with complete information indeed knows more about the situations than those trained with incomplete information, in particular at the beginning of the game.

Original: Thus there is less surprise.
Revised: As shown in Fig. 3(c), the distribution of surprise is different between complete and incomplete information bot.

Original: However, even for the complete information bot, there are quite a few sudden drops.
Revised: However, even for the complete information bot, there are quite a few sudden drops.

Original: This is due to the fact of complicated dynamics of the game.
Revised: This is due to the fact of complicated dynamics of the game.

Original: Lots of small factors, e.g., path planning, collision detection, could lead to substantial different consequences of the game.
Revised: Lots of small factors, e.g., path planning, collision detection, could lead to substantial different consequences of the game.

Original: For example, the consequence of an even battle is often determined by the micro-management of each player, for which our model does not have control.
Revised: For example, the consequence of an even battle is often determined by the micro-management of each player, for which our model does not have control.

Original: Different learned strategy .
Revised: Different learned strategy .

Original: Although the ﬁnal performance of PrevSeen and BuildHistory are similar, their behavior against rule-based AI is quite different.
Revised: Although the ﬁnal performance of PrevSeen and BuildHistory are similar, their behavior against rule-based AI is quite different.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 4 AI trained from
Revised: 4, AI trained from

Original: PrevSeen learned to explore the map and/or rush the enemy with a few tanks at the beginning of the game.
Revised: PrevSeen learned to explore the map and/or rush the enemy with a few tanks at the beginning of the game.

Original: Quantitatively , the number of nonzero seen-then-hidden events per game from PrevSeen is twice ( 3405 ) than that ( 1736 ) from BuildHistory (Fig. 5).
Revised: Quantitatively, the number of nonzero seen-then-hidden events per game from PrevSeen is twice ( 3405 ) than that ( 1736 ) from BuildHistory (Fig. 5).

Original: We haven’t seen similar behaviors in prior works.
Revised: We haven’t seen similar behaviors in prior works.

Original: [Figure]
Revised: [Figure]

Original: 4. R ECONSTRUCTION FROM THE LATENT STATE
Revised: 4. R ECONSTRUCTION FROM THE LATENT STATE

Original: With the latent state, we can reconstruct the original 20x20 channels.
Revised: With the latent state, we can reconstruct the original 20x20 channels.

Original: We choose to predict one channel at a time.
Revised: We choose to predict one channel at a time.

Original: A joint model that predicts all input channels at once works poorly since dense channels (with many non-zero entries)would dominate the prediction.
Revised: A joint model that predicts all input channels at once works poorly since dense channels (with many non-zero entries) dominate the prediction.

Original: We deﬁne the normalized reconstruction accuracy (NRA) to be 1 − (cid:107) c − c ∗ (cid:107) / (cid:107) c ∗ (cid:107) , where c is some channel and c ∗ is its ground truth value.
Revised: We deﬁne the normalized reconstruction accuracy (NRA) to be 1 − (cid:107) c − c ∗ (cid:107) / (cid:107) c ∗ (cid:107) , where c is some channel and c ∗ is its ground truth value.

Original: Fig.
Revised: Fig.

Original: 6 shows the results.
Revised: 6 shows the results.

Original: The model correctly ﬁnds the most relevant features from the input (e.g., base HP ratio, the amount of available resource).
Revised: The model correctly ﬁnds the most relevant features from the input (e.g., base HP ratio, the amount of available resource).

Original: We discover that the agent can partially recover the exact location of the workers during early stage of training, but ignores them at convergence, showing that the model learns to discard less relevant features.
Revised: We discover that the agent can partially recover the exact location of the workers during early stage of training, but ignores them at convergence, showing that the model learns to discard less relevant features.

Original: Interestingly, the prev-seen feature channels are not predicted well for the full-ﬂedged agent, showing they are not important in the decision.
Revised: Interestingly, the prev-seen feature channels are not predicted well for the full-ﬂedged agent, showing they are not important in the decision.

Original: However, models equipped with these channels train much faster.
Revised: However, models equipped with these channels train much faster.

Original: We also try predicting the 5x5 down-sampled version of the original 20x20 channels, by removing all upsampling layers.
Revised: We also try predicting the 5x5 down-sampled version of the original 20x20 channels, by removing all upsampling layers.

Original: Overall the normalized reconstruction accuracy is much higher, as shown in the dark blue bars.
Revised: Overall the normalized reconstruction accuracy is much higher, as shown in the dark blue bars.

Original: This is because the latent state is also a 5 × 5 down-sampled state of the original game state, and the model only needs to reconstruct the features in a rough region instead of in each grid.
Revised: This is because the latent state is also a 5 × 5 down-sampled state of the original game state, and the model only needs to reconstruct the features in a rough region instead of in each grid.

Original: In particular, we see the emergence of multiple relevant features such as the location of own
Revised: In particular, we see the emergence of multiple relevant features such as the location of own

Original: RANGE ATTACKER and BASE , afﬁliation and unit HP ratio, etc.
Revised: RANGE ATTACKER and BASE , afﬁliation and unit HP ratio, etc.

Original: We also see that the agent learns to pay attention to the opponent’s WORKER (which gathers resources) and MELEE ATTACKER (which attacks us).
Revised: We also see that the agent learns to pay attention to the opponent’s WORKER (which gathers resources) and MELEE ATTACKER (which attacks us).

Original: The model pays attention to the opponent’s MELEE ATTACKER but not so much to
Revised: The model pays attention to the opponent’s MELEE ATTACKER but not so much to

Original: RANGE ATTACKER because the rule based AI mostly builds MELEE ATTACKER .
Revised: RANGE ATTACKER because the rule based AI mostly builds MELEE ATTACKER .

Original: 4. L EARN A FORWARD MODEL
Revised: 4. L EARN A FORWARD MODEL

Original: We train multiple forward models following Sec. 3.2.
Revised: We train multiple forward models following Sec. 3.2, each with different training paradigm deﬁned as follows.

Revised: Pred1 : basic forward model (Fig. 1(b)); MatchPi : enforce the predicted future state to predict the future policy well (Fig. 1(c))

Revised: ; MatchA : enforce the predicted future state to predict the future action well (Fig. 1(d)); PredN : predict long-term future states (Fig. 1(e)).

Original: For evaluation, at a time step t , we recursively apply the learned forward model to estimate the current latent state ˆ h t from previous state h t − t (cid:48) that is computed from observation s t − t (cid:48) .
Revised: For evaluation, at a time step t , we recursively apply the learned forward model to estimate the current latent state ˆ h t from previous state h t − t (cid:48) that is computed from observation s t − t (cid:48) .

Original: Then we use ˆ h t to make decision for time t , and check its win rate against the rule-based system.
Revised: Then we use ˆ h t to make decision for time t , and check its win rate against the rule-based system.

Original: As a baseline, we use identity forward function ˆ h t = h t − t (cid:48) .
Revised: As a baseline, we use identity forward function ˆ h t = h t − t (cid:48) .

Original: We evaluate these training paradigms in 5 independently trained models using model-free approaches.
Revised: We evaluate these training paradigms in 5 independently trained models using model-free approaches.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 7, MatchPi is the most stable method with highest win rate, consistently higher than the baseline approach that uses ˆ h t = h t − t (cid:48) .
Revised: 7, MatchPi is the most stable method with highest win rate, consistently higher than the baseline approach that uses ˆ h t = h t − t (cid:48) .

Original: MatchA also learns quite fast, but runs into numerical instabilities after a few hundreds of iterations.
Revised: MatchA also learns quite fast, but runs into numerical instabilities after a few hundreds of iterations.

Original: PredN is also quite stable but its learning speed is not as fast compared to MatchA .
Revised: PredN is also quite stable but its learning speed is not as fast compared to MatchA .

Original: When the delay is large, we clearly see the decay of the win rate performance.
Revised: When the delay is large, we clearly see the decay of the win rate performance.

Original: 4. U SING F ORWARD MODELING IN MCTS
Revised: 4. U SING F ORWARD MODELING IN MCTS

Original: The performance metric used in Sec.
Revised: The performance metric used in Sec.

Original: 4.3 is only an intermediate metric.
Revised: 4.3 is only an intermediate metric.

Original: To ﬁnally test the performance of forward models, we plug them into a planning algorithm, e.g., Monte-Carlo Tree Search (MCTS).
Revised: To ﬁnally test the performance of forward models, we plug them into a planning algorithm, e.g., Monte-Carlo Tree Search (MCTS).

Original: Note that traditional MCTS can only be applied when both the complete game state and the dynamics of the game environments are known.
Revised: Note that traditional MCTS can only be applied when both the complete game state and the dynamics of the game environments are known.

Original: However, neither conditions are satisﬁed for
Revised: However, neither conditions are satisﬁed for

Original: RTS games, thus learned forward models and value function are necessary.
Revised: RTS games, thus learned forward models and value function are necessary.

Original: In MCTS, We use shared tree (tree parallelization) among multiple threads.
Revised: In MCTS, We use shared tree (tree parallelization) among multiple threads.

Original: This is because compared to other approaches (e.g., root parallelization Chaslot et al.
Revised: This is because compared to other approaches (e.g., root parallelization Chaslot et al.

Original: (2008)), expanding a tree node, as a costly operation that requires feed-forwarding a neural network, only happens once.
Revised: (2008)), expanding a tree node, as a costly operation that requires feed-forwarding a neural network, only happens once.

Original: To increase diversity of initial exploration, we use virtual loss of size 5 .
Revised: To increase diversity of initial exploration, we use virtual loss of size 5 .

Original: During the execution of MCTS, the network performs a projection step that maps the input state to its latent representation, a forwarding step that predicts the latent state h t +

Revised: During the execution of MCTS, the network performs a projection step that maps the input state to its latent representation, a forwarding step that predicts the latent state h t +

Original: given h t and a t , and a prediction step that gives the value of the predicted state for back-propagation in MCTS.
Revised: given h t and a t , and a prediction step that gives the value of the predicted state for back-propagation in MCTS.

Original: Interestingly, the models with strong performance in delayed prediction task (e.g., MatchPi ) does not perform well in MCTS.
Revised: Interestingly, the models with strong performance in delayed prediction task (e.g., MatchPi ) does not perform well in MCTS.

Original: On the contrary, PredN gives a consistently strong performance (25%) over 5 independently trained models, compared to the random baseline (17% win rate).
Revised: On the contrary, PredN gives a consistently strong performance (25%) over 5 independently trained models, compared to the random baseline (17% win rate).

Original: Note that without a good forward model and a value function, we cannot even run reduced space MCTS.
Revised: Note that without a good forward model and a value function, we cannot even run reduced space MCTS.

Original: The proposed forward model makes it possible.
Revised: The proposed forward model makes it possible.

Original: However, compared to the performance of model-free  approach ( > 80% ), there is still a long way to go.
Revised: However, compared to the performance of model-free [Figure] approach ( > 80% ), there is still a long way to go.

Original: We also tried removing short-term predictions (Fig. 1(f)), the performance is similar.
Revised: We also tried removing short-term predictions (Fig. 1(f)), the performance is similar.

Original: We have also tried combining forward model with action probability given by model-free approach using PUCT
Revised: We have also tried combining forward model with action probability given by model-free approach using PUCT

Original: [Rosin (2011)].
Revised: [Rosin (2011)].

Original: However, there is no gain due to the fact that the model-free part has a much stronger performance, overshadowing the contribution of forward models.[Figure]
Revised: However, there is no gain due to the fact that the model-free part has a much stronger performance, overshadowing the contribution of forward models.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: Latent space learned by model-free reinforcement learning encodes important information for an agent to make sensible decisions to maximize the reward in a complicated simulated environment.
Revised: Latent space learned by model-free reinforcement learning encodes important information for an agent to make sensible decisions to maximize the reward in a complicated simulated environment.

Original: In this paper, we verify the power of latent space of successfully trained model-free agent, and propose several methods to learn forward models on this space, in a real-time strategy game with incomplete information.
Revised: In this paper, we verify the power of latent space of successfully trained model-free agent, and propose several methods to learn forward models on this space, in a real-time strategy game with incomplete information.

Original: Despite an extremely hard problem, we learn forward models that make it possible to use planning approaches such as Monte Carlo Tree Search, and show consistently positive gains over baselines.
Revised: Despite an extremely hard problem, we learn forward models that make it possible to use planning approaches such as Monte Carlo Tree Search, and show consistently positive gains over baselines.

Original: A lot of future works follow.
Revised: A lot of future works follow.

Original: As a ﬁrst step, although we show that it is possible to learn a forward model for incomplete information Real-time Strategy games to enable model-based planning in the latent space, it remains an open problem how to improve its performance.
Revised: As a ﬁrst step, although we show that it is possible to learn a forward model for incomplete information Real-time Strategy games to enable model-based planning in the latent space, it remains an open problem how to improve its performance.

Original: It is possible that despite a good forward model is learned, the value function is not good enough, e.g., putting too much focus on the on-policy trajectory, for Monte-Carlo Tree Search.
Revised: It is possible that despite a good forward model is learned, the value function is not good enough, e.g., putting too much focus on the on-policy trajectory, for Monte-Carlo Tree Search.

Original: Also, in this paper we use predeﬁnedglobal actions for the game.
Revised: Also, in this paper we use predeﬁnedglobal actions for the game.

Original: How to automatically learn global actions from unit-based commands that are exponentially large, is still an challenging issue to solve.
Revised: How to automatically learn global actions from unit-based commands that are exponentially large, is still an challenging issue to solve.