Original: Grounded Reinforcement Learning: Learning to Win the Game under Human Commands
Revised: Grounded Reinforcement Learning: Learning to Win the Game under Human Commands

Original: Abstract
Revised: Abstract

Original: We consider the problem of building a reinforcement learning (RL) agent that can both accomplish non-trivial tasks, like winning a real-time strategy game, and strictly follow high-level language commands from humans, like “attack”, even if a command is sub-optimal.
Revised: We consider the problem of building a reinforcement learning (RL) agent that can both accomplish non-trivial tasks, like winning a real-time strategy game, and strictly follow high-level language commands from humans, like “attack”, even if a command is sub-optimal.

Original: We call this novel yet important problem, Grounded
Revised: We call this novel yet important problem, Grounded

Original: Reinforcement Learning (GRL).
Revised: Reinforcement Learning (GRL).

Original: Compared with other language grounding tasks,
Revised: Compared with other language grounding tasks,

Original: GRL is particularly non-trivial and cannot be simply solved by pure RL or behavior cloning (BC).
Revised: GRL is particularly non-trivial and cannot be simply solved by pure RL or behavior cloning (BC).

Original: From the RL perspective, it is extremely challenging to derive a precise reward function for human preferences since the commands are abstract and the valid behaviors are highly complicated and multi-modal.
Revised: From the RL perspective, it is extremely challenging to derive a precise reward function for human preferences since the commands are abstract and the valid behaviors are highly complicated and multi-modal.

Original: From the BC perspective, it is impossible to obtain perfect demonstrations since human strategies in complex games are typically sub-optimal.
Revised: From the BC perspective, it is impossible to obtain perfect demonstrations since human strategies in complex games are typically sub-optimal.

Original: We tackle GRL via a simple, tractable, and practical constrained RL objective and develop an iterative RL algorithm,
Revised: We tackle GRL via a simple, tractable, and practical constrained RL objective and develop an iterative RL algorithm,

Original: REinforced demonstration Distillation (RED), to obtain a strong GRL policy.
Revised: REinforced demonstration Distillation (RED), to obtain a strong GRL policy.

Original: We evaluate the policies derived by RED, BC and pure RL methods on a simplified real-time strategy game, MiniRTS.
Revised: We evaluate the policies derived by RED, BC and pure RL methods on a simplified real-time strategy game, MiniRTS.

Original: Experiment results and human studies show that the RED policy is able to consistently follow human commands and, at the same time, achieve a higher win rate than the baselines.
Revised: Experiment results and human studies show that the RED policy is able to consistently follow human commands and, at the same time, achieve a higher win rate than the baselines.

Original: We release our code and present more examples at https://sites.google.com/view/grounded-rl .
Revised: We release our code and present more examples at https://sites.google.com/view/grounded-rl .

Original:  Introduction
Revised:  Introduction

Original: Building assistive agents that can help humans accomplish complex tasks remains a long-standing challenge in artificial intelligence.
Revised: Building assistive agents that can help humans accomplish complex tasks remains a long-standing challenge in artificial intelligence.

Original: Natural language, as the most generic protocol for humans to exchange and share knowledge, becomes a general and important interface for human-AI interaction.
Revised: Natural language, as the most generic protocol for humans to exchange and share knowledge, becomes a general and important interface for human-AI interaction.

Original: Decades of research efforts have been continuously made to develop AI systems that can ground agent behaviors to natural language commands [69, 64, 46].
Revised: Decades of research efforts have been continuously made to develop AI systems that can ground agent behaviors to natural language commands [69, 64, 46].

Original: Recently, as deep reinforcement learning (RL) techniques have been widely used to develop interactive agents to solve a variety of challenging problems, it becomes a new trend to cast language grounding as an RL problem to train agents that can automatically ground language concepts to visual objects or behaviors in an interactive fashion [29, 13, 15].
Revised: Recently, as deep reinforcement learning (RL) techniques have been widely used to develop interactive agents to solve a variety of challenging problems, it becomes a new trend to cast language grounding as an RL problem to train agents that can automatically ground language concepts to visual objects or behaviors in an interactive fashion [29, 13, 15].

Original: In particular, an RL agent is typically presented with a language command describing the goal of the RL task and will receive a success reward when the desired goal state under the command is achieved.
Revised: In particular, an RL agent is typically presented with a language command describing the goal of the RL task and will receive a success reward when the desired goal state under the command is achieved.

Original: A well-trained RL agent can often exhibit strong generalization capabilities to novel commands.
Revised: A well-trained RL agent can often exhibit strong generalization capabilities to novel commands.

Original: Despite the simplicity and effectiveness, such an RL formulation requires a language generator to repeatedly output random commands and an explicit reward function to determine whether the command is accomplished.
Revised: Despite the simplicity and effectiveness, such an RL formulation requires a language generator to repeatedly output random commands and an explicit reward function to determine whether the command is accomplished.

Original: Hence, most existing RL works focus on simple navigation problems with template-based compositional languages over objects and attributes, e.g., “go to the red box next to the blue wall” [56, 68, 34, 15].
Revised: Hence, most existing RL works focus on simple navigation problems with template-based compositional languages over objects and attributes, e.g., “go to the red box next to the blue wall” [56, 68, 34, 15].

Original: To train agents that can interpret general human languages, another research direction is to leverage paired behavior-command data collected from human demonstrators.
Revised: To train agents that can interpret general human languages, another research direction is to leverage paired behavior-command data collected from human demonstrators.

Original: Representative applications include robotic manipulation, where the robot needs to map language commands to predefined motion skills [60, 35, 61, 65], and vision-and-language navigation, where an agent learns from human demonstrations to navigate towards a desired location in a 3D environment [51, 13, 29, 71].
Revised: Representative applications include robotic manipulation, where the robot needs to map language commands to predefined motion skills [60, 35, 61, 65], and vision-and-language navigation, where an agent learns from human demonstrations to navigate towards a desired location in a 3D environment [51, 13, 29, 71].

Original: Since an accurate command-following reward is no longer accessible, most works apply behavior cloning (BC) to directly imitate human behaviors or adopt inverse reinforcement learning (IRL) to learn a language-conditioned reward function [19, 52, 30, 23, 79, 17].
Revised: Since an accurate command-following reward is no longer accessible, most works apply behavior cloning (BC) to directly imitate human behaviors or adopt inverse reinforcement learning (IRL) to learn a language-conditioned reward function [19, 52, 30, 23, 79, 17].

Original: Although the use of human data enables natural commands, pure imitation-based methods require massive demonstrations, which can be expensive to collect.
Revised: Although the use of human data enables natural commands, pure imitation-based methods require massive demonstrations, which can be expensive to collect.

Original: More importantly, both BC and IRL algorithms assume that the demonstrations are optimal [19, 52, 30, 23, 79, 17], in the sense that (i) each behavior demonstration is optimal under its paired language command, and (ii) the language commands are optimal for the underlying task.
Revised: More importantly, both BC and IRL algorithms assume that the demonstrations are optimal [19, 52, 30, 23, 79, 17], in the sense that (i) each behavior demonstration is optimal under its paired language command, and (ii) the language commands are optimal for the underlying task.

Original: Such an assumption is reasonable in restricted domains like manipulation or navigation, where the optimal behavior is simply the shortest trajectory to the goal while human commands typically provide strong guidance towards task completion.
Revised: Such an assumption is reasonable in restricted domains like manipulation or navigation, where the optimal behavior is simply the shortest trajectory to the goal while human commands typically provide strong guidance towards task completion.

Original: However, in more complex problems like playing real-time strategy games, it is often the case that human players can be highly biased or sub-optimal [10].
Revised: However, in more complex problems like playing real-time strategy games, it is often the case that human players can be highly biased or sub-optimal [10].

Original: Let’s consider a concrete example in a simplified real-time strategy game, MiniRTS [33].
Revised: Let’s consider a concrete example in a simplified real-time strategy game, MiniRTS [33].

Original: Fig.
Revised: Fig.

Original: 1(C) shows a scenario where a human commander gives an abstract command “attack” to a human executor to attack enemy units.
Revised: 1(C) shows a scenario where a human commander gives an abstract command “attack” to a human executor to attack enemy units.

Original: However, the action taken by the human executor is sub-optimal: a unit with full
Revised: However, the action taken by the human executor is sub-optimal: a unit with full

Original: HP is under attack while a clear better strategy is to attack another dying unit.
Revised: HP is under attack while a clear better strategy is to attack another dying unit.

Original: Moreover, the human commander can be sub-optimal as well.
Revised: Moreover, the human commander can be sub-optimal as well.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1(E), very few enemy units are remaining, so the optimal command should be simply an “attack” for the win.
Revised: 1(E), very few enemy units are remaining, so the optimal command should be simply an “attack” for the win.

Original: However, the commander sends a
Revised: However, the commander sends a

Original: “retreat” command.
Revised: “retreat” command.

Original: This could be possibly due to the partially observed game state or human biases, but an obedient executor should still faithfully control the units to stop attacking and retreat.
Revised: This could be possibly due to the partially observed game state or human biases, but an obedient executor should still faithfully control the units to stop attacking and retreat.

Original: We study this novel RL challenge, i.e., learning an strong agent capable of not only achieving a high win rate in a real-time strategy game, e.g., MiniRTS, but also strictly following high-level language commands, even if a sub-optimal command is given.
Revised: We study this novel RL challenge, i.e., learning an strong agent capable of not only achieving a high win rate in a real-time strategy game, e.g., MiniRTS, but also strictly following high-level language commands, even if a sub-optimal command is given.

Original: We call this problem, Grounded
Revised: We call this problem, Grounded

Original: Reinforcement Learning (GRL), which assumes an interactive RL environment with a dataset of (sub-optimal) human demonstrations reflecting proper human behaviors under language commands.
Revised: Reinforcement Learning (GRL), which assumes an interactive RL environment with a dataset of (sub-optimal) human demonstrations reflecting proper human behaviors under language commands.

Original: In this setting, the rewards are not associated with the language commands, and it is non-trivial to verify whether a high-level language command is completed or not.
Revised: In this setting, the rewards are not associated with the language commands, and it is non-trivial to verify whether a high-level language command is completed or not.

Original: So the policy can only learn about how humans follow commands from the demonstrations.
Revised: So the policy can only learn about how humans follow commands from the demonstrations.

Original: We remark that GRL is different from standard language grounding problems since the primary mission is not towards better language understanding via interactions [11].
Revised: We remark that GRL is different from standard language grounding problems since the primary mission is not towards better language understanding via interactions [11].

Original: Instead, GRL focuses on the opposite direction, i.e., learning strong winning strategies while taking human commands as high-level behavior regulations, which is more related to the concept of developing human-compatible AI [57].
Revised: Instead, GRL focuses on the opposite direction, i.e., learning strong winning strategies while taking human commands as high-level behavior regulations, which is more related to the concept of developing human-compatible AI [57].

Original: To tackle the GRL problem, we propose a simple yet effective constrained RL objective as a tractable approximation and developed an iterative RL algorithm, REinforced demonstration Distillation (RED), to derive a strong language-conditioned policy.
Revised: To tackle the GRL problem, we propose a simple yet effective constrained RL objective as a tractable approximation and developed an iterative RL algorithm, REinforced demonstration Distillation (RED), to derive a strong language-conditioned policy.

Original: RED adopts unconditioned RL training to encourage the policy to explore stronger winning strategies and periodically applies self-distillation and BC over demonstrations to ensure that the policy behavior is consistent with human preferences.
Revised: RED adopts unconditioned RL training to encourage the policy to explore stronger winning strategies and periodically applies self-distillation and BC over demonstrations to ensure that the policy behavior is consistent with human preferences.

Original: We evaluate the performance of RED as well as baseline methods including BC-based methods and
Revised: We evaluate the performance of RED as well as baseline methods including BC-based methods and

Original: RL variants on the MiniRTS environment.
Revised: RL variants on the MiniRTS environment.

Original: Simulation results show that RED policy achieves strong command-following capabilities and a higher win rate than baselines under various types of test-time commands.
Revised: Simulation results show that RED policy achieves strong command-following capabilities and a higher win rate than baselines under various types of test-time commands.

Original: We also conduct human evaluation by inviting 30 volunteers to play with policies trained by different methods.
Revised: We also conduct human evaluation by inviting 30 volunteers to play with policies trained by different methods.

Original: RED policy appears to be the most obedient under human votes and leads to a much higher collaborative win rate with human commanders.
Revised: RED policy appears to be the most obedient under human votes and leads to a much higher collaborative win rate with human commanders.

Original:  Related Work
Revised:  Related Work

Original: Language grounding.
Revised: Language grounding.

Original: The idea of grounding languages to behaviors or concepts can be traced backed to 1970s [69], and most early works assume simple domain-specific languages and a predefined set of skills [45, 77, 37, 14, 47, 5].
Revised: The idea of grounding languages to behaviors or concepts can be traced backed to 1970s [69], and most early works assume simple domain-specific languages and a predefined set of skills [45, 77, 37, 14, 47, 5].

Original: Thanks to the recent advances of deep RL, it becomes feasible to ground languages to visual concepts and non-trivial behaviors in a simulated world.
Revised: Thanks to the recent advances of deep RL, it becomes feasible to ground languages to visual concepts and non-trivial behaviors in a simulated world.

Original: Many 3D environments with language-described goals have been developed over different domains, including maze exploration [15], visual navigation [13, 29, 71] and robot control [35, 61, 65], allowing end-to-end concept learning and grounding.
Revised: Many 3D environments with language-described goals have been developed over different domains, including maze exploration [15], visual navigation [13, 29, 71] and robot control [35, 61, 65], allowing end-to-end concept learning and grounding.

Original: RL training requires a goal generator and a reward function.
Revised: RL training requires a goal generator and a reward function.

Original: Hence, these testbeds only adopt template-based languages over objects (e.g., box or cube) and attributes (e.g., spatial relation or color) and assume an oracle that can verify whether a state is consistent with the language description.
Revised: Hence, these testbeds only adopt template-based languages over objects (e.g., box or cube) and attributes (e.g., spatial relation or color) and assume an oracle that can verify whether a state is consistent with the language description.

Original: We focus on following high-level natural commands.
Revised: We focus on following high-level natural commands.

Original: Regarding natural commands, recent visual navigation benchmarks [3, 49, 61, 62] started to provide large-scale human demonstrations with the goal or pathway specified by natural languages.
Revised: Regarding natural commands, recent visual navigation benchmarks [3, 49, 61, 62] started to provide large-scale human demonstrations with the goal or pathway specified by natural languages.

Original: Behavior cloning (BC) is feasible since every human description is paired with a successful trajectory [19, 52, 30].
Revised: Behavior cloning (BC) is feasible since every human description is paired with a successful trajectory [19, 52, 30].

Original: RL fine-tuning can be also applied, since a precise success measurement (i.e., distance to the goal position) w.r.t.
Revised: RL fine-tuning can be also applied, since a precise success measurement (i.e., distance to the goal position) w.r.t.

Original: each language instruction is known [22, 67].
Revised: each language instruction is known [22, 67].

Original: Some works adopt inverse RL (IRL) to learn a language-conditioned reward function [23, 79] thanks to the fact that the demonstrations are actually optimal.
Revised: Some works adopt inverse RL (IRL) to learn a language-conditioned reward function [23, 79] thanks to the fact that the demonstrations are actually optimal.

Original: We consider a more challenging game MiniRTS [33] with abstract language commands and a complex behavior space.
Revised: We consider a more challenging game MiniRTS [33] with abstract language commands and a complex behavior space.

Original: BC and IRL methods work poorly in MiniRTS since both language commands and game trajectories from human annotators are highly sub-optimal.
Revised: BC and IRL methods work poorly in MiniRTS since both language commands and game trajectories from human annotators are highly sub-optimal.

Original: There are also successful attempts to leverage powerful pretrained language models to map general languages to the goal space [36, 44] or a predefined set of skills [31, 60, 2] so that massive paired demonstrations can be no longer necessary.
Revised: There are also successful attempts to leverage powerful pretrained language models to map general languages to the goal space [36, 44] or a predefined set of skills [31, 60, 2] so that massive paired demonstrations can be no longer necessary.

Original: We primarily consider the problem of policy learning, so the use of pretrained model is orthogonal to our current focus but remains an exciting future direction.
Revised: We primarily consider the problem of policy learning, so the use of pretrained model is orthogonal to our current focus but remains an exciting future direction.

Original: There are also parallel works on language grounding in other domains, such as text games [38], answer questioning [4] and communication [39], where languages serve as state descriptor or action space rather than commands to follow.
Revised: There are also parallel works on language grounding in other domains, such as text games [38], answer questioning [4] and communication [39], where languages serve as state descriptor or action space rather than commands to follow.

Original: Our work is also related to ad-hoc team play [66, 12] since the policy cooperates with arbitrary commands.
Revised: Our work is also related to ad-hoc team play [66, 12] since the policy cooperates with arbitrary commands.

Original: The difference is that the commander is not assumed to be optimal under the game reward.
Revised: The difference is that the commander is not assumed to be optimal under the game reward.

Original: The policy must be assistive [28, 70] or even obedient [50].
Revised: The policy must be assistive [28, 70] or even obedient [50].

Original: Constrained reinforcement learning.
Revised: Constrained reinforcement learning.

Original: We cast grounded RL as a constrained RL (CRL) formulation for tractability.
Revised: We cast grounded RL as a constrained RL (CRL) formulation for tractability.

Original: CRL algorithms often leverage the Lagrangian multiplier [55] or projected gradients [1, 72] for policy improvement assuming simple (closed-form) constraints over states.
Revised: CRL algorithms often leverage the Lagrangian multiplier [55] or projected gradients [1, 72] for policy improvement assuming simple (closed-form) constraints over states.

Original: In our setting, constraints can be only implicitly learned from human demonstrations.
Revised: In our setting, constraints can be only implicitly learned from human demonstrations.

Original: We simply adopt the behavior cloning objective, i.e., negative log likelihood, as the constraint satisfying metric, which is related to RL with demonstrations (RLwD) [32, 25].
Revised: We simply adopt the behavior cloning objective, i.e., negative log likelihood, as the constraint satisfying metric, which is related to RL with demonstrations (RLwD) [32, 25].

Original: However, due to limited data and a significant distribution shift between demonstrations and on-policy trajectories, RLwD methods can perform poorly.
Revised: However, due to limited data and a significant distribution shift between demonstrations and on-policy trajectories, RLwD methods can perform poorly.

Original: In addition, [73] iteratively optimizes the RL objective, projects the policy on a region around a reference policy, and enforces the policy satisfies the cost constraint.
Revised: In addition, [73] iteratively optimizes the RL objective, projects the policy on a region around a reference policy, and enforces the policy satisfies the cost constraint.

Original: [27] updates the policies on the mixture of the expert replay buffer and the rollout trajectories.
Revised: [27] updates the policies on the mixture of the expert replay buffer and the rollout trajectories.

Original: The most related work to our RED algorithm is Supervised Seeded Iterated Learning (SSIL) [42], which is designed for learning grounded communication and adopts a conceptually similar iterative framework by alternating between RL and self-distillation.
Revised: The most related work to our RED algorithm is Supervised Seeded Iterated Learning (SSIL) [42], which is designed for learning grounded communication and adopts a conceptually similar iterative framework by alternating between RL and self-distillation.

Original: SSIL suggests to jointly optimize RL and BC objectives assuming human communications are perfect demonstrations.
Revised: SSIL suggests to jointly optimize RL and BC objectives assuming human communications are perfect demonstrations.

Original: By contrast, we find that an additional
Revised: By contrast, we find that an additional

Original: BC loss can be harmful to RL training due to the distribution shift issue.
Revised: BC loss can be harmful to RL training due to the distribution shift issue.

Original: Similar empirical phenomena have been also reported in multi-task RL literature [75, 63, 74].
Revised: Similar empirical phenomena have been also reported in multi-task RL literature [75, 63, 74].

Original:  Preliminary 3.1 The MiniRTS Environment and Dataset
Revised:  Preliminary 3.1 The MiniRTS Environment and Dataset

Original: MiniRTS [33] is a grid-world RL environment (Fig. 1) that distills the key features of complex real-time strategy games.
Revised: MiniRTS [33] is a grid-world RL environment (Fig. 1) that distills the key features of complex real-time strategy games.

Original: It has two parties, a player (blue) controlled by a human/policy against a built-in script AI (red).
Revised: It has two parties, a player (blue) controlled by a human/policy against a built-in script AI (red).

Original: The player controls units to collect resources, do construction and kill all the enemy units or destroy the enemy base to win a game.
Revised: The player controls units to collect resources, do construction and kill all the enemy units or destroy the enemy base to win a game.

Original: A total of 7 unit types form a rock-paperscissors attacking dynamics.
Revised: A total of 7 unit types form a rock-paperscissors attacking dynamics.

Original: Learning a strong policy is challenging due to partially observed states, complicated micro-management over dozens of units, and extremely diverse possible strategies.
Revised: Learning a strong policy is challenging due to partially observed states, complicated micro-management over dozens of units, and extremely diverse possible strategies.

Original: In addition to an RL environment, MiniRTS provides a command-following dataset, which is collected by two humans playing collaboratively against the script opponent.
Revised: In addition to an RL environment, MiniRTS provides a command-following dataset, which is collected by two humans playing collaboratively against the script opponent.

Original: One is the commander, who gives high-level strategic language commands like “attack”, “retreat”, or “build an archer”.
Revised: One is the commander, who gives high-level strategic language commands like “attack”, “retreat”, or “build an archer”.

Original: The other one is the executor, who controls the units to follow the language command.
Revised: The other one is the executor, who controls the units to follow the language command.

Original: We remark that human data were collected against much weaker scripts than the default game opponent.
Revised: We remark that human data were collected against much weaker scripts than the default game opponent.

Original: Even in such an easier mode, the collected game episodes are highly sub-optimal with a win rate of merely 47.2%.
Revised: Even in such an easier mode, the collected game episodes are highly sub-optimal with a win rate of merely 47.2%.

Original: The dataset contains a total of 63,285 commands with an average of 13.7 per game while the average game horizon is 151.7.
Revised: The dataset contains a total of 63,285 commands with an average of 13.7 per game while the average game horizon is 151.7.

Original: So, each single command may take a long horizon to accomplish.
Revised: So, each single command may take a long horizon to accomplish.

Original: For notation simplicity in this paper, we assume an independent command is given to the executor at each time step, although a command should be repeated for a few steps.
Revised: For notation simplicity in this paper, we assume an independent command is given to the executor at each time step, although a command should be repeated for a few steps.

Original: Full details are in appendix.
Revised: Full details are in appendix.

Original: 3.2 Notation
Revised: 3.2 Notation

Original: Reinforcement learning.
Revised: Reinforcement learning.

Original: We formulate the MiniRTS environment as a Partially Observable Markov
Revised: We formulate the MiniRTS environment as a Partially Observable Markov

Original: Decision Process (POMDP), which is represented by a tuple ⟨S , A , O , O, r, p ⟩ .
Revised: Decision Process (POMDP), which is represented by a tuple ⟨S , A , O , O, r, p ⟩ .

Original: S is the state space.
Revised: S is the state space.

Original: A is the action space.
Revised: A is the action space.

Original: O is the observation space.
Revised: O is the observation space.

Original: r : S ×
Revised: r : S ×

Original: A → R is the reward function.
Revised: A → R is the reward function.

Original: p : S ×
Revised: p : S ×

Original: A × S → [0 , 1] is the transition probability with p ( s ′ | s, a ) denoting the probability from state s to state s ′ after taking action a .
Revised: A × S → [0 , 1] is the transition probability with p ( s ′ | s, a ) denoting the probability from state s to state s ′ after taking action a .

Original: At each time step, the agent (i.e., the game player) observes o t
Revised: At each time step, the agent (i.e., the game player) observes o t

Original: = O ( s t ) , produces an action a t according to its policy π θ parameterized by θ , and receives a reward r t = r
Revised: = O ( s t ) , produces an action a t according to its policy π θ parameterized by θ , and receives a reward r t = r

Original: ( s t , a t ) .
Revised: ( s t , a t ) .

Original: The optimal policy should achieve high win rates in the game, namely θ ⋆ = arg max θ E a t ,s t
Revised: The optimal policy should achieve high win rates in the game, namely θ ⋆ = arg max θ E a t ,s t

Original: [ (cid:80) t r ( s t , a t )] .
Revised: [ (cid:80) t r ( s t , a t )] .

Original: Note that we omit the discount factor γ for conciseness only.
Revised: Note that we omit the discount factor γ for conciseness only.

Original: Command-following policy.
Revised: Command-following policy.

Original: We represent the policy by π θ ( a t |
Revised: We represent the policy by π θ ( a t |

Original: o t , c t ) , which conditions on an observation o t ∈ O and a language command c t ∈ C at each time step t .
Revised: o t , c t ) , which conditions on an observation o t ∈ O and a language command c t ∈ C at each time step t .

Original: c t can be generated from arbitrary distributions.
Revised: c t can be generated from arbitrary distributions.

Original: C denotes the space of possible natural language commands.
Revised: C denotes the space of possible natural language commands.

Original: An LSTM is used to encode a command c to an embedding.
Revised: An LSTM is used to encode a command c to an embedding.

Original: We use a special command “ NA ” to denote “ no command ”, which corresponds to a zero embedding.
Revised: We use a special command “ NA ” to denote “ no command ”, which corresponds to a zero embedding.

Original: When setting every c t to be NA , the command-following policy π θ degenerates to a conventional policy in classical RL without language commands.
Revised: When setting every c t to be NA , the command-following policy π θ degenerates to a conventional policy in classical RL without language commands.

Original: Human demonstrations.
Revised: Human demonstrations.

Original: We denote human demonstrations as a dataset of trajectories, i.e., D = { τ 1 , τ 2 , . . . } .
Revised: We denote human demonstrations as a dataset of trajectories, i.e., D = { τ 1 , τ 2 , . . . } .

Original: Each trajectory τ i , denoted by τ i =
Revised: Each trajectory τ i , denoted by τ i =

Original: ( s i 0 , a i 0 , c i 0 , s i 1 , a i 1 , c i 1 , . . . ) , consists of transitions from a single game played by a human executor and the paired command sequence from a human commander.
Revised: ( s i 0 , a i 0 , c i 0 , s i 1 , a i 1 , c i 1 , . . . ) , consists of transitions from a single game played by a human executor and the paired command sequence from a human commander.

Original: Human behaviors can be sub-optimal: i.e., given the observation o it
Revised: Human behaviors can be sub-optimal: i.e., given the observation o it

Original: = O
Revised: = O

Original: ( s it ) and the command c it , the chosen action a it can result in low rewards.
Revised: ( s it ) and the command c it , the chosen action a it can result in low rewards.

Original: Likewise, a command c it can be arbitrary w.r.t.
Revised: Likewise, a command c it can be arbitrary w.r.t.

Original: human preferences.
Revised: human preferences.

Original: Finally, since human commanders are always asked to generate language commands, D does not contain any NA command, i.e., c it ̸ = NA .
Revised: Finally, since human commanders are always asked to generate language commands, D does not contain any NA command, i.e., c it ̸ = NA .

Original:  Method 4.1 Grounded Reinforcement Learning: Problem Formulation
Revised:  Method 4.1 Grounded Reinforcement Learning: Problem Formulation

Original: The mission of GRL is to learn a policy π θ such that π θ can achieve high rewards in the environment and follow any possible command c ∈ C by achieving consistent behaviors with human demonstrations D .
Revised: The mission of GRL is to learn a policy π θ such that π θ can achieve high rewards in the environment and follow any possible command c ∈ C by achieving consistent behaviors with human demonstrations D .

Original: We formulate the GRL problem as the following RL objective J G :
Revised: We formulate the GRL problem as the following RL objective J G :

Original: [Equation]
Revised: [Equation]

Original: where K denotes a distance metric between the policy behaviors from π θ and the human data D .
Revised: where K denotes a distance metric between the policy behaviors from π θ and the human data D .

Original: Remark #1: The commands can be arbitrary.
Revised: Remark #1: The commands can be arbitrary.

Original: π θ needs to follow any possible command c ∈ C while still attains the highest rewards under the behavior constraint K ( π θ , D ) ≤ δ .
Revised: π θ needs to follow any possible command c ∈ C while still attains the highest rewards under the behavior constraint K ( π θ , D ) ≤ δ .

Original: Remark #2: Since the human actions in D can be sub-optimal w.r.t.
Revised: Remark #2: Since the human actions in D can be sub-optimal w.r.t.

Original: the paired commands, it can be problematic to directly perform behavior cloning (BC) over D .
Revised: the paired commands, it can be problematic to directly perform behavior cloning (BC) over D .

Original: Therefore, we utilize a threshold δ so that the command-following policy π θ can possibly deviate from the sub-optimal behaviors in D .
Revised: Therefore, we utilize a threshold δ so that the command-following policy π θ can possibly deviate from the sub-optimal behaviors in D .

Original: Note that Eq.
Revised: Note that Eq.

Original: 1 is generally intractable and is presented only for the purpose of problem definition.
Revised: 1 is generally intractable and is presented only for the purpose of problem definition.

Original: 4.2 Constrained RL
Revised: 4.2 Constrained RL

Original: as a Tractable Approximation for GRL
Revised: as a Tractable Approximation for GRL

Original: There are two issues when directly optimizing Eq.
Revised: There are two issues when directly optimizing Eq.

Original: First, the command c t can be arbitrary.
Revised: First, the command c t can be arbitrary.

Original: Second, the choice of behavior distance metric K ( π θ , D ) should be specified.
Revised: Second, the choice of behavior distance metric K ( π θ , D ) should be specified.

Original: Training commands.
Revised: Training commands.

Original: An obvious choice is to sample a random command c t from C at each time step t , which is perfectly aligned with Eq.
Revised: An obvious choice is to sample a random command c t from C at each time step t , which is perfectly aligned with Eq.

Original: However, inconsistent commands from random sampling are typically harmful for a win, making the RL agent tend to ignore the commands.
Revised: However, inconsistent commands from random sampling are typically harmful for a win, making the RL agent tend to ignore the commands.

Original: An alternative is a “human proxy” by learning a command model h ϕ ( c t | s t ) over human data D , leading to the objective E c t ∼ h ϕ ( s t ) ,a t ∼ π θ ( o t ,c t ) [ (cid:80) t r ( a t , s t )] .
Revised: An alternative is a “human proxy” by learning a command model h ϕ ( c t | s t ) over human data D , leading to the objective E c t ∼ h ϕ ( s t ) ,a t ∼ π θ ( o t ,c t ) [ (cid:80) t r ( a t , s t )] .

Original: Although this is reasonable, we need to be aware that human commands are sub-optimal and biased, which may further limit the chance for the policy π θ to discover strong strategies during RL training.
Revised: Although this is reasonable, we need to be aware that human commands are sub-optimal and biased, which may further limit the chance for the policy π θ to discover strong strategies during RL training.

Original: Thus, we propose an extreme version: i.e., directly performing unconditioned RL training by setting every command c t to be NA .
Revised: Thus, we propose an extreme version: i.e., directly performing unconditioned RL training by setting every command c t to be NA .

Original: This allows the policy to freely explore the strategy space during RL training for the best winning strategies.
Revised: This allows the policy to freely explore the strategy space during RL training for the best winning strategies.

Original: Command-following metric.
Revised: Command-following metric.

Original: The simplest distance metric K is the behavior cloning (BC) loss, i.e., negative log-likelihood (NLL) of π θ for ( s t , a t , c t ) ∈ D .
Revised: The simplest distance metric K is the behavior cloning (BC) loss, i.e., negative log-likelihood (NLL) of π θ for ( s t , a t , c t ) ∈ D .

Original: Specifically, we can define BC objective
Revised: Specifically, we can define BC objective

Original: L ( θ ) =
Revised: L ( θ ) =

Original: E D [ − log π θ ( a t | o t , c t )] and constrain L ( θ ) ≤ δ to ensure the policy behavior does not deviate from the demonstrations too much.
Revised: E D [ − log π θ ( a t | o t , c t )] and constrain L ( θ ) ≤ δ to ensure the policy behavior does not deviate from the demonstrations too much.

Original: There can be alternatives, such as KL-divergence or a learned metric.
Revised: There can be alternatives, such as KL-divergence or a learned metric.

Original: Regarding KL-divergence, since we are measuring the distance between samples
Revised: Regarding KL-divergence, since we are measuring the distance between samples

Original: D and a distribution π θ , KL ( D|| π θ ) is equivalent to L ( θ ) while KL ( π θ ||D ) requires an behavior proxy over D , which may involve additional biases.
Revised: D and a distribution π θ , KL ( D|| π θ ) is equivalent to L ( θ ) while KL ( π θ ||D ) requires an behavior proxy over D , which may involve additional biases.

Original: Regarding a learned metric, we can apply inverse
Revised: Regarding a learned metric, we can apply inverse

Original: RL to derive a reward function on whether the command is accomplished [23, 6, 7].
Revised: RL to derive a reward function on whether the command is accomplished [23, 6, 7].

Original: However, we empirically notice that a learning-based metric works poorly on MiniRTS.
Revised: However, we empirically notice that a learning-based metric works poorly on MiniRTS.

Original: We hypothesis that this is because the demonstrations are sub-optimal and possibly limited in size for such a complex game.
Revised: We hypothesis that this is because the demonstrations are sub-optimal and possibly limited in size for such a complex game.

Original: Tractable objective.
Revised: Tractable objective.

Original: To sum up, we propose the following constrained RL objective J C ( θ ) as a tractable objective for the GRL problem, i.e.,
Revised: To sum up, we propose the following constrained RL objective J C ( θ ) as a tractable objective for the GRL problem, i.e.,

Original: [Equation]
Revised: [Equation]

Original: subject
Revised: subject

Original: [Equation]
Revised: [Equation]

Original: Note that in Eq.
Revised: Note that in Eq.

Original: (2), the sample distribution from the RL objective J C is disjoint from the BC objective L ( θ ; D ) since we force c t =
Revised: (2), the sample distribution from the RL objective J C is disjoint from the BC objective L ( θ ; D ) since we force c t =

Original: NA in RL training while D does not contain NA at all.
Revised: NA in RL training while D does not contain NA at all.

Original: Hence, Eq.
Revised: Hence, Eq.

Original: (2) may look a bit ill-posed: an obvious solution is a “switching” policy which takes either the pure RL strategy or the BC strategy with or without a command.
Revised: (2) may look a bit ill-posed: an obvious solution is a “switching” policy which takes either the pure RL strategy or the BC strategy with or without a command.

Original: However, we will empirically show that such a switching policy does not provide strong performances.
Revised: However, we will empirically show that such a switching policy does not provide strong performances.

Original: The insight is that we are training a conditioned neural policy .
Revised: The insight is that we are training a conditioned neural policy .

Original: It is often observed in practice (e.g., multi-task learning) that a neural network can implicitly distill strategies from different modalities via its shared representations [20, 54, 76].
Revised: It is often observed in practice (e.g., multi-task learning) that a neural network can implicitly distill strategies from different modalities via its shared representations [20, 54, 76].

Original: Therefore, the winning strategy discovered from RL training can empirically improve the sub-optimal command-following demonstrations within a bounded range.
Revised: Therefore, the winning strategy discovered from RL training can empirically improve the sub-optimal command-following demonstrations within a bounded range.

Original: This phenomenon can be also justified by some recent theoretical findings [48, 16].
Revised: This phenomenon can be also justified by some recent theoretical findings [48, 16].

Original: 4.3 Reinforced Demonstration Distillation
Revised: 4.3 Reinforced Demonstration Distillation

Original: A straightforward way to solve the constrained optimization problem in Eq.
Revised: A straightforward way to solve the constrained optimization problem in Eq.

Original: (2) is to adopt the
Revised: (2) is to adopt the

Original: Lagrangian multiplier, which leads to a joint optimization problem, i.e.,
Revised: Lagrangian multiplier, which leads to a joint optimization problem, i.e.,

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: (3) treats the BC objective L ( θ ; D ) as an auxiliary loss of the stardard RL objective J C ( θ ) .
Revised: (3) treats the BC objective L ( θ ; D ) as an auxiliary loss of the stardard RL objective J C ( θ ) .

Original: The constraint in Eq.
Revised: The constraint in Eq.

Original: (2) can be satisfied by tuning the coefficient β .
Revised: (2) can be satisfied by tuning the coefficient β .

Original: However, due to the distribution shift issue between on-policy samples and demonstrations, the supervised learning loss may interfere with the policy gradient and further makes the training process sensitive and unstable [21, 40].
Revised: However, due to the distribution shift issue between on-policy samples and demonstrations, the supervised learning loss may interfere with the policy gradient and further makes the training process sensitive and unstable [21, 40].

Original: Iterative solution.
Revised: Iterative solution.

Original: Inspired by the recent advances in self-imitation [53] and self-distillation [78, 43, 42], we adopt an iterative framework to decouple the RL loss and the BC loss in Eq.
Revised: Inspired by the recent advances in self-imitation [53] and self-distillation [78, 43, 42], we adopt an iterative framework to decouple the RL loss and the BC loss in Eq.

Original: The idea is simple: for RL training, we solely estimate the unconstrained policy gradient over J C without considering the BC loss; after policy improvement, we distill the demonstrations into the policy by running pure behavior cloning over both human demonstrations and self-generated winning trajectories.
Revised: The idea is simple: for RL training, we solely estimate the unconstrained policy gradient over J C without considering the BC loss; after policy improvement, we distill the demonstrations into the policy by running pure behavior cloning over both human demonstrations and self-generated winning trajectories.

Original: By repeatedly alternating between pure RL and pure BC, we are able to achieve a stable learning process.
Revised: By repeatedly alternating between pure RL and pure BC, we are able to achieve a stable learning process.

Original: We call this method, REinforced demonstration Distillation (RED).
Revised: We call this method, REinforced demonstration Distillation (RED).

Original: Specifically, let θ k denote the parameters at iteration k , then we have the following update rule.
Revised: Specifically, let θ k denote the parameters at iteration k , then we have the following update rule.

Original: [Equation]
Revised: [Equation]

Original: Here α denotes the learning rate.
Revised: Here α denotes the learning rate.

Original: By repeating the update rules for N iterations, we will derive our final parameter θ ⋆ .
Revised: By repeating the update rules for N iterations, we will derive our final parameter θ ⋆ .

Original: The remaining issue is to ensure the constraint is satisfied, i.e., L ( θ ; D ) ≤ δ .
Revised: The remaining issue is to ensure the constraint is satisfied, i.e., L ( θ ; D ) ≤ δ .

Original: We empirically notice that this can be accomplished by controlling the ratio between the size of on-policy samples ∥D k ∥ and the size of demonstrations ∥D∥ .
Revised: We empirically notice that this can be accomplished by controlling the ratio between the size of on-policy samples ∥D k ∥ and the size of demonstrations ∥D∥ .

Original: We present an intuitive justification of the self-distillation process from the perspective of sample distributions [9, 8].
Revised: We present an intuitive justification of the self-distillation process from the perspective of sample distributions [9, 8].

Original: We adopt RL for the highest rewards, so the unconditioned policy may frequently visit those states approaching a win.
Revised: We adopt RL for the highest rewards, so the unconditioned policy may frequently visit those states approaching a win.

Original: As a result, for games played by strong humans with aggressive commands, the paired game states will be more aligned with the RL state distribution while the states produced by biased or defensive human commanders will be more apart.
Revised: As a result, for games played by strong humans with aggressive commands, the paired game states will be more aligned with the RL state distribution while the states produced by biased or defensive human commanders will be more apart.

Original: As we are distilling two drift distributions into a single neural policy π θ , it will be more likely to get actions over those overlapping states improved within a bounded BC loss during optimization (see evidences in Sec. 5.3).
Revised: As we are distilling two drift distributions into a single neural policy π θ , it will be more likely to get actions over those overlapping states improved within a bounded BC loss during optimization (see evidences in Sec. 5.3).

Original: 4.4 Implementations
Revised: 4.4 Implementations

Original: We highlight some critical implementation factors here.
Revised: We highlight some critical implementation factors here.

Original: More details are in appendix.
Revised: More details are in appendix.

Original: Policy architecture.
Revised: Policy architecture.

Original: We adopt a similar policy architecture to the provided executor model in
Revised: We adopt a similar policy architecture to the provided executor model in

Original: MiniRTS [33] but with a modified action space and an additional value head as the critic.
Revised: MiniRTS [33] but with a modified action space and an additional value head as the critic.

Original: The original model outputs an action for every controllable unit, which makes RL training extremely slow.
Revised: The original model outputs an action for every controllable unit, which makes RL training extremely slow.

Original: We introduce an additional 0/1 selection action on each unit denoting whether this unit should act or not , which substantially reduces the action dimension.
Revised: We introduce an additional 0/1 selection action on each unit denoting whether this unit should act or not , which substantially reduces the action dimension.

Original: Policy learning.
Revised: Policy learning.

Original: We use PPO [59] for RL training, which involves multiple mini-batch policy gradient steps.
Revised: We use PPO [59] for RL training, which involves multiple mini-batch policy gradient steps.

Original: Since RL from scratch fails completely in practice, we fine-tune the BC policy and pretrain the value head with the policy parameters frozen.
Revised: Since RL from scratch fails completely in practice, we fine-tune the BC policy and pretrain the value head with the policy parameters frozen.

Original: We also empirically find learning rate warmup [41] particularly helpful for stabilizing training with a pretrained model: we start with learning rate 0 and then linearly increase it to the desired value α .
Revised: We also empirically find learning rate warmup [41] particularly helpful for stabilizing training with a pretrained model: we start with learning rate 0 and then linearly increase it to the desired value α .

Original: Creating D k .
Revised: Creating D k .

Original: Note that in the early stage of training phase, the winning rate can be low.
Revised: Note that in the early stage of training phase, the winning rate can be low.

Original: In order to obtain sufficient data for D k , we implement a queue to store all past winning trajectories and use bootstrapped sampling if the queue is still not full.
Revised: In order to obtain sufficient data for D k , we implement a queue to store all past winning trajectories and use bootstrapped sampling if the queue is still not full.

Original: For the ratio between ∥D k ∥ and ∥D∥ , we ensure the BC loss (i.e., negative log likelihood) of π ⋆θ over the validation set is at most 10% more than the pure BC policy (i.e., no RL training).
Revised: For the ratio between ∥D k ∥ and ∥D∥ , we ensure the BC loss (i.e., negative log likelihood) of π ⋆θ over the validation set is at most 10% more than the pure BC policy (i.e., no RL training).

Original: We can also run sub-sampling over D to reduce the required sample size.
Revised: We can also run sub-sampling over D to reduce the required sample size.

Original: Empirically, we find that simply setting ∥D k ∥ =
Revised: Empirically, we find that simply setting ∥D k ∥ =

Original: ∥D∥ leads to strong performances.
Revised: ∥D∥ leads to strong performances.

Original:  Experiment
Revised:  Experiment

Original: All the simulation win rates are based on 1200 test games and repeated over 3 random seeds.
Revised: All the simulation win rates are based on 1200 test games and repeated over 3 random seeds.

Original: More details and additional results including emergent behaviors are deferred to appendix.
Revised: More details and additional results including emergent behaviors are deferred to appendix.

Original: Baselines.
Revised: Baselines.

Original: We consider the following methods in addition to our RED algorithm as baselines.
Revised: We consider the following methods in addition to our RED algorithm as baselines.

Original: “ RL ” first pretrains the policy over demonstrations and then performs pure unconditioned RL training
Revised: “ RL ” first pretrains the policy over demonstrations and then performs pure unconditioned RL training

Original: (i
Revised: (i

Original: e., conditioning on “ NA ” command) without any command-following constraint.
Revised: e., conditioning on “ NA ” command) without any command-following constraint.

Original: “ Switch ” is the “switching” policy, which consists of two policies, a pure BC policy from demonstrations and a pure RL policy without commands.
Revised: “ Switch ” is the “switching” policy, which consists of two policies, a pure BC policy from demonstrations and a pure RL policy without commands.

Original: When a langauge command is given, it runs the BC policy and uses the RL policy otherwise.
Revised: When a langauge command is given, it runs the BC policy and uses the RL policy otherwise.

Original: “ Joint ” optimizes the soft objective J softC in Eq.
Revised: “ Joint ” optimizes the soft objective J softC in Eq.

Original: β is selected via a grid-search process.
Revised: β is selected via a grid-search process.

Original: “ IRL ” learns a reward function from demonstrations [24] and combines the game reward and the learned language reward to train a conditioned policy.
Revised: “ IRL ” learns a reward function from demonstrations [24] and combines the game reward and the learned language reward to train a conditioned policy.

Original: Evaluation.
Revised: Evaluation.

Original: To evaluate a strong GRL policy, we need to answer the following questions.
Revised: To evaluate a strong GRL policy, we need to answer the following questions.

Original: Does the policy learn a strong winning strategy?
Revised: Does the policy learn a strong winning strategy?

Original: The win rate can be a direct metric for this question.
Revised: The win rate can be a direct metric for this question.

Original: A RED policy should largely improves a pure BC policy with a high win rate.
Revised: A RED policy should largely improves a pure BC policy with a high win rate.

Original: Does the policy follow the commands well?
Revised: Does the policy follow the commands well?

Original: A good GRL policy should keep its winning strategy while follow arbitrary commands
Revised: A good GRL policy should keep its winning strategy while follow arbitrary commands

Original: A precise evaluation of this question is non-trivial since the constraint satisfying condition, i.e., BC loss, can be a very misleading signal 2 .
Revised: A precise evaluation of this question is non-trivial since the constraint satisfying condition, i.e., BC loss, can be a very misleading signal 2 .

Original: We adopt an approximate measurement via win rates in Sec.
Revised: We adopt an approximate measurement via win rates in Sec.

Original: and then conduct human evaluation in Sec. 5.4.
Revised: and then conduct human evaluation in Sec. 5.4.

Original: Ablation studies on algorithmic hyper-parameters of our RED algorithm are also conducted in Sec. 5.
Revised: Ablation studies on algorithmic hyper-parameters of our RED algorithm are also conducted in Sec. 5.

Original: 5.1 Main Results
Revised: 5.1 Main Results

Original: Q#1: is the RED policy strong?
Revised: Q#1: is the RED policy strong?

Original: We measure the win rates of different policies under two test-time command strategies, i.e., a pure NA strategy, which always gives an NA , and an Oracle strategy, which gives “optimal” commands.
Revised: We measure the win rates of different policies under two test-time command strategies, i.e., a pure NA strategy, which always gives an NA , and an Oracle strategy, which gives “optimal” commands.

Original: For the NA strategy, the GRL problem degenerates to the conventional
Revised: For the NA strategy, the GRL problem degenerates to the conventional

Original: RL setting, so a strong GRL policy should at least achieve comparable performances to the RL policy and substantially outperforms the BC (i.e., Switch ) policy.
Revised: RL setting, so a strong GRL policy should at least achieve comparable performances to the RL policy and substantially outperforms the BC (i.e., Switch ) policy.

Original: For the Oracle strategy, it was scripted to give carefully tuned commands based on the ground-truth game state to instruct the policy to build dominating units according to the rock-paper-scissors dynamics.
Revised: For the Oracle strategy, it was scripted to give carefully tuned commands based on the ground-truth game state to instruct the policy to build dominating units according to the rock-paper-scissors dynamics.

Original: Hence, the GRL problem is converted to a fully cooperative game, and following Oracle commands becomes the optimal executor strategy.
Revised: Hence, the GRL problem is converted to a fully cooperative game, and following Oracle commands becomes the optimal executor strategy.

Original: A strong GRL policy should achieve the highest possible win rate.
Revised: A strong GRL policy should achieve the highest possible win rate.

Original: Meanwhile, the policy with the highest win rate must produce a good command-following behavior.
Revised: Meanwhile, the policy with the highest win rate must produce a good command-following behavior.

Original: We also report the win rates of different methods with random commands and a “human proxy” commander, which is learned from human data.
Revised: We also report the win rates of different methods with random commands and a “human proxy” commander, which is learned from human data.

Original: Since Random and Human Proxy are sub-optimal, a good GRL policy should have a substantially lower win rate than the NA case but still outperforms BC ( Switch ) policy.
Revised: Since Random and Human Proxy are sub-optimal, a good GRL policy should have a substantially lower win rate than the NA case but still outperforms BC ( Switch ) policy.

Original: The results are summarized in Tab. 1.
Revised: The results are summarized in Tab. 1.

Original: The win rate by RED is the highest under oracle commands and is comparable to RL and Switch without commands ( NA ).
Revised: The win rate by RED is the highest under oracle commands and is comparable to RL and Switch without commands ( NA ).

Original: Both RED and Joint policies substantially outperform Switch (BC) policy with sub-optimal commands (Random and Human Proxy), suggesting the conditioned behaviors are improved.
Revised: Both RED and Joint policies substantially outperform Switch (BC) policy with sub-optimal commands (Random and Human Proxy), suggesting the conditioned behaviors are improved.

Original: Neither RL nor IRL is able to make the policy obedient – even under random commands, their win rates remain comparable to the NA case.
Revised: Neither RL nor IRL is able to make the policy obedient – even under random commands, their win rates remain comparable to the NA case.

Original: We also note that commands from human proxy generally yield an even lower win rate than random commands.
Revised: We also note that commands from human proxy generally yield an even lower win rate than random commands.

Original: We remark that learning a good human commander model is extremely hard – not only due to the limited human command data in the dataset but also because most of the commands are highly sub-optimal.
Revised: We remark that learning a good human commander model is extremely hard – not only due to the limited human command data in the dataset but also because most of the commands are highly sub-optimal.

Original: [Table]
Revised: [Table]

Original: An obedient policy should have a decreasing win rate.
Revised: An obedient policy should have a decreasing win rate.

Original: Here we present an additional criterion: intuitively, as a command-following policy, its win rate should decrease if “worse” commands are fed; otherwise, if the win rate does not decay, the policy must not follow the commands well.
Revised: Here we present an additional criterion: intuitively, as a command-following policy, its win rate should decrease if “worse” commands are fed; otherwise, if the win rate does not decay, the policy must not follow the commands well.

Original: Hence, we interpolate between the NA strategy and the full Random strategy and evaluate the win rate of different policies with different ratios of random commands.
Revised: Hence, we interpolate between the NA strategy and the full Random strategy and evaluate the win rate of different policies with different ratios of random commands.

Original: The results are shown in
Revised: The results are shown in

Original: Fig.
Revised: Fig.

Original: We can observe that the RL policy and IRL policy are clearly not following the commands.
Revised: We can observe that the RL policy and IRL policy are clearly not following the commands.

Original: Among the remaining command-following policies, RED policy generally produces a higher win rate than Switch and Joint , which provides evidences that our RED algorithm leads to improved strategies under the command-following requirement.
Revised: Among the remaining command-following policies, RED policy generally produces a higher win rate than Switch and Joint , which provides evidences that our RED algorithm leads to improved strategies under the command-following requirement.

Original: We also try to evaluate the command-following ability directly.
Revised: We also try to evaluate the command-following ability directly.

Original: At the beginning of a game, we provide a sequence of commands to guide the policies to build a specific army unit, and evaluate the success rates in 30 steps.
Revised: At the beginning of a game, we provide a sequence of commands to guide the policies to build a specific army unit, and evaluate the success rates in 30 steps.

Original: The results are listed in Tab. 2.
Revised: The results are listed in Tab. 2.

Original: We can observe that the Switch, Joint, and
Revised: We can observe that the Switch, Joint, and

Original: RED policies achieve significantly higher success rates than IRL.
Revised: RED policies achieve significantly higher success rates than IRL.

Original: This suggests that Switch, Joint, and RED faithfully follow the commands, while IRL tends to be command-ignorant.
Revised: This suggests that Switch, Joint, and RED faithfully follow the commands, while IRL tends to be command-ignorant.

Original: Command-Ignorant
Revised: Command-Ignorant

Original: [Table]
Revised: [Table]

Original: -Following
Revised: -Following

Original: [Table]
Revised: [Table]

Original: It is also worth noting that RL still achieves reasonable success rates.
Revised: It is also worth noting that RL still achieves reasonable success rates.

Original: This is probably caused by the fact that RL policy is initialized with the BC policy and keeps a weak command-following ability.
Revised: This is probably caused by the fact that RL policy is initialized with the BC policy and keeps a weak command-following ability.

Original: Building a single unit at the beginning of a game is relatively easy.
Revised: Building a single unit at the beginning of a game is relatively easy.

Original: However, following commands in a complex situation later in the game may be more difficult.
Revised: However, following commands in a complex situation later in the game may be more difficult.

Original: 5.2 Out-of-Distribution Commands
Revised: 5.2 Out-of-Distribution Commands

Original: As mentioned in Sec. 3.2, we adopt an LSTM encoder to encode arbitrary natural language commands into fixed-length sentence embeddings.
Revised: As mentioned in Sec. 3.2, we adopt an LSTM encoder to encode arbitrary natural language commands into fixed-length sentence embeddings.

Original: During the training process, there are a total of 38,  different commands in the training set.
Revised: During the training process, there are a total of 38,  different commands in the training set.

Original: We also evaluate how the policies perform on Out-ofDistribution (OOD) commands.
Revised: We also evaluate how the policies perform on Out-ofDistribution (OOD) commands.

Original: As shown in Tab. 3, we try 4 different ways of adding noise to Oracle commands and report the win rates.
Revised: As shown in Tab. 3, we try 4 different ways of adding noise to Oracle commands and report the win rates.

Original: We can observe that the agents under noisy oracle commands perform better than Random commands but worse than Oracle .
Revised: We can observe that the agents under noisy oracle commands perform better than Random commands but worse than Oracle .

Original: Intuitively, we can observe that the win rate roughly follows “ Random ” < “ Drop ” ∼ “ Replace ” < “ Shuffle ” ∼ “ Insert ” <
Revised: Intuitively, we can observe that the win rate roughly follows “ Random ” < “ Drop ” ∼ “ Replace ” < “ Shuffle ” ∼ “ Insert ” <

Original: “ Oracle ”.
Revised: “ Oracle ”.

Original: This suggests that the LSTM encoder is sensitive to input words and word order.
Revised: This suggests that the LSTM encoder is sensitive to input words and word order.

Original: We also find that strong-following policies are affected more by noisy commands than weak-following policies, indicating that strong-following policies are more sensitive to the commands.
Revised: We also find that strong-following policies are affected more by noisy commands than weak-following policies, indicating that strong-following policies are more sensitive to the commands.

Original: In addition, we also investigate the influence of Out-of-Vocabulary (OOV) words in Appendix D.5, and visualize the command representations in Appendix D.6.
Revised: In addition, we also investigate the influence of Out-of-Vocabulary (OOV) words in Appendix D.5, and visualize the command representations in Appendix D.6.

Original: [Table]
Revised: [Table]

Original: , we delete 50% of words in each command.
Revised: , we delete 50% of words in each command.

Original: For Replace, we replace 50% of the words with random words.
Revised: For Replace, we replace 50% of the words with random words.

Original: For Insert, we insert random words and make the original command twice longer.
Revised: For Insert, we insert random words and make the original command twice longer.

Original: For Shuffle, we shuffle the words in each command.
Revised: For Shuffle, we shuffle the words in each command.

Original: 5.3 Ablation Study
Revised: 5.3 Ablation Study

Original: Training commands.
Revised: Training commands.

Original: We test the performance of RED with different training commands in Fig.
Revised: We test the performance of RED with different training commands in Fig.

Original: The default RED yields the best strategy, i.e., the highest win rate with NA and oracle commands.
Revised: The default RED yields the best strategy, i.e., the highest win rate with NA and oracle commands.

Original: Training with random commands or human proxy makes the policy command-ignorant – both two variants achieve similar win rates under NA and random commands.
Revised: Training with random commands or human proxy makes the policy command-ignorant – both two variants achieve similar win rates under NA and random commands.

Original: [Table]
Revised: [Table]

Original: Ratio of ∥D k ∥ and ∥D∥ .
Revised: Ratio of ∥D k ∥ and ∥D∥ .

Original: We test different ratios of on-policy samples, i.e., ∥D k ∥ , to the demonstration size, i.e., ∥D∥ , in Tab. 4.
Revised: We test different ratios of on-policy samples, i.e., ∥D k ∥ , to the demonstration size, i.e., ∥D∥ , in Tab. 4.

Original: We report the test-time win rates with oracle and NA commands as well as the constraint satisfaction condition, including both NLL (i.e., L ( θ ; D ) from Eq. 2) and the action prediction accuracy, on the validation set.
Revised: We report the test-time win rates with oracle and NA commands as well as the constraint satisfaction condition, including both NLL (i.e., L ( θ ; D ) from Eq. 2) and the action prediction accuracy, on the validation set.

Original: The results show that the validation NLL can be indeed controlled by tuning the dataset ratio, i.e., more RL samples consistently yielding a higher NLL.
Revised: The results show that the validation NLL can be indeed controlled by tuning the dataset ratio, i.e., more RL samples consistently yielding a higher NLL.

Original: By contrast, we also find that a lower NLL does NOT necessarily lead to a higher action prediction accuracy.
Revised: By contrast, we also find that a lower NLL does NOT necessarily lead to a higher action prediction accuracy.

Original: But we empirically find that simply setting a 1:1 ratio leads to a sufficiently good policy, which is the default choice for our RED algorithm.
Revised: But we empirically find that simply setting a 1:1 ratio leads to a sufficiently good policy, which is the default choice for our RED algorithm.

Original: Learned strategy.
Revised: Learned strategy.

Original: We identify the five most common categories of commands and measure the validation action prediction accuracy for both RED and BC policies to examine under what conditions the actions are changed – noting that RED policy uses BC policy as a warm-start.
Revised: We identify the five most common categories of commands and measure the validation action prediction accuracy for both RED and BC policies to examine under what conditions the actions are changed – noting that RED policy uses BC policy as a warm-start.

Original: Fig. 4 shows the categorized accuracy differences with the dashed line denoting the overall mean difference.
Revised: Fig.shows the categorized accuracy differences with the dashed line denoting the overall mean difference.

Original: We can observe that RED policy disagrees with human actions the most under commands belonging to “attack” and “defend” categories, which both represent a highly complicated space of possible behaviors (e.g., which enemy unit to attack, how to defend).
Revised: We can observe that RED policy disagrees with human actions the most under commands belonging to “attack” and “defend” categories, which both represent a highly complicated space of possible behaviors (e.g., which enemy unit to attack, how to defend).

Original: We believe these action changes are caused by policy improvement.
Revised: We believe these action changes are caused by policy improvement.

Original: While for commands with a low uncertainty belonging to “mine” and “stop” categories, RED policy simply follows humans.
Revised: While for commands with a low uncertainty belonging to “mine” and “stop” categories, RED policy simply follows humans.

Original: These findings provide empirical supports to our intuitive justification in Sec. 4.3.
Revised: These findings provide empirical supports to our intuitive justification in Sec. 4.3.

Original: 5.4 Human Evaluation
Revised: 5.4 Human Evaluation

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: We pick 4 policies, i.e., RED, Switch, Joint and RL, and invite 30 college student volunteers under department permission to complete a two-stage study.
Revised: We pick 4 policies, i.e., RED, Switch, Joint and RL, and invite 30 college student volunteers under department permission to complete a two-stage study.

Original: In the first stage, we shuffle the order of the 4 policies and ask each student to keep playing with the 4 policies using any commands.
Revised: In the first stage, we shuffle the order of the 4 policies and ask each student to keep playing with the 4 policies using any commands.

Original: Once the student feels familiar with the game and policies enough, he/she is asked to rank the policies by the level of how they follow commands.
Revised: Once the student feels familiar with the game and policies enough, he/she is asked to rank the policies by the level of how they follow commands.

Original: In the second part, each student is asked to play 2 games per policy, and we measure the average winning rate of different policies over the 60 collected game episodes.
Revised: In the second part, each student is asked to play 2 games per policy, and we measure the average winning rate of different policies.

Original: The results of ranking votes and win rates are shown in Fig.
Revised: The results of ranking votes and win rates are shown in Fig.

Original: 5 and Fig.
Revised: 5 and Fig.

Original: Regarding the votes, the average rank of RED, Switch, Joint and RL are 2.3, 2.4, 2.5, 2.8, respectively.
Revised: Regarding the votes, the average rank of RED, Switch, Joint and RL are 2.3, 2.4, 2.5, 2.8, respectively.

Original: Fig.
Revised: Fig.

Original: 5 visualizes the vote percentage for different policies.
Revised: 5 visualizes the vote percentage for different policies.

Original: We notice a very interesting phenomenon that every policy has almost the same 1st-rank votes.
Revised: We notice a very interesting phenomenon that every policy has almost the same 1st-rank votes.

Original: Knowing that we shuffle the policy order while the students often spend a long time on the first policy to get familiar with MiniRTS, we hypothesize that the students are often biased towards the first policy they encountered.
Revised: Knowing that we shuffle the policy order while the students often spend a long time on the first policy to get familiar with MiniRTS, we hypothesize that the students are often biased towards the first policy they encountered.

Original: Nevertheless, RED has substantially more votes for the 2nd-rank, indicating a better command-following capacity.
Revised: Nevertheless, RED has substantially more votes for the 2nd-rank, indicating a better command-following capacity.

Original: RL has the most 4th-rank votes, which is consistent with our expectation.
Revised: RL has the most 4th-rank votes, which is consistent with our expectation.

Original: Regarding the win rates (Fig. 6), RED outperforms
Revised: Regarding the win rates (Fig. 6), RED outperforms

Original: Switch and Joint with a clear margin.
Revised: Switch and Joint with a clear margin.

Original: The win rate of RL is comparable with RED.
Revised: The win rate of RL is comparable with RED.

Original: We believe this is due to the fact that RL policy often ignores human commands and simply acts for winning.
Revised: We believe this is due to the fact that RL policy often ignores human commands and simply acts for winning.

Original:  Conclusion
Revised:  Conclusion

Original: We tackle a new problem, grounded reinforcement learning (GRL), which aims to learn an agent that can not only get high rewards but also faithfully follow natural language commands from humans.
Revised: We tackle a new problem, grounded reinforcement learning (GRL), which aims to learn an agent that can not only get high rewards but also faithfully follow natural language commands from humans.

Original: We proposed a tractable objective, and developed an iterative RL algorithm RED, and evaluated the derived policy on a real-time strategy game MiniRTS.
Revised: We proposed a tractable objective, and developed an iterative RL algorithm RED, and evaluated the derived policy on a real-time strategy game MiniRTS.

Original: Both simulation and human results show that
Revised: Both simulation and human results show that

Original: a
Revised: a

Original: RED policy achieves high win rates while exhibits strong command-following capabilities.
Revised: RED policy achieves high win rates while exhibits strong command-following capabilities.

Original: Limitation and social impact.
Revised: Limitation and social impact.

Original: As an initial study on the GRL problem, the RED algorithm is only evaluated on MiniRTS.
Revised: As an initial study on the GRL problem, the RED algorithm is only evaluated on MiniRTS.

Original: This is because MiniRTS is the only public environment that is both complex (beyond “reaching goals”) and provides natural language data.
Revised: This is because MiniRTS is the only public environment that is both complex (beyond “reaching goals”) and provides natural language data.

Original: In addition, although we draw connections to existing theories in multi-task learning to justify why RED works, the theoretical principle of the underlying optimization process remains an open problem.
Revised: In addition, although we draw connections to existing theories in multi-task learning to justify why RED works, the theoretical principle of the underlying optimization process remains an open problem.

Original: Finally, even though it is in general debatable whether a robot should be fully obedient [18, 26, 50, 58], we believe that in restricted single-agent domains, like playing video games, strictly enforcing command-following behaviors should not result in worse negative social impact than BC or other RL methods. 
Revised: Finally, even though it is in general debatable whether a robot should be fully obedient [18, 26, 50, 58], we believe that in restricted single-agent domains, like playing video games, strictly enforcing command-following behaviors should not result in worse negative social impact than BC or other RL methods. Acknowledgement

Revised: Yi Wu is supported by 2030 Innovation Megaprojects of China (Programme on New Generation

Revised: Artificial Intelligence)

Revised: Grant No. 2021AAA0150000.

Original: A MiniRTS Details & Dataset
Revised: A MiniRTS Details & Dataset

Original: In this section, we describe the details of MiniRTS Environment and human dataset.
Revised: In this section, we describe the details of MiniRTS Environment and human dataset.

Original: Our work is based on the game and dataset released at https://github.com/facebookresearch/minirts under CC BY-NC 4.0 license.
Revised: Our work is based on the game and dataset released at https://github.com/facebookresearch/minirts under CC BY-NC 4.0 license.

Original: The data do not contain any personally identifiable information or offensive content.
Revised: The data do not contain any personally identifiable information or offensive content.

Revised: A.1 Game Design

Revised: rock-paper

Revised: scissors attack graph

Revised: each army type

Original:  For example restrains “spearman” but is retrained 
Revised: has some units it is effective against and vulnerable to. For example, “swordman” restrains “spearman” but is retrained by “cavarly”. “swordman”, ”spearman” and “cavalry” all are effective against “archer”

Original: A.1 Game Design Game Units 
Revised: Game Units There are

Original: There are 3 kinds of units in MiniRTS, including resource units, building units, and army units.
Revised: 3 kinds of units in MiniRTS including resource units, building units, and army units.

Original: • Resource Units: Resource units are stationary and neutral.
Revised: • Resource Units: Resource units are stationary and neutral.

Original: Resource units cannot be constructed by anyone and are created at the beginning of a game.
Revised: Resource units cannot be constructed by anyone and are created at the beginning of a game.

Original: Only “peasant” (an army unit type) of both teams can mine from the resource units.
Revised: Only “peasant” (an army unit type) of both teams can mine from the resource units.

Original: One mine action could gather resources from the resource units, and the mined resources are necessary to build new building units or army units.
Revised: One mine action could gather resources from the resource units, and the mined resources are necessary to build new building units or army units.

Original: • Building Units: MiniRTS supports 6 different building unit types.
Revised: • Building Units: MiniRTS supports 6 different building unit types.

Original: 5 of the building unit types can produce particular army units by consuming resources (Fig. 8).
Revised: 5 of the building unit types can produce particular army units by consuming resources (Fig. 8).

Original: The “guard tower” can not produce army units but can attack enemies.
Revised: The “guard tower” can not produce army units but can attack enemies.

Original: All the building units cannot move.
Revised: All the building units cannot move.

Original: Building units can be constructed by “peasant” at any available map location.
Revised: Building units can be constructed by “peasant” at any available map location.

Original: • Army Units: 7 types of army units can move and attack enemies.
Revised: • Army Units: 7 types of army units can move and attack enemies.

Original: Specifically, a “peasant” can mine resources from resource units and construct building units with mined resources, but its attack power is low.
Revised: Specifically, a “peasant” can mine resources from resource units and construct building units with mined resources, but its attack power is low.

Original: The other 6 army unit types and “guard tower” are designed with a rock-paper-scissors dynamic.
Revised: The other 6 army unit types and “guard tower” are designed with a rock-paper-scissors dynamic.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 7, each type has some units that it is effective against and vulnerable to.
Revised: 7, each type has some units that it is effective against and vulnerable to.

Original: Game Map
Revised: Game Map

Original: The game map is a discrete grid of 32x32, where each cell can either be grass or river.
Revised: The game map is a discrete grid of 32x32, where each cell can either be grass or river.

Original: The grass cell is available for constructing building units and is passable for any army unit, while the river cell only allows the “dragon” to go through.
Revised: The grass cell is available for constructing building units and is passable for any army unit, while the river cell only allows the “dragon” to go through.

Original: The map of each game is generated randomly.
Revised: The map of each game is generated randomly.

Original: [Table]

Original: When initializing the map, one “town hall” and three “peasant” are placed for each player, then river cells and several resource units are added randomly.
Revised: When initializing the map, one “town hall” and three “peasant” are placed for each player, then river cells and several resource units are added randomly.

Original: This generation phase ensures at least one path between two players’ “town hall”, and there are approximately equal resource units around each “town hall”.
Revised: This generation phase ensures at least one path between two players’ “town hall”, and there are approximately equal resource units around each “town hall”.

Original: In addition, the map is partially observable due to “fog-of-war”.
Revised: In addition, the map is partially observable due to “fog-of-war”.

Original: A.2 MiniRTS as an RL Environment
Revised: A.2 MiniRTS as an RL Environment

Original: As an RL environment, a player is controlled by the RL agent while the opponent is a built-in script
Revised: As an RL environment, a player is controlled by the RL agent while the opponent is a built-in script

Original: AI.
Revised: AI.

Original: The RL agent needs to control units to collect resources, do construction and kill all the enemy units or destroy the enemy base (i.e., “town hall”) to win a game.
Revised: The RL agent needs to control units to collect resources, do construction and kill all the enemy units or destroy the enemy base (i.e., “town hall”) to win a game.

Original: We limit the length of a game to a maximum of 256 time steps. 
Revised: We limit the length of a game to a maximum of 256 time steps. [Table]

Original: Observation Space
Revised: Observation Space

Original: The observation of an agent includes a 32x32 map and extra states of the game (e.g., health points of the observable units, the amount of resources, etc.).
Revised: The observation of an agent includes a 32x32 map and extra states of the game (e.g., health points of the observable units, the amount of resources, etc.).

Original: The regions not visited are masked in the observation, and unseen enemy units are removed.
Revised: The regions not visited are masked in the observation, and unseen enemy units are removed.

Original: Action Space
Revised: Action Space

Original: At each time step, the environment requires an action for each controlled unit, leading to a particularly large action space.
Revised: At each time step, the environment requires an action for each controlled unit, leading to a particularly large action space.

Original: We introduce an additional 0/1 action for each unit, denoting whether this unit should act or not.
Revised: We introduce an additional 0/1 action for each unit, denoting whether this unit should act or not.

Original: We generate the same action for those selected units, which reduces the action dimension substantially.
Revised: We generate the same action for those selected units, which reduces the action dimension substantially.

Original: In particular, our policy network outputs a common action as well as a 0/1 flag for each unit.
Revised: In particular, our policy network outputs a common action as well as a 0/1 flag for each unit.

Original: Then we convert this into the standard MiniRTS format in the following way: for a unit assigned “1”, it executes the common output action, for a unit assigned “0”, it executes the action CONTINUE.
Revised: Then we convert this into the standard MiniRTS format in the following way: for a unit assigned “1”, it executes the common output action, for a unit assigned “0”, it executes the action CONTINUE.

Original: After determining which units should act, following [33], we first predict an action type (e.g., MOVE, ATTACK), then predict the action outputs based on the action type.
Revised: After determining which units should act, following [33], we first predict an action type (e.g., MOVE, ATTACK), then predict the action outputs based on the action type.

Original: We summarize all available action types and their structure in Tab. 5.
Revised: We summarize all available action types and their structure in Tab. 5.

Original: For IDLE, the selected units would do nothing.
Revised: For IDLE, the selected units would do nothing.

Original: For CONTINUE, the units would continue their previous actions.
Revised: For CONTINUE, the units would continue their previous actions.

Original: For GATHER, we should also tell the units the ID of the target resource unit.
Revised: For GATHER, we should also tell the units the ID of the target resource unit.

Original: For ATTACK, we should tell the units the enemy unit ID.
Revised: For ATTACK, we should tell the units the enemy unit ID.

Original: For TRAIN, we should tell the units the army type they should train.
Revised: For TRAIN, we should tell the units the army type they should train.

Original: For BUILD, we should tell the units the building type and the position (x,y) to build.
Revised: For BUILD, we should tell the units the building type and the position (x,y) to build.

Original: For MOVE, we should tell the units the target position.
Revised: For MOVE, we should tell the units the target position.

Original: Reward This environment supports a sparse reward.
Revised: Reward This environment supports a sparse reward.

Original: At the end of a game, the reward is 1 if the agent wins and -1 if the agent loses.
Revised: At the end of a game, the reward is 1 if the agent wins and -1 if the agent loses.

Original: And the agent would receive the reward of 0 all the other time steps.
Revised: And the agent would receive the reward of 0 all the other time steps.

Original: A.3 Built-in Script AI
Revised: A.3 Built-in Script AI

Original: The authors of MiniRTS [33] provide several built-in script AIs.
Revised: The authors of MiniRTS [33] provide several built-in script AIs.

Original: We find that an unconditioned
Revised: We find that an unconditioned

Original: RL agent achieves comparable win rates against both medium level and strong level script AIs, and we choose the medium level AI as the opponent for the convenience of designing the oracle commands.
Revised: RL agent achieves comparable win rates against both medium level and strong level script AIs, and we choose the medium level AI as the opponent for the convenience of designing the oracle commands.

Original: This script first sends all 3 initially available peasants to mine from the closest resource unit.
Revised: This script first sends all 3 initially available peasants to mine from the closest resource unit.

Original: It randomly chooses one army unit type from “spearman”, “swordman”, “cavalry”, “archer”, “dragon” and “catapult” and determines an army size n between 3 and 7.
Revised: It randomly chooses one army unit type from “spearman”, “swordman”, “cavalry”, “archer”, “dragon” and “catapult” and determines an army size n between 3 and 7.

Original: It constructs a building unit corresponding to the selected army (Fig.
Revised: It constructs a building unit corresponding to the selected army (Fig.

Original: 8), then trains n units of the selected army type and sends them to attack.
Revised: 8), then trains n units of the selected army type and sends them to attack.

Original: The script continuously trains the army units and maintains the army size of n .
Revised: The script continuously trains the army units and maintains the army size of n .

Original: A.4 Dataset
Revised: A.4 Dataset

Original: The authors of MiniRTS [33] split the dataset into a training set and a validation set.
Revised: The authors of MiniRTS [33] split the dataset into a training set and a validation set.

Original: The training set includes 4,171 trajectories, 57,293 commands and 634,799 transitions.
Revised: The training set includes 4,171 trajectories, 57,293 commands and 634,799 transitions.

Original: While the validation set contains 433 trajectories, 5,992 commands, and 63,649 transitions.
Revised: While the validation set contains 433 trajectories, 5,992 commands, and 63,649 transitions.

Original: When training the policy, we only use data in the training set.
Revised: When training the policy, we only use data in the training set.

Original: B Implementation Details
Revised: B Implementation Details

Original: In this section, we introduce the implementation details and hyper-parameters.
Revised: In this section, we introduce the implementation details and hyper-parameters.

Original: We train all the policies on a server with 8 RTX-3090 GPUs.
Revised: We train all the policies on a server with 8 RTX-3090 GPUs.

Original: All the policies warm-start with the BC pretrained model.
Revised: All the policies warm-start with the BC pretrained model.

Original: B.1 RED Implementation
Revised: B.1 RED Implementation

Original: Policy Architecture We adopt a similar policy architecture to the provided executor model in
Revised: Policy Architecture We adopt a similar policy architecture to the provided executor model in

Original: MiniRTS [33] but with a modified action space and an additional value head as the critic.
Revised: MiniRTS [33] but with a modified action space and an additional value head as the critic.

Original: The original model outputs an action for every controllable unit, which makes RL training extremely slow.
Revised: The original model outputs an action for every controllable unit, which makes RL training extremely slow.

Original: We introduce an additional 0/1 selection action on each unit, denoting whether this unit should act or not , and only generate the same action for those selected units based on the average of their features, which substantially reduces the action dimension.
Revised: We introduce an additional 0/1 selection action on each unit, denoting whether this unit should act or not , and only generate the same action for those selected units based on the average of their features, which substantially reduces the action dimension.

Original: RL Phase We train the policy through a parallel PPO training process.
Revised: RL Phase We train the policy through a parallel PPO training process.

Original: We use 1024 parallel workers to collect transitions ( s, a, r, s ′ ) from the environments synchronously ( c is NA for RED).
Revised: We use 1024 parallel workers to collect transitions ( s, a, r, s ′ ) from the environments synchronously ( c is NA for RED).

Original: Once 128 workers get 256 data points each, we split the collected data ( 128 × 256 data points) intobatches and run one epoch of PPO training by setting the discount factor γ as 0.999.
Revised: Once 128 workers get 256 data points each, we split the collected data ( 128 × 256 data points) intobatches and run one epoch of PPO training by setting the discount factor γ as 0.999.

Original: We use GAE advantage for each data point.
Revised: We use GAE advantage for each data point.

Original: In each iteration, we repeat 100 training epochs as described above.
Revised: In each iteration, we repeat 100 training epochs as described above.

Original: We collect winning trajectories and store them as D k .
Revised: We collect winning trajectories and store them as D k .

Original: In the early stage of the training phase, the winning rate can be low.
Revised: In the early stage of the training phase, the winning rate can be low.

Original: To obtain sufficient data for D k , we implement a queue to store all past winning trajectories and use bootstrapped sampling if the queue is still not full.
Revised: To obtain sufficient data for D k , we implement a queue to store all past winning trajectories and use bootstrapped sampling if the queue is still not full.

Original: Note that in each iteration, 100 epochs of PPO training produces 100 × 128 × 256 = 3 , 276 , 800 transitions, and there are 634 , 799 transitions in the training set of D .
Revised: Note that in each iteration, 100 epochs of PPO training produces 100 × 128 × 256 = 3 , 276 , 800 transitions, and there are 634 , 799 transitions in the training set of D .

Original: We limit the capacity of D k to the same as the size of D .
Revised: We limit the capacity of D k to the same as the size of D .

Original: We report the results on different ratios of ∥D k ∥ and ∥D∥ in Sec. 5.2.
Revised: We report the results on different ratios of ∥D k ∥ and ∥D∥ in Sec. 5.2.

Original: BC Phase We adopt a similar BC training process to MiniRTS [33].
Revised: BC Phase We adopt a similar BC training process to MiniRTS [33].

Original: Since we introduce an additional 0/1 action to denote whether each controllable unit should act or not, we also need to train this action in the BC phase.
Revised: Since we introduce an additional 0/1 action to denote whether each controllable unit should act or not, we also need to train this action in the BC phase.

Original: In a transition of D , each controllable unit is paired with an action type.
Revised: In a transition of D , each controllable unit is paired with an action type.

Original: We first find the most common action type across all units, then label the units with this action type as 1 and the rest units as 0.
Revised: We first find the most common action type across all units, then label the units with this action type as 1 and the rest units as 0.

Original: Finally, based on these labels, we train the additional 0/1 actions for all units.
Revised: Finally, based on these labels, we train the additional 0/1 actions for all units.

Original: We run one epoch of BC training in each iteration by setting the batch size as 2048 .
Revised: We run one epoch of BC training in each iteration by setting the batch size as 2048 .

Original: Optimizer & Learning Rate We use Adam optimizer and adopt separate optimizers for RL and
Revised: Optimizer & Learning Rate We use Adam optimizer and adopt separate optimizers for RL and

Original: BC training.
Revised: BC training.

Original: We run a total of 40 iterations (i.e. 4,000 PPO epochs.)
Revised: We run a total of 40 iterations (i.e. 4,000 PPO epochs.)

Original: to train a RED policy.
Revised: to train a RED policy.

Original: For BC training, we set β = (0 . 9 , 0 . 999) and fix the learning rate as 2 e − 4 .
Revised: For BC training, we set β = (0 . 9 , 0 . 999) and fix the learning rate as 2 e − 4 .

Original: For RL training, we also set β = (0 . 9 , 0 . 999) but adapt the learning rate throughout all the 4,000 PPO epochs.
Revised: For RL training, we also set β = (0 . 9 , 0 . 999) but adapt the learning rate throughout all the 4,000 PPO epochs.

Original: In the firstepochs, we start with a learning rate of 0 and linearly increase it to the desired value 5 e − 5 .
Revised: In the firstepochs, we start with a learning rate of 0 and linearly increase it to the desired value 5 e − 5 .

Original: In the rest 3500 epochs, we decrease the learning rate from 5 e − 5 to 0 linearly.
Revised: In the rest 3500 epochs, we decrease the learning rate from 5 e − 5 to 0 linearly.

Original: B.2 Baseline Implementation
Revised: B.2 Baseline Implementation

Original: RL Implementation For pure RL baseline, we adopt the same architecture and hyper-parameters as RED.
Revised: RL Implementation For pure RL baseline, we adopt the same architecture and hyper-parameters as RED.

Original: We run 4,000 PPO epochs.
Revised: We run 4,000 PPO epochs.

Original: The batch size and number of transitions in each epoch are the same as RED.
Revised: The batch size and number of transitions in each epoch are the same as RED.

Original: We also adopt the same optimizer and learning rate.
Revised: We also adopt the same optimizer and learning rate.

Original: The only difference is that there is no BC phase during the pure RL training.
Revised: The only difference is that there is no BC phase during the pure RL training.

Original: RL training also starts with the BC pretrained policy.
Revised: RL training also starts with the BC pretrained policy.

Original: Joint Implementation For joint RL baseline, we modify the loss in the pure RL training by adding the NLL loss.
Revised: Joint Implementation For joint RL baseline, we modify the loss in the pure RL training by adding the NLL loss.

Original: The batch size to compute the NLL loss is the same as RL training.
Revised: The batch size to compute the NLL loss is the same as RL training.

Original: The architecture and all the hyper-parameters are the same as RED and pure RL training.
Revised: The architecture and all the hyper-parameters are the same as RED and pure RL training.

Original: We tune the weight β of the
Revised: We tune the weight β of the

Original: NLL loss via a grid-search process, which is described in Sec.
Revised: NLL loss via a grid-search process, which is described in Sec.

Original: D.1.
Revised: D.1.

Original: IRL Implementation We adopt AIRL [24] for IRL training.
Revised: IRL Implementation We adopt AIRL [24] for IRL training.

Original: The policy network has the same architecture and hyper-parameters as RED.
Revised: The policy network has the same architecture and hyper-parameters as RED.

Original: We train a discriminator network D ϕ ( s, c, a ) in the form of
Revised: We train a discriminator network D ϕ ( s, c, a ) in the form of

Original: [Equation]
Revised: [Equation]

Original: where π θ is the policy network.
Revised: where π θ is the policy network.

Original: The hyper-parameters are the same as the policy network.
Revised: The hyper-parameters are the same as the policy network.

Original: For the first 11 iterations, we run 25 discriminator epochs before each policy epoch.
Revised: For the first 11 iterations, we run 25 discriminator epochs before each policy epoch.

Original: For the rest of the iterations, we run a single discriminator epoch before each policy epoch.
Revised: For the rest of the iterations, we run a single discriminator epoch before each policy epoch.

Original: The architecture of f ϕ ( s, c, a ) is similar to the policy network.
Revised: The architecture of f ϕ ( s, c, a ) is similar to the policy network.

Original: We extract a fixed-length global feature vector, a fixed-length command feature vector, and fixed-length feature vectors for each unit and each position on the map in the same way as in MiniRTS [33].
Revised: We extract a fixed-length global feature vector, a fixed-length command feature vector, and fixed-length feature vectors for each unit and each position on the map in the same way as in MiniRTS [33].

Original: Recall that an action contains an action type and possibly action outputs.
Revised: Recall that an action contains an action type and possibly action outputs.

Original: We encode action type using an embedding layer.
Revised: We encode action type using an embedding layer.

Original: Action types IDLE and CONTINUE do not have any action output.
Revised: Action types IDLE and CONTINUE do not have any action output.

Original: For action types GATHER, ATTACK, and TRAIN, we encode action outputs by their corresponding unit embedding.
Revised: For action types GATHER, ATTACK, and TRAIN, we encode action outputs by their corresponding unit embedding.

Original: For action type MOVE, we encode action outputs by the embedding of the moving location.
Revised: For action type MOVE, we encode action outputs by the embedding of the moving location.

Original: For action type BUILD, we encode the building type using an embedding layer and then add it to the embedding of the building location to obtain the embedding of action output.
Revised: For action type BUILD, we encode the building type using an embedding layer and then add it to the embedding of the building location to obtain the embedding of action output.

Original: The embedding of action is then obtained by concatenating the embedding of the action type and action outputs.
Revised: The embedding of action is then obtained by concatenating the embedding of the action type and action outputs.

Original: In addition, we encode unit selection features by averaging the extracted features of the units that are selected.
Revised: In addition, we encode unit selection features by averaging the extracted features of the units that are selected.

Original: Finally, we concatenate the global feature, the command feature, the action feature, the unit selection feature, and the global continue flag together and feed it to a linear layer to get the value of f ϕ ( s, c, a ) .
Revised: Finally, we concatenate the global feature, the command feature, the action feature, the unit selection feature, and the global continue flag together and feed it to a linear layer to get the value of f ϕ ( s, c, a ) .

Original: The discriminator objective is given by
Revised: The discriminator objective is given by

Original: [Equation]
Revised: [Equation]

Original: We combine environment rewards and the intrinsic rewards, namely
Revised: We combine environment rewards and the intrinsic rewards, namely

Original: [Equation]
Revised: [Equation]

Original: wherer
Revised: where

Original: [Equation]
Revised: [Equation]

Original:  and µ, σ are the mean and variance of r disc ( s, c, a ) within a sample batch, respectively.
Revised: a )) , (9) and µ, σ are the mean and variance of r disc ( s, c, a ) within a sample batch, respectively.

Original: The tuning process of weight β i of the intrinsic reward is described in Sec.
Revised: The tuning process of weight β i of the intrinsic reward is described in Sec.

Original: D.3.
Revised: D.3.

Original: C Experimental Details C.1 Oracle Commander
Revised: C Experimental Details C.1 Oracle Commander

Original: Since the built-in script AI only builds army units of a single type in a game.
Revised: Since the built-in script AI only builds army units of a single type in a game.

Original: We script an oracle command strategy according to the ground truth of enemy units and the attack graph (Fig. 7).
Revised: We script an oracle command strategy according to the ground truth of enemy units and the attack graph (Fig. 7).

Original: This commander would ask the agent to build the appropriate army units step by step in the early stage of the game.
Revised: This commander would ask the agent to build the appropriate army units step by step in the early stage of the game.

Original: Then, it sends NA and asks the policy to play on its own.
Revised: Then, it sends NA and asks the policy to play on its own.

Original: A strong GRL policy would follow these commands to build the correct army units, play itself in the game’s reset, and achieve a higher win rate.
Revised: A strong GRL policy would follow these commands to build the correct army units, play itself in the game’s reset, and achieve a higher win rate.

Original: • Enemy “spearman”: (i) “mine with all idle peasant” in the first 2 time steps.
Revised: • Enemy “spearman”: (i) “mine with all idle peasant” in the first 2 time steps.

Original: (ii) “buildpeasant” until there are 6 peasants.
Revised: (ii) “buildpeasant” until there are 6 peasants.

Original: (iii) “build a blacksimth” until there is a blacksmith.
Revised: (iii) “build a blacksimth” until there is a blacksmith.

Original: (iv) “build another swordman” when the number of swordman is less than 5 otherwise NA .
Revised: (iv) “build another swordman” when the number of swordman is less than 5 otherwise NA .

Original: • Enemy “swordman”: (i) “mine with all idle peasant” in the first 2 time steps.
Revised: • Enemy “swordman”: (i) “mine with all idle peasant” in the first 2 time steps.

Original: (ii) “buildpeasant” until there are 6 peasants.
Revised: (ii) “buildpeasant” until there are 6 peasants.

Original: (iii) “build a stable” until there is a stable.
Revised: (iii) “build a stable” until there is a stable.

Original: (iv) “build another cavalry” when the number of cavalry is less than 5 otherwise NA .
Revised: (iv) “build another cavalry” when the number of cavalry is less than 5 otherwise NA .

Original: • Enemy “cavalry”: (i) “mine with all idle peasant” in the first 2 time 
Revised: • Enemy “cavalry”: (i) “mine with all idle peasant” in the first 2 time steps.

Revised: (ii) “buildpeasant” until there are 6 peasants.

Revised: (iii) “build a barrack” until there is a barrack.

Revised: (iv) “build a spearman” when the number of swordman is less than 5 otherwise NA .

Original: • Enemy “dragon”: (i) “mine with all idle peasant” in the first 2 time steps.
Revised: • Enemy “dragon”: (i) “mine with all idle peasant” in the first 2 time steps.

Original: (ii) “buildpeasant” until there are 6 peasants.
Revised: (ii) “buildpeasant” until there are 6 peasants.

Original: (iii) “build a workshop” until there is a workshop.
Revised: (iii) “build a workshop” until there is a workshop.

Original: (iv) “make archer” when the number of archer is less than 5 otherwise NA .
Revised: (iv) “make archer” when the number of archer is less than 5 otherwise NA .

Original: • Enemy “archer”:
Revised: • Enemy “archer”:

Original: (i) “mine with all idle peasant” in the first 2 time steps.
Revised: (i) “mine with all idle peasant” in the first 2 time steps.

Original: (ii) “buildpeasant” until there are 6 peasants.
Revised: (ii) “buildpeasant” until there are 6 peasants.

Original: (iii) Randomly select a building command from “build a blacksimth”, “build a stable” and “build a barrack” until the corresponding building unit is constructed.
Revised: (iii) Randomly select a building command from “build a blacksimth”, “build a stable” and “build a barrack” until the corresponding building unit is constructed.

Original: (iv) Select from “build another swordman”, “build another cavalry” and “build a spearman” according to the constructed building unit when the number of the corresponding army unit is less than 5 otherwise NA .
Revised: (iv) Select from “build another swordman”, “build another cavalry” and “build a spearman” according to the constructed building unit when the number of the corresponding army unit is less than 5 otherwise NA .

Original: • Enemy “catapult”:
Revised: • Enemy “catapult”:

Original: We empirically find that the best strategy to deal with the “catapult” is to send NA all the time (according to the policies of Joint, RED and Switch).
Revised: We empirically find that the best strategy to deal with the “catapult” is to send NA all the time (according to the policies of Joint, RED and Switch).

Original: We hypothesize it is because that all the other army types are effective against “catapult”, and building commands as before are not helpful to play against catapult.
Revised: We hypothesize it is because that all the other army types are effective against “catapult”, and building commands as before are not helpful to play against catapult.

Original: C.2 “Human Proxy” Command Generator
Revised: C.2 “Human Proxy” Command Generator

Original: Training a command generator is not our focus.
Revised: Training a command generator is not our focus.

Original: We use the best commander (instructor) model trained and released by MiniRTS [33].
Revised: We use the best commander (instructor) model trained and released by MiniRTS [33].

Original: Please refer to the model at https://github.com/ facebookresearch/minirts .
Revised: Please refer to the model at https://github.com/ facebookresearch/minirts .

Original: C.3 Learned Strategy
Revised: C.3 Learned Strategy

Original: How to Categorize Commands We identify the five most common categories of commands as follows:
Revised: How to Categorize Commands We identify the five most common categories of commands as follows:

Original: • Attack : Commands containing “attack” or “kill” belong to this category.
Revised: • Attack : Commands containing “attack” or “kill” belong to this category.

Original: This type of command typically instructs the agent to attack enemy units, but does not necessarily specify which unit to attack.
Revised: This type of command typically instructs the agent to attack enemy units, but does not necessarily specify which unit to attack.

Original: • Defend : Commands containing “defend” belong to this category.
Revised: • Defend : Commands containing “defend” belong to this category.

Original: This type of command typically asks the agent to fight with invading enemy units but usually does not give more detail.
Revised: This type of command typically asks the agent to fight with invading enemy units but usually does not give more detail.

Original: • Build : Commands containing “build”, “create” or “make” belong to this category.
Revised: • Build : Commands containing “build”, “create” or “make” belong to this category.

Original: This type of command typically instructs the model to build a specific type of building unit or army unit.
Revised: This type of command typically instructs the model to build a specific type of building unit or army unit.

Original: • Mine : Commands containing “mine”, “mining” or “mineral” belong to this category.
Revised: • Mine : Commands containing “mine”, “mining” or “mineral” belong to this category.

Original: This type of command typically instructs the model to mine from the resource units.
Revised: This type of command typically instructs the model to mine from the resource units.

Original: • Stop : Commands containing “stop” belong to this category.
Revised: • Stop : Commands containing “stop” belong to this category.

Original: Such commands typically claim that the current action should be terminated.
Revised: Such commands typically claim that the current action should be terminated.

Original: Action Prediction Accuracy
Revised: Action Prediction Accuracy

Original: The actions in the transitions of D are extremely complex.
Revised: The actions in the transitions of D are extremely complex.

Original: In each transition, there are many controllable units, and each unit has its action type and the corresponding action output (see Tab. 5).
Revised: In each transition, there are many controllable units, and each unit has its action type and the corresponding action output (see Tab. 5).

Original: So it is not so direct to compute the action prediction accuracy.
Revised: So it is not so direct to compute the action prediction accuracy.

Original: When predicting the action prediction, we concentrate on the units whose action types are not CONTINUE, since the units with CONTINUE would continue their previous actions, which should be considered in the previous transitions.
Revised: When predicting the action prediction, we concentrate on the units whose action types are not CONTINUE, since the units with CONTINUE would continue their previous actions, which should be considered in the previous transitions.

Original: In detail, for a single transition, we skip the actions of determining whether the units should act or not, and treat the units whose actions are not CONTINUE as the selected units.
Revised: In detail, for a single transition, we skip the actions of determining whether the units should act or not, and treat the units whose actions are not CONTINUE as the selected units.

Original: We predict the action type and action output based on the selected units.
Revised: We predict the action type and action output based on the selected units.

Original: We think the prediction is correct if the predicted action type and action output exactly match with any selected unit.
Revised: We think the prediction is correct if the predicted action type and action output exactly match with any selected unit.

Original: The accuracy is then computed as the correct prediction ratio over all the transitions.
Revised: The accuracy is then computed as the correct prediction ratio over all the transitions.

Original: In addition to action prediction accuracy , we also evaluate NLL difference of RED policy and BC policy in Sec.
Revised: In addition to action prediction accuracy , we also evaluate NLL difference of RED policy and BC policy in Sec.

Original: D.4.
Revised: D.4.

Original: D Additional Results D.1 Grid Search on β for Baseline “Joint”
Revised: D Additional Results D.1 Grid Search on β for Baseline “Joint”

Original: We conduct the experiments on joint training with different β , and the results are listed in Tab. 6.
Revised: We conduct the experiments on joint training with different β , and the results are listed in Tab. 6.

Original: We can observe that the policy with a larger β performs worse when no commands are provided ( NA ), which indicates that the joint objective may be harmful to RL training.
Revised: We can observe that the policy with a larger β performs worse when no commands are provided ( NA ), which indicates that the joint objective may be harmful to RL training.

Original: In addition, the win rates tested with random commands suggest that a smaller β may result in the policy being less obedient.
Revised: In addition, the win rates tested with random commands suggest that a smaller β may result in the policy being less obedient.

Original: [Table]

Original: D.2 Adaptive β
Revised: D.2 Adaptive β

Original: Although it is empirically a common practice to use a fixed β in the Lagrangian method [Table] we additionally provide experiments in which the coefficient β is updated during the training process.
Revised: Although it is empirically a common practice to use a fixed β in the Lagrangian method, we additionally provide experiments in which the coefficient β is updated during the training process.

Original: It is worth noting that a similar technique can also be applied to our proposed method RED.
Revised: It is worth noting that a similar technique can also be applied to our proposed method RED.

Original: In particular, during the BC phase
Revised: In [Table]

Original: [Table]
Revised: [Table]

Original:  we add a coefficient β on the behavior clone loss on demonstrations
Revised: particular, during the BC phase, we add a coefficient β on the behavior clone loss on demonstrations

Original: D .
Revised: D .

Original: This coefficient β is updated during the training process.
Revised: This coefficient β is updated during the training process.

Original: The results are listed in Tab. 7.
Revised: The results are listed in Tab. 7.

Original: [Table]

Original: One can see that using an adaptive β improves the performance of both Joint and RED.
Revised: One can see that using an adaptive β improves the performance of both Joint and RED.

Original: While having similar performance under the Oracle commander, RED with adaptive β significantly outperforms
Revised: While having similar performance under the Oracle commander, RED with adaptive β significantly outperforms

Original: Joint under the NA commander.
Revised: Joint under the NA commander.

Original: This suggests that iterating between RL and BC ensures better RL performance than mixing the two objectives.
Revised: This suggests that iterating between RL and BC ensures better RL performance than mixing the two objectives.

Original: D.3 Grid Search on β i for Baseline “IRL”
Revised: D.3 Grid Search on β i for Baseline “IRL”

Original: We adopt AIRL [24] as the algorithm for IRL experiments.
Revised: We adopt AIRL [24] as the algorithm for IRL experiments.

Original: We learn a reward function from demonstrations and combine the game reward and the learned language reward to train a conditioned policy (See Eq.
Revised: We learn a reward function from demonstrations and combine the game reward and the learned language reward to train a conditioned policy (See Eq.

Original: We use the human proxy command generator to provide commands when training, since the learned language reward is conditional on game states and human commands.
Revised: We use the human proxy command generator to provide commands when training, since the learned language reward is conditional on game states and human commands.

Original: We evaluate the performance with different β i .
Revised: We evaluate the performance with different β i .

Original: and the results are listed in Tab.
Revised: and the results are listed in Tab.

Original: [Table]
Revised: [Table]

Original: We can observe that the win rates tested with random commands are comparable with tested with NA , which indicates that IRL is not able to make the policy obedient, although we increase the weight of the learned reward.
Revised: We can observe that the win rates tested with random commands are comparable with tested with NA , which indicates that IRL is not able to make the policy obedient, although we increase the weight of the learned reward.

Original: In addition, the learned reward may be harmful to the RL training, larger β i leads to lower win rates tested on the oracle and NA commands.
Revised: In addition, the learned reward may be harmful to the RL training, larger β i leads to lower win rates tested on the oracle and NA commands.

Revised: D.4 Validation NLL for Various Command Types

Original: In addition to action prediction accuracy, we also evaluate NLL difference of RED policy and BC policy in Fig.
Revised: In addition to action prediction accuracy, we also evaluate NLL difference of RED policy and BC policy in Fig.

Original: 9, on the validation set.
Revised: 9, on the validation set.

Original: The dashed line denotes the overall validation difference.
Revised: The dashed line denotes the overall validation difference.

Original: The conclusion is the same as action prediction accuracy.
Revised: The conclusion is the same as action prediction accuracy.

Original: RED disagrees with the commands with a highly complicated space of possible behaviors (e.g., the command belonging “attack”, “defend”) the most, and follows humans’ action on the commands with low uncertainty (e.g., the commands belonging to “stop”).
Revised: RED disagrees with the commands with a highly complicated space of possible behaviors (e.g., the command belonging “attack”, “defend”) the most, and follows humans’ action on the commands with low uncertainty (e.g., the commands belonging to “stop”).

Original: D.5 Performance on Commands with OOV Words
Revised: D.5 Performance on Commands with OOV Words

Original: We also evaluate the policy performances on commands with Out-of-Vocabulary (OOV) words.
Revised: We also evaluate the policy performances on commands with Out-of-Vocabulary (OOV) words.

Original: It is worth noting that we use a special token [unk] to handle all the OOV words.
Revised: It is worth noting that we use a special token [unk] to handle all the OOV words.

Original: We insert the special token in different positions of the oracle commands to evaluate the influence of OOV words.
Revised: We insert the special token in different positions of the oracle commands to evaluate the influence of OOV words.

Original: We describe the ways to insert the OOV words as follows.
Revised: We describe the ways to insert the OOV words as follows.

Original: And the results are listed in Tab. 9.
Revised: And the results are listed in Tab. 9.

Original: • All-OOV: We replace all words with [unk] tokens in each command.
Revised: • All-OOV: We replace all words with [unk] tokens in each command.

Original: • Pre-OOV: We insert a sequence of [unk] tokens in front of each command, which makes the command twice the length of the original.
Revised: • Pre-OOV: We insert a sequence of [unk] tokens in front of each command, which makes the command twice the length of the original.

Original: • Post-OOV: We insert a sequence of [unk] tokens at the end of each command, which makes the command twice the length of the original.
Revised: • Post-OOV: We insert a sequence of [unk] tokens at the end of each command, which makes the command twice the length of the original.

Original: • Mid-OOV: We insert
Revised: • Mid-OOV: We insert

Original: [unk] tokens between any two words of each command.
Revised: [unk] tokens between any two words of each command.

Original: [Table]
Revised: [Table]

Original: Switch

Original: Joint

Original: RED

Original: [Table]

Original: [Table] [Table] [Table] The results show that “All-OOV” commands are terrible for our strong following policies (i.e., RED,
Revised: The results show that “All-OOV” commands are terrible for our strong following policies (i.e., RED,

Original: Joint, and Switch).
Revised: Joint, and Switch).

Original: RL and IRL policies perform near the same under the NA command and All-OOV commands, which suggests they are command-ignorant.
Revised: RL and IRL policies perform near the same under the NA command and All-OOV commands, which suggests they are command-ignorant.

Original: Performances on Pre-OOV , Post-OOV and
Revised: Performances on Pre-OOV , Post-OOV and

Original: Mid-OOV indicate that the policies can also perform well when some OOV words are inserted.
Revised: Mid-OOV indicate that the policies can also perform well when some OOV words are inserted.

Original: In
Revised: In

Original: Pre-OOV and Post-OOV commands, the original sequences are kept, which outperform Mid-OOV .
Revised: Pre-OOV and Post-OOV commands, the original sequences are kept, which outperform Mid-OOV .

Original: Particularly in Pre-OOV commands, the performances are near the same with Oracle .
Revised: Particularly in Pre-OOV commands, the performances are near the same with Oracle .

Original: D.6 Clustering Analysis of Command Representations.
Revised: D.6 Clustering Analysis of Command Representations.

Original: We generate commands using some templates and visualize their embeddings with t-SNE.
Revised: We generate commands using some templates and visualize their embeddings with t-SNE.

Original: The templates are listed as follows:
Revised: The templates are listed as follows:

Original: • build building units: {build | create | make} {blacksmith | stable | barrack | workshop | guard tower}, {build | create | make} a {blacksmith | stable | barrack | workshop | guard tower}, {build | create | make} another {blacksmith | stable | barrack | workshop | guard tower}.
Revised: • build building units: {build | create | make} {blacksmith | stable | barrack | workshop | guard tower}, {build | create | make} a {blacksmith | stable | barrack | workshop | guard tower}, {build | create | make} another {blacksmith | stable | barrack | workshop | guard tower}.

Original: • build army units: {build | create | make} {swordman | cavalry | spearman | archer | dragon | catapult}, {build | create | make} a {swordman | cavalry | spearman | archer | dragon | catapult}, {build | create | make} another {swordman | cavalry | spearman | archer | dragon | catapult}, {build | create | make} 3 {swordman | cavalry | spearman | archer | dragon | catapult}, {build | create | make} 5 {swordman | cavalry | spearman | archer | dragon | catapult}, {build | create | make} 7 {swordman | cavalry | spearman | archer | dragon | catapult}.
Revised: • build army units: {build | create | make} {swordman | cavalry | spearman | archer | dragon | catapult}, {build | create | make} a {swordman | cavalry | spearman | archer | dragon | catapult}, {build | create | make} another {swordman | cavalry | spearman | archer | dragon | catapult}, {build | create | make} 3 {swordman | cavalry | spearman | archer | dragon | catapult}, {build | create | make} 5 {swordman | cavalry | spearman | archer | dragon | catapult}, {build | create | make} 7 {swordman | cavalry | spearman | archer | dragon | catapult}.

Original: • attack: attack, kill.
Revised: • attack: attack, kill.

Original: • scout: scout, scout the map, go around the map.
Revised: • scout: scout, scout the map, go around the map.

Original: • mine: mine, {gather | collect | mine} {minerals | resources}.
Revised: • mine: mine, {gather | collect | mine} {minerals | resources}.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 10, similar commands are encoded into similar embeddings.
Revised: 10, similar commands are encoded into similar embeddings.

Original: In addition, the commands that build building units are close to each other and far from the commands that build army units, which suggests that the LSTM module is able to learn the semantic meanings of different commands.
Revised: In addition, the commands that build building units are close to each other and far from the commands that build army units, which suggests that the LSTM module is able to learn the semantic meanings of different commands.

Original: E Human Evaluation Details E.1 Ethics Statement
Revised: E Human Evaluation Details E.1 Ethics Statement

Original: The experiments are permitted under our department committee.
Revised: The experiments are permitted under our department committee.

Original: There is not any personally identifiable information or sensitive personally identifiable information involved in the experiment.
Revised: There is not any personally identifiable information or sensitive personally identifiable information involved in the experiment.

Original: When conducting the human evaluation experiments, we informed the participants in advance of the purpose of the experiment and the time the experiment might take.
Revised: When conducting the human evaluation experiments, we informed the participants in advance of the purpose of the experiment and the time the experiment might take.

Original: All the participants are fully informed and participate voluntarily with signed confirmation.
Revised: All the participants are fully informed and participate voluntarily with signed confirmation.

Original: We have controlled each experiment to be completed within 1 hour.
Revised: We have controlled each experiment to be completed within 1 hour.

Original: We invite 28 undergrad students and 2 Ph.D. students under department permission to complete a two-stage study.
Revised: We invite 28 undergrad students and 2 Ph.D. students under department permission to complete a two-stage study.

Original: We release this human evaluation as an optional project in an undergrad course.
Revised: We release this human evaluation as an optional project in an undergrad course.

Original: The students who are interested in this project can participate.
Revised: The students who are interested in this project can participate.

Original: The Ph.D. students are well paid as research assistants.
Revised: The Ph.D. students are well paid as research assistants.

Original: E.2 Evaluation Process
Revised: E.2 Evaluation Process

Original: We first introduce the rules of the game as described in Sec.
Revised: We first introduce the rules of the game as described in Sec.

Original: A.1 and Fig.
Revised: A.1 and Fig.

Original: 7, and ask them to play the game by providing the commands.
Revised: 7, and ask them to play the game by providing the commands.

Original: To avoid the participants not knowing what to do at the beginning, we give some basic commands as shown in Tab. 10.
Revised: To avoid the participants not knowing what to do at the beginning, we give some basic commands as shown in Tab. 10.

Original: In addition to these basic commands, the participants can also input arbitrary commands.
Revised: In addition to these basic commands, the participants can also input arbitrary commands.

Original: For example, some participants find that “go around the map” and “scout the map” can make the units explore the map and find the enemies.
Revised: For example, some participants find that “go around the map” and “scout the map” can make the units explore the map and find the enemies.

Revised: mine all peasant mine attack defend

Original: [Table]
Revised: [Table]

Original: We pick 4 policies.
Revised: We pick 4 policies.

Original: i.e., RED, Switch, Joint and RL.
Revised: i.e., RED, Switch, Joint and RL.

Original: In the first stage of the human evaluation, we shuffle the order of 4 policies and ask each volunteer to play with these policies using arbitrary commands.
Revised: In the first stage of the human evaluation, we shuffle the order of 4 policies and ask each volunteer to play with these policies using arbitrary commands.

Original: Once the volunteer feels familiar with the game policies, he/she is asked to rank the policies according to how the policies follow commands.
Revised: Once the volunteer feels familiar with the game policies, he/she is asked to rank the policies according to how the policies follow commands.

Original: In the second stage, each volunteer is asked to play 2 games per policy and try their best to win.
Revised: In the second stage, each volunteer is asked to play 2 games per policy and try their best to win.

Original: In this stage, the volunteers can turn off the “fog-of-war” and observe the enemy units (“fog-of-war” is always turned on for the executor policies).
Revised: In this stage, the volunteers can turn off the “fog-of-war” and observe the enemy units (“fog-of-war” is always turned on for the executor policies).

Original: Note that we shuffle the policy order, and the students often spend a long time on the first policy to get familiar with MiniRTS, which can make the students biased towards the first policy they encountered.
Revised: Note that we shuffle the policy order, and the students often spend a long time on the first policy to get familiar with MiniRTS, which can make the students biased towards the first policy they encountered.

Original: We think that a better evaluation process is to arrange an additional practice stage, allowing participants to play with a BC policy and get familiar with the game.
Revised: We think that a better evaluation process is to arrange an additional practice stage, allowing participants to play with a BC policy and get familiar with the game.

Original: E.3 Gameplay Interface
Revised: E.3 Gameplay Interface

Original: We present a screenshot of the gameplay interface in Fig.
Revised: We present a screenshot of the gameplay interface in Fig.

Original: The user can select a command from a set of recommendations using the "Select Command" button or input an arbitrary (English) command in the text box below.
Revised: The user can select a command from a set of recommendations using the "Select Command" button or input an arbitrary (English) command in the text box below.

Original: If the user types ENTER in the text box while leaving it empty, the previous command will be issued.
Revised: If the user types ENTER in the text box while leaving it empty, the previous command will be issued.

Original: This functionality is designed because we notice that it is a natural choice to keep sending the same command for several continuous time steps.
Revised: This functionality is designed because we notice that it is a natural choice to keep sending the same command for several continuous time steps.

Original: To send an empty command (i.e., the NA command), the user can press the “Send Empty Command” button.
Revised: To send an empty command (i.e., the NA command), the user can press the “Send Empty Command” button.

Original: Since some users may not be good at playing real-time strategy games, one can optionally toggle the fog of war using the “Fog of War: ON(OFF)”.
Revised: Since some users may not be good at playing real-time strategy games, one can optionally toggle the fog of war using the “Fog of War: ON(OFF)”.

Original: This lowers the difficulty of the game.
Revised: This lowers the difficulty of the game.

Original: E.4 Emergent Behaviors
Revised: E.4 Emergent Behaviors

Original: We find some interesting emergent behaviors during human evaluation.
Revised: We find some interesting emergent behaviors during human evaluation.

Original: Please refer to our videos at https://sites.google.com/view/grounded-rl .
Revised: Please refer to our videos at https://sites.google.com/view/grounded-rl .

Original: Some human participants prefer to build dragons since dragons sound powerful and cool.
Revised: Some human participants prefer to build dragons since dragons sound powerful and cool.

Original: But building dragons makes it more challenging to win because dragons are expensive.
Revised: But building dragons makes it more challenging to win because dragons are expensive.

Original: If the player builds dragons at the beginning of a game, he/she will get few dragons ready when the enemies start to attack him/her, leading to a game loss.
Revised: If the player builds dragons at the beginning of a game, he/she will get few dragons ready when the enemies start to attack him/her, leading to a game loss.

Original: So the RL trained policies prefer to build “spearman”, “swordman” or “cavalry”, since win rates of these units are higher.
Revised: So the RL trained policies prefer to build “spearman”, “swordman” or “cavalry”, since win rates of these units are higher.

Original: Fortunately, humans are intelligent and creative.
Revised: Fortunately, humans are intelligent and creative.

Original: Some participants figure out the behavior pattern of the built-in script AI.
Revised: Some participants figure out the behavior pattern of the built-in script AI.

Original: They find that the script AI will quickly build several army units and send them to attack.
Revised: They find that the script AI will quickly build several army units and send them to attack.

Original: It is vital to resist the first two waves of attacks.
Revised: It is vital to resist the first two waves of attacks.

Original: A participant instructs the policy to construct several towers in the early stage of the game to resist the attack.
Revised: A participant instructs the policy to construct several towers in the early stage of the game to resist the attack.

Original: After ensuring the safety, the student asks the policy to build dragons and then sends the dragons to find the enemies to attack, which finally results in a win.
Revised: After ensuring the safety, the student asks the policy to build dragons and then sends the dragons to find the enemies to attack, which finally results in a win.

Original: It is difficult for pure RL to learn such a complicated strategy since building other army units (e.g., “spearman”, “swordman” or “cavalry”) is more direct for a fast win.
Revised: It is difficult for pure RL to learn such a complicated strategy since building other army units (e.g., “spearman”, “swordman” or “cavalry”) is more direct for a fast win.

Original: A well-trained RED policy can follow human commands to execute this complicated strategy.
Revised: A well-trained RED policy can follow human commands to execute this complicated strategy.

Original: In addition, although the human commands are vague, RED policy can take suitable actions to follow the commands.
Revised: In addition, although the human commands are vague, RED policy can take suitable actions to follow the commands.

Original: For example, it builds towers close to each other to strengthen the defense.
Revised: For example, it builds towers close to each other to strengthen the defense.

Original: After finding the enemies, it automatically sends the dragons to attack them.
Revised: After finding the enemies, it automatically sends the dragons to attack them.

Original: Here we present the example of how the human commander instructs the RED policy to win a game using dragon by several commands in Fig.
Revised: Here we present the example of how the human commander instructs the RED policy to win a game using dragon by several commands in Fig.

Original: There are 3 phases.
Revised: There are 3 phases.

Original: In phase 1, the human player asks the policy to build towers by sending “build a guard tower” .
Revised: In phase 1, the human player asks the policy to build towers by sending “build a guard tower” .

Original: In phase 2, the human player instructs the policy to build dragons using commands “build a workshop” and “build dragon” .
Revised: In phase 2, the human player instructs the policy to build dragons using commands “build a workshop” and “build dragon” .

Original: In phase 3, the human player sends “scout the map” , and the policy makes the dragons fly around the map and find the enemies.
Revised: In phase 3, the human player sends “scout the map” , and the policy makes the dragons fly around the map and find the enemies.

Original: We describe these 3 phases in detail in the following.
Revised: We describe these 3 phases in detail in the following.

Original: Phase 1: Build Towers As shown in Fig. 13, in the early stage of the game, to ensure safety, the human player gives the command “build a guard tower” at every time step.
Revised: Phase 1: Build Towers As shown in Fig. 13, in the early stage of the game, to ensure safety, the human player gives the command “build a guard tower” at every time step.

Original: So the RED policy lets one peasant build the tower, and the rest peasants keep mining resources from the resource units.
Revised: So the RED policy lets one peasant build the tower, and the rest peasants keep mining resources from the resource units.

Original: Since the command is provided at every time step, the peasant keeps constructing towers one after another.
Revised: Since the command is provided at every time step, the peasant keeps constructing towers one after another.

Original: Phase 2: Build Dragons As shown in Fig.
Revised: Phase 2: Build Dragons As shown in Fig.

Original: 14, after building enough towers, the human player changes the command to “build a workshop” , where the workshop can produce dragons.
Revised: 14, after building enough towers, the human player changes the command to “build a workshop” , where the workshop can produce dragons.

Original: Then one peasant starts to build the workshop immediately.
Revised: Then one peasant starts to build the workshop immediately.

Original: Once the workshop is built, the human player changes the command and keeps asking the policy to “build dragon” .
Revised: Once the workshop is built, the human player changes the command and keeps asking the policy to “build dragon” .

Original: Then, the workshop keeps building the dragons, and all the peasants return to mine resources.
Revised: Then, the workshop keeps building the dragons, and all the peasants return to mine resources.

Original: In this phase, some enemy units started to attack but are all defended by the towers.
Revised: In this phase, some enemy units started to attack but are all defended by the towers.

Original: Phase 3: Find the Enemies As shown in Fig.
Revised: Phase 3: Find the Enemies As shown in Fig.

Original: 15, after building 4 dragons, the human player changes the command to “scout the map” , and the dragons fly around the map to explore.
Revised: 15, after building 4 dragons, the human player changes the command to “scout the map” , and the dragons fly around the map to explore.

Original: Once the dragons find the enemies, RED policy sends them to attack the enemies automatically.
Revised: Once the dragons find the enemies, RED policy sends them to attack the enemies automatically.