Original: Grounded Reinforcement Learning: Learning to Win the Game under Human CommandsAuthor s
Revised: Grounded Reinforcement Learning: Learning to Win the Game under Human Commands

Original: Abstract
Revised: Abstract

Original: We consider the problem of building a reinforcement learning (RL) agent thatcan both accomplish non-trivial tasks, like winning a real-time strategy game, andstrictly follow high-level language commands from humans, like “attack”, even ifa command is sub-optimal.
Revised: We consider the problem of building a reinforcement learning (RL) agent thatcan both accomplish non-trivial tasks, like winning a real-time strategy game, andstrictly follow high-level language commands from humans, like “attack”, even ifa command is sub-optimal.

Original: We call this novel yet important problem, Grounded
Revised: We call this novel yet important problem, Grounded

Original: Reinforcement Learning (GRL).
Revised: Reinforcement Learning (GRL).

Original: Compared with other language grounding tasks,
Revised: Compared with other language grounding tasks,

Original: GRL is particularly non-trivial and cannot be simply solved by pure RL or behaviorcloning (BC).
Revised: GRL is particularly non-trivial and cannot be simply solved by pure RL or behaviorcloning (BC).

Original: From the RL perspective, it is extremely challenging to derive aprecise reward function for human preferences since the commands are abstractand the valid behaviors are highly complicated and multi-modal.
Revised: From the RL perspective, it is extremely challenging to derive aprecise reward function for human preferences since the commands are abstractand the valid behaviors are highly complicated and multi-modal.

Original: From the BCperspective, it is impossible to obtain perfect demonstrations since human strategiesin complex games are typically sub-optimal.
Revised: From the BCperspective, it is impossible to obtain perfect demonstrations since human strategiesin complex games are typically sub-optimal.

Original: We tackle GRL via a simple, tractable,and practical constrained RL objective and develop an iterative RL algorithm,
Revised: We tackle GRL via a simple, tractable,and practical constrained RL objective and develop an iterative RL algorithm,

Original: REinforced demonstration Distillation (RED), to obtain a strong GRL policy.
Revised: REinforced demonstration Distillation (RED), to obtain a strong GRL policy.

Original: Weevaluate the policies derived by RED, BC and pure RL methods on a simplifiedreal-time strategy game, MiniRTS.
Revised: Weevaluate the policies derived by RED, BC and pure RL methods on a simplifiedreal-time strategy game, MiniRTS.

Original: Experiment results and human studies showthat the RED policy is able to consistently follow human commands and achieve ahigher win rate than the baselines.
Revised: Experiment results and human studies showthat the RED policy is able to consistently follow human commands and, at thesame time, achieve a higher win rate than the baselines.

Original:  Introduction
Revised:  Introduction

Original: Building assistive agents that can help humans accomplish complex tasks remains a long-standingchallenge in artificial intelligence.
Revised: Building assistive agents that can help humans accomplish complex tasks remains a long-standingchallenge in artificial intelligence.

Original: Natural language, as the most generic protocol for humans toexchange and share knowledge, becomes a general and important interface for human-AI interaction.
Revised: Natural language, as the most generic protocol for humans toexchange and share knowledge, becomes a general and important interface for human-AI interaction.

Original: Decades of research efforts have been continuously made to develop AI systems that can groundagent behaviors to natural language commands [67, 62, 44].
Revised: Decades of research efforts have been continuously made to develop AI systems that can groundagent behaviors to natural language commands [69, 64, 46].

Original: Recently, as deep reinforcement learning (RL) techniques have been widely used to develop interactiveagents to solve a variety of challenging problems, it becomes a new trend to cast language groundingas an RL problem to train agents that can automatically ground language concepts to visual objects orbehaviors in an interactive fashion [27, 13, 15].
Revised: Recently, as deep reinforcement learning (RL) techniques have been widely used to develop interactiveagents to solve a variety of challenging problems, it becomes a new trend to cast language groundingas an RL problem to train agents that can automatically ground language concepts to visual objects orbehaviors in an interactive fashion [29, 13, 15].

Original: In particular, an RL agent is typically presented witha language command describing the goal of the RL task and will receive a success reward when thedesired goal state under the command is achieved.
Revised: In particular, an RL agent is typically presented witha language command describing the goal of the RL task and will receive a success reward when thedesired goal state under the command is achieved.

Original: A well-trained RL agent can often exhibit stronggeneralization capabilities to novel commands.
Revised: A well-trained RL agent can often exhibit stronggeneralization capabilities to novel commands.

Original: Despite the simplicity and effectiveness, such an RL formulation requires a language generator to repeatedly output random commands and an explicit reward function to determine whether the command is accomplished.
Revised: Despite the simplicity and effectiveness, such an RLformulation requires a language generator to repeatedly output random commands and an explicitreward function to determine whether the command is accomplished.

Original: Hence, most existing RL worksfocus on simple navigation problems with template-based compositional languages over objects andattributes, e.g., “go to the red box next to the blue wall” [54, 66, 32, 15].
Revised: Hence, most existing RL worksfocus on simple navigation problems with template-based compositional languages over objects andattributes, e.g., “go to the red box next to the blue wall” [56, 68, 34, 15].

Original: To train agents that can interpret general human languages, another research direction is to leveragepaired behavior-command data collected from human demonstrators.
Revised: To train agents that can interpret general human languages, another research direction is to leveragepaired behavior-command data collected from human demonstrators.

Original: Representative applications include robotic manipulation, where the robot needs to map language commands to predefinedmotion skills [58, 33, 59, 63], and vision-and-language navigation, where an agent learns from humandemonstrations to navigate towards a desired location in a 3D environment [49, 13, 27, 69].
Revised: Representative applications include robotic manipulation, where the robot needs to map language commands to predefinedmotion skills [60, 35, 61, 65], and vision-and-language navigation, where an agent learns fromhuman demonstrations to navigate towards a desired location in a 3D environment [51, 13, 29, 71].

Original: Sincean accurate command-following reward is no longer accessible, most works apply behavior cloning(BC) to directly imitate human behaviors or adopt inverse reinforcement learning (IRL) to learn alanguage-conditioned reward function [18, 50, 28, 22, 76].
Revised: Since an accurate command-following reward is no longer accessible, most works apply behaviorcloning (BC) to directly imitate human behaviors or adopt inverse reinforcement learning (IRL) tolearn a language-conditioned reward function [19, 52, 30, 23, 79, 17].

Original: Although the use of human data enablesnatural commands, pure imitation-based methods require massive demonstrations, which can beexpensive to collect.
Revised: Although the use of humandata enables natural commands, pure imitation-based methods require massive demonstrations,which can be expensive to collect.

Original: More importantly, both BC and IRL algorithms assume that the demonstrationsare optimal [18, 50, 28, 22, 76], in the sense that (i) each behavior demonstration is optimal under itspaired language command, and (ii) the language commands are optimal for the underlying task.
Revised: More importantly, both BC and IRL algorithms assume that thedemonstrations are optimal [19, 52, 30, 23, 79, 17], in the sense that (i) each behavior demonstrationis optimal under its paired language command, and (ii) the language commands are optimal forthe underlying task.

Original: Suchan assumption is reasonable in restricted domains like manipulation or navigation, where the optimalbehavior is simply the shortest trajectory to the goal while human commands typically provide strongguidance towards task completion.
Revised: Such an assumption is reasonable in restricted domains like manipulation ornavigation, where the optimal behavior is simply the shortest trajectory to the goal while humancommands typically provide strong guidance towards task completion.

Original: However, in more complex problems like playing real-timestrategy games, it is often the case that human players can be highly biased or sub-optimal [10].
Revised: However, in more complexproblems like playing real-time strategy games, it is often the case that human players can be highlybiased or sub-optimal [10].

Original: Let’s consider a concrete example in a simplified real-time strategy game, MiniRTS [31].
Revised: Let’s consider a concrete example in a simplified real-time strategy game, MiniRTS [33].

Original: Fig.
Revised: Fig.

Original: 1(C)shows a scenario where a human commander gives an abstract command “attack” to a human executorto attack enemy units.
Revised: 1(C)shows a scenario where a human commander gives an abstract command “attack” to a human executorto attack enemy units.

Original: However, the action taken by the human executor is sub-optimal: a unit with full
Revised: However, the action taken by the human executor is sub-optimal: a unit with full

Original: HP is under attack while a clear better strategy is to attack another dying unit.
Revised: HP is under attack while a clear better strategy is to attack another dying unit.

Original: Moreover, the humancommander can be sub-optimal as well.
Revised: Moreover, the humancommander can be sub-optimal as well.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1(E), very few enemy units are remaining,so the optimal command should be simply an “attack” for the win.
Revised: 1(E), very few enemy units are remaining,so the optimal command should be simply an “attack” for the win.

Original: However, the commander sends a“retreat” command.
Revised: However, the commander sends a“retreat” command.

Original: This could be possibly due to the partially observed game state or human biases,but an obedient executor should still faithfully control the units to stop attacking and retreat.
Revised: This could be possibly due to the partially observed game state or human biases,but an obedient executor should still faithfully control the units to stop attacking and retreat.

Original: We study this novel RL challenge, i.e., learning an strong agent capable of not only achieving a highwin rate in a real-time strategy game, e.g., MiniRTS, but also strictly following high-level languagecommands, even if a sub-optimal command is given.
Revised: We study this novel RL challenge, i.e., learning an strong agent capable of not only achieving ahigh win rate in a real-time strategy game, e.g., MiniRTS, but also strictly following high-levellanguage commands, even if a sub-optimal command is given.

Original: We call this problem, GroundedReinforcement
Revised: We call this problem, Grounded

Original:  Learning (GRL), which assumes an interactive RL environment with a dataset of (sub-optimal) humandemonstrations reflecting proper human behaviors under language commands.
Revised: Reinforcement Learning (GRL), which assumes an interactive RL environment with a dataset of(sub-optimal) human demonstrations reflecting proper human behaviors under language commands.

Revised: In this setting, the rewards are not associated with the language commands, and it is non-trivial toverify whether a high-level language command is completed or not.

Revised: So the policy can only learnabout how humans follow commands from the demonstrations.

Original: We remark that GRL isdifferent from standard language grounding problems since the primary mission is not towards betterlanguage understanding via interactions [11].
Revised: We remark that GRL is differentfrom standard language grounding problems since the primary mission is not towards better languageunderstanding via interactions [11].

Original: Instead, GRL focuses on the opposite direction, i.e.,learning strong winning strategies while taking human commands as high-level behavior regulations,which is more related to the concept of developing human-compatible AI [55].
Revised: Instead, GRL focuses on the opposite direction, i.e., learningstrong winning strategies while taking human commands as high-level behavior regulations, which ismore related to the concept of developing human-compatible AI [57].

Original: To tackle the GRL problem, we propose a simple yet effective constrained RL objective as a tractableapproximation and developed an iterative RL algorithm, REinforced demonstration Distillation(RED), to derive a strong language-conditioned policy.
Revised: To tackle the GRL problem, we propose a simple yet effective constrained RL objective as a tractableapproximation and developed an iterative RL algorithm, REinforced demonstration Distillation(RED), to derive a strong language-conditioned policy.

Original: RED adopts unconditioned RL training toencourage the policy to explore stronger winning strategies and periodically applies self-distillationand BC over demonstrations to ensure that the policy behavior is consistent with human preferences.
Revised: RED adopts unconditioned RL training toencourage the policy to explore stronger winning strategies and periodically applies self-distillationand BC over demonstrations to ensure that the policy behavior is consistent with human preferences.

Original: We evaluate the performance of RED as well as baseline methods including BC-based methods and
Revised: We evaluate the performance of RED as well as baseline methods including BC-based methods and

Original: RL variants on the MiniRTS environment.
Revised: RL variants on the MiniRTS environment.

Original: Simulation results show that RED policy achieves strongcommand-following capabilities and a higher win rate than baselines under various types of test-timecommands.
Revised: Simulation results show that RED policy achieves strongcommand-following capabilities and a higher win rate than baselines under various types of test-timecommands.

Original: We also conduct human evaluation by inviting 30 volunteers to play with policies trainedby different methods.
Revised: We also conduct human evaluation by inviting 30 volunteers to play with policies trainedby different methods.

Original: RED policy appears to be the most obedient under human votes and leads to amuch higher collaborative win rate with human commanders.
Revised: RED policy appears to be the most obedient under human votes and leads to amuch higher collaborative win rate with human commanders.

Original:  Related Work
Revised:  Related Work

Original: Language grounding.
Revised: Language grounding.

Original: The idea of grounding languages to behaviors or concepts can be tracedbacked to 1970s [67], and most early works assume simple domain-specific languages and a predefined set of skills [43, 74, 35, 14, 45, 5].
Revised: The idea of grounding languages to behaviors or concepts can be tracedbacked to 1970s [69], and most early works assume simple domain-specific languages and a predefined set of skills [45, 77, 37, 14, 47, 5].

Original: Thanks to the recent advances of deep RL, it becomesfeasible to ground languages to visual concepts and non-trivial behaviors in a simulated world.
Revised: Thanks to the recent advances of deep RL, it becomesfeasible to ground languages to visual concepts and non-trivial behaviors in a simulated world.

Original: Many3D environments with language-described goals have been developed over different domains, including maze exploration [15], visual navigation [13, 27, 69] and robot control [33, 59, 63], allowingend-to-end concept learning and grounding.
Revised: Many3D environments with language-described goals have been developed over different domains, including maze exploration [15], visual navigation [13, 29, 71] and robot control [35, 61, 65], allowingend-to-end concept learning and grounding.

Original: RL training requires a goal generator and a rewardfunction.
Revised: RL training requires a goal generator and a rewardfunction.

Original: Hence, these testbeds only adopt template-based languages over objects (e.g., box or cube)and attributes (e.g., spatial relation or color) and assume an oracle that can verify whether a state isconsistent with the language description.
Revised: Hence, these testbeds only adopt template-based languages over objects (e.g., box or cube)and attributes (e.g., spatial relation or color) and assume an oracle that can verify whether a state isconsistent with the language description.

Original: We focus on following high-level natural commands.
Revised: We focus on following high-level natural commands.

Original: Regarding natural commands, recent visual navigation benchmarks [3, 47, 59, 60] started to providelarge-scale human demonstrations with the goal or pathway specified by natural languages.
Revised: Regarding natural commands, recent visual navigation benchmarks [3, 49, 61, 62] started to providelarge-scale human demonstrations with the goal or pathway specified by natural languages.

Original: Behaviorcloning (BC) is feasible since every human description is paired with a successful trajectory [18, 50,28].
Revised: Behaviorcloning (BC) is feasible since every human description is paired with a successful trajectory [19, 52,30].

Original: RL fine-tuning can be also applied, since a precise success measurement (i.e., distance to thegoal position) w.r.t.
Revised: RL fine-tuning can be also applied, since a precise success measurement (i.e., distance to thegoal position) w.r.t.

Original: each language instruction is known [21, 65].
Revised: each language instruction is known [22, 67].

Original: Some works adopt inverse RL (IRL)to learn a language-conditioned reward function [22, 76] thanks to the fact that the demonstrationsare actually optimal.
Revised: Some works adopt inverse RL (IRL)to learn a language-conditioned reward function [23, 79] thanks to the fact that the demonstrationsare actually optimal.

Original: We consider a more challenging game MiniRTS [31] with abstract languagecommands and a complex behavior space.
Revised: We consider a more challenging game MiniRTS [33] with abstract languagecommands and a complex behavior space.

Original: BC and IRL methods work poorly in MiniRTS since bothlanguage commands and game trajectories from human annotators are highly sub-optimal.
Revised: BC and IRL methods work poorly in MiniRTS since bothlanguage commands and game trajectories from human annotators are highly sub-optimal.

Original: There are also successful attempts to leverage powerful pretrained language models to map generallanguages to the goal space [34, 42] or a predefined set of skills [29, 58, 2] so that massive paireddemonstrations can be no longer necessary.
Revised: There are also successful attempts to leverage powerful pretrained language models to map generallanguages to the goal space [36, 44] or a predefined set of skills [31, 60, 2] so that massive paireddemonstrations can be no longer necessary.

Original: We primarily consider the problem of policy learning, sothe use of pretrained model is orthogonal to our current focus but remains an exciting future direction.
Revised: We primarily consider the problem of policy learning, sothe use of pretrained model is orthogonal to our current focus but remains an exciting future direction.

Original: There are also parallel works on language groundingin other domains, such as text games [36],answer questioning [4] and communication [37], where languages serve as state descriptor or actionspace rather than commands to follow. Our work is also related to ad-hoc team play [64, 12] sincethe policy cooperates with arbitrary commands. The difference is that the commander is not assumedto be optimal under the game reward. The policy must be assistive [26, 68] or even obedient [48].
Revised: There are also parallel works on language grounding

Original: Constrained reinforcement learning.
Revised: Constrained reinforcement learning.

Original: We cast grounded RL as a constrained RL (CRL) formulation for tractability.
Revised: We cast grounded RL as a constrained RL (CRL) formulation for tractability.

Original: CRL algorithms often leverage the Lagrangian multiplier [53] or projectedgradients [1, 70] for policy improvement assuming simple (closed-form) constraints over states.
Revised: CRL algorithms often leverage the Lagrangian multiplier [55] or projectedgradients [1, 72] for policy improvement assuming simple (closed-form) constraints over states.

Original: In our setting, constraints can be only implicitly learned from human demonstrations.
Revised: In our setting, constraints can be only implicitly learned from human demonstrations.

Original: We simply adopt the behavior cloning objective, i.e., negative log likelihood, as the constraint satisfying metric,which is related to RL with demonstrations (RLwD) [30, 24].
Revised: We simplyadopt the behavior cloning objective, i.e., negative log likelihood, as the constraint satisfying metric,which is related to RL with demonstrations (RLwD) [32, 25].

Original: However, due to limited data and asignificant distribution shift between demonstrations and on-policy trajectories, RLwD methods canperform poorly.
Revised: However, due to limited data and asignificant distribution shift between demonstrations and on-policy trajectories, RLwD methods canperform poorly.

Revised: [73] iteratively optimizes the RL objective, projects the policy on a region around areference policy, and enforces the policy satisfies the cost constraint.

Revised: [27] updates the policies onthe mixture of the expert replay buffer and the rollout trajectories.

Original: The most related work to our RED algorithm is Supervised Seeded Iterated Learning(SSIL) [40], which is designed for learning grounded communication and adopts a conceptuallysimilar iterative framework by alternating between RL and self-distillation.
Revised: The most related work to our RED algorithm is Supervised Seeded Iterated Learning (SSIL) [42], which is designed for learninggrounded communication and adopts a conceptually similar iterative framework by alternating between RL and self-distillation.

Original: SSIL suggests to jointlyoptimize RL and BC objectives assuming human communications are perfect demonstrations.
Revised: SSIL suggests to jointly optimize RL and BC objectives assuminghuman communications are perfect demonstrations.

Original: Bycontrast, we find that an additional BC loss can be harmful to RL training due to the distribution shiftissue.
Revised: By contrast, we find that an additional BC losscan be harmful to RL training due to the distribution shift issue.

Original: Similar empirical phenomena have been also reported in multi-task RL literature [72, 61, 71].
Revised: Similar empirical phenomena havebeen also reported in multi-task RL literature [75, 63, 74].

Original:  Preliminary
Revised:  Preliminary

Original: 3.1 The MiniRTS Environment and Dataset
Revised: 3.1 The MiniRTS Environment and Dataset

Original: MiniRTS [31] is a grid-world RL environment (Fig. 1) that distills the key features of complexreal-time strategy games.
Revised: MiniRTS [33] is a grid-world RL environment (Fig. 1) that distills the key features of complexreal-time strategy games.

Original: It has two parties, a player (blue) controlled by a human/policy against abuilt-in script AI (red).
Revised: It has two parties, a player (blue) controlled by a human/policy against abuilt-in script AI (red).

Original: The player controls units to collect resources, do construction and kill allthe enemy units or destroy the enemy base to win a game.
Revised: The player controls units to collect resources, do construction and kill allthe enemy units or destroy the enemy base to win a game.

Original: A total of 7 unit types form a rock-paperscissor attacking dynamics.
Revised: A total of 7 unit types form a rock-paperscissors attacking dynamics.

Original: Learning a strong policy is challenging due to partially observed states,complicated micro-management over dozens of units, and extremely diverse possible strategies.
Revised: Learning a strong policy is challenging due to partially observed states,complicated micro-management over dozens of units, and extremely diverse possible strategies.

Original: In addition to an RL environment, MiniRTS provides a command-following dataset, which is collectedby two humans playing collaboratively against the script opponent.
Revised: In addition to an RL environment, MiniRTS provides a command-following dataset, which is collectedby two humans playing collaboratively against the script opponent.

Original: One is the commander, whogives high-level strategic language commands like “attack”, “retreat”, or “build an archer”.
Revised: One is the commander, whogives high-level strategic language commands like “attack”, “retreat”, or “build an archer”.

Original: Theother one is the executor, who controls the units to follow the language command.
Revised: Theother one is the executor, who controls the units to follow the language command.

Original: We remark thathuman data were collected against much weaker scripts than the default game opponent.
Revised: We remark thathuman data were collected against much weaker scripts than the default game opponent.

Original: Even insuch an easier mode, the collected game episodes are highly sub-optimal with a win rate of merely47.2%.
Revised: Even insuch an easier mode, the collected game episodes are highly sub-optimal with a win rate of merely47.2%.

Original: The dataset contains a total of 63,285 commands with an average of 13.7 per game while theaverage game horizon is 151.7.
Revised: The dataset contains a total of 63,285 commands with an average of 13.7 per game while theaverage game horizon is 151.7.

Original: So, each single command may take a long horizon to accomplish.
Revised: So, each single command may take a long horizon to accomplish.

Original: For notation simplicity in this paper, we assume an independent command is given to the executor ateach time step, although a command should be repeated for a few steps.
Revised: For notation simplicity in this paper, we assume an independent command is given to the executor ateach time step, although a command should be repeated for a few steps.

Original: Full details are in appendix.
Revised: Full details are in appendix.

Original: 3.2 Notation
Revised: 3.2 Notation

Original: Reinforcement learning.
Revised: Reinforcement learning.

Original: We formulate the MiniRTS environment as a Partially Observable Markov
Revised: We formulate the MiniRTS environment as a Partially Observable Markov

Original: Decision Process (POMDP), which is represented by a tuple ⟨S , A , O , O, r, p ⟩ .
Revised: Decision Process (POMDP), which is represented by a tuple ⟨S , A , O , O, r, p ⟩ .

Original: S is the state space.
Revised: S is the state space.

Original: A is the action space.
Revised: A is the action space.

Original: O is the observation space.
Revised: O is the observation space.

Original: r : S ×
Revised: r : S ×

Original: A → R is the reward function.p : S ×
Revised: A → R is the reward function.p : S ×

Original: A × S → [0 , 1] is the transition probability with p ( s ′ | s, a ) denoting the probability fromstate s to state s ′ after taking action a .
Revised: A × S → [0 , 1] is the transition probability with p ( s ′ | s, a ) denoting the probability fromstate s to state s ′ after taking action a .

Original: At each time step, the agent (i.e., the game player) observeso t
Revised: At each time step, the agent (i.e., the game player) observeso t

Original: = O ( s t ) , produces an action a t according to its policy π θ parameterized by θ , and receivesa reward r t = r ( s t , a t ) .
Revised: = O ( s t ) , produces an action a t according to its policy π θ parameterized by θ , and receivesa reward r t = r ( s t , a t ) .

Original: The optimal policy should achieve high win rates in the game, namelyθ ⋆ = arg max θ E a t ,s t
Revised: The optimal policy should achieve high win rates in the game, namelyθ ⋆ = arg max θ E a t ,s t

Original: [ (cid:80) t r ( s t , a t )] .
Revised: [ (cid:80) t r ( s t , a t )] .

Original: Note that we omit the discount factor γ for conciseness only.
Revised: Note that we omit the discount factor γ for conciseness only.

Original: Command-following policy.
Revised: Command-following policy.

Original: We represent the policy by π θ ( a t |
Revised: We represent the policy by π θ ( a t |

Original: o t , c t ) , which conditions on anobservation o t ∈ O and a language command c t ∈ C at each time step t .
Revised: o t , c t ) , which conditions on anobservation o t ∈ O and a language command c t ∈ C at each time step t .

Original: c t can be generated fromarbitrary distributions.
Revised: c t can be generated fromarbitrary distributions.

Original: Here C denotes the space of possible natural language commands.
Revised: C denotes the space of possible natural language commands.

Original: Particularin the paper, we consider a finite set of top 500 commands with the highest frequencies from thehuman demonstrations as a simplification. It is also possible to extend C to arbitrary languages usinga pretrained language model, which we leaves as future work.
Revised: An LSTM is usedto encode a command c to an embedding.

Original: Note that we augment C with a special“ NA ” command denoting “ no command ”.
Revised: We use a special command “ NA ” to denote “ no command ”,which corresponds to a zero embedding.

Original: When setting every c t to be NA , the command-followingpolicy π θ degenerates to a conventional policy in classical RL without language commands.
Revised: When setting every c t to be NA , the command-followingpolicy π θ degenerates to a conventional policy in classical RL without language commands.

Original: Human demonstrations.
Revised: Human demonstrations.

Original: We denote human demonstrations as a dataset of trajectories, i.e., D ={ τ 1 , τ 2 , . . . } .
Revised: We denote human demonstrations as a dataset of trajectories, i.e., D ={ τ 1 , τ 2 , . . . } .

Original: Each trajectory τ i , denoted by τ i =
Revised: Each trajectory τ i , denoted by τ i =

Original: ( s i 0 , a i 0 , c i 0 , s i 1 , a i 1 , c i 1 , . . . ) , consists of transitionsfrom a single game played by a human executor and the paired command sequence from a humancommander.
Revised: ( s i 0 , a i 0 , c i 0 , s i 1 , a i 1 , c i 1 , . . . ) , consists of transitionsfrom a single game played by a human executor and the paired command sequence from a humancommander.

Original: Human behaviors can be sub-optimal: i.e., given the observation o it
Revised: Human behaviors can be sub-optimal: i.e., given the observation o it

Original: = O
Revised: = O

Original: ( s it ) and thecommand c it , the chosen action a it can result in low rewards.
Revised: ( s it ) and thecommand c it , the chosen action a it can result in low rewards.

Original: Likewise, a command c it can be arbitrary w.r.t.
Revised: Likewise, a command c it can be arbitraryw.r.t.

Original: human preferences.
Revised: human preferences.

Original: Finally, since human commanders are always asked to generate languagecommands, D does not contain any NA command, i.e., c it ̸ = NA .
Revised: Finally, since human commanders are always asked to generate languagecommands, D does not contain any NA command, i.e., c it ̸ = NA .

Original:  Method
Revised:  Method

Original: 4.1 Grounded Reinforcement Learning: Problem Formulation
Revised: 4.1 Grounded Reinforcement Learning: Problem Formulation

Original: The mission of GRL is to learn a policy π θ such that π θ can achieve high rewards in the environment and follow any possible command c ∈ C by achieving consistent behaviors with humandemonstrations D .
Revised: The mission of GRL is to learn a policy π θ such that π θ can achieve high rewards in the environment and follow any possible command c ∈ C by achieving consistent behaviors with humandemonstrations D .

Original: We formulate the GRL problem as the following RL objective J G : 
Revised: We formulate the GRL problem as the following RL objective J G : [Equation] subject to K ( π [Equation]

Revised: , D ) ≤ δ

Original: [Equation]
Revised: [Equation]

Original:  where K denotes a distance metric between the policy behaviors from π θ and the human data D .
Revised: (1) where K denotes a distance metric between the policy behaviors from π θ and the human data D .

Original: Remark #1: The commands can be arbitrary.
Revised: Remark #1: The commands can be arbitrary.

Original: π θ needs to follow any possible command c ∈ C whilestill attains the highest rewards under the behavior constraint K ( π θ , D ) ≤ δ .
Revised: π θ needs to follow any possible command c ∈ C whilestill attains the highest rewards under the behavior constraint K ( π θ , D ) ≤ δ .

Original: Remark #2: Since the human actions in D can be sub-optimal w.r.t.
Revised: Remark #2: Since the human actions in D can be sub-optimal w.r.t.

Original: the paired commands, it can beproblematic to directly perform behavior cloning (BC) over D .
Revised: the paired commands, it can beproblematic to directly perform behavior cloning (BC) over D .

Original: Therefore, we utilize a threshold δ sothat the command-following policy π θ can possibly deviate from the sub-optimal behaviors in D .
Revised: Therefore, we utilize a threshold δ sothat the command-following policy π θ can possibly deviate from the sub-optimal behaviors in D .

Original: Note that Eq.
Revised: Note that Eq.

Original: 1 is generally intractable and is presented only for the purpose of problem definition.
Revised: 1 is generally intractable and is presented only for the purpose of problem definition.

Original: 4.2 Constrained RL as a Tractable Approximation for GRL
Revised: 4.2 Constrained RL as a Tractable Approximation for GRL

Original: There are two issues when directly optimizing Eq.
Revised: There are two issues when directly optimizing Eq.

Original: First, the command c t can be arbitrary.
Revised: First, the command c t can be arbitrary.

Original: Second,the choice of behavior distance metric K ( π θ , D ) should be specified.
Revised: Second,the choice of behavior distance metric K ( π θ , D ) should be specified.

Original: Training commands.
Revised: Training commands.

Original: An obvious choice is to sample a random command c t from C at each timestep t , which is perfectly aligned with Eq.
Revised: An obvious choice is to sample a random command c t from C at each timestep t , which is perfectly aligned with Eq.

Original: However, inconsistent commands from randomsampling are typically harmful for a win, making the RL agent tend to ignore the commands.
Revised: However, inconsistent commands from randomsampling are typically harmful for a win, making the RL agent tend to ignore the commands.

Original: Analternative is a “human proxy” by learning a command model h ϕ ( c t | s t ) over human data D , leadingto the objective E c t ∼ h ϕ ( s t ) ,a t ∼ π
Revised: Analternative is a “human proxy” by learning a command model h ϕ ( c t | s t ) over human data D , leadingto the objective E c t ∼ h ϕ ( s t ) ,a t ∼ π

Original: θ
Revised: θ

Original: ( o t ,c t ) [ (cid:80) t r ( a t , s t )] .
Revised: ( o t ,c t ) [ (cid:80) t r ( a t , s t )] .

Original: Although this is reasonable, we need to be 182 aware that human commands are sub-optimal and biased, which may further limit the chance for thepolicy π θ to discover strong strategies during RL training.
Revised: Although this is reasonable, we need to be 187 aware that human commands are sub-optimal and biased, which may further limit the chance for thepolicy π θ to discover strong strategies during RL training.

Original: Thus, we propose an extreme version: i.e.,directly performing unconditioned RL training by setting every command c t to be NA .
Revised: Thus, we propose an extreme version: i.e.,directly performing unconditioned RL training by setting every command c t to be NA .

Original: This allows thepolicy to freely explore the strategy space during RL training for the best winning strategies.
Revised: This allows thepolicy to freely explore the strategy space during RL training for the best winning strategies.

Original: Command-following metric.
Revised: Command-following metric.

Original: The simplest distance metric K is the behavior cloning (BC) loss,i.e., negative log-likelihood (NLL) of π θ for ( s t , a t , c t ) ∈ D .
Revised: The simplest distance metric K is the behavior cloning (BC) loss,i.e., negative log-likelihood (NLL) of π θ for ( s t , a t , c t ) ∈ D .

Original: Specifically, we can define BC objective
Revised: Specifically, we can define BC objective

Original: L ( θ ) =
Revised: L ( θ ) =

Original: E D [ − log π θ ( a t | o t , c t )] and constrain L ( θ ) ≤ δ to ensure the policy behavior does notdeviate from the demonstrations too much.
Revised: E D [ − log π θ ( a t | o t , c t )] and constrain L ( θ ) ≤ δ to ensure the policy behavior does notdeviate from the demonstrations too much.

Original: There can be alternatives, such as KL-divergence or alearned metric.
Revised: There can be alternatives, such as KL-divergence or alearned metric.

Original: Regarding KL-divergence, since we are measuring the distance between samples
Revised: Regarding KL-divergence, since we are measuring the distance between samples

Original: D and a distribution π θ , KL ( D|| π θ ) is equivalent to L ( θ ) while KL ( π θ ||D ) requires an behaviorproxy over D , which may involve additional biases.
Revised: D and a distribution π θ , KL ( D|| π θ ) is equivalent to L ( θ ) while KL ( π θ ||D ) requires an behaviorproxy over D , which may involve additional biases.

Original: Regarding a learned metric, we can apply inverse
Revised: Regarding a learned metric, we can apply inverse

Original: RL to derive a reward function on whether the command is accomplished [22, 6, 7].
Revised: RL to derive a reward function on whether the command is accomplished [23, 6, 7].

Original: However, weempirically notice that a learning-based metric works poorly on MiniRTS.
Revised: However, weempirically notice that a learning-based metric works poorly on MiniRTS.

Original: We hypothesis that this isbecause the demonstrations are sub-optimal and possibly limited in size for such a complex game.
Revised: We hypothesis that this isbecause the demonstrations are sub-optimal and possibly limited in size for such a complex game.

Original: Tractable objective.
Revised: Tractable objective.

Original: To sum up, we propose the following constrained RL objective J C ( θ ) as atractable objective for the GRL problem, i.e.,
Revised: To sum up, we propose the following constrained RL objective J C ( θ ) as atractable objective for the GRL problem, i.e.,

Original: [Equation]
Revised: [Equation]

Original: a t ) (cid:35) subject toL ( θ ; D ) =
Revised: subject to

Original: E D [ − log π θ ( a t | o t , c t )] ≤ δ.
Revised: [Equation]

Original: Note that in Eq.
Revised: Note that in Eq.

Original: (2), the sample distribution from the RL objective J C is disjoint from the BCobjective L ( θ ; D ) since we force c t =
Revised: (2), the sample distribution from the RL objective J C is disjoint from the BCobjective L ( θ ; D ) since we force c t =

Original: NA in RL training while D does not contain NA at all.
Revised: NA in RL training while D does not contain NA at all.

Original: Hence, Eq.
Revised: Hence, Eq.

Original: (2) may look a bit ill-posed: an obvious solution is a “switching” policy which takeseither the pure RL strategy or the BC strategy with or without a command.
Revised: (2) may look a bit ill-posed: an obvious solution is a “switching” policy which takeseither the pure RL strategy or the BC strategy with or without a command.

Original: However, we will empirically show that such a switching policy does not provide strong performances.
Revised: However, we willempirically show that such a switching policy does not provide strong performances.

Original: The insightis that we are training a conditioned neural policy .
Revised: The insightis that we are training a conditioned neural policy .

Original: It is often observed in practice (e.g., multi-tasklearning) that a neural network can implicitly distill strategies from different modalities via itsshared representations [19, 52, 73].
Revised: It is often observed in practice (e.g., multi-tasklearning) that a neural network can implicitly distill strategies from different modalities via itsshared representations [20, 54, 76].

Original: Therefore, the winning strategy discovered from RL training canempirically improve the sub-optimal command-following demonstrations within a bounded range.
Revised: Therefore, the winning strategy discovered from RL training canempirically improve the sub-optimal command-following demonstrations within a bounded range.

Original: This phenomenon can be also justified by some recent theoretical findings [46, 16].
Revised: This phenomenon can be also justified by some recent theoretical findings [48, 16].

Original: 4.3 Reinforced Demonstration Distillation
Revised: 4.3 Reinforced Demonstration Distillation

Original: A straightforward way to solve the constrained optimization problem in Eq.
Revised: A straightforward way to solve the constrained optimization problem in Eq.

Original: (2) is to adopt the
Revised: (2) is to adopt the

Original: Lagrangian multiplier, which leads to a joint optimization problem, i.e.,
Revised: Lagrangian multiplier, which leads to a joint optimization problem, i.e.,

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: (3) treats the BC objective L ( θ ; D ) as an auxiliary loss of the stardard RL objective J C ( θ ) .
Revised: (3) treats the BC objective L ( θ ; D ) as an auxiliary loss of the stardard RL objective J C ( θ ) .

Original: Theconstraint in Eq.
Revised: Theconstraint in Eq.

Original: (2) can be satisfied by tuning the coefficient β .
Revised: (2) can be satisfied by tuning the coefficient β .

Original: However, due to the distributionshift issue between on-policy samples and demonstrations, the supervised learning loss may interferewith the policy gradient and further makes the training process sensitive and unstable [20, 38].
Revised: However, due to the distributionshift issue between on-policy samples and demonstrations, the supervised learning loss may interferewith the policy gradient and further makes the training process sensitive and unstable [21, 40].

Original: Iterative solution.
Revised: Iterative solution.

Original: Inspired by the recent advances in self-imitation [51] and self-distillation [75,41, 40], we adopt an iterative framework to decouple the RL loss and the BC loss in Eq.
Revised: Inspired by the recent advances in self-imitation [53] and self-distillation [78,43, 42], we adopt an iterative framework to decouple the RL loss and the BC loss in Eq.

Original: Theidea is simple: for RL training, we solely estimate the unconstrained policy gradient over J C withoutconsidering the BC loss; after policy improvement, we distill the demonstrations into the policyby running pure behavior cloning over both human demonstrations and self-generated winningtrajectories.
Revised: Theidea is simple: for RL training, we solely estimate the unconstrained policy gradient over J C withoutconsidering the BC loss; after policy improvement, we distill the demonstrations into the policyby running pure behavior cloning over both human demonstrations and self-generated winningtrajectories.

Original: By repeatedly alternating between pure RL and pure BC, we are able to achieve a stablelearning process.
Revised: By repeatedly alternating between pure RL and pure BC, we are able to achieve a stablelearning process.

Original: We call this method, REinforced demonstration Distillation (RED).
Revised: We call this method, REinforced demonstration Distillation (RED).

Original: Specifically, let θ k denote the parameters at iteration k , then we have the following update rule.
Revised: Specifically, let θ k denote the parameters at iteration k , then we have the following update rule.

Original: [Equation]
Revised: [Equation]

Original: Here α denotes the learning rate.
Revised: Here α denotes the learning rate.

Original: By repeating the update rules for N iterations, we will derive ourfinal parameter θ ⋆ .
Revised: By repeating the update rules for N iterations, we will derive ourfinal parameter θ ⋆ .

Original: The remaining issue is to ensure the constraint is satisfied, i.e., L ( θ ; D ) ≤ δ .
Revised: The remaining issue is to ensure the constraint is satisfied, i.e., L ( θ ; D ) ≤ δ .

Original: Weempirically notice that this can be accomplished by controlling the ratio between the size of on-policysamples ∥D k ∥ and the size of demonstrations ∥D∥ .
Revised: Weempirically notice that this can be accomplished by controlling the ratio between the size of on-policysamples ∥D k ∥ and the size of demonstrations ∥D∥ .

Original: We present an intuitive justification of the self-distillation process from the perspective of sampledistributions [9, 8].
Revised: We present an intuitive justification of the self-distillation process from the perspective of sampledistributions [9, 8].

Original: We adopt RL for the highest rewards, so the unconditioned policy may frequentlyvisit those states approaching a win.
Revised: We adopt RL for the highest rewards, so the unconditioned policy may frequentlyvisit those states approaching a win.

Original: As a result, for games played by strong humans with aggressivecommands, the paired game states will be more aligned with the RL state distribution while the statesproduced by biased or defensive human commanders will be more apart.
Revised: As a result, for games played by strong humans with aggressivecommands, the paired game states will be more aligned with the RL state distribution while the statesproduced by biased or defensive human commanders will be more apart.

Original: As we are distilling two driftdistributions into a single neural policy π θ , it will be more likely to get actions over those overlappingstates improved within a bounded BC loss during optimization (see evidences in Sec. 5.2).
Revised: As we are distilling two driftdistributions into a single neural policy π θ , it will be more likely to get actions over those overlappingstates improved within a bounded BC loss during optimization (see evidences in Sec. 5.2).

Original: 4.4 Implementations
Revised: 4.4 Implementations

Original: We highlight some critical implementation factors here.
Revised: We highlight some critical implementation factors here.

Original: More details are in appendix.
Revised: More details are in appendix.

Original: Policy architecture.
Revised: Policy architecture.

Original: We adopt a similar policy architecture to the provided executor model in
Revised: We adopt a similar policy architecture to the provided executor model in

Original: MiniRTS [31] but with a modified action space and an additional value head as the critic.
Revised: MiniRTS [33] but with a modified action space and an additional value head as the critic.

Original: The originalmodel outputs an action for every controllable unit, which makes RL training extremely slow.
Revised: The originalmodel outputs an action for every controllable unit, which makes RL training extremely slow.

Original: Weintroduce an additional 0/1 selection action on each unit denoting whether this unit should act or not ,which substantially reduces the action dimension.
Revised: Weintroduce an additional 0/1 selection action on each unit denoting whether this unit should act or not ,which substantially reduces the action dimension.

Original: Policy learning.
Revised: Policy learning.

Original: We use PPO [57] for RL training, which involves multiple mini-batch policygradient steps.
Revised: We use PPO [59] for RL training, which involves multiple mini-batch policygradient steps.

Original: Since RL from scratch fails completely in practice, we fine-tune the BC policy andpretrain the value head with the policy parameters frozen.
Revised: Since RL from scratch fails completely in practice, we fine-tune the BC policy andpretrain the value head with the policy parameters frozen.

Original: We also empirically find learning ratewarmup [39] particularly helpful for stabilizing training with a pretrained model: we start withlearning rate 

Revised: We also empirically find learning rate warmup [41] particularly helpful for stabilizing training with a pretrained model: we start withlearning rate

Original: and then linearly increase it to the desired value α .
Revised: and then linearly increase it to the desired value α .

Original: Creating D k .
Revised: Creating D k .

Original: Note that in the early stage of training phase, the winning rate can be low.
Revised: Note that in the early stage of training phase, the winning rate can be low.

Original: In orderto obtain sufficient data for D k , we implement a queue to store all past winning trajectories and use bootstrapped sampling if the queue is still not full.
Revised: In orderto obtain sufficient data for D k , we implement a queue to store all past winning trajectories and usebootstrapped sampling if the queue is still not full.

Original: For the ratio between ∥D k ∥ and ∥D∥ , we ensurethe BC loss (i.e., negative log likelihood) of π ⋆θ over the validation set is at most 10% more than thepure BC policy (i.e., no RL training).
Revised: For the ratio between ∥D k ∥ and ∥D∥ , we ensurethe BC loss (i.e., negative log likelihood) of π ⋆θ over the validation set is at most 10% more than thepure BC policy (i.e., no RL training).

Original: We can also run sub-sampling over D to reduce the requiredsample size.
Revised: We can also run sub-sampling over D to reduce the requiredsample size.

Original: Empirically, we find that simply setting ∥D k ∥ =
Revised: Empirically, we find that simply setting ∥D k ∥ =

Original: ∥D∥ leads to strong performances.
Revised: ∥D∥ leads to strong performances.

Original:  Experiment
Revised:  Experiment

Original: All the simulation win rates are based on 1200 test games and repeated over 3 random seeds.
Revised: All the simulation win rates are based on 1200 test games and repeated over 3 random seeds.

Original: Moredetails and additional results including emergent behaviors are deferred to appendix.
Revised: Moredetails and additional results including emergent behaviors are deferred to appendix.

Original: Baselines.
Revised: Baselines.

Original: We consider the following methods in addition to our RED algorithm as baselines.
Revised: We consider the following methods in addition to our RED algorithm as baselines.

Original: “ RL ” first pretrains the policy over demonstrations and then performs pure unconditioned RL
Revised: “ RL ” first pretrains the policy over demonstrations and then performs pure unconditioned RLtraining (i

Original: training without any command-following constraint.
Revised: e., conditioning on “ NA ” command) without any command-following constraint.2.

Original: “ Switch ” is the “switching” policy, which consists of two policies, a pure BC policy from demonstrations and a pure RL policy without commands.
Revised: “ Switch ” is the “switching” policy, which consists of two policies, a pure BC policy from demonstrations and a pure RL policy without commands.

Original: When a langauge command is given, it runsthe BC policy and uses the RL policy otherwise.3.
Revised: When a langauge command is given, it runsthe BC policy and uses the RL policy otherwise.3.

Original: “ Joint ” optimizes the soft objective J softC in Eq.
Revised: “ Joint ” optimizes the soft objective J softC in Eq.

Original: β is selected via a grid-search process.4.
Revised: β is selected via a grid-search process.4.

Original: “ IRL ” learns a reward function from demonstrations [23] and combines the game reward and thelearned language reward to train a conditioned policy.
Revised: “ IRL ” learns a reward function from demonstrations [24] and combines the game reward and thelearned language reward to train a conditioned policy.

Original: Evaluation.
Revised: Evaluation.

Original: To evaluate a strong GRL policy, we need to answer the following questions.
Revised: To evaluate a strong GRL policy, we need to answer the following questions.

Original: Does the policy learn a strong winning strategy?
Revised: Does the policy learn a strong winning strategy?

Original: The win rate can be a direct metric for thisquestion.
Revised: The win rate can be a direct metric for thisquestion.

Original: A RED policy should largely improves a pure BC policy with a high win rate.
Revised: A RED policy should largely improves a pure BC policy with a high win rate.2.

Original: Does the policy follow the commands well?
Revised: Does the policy follow the commands well?

Original: A good GRL policy should keep its winning strategywhile follow arbitrary commands.
Revised: A good GRL policy should keep its winning strategywhile follow arbitrary commands.

Original: A precise evaluation of this question is non-trivial since theconstraint satisfying condition, i.e., BC loss, can be a very misleading signal 2 .
Revised: A precise evaluation of this question is non-trivial since theconstraint satisfying condition, i.e., BC loss, can be a very misleading signal 2 .

Original: We adopt anapproximate measurement via win rates in Sec.
Revised: We adopt anapproximate measurement via win rates in Sec.

Original: and then conduct human evaluation in Sec.
Revised: and then conduct human evaluation in Sec.

Original: 5.3.Ablation studies on algorithmic hyper-parameters of our RED algorithm are also conducted in Sec.
Revised: Ablation studies on algorithmic hyper-parameters of our RED algorithm are also conducted in Sec.

Original: 5.1 Main Results
Revised: 5.1 Main Results

Original: Q#1: is the RED policy strong?
Revised: Q#1: is the RED policy strong?

Original: We measure the win rates of different policies under two test-timecommand strategies, i.e., a pure NA strategy, which always gives an NA , and an Oracle strategy, whichgives “optimal” commands.
Revised: We measure the win rates of different policies under two test-timecommand strategies, i.e., a pure NA strategy, which always gives an NA , and an Oracle strategy, whichgives “optimal” commands.

Original: For the NA strategy, the GRL problem degenerates to the conventional
Revised: For the NA strategy, the GRL problem degenerates to the conventional

Original: RL setting, so a strong GRL policy should at least achieve comparable performances to the RL policyand substantially outperforms the BC (i.e., Switch ) policy.
Revised: RL setting, so a strong GRL policy should at least achieve comparable performances to the RL policyand substantially outperforms the BC (i.e., Switch ) policy.

Original: For the Oracle strategy, it was scriptedto give carefully tuned commands based on the ground-truth game state to instruct the policy tobuild dominating units according to the rock-paper-scissor dynamics.
Revised: For the Oracle strategy, it was scriptedto give carefully tuned commands based on the ground-truth game state to instruct the policy tobuild dominating units according to the rock-paper-scissors dynamics.

Original: Hence, the GRL problem isconverted to a fully cooperative game, and following Oracle commands becomes the optimal executorstrategy.
Revised: Hence, the GRL problemis converted to a fully cooperative game, and following Oracle commands becomes the optimalexecutor strategy.

Original: A strong GRL policy should achieve the highest possible win rate.
Revised: A strong GRL policy should achieve the highest possible win rate.

Original: Meanwhile, the policywith the highest win rate must produce a good command-following behavior.
Revised: Meanwhile, thepolicy with the highest win rate must produce a good command-following behavior.

Original: We also report thewin rates of different methods with random commands and a “human proxy” commander, which islearned from human data.
Revised: We also reportthe win rates of different methods with random commands and a “human proxy” commander, whichis learned from human data.

Original: Since Random and Human Proxy are sub-optimal, a good GRL policyshould have a substantially lower win rate than the NA case but still outperforms BC ( Switch ) policy.[Table]
Revised: Since Random and Human Proxy are sub-optimal, a good GRL policyshould have a substantially lower win rate than the NA case but still outperforms BC ( Switch ) policy.

Original: The results are summarized in Tab. 1.
Revised: The results are summarized in Tab. 1.

Original: The win rate by RED is the highest under oracle commands andis comparable to RL and Switch without commands ( NA ).
Revised: The win rate by RED is the highest under oracle commands andis comparable to RL and Switch without commands ( NA ).

Original: Both RED and Joint policies substantially outperform Switch (BC) policy with sub-optimal commands (Random and Human Proxy), suggestingthe conditioned behaviors are improved.
Revised: Both RED and Joint policies substantiallyoutperform Switch (BC) policy with sub-optimal commands (Random and Human Proxy), suggestingthe conditioned behaviors are improved.

Original: Neither RL nor IRL is able to make the policy obedient –even under random commands, their win rates remain comparable to the NA case.
Revised: Neither RL nor IRL is able to make the policy obedient –even under random commands, their win rates remain comparable to the NA case.

Original: We also note thatcommands from human proxy generally yield an even lower win rate than random commands.
Revised: We also note thatcommands from human proxy generally yield an even lower win rate than random commands.

Original: Weremark that learning a good human commander model is extremely hard – not only due to the limitedhuman command data in the dataset but also because most of the commands are highly sub-optimal. 
Revised: Weremark that learning a good human commander model is extremely hard – not only due to the limitedhuman command data in the dataset but also because most of the commands are highly sub-optimal. [Table]

Original: An obedient policy should have a decreasing win rate.
Revised: An obedient policy should have a decreasing win rate.

Original: Q#2: is the RED policy grounded?
Revised: Q#2: is the RED policy grounded?

Original: Note that Tab. 1 hasalready provided evidences on the command-following performance of the RED policy, since the highest win rates can be onlyachieved by following the Oracle commands.
Revised: Note that Tab. 1 hasalready provided evidences on the command-following perfor- mance of the RED policy, since the highest win rates can be onlyachieved by following the Oracle commands.

Original: Here we presentan additional criterion: intuitively, as a command-followingpolicy, its win rate should decrease if “worse” commands arefed; otherwise, if the win rate does not decay, the policy mustnot follow the commands well.
Revised: Here we presentan additional criterion: intuitively, as a command-followingpolicy, its win rate should decrease if “worse” commands arefed; otherwise, if the win rate does not decay, the policy mustnot follow the commands well.

Original: Hence, we interpolate betweenthe NA strategy and the full Random strategy and evaluate thewin rate of different policies with different ratios of randomcommands.
Revised: Hence, we interpolate betweenthe NA strategy and the full Random strategy and evaluate thewin rate of different policies with different ratios of randomcommands.

Original: The results are shown in Fig.
Revised: The results are shown in Fig.

Original: We can observethat the RL policy and IRL policy are clearly not following thecommands.
Revised: We can observethat the RL policy and IRL policy are clearly not following thecommands.

Original: Among the remaining command-following policies,the RED policy generally produces a higher win rate than Switch and Joint , which provide evidences that our RED algorithm leads to improved strategies underthe command-following requirement.
Revised: Among the remaining command-following policies, RED policy generally produces a higher win rate than Switch and Joint , which provides evidencesthat our RED algorithm leads to improved strategies under the command-following requirement.

Original: 5.2 Ablation Study
Revised: 5.2 Ablation Study

Original: Training commands.
Revised: Training commands.

Original: We test the performance of RED with different training commands in Fig.
Revised: We test the performance of RED with different training commands in Fig.

Original: The default RED yields the best strategy, i.e., the highest win rate with NA and oracle commands.
Revised: The default RED yields the best strategy, i.e., the highest win rate with NA and oracle commands.

Original: Training with random commands or human proxy makes the policy command-ignorant – both twovariants achieve similar win rates under NA and random commands. 
Revised: Training with random commands or human proxy makes the policy command-ignorant – both twovariants achieve similar win rates under NA and random commands. [Table] Win [Figure]

Original: a command-ignoring policy.
Revised: [Table] [Table] [Table] [Table]

Original: ∥D k ∥ : ∥D∥
Revised: NA [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Revised: Constraint Satisfaction on

Original: [Table]
Revised: [Table]

Original:  [Table]
Revised: NLL [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Test Win Rate with Different Commands
Revised: Accuracy [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Ratio of ∥D k ∥ and ∥D∥ .
Revised: Ratio of ∥D k ∥ and ∥D∥ .

Original: We test different ratio of on-policy samples, i.e., ∥D k ∥ , tothe demonstration size, i.e., ∥D∥ , in Tab. 2.
Revised: We test different ratios of on-policy samples, i.e., ∥D k ∥ , to thedemonstration size, i.e., ∥D∥ , in Tab. 2.

Original: We report the test-time win rates with oracleand NA commands as well as the constraintsatisfaction condition, including both NLL(i.e. , L ( θ ; D ) from Eq. 2) and the action prediction accuracy, on the validation set.
Revised: We report the test-time win rates with oracle and NAcommands as well as the constraint satisfactioncondition, including both NLL (i.e., L ( θ ; D )from Eq. 2) and the action prediction accuracy,on the validation set.

Original: Theresults show that the validation NLL can beindeed controlled by tuning the dataset ratio,i.e., more RL samples consistently yieldinga higher NLL.
Revised: The results show thatthe validation NLL can be indeed controlled bytuning the dataset ratio, i.e., more RL samplesconsistently yielding a higher NLL.

Original: By contrast, we also find thata lower NLL does NOT necessarily lead to ahigher action prediction accuracy.
Revised: By contrast, we also find that a lower NLL does NOTnecessarily lead to a higher action predictionaccuracy.

Original: But we empirically find thatsimply setting a 1:1 ratio leads to asufficiently good policy, which is the default choice for our RED algorithm.
Revised: But we empirically find that

Original: Learned strategy.
Revised: Learned strategy.

Original: We identify the five most common categoriesof commands and measure the validation action prediction accuracyfor both RED and BC policies to examine under what conditionsthe actions are changed – noting that RED policy uses BC policyas  warm-start.
Revised: We identify the five most common categoriesof commands and measure the validation action prediction accuracyfor both RED and BC policies to examine under what conditionsthe actions are changed – noting that RED policy uses BC policyas a warm-start.

Original: Fig.
Revised: Fig.

Original: 4 shows the categorized accuracy differenceswith the dashed line denoting the overall mean difference.
Revised: 4 shows the categorized accuracy differenceswith the dashed line denoting the overall mean difference.

Original: We canobserve that RED policy disagrees with human actions most undercommands belonging to “attack” and “defend” categories, whichboth represent a highly complicated space of possible behaviors (e.g.,which enemy unit to attack, how to defend).
Revised: We canobserve that RED policy disagrees with human actions the mostunder commands belonging to “attack” and “defend” categories,which both represent a highly complicated space of possible behaviors (e.g., which enemy unit to attack, how to defend).

Original: We believe these actionchanges are caused by policy improvement.
Revised: We believethese action changes are caused by policy improvement.

Original: While for commandswith a low uncertainty belong to “mine” and “stop” categories, REDpolicy simply follows humans.
Revised: While forcommands with a low uncertainty belonging to “mine” and “stop”categories, RED policy simply follows humans.

Original: These findings provide empiricalsupports to our intuitive justification in Sec. 4.3.
Revised: These findingsprovide empirical supports to our intuitive justification in Sec. 4.3.

Original: 5.3 Human Evaluation
Revised: 5.3 Human Evaluation

Original: [Figure]
Revised: when cooperating with the 4 models.

Original: [Figure]
Revised: Each model is tested over 60 games.

Original: We pick 4 policies, i.e., RED, Switch, Joint and RL, and invite 30 collegestudent volunteers under departmentpermission to complete a two-stagestudy.
Revised: We pick 4 policies, i.e., RED, Switch, Jointand RL, and invite 30 college student volunteers under department permission to complete a two-stage study.

Original: In the first stage, we shuffle theorder of the 4 policies and ask eachstudent to keep playing with the 4 policies using any commands.
Revised: In the first stage,we shuffle the order of the 4 policies andask each student to keep playing with the4 policies using any commands.

Original: Once thestudent feels familiar with the gameand policies enough, he/she is askedto rank the policies by the level of howthey follow commands.
Revised: Once thestudent feels familiar with the game andpolicies enough, he/she is asked to rank thepolicies by the level of how they followcommands.

Original: In the secondpart, each student is asked to play 2 games per policy, and we measure the average winning rate ofdifferent policies over the 60 collected game episodes.
Revised: In the second part, each student is asked to play 2 games per policy, and we measure theaverage winning rate of different policies over the 60 collected game episodes.

Original: The results of ranking votes and win rates are shown in Fig.
Revised: The results of ranking votes and win rates are shown in Fig.

Original: 5 and Fig.
Revised: 5 and Fig.

Original: Regarding the votes, theaverage rank of RED, Switch, Joint and RL are 2.3, 2.4, 2.5, 2.8, respectively.
Revised: Regarding the votes, theaverage rank of RED, Switch, Joint and RL are 2.3, 2.4, 2.5, 2.8, respectively.

Original: Fig.
Revised: Fig.

Original: 5 visualizes thevote percentage for different policies.
Revised: 5 visualizes thevote percentage for different policies.

Original: We notice a very interesting phenomenon that every policy hasalmost the same 1st-rank votes.
Revised: We notice a very interesting phenomenon that every policy hasalmost the same 1st-rank votes.

Original: Knowing that we shuffle the policy order while the students oftenspend a long time on the first policy to get familiar with MiniRTS, we hypothesize that the studentsare often biased towards the first policy they encountered.
Revised: Knowing that we shuffle the policy order while the students oftenspend a long time on the first policy to get familiar with MiniRTS, we hypothesize that the studentsare often biased towards the first policy they encountered.

Original: Nevertheless, RED has substantially morevotes for the 2nd-rank, indicating a better command-following capacity.
Revised: Nevertheless, RED has substantially morevotes for the 2nd-rank, indicating a better command-following capacity.

Original: RL has the most 4th-rankvotes, which is consistent with our expectation.
Revised: RL has the most 4th-rankvotes, which is consistent with our expectation.

Original: Regarding the win rates (Fig. 6), RED outperforms
Revised: Regarding the win rates (Fig. 6), RED outperforms

Original: Switch and Joint with a clear margin.
Revised: Switch and Joint with a clear margin.

Original: The win rate of RL is comparable with RED.
Revised: The win rate of RL is comparable with RED.

Original: We believe this isdue to the fact that RL policy often ignores human commands and simply acts for winning.
Revised: We believe this isdue to the fact that RL policy often ignores human commands and simply acts for winning.

Original:  Conclusion
Revised:  Conclusion

Original: We tackle a new problem, grounded reinforcement learning (GRL), which aims to learn an agent thatcan not only get high rewards but also faithfully follow natural language commands from humans.
Revised: We tackle a new problem, grounded reinforcement learning (GRL), which aims to learn an agent thatcan not only get high rewards but also faithfully follow natural language commands from humans.

Original: We proposed a tractable objective, and developed an iterative RL algorithm RED, and evaluated thederived policy on a real-time strategy game MiniRTS.
Revised: We proposed a tractable objective, and developed an iterative RL algorithm RED, and evaluated thederived policy on a real-time strategy game MiniRTS.

Original: Both simulation and human results show that
Revised: Both simulation and human results show that

Original: a
Revised: a

Original: RED policy achieves high win rates while exhibits strong command-following capabilities.
Revised: RED policy achieves high win rates while exhibits strong command-following capabilities.

Original: Limitation and social impact.
Revised: Limitation and social impact.

Original: As an initial study on the GRL problem, the RED algorithm isonly evaluated on MiniRTS.
Revised: As an initial study on the GRL problem, the RED algorithm isonly evaluated on MiniRTS.

Original: This is because MiniRTS is the only public environment that is bothcomplex (beyond “reaching goals”) and provides natural language data.
Revised: This is because MiniRTS is the only public environment that is bothcomplex (beyond “reaching goals”) and provides natural language data.

Original: In addition, although wedraw connections to existing theories in multi-task learning to justify why RED works, the theoreticalprinciple of the underlying optimization process remains an open problem.
Revised: In addition, although wedraw connections to existing theories in multi-task learning to justify why RED works, the theoreticalprinciple of the underlying optimization process remains an open problem.

Original: Finally, even though itis in general debatable whether a robot should be fully obedient [17, 25, 48, 56], we believe that inrestricted single-agent domains, like playing video games, strictly enforcing command-followingbehaviorsis compatible with human values and should not result in any negative social impact.
Revised: Finally, even though itis in general debatable whether a robot should be fully obedient [18, 26, 50, 58], we believe that inrestricted single-agent domains, like playing video games, strictly enforcing command-followingbehaviors should not result in worse negative social impact than BC or other RL methods.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes] See Section.1.(b)
Revised: [Yes] See Section.

Original:  Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Section.
Revised: [Yes] See Section.

Original: 6.(c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] SeeSection.
Revised: [Yes] SeeSection.

Original: 6.(d)
Revised: 6.(d)

Original: Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A](b)
Revised: [N/A](b)

Original: Did you include complete proofs of all theoretical results?
Revised: Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]3.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] In supplementalmaterial. (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: [Yes] In supplementalmaterial.(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] See Appendix.
Revised: [Yes] See Appendix.

Original: B.
Revised: B.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] See Section.5.
Revised: [Yes] See Section.5.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Appendix.
Revised: [Yes] See Appendix.

Original: B.4.
Revised: B.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Section.3.
Revised: [Yes] See Section.3.

Original: [Table]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] See Appendix.
Revised: [Yes] See Appendix.

Original: A.
Revised: A.

Original: [Table] (c) Did you include any new assets either in the supplemental material or as a URL?[N/A]
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?[N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re [Table] using/curating?

Original: [Yes] See Appendix.
Revised: [Yes] See Appendix.

Original: A.
Revised: A.

Original: [Table]
Revised: [Table]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Revised: [Table]

Original: information or offensive content?
Revised: information or offensive content?

Original: [Yes] See Appendix.
Revised: [Yes] See Appendix.

Original: A.
Revised: A.

Revised: [Table]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original:  (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: [Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table] applicable?

Original: [Yes] See Appendix.
Revised: [Yes] See Appendix.

Original: D.
Revised: E.

Original:  (b) Did you describe any potential participant risks, with links to Institutional Review 
Revised: [Table] (b) Did you describe any potential participant risks, with links to Institutional Review [Table]

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [Yes] See Appendix.
Revised: [Yes] See Appendix.

Original: D.
Revised: E.

Original:  (c) Did you include the estimated hourly wage paid to participants and the total amount 
Revised: [Table] (c) Did you include the estimated hourly wage paid to participants and the total amount [Table]

Original: spent on participant compensation?
Revised: spent on participant compensation?

Revised: [Yes] See Appendix.

Revised: E.

Revised: [Table]