Original: ACTRCE:
Revised: ACTRCE:

Original: A UGMENTING
Revised: A UGMENTING

Original: E XPERIENCE
Revised: E XPERIENCE

Original: VIA
Revised: VIA

Original: T EACHER ’ S A DVICE
Revised: T EACHER ’ S A DVICE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Sparse reward is one of the most challenging problems in reinforcement learning (RL).
Revised: Sparse reward is one of the most challenging problems in reinforcement learning (RL).

Original: Hindsight Experience Replay (HER) attempts to address this issue by converting a failure experience to a successful one by relabeling the goals.
Revised: Hindsight Experience Replay (HER) attempts to address this issue by converting a failure experience to a successful one by relabeling the goals.

Original: Despite its effectiveness, HER has limited applicability because it lacks a compact and universal goal representation.
Revised: Despite its effectiveness, HER has limited applicability because it lacks a compact and universal goal representation.

Original: We present Augmenting experienCe via TeacheR’s
Revised: We present Augmenting experienCe via TeacheR’s

Original: adviCE (ACTRCE), an efﬁcient reinforcement learning technique that extends
Revised: adviCE (ACTRCE), an efﬁcient reinforcement learning technique that extends

Original: the
Revised: the

Original: HER framework using natural language as the goal representation.
Revised: HER framework using natural language as the goal representation.

Original: We ﬁrst analyze the differences among goal representation and show that ACTRCE can efﬁciently solve difﬁcult reinforcement learning problems in challenging 3D navigation tasks, whereas HER with non-language goal representation failed to learn.
Revised: We ﬁrst analyze the differences among goal representation and show that ACTRCE can efﬁciently solve difﬁcult reinforcement learning problems in challenging 3D navigation tasks, whereas HER with non-language goal representation failed to learn.

Original: We also show that with language goal representations, the agent can generalize to unseen instructions, and even generalize to instructions with unseen lexicons .
Revised: We also show that with language goal representations, the agent can generalize to unseen instructions, and even generalize to instructions with unseen lexicons .

Original: We further demonstrate it is crucial to use hindsight advice to solve challenging tasks, but we also found that little amount of hindsight advice is sufﬁcient for the learning to take off, showing the practical aspect of the method.
Revised: We further demonstrate it is crucial to use hindsight advice to solve challenging tasks, but we also found that little amount of hindsight advice is sufﬁcient for the learning to take off, showing the practical aspect of the method.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Many impressive deep reinforcement learning (deep RL) applications rely on carefully-crafted reward functions to encourage the desired behavior.
Revised: Many impressive deep reinforcement learning (deep RL) applications rely on carefully-crafted reward functions to encourage the desired behavior.

Original: However, designing a good reward function is nontrivial (Ng et al., 1999), and requires a signiﬁcant engineering effort.
Revised: However, designing a good reward function is nontrivial (Ng et al., 1999), and requires a signiﬁcant engineering effort.

Original: For example, even for the seemingly simple task of stacking Lego blocks, Popov et al.
Revised: For example, even for the seemingly simple task of stacking Lego blocks, Popov et al.

Original: (2017) needed 5 complicated reward terms with different importance weights.
Revised: (2017) needed 5 complicated reward terms with different importance weights.

Original: Moreover, handcrafted reward shaping (Gullapalli & Barto, 1992) can lead to biased learning, which may cause the agent to learn unexpected and undesired behaviors (Clark & Amodei, 2016).
Revised: Moreover, handcrafted reward shaping (Gullapalli & Barto, 1992) can lead to biased learning, which may cause the agent to learn unexpected and undesired behaviors (Clark & Amodei, 2016).

Original: One approach to avoid deﬁning a complicated reward function is to use a sparse and binary reward function, i.e., give only a positive or negative reward at the terminal state, depending on the success of the task.
Revised: One approach to avoid deﬁning a complicated reward function is to use a sparse and binary reward function, i.e., give only a positive or negative reward at the terminal state, depending on the success of the task.

Original: However, the sparse reward makes learning difﬁcult.
Revised: However, the sparse reward makes learning difﬁcult.

Original: Hindsight Experience Replay (HER) (Andrychowicz et al., 2017) attempts to address this issue.
Revised: Hindsight Experience Replay (HER) (Andrychowicz et al., 2017) attempts to address this issue.

Original: The main idea of HER is to utilize failed experiences by substituting with a fake goal in order to convert them to successful experiences.
Revised: The main idea of HER is to utilize failed experiences by substituting with a fake goal in order to convert them to successful experiences.

Original: For their algorithm to work, Andrychowicz et al. made the non-trivial assumption that for every state in the environment, there exists a goal which is achieved in that state.
Revised: For their algorithm to work, Andrychowicz et al. made the non-trivial assumption that for every state in the environment, there exists a goal which is achieved in that state.

Original: As the authors pointed out, this assumption can be trivially satisﬁed by choosing the goal space to be the state space.
Revised: As the authors pointed out, this assumption can be trivially satisﬁed by choosing the goal space to be the state space.

Original: However, representing the goal using the enormous state space is very inefﬁcient and contains much redundant information.
Revised: However, representing the goal using the enormous state space is very inefﬁcient and contains much redundant information.

Original: For example, if we want to ask an agent to avoid collisions while driving where the state is the raw pixel value from the camera, then there can be many states (i.e. frames) that achieve this goal.
Revised: For example, if we want to ask an agent to avoid collisions while driving where the state is the raw pixel value from the camera, then there can be many states (i.e. frames) that achieve this goal.

Original: It is redundant to represent the goal using each state.
Revised: It is redundant to represent the goal using each state.

Original: Therefore, we need a goal representation that is (1) expressive and ﬂexible enough to satisfy the assumption in HER, while also being (2) compact and informative where similar goals are represented using similar features.
Revised: Therefore, we need a goal representation that is (1) expressive and ﬂexible enough to satisfy the assumption in HER, while also being (2) compact and informative where similar goals are represented using similar features.

Original: Natural language representation of goals satisﬁes both requirements.
Revised: Natural language representation of goals satisﬁes both requirements.

Original: First, language can ﬂexibly describe goals across tasks and environments.
Revised: First, language can ﬂexibly describe goals across tasks and environments.

Original: Second, language representation is abstract, hence able to compress any redundant information in the states.
Revised: Second, language representation is abstract, hence able to compress any redundant information in the states.

Original: Recall the previous driving example, for which we can simply describe “avoid collisions" to represent all states that satisfy this goal.
Revised: Recall the previous driving example, for which we can simply describe “avoid collisions" to represent all states that satisfy this goal.

Original: Moreover, the compositional nature of language provides transferable features for generalizing across goals.
Revised: Moreover, the compositional nature of language provides transferable features for generalizing across goals.

Original: In this paper, we combine the HER framework with natural language goal representation, and propose an efﬁcient technique called Augmenting experienCe via TeacheR’s adviCE (ACTRCE; pronounced “actress”) to a broad range of reinforcement learning problems.
Revised: In this paper, we combine the HER framework with natural language goal representation, and propose an efﬁcient technique called Augmenting experienCe via TeacheR’s adviCE (ACTRCE; pronounced “actress”) to a broad range of reinforcement learning problems.

Original: Our method works as follows.
Revised: Our method works as follows.

Original: Whenever an agent ﬁnishes an episode, a teacher gives advice in natural language to the agent based on the episode.
Revised: Whenever an agent ﬁnishes an episode, a teacher gives advice in natural language to the agent based on the episode.

Original: The agent takes the advice to form a new experience with a corresponding reward, alleviating the sparse reward problem.
Revised: The agent takes the advice to form a new experience with a corresponding reward, alleviating the sparse reward problem.

Original: For example, a teacher can describe what the agent has achieved in the episode, and the agent can replace the original goal with the advice and a reward of 1.
Revised: For example, a teacher can describe what the agent has achieved in the episode, and the agent can replace the original goal with the advice and a reward of 1.

Original: We show many beneﬁts brought by language goal representation when combining with hindsight advice.
Revised: We show many beneﬁts brought by language goal representation when combining with hindsight advice.

Original: The agent can efﬁciently solve reinforcement learning problems in challenging 2D and 3D environments; it can generalize to unseen instructions, and even generalize to instruction with unseen lexicons .
Revised: The agent can efﬁciently solve reinforcement learning problems in challenging 2D and 3D environments; it can generalize to unseen instructions, and even generalize to instruction with unseen lexicons .

Original: We further demonstrate it is crucial to use hindsight advice to solve challenging tasks, but we also found that little amount of hindsight advice is sufﬁcient for the learning to take off, showing the practical aspect of the method.
Revised: We further demonstrate it is crucial to use hindsight advice to solve challenging tasks, but we also found that little amount of hindsight advice is sufﬁcient for the learning to take off, showing the practical aspect of the method.

Original: We note that our work is also interesting from a language learning perspective.
Revised: We note that our work is also interesting from a language learning perspective.

Original: Learning to achieve goals described in natural language is part of a class of problem called language grounding (Harnad, 1990), which has recently received increasing interest, as grounding is believed to be necessary for more general understanding of natural language.
Revised: Learning to achieve goals described in natural language is part of a class of problem called language grounding (Harnad, 1990), which has recently received increasing interest, as grounding is believed to be necessary for more general understanding of natural language.

Original: Early attempts to ground language in a simulated physical world (Winograd, 1972; Siskind, 1994) consisted of hard coded rules which could not scale beyond their original domain.
Revised: Early attempts to ground language in a simulated physical world (Winograd, 1972; Siskind, 1994) consisted of hard coded rules which could not scale beyond their original domain.

Original: Recent work has been using reinforcement learning techniques to address this problem (Hermann et al., 2017; Chaplot et al., 2017b).
Revised: Recent work has been using reinforcement learning techniques to address this problem (Hermann et al., 2017; Chaplot et al., 2017b).

Original: Our work combines reinforcement learning and rich language advice, providing an efﬁcient technique for language grounding.
Revised: Our work combines reinforcement learning and rich language advice, providing an efﬁcient technique for language grounding.

Original:  B ACKGROUND 2. R EINFORCEMENT L EARNING AND D EEP Q-N ETWORKS
Revised:  B ACKGROUND 2. R EINFORCEMENT L EARNING AND D EEP Q-N ETWORKS

Original: We ﬁrst review the traditional reinforcement learning setting, where an agent interacts with an inﬁnite-horizon, discounted Markov Decision Process ( S , A , γ, P, r ) .
Revised: We ﬁrst review the traditional reinforcement learning setting, where an agent interacts with an inﬁnite-horizon, discounted Markov Decision Process ( S , A , γ, P, r ) .

Original: Here, S is the state space,
Revised: Here, S is the state space,

Original: A is the action space, γ the discount factor, P the transition model and r is a reward function r : S ×
Revised: A is the action space, γ the discount factor, P the transition model and r is a reward function r : S ×

Original: A → R .
Revised: A → R .

Original: We consider a policy π θ ( a | s ) parameterized by θ .
Revised: We consider a policy π θ ( a | s ) parameterized by θ .

Original: At time t , the agent chooses an action a t ∈ A according to its policy π θ ( a | s t ) where s t ∈ S is the current state.
Revised: At time t , the agent chooses an action a t ∈ A according to its policy π θ ( a | s t ) where s t ∈ S is the current state.

Original: The environment in turn produces a reward r ( s t , a t ) and transitions to the next state s t +

Revised: The environment in turn produces a reward r ( s t , a t ) and transitions to the next state s t +

Original: according to the transition probability P ( s t +1 | s t , a t ) .
Revised: according to the transition probability P ( s t +1 | s t , a t ) .

Original: The goal of the agent is to maximize the expected γ -discounted cumulative return E π [ (cid:80) ∞ t =

Revised: The goal of the agent is to maximize the expected γ -discounted cumulative return E π [ (cid:80) ∞ t =

Original: γ t r ( s t , a t )] with respect to the policy parameters θ .
Revised: γ t r ( s t , a t )] with respect to the policy parameters θ .

Original: The action-value function is denoted as Q π ( s t , a t ) =
Revised: The action-value function is denoted as Q π ( s t , a t ) =

Original: E π [ (cid:80) ∞
Revised: E π [ (cid:80) ∞

Original: i =
Revised: i =

Original: t γ i − t r
Revised: t γ i − t r

Original: ( s i , a i ) | s = s t , a = a t ] , which is the expected discounted sum of rewards obtained by performing an action a at state s and following the policy π thereafter.
Revised: ( s i , a i ) | s = s t , a = a t ] , which is the expected discounted sum of rewards obtained by performing an action a at state s and following the policy π thereafter.

Original: We denote π ∗ as the optimal policy such that Q π ∗ ( s, a ) ≥ Q π ( s, a ) for every s ∈ S , a ∈ A and policy π .
Revised: We denote π ∗ as the optimal policy such that Q π ∗ ( s, a ) ≥ Q π ( s, a ) for every s ∈ S , a ∈ A and policy π .

Original: The optimal Q-function Q ∗ =
Revised: The optimal Q-function Q ∗ =

Original: Q π ∗ satisﬁes the Bellman equation :
Revised: Q π ∗ satisﬁes the Bellman equation :

Original: [Equation]
Revised: [Equation]

Original: Q-learning (Sutton & Barto, 2018) is an off-policy, model-free RL algorithm that is based on the Bellman equation.
Revised: Q-learning (Sutton & Barto, 2018) is an off-policy, model-free RL algorithm that is based on the Bellman equation.

Original: The algorithm uses semi-gradient descent (Sutton & Barto, 2018) to minimize the squared Temporal Difference (TD) error:
Revised: The algorithm uses semi-gradient descent (Sutton & Barto, 2018) to minimize the squared Temporal Difference (TD) error:

Original: L = E ( Q ( s t , a t ) − y t ) 2 , where the y t =
Revised: L = E ( Q ( s t , a t ) − y t ) 2 , where the y t =

Original: r t + γ max a (cid:48) ∈A Q ( s t +1 , a (cid:48) ) is the TD target .
Revised: r t + γ max a (cid:48) ∈A Q ( s t +1 , a (cid:48) ) is the TD target .

Original: Deep Q-Network (Mnih et al., 2013) builds on the Q-learning algorithm and uses a neural network to approximate Q ∗ .
Revised: Deep Q-Network (Mnih et al., 2013) builds on the Q-learning algorithm and uses a neural network to approximate Q ∗ .

Original: It also uses a replay buffer as well as a target network to improve training stability.
Revised: It also uses a replay buffer as well as a target network to improve training stability.

Original: 2. G OAL O RIENTED R EINFORCEMENT L EARNING
Revised: 2. G OAL O RIENTED R EINFORCEMENT L EARNING

Original: We review the goal-oriented reinforcement learning framework following Schaul et al. (2015).
Revised: We review the goal-oriented reinforcement learning framework following Schaul et al. (2015).

Original: We augment the previously deﬁned inﬁnite-horizon, discounted Markov Decision Process with a goal space G .
Revised: We augment the previously deﬁned inﬁnite-horizon, discounted Markov Decision Process with a goal space G .

Original: A goal g is chosen from G and stays ﬁxed for each episode.
Revised: A goal g is chosen from G and stays ﬁxed for each episode.

Original: The goal induces a reward function r g : S ×
Revised: The goal induces a reward function r g : S ×

Original: A → R , that assigns reward to a state conditioned on the given goal.
Revised: A → R , that assigns reward to a state conditioned on the given goal.

Original: At time step t , the agent π ( a t | s t , g ) chooses an action conditioned on the current state s t , as well as the current goal g .
Revised: At time step t , the agent π ( a t | s t , g ) chooses an action conditioned on the current state s t , as well as the current goal g .

Original: The agent’s objective is to maximize the expected discounted cumulative return given the goal, i.e., E π [ (cid:80) ∞ t =

Revised: The agent’s objective is to maximize the expected discounted cumulative return given the goal, i.e., E π [ (cid:80) ∞ t =

Original: γ t r ( s t , a t , g )] .
Revised: γ t r ( s t , a t , g )] .

Original: [Figure]
Revised: [Figure]

Original: 2. H INDSIGHT E XPERIENCE
Revised: 2. H INDSIGHT E XPERIENCE

Original: [Figure]
Revised: [Figure]

Original: We follow Andrychowicz et al.
Revised: We follow Andrychowicz et al.

Original: and consider a speciﬁc family of goal conditioned reward functions, where the reward is either 0 or 1 , depending on the resulting state, r g ( s t , a t )
Revised: and consider a speciﬁc family of goal conditioned reward functions, where the reward is either 0 or 1 , depending on the resulting state, r g ( s t , a t )

Original: =
Revised: =

Original: f g ( s t +1 ) , where f g : S → { 0 , 1 } .
Revised: f g ( s t +1 ) , where f g : S → { 0 , 1 } .

Original: In many scenarios, the function f g acts as a predicate that determines whether the agent is successful or not according to g , and only assigns positive reward at the terminal state.
Revised: In many scenarios, the function f g acts as a predicate that determines whether the agent is successful or not according to g , and only assigns positive reward at the terminal state.

Original: Hence the reward function is very sparse and makes it difﬁcult for an agent to learn.
Revised: Hence the reward function is very sparse and makes it difﬁcult for an agent to learn.

Original: HER proposes a solution to this problem.
Revised: HER proposes a solution to this problem.

Original: The agent ﬁrst collects an episode of experiences s 0 , a 0 , r 1 , ..., s T − 1 , a T − 1 , r T , s T under the goal g , where r T =
Revised: The agent ﬁrst collects an episode of experiences s 0 , a 0 , r 1 , ..., s T − 1 , a T − 1 , r T , s T under the goal g , where r T =

Original: r g ( s T − 1 , a T − 1 ) is the terminal reward.
Revised: r g ( s T − 1 , a T − 1 ) is the terminal reward.

Original: If r
Revised: If r

Original: T is zero, one can replace g with g (cid:

Revised: T is zero, one can replace g with g (cid:

Original: ) ∈
Revised: ) ∈

Original: G such that r g (cid:48) ( s T − 1 , a T − 1 ) = 1 .
Revised: G such that r g (cid:48) ( s T − 1 , a T − 1 ) = 1 .

Original: With an off-policy algorithm, one is able to learn from the goal transformed experience.
Revised: With an off-policy algorithm, one is able to learn from the goal transformed experience.

Original: In order to ﬂexibly relabel with a desirable goal, the authors assume that there exists a representation map from the state space to the goal space T : S → G , so that for each s ∈ S , the corresponding goal representation satisﬁes the predicate f g ( T ( s ))
Revised: In order to ﬂexibly relabel with a desirable goal, the authors assume that there exists a representation map from the state space to the goal space T : S → G , so that for each s ∈ S , the corresponding goal representation satisﬁes the predicate f g ( T ( s ))

Original: = 1 .
Revised: = 1 .

Original: However, such mapping is not simple to construct.
Revised: However, such mapping is not simple to construct.

Original: Although the author pointed out a trivial construction can be done by taking G = S and f g ( s ) =
Revised: Although the author pointed out a trivial construction can be done by taking G = S and f g ( s ) =

Original: [ g = s ] , such goal representation is redundant and uninformative, and largely limits the applicability of the algorithm.
Revised: [ g = s ] , such goal representation is redundant and uninformative, and largely limits the applicability of the algorithm.

Original: Consider an example where the state is the ﬁrst-person raw pixel observation in a 3-D environment, and the goal is to walk towards a particular object.
Revised: Consider an example where the state is the ﬁrst-person raw pixel observation in a 3-D environment, and the goal is to walk towards a particular object.

Original: There are many possible directions to approach the object from, which results in many different possible states that satisfy the goal.
Revised: There are many possible directions to approach the object from, which results in many different possible states that satisfy the goal.

Original: However, no subset of the raw pixel observation can abstractly represent the goal.
Revised: However, no subset of the raw pixel observation can abstractly represent the goal.

Original:  A UGMENTING E XPERIENCE VIA T EACHER ’ S A DVICE (ACTRCE)
Revised:  A UGMENTING E XPERIENCE VIA T EACHER ’ S A DVICE (ACTRCE)

Original: For a goal oriented MDP, {S , G , A , γ, P, r g } , and a parameterized agent π ( a t | s t , g ) , our objective is to maximize the expected discounted cumulative return, i.e., E π [ (cid:80) ∞ t =

Revised: For a goal oriented MDP, {S , G , A , γ, P, r g } , and a parameterized agent π ( a t | s t , g ) , our objective is to maximize the expected discounted cumulative return, i.e., E π [ (cid:80) ∞ t =

Original: γ t r g ( s t , a t )] for each g ∈ G .
Revised: γ t r g ( s t , a t )] for each g ∈ G .

Original: Following HER, we assume there exists one or more more goal representation mappings T : S → G , which describe what goal is satisﬁed at each state.
Revised: Following HER, we assume there exists one or more more goal representation mappings T : S → G , which describe what goal is satisﬁed at each state.

Original: For example, in the original HER paper, the mapping T is simply a subspace projection from the state.
Revised: For example, in the original HER paper, the mapping T is simply a subspace projection from the state.

Original: Such representation can work well for their robotic tasks, where subset of the state (i.e., the coordinates of objects and body parts) can reasonably well represent the goal.
Revised: Such representation can work well for their robotic tasks, where subset of the state (i.e., the coordinates of objects and body parts) can reasonably well represent the goal.

Original: However, in general, such mapping can not abstract meaningful information about the goal, and will cause redundancy.
Revised: However, in general, such mapping can not abstract meaningful information about the goal, and will cause redundancy.

Original: Language provides an abstract representation for a goal and hence reduces redundancy in representation.
Revised: Language provides an abstract representation for a goal and hence reduces redundancy in representation.

Original: This motivates our proposal to use natural language for representing the goal space.
Revised: This motivates our proposal to use natural language for representing the goal space.

Original: Concretely, for each state s ∈ S , we deﬁne T as a teacher that gives an advice T ( s ) , a natural language description of the goal that is achieved in the state s .
Revised: Concretely, for each state s ∈ S , we deﬁne T as a teacher that gives an advice T ( s ) , a natural language description of the goal that is achieved in the state s .

Original: To implement such a teacher, one can ask a human to look at a state and describe the goal in words, or generate a goal description automatically, as part of a narration or accessibility feature in a game.
Revised: To implement such a teacher, one can ask a human to look at a state and describe the goal in words, or generate a goal description automatically, as part of a narration or accessibility feature in a game.

Original: Given T , we can convert a failure trajectory to a successful one by relabeling the original goal with the teacher advice.
Revised: Given T , we can convert a failure trajectory to a successful one by relabeling the original goal with the teacher advice.

Original: To illustrate, say the original goal for the episode is "Go to the armor", but at this particular time step, the agent has reached the blue torch instead.
Revised: To illustrate, say the original goal for the episode is "Go to the armor", but at this particular time step, the agent has reached the blue torch instead.

Original: The teacher then tells the agent that the goal that reﬂects the current state is "Go to blue torch".
Revised: The teacher then tells the agent that the goal that reﬂects the current state is "Go to blue torch".

Original: The agent can hence take the advice and use it as a positive experience.
Revised: The agent can hence take the advice and use it as a positive experience.

Original: Besides positive reward signals, we also ﬁnd that negative experience is necessary for training.
Revised: Besides positive reward signals, we also ﬁnd that negative experience is necessary for training.

Original: Hence, in addition to a teacher who describes the achieved goal with a positive reward, we also introduce a teacher who gives negative reward with a description of a goal that has not been achieved.
Revised: Hence, in addition to a teacher who describes the achieved goal with a positive reward, we also introduce a teacher who gives negative reward with a description of a goal that has not been achieved.

Original: We further consider the scenario where there is more than one goal that can be satisﬁed in a state, e.g.,
Revised: We further consider the scenario where there is more than one goal that can be satisﬁed in a state, e.g.,

Original: algorithm
Revised: algorithm

Original: • A language goal space G
Revised: • A language goal space G

Original: a desired goal space
Revised: a desired goal space

Original: the state that is described as “Reach a blue torch" may also be described as “Reach the largest blue object".
Revised: the state that is described as “Reach a blue torch" may also be described as “Reach the largest blue object".

Original: Each different description of the goal corresponds to a different teacher (see more discussions and experiments on variations of teachers in Appendix E).
Revised: Each different description of the goal corresponds to a different teacher (see more discussions and experiments on variations of teachers in Appendix E).

Original: Therefore, in general, there is a group of teachers of different kinds, each giving different advice.
Revised: Therefore, in general, there is a group of teachers of different kinds, each giving different advice.

Original: We then relabel the original goal with each advice and corresponding positive or negative reward, and augment the replay buffer with these trajectories.
Revised: We then relabel the original goal with each advice and corresponding positive or negative reward, and augment the replay buffer with these trajectories.

Original: Algorithm 1 describes our proposed approach formally.
Revised: Algorithm 1 describes our proposed approach formally.

Original: Note that in the above formulation, we assume a MDP setting, and let the teacher give advice based solely on the terminal state g (cid:48)
Revised: Note that in the above formulation, we assume a MDP setting, and let the teacher give advice based solely on the terminal state g (cid:48)

Original: =
Revised: =

Original: T ( s T ) .
Revised: T ( s T ) .

Original: By contrast, in the POMDP setting, we ask the teacher to give advice based on the history of states and actions during the episode, i.e., g (cid:48) =
Revised: By contrast, in the POMDP setting, we ask the teacher to give advice based on the history of states and actions during the episode, i.e., g (cid:48) =

Original: T ( { s 0 , a 0 , . . .
Revised: T ( { s 0 , a 0 , . . .

Original: , s T } ) .
Revised: , s T } ) .

Original: 3. L ANGUAGE REPRESENTATION
Revised: 3. L ANGUAGE REPRESENTATION

Original: There remains a question of how we deal with the natural language goal representation, a sequence of discrete symbols.
Revised: There remains a question of how we deal with the natural language goal representation, a sequence of discrete symbols.

Original: In this paper, we explore two standard ways to convert a language goal into a continuous vector representation for neural network training.
Revised: In this paper, we explore two standard ways to convert a language goal into a continuous vector representation for neural network training.

Original: One way is to represent each word as a one-hot vector, and use a recurrent neural network (RNN) to sequentially embed each word into its hidden state.
Revised: One way is to represent each word as a one-hot vector, and use a recurrent neural network (RNN) to sequentially embed each word into its hidden state.

Original: We can then obtain some function of its hidden states (e.g., last hidden state, an attention over all hidden states) as the representation of the goal.
Revised: We can then obtain some function of its hidden states (e.g., last hidden state, an attention over all hidden states) as the representation of the goal.

Original: The other way is to use a pre-trained language component obtained by other approaches and dataset (e.g. Mikolov et al.
Revised: The other way is to use a pre-trained language component obtained by other approaches and dataset (e.g. Mikolov et al.

Original: (2013)) to represent the given language instructions.
Revised: (2013)) to represent the given language instructions.

Original: Since a pre-trained sentence embedding deﬁnes a reasonable similarity metric among sentences, one can expect the agent to understand unseen lexicons that are closely related to the language goals in training.
Revised: Since a pre-trained sentence embedding deﬁnes a reasonable similarity metric among sentences, one can expect the agent to understand unseen lexicons that are closely related to the language goals in training.

Original: This allows the agent to gain better natural language understanding, and potentially be more robust to the language advice it gets from teachers (advice from humans can be quite noisy).
Revised: This allows the agent to gain better natural language understanding, and potentially be more robust to the language advice it gets from teachers (advice from humans can be quite noisy).

Original: To integrate language representation of goals into the model, we design the architecture as follows.
Revised: To integrate language representation of goals into the model, we design the architecture as follows.

Original: The architecture consists of 3 modules.
Revised: The architecture consists of 3 modules.

Original: One language component that takes an instruction and convert it into a continuous vector, and we apply an attention vector.
Revised: One language component that takes an instruction and convert it into a continuous vector, and we apply an attention vector.

Original: The second component processes the observation to obtain a image representation using convolution neural networks.
Revised: The second component processes the observation to obtain a image representation using convolution neural networks.

Original: We then use gated attention from Chaplot et al.
Revised: We then use gated attention from Chaplot et al.

Original: (2017b)
Revised: (2017b)

Original: to fuse goal information and the observation.
Revised: to fuse goal information and the observation.

Original: The third component then take the result of the fused representation and compute the output value.
Revised: The third component then take the result of the fused representation and compute the output value.

Original: Figure.(1) shows the computational graph.
Revised: Figure.(1) shows the computational graph.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In the following experiments, we demonstrate the effectiveness of the proposed method and discuss qualitative differences from the baseline.
Revised: In the following experiments, we demonstrate the effectiveness of the proposed method and discuss qualitative differences from the baseline.

Original: We ﬁrst describe the experimental set up with ourenvironments, KrazyGrid World and ViZDoom.
Revised: We ﬁrst describe the experimental set up with ourenvironments, KrazyGrid World and ViZDoom.

Original: In the subsequent sections, we investigate:
Revised: In the subsequent sections, we investigate:

Original: [Figure]
Revised: [Figure]

Original: A comprehensive comparison between goal representations in hindsight advice, generalization, and semantic similarities.
Revised: A comprehensive comparison between goal representations in hindsight advice, generalization, and semantic similarities.

Original: We compare different goal representations: a naive one hot vector for each instruction, a Gated Recurrent Unit (GRU) (Chung et al., 2014) that embeds the discrete language tokens, and a pre-trained word embedding.
Revised: We compare different goal representations: a naive one hot vector for each instruction, a Gated Recurrent Unit (GRU) (Chung et al., 2014) that embeds the discrete language tokens, and a pre-trained word embedding.

Original: We show that as the number of instructions increases, the one hot approach does not scale as well as the GRU and pre-trained embedding.
Revised: We show that as the number of instructions increases, the one hot approach does not scale as well as the GRU and pre-trained embedding.

Original: From visualizing the representation, we can see that structures emerge from GRU and pre-trained embeddings.
Revised: From visualizing the representation, we can see that structures emerge from GRU and pre-trained embeddings.

Original: We also demonstrate that pre-trained embedding representation can generalize to goals containing out of training vocabulary words.
Revised: We also demonstrate that pre-trained embedding representation can generalize to goals containing out of training vocabulary words.

Original: Does the hindsight language advice help with learning?
Revised: Does the hindsight language advice help with learning?

Original: How much advice do we need?
Revised: How much advice do we need?

Original: We show signiﬁcant improvement in sample efﬁciency by using advice from teachers.
Revised: We show signiﬁcant improvement in sample efﬁciency by using advice from teachers.

Original: In many of the challenging tasks, the agent cannot learn at all without teachers’ advice.
Revised: In many of the challenging tasks, the agent cannot learn at all without teachers’ advice.

Original: We also show that signiﬁcant improvement can be achieved even if we provide limited amount of teachers’ advice, showing low burden of the method in practice.
Revised: We also show that signiﬁcant improvement can be achieved even if we provide limited amount of teachers’ advice, showing low burden of the method in practice.

Original: 4. E NVIRONMENTS AND T RAINING
Revised: 4. E NVIRONMENTS AND T RAINING

Original: We tested our method in a 2D grid world (KrazyGrid World (Stadie et al., 2018)), as well as a 3D environment (ViZDoom (Chaplot et al., 2017b)).
Revised: We tested our method in a 2D grid world (KrazyGrid World (Stadie et al., 2018)), as well as a 3D environment (ViZDoom (Chaplot et al., 2017b)).

Original: We describe the environment and our modiﬁcations below.
Revised: We describe the environment and our modiﬁcations below.

Original: KrazyGrid World (KGW) (Stadie et al., 2018): KrazyGrid World is a 2D grid environment.
Revised: KrazyGrid World (KGW) (Stadie et al., 2018): KrazyGrid World is a 2D grid environment.

Original: For our experiments we chose to use 4 functionality of tiles: Goal, Lava, Normal, and Agent.
Revised: For our experiments we chose to use 4 functionality of tiles: Goal, Lava, Normal, and Agent.

Original: We added an additional colour attribute in: Red, Blue, Green.
Revised: We added an additional colour attribute in: Red, Blue, Green.

Original: The desired goals in this environment is to reach goals of different colours.
Revised: The desired goals in this environment is to reach goals of different colours.

Original: Appendix A lists all language goals and additional environment details.
Revised: Appendix A lists all language goals and additional environment details.

Original: ViZDoom (Kempka et al., 2016; Chaplot et al., 2017b):
Revised: ViZDoom (Kempka et al., 2016; Chaplot et al., 2017b):

Original: The 3D learning environment is based on the ﬁrst person shooter game Doom.
Revised: The 3D learning environment is based on the ﬁrst person shooter game Doom.

Original: The state is a raw pixel image from ﬁrst person perspective of a room containing several random doom objects of various types, colours, and sizes.
Revised: The state is a raw pixel image from ﬁrst person perspective of a room containing several random doom objects of various types, colours, and sizes.

Original: The goal for the episode is a natural language instruction in the form "Go to the [ target attribute ] [ object ]", such as " Go to the green torch ".
Revised: The goal for the episode is a natural language instruction in the form "Go to the [ target attribute ] [ object ]", such as " Go to the green torch ".

Original: See Appendix B for list of instructions and more description of ViZDoom.
Revised: See Appendix B for list of instructions and more description of ViZDoom.

Original: Compositions of goals.
Revised: Compositions of goals.

Original: We refer language goals introduced in KGW and ViZDoom sections as singleton tasks.
Revised: We refer language goals introduced in KGW and ViZDoom sections as singleton tasks.

Original: We further consider a set of more challenging tasks – tasks that are composed of singleton tasks.
Revised: We further consider a set of more challenging tasks – tasks that are composed of singleton tasks.

Original: Given two singleton tasks A , B , we take composition function “and" and “or" to form two new tasks “ A and B ", and “ A or B ".
Revised: Given two singleton tasks A , B , we take composition function “and" and “or" to form two new tasks “ A and B ", and “ A or B ".

Original: The task “ A and B " is considered as complete when the agent completes both A and B within an episode, and “ A or B " is considered as complete when one of the task is achieved.
Revised: The task “ A and B " is considered as complete when the agent completes both A and B within an episode, and “ A or B " is considered as complete when one of the task is achieved.

Original: In our experiments, we consider all combinations of singleton tasks with both compositions “and" and “or" for KGW, and “and" for ViZDoom.
Revised: In our experiments, we consider all combinations of singleton tasks with both compositions “and" and “or" for KGW, and “and" for ViZDoom.

Original: Training details.
Revised: Training details.

Original: We used the DQN algorithm as the base reinforcement learning algorithm in both environments.
Revised: We used the DQN algorithm as the base reinforcement learning algorithm in both environments.

Original: A detailed architecture description for all of our experienments can be found in Appendix C. We carried out multi environment training as follows.
Revised: A detailed architecture description for all of our experienments can be found in Appendix C. We carried out multi environment training as follows.

Original: First we sample 16 different environments.
Revised: First we sample 16 different environments.

Original: We collected data from all 16 environments.
Revised: We collected data from all 16 environments.

Original: We updated the agents with an average gradient.
Revised: We updated the agents with an average gradient.

Original: We then resampled 16 environments.
Revised: We then resampled 16 environments.

Original: Further training details can be found in Appendix D.
Revised: Further training details can be found in Appendix D.

Original: [Figure]
Revised: [Figure]

Original: 4. I NVESTIGATING DIFFERENT TYPES OF GOAL REPRESENTATION
Revised: 4. I NVESTIGATING DIFFERENT TYPES OF GOAL REPRESENTATION

Original: In this section, we investigate the hypothesis that using language for goal representation helps learning in various aspects.
Revised: In this section, we investigate the hypothesis that using language for goal representation helps learning in various aspects.

Original: We compare language-based goal representations and a non-language goal representation.
Revised: We compare language-based goal representations and a non-language goal representation.

Original: In summary, we show that when the difﬁculty of the tasks increases, language goal representations became more effective in providing learning signals.
Revised: In summary, we show that when the difﬁculty of the tasks increases, language goal representations became more effective in providing learning signals.

Original: We also show how one can use language goal representation to generalize to unseen goals in training, whereas non-language goal representation could not.
Revised: We also show how one can use language goal representation to generalize to unseen goals in training, whereas non-language goal representation could not.

Original: With a pre-trained sentence embedding, we can even generalize to instructions consisting of unseen lexicons , showing the robustness of the language goal representation.
Revised: With a pre-trained sentence embedding, we can even generalize to instructions consisting of unseen lexicons , showing the robustness of the language goal representation.

Original: Three goal representations are described as follows,
Revised: Three goal representations are described as follows,

Original: Language Sentence Representation with GRUs : For each instruction, we represent each word as a one hot vector, and sequentially embed words into a GRU.
Revised: Language Sentence Representation with GRUs : For each instruction, we represent each word as a one hot vector, and sequentially embed words into a GRU.

Original: We use take the last hidden state representation of the GRU for representing the instruction.
Revised: We use take the last hidden state representation of the GRU for representing the instruction.

Original: We use the same GRU architecture from
Revised: We use the same GRU architecture from

Original: Chaplot et al.
Revised: Chaplot et al.

Original: (2017b), with hidden size 256.
Revised: (2017b), with hidden size 256.

Original: Pre-trained Language Sentence Representation : We also consider models where the language component is pre-trained.
Revised: Pre-trained Language Sentence Representation : We also consider models where the language component is pre-trained.

Original: We use InferLite (Kiros & Chan, 2018) as the pre-trained sentence representation and hold the parameters ﬁxed during training.
Revised: We use InferLite (Kiros & Chan, 2018) as the pre-trained sentence representation and hold the parameters ﬁxed during training.

Original: InferLite is a lightweight form of generic sentence encoder trained to perform natural language inference (Bowman et al., 2015;
Revised: InferLite is a lightweight form of generic sentence encoder trained to perform natural language inference (Bowman et al., 2015;

Original: Williams et al., 2018) resembling InferSent (Conneau et al., 2017) but without the use of RNNs.
Revised: Williams et al., 2018) resembling InferSent (Conneau et al., 2017) but without the use of RNNs.

Original: The original sentence embedding vector is of dimension 4096, and we use a learned linear layer to project it down to 256 for keeping every other part of the model same as the other two.
Revised: The original sentence embedding vector is of dimension 4096, and we use a learned linear layer to project it down to 256 for keeping every other part of the model same as the other two.

Original: One-Hot Representation : The non-language baseline represents each instruction as a one hot vector, which does not contain any language structure.
Revised: One-Hot Representation : The non-language baseline represents each instruction as a one hot vector, which does not contain any language structure.

Original: We embed this one-hot goal representation to a vector with the same number of dimensions as the GRU representation.
Revised: We embed this one-hot goal representation to a vector with the same number of dimensions as the GRU representation.

Original: This embedding is learned independently for each goal encountered.
Revised: This embedding is learned independently for each goal encountered.

Original: 4.2. H OW DOES EACH GOAL REPRESENTATION PERFORM ?
Revised: 4.2. H OW DOES EACH GOAL REPRESENTATION PERFORM ?

Original: We use all three goal representations with hindsight advice for learning both singleton tasks and compositional tasks of ViZDoom environment.
Revised: We use all three goal representations with hindsight advice for learning both singleton tasks and compositional tasks of ViZDoom environment.

Original: The singleton tasks consists of 7 objects, and the compositional tasks consists of 5 objects.
Revised: The singleton tasks consists of 7 objects, and the compositional tasks consists of 5 objects.

Original: The result of comparisons is plotted in Figure 2.
Revised: The result of comparisons is plotted in Figure 2.

Original: We see that in an easier benchmark—singleton—tasks, agents using one-hot goal representations are still able to learn as quickly as agents using the other two goal representations.
Revised: We see that in an easier benchmark—singleton—tasks, agents using one-hot goal representations are still able to learn as quickly as agents using the other two goal representations.

Original: However, in a much more challenging benchmark— compositional–tasks, the one-hot representations can only achieve a 24% success rates, whereas ACTRCE with language representations can achieve 97% .
Revised: However, in a much more challenging benchmark— compositional–tasks, the one-hot representations can only achieve a 24% success rates, whereas ACTRCE with language representations can achieve 97% .

Original: Since one hot representations represent each goal independently, the agent is unable to generalize to any new unseen instructions.
Revised: Since one hot representations represent each goal independently, the agent is unable to generalize to any new unseen instructions.

Original: With language representation, we can easily generalize to unseen instructions that consists of seen lexicons .
Revised: With language representation, we can easily generalize to unseen instructions that consists of seen lexicons .

Original: We reported the generalization results in Table 1, under “ZSL" (zero-shot learning).
Revised: We reported the generalization results in Table 1, under “ZSL" (zero-shot learning).

Original: Remarkably, with GRU language goal representation, the agent is able to generalize to unseen instructions 83% of the time, showing great generalization ability of language goal representations.
Revised: Remarkably, with GRU language goal representation, the agent is able to generalize to unseen instructions 83% of the time, showing great generalization ability of language goal representations.

Original: 4.2. V ISUALIZATION OF LEARNED EMBEDDINGS
Revised: 4.2. V ISUALIZATION OF LEARNED EMBEDDINGS

Original: We carried out a visualization analysis to see the statistical relations between the learned embeddings of goals.
Revised: We carried out a visualization analysis to see the statistical relations between the learned embeddings of goals.

Original: We took the model trained in the singleton VizDoom benchmark, where the agent was able
Revised: We took the model trained in the singleton VizDoom benchmark, where the agent was able

Original: [Table]
Revised: [Table]

Original: to learn with all three goal representations.
Revised: to learn with all three goal representations.

Original: However
Revised: However

Original: [Table]
Revised: [Table]

Original: we ﬁnd there are huge differences in learned embeddings.
Revised: we ﬁnd there are huge differences in learned embeddings.

Original: For each goal instruction, we extract the corresponding learned embedding for all three goal representation, which is the continuous vector before the attention layer (recall Fig.(1)), all of size 256.
Revised: For each goal instruction, we extract the corresponding learned embedding for all three goal representation, which is the continuous vector before the attention layer (recall Fig.(1)), all of size 256.

Original: We then calculated the correlation matrix for each goal representations.
Revised: We then calculated the correlation matrix for each goal representations.

Original: We plot all three correlation matrices in Figure.(3).
Revised: We plot all three correlation matrices in Figure.(3).

Original: The rows and columns are grouped by object type, then colour and size.
Revised: The rows and columns are grouped by object type, then colour and size.

Original: We found that GRU and InferLite embeddings have very similar block-like structure, due to clustering of colours and shapes, while each one-hot goal embeddings share almost no correlation between each other.
Revised: We found that GRU and InferLite embeddings have very similar block-like structure, due to clustering of colours and shapes, while each one-hot goal embeddings share almost no correlation between each other.

Original: We further performed t-SNE (Maaten & Hinton, 2008) embeddings and observe meaningful clustering with language goal representations and sporadic embeddings for one-hot goals.
Revised: We further performed t-SNE (Maaten & Hinton, 2008) embeddings and observe meaningful clustering with language goal representations and sporadic embeddings for one-hot goals.

Original: See more details in Appendix J.1.
Revised: See more details in Appendix K.1.

Original: 4.2. G ENERALIZING TO UNSEEN LEXICONS WITH PRE - TRAINED WORD EMBEDDING
Revised: 4.2. G ENERALIZING TO UNSEEN LEXICONS WITH PRE - TRAINED WORD EMBEDDING

Original: We now show how to use pre-trained embeddings to allow our model to generalize to unseen lexicons at test time through representation transfer, similar in spirit to DeViSE (Frome et al., 2013) that was used for image classiﬁcation with unseen classes.
Revised: We now show how to use pre-trained embeddings to allow our model to generalize to unseen lexicons at test time through representation transfer, similar in spirit to DeViSE (Frome et al., 2013) that was used for image classiﬁcation with unseen classes.

Original: We took the agent trained in singleton tasks with
Revised: We took the agent trained in singleton tasks with

Original: InferLite goal representations, and replaced one word in each original instruction with its synonym 1 .
Revised: InferLite goal representations, and replaced one word in each original instruction with its synonym 1 .

Original: We evaluate the performance of the agent on these new instructions that contain unseen lexicons.
Revised: We evaluate the performance of the agent on these new instructions that contain unseen lexicons.

Original: The results are shown in Tab.(3).
Revised: The results are shown in Tab.(3).

Original: We ﬁnd the agent is able to achieve tasks above 66% of time.
Revised: We ﬁnd the agent is able to achieve tasks above 66% of time.

Original: The ability of understanding sentences of similar meanings become useful when one implements the method with advice comes from humans.
Revised: The ability of understanding sentences of similar meanings become useful when one implements the method with advice comes from humans.

Original: Since humans can describe the same meaning in many different ways, understanding synonyms can improve the robustness of learning in general noisy settings.
Revised: Since humans can describe the same meaning in many different ways, understanding synonyms can improve the robustness of learning in general noisy settings.

Original: [Table]
Revised: [Table]

Original: 4. D O WE NEED HINDSIGHT ADVICE ?
Revised: 4. D O WE NEED HINDSIGHT ADVICE ?

Original: In the previous section, we demonstrate the effectiveness of language goal representation from various perspectives.
Revised: In the previous section, we demonstrate the effectiveness of language goal representation from various perspectives.

Original: In this section, we show how hindsight advice plays an important role in learning.
Revised: In this section, we show how hindsight advice plays an important role in learning.

Original: We compared our method (denoted as “ACTRCE") to the algorithm DQN, the base algorithm without any hindsight language advice.
Revised: We compared our method (denoted as “ACTRCE") to the algorithm DQN, the base algorithm without any hindsight language advice.

Original: We show that without hindsight advice, DQN is not able to learn many challenging tasks.
Revised: We show that without hindsight advice, DQN is not able to learn many challenging tasks.

Original: However, we also found that little advice (1%) is sufﬁcient for learning to take off, showing the practicality of our method.
Revised: However, we also found that little advice (1%) is sufﬁcient for learning to take off, showing the practicality of our method.

Original: For language representation, we chose to use recurrent neural networks for embedding language goals, for both our method and the baseline method.
Revised: For language representation, we chose to use recurrent neural networks for embedding language goals, for both our method and the baseline method.

Original: 4.3. C OMPARISON OF ACTRCE TO DQN
Revised: 4.3. C OMPARISON OF ACTRCE TO DQN

Original: Singleton tasks.
Revised: Singleton tasks.

Original: For experiments on KGW, we tried two different kind of grids, one with 3 lavas and the other with 6 lavas, and both with 3 goals of different colours.
Revised: For experiments on KGW, we tried two different kind of grids, one with 3 lavas and the other with 6 lavas, and both with 3 goals of different colours.

Original: Fig.
Revised: Fig.

Original: 4 (a) and (b) show the
Revised: 4 (a) and (b) show the

Original: [Figure]
Revised: [Figure]

Original: shown in blue curve, which failed to learn anything at all.
Revised: shown in blue curve, which failed to learn anything at all.

Original: Our method (shown in green) quickly learned and achieved good results on both environments (around 80% success rate).
Revised: Our method (shown in green) quickly learned and achieved good results on both environments (around 80% success rate).

Original: In ViZDoom experiments, we trained our agent inconﬁgurations: 5 objects in (1) easy and (2) hard mode, and (3) 7-objects in the hard mode with 50% larger room size.
Revised: In ViZDoom experiments, we trained our agent inconﬁgurations: 5 objects in (1) easy and (2) hard mode, and (3) 7-objects in the hard mode with 50% larger room size.

Original: While the DQN and A3C implementation from Chaplot et al.
Revised: While the DQN and A3C implementation from Chaplot et al.

Original: (2017a) were able to learn in the 5 objects task (see Appendix G for details), we found that in the more difﬁcult 7-objects environment, only the agent trained with ACTRCE was able learn at all.
Revised: (2017a) were able to learn in the 5 objects task (see Appendix G for details), we found that in the more difﬁcult 7-objects environment, only the agent trained with ACTRCE was able learn at all.

Original: Figure 4 (e) illustrates the training instruction average success rate (over 100 episode chunks) vs. the number of frames set, when using ACTRCE versus DQN baseline.
Revised: Figure 4 (e) illustrates the training instruction average success rate (over 100 episode chunks) vs. the number of frames set, when using ACTRCE versus DQN baseline.

Original: Table 1 summarizes the agent’s
Revised: Table 1 summarizes the agent’s

Original: Multitask and Zero-Shot generalization performance.
Revised: Multitask and Zero-Shot generalization performance.

Original: Compositional tasks.
Revised: Compositional tasks.

Original: In KGW, we carried out the experiments in a grid with 3 goals and 3 lavas.
Revised: In KGW, we carried out the experiments in a grid with 3 goals and 3 lavas.

Original: A comparison of average success rates over all goals of ACTRCE, and baseline DQN is shown in in Fig.
Revised: A comparison of average success rates over all goals of ACTRCE, and baseline DQN is shown in in Fig.

Original: 4 (C).
Revised: 4 (C).

Original: The shaded area represents the standard deviation over 2 random seeds.
Revised: The shaded area represents the standard deviation over 2 random seeds.

Original: We observed that the baseline still could not learn the task while ACTRCE learned efﬁciently and achieved good performance in this highly challenging environment.
Revised: We observed that the baseline still could not learn the task while ACTRCE learned efﬁciently and achieved good performance in this highly challenging environment.

Original: In ViZDoom, we chose to use 5 objects in easy mode for the compositional task, where all 5 objects appeared in front of the agent at the beginning of the episode.
Revised: In ViZDoom, we chose to use 5 objects in easy mode for the compositional task, where all 5 objects appeared in front of the agent at the beginning of the episode.

Original: Figure 4 (b) shows the average success rate when using ACTRCE vs. baseline DQN.
Revised: Figure 4 (b) shows the average success rate when using ACTRCE vs. baseline DQN.

Original: Likewise, our agent was able to learn very well with hindsight advice, whereas the baseline DQN failed to learn.
Revised: Likewise, our agent was able to learn very well with hindsight advice, whereas the baseline DQN failed to learn.

Original: 4.3. H OW MUCH ADVICE DO WE NEED ?
Revised: 4.3. H OW MUCH ADVICE DO WE NEED ?

Original: We consider a variant of ACTRCE where the teacher provides hindsight advice only in the ﬁrst { 10% , 1% } of frames during training, and stops giving any advice for the remaining portion of the training.
Revised: We consider a variant of ACTRCE where the teacher provides hindsight advice only in the ﬁrst { 10% , 1% } of frames during training, and stops giving any advice for the remaining portion of the training.

Original: We perform this experiment in the ViZDoom environment single target mode with 7 objects, using the GRU as the sentence embedding.
Revised: We perform this experiment in the ViZDoom environment single target mode with 7 objects, using the GRU as the sentence embedding.

Original: Figure 4 illustrates the training average success rate over the frames, and Table 5 lists the ﬁnal MT and ZSL performance on the trained agents.
Revised: Figure 4 illustrates the training average success rate over the frames, and Table 5 lists the ﬁnal MT and ZSL performance on the trained agents.

Original: We observe that the agent is still able to learn comparably well even given very little (1%) advice, indicating of the method’s robustness in practical settings.
Revised: We observe that the agent is still able to learn comparably well even given very little (1%) advice, indicating of the method’s robustness in practical settings.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Several approaches have used natural language in the context of reinforcement learning.
Revised: Several approaches have used natural language in the context of reinforcement learning.

Original: In pioneering work, Maclin & Shavlik (1994) translated language advice into a short program, implemented as a neural network.
Revised: In pioneering work, Maclin & Shavlik (1994) translated language advice into a short program, implemented as a neural network.

Original: This advice network encouraged the policy to take the suggested action.
Revised: This advice network encouraged the policy to take the suggested action.

Original: Similarly, Kuhlmann et al.
Revised: Similarly, Kuhlmann et al.

Original: (2004) exploited natural language advice for a RL agent learning to play a (simulated) soccer game.
Revised: (2004) exploited natural language advice for a RL agent learning to play a (simulated) soccer game.

Original: In Ling & Fidler (2017), human feedback in the form of natural language was exploited to shape the reward of an image captioning RL agent.
Revised: In Ling & Fidler (2017), human feedback in the form of natural language was exploited to shape the reward of an image captioning RL agent.

Original: Kaplan et al.
Revised: Kaplan et al.

Original: (2017) introduced an agent that learns to use English instructions to self-monitor its progress to beat Atari games.
Revised: (2017) introduced an agent that learns to use English instructions to self-monitor its progress to beat Atari games.

Original: This was accomplished by implementing a multi-modal embedding of the pairs of game frames and instructions to determine if the instruction is satisﬁed, then provide additional reward to the agent.
Revised: This was accomplished by implementing a multi-modal embedding of the pairs of game frames and instructions to determine if the instruction is satisﬁed, then provide additional reward to the agent.

Original: Andreas et al.
Revised: Andreas et al.

Original: (2017) proposed to use language as the latent parameter space for few-shot learning problems, including policy search.
Revised: (2017) proposed to use language as the latent parameter space for few-shot learning problems, including policy search.

Original: The reverse has also been studied: using reinforcement learning to learn grounded language, referred to as task-oriented language grounding.
Revised: The reverse has also been studied: using reinforcement learning to learn grounded language, referred to as task-oriented language grounding.

Original: Misra et al.
Revised: Misra et al.

Original: (2017) mapped language instructions and visual observations to actions for manipulating blocks on a 2D plane.
Revised: (2017) mapped language instructions and visual observations to actions for manipulating blocks on a 2D plane.

Original: Yu et al.
Revised: Yu et al.

Original: (2018) grounded language in 2D maze environment, via sentence-directed navigation and question answering task.
Revised: (2018) grounded language in 2D maze environment, via sentence-directed navigation and question answering task.

Original: They introduced the notion of extrapolation zero-shot sentence, where new words are transferred from other use cases and models.
Revised: They introduced the notion of extrapolation zero-shot sentence, where new words are transferred from other use cases and models.

Original: In their work, the unseen word in the navigation sentence was seen (transferred from) during the question answering task.
Revised: In their work, the unseen word in the navigation sentence was seen (transferred from) during the question answering task.

Original: In contrast, we transfer the unseen synonym words from a pre-trained sentence embedding InferLite model.
Revised: In contrast, we transfer the unseen synonym words from a pre-trained sentence embedding InferLite model.

Original: Hermann et al.
Revised: Hermann et al.

Original: (2017) presented an agent that learns to execute written instructions in a 3D environment through reinforcement learning and unsupervised learning.
Revised: (2017) presented an agent that learns to execute written instructions in a 3D environment through reinforcement learning and unsupervised learning.

Original: Our work builds on Chaplot et al.
Revised: Our work builds on Chaplot et al.

Original: (2017b) who proposed a gated-attention architecture for combining the language and image features to learn a policy that execute written instruction in a 3D environment.
Revised: (2017b) who proposed a gated-attention architecture for combining the language and image features to learn a policy that execute written instruction in a 3D environment.

Original: Our work also relates to augmenting the experience using a experience replay buffer.
Revised: Our work also relates to augmenting the experience using a experience replay buffer.

Original: Lin (1992) introduced experience replay (ER) to speed up the credit assignment process.
Revised: Lin (1992) introduced experience replay (ER) to speed up the credit assignment process.

Original: The technique was later popularized by DQN (Mnih et al.
Revised: The technique was later popularized by DQN (Mnih et al.

Original: , for Atari games
Revised: , for Atari games

Original: Further works focus on sampling techniques from the replay buffer to accelerate learning.
Revised: Further works focus on sampling techniques from the replay buffer to accelerate learning.

Original: Hindsight Experience Replay (Andrychowicz et al., 2017) builds on ER by providing additional experiences in a goal conditional task.
Revised: Hindsight Experience Replay (Andrychowicz et al., 2017) builds on ER by providing additional experiences in a goal conditional task.

Original: Our work extends this technique to use a natural language goal representation.
Revised: Our work extends this technique to use a natural language goal representation.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work we propose the ACTRCE method that uses natural language as a goal representation for hindsight advice.
Revised: In this work we propose the ACTRCE method that uses natural language as a goal representation for hindsight advice.

Original: The main point of the paper is to show that using language as goal representations can bring many beneﬁts, when combined with hindsight advice.
Revised: The main point of the paper is to show that using language as goal representations can bring many beneﬁts, when combined with hindsight advice.

Original: We analyzed the differences among goal representation and show that ACTRCE can efﬁciently solve difﬁcult reinforcement learning problems in challenging 3D navigation tasks, whereas HER with non-language goal representation failed to learn.
Revised: We analyzed the differences among goal representation and show that ACTRCE can efﬁciently solve difﬁcult reinforcement learning problems in challenging 3D navigation tasks, whereas HER with non-language goal representation failed to learn.

Original: We also show that with language goal representations, the agent can generalize to unseen instructions.
Revised: We also show that with language goal representations, the agent can generalize to unseen instructions.

Original: With pre-trained language component, the agent can even generalize to instructions with unseen lexicons , demonstrating its potential to deal with noisy natural language advice from humans.
Revised: With pre-trained language component, the agent can even generalize to instructions with unseen lexicons , demonstrating its potential to deal with noisy natural language advice from humans.

Original: Although ACTRCE algorithm crucially relies on hindsight advice, we showed that little amount of advice is sufﬁcient for the learning to take off, showing its great practicality.
Revised: Although ACTRCE algorithm crucially relies on hindsight advice, we showed that little amount of advice is sufﬁcient for the learning to take off, showing its great practicality.

Original: A K RAZY G RID W ORLD L ANGUAGE G OALS A. M ORE DETIALS ON THE ENVIRONMENT
Revised: A K RAZY G RID W ORLD L ANGUAGE G OALS A. M ORE DETIALS ON THE ENVIRONMENT

Original: KrazyGrid World is a 2D grid environment.
Revised: KrazyGrid World is a 2D grid environment.

Original: For our experiments we chose to use 4 functionality of tiles: Goal, Lava, Normal, and Agent.
Revised: For our experiments we chose to use 4 functionality of tiles: Goal, Lava, Normal, and Agent.

Original: We added an additional colour attribute ranging in 3 colours:
Revised: We added an additional colour attribute ranging in 3 colours:

Original: Red, Blue, Green.
Revised: Red, Blue, Green.

Original: There are 4 discrete actions, “up", “down", “right", “left".
Revised: There are 4 discrete actions, “up", “down", “right", “left".

Original: The desired goals in this environment is to reach goals of different colours.
Revised: The desired goals in this environment is to reach goals of different colours.

Original: We use a global view of the grid state as the observation to the agent.
Revised: We use a global view of the grid state as the observation to the agent.

Original: Each tile in the grid is represented by a concatenation of its functionality and colour attribute, both in one hot vectors.
Revised: Each tile in the grid is represented by a concatenation of its functionality and colour attribute, both in one hot vectors.

Original: The grid state also includes a one hot vector that represents the agent’s position.
Revised: The grid state also includes a one hot vector that represents the agent’s position.

Original: In our experiments, we use an environment of grid size 9 × 9, with 3 goals in the environment, each has a distinct colour.
Revised: In our experiments, we use an environment of grid size 9 × 9, with 3 goals in the environment, each has a distinct colour.

Original: We tried various number of lavas in the environment, and made sure there is at least 1 lava of each colour.
Revised: We tried various number of lavas in the environment, and made sure there is at least 1 lava of each colour.

Original: In the simple task setting, the episode terminates when the agent reaches a lava or a goal, or the episode runs over a maximum time step T = 25 .
Revised: In the simple task setting, the episode terminates when the agent reaches a lava or a goal, or the episode runs over a maximum time step T = 25 .

Original: We automatically generates descriptions give any states of the environment as the teacher.
Revised: We automatically generates descriptions give any states of the environment as the teacher.

Original: Compositional We enlarged the goal space by considering the task of compositions of goals.
Revised: Compositional We enlarged the goal space by considering the task of compositions of goals.

Original: As the goal space changed, we also had to make several modiﬁcations to the environment.
Revised: As the goal space changed, we also had to make several modiﬁcations to the environment.

Original: We let the episode terminate when the agent reaches a lava or reaches 2 different goals, or runs over the maximum time step of 50 .
Revised: We let the episode terminate when the agent reaches a lava or reaches 2 different goals, or runs over the maximum time step of 50 .

Original: Since we still included simple goals (e.g., “Reach blue goal") in the goal space, we added an extra action called “ﬂag".
Revised: Since we still included simple goals (e.g., “Reach blue goal") in the goal space, we added an extra action called “ﬂag".

Original: This action can be used for the agent to terminate the episode when it thinks the given goal is accomplished.
Revised: This action can be used for the agent to terminate the episode when it thinks the given goal is accomplished.

Original: The environment will be terminated if the agent indeed has accomplished the given task, and will stay unchanged otherwise.
Revised: The environment will be terminated if the agent indeed has accomplished the given task, and will stay unchanged otherwise.

Original: A. LIST OF LANGUAGE INSTURCTIONS
Revised: A. LIST OF LANGUAGE INSTURCTIONS

Original: Single goal setting The entire goal space G consists of 8 goals: { Reach red goal, Reach blue goal,
Revised: Single goal setting The entire goal space G consists of 8 goals: { Reach red goal, Reach blue goal,

Original: Reach green goal, Reach red lava, Reach blue lava, Reach green lava, Avoid any lava, Avoid any goal } .
Revised: Reach green goal, Reach red lava, Reach blue lava, Reach green lava, Avoid any lava, Avoid any goal } .

Original: The desired goal space G d consists of 3 goals { Reach red goal, Reach blue goal, Reach green goal } .
Revised: The desired goal space G d consists of 3 goals { Reach red goal, Reach blue goal, Reach green goal } .

Original: Compositional goal setting The desired goal space G d consists of 21 goals { Reach red goal, Reach blue goal, Reach green goal, Reach red goal and Reach blue goal, Reach blue goal and Reach red goal, Reach red goal or Reach blue goal, Reach blue goal or Reach red goal, Reach red goal and
Revised: Compositional goal setting The desired goal space G d consists of 21 goals { Reach red goal, Reach blue goal, Reach green goal, Reach red goal and Reach blue goal, Reach blue goal and Reach red goal, Reach red goal or Reach blue goal, Reach blue goal or Reach red goal, Reach red goal and

Original: Reach green goal, Reach green goal and Reach red goal, Reach red goal or Reach green goal, Reach green goal or Reach red goal, Reach blue goal and Reach green goal, Reach green goal and Reach blue goal, Reach blue goal or Reach green goal, Reach green goal or Reach blue goal, Reach red goal and Avoid any lava, Avoid any lava and Reach red goal, Reach blue goal and Avoid any lava, Avoid any lava and Reach blue goal, Reach green goal and Avoid any lava, Avoid any lava and Reach green goal } .
Revised: Reach green goal, Reach green goal and Reach red goal, Reach red goal or Reach green goal, Reach green goal or Reach red goal, Reach blue goal and Reach green goal, Reach green goal and Reach blue goal, Reach blue goal or Reach green goal, Reach green goal or Reach blue goal, Reach red goal and Avoid any lava, Avoid any lava and Reach red goal, Reach blue goal and Avoid any lava, Avoid any lava and Reach blue goal, Reach green goal and Avoid any lava, Avoid any lava and Reach green goal } .

Original: The entire goal space G has another of 17 goals: { Reach red lava or Reach blue lava, Reach blue lava or Reach red lava, Reach red lava or Reach green lava, Reach green lava or Reach red lava, Reach blue lava or Reach green lava, Reach green lava or Reach blue lava, Reach red lava, Reach red lava and Avoid any goal, Avoid any goal and Reach red lava, Reach blue lava, Reach blue lava and Avoid any goal, Avoid any goal and Reach blue lava, Reach green lava, Reach green lava and Avoid any goal, Avoid any goal and Reach green lava, Avoid any lava and Avoid any goal, Avoid any goal and
Revised: The entire goal space G has another of 17 goals: { Reach red lava or Reach blue lava, Reach blue lava or Reach red lava, Reach red lava or Reach green lava, Reach green lava or Reach red lava, Reach blue lava or Reach green lava, Reach green lava or Reach blue lava, Reach red lava, Reach red lava and Avoid any goal, Avoid any goal and Reach red lava, Reach blue lava, Reach blue lava and Avoid any goal, Avoid any goal and Reach blue lava, Reach green lava, Reach green lava and Avoid any goal, Avoid any goal and Reach green lava, Avoid any lava and Avoid any goal, Avoid any goal and

Original: Avoid any lava } .
Revised: Avoid any lava } .

Original: B V I ZD OOM E NVIRONMENT D ETAIL B. E NVIRONMENT D ESCRIPTION
Revised: B V I ZD OOM E NVIRONMENT D ETAIL B. E NVIRONMENT D ESCRIPTION

Original: ViZDoom (Kempka et al., 2016; Chaplot et al., 2017b):
Revised: ViZDoom (Kempka et al., 2016; Chaplot et al., 2017b):

Original: The 3D learning environment is based on the ﬁrst person shooter game Doom.
Revised: The 3D learning environment is based on the ﬁrst person shooter game Doom.

Original: At each time step, the state is a raw pixel image from ﬁrst person perspective of the 3D environment.
Revised: At each time step, the state is a raw pixel image from ﬁrst person perspective of the 3D environment.

Original: The environment consists of a room containing several random doom objects of various types, colours, and sizes.
Revised: The environment consists of a room containing several random doom objects of various types, colours, and sizes.

Original: The agent can navigate the environment via 3 possible actions: turn left, turn right, and move forward.
Revised: The agent can navigate the environment via 3 possible actions: turn left, turn right, and move forward.

Original: The goal for the episode is a natural language instruction in the form "Go to the [ target attribute ] [ object ]", such as " Go to the green torch ".
Revised: The goal for the episode is a natural language instruction in the form "Go to the [ target attribute ] [ object ]", such as " Go to the green torch ".

Original: Only one target object is present in each episode.
Revised: Only one target object is present in each episode.

Original: The episode terminates either when the agent reaches an object (regardless of whether it is correct or not), or the maximum time step is reached ( T = 30 ).
Revised: The episode terminates either when the agent reaches an object (regardless of whether it is correct or not), or the maximum time step is reached ( T = 30 ).

Original: A reward of 1 is given if the correct object is reached, otherwise a reward of 0 is given.
Revised: A reward of 1 is given if the correct object is reached, otherwise a reward of 0 is given.

Original: The environment has several difﬁculty modes, depending on how the objects and the agent are distributed at the beginning of the episode.
Revised: The environment has several difﬁculty modes, depending on how the objects and the agent are distributed at the beginning of the episode.

Original: In the easy mode, the agent is spawned at a ﬁxed location facing forward.
Revised: In the easy mode, the agent is spawned at a ﬁxed location facing forward.

Original: The objects are placed evenly spaced apart along a horizontal row in front of the agent.
Revised: The objects are placed evenly spaced apart along a horizontal row in front of the agent.

Original: In hard , both the objects and the agent are randomly spawned, and the objects are not necessarily in view.
Revised: In hard , both the objects and the agent are randomly spawned, and the objects are not necessarily in view.

Original: We focus on the easy and hard mode for our experiments.
Revised: We focus on the easy and hard mode for our experiments.

Original: [Figure]
Revised: [Figure]

Original: The compositional instructions consist of two single object instructions from the original training set joined by the word "and", such as " Go to the red torch and Go to the pillar ".
Revised: The compositional instructions consist of two single object instructions from the original training set joined by the word "and", such as " Go to the red torch and Go to the pillar ".

Original: We did not include any superlative instructions, such as " Go to the largest object ".
Revised: We did not include any superlative instructions, such as " Go to the largest object ".

Original: We ensured that the desirable instructions were unambiguous–given two instructions, the set of objects satisfying the ﬁrst instruction is mutually exclusive from the set of objects satisﬁed in the second instruction.
Revised: We ensured that the desirable instructions were unambiguous–given two instructions, the set of objects satisfying the ﬁrst instruction is mutually exclusive from the set of objects satisﬁed in the second instruction.

Original: For example, "Go to the torch and Go to the pillar" have mutually exclusive set of valid objects.
Revised: For example, "Go to the torch and Go to the pillar" have mutually exclusive set of valid objects.

Original: An ambiguous compositional instruction may have objects that satisfy both instructions.
Revised: An ambiguous compositional instruction may have objects that satisfy both instructions.

Original: For example, "Go to the blue object and Go to the torch" is ambiguous because a blue torch satisﬁes both instructions.
Revised: For example, "Go to the blue object and Go to the torch" is ambiguous because a blue torch satisﬁes both instructions.

Original: This is illustrated in Figure 5.
Revised: This is illustrated in Figure 5.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: For the ViZDoom composition task, we modify the raw pixel image of the environment to include a basic head-up display (HUD) consisting of black rectangle in the bottom left of the screen.
Revised: For the ViZDoom composition task, we modify the raw pixel image of the environment to include a basic head-up display (HUD) consisting of black rectangle in the bottom left of the screen.

Original: When the agent reaches an object, a small thumbnail image of the reached object will appear inside the HUD.
Revised: When the agent reaches an object, a small thumbnail image of the reached object will appear inside the HUD.

Original: The HUD can show up to 2 objects reached, and the episode terminates once the again has reached a second object.
Revised: The HUD can show up to 2 objects reached, and the episode terminates once the again has reached a second object.

Original: Figure 6 illustrates what the agent sees as the input image throughout a composition task episode.
Revised: Figure 6 illustrates what the agent sees as the input image throughout a composition task episode.

Original: [Figure]
Revised: [Figure]

Original: B. V I ZD OOM S YNONYM I NSTRUCTIONS
Revised: B. V I ZD OOM S YNONYM I NSTRUCTIONS

Original: We generate the synonym instructions by replacing an original word in the instruction with one of the synonyms listed in Table
Revised: We generate the synonym instructions by replacing an original word in the instruction with one of the synonyms listed in Table

Original: [Table]
Revised: [Table]

Original: B. V I ZD OOM T EACHER A DVICE G ENERATION
Revised: B. V I ZD OOM T EACHER A DVICE G ENERATION

Original: Singleton task.
Revised: Singleton task.

Original: When the agent does reach an object (correct or incorrect), we generate a positive teacher advice by sampling from the training set of instructions that can describe the reached object.
Revised: When the agent does reach an object (correct or incorrect), we generate a positive teacher advice by sampling from the training set of instructions that can describe the reached object.

Original: Additionally, we also generate a negative feedback by randomly picking one of the object in the current environment that was not reached, then sampling an instruction that described that unreached object, while ensuring that it will not describe the reached object.
Revised: Additionally, we also generate a negative feedback by randomly picking one of the object in the current environment that was not reached, then sampling an instruction that described that unreached object, while ensuring that it will not describe the reached object.

Original: When the agent did not reach any object
Revised: When the agent did not reach any object

Original: [Table]
Revised: [Table]

Original: we simply do not give any positive teacher advice (i
Revised: we simply do not give any positive teacher advice (i

Original: [Table]
Revised: [Table]

Original: e. the achieved goal).
Revised: e. the achieved goal).

Original: We had originally tried to give the advice of “ Reach no object ” when the agent did not reach any objects at the end of the episode
Revised: We had originally tried to give the advice of “ Reach no object ” when the agent did not reach any objects at the end of the episode

Original: [Table]
Revised: [Table]

Original: but it did not improve the performance.
Revised: but it did not improve the performance.

Original: However, we still give the unachieved goal, from any of the objects in the current environment.
Revised: However, we still give the unachieved goal, from any of the objects in the current environment.

Original: Compositional task.
Revised: Compositional task.

Original: We follow a similar process for compositional task, depending on the number of objects reached.
Revised: We follow a similar process for compositional task, depending on the number of objects reached.

Original: If the agent reached 0 or 1 object only, we generate a singleton advice in similar fashion as singleton task.
Revised: If the agent reached 0 or 1 object only, we generate a singleton advice in similar fashion as singleton task.

Original: We later found that not giving any advice when reaching 0/1 object also performed equally well and slightly faster learning.
Revised: We later found that not giving any advice when reaching 0/1 object also performed equally well and slightly faster learning.

Original: If the agent reached 2 objects during the episode, for positive advice we sample generate singleton instructions for each of the two objects, ensuring that they were not the same instructions, then merge them with " and " keyword.
Revised: If the agent reached 2 objects during the episode, for positive advice we sample generate singleton instructions for each of the two objects, ensuring that they were not the same instructions, then merge them with " and " keyword.

Original: For negative advice, we generate singleton instructions for the unreached objects and combine two of them, ensuring that the instructions do not refer to the reached objects at all.
Revised: For negative advice, we generate singleton instructions for the unreached objects and combine two of them, ensuring that the instructions do not refer to the reached objects at all.

Original: C A RCHITECTURES KrazyGrid World
Revised: C A RCHITECTURES KrazyGrid World

Original: • Singleton
Revised: • Singleton

Original: We use series of convolution layers follows with ReLU activation functions for prepossessing the grid observation: 32 of 3x3 kernel with stride 1, followed by 64 of 3x kernel size with stride of 2 and then 64 of 3x3 kernel with a stride of 1 and 128 of 3x kernel with a stride of 2.
Revised: We use series of convolution layers follows with ReLU activation functions for prepossessing the grid observation: 32 of 3x3 kernel with stride 1, followed by 64 of 3x kernel size with stride of 2 and then 64 of 3x3 kernel with a stride of 1 and 128 of 3x kernel with a stride of 2.

Original: We input the language sentences as word level one hot vectors to a LSTM of hidden size 128.
Revised: We input the language sentences as word level one hot vectors to a LSTM of hidden size 128.

Original: The LSTM’s last hidden vector is passed into a fully connected layer with 128 output units with sigmoid activation.
Revised: The LSTM’s last hidden vector is passed into a fully connected layer with 128 output units with sigmoid activation.

Original: This acts as the attention vector that is multiplied channel-wise with the 64 feature maps from the convolutional later.
Revised: This acts as the attention vector that is multiplied channel-wise with the 64 feature maps from the convolutional later.

Original: The gated feature maps are then ﬂattened and passed through a fully connected layer with ReLU activation with 256 units.
Revised: The gated feature maps are then ﬂattened and passed through a fully connected layer with ReLU activation with 256 units.

Original: We then pass into a linear layer to predict the 4 action values.
Revised: We then pass into a linear layer to predict the 4 action values.

Original: Compositional We use series of convolution layers follows with ReLU activation functions for prepossessing the grid observation: 32 of 1x1 kernel with stride 1, followed by 32 of 3x kernel size with stride of 2.
Revised: Compositional We use series of convolution layers follows with ReLU activation functions for prepossessing the grid observation: 32 of 1x1 kernel with stride 1, followed by 32 of 3x kernel size with stride of 2.

Original: We input the language sentences as word level one hot vectors to a bidirection LSTM, of hidden size 40.
Revised: We input the language sentences as word level one hot vectors to a bidirection LSTM, of hidden size 40.

Original: After obtaining the preprocessed observation, we augment with the history vector provided by the environment as a query to attend to all the hidden states in the Bi-direction LSTMs via Luong attention Luong et al.
Revised: After obtaining the preprocessed observation, we augment with the history vector provided by the environment as a query to attend to all the hidden states in the Bi-direction LSTMs via Luong attention Luong et al.

Original: We then keep processing the observation using two more convolution layers follows with ReLU activation functions: 64 of 3x3 kernel with stride 2, followed by 64 of 3x3 kernel size with stride of 2.
Revised: We then keep processing the observation using two more convolution layers follows with ReLU activation functions: 64 of 3x3 kernel with stride 2, followed by 64 of 3x3 kernel size with stride of 2.

Original: At each of these two layers, we use the context vector formed by language attention to attend back the observation by gating on 64 feature maps in a channel-wise fashion.
Revised: At each of these two layers, we use the context vector formed by language attention to attend back the observation by gating on 64 feature maps in a channel-wise fashion.

Original: The ﬁnal gated feature maps are then ﬂattened and passed through a fully connected layer with ReLU activation with 256 units.
Revised: The ﬁnal gated feature maps are then ﬂattened and passed through a fully connected layer with ReLU activation with 256 units.

Original: We then pass into a linear layer to predict theaction values.
Revised: We then pass into a linear layer to predict theaction values.

Original: ViZDooms
Revised: ViZDooms

Original: Our architecture is almost identical to Chaplot et al.
Revised: Our architecture is almost identical to Chaplot et al.

Original: (2017b), except that we uses a linear output layer for the action values,
Revised: (2017b), except that we uses a linear output layer for the action values,

Original: and we did not use dueling architecture.
Revised: and we did not use dueling architecture.

Original: The state input to the network is RGB image of size 3 × 300 × 168 .
Revised: The state input to the network is RGB image of size 3 × 300 × 168 .

Original: A series of convolution layers follows with
Revised: A series of convolution layers follows with

Original: ReLU activation functions: 128 of 8x8 kernel with stride 4, followed by 64 of 4x4 kernel size with stride of 2 and then 64 of 4x4 kernel with a stride of 2.
Revised: ReLU activation functions: 128 of 8x8 kernel with stride 4, followed by 64 of 4x4 kernel size with stride of 2 and then 64 of 4x4 kernel with a stride of 2.

Original: To process the language input, we ﬁrst use an embedding matrix to embed the voculary to a vector of size 32.
Revised: To process the language input, we ﬁrst use an embedding matrix to embed the voculary to a vector of size 32.

Original: We use the Gated Recurrent Unit (GRU) Chung et al.
Revised: We use the Gated Recurrent Unit (GRU) Chung et al.

Original: (2014) of size 256 as the recurrent cell to process word embeddings.
Revised: (2014) of size 256 as the recurrent cell to process word embeddings.

Original: The GRU’s last hidden vector is passed into a fully connected layer with 64 output units with sigmoid activation.
Revised: The GRU’s last hidden vector is passed into a fully connected layer with 64 output units with sigmoid activation.

Original: This acts as the attention vector that is multiplied channel-wise with the 64 feature maps from the convolutional later.
Revised: This acts as the attention vector that is multiplied channel-wise with the 64 feature maps from the convolutional later.

Original: The gated feature maps are then ﬂattened and passed through a fully connected layer with ReLU activation with 256 units.
Revised: The gated feature maps are then ﬂattened and passed through a fully connected layer with ReLU activation with 256 units.

Original: We then pass into the LSTM with 256 hidden units, and its hidden units go through a linear layer to predict the 3 action values.
Revised: We then pass into the LSTM with 256 hidden units, and its hidden units go through a linear layer to predict the 3 action values.

Original: D T RAINING D ETAILS
Revised: D T RAINING D ETAILS

Original: KrazyGrid World For KrazyGrid World experiments, we tune the following hyperparameters within corresponding range: learning rate { 0 . 0003 , 0 .
Revised: KrazyGrid World For KrazyGrid World experiments, we tune the following hyperparameters within corresponding range: learning rate { 0 . 0003 , 0 .

Original: 001 , 0 . 003 } , replay buffer size of { 5000 , 10000 } , training the network every { 1 , 2 , 4 } frames.
Revised: 001 , 0 . 003 } , replay buffer size of { 5000 , 10000 } , training the network every { 1 , 2 , 4 } frames.

Original: We generate episodes with (cid:15) -greedy policy starting at (cid:15) = 1 . 0 then decaying linearly to 0 .
Revised: We generate episodes with (cid:15) -greedy policy starting at (cid:15) = 1 . 0 then decaying linearly to 0 .

Original: 01 by 10000 frames and remain at 0 .
Revised: 01 by 10000 frames and remain at 0 .

Original: We use
Revised: We use

Original: Double DQN Hasselt et al.
Revised: Double DQN Hasselt et al.

Original: (2016) to reduce the Q-value overestimation and Huber loss ( δ = 1 ) for stable gradients.
Revised: (2016) to reduce the Q-value overestimation and Huber loss ( δ = 1 ) for stable gradients.

Original: ViZDoom For ViZDoom environment, we use the same set of 56 training instructions to train the agent, and a set of 15 held out test instructions for zero-shot evaluation as in Chaplot et al.
Revised: ViZDoom For ViZDoom environment, we use the same set of 56 training instructions to train the agent, and a set of 15 held out test instructions for zero-shot evaluation as in Chaplot et al.

Original: (2017b).
Revised: (2017b).

Original: We also used their code Chaplot et al.
Revised: We also used their code Chaplot et al.

Original: (2017a)
Revised: (2017a)

Original: for reproducing training using Asynchronous
Revised: for reproducing training using Asynchronous

Original: Advantage Actor Critic (A3C) Mnih et al.
Revised: Advantage Actor Critic (A3C) Mnih et al.

Original: We implemented our version of DQN on top of the implementation of Arnold Lample & Chaplot (2017);
Revised: We implemented our version of DQN on top of the implementation of Arnold Lample & Chaplot (2017);

Original: Lample (2017).
Revised: Lample (2017).

Original: The cyclic buffer replay buffer contains the last 10000 and 20000 most recent transitions for the easy and hard mode, respectively, chosen from the range { 1000 , 10000 , 100000 } and ﬁne tuned.
Revised: The cyclic buffer replay buffer contains the last 10000 and 20000 most recent transitions for the easy and hard mode, respectively, chosen from the range { 1000 , 10000 , 100000 } and ﬁne tuned.

Original: We generate episodes with (cid:15) -greedy policy starting at (cid:15) = 1 . 0 then decaying linearly to 0 .
Revised: We generate episodes with (cid:15) -greedy policy starting at (cid:15) = 1 . 0 then decaying linearly to 0 .

Original: 01 by 10000 frames and remain at 0 .
Revised: 01 by 10000 frames and remain at 0 .

Original: We found that this performed better than using a larger epsilon = 0 .
Revised: We found that this performed better than using a larger epsilon = 0 .

Original: Only after the ﬁrst 1000 frames have been collected that the training begins, selected from a range of { 1000 , 10000 , 100000 } .
Revised: Only after the ﬁrst 1000 frames have been collected that the training begins, selected from a range of { 1000 , 10000 , 100000 } .

Original: We use Double DQN Hasselt et al.
Revised: We use Double DQN Hasselt et al.

Original: (2016) to reduce the Q-value overestimation and Huber loss ( δ = 1 ) for stable gradients.
Revised: (2016) to reduce the Q-value overestimation and Huber loss ( δ = 1 ) for stable gradients.

Original: We use the Adam optimizer Kingma & Ba (2014) with a learning rate of 0.0001.
Revised: We use the Adam optimizer Kingma & Ba (2014) with a learning rate of 0.0001.

Original: The network is updated every 4 frames on easy and 16 frames on difﬁcult mode.
Revised: The network is updated every 4 frames on easy and 16 frames on difﬁcult mode.

Original: While updating less often reduces sample efﬁciency, in practice we found that this speeds up training wall clock time and converges to a better performance.
Revised: While updating less often reduces sample efﬁciency, in practice we found that this speeds up training wall clock time and converges to a better performance.

Original: When sampling from the replay buffer, we sample the start a random episode and select 32 consecutive frames from the replay buffer.
Revised: When sampling from the replay buffer, we sample the start a random episode and select 32 consecutive frames from the replay buffer.

Original: This approach leads to more accurate estimates of the hidden state of the LSTM.
Revised: This approach leads to more accurate estimates of the hidden state of the LSTM.

Original: In addition, the sequential mini-batch allows us to perform n -step Q learning as outlined in Mnih et al.
Revised: In addition, the sequential mini-batch allows us to perform n -step Q learning as outlined in Mnih et al.

Original: The correlation between samples in the mini-batch is alleviated by running 16 parallel threads to send gradient updates to the shared network parameters.
Revised: The correlation between samples in the mini-batch is alleviated by running 16 parallel threads to send gradient updates to the shared network parameters.

Original: We synchronize the training thread model with the shared network each time before computing the training loss and gradient.
Revised: We synchronize the training thread model with the shared network each time before computing the training loss and gradient.

Original: The target network is synchronized with the current model once every 500 time steps.
Revised: The target network is synchronized with the current model once every 500 time steps.

Original: and use one additional thread to evaluate the multi-task success rate throughout the training process.
Revised: and use one additional thread to evaluate the multi-task success rate throughout the training process.

Original: E D IFFERENT TYPES OF TEACHERS
Revised: E D IFFERENT TYPES OF TEACHERS

Original: In this section we described the details of teacher types.
Revised: In this section we described the details of teacher types.

Original: Firstly, note that not all of the language descriptions describe favorable behaviors.
Revised: Firstly, note that not all of the language descriptions describe favorable behaviors.

Original: For example, in the KGW, a desirable goal is “reach a goal", but there are also undesirable states that corresponds to goal “reach a lava".
Revised: For example, in the KGW, a desirable goal is “reach a goal", but there are also undesirable states that corresponds to goal “reach a lava".

Original: Hence we consider a subset G d ⊆ G to denote all desired goals that the agent is expected to perform.
Revised: Hence we consider a subset G d ⊆ G to denote all desired goals that the agent is expected to perform.

Original: At each episode, we ﬁrst sample a goal g ∈ G d from the set of all desired language goal spaces, and ask the agent to explore the environments conditioned on the goal.
Revised: At each episode, we ﬁrst sample a goal g ∈ G d from the set of all desired language goal spaces, and ask the agent to explore the environments conditioned on the goal.

Original: When the episode ends, we obtained an advice from a teacher T by observing the terminal state, g ∗ =
Revised: When the episode ends, we obtained an advice from a teacher T by observing the terminal state, g ∗ =

Original: T ( s T ) .
Revised: T ( s T ) .

Original: Depending on the kind of teacher, we obtain a different kind of advice or no advice.
Revised: Depending on the kind of teacher, we obtain a different kind of advice or no advice.

Original: We consider three kinds of teacher as follows:
Revised: We consider three kinds of teacher as follows:

Original: • Optimistic A teacher who gives advice only when the agent achieved a desirable goal, i.e., when g ∗ ∈ G d .
Revised: • Optimistic A teacher who gives advice only when the agent achieved a desirable goal, i.e., when g ∗ ∈ G d .

Original: When the agent performs poorly, there is no advice from this teacher.
Revised: When the agent performs poorly, there is no advice from this teacher.

Original: • Knowledgeable A teacher who describes what the agent has achieved in all scenarios, including the undesirable behaviors, as advice to the agent.
Revised: • Knowledgeable A teacher who describes what the agent has achieved in all scenarios, including the undesirable behaviors, as advice to the agent.

Original: Discouraging A teacher who describes a desired goal g ∗ ∈ G d that the agent has not achieved as advice to the agent.
Revised: Discouraging A teacher who describes a desired goal g ∗ ∈ G d that the agent has not achieved as advice to the agent.

Original: In this case, the trajectory with goal replaced by the teacher’s advice will receive a reward of 0.
Revised: In this case, the trajectory with goal replaced by the teacher’s advice will receive a reward of 0.

Original: E. H OW DOES EACH TEACHER PERFORM ?
Revised: E. H OW DOES EACH TEACHER PERFORM ?

Original: In this section, we investigated how different teachers help in giving advice.
Revised: In this section, we investigated how different teachers help in giving advice.

Original: We compared our method to the DQN algorithm.
Revised: We compared our method to the DQN algorithm.

Original: We denote the method using only optimistic and discouraging teachers as “ACTRCE − ".
Revised: We denote the method using only optimistic and discouraging teachers as “ACTRCE − ".

Original: We denote the method using knowledgeable teachers as well as discouraging teachers as ACTRCE (this is the version we use in the main paper).
Revised: We denote the method using knowledgeable teachers as well as discouraging teachers as ACTRCE (this is the version we use in the main paper).

Original: We evaluated three methods in KrazyGrid
Revised: We evaluated three methods in KrazyGrid

Original: World and the results are as follows.
Revised: World and the results are as follows.

Original: We tried two different kind of grids, one with 3 lavas and the other with 6 lavas, and both with 3 goals of different colours.
Revised: We tried two different kind of grids, one with 3 lavas and the other with 6 lavas, and both with 3 goals of different colours.

Original: Fig.
Revised: Fig.

Original: 4 (a) and (b) show the average success rate over all goals on 16-environments training.
Revised: 4 (a) and (b) show the average success rate over all goals on 16-environments training.

Original: The shaded area represents the standard deviation over 2 random seeds.
Revised: The shaded area represents the standard deviation over 2 random seeds.

Original: The baseline DQN is shown in blue curve, which failed to learn anything at all.
Revised: The baseline DQN is shown in blue curve, which failed to learn anything at all.

Original: “ACTRCE − " (shown in orange) quickly learned and achieved good results on both environments.
Revised: “ACTRCE − " (shown in orange) quickly learned and achieved good results on both environments.

Original: However, we observed that when the number of lavas increased, the task became harder, and ACTRCE − performed worse after 32 millions frames of training (the performance dropped from 83% to 63% ).
Revised: However, we observed that when the number of lavas increased, the task became harder, and ACTRCE − performed worse after 32 millions frames of training (the performance dropped from 83% to 63% ).

Original: On the other hand, we observed that having knowledgeable teachers always helped speed up learning.
Revised: On the other hand, we observed that having knowledgeable teachers always helped speed up learning.

Original: In particular, when the number of lavas increased, the task became harder for ACTRCE − .
Revised: In particular, when the number of lavas increased, the task became harder for ACTRCE − .

Original: However, with knowledgeable teachers, language advice was provided even when the agent reached lava, and hence the amount of advice per step was kept the same in the more difﬁcult setting.
Revised: However, with knowledgeable teachers, language advice was provided even when the agent reached lava, and hence the amount of advice per step was kept the same in the more difﬁcult setting.

Original: Therefore, we observed ACTRCE learning at a similar rate in the more difﬁcult setting, leaving a big gap from ACTRCE − aftermillions frames of training ( 80% vs 63% ).
Revised: Therefore, we observed ACTRCE learning at a similar rate in the more difﬁcult setting, leaving a big gap from ACTRCE − aftermillions frames of training ( 80% vs 63% ).

Original: F C AN LEARNING EASY GOALS HELP LEARNING DIFFICULT GOALS ?
Revised: F C AN LEARNING EASY GOALS HELP LEARNING DIFFICULT GOALS ?

Original: As we mentioned, there are desirable tasks as well as undesirable tasks.
Revised: As we mentioned, there are desirable tasks as well as undesirable tasks.

Original: One would worry what if the desirable tasks are very difﬁcult, would the agent only learns to accomplish easier tasks which are undesirable?
Revised: One would worry what if the desirable tasks are very difﬁcult, would the agent only learns to accomplish easier tasks which are undesirable?

Original: Hence, we would like to ask whether learning the easier tasks from hindsight can
Revised: Hence, we would like to ask whether learning the easier tasks from hindsight can

Original: [Figure]
Revised: [Figure]

Original: provide any learning signal for more difﬁcult tasks.
Revised: provide any learning signal for more difﬁcult tasks.

Original: We hypothesize that it is possible in the KGW setting as learning to “reach lava" can aid learning controllers, which is used to “reach goals".
Revised: We hypothesize that it is possible in the KGW setting as learning to “reach lava" can aid learning controllers, which is used to “reach goals".

Original: We designed the following transfer learning experiment to further our hypothesis.
Revised: We designed the following transfer learning experiment to further our hypothesis.

Original: First, we artiﬁcially constructed a pessimistic teacher who only gives advice when the agent achieved undesirable goals, i.e., when g ∗ ∈ G\G d .
Revised: First, we artiﬁcially constructed a pessimistic teacher who only gives advice when the agent achieved undesirable goals, i.e., when g ∗ ∈ G\G d .

Original: We pretrained the agents with only the pessimistic teacher for 5 million frames.
Revised: We pretrained the agents with only the pessimistic teacher for 5 million frames.

Original: Now, we carried out training using ACTRCE − (with optimistic and discouraging teachers) with the pretrained agent and compared to unpretrained ones.
Revised: Now, we carried out training using ACTRCE − (with optimistic and discouraging teachers) with the pretrained agent and compared to unpretrained ones.

Original: Results are shown in Figure 8.
Revised: Results are shown in Figure 8.

Original: We found that by pretraining the agent using pessimistic agents, even though the language goals in the pretraining phase have no overlapping with the actual training goals, the agent learned much faster than the unpretrained ones.
Revised: We found that by pretraining the agent using pessimistic agents, even though the language goals in the pretraining phase have no overlapping with the actual training goals, the agent learned much faster than the unpretrained ones.

Original: In particular, in the environment with 3 lavas, the pretrained agent learned the fastest.
Revised: In particular, in the environment with 3 lavas, the pretrained agent learned the fastest.

Original: In the environment with 6 lavas, pretrained agents learned as fast as ACTRCE, leaving a big gap from ACTRCE − , even though it was given the same amount of advice during training as
Revised: In the environment with 6 lavas, pretrained agents learned as fast as ACTRCE, leaving a big gap from ACTRCE − , even though it was given the same amount of advice during training as

Original: ACTRCE − .
Revised: ACTRCE − .

Original: This provides an evidence that learning easier goals can sometimes provide learning signals for harder goals, especially when both tasks require similar modules
Revised: This provides an evidence that learning easier goals can sometimes provide learning signals for harder goals, especially when both tasks require similar modules

Original: [Figure]
Revised: [Figure]

Original: G V I ZD OOM 5 O BJECTS S INGLE T ARGET E XPERIMENTS
Revised: G V I ZD OOM 5 O BJECTS S INGLE T ARGET E XPERIMENTS

Original: In this section we show experiments done on ViZDoom with single target for 5 objects on easy and hard mode.
Revised: In this section we show experiments done on ViZDoom with single target for 5 objects on easy and hard mode.

Original: On the easy task, the baseline DQN is able to consistently learn to solve the task, and with ACTRCE the performance was similar.
Revised: On the easy task, the baseline DQN is able to consistently learn to solve the task, and with ACTRCE the performance was similar.

Original: However, as the task difﬁculty increased to theobjects on hard mode, we found that DQN’s learning was less consistent across seeds, in contrast with ACTRCE’s low variance across seeds.
Revised: However, as the task difﬁculty increased to theobjects on hard mode, we found that DQN’s learning was less consistent across seeds, in contrast with ACTRCE’s low variance across seeds.

Original: We also ran the A3C baseline from Chaplot et al.
Revised: We also ran the A3C baseline from Chaplot et al.

Original: (2017b), from their implementation available online (Chaplot et al.
Revised: (2017b), from their implementation available online (Chaplot et al.

Original: (2017a)).
Revised: (2017a))

Revised: using the hyperparameter settings stated in their paper.

Original: On the easy task, we were able to solve the task with A3C [Table] but with an order of magnitude less sample efﬁciency compared to our DQN/ACTRCE implementation.
Revised: On the easy task, we were able to solve the task with A3C, but with an order of magnitude less sample efﬁciency compared to our DQN/ACTRCE implementation.

Original: On the hard task, we were unable to reproduce the results with our limited computational budget.
Revised: On the hard task, we were unable to reproduce the results with our limited computational budget.

Original: A similar trend of an order of magnitude difference in sample efﬁciency is observed between A3C and DQN/ACTRCE on the hard task
Revised: A similar trend of an order of magnitude difference in sample efﬁciency is observed between A3C and DQN/ACTRCE on the hard task

Original: [Table]
Revised: [Table]

Original: The standard deviations are calculated across 2 seeds.
Revised: calculated across 2 seeds.

Original: MT was evaluated throughout the training process, while ZSL was evaluated at the end of training.
Revised: MT was evaluated throughout the training process, while ZSL was evaluated at the end of training

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: H V I ZD OOM A VERAGE E PISODE L ENGTH
Revised: H V I ZD OOM A VERAGE E PISODE L ENGTH

Original: We report on the average episode length (averaged over 100 episodes) during training for 3 ViZDoom scenarios: single target 5 and 7 objects in hard mode, and composition target 5 objects in easy mode.
Revised: We report on the average episode length (averaged over 100 episodes) during training for 3 ViZDoom scenarios: single target 5 and 7 objects in hard mode, and composition target 5 objects in easy mode.

Original: The GRU hidden state representation of the sentence were used in the plots in Figure 11.
Revised: The GRU hidden state representation of the sentence were used in the plots in Figure 11.

Original: We observe that the average episode length is slowly decreasing when using ACTRCE, while the baseline DQN remains fairly ﬂat for the harder 7 objects (single target) and 5 objects (compositional).
Revised: We observe that the average episode length is slowly decreasing when using ACTRCE, while the baseline DQN remains fairly ﬂat for the harder 7 objects (single target) and 5 objects (compositional).

Original: [Figure]
Revised: [Figure]

Revised: I V I ZD OOM C UMULATIVE S UCCESS R ATE VS . E PISODE L ENGTH C URVE

Revised: We take the ﬁnal trained model and run 100 episodes, noting whether each run was successful or not (binary value).

Revised: We construct a cumulative success rate (CSR) versus episode length curve, where the x-axis is the episode length, and the y-axis is the fraction of total number of episodes that were successful and had episode length less than or equal to the x-axis value:

Revised: [Equation]

Revised: Therefore, the curve is monotonically increasing, with the y-axis being the overall success rate when the x-axis value is the maximum episode length.

Revised: The better the model, the larger the area under this curve will be—similar in spirit to the precision-recall curve—because it will be able to have more of successful trajectories that are short early on.

Revised: Figure 12 shows the Multi-task (MT) cumulative success rate for the 3 ViZDoom environment tasks, using GRU hidden state language encoding: single target 5 and 7 objects in hard mode, and composition target 5 objects in easy mode.

Revised: In the 5 objects hard mode (Figure 12a), we observe that all 3 training algorithms had similar performance until around episode length of 20, where ACTRCE has more successful trajectories which are longer.

Revised: In the 7 objects hard mode (Figure 12b), ACTRCE maintains a similar behaviour while the baseline DQN essentially was only able to get success on very short episodes (i.e. when the target object was very close by).

Revised: Lastly, in the 5 objects composition task (Figure 12c), the curve for ACTRCE indicates that there were 2 groups of trajectories: one requiring less than 10 time-steps, while another requiring over 20 time-steps.

Revised: The former group occurs when the two target objects are adjacent to one another, making it easy to reach the second object after the ﬁrst in only a few time steps.

Revised: The latter group occurs when the target objects are not adjacent to each other, and thus requires the agent to more carefully turn around and avoid hitting other objects when trying to reach the second target.

Revised: 30 Episode Length 0.

Revised: S u cc e ss R a t e w i

Revised: t h i n

Revised: E p s L e n

Revised: ViZDoom Single (5 objects hard)

Revised: DQN ACTRCE A3C     30 Episode Length 0.0 0.

Revised: S u cc e ss R a t e w i

Revised: t h i n

Revised: E p s L e n

Revised: ViZDoom Single (7 objects hard)

Revised: DQN ACTRCE 10 15 20 25 30 35 40 Episode Length 0.

Revised: S u cc e ss R a t e w i

Revised: t h i n

Revised: E p s L e n

Revised: ViZDoom Composition (5 objects)

Revised: DQN ACTRCE

Original: [Figure]
Revised: [Figure]

Original: I A DDITIONAL DETAILS ON H INDSIGHT A DVICE A NNEALING
Revised: J A DDITIONAL DETAILS ON H INDSIGHT A DVICE A NNEALING

Original: We report the results for the hindsight advice annealing when using InferLite and One-Hot representation when providing hindsight advice on the ﬁrst 10% and 1% of training:
Revised: We report the results for the hindsight advice annealing when using InferLite and One-Hot representation when providing hindsight advice on the ﬁrst 10% and 1% of training:

Original: Method
Revised: [Table]

Original: MT
Revised: MT

Original: ZSL
Revised: [Table] [Figure] ACTRCE (InferLite)

Original: [Table]
Revised: [Figure]

Revised: ACTRCE (InferLite)

Original: [Figure]
Revised: [Figure]

Original: cc
Revised: ACTRCE (OneHot [Figure] − ACTRCE (OneHot)

Original: [Figure]
Revised: [Figure]

Original: ss
Revised: [Figure] [Figure] − ACTRCE (OneHot)

Original: R a t e
Revised: [Figure] [Figure]

Original: Limit Advice: ViZDoom (Single) InferLite
Revised: Limit Advice: ViZDoom (Single) InferLite

Original: [Figure]
Revised: [Figure]

Original: Limit Advice: ViZDoom (Single) One-Hot
Revised: Limit Advice: ViZDoom (Single) One-Hot

Original: [Figure]
Revised: [Figure]

Original: J V ISUALIZATION
Revised: K V ISUALIZATION

Original: For each single-target instruction in the training and test set (70 instructions), we obtain an embedding vector of dimension 256 using either GRU, InferLite, or One Hot (storing entire embedding).
Revised: For each single-target instruction in the training and test set (70 instructions), we obtain an embedding vector of dimension 256 using either GRU, InferLite, or One Hot (storing entire embedding).

Original: From this embedding matrix, we can visualize the learned instruction embedding in several ways.
Revised: From this embedding matrix, we can visualize the learned instruction embedding in several ways.

Original: J. E MBEDDING C ORRELATION C OMPARISON
Revised: K. E MBEDDING C ORRELATION C OMPARISON

Original: For each pair of instructions i and j , we obtain their embeddings vectors v i and v j , and compute their correlation distance , cdist ( v i , v j ) .
Revised: For each pair of instructions i and j , we obtain their embeddings vectors v i and v j , and compute their correlation distance , cdist ( v i , v j ) .

Original: We deﬁne the correlation distance between two vectors u and v as:
Revised: We deﬁne the correlation distance between two vectors u and v as:

Original: [Equation]
Revised: [Equation]

Original: We divide each matrix by the maximum entry as to scale the results to [0 , 1] .
Revised: We divide each matrix by the maximum entry as to scale the results to [0 , 1] .

Original: G o t o t h e s h
Revised: G o t o t h e s h

Original: o r t
Revised: o r t

Original: r
Revised: r

Original: e
Revised: e

Original: d o b
Revised: d o b

Original: j e c t
Revised: j e c t

Original: G o t o t h e t
Revised: G o t o t h e t

Original: a ll r e d o b j e c t
Revised: a ll r e d o b j e c t

Original: G o t o t
Revised: G o t o t

Original: h e r e d s h o r t
Revised: h e r e d s h o r t

Original: o
Revised: o

Original: b
Revised: b

Original: j e c t
Revised: j e c t

Original: G o t o t h e r e d t a
Revised: G o t o t h e r e d t a

Original: ll o b
Revised: ll o b

Original: j e c t
Revised: j e c t

Original: G o t o t
Revised: G o t o t

Original: h e
Revised: h e

Original: r e d o b j e c t
Revised: r e d o b j e c t

Original: G o t o t h e l a r g e s t r e d o b j e c t G o t o t h e s m a ll e s t r e d o b j e c t
Revised: G o t o t h e l a r g e s t r e d o b j e c t G o t o t h e s m a ll e s t r e d o b j e c t

Original: G o t o t h e t a ll b l u e o b j e c t
Revised: G o t o t h e t a ll b l u e o b j e c t

Original: G o t o t h e s h
Revised: G o t o t h e s h

Original: o r t
Revised: o r t

Original: b
Revised: b

Original: l
Revised: l

Original: u e o b j e c t
Revised: u e o b j e c t

Original: G o t o t
Revised: G o t o t

Original: h e
Revised: h e

Original: b l u e t a
Revised: b l u e t a

Original: ll o b
Revised: ll o b

Original: j e c t
Revised: j e c t

Original: G o t o t
Revised: G o t o t

Original: h e
Revised: h e

Original: b l u e s h o r t
Revised: b l u e s h o r t

Original: o
Revised: o

Original: b
Revised: b

Original: j e c t
Revised: j e c t

Original: G o t o t
Revised: G o t o t

Original: h e
Revised: h e

Original: b l u e o b j e c t
Revised: b l u e o b j e c t

Original: G o t o t h e s m a ll e s t b l u e o b
Revised: G o t o t h e s m a ll e s t b l u e o b

Original: j e c t
Revised: j e c t

Original: G o t o t h e l a r g e s t b l u e o b j e c t G o t o t h e t all g r e e n o b
Revised: G o t o t h e l a r g e s t b l u e o b j e c t G o t o t h e t a

Original:  j e c t
Revised: ll g r ee n o b j e c t

Original: G o t o t h e s h o r t g r e e n o b
Revised: G o t o t h e s h o r t g r ee n o b

Original: j e c t
Revised: j e c t

Original: G o t o t
Revised: G o t o t

Original: h e g r ee n t a ll o
Revised: h e g r ee n t a ll o

Original: b j e c t
Revised: b j e c t

Original: G o t o t h e g r ee n s h
Revised: G o t o t h e g r ee n s h

Original: o r t
Revised: o r t

Original: o
Revised: o

Original: b
Revised: b

Original: j e c t
Revised: j e c t

Original: G o t o t
Revised: G o t o t

Original: h e g r e e n o b j e c t
Revised: h e g r ee n o b j e c t

Original: G o t o t h e s m a ll e s t g r e e n o b
Revised: G o t o t h e s m a ll e s t g r ee n o b

Original: j e c t
Revised: j e c t

Original: G o t o t h e l a r g e s t g r e e n o b
Revised: G o t o t h e l a r g e s t g r ee n o b

Original: j e c t
Revised: j e c t

Original: G o t o t
Revised: G o t o t

Original: h e y e ll o w o
Revised: h e y e ll o w o

Original: b
Revised: b

Original: j e c t
Revised: j e c t

Original: G o t o t h e s m a ll e s t
Revised: G o t o t h e s m a ll e s t

Original: y e ll o w o b
Revised: y e ll o w o b

Original: j e c t
Revised: j e c t

Original: G o t o t h e l a r g e s t
Revised: G o t o t h e l a r g e s t

Original: y e ll o w o b
Revised: y e ll o w o b

Original: j e c t
Revised: j e c t

Original: G o t o t h e l a r g e s t
Revised: G o t o t h e l a r g e s t

Original: o b j e c t
Revised: o b j e c t

Original: G o t o t h e s m a ll e s t o b
Revised: G o t o t h e s m a ll e s t o b

Original: j e c t
Revised: j e c t

Original: G o t o t h e s h
Revised: G o t o t h e s h

Original: o r t
Revised: o r t

Original: o
Revised: o

Original: b
Revised: b

Original: j e c t G o t o t
Revised: j e c t G o t o t

Original: h e t a
Revised: h e t a

Original: ll o b
Revised: ll o b

Original: j e c t G o t o n o o b j e c t G o t o t
Revised: j e c t G o t o n o o b j e c t G o t o t

Original: h e
Revised: h e

Original: r e d a r m o r
Revised: r e d a r m o r

Original: G o t o t h e g r e e n a r m 
Revised: G o t o t h e g r ee n a r m o r

Original: o r G o t o t h e a r m o r
Revised: G o t o t h e a r m o r

Original: G o t o t h e s h
Revised: G o t o t h e s h

Original: o r t r e d p ill a r
Revised: o r t r e d p ill a r

Original: G o t o t h e t a ll r e d p ill a r
Revised: G o t o t h e t a ll r e d p ill a r

Original: G o t o t
Revised: G o t o t

Original: h e r e d s h o r t p ill a r
Revised: h e r e d s h o r t p ill a r

Original: G o t o t h e r e d t a ll p ill a r G o t o t h e r e d p ill a r
Revised: G o t o t h e r e d t a ll p ill a r G o t o t h e r e d p ill a r

Original: G o t o t h e t a
Revised: G o t o t h e t a

Original: ll g r ee n p ill a r
Revised: ll g r ee n p ill a r

Original: G o t o t h e s h
Revised: G o t o t h e s h

Original: o r t g r ee n p ill a r
Revised: o r t g r ee n p ill a r

Original: G o t o t
Revised: G o t o t

Original: h e g r ee n t a ll p ill a r
Revised: h e g r ee n t a ll p ill a r

Original: G o t o t h e g r ee n s h
Revised: G o t o t h e g r ee n s h

Original: o r t
Revised: o r t

Original: p ill a r
Revised: p ill a r

Original: G o t o t h e g r ee n p ill a r G o t o t h e p ill a r
Revised: G o t o t h e g r ee n p ill a r G o t o t h e p ill a r

Original: G o t o t h e t a ll p ill a r
Revised: G o t o t h e t a ll p ill a r

Original: G o t o t h e s h
Revised: G o t o t h e s h

Original: o r t p ill a r
Revised: o r t p ill a r

Original: G o t o t h e s h
Revised: G o t o t h e s h

Original: o r t r e d t
Revised: o r t r e d t

Original: o r c h
Revised: o r c h

Original: G o t o t h e t a ll r e d t
Revised: G o t o t h e t a ll r e d t

Original: o r c h
Revised: o r c h

Original: G o t o t
Revised: G o t o t

Original: h e r e d s h o r t t
Revised: h e r e d s h o r t t

Original: o r c h
Revised: o r c h

Original: G o t o t h e r e d t a ll t o r c h
Revised: G o t o t h e r e d t a ll t o r c h

Original: G o t o t
Revised: G o t o t

Original: h e r e d t
Revised: h e r e d t

Original: o r c h
Revised: o r c h

Original: G o t o t h e s h
Revised: G o t o t h e s h

Original: o r t
Revised: o r t

Original: b l u e t
Revised: b l u e t

Original: o r c h
Revised: o r c h

Original: G o t o t h e t a ll b l u e t
Revised: G o t o t h e t a ll b l u e t

Original: o r c h
Revised: o r c h

Original: G o t o t h eb
Revised: G o t o t h e

Original: [Table]
Revised: b l u e t a ll t o r c h G o t o t

Original: u
Revised: h e

Original: [Table]
Revised: b l u e s h o r t t o r c h G o t o t

Original: t a
Revised: h e

Original: [Table]
Revised: b l u e t o r c h G o t o t h e t a ll g r ee n t o r c h

Revised: G o t o t h e s h

Original: t o 
Revised: o r t

Original: [Table] c 
Revised: g r ee n t o r c h

Original: [Table] t
Revised: G o t o t

Original: [Table]
Revised: h e g r ee n t a ll t o r c h G o t o t h e g r ee n s h

Original:  t
Revised: o r t t

Original: [Table]
Revised: o r c h G o t o t h e g r ee n t o r c h G o t o t h e t o r c h

Original:  r 
Revised: G o t o t h e s h o r t t o r c h G o t o t h e t a ll t o r c h G o t o t h e r e d

Original: [Table]
Revised: [Table]

Original: o r c

Original: [Table]
Revised: [Table]

Original: t

Original: [Table]
Revised: [Table]

Revised: G o t o t h e s h

Original: n t
Revised: o r t

Original: [Table]

Original: t
Revised: r

Original: [Table]

Original: n
Revised: e

Original: [Table] t [Table]
Revised: d o b

Original:  t[Table]
Revised: j e c t

Original:  t [Table] t[Table]
Revised: G o t o t h e t

Original:  r e d o b j e c t
Revised: a ll r e d o b j e c t

Original: G o t o 
Revised: G o t o t

Original: [Table]
Revised: h e r e d s h o r t o b j e c t

Original:  t o t h e r e d t a
Revised: G o t o t h e r e d t a

Original: ll o b
Revised: ll o b

Original: j e c t
Revised: j e c t

Original: G o t o t
Revised: G o t o t

Original: h e
Revised: h e

Original: r e d o b j e c t
Revised: r e d o b j e c t

Original: G o t o t h e l a r g e s t r e d o b j e c t G o t o t h e s m a ll e s t r e d o b j e c t
Revised: G o t o t h e l a r g e s t r e d o b j e c t G o t o t h e s m a ll e s t r e d o b j e c t

Original: G o t o t h e t a ll b l u e o b j e c t
Revised: G o t o t h e t a ll b l u e o b j e c t

Original: G o t o t h e s h
Revised: G o t o t h e s h

Original: o r t
Revised: o r t

Original: b
Revised: b

Original: l
Revised: l

Original: u e o b j e c t
Revised: u e o b j e c t

Original: G o t o t
Revised: G o t o t

Original: h e
Revised: h e

Original: b l u e t a
Revised: b l u e t a

Original: ll o b
Revised: ll o b

Original: j e c t
Revised: j e c t

Original: G o t o t
Revised: G o t o t

Original: h e
Revised: h e

Original: b l u e s h o 
Revised: b l u e s h o r t

Original: [Table]
Revised: o

Original: t o b j e c t
Revised: b j e c t

Original: G o t o t
Revised: G o t o t

Original: h e
Revised: h e

Original: b l u e o b j e c t
Revised: b l u e o b j e c t

Original: G o t o t h e s m a ll e s t b l u e o b
Revised: G o t o t h e s m a ll e s t b l u e o b

Original: j e c t
Revised: j e c t

Original: G o t o t h e l a r g e s t b l u e o b j e c t G o t o t h e t all g r e e n o b
Revised: G o t o t h e l a r g e s t b l u e o b j e c t G o t o t h e t a

Original:  j e c t
Revised: ll g r ee n o b j e c t

Original: G o t o t h e s h o r t g r e e n o b
Revised: G o t o t h e s h o r t g r ee n o b

Original: j e c t
Revised: j e c t

Original: G o t o t
Revised: G o t o t

Original: h e g r ee n t a ll o
Revised: h e g r ee n t a ll o

Original: b j e c t
Revised: b j e c t

Original: G o t o t h e g r ee n s h
Revised: G o t o t h e g r ee n s h

Original: o r t
Revised: o r t

Original: o
Revised: o

Original: b
Revised: b

Original: j e c t
Revised: j e c t

Original: G o t o t
Revised: G o t o t

Original: h e g r e e n o b j e c t
Revised: h e g r ee n o b j e c t

Original: G o t o t h e s m a ll e s t g r e e n o b
Revised: G o t o t h e s m a ll e s t g r ee n o b

Original: j e c t
Revised: j e c t

Original: G o t o t h e l a r g e s t g r e e n o b
Revised: G o t o t h e l a r g e s t g r ee n o b

Original: j e c t
Revised: j e c t

Original: G o t o t
Revised: G o t o t

Original: h e y e ll o w o
Revised: h e y e ll o w o

Original: b
Revised: b

Original: j e c t
Revised: j e c t

Original: G o t o t h e s m a ll e s t
Revised: G o t o t h e s m a ll e s t

Original: y e ll o w o b
Revised: y e ll o w o b

Original: j e c t
Revised: j e c t

Original: G o t o t h e l a r g e s t
Revised: G o t o t h e l a r g e s t

Original: y e ll o w o b
Revised: y e ll o w o b

Original: j e c t
Revised: j e c t

Original: G o t o t h e l a r g e s t
Revised: G o t o t h e l a r g e s t

Original: o b j e c t
Revised: o b j e c t

Original: G o t o t h e s m a ll e s t o b
Revised: G o t o t h e s m a ll e s t o b

Original: j e c t
Revised: j e c t

Original: G o t o t h e s h
Revised: G o t o t h e s h

Original: o r t
Revised: o r t

Original: o
Revised: o

Original: b
Revised: b

Original: j e c t G o t o t
Revised: j e c t G o t o t

Original: h e t a
Revised: h e t a

Original: ll o b
Revised: ll o b

Original: j e c t G o t o n o o b j e c t G o t o t
Revised: j e c t G o t o n o o b j e c t G o t o t

Original: h e
Revised: h e

Original: r e d a r m o r
Revised: r e d a r m o r

Original: G o t o t h e g r e e n a r m 
Revised: G o t o t h e g r ee n a r m o r

Original: o r G o t o t h e a r m o r
Revised: G o t o t h e a r m o r

Original: G o t o t h e s h
Revised: G o t o t h e s h

Original: o r t r e d p ill a r
Revised: o r t r e d p ill a r

Original: G o t o t h e t a ll r e d p ill a r
Revised: G o t o t h e t a ll r e d p ill a r

Original: G o t o t
Revised: G o t o t

Original: h e r e d s h o r t p ill a r
Revised: h e r e d s h o r t p ill a r

Original: G o t o t h e r e d t a ll p ill a r G o t o t h e r e d p ill a r
Revised: G o t o t h e r e d t a ll p ill a r G o t o t h e r e d p ill a r

Original: G o t o t h e t a
Revised: G o t o t h e t a

Original: ll g r ee n p ill a r
Revised: ll g r ee n p ill a r

Original: G o t o t h e s h
Revised: G o t o t h e s h

Original: o r t g r ee n p ill a r
Revised: o r t g r ee n p ill a r

Original: G o t o t
Revised: G o t o t

Original: h e g r ee n t a ll p ill a r
Revised: h e g r ee n t a ll p ill a r

Original: G o t o t h e g r ee n s h
Revised: G o t o t h e g r ee n s h

Original: o r t
Revised: o r t

Original: p ill a r
Revised: p ill a r

Original: G o t o t h e g r ee n p ill a r G o t o t h e p ill a r
Revised: G o t o t h e g r ee n p ill a r G o t o t h e p ill a r

Original: G o t o t h e t a ll p ill a r
Revised: G o t o t h e t a ll p ill a r

Original: G o t o t h e s h
Revised: G o t o t h e s h

Original: o r t p ill a r
Revised: o r t p ill a r

Original: G o t o t h e s h
Revised: G o t o t h e s h

Original: o r t r e d t
Revised: o r t r e d t

Original: o r c h
Revised: o r c h

Original: G o t o t h e t a ll r e d t
Revised: G o t o t h e t a ll r e d t

Original: o r c h
Revised: o r c h

Original: G o t o t
Revised: G o t o t

Original: h e r e d s h o r t t
Revised: h e r e d s h o r t t

Original: o r c h
Revised: o r c h

Original: G o t o t h e r e d t a ll t o r c h
Revised: G o t o t h e r e d t a ll t o r c h

Original: G o t o t
Revised: G o t o t

Original: h e r e d t
Revised: h e r e d t

Original: o r c h
Revised: o r c h

Original: G o t o t h e s h
Revised: G o t o t h e s h

Original: o r t
Revised: o r t

Original: b l u e t
Revised: b l u e t

Original: o r c h
Revised: o r c h

Original: G o t o t h e t a ll b l u e t
Revised: G o t o t h e t a ll b l u e t

Original: o r c h
Revised: o r c h

Original: G o t o t
Revised: G o t o t

Original: h e
Revised: h e

Original: b l u e t a ll t o r c h
Revised: b l u e t a ll t o r c h

Original: G o t o t
Revised: G o t o t

Original: h e
Revised: h e

Original: b l u e s h
Revised: b l u e s h

Original: o r t t
Revised: o r t t

Original: o r c h
Revised: o r c h

Original: G o t o t
Revised: G o t o t

Original: h e
Revised: h e

Original: b l u e t o r c h
Revised: b l u e t o r c h

Original: G o t o t h e t a
Revised: G o t o t h e t a

Original: ll g r ee n t
Revised: ll g r ee n t

Original: o r c h
Revised: o r c h

Original: G o t o t h e s h
Revised: G o t o t h e s h

Original: o r t
Revised: o r t

Original: g r ee n t
Revised: g r ee n t

Original: o r c h
Revised: o r c h

Original: G o t o t
Revised: G o t o t

Original: h e g r ee n t a ll t o r c h
Revised: h e g r ee n t a ll t o r c h

Original: G o t o t h e g r ee n s h
Revised: G o t o t h e g r ee n s h

Original: o r t t
Revised: o r t t

Original: o r c h
Revised: o r c h

Original: G o t o t h e g r e e n t o r c h G o t o t h e t
Revised: G o t o t h e g r ee n t o r c h G o t o t h e t

Original: o r c 
Revised: o r c h

Original: [Table]
Revised: G o t o t h e s h o r t t o r c h G o t o t h e t a ll t o r c h G o t o t

Original: to the green tall torch Go to the
Revised: h e r e d k e y c a r d G o t o t

Original: [Figure]
Revised: h e b l u e k e y c a r d G o t o t h e y e ll o w k e y c

Original: green
Revised: a r d

Original: [Figure]
Revised: G o t o t h e k e y c a r d G o t o t h e r e d s k u ll k e y

Original: Go [Figure]
Revised: G o t o t h e

Original: to the short
Revised: b l u e s k u ll k e y

Original: [Figure]
Revised: G o t o t h e y e ll o w s k u ll k e y

Original: Go to the tall
Revised: G o t o t h e s k u ll k e y

Original: [Figure]
Revised: [Table]

Original: Go to the red keycard Go to the blue keycard Go to the
Revised: K.

Original: [Figure]
Revised: [Figure]

Original: keycard Go to the keycard Go to the red skullkey Go to the blue skullkey Go to the yellow skullkey

Original: Go to the skullkey 0.

Original: J. T -SNE P LOT C OMPARISON

Original: We use t-SNE to visualize the space of instruction embeddings.
Revised: We use t-SNE to visualize the space of instruction embeddings.

Original: In the following ﬁgure, the shape of the datapoint indicates the type of object, such as triangle for armor, diamond for pillar, etc., and the colour indicate the object’s colour, with black being used for when no colour is speciﬁed.
Revised: In the following ﬁgure, the shape of the datapoint indicates the type of object, such as triangle for armor, diamond for pillar, etc., and the colour indicate the object’s colour, with black being used for when no colour is speciﬁed.

Original: Finally the sizeof the data point corresponds to the size indicated in the object, from ’smallest’, ’small’, none speciﬁed, ’large’, and ’largest’. For the instructions with synonym word replacement, we simply leavethe colour as black and the default size
Revised: Finally the size none speciﬁed leave

Original: [Figure]
Revised: [Figure]