Original: R OBUST INSTRUCTION - FOLLOWING IN A SITUATED AGENT VIA TRANSFER - LEARNING FROM TEXT
Revised: R OBUST INSTRUCTION - FOLLOWING IN A SITUATED AGENT VIA TRANSFER - LEARNING FROM TEXT

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recent work has described neural-network-based agents that are trained to execute language-like commands in simulated worlds, as a step towards an intelligent agent or robot that can be instructed by human users.
Revised: Recent work has described neural-network-based agents that are trained to execute language-like commands in simulated worlds, as a step towards an intelligent agent or robot that can be instructed by human users.

Original: However, the instructions that such agents are trained to follow are typically generated from templates (by an environment simulator) and do not reﬂect the varied or ambiguous expressions used by real people.
Revised: However, the instructions that such agents are trained to follow are typically generated from templates (by an environment simulator) and do not reﬂect the varied or ambiguous expressions used by real people.

Original: We address this issue by integrating language encoders that are pretrained on large text corpora into a situated, instruction-following agent.
Revised: We address this issue by integrating language encoders that are pretrained on large text corpora into a situated, instruction-following agent.

Original: In a procedurally-randomized ﬁrst-person 3D world we ﬁrst train agents to follow synthetic instructions requiring the identiﬁcation, manipulation and relative positioning of visually-realistic object models.
Revised: In a procedurally-randomized ﬁrst-person 3D world we ﬁrst train agents to follow synthetic instructions requiring the identiﬁcation, manipulation and relative positioning of visually-realistic object models.

Original: We then show how these abilities can transfer to a context where humans provide instructions in natural language, but only when agents are endowed with language encoding components that were pretrained on text-data.
Revised: We then show how these abilities can transfer to a context where humans provide instructions in natural language, but only when agents are endowed with language encoding components that were pretrained on text-data.

Original: We explore techniques for integrating text-trained and environment-trained components into an agent, observing clear advantages for the fully-contextual phrase representations computed by the well-known BERT model and additional gains by integrating a self-attention operation optimized to adapt BERT’s representations for the agent’s tasks and environment.
Revised: We explore techniques for integrating text-trained and environment-trained components into an agent, observing clear advantages for the fully-contextual phrase representations computed by the well-known BERT model and additional gains by integrating a self-attention operation optimized to adapt BERT’s representations for the agent’s tasks and environment.

Original: These results bridge the gap between two successful strands of recent AI research: agent-centric behavior optimization and text-based representation learning
Revised: These results bridge the gap between two successful strands of recent AI research: agent-centric behavior optimization and text-based representation learning

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Developing machines that can follow natural human commands, particularly those pertaining to an environment shared by both machine and human, is a long-standing and elusive goal of AI (Winograd, 1972).
Revised: Developing machines that can follow natural human commands, particularly those pertaining to an environment shared by both machine and human, is a long-standing and elusive goal of AI (Winograd, 1972).

Original: Recent work has applied end-to-end, representation-learning-based methods to this challenge, where a neural-network-based agent is optimized to process language input, perceive its surroundings and execute appropriate movements jointly (Oh et al., 2017; Hermann et al., 2017;
Revised: Recent work has applied end-to-end, representation-learning-based methods to this challenge, where a neural-network-based agent is optimized to process language input, perceive its surroundings and execute appropriate movements jointly (Oh et al., 2017; Hermann et al., 2017;

Original: Chaplot et al., 2018).
Revised: Chaplot et al., 2018).

Original: End-to-end learning promises a way to deal ﬂexibly with the complexity of the physical, visual and linguistic world without relying on (potentially brittle) hand-crafted features, rules or policies.
Revised: End-to-end learning promises a way to deal ﬂexibly with the complexity of the physical, visual and linguistic world without relying on (potentially brittle) hand-crafted features, rules or policies.

Original: Nevertheless, the cost of this ﬂexibility is the large number of environment interactions (samples) required for a randomly-initialized network to learn behaviour policies from raw experience.
Revised: Nevertheless, the cost of this ﬂexibility is the large number of environment interactions (samples) required for a randomly-initialized network to learn behaviour policies from raw experience.

Original: To make the approach feasible, many studies thus employ a synthetic language that is generated on demand from templates by the environment simulator (Chevalier-Boisvert et al., 2018; Jiang et al., 2019; Yu et al., 2018b;a).
Revised: To make the approach feasible, many studies thus employ a synthetic language that is generated on demand from templates by the environment simulator (Chevalier-Boisvert et al., 2018; Jiang et al., 2019; Yu et al., 2018b;a).

Original: The studies that do combine both end-to-end learning with natural-language data do so in less realistic grid-like environments (Misra et al., 2017) or grant agents access to privileged global observations to make learning more tractable (Misra et al., 2018).
Revised: The studies that do combine both end-to-end learning with natural-language data do so in less realistic grid-like environments (Misra et al., 2017) or grant agents access to privileged global observations to make learning more tractable (Misra et al., 2018).

Original: Here, we take a different learning-based approach to instruction-following that is robust to human commands.
Revised: Here, we take a different learning-based approach to instruction-following that is robust to human commands.

Original: We train agent policies to respond to synthetic, template-based language but also endow them with powerful language encoders that are pretrained on natural language text.
Revised: We train agent policies to respond to synthetic, template-based language but also endow them with powerful language encoders that are pretrained on natural language text.

Original: The synthetic instructions that our agents are trained to follow require mastery of 26 ﬁne-grained motor actions in order to identify and manipulate visually-realistic models from the ShapeNet dataset (Chang et al., 2015) in a 3D room.
Revised: The synthetic instructions that our agents are trained to follow require mastery of 26 ﬁne-grained motor actions in order to identify and manipulate visually-realistic models from the ShapeNet dataset (Chang et al., 2015) in a 3D room.

Original: The visual realism of the world makes it possible to elicit diverse natural human ways of instructing and/or referring to things, and to study the agents’ robustness to this diversity.
Revised: The visual realism of the world makes it possible to elicit diverse natural human ways of instructing and/or referring to things, and to study the agents’ robustness to this diversity.

Original: Unsurprisingly, we ﬁnd that agents that are trained on template-based commands do not cope well with the diversity and variation of natural keyboard-typed language.
Revised: Unsurprisingly, we ﬁnd that agents that are trained on template-based commands do not cope well with the diversity and variation of natural keyboard-typed language.

Original: In contrast, when powerful
Revised: In contrast, when powerful

Original: [Figure]
Revised: [Table]

Revised: Visual

Original: [Table]
Revised: [Table]

Original: interaction manipulation [Figure] language
Revised: interaction manipulation realism language

Original: Oh [Table]
Revised: Oh et al.

Original: Hermann et al.
Revised: Hermann et al.

Original: [Table]
Revised: Chaplot et al.

Original: [Table] et [Table]
Revised: Misra et al. Wu et al.

Original: [Table]
Revised: Jiang et al.

Original: This work pretrained language encoders are integrated 
Revised: This work pretrained language encoders are integrated into our agent, we ﬁnd that agents can satisfy human with

Original: To better analyze the generalization that supports this robustness, we probe trained agents with speciﬁc modiﬁcations to their training instructions.
Revised: To better analyze the generalization that supports this robustness, we probe trained agents with speciﬁc modiﬁcations to their training instructions.

Original: We ﬁnd that methods based on conventional (context-free) word embeddings support generalization that is driven by lexical similarity (executing lift a vehicle when trained to lift a car ), but capture phrasal equivalence less well (failing at put a plate on the container when trained to put the dish on the tray ).
Revised: We ﬁnd that methods based on conventional (context-free) word embeddings support generalization that is driven by lexical similarity (executing lift a vehicle when trained to lift a car ), but capture phrasal equivalence less well (failing at put a plate on the container when trained to put the dish on the tray ).

Original: In contrast, methods that integrate contextual word representations support both types of generalization.
Revised: In contrast, methods that integrate contextual word representations support both types of generalization.

Original: We also ﬁnd that robustness to synonyms can often be improved when pretrained encoders are combined with an additional (cross-modal) self-attention tuned to the agent’s environmental objectives.
Revised: We also ﬁnd that robustness to synonyms can often be improved when pretrained encoders are combined with an additional (cross-modal) self-attention tuned to the agent’s environmental objectives.

Original: Ablation experiments highlight the role that WordPiece tokenization (Schuster & Nakajima, 2012) plays in robustness to human instructions.
Revised: Ablation experiments highlight the role that WordPiece tokenization (Schuster & Nakajima, 2012) plays in robustness to human instructions.

Original: This motivates the addition of typo noise to our training pipeline, which further improves the accuracy of the agent’s responses.
Revised: This motivates the addition of typo noise to our training pipeline, which further improves the accuracy of the agent’s responses.

Original: Overall, our principal contributions are the following:
Revised: Overall, our principal contributions are the following:

Original: • We train an agent that can  interpret human language commands and overcomes more behavioural and environmental challenges than previous work (Table 1).
Revised: • We train an agent that can both interpret human language commands and overcome a similar range of behavioural and environmental challenges to state-of-the-art policy-learning approaches (Table 1).

Original: • We develop the techniques of transfer-learning from a text representation-learning model to an embodied agent.
Revised: • We develop the techniques of transfer-learning from a text representation-learning model to an embodied agent.

Original: Note that this is very different from transfer between
Revised: Note that this is very different from transfer between

Original: language
Revised: language

Original: classiﬁcation tasks (
Revised: classiﬁcation tasks (

Original: Collobert
Revised: Collobert

Original: Weston
Revised: Weston

Original: Devlin et al
Revised: Devlin et al

Original: ); in our evaluations the agent must interpret unfamiliar instructions zero-shot (i
Revised: ); in our evaluations the agent must interpret unfamiliar instructions zero-shot (i

Original: e. without additional learning steps), and must realize complex language-conditional behaviours (decoding ≈
Revised: e. without additional learning steps), and must realize complex language-conditional behaviours (decoding ≈

Original: appropriate
Revised: appropriate

Original: actions) in spite of this uncertainty.
Revised: actions) in spite of this uncertainty.

Original: 1. H IGH - LEVEL A PPROACH
Revised: 1. H IGH - LEVEL A PPROACH

Original: We experiment in a simulated room in the Unity 1 game engine (Fig 1).
Revised: We experiment in a simulated room in the Unity 1 game engine (Fig 1).

Original: Importantly, the realism of the visual assets in this room enable human observers to recognize the objects (and hence refer to them in natural ways).
Revised: Importantly, the realism of the visual assets in this room enable human observers to recognize the objects (and hence refer to them in natural ways).

Original: Our experimental pipeline then involves the following steps:
Revised: Our experimental pipeline then involves the following steps:

Original: Deﬁne a (language-dependent) task, including initial conditions for the simulated room, a template-based process for generating synthetic language instructions and a reward function for checking game states and exposing reward if the agent succeeds.
Revised: Deﬁne a (language-dependent) task, including initial conditions for the simulated room, a template-based process for generating synthetic language instructions and a reward function for checking game states and exposing reward if the agent succeeds.

Original: Train agents with different language encoders to solve the task when conditioned on the synthetic language produced by the environment generator. 3.
Revised: Train agents with different language encoders to solve the task when conditioned on the synthetic language produced by the environment generator. 3.

Original: Evaluate the trained agent on 1000 episodes of the task with the synthetic instructions replaced with synonyms or free-form commands typed by human annotators.
Revised: Evaluate the trained agent on 1000 episodes of the task with the synthetic instructions replaced with synonyms or free-form commands typed by human annotators.

Original:  A RCHITECTURES FOR INSTRUCTION - FOLLOWING AGENTS
Revised:  A RCHITECTURES FOR INSTRUCTION - FOLLOWING AGENTS

Original: Since the object of our study is instruction-following, we consider an agent architecture that is as conventional as possible in aspects not related to language processing.
Revised: Since the object of our study is instruction-following, we consider an agent architecture that is as conventional as possible in aspects not related to language processing.

Original: It consists of standard components for computing visual representations of pixels, embedding text strings and predicting actions contingent on these and memory of past inputs.
Revised: It consists of standard components for computing visual representations of pixels, embedding text strings and predicting actions contingent on these and memory of past inputs.

Original: Visual processing The visual observations received by the agent at each timestep are 96 × 72 ×real-valued tensors, which are processed by a 3-layer residual convnet.
Revised: Visual processing The visual observations received by the agent at each timestep are 96 × 72 ×real-valued tensors, which are processed by a 3-layer residual convnet.

Original: Memory core The output of the visual processing is combined with language information according to a particular encoding strategy, as described below.
Revised: Memory core The output of the visual processing is combined with language information according to a particular encoding strategy, as described below.

Original: In all conditions, some combination of vision and language input at each timestep passes into a LSTM memory core with hidden dim.
Revised: In all conditions, some combination of vision and language input at each timestep passes into a LSTM memory core with hidden dim.

Original: Action and value prediction The state of the memory core at each timestep is passed through a linear layer and softmax to compute a distribution over 26 actions.
Revised: Action and value prediction The state of the memory core at each timestep is passed through a linear layer and softmax to compute a distribution over 26 actions.

Original: Independently, the memory state is passed to a linear layer to yield a scalar value prediction.
Revised: Independently, the memory state is passed to a linear layer to yield a scalar value prediction.

Original: Training algorithm
Revised: Training algorithm

Original: The agent is trained using an importance-weighted actor-critic algorithm with a central learner and distributed actors (Espeholt et al., 2018).
Revised: The agent is trained using an importance-weighted actor-critic algorithm with a central learner and distributed actors (Espeholt et al., 2018).

Original: 2. L ANGUAGE ENCODING
Revised: 2. L ANGUAGE ENCODING

Original: The agent must process both string observations in the simulated environment, during training, and human instructions, also encoded as strings, during evaluation.
Revised: The agent must process both string observations in the simulated environment, during training, and human instructions, also encoded as strings, during evaluation.

Original: Its representations of language must be combined with visual information to make decisions about how to act.
Revised: Its representations of language must be combined with visual information to make decisions about how to act.

Original: We compare various different ways of achieving this encoding.
Revised: We compare various different ways of achieving this encoding.

Original: For methods that transfer knowledge from unsupervised text-based learning, we take weights from the well-known BERT model (Devlin et al., 2018), specifically the uncased BERT BASE model made available by the authors.
Revised: For methods that transfer knowledge from unsupervised text-based learning, we take weights from the well-known BERT model (Devlin et al., 2018), specifically the uncased BERT BASE model made available by the authors.

Original: BERT + mean pool For a given input of w words, BERT BASE returns w context-dependent (sub)- word representations of 768 units.
Revised: BERT + mean pool For a given input of w words, BERT BASE returns w context-dependent (sub)- word representations of 768 units.

Original: In this condition, a mean pooling operation is applied over the w dimension to yield a single representation of dimension 768 , which is concatenated with the ﬂattened output of the visual processing module.
Revised: In this condition, a mean pooling operation is applied over the w dimension to yield a single representation of dimension 768 , which is concatenated with the ﬂattened output of the visual processing module.

Original: This multi-modal representation is passed through a single layer MLP with tanh activation and output dimension 128 before entering the memory core of the agent.
Revised: This multi-modal representation is passed through a single layer MLP with tanh activation and output dimension 128 before entering the memory core of the agent.

Original: We apply the standard BERT BASE WordPiece(subword) vocabulary (of size 30,000). 
Revised: We apply the standard BERT BASE WordPiece vocabulary (of size 30,000). Note Unity See

Revised: Available at https://tfhub.dev/google/bert_uncased_L-12_H-768_A-12/  that WordPiece encodes language in terms of subwords (a mix of characters, common word chunks, morphemes and words) rather than the word-level vocabulary applied in more traditional neural language models.

Original: BERT + self-attention layer When applying BERT to text classiﬁcation, performance can sometimes be improved by ﬁne-tuning the weights in the BERT encoder to suit the task.
Revised: BERT + self-attention layer When applying BERT to text classiﬁcation, performance can sometimes be improved by ﬁne-tuning the weights in the BERT encoder to suit the task.

Original: Because of the large number of gradient updates required to learn a complex behaviour policy, ﬁne-tuning the BERT weights in this way would cause substantial overﬁtting to the synthetic environment language.
Revised: Because of the large number of gradient updates required to learn a complex behaviour policy, ﬁne-tuning the BERT weights in this way would cause substantial overﬁtting to the synthetic environment language.

Original: We therefore keep the BERT weights frozen, but experiment with an additional learned self-attention layer (Vaswani et al., 2017) to create a language encoder whose bottom layers are pretrained and whose ﬁnal layer optimizes these representations to the present environment or tasks.
Revised: We therefore keep the BERT weights frozen, but experiment with an additional learned self-attention layer (Vaswani et al., 2017) to create a language encoder whose bottom layers are pretrained and whose ﬁnal layer optimizes these representations to the present environment or tasks.

Original: This additional layer has 4 attention heads, and uses 64 dimensional key and value embeddings.
Revised: This additional layer has 4 attention heads, and uses 64 dimensional key and value embeddings.

Original: BERT+ cross-modal self-attention We also consider a cross-modal self-attention layer of the type suggested by e.g. Tsai et al.
Revised: BERT+ cross-modal self-attention We also consider a cross-modal self-attention layer of the type suggested by e.g. Tsai et al.

Original: ; Lu et al.
Revised: ; Lu et al.

Original: (2019), which in our application provides an explicit pathway for the agent to bind visual experience to speciﬁc (contextual) word representations.
Revised: (2019), which in our application provides an explicit pathway for the agent to bind visual experience to speciﬁc (contextual) word representations.

Original: In this case, we treat each of the output channels of the visual processing network as word-like entities, by passing them through a linear layer of output size 768 to match the BERT output.
Revised: In this case, we treat each of the output channels of the visual processing network as word-like entities, by passing them through a linear layer of output size 768 to match the BERT output.

Original: The embeddings for all words and the visual channels are then processed with a single self-attention layer, whose parameters are again learned with all other agent parameters.
Revised: The embeddings for all words and the visual channels are then processed with a single self-attention layer, whose parameters are again learned with all other agent parameters.

Original: Pretrained (sub)-word embeddings Language classiﬁcation experiments with BERT show the value of highly context-dependent (sub)-word representations, but transfer in that context is also possible with more conventional (context-independent) word embeddings (Collobert & Weston, 2008).
Revised: Pretrained (sub)-word embeddings Language classiﬁcation experiments with BERT show the value of highly context-dependent (sub)-word representations, but transfer in that context is also possible with more conventional (context-independent) word embeddings (Collobert & Weston, 2008).

Original: To measure the effect of this distinction, we consider a simpler encoder based on the (context independent) input (sub-)word embeddings from BERT, which are also of dimension 768 .
Revised: To measure the effect of this distinction, we consider a simpler encoder based on the (context independent) input (sub-)word embeddings from BERT, which are also of dimension 768 .

Original: 4 Taking a mean of these context-independent vectors would yield a word-order-invariant representation of language.
Revised: 4 Taking a mean of these context-independent vectors would yield a word-order-invariant representation of language.

Original: We therefore process them with single-layer transformer with 4 attention heads.
Revised: We therefore process them with single-layer transformer with 4 attention heads.

Original: As in
Revised: As in

Original: BERT + mean pool , this output is averaged, reduced by a 
Revised: BERT + mean pool , this output is averaged, reduced by a single-layer MLP (from 768 to 128 units) and passed to the agent’s core memory.

Original: Typo noise Finally, in one condition we introduce typo noise (each key character was replaced with that of an adjacent character on a standard keyboard, with a small probability) to the synthetic language strings produced by the environment, allowing the agent to learn to compensate during training.
Revised: Typo noise Finally, in one condition we introduce typo noise (each key character was replaced with that of an adjacent character on a standard keyboard, with a probability of 0 . 01 ) to the synthetic language strings produced by the environment, allowing the agent to learn to compensate during training.

Original: No modiﬁcations were made to the evaluation stimuli.
Revised: No modiﬁcations were made to the evaluation stimuli.

Original: Much previous work applies typing noise for language classiﬁer robustness; see e.g. (Pruthi et al., 2019) for a recent survey.
Revised: Much previous work applies typing noise for language classiﬁer robustness; see e.g. (Pruthi et al., 2019) for a recent survey.

Original: We compare to various baselines designed to isolate speciﬁc components of these encoders.
Revised: We compare to various baselines designed to isolate speciﬁc components of these encoders.

Original: Random mean pool As a direct baseline for BERT mean pool , we consider an identical architecture but in which the contextual BERT embeddings are replaced by ﬁxed random (subword-speciﬁc) vectors (also of dimension 768 ).
Revised: Random mean pool As a direct baseline for BERT mean pool , we consider an identical architecture but in which the contextual BERT embeddings are replaced by ﬁxed random (subword-speciﬁc) vectors (also of dimension 768 ).

Original: The weights in the rest of the agent network are trained as in other conditions.
Revised: The weights in the rest of the agent network are trained as in other conditions.

Original: We note that random sentence vectors are a competitive baseline on many language classiﬁcation tasks (Wieting & Kiela, 2019).
Revised: We note that random sentence vectors are a competitive baseline on many language classiﬁcation tasks (Wieting & Kiela, 2019).

Original: Word-level and WordPiece transformers
Revised: Word-level and WordPiece transformers

Original: In addition to pre-trained weights, a potentially important aspect of encoders based on BERT is WordPiece tokenization, which can afford greater ability to make sense of typos or rare words with familiar morphemes than in a more conventional wordlevel encoder.
Revised: In addition to pre-trained weights, a potentially important aspect of encoders based on BERT is WordPiece tokenization, which can afford greater ability to make sense of typos or rare words with familiar morphemes than in a more conventional wordlevel encoder.

Original: To isolate the effect of tokenization from that of pretrained weights, we compared two further encoders.
Revised: To isolate the effect of tokenization from that of pretrained weights, we compared two further encoders.

Original: In one, we split all input strings by whitespace and hash each word string to a unique index representing an input to a single-layer transformer with 4 attention heads and embedding size 768 (chosen to match BERT BASE ), the output of which is processed identically to the
Revised: In one, we split all input strings by whitespace and hash each word string to a unique index representing an input to a single-layer transformer with 4 attention heads and embedding size 768 (chosen to match BERT BASE ), the output of which is processed identically to the

Original: Random mean pool condition.
Revised: Random mean pool condition.

Original: We contrast this with an otherwise identical condition in which the
Revised: We contrast this with an otherwise identical condition in which the

Original: WordPiece tokenization from BERT is applied rather than splitting on white space.
Revised: WordPiece tokenization from BERT is applied rather than splitting on white space.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We experiment with a lifting task , which focuses on object identiﬁcation, and a putting task which focuses on object relations and manipulation.
Revised: We experiment with a lifting task , which focuses on object identiﬁcation, and a putting task which focuses on object relations and manipulation.

Original: In both tasks the locations of all objects and the initial position and orientation of the agent are chosen at random on ﬂoating point scales, so it is highly unlikely that any two episodes are spatially identical (whether during training or evaluation).
Revised: In both tasks the locations of all objects and the initial position and orientation of the agent are chosen at random on ﬂoating point scales, so it is highly unlikely that any two episodes are spatially identical (whether during training or evaluation).

Original: The action set consists of movements ( move- { forward,back,left,right } ), turns ( turn- { up,down,left,right } ), and object pick-up and manipulation with 4 DoF.
Revised: The action set consists of movements ( move- { forward,back,left,right } ), turns ( turn- { up,down,left,right } ), and object pick-up and manipulation with 4 DoF.

Original: The placement of objects is assisted by a visible vertical line.
Revised: The placement of objects is assisted by a visible vertical line.

Original: See Appendix C a full description.
Revised: See Appendix C a full description.

Original: 3. L IFTING WITH HUMAN INSTRUCTIONS
Revised: 3. L IFTING WITH HUMAN INSTRUCTIONS

Original: In an episode of the lifting task, the environment selects two movable objects at random from a global object set, and generates a language instruction of the form Lift a X , where X is the correct name for (exactly) one of those objects.
Revised: In an episode of the lifting task, the environment selects two movable objects at random from a global object set, and generates a language instruction of the form Lift a X , where X is the correct name for (exactly) one of those objects.

Original: To achieve reward, the agent must locate the X , lift it more than 1m above the ground and keep it there for 5 time steps.
Revised: To achieve reward, the agent must locate the X , lift it more than 1m above the ground and keep it there for 5 time steps.

Original: If the reward function in the environment determines that these conditions are met, a positive reward of +1 is emitted and the episode ends.
Revised: If the reward function in the environment determines that these conditions are met, a positive reward of +1 is emitted and the episode ends.

Original: If a lifting takes place with the incorrect object, the episode also ends with reward 0 .
Revised: If a lifting takes place with the incorrect object, the episode also ends with reward 0 .

Original: For a global set of recognisable objects, we use the ShapeNet dataset (Chang et al., 2015), which contains over 12, 3D rendered models.
Revised: For a global set of recognisable objects, we use the ShapeNet dataset (Chang et al., 2015), which contains over 12, 3D rendered models.

Original: For performance reasons we discard models with more than 8,000 vertices or an OBJ ﬁle size greater than 1 MB and consider only those tagged into a synset in the WordNet taxonomy.
Revised: For performance reasons we discard models with more than 8,000 vertices or an OBJ ﬁle size greater than 1 MB and consider only those tagged into a synset in the WordNet taxonomy.

Original: We use the name of the ﬁrst lemma of that synset as a proxy for the name of the object.
Revised: We use the name of the ﬁrst lemma of that synset as a proxy for the name of the object.

Original: From these names, we selected 80 to for the environment, each referring to a set of object models with a minimum of 12 exemplars.
Revised: From these names, we selected 80 to for the environment, each referring to a set of object models with a minimum of 12 exemplars.

Original: See appendix D for more details.
Revised: See appendix D for more details.

Original: Template lang.
Revised: [Table]

Original: Natural (training) Synonym referring expression Lift a ﬂag
Revised: banner

Original: Lift a
Revised: [Table]

Original: Lift the indian ﬂag Lift a ﬂag banner Lift the ﬂag.
Revised: [Table]

Original: Lift a pillow
Revised: cushion

Original: Lift a Lift a pillows Lift a cushion cushion Lift a paper
Revised: [Table]

Original: For all language encoding strategies described above, the agent was able to learn the task, and a well-trained policy completed episodes in an average of ≈ 20 timesteps with accuracy around 90%.
Revised: For all language encoding strategies described above, the agent was able to learn the task, and a well-trained policy completed episodes in an average of ≈ 20 timesteps with accuracy around 90%.

Original: The failure of agents to reach perfect performance on the training set is unimportant for the present study; we suspect that the agent’s comparatively small convolutional network fails to perceive important distinctions between the more intricate ShapeNet models.
Revised: The failure of agents to reach perfect performance on the training set is unimportant for the present study; we suspect that the agent’s comparatively small convolutional network fails to perceive important distinctions between the more intricate ShapeNet models.

Original: [Table]
Revised: [Table]

Original: We consider two evaluation settings.
Revised: We consider two evaluation settings.

Original: In the synonym evaluation, we ran the environment for 1, episodes with the noun in the environment template instruction replaced by a synonym ( Lift a X becomes Lift a X ∗ where X ≈ X ∗ ).
Revised: In the synonym evaluation, we ran the environment for 1, episodes with the noun in the environment template instruction replaced by a synonym ( Lift a X becomes Lift a X ∗ where X ≈ X ∗ ).

Original: The synonyms were provided by native English speaking subjects.
Revised: The synonyms were provided by native English speaking subjects.

Original: In the natural referring expression evaluation, we gave 40 annotators access to a room containing a single ShapeNet model via a crowd-sourcing platform.
Revised: In the natural referring expression evaluation, we gave 40 annotators access to a room containing a single ShapeNet model via a crowd-sourcing platform.

Original: We asked them to write down what they found in the room and then hit a button that restarted the environment with a new object in the room.
Revised: We asked them to write down what they found in the room and then hit a button that restarted the environment with a new object in the room.

Original: As illustrated in Table 2, unlike the synonyms, the natural referring expressions involve variation in articles as well as nouns ( a pencil might become the pen ), may include spelling mistakes or typos, can refer entirely incorrectly to the intended object (if the subject fails to recognize the ShapeNet model), but may also match the training instruction exactly.
Revised: As illustrated in Table 2, unlike the synonyms, the natural referring expressions involve variation in articles as well as nouns ( a pencil might become the pen ), may include spelling mistakes or typos, can refer entirely incorrectly to the intended object (if the subject fails to recognize the ShapeNet model), but may also match the training instruction exactly.

Original: Moreover, unlike the synonym test, there are 30 − 40 natural referring expressions for each of the 80 environment nouns, from which we sample randomly when evaluating the agent (again on 1,000 evaluation espisodes).
Revised: Moreover, unlike the synonym test, there are 30 − 40 natural referring expressions for each of the 80 environment nouns, from which we sample randomly when evaluating the agent (again on 1,000 evaluation espisodes).

Original: 3. P UTTING OBJECTS ON OTHER OBJECTS WITH HUMAN INSTRUCTIONS
Revised: 3. P UTTING OBJECTS ON OTHER OBJECTS WITH HUMAN INSTRUCTIONS

Original: The strength of models like BERT is their ability to combine lexical representations into phrasal or sentence representations.
Revised: The strength of models like BERT is their ability to combine lexical representations into phrasal or sentence representations.

Original: To study this capacity in the context of instruction-following, we devised a putting task involving the verb ‘to put’, which, in the imperative ( put the cup on the tray ) takes two arguments, the direct object (D.O.) cup and the indirect object (I.O.) tray .
Revised: To study this capacity in the context of instruction-following, we devised a putting task involving the verb ‘to put’, which, in the imperative ( put the cup on the tray ) takes two arguments, the direct object (D.O.) cup and the indirect object (I.O.) tray .

Original: In terms of the behaviours required, the putting task focuses on manipulation and object relations rather than object identiﬁcation or reference.
Revised: In terms of the behaviours required, the putting task focuses on manipulation and object relations rather than object identiﬁcation or reference.

Original: The environment was conﬁgured to begin each episode with three randomly-chosen moveable objects and two larger immovable objects (a bed and a tray), each randomly positioned in the room.
Revised: The environment was conﬁgured to begin each episode with three randomly-chosen moveable objects and two larger immovable objects (a bed and a tray), each randomly positioned in the room.

Original: In each episode of this task, the agent receives an instruction Put a D.O. on the I.O. , where D.O. is any of the three moveable objects (chosen from a global set of ten) and I.O. is either bed or tray .
Revised: In each episode of this task, the agent receives an instruction Put a D.O. on the I.O. , where D.O. is any of the three moveable objects (chosen from a global set of ten) and I.O. is either bed or tray .

Original: The environment checks whether an instance of D.O. is at rest (and not held by the agent) on top of the I.O. , returning a positive reward +1 if so and ending the episode.
Revised: The environment checks whether an instance of D.O. is at rest (and not held by the agent) on top of the I.O. , returning a positive reward +1 if so and ending the episode.

Original: If the object D.O. is placed on something other than a I.O. , or if another movable object is placed on the bed or the tray then again the episode ends immediately with reward 0 .
Revised: If the object D.O. is placed on something other than a I.O. , or if another movable object is placed on the bed or the tray then again the episode ends immediately with reward 0 .

Original: As before, we ﬁrst trained all agents on the putting task with synthetic environment language instructions.
Revised: As before, we ﬁrst trained all agents on the putting task with synthetic environment language instructions.

Original: Training a policy on this task with reinforcement learning required a bespoke task curriculum (see Appendix C for details); a well-trained policy completes each episode in an average of ≈actions/timesteps.
Revised: Training a policy on this task with reinforcement learning required a bespoke task curriculum (see Appendix C for details); a well-trained policy completes each episode in an average of ≈actions/timesteps.

Original: To gather the evaluation stimuli, we again crowd-sourced humans to provide both natural synonyms for each of the 12 objects in the global set for this task and, in this case, entirely free-form natural human instructions.
Revised: To gather the evaluation stimuli, we again crowd-sourced humans to provide both natural synonyms for each of the 12 objects in the global set for this task and, in this case, entirely free-form natural human instructions.

Original: To obtain natural instructions, we instantiated an environment with only one of the global set of moveable objects, coloured red, and one of either the bed or the tray, coloured white, and asked subjects to ask somebody to place the red object on top of the white object without mentioning their color .
Revised: To obtain natural instructions, we instantiated an environment with only one of the global set of moveable objects, coloured red, and one of either the bed or the tray, coloured white, and asked subjects to ask somebody to place the red object on top of the white object without mentioning their color .

Original: From these instructions, we deﬁned four evaluations, illustrated in Table 3: D.O. synonym , I.O. synonym and D.O. & I.O. synonym , in which particular parts of the original template command were replaced with synonyms, and Natural instruction , the fully free-form human instruction, which can include orthographic errors and misidentiﬁed objects.
Revised: From these instructions, we deﬁned four evaluations, illustrated in Table 3: D.O. synonym , I.O. synonym and D.O. & I.O. synonym , in which particular parts of the original template command were replaced with synonyms [Table] and Natural instruction , the fully free-form human instruction, which can include orthographic errors and misidentiﬁed objects [Table]

Original: 3. D ISCUSSION OF RESULTS
Revised: 3. D ISCUSSION OF RESULTS

Original: [Table]
Revised: [Table]

Original: The accuracies for both lifting tasks are presented in Table 4 and for putting tasks in Table 5.
Revised: The accuracies for both lifting tasks are presented in Table 4 and for putting tasks in Table 5.

Original: The results reveal the following main effects of language encoding on model performance:
Revised: The results reveal the following main effects of language encoding on model performance:

Original: Substantial transfer from text requires contextual encoders Agents with weights that are pretrained on text data exhibit substantially higher accuracy on both the lifting and the putting tasks.
Revised: Substantial transfer from text requires contextual encoders Agents with weights that are pretrained on text data exhibit substantially higher accuracy on both the lifting and the putting tasks.

Original: This effect is greatest in the more focused synonym evaluations, but also holds for the the freeform human instructions.
Revised: This effect is greatest in the more focused synonym evaluations, but also holds for the the freeform human instructions.

Original: A small transfer effect can be seen by comparing the word embeddings + Transformer condition (62% accuracy on the synonym evaluation, lifting task and 57% accuracy  on the D.O. synonym evaluation, putting task) with the WordPiece Transformer (57% and 20%).
Revised: A small transfer effect can be seen by comparing the word embeddings + Transformer condition (62% accuracy on the synonym evaluation, lifting task and 57% accuracy [Table] show mean across 1,000 random episodes. on the D.O. synonym evaluation, putting task) with the WordPiece Transformer (57% and 20%).

Original: However, overall the transfer effect is much stronger in the case of the full context-dependent BERT representations.
Revised: However, overall the transfer effect is much stronger in the case of the full context-dependent BERT representations.

Original: On the same two evaluations, BERT + mean pool achieves 77% and 94% accuracy 
Revised: On the same two evaluations, BERT + mean pool achieves 77% and 94% accuracy respectively.

Original: [Table]
Revised: The gains from transferring via BERT representations vs. just (sub)word embeddings are greatest for the (longer) putting instructions than for the lifting instructions, and greatest of all in the D.O & I.O. synonym evaluation . These are cases where one would expect the marginal value of powerful sentential (rather than just lexical) representations should be greatest.

Original: Tuning via self-attention layers (with typo noise) helps Interestingly, we ﬁnd that tuned self-attention layers do not improve generalization performance over using BERT and mean pooling.
Revised: Tuning via self-attention layers (with typo noise) helps Interestingly, we ﬁnd that tuned self-attention layers do not improve generalization performance over using BERT and mean pooling.

Original: This may be simply because the additional layers cause a degree of overﬁtting to the template environment language during training.
Revised: This may be simply because the additional layers cause a degree of overﬁtting to the template environment language during training.

Original: However, typo noise mitigates this issue, so that the strongest evaluation performance on the putting tasks overall is observed with a combination of a tuned cross-modal self-attention layer and typo-noise training.
Revised: However, typo noise mitigates this issue, so that the strongest evaluation performance on the putting tasks overall is observed with a combination of a tuned cross-modal self-attention layer and typo-noise training.

Original: Indeed, the value of typo noise as a regularizer can be seen by the fact that it improves the robustness of agents with tuned self-attention layers even in the synonym evaluations (for both lifting and putting), which do not involve any typos.
Revised: Indeed, the value of typo noise as a regularizer can be seen by the fact that it improves the robustness of agents with tuned self-attention layers even in the synonym evaluations (for both lifting and putting), which do not involve any typos.

Original: Thus, the BERT + CMSA + TN model performs better than all others on two of the three synonym evaluations in the putting task.
Revised: Thus, the BERT + CMSA + TN model performs better than all others on two of the three synonym evaluations in the putting task.

Original: One way in which the appropriately-tuned self-attention layer might make the agent more robust to synonyms in this case is by spreading out task-relevant object-nouns in the agent’s language representation space (leaving those words closer to synonyms than to potential confounding words).
Revised: One way in which the appropriately-tuned self-attention layer might make the agent more robust to synonyms in this case is by spreading out task-relevant object-nouns in the agent’s language representation space (leaving those words closer to synonyms than to potential confounding words).

Original: The degree to which this happens when the object-nouns and their synonymns in our environments are passed ﬁrst through BERT and then through an agent-tuned self-attention layer (compared to passing through the same layer but with random weights) is shown in Fig 2.
Revised: The degree to which this happens when the object-nouns and their synonymns in our environments are passed ﬁrst through BERT and then through a (BERT + SA) agent-tuned self-attention layer (compared to passing through the same layer but with random weights) is shown in Fig 2.

Original: WordPiece tokenization adds robustness
Revised: WordPiece tokenization adds robustness

Original: In the two evaluations involving natural language instructions from humans, a comparison of Word-level Transformer and WordPiece Transformer shows that some robustness is obtained simply from WordPiece encoding, which in turn must play some part in all BERT-based conditions (e.g. improving from 61 % to 66 % in the natural referring expression evaluation and from 36 % to 42 % in the natural instruction putting evaluation).
Revised: In the two evaluations involving natural language instructions from humans, a comparison of Word-level Transformer and WordPiece Transformer shows that some robustness is obtained simply from WordPiece encoding, which in turn must play some part in all BERT-based conditions (e.g. improving from 61 % to 66 % in the natural referring expression evaluation and from 36 % to 42 % in the natural instruction putting evaluation).

Original: As mentioned above, BERT-based agents with WordPiece encoding are particularly robust to human instructions when trained with typo noise, and this is most effective when combined with tuned self-attention layers.
Revised: As mentioned above, BERT-based agents with WordPiece encoding are particularly robust to human instructions when trained with typo noise, and this is most effective when combined with tuned self-attention layers.

Original: This makes intuitive sense, as a learnable self-attention layer should provide the agent with more ﬂexibility to learn to correct for typos during training.
Revised: This makes intuitive sense, as a learnable self-attention layer should provide the agent with more ﬂexibility to learn to correct for typos during training.

Original: Indeed, on the natural instruction evaluation of the putting task, where typos or spelling errors are most common, the cross- modal attention agent trained with typo noise achieves 70% accuracy.
Revised: Indeed, on the natural instruction evaluation of the putting task, where typos or spelling errors are most common, the crossmodal attention agent trained with typo noise achieves 70% accuracy.

Original: Note that 100% accuracy on this evaluation may be impossible, even for humans, because visual ambiguity or human error mean that instructions can sometimes refer in entirely mistaken ways to the objects in the room.
Revised: Note that 100% accuracy on this evaluation may be impossible, even for humans, because visual ambiguity or human error mean that instructions can sometimes refer in entirely mistaken ways to the objects in the room.

Revised: Multitask learning is possible The performance of an agent trained on both the putting and lifting tasks is not substantially lower than agents that are specialized to each of the tasks individually.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Most closely related to our work is an experiment by Chan et al.
Revised: Most closely related to our work is an experiment by Chan et al.

Original: (2019), who showed how an agent trained with InferLite sentence representations (Kiros & Chan, 2018) can be robust to synonym replacements in template instructions.
Revised: (2019), who showed how an agent trained with InferLite sentence representations (Kiros & Chan, 2018) can be robust to synonym replacements in template instructions.

Original: The task itself involves object identiﬁcation in the VizDoom environment (Kempka et al., 2016), which requires only 3 motor actions.
Revised: The task itself involves object identiﬁcation in the VizDoom environment (Kempka et al., 2016), which requires only 3 motor actions.

Original: Our work develops this insight substantially, applying a similar approach to a visually-realistic environment requiring ﬁnegrained object manipulation, integrating context-dependent pre-trained models with sub-word tokenization (BERT), analysing architectures and training strategies for integrating such models and extending from synonym replacements to free-form instructions typed by humans.
Revised: Our work develops this insight substantially, applying a similar approach to a visually-realistic environment requiring ﬁnegrained object manipulation, integrating context-dependent pre-trained models with subword tokenization (BERT), analysing architectures and training strategies for integrating such models and extending from synonym replacements to free-form instructions typed by humans.

Original: Much recent work applies deep learning and policy optimization in end-to-end approaches to learning instruction following (Chaplot et al., 2018;
Revised: Much recent work applies deep learning and policy optimization in end-to-end approaches to learning instruction following (Chaplot et al., 2018;

Original: Oh et al., 2017; Bahdanau et al., 2018; ChevalierBoisvert et al., 2018; Yu et al., 2018b;a; Jiang et al., 2019).
Revised: Oh et al., 2017; Bahdanau et al., 2018; ChevalierBoisvert et al., 2018; Yu et al., 2018b;a; Jiang et al., 2019).

Original: As noted in the introduction, these studies do not typically involve natural language.
Revised: As noted in the introduction, these studies do not typically involve natural language.

Original: Our work differs from studies that do (Misra et al., 2017; 2018) in several ways.
Revised: Our work differs from studies that do (Misra et al., 2017; 2018) in several ways.

Original: First, we consider generalization and robustness (rather than performance when trained on a set of natural language commands).
Revised: First, we consider generalization and robustness (rather than performance when trained on a set of natural language commands).

Original: Second, we apply our method to learn much more complex policies requiring object manipulation and positioning and mastery of ﬁne-grained action sets.
Revised: Second, we apply our method to learn much more complex policies requiring object manipulation and positioning and mastery of ﬁne-grained action sets.

Original: Our approach makes learning such policies possible because we combine (potentially inﬁnite) template-based RL training with text-based pretraining and transfer.
Revised: Our approach makes learning such policies possible because we combine (potentially inﬁnite) template-based RL training with text-based pretraining and transfer.

Original: A limitation of all studies above is the reliance on simulation.
Revised: A limitation of all studies above is the reliance on simulation.

Original: Both object identiﬁcation and manipulation are likely far harder in reality, and it remains to be seen whether our methods scale to robot language understanding (Tellex et al., 2011; 2012; Walter et al., 2014).
Revised: Both object identiﬁcation and manipulation are likely far harder in reality, and it remains to be seen whether our methods scale to robot language understanding (Tellex et al., 2011; 2012; Walter et al., 2014).

Original: See also (Anderson et al., 2018;
Revised: See also (Anderson et al., 2018;

Original: Wang et al., 2019) for recent improvements to visual realism in simulated environments.
Revised: Wang et al., 2019) for recent improvements to visual realism in simulated environments.

Original: Finally, there is a long history of building in knowledge about the structure of language and/or its environment into instruction-following systems rather than learning it end-to-end.
Revised: Finally, there is a long history of building in knowledge about the structure of language and/or its environment into instruction-following systems rather than learning it end-to-end.

Original: In Winograd (1972)’s SHRDLU, syntactic modules parsed the language input into a logical form, and handwritten rules were applied to connect such forms to the environment.
Revised: In Winograd (1972)’s SHRDLU, syntactic modules parsed the language input into a logical form, and handwritten rules were applied to connect such forms to the environment.

Original: More recent pipeline-based approaches use learning algorithms to map language to a program that can then interface with a planner (Chen & Mooney, 2011; Matuszek et al., 2013; Wang et al., 2016), and/or a controller, both of which may have priviledged information about how the world connects to the program.
Revised: More recent pipeline-based approaches use learning algorithms to map language to a program that can then interface with a planner (Chen & Mooney, 2011; Matuszek et al., 2013; Wang et al., 2016), and/or a controller, both of which may have priviledged information about how the world connects to the program.

Original: It is likely that pretrained language encoders could add robustness to parts of these approaches, much as they do here.
Revised: It is likely that pretrained language encoders could add robustness to parts of these approaches, much as they do here.

Original: Our focus on end-to-end learning, however, is motivated by the intuition that it may eventually scale or adapt more ﬂexibly to arbitrary environments or problems than pipeline approaches.
Revised: Our focus on end-to-end learning, however, is motivated by the intuition that it may eventually scale or adapt more ﬂexibly to arbitrary environments or problems than pipeline approaches.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this work, we have developed an agent that can follow natural human instructions requiring the identiﬁcation, manipulation and positioning of visually-realistic assets.
Revised: In this work, we have developed an agent that can follow natural human instructions requiring the identiﬁcation, manipulation and positioning of visually-realistic assets.

Original: Our method relies on zeroshot transfer from template language instructions to those given by human annotators when asked to refer and instruct in natural ways.
Revised: Our method relies on zeroshot transfer from template language instructions to those given by human annotators when asked to refer and instruct in natural ways.

Original: The results show that, with powerful pretrained language encoders, this transfer effect is sufﬁciently strong to permit decoding of complex language-dependent motor behaviours, despite the shift in distribution of the agent’s input.
Revised: The results show that, with powerful pretrained language encoders, this transfer effect is sufﬁciently strong to permit decoding of complex language-dependent motor behaviours, despite the shift in distribution of the agent’s input.

Original: More generally, we hope that this contribution serves to bring research on text-based and situated language learning closer together.
Revised: More generally, we hope that this contribution serves to bring research on text-based and situated language learning closer together.

Original: To facilitate research in this direction, we provide our dataset of natural instructions and referring expressions aligned to ShapeNet models.
Revised: To facilitate research in this direction, we provide our dataset of natural instructions and referring expressions aligned to ShapeNet models.

Original: We have explored only a tiny fraction of the many possible ways to transfer knowledge from a text corpus to a situated environment.
Revised: We have explored only a tiny fraction of the many possible ways to transfer knowledge from a text corpus to a situated environment.

Original: For instance, our approach involves freezing rather than ﬁne-tuning the BERT encoder weights to our desired behaviour policy, to avoid overﬁtting, but techniques such as knowledge distillation (Hinton et al., 2015) could point to more elegant ways to learn jointly from text and environmental experiences.
Revised: For instance, our approach involves freezing rather than ﬁne-tuning the BERT encoder weights to our desired behaviour policy, to avoid overﬁtting, but techniques such as knowledge distillation (Hinton et al., 2015) could point to more elegant ways to learn jointly from text and environmental experiences.

Original: Moreover, we have focused on BERT, but improvements may be possible by applying alternative general-purpose language encoders, such as
Revised: Moreover, we have focused on BERT, but improvements may be possible by applying alternative general-purpose language encoders, such as

Original: GPT-2 (Radford et al., 2019), Roberta (Liu et al., 2019) and Transformer XL (Dai et al., 2019).
Revised: GPT-2 (Radford et al., 2019), Roberta (Liu et al., 2019) and Transformer XL (Dai et al., 2019)

Original: A F ULL EXPERIMENTAL RESULTS
Revised: A F ULL EXPERIMENTAL RESULTS

Original: The results presented in charts here are those shown in the tables in the paper, with two additional conditions involving typo noise.
Revised: The results presented in charts here are those shown in the tables in the paper, with two additional conditions involving typo noise.

Original: B I NSTRUCTIONS TO HUMAN ANNOTATORS
Revised: B I NSTRUCTIONS TO HUMAN ANNOTATORS

Original: Human annotators use a keyboard and mouse to control a player in the environment simulator, as is standard in ﬁrst-person video games.
Revised: Human annotators use a keyboard and mouse to control a player in the environment simulator, as is standard in ﬁrst-person video games.

Original: The annotators were given the following instructions as part of the each annotation task.
Revised: The annotators were given the following instructions as part of the each annotation task.

Original: B. N ATURAL REFERRING EXPRESSIONS
Revised: B. N ATURAL REFERRING EXPRESSIONS

Original: This is a task called Name The Object .
Revised: This is a task called Name The Object .

Original: You will ﬁnd yourself in a room containing a single object.
Revised: You will ﬁnd yourself in a room containing a single object.

Original: Please move around the room to get a good view of the object.
Revised: Please move around the room to get a good view of the object.

Original: When you know what the object is:
Revised: When you know what the object is:

Original: Hit Enter 2.
Revised: Hit Enter 2.

Original: Type the name of the object 3.
Revised: Type the name of the object 3.

Original: Hit Enter again
Revised: Hit Enter again

Original: Examples of good responses:
Revised: Examples of good responses:

Original: A kettle 2.
Revised: A kettle 2.

Original: Some trousers 3.
Revised: Some trousers 3.

Original: A pair of scissors 4.
Revised: A pair of scissors 4.

Original: A tennis ball
Revised: A tennis ball

Original: Please don’t describe the object.
Revised: Please don’t describe the object.

Original: Just write down what you see, with an article like ‘a’ or ‘some’ if appropriate.
Revised: Just write down what you see, with an article like ‘a’ or ‘some’ if appropriate.

Original: Example of bad responses
Revised: Example of bad responses

Original: A brown ball 2.
Revised: A brown ball 2.

Original: A large piano with long black legs 3.
Revised: A large piano with long black legs 3.

Original: A small thing with lumps on the side
Revised: A small thing with lumps on the side

Original: You should not need more than 4-5 different words, and most objects will require just 1 or 2 words to name.
Revised: You should not need more than 4-5 different words, and most objects will require just 1 or 2 words to name.

Original: Sometimes, you might be unsure what the object is.
Revised: Sometimes, you might be unsure what the object is.

Original: If that’s the case, just make your best guess.
Revised: If that’s the case, just make your best guess.

Original: B. F ULL HUMAN INSTRUCTIONS
Revised: B. F ULL HUMAN INSTRUCTIONS

Original: This is a task called Ask to put .
Revised: This is a task called Ask to put .

Original: You will ﬁnd yourself in a room.
Revised: You will ﬁnd yourself in a room.

Original: Your job is to imagine giving an instruction to somebody else so that that person puts the red object on to the white object.
Revised: Your job is to imagine giving an instruction to somebody else so that that person puts the red object on to the white object.

Original: Move around the room to get a good look at what the two objects are.
Revised: Move around the room to get a good look at what the two objects are.

Original: When you are ready to give your instruction:
Revised: When you are ready to give your instruction:

Original: Hit Enter 2.
Revised: Hit Enter 2.

Original: Type your instruction 3.
Revised: Type your instruction 3.

Original: Hit Enter again
Revised: Hit Enter again

Original: Examples of good instructions:
Revised: Examples of good instructions:

Original: Place the cup onto the table 2.
Revised: Place the cup onto the table 2.

Original: Put the ball on the plate 3.
Revised: Put the ball on the plate 3.

Original: Move the pencil onto the box
Revised: Move the pencil onto the box

Original: Words to avoid Your instruction must not contain words for colours or other properties of the object.
Revised: Words to avoid Your instruction must not contain words for colours or other properties of the object.

Original: Please do not use the words red, white, scarlet, dark, large etc.
Revised: Please do not use the words red, white, scarlet, dark, large etc.

Original: in your instruction.
Revised: in your instruction.

Original: Instead, refer to objects by their name as you recognise them.
Revised: Instead, refer to objects by their name as you recognise them.

Original: If you don’t recognise what an object is, just make your best guess.
Revised: If you don’t recognise what an object is, just make your best guess.

Original: Examples of bad instructions:
Revised: Examples of bad instructions:

Original: Put the red thing on the table 2.
Revised: Put the red thing on the table 2.

Original: Put the large object on the small object 3.
Revised: Put the large object on the small object 3.

Original: Put the small round thing on the chest
Revised: Put the small round thing on the chest

Original: Keep your language varied.
Revised: Keep your language varied.

Original: Try to use various ways to express your instruction in different episodes to keep things interesting.
Revised: Try to use various ways to express your instruction in different episodes to keep things interesting.

Original: C A GENT ARCHITECTURE AND TRAINING DETAILS
Revised: C A GENT ARCHITECTURE AND TRAINING DETAILS

Original: To process visual input, the agent uses a residual convolutional network with 64 , 64 , 32 channels in the ﬁrst, second and third layers respectively and 2 residual blocks in each layer.
Revised: To process visual input, the agent uses a residual convolutional network with 64 , 64 , 32 channels in the ﬁrst, second and third layers respectively and 2 residual blocks in each layer.

Original: As the agent learns, actors carry replicas of the latest learner network, interact with the environment independently and send trajectories (observations and agent actions) back to a central learner.
Revised: As the agent learns, actors carry replicas of the latest learner network, interact with the environment independently and send trajectories (observations and agent actions) back to a central learner.

Original: The learning algorithm (Espeholt et al., 2018) modiﬁes the learner weights to optimize an actor-critic objective, with updates importance-weighted to correct for differences between the actor policy and the current state of the learner policy.
Revised: The learning algorithm (Espeholt et al., 2018) modiﬁes the learner weights to optimize an actor-critic objective, with updates importance-weighted to correct for differences between the actor policy and the current state of the learner policy.

Revised: We used a learning rate of 0 . 0005 , a learner batch size of 32 , an agent unroll length of 50 , a discounting factor of 0 . 99 , an epsilon (epsilon-greedy policy) of 1 e −and an entropy cost of 0 .

Revised: Mnih et al.

Original: In order to train the agent on the putting task , it was necessary to combine episodes of the task itself with episodes of simpler tasks, in a form of curriculum (although learning in parallel on all tasks).
Revised: In order to train the agent on the putting task , it was necessary to combine episodes of the task itself with episodes of simpler tasks, in a form of curriculum (although learning in parallel on all tasks).

Original: In particular, we trained it concurrently on a lifting task that involved the same moveable objects as the putting task (so that the agent could receive signal about their names without relying on a complete act of putting).
Revised: In particular, we trained it concurrently on a lifting task that involved the same moveable objects as the putting task (so that the agent could receive signal about their names without relying on a complete act of putting).

Original: We also found it beneﬁcial to add a put-near task to the curriculum, where the instructions were of the form Put a cup near the tray rather than Put a cup on the tray , and a reward was emitted if the agent moved the cup a short distance from the tray and placed it on the ground.
Revised: We also found it beneﬁcial to add a put-near task to the curriculum, where the instructions were of the form Put a cup near the tray rather than Put a cup on the tray , and a reward was emitted if the agent moved the cup a short distance from the tray and placed it on the ground.

Revised: In the lifting task , training the agent requires approximately 200 million frames of experience (approximately 7 million episodes), which takes about 24 hours with 250 actors on GPU.

Revised: In the putting task , training in each condition was stopped after 30 million episodes, approximately three days of training.

Original: Body movement actions
Revised: Body movement actions

Original: Movement and grip actions
Revised: Movement and grip actions

Original: Object manipulation
Revised: Object manipulation

Original: NOOP
Revised: NOOP

Original: GRAB
Revised: GRAB

Original: GRAB + SPIN OBJECT
Revised: GRAB + SPIN OBJECT

Original: RIGHT
Revised: RIGHT

Original: MOVE FORWARD
Revised: MOVE FORWARD

Original: GRAB + MOVE FORWARD
Revised: GRAB + MOVE FORWARD

Original: GRAB + SPIN OBJECT LEFT
Revised: GRAB + SPIN OBJECT LEFT

Original: MOVE BACKWARD
Revised: MOVE BACKWARD

Original: GRAB + MOVE BACKWARD
Revised: GRAB + MOVE BACKWARD

Original: GRAB + SPIN OBJECT UP
Revised: GRAB + SPIN OBJECT UP

Original: MOVE RIGHT
Revised: MOVE RIGHT

Original: GRAB + MOVE RIGHT
Revised: GRAB + MOVE RIGHT

Original: GRAB + SPIN OBJECT DOWN
Revised: GRAB + SPIN OBJECT DOWN

Original: MOVE LEFT
Revised: MOVE LEFT

Original: GRAB + MOVE BACKWARD
Revised: GRAB + MOVE LEFT

Original: GRAB + SPIN OBJECT FORWARD
Revised: GRAB + SPIN OBJECT FORWARD

Original: LOOK RIGHT
Revised: LOOK RIGHT

Original: GRAB + LOOK RIGHT
Revised: GRAB + LOOK RIGHT

Original: GRAB + SPIN OBJECT BACKWARD
Revised: GRAB + SPIN OBJECT BACKWARD

Original: LOOK LEFT
Revised: LOOK LEFT

Original: GRAB + LOOK LEFT
Revised: GRAB + LOOK LEFT

Original: GRAB + PUSH OBJECT AWAY
Revised: GRAB + PUSH OBJECT AWAY

Original: LOOK UP
Revised: LOOK UP

Original: GRAB + LOOK UP
Revised: GRAB + LOOK UP

Original: GRAB + PULL OBJECT CLOSE
Revised: GRAB + PULL OBJECT CLOSE

Original: LOOK DOWN
Revised: LOOK DOWN

Original: GRAB + LOOK DOWN
Revised: GRAB + LOOK DOWN

Original: D F URTHER ENVIRONMENT DETAILS
Revised: D F URTHER ENVIRONMENT DETAILS

Original: For all experiments, the environment is a Unity room of dimension 4m x 4m. The walls, ﬂoor and ceiling are always the same color, but we add a window and door (positioned randomly per episode) to give some sense of absolute location to the agent.
Revised: For all experiments, the environment is a Unity room of dimension 4m x 4m. The walls, ﬂoor and ceiling are always the same color, but we add a window and door (positioned randomly per episode) to give some sense of absolute location to the agent.

Original: When importing ShapeNet models, we use the scaling provided in the metadata, unless it it is very small (less than 0.000001), in which case we interpret the coordinates in the OBJ ﬁle as meters.
Revised: When importing ShapeNet models, we use the scaling provided in the metadata, unless it it is very small (less than 0.000001), in which case we interpret the coordinates in the OBJ ﬁle as meters.

Original: All objects have rigid bodies with collision meshes generated using Unity’s built in MeshCollider, with convex set to true.
Revised: All objects have rigid bodies with collision meshes generated using Unity’s built in MeshCollider, with convex set to true.

Original: The masses of all movable objects are set to 1 kg, so that our avatar has enough strength to pick all of them up (except for beds and trays, which are made kinetic).
Revised: The masses of all movable objects are set to 1 kg, so that our avatar has enough strength to pick all of them up (except for beds and trays, which are made kinetic).

Original: When selecting ShapeNet models, for performance reasons we discarded all models with a vertex count higher than 8000, and an OBJ ﬁle size greater than 1 MB.
Revised: When selecting ShapeNet models, for performance reasons we discarded all models with a vertex count higher than 8000, and an OBJ ﬁle size greater than 1 MB.

Original: The native ShapeNet category names are often not natural everyday names (for instance, they can be highly speciﬁc, like ”dual shock analog controller”).
Revised: The native ShapeNet category names are often not natural everyday names (for instance, they can be highly speciﬁc, like ”dual shock analog controller”).

Original: To mitigate this, we used ShapeNet’s WordNet tags, grouping models into categories according to WordNet synsets and assigning the name of the ﬁrst synset lemma to the category.
Revised: To mitigate this, we used ShapeNet’s WordNet tags, grouping models into categories according to WordNet synsets and assigning the name of the ﬁrst synset lemma to the category.

Original: Because depth-perception is challenging without binocular vision, the agent is assisted in manipulation by a visual guide (bottom-right) that highlights objects within grasping range and drops a vertical line from held objects.
Revised: Because depth-perception is challenging without binocular vision, the agent is assisted in manipulation by a visual guide (bottom-right) that highlights objects within grasping range and drops a vertical line from held objects.

Original: Full lists of objects and synonyms are on the following pages.
Revised: Full lists of objects and synonyms are on the following pages.

Original: name
Revised: name

Original: [Table]
Revised: [Table]

Original: drawers cupboard
Revised: drawers cupboard

Original: [Table]
Revised: [Table]

Original: table desk
Revised: table desk

Original: [Table]
Revised: [Table]

Original: seat 6bcabd23a81dd790e386ecf78eadd61c
Revised: seat 6bcabd23a81dd790e386ecf78eadd61c

Original: [Table]
Revised: [Table]

Original: sofa couch 11d5e99e8faa10ff n4263630 television receiver aerial cebb35bd663c82d3554a13580615ae n4413042 lamp light 62ab9c2f7b826fbcb910025244eec99a n3641940 desk bench a2c81b5c0ce9231245bc5a n
Revised: sofa couch 11d5e99e8faa10ff n4263630 television receiver aerial cebb35bd663c82d3554a13580615ae n4413042 lamp light 62ab9c2f7b826fbcb910025244eec99a n3641940 desk bench a2c81b5c0ce9231245bc5a n

Original: [Table]
Revised: [Table]

Original: tin 15b5c54a8ddc84353a5acb1c5cbd19ff n
Revised: tin 15b5c54a8ddc84353a5acb1c5cbd19ff n

Original: [Table]
Revised: [Table]

Original: ﬂute 415a96bfbb45b3ee35836c728d n4529463 bed bunk bbd23cc1e56a24204e649bb31ff339dd
Revised: ﬂute 415a96bfbb45b3ee35836c728d n4529463 bed bunk bbd23cc1e56a24204e649bb31ff339dd

Original: n2821967 ﬂoor lamp lamp 54017ba00148a652846fa729d90e
Revised: n2821967 ﬂoor lamp lamp 54017ba00148a652846fa729d90e

Original: n3371905 cabinet cupboard f5024636f3514eb51d0662c550f8f n2936496 book magazine c8691c86e110318ef2bc9da1ba799c
Revised: n3371905 cabinet cupboard f5024636f3514eb51d0662c550f8f n2936496 book magazine c8691c86e110318ef2bc9da1ba799c

Original: n6422547 pencil pen 839923a17af633dfb29a5ca18b250f3b n3914323 laptop
Revised: n6422547 pencil pen 839923a17af633dfb29a5ca18b250f3b n3914323 laptop

Original: [Table]
Revised: [Table]

Original: n2741540 cupboard wardrobe 3bb80aa0267a12bed00bff798ed59ff n3152990 pen pencil 628e5c83b1762320873ca101f05858b n3913116 bag sack b2c9ac70c58c90fa6dd2d391b72f n2776843 soda can drink 16526d147e837c386829bf9ee210f5e n4262696 sword knife 555c17f73cd6d530603c267665ac68a
Revised: n2741540 cupboard wardrobe 3bb80aa0267a12bed00bff798ed59ff n3152990 pen pencil 628e5c83b1762320873ca101f05858b n3913116 bag sack b2c9ac70c58c90fa6dd2d391b72f n2776843 soda can drink 16526d147e837c386829bf9ee210f5e n4262696 sword knife 555c17f73cd6d530603c267665ac68a

Original: n4380981 bookshelf shelf 586356e8809b6a678d44b95ca8abc7b n2874800 ﬁreplace burner df1bd1065e7f7cde5e29ce2c9d37b
Revised: n4380981 bookshelf shelf 586356e8809b6a678d44b95ca8abc7b n2874800 ﬁreplace burner df1bd1065e7f7cde5e29ce2c9d37b

Original: n3351301 curtain drape 6f3da555075ec7f9e17e45953c2e n3155743 battery cell
Revised: n3351301 curtain drape 6f3da555075ec7f9e17e45953c2e n3155743 battery cell

Original: 62733b55e76a3b718c9d9ab13336021b n2813606 bookcase bookshelf bc80335bbfda741df1783a44a88d n2874241 pizza pie caca4c8d409cddc66b04c0f74e5b376e
Revised: 62733b55e76a3b718c9d9ab13336021b n2813606 bookcase bookshelf bc80335bbfda741df1783a44a88d n2874241 pizza pie caca4c8d409cddc66b04c0f74e5b376e

Original: n7889783 hammer mallet a49a6c15dcef467bc84c00e08b1e25d
Revised: n7889783 hammer mallet a49a6c15dcef467bc84c00e08b1e25d

Original: n3486255 microwave oven b3bf04a02a596b139220647403cfb n3766619 cereal box oatmeal dc394c7fdea7e07887e775fad2c0bf
Revised: n3486255 microwave oven b3bf04a02a596b139220647403cfb n3766619 cereal box oatmeal dc394c7fdea7e07887e775fad2c0bf

Original: n3001610 food meal d92f30d9e38cf61ce69bbdea737daae
Revised: n3001610 food meal d92f30d9e38cf61ce69bbdea737daae

Original: n21445 screen monitor 4d7fb62f0ed368a18f62bdf4e n4159912 glass beaker 89cf9af7513ecd0947bdf66811027e n3443167 wine bottle
Revised: n21445 screen monitor 4d7fb62f0ed368a18f62bdf4e n4159912 glass beaker 89cf9af7513ecd0947bdf66811027e n3443167 wine bottle

Original: magnum e101cc44ead036294bc79c881a0e818b n4599016 lamppost streetlight 9cf4a30ab7e41c85c671a0255ba06fe n3642472 oven cooker 69c57dd0f6abdab7ac51268fdb437a9e
Revised: magnum e101cc44ead036294bc79c881a0e818b n4599016 lamppost streetlight 9cf4a30ab7e41c85c671a0255ba06fe n3642472 oven cooker 69c57dd0f6abdab7ac51268fdb437a9e

Original: n3868196 camera polaroid 48e26e789524c158e1e4b7162e96446c n2946154 globe world 3e97d91fda31a1c56ab57877a8c22e
Revised: n3868196 camera polaroid 48e26e789524c158e1e4b7162e96446c n2946154 globe world 3e97d91fda31a1c56ab57877a8c22e

Original: n3445436 cd player stereo fe90d87deaa5a8a5336d4ad4cfab5bfe n2991759 wardrobe cupboard cb48ec828b688a78d747fd5e045d
Revised: n3445436 cd player stereo fe90d87deaa5a8a5336d4ad4cfab5bfe n2991759 wardrobe cupboard cb48ec828b688a78d747fd5e045d

Original: n4557470 bible text 6ad85ddb5a110664e632c30f54a9aa
Revised: n4557470 bible text 6ad85ddb5a110664e632c30f54a9aa

Original: n6434286 pillow cushion 8b0c10a775c4c4edc1ebca21882cca5d
Revised: n6434286 pillow cushion 8b0c10a775c4c4edc1ebca21882cca5d

Original: n3944520 mug cup ea33ad442b032208d778b73d04298f n3802912 chandelier light 37d81dd3c640a6e2b3087a7528d1dd6a
Revised: n3944520 mug cup ea33ad442b032208d778b73d04298f n3802912 chandelier light 37d81dd3c640a6e2b3087a7528d1dd6a

Original: [Table] [Table]
Revised: [Table] [Table]