Original: Guiding Multi-Step Rearrangement Tasks with Natural Language Instructions
Revised: Guiding Multi-Step Rearrangement Tasks with Natural Language Instructions

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract: Enabling human operators to interact with robotic agents using naturallanguage would allow non-experts to intuitively instruct these agents.
Revised: Abstract: Enabling human operators to interact with robotic agents using naturallanguage would allow non-experts to intuitively instruct these agents.

Original: Towardsthis goal, we propose a novel Transformer-based model which enables a user toguide a robot arm through a 3D multi-step manipulation task with natural language commands.
Revised: Towardsthis goal, we propose a novel Transformer-based model which enables a user toguide a robot arm through a 3D multi-step manipulation task with natural language commands.

Original: Our system maps images and commands to masks over graspor place locations, grounding the language directly in perceptual space.
Revised: Our system maps images and commands to masks over graspor place locations, grounding the language directly in perceptual space.

Original: In a suiteof block rearrangement tasks, we show that these masks can be combined withan existing manipulation framework without re-training to greatly improve learning efﬁciency.
Revised: In a suiteof block rearrangement tasks, we show that these masks can be combined with anexisting manipulation framework without re-training, greatly improving learningefﬁciency.

Original: Our masking model is several orders of magnitude more sampleefﬁcient than typical Transformer models, operating with hundreds, not millions,of examples.
Revised: Our masking model is several orders of magnitude more sample efﬁcient than typical Transformer models, operating with hundreds, not millions, ofexamples.

Original: Our modular design allows us to leverage supervised and reinforcement learning, providing an easy interface for experimentation with different architectures.
Revised: Our modular design allows us to leverage supervised and reinforcementlearning, providing an easy interface for experimentation with different architectures.

Original: Our model completes block manipulation tasks with synthetic commands 530% more often than a UNet-based baseline. We demonstrate that ourmodel learns to localize actions correctly and to create a mapping of symbols toperceptual input that supports compositional reasoning.
Revised: Our model completes block manipulation tasks with synthetic commands530% more often than a UNet-based baseline, and learns to localize actions correctly while creating a mapping of symbols to perceptual input that supports compositional reasoning.

Original: We provide a valuableresource for 3D manipulation instruction following research by porting an existing 3D block dataset with crowdsourced language to a simulated environment.
Revised: We provide a valuable resource for 3D manipulation instruction following research by porting an existing 3D block dataset with crowdsourcedlanguage to a simulated environment.

Original: Our method’s 25 .
Revised: Our method’s 25 .

Original: 3% absolute improvement in identifying the correct block on theported dataset indicates that we handle syntactic and lexical variation effectively.
Revised: 3% absolute improvementin identifying the correct block on the ported dataset demonstrates its ability tohandle syntactic and lexical variation.

Original:  Introduction
Revised:  Introduction

Original: Reinforcement Learning (RL) is a powerful tool for developing perception-driven robotic systems.
Revised: Reinforcement Learning (RL) is a powerful tool for developing perception-driven robotic systems.

Original: Most RL algorithms train an agent to perform a single task based on an given reward function.
Revised: Most RL algorithms train an agent to perform a single task based on an given reward function.

Original: However, what if we would like to adapt or extend the learned policy of a robot to novel or morespecialized tasks by providing instructions?
Revised: However, what if we would like to adapt or extend the learned policy of a robot to novel or morespecialized tasks by providing instructions?

Original: Could we exploit previously learned RL policies for arobot to solve novel tasks now speciﬁed using natural language?
Revised: Could we exploit previously learned RL policies for arobot to solve novel tasks now speciﬁed using natural language?

Original: The ability to adapt learned policiesthrough natural language to perform novel tasks would open the door to a modular integration ofperception, language, and action, avoiding the complexities of joint training over all modalities simultaneously.
Revised: The ability to adapt learned policiesthrough natural language to perform novel tasks would open the door to a modular integration ofperception, language, and action, avoiding the complexities of joint training over all modalities simultaneously.

Original: For example, in our experiments, we constrain a general-purpose block manipulationpolicy using natural language, without needing any language data to train the policy itself.
Revised: For example, in our experiments, we constrain a general-purpose block manipulationpolicy using natural language, without needing any language data to train the policy itself.

Original: This kind of modular learning should be capable of injecting guidance from language into the RLdriven action policy and simultaneously address the challenges ofunderstanding natural languagecomputationally.
Revised: This kind of modular learning requires injecting guidance from language into the RL-driven actionpolicy while simultaneously addressing the challenges of natural language understanding .

Original: For example, natural language semantics is often underspeciﬁed, especially withrespect to a physical environment. Correctly choosing one out of many potentially valid actionsfor a command such as “Move the green block to the left of the right block” involves pragmatic andcommon-sense reasoning.
Revised: Forexample, natural language semantics is often underspeciﬁed, especially with respect to a physicalenvironment; correctly choosing one out of many potentially valid actions for a command suchas “Move the green block to the left of the right block” involves pragmatic and common-sensereasoning.

Original: Furthermore, in order to map from language to action, we need to addressthe symbol grounding problem [1], i.e. how does one map symbols (e.g. “left”, “right”, “block”,“red”) to the physical world?
Revised: Furthermore, in order to map from language to action, we need to address the symbolgrounding problem [1], i.e. how does one map symbols (e.g. “left”, “right”, “block”, “red”) to thephysical world?

Original: We leverage Transformer encoders to learn a strategy from data, inplace of the manually deﬁned data fusion strategies found in prior work [2, 3, 4, 5, 6, 7, 8].
Revised: We leverage Transformer encoders to learn a strategy from data, in place of themanually deﬁned data fusion strategies found in prior work [2, 3, 4, 5, 6, 7, 8].

Original: [Figure]
Revised: [Figure]

Original: In our work, we factorize language-driven action into learning how to act and learning where to act ,similar to Misra et al.
Revised: In our work, we factorize language-driven action into learning how to act and learning where to act ,following Misra et al.

Original: Learning how to act is accomplished by the SPOT-Q algorithm [9] whichteaches an agent to re-arrange colored blocks into stacks and rows by grasping and placing them.
Revised: Learning how to act is accomplished by the SPOT-Q algorithm [9] whichteaches an agent to re-arrange colored blocks into stacks and rows by grasping and placing them.

Original: We use SPOT-Q models pre-trained without any language data for our manipulation experiments,demonstrating that our method lets us leverage existing manipulation models without additionaltraining.
Revised: We use SPOT-Q models pre-trained without any language data for our manipulation experiments,demonstrating that our method lets us leverage existing manipulation models without additionaltraining.

Original: SPOT-Q[9] identiﬁes regions worth exploring, e.g. containing an object, to encode a set ofvalid action regions for grasping and placing.
Revised: SPOT-Q identiﬁes regions worth exploring, e.g. containing an object, to encode a setof valid action regions for grasping and placing.

Original: We produce segmentation masks that are jointly conditioned on both the language input and an image of the state, ﬁnd their intesection with SPOT-Q’smasks, then apply this new mask tothe set of pixel-wise Q-values from the SPOT-Q RL framework.
Revised: We produce segmentation masks that are jointlyconditioned on both the language input and an image of the state and apply them with to set ofpixel-wise Q-values from the SPOT-Q RL framework.

Original: This process provides us with the best action for a given state and command, as Figure 1 illustrates.
Revised: This process provides us with the best actionfor a given state and command, as Figure 1 illustrates.

Original: Factorizing the learning process this way provides several advantages.
Revised: Factorizing the learning process this way provides several advantages.

Original: Firstly, it offers a moreefﬁcient path to learning.
Revised: Firstly, it offers a moreefﬁcient path to learning.

Original: Separating the language grounding model from the reinforcement learningmodel, leads to a less complex RL model that receives more direct feedback for tasks.
Revised: Separating the language grounding model from the reinforcement learningmodel, leads to a less complex RL model that receives more direct feedback for tasks.

Original: Moreover,the separation of action and language understanding allows us to learn the language component viasupervised learning, which is typically more sample efﬁcient.
Revised: Moreover,the separation of action and language understanding allows us to learn the language component viasupervised learning, which is typically more sample efﬁcient.

Original: Secondly, the modular nature of themodel allows us to swap components easily: we can train a masking model on simulated imagesor real images (cf.
Revised: Secondly, the modular nature of themodel allows us to swap components easily: we can train a masking model on simulated imagesor real images (cf.

Original: Section 5.1), allowing us to use the same action model for both simulated andreal tasks without re-training.
Revised: Section 5.1), allowing us to use the same action model for both simulated andreal tasks without re-training.

Original: In an end-to-end model, a change to the input space would likelyrequire re-training from scratch.
Revised: In an end-to-end model, a change to the input space would likelyrequire re-training from scratch.

Original: Finally, the factorized model output is more interpretable; whenan action is unsuccessful, we can inspect our model to localize the point of failure, be it a lack ofcomprehension or a mistake while acting.
Revised: Finally, the factorized model output is more interpretable; whenan action is unsuccessful, we can inspect our model to localize the point of failure, be it a lack ofcomprehension or a mistake while acting.

Original: This may not be evident from the agent’s action alone.
Revised: This may not be evident from the agent’s action alone.

Original: Note that our implementation of this factorization is not without disadvantages: if the languagecontains information on how to act this information could be lost.
Revised: Note that our implementation of this factorization is not without disadvantages: if the languagecontains information on how to act this information could be lost.

Original: In summary, our contributions are:
Revised: In summary, our contributions are:

Original: We introduce a novel Transformer-based model for mapping language commands to masks
Revised: We introduce a novel Transformer-based model for mapping language commands to masks

Original: over an action space.
Revised: over an action space.

Original: Our approach’s modular nature enables us to seamlessly integrate
Revised: Our approach’s modular nature enables us to seamlessly integrate

Original: an
Revised: an

Original: existing Q-learning framework proposed by Hundt et al.
Revised: existing Q-learning framework proposed by Hundt et al.

Original: [9] for block manipulation tasks.
Revised: [9] for block manipulation tasks.

Original: We show our model succeeds in a variety of simulated and real data settings with orders of
Revised: We show our model succeeds in a variety of simulated and real data settings with orders of

Original: magnitude less data than typically available to Transformers, furthering the potential use of
Revised: magnitude less data than typically available to Transformers, furthering the potential use of

Original: Transformers in the low-data regimes commonly found in robotics.
Revised: Transformers in the low-data regimes commonly found in robotics.

Original: We demonstrate this model’s utility on an existing dataset (introduced by Bisk et al.
Revised: We demonstrate this model’s utility on an existing dataset (introduced by Bisk et al.

Original: involving a challenging block re-arrangement task with crowdsourced natural language commands containing syntactic diversity as well as complex natural language phenomena.
Revised: involving a challenging block re-arrangement task with crowdsourced natural languagecommands containing syntactic diversity as well as complex natural language phenomena.4.

Original: We release our real and simulated datasets, plus a ﬁltered version of the Bisk et al.
Revised: We release our real and simulated datasets, plus a ﬁltered version of the Bisk et al.

Original: [10]dataset containing the subset of examples feasible under a realistic model of physics.
Revised: [10]dataset containing the subset of examples feasible under a realistic model of physics.

Original:  Related Work
Revised:  Related Work

Original: We describe related work in the areas of natural language instruction following on robots, QLearning for multistep manipulation tasks, and Transformer models for Computer Vision.
Revised: We describe related work in the areas of natural language instruction following on robots, QLearning for multistep manipulation tasks, and Transformer models for Computer Vision.

Original: Robots and Natural Language Commands – We follow the Problem-Solution Sequences (PSS)paradigm [11], wherein a multi-step manipulation task (e.g. “build a stack”) is framed as a sequence of incremental images or state representations.
Revised: Robots and Natural Language Commands – We follow the Problem-Solution Sequences (PSS)paradigm [11], where a multi-step manipulation task (e.g. “build a stack”) is framed as a sequence of incremental images or state representations.

Original: These may be paired with per-step instructions onhow to accomplish the intermediate goal (e.g., “stack the red block on the blue block”).
Revised: These may be paired with per-step instructions onhow to accomplish the intermediate goal (e.g., “stack the red block on the blue block”).

Original: Several
Revised: Several

Original: PSS datasets have been proposed [11, 10] and modeled in 2D [4, 12] and 3D [10, 3].
Revised: PSS datasets have been proposed [11, 10] and modeled in 2D [4, 12] and 3D [10].

Original: We translatethe 3D dataset proposed by Bisk et al.
Revised: We translate the3D dataset proposed by Bisk et al.

Original: [10] to a simulated environment in Section 5.2.
Revised: [10] to a simulated environment in Section 5.2.

Original: Our experiments on this dataset and in simulation involve retrieving and placing blocks; Nguyen et al.
Revised: The sequentialnature of PSS relates to the hierarchical RL setting explored by Andreas et al.

Original: [13]use natural language constraints to guide object retrieval, but do not model placement.
Revised: [13], who learn taskspeciﬁc sub-policies indexed by linguistic sub-goals. Similar approaches are applied to zero-shotsub-goal sequence generalization[14], non-linearly-ordered sub-goal execution[15], and choosinginterpretable linguistic sub-goals [16]. While we also assume access to step-wise annotated instructions, we focus on guiding a low-level robotic manipulation policy with a supervised masking modelrather than a task-level policy.

Original: RL withformal languages, e.g. programs[14].
Revised: Gangopadhyay et al. [17] and Sun et al. [18] learn policies guidedby symbolic programs, while Perera et al. [19] ﬁrst parse language into a symbolic representations.

Original: Imitation learning [15] and semantic parsing [16] have alsobeen used for block manipulation tasks; our work uses Q-learning for policies, and does not specifyan ontology or semantic parse.
Revised: Our masking-based method ties the language directly to the perceptual space, eschewing intermediate symbol ontologies. Language has been used for reward-shaping [20, 21, 22], and reward modellearning [23]; we instead use the language-agnostic reward shaping in SPOT-Q. Stepputtis et al.[24] and Lynch and Sermanet [25] use language-conditioned imitation learning for manipulationpolicies; we instead use Q-learning.

Original: TransporterNets
Revised: TransporterNets

Original: [17] imitates multi-step tasks from demonstration,without natural language input.
Revised: [26] learn to perform multi-step tasks fromdemonstration, but lacks language input.

Original:  Concept2Robot [18] is among the most relevant work since it completes language-described single-step tasks via a combination of learning from demonstration and 
Revised: Like our work, Concept2Robot [27] completes languagedescribed tasks by combining learning from demonstration and RL.

Original: RL, however, it does not consider several key aspects of this work: we evaluate challenging multistep tasks withthe possibility of progress reversal [9]; we succeed on much smaller datasets; weconsider crowdsourced commands; and we evaluate on real robot data.
Revised: However, it differs from oursystem along multiple axes: we examine challenging multi-step tasks with progress reversal [9]; wesucceed on much smaller datasets; we consider crowdsourced commands; and we evaluate on realrobot data.

Original: Navigating according to natural language instructions[19, 20, 4, 8, 7, 5] includes methods such as Jiang et al.
Revised: Our experiments involve block retrieval and placement; Nguyen et al.

Original: [21], who condition imitation learning based on unstructured data of a robot moving anda human describing the actions, but their simulation-only experiments require millions of examples.
Revised: [28] guide objectretrieval with language, but do not model placement.

Original: Visual Goal Prediction (VGP), introduced by Misra et al.
Revised: Visual Goal Prediction (VGP), introduced by Misra et al.

Original: [4], frames the problem of choosing anaction region as a semantic segmentation task in which each pixel as a separate binary decision: actor do not act.
Revised: [4] for language-conditioned navigation[29, 30, 4, 8, 7, 5], frames the problem of choosing an action region as a semantic segmentation taskin which each pixel as a separate binary decision: act or do not act.

Original: VGP has directed a simulated [5] and real quadcopter [8, 7].
Revised: VGP has directed a simulated[5] and real quadcopter [8, 7].

Original: We implement VGPwith UNet[22], used by [4, 5, 7, 8],to serve as our baseline.
Revised: We implement VGP with UNet[31, 4, 5, 7, 8], as a baseline.

Original: Language has also been used forreward-shaping [23, 24, 25], for providing language-like goal speciﬁcations in a grid world[26], andfor compositional language abstraction directly within an RL model from simulation state[27] (notpixels).

Original: By contrast, we separate natural language and RL via the language-agnostic reward shapingin SPOT-Q[9], and operate with just hundreds of examples of real data.

Original: Q -learning for Multi Step Tasks – Q-Learning for robotics is an active area of research into algorithms and supporting datasets [3], including various RL mechanisms for tabletop clutter removal[28, 29, 30, 31], and multi-step formal RL[14, 32], but NLP is not addressed.
Revised: Q-learning for Multi-Step Tasks – Q-Learning for robotics is an active area of research into algorithms and supporting datasets [3], including various RL mechanisms for tabletop clutter removal[32, 33, 34, 35], and multi-step formal RL[14, 13], but NLP is not addressed.

Original: “Good Robot!”
Revised: “Good Robot!”

Original: [9]was the ﬁrst to complete multi-step tasks with consideration of progress reversal , a failure casewhere actions undo previous progress such that ⌦ ( n ) additional actions are required to recover, e.g.toppling the tower in a partially completed stacking task.
Revised: [9]was the ﬁrst to complete multi-step tasks with consideration of progress reversal , a failure casewhere actions undo previous progress such that ⌦ ( n ) additional actions are required to recover, e.g.toppling the tower in a partially completed stacking task.

Original: “Good Robot!”
Revised: “Good Robot!”

Original: [9] also shows that actionmasking (segmentation) accelerates robot learning.
Revised: [9] also shows that actionmasking (segmentation) accelerates robot learning.

Original: Mohammed et al.
Revised: Mohammed et al.

Original: [33] reviews RL grasping.
Revised: [36] reviews RL grasping.

Original: Transformer Encoders for Vision – Though Transformer encoders, which build representationsusing self-attention, have become standard in NLP, computer vision tasks are still dominated by convolutional models, of which UNet [22] is a paradigmatic instance.
Revised: Transformer Encoders for Vision – Though Transformer encoders, which build representationsusing self-attention, have become standard in NLP, computer vision tasks are still dominated by convolutional models, of which UNet [31] is a paradigmatic instance.

Original: The use of Transformers in visiontasks has been held back by the quadratic complexity of the Transformer’s self-attention layer; animage input treated as a sequence of pixels would quickly become computationally intractable [34].
Revised: The use of Transformers in visiontasks has been held back by the quadratic complexity of the Transformer’s self-attention layer; animage input treated as a sequence of pixels would quickly become computationally intractable [37].

Original: Dosovitskiy et al.
Revised: Dosovitskiy et al.

Original: [34] introduce a minimal modiﬁcation to perform image recognition using Transformers, tiling the image into square patches, which they concatenate into a sequence and augmentwith learned linear positional embeddings.
Revised: [37] introduce a minimal modiﬁcation to perform image recognition using Transformers, tiling the image into square patches, which they concatenate into a sequence and augmentwith learned linear positional embeddings.

Original: Though the model presented by Dosovitskiy et al.
Revised: Though the model presented by Dosovitskiy et al.

Original: performs image-level recognition well, it cannot perform pixel-level tasks (e.g., semantic segmentation)as-is.
Revised: performs image-level recognition well, it cannot perform pixel-level tasks (e.g., semantic segmentation)as-is.

Original: Ranftl et al.
Revised: Ranftl et al.

Original: [35] present an extension to this model for pixel-level prediction.
Revised: [38] present an extension to this model for pixel-level prediction.

Original:  Models
Revised:  Models

Original: We propose a Transformer-based model to convert image and natural language inputs into separatemasks for grasping and placing, thereby allowing users to guide multi-step manipulation tasks.
Revised: We propose a Transformer-based model to convert image and natural language inputs into separatemasks for grasping and placing, thereby allowing users to guide multi-step manipulation tasks.

Original: We begin by tiling the n ⇥ n birds-eye image of the scene with C colored blocks into k ⇥ k patches,whose RGB pixel values are concatenated into one vector, resulting in patch vectors
Revised: We begin by tiling the n ⇥ n birds-eye image of the scene with C colored blocks into k ⇥ k patches,whose RGB pixel values are concatenated into one vector, resulting in patch vectors

Original: I = i 1 , . .
Revised: I = i 1 , . .

Original: , i L .
Revised: , i L .

Original: The resulting vectors are augmented with a ﬁxed sinusoidal positional encoding [36].
Revised: The resulting vectors are augmented with a ﬁxed sinusoidal positional encoding [39].

Original: Similarly,each token in the command string W = w 1 , . . .
Revised: Similarly,each token in the command string W = w 1 , . . .

Original: w m is embedded into a continuous space resultingin vectors e 1 , . . .
Revised: w m is embedded into a continuous space resultingin vectors e 1 , . . .

Original: , e m , and those vectors are augmented with positional encodings as well.
Revised: , e m , and those vectors are augmented with positional encodings as well.

Original: The goal is to train a model that produces a pair of language-deﬁned spatial masks ˆ M place 2 R n ⇥
Revised: The goal is to train a model that produces a pair of language-deﬁned spatial masks ˆ M place 2 R n ⇥

Original: nand ˆ M grasp 2 R n ⇥ n , and an optional pair of state reconstructions
Revised: nand ˆ M grasp 2 R n ⇥ n , and an optional pair of state reconstructions

Original: ˆ S grasp , ˆ S place 2 R C ⇥ n ⇥ n given by:
Revised: ˆ S grasp , ˆ S place 2 R C ⇥ n ⇥ n given by:

Original: [Equation]

Original:  f shared

Original: [Equation]
Revised: [Equation]

Original:

Original: [Equation] ◆ ( [Equation] where f . are parameterized by Transformer encoders with output size k , H m 2 R k ⇥ 2 , H s 2 R k ⇥ C , is the softmax function, and t 2 { grasp , place } .
Revised: where f . are parameterized by Transformer encoders with output size k , H m 2 R k ⇥ 2 , H s 2 R k ⇥ C , is the softmax function, and t 2 { grasp , place } .

Original: The reconstruction function R up-samples kpatches by copying the patch value across each of its constituent pixels, resulting in an n ⇥ n image.
Revised: The reconstruction function R up-samples kpatches by copying the patch value across each of its constituent pixels, resulting in an n ⇥ n image.

Original: Note that structurally, the “shared,” “grasp,” and “place” Transformers are identical, but while the“shared” Transformer is used both for grasp and place prediction, the others are task-speciﬁc.
Revised: Note that structurally, the “shared,” “grasp,” and “place” Transformers are identical, but while the“shared” Transformer is used both for grasp and place prediction, the others are task-speciﬁc.

Original: We train the masking model separately from the manipulation model.
Revised: We train the masking model separately from the manipulation model.

Original: For the latter model, weuse a pre-trained SPOT-Q model from Hundt et al.
Revised: For the latter model, weuse a pre-trained SPOT-Q model from Hundt et al.

Original: [9], which produces pixel-wise Q-values forpicking and placing, making it easy to integrate with our approach.
Revised: [9], which produces pixel-wise Q-values forpicking and placing, making it easy to integrate with our approach.

Original: Furthermore, pixel-wise Qvalues facilitate potential future work on extending our method to non-cubic shapes.
Revised: Furthermore, pixel-wise Qvalues facilitate potential future work on extending our method to non-cubic shapes.

Original: All maskingmodels are trained with a multi-task loss, composed of a pixel-wise binary cross-entropy loss for themasks, and an optional auxiliary reconstruction loss:
Revised: All maskingmodels are trained with a multi-task loss, composed of a pixel-wise binary cross-entropy loss for themasks, and an optional auxiliary reconstruction loss:

Original: [Equation]
Revised: [Equation]

Original: The reconstruction loss here forces the model to semantically segment the image by identifying pixelwith its block ID, or labeling it as background.
Revised: The reconstruction loss here forces the model to semantically segment the image by identifying pixelwith its block ID, or labeling it as background.

Original: As in Nguyen and Salazar [37], we use a pre-norm
Revised: As in Nguyen and Salazar [40], we use a pre-norm

Original: Transformer layer with scaled initialization.
Revised: Transformer layer with scaled initialization.

Original: Our learning rate schedule is based on Vaswani et al.[36], with a varied number of warmup steps as in Nguyen and Salazar.
Revised: Our learning rate schedule is based on Vaswani et al.[39], with a varied number of warmup steps as in Nguyen and Salazar.

Original: A comprehensive report ofall hyperparameters and a ﬁgure of our architecture is given in Appendix A.
Revised: A comprehensive report ofall hyperparameters and a ﬁgure of our architecture is given in Appendix A.

Original:  Our model relies on a learned function from states and actions to expected rewards,
Revised: Q-Learning – Our model relies on a learned function from states and actions to expected rewards,

Original: Q ( s t , a ) , which we learn via the SPOT-Q algorithm [9], which introduces novel reward shapingprocesses for improving efﬁciency and sim-to-real transfer.
Revised: Q ( s t , a ) , which we learn via the SPOT-Q algorithm [9], which introduces novel reward shapingprocesses for improving efﬁciency.

Original: More details are given in Appendix A.3.
Revised: More details are given in Appendix A.3.

Original: Given a pixel-wise n ⇥ n state space of an image and two actions, grasp and place, the output of thisfunction can be expressed as a tensor Q 2 R n ⇥ n ⇥ 2 .
Revised: Given a pixel-wisen ⇥ n state space of an image and two actions, grasp and place, the output of this function can beexpressed as a tensor Q 2 R n ⇥ n ⇥ 2 .

Original:  Datasets and Resources
Revised:  Datasets and Resources

Original: 4.1 Simulation Experiments with Synthetic Commands
Revised: 4.1 Simulation Experiments with Synthetic Commands

Original: We create a dataset of commands and grasp/place masks to train our masking model and demonstrate its performance in simulated environment on a robot arm.
Revised: We create a dataset of commands and grasp/place masks to train our masking model and demonstrate its performance in simulated environment on a robot arm.

Original: Using trajectories from Hundt et al.[9], we annotate successful grasp-and-place action pairs with synthetic commands generated from atemplate, and use simulation state information to construct the reference masks M t and state information S t .
Revised: Using trajectories from Hundt et al.[9], we annotate successful grasp-and-place action pairs with synthetic commands generated from atemplate, and use simulation state information to construct the reference masks M t and state information S t .

Original: While synthetic language makes simplifying assumptions about the syntactic and lexicaldiversity of natural language, it provides a controllable test-bed for symbolic reasoning, and is oftenused to study compositionality [38, 39, 40], as we do in Section 5.1.
Revised: While synthetic language makes simplifying assumptions about the syntactic and lexicaldiversity of natural language, it provides a controllable test-bed for symbolic reasoning, and is oftenused to study compositionality [41, 42, 43], as we do in Section 5.1.

Original: For row-making, the template is“move the color a block next to the color b block”, and for stacking the template is “stack thecolor a block on the color b block”.
Revised: For row-making, the template is“move the color a block next to the color b block”, and for stacking the template is “stack thecolor a block on the color b block”.

Original: We train supervised VGP models (cf.
Revised: We train supervised VGP models (cf.

Original: Section 3) on thesecommands, and combine the produced masks with the Q-values from the SPOT-Q framework in asimulated environment to execute the actions in order.
Revised: Section 3) on thesecommands, and combine the produced masks with the Q-values from the SPOT-Q framework in asimulated environment to execute the actions in order.

Original: The visual input to our models is the concatenation of a birds-eye color image and depth image.
Revised: The visual input to our models is the concatenation of a birds-eye color image and depth image.

Original: The training data is augmented by ﬂipping eachimage across 4 axes, manipulating the command accordingly, as well as by the addition of pixelwise Gaussian noise drawn from N (0 , 0 . 05) .
Revised: The training data is augmented by ﬂipping eachimage across 4 axes, manipulating the command accordingly, as well as by the addition of pixelwise Gaussian noise drawn from N (0 , 0 . 05) .

Original: This results in train/dev/test splits of 5520 / 46 / 46 and8970 / 74 / 76 for row-making and stacking, respectively, where each sample is an input image andcommand paired with a reference place and grasp mask.
Revised: This results in train/dev/test splits of 5520 / 46 / 46 and8970 / 74 / 76 for row-making and stacking, respectively, where each sample is an input image andcommand paired with a reference place and grasp mask.

Original: Experiments and results on VGP metricsand in the simulation for this data are given in Section 5.1.
Revised: Experiments and results on VGP metricsand in the simulation for this data are given in Section 5.1.

Original: 4.2 Transfer to Real Images
Revised: 4.2 Transfer to Real Images

Original: In order to be useful in real-life settings, our method must also be applicable to real data.
Revised: In order to be useful in real-life settings, our method must also be applicable to real data.

Original: However,unlike simulated data, data from a real robot is expensive and time-consuming to collect.
Revised: However,unlike simulated data, data from a real robot is expensive and time-consuming to collect.

Original: Therefore,it is natural to ask: how much real data is needed to train an adequate VGP model?
Revised: Therefore,it is natural to ask: how much real data is needed to train an adequate VGP model?

Original: The Hundt et al.
Revised: The Hundt et al.

Original: [9] real robot trial logs include 1496 actions each for row-making and stacking.
Revised: [9] real robot trial logs include 1496 actions each for row-making and stacking.

Original: We split these into an 80/10/10 train/dev/test split and augment the examples by ﬂipping, producing5980 , 747 , 748 datapoints.
Revised: We split these into an 80/10/10 train/dev/test split and augment the examples by ﬂipping, producing5980 , 747 , 748 datapoints.

Original: Image collection details are given in Appendix B.2.
Revised: Image collection details are given in Appendix B.2.

Original: Given the small sizeof the real dataset, we pre-train a model on simulated images and ﬁne-tune it with the real images.
Revised: Given the small sizeof the real dataset, we pre-train a model on simulated images and ﬁne-tune it with the real images.

Original: 4.3 Compositionality
Revised: 4.3 Compositionality

Original: In linguistics, compositionality is typically taken to mean that the meaning of a whole utteranceis composed of the meanings of its parts.
Revised: In linguistics, compositionality is typically taken to mean that the meaning of a whole utteranceis composed of the meanings of its parts.

Original: For example, the meaning of “red block” is composed of the meanings of “red” and “block”.
Revised: For example, the meaning of “red block” is composed of the meanings of “red” and “block”.

Original: Compositionality allows speakers to re-combine knownsymbols in novel ways, and underpins human language use.
Revised: Compositionality allows speakers to re-combine knownsymbols in novel ways, and underpins human language use.

Original: In our task, a compositional reasonershould correctly execute the instruction “stack the red block on the blue block” without ever havingseen that particular combination; it should be able to localize red and blue blocks from other colorcontexts (e.g. red/green, blue/yellow, etc.).
Revised: In our task, a compositional reasonershould correctly execute the instruction “stack the red block on the blue block” without ever havingseen that particular combination; it should be able to localize red and blue blocks from other colorcontexts (e.g. red/green, blue/yellow, etc.).

Original: However, there is no explicit constraint during trainingto enforce this; the model could learn to interpret each of the 23 possible color pairs atomically.
Revised: However, there is no explicit constraint during trainingto enforce this; the model could learn to interpret each of the 23 possible color pairs atomically.

Original: Wetest our model’s compositional reasoning by examining the ability of our system to generalize tounseen color combinations.
Revised: Wetest our model’s compositional reasoning by examining the ability of our system to generalize tounseen color combinations.

Original: We hold out 6 color pairs during training, 26% of the 23 order-invariantpairs, and then test exclusively on those pairs.
Revised: We hold out 6 color pairs during training, 26% of the 23 order-invariantpairs, and then test exclusively on those pairs.

Original: For example, we train a masking model with all colorcombinations except those made only of red, blue, yellow, and green and then test only on “stack the{ red,blue,yellow,green } block on the { red,blue,yellow,green } block” examples.
Revised: For example, we train a masking model with all colorcombinations except those made only of red, blue, yellow, and green and then test only on “stack the{ red,blue,yellow,green } block on the { red,blue,yellow,green } block” examples.

Original: We hold-out each 195 possible subset, training and evaluating a model as described above, average the metrics across allsubsets, and compare against a random train and test split baseline, where all color pairs are seen attrain time.
Revised: We hold-out each 196 possible subset, training and evaluating a model as described above, average the metrics across allsubsets, and compare against a random train and test split baseline, where all color pairs are seen attrain time.

Original: For efﬁciency, we train on each task separately and do not augment with Gaussian noise.
Revised: For efﬁciency, we train on each task separately and do not augment with Gaussian noise.

Original: 4.4 Naturalistic Language Commands with Complex Scenes
Revised: 4.4 Naturalistic Language Commands with Complex Scenes

Original: Language sourced from human speakers is syntactically and lexically diverse, with mistakes, misspellings, and other idiosyncrasies that make far more challenging to model than template data.
Revised: Language sourced from human speakers is syntactically and lexically diverse, with mistakes, misspellings, and other idiosyncrasies that make far more challenging to model than template data.

Original: Analyzing synthetic language showcases our model’s ability to integrate discrete symbolic commandswith visual perception and action, as well as our model’s ability to generalize in a compositionalfashion, but this does not imply that our model generalizes to the complexities of real language.
Revised: Analyzing synthetic language showcases our model’s ability to integrate discrete symbolic commandswith visual perception and action, as well as our model’s ability to generalize in a compositionalfashion, but this does not imply that our model generalizes to the complexities of real language.

Original: We examine a 3D block manipulation dataset with human natural language instructions that wasintroduced by Bisk et al.
Revised: We examine a 3D block manipulation dataset with human natural language instructions that wasintroduced by Bisk et al.

Original: [10] (cf.
Revised: [10] (cf.

Original: Fig. 2).
Revised: Fig. 2).

Original: This dataset is set apart from other block-world data byits focus on complex instructions.
Revised: This dataset is set apart from other block-world data byits focus on complex instructions.

Original: Rather than using synthetic utterances, instructions were sourcedfrom crowdworkers (9 per example), who were shown a pair of images and asked to provide instructions to a robot, with the goal of manipulating the blocks in the ﬁrst image such that they result in thesecond image.
Revised: Rather than using synthetic utterances, instructions were sourcedfrom crowdworkers (9 per example), who were shown a pair of images and asked to provide instructions to a robot, with the goal of manipulating the blocks in the ﬁrst image such that they result in thesecond image.

Original: This collection method result in more syntactically diverse utterances with complexlinguistic concepts, such as “mirroring” and “balancing”.
Revised: This collection method result in more syntactically diverse utterances with complexlinguistic concepts, such as “mirroring” and “balancing”.

Original: [Figure]
Revised: [Figure]

Original: The dataset is comprised of a sequence of state representations paired with natural language instructions on how to transition between states.
Revised: The dataset is comprised of a sequence of state representations paired with natural language instructions on how to transition between states.

Original: While theblocks dataset is designed for robot manipulation,several assumptions were made to simplify annotation by crowdworkers.
Revised: While theblocks dataset is designed for robot manipulation,several assumptions were made to simplify annotation by crowdworkers.

Original: Crucially, physics in the environment were turned off while the structures werebeing constructed and annotated, resulting in somephysically infeasible structures.
Revised: Crucially, physics in the environment were turned off while the structures werebeing constructed and annotated, resulting in somephysically infeasible structures.

Original: To facilitate research in the blocks domain, and make the datasetmore useful for training and evaluating embodied agents, we create a ﬁltered version of the dataset,where physically infeasible examples are removed.
Revised: To facilitate research in the blocks domain, and make the datasetmore useful for training and evaluating embodied agents, we create a ﬁltered version of the dataset,where physically infeasible examples are removed.

Original: Our ﬁltering process removes 49 .
Revised: Our ﬁltering process removes 49 .

Original: 93%and 38 .
Revised: 93%and 38 .

Original: 23% of train, dev, and test, respectively.
Revised: 23% of train, dev, and test, respectively.

Original: Appendix C.1 contains more details on the translation process and data preprocessing.
Revised: Appendix C.1 contains more details on the translation process and data preprocessing.

Original: To model the dataset’s world state sequences in the VGPframework, we translate the 3D Cartesian coordinates of each block given by the dataset’s state representation into a top-down grid view, where each pixel corresponds to the ID of the block locatedat that position.
Revised: To model the dataset’s world state sequences in the VGPframework, we translate the 3D Cartesian coordinates of each block given by the dataset’s state representation into a top-down grid view, where each pixel corresponds to the ID of the block locatedat that position.

Original: We refer to this image as the scene’s “state representation.”
Revised: We refer to this image as the scene’s “state representation.”

Original: While the blocks dataset contains all the state information required to produce image-like state representations, this cannot generally be assumed to be the case.
Revised: While the blocks dataset contains all the state information required to produce image-like staterepresentations, this cannot generally be assumed to be the case.

Original: A more likely scenario mimicks Section4.1, where the model’s input is a combined color and depth image.
Revised: A more likely scenario mimicks Section 4.1, where the model’s input is a combined color and depth image.

Original: After translating  thedataset intothe simulated environment used in Section 4.1, we collect color and depth images for the wholeblocks dataset.
Revised: After translating thedataset into the simulated environment used in Section 4.1, we collect color and depth images forthe whole blocks dataset.

Original: In the original dataset, each block type is identiﬁed by a company logo.
Revised: In the original dataset, each block type is identiﬁed by a company logo.

Original: Since thelogos were downsampled by the simulator, we also assign a color to each type.
Revised: Since the logos were downsampled by the simulator, we also assign a color to each type.

Original: Some limitationsof the dataset include the demographics and labor practices underlying its original creation are unknown [10, 41] (our revisions were made by the authors); and it is intended for robotics research,not commercial deployment.
Revised: Thedemographics and labor practices underlying its original creation are unknown [10, 44]. We ﬁlteredtheir dataset without crowdworkers; and it is intended for robotics research, not product deployment.

Original: Experiments and results for this dataset are in Sec.
Revised: Experiments and results for this dataset are in Sec.

Original:  Experiments
Revised:  Experiments

Original: 5.1 Simulation Experiments with Synthetic Commands
Revised: 5.1 Simulation Experiments with Synthetic Commands

Original: An operator guiding a robot through a manipulation task may have a particular outcome in mind: forexample, a stack with a particular order of blocks.
Revised: An operator guiding a robot through a manipulation task may have a particular outcome in mind: forexample, a stack with a particular order of blocks.

Original: Language can naturally express such constraints; i.e. “stack the red block on the yellow block.”
Revised: Language can naturally express such constraints; i.e. “stack the red block on the yellow block.”

Original: Following the template generation and data annotationin Section 4.1, we train baseline and Transformer VGP models to produce output masks ˆ M grasp andˆ M place and combine them with a SPOT-Q model to manipulate blocks in a simulated environment.
Revised: Following the template generation and data annotationin Section 4.1, we train baseline and Transformer VGP models to produce output masks ˆ M grasp andˆ M place and combine them with a SPOT-Q model to manipulate blocks in a simulated environment.

Original: These masks are ﬁltered and thresholded using 0-1 block masks, following Hundt et al.
Revised: These masks are ﬁltered and thresholded using 0-1 block masks, following Hundt et al.

Original: Toevaluate the VGP system in isolation, we report the following metrics:
Revised: Toevaluate the VGP system in isolation, we report the following metrics:

Original: • Block Accuracy is the percentage predictions for which the grasp mask overlaps with the
Revised: • Block Accuracy is the percentage predictions for which the grasp mask overlaps with the

Original: correct block,
Revised: correct block,

Original: i
Revised: i

Original: e. the model chooses the correct block to pick up.
Revised: e. the model chooses the correct block to pick up.

Original: Higher is better.
Revised: Higher is better.

Original: • Idealized Error Score (IES) captures the performance of an idealized robot on the manipulation task, choosing a block to move using the grasp mask, and a location using the
Revised: • Idealized Error Score (IES) captures the performance of an idealized robot on the manipulation task, choosing a block to move using the grasp mask, and a location using the

Original: place mask.
Revised: place mask.

Original: After the chosen block is teleported to the predicted location, we measure the
Revised: After the chosen block is teleported to the predicted location, we measure the

Original: Euclidean distance between the block’s new location and its reference location in block
Revised: Euclidean distance between the block’s new location and its reference location in block

Original: lengths.
Revised: lengths.

Original: Lower is better.
Revised: Lower is better.

Original: Our baseline is a UNet-based VGP model; speciﬁcally, we extend the L ING UN ET model presentedby Misra et al.
Revised: Our baseline is a UNet-based VGP model; speciﬁcally, we extend the L ING UN ET model presentedby Misra et al.

Original: [4] with an attentional fusion mechanism.
Revised: [4] with an attentional fusion mechanism.

Original: While the original L ING UN ET used theﬁnal state of an LSTM for the language representation, we use a weighted average of BiLSTM statesproduced by an attention mechanism.
Revised: While the original L ING UN ET used theﬁnal state of an LSTM for the language representation, we use a weighted average of BiLSTM statesproduced by an attention mechanism.

Original: We also ran a small end-to-end Q-Learning experiment thatbackpropagates through both the FCN and Transformer, but the model did not converge.
Revised: We also ran a small end-to-end Q-Learning experiment thatbackpropagates through both the FCN and Transformer, but the model did not converge.

Original: [Table]
Revised: [Table]

Original: and
Revised: and

Original: [Table]
Revised: [Table]

Original: Score (IES, measured in blocks), is better.
Revised: Score (IES, measured in blocks), is better.

Original: TFMR is
Revised: TFMR is

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Stacks Stacks 27.9% 1.
Revised: Stacks Stacks 27.9% 1.

Original: TFMR Stacks Stacks 81.5% 0.
Revised: TFMR Stacks Stacks 81.5% 0.

Original: UNet
Revised: UNet

Original: Both
Revised: Both

Original: Stacks 75.2% 0.
Revised: Stacks 75.2% 0.

Original: TFMR
Revised: TFMR

Original: Both
Revised: Both

Original: Stacks 90.4% 0.
Revised: Stacks 90.4% 0.

Original: Further details on the masking, as well as the simulation environment, can be found in Appendix B.1.
Revised: Further details on the masking, as well as the simulation environment, can be found in Appendix B.1.

Original: The ﬁltered/thresholded masks ˆ M grasp and ˆ M placeare stacked into a single mask ˆ M 2 R n ⇥ n ⇥ 2 .
Revised: The ﬁltered/thresholded masks ˆ M grasp and ˆ M placeare stacked into a single mask ˆ M 2 R n ⇥ n ⇥ 2 .

Original: A ﬁnalmatrix of Q 0 of Q values constrained by the languageis given by Q 0 =
Revised: A ﬁnalmatrix of Q 0 of Q values constrained by the languageis given by Q 0 =

Original: ˆ M  Q , where  is the Hadamardproduct; Q 0 is then used to choose an action and alocation at which to grasp/place in the simulation.
Revised: ˆ M  Q , where  is the Hadamardproduct; Q 0 is then used to choose an action and alocation at which to grasp/place in the simulation.

Original: We count stacks and rows as successful only withthe speciﬁed color order, plus we tally the number ofactions this required, and the percentage of times thegripper picked up the correct block.
Revised: We count stacks and rows as successful only withthe speciﬁed color order, plus we tally the number ofactions this required, and the percentage of times thegripper picked up the correct block.

Original: We contrast ourmodel with the SPOT-Q framework alone, which hasno access to the order constraints, and with our UNetbaseline.
Revised: We contrast ourmodel with the SPOT-Q framework alone, which hasno access to the order constraints, and with our UNetbaseline.

Original: Each model was evaluated with the same SPOT-Q model and tested on 100 trials.
Revised: Each model was evaluated with the same SPOT-Q model and tested on 100 trials.

Original: Hundtet al.
Revised: Hundtet al.

Original: [9] set a 30 action limit for failed trials; we added several evaluation heuristics to terminatefailed trials when an irrecoverable state was reached, measuring color ordering errors.
Revised: [9] set a 30 action limit for failed trials; we added several evaluation heuristics to terminatefailed trials when an irrecoverable state was reached, measuring color ordering errors.

Original: Stackingtasks use 8 block colors and row-making tasks have 4.
Revised: Stackingtasks use 8 block colors and row-making tasks have 4.

Original: [Table]
Revised: [Table]

Original: Table 1 shows the visual goal prediction metrics for all model variants trained on one or bothtasks.
Revised: Table 1 shows the visual goal prediction metrics for all model variants trained on one or bothtasks.

Original: We see that the Transformer-based models (TFMR) uniformly out-perform the UNetbaseline across all metrics, often by large margins.
Revised: We see that the Transformer-based models (TFMR) uniformly out-perform the UNetbaseline across all metrics, often by large margins.

Original: For both UNet and Transformer architectures, training on two tasks simultaneouslycovers a broader range of semantically different examples with a corresponding increase inperformance on all metrics.
Revised: For both UNet and Transformer architectures, training on two tasks simultaneouslycovers a broader range of semantically different examples with a corresponding increase inperformance on all metrics.

Original: Notably, these examples differ by task: ﬁrstly, the row-makingtasks contain a subset of the colors available in stacking.
Revised: Notably, these examples differ by task: ﬁrstly, the row-makingtasks contain a subset of the colors available in stacking.

Original: Secondly, while the grasp data acrossboth tasks looks roughly identical, the data for place masks looks quite different.
Revised: Secondly, while the grasp data acrossboth tasks looks roughly identical, the data for place masks looks quite different.

Original: For example, both“stack the blue block on the red block” and “move the blue block next to the red block” result ingrasp masks over the blue block, but whereas the place mask for stacking picks out the block onwhich to stack (the red block), the row mask speciﬁes a location near the target block.
Revised: For example, both“stack the blue block on the red block” and “move the blue block next to the red block” result ingrasp masks over the blue block, but whereas the place mask for stacking picks out the block onwhich to stack (the red block), the row mask speciﬁes a location near the target block.

Original: Thus, it isintriguing that combining tasks beneﬁts not only the block accuracy, but also the IES.
Revised: Thus, it isintriguing that combining tasks beneﬁts not only the block accuracy, but also the IES.

Original: Table 2 shows the task completion percentage for the best UNet and Transformer models (trainedon both tasks) in the simulated environment, as well as the percentage of successful grasps thatpicked up the correct color, and the average number of actions taken to complete a successful trial.
Revised: Table 2 shows the task completion percentage for the best UNet and Transformer models (trainedon both tasks) in the simulated environment, as well as the percentage of successful grasps thatpicked up the correct color, and the average number of actions taken to complete a successful trial.

Original: In the aggregate, we see that high performance on VGP metrics does not translate to equivalent performance in the simulation environment.
Revised: In the aggregate, we see that high performance on VGP metrics does not translate to equivalent performance in the simulation environment.

Original: For example, both for stacking and row-making, thepercentage of correct color grasps is substantially lower that the VGP equivalent metric, block accuracy.
Revised: For example, both for stacking and row-making, thepercentage of correct color grasps is substantially lower that the VGP equivalent metric, block accuracy.

Original: For the Transformer model, this may be due to the imprecision of the patch-wise tiling ofthe image: when heuristically combined with a common-sense mask, a patch may cover multipleadjacent blocks.
Revised: For the Transformer model, this may be due to the imprecision of the patch-wise tiling ofthe image: when heuristically combined with a common-sense mask, a patch may cover multipleadjacent blocks.

Original: This is an inherent limitation of our approach.
Revised: This is an inherent limitation of our approach.

Original: When compared to the RL-only and
Revised: When compared to the RL-only and

Original: UNet baselines, our Transformer-based model succeeds in signiﬁcantly more trials.
Revised: UNet baselines, our Transformer-based model succeeds in signiﬁcantly more trials.

Original: Additionally,the mean number of actions across successful trials is lower, indicating that the Transformer-basedmasks result in fewer actions per successful trial.
Revised: Additionally,the mean number of actions across successful trials is lower, indicating that the Transformer-basedmasks result in fewer actions per successful trial.

Original: Note that for stacking, both baselines succeededon 

Revised: Note that for stacking, both baselines succeededon

Original: out of the 100 trials; due to toppling and occlusion, stacking poses a signiﬁcantly harder taskthan row-making, and it is harder to recover from a misplaced block, even if the trial is not ended.
Revised: out of the 100 trials; due to toppling and occlusion, stacking poses a signiﬁcantly harder taskthan row-making, and it is harder to recover from a misplaced block, even if the trial is not ended.

Original: We see that even a fairly high block accuracy and low IES are insufﬁcient for high task performance.
Revised: We see that even a fairly high block accuracy and low IES are insufﬁcient for high task performance.

Original: This highlights the one of the main challenges of multistep tasks: small error rates compound acrossmultiple actions, yielding low task performance.
Revised: This highlights the one of the main challenges of multistep tasks: small error rates compound acrossmultiple actions, yielding low task performance.

Original: A full error analysis is given in Appendix B.

Revised: A full error analysis is given in Appendix B.

Original: Transfer to Real Images – In Figure 3, we plot the
Revised: Transfer to Real Images – In Figure 3, we plot the

Original: IES of row-making and stacking models when trainedon increasing subsets of real data.
Revised: IES of row-making and stacking models when trainedon increasing subsets of real data.

Original: We see that for bothrows and stacks, performance improves slightly as we increase the amount of training data, with diminishing returns after 30% of the available data (1794 examples) areused.
Revised: We see that for bothrows and stacks, performance improves slightly as we increase the amount of training data, with diminishing returns after 30% of the available data (1794 examples) areused.

Original: Given that Transformers typically require severalorders of magnitude more data to reach good performance[42, 37, 43], the fact that our architecture can be trainedwith mere thousands of simulated training examples, andsuccessfully ﬁnetuned with even fewer real examples, extracted from only a few hundred real images, represents a signiﬁcant achievement.
Revised: Given that Transformers typically require severalorders of magnitude more data to reach good performance[45, 40, 46], the fact that our architecture can be trainedwith mere thousands of simulated training examples, andsuccessfully ﬁnetuned with even fewer real examples, extracted from only a few hundred real images, represents a signiﬁcant achievement.

Original: For a proof-of-concept demonstration of our model running on a real robot, see Appendix B.4 and the supplementary materials.
Revised: For a proof-of-concept demonstration of our model running on a real robot, see Appendix B.4 and the supplementary materials.

Original: Compositionality – By generating held-out data splits as described in Section 4.3, we can testthe ability of our model to generalize compositionally, i.e. to recombine known symbols in unseencontexts.
Revised: Compositionality – By generating held-out data splits as described in Section 4.3, we can testthe ability of our model to generalize compositionally, i.e. to recombine known symbols in unseencontexts.

Original: A model that has learned compositional color representations will be able to combinecolor representations across contexts; for example, if it has seen red blocks in the context of greenblocks, it should generalize to a blue block context.
Revised: A model that has learned compositional color representations will be able to combinecolor representations across contexts; for example, if it has seen red blocks in the context of greenblocks, it should generalize to a blue block context.

Original: We would therefore expect the model to performroughly on par with a model trained on a random data split, where all contexts are seen at train time.
Revised: We would therefore expect the model to performroughly on par with a model trained on a random data split, where all contexts are seen at train time.

Original: Conversely, if the model has learned to interpret each utterance as an atomic statement, we shouldexpect much lower performance on the held-out subset.
Revised: Conversely, if the model has learned to interpret each utterance as an atomic statement, we shouldexpect much lower performance on the held-out subset.

Original: [Table]
Revised: [Table]

Original: Our results are reported in Table 3.
Revised: Our results are reported in Table 3.

Original: Both rows andstacks see a slight drop in block accuracy from the baseline.
Revised: Both rows andstacks see a slight drop in block accuracy from the baseline.

Original: Nevertheless, the lack of a large disparity betweenthe baseline and the held-out condition suggests that themodel has largely learned to generalize compositionallyto unseen block combinations.
Revised: Nevertheless, the lack of a large disparity betweenthe baseline and the held-out condition suggests that themodel has largely learned to generalize compositionallyto unseen block combinations.

Original: This ﬁnding suggests that,although the model certainly has the capacity to memorize 23 combinations, and although it is not in any waytrained to perform compositional reasoning, it naturallylearns to bind symbols to percepts in a way that generalizes across contexts (i.e. “red block” refersto one particular block, regardless of its context).
Revised: This ﬁnding suggests that,although the model certainly has the capacity to memorize 23 combinations, and although it is not in any waytrained to perform compositional reasoning, it naturallylearns to bind symbols to percepts in a way that generalizes across contexts (i.e. “red block” refersto one particular block, regardless of its context).

Original: These result are promising but limited to simulatedlanguage; accordingly, in Section 5.2 we evaluate our results with real language data.
Revised: These result are promising but limited to simulatedlanguage; accordingly, in Section 5.2 we evaluate our results with real language data.

Original: 5.2 Naturalistic Language Commands with Complex Scenes
Revised: 5.2 Naturalistic Language Commands with Complex Scenes

Original: In this experiment we model the naturalistic data described in Section 4.4, where state representations are coupled instead with diverse crowdsourced commands, in contrast to the template generated commands in Section 5.1.
Revised: In this experiment we model the naturalistic data described in Section 4.4, where state representations are coupled instead with diverse crowdsourced commands, in contrast to the template generated commands in Section 5.1.

Original: We train models on state representations (cf.
Revised: We train models on state representations (cf.

Original: Section 4.4) as well asbirds-eye color and depth images.
Revised: Section 4.4) as well asbirds-eye color and depth images.

Original: This highlights the ﬂexibility of our model: we are able to trainfrom images and from state representations with no modiﬁcations to the architecture.
Revised: This highlights the ﬂexibility of our model: we are able to trainfrom images and from state representations with no modiﬁcations to the architecture.

Original: Large pre-trained encoders have been shown to encode useful linguistic features at a number oflevels of analysis [44, 45, 46]; accordingly, we make use of bert-base-cased [47] as an inputfeature, contrasting it with GloVe [48] word embeddings.
Revised: Large pre-trained encoders have been shown to encode useful linguistic features at a number oflevels of analysis [47, 48, 49]; accordingly, we make use of bert-base-cased [50] as an inputfeature, contrasting it with GloVe [51] word embeddings.

Original: Tuning and metrics are as in Sec. 5.1.
Revised: Tuning and metrics are as in Sec. 5.1.

Original: [Table]
Revised: [Table]

Original: Table 4 shows the VGP metrics for modelstrained from image-like representations ofthe state (described in Section 4.4) as wellas from images.
Revised: Table 4 shows the VGP metrics for modelstrained from image-like representations ofthe state (described in Section 4.4) as wellas from images.

Original: For brevity, only thebest model for each embedding type is reported, with full results in Appendix C.5.
Revised: For brevity, only thebest model for each embedding type is reported, with full results in Appendix C.5.

Original: On state representations, the Transformerbased models show a clear advantage overthe baseline, with IES being substantiallylower for both Transformer models.
Revised: On state representations, the Transformerbased models show a clear advantage overthe baseline, with IES being substantiallylower for both Transformer models.

Original: Whenusing GloVe embeddings with state representation input, 2 ⇥ 2 patches resulted in the best performance, while a patch size of 4 yielded thebest results for BERT embeddings.
Revised: Whenusing GloVe embeddings with state representation input, 2 ⇥ 2 patches resulted in the best performance, while a patch size of 4 yielded thebest results for BERT embeddings.

Original: Overall, while the BERT-based model has slightly higher blockaccuracy, the IES are equal between the two models when trained on state representations.
Revised: Overall, while the BERT-based model has slightly higher blockaccuracy, the IES are equal between the two models when trained on state representations.

Original: Whilethese metrics measure performance per step, the results here suggest that our  system can be appliedto real language as well as simulated language.
Revised: Whilethese metrics measure performance per step, the results here suggest that our masking system canbe applied to real language as well as simulated language.

Original: In Table 4, we also see that the performance shifts substantially when changing the input from stateto images.
Revised: In Table 4, we also see that the performance shifts substantially when changing the input from stateto images.

Original: While the baseline improves slightly in IES, the Transformers’ performance decreasesdramatically.
Revised: While the baseline improves slightly in IES, the Transformers’ performance decreasesdramatically.

Original: These results stand in contrast to those in Section 5.1, where a Transformer trainedon images unequivocally outperformed the baseline.
Revised: These results stand in contrast to those in Section 5.1, where a Transformer trainedon images unequivocally outperformed the baseline.

Original: Nevertheless, the best Transformer model stilloutperforms the baseline.
Revised: Nevertheless, the best Transformer model stilloutperforms the baseline.

Original: We speculate that the baseline models may perform better here becausethe architecture was developed for image segmentation problems.
Revised: We speculate that the baseline models may perform better here becausethe architecture was developed for image segmentation problems.

Original: Interestingly, while a BERT-based
Revised: Interestingly, while a BERT-based

Original: Transformer performed well when trained from state, the same models’ performance here is greatlydiminished.
Revised: Transformer performed well when trained from state, the same models’ performance here is greatlydiminished.

Original: Improving the Transformer’s real-world performance is a direction for future work.
Revised: Improving the Transformer’s real-world performance is a direction for future work.

Original: Data Translation and Filtering
Revised: Data Translation and Filtering

Original: In Section 5.1, we observed that high performance in a static evaluation against a dataset did not entail equally high performance when using a realistic robot formulti-step tasks in a simulated environment.
Revised: In Section 5.1, we observed that high performance in a static evaluation against a dataset did not entail equally high performance when using a realistic robot formulti-step tasks in a simulated environment.

Original: In its current form, the blocks dataset can only be evaluated using static metrics.
Revised: In its current form, the blocks dataset can only be evaluated using static metrics.

Original: Furthermore, the images provided in the dataset are taken from a ﬁxed 45degree projection, and our method assumes access to birds-eye images.
Revised: Furthermore, the images provided in the dataset are taken from a ﬁxed 45degree projection, and our method assumes access to birds-eye images.

Original: To obtain birds-eye imagesof the environment and to facilitate further research into combining rich, naturalistic language withaction in manipulation tasks, we translate the dataset into the same environment used in Section 5.1,and ﬁlter the dataset to only physically feasible examples; this process is described in Section 4.4.
Revised: To obtain birds-eye imagesof the environment and to facilitate further research into combining rich, naturalistic language withaction in manipulation tasks, we translate the dataset into the same environment used in Section 5.1,and ﬁlter the dataset to only physically feasible examples; this process is described in Section 4.4.

Original: ⇤ indicates signiﬁcant improvement over UNet as per the Wilcoxon ranked-sign test.
Revised: ⇤ indicates signiﬁcant improvement over UNet as per the Wilcoxon ranked-sign test.

Original: [Table]
Revised: [Table]

Original: Table 5 reports the VGP metrics on the thephysically feasible subset of the data for the
Revised: Table 5 reports the VGP metrics on the thephysically feasible subset of the data for the

Original: Transformer models.
Revised: Transformer models.

Original: Firstly, we see thatperformance decreases across most metrics.
Revised: Firstly, we see thatperformance decreases across most metrics.

Original: However, for UNet, IES improves onthe physically feasible subset.
Revised: However, for UNet, IES improves onthe physically feasible subset.

Original: In contrast,the Transformer models decrease in performance across both metrics.
Revised: In contrast,the Transformer models decrease in performance across both metrics.

Original: Nevertheless,the Transformer model still performs on parwith the baseline on IES, and substantially better on block accuracy.
Revised: Nevertheless,the Transformer model still performs on parwith the baseline on IES, and substantially better on block accuracy.

Original: These results suggest that thephysically feasible subset is more challenging to model, even when evaluating the models statically.
Revised: These results suggest that thephysically feasible subset is more challenging to model, even when evaluating the models statically.

Original: In other words, even when the evaluation and models are physically unconstrained, modeling onlyphysically feasible scenes is harder than modeling all scenes.
Revised: In other words, even when the evaluation and models are physically unconstrained, modeling onlyphysically feasible scenes is harder than modeling all scenes.

Original: This result highlights the difﬁculty ofdeveloping embodied agents and the importance of testing them in realistic environments.
Revised: This result highlights the difﬁculty ofdeveloping embodied agents and the importance of testing them in realistic environments.

Original:  Conclusion
Revised:  Conclusion

Original: We have presented a general approach to multi-step rearrangement and manipulation tasks that factorizes each step into learning how to act and learning where to act.
Revised: We have presented a general approach to multi-step rearrangement and manipulation tasks that factorizes each step into learning how to act and learning where to act.

Original: To decide where to act, wehave presented a Transformer-based model for converting state representations and natural languageinstructions to masks over an action space that learns with remarkably few examples.
Revised: To decide where to act, wehave presented a Transformer-based model for converting state representations and natural languageinstructions to masks over an action space that learns with remarkably few examples.

Original: We haveshown the ability of our model to act in a realistic simulated environment for two multi-step blockrearrangement tasks, and naturally learn a compositional mapping from text to actions in a perceptual space.
Revised: We haveshown the ability of our model to act in a realistic simulated environment for two multi-step blockrearrangement tasks, and naturally learn a compositional mapping from text to actions in a perceptual space.

Original: We have demonstrated our model performs well under both these realistic physicalassumptions and with real and complex natural language via a dataset of crowdsourced instructions.
Revised: We have demonstrated our model performs well under both these realistic physicalassumptions and with real and complex natural language via a dataset of crowdsourced instructions.