Original: Guiding Multi-Step Rearrangement Tasks with Natural Language Instructions
Revised: Guiding Multi-Step Rearrangement Tasks with Natural Language Instructions

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract: Enabling human operators to interact with robotic agents using naturallanguage would allow non-experts to intuitively instruct these agents.
Revised: Abstract: Enabling human operators to interact with robotic agents using naturallanguage would allow non-experts to intuitively instruct these agents.

Original: Towardsthis goal, we propose a novel Transformer-based model which enables a user toguide a robot arm through a 3D multi-step manipulation task with natural language commands.
Revised: Towardsthis goal, we propose a novel Transformer-based model which enables a user toguide a robot arm through a 3D multi-step manipulation task with natural language commands.

Original: Our system maps images and commands to masks over graspor place locations, grounding the language directly in perceptual space.
Revised: Our system maps images and commands to masks over graspor place locations, grounding the language directly in perceptual space.

Original: In a suiteof block rearrangement tasks, we show that these masks can be combined with anexisting manipulation framework without re-training, greatly improving learningefﬁciency.
Revised: In a suiteof block rearrangement tasks, we show that these masks can be combined with anexisting manipulation framework without re-training, greatly improving learningefﬁciency.

Original: Our masking model is several orders of magnitude more sample efﬁcient than typical Transformer models, operating with hundreds, not millions, ofexamples.
Revised: Our masking model is several orders of magnitude more sample efﬁcient than typical Transformer models, operating with hundreds, not millions, ofexamples.

Original: Our modular design allows us to leverage supervised and reinforcementlearning, providing an easy interface for experimentation with different architectures.
Revised: Our modular design allows us to leverage supervised and reinforcementlearning, providing an easy interface for experimentation with different architectures.

Original: Our model completes block manipulation tasks with synthetic commands530% more often than a UNet-based baseline, and learns to localize actions correctly while creating a mapping of symbols to perceptual input that supports compositional reasoning.
Revised: Our model completes block manipulation tasks with synthetic commands530% more often than a UNet-based baseline, and learns to localize actions correctly while creating a mapping of symbols to perceptual input that supports compositional reasoning.

Original: We provide a valuable resource for 3D manipulation instruction following research by porting an existing 3D block dataset with crowdsourcedlanguage to a simulated environment.
Revised: We provide a valuable resource for 3D manipulation instruction following research by porting an existing 3D block dataset with crowdsourcedlanguage to a simulated environment.

Original: Our method’s 25 .
Revised: Our method’s 25 .

Original: 3% absolute improvementin identifying the correct block on the ported dataset demonstrates its ability tohandle syntactic and lexical variation.
Revised: 3% absolute improvementin identifying the correct block on the ported dataset demonstrates its ability tohandle syntactic and lexical variation.

Original:  Introduction
Revised:  Introduction

Original: Reinforcement Learning (RL) is a powerful tool for developing perception-driven robotic systems.
Revised: Reinforcement Learning (RL) is a powerful tool for developing perception-driven robotic systems.

Original: Most RL algorithms train an agent to perform a single task based on an given reward function.
Revised: Most RL algorithms train an agent to perform a single task based on an given reward function.

Original: However, what if we would like to adapt or extend the learned policy of a robot to novel or morespecialized tasks by providing instructions?
Revised: However, what if we would like to adapt or extend the learned policy of a robot to novel or morespecialized tasks by providing instructions?

Original: Could we exploit previously learned RL policies for arobot to solve novel tasks now speciﬁed using natural language?
Revised: Could we exploit previously learned RL policies for arobot to solve novel tasks now speciﬁed using natural language?

Original: The ability to adapt learned policiesthrough natural language to perform novel tasks would open the door to a modular integration ofperception, language, and action, avoiding the complexities of joint training over all modalities simultaneously.
Revised: The ability to adapt learned policiesthrough natural language to perform novel tasks would open the door to a modular integration ofperception, language, and action, avoiding the complexities of joint training over all modalities simultaneously.

Original: For example, in our experiments, we constrain a general-purpose block manipulationpolicy using natural language, without needing any language data to train the policy itself.
Revised: For example, in our experiments, we constrain a general-purpose block manipulationpolicy using natural language, without needing any language data to train the policy itself.

Original: This kind of modular learning requires injecting guidance from language into the RL-driven actionpolicy while simultaneously addressing the challenges of natural language understanding .
Revised: This kind of modular learning requires injecting guidance from language into the RL-driven actionpolicy while simultaneously addressing the challenges of natural language understanding .

Original: Forexample, natural language semantics is often underspeciﬁed, especially with respect to a physicalenvironment; correctly choosing one out of many potentially valid actions for a command suchas “Move the green block to the left of the right block” involves pragmatic and common-sensereasoning.
Revised: Forexample, natural language semantics is often underspeciﬁed, especially with respect to a physicalenvironment; correctly choosing one out of many potentially valid actions for a command suchas “Move the green block to the left of the right block” involves pragmatic and common-sensereasoning.

Original: Furthermore, in order to map from language to action, we need to address the symbolgrounding problem [1], i.e. how does one map symbols (e.g. “left”, “right”, “block”, “red”) to thephysical world?
Revised: Furthermore, in order to map from language to action, we need to address the symbolgrounding problem [1], i.e. how does one map symbols (e.g. “left”, “right”, “block”, “red”) to thephysical world?

Original: We leverage Transformer encoders to learn a strategy from data, in place of themanually deﬁned data fusion strategies found in prior work [2, 3, 4, 5, 6, 7, 8].
Revised: We leverage Transformer encoders to learn a strategy from data, in place of themanually deﬁned data fusion strategies found in prior work [2, 3, 4, 5, 6, 7, 8].

Original: [Figure]
Revised: [Figure]

Original: In our work, we factorize language-driven action into learning how to act and learning where to act ,following Misra et al.
Revised: In our work, we factorize language-driven action into learning how to act and learning where to act ,following Misra et al.

Original: Learning how to act is accomplished by the SPOT-Q algorithm [9] whichteaches an agent to re-arrange colored blocks into stacks and rows by grasping and placing them.
Revised: Learning how to act is accomplished by the SPOT-Q algorithm [9] whichteaches an agent to re-arrange colored blocks into stacks and rows by grasping and placing them.

Original: We use SPOT-Q models pre-trained without any language data for our manipulation experiments,demonstrating that our method lets us leverage existing manipulation models without additionaltraining.
Revised: We use SPOT-Q models pre-trained without any language data for our manipulation experiments,demonstrating that our method lets us leverage existing manipulation models without additionaltraining.

Original: SPOT-Q identiﬁes regions worth exploring, e.g. containing an object, to encode a setof valid action regions for grasping and placing.
Revised: SPOT-Q identiﬁes regions worth exploring, e.g. containing an object, to encode a setof valid action regions for grasping and placing.

Original: We produce segmentation masks that are jointlyconditioned on both the language input and an image of the state and apply them with to set ofpixel-wise Q-values from the SPOT-Q RL framework.
Revised: We produce segmentation masks that are jointlyconditioned on both the language input and an image of the state and apply them with to set ofpixel-wise Q-values from the SPOT-Q RL framework.

Original: This process provides us with the best actionfor a given state and command, as Figure 1 illustrates.
Revised: This process provides us with the best actionfor a given state and command, as Figure 1 illustrates.

Original: Factorizing the learning process this way provides several advantages.
Revised: Factorizing the learning process this way provides several advantages.

Original: Firstly, it offers a moreefﬁcient path to learning.
Revised: Firstly, it offers a moreefﬁcient path to learning.

Original: Separating the language grounding model from the reinforcement learningmodel, leads to a less complex RL model that receives more direct feedback for tasks.
Revised: Separating the language grounding model from the reinforcement learningmodel, leads to a less complex RL model that receives more direct feedback for tasks.

Original: Moreover,the separation of action and language understanding allows us to learn the language component viasupervised learning, which is typically more sample efﬁcient.
Revised: Moreover,the separation of action and language understanding allows us to learn the language component viasupervised learning, which is typically more sample efﬁcient.

Original: Secondly, the modular nature of themodel allows us to swap components easily: we can train a masking model on simulated imagesor real images (cf.
Revised: Secondly, the modular nature of themodel allows us to swap components easily: we can train a masking model on simulated imagesor real images (cf.

Original: Section 5.1), allowing us to use the same action model for both simulated andreal tasks without re-training.
Revised: Section 5.1), allowing us to use the same action model for both simulated andreal tasks without re-training.

Original: In an end-to-end model, a change to the input space would likelyrequire re-training from scratch.
Revised: In an end-to-end model, a change to the input space would likelyrequire re-training from scratch.

Original: Finally, the factorized model output is more interpretable; whenan action is unsuccessful, we can inspect our model to localize the point of failure, be it a lack ofcomprehension or a mistake while acting.
Revised: Finally, the factorized model output is more interpretable; whenan action is unsuccessful, we can inspect our model to localize the point of failure, be it a lack ofcomprehension or a mistake while acting.

Original: This may not be evident from the agent’s action alone.
Revised: This may not be evident from the agent’s action alone.

Original: Note that our implementation of this factorization is not without disadvantages: if the languagecontains information on how to act this information could be lost.
Revised: Note that our implementation of this factorization is not without disadvantages: if the languagecontains information on how to act this information could be lost.

Original: In summary, our contributions are:
Revised: In summary, our contributions are:

Original: We introduce a novel Transformer-based model for mapping language commands to masks
Revised: We introduce a novel Transformer-based model for mapping language commands to masks

Original: over an action space.
Revised: over an action space.

Original: Our approach’s modular nature enables us to seamlessly integrate
Revised: Our approach’s modular nature enables us to seamlessly integrate

Original: an
Revised: an

Original: existing Q-learning framework proposed by Hundt et al.
Revised: existing Q-learning framework proposed by Hundt et al.

Original: [9] for block manipulation tasks.
Revised: [9] for block manipulation tasks.

Original: We show our model succeeds in a variety of simulated and real data settings with orders of
Revised: We show our model succeeds in a variety of simulated and real data settings with orders of

Original: magnitude less data than typically available to Transformers, furthering the potential use of
Revised: magnitude less data than typically available to Transformers, furthering the potential use of

Original: Transformers in the low-data regimes commonly found in robotics.
Revised: Transformers in the low-data regimes commonly found in robotics.

Original: We demonstrate this model’s utility on an existing dataset (introduced by Bisk et al.
Revised: We demonstrate this model’s utility on an existing dataset (introduced by Bisk et al.

Original: involving a challenging block re-arrangement task with crowdsourced natural languagecommands containing syntactic diversity as well as complex natural language phenomena.4.
Revised: involving a challenging block re-arrangement task with crowdsourced natural languagecommands containing syntactic diversity as well as complex natural language phenomena.4.

Original: We release our real and simulated datasets, plus a ﬁltered version of the Bisk et al.
Revised: We release our real and simulated datasets, plus a ﬁltered version of the Bisk et al.

Original: [10]dataset containing the subset of examples feasible under a realistic model of physics.
Revised: [10]dataset containing the subset of examples feasible under a realistic model of physics.

Original:  Related Work
Revised:  Related Work

Original: We describe related work in the areas of natural language instruction following on robots, QLearning for multistep manipulation tasks, and Transformer models for Computer Vision.
Revised: We describe related work in the areas of natural language instruction following on robots, QLearning for multistep manipulation tasks, and Transformer models for Computer Vision.

Original: Robots and Natural Language Commands – We follow the Problem-Solution Sequences (PSS)paradigm [11], where a multi-step manipulation task (e.g. “build a stack”) is framed as a sequence of incremental images or state representations.
Revised: Robots and Natural Language Commands – We follow the Problem-Solution Sequences (PSS)paradigm [11], where a multi-step manipulation task (e.g. “build a stack”) is framed as a sequence of incremental images or state representations.

Original: These may be paired with per-step instructions onhow to accomplish the intermediate goal (e.g., “stack the red block on the blue block”).
Revised: These may be paired with per-step instructions onhow to accomplish the intermediate goal (e.g., “stack the red block on the blue block”).

Original: Several
Revised: Several

Original: PSS datasets have been proposed [11, 10] and modeled in 2D [4, 12] and 3D [10].
Revised: PSS datasets have been proposed [11, 10] and modeled in 2D [4, 12] and 3D [10].

Original: We translate the3D dataset proposed by Bisk et al.
Revised: We translate the3D dataset proposed by Bisk et al.

Original: [10] to a simulated environment in Section 5.2.
Revised: [10] to a simulated environment in Section 5.2.

Original: The sequentialnature of PSS relates to the hierarchical RL setting explored by Andreas et al.
Revised: The sequentialnature of PSS relates to the hierarchical RL setting explored by Andreas et al.

Original: [13], who learn taskspeciﬁc sub-policies indexed by linguistic sub-goals.
Revised: [13], who learn taskspeciﬁc sub-policies indexed by linguistic sub-goals.

Original: Similar approaches are applied to zero-shotsub-goal sequence generalization[14], non-linearly-ordered sub-goal execution[15], and choosinginterpretable linguistic sub-goals [16].
Revised: Similar approaches are applied to zero-shotsub-goal sequence generalization[14], non-linearly-ordered sub-goal execution[15], and choosinginterpretable linguistic sub-goals [16].

Original: While we also assume access to step-wise annotated instructions, we focus on guiding a low-level robotic manipulation policy with a supervised masking modelrather than a task-level policy.
Revised: While we also assume access to step-wise annotated instructions, we focus on guiding a low-level robotic manipulation policy with a supervised masking modelrather than a task-level policy.

Original: Gangopadhyay et al.
Revised: Gangopadhyay et al.

Original: [17] and Sun et al.
Revised: [17] and Sun et al.

Original: [18] learn policies guidedby symbolic programs, while Perera et al.
Revised: [18] learn policies guidedby symbolic programs, while Perera et al.

Original: [19] ﬁrst parse language into a symbolic representations.
Revised: [19] ﬁrst parse language into a symbolic representations.

Original: Our masking-based method ties the language directly to the perceptual space, eschewing intermediate symbol ontologies.
Revised: Our masking-based method ties the language directly to the perceptual space, eschewing intermediate symbol ontologies.

Original: Language has been used for reward-shaping [20, 21, 22], and reward modellearning [23]; we instead use the language-agnostic reward shaping in SPOT-Q. Stepputtis et al.[24] and Lynch and Sermanet [25] use language-conditioned imitation learning for manipulationpolicies; we instead use Q-learning.
Revised: Language has been used for reward-shaping [20, 21, 22], and reward modellearning [23]; we instead use the language-agnostic reward shaping in SPOT-Q. Stepputtis et al. [24]and Lynch and Sermanet [25] use language-conditioned imitation learning for manipulation policies;we instead use Q-learning.

Original: TransporterNets
Revised: TransporterNets

Original: [26] learn to perform multi-step tasks fromdemonstration, but lacks language input.
Revised: [26] learn to perform multi-step tasks from demonstration, but lacks language input.

Original: Like our work, Concept2Robot [27] completes languagedescribed tasks by combining learning from demonstration and RL.
Revised: Like our work, Concept2Robot [27] completes language-describedtasks by combining learning from demonstration and RL.

Original: However, it differs from oursystem along multiple axes: we examine challenging multi-step tasks with progress reversal [9]; wesucceed on much smaller datasets; we consider crowdsourced commands; and we evaluate on realrobot data.
Revised: However, it differs from our system alongmultiple axes: we examine challenging multi-step tasks with progress reversal [9]; we succeed onmuch smaller datasets; we consider crowdsourced commands; and we evaluate on real robot data.

Original: Our experiments involve block retrieval and placement; Nguyen et al.
Revised: Nguyen et al.

Original: [28] guide objectretrieval with language, but do not model placement.
Revised: [28] guide object retrieval with language, but do not model placement.

Original: Visual Goal Prediction (VGP), introduced by Misra et al.
Revised: Visual Goal Prediction (VGP), introduced by Misra et al.

Original: [4] for language-conditioned navigation[29, 30, 4, 8, 7, 5], frames the problem of choosing an action region as a semantic segmentation taskin which each pixel as a separate binary decision: act or do not act.
Revised: [4] for language-conditioned navigation[29, 30, 4, 8, 7, 5], frames the problem of choosing an action region as a semantic segmentation taskin which each pixel as a separate binary decision: act or do not act.

Original: VGP has directed a simulated[5] and real quadcopter [8, 7].
Revised: VGP has directed a simulated[5] and real quadcopter [8, 7].

Original: We implement VGP with UNet[31, 4, 5, 7, 8], as a baseline.
Revised: We implement VGP with UNet[31, 4, 5, 7, 8], as a baseline.

Revised: Q-learning for Multi-Step Tasks – Mohammed et al.

Original: Q-learning for Multi-Step Tasks – Q-Learning  for robotics is an active area of research into algorithms and supporting datasets [3], including various RL mechanisms for tabletop clutter removal[32, 33, 34, 35], and multi-step formal RL[14, 13], but NLP is not addressed.
Revised: [32] reviews RL for robot grasping. Zenget al. [33] shows Q-Learning outperforms supervised action segmentation for table decluttering[34,35, 36], instead, “Good Robot!” [9] segments actions worth exploring to accelerate Q-Learning ofsingle-task, multi-step robot learning, all without language input.

Original: “Good Robot!”
Revised: “Good Robot!”

Original: [9]was the ﬁrst to complete multi-step tasks with consideration of progress reversal , a failure casewhere actions undo previous progress such that ⌦ ( n ) additional actions are required to recover, e.g.toppling the tower in a partially completed stacking task.
Revised: [9] was also theﬁrst to complete multi-step tasks with consideration of progress reversal , a failure case where actionsundo previous progress such that Ω( n ) additional actions are required to recover, e.g. toppling thetower in a partially completed stacking task.

Original: “Good Robot!”

Original: [9] also shows that actionmasking (segmentation) accelerates robot learning. Mohammed et al. [36] reviews RL grasping.
Revised: We segment actions worth exploring via linguistic inputfor multi-step, multi-task execution considering progress reversal and low-data[3, 34] conditions.

Original: Transformer Encoders for Vision – Though Transformer encoders, which build representationsusing self-attention, have become standard in NLP, computer vision tasks are still dominated by convolutional models, of which UNet [31] is a paradigmatic instance.
Revised: Transformer Encoders for Vision – Though Transformer encoders, which build representationsusing self-attention, have become standard in NLP, computer vision tasks are still dominated by convolutional models, of which UNet [31] is a paradigmatic instance.

Original: The use of Transformers in visiontasks has been held back by the quadratic complexity of the Transformer’s self-attention layer; animage input treated as a sequence of pixels would quickly become computationally intractable [37].
Revised: The use of Transformers in visiontasks has been held back by the quadratic complexity of the Transformer’s self-attention layer; animage input treated as a sequence of pixels would quickly become computationally intractable [37].

Original: Dosovitskiy et al.
Revised: Dosovitskiy et al.

Original: [37] introduce a minimal modiﬁcation to perform image recognition using Transformers, tiling the image into square patches, which they concatenate into a sequence and augmentwith learned linear positional embeddings.
Revised: [37] introduce a minimal modiﬁcation to perform image recognition using Transformers, tiling the image into square patches, which they concatenate into a sequence and augmentwith learned linear positional embeddings.

Original: Though the model presented by Dosovitskiy et al.
Revised: Though the model presented by Dosovitskiy et al.

Original: performs image-level recognition well, it cannot perform pixel-level tasks (e.g., semantic segmentation)as-is.
Revised: performs image-level recognition well, it cannot perform pixel-level tasks (e.g., semantic segmentation)as-is.

Original: Ranftl et al.
Revised: Ranftl et al.

Original: [38] present an extension to this model for pixel-level prediction.
Revised: [38] present an extension to this model for pixel-level prediction.

Original:  Models
Revised:  Models

Original: We propose a Transformer-based model to convert image and natural language inputs into separatemasks for grasping and placing, thereby allowing users to guide multi-step manipulation tasks.
Revised: We propose a Transformer-based model to convert image and natural language inputs into separatemasks for grasping and placing, thereby allowing users to guide multi-step manipulation tasks.

Original: We begin by tiling the n ⇥ n birds-eye image of the scene with C colored blocks into k ⇥ k patches,whose RGB pixel values are concatenated into one vector, resulting in patch vectors
Revised: We begin by tiling the n × n birds-eye image of the scene with C colored blocks into k × k patches,whose RGB pixel values are concatenated into one vector, resulting in patch vectors

Original: I = i 1 , . .
Revised: I = i 1 , . .

Original: , i L .
Revised: , i L .

Original: The resulting vectors are augmented with a ﬁxed sinusoidal positional encoding [39].
Revised: The resulting vectors are augmented with a ﬁxed sinusoidal positional encoding [39].

Original: Similarly,each token in the command string W = w 1 , . . .
Revised: Similarly,each token in the command string W = w 1 , . . .

Original: w m is embedded into a continuous space resultingin vectors e 1 , . . .
Revised: w m is embedded into a continuous space resultingin vectors e 1 , . . .

Original: , e m , and those vectors are augmented with positional encodings as well.
Revised: , e m , and those vectors are augmented with positional encodings as well.

Original: The goal is to train a model that produces a pair of language-deﬁned spatial masks ˆ M place 2 R n ⇥
Revised: The goal is to train a model that produces a pair of language-deﬁned spatial masks ˆ M place ∈ R n × nand

Original: nand ˆ M grasp 2 R n ⇥ n , and an optional pair of state reconstructions
Revised: ˆ M grasp ∈ R n × n , and an optional pair of state reconstructions

Original: ˆ S grasp , ˆ S place 2 R C ⇥ n ⇥ n given by:
Revised: ˆ S grasp , ˆ S place ∈ R C × n × n given by:

Original: [Equation]
Revised: [Equation]

Original: where f . are parameterized by Transformer encoders with output size k , H m 2 R k ⇥ 2 , H s 2 R k ⇥ C , is the softmax function, and t 2 { grasp , place } .
Revised: where f . are parameterized by Transformer encoders with output size k , H m ∈ R k × 2 , H s ∈ R k × C ,σ is the softmax function, and t ∈ { grasp , place } .

Original: The reconstruction function R up-samples kpatches by copying the patch value across each of its constituent pixels, resulting in an n ⇥ n image.
Revised: The reconstruction function R up-samples kpatches by copying the patch value across each of its constituent pixels, resulting in an n × n image.

Original: Note that structurally, the “shared,” “grasp,” and “place” Transformers are identical, but while the“shared” Transformer is used both for grasp and place prediction, the others are task-speciﬁc.
Revised: Note that structurally, the “shared,” “grasp,” and “place” Transformers are identical, but while the“shared” Transformer is used both for grasp and place prediction, the others are task-speciﬁc.

Original: We train the masking model separately from the manipulation model.
Revised: We train the masking model separately from the manipulation model.

Original: For the latter model, weuse a pre-trained SPOT-Q model from Hundt et al.
Revised: For the latter model, weuse a pre-trained SPOT-Q model from Hundt et al.

Original: [9], which produces pixel-wise Q-values forpicking and placing, making it easy to integrate with our approach.
Revised: [9], which produces pixel-wise Q-values forpicking and placing, making it easy to integrate with our approach.

Original: Furthermore, pixel-wise Qvalues facilitate potential future work on extending our method to non-cubic shapes.
Revised: Furthermore, pixel-wise Qvalues facilitate potential future work on extending our method to non-cubic shapes.

Original: All maskingmodels are trained with a multi-task loss, composed of a pixel-wise binary cross-entropy loss for themasks, and an optional auxiliary reconstruction loss:
Revised: All maskingmodels are trained with a multi-task loss, composed of a pixel-wise binary cross-entropy loss for themasks, and an optional auxiliary reconstruction loss:

Original: [Equation]
Revised: [Equation]

Original: The reconstruction loss here forces the model to semantically segment the image by identifying pixelwith its block ID, or labeling it as background.
Revised: The reconstruction loss here forces the model to semantically segment the image by identifying pixelwith its block ID, or labeling it as background.

Original: As in Nguyen and Salazar [40], we use a pre-norm
Revised: As in Nguyen and Salazar [40], we use a pre-norm

Original: Transformer layer with scaled initialization.
Revised: Transformer layer with scaled initialization.

Original: Our learning rate schedule is based on Vaswani et al.[39], with a varied number of warmup steps as in Nguyen and Salazar.
Revised: Our learning rate schedule is based on Vaswani et al.[39], with a varied number of warmup steps as in Nguyen and Salazar.

Original: A comprehensive report ofall hyperparameters and a ﬁgure of our architecture is given in Appendix A.
Revised: A comprehensive report ofall hyperparameters and a ﬁgure of our architecture is given in Appendix A.

Original: Q-Learning – Our model relies on a learned function from states and actions to expected rewards,
Revised: Q-Learning – Our model relies on a learned function from states and actions to expected rewards,

Original: Q ( s t , a ) , which we learn via the SPOT-Q algorithm [9], which introduces novel reward shapingprocesses for improving efﬁciency.
Revised: Q ( s t , a ) , which we learn via the SPOT-Q algorithm [9], which introduces novel reward shapingprocesses for improving efﬁciency.

Original: More details are given in Appendix A.3.
Revised: More details are given in Appendix A.3.

Original: Given a pixel-wisen ⇥ n state space of an image and two actions, grasp and place, the output of this function can beexpressed as a tensor Q 2 R n ⇥ n ⇥ 2 .
Revised: Given a pixel-wisen × n state space of an image and two actions, grasp and place, the output of this function can beexpressed as a tensor Q ∈ R n × n × 2 .

Original:  Datasets and Resources
Revised:  Datasets and Resources

Original: 4.1 Simulation Experiments with Synthetic Commands
Revised: 4.1 Simulation Experiments with Synthetic Commands

Original: We create a dataset of commands and grasp/place masks to train our masking model and demonstrate its performance in simulated environment on a robot arm.
Revised: We create a dataset of commands and grasp/place masks to train our masking model and demonstrate its performance in simulated environment on a robot arm.

Original: Using trajectories from Hundt et al.[9], we annotate successful grasp-and-place action pairs with synthetic commands generated from atemplate, and use simulation state information to construct the reference masks M t and state information S t .
Revised: Using trajectories from Hundt et al.[9], we annotate successful grasp-and-place action pairs with synthetic commands generated from atemplate, and use simulation state information to construct the reference masks M t and state information S t .

Original: While synthetic language makes simplifying assumptions about the syntactic and lexicaldiversity of natural language, it provides a controllable test-bed for symbolic reasoning, and is oftenused to study compositionality [41, 42, 43], as we do in Section 5.1.
Revised: While synthetic language makes simplifying assumptions about the syntactic and lexicaldiversity of natural language, it provides a controllable test-bed for symbolic reasoning, and is oftenused to study compositionality [41, 42, 43], as we do in Section 5.1.

Original: For row-making, the template is“move the color a block next to the color b block”, and for stacking the template is “stack thecolor a block on the color b block”.
Revised: For row-making, the template is“move the color a block next to the color b block”, and for stacking the template is “stack thecolor a block on the color b block”.

Original: We train supervised VGP models (cf.
Revised: We train supervised VGP models (cf.

Original: Section 3) on thesecommands, and combine the produced masks with the Q-values from the SPOT-Q framework in asimulated environment to execute the actions in order.
Revised: Section 3) on thesecommands, and combine the produced masks with the Q-values from the SPOT-Q framework in asimulated environment to execute the actions in order.

Original: The visual input to our models is the concatenation of a birds-eye color image and depth image.
Revised: The visual input to our models is the concatenation of a birds-eye color image and depth image.

Original: The training data is augmented by ﬂipping eachimage across 4 axes, manipulating the command accordingly, as well as by the addition of pixelwise Gaussian noise drawn from N (0 , 0 . 05) .
Revised: The training data is augmented by ﬂipping eachimage across 4 axes, manipulating the command accordingly, as well as by the addition of pixelwise Gaussian noise drawn from N (0 , 0 . 05) .

Original: This results in train/dev/test splits of 5520 / 46 / 46 and8970 / 74 / 76 for row-making and stacking, respectively, where each sample is an input image andcommand paired with a reference place and grasp mask.
Revised: This results in train/dev/test splits of 5520 / 46 / 46 and8970 / 74 / 76 for row-making and stacking, respectively, where each sample is an input image andcommand paired with a reference place and grasp mask.

Original: Experiments and results on VGP metricsand in the simulation for this data are given in Section 5.1.
Revised: Experiments and results on VGP metricsand in the simulation for this data are given in Section 5.1.

Original: 4.2 Transfer to Real Images
Revised: 4.2 Transfer to Real Images

Original: In order to be useful in real-life settings, our method must also be applicable to real data.
Revised: In order to be useful in real-life settings, our method must also be applicable to real data.

Original: However,unlike simulated data, data from a real robot is expensive and time-consuming to collect.
Revised: However,unlike simulated data, data from a real robot is expensive and time-consuming to collect.

Original: Therefore,it is natural to ask: how much real data is needed to train an adequate VGP model?
Revised: Therefore,it is natural to ask: how much real data is needed to train an adequate VGP model?

Original: The Hundt et al.
Revised: The Hundt et al.

Original: [9] real robot trial logs include 1496 actions each for row-making and stacking.
Revised: [9] real robot trial logs include 1496 actions each for row-making and stacking.

Original: We split these into an 80/10/10 train/dev/test split and augment the examples by ﬂipping, producing5980 , 747 , 748 datapoints.
Revised: We split these into an 80/10/10 train/dev/test split and augment the examples by ﬂipping, producing5980 , 747 , 748 datapoints.

Original: Image collection details are given in Appendix B.2.
Revised: Image collection details are given in Appendix B.2.

Original: Given the small sizeof the real dataset, we pre-train a model on simulated images and ﬁne-tune it with the real images.
Revised: Given the small sizeof the real dataset, we pre-train a model on simulated images and ﬁne-tune it with the real images.

Original: 4.3 Compositionality
Revised: 4.3 Compositionality

Original: In linguistics, compositionality is typically taken to mean that the meaning of a whole utterance iscomposed of the meanings of its parts; the meaning of “stack the red block on the blue block” is in part made up of the independent meanings of “red” and “blue”.
Revised: In linguistics, compositionality is typically taken to mean that the meaning of a whole utterance iscomposed of the meanings of its parts; the meaning of “stack the red block on the blue block” is in part made up of the independent meanings of “red” and “blue”.

Original: Compositionality allows speakersto re-combine known symbols in novel ways, and underpins human language use.
Revised: Compositionality lets speakersto re-combine known symbols in new ways, underpinning human language use.

Original: In our task, acompositional reasoner should correctly executethe instruction “stack the red block on the blueblock” withoutever having seen that combination; it should be able to localize red and blue blocksfrom other color contexts (e.g. red/green, blue/yellow, etc.).
Revised: In our task, acompositional reasoner should correctly execute “stack the red block on the blue block” (referredto here as stack(r, b) ) without having seen that combination; it should be able to localize redand blue blocks from other color contexts (e.g. stack(r, g) , stack(y, b) , etc.).

Original: However, there is no explicit constraintduring training to enforce this; the model could learn to interpret each of the 23 possible color pairsatomically.
Revised: However,there is no explicit constraint during training to enforce this; the model could learn to interpret eachof the 23 possible color pairs atomically.

Original: We test our model’s compositional reasoning by examining its ability to generalize tounseen color combinations.
Revised: We test our model’s compositional reasoning via its abilityto generalize to unseen color combinations.

Original: We hold out 6 color pairs during training, 26% of the 23 order-invariantpairs, andthen test exclusively on those pairs.
Revised: We hold out 6 color pairs during training ( 26% ofthe 23 order-invariant pairs) and test exclusively on those pairs.

Original: For example, we train a masking model with all colorcombinations except those made only of red, blue, yellow, and green and then test only on “stack the{ red,blue,yellow,green } block on the { red,blue,yellow,green }block” examples.
Revised: For example, we train a maskingmodel with all color combinations except those made only of red, blue, yellow, and green and thentest only on stack( { r,b,g,y } , { r,b,g,y } examples.

Original: We hold-out each196 possible subset, training and evaluating a model as described above, average the metrics across allsubsets, and compare against a random train and test split baseline, where all color pairs are seen attrain time.
Revised: We hold-out each subset, training andevaluating a model as described above, average the metrics across all subsets, and compare againsta random train and test split baseline, where all color pairs are seen at train time.

Original: For efﬁciency, we train on each task separately and do not augment with Gaussian noise.
Revised: For efﬁciency, wetrain on each task separately and do not augment with Gaussian noise.

Original: 4.4 Naturalistic Language Commands with Complex Scenes
Revised: 4.4 Naturalistic Language Commands with Complex Scenes

Original: Language sourced from human speakers is syntactically and lexically diverse, with mistakes, misspellings, and other idiosyncrasies that make far more challenging to model than template data.
Revised: Language sourced from human speakers is syntactically and lexically diverse, with mistakes, misspellings, and other idiosyncrasies that make far more challenging to model than template data.

Original: Analyzing synthetic language showcases our model’s ability to integrate discrete symbolic commandswith visual perception and action, as well as our model’s ability to generalize in a compositionalfashion, but this does not imply that our model generalizes to the complexities of real language.
Revised: Analyzing synthetic language showcases our model’s ability to integrate discrete symbolic commandswith visual perception and action, as well as our model’s ability to generalize in a compositionalfashion, but this does not imply that our model generalizes to the complexities of real language.

Original: We examine a 3D block manipulation dataset with human natural language instructions that wasintroduced by Bisk et al.
Revised: We examine a 3D block manipulation dataset with human natural language instructions that wasintroduced by Bisk et al.

Original: [10] (cf.
Revised: [10] (cf.

Original: Fig. 2).
Revised: Fig. 2).

Original: This dataset is set apart from other block-world data byits focus on complex instructions.
Revised: This dataset is set apart from other block-world data byits focus on complex instructions.

Original: Rather than using synthetic utterances, instructions were sourcedfrom crowdworkers (9 per example), who were shown a pair of images and asked to provide instructions to a robot, with the goal of manipulating the blocks in the ﬁrst image such that they result in thesecond image.
Revised: Rather than using synthetic utterances, instructions were sourcedfrom crowdworkers (9 per example), who were shown a pair of images and asked to provide instructions to a robot, with the goal of manipulating the blocks in the ﬁrst image such that they result in thesecond image.

Original: This collection method result in more syntactically diverse utterances with complexlinguistic concepts, such as “mirroring” and “balancing”.
Revised: This collection method result in more syntactically diverse utterances with complexlinguistic concepts, such as “mirroring” and “balancing”.

Original: [Figure]
Revised: [Figure]

Original: The dataset is comprised of a sequence of state representations paired with natural language instructions on how to transition between states.
Revised: The dataset is comprised of a sequence of state representations paired with natural language instructions on how to transition between states.

Original: While theblocks dataset is designed for robot manipulation,several assumptions were made to simplify annotation by crowdworkers.
Revised: While theblocks dataset is designed for robot manipulation,several assumptions were made to simplify annotation by crowdworkers.

Original: Crucially, physics in the environment were turned off while the structures werebeing constructed and annotated, resulting in somephysically infeasible structures.
Revised: Crucially, physics in the environment were turned off while the structures werebeing constructed and annotated, resulting in somephysically infeasible structures.

Original: To facilitate research in the blocks domain, and make the datasetmore useful for training and evaluating embodied agents, we create a ﬁltered version of the dataset,where physically infeasible examples are removed.
Revised: To facilitate research in the blocks domain, and make the datasetmore useful for training and evaluating embodied agents, we create a ﬁltered version of the dataset,where physically infeasible examples are removed.

Original: Our ﬁltering process removes 49 .
Revised: Our ﬁltering process removes 49 .

Original: 93%and 38 .
Revised: 93%and 38 .

Original: 23% of train, dev, and test, respectively.
Revised: 23% of train, dev, and test, respectively.

Original: Appendix C.1 contains more details on the translation process and data preprocessing.
Revised: Appendix C.1 contains more details on the translation process and data preprocessing.

Original: To model the dataset’s world state sequences in the VGPframework, we translate the 3D Cartesian coordinates of each block given by the dataset’s state representation into a top-down grid view, where each pixel corresponds to the ID of the block locatedat that position.
Revised: To model the dataset’s world state sequences in the VGPframework, we translate the 3D Cartesian coordinates of each block given by the dataset’s state representation into a top-down grid view, where each pixel corresponds to the ID of the block locatedat that position.

Original: We refer to this image as the scene’s “state representation.”
Revised: We refer to this image as the scene’s “state representation.”

Original: While the blocks dataset contains all the state information required to produce image-like staterepresentations, this cannot generally be assumed to be the case.
Revised: While the blocks dataset contains all the state information required to produce image-like staterepresentations, this cannot generally be assumed to be the case.

Original: A more likely scenario mimicks
Revised: A more likely scenario mimicks

Original: Section 4.1, where the model’s input is a combined color and depth image.
Revised: Section 4.1, where the model’s input is a combined color and depth image.

Original: After translating thedataset into the simulated environment used in Section 4.1, we collect color and depth images forthe whole blocks dataset.
Revised: After translating thedataset into the simulated environment used in Section 4.1, we collect color and depth images forthe whole blocks dataset.

Original: In the original dataset, each block type is identiﬁed by a company logo.
Revised: In the original dataset, each block type is identiﬁed by a company logo.

Original: Since the logos were downsampled by the simulator, we also assign a color to each type.
Revised: Since the logos were downsampled by the simulator, we also assign a color to each type.

Original: Thedemographics and labor practices underlying its original creation are unknown [10, 44].
Revised: Thedemographics and labor practices underlying its original creation are unknown [10, 44].

Original: We ﬁlteredtheir dataset without crowdworkers; and it is intended for robotics research, not product deployment.
Revised: We ﬁlteredtheir dataset without crowdworkers; and it is intended for robotics research, not product deployment.

Original: Experiments and results for this dataset are in Sec.
Revised: Experiments and results for this dataset are in Sec.

Original:  Experiments
Revised:  Experiments

Original: 5.1 Simulation Experiments with Synthetic Commands
Revised: 5.1 Simulation Experiments with Synthetic Commands

Original: An operator guiding a robot through a manipulation task may have a particular outcome in mind: forexample, a stack with a particular order of blocks.
Revised: An operator guiding a robot through a manipulation task may have a particular outcome in mind: forexample, a stack with a particular order of blocks.

Original: Language can naturally express such constraints; i.e. “stack the red block on the yellow block.”
Revised: Language can naturally express such constraints; i.e. “stack the red block on the yellow block.”

Original: Following the template generation and data annotationin Section 4.1, we train baseline and Transformer VGP models to produce output masks ˆ M grasp andˆ M place and combine them with a SPOT-Q model to manipulate blocks in a simulated environment.
Revised: Following the template generation and data annotationin Section 4.1, we train baseline and Transformer VGP models to produce output masks ˆ M grasp andˆ M place and combine them with a SPOT-Q model to manipulate blocks in a simulated environment.

Original: These masks are ﬁltered and thresholded using 0-1 block masks, following Hundt et al.
Revised: These masks are ﬁltered and thresholded using 0-1 block masks, following Hundt et al.

Original: Toevaluate the VGP system in isolation, we report the following metrics:
Revised: Toevaluate the VGP system in isolation, we report the following metrics:

Original: • Block Accuracy is the percentage predictions for which the grasp mask overlaps with the
Revised: • Block Accuracy is the percentage predictions for which the grasp mask overlaps with the

Original: correct block,
Revised: correct block,

Original: i
Revised: i

Original: e. the model chooses the correct block to pick up.
Revised: e. the model chooses the correct block to pick up.

Original: Higher is better.
Revised: Higher is better.

Original: • Idealized Error Score (IES) captures the performance of an idealized robot on the manipulation task, choosing a block to move using the grasp mask, and a location using the
Revised: • Idealized Error Score (IES) captures the performance of an idealized robot on the manipulation task, choosing a block to move using the grasp mask, and a location using the

Original: place mask.
Revised: place mask.

Original: After the chosen block is teleported to the predicted location, we measure the
Revised: After the chosen block is teleported to the predicted location, we measure the

Original: Euclidean distance between the block’s new location and its reference location in block
Revised: Euclidean distance between the block’s new location and its reference location in block

Original: lengths.
Revised: lengths.

Original: Lower is better.
Revised: Lower is better.

Original: Our baseline is a UNet-based VGP model; speciﬁcally, we extend the L ING UN ET model presentedby Misra et al.
Revised: Our baseline is a UNet-based VGP model; speciﬁcally, we extend the L ING UN ET model presentedby Misra et al.

Original: [4] with an attentional fusion mechanism.
Revised: [4] with an attentional fusion mechanism.

Original: While the original L ING UN ET used theﬁnal state of an LSTM for the language representation, we use a weighted average of BiLSTM statesproduced by an attention mechanism.
Revised: While the original L ING UN ET used theﬁnal state of an LSTM for the language representation, we use a weighted average of BiLSTM statesproduced by an attention mechanism.

Original: We also ran a small end-to-end Q-Learning experiment thatbackpropagates through both the FCN and Transformer, but the model did not converge.[Table] and
Revised: We also ran a small end-to-end Q-Learning experiment thatbackpropagates through both the FCN and Transformer, but the model did not converge.

Original: [Table]
Revised: [Table]

Original: Score (IES, measured in blocks), is better.

Original: TFMR is

Original: [Table]

Original: [Table]

Original: Stacks Stacks 27.9% 1.
Revised: Stacks Stacks 27.9% 1.

Original: TFMR Stacks Stacks 81.5% 0.
Revised: TFMR Stacks Stacks 81.5% 0.

Original: UNet
Revised: UNet

Original: Both
Revised: Both

Original: Stacks 75.2% 0.
Revised: Stacks 75.2% 0.

Original: TFMR
Revised: TFMR

Original: Both
Revised: Both

Original: Stacks 90.4% 0.
Revised: Stacks 90.4% 0.

Original: Further details on the masking, as well as the simulation environment, can be found in Appendix B.1.
Revised: Further details on the masking, as well as the simulation environment, can be found in Appendix B.1.

Original: The ﬁltered/thresholded masks ˆ M grasp and ˆ M placeare stacked into a single mask ˆ M 2 R n ⇥ n ⇥ 2 .
Revised: The ﬁltered/thresholded masks ˆ M grasp and ˆ M placeare stacked into a single mask ˆ M ∈ R n × n × 2 .

Original: A ﬁnalmatrix of Q 0 of Q values constrained by the languageis given by Q 0 =
Revised: A ﬁnalmatrix of Q (cid:48) of Q values constrained by the languageis given by Q (cid:48) =

Original: ˆ M  Q , where  is the Hadamardproduct; Q 0 is then used to choose an action and alocation at which to grasp/place in the simulation.
Revised: ˆ M (cid:12) Q , where (cid:12) is the Hadamardproduct; Q (cid:48) is then used to choose an action and alocation at which to grasp/place in the simulation.

Original: We count stacks and rows as successful only withthe speciﬁed color order, plus we tally the number ofactions this required, and the percentage of times thegripper picked up the correct block.
Revised: We count stacks and rows as successful only withthe speciﬁed color order, plus we tally the number ofactions this required, and the percentage of times thegripper picked up the correct block.

Original: We contrast ourmodel with the SPOT-Q framework alone, which hasno access to the order constraints, and with our UNetbaseline.
Revised: We contrast ourmodel with the SPOT-Q framework alone, which hasno access to the order constraints, and with our UNetbaseline.

Original: Each model was evaluated with the same SPOT-Q model and tested on 100 trials.
Revised: Each model was evaluated with the same SPOT-Q model and tested on 100 trials.

Original: Hundtet al.
Revised: Hundtet al.

Original: [9] set a 30 action limit for failed trials; we added several evaluation heuristics to terminatefailed trials when an irrecoverable state was reached, measuring color ordering errors.
Revised: [9] set a 30 action limit for failed trials; we added several evaluation heuristics to terminatefailed trials when an irrecoverable state was reached, measuring color ordering errors.

Original: Stackingtasks use 8 block colors and row-making tasks have 4. 
Revised: Stackingtasks use 8 block colors and row-making tasks have 4. [Table] UNet

Original: [Table]
Revised: [Table]

Original:  Table 1 shows the visual goal prediction metrics for all model variants trained on one or bothtasks.
Revised: NaN [Table] [Table] Table 1 shows the visual goal prediction metrics for all model variants trained on one or bothtasks.

Original: We see that the Transformer-based models (TFMR) uniformly out-perform the UNetbaseline across all metrics, often by large margins.
Revised: We see that the Transformer-based models (TFMR) uniformly out-perform the UNetbaseline across all metrics, often by large margins.

Original: For both UNet and Transformer architectures, training on two tasks simultaneouslycovers a broader range of semantically different examples with a corresponding increase inperformance on all metrics.
Revised: For both UNet and Transformer architectures, training on two tasks simultaneouslycovers a broader range of semantically different examples with a corresponding increase inperformance on all metrics.

Original: Notably, these examples differ by task: ﬁrstly, the row-makingtasks contain a subset of the colors available in stacking.
Revised: Notably, these examples differ by task: ﬁrstly, the row-makingtasks contain a subset of the colors available in stacking.

Original: Secondly, while the grasp data acrossboth tasks looks roughly identical, the data for place masks looks quite different.
Revised: Secondly, while the grasp data acrossboth tasks looks roughly identical, the data for place masks looks quite different.

Original: For example, both“stack the blue block on the red block” and “move the blue block next to the red block” result ingrasp masks over the blue block, but whereas the place mask for stacking picks out the block onwhich to stack (the red block), the row mask speciﬁes a location near the target block.
Revised: For example, both“stack the blue block on the red block” and “move the blue block next to the red block” result ingrasp masks over the blue block, but whereas the place mask for stacking picks out the block onwhich to stack (the red block), the row mask speciﬁes a location near the target block.

Original: Thus, it isintriguing that combining tasks beneﬁts not only the block accuracy, but also the IES.
Revised: Thus, it isintriguing that combining tasks beneﬁts not only the block accuracy, but also the IES.

Original: Table 2 shows the task completion percentage for the best UNet and Transformer models (trainedon both tasks) in the simulated environment, as well as the percentage of successful grasps thatpicked up the correct color, and the average number of actions taken to complete a successful trial.
Revised: Table 2 shows the task completion percentage for the best UNet and Transformer models (trainedon both tasks) in the simulated environment, as well as the percentage of successful grasps thatpicked up the correct color, and the average number of actions taken to complete a successful trial.

Original: In the aggregate, we see that high performance on VGP metrics does not translate to equivalent performance in the simulation environment.
Revised: In the aggregate, we see that high performance on VGP metrics does not translate to equivalent performance in the simulation environment.

Original: For example, both for stacking and row-making, thepercentage of correct color grasps is substantially lower that the VGP equivalent metric, block accuracy.
Revised: For example, both for stacking and row-making, thepercentage of correct color grasps is substantially lower that the VGP equivalent metric, block accuracy.

Original: For the Transformer model, this may be due to the imprecision of the patch-wise tiling ofthe image: when heuristically combined with a common-sense mask, a patch may cover multipleadjacent blocks.
Revised: For the Transformer model, this may be due to the imprecision of the patch-wise tiling ofthe image: when heuristically combined with a common-sense mask, a patch may cover multipleadjacent blocks.

Original: This is an inherent limitation of our approach.
Revised: This is an inherent limitation of our approach.

Original: When compared to the RL-only and
Revised: When compared to the RL-only and

Original: UNet baselines, our Transformer-based model succeeds in signiﬁcantly more trials.
Revised: UNet baselines, our Transformer-based model succeeds in signiﬁcantly more trials.

Original: Additionally,the mean number of actions across successful trials is lower, indicating that the Transformer-basedmasks result in fewer actions per successful trial.
Revised: Additionally,the mean number of actions across successful trials is lower, indicating that the Transformer-basedmasks result in fewer actions per successful trial.

Original: Note that for stacking, both baselines succeededon 

Revised: Note that for stacking, both baselines succeededon

Original: out of the 100 trials; due to toppling and occlusion, stacking poses a signiﬁcantly harder taskthan row-making, and it is harder to recover from a misplaced block, even if the trial is not ended.
Revised: out of the 100 trials; due to toppling and occlusion, stacking poses a signiﬁcantly harder taskthan row-making, and it is harder to recover from a misplaced block, even if the trial is not ended.

Original: We see that even a fairly high block accuracy and low IES are insufﬁcient for high task performance.
Revised: We see that even a fairly high block accuracy and low IES are insufﬁcient for high task performance.

Original: This highlights the one of the main challenges of multistep tasks: small error rates compound acrossmultiple actions, yielding low task performance.
Revised: This highlights the one of the main challenges of multistep tasks: small error rates compound acrossmultiple actions, yielding low task performance.

Original: A full error analysis is given in Appendix B.

Revised: A full error analysis is given in Appendix B.

Original:  Transfer to Real Images – In Figure 3, we plot the
Revised: [Figure] Transfer to Real Images – In Figure 3, we plot the

Original: IES of row-making and stacking models when trainedon increasing subsets of real data.
Revised: IES of row-making and stacking models when trainedon increasing subsets of real data.

Original: We see that for bothrows and stacks, performance improves slightly as we increase the amount of training data, with diminishing returns after 30% of the available data (1794 examples) areused.
Revised: We see that for bothrows and stacks, performance improves slightly as we increase the amount of training data, with diminishing returns after 30% of the available data (1794 examples) areused.

Original: Given that Transformers typically require severalorders of magnitude more data to reach good performance[45, 40, 46], the fact that our architecture can be trainedwith mere thousands of simulated training examples, andsuccessfully ﬁnetuned with even fewer real examples, extracted from only a few hundred real images, represents a signiﬁcant achievement.
Revised: Given that Transformers typically require severalorders of magnitude more data to reach good performance[45, 40, 46], the fact that our architecture can be trainedwith mere thousands of simulated training examples, andsuccessfully ﬁnetuned with even fewer real examples, extracted from only a few hundred real images, represents a signiﬁcant achievement.

Original: For a proof-of-concept demonstration of our model running on a real robot, see Appendix B.4 and the supplementary materials.
Revised: For a proof-of-concept demonstration of our model running on a real robot, see Appendix B.4 and the supplementary materials.

Original: Compositionality – By generating held-out data splits as described in Section 4.3, we can testthe ability of our model to generalize compositionally, i.e. to recombine known symbols in unseencontexts.
Revised: Compositionality – By generating held-out data splits as described in Section 4.3, we can testthe ability of our model to generalize compositionally, i.e. to recombine known symbols in unseencontexts.

Original: A model that has learned compositional color representations will be able to combinecolor representations across contexts; for example, if it has seen red blocks in the context of greenblocks, it should generalize to a blue block context.
Revised: A model that has learned compositional color representations will be able to combinecolor representations across contexts; for example, if it has seen red blocks in the context of greenblocks, it should generalize to a blue block context.

Original: We would therefore expect the model to performroughly on par with a model trained on a random data split, where all contexts are seen at train time.
Revised: We would therefore expect the model to performroughly on par with a model trained on a random data split, where all contexts are seen at train time.

Original: Conversely, if the model has learned to interpret each utterance as an atomic statement, we shouldexpect much lower performance on the held-out subset.
Revised: Conversely, if the model has learned to interpret each utterance as an atomic statement, we shouldexpect much lower performance on the held-out subset.

Original: [Table]
Revised: [Table]

Original: Our results are reported in Table 3.
Revised: Our results are reported in Table 3.

Original: Both rows andstacks see a slight drop in block accuracy from the baseline.
Revised: Both rows andstacks see a slight drop in block accuracy from the baseline.

Original: Nevertheless, the lack of a large disparity betweenthe baseline and the held-out condition suggests that themodel has largely learned to generalize compositionallyto unseen block combinations.
Revised: Nevertheless, the lack of a large disparity betweenthe baseline and the held-out condition suggests that themodel has largely learned to generalize compositionallyto unseen block combinations.

Original: This ﬁnding suggests that,although the model certainly has the capacity to memorize 23 combinations, and although it is not in any waytrained to perform compositional reasoning, it naturallylearns to bind symbols to percepts in a way that generalizes across contexts (i.e. “red block” refersto one particular block, regardless of its context).
Revised: This ﬁnding suggests that,although the model certainly has the capacity to memorize 23 combinations, and although it is not in any waytrained to perform compositional reasoning, it naturallylearns to bind symbols to percepts in a way that generalizes across contexts (i.e. “red block” refersto one particular block, regardless of its context).

Original: These result are promising but limited to simulatedlanguage; accordingly, in Section 5.2 we evaluate our results with real language data.
Revised: These result are promising but limited to simulatedlanguage; accordingly, in Section 5.2 we evaluate our results with real language data.

Original: 5.2 Naturalistic Language Commands with Complex Scenes
Revised: 5.2 Naturalistic Language Commands with Complex Scenes

Original: In this experiment we model the naturalistic data described in Section 4.4, where state representations are coupled instead with diverse crowdsourced commands, in contrast to the template generated commands in Section 5.1.
Revised: In this experiment we model the naturalistic data described in Section 4.4, where state representations are coupled instead with diverse crowdsourced commands, in contrast to the template generated commands in Section 5.1.

Original: We train models on state representations (cf.
Revised: We train models on state representations (cf.

Original: Section 4.4) as well asbirds-eye color and depth images.
Revised: Section 4.4) as well asbirds-eye color and depth images.

Original: This highlights the ﬂexibility of our model: we are able to trainfrom images and from state representations with no modiﬁcations to the architecture.
Revised: This highlights the ﬂexibility of our model: we are able to trainfrom images and from state representations with no modiﬁcations to the architecture.

Original: Large pre-trained encoders have been shown to encode useful linguistic features at a number oflevels of analysis [47, 48, 49]; accordingly, we make use of bert-base-cased [50] as an inputfeature, contrasting it with GloVe [51] word embeddings.
Revised: Large pre-trained encoders have been shown to encode useful linguistic features at a number oflevels of analysis [47, 48, 49]; accordingly, we make use of bert-base-cased [50] as an inputfeature, contrasting it with GloVe [51] word embeddings.

Original: Tuning and metrics are as in Sec. 5.1.
Revised: Tuning and metrics are as in Sec. 5.1.

Original: [Table]
Revised: [Table]

Original: Table 4 shows the VGP metrics for modelstrained from image-like representations ofthe state (described in Section 4.4) as wellas from images.
Revised: Table 4 shows the VGP metrics for modelstrained from image-like representations ofthe state (described in Section 4.4) as wellas from images.

Original: For brevity, only thebest model for each embedding type is reported, with full results in Appendix C.5.
Revised: For brevity, only thebest model for each embedding type is reported, with full results in Appendix C.5.

Original: On state representations, the Transformerbased models show a clear advantage overthe baseline, with IES being substantiallylower for both Transformer models.
Revised: On state representations, the Transformerbased models show a clear advantage overthe baseline, with IES being substantiallylower for both Transformer models.

Original: Whenusing GloVe embeddings with state representation input, 2 ⇥ 2 patches resulted in the best performance, while a patch size of 4 yielded thebest results for BERT embeddings.
Revised: Whenusing GloVe embeddings with state representation input, 2 × 2 patches resulted in the best performance; a patch size of 4 yielded the bestresults for BERT embeddings.

Original: Overall, while the BERT-based model has slightly higher blockaccuracy, the IES are equal between the two models when trained on state representations.
Revised: Overall, while the BERT-based model has slightly higher block accuracy, the IES are equal between the two models when trained on state representations.

Original: Whilethese metrics measure performance per step, the resultshere suggest that our masking system canbe applied to reallanguage as well as simulated language.
Revised: While thesemetrics measure per-step performance, these results suggest that our masking model can be appliedto real as well as simulated language.

Original: In Table 4, we also see that the performance shifts substantially when changing the input from stateto images.
Revised: In Table 4, we also see that the performance shifts substantially when changing the input from stateto images.

Original: While the baseline improves slightly in IES, the Transformers’ performance decreasesdramatically.
Revised: While the baseline improves slightly in IES, the Transformers’ performance decreasesdramatically.

Original: These results stand in contrast to those in Section 5.1, where a Transformer trainedon images unequivocally outperformed the baseline.
Revised: These results stand in contrast to those in Section 5.1, where a Transformer trainedon images unequivocally outperformed the baseline.

Original: Nevertheless, the best Transformer model stilloutperforms the baseline.
Revised: Nevertheless, the best Transformer model stilloutperforms the baseline.

Original: We speculate that the baseline models may perform better here becausethe architecture was developed for image segmentation problems.
Revised: We speculate that the baseline models may perform better here becausethe architecture was developed for image segmentation problems.

Original: Interestingly, while a BERT-based
Revised: Interestingly, while a BERT-based

Original: Transformer performed well when trained from state, the same models’ performance here is greatlydiminished.
Revised: Transformer performed well when trained from state, the same models’ performance here is greatlydiminished.

Original: Improving the Transformer’s real-world performance is a direction for future work.
Revised: Improving the Transformer’s real-world performance is a direction for future work.

Original: Data Translation and Filtering
Revised: Data Translation and Filtering

Original: In Section 5.1, we observed that high performance in a static evaluation against a dataset did not entail equally high performance when using a realistic robot formulti-step tasks in a simulated environment.
Revised: In Section 5.1, we observed that high performance in a static evaluation against a dataset did not entail equally high performance when using a realistic robot formulti-step tasks in a simulated environment.

Original: In its current form, the blocks dataset can only be evaluated using static metrics.
Revised: In its current form, the blocks dataset can only be evaluated using static metrics.

Original: Furthermore, the images provided in the dataset are taken from a ﬁxed 45degree projection, and our method assumes access to birds-eye images.
Revised: Furthermore, the images provided in the dataset are taken from a ﬁxed 45degree projection, and our method assumes access to birds-eye images.

Original: To obtain birds-eye imagesof the environment and to facilitate further research into combining rich, naturalistic language withaction in manipulation tasks, we translate the dataset into the same environment used in Section 5.1,and ﬁlter the dataset to only physically feasible examples; this process is described in Section 4.4.
Revised: To obtain birds-eye imagesof the environment and to facilitate further research into combining rich, naturalistic language withaction in manipulation tasks, we translate the dataset into the same environment used in Section 5.1,and ﬁlter the dataset to only physically feasible examples; this process is described in Section 4.4.

Original: ⇤ indicates signiﬁcant improvement over UNet as per the Wilcoxon ranked-sign test.
Revised: indicates signiﬁcant improvement over UNet as per the Wilcoxon ranked-sign test.

Original: [Table]
Revised: [Table]

Original: Table 5 reports the VGP metrics on the thephysically feasible subset of the data for the
Revised: Table 5 reports the VGP metrics on the thephysically feasible subset of the data for the

Original: Transformer models.
Revised: Transformer models.

Original: Firstly, we see thatperformance decreases across most metrics.
Revised: Firstly, we see thatperformance decreases across most metrics.

Original: However, for UNet, IES improves onthe physically feasible subset.
Revised: However, for UNet, IES improves onthe physically feasible subset.

Original: In contrast,the Transformer models decrease in performance across both metrics.
Revised: In contrast,the Transformer models decrease in performance across both metrics.

Original: Nevertheless,the Transformer model still performs on parwith the baseline on IES, and substantially better on block accuracy.
Revised: Nevertheless,the Transformer model still performs on parwith the baseline on IES, and substantially better on block accuracy.

Original: These results suggest that thephysically feasible subset is more challenging to model, even when evaluating the models statically.
Revised: These results suggest that thephysically feasible subset is more challenging to model, even when evaluating the models statically.

Original: In other words, even when the evaluation and models are physically unconstrained, modeling onlyphysically feasible scenes is harder than modeling all scenes.
Revised: In other words, even when the evaluation and models are physically unconstrained, modeling onlyphysically feasible scenes is harder than modeling all scenes.

Original: This result highlights the difﬁculty ofdeveloping embodied agents and the importance of testing them in realistic environments.
Revised: This result highlights the difﬁculty ofdeveloping embodied agents and the importance of testing them in realistic environments.

Original:  Conclusion
Revised:  Conclusion

Original: We have presented a general approach to multi-step rearrangement and manipulation tasks that factorizes each step into learning how to act and learning where to act.
Revised: We have presented a general approach to multi-step rearrangement and manipulation tasks that factorizes each step into learning how to act and learning where to act.

Original: To decide where to act, wehave presented a Transformer-based model for converting state representations and natural languageinstructions to masks over an action space that learns with remarkably few examples.
Revised: To decide where to act, wehave presented a Transformer-based model for converting state representations and natural languageinstructions to masks over an action space that learns with remarkably few examples.

Original: We haveshown the ability of our model to act in a realistic simulated environment for two multi-step blockrearrangement tasks, and naturally learn a compositional mapping from text to actions in a perceptual space. 
Revised: We haveshown the ability of our model to act in a realistic simulated environment for two multi-step blockrearrangement tasks, and naturally learn a compositional mapping from text to actions in a perceptual space. We have demonstrated our model performs well under both these realistic physicalassumptions and with real and complex natural language via a dataset of crowdsourced instructions. A.2 Hyperparameters

Revised: The number of shared and combined layers is treated as a hyperparameter n ∈ [0 , 2 , 4 , 6] , aswell as the number of warmup steps w ∈ [100 , 400 , 1000 , 4000] , the dropout probability p ∈[0 .

Revised: 40] , and the loss weight ratio γ ∈ [0 . 01 , 0 . 1 , 0 . 2] .

Original: We have demonstrated our model performs well under both these realistic physicalassumptions and with real and complex natural language via a dataset of crowdsourced instructions.
Revised: Due to the larger number of hyperparameters, we sample 70 random parameter conﬁgurations. A.3 Q-Learning Our model relies on a learned function from states and actions to expected rewards, Q ( s t , a ) . Thisfunction is learned in the context of a Markov Decision Process ( S, A, P, R, γ ) composed of a setof states S , a set of actions A , a transition function P : S × S ×

Revised: A → R , a reward function

Revised: R : S ×

Revised: A → R , and a discount factor γ , where 0 ≤ γ ≤ 1 .

Revised: At time t , an agent observes state S t andchooses action a t =

Revised: π ( s t ) , where π is a policy.

Revised: The function Q , approximating the expected rewardfor each state-action pair, can be used to extract a deterministic policy π that maximizes the expectedreward:

Revised: π ( s t ) =

Revised: argmax Q ( s t , a ) , ∀ a ∈ A .

Revised: We learn Q to maximize the reward R over time byminimizing | Q ( s t , a t ) − y t

Revised: | , where y t =

Revised: R ( s t +1 , a t ) + γQ ( s t +1 , π ( s t +1 )) .

Revised: In a multi-step task, thedeﬁnition of R has a major impact on the learning efﬁciency of the agent.

Revised: Speciﬁcally, relying on asingle reward at the end of a task yields a very sparse reward signal and results in inefﬁcient learning.

Revised: In the multi-step tasks considered here (stacking and row-making) there are intermediate subtasksthat naturally lend themselves to reward shaping, i.e. providing smaller intermediate rewards tothe agent when the chosen action reﬂects an incremental progression towards the goal state.

Revised: The

Revised: SPOT-Q algorithm, introduced by Hundt et al.

Revised: [9], uses intermediate rewards to shape the learningof Q ( s t , a ) for row-making and stacking.

Revised: We use SPOT-Q to learn the function Q ( s t , a ) .

Revised: Given apixel-wise n × n state space of an image and two actions (grasp and place), this function can beexpressed as a tensor Q ∈ R n × n × 2 .

Revised: B Experiment 1B.1 Combination Heuristics

Revised: In the case of stacking, following Hundt et al.

Revised: [9], we intersect each mask with a common-sensemask that assigns a value of 0 where there are no blocks (i.e. an empty space cannot be the locationof a grasp or a place).

Revised: For row-making, we allow placing in empty locations, but intersect the graspmask with a common-sense mask.

Revised: For place and grasp actions in stacking, and grasp actions inrow-making, we then ﬁnd the maximum location s ∗ in mask M : s ∗ = argmax M .

Revised: Because thereconstruction function copies a patch value across all of its pixels, for a patch size of p there willbe p 2 pixels with the value of s ∗ .

Revised: We intersect this patch with the common-sense mask to ﬁnd thesingle block with the highest value in M .

Revised: For place actions in row-making, we simply threshold Mto obtain a set of valid patches to intersect with the Q values.

Revised: Both processes produce binary masks.

Revised: B.2 Simulation

Revised: We use the CopelliaSim simulator [52] for experiments involving a simulated robot.

Revised: The simulatedagent collects observations via a ﬁxed RGB-D camera, whose images are projected to a birds-eyeview, as shown in Hundt et al.

Revised: The agent operates over a discretized spatial and angular actionspace, and movement to a particular location and angle is performed by an inverse kinematics solverbuilt into the simulator.

Revised: Images All real images were collected via a UR5 robot, Robotiq 2f-85 2-ﬁnger gripper, and Primesense Carmine RGB-D camera.

Revised: All of these conditions are different from the simulator, except the

Revised: UR5.

Revised: The images were collected under variedlighting conditions.

Revised: The data is saved in the sameﬁle format as the experiments in Tables 1 and 2.with a birds-eye view, 224 × 224 RBG and depthmap images, etc.

Revised: Early Termination

Revised: The step-by-step nature ofthe instructions means that the agent is sometimesunable to reverse course: for example, if the ﬁrstinstruction were “stack the green block on the blueblock” but the agent mistakenly grasped the redblock and placed it on the green block, the greenblock would become unreachable.

Revised: Since neither the Q-value model nor the language understanding model were trained on unstacking tasks, there would be no way to reverse course here withoutadding an external observer.

Revised: Thus, in these cases, we terminate the trial.

Revised: Similarly, if the stack istoppled, we terminate the trial, as without an external observer we cannot determine which step ofthe process we have regressed to, since the toppling of partial stacks is not unusual.

Revised: Since blocks aretypically not occluded in row-making, we do not need these heuristics, and the only way for a trialto fail is by timing out after 30 actions.

Revised: Simulator For experiments involving a simulated robot, the CopelliaSim simulator [52] was used.

Revised: The simulated agent collects observations via a ﬁxed RGB-D camera, whose images are projectto a birds-eye view.

Revised: The agent operates over a discretized spatial and angular action space, andmovement to a particular location and angle is performed by an inverse kinematics solver built intothe simulator.

Revised: B.3 Error analysis

Revised: As mentioned in §5.1, stacking tasks may terminate early for a number of reasons, including irreversible actions that make a successful stack impossible.

Revised: We conduct an error analysis to determinewhat percentage of the failures are due to errors in the perception/reasoning component (i.e. themasking) and what percentage are due to physical failures of the grasping arm.

Revised: We ﬁnd that 61.97%of the failures could be attributed to timing out after 30 actions.

Revised: Qualitatively, we found that this

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: often happens when a block tumbles outside of the work area, rendering it impossible to grasp.

Revised: Another 7.04% can be attributed to toppled stacks, where a stack height of 2 or 3 was reached before aplace action destroyed the whole stack, ending the trial.

Revised: Finally, 30.99% were due to incorrect blockorderings which occluded crucial blocks; this is due to the 21.42% of grasp actions that picked upthe incorrect block.

Revised: B.4 Proof-of-Concept

Revised: Due to restrictions induced by the ongoing COVID-19 crisis, we were unable to run sufﬁcient trials to include quantitative real results.

Revised: Nevertheless, we include a proof-of-concept demonstrationvideo in which we run the Transformer-based masking model with a real robot arm to successfullycomplete a real stack in the bottom to top order green, yellow, blue, red.

Revised: C Experiment

Revised: C.1 Data Translation and Preprocessing

Revised: Note that in the original dataset, 4.82% of the examples have multiple blocks moving in a singleframe.

Revised: As this breaks the assumptions made in § 1, we ignore these examples both while trainingand evaluating.

Revised: For training, we convert the 64 × 64 state image grids into binary masks, where all elements are zeroexcept the pixels corresponding to the block which is moved.

Revised: C.2 Data Filtering

Revised: We ﬁlter out physically infeasible examples, which are typically due to unstable structures topplingor blocks being placed in overlapping areas.

Revised: Because the images captured from the simulated environment are of a lower resolution than the originals, many of these logos are difﬁcult to read in thesimulated images; to aid with block discrimination, we assign a color to each block in addition to itslogo.

Revised: Table 6 reports the same statistics given in Bisk et al.

Revised: [10] on the ﬁltered subset of the data.

Revised: C.3 Dataset Ambiguity

Revised: A remaining limitation of the dataset is that some descriptions are ambiguous and not reliably actionable, since annotators did not attempt to execute the described actions.

Revised: Some of the naturallanguage descriptions are ambiguous, as depicted in Fig.

Revised: C.4 Qualitative Error Analysis

Revised: Based on our results in Tables 4, we qualitatively examine some of the errors made by the

Revised: Transformer-based model, where we observe several patterns.

Revised: Looking at the validation exampleswith the highest IES, we observe that they often correspond to instances where the source block wasmis-identiﬁed, leading the wrong block to be moved, and yielding a large IES.

Revised: More interestingly,in the cases where the correct block was moved, the instruction often contains higher-order conceptsas well as linguistic complexities such as ellipsis.

Revised: For example, in Figure 6 there is a reference to a“disconnected square”, which is a rare, higher-order geometric concept.

Revised: In addition, the subsequentclause lacks an explicit reference back to the square; the annotator chose to leave this referenceas implicit, given that the square is raised to a salient position by the previous clause.

Revised: This type ofnoun-phrase ellipsis is common in natural language [53], and reﬂects the type of advanced pragmaticreasoning required to handle natural language.

Revised: Other examples of a higher-order concept instruction that the model performs poorly on are: “Takethe Mercedes Benz block on the Burger King block without hiding the right top edge” (hiding),“targt [sic] goes 1/2 under Adidas with the right side hanging off” (hanging), and “take the Stella

Revised: Artois block and place it on top of the Nvidia block, lined up perfectly” (perfectly).

Revised: The model alsostruggles with long coreference chains, even when the anaphora are explicit, e.g. “Place the blockthat is to the right of the Starbucks block and make it the highest block on the board by placing it onthe Mercedes block.

Revised: It should be in line with the bottom block.”

Revised: [Table]

Revised: Transformer

Revised: GloVe

Revised: [Table]

Revised: Transformer

Revised: GloVe

Revised: [Table]

Revised: Transformer

Revised: [Table]

Revised: Transformer

Revised: [Table]

Revised: Transformer

Revised: [Table]

Revised: Transformer

Revised: [Table]

Revised: Transformer

Revised: [Table]

Revised: Transformer

Revised: [Table]

Revised: Transformer

Revised: [Table]

Revised: Transformer

Revised: GloVe

Revised: [Table]

Revised: Transformer

Revised: [Table]

Revised: Transformer

Revised: [Table]

Revised: Transformer

Revised: [Table]

Revised: Transformer

Revised: [Table]

Revised: Transformer

Revised: [Table]

Revised: Transformer

Revised: [Table]

Revised: [Table]

Revised: Yes

Revised: Image 45.

Revised: UNet

Revised: GloVe

Revised: No

Revised: Image 56.

Revised: Transformer

Revised: GloVe

Revised: Yes

Revised: State 88.

Revised: Transformer

Revised: GloVe

Revised: No

Revised: State 90.

Revised: Transformer

Revised: GloVe

Revised: Yes

Revised: State 89.

Revised: Transformer

Revised: GloVe

Revised: No

Revised: State 85.

Revised: Transformer

Revised: BERT

Revised: Yes

Revised: State 86.

Revised: Transformer

Revised: BERT

Revised: No

Revised: State 88.

Revised: Transformer

Revised: BERT

Revised: Yes

Revised: State 88.

Revised: Transformer

Revised: BERT

Revised: No

Revised: State 72.

Revised: Transformer

Revised: GloVe

Revised: Yes

Revised: Image 86.

Revised: Transformer

Revised: GloVe

Revised: No

Revised: Image 83.

Revised: Transformer

Revised: GloVe

Revised: Yes

Revised: Image 75.

Revised: Transformer

Revised: GloVe

Revised: No

Revised: Image 84.

Revised: Transformer

Revised: BERT

Revised: Yes

Revised: Image 88.

Revised: Transformer

Revised: BERT

Revised: No

Revised: Image 66.

Revised: Transformer

Revised: BERT

Revised: Yes

Revised: Image 88.

Revised: Transformer

Revised: BERT

Revised: No

Revised: Image 86.