Original: Guiding Multi-Step Rearrangement Tasks with Natural Language Instructions
Revised: Guiding Multi-Step Rearrangement Tasks with Natural Language Instructions

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract: Enabling human operators to interact with robotic agents using naturallanguage would allow non-experts to intuitively instruct these agents.
Revised: Abstract: Enabling human operators to interact with robotic agents using natural language would allow non-experts to intuitively instruct these agents.

Original: Towardsthis goal, we propose a novel Transformer-based model which enables a user toguide a robot arm through a 3D multi-step manipulation task with natural language commands.
Revised: Towards this goal, we propose a novel Transformer-based model which enables a user to guide a robot arm through a 3D multi-step manipulation task with natural language commands.

Original: Our system maps images and commands to masks over graspor place locations, grounding the language directly in perceptual space.
Revised: Our system maps images and commands to masks over grasp or place locations, grounding the language directly in perceptual space.

Original: In a suiteof block rearrangement tasks, we show that these masks can be combined with anexisting manipulation framework without re-training, greatly improving learningefﬁciency.
Revised: In a suite of block rearrangement tasks, we show that these masks can be combined with an existing manipulation framework without re-training, greatly improving learning efﬁciency.

Original: Our masking model is several orders of magnitude more sample efﬁcient than typical Transformer models, operating with hundreds, not millions, ofexamples.
Revised: Our masking model is several orders of magnitude more sample efﬁcient than typical Transformer models, operating with hundreds, not millions, of examples.

Original: Our modular design allows us to leverage supervised and reinforcementlearning, providing an easy interface for experimentation with different architectures.
Revised: Our modular design allows us to leverage supervised and reinforcement learning, providing an easy interface for experimentation with different architectures 2 .

Original: Our model completes block manipulation tasks with synthetic commands530% more often than a UNet-based baseline, and learns to localize actions correctly while creating a mapping of symbols to perceptual input that supports compositional reasoning.
Revised: Our model completes block manipulation tasks with synthetic commands 530% more often than a UNet-based baseline, and learns to localize actions correctly while creating a mapping of symbols to perceptual input that supports compositional reasoning.

Original: We provide a valuable resource for 3D manipulation instruction following research by porting an existing 3D block dataset with crowdsourcedlanguage to a simulated environment.
Revised: We provide a valuable resource for 3D manipulation instruction following research by porting an existing 3D block dataset with crowdsourced language to a simulated environment.

Original: Our method’s 25 .
Revised: Our method’s 25 .

Original: 3% absolute improvementin identifying the correct block on the ported dataset demonstrates its ability tohandle syntactic and lexical variation.
Revised: 3% absolute improvement in identifying the correct block on the ported dataset demonstrates its ability to handle syntactic and lexical variation.

Original:  Introduction
Revised:  Introduction

Original: Reinforcement Learning (RL) is a powerful tool for developing perception-driven robotic systems.
Revised: Reinforcement Learning (RL) is a powerful tool for developing perception-driven robotic systems.

Original: Most RL algorithms train an agent to perform a single task based on an given reward function.
Revised: Most RL algorithms train an agent to perform a single task based on an given reward function.

Original: However, what if we would like to adapt or extend the learned policy of a robot to novel or morespecialized tasks by providing instructions?
Revised: However, what if we would like to adapt or extend the learned policy of a robot to novel or more specialized tasks by providing instructions?

Original: Could we exploit previously learned RL policies for arobot to solve novel tasks now speciﬁed using natural language?
Revised: Could we exploit previously learned RL policies for a robot to solve novel tasks now speciﬁed using natural language?

Original: The ability to adapt learned policiesthrough natural language to perform novel tasks would open the door to a modular integration ofperception, language, and action, avoiding the complexities of joint training over all modalities simultaneously.
Revised: The ability to adapt learned policies through natural language to perform novel tasks would open the door to a modular integration of perception, language, and action, avoiding the complexities of joint training over all modalities simultaneously.

Original: For example, in our experiments, we constrain a general-purpose block manipulationpolicy using natural language, without needing any language data to train the policy itself.
Revised: For example, in our experiments, we constrain a general-purpose block manipulation policy using natural language, without needing any language data to train the policy itself.

Original: This kind of modular learning requires injecting guidance from language into the RL-driven actionpolicy while simultaneously addressing the challenges of natural language understanding .
Revised: This kind of modular learning requires injecting guidance from language into the RL-driven action policy while simultaneously addressing the challenges of natural language understanding.

Original: Forexample, natural language semantics is often underspeciﬁed, especially with respect to a physicalenvironment; correctly choosing one out of many potentially valid actions for a command suchas “Move the green block to the left of the right block” involves pragmatic and common-sensereasoning.
Revised: Natural language semantics is often underspeciﬁed, especially with respect to a physical environment; correctly choosing one out of many potentially valid actions for a command such as “Move the green block to the left of the right block” involves pragmatic and common-sense reasoning.

Original: Furthermore, in order to map from language to action, we need to address the symbolgrounding problem [1], i.e.  how doesone map symbols (e.g. “left”, “right”, “block”, “red”) to thephysical world?
Revised: Furthermore, in order to map from language to action, we need to address the symbol grounding problem [1], i.e. [Figure] how does map “block” the physical world?

Original: We leverage Transformer encoders to learn a strategy from data, in place of themanually deﬁned data fusion strategies found in prior work [2, 3, 4, 5, 6, 7, 8].[Figure]
Revised: We leverage Transformer encoders to learn a strategy from data, in place of the manually deﬁned data fusion strategies found in prior work [2, 3, 4, 5, 6, 7, 8].

Original: In our work, we factorize language-driven action into learning how to act and learning where to act ,following Misra et al.
Revised: In our work, we factorize language-driven action into learning how to act and learning where to act , following Misra et al.

Original: Learning how to act is accomplished by the SPOT-Q algorithm [9] whichteaches an agent to re-arrange colored blocks into stacks and rows by grasping and placing them.
Revised: Learning how to act is accomplished by the SPOT-Q algorithm [9] which teaches an agent to re-arrange colored blocks into stacks and rows by grasping and placing them.

Original: We use SPOT-Q models pre-trained without any language data for our manipulation experiments,demonstrating that our method lets us leverage existing manipulation models without additionaltraining.
Revised: We use SPOT-Q models pre-trained without any language data for our manipulation experiments, demonstrating that our method lets us leverage existing manipulation models without additional training.

Original: SPOT-Q identiﬁes regions worth exploring, e.g. containing an object, to encode a setof valid action regions for grasping and placing.
Revised: SPOT-Q identiﬁes regions worth exploring, i.e. containing an object, to encode a set of valid action regions for grasping and placing.

Original: We produce segmentation masks that are jointlyconditioned on both the language input and an image of the state and apply them with to set ofpixel-wise Q-values from the SPOT-Q RL framework.
Revised: We produce segmentation masks that are jointly conditioned on both the language input and an image of the state and apply them with to set of pixel-wise Q-values from the SPOT-Q RL framework.

Original: This process  provides us with the best actionfor a given state and command, as Figure 1 illustrates.
Revised: This process ideally provides us with the best action for a given state and command, as Figure 1 illustrates.

Original: Factorizing the learning process this way provides several advantages.
Revised: Factorizing the learning process this way provides several advantages.

Original: Firstly, it offers a moreefﬁcient path to learning.
Revised: Firstly, it offers a more efﬁcient path to learning.

Original: Separating the language grounding model from the reinforcement learningmodel, leads to a less complex RL model that receives more direct feedback for tasks.
Revised: Separating the language grounding model from the reinforcement learning model leads to a less complex RL model that receives more direct feedback for tasks.

Original: Moreover,the separation of action and language understanding allows us to learn the language component viasupervised learning, which is typically more sample efﬁcient.
Revised: Moreover, the separation of action and language understanding allows us to learn the language component via supervised learning, which is typically more sample efﬁcient.

Original: Secondly, the modular nature of themodel allows us to swap components easily: we can train a masking model on simulated imagesor real images (cf.
Revised: Secondly, the modular nature of the model allows us to swap components easily: we can train a masking model on simulated images or real images (cf.

Original: Section 5.1), allowing us to use the same action model for both simulated andreal tasks without re-training.
Revised: Section 5.1), allowing us to use the same action model for both simulated and real tasks without re-training.

Original: In an end-to-end model, a change to the input space would likelyrequire re-training from scratch.
Revised: In an end-to-end model, a change to the input space would likely require re-training from scratch.

Original: Finally, the factorized model output is more interpretable; whenan action is unsuccessful, we can inspect our model to localize the point of failure, be it a lack ofcomprehension or a mistake while acting.
Revised: Finally, the factorized model output is more interpretable; when an action is unsuccessful, we can inspect our model to localize the point of failure, be it a lack of comprehension or a mistake while acting.

Original: This may not be evident from the agent’s action alone.
Revised: This may not be evident from the agent’s action alone.

Original: Note that our implementation of this factorization is not without disadvantages: if the languagecontains information on how to act this information could be lost.
Revised: Note that our implementation of this factorization is not without disadvantages: if the language contains information on how to act this data could be lost.

Original: In summary, our contributions are:
Revised: In summary, our contributions are:

Original: We introduce a novel Transformer-based model for mapping language commands to masks over an action space.
Revised: We introduce a novel Transformer-based model for mapping language commands to masks over an action space.

Original: Our approach’s modular nature enables us to seamlessly integrate an existing Q-learning framework proposed by Hundt et al.
Revised: Our approach’s modular nature enables us to seamlessly integrate an existing Q-learning framework proposed by Hundt et al.

Original: [9] for block manipulation tasks.
Revised: [9] for block manipulation tasks.

Original: We show our model succeeds in a variety of simulated and real data settings with orders of magnitude less data than typically available to Transformers, furthering the potential use of Transformers in the low-data regimes commonly found in robotics.
Revised: We show our model succeeds in a variety of simulated and real data settings with orders of magnitude less data than typically available to Transformers, furthering the potential use of Transformers in the low-data regimes commonly found in robotics.

Original: We demonstrate this model’s utility on an existing dataset (introduced by Bisk et al.
Revised: We demonstrate this model’s utility on an existing dataset (introduced by Bisk et al.

Original:  involving a challenging block re-arrangement task with crowdsourced natural languagecommands containing syntactic diversity as well as complex natural language phenomena.4.
Revised: [10]) involving a challenging block re-arrangement task with crowdsourced natural language commands containing syntactic diversity as well as complex natural language phenomena.

Original: We release our real and simulated datasets, plus a ﬁltered version of the Bisk et al.
Revised: We release our real and simulated datasets, plus a ﬁltered version of the Bisk et al.

Original: [10]dataset containing the subset of examples feasible under a realistic model of physics.
Revised: [10] dataset containing the subset of examples feasible under a realistic model of physics.

Original:  Related Work
Revised:  Related Work

Original: We describe related work in the areas of natural language instruction following on robots, QLearning for multistep manipulation tasks, and Transformer models for Computer Vision.
Revised: We describe related work in the areas of natural language instruction following on robots, QLearning for multistep manipulation tasks, and Transformer models for Computer Vision.

Original: Robots and Natural Language Commands – We follow the Problem-Solution Sequences (PSS)paradigm [11], where a multi-step manipulation task (e.g. “build a stack”) is framed as a sequence of incremental images or state representations.
Revised: Robots and Natural Language Commands – We follow the Problem-Solution Sequences (PSS) paradigm [11], where a multi-step manipulation task (e.g. “build a stack”) is framed as a sequence of incremental images or state representations.

Original: These may be paired with per-step instructions onhow to accomplish the intermediate goal (e.g., “stack the red block on the blue block”).
Revised: These may be paired with per-step instructions on how to accomplish the intermediate goal (e.g., “stack the red block on the blue block”).

Original: Several
Revised: Several

Original: PSS datasets have been proposed [11, 10] and modeled in 2D [4, 12] and 3D [10].
Revised: PSS datasets have been proposed [11, 10] and modeled in 2D [4, 12] and 3D [10].

Original: We translate the3D dataset proposed by Bisk et al.
Revised: We translate the 3D dataset proposed by Bisk et al.

Original: [10] to a simulated environment in Section 5.2.
Revised: [10] to a simulated environment in Section 5.2.

Original: The sequentialnature of PSS relates to the hierarchical RL setting explored by Andreas et al.
Revised: The sequential nature of PSS relates to the hierarchical RL setting explored by Andreas et al.

Original: [13], who learn taskspeciﬁc sub-policies indexed by linguistic sub-goals.
Revised: [13], who learn taskspeciﬁc sub-policies indexed by linguistic sub-goals.

Original: Similar approaches are applied to zero-shotsub-goal sequence generalization[14], non-linearly-ordered sub-goal execution[15], and choosinginterpretable linguistic sub-goals [16].
Revised: Similar approaches are applied to zero-shot sub-goal sequence generalization[14], non-linearly-ordered sub-goal execution[15], and choosing interpretable linguistic sub-goals [16].

Original: While we also assume access to step-wise annotated instructions, we focus on guiding a low-level robotic manipulation policy with a supervised masking modelrather than a task-level policy.
Revised: While we also assume access to step-wise annotated instructions, we focus on guiding a low-level robotic manipulation policy with a supervised masking model rather than a task-level policy.

Original: Gangopadhyay et al.
Revised: Gangopadhyay et al.

Original: [17] and Sun et al.
Revised: [17] and Sun et al.

Original: [18] learn policies guidedby symbolic programs, while Perera et al.
Revised: [18] learn policies guided by symbolic programs, while Perera et al.

Original: [19] ﬁrst parse language into a symbolic representations.
Revised: [19] ﬁrst parse language into a symbolic representations.

Original: Our masking-based method ties the language directly to the perceptual space, eschewing intermediate symbol ontologies.
Revised: Our masking-based method ties the language directly to the perceptual space, eschewing intermediate symbol ontologies.

Original: Language has been used for reward-shaping [20, 21, 22], and reward modellearning [23]; we instead use the language-agnostic reward shaping in SPOT-Q. Stepputtis et al.
Revised: Language has been used for reward-shaping [20, 21, 22], and reward model learning [23]; we instead use the language-agnostic reward shaping in SPOT-Q. Stepputtis et al.

Original: [24]and Lynch and Sermanet [25] use language-conditioned imitation learning for manipulation policies;we instead use Q-learning.
Revised: [24] and Lynch and Sermanet [25] use language-conditioned imitation learning for manipulation policies; we instead use Q-learning.

Original: TransporterNets
Revised: TransporterNets

Original: [26] learn to perform multi-step tasks from demonstration, but lacks language input.
Revised: [26] learn to perform multi-step tasks from demonstration, but lacks language input.

Original: Like our work, Concept2Robot [27] completes language-describedtasks by combining learning from demonstration and RL.
Revised: Like our work, Concept2Robot [27] completes language-described tasks by combining learning from demonstration and RL.

Original: However, it differs from our system alongmultiple axes: we examine challenging multi-step tasks with progress reversal [9]; we succeed onmuch smaller datasets; we consider crowdsourced commands; and we evaluate on real robot data.
Revised: However, it differs from our system along multiple axes: we examine challenging multi-step tasks with progress reversal [9]; we succeed on much smaller datasets; we consider crowdsourced commands; and we evaluate on real robot data.

Original: Nguyen et al.
Revised: Nguyen et al.

Original: [28] guide object retrieval with language, but do not model placement.
Revised: [28] guide object retrieval with language, but do not model placement.

Original: Visual Goal Prediction (VGP), introduced by Misra et al.
Revised: Visual Goal Prediction (VGP), introduced by Misra et al.

Original: [4] for language-conditioned navigation[29, 30, 4, 8, 7, 5], frames the problem of choosing an action region as a semantic segmentation taskin which each pixel as a separate binary decision: act or do not act.
Revised: [4] for language-conditioned navigation [29, 30, 4, 8, 7, 5], frames the problem of choosing an action region as a semantic segmentation task in which each pixel as a separate binary decision: act or do not act.

Original: VGP has directed a simulated[5] and real quadcopter [8, 7].
Revised: VGP has directed a simulated [5] and real quadcopter [8, 7].

Original: We implement VGP with UNet[31, 4, 5, 7, 8], as a baseline.
Revised: We implement VGP with UNet[31, 4, 5, 7, 8], as a baseline.

Original: Q-learning for Multi-Step Tasks – Mohammed et al.
Revised: Q-learning for Multi-Step Tasks – Mohammed et al.

Original: [32] reviews RL for robot grasping.
Revised: [32] reviews RL for robot grasping.

Original: Zenget al.
Revised: Zeng et al.

Original: [33] shows Q-Learning outperforms supervised action segmentation for table decluttering[34,35, 36], instead, “Good Robot!”
Revised: [33] shows Q-Learning outperforms supervised action segmentation for table decluttering[34, 35, 36], instead, “Good Robot!”

Original: [9] segments actions worth exploring to accelerate Q-Learning ofsingle-task, multi-step robot learning, all without language input.
Revised: [9] segments actions worth exploring to accelerate Q-Learning of single-task, multi-step robot learning, all without language input.

Original: “Good Robot!”
Revised: “Good Robot!”

Original: [9] was also theﬁrst to complete multi-step tasks with consideration of progress reversal , a failure case where actionsundo previous progress such that Ω( n ) additional actions are required to recover, e.g. toppling thetower in a partially completed stacking task.
Revised: [9] was also the ﬁrst to complete multi-step tasks with consideration of progress reversal , a failure case where actions undo previous progress such that Ω( n ) additional actions are required to recover, e.g. toppling the tower in a partially completed stacking task.

Original: We segment actions worth exploring via linguistic inputfor multi-step, multi-task execution considering progress reversal and low-data[3, 34] conditions.
Revised: We segment actions worth exploring via linguistic input for multi-step, multi-task execution considering progress reversal and low-data[3, 34] conditions.

Original: Transformer Encoders for Vision – Though Transformer encoders, which build representationsusing self-attention, have become standard in NLP, computer vision tasks are still dominated by convolutional models, of which UNet [31] is a paradigmatic instance.
Revised: Transformer Encoders for Vision – Though Transformer encoders, which build representations using self-attention, have become standard in NLP, computer vision tasks are still dominated by convolutional models, of which UNet [31] is a paradigmatic instance.

Original: The use of Transformers in visiontasks has been held back by the quadratic complexity of the Transformer’s self-attention layer; animage input treated as a sequence of pixels would quickly become computationally intractable [37].
Revised: The use of Transformers in vision tasks has been held back by the quadratic complexity of the Transformer’s self-attention layer; an image input treated as a sequence of pixels would quickly become computationally intractable [37].

Original: Dosovitskiy et al.
Revised: Dosovitskiy et al.

Original: [37] introduce a minimal modiﬁcation to perform image recognition using Transformers, tiling the image into square patches, which they concatenate into a sequence and augmentwith learned linear positional embeddings.
Revised: [37] introduce a minimal modiﬁcation to perform image recognition using Transformers, tiling the image into square patches, which they concatenate into a sequence and augment with learned linear positional embeddings.

Original: Though the model presented by Dosovitskiy et al.
Revised: Though the model presented by Dosovitskiy et al.

Original: performs image-level recognition well, it cannot perform pixel-level tasks (e.g., semantic segmentation)as-is.
Revised: performs image-level recognition well, it cannot perform pixel-level tasks (e.g., semantic segmentation) as-is.

Original: Ranftl et al.
Revised: Ranftl et al.

Original: [38] present an extension to this model for pixel-level prediction.
Revised: [38] present an extension to this model for pixel-level prediction.

Original:  Models
Revised:  Models

Original: We propose a Transformer-based model to convert image and natural language inputs into separatemasks for grasping and placing, thereby allowing users to guide multi-step manipulation tasks.
Revised: We propose a Transformer-based model to convert image and natural language inputs into separate masks for grasping and placing, thereby allowing users to guide multi-step manipulation tasks.

Original: We begin by tiling the n × n birds-eye image of the scene with C colored blocks into k × k patches,whose RGB pixel values are concatenated into one vector, resulting in patch vectors
Revised: We begin by tiling the n × n birds-eye image of the scene with C colored blocks into k × k patches, whose RGB pixel values are concatenated into one vector, resulting in patch vectors

Original: I = i 1 , . .
Revised: I = i 1 , . .

Original: , i L .
Revised: , i L .

Original: The resulting vectors are augmented with a ﬁxed sinusoidal positional encoding [39].
Revised: The resulting vectors are augmented with a ﬁxed sinusoidal positional encoding [39].

Original: Similarly,each token in the command string W = w 1 , . . .
Revised: Similarly, each token in the command string W = w 1 , . . .

Original: w m is embedded into a continuous space resultingin vectors e 1 , . . .
Revised: w m is embedded into a continuous space resulting in vectors e 1 , . . .

Original: , e m , and those vectors are augmented with positional encodings as well.
Revised: , e m , and those vectors are augmented with positional encodings as well.

Original: The goal is to train a model that produces a pair of language-deﬁned spatial masks ˆ M place ∈ R n × nand ˆ M grasp ∈ R n × n , and an optional pair of state reconstructions ˆ S grasp , 
Revised: The goal is to train a model that produces a pair of language-deﬁned spatial masks ˆ M place ∈ R n × n and ˆ M grasp ∈ R n × n , and an optional pair of state reconstructions ˆ S grasp , ˆ

Original: ˆ S place ∈ R C × n × n given by:
Revised: S place ∈ R C × n × n given by:

Original: [Equation]
Revised: [Equation]

Original: where f . are parameterized by Transformer encoders with output size k , H m
Revised: where f . are parameterized by Transformer encoders with output size k , H m

Original: ∈ R k × 2 , H s ∈ R k × C ,σ is the softmax function, and t ∈ { grasp , place } .
Revised: ∈ R k × 2 , H s ∈ R k × C , σ is the softmax function, and t ∈ { grasp , place } .

Original: The reconstruction function R up-samples kpatches by copying the patch value across each of its constituent pixels, resulting in an n × n image.
Revised: The reconstruction function R up-samples k patches by copying the patch value across each of its constituent pixels, resulting in an n × n image.

Original: Note that structurally, the “shared,” “grasp,” and “place” Transformers are identical, but while the“shared” Transformer is used both for grasp and place prediction, the others are task-speciﬁc.
Revised: Note that structurally, the “shared,” “grasp,” and “place” Transformers are identical, but while the “shared” Transformer is used both for grasp and place prediction, the others are task-speciﬁc.

Original: We train the masking model separately from the manipulation model.
Revised: We train the masking model separately from the manipulation model.

Original: For the latter model, weuse a pre-trained SPOT-Q model from Hundt et al.
Revised: For the latter model, we use a pre-trained SPOT-Q model from Hundt et al.

Original: [9], which produces pixel-wise Q-values forpicking and placing, making it easy to integrate with our approach.
Revised: [9], which produces pixel-wise Q-values for picking and placing, making it easy to integrate with our approach.

Original: Furthermore, pixel-wise Qvalues facilitate potential future work on extending our method to non-cubic shapes.
Revised: Furthermore, pixel-wise Qvalues facilitate potential future work on extending our method to non-cubic shapes.

Original: All maskingmodels are trained with a multi-task loss, composed of a pixel-wise binary cross-entropy loss for themasks, and an optional auxiliary reconstruction loss:
Revised: All masking models are trained with a multi-task loss, composed of a pixel-wise binary cross-entropy loss for the masks, and an optional auxiliary reconstruction loss:

Original: [Equation]
Revised: [Equation]

Original: The reconstruction loss here forces the model to semantically segment the image by identifying pixelwith its block ID, or labeling it as background.
Revised: The reconstruction loss here forces the model to semantically segment the image by identifying pixel with its block ID, or labeling it as background.

Original: As in Nguyen and Salazar [40], we use a pre-norm
Revised: As in Nguyen and Salazar [40], we use a pre-norm

Original: Transformer layer with scaled initialization.
Revised: Transformer layer with scaled initialization.

Original: Our learning rate schedule is based on Vaswani et al.[39], with a varied number of warmup steps as in Nguyen and Salazar.
Revised: Our learning rate schedule is based on Vaswani et al. [39], with a varied number of warmup steps as in Nguyen and Salazar.

Original: A comprehensive report ofall hyperparameters and a ﬁgure of our architecture is given in Appendix A.
Revised: A comprehensive report of all hyperparameters and a ﬁgure of our architecture is given in Appendix A.

Original: Q-Learning – Our model relies on a learned function from states and actions to expected rewards,
Revised: Q-Learning – Our model relies on a learned function from states and actions to expected rewards,

Original: Q ( s t , a ) , which we learn via the SPOT-Q algorithm [9], which introduces novel reward shapingprocesses for improving efﬁciency.
Revised: Q ( s t , a ) , which we learn via the SPOT-Q algorithm [9], which introduces novel reward shaping processes for improving efﬁciency.

Original: More details are given in Appendix A.3.
Revised: More details are given in Appendix A.3.

Original: Given a pixel-wisen × n state space of an image and two actions, grasp and place, the output of this function can beexpressed as a tensor Q ∈ R n × n × 2 .
Revised: Given a pixel-wise n × n state space of an image and two actions, grasp and place, the output of this function can be expressed as a tensor Q ∈ R n × n × 2 .

Original:  Datasets and Resources 
Revised:  Datasets and Resources 4.1 Simulation Experiments with Synthetic Commands

Original: 4.1 Simulation Experiments with Synthetic Commands

Original: We create a dataset of commands and grasp/place masks to train our masking model and demonstrate its performance in simulated environment on a robot arm.
Revised: We create a dataset of commands and grasp/place masks to train our masking model and demonstrate its performance in simulated environment on a robot arm.

Original: Using trajectories from Hundt et al.[9], we annotate successful grasp-and-place action pairs with synthetic commands generated from atemplate, and use simulation state information to construct the reference masks M t and state information S t .
Revised: Using trajectories from Hundt et al. [9], we annotate successful grasp-and-place action pairs with synthetic commands generated from a template, and use simulation state information to construct the reference masks M t and state information S t .

Original: While synthetic language makes simplifying assumptions about the syntactic and lexicaldiversity of natural language, it provides a controllable test-bed for symbolic reasoning, and is oftenused to study compositionality [41, 42, 43], as we do in Section 5.1.
Revised: While synthetic language makes simplifying assumptions about the syntactic and lexical diversity of natural language, it provides a controllable test-bed for symbolic reasoning, and is often used to study compositionality [41, 42, 43], as we do in Section 5.1.

Original: For row-making, the template is“move the color a block next to the color b block”, and for stacking the template is “stack thecolor a block on the color b block”.
Revised: For row-making, the template is “move the color a block next to the color b block”, and for stacking the template is “stack the color a block on the color b block”.

Original: We train supervised VGP models (cf.
Revised: We train supervised VGP models (cf.

Original: Section 3) on thesecommands, and combine the produced masks with the Q-values from the SPOT-Q framework in asimulated environment to execute the actions in order.
Revised: Section 3) on these commands, and combine the produced masks with the Q-values from the SPOT-Q framework in a simulated environment to execute the actions in order.

Original: The visual input to our models is the concatenation of a birds-eye color image and depth image.
Revised: The visual input to our models is the concatenation of a birds-eye color image and depth image.

Original: The training data is augmented by ﬂipping eachimage across 4 axes, manipulating the command accordingly, as well as by the addition of pixelwise Gaussian noise drawn from N (0 , 0 . 05) .
Revised: The training data is augmented by ﬂipping each image across 4 axes, manipulating the command accordingly, as well as by the addition of pixelwise Gaussian noise drawn from N (0 , 0 . 05) .

Original: This results in train/dev/test splits of 5520 / 46 / 46 and8970 / 74 / 76 for row-making and stacking, respectively, where each sample is an input image andcommand paired with a reference place and grasp mask.
Revised: This results in train/dev/test splits of 5520 / 46 / 46 and 8970 / 74 / 76 for row-making and stacking, respectively, where each sample is an input image and command paired with a reference place and grasp mask.

Original: Experiments and results on VGP metricsand in the simulation for this data are given in Section 5.1.
Revised: Experiments and results on VGP metrics and in the simulation for this data are given in Section 5.1.

Original: 4.2 Transfer to Real Images
Revised: 4.2 Transfer to Real Images

Original: In order to be useful in real-life settings, our method must also be applicable to real data.
Revised: In order to be useful in real-life settings, our method must also be applicable to real data.

Original: However,unlike simulated data, data from a real robot is expensive and time-consuming to collect.
Revised: However, unlike simulated data, data from a real robot is expensive and time-consuming to collect.

Original: Therefore,it is natural to ask: how much real data is needed to train an adequate VGP model?
Revised: Therefore, it is natural to ask: how much real data is needed to train an adequate VGP model?

Original: The Hundt et al.
Revised: The Hundt et al.

Original: [9] real robot trial logs include 1496 actions each for row-making and stacking.
Revised: [9] real robot trial logs include 1496 actions each for row-making and stacking.

Original: We split these into an 80/10/10 train/dev/test split and augment the examples by ﬂipping, producing5980 , 747 , 748 datapoints.
Revised: We split these into an 80/10/10 train/dev/test split and augment the examples by ﬂipping, producing 5980 , 747 , 748 datapoints.

Original: Image collection details are given in Appendix B.2.
Revised: Image collection details are given in Appendix B.2.

Original: Given the small sizeof the real dataset, we pre-train a model on simulated images and ﬁne-tune it with the real images.
Revised: Given the small size of the real dataset, we pre-train a model on simulated images and ﬁne-tune it with the real images.

Original: 4.3 Compositionality
Revised: 4.3 Compositionality

Original: In linguistics, compositionality is typically taken to mean that the meaning of a whole utterance iscomposed of the meanings of its parts; the meaning of “stack the red block on the blue block” is in part made up of the independent meanings of “red” and “blue”.
Revised: In linguistics, compositionality is typically taken to mean that the meaning of a whole utterance is composed of the meanings of its parts; the meaning of “stack the red block on the blue block” is in part made up of the independent meanings of “red” and “blue”.

Original: Compositionality lets speakersto re-combine known symbols in new ways, underpinning human language use.
Revised: Compositionality lets speakers re-combine known symbols in new ways, underpinning human language use.

Original: In our task, acompositional reasoner should correctly execute “stack the red block on the blue block” (referredto here as stack(r, b) ) without having seen that combination; it should be able to localize redand blue blocks from other color contexts (e.g. stack(r, g) , stack(y, b) , etc.).
Revised: In our task, a compositional reasoner should correctly execute “stack the red block on the blue block” (referred to here as stack(r, b) ) without having seen that combination; it should be able to localize red and blue blocks from other color contexts (e.g. stack(r, g) , stack(y, b) , etc.).

Original: However,there is no explicit constraint during training to enforce this; the model could learn to interpret eachof the 23 possible color pairs atomically.
Revised: However, there is no explicit constraint during training to enforce this; the model could learn to interpret each of the 23 possible color pairs atomically.

Original: We test our model’s compositional reasoning via its abilityto generalize to unseen color combinations.
Revised: We test our model’s compositional reasoning via its ability to generalize to unseen color combinations.

Original: We hold out 6 color pairs during training ( 26% ofthe 23 order-invariant pairs) and test exclusively on those pairs.
Revised: We hold out 6 color pairs during training ( 26% of theorder-invariant pairs) and test exclusively on those pairs.

Original: For example, we train a maskingmodel with all color combinations except those made only of red, blue, yellow, and green  and thentest only on stack( { r,b,g,y } , { r,b,g,y }  examples.
Revised: For example, we train a masking model with all color combinations except those made only of red, blue, yellow, and green blocks and then test only on stack( { r,b,g,y } , { r,b,g,y } ) examples.

Original: We hold-out each subset, training andevaluating a model as described above, average the metrics across all subsets, and compare againsta random train and test split baseline, where all color pairs are seen at train time.
Revised: We hold-out each subset, training and evaluating a model as described above, average the metrics across all subsets, and compare against a random train and test split baseline, where all color pairs are seen at train time.

Original: For efﬁciency, wetrain on each task separately and do not augment with Gaussian noise.
Revised: For efﬁciency, we train on each task separately and do not augment with Gaussian noise.

Original: 4.4 Naturalistic Language Commands with Complex Scenes
Revised: 4.4 Naturalistic Language Commands with Complex Scenes

Original: Language sourced from human speakers is syntactically and lexically diverse, with mistakes, misspellings, and other idiosyncrasies that make far more challenging to model than template data.
Revised: Language sourced from human speakers is syntactically and lexically diverse, with mistakes, misspellings, and other idiosyncrasies that make far more challenging to model than template data.

Original: Analyzing synthetic language showcases our model’s ability to integrate discrete symbolic commandswith visual perception and action, as well as our model’s ability to generalize in a compositionalfashion, but this does not imply that our model generalizes to the complexities of real language.
Revised: Analyzing synthetic language showcases our model’s ability to integrate discrete symbolic commands with visual perception and action, as well as our model’s ability to generalize in a compositional fashion, but this does not imply that our model generalizes to the complexities of real language.

Original: We examine a 3D block manipulation dataset with human natural language instructions that wasintroduced by Bisk et al.
Revised: We examine a 3D block manipulation dataset with human natural language instructions that was introduced by Bisk et al.

Original: [10] (cf.
Revised: [10] (cf.

Original: Fig. 2).
Revised: Fig. 2).

Original: This dataset is set apart from other block-world data byits focus on complex instructions.
Revised: This dataset is set apart from other block-world data by its focus on complex instructions.

Original: Rather than using synthetic utterances, instructions were sourcedfrom crowdworkers (9 per example), who were shown a pair of images and asked to provide instructions to a robot, with the goal of manipulating the blocks in the ﬁrst image such that they result in thesecond image.
Revised: Rather than using synthetic utterances, instructions were sourced from crowdworkers (9 per example), who were shown a pair of images and asked to provide instructions to a robot, with the goal of manipulating the blocks in the ﬁrst image such that they result in the second image.

Original: This collection method result in more syntactically diverse utterances with complexlinguistic concepts, such as “mirroring” and “balancing”.
Revised: This collection method result in more syntactically diverse utterances with complex linguistic concepts, such as “mirroring” and “balancing”.

Original: [Figure]
Revised: [Figure]

Original: The dataset is comprised of a sequence of state representations paired with natural language instructions on how to transition between states.
Revised: The dataset is comprised of a sequence of state representations paired with natural language instructions on how to transition between states.

Original: While theblocks dataset is designed for robot manipulation,several assumptions were made to simplify annotation by crowdworkers.
Revised: While the blocks dataset is designed for robot manipulation, several assumptions were made to simplify annotation by crowdworkers.

Original: Crucially, physics in the environment were turned off while the structures werebeing constructed and annotated, resulting in somephysically infeasible structures.
Revised: Crucially, physics in the environment were turned off while the structures were being constructed and annotated, resulting in some physically infeasible structures.

Original: To facilitate research in the blocks domain, and make the datasetmore useful for training and evaluating embodied agents, we create a ﬁltered version of the dataset,where physically infeasible examples are removed.
Revised: To facilitate research in the blocks domain, and make the dataset more useful for training and evaluating embodied agents, we create a ﬁltered version of the dataset, where physically infeasible examples are removed.

Original: Our ﬁltering process removes 49 .
Revised: Our ﬁltering process removes 49 .

Original: 93%and 38 .
Revised: 93% and 38 .

Original: 23% of train, dev, and test, respectively.
Revised: 23% of train, dev, and test, respectively.

Original: Appendix C.1 contains more details on the translation process and data preprocessing.
Revised: Appendix C.1 contains more details on the translation process and data preprocessing.

Original: To model the dataset’s world state sequences in the VGPframework, we translate the 3D Cartesian coordinates of each block given by the dataset’s state representation into a top-down grid view, where each pixel corresponds to the ID of the block locatedat that position.
Revised: To model the dataset’s world state sequences in the VGP framework, we translate the 3D Cartesian coordinates of each block given by the dataset’s state representation into a top-down grid view, where each pixel corresponds to the ID of the block located at that position.

Original: We refer to this image as the scene’s “state representation.”
Revised: We refer to this image as the scene’s “state representation.”

Original: While the blocks dataset contains all the state information required to produce image-like staterepresentations, this cannot generally be assumed to be the case.
Revised: While the blocks dataset contains all the state information required to produce image-like state representations, this cannot generally be assumed to be the case.

Original: A more likely scenario mimicks
Revised: A more likely scenario mimicks

Original: Section 4.1, where the model’s input is a combined color and depth image.
Revised: Section 4.1, where the model’s input is a combined color and depth image.

Original: After translating thedataset into the simulated environment used in Section 4.1, we collect color and depth images forthe whole blocks dataset.
Revised: After translating the dataset into the simulated environment used in Section 4.1, we collect color and depth images for the whole blocks dataset.

Original: In the original dataset, each block type is identiﬁed by a company logo.
Revised: In the original dataset, each block type is identiﬁed by a company logo.

Original: Since the logos were downsampled by the simulator, we also assign a color to each type.
Revised: Since the logos were downsampled by the simulator, we also assign a color to each type.

Original: Thedemographics and labor practices underlying its original creation are unknown [10, 44].
Revised: The demographics and labor practices underlying its original creation are unknown [10, 44].

Original: We ﬁlteredtheir dataset without crowdworkers; and it is intended for robotics research, not product deployment.
Revised: We ﬁltered their dataset without crowdworkers; and it is intended for robotics research, not product deployment.

Original: Experiments and results for this dataset are in Sec.
Revised: Experiments and results for this dataset are in Sec.

Original:  Experiments 
Revised:  Experiments 5.1 Simulation Experiments with Synthetic Commands

Original: 5.1 Simulation Experiments with Synthetic Commands

Original: An operator guiding a robot through a manipulation task may have a particular outcome in mind: forexample, a stack with a particular order of blocks.
Revised: An operator guiding a robot through a manipulation task may have a particular outcome in mind: for example, a stack with a particular order of blocks.

Original: Language can naturally express such constraints; i.e. “stack the red block on the yellow block.”
Revised: Language can naturally express such constraints; i.e. “stack the red block on the yellow block.”

Original: Following the template generation and data annotationin Section 4.1, we train baseline and Transformer VGP models to produce output masks ˆ M grasp andˆ M place and combine them with a SPOT-Q model to manipulate blocks in a simulated environment.
Revised: Following the template generation and data annotation in Section 4.1, we train baseline and Transformer VGP models to produce output masks ˆ M grasp and ˆ M place and combine them with a SPOT-Q model to manipulate blocks in a simulated environment.

Original: These masks are ﬁltered and thresholded using 0-1 block masks, following Hundt et al.
Revised: These masks are ﬁltered and thresholded using 0-1 block masks, following Hundt et al.

Original: Toevaluate the VGP system in isolation, we report the following metrics:
Revised: To evaluate the VGP system in isolation, we report the following metrics:

Original: • Block Accuracy is the percentage predictions for which the grasp mask overlaps with the correct block, i
Revised: • Block Accuracy is the percentage predictions for which the grasp mask overlaps with the correct block, i

Original: e. the model chooses the correct block to pick up.
Revised: e. the model chooses the correct block to pick up.

Original: Higher is better.
Revised: Higher is better.

Original: • Idealized Error Score (IES) captures the performance of an idealized robot on the manipulation task, choosing a block to move using the grasp mask, and a location using the place mask.
Revised: • Idealized Error Score (IES) captures the performance of an idealized robot on the manipulation task, choosing a block to move using the grasp mask, and a location using the place mask.

Original: After the chosen block is teleported to the predicted location, we measure the Euclidean distance between the block’s new location and its reference location in block lengths.
Revised: After the chosen block is teleported to the predicted location, we measure the Euclidean distance between the block’s new location and its reference location in block lengths.

Original: Lower is better.
Revised: Lower is better.

Original: Our baseline is a UNet-based VGP model; speciﬁcally, we extend the L ING UN ET model presentedby Misra et al.
Revised: Our baseline is a UNet-based VGP model; speciﬁcally, we extend the L ING UN ET model presented by Misra et al.

Original: [4] with an attentional fusion mechanism.
Revised: [4] with an attentional fusion mechanism.

Original: While the original L ING UN ET used theﬁnal state of an LSTM for the language representation, we use a weighted average of BiLSTM statesproduced by an attention mechanism.
Revised: While the original L ING UN ET used the ﬁnal state of an LSTM for the language representation, we use a weighted average of BiLSTM states produced by an attention mechanism.

Original: We also ran asmall end-to-end Q-Learning experiment thatbackpropagates through both the FCN and Transformer, but the model did not converge.
Revised: We also ran a end-to-end Q-Learning experiment that backpropagates through both the FCN and Transformer, but the model did not converge.

Revised: Higher accuracy and lower Idealized Error

Revised: Score (IES, measured in blocks), is better.

Revised: TFMR is Transformer.

Revised: Model Train

Revised: Test

Original: [Table]
Revised: [Table]

Revised: TFMR

Revised: Both

Revised: Rows 91.3% 0.

Revised: UNet

Original: Stacks Stacks 27.9% 1.
Revised: Stacks Stacks 27.9% 1.

Original: TFMR Stacks Stacks 81.5% 0.
Revised: TFMR Stacks Stacks 81.5% 0.

Original: UNet
Revised: UNet

Original: Both
Revised: Both

Original: Stacks 75.2% 0.
Revised: Stacks 75.2% 0.

Original: TFMR
Revised: TFMR

Original: Both
Revised: Both

Original: Stacks 90.4% 0.
Revised: Stacks 90.4% 0.

Original: Further details on the masking, as well as the simulation environment, can be found in Appendix B.1.
Revised: Further details on the masking, as well as the simulation environment, can be found in Appendix B.1.

Original: The ﬁltered/thresholded masks ˆ M grasp and ˆ M placeare stacked into a single mask ˆ M ∈ R n × n × 2 .
Revised: The ﬁltered/thresholded masks ˆ M grasp and ˆ M place are stacked into a single mask ˆ M ∈ R n × n × 2 .

Original: A ﬁnalmatrix of Q (cid:48) of Q values constrained by the languageis given by Q (cid:48)
Revised: A ﬁnal matrix of Q (cid:48) of Q values constrained by the language is given by Q (cid:48)

Original: =
Revised: =

Original: ˆ M (cid:12) Q , where (cid:12) is the Hadamardproduct; Q (cid:48) is then used to choose an action and alocation at which to grasp/place in the simulation.
Revised: ˆ M (cid:12) Q , where (cid:12) is the Hadamard product; Q (cid:48) is then used to choose an action and a location at which to grasp/place in the simulation.

Original: We count stacks and rows as successful only withthe speciﬁed color order, plus we tally the number ofactions this required, and the percentage of times thegripper picked up the correct block.
Revised: We count stacks and rows as successful only with the speciﬁed color order, also tallying the number of actions this required and the percentage of times the gripper picked up the correct block.

Original: We contrast ourmodel with the SPOT-Q framework alone, which hasno access to the order constraints, and with our UNetbaseline.
Revised: We contrast our model with the SPOT-Q framework alone, which has no access to the order constraints, and with our UNet baseline.

Original: Each model was evaluated with the same SPOT-Q model and tested on 100 trials.
Revised: Each model was evaluated with the same SPOT-Q model and tested on 100 trials.

Original: Hundtet al.
Revised: Hundt et al.

Original: [9] set a 30 action limit for failed trials; we added several evaluation heuristics to terminatefailed trials when an irrecoverable state was reached, measuring color ordering errors.
Revised: [9] set a 30 action limit for failed trials; we added several evaluation heuristics to terminate failed trials when an irrecoverable state was reached, measuring color ordering errors.

Original: Stackingtasks use 8block colors and row-making tasks have 4.
Revised: Stacking tasks use 8 colors and row-making tasks use 4.

Original: [Table]
Revised: [Table]

Original: UNet

Original: [Table]

Original: NaN [Table] [Table] Table 1 shows the visual goal prediction metrics for all model variants trained on one or bothtasks.
Revised: Table 1 shows the visual goal prediction metrics for all model variants trained on one or both tasks.

Original: We see that the Transformer-based models (TFMR) uniformly out-perform the UNetbaseline across all metrics, often by large margins.
Revised: We see that the Transformer-based models (TFMR) uniformly out-perform the UNet baseline across all metrics, often by large margins.

Original: For both UNet and Transformer architectures, training on two tasks simultaneouslycovers a broader range of semantically different examples with a corresponding increase inperformance on all metrics.
Revised: For both UNet and Transformer architectures, training on two tasks simultaneously covers a broader range of semantically different examples with a corresponding increase in performance on all metrics.

Original: Notably, these examples differ by task: ﬁrstly, the row-makingtasks contain a subset of the colors available in stacking.
Revised: Notably, these examples differ by task: ﬁrstly, row-making uses a subset of the colors available in stacking.

Original: Secondly, while the grasp data acrossboth tasks looks roughly identical, the data for place masks looks quite different.
Revised: Secondly, while the grasp data across both tasks is roughly identical, the data for place masks is quite different.

Original: For example, both“stack the blue block on the red block” and “move the blue block next to the red block” result ingrasp masks over the blue block, but whereas the place mask for stacking picks out the block onwhich to stack (the red block), the row mask speciﬁes a location near the target block.
Revised: For example, both “stack the blue block on the red block” and “move the blue block next to the red block” result in grasp masks over the blue block, but whereas the place mask for stacking picks out the block on which to stack (the red block), the row mask speciﬁes a location near the target block.

Original: Thus, it isintriguing that combining tasks beneﬁts not only the block accuracy, but also the IES.
Revised: Thus, it is noteworthy that combining tasks beneﬁts not only the block accuracy, but also the IES.

Original: Table 2 shows the task completion percentage for the best UNet and Transformer models (trainedon both tasks) in the simulated environment, as well as the percentage of successful grasps thatpicked up the correct color, and the average number of actions taken to complete a successful trial.
Revised: Table 2 shows the task completion percentage for the best UNet and Transformer models (trained on both tasks) in the simulated environment, as well as the percentage of successful grasps that picked up the correct color, and the average number of actions taken to complete a successful trial.

Original: In the aggregate, we see that high performance on VGP metrics does not translate to equivalent performance in the simulation environment.
Revised: In the aggregate, we see that high performance on VGP metrics does not translate to equivalent performance in the simulation environment.

Original: For example, both for stacking and row-making, thepercentage of correct color grasps is substantially lower that the VGP equivalent metric, block accuracy.
Revised: For example, both for stacking and row-making, the percentage of correct color grasps is substantially lower that the VGP equivalent metric, block accuracy.

Original: For the Transformer model, this may be due to the imprecision of the patch-wise tiling ofthe image: when heuristically combined with a common-sense mask, a patch may cover multipleadjacent blocks.
Revised: For the Transformer model, this may be due to the imprecision of the patch-wise tiling of the image: when heuristically combined with a common-sense mask, a patch may cover multiple adjacent blocks.

Original: This is an inherent limitation of our approach.
Revised: This is an inherent limitation of our approach.

Original: When compared to the RL-only and
Revised: When compared to the RL-only and

Original: UNet baselines, our Transformer-based model succeeds in signiﬁcantly more trials.
Revised: UNet baselines, our Transformer-based model succeeds in signiﬁcantly more trials.

Original: Additionally,the mean number of actions across successful trials is lower, indicating that the Transformer-basedmasks result in fewer actions per successful trial.
Revised: Additionally, the mean number of actions across successful trials is lower, indicating that the Transformer-based masks result in fewer actions per successful trial.

Original: Note that for stacking, both baselines [-succeededon 
-] out of the 100 trials; due to toppling and occlusion, stacking poses a signiﬁcantly harder taskthan row-making, and it is harder to recover from a misplaced block, even if the trial is not ended.
Revised: Note that for stacking, both baselines [-succeededon 
-]succeeded on 0 out of the 100 trials; due to toppling and occlusion, stacking poses a signiﬁcantly harder task than row-making, and it is harder to recover from a misplaced block, even if the trial is not ended.

Original: We see that even a fairly high block accuracy and low IES are insufﬁcient for high task performance.
Revised: We see that even a fairly high block accuracy and low IES are insufﬁcient for high task performance.

Original: This highlights the one of the main challenges of multistep tasks: small error rates compound acrossmultiple actions, yielding low task performance.
Revised: This highlights the one of the main challenges of multistep tasks: small error rates compound across multiple actions, yielding low task performance.

Original: A full error analysis is given in Appendix [-B.
-]
Revised: A full error analysis is given in Appendix [-B.
-]B.

Original: [Figure]
Revised: [Figure]

Original: Transfer to Real Images – In Figure 3, we plot the
Revised: Transfer to Real Images – In Figure 3, we plot the

Original: IES of row-making and stacking models when trainedon increasing subsets of real data.
Revised: IES of row-making and stacking models when trained on increasing subsets of real data.

Original: We see that for bothrows and stacks, performance improves slightly as we increase the amount of training data, with diminishing returns after 30% of the available data (1794 examples) areused.
Revised: We see that for both rows and stacks, performance improves slightly as we increase the amount of training data, with diminishing returns after 30% of the available data (1794 examples) are used.

Original: Given that Transformers typically require severalorders of magnitude more data to reach good performance[45, 40, 46], the fact that our architecture can be trainedwith mere thousands of simulated training examples, andsuccessfully ﬁnetuned with even fewer real examples, extracted from only a few hundred real images, represents a signiﬁcant achievement.
Revised: Given that Transformers typically require several orders of magnitude more data to reach good performance [45, 40, 46], the fact that our architecture can be trained with mere thousands of simulated training examples, and successfully ﬁnetuned with even fewer real examples, extracted from only a few hundred real images, represents a signiﬁcant achievement.

Original: For a proof-of-concept demonstration of our model running on a real robot, see Appendix B.4 and the supplementary materials.
Revised: For a proof-of-concept demonstration of our model running on a real robot, see Appendix B.4 and the supplementary materials.

Original: Compositionality – By generating held-out data splits as described in Section 4.3, we can testthe ability of our model to generalize compositionally, i.e. to recombine known symbols in unseencontexts.
Revised: Compositionality – By generating held-out data splits as described in Section 4.3, we can test the ability of our model to generalize compositionally, i.e. to recombine known symbols in new contexts.

Original: A model that has learned compositional color representations will be able to combinecolor representations across contexts; for example, if it has seen red blocks in the context of greenblocks, it should generalize to a blue block context.
Revised: A model with compositional color representations will be able to combine colors across contexts; for example, if it has seen red blocks in the context of green blocks, it should generalize to a blue block context.

Original: We would therefore expect the model to performroughly on par with a model trained on a random data split, where all contexts are seen at train time.
Revised: We would therefore expect the model to perform roughly on par with a model trained on a i.i.d. data split, where all contexts are seen at train time.

Original: Conversely, if the model has learned to interpret each utterance as an atomic statement, we shouldexpect much lowerperformance on the held-out subset.
Revised: Conversely, if the model has learned to interpret each utterance atomically, we should expect much lower held-out performance.

Original: [Table]
Revised: [Table]

Original: Our results are reported in Table 3.
Revised: Our results are reported in Table 3.

Original: Both rows andstacks see a slight drop in block accuracy from the baseline.
Revised: Both rows and stacks see a slight drop in block accuracy from the baseline.

Original: Nevertheless, the lack of a large disparity betweenthe baseline and the held-out condition suggests that themodel has largely learned to generalize compositionallyto unseen block combinations.
Revised: Nevertheless, the lack of a large gap between the baseline and the held-out conditions suggests that the model has largely learned to generalize compositionally to unseen block combinations.

Original: Thisﬁnding suggests that,although the model certainly has the capacity to memorize 23 combinations, and although it is not in any waytrained to perform compositional reasoning, it naturallylearns to bind symbols to percepts in a way that generalizes across contexts (i.e. “red block” refersto one particular block, regardless of its context).
Revised: This suggests that, although the model certainly has the capacity to memorize 23 combinations, and although it is not trained to perform compositional reasoning, it learns to bind symbols to percepts in a way that generalizes across contexts (i.e. “red block” refers to one particular block, independent of its context).

Original: These result are promising but limited to simulatedlanguage; accordingly, in Section 5.2 we evaluate our results with real language data.
Revised: These results are promising but limited to simulated language; accordingly, in Section 5.2 we evaluate our model with real language data.

Original: 5.2 Naturalistic Language Commands with Complex Scenes
Revised: 5.2 Naturalistic Language Commands with Complex Scenes

Original: In this experiment we model the naturalistic data described in Section 4.4, where state representations are coupled instead with diverse crowdsourced commands, in contrast to the template generated commands in Section 5.1.
Revised: In this experiment we model the naturalistic data described in Section 4.4, where state representations are coupled instead with diverse crowdsourced commands, in contrast to the template generated commands in Section 5.1.

Original: We train models on state representations (cf.
Revised: We train models on state representations (cf.

Original: Section 4.4) as well asbirds-eye color and depth images.
Revised: Section 4.4) as well as birds-eye color and depth images.

Original: This highlights the ﬂexibility of our model: we are able to trainfrom images and from state representations with no modiﬁcations to the architecture.
Revised: This highlights the ﬂexibility of our model: we are able to train from images and from state representations with no modiﬁcations to the architecture.

Original: Large pre-trained encoders have been shown to encode useful linguistic features at a number oflevels of analysis [47, 48, 49]; accordingly, we make use of bert-base-cased [50] as an inputfeature, contrasting it with GloVe [51] word embeddings.
Revised: Large pre-trained encoders have been shown to encode useful linguistic features at a number of levels of analysis [47, 48, 49]; accordingly, we make use of bert-base-cased [50] as an input feature, contrasting it with GloVe [51] word embeddings.

Original: Tuning and metrics are as in Sec. 5.1.
Revised: Tuning and metrics are as in Sec. 5.1.

Original: [Table]
Revised: [Table]

Original: Table 4 shows the VGP metrics for modelstrained from image-like representations ofthe state (described in Section 4.4) as wellas from images.
Revised: Table 4 shows the VGP metrics for models trained from image-like representations of the state (described in Section 4.4) as well as from images.

Original: For brevity, only thebest model for each embedding type is reported, with full results in Appendix C.5.
Revised: For brevity, only the best model for each embedding type is reported, with full results in Appendix C.5.

Original: On state representations, the Transformerbased models show a clear advantage overthe baseline, with IES being substantiallylower for both Transformer models.
Revised: On state representations, the Transformerbased models show a clear advantage over the baseline, with IES being substantially lower for both Transformer models.

Original: Whenusing GloVe embeddings with state representation input, 2 × 2 patches resulted in the best performance; a patch size of 4 yielded the bestresults for BERT embeddings.
Revised: When using GloVe embeddings with state representation input, 2 × 2 patches resulted in the best performance; a patch size of 4 yielded the best results for BERT embeddings.

Original: Overall, while the BERT-based model has slightly higher block accuracy, the IES are equal between the two models when trained on state representations.
Revised: Overall, while the BERT-based model has slightly higher block accuracy, the IES are equal between the two models when trained on state representations.

Original: While thesemetrics measure per-step performance, these results suggest that our masking model can be appliedto real as well as simulated language.
Revised: While these metrics measure per-step performance, these results suggest that our masking model can be applied to real as well as simulated language.

Original: In Table 4, we also see that the performance shifts substantially when changing the input from stateto images.
Revised: In Table 4, we also see that the performance shifts substantially when changing the input from state to images.

Original: While the baseline improves slightly in IES, the Transformers’ performance decreasesdramatically.
Revised: While the baseline improves slightly in IES, the Transformers’ performance decreases dramatically.

Original: These results stand in contrast to those in Section 5.1, where a Transformer trainedon images unequivocally outperformed the baseline.
Revised: These results stand in contrast to those in Section 5.1, where a Transformer trained on images unequivocally outperformed the baseline.

Original: Nevertheless, the best Transformer model stilloutperforms the baseline.
Revised: Nevertheless, the best Transformer model still outperforms the baseline.

Original: We speculate that the baseline models may perform better here becausethe architecture was developed for image segmentation problems.
Revised: We speculate that the baseline models may perform better here because the architecture was developed for image segmentation problems.

Original: Interestingly, while a BERT-based
Revised: Interestingly, while a BERT-based

Original: Transformer performed well when trained from state, the same models’ performance here is greatlydiminished.
Revised: Transformer performed well when trained from state, the same models’ performance here is greatly diminished.

Original: Improving the Transformer’s real-world performance is a direction for future work.
Revised: Improving the Transformer’s real-world performance is a direction for future work.

Original: Data Translation and Filtering
Revised: Data Translation and Filtering

Original: In Section 5.1, we observed that high performance in a static evaluation against a dataset did not entail equally high performance when using a realistic robot formulti-step tasks in a simulated environment.
Revised: In Section 5.1, we observed that high performance in a static evaluation against a dataset did not entail equally high performance when using a realistic robot for multi-step tasks in a simulated environment.

Original: In its current form, the blocks dataset can only be evaluated using static metrics.
Revised: In its current form, the blocks dataset can only be evaluated using static metrics.

Original: Furthermore, the images provided in the dataset are taken from a ﬁxed 45degree projection, and our method assumes access to birds-eye images.
Revised: Furthermore, the images provided in the dataset are taken from a ﬁxeddegree projection, and our method assumes access to birds-eye images.

Original: To obtain birds-eye imagesof the environment and to facilitate further research into combining rich, naturalistic language withaction in manipulation tasks, we translate the dataset into the same environment used in Section 5.1,and ﬁlter the dataset to only physically feasible examples; this process is described in Section 4.4.
Revised: To obtain birds-eye images of the environment and to facilitate further research into combining rich, naturalistic language with action in manipulation tasks, we translate the dataset into the same environment used in Section 5.1, and ﬁlter the dataset to only physically feasible examples; this process is described in Section 4.4.

Original: indicates signiﬁcant improvement over UNet as per the Wilcoxon ranked-sign test.
Revised: indicates signiﬁcant improvement over ranked

Original: [Table]
Revised: [Table]

Original: Table 5 reports the VGP metrics on the thephysically feasible subset of the data for the
Revised: Table 5 reports the VGP metrics on the the physically feasible subset of the data for the

Original: Transformer models.
Revised: Transformer models.

Original: Firstly, we see thatperformance decreases across most metrics.
Revised: Firstly, we see that performance decreases across most metrics.

Original: However, for UNet, IES improves onthe physically feasible subset.
Revised: However, for UNet, IES improves on the physically feasible subset.

Original: In contrast,the Transformer models decrease in performance across both metrics.
Revised: In contrast, the Transformer models decrease in performance across both metrics.

Original: Nevertheless,the Transformer model still performs on parwith the baseline on IES, and substantially better on block accuracy.
Revised: Nevertheless, the Transformer model still performs on par with the baseline on IES, and substantially better on block accuracy.

Original: These results suggest that thephysically feasible subset is more challenging to model, even when evaluating the models statically.
Revised: These results suggest that the physically feasible subset is more challenging to model, even when evaluating the models statically.

Original: In other words, even when the evaluation and models are physically unconstrained, modeling onlyphysically feasible scenes is harder than modeling all scenes.
Revised: In other words, even when the evaluation and models are physically unconstrained, modeling only physically feasible scenes may be harder than modeling all scenes.

Original: This result highlights the difﬁculty ofdeveloping embodied agents and the importance of testing them in realistic environments.
Revised: This result highlights the difﬁculty of developing embodied agents and the importance of testing them in realistic environments.

Original:  Conclusion
Revised:  Conclusion

Original: We have presented a general approach to multi-step rearrangement and manipulation tasks that factorizes each step into learning how to act and learning where to act.
Revised: We have presented a general approach to multi-step rearrangement and manipulation tasks that factorizes each step into learning how to act and learning where to act.

Original: To decide where to act, wehave presented a Transformer-based model for converting state representations and natural languageinstructions to masks over an action space that learns with remarkably few examples.
Revised: To decide where to act, we have presented a Transformer-based model for converting state representations and natural language instructions to masks over an action space that learns with remarkably few examples.

Original: We haveshown the ability of our model to act in a realistic simulated environment for two multi-step blockrearrangement tasks, and naturally learn a compositional mapping from text to actions in a perceptual space.
Revised: We have shown the ability of our model to act in a realistic simulated environment for two multi-step block rearrangement tasks, and naturally learn a compositional mapping from text to actions in a perceptual space.

Original: We have demonstrated  our model performs well under both these realistic physicalassumptions and with real and complex natural language via a dataset of crowdsourced instructions.  
Revised: We have demonstrated that our model performs well under both these realistic physical assumptions and with real and complex natural language via a dataset of crowdsourced instructions.  Acknowledgements

Original: A.2 Hyperparameters
Revised: We thank the anonymous reviewers and the area chair for their input and engagement. We would also like to thank Benjamin Van Durme, Tae Soo Kim and Jonathan Jones for their helpful feedback. This material is based upon work supported by NSF Award #1763705 and by the Ofﬁce of Naval Research under grant N00014-19-1-2076.

Revised: Elias Stengel-Eskin is supported by an NSF GRFP.

Revised: A.2 Hyperparameters

Original: The number of shared and combined layers is treated as a hyperparameter n ∈ [0 , 2 , 4 , 6] , aswell as the number of warmup steps w ∈ [100 , 400 , 1000 , 4000] , the dropout probability p ∈[0 .
Revised: The number of shared and combined layers is treated as a hyperparameter n ∈ [0 , 2 , 4 , 6] , as well as the number of warmup steps w ∈ [100 , 400 , 1000 , 4000] , the dropout probability p ∈ [0 . 25 , 0 .

Original: 40] , and the loss weight ratio γ ∈ [0 . 01 , 0 . 1 , 0 . 2] .
Revised: 40] , and the loss weight ratio γ ∈ [0 . 01 , 0 . 1 , 0 . 2] .

Revised: Each model has a hidden dimension of 256 and a feed-forward dimension of 512.

Original: Due to the larger number of hyperparameters, we sample 70 random parameter conﬁgurations.
Revised: Due to the larger number of hyperparameters, we sample 70 random parameter conﬁgurations.

Revised: stack the blue block on the red block

Revised: [Figure]

Original: A.3 Q-Learning
Revised: A.3 Q-Learning

Original: Our model relies on a learned function from states and actions to expected rewards, Q ( s t , a ) .
Revised: Our model relies on a learned function from states and actions to expected rewards, Q ( s t , a ) .

Original: Thisfunction is learned in the context of a Markov Decision Process ( S, A, P, R, γ ) composed of a setof states S , a set of actions A , a transition function P : S × S ×
Revised: This function is learned in the context of a Markov Decision Process ( S, A, P, R, γ ) composed of a set of states S , a set of actions A , a transition function P : S × S ×

Original: A → R , a reward function
Revised: A → R , a reward function

Original: R : S ×
Revised: R : S ×

Original: A → R , and a discount factor γ , where 0 ≤ γ ≤ 1 .
Revised: A → R , and a discount factor γ , where 0 ≤ γ ≤ 1 .

Original: At time t , an agent observes state S t andchooses action a t =
Revised: At time t , an agent observes state S t and chooses action a t =

Original: π ( s t ) , where π is a policy.
Revised: π ( s t ) , where π is a policy.

Original: The function Q , approximating the expected rewardfor each state-action pair, can be used to extract a deterministic policy π that maximizes the expectedreward: π ( s t ) =
Revised: The function Q , approximating the expected reward for each state-action pair, can be used to extract a deterministic policy π that maximizes the expected reward: π ( s t ) =

Original: argmax Q ( s t , a ) , ∀ a ∈ A .
Revised: argmax Q ( s t , a ) , ∀ a ∈ A .

Original: We learn Q to maximize the reward R over time byminimizing | Q ( s t , a t ) − y t | , where y t =
Revised: We learn Q to maximize the reward R over time by minimizing | Q ( s t , a t ) − y t | , where y t =

Original: R ( s t +1 , a t ) + γQ ( s t +1 , π ( s t +1 )) .
Revised: R ( s t +1 , a t ) + γQ ( s t +1 , π ( s t +1 )) .

Original: In a multi-step task, thedeﬁnition of R has a major impact on the learning efﬁciency of the agent.
Revised: In a multi-step task, the deﬁnition of R has a major impact on the learning efﬁciency of the agent.

Original: Speciﬁcally, relying on asingle reward at the end of a task yields a very sparse reward signal and results in inefﬁcient learning.
Revised: Speciﬁcally, relying on a single reward at the end of a task yields a very sparse reward signal and results in inefﬁcient learning.

Original: In the multi-step tasks considered here (stacking and row-making) there are intermediate subtasksthat naturally lend themselves to reward shaping, i.e. providing smaller intermediate rewards tothe agent when the chosen action reﬂects an incremental progression towards the goal state.
Revised: In the multi-step tasks considered here (stacking and row-making) there are intermediate subtasks that naturally lend themselves to reward shaping, i.e. providing smaller intermediate rewards to the agent when the chosen action reﬂects an incremental progression towards the goal state.

Original: The
Revised: The

Original: SPOT-Q algorithm, introduced by Hundt et al.
Revised: SPOT-Q algorithm, introduced by Hundt et al.

Original: [9], uses intermediate rewards to shape the learningof Q ( s t , a ) for row-making and stacking.
Revised: [9], uses intermediate rewards to shape the learning of Q ( s t , a ) for row-making and stacking.

Original: We use SPOT-Q to learn the function Q ( s t , a ) .
Revised: We use SPOT-Q to learn the function Q ( s t , a ) .

Original: Given apixel-wise n × n state space of an image and two actions (grasp and place), this function can beexpressed as a tensor Q ∈ R n × n × 2 .
Revised: Given a pixel-wise n × n state space of an image and two actions (grasp and place), this function can be expressed as a tensor Q ∈ R n × n × 2 .

Original: B Experiment 1B.1 Combination Heuristics
Revised: B ExperimentB.1 Combination Heuristics

Original: In the case of stacking, following Hundt et al.
Revised: In the case of stacking, following Hundt et al.

Original: [9], we intersect each mask with a common-sensemask that assigns a value of 0 where there are no blocks (i.e. an empty space cannot be the locationof a grasp or a place).
Revised: [9], we intersect each mask with a common-sense mask that assigns a value of 0 where there are no blocks (i.e. an empty space cannot be the location of a grasp or a place).

Original: For row-making, we allow placing in empty locations, but intersect the graspmask with a common-sense mask.
Revised: For row-making, we allow placing in empty locations, but intersect the grasp mask with a common-sense mask.

Original: For place and grasp actions in stacking, and grasp actions inrow-making, we then ﬁnd the maximum location s ∗ in mask M : s ∗ = argmax M .
Revised: For place and grasp actions in stacking, and grasp actions in row-making, we then ﬁnd the maximum location s ∗ in mask M : s ∗ = argmax M .

Original: Because thereconstruction function copies a patch value across all of its pixels, for a patch size of p there willbe p 2 pixels with the value of s ∗ .
Revised: Because the reconstruction function copies a patch value across all of its pixels, for a patch size of p there will be p 2 pixels with the value of s ∗ .

Original: We intersect this patch with the common-sense mask to ﬁnd thesingle block with the highest value in M .
Revised: We intersect this patch with the common-sense mask to ﬁnd the single block with the highest value in M .

Original: For place actions in row-making, we simply threshold Mto obtain a set of valid patches to intersect with the Q values.
Revised: For place actions in row-making, we simply threshold M to obtain a set of valid patches to intersect with the Q values.

Original: Both processes produce binary masks.
Revised: Both processes produce binary masks.

Original: B.2 Simulation
Revised: B.2 Simulation

Original: We use the CopelliaSim simulator [52] for experiments involving a simulated robot.
Revised: We use the CopelliaSim simulator [52] for experiments involving a simulated robot.

Original: The simulatedagent collects observations via a ﬁxed RGB-D camera, whose images are projected to a birds-eyeview, as shown in Hundt et al.

Original: The agent operates over a discretized spatial and angular actionspace, and movement to a particular location and angle is performed by an inverse kinematics solverbuilt into the simulator.

Original: Images All real images were collected via a UR5 robot, Robotiq 2f-85 2-ﬁnger gripper, and Primesense Carmine RGB-D camera.
Revised: Images All real images were collected via a UR5 robot, Robotiq 2f-85 2-ﬁnger gripper, and Primesense Carmine RGB-D camera.

Original: All of these conditions are different from the simulator, except the
Revised: All of these conditions are different from the simulator, except the

Original: UR5.
Revised: UR5.

Original: The images were collected under variedlighting conditions.
Revised: The images were collected under varied lighting conditions.

Original: The data is saved in the sameﬁle format as the experiments in Tables 1 and 2.with a birds-eye view, 224 × 224 RBG and depthmap images, etc.
Revised: The data is saved in the same ﬁle format as the experiments in Tables 1 and 2. with a birds-eye view, 224 × 224 RBG and depthmap images, etc.

Original: Early Termination
Revised: Early Termination

Original: The step-by-step nature ofthe instructions means that the agent is sometimesunable to reverse course: for example, if the ﬁrstinstruction were “stack the green block on the blueblock” but the agent mistakenly grasped the redblock and placed it on the green block, the greenblock would become unreachable.
Revised: The step-by-step nature of the instructions means that the agent is sometimes unable to reverse course: for example, if the ﬁrst instruction were “stack the green block on the blue block” but the agent mistakenly grasped the red block and placed it on the green block, the green block would become unreachable.

Original: Since neither the Q-value model nor the language understanding model were trained on unstacking tasks, there would be no way to reverse course here withoutadding an external observer.
Revised: Since neither the Q-value model nor the language understanding model were trained on unstacking tasks, there would be no way to reverse course here without adding an external observer.

Original: Thus, in these cases, we terminate the trial.
Revised: Thus, in these cases, we terminate the trial.

Original: Similarly, if the stack istoppled, we terminate the trial, as without an external observer we cannot determine which step ofthe process we have regressed to, since the  toppling ofpartial stacks is not unusual.
Revised: Similarly, if the stack is toppled, we terminate the trial, as without an external observer we cannot determine which step of the process we have regressed to, as the partial toppling of stacks is not unusual.

Original: Since blocks aretypically not occluded in row-making, we do not need these heuristics, and the only way for a trialto fail is by timing out after 30 actions.
Revised: Because blocks are typically not occluded in row-making, we do not need these heuristics, and the only way for a trial to fail is by timing out after 30 actions.

Original: Simulator For experiments involving a simulated robot, the CopelliaSim simulator [52] was used.
Revised: Simulator For experiments involving a simulated robot, the CopelliaSim simulator [52] was used.

Original: The simulated agent collects observations via a ﬁxed RGB-D camera, whose images are projectto a birds-eye view.
Revised: The simulated agent collects observations via a ﬁxed RGB-D camera, whose images are project to a birds-eye view.

Original: The agent operates over a discretized spatial and angular action space, andmovement to a particular location and angle is performed by an inverse kinematics solver built intothe simulator.
Revised: The agent operates over a discretized spatial and angular action space, and movement to a particular location and angle is performed by an inverse kinematics solver built into the simulator.

Original: B.3 Error analysis
Revised: B.3 Error analysis

Original: As mentioned in §5.1, stacking tasks may terminate early for a number of reasons, including irreversible actions that make a successful stack impossible.
Revised: As mentioned in §5.1, stacking tasks may terminate early for a number of reasons, including irreversible actions that make a successful stack impossible.

Original: We conduct an error analysis to determinewhat percentage of the failures are due to errors in the perception/reasoning component (i.e. themasking) and what percentage are due to physical failures of the grasping arm.
Revised: We conduct an error analysis to determine what percentage of the failures are due to errors in the perception/reasoning component (i.e. the masking) and what percentage are due to physical failures of the grasping arm.

Original: We ﬁnd that 61.97%of the failures could be attributed to timing out after 30 actions.
Revised: We ﬁnd that 61.97% of the failures could be attributed to timing out after 30 actions.

Original: Qualitatively, we found that this

Original: [Table] [Table] [Table] often happens when a block tumbles outside of the work area, rendering it impossible to grasp.
Revised: Qualitatively, we found that this often happens when a block tumbles outside of the work area, rendering it impossible to grasp.

Original: Another 7.04% can be attributed to toppled stacks, where a stack height of 2 or 3 was reached before aplace action destroyed the whole stack, ending the trial.
Revised: Another 7.04% can be attributed to toppled stacks, where a stack height of 2 or 3 was reached before a place action destroyed the whole stack, ending the trial.

Original: Finally, 30.99% were due to incorrect blockorderings which occluded crucial blocks; this is due to the 21.42% of grasp actions that picked upthe incorrect block.
Revised: Finally, 30.99% were due to incorrect block orderings which occluded crucial blocks; this is due to the 21.42% of grasp actions that picked up the incorrect block.

Original: B.4 Proof-of-Concept
Revised: B.4 Proof-of-Concept

Original: Due to restrictions induced by the ongoing COVID-19 crisis, we were unable to run sufﬁcient trials to include quantitative real results.
Revised: Due to restrictions induced by the ongoing COVID-19 crisis, we were unable to run sufﬁcient trials to include quantitative real results.

Original: Nevertheless, we include a proof-of-concept demonstrationvideo in which we run the Transformer-based masking model with a real robot arm to successfullycomplete a real stack in the bottom to top order green, yellow, blue, red.
Revised: Nevertheless, we include a proof-of-concept demonstration video in which we run the Transformer-based masking model with a real robot arm to successfully complete a real stack in the bottom to top order green, yellow, blue, red.

Original: C Experiment
Revised: C ExperimentC.1 Data Translation and Preprocessing

Original: C.1 Data Translation and Preprocessing

Original: Note that in the original dataset, 4.82% of the examples have multiple blocks moving in a singleframe.
Revised: Note that in the original dataset, 4.82% of the examples have multiple blocks moving in a single frame.

Original: As this breaks the assumptions made in § 1, we ignore these examples both while trainingand evaluating.
Revised: As this breaks the assumptions made in § 1, we ignore these examples both while training and evaluating.

Original: For training, we convert the 64 × 64 state image grids into binary masks, where all elements are zeroexcept the pixels corresponding to the block which is moved.
Revised: For training, we convert the 64 × 64 state image grids into binary masks, where all elements are zero except the pixels corresponding to the block which is moved.

Original: C.2 Data Filtering
Revised: C.2 Data Filtering

Original: We ﬁlter out physically infeasible examples, which are typically due to unstable structures topplingor blocks being placed in overlapping areas.
Revised: We ﬁlter out physically infeasible examples, which are typically due to unstable structures toppling or blocks being placed in overlapping areas.

Original: Because the images captured from the simulated environment are of a lower resolution than the originals, many of these logos are difﬁcult to read in thesimulated images; to aid with block discrimination, we assign a color to each block in addition to itslogo.
Revised: Because the images captured from the simulated environment are of a lower resolution than the originals, many of these logos are difﬁcult to read in the simulated images; to aid with block discrimination, we assign a color to each block in addition to its logo.

Original: Table 6 reports the same statistics given in Bisk et al.
Revised: Table 6 reports the same statistics given in Bisk et al.

Original: [10] on the ﬁltered subset of the data. 
Revised: [10] on the ﬁltered subset of the data. [Table] [Table]

Original: C.3 Dataset Ambiguity
Revised: Dataset Ambiguity

Original: A remaining limitation of the dataset is that some descriptions are ambiguous and not reliably actionable, since annotators did not attempt to execute the described actions.
Revised: A remaining limitation of the dataset is that some descriptions are ambiguous and not reliably actionable, since annotators did not attempt to execute the described actions.

Original: Some of the naturallanguage descriptions are ambiguous, as depicted in Fig.
Revised: Some of the natural language descriptions are ambiguous, as depicted in Fig.

Original: C.4 Qualitative Error Analysis
Revised: C.4 Qualitative Error Analysis

Original: Based on our results in Tables 4, we qualitatively examine some of the errors made by the
Revised: Based on our results in Tables 4, we qualitatively examine some of the errors made by the

Original: Transformer-based model, where we observe several patterns.
Revised: Transformer-based model, where we observe several patterns.

Original: Looking at the validation exampleswith the highest IES, we observe that they often correspond to instances where the source block wasmis-identiﬁed, leading the wrong block to be moved, and yielding a large IES.
Revised: Looking at the validation examples with the highest IES, we observe that they often correspond to instances where the source block was mis-identiﬁed, leading the wrong block to be moved, and yielding a large IES.

Original: More interestingly,in the cases where the correct block was moved, the instruction often contains higher-order conceptsas well as linguistic complexities such as ellipsis.
Revised: More interestingly, in the cases where the correct block was moved, the instruction often contains higher-order concepts as well as linguistic complexities such as ellipsis.

Original: For example, in Figure 6 there is a reference to a“disconnected square”, which is a rare, higher-order geometric concept.
Revised: For example, in Figure 6 there is a reference to a “disconnected square”, which is a rare, higher-order geometric concept.

Original: In addition, the subsequentclause lacks an explicit reference back to the square; the annotator chose to leave this referenceas implicit, given that the square is raised to a salient position by the previous clause.
Revised: In addition, the subsequent clause lacks an explicit reference back to the square; the annotator chose to leave this reference as implicit, given that the square is raised to a salient position by the previous clause.

Original: This type ofnoun-phrase ellipsis is common in natural language [53], and reﬂects the type of advanced pragmaticreasoning required to handle natural language.
Revised: This type of noun-phrase ellipsis is common in natural language [53], and reﬂects the type of advanced pragmatic reasoning required to handle natural language.

Original: Other examples of a higher-order concept instruction that the model performs poorly on are: “Takethe Mercedes Benz block on the Burger King block without hiding the right top edge” (hiding),“targt [sic] goes 1/2 under Adidas with the right side hanging off” (hanging), and “take the Stella
Revised: Other examples of a higher-order concept instruction that the model performs poorly on are: “Take the Mercedes Benz block on the Burger King block without hiding the right top edge” (hiding),

Original: Artois block and place it on top of the Nvidia block, lined up perfectly” (perfectly). The model alsostruggles with long coreference chains, even when the anaphora are explicit, e.g. “Place the blockthat is to the right of the Starbucks block and make it the highest block on the board by placing it onthe Mercedes block.
Revised: “There is a disconnected “square” at the bottom.

Original: It should be in line with the bottom block.” [Table]
Revised: Place Shell on top of the lower left corner.”

Original: Transformer GloVe
Revised: “Add Shell as the second block in the four-block row.”

Original: [Table]
Revised: [Table]

Original: Transformer
Revised: Transformer

Original: GloVe

Original: [Table]
Revised: [Table]

Original: Transformer
Revised: Transformer

Original: [Table]
Revised: [Table]

Original: Transformer
Revised: Transformer

Original: [Table]
Revised: [Table]

Original: Transformer
Revised: Transformer

Original: [Table]
Revised: [Table]

Original: Transformer
Revised: Transformer

Original: [Table]
Revised: [Table]

Original: Transformer
Revised: Transformer

Original: [Table]
Revised: [Table]

Original: Transformer
Revised: Transformer

Original: [Table]
Revised: [Table]

Original: Transformer
Revised: Transformer

Original: [Table]
Revised: [Table]

Original: Transformer
Revised: Transformer

Original: GloVe

Original: [Table]
Revised: [Table]

Original: Transformer
Revised: Transformer

Original: [Table]
Revised: [Table]

Original: Transformer
Revised: Transformer

Original: [Table]
Revised: [Table]

Original: Transformer
Revised: Transformer

Original: [Table]
Revised: [Table]

Original: Transformer
Revised: Transformer

Original: [Table]
Revised: [Table]

Original: Transformer
Revised: Transformer

Original: [Table]
Revised: [Table]

Original: Transformer
Revised: Transformer

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Yes

Original: Image 45.
Revised: Image 45.

Original: UNet
Revised: UNet

Original: GloVe
Revised: GloVe

Original: No
Revised: No

Original: Image 56.
Revised: Image 56.

Original: Transformer
Revised: Transformer

Original: GloVe
Revised: GloVe

Original: Yes
Revised: Yes

Original: State 88.
Revised: State 88.

Original: Transformer
Revised: Transformer

Original: GloVe
Revised: GloVe

Original: No
Revised: No

Original: State 90.
Revised: State 90.

Original: Transformer
Revised: Transformer

Original: GloVe
Revised: GloVe

Original: Yes
Revised: Yes

Original: State 89.
Revised: State 89.

Original: Transformer
Revised: Transformer

Original: GloVe
Revised: GloVe

Original: No
Revised: No

Original: State 85.
Revised: State 85.

Original: Transformer
Revised: Transformer

Original: BERT
Revised: BERT

Original: Yes
Revised: Yes

Original: State 86.
Revised: State 86.

Original: Transformer
Revised: Transformer

Original: BERT
Revised: BERT

Original: No
Revised: No

Original: State 88.
Revised: State 88.

Original: Transformer
Revised: Transformer

Original: BERT
Revised: BERT

Original: Yes
Revised: Yes

Original: State 88.
Revised: State 88.

Original: Transformer
Revised: Transformer

Original: BERT
Revised: BERT

Original: No
Revised: No

Original: State 72.
Revised: State 72.

Original: Transformer
Revised: Transformer

Original: GloVe
Revised: GloVe

Original: Yes
Revised: Yes

Original: Image 86.
Revised: Image 86.

Original: Transformer
Revised: Transformer

Original: GloVe
Revised: GloVe

Original: No
Revised: No

Original: Image 83.
Revised: Image 83.

Original: Transformer
Revised: Transformer

Original: GloVe
Revised: GloVe

Original: Yes
Revised: Yes

Original: Image 75.
Revised: Image 75.

Original: Transformer
Revised: Transformer

Original: GloVe
Revised: GloVe

Original: No
Revised: No

Original: Image 84.
Revised: Image 84.

Original: Transformer
Revised: Transformer

Original: BERT
Revised: BERT

Original: Yes
Revised: Yes

Original: Image 88.
Revised: Image 88.

Original: Transformer
Revised: Transformer

Original: BERT
Revised: BERT

Original: No
Revised: No

Original: Image 66.
Revised: Image 66.

Original: Transformer
Revised: Transformer

Original: BERT
Revised: BERT

Original: Yes
Revised: Yes

Original: Image 88.
Revised: Image 88.

Original: Transformer
Revised: Transformer

Original: BERT
Revised: BERT

Original: No
Revised: No

Original: Image 86.
Revised: Image 86.