Original: P ROGRAMMATICALLY G ROUNDED , C OMPOSITIONALLY G ENERALIZABLE R OBOTIC M ANIPULATION
Revised: P ROGRAMMATICALLY G ROUNDED , C OMPOSITIONALLY G ENERALIZABLE R OBOTIC M ANIPULATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Robots operating in the real world require both rich manipulation skills as well as the ability to semantically reason about when to apply those skills.
Revised: Robots operating in the real world require both rich manipulation skills as well as the ability to semantically reason about when to apply those skills.

Original: Towards this goal, recent works have integrated semantic representations from large-scale pretrained vision-language (VL) models into manipulation models, imparting them with more general reasoning capabilities.
Revised: Towards this goal, recent works have integrated semantic representations from large-scale pretrained vision-language (VL) models into manipulation models, imparting them with more general reasoning capabilities.

Original: However, we show that the conventional pretraining-finetuning pipeline for integrating such representations entangles the learning of domain-specific action information and domain-general visual information, leading to less data-efficient training and poor generalization to unseen objects and tasks.
Revised: However, we show that the conventional pretraining-finetuning pipeline for integrating such representations entangles the learning of domain-specific action information and domain-general visual information, leading to less data-efficient training and poor generalization to unseen objects and tasks.

Original: To this end, we propose PROGPORT , a modular approach to better leverage pretrained VL models by exploiting the syntactic and semantic structures of an input language instruction.
Revised: To this end, we propose PROGPORT , a modular approach to better leverage pretrained VL models by exploiting the syntactic and semantic structures of an input language instruction.

Original: Our framework uses a semantic parser to recover an executable program that is composed of functional modules grounded on vision and action across different modalities.
Revised: Our framework uses a semantic parser to recover an executable program that is composed of functional modules grounded on vision and action across different modalities.

Original: Each functional module is realized as a combination of deterministic computation and learnable neural networks.
Revised: Each functional module is realized as a combination of deterministic computation and learnable neural networks.

Original: The execution of the program produces parameters to general manipulation primitives for a robotic end-effector.
Revised: The execution of the program produces parameters to general manipulation primitives for a robotic end-effector.

Original: The entire modular network can be trained with end-toend imitation learning objectives.
Revised: The entire modular network can be trained with end-toend imitation learning objectives.

Original: Experiments show that our model successfully disentangles action and perception, translating to improved zero-shot and compositional generalization in a variety of manipulation behaviors.
Revised: Experiments show that our model successfully disentangles action and perception, translating to improved zero-shot and compositional generalization in a variety of manipulation behaviors.

Original: Project webpage at: https://progport.github.io
Revised: Project webpage at: https://progport.github.io

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Robotic manipulation models that map directly from raw pixels to actions are capable of learning diverse and complex behaviors through imitation.
Revised: Robotic manipulation models that map directly from raw pixels to actions are capable of learning diverse and complex behaviors through imitation.

Original: To enable more abstract goal specification, many such models also take as input natural language instructions.
Revised: To enable more abstract goal specification, many such models also take as input natural language instructions.

Original: However, this vision-language manipulation setting introduces a new problem: the agent must jointly learn to ground language tokens to its perceptual inputs, and correspond this grounded understanding with the desired actions.
Revised: However, this vision-language manipulation setting introduces a new problem: the agent must jointly learn to ground language tokens to its perceptual inputs, and correspond this grounded understanding with the desired actions.

Original: Moreover, to fully leverage the flexibility of language, the agent must handle novel vocabulary and compositions not explicitly seen during training, but specified at test time (Fig. 1).
Revised: Moreover, to fully leverage the flexibility of language, the agent must handle novel vocabulary and compositions not explicitly seen during training, but specified at test time (Fig. 1).

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: To these ends, many recent works have relied on large pretrained vision-language (VL) models such as CLIP (Radford et al., 2021) to tackle both grounding and zero-shot generalization.
Revised: To these ends, many recent works have relied on large pretrained vision-language (VL) models such as CLIP (Radford et al., 2021) to tackle both grounding and zero-shot generalization.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2a, these works generally treat the pretrained VL model as a semantic prior, for example, by partially initializing the weights of image and text encoders with a pretrained VL model (Ahn et al., 2022; Khandelwal et al., 2022; Shridhar et al., 2022a).
Revised: 2a, these works generally treat the pretrained VL model as a semantic prior, for example, by partially initializing the weights of image and text encoders with a pretrained VL model (Ahn et al., 2022; Khandelwal et al., 2022; Shridhar et al., 2022a).

Original: These models are then updated via imitating expert demonstrations.
Revised: These models are then updated via imitating expert demonstrations.

Original: However, this training scheme entangles the learning of domainspecific control policies and domain-independent vision-language grounding.
Revised: However, this training scheme entangles the learning of domainspecific control policies and domain-independent vision-language grounding.

Original: Specifically, we find that these VL-enabled agents overfit to their task-specific data, leveraging shortcuts during training to successfully optimize their imitation learning (IL) objective, without learning a generalizable grounding of language to vision.
Revised: Specifically, we find that these VL-enabled agents overfit to their task-specific data, leveraging shortcuts during training to successfully optimize their imitation learning (IL) objective, without learning a generalizable grounding of language to vision.

Original: This phenomenon is particularly apparent when the agent is given language goals with unknown concepts and objects, or novel compositions of known concepts and objects.
Revised: This phenomenon is particularly apparent when the agent is given language goals with unknown concepts and objects, or novel compositions of known concepts and objects.

Original: For example, an agent that overfits to packing shapes into a box can fail to generalize to other manipulation behaviors (e.g. pushing) involving the same shapes.
Revised: For example, an agent that overfits to packing shapes into a box can fail to generalize to other manipulation behaviors (e.g. pushing) involving the same shapes.

Original: In this work, we introduce PROGPORT , a program-based modular approach that enables more faithful vision-language grounding when incorporating pretrained VL models for robotic manipulation.
Revised: In this work, we introduce PROGPORT , a program-based modular approach that enables more faithful vision-language grounding when incorporating pretrained VL models for robotic manipulation.

Original: Given the natural language instruction, we first use a Combinatory Categorial Grammar (CCG)
Revised: Given the natural language instruction, we first use a Combinatory Categorial Grammar (CCG)

Original: (Steedman, 1996) to parse the sentence into a “manipulation program,” based on a compact but general domainspecific language (DSL).
Revised: (Steedman, 1996) to parse the sentence into a “manipulation program,” based on a compact but general domainspecific language (DSL).

Original: The program consists of functional modules that are either visual grounding modules (e.g., locate all objects of a given category) or action policies (e.g., produce a control parameter).
Revised: The program consists of functional modules that are either visual grounding modules (e.g., locate all objects of a given category) or action policies (e.g., produce a control parameter).

Original: This enables us to directly leverage a pretrained VL model to ground singular, independent categories or attribute descriptors to their corresponding pixels, and thus disentangles the learning of visual grounding and action policies (Fig. 2b).
Revised: This enables us to directly leverage a pretrained VL model to ground singular, independent categories or attribute descriptors to their corresponding pixels, and thus disentangles the learning of visual grounding and action policies (Fig. 2b).

Original: Our programmatically structured, modular design enables PROGPORT to successfully learn more performant imitation policies with fewer data across 10 diverse tasks in a tabletop manipulation environment.
Revised: Our programmatically structured, modular design enables PROGPORT to successfully learn more performant imitation policies with fewer data across 10 diverse tasks in a tabletop manipulation environment.

Original: We show that after training on a subset of objects, visual properties, and actions, our model can zero-shot generalize to completely different subsets and reason over novel compositions of language descriptors, without further finetuning (Fig. 1).
Revised: We show that after training on a subset of objects, visual properties, and actions, our model can zero-shot generalize to completely different subsets and reason over novel compositions of language descriptors, without further finetuning (Fig. 1).

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Problem formulation .
Revised: Problem formulation .

Original: We take a manipulation primitive-based approach to robot learning.
Revised: We take a manipulation primitive-based approach to robot learning.

Original: At a high level, we assume our robot is given a set of primitives P (e.g., pick, place, and push).
Revised: At a high level, we assume our robot is given a set of primitives P (e.g., pick, place, and push).

Original: Such primitives are usually defined by interaction modes between the robot and objects, parameterized by continuous parameters, and their composition spans a wide range of tasks.
Revised: Such primitives are usually defined by interaction modes between the robot and objects, parameterized by continuous parameters, and their composition spans a wide range of tasks.

Original: Throughout the paper, we will be using the simple primitive set P = { pick , place } in a tabletop environment as the example, and extend our framework to other primitives such as pushing in the experiment section.
Revised: Throughout the paper, we will be using the simple primitive set P = { pick , place } in a tabletop environment as the example, and extend our framework to other primitives such as pushing in the experiment section.

Original: Therefore, our goal is to learn a policy π mapping an input observation o t at time t to an action a t .
Revised: Therefore, our goal is to learn a policy π mapping an input observation o t at time t to an action a t .

Original: Each a t is a tuple of two control parameters (T pick , T place ) , which, in the pick-and-place setting, parameterizes the poses for the robot end effector when picking and placing objects, respectively.
Revised: Each a t is a tuple of two control parameters (T pick , T place ) , which, in the pick-and-place setting, parameterizes the poses for the robot end effector when picking and placing objects, respectively.

Original: This formulation also generalizes to other primitives such as pushing, in which case the control parameters are the pre and post-push locations.
Revised: This formulation also generalizes to other primitives such as pushing, in which case the control parameters are the pre and post-push locations.

Original: More specifically, in a language-guided tabletop manipulation task, our observation is given by o t = ( x t , l t ) .
Revised: More specifically, in a language-guided tabletop manipulation task, our observation is given by o t = ( x t , l t ) .

Original: Here, x t ∈ R H
Revised: Here, x t ∈ R H

Original: × W × 4 is the top-down orthographic RGB-D projection of the 3D visual scene, while l t is a natural language description of the desired goal.
Revised: × W × 4 is the top-down orthographic RGB-D projection of the 3D visual scene, while l t is a natural language description of the desired goal.

Original: The control parameters T pick , T place both lie in SE ( 2 ) .
Revised: The control parameters T pick , T place both lie in SE ( 2 ) .

Original: To learn such a policy, we assume access to a set of expert demonstrations D
Revised: To learn such a policy, we assume access to a set of expert demonstrations D

Original: = { d 1 ,d 2 ,...,d n } , each consisting of discrete time-indexed scene-language-action tuples so that d i = {( x 1 , l 1 , a 1 ) , ( x 2 , l 2 , a 2 ) ,..., ( x j , l j , a j )} .
Revised: = { d 1 ,d 2 ,...,d n } , each consisting of discrete time-indexed scene-language-action tuples so that d i = {( x 1 , l 1 , a 1 ) , ( x 2 , l 2 , a 2 ) ,..., ( x j , l j , a j )} .

Original: The policy π can be trained with an imitation learning objective by randomly sampling a particular tuple at time t from any demonstration.
Revised: The policy π can be trained with an imitation learning objective by randomly sampling a particular tuple at time t from any demonstration.

Original: Vision-language manipulation .
Revised: Vision-language manipulation .

Original: Akin to Transporter Networks (Zeng et al., 2020), our policy π is composed of two functions, Q pick and Q place , outputting dense pixelwise action values over R H × W , and operating in sequence.
Revised: Akin to Transporter Networks (Zeng et al., 2020), our policy π is composed of two functions, Q pick and Q place , outputting dense pixelwise action values over R H × W , and operating in sequence.

Original: Q pick first decides a particular pixel location T pick in the top-down visual field to pick the target object; Q place then conditions on the pick location to decide a particular place pose T place for the object:
Revised: Q pick first decides a particular pixel location T pick in the top-down visual field to pick the target object; Q place then conditions on the pick location to decide a particular place pose T place for the object:

Original: [Equation]
Revised: [Equation]

Original: where poses representing the manipulation end effector primitives are apriori discretized into the set { τ } , composed of discrete pixel locations and a discrete set of 2D rotation angles.
Revised: where poses representing the manipulation end effector primitives are apriori discretized into the set { τ } , composed of discrete pixel locations and a discrete set of 2D rotation angles.

Original: Recently,
Revised: Recently,

Original: CLIPort (Shridhar et al., 2022a) proposed to integrate pretrained vision-language (VL) models into such frameworks.
Revised: CLIPort (Shridhar et al., 2022a) proposed to integrate pretrained vision-language (VL) models into such frameworks.

Original: Shown in Fig.
Revised: Shown in Fig.

Original: 2a, in CLIPort, both Q pick and Q place are based on two-stream fullyconnected network (FCN) architectures.
Revised: 2a, in CLIPort, both Q pick and Q place are based on two-stream fullyconnected network (FCN) architectures.

Original: The encoders of the FCNs are initialized from pretrained weights of CLIP (Radford et al., 2021), while the control parameter decoders are randomly initialized and optimized.
Revised: The encoders of the FCNs are initialized from pretrained weights of CLIP (Radford et al., 2021), while the control parameter decoders are randomly initialized and optimized.

Original: Similar paradigms have also been applied to other manipulation environments such as
Revised: Similar paradigms have also been applied to other manipulation environments such as

Original: Khandelwal et al.
Revised: Khandelwal et al.

Original: and Ahn et al. (2022).
Revised: and Ahn et al. (2022).

Original: While the pretraining-finetuning framework can leverage features from the pretrained CLIP encoders to inform the model on a wide array of vision-language semantics, its monolithic structure entangles the learning of vision-language grounding and action policies.
Revised: While the pretraining-finetuning framework can leverage features from the pretrained CLIP encoders to inform the model on a wide array of vision-language semantics, its monolithic structure entangles the learning of vision-language grounding and action policies.

Original: This has two drawbacks: poor data efficiency and overfitting.
Revised: This has two drawbacks: poor data efficiency and overfitting.

Original: More specifically, since training data will only contain a finite number of compositions of action, category, and property terms, the model usually fails to generalize to manipulation instructions that contain novel compositions of seen terms and unseen terms.
Revised: More specifically, since training data will only contain a finite number of compositions of action, category, and property terms, the model usually fails to generalize to manipulation instructions that contain novel compositions of seen terms and unseen terms.

Original:  P ROG P ORT
Revised:  P ROG P ORT

Original: The key idea of our framework PROGPORT is that reliably utilizing pretrained vision-language (VL) models in manipulation involves leveraging the structure of natural language.
Revised: The key idea of our framework PROGPORT is that reliably utilizing pretrained vision-language (VL) models in manipulation involves leveraging the structure of natural language.

Original: Given the visual input and natural language instruction, PROGPORT parses the instruction into a program based on a domain-specific language (DSL).
Revised: Given the visual input and natural language instruction, PROGPORT parses the instruction into a program based on a domain-specific language (DSL).

Original: Each generated program corresponds to a hierarchical structure of functional modules implemented by neural networks, each fulfilling a specific operation over the vision or action representation.
Revised: Each generated program corresponds to a hierarchical structure of functional modules implemented by neural networks, each fulfilling a specific operation over the vision or action representation.

Original: The program executor executes the program upon the visual input and derives the control parameters to action primitives.
Revised: The program executor executes the program upon the visual input and derives the control parameters to action primitives.

Original: This compositional design disentangles the learning for visual grounding modules (e.g., locating the objects being referred to) and the action modules (e.g., predicting the control parameters).
Revised: This compositional design disentangles the learning for visual grounding modules (e.g., locating the objects being referred to) and the action modules (e.g., predicting the control parameters).

Original: Concretely, it enables us to directly leverage pretrained VL models such as CLIP as the visual grounding module without further finetuning, and to only focus on learning action modules, which improves the transparency, data efficiency, and the generalization of the system.
Revised: Concretely, it enables us to directly leverage pretrained VL models such as CLIP as the visual grounding module without further finetuning, and to only focus on learning action modules, which improves the transparency, data efficiency, and the generalization of the system.

Original: A visual overview of our framework is available in Fig.
Revised: A visual overview of our framework is available in Fig.

Original: In the remainder of this section, we first describe the structure of our manipulation programs, and how they are derived from natural language instructions (Section 3.1).
Revised: In the remainder of this section, we first describe the structure of our manipulation programs, and how they are derived from natural language instructions (Section 3.1).

Original: We then detail the neural modules which realize their execution (Section 3.2).
Revised: We then detail the neural modules which realize their execution (Section 3.2).

Original: Finally, we specify the training paradigm (Section 3.2).
Revised: Finally, we specify the training paradigm (Section 3.2).

Original: 3. P ROGRAMMATIC R EPRESENTATIONS OF M ANIPULATION I NSTRUCTIONS
Revised: 3. P ROGRAMMATIC R EPRESENTATIONS OF M ANIPULATION I NSTRUCTIONS

Original: At the core of the proposed PROGPORT framework is a programmatic representation for instructions.
Revised: At the core of the proposed PROGPORT framework is a programmatic representation for instructions.

Original: Illustrated in Fig.
Revised: Illustrated in Fig.

Original: 2b, we use a program to represent the underlying semantics of the natural language instruction.
Revised: 2b, we use a program to represent the underlying semantics of the natural language instruction.

Original: Each program has a hierarchical structure composed of functional modules.
Revised: Each program has a hierarchical structure composed of functional modules.

Original: Each functional module takes the outputs from its predecessors, the visual input, and optionally additional concepts from the language (e.g., red ) and computes the outputs (e.g., an attention mask).
Revised: Each functional module takes the outputs from its predecessors, the visual input, and optionally additional concepts from the language (e.g., red ) and computes the outputs (e.g., an attention mask).

Original: [Figure]
Revised: [Figure]

Original: We present the detailed definition of our domain-specific language (DSL) in Appendix A.1.
Revised: We present the detailed definition of our domain-specific language (DSL) in Appendix A.1.

Original: All functional modules can be categorized into two groups: visual grounding modules and action modules .
Revised: All functional modules can be categorized into two groups: visual grounding modules and action modules .

Original: Visual grounding modules take as input language descriptors, and return spatial masks over the corresponding locations in the visual scene.
Revised: Visual grounding modules take as input language descriptors, and return spatial masks over the corresponding locations in the visual scene.

Original: In this paper, the only visual grounding module we consider is the filter operation, which localizes objects that have certain properties (as in filter(blue, ...) will return blue objects in the scene).
Revised: In this paper, the only visual grounding module we consider is the filter operation, which localizes objects that have certain properties (as in filter(blue, ...) will return blue objects in the scene).

Original: In contrast, action modules generate control parameters for robot primitives.
Revised: In contrast, action modules generate control parameters for robot primitives.

Original: They take as input the outputs from visual grounding modules.
Revised: They take as input the outputs from visual grounding modules.

Original: For example, the goal module generates a 2D pose that has a specified spatial relationship with another object (as in goal(in, ...) will return a location in the stated object).
Revised: For example, the goal module generates a 2D pose that has a specified spatial relationship with another object (as in goal(in, ...) will return a location in the stated object).

Original: The do(pack, ..., ...) operator takes three arguments: the action name pack , the object to be packed, and the goal location, both specified as masks over the input image, and generates the control parameter for the robot’s next move.
Revised: The do(pack, ..., ...) operator takes three arguments: the action name pack , the object to be packed, and the goal location, both specified as masks over the input image, and generates the control parameter for the robot’s next move.

Original: Semantic parsing with a combinatory categorial grammar .
Revised: Semantic parsing with a combinatory categorial grammar .

Original: To parse language instructions into a program structure, we use the Combinatory Categorial Grammar (CCG) formalism (Steedman, 1996).
Revised: To parse language instructions into a program structure, we use the Combinatory Categorial Grammar (CCG) formalism (Steedman, 1996).

Original: A CCG parser is composed of a small (less than 10) set of universal categorial grammar rules and a domain-specific lexicon.
Revised: A CCG parser is composed of a small (less than 10) set of universal categorial grammar rules and a domain-specific lexicon.

Original: Each lexicon entry maps a natural language word (e.g., red ) to its syntactic type (e.g., N/N ) and semantic form (e.g., λx.
Revised: Each lexicon entry maps a natural language word (e.g., red ) to its syntactic type (e.g., N/N ) and semantic form (e.g., λx.

Original: filter ( red ,x ) ).
Revised: filter ( red ,x ) ).

Original: The notation N/N indicates that this word can be combined with a word on its right with type N (noun), and the semantic form indicates that this word corresponds to a visual grounding module that selects red objects from the scene.
Revised: The notation N/N indicates that this word can be combined with a word on its right with type N (noun), and the semantic form indicates that this word corresponds to a visual grounding module that selects red objects from the scene.

Original: PROGPORT uses a predefined lexicon, manually specified by the programmers.
Revised: PROGPORT uses a predefined lexicon, manually specified by the programmers.

Original: We choose to use a predefined CCG for three main reasons.
Revised: We choose to use a predefined CCG for three main reasons.

Original: First, CCG enables a compact definition of a domain-specific semantic parser.
Revised: First, CCG enables a compact definition of a domain-specific semantic parser.

Original: For each word in the vocabulary, we only need to annotate 1 or 2 entries.
Revised: For each word in the vocabulary, we only need to annotate 1 or 2 entries.

Original: Second, the
Revised: Second, the

Original: CCG parser is naturally compositional: given only the syntax and semantic forms for individual words, it is able to parse whole sentences with complex linguistic structures such as nested prepositional phrases.
Revised: CCG parser is naturally compositional: given only the syntax and semantic forms for individual words, it is able to parse whole sentences with complex linguistic structures such as nested prepositional phrases.

Original: Third, and most importantly, the CCG parser can handle unknown words at test time.
Revised: Third, and most importantly, the CCG parser can handle unknown words at test time.

Original: In particular, following work on syntactic bootstrapping (Abend et al., 2017; Alishahi & Stevenson, 2008), for a novel word (e.g., pack the daxy shape into the box ), the parser first “guesses” its syntactic type by ensuring that the entire sentence parses successfully (in this case, the inferred syntactic type is N/N ), and predicts its semantic form following a prior distribution of p ( semantics ∣ syntax ) , under the current lexicon.
Revised: In particular, following work on syntactic bootstrapping (Abend et al., 2017; Alishahi & Stevenson, 2008), for a novel word (e.g., pack the daxy shape into the box ), the parser first “guesses” its syntactic type by ensuring that the entire sentence parses successfully (in this case, the inferred syntactic type is N/N ), and predicts its semantic form following a prior distribution of p ( semantics ∣ syntax ) , under the current lexicon.

Original: In this case, the predicted semantic form is λx.
Revised: In this case, the predicted semantic form is λx.

Original: filter ( daxy ,x ) .
Revised: filter ( daxy ,x ) .

Original: However, note that with too many novel words, the CCG parser fails to successfully guess the syntactic type of each new word, and such inference fails.
Revised: However, note that with too many novel words, the CCG parser fails to successfully guess the syntactic type of each new word, and such inference fails.

Original: More generally, when training on demonstrations with larger vocabularies and more complex language instructions, predefining the lexicon becomes unscable.
Revised: More generally, when training on demonstrations with larger vocabularies and more complex language instructions, predefining the lexicon becomes unscable.

Original: Alternate techniques such as pretraining to learn a CCG can alleviate this issue (Bisk &
Revised: Alternate techniques such as pretraining to learn a CCG can alleviate this issue (Bisk &

Original: Hockenmaier, 2013; Zettlemoyer & Collins, 2012).
Revised: Hockenmaier, 2013; Zettlemoyer & Collins, 2012).

Original: 3. P ROG P ORT A RCHITECTURE
Revised: 3. P ROG P ORT A RCHITECTURE

Original: PROGPORT executes the parsed programs by following their hierarchical structures.
Revised: PROGPORT executes the parsed programs by following their hierarchical structures.

Original: Overall, both visual grounding and action modules consume outputs from their predecessors and individual concepts parsed by the CCG, instead of the entire language instruction, and produce outputs.
Revised: Overall, both visual grounding and action modules consume outputs from their predecessors and individual concepts parsed by the CCG, instead of the entire language instruction, and produce outputs.

Original: By disentangling manipulation in such a structured fashion, our architecture can learn to flexibly leverage vision-semantic priors across a myriad of specific manipulation contexts.
Revised: By disentangling manipulation in such a structured fashion, our architecture can learn to flexibly leverage vision-semantic priors across a myriad of specific manipulation contexts.

Original: Visual grounding modules .
Revised: Visual grounding modules .

Original: The key functionality of a visual grounding module is to localize visual regions corresponding to objects with a given language-specified property (i.e., filter ( flower ,... ) ).
Revised: The key functionality of a visual grounding module is to localize visual regions corresponding to objects with a given language-specified property (i.e., filter ( flower ,... ) ).

Original: To imbue our manipulation model with semantic understanding of diverse object properties and spatial relationships, including those unseen in the manipulation demonstration data, we borrow from the recent zero-shot semantic segmentation model MASKCLIP (Zhou et al., 2022).
Revised: To imbue our manipulation model with semantic understanding of diverse object properties and spatial relationships, including those unseen in the manipulation demonstration data, we borrow from the recent zero-shot semantic segmentation model MASKCLIP (Zhou et al., 2022).

Original: Given the input image x t and a singular concept c (e.g., flower ), we use the image encoder G and text encoder H from a pretrained CLIP model to obtain deep features G( x t ) ∈
Revised: Given the input image x t and a singular concept c (e.g., flower ), we use the image encoder G and text encoder H from a pretrained CLIP model to obtain deep features G( x t ) ∈

Original: R H ′
Revised: R H ′

Original: × W ′ × D 1 and H( c ) ∈ R D 2 , respectively * .
Revised: × W ′ × D 1 and H( c ) ∈ R D 2 , respectively * .

Original: Here, we denote the spatial dimensions of the visual features as H ′ × W ′ , and the feature dimensions of the visual and language streams as D 1 ,D 2 , respectively.
Revised: Here, we denote the spatial dimensions of the visual features as H ′ × W ′ , and the feature dimensions of the visual and language streams as D 1 ,D 2 , respectively.

Original: Recall that vanilla CLIP uses attention pooling to pool the visual features at different spatial locations into a single feature vector, and apply a linear layer to align the vector with the language encoding.
Revised: Recall that vanilla CLIP uses attention pooling to pool the visual features at different spatial locations into a single feature vector, and apply a linear layer to align the vector with the language encoding.

Original: We adopt a simple modification from MASKCLIP to achieve a similar semantic alignment of features, while preserving the spatiality of G( x t ) .
Revised: We adopt a simple modification from MASKCLIP to achieve a similar semantic alignment of features, while preserving the spatiality of G( x t ) .

Original: In particular, we initialize two 1 × 1 convolution layers
Revised: In particular, we initialize two 1 × 1 convolution layers

Original: C v and C l with weights from the penultimate and last linear layers of the CLIP attention pooling mechanism, respectively.
Revised: C v and C l with weights from the penultimate and last linear layers of the CLIP attention pooling mechanism, respectively.

Original: The C v layer is initialized from the value encoder in the QKV-attention layer (Vaswani et al., 2017), while the C l layer is initialized from the last linear layer.
Revised: The C v layer is initialized from the value encoder in the QKV-attention layer (Vaswani et al., 2017), while the C l layer is initialized from the last linear layer.

Original: Applying them sequentially will project the visual feature into the same space as the language feature.
Revised: Applying them sequentially will project the visual feature into the same space as the language feature.

Original: Finally, our grounding module F grounding generates a dense mask over image regions of concept c via:
Revised: Finally, our grounding module F grounding generates a dense mask over image regions of concept c via:

Original: [Equation]
Revised: [Equation]

Original: where TILE defines a spatial broadcast operator, in this case for the language embedding vector to the spatial dimensions of the visual features, the ⊙ is the Hadamard product, and the summation is over the feature dimension D 2 .
Revised: where TILE defines a spatial broadcast operator, in this case for the language embedding vector to the spatial dimensions of the visual features, the ⊙ is the Hadamard product, and the summation is over the feature dimension D 2 .

Original: More details are available in Appendix A.4.
Revised: More details are available in Appendix A.4.

Original: This formulation of our semantic module allows us to leverage CLIP’s large-scale vision-language pretraining to generalize semantics to open vocabulary categories, including objects and their visual properties.
Revised: This formulation of our semantic module allows us to leverage CLIP’s large-scale vision-language pretraining to generalize semantics to open vocabulary categories, including objects and their visual properties.

Original: Moreover, since our semantic program has decomposed the input language phrase l t into individual visual concepts, we can compose multiple filter programs with the intersection operation.
Revised: Moreover, since our semantic program has decomposed the input language phrase l t into individual visual concepts, we can compose multiple filter programs with the intersection operation.

Original: Specifically, this is implemented as in NS-CL (Mao et al., 2019) by taking the element-wise min of the generated masks, parsing out image regions that do not satisfy the combination of language descriptors.
Revised: Specifically, this is implemented as in NS-CL (Mao et al., 2019) by taking the element-wise min of the generated masks, parsing out image regions that do not satisfy the combination of language descriptors.

Original: Action modules .
Revised: Action modules .

Original: At their core, our action modules extend the architecture from CLIPORT , and finally generate the control parameters (T pick , T place ) .
Revised: At their core, our action modules extend the architecture from CLIPORT , and finally generate the control parameters (T pick , T place ) .

Original: In the pick-and-place setting, the visual grounding of the object to be picked (i.e., the mask output by the corresponding filter operations) can be directly interpreted as a distribution for T pick .
Revised: In the pick-and-place setting, the visual grounding of the object to be picked (i.e., the mask output by the corresponding filter operations) can be directly interpreted as a distribution for T pick .

Original: Therefore, in this section, we will focus on how PROGPORT predicts T place conditioned on T pick , the visual grounding of the target location (e.g., the orange box ), and the desired relation (e.g., in ).
Revised: Therefore, in this section, we will focus on how PROGPORT predicts T place conditioned on T pick , the visual grounding of the target location (e.g., the orange box ), and the desired relation (e.g., in ).

Original: The corresponding module is the “goal” operator.
Revised: The corresponding module is the “goal” operator.

Original: There are two design principles.
Revised: There are two design principles.

Original: First, besides the image and text, the action modules should also be conditioned on the grounding results from visual grounding modules (i.e., a spatial grounding map indicating the position of target objects).
Revised: First, besides the image and text, the action modules should also be conditioned on the grounding results from visual grounding modules (i.e., a spatial grounding map indicating the position of target objects).

Original: Second, we want to prevent overfitting the action module to visual categories presented at training time.
Revised: Second, we want to prevent overfitting the action module to visual categories presented at training time.

Original: Therefore, we extend the architecture for Q place to take the spatial grounding maps from previous visual modules with the following modifications.
Revised: Therefore, we extend the architecture for Q place to take the spatial grounding maps from previous visual modules with the following modifications.

Original: First, we bilinearly upsample or downsample the input masks to the spatial dimensions of different layers within the CLIPort feature backbone, and multiply the interpolated grounding map with these intermediate layers.
Revised: First, we bilinearly upsample or downsample the input masks to the spatial dimensions of different layers within the CLIPort feature backbone, and multiply the interpolated grounding map with these intermediate layers.

Original: In contrast to other fusion strategies such as concatenation, this fusion strategy ensures that the grounding map will not simply be ignored by the action modules.
Revised: In contrast to other fusion strategies such as concatenation, this fusion strategy ensures that the grounding map will not simply be ignored by the action modules.

Original: Second, we upsample the grounding map to the spatial dimensions of the output action maps, and perform direct element-wise matrix multiplication with the action map.
Revised: Second, we upsample the grounding map to the spatial dimensions of the output action maps, and perform direct element-wise matrix multiplication with the action map.

Original: Since the grounding map can be viewed as a mask of the objects being referred to, this multiplication effectively forces the action modules to attend to regions where pick objects and place locations are described in the manipulation instruction.
Revised: Since the grounding map can be viewed as a mask of the objects being referred to, this multiplication effectively forces the action modules to attend to regions where pick objects and place locations are described in the manipulation instruction.

Original: Overall, the end-effector poses in SE ( 2 ) can be output by appropriately adapting Eq.
Revised: Overall, the end-effector poses in SE ( 2 ) can be output by appropriately adapting Eq.

Original: [Equation]
Revised: [Equation]

Original: where Up (⋅) is an upsampling operator, and ⊙ is the Hadamard product.
Revised: where Up (⋅) is an upsampling operator, and ⊙ is the Hadamard product.

Original: More details are available in Appendix A.4.
Revised: More details are available in Appendix A.4.

Original: In the simple SE ( 2 ) pick-and-place setting under the Transporter or CLIPort framework, simply specifying the locations of the object and its target pose is sufficient, and thus do(pack, ..., ...) operates as an identity function.
Revised: In the simple SE ( 2 ) pick-and-place setting under the Transporter or CLIPort framework, simply specifying the locations of the object and its target pose is sufficient, and thus do(pack, ..., ...) operates as an identity function.

Original: Example.
Revised: Example.

Original: Recall that the program for the instruction “pack the blue hexagon in the orange box” is given by do(goal(filter(filter(hexagon), blue), filter(filter(box), orange), in), pack) .
Revised: Recall that the program for the instruction “pack the blue hexagon in the orange box” is given by do(goal(filter(filter(hexagon), blue), filter(filter(box), orange), in), pack) .

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 3, our first set of filter modules take the input image and output masks corresponding to “hexagon” shapes and “box” objects.
Revised: 3, our first set of filter modules take the input image and output masks corresponding to “hexagon” shapes and “box” objects.

Original: The next stage similarly filters for “blue” and “orange” objects.
Revised: The next stage similarly filters for “blue” and “orange” objects.

Original: To compose these adjectives with their corresponding nouns from the previous stage, we take their elementwise min, returning masks corresponding to “blue hexagon” and “orange box”, respectively.
Revised: To compose these adjectives with their corresponding nouns from the previous stage, we take their elementwise min, returning masks corresponding to “blue hexagon” and “orange box”, respectively.

Original: Next, our goal module takes the masks for the referred objects, which correspond to the picked object and its general place region, and outputs a distribution over SE ( 2 ) pose for the object that is semantically consistent with the “in” target location.
Revised: Next, our goal module takes the masks for the referred objects, which correspond to the picked object and its general place region, and outputs a distribution over SE ( 2 ) pose for the object that is semantically consistent with the “in” target location.

Original: Finally, our do module outputs the final control parameters to the robot end-effector.
Revised: Finally, our do module outputs the final control parameters to the robot end-effector.

Original: In this case, it contains T pick from the execution of filter(filter(hexagon), blue) , and the T place output by the goal module.
Revised: In this case, it contains T pick from the execution of filter(filter(hexagon), blue) , and the T place output by the goal module.

Original: Training .
Revised: Training .

Original: Our model can be trained in an end-to-end fashion, given the observation-action tuples ( x t , l t , a t ) , via an imitation learning objective.
Revised: Our model can be trained in an end-to-end fashion, given the observation-action tuples ( x t , l t , a t ) , via an imitation learning objective.

Original: Note that each action a t is itself a tuple described by (T pick , T place ) .
Revised: Note that each action a t is itself a tuple described by (T pick , T place ) .

Original: Thus, given a random observation-action tuple ( x t , l t , a t ) sampled from some expert demonstration d i ∈ D at time t , we fit our model by minimizing the cross-entropy loss:
Revised: Thus, given a random observation-action tuple ( x t , l t , a t ) sampled from some expert demonstration d i ∈ D at time t , we fit our model by minimizing the cross-entropy loss:

Original: [Equation]
Revised: [Equation]

Revised: Q place ( τ i ∣ x t , l t , T pick )] (4)

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we will show that the modularity of PROGPORT enables better vision-language grounding and generalization, which empirically translates to strong zero-shot generalization (i.e., understanding instructions with unseen visual concepts), and compositional generalzation (i.e., understanding novel combination of previously seen visual and action concepts). 
Revised: In this section, we will show that the modularity of PROGPORT enables better vision-language grounding and generalization, which empirically translates to strong zero-shot generalization (i.e., understanding instructions with unseen visual concepts), and compositional generalzation (i.e., understanding novel combination of previously seen visual and action concepts). We introduce a new operator relate and two new tasks involving prepositional phrases. Zero-shot generalization is evaluated in packing-prepositions , and compositional generalization in packingnested-prepositions . Please see Appendix A.5 for more details.

Original: Dataset .
Revised: Dataset .

Original: Our dataset is an extension of the benchmark proposed by CLIPORT (Shridhar et al., 2022a), which itself is based on the Ravens benchmark from Zeng et al.
Revised: Our dataset is an extension of the benchmark proposed by CLIPORT (Shridhar et al., 2022a), which itself is based on the Ravens benchmark from Zeng et al.

Original: The CLIPORT benchmark consists of 10 language-conditioned manipulation tasks in a PyBullet simulation environment, where all episodes are performed by a Universal Robot UR5e with a suction gripper end-effector.
Revised: The CLIPORT benchmark consists of 10 language-conditioned manipulation tasks in a PyBullet simulation environment, where all episodes are performed by a Universal Robot UR5e with a suction gripper end-effector.

Original: Every episode in a task is constructed by sampling a set of objects and object properties (such as object type, color, size etc.), and manipulating a subset of the objects to some desired conformation, which is described by a language goal.
Revised: Every episode in a task is constructed by sampling a set of objects and object properties (such as object type, color, size etc.), and manipulating a subset of the objects to some desired conformation, which is described by a language goal.

Original: Concretely, each training sample in an episode consists of a (visual scene, language instruction) input tuple, and a (pick pose, place pose) target output tuple.

Original: The visual input is a 640 × 480 top-down RGB-D orthographic projection from 3 different camera views.

Original: The language input is obtained by completing a template with the specified objects and object properties.

Original: The pick and place poses are obtained from a successful rollout of an oracle agent.

Original: Model details, training, and evaluation .
Revised: Model details, training, and evaluation .

Original: Our visual grounding modules use the weights from the frozen, pretrained ViT-B/16 CLIP encoder.
Revised: Our visual grounding modules use the weights from the frozen, pretrained ViT-B/16 CLIP encoder.

Original: We train for 200k iterations with AdamW and a learning rate of 0.0002.
Revised: We train for 200k iterations with AdamW and a learning rate of 0.0002.

Original: We use either n = 100 or n = 1000 training demonstrations, and evaluate ontest demonstrations.
Revised: We use either n = 100 or n = 1000 training demonstrations, and evaluate ontest demonstrations.

Original: Our evaluation metric follows Shridhar et al.
Revised: Our evaluation metric follows Shridhar et al.

Original: (2022a)
Revised: (2022a)

Original: based on the same CLIP encoders, where a dense score between 0 (failure) and 100 (success) represents the fraction of the episode completed correctly.
Revised: based on the same CLIP encoders, where a dense score between 0 (failure) and 100 (success) represents the fraction of the episode completed correctly.

Original: For baselines, we compare to CLIPORT across two settings: the single setting, where the model trains and tests on a single task at a time, and the multi setting, where the model trains on all tasks at once, and is independently evaluated on each individual task.
Revised: For baselines, we compare to CLIPORT across two settings: the single setting, where the model trains and tests on a single task at a time, and the multi setting, where the model trains on all tasks at once, and is independently evaluated on each individual task.

Original: 4. Z ERO -S HOT G ENERALIZATION
Revised: 4. Z ERO -S HOT G ENERALIZATION

Original: To evaluate zero-shot generalization, all of our main results in Table 1 use the “unseen” variants of the tasks, as described in Shridhar et al.
Revised: To evaluate zero-shot generalization, all of our main results in Table 1 use the “unseen” variants of the tasks, as described in Shridhar et al.

Original: (2022a).
Revised: (2022a).

Original: Concretely, the train and test splits of each task involve different objects and object properties, with minimal overlap.
Revised: Concretely, the train and test splits of each task involve different objects and object properties, with minimal overlap.

Original: For example, out of 11 total possible colors for objects, only 3 are shared for both train and test episodes.
Revised: For example, out of 11 total possible colors for objects, only 3 are shared for both train and test episodes.

Original: Or for tasks involving real-world objects from the Google Scanned Objects dataset (Downs et al., 2022), a total of 56 household items are split into mutually exclusive sets of 37 train items and 19 test items.
Revised: Or for tasks involving real-world objects from the Google Scanned Objects dataset (Downs et al., 2022), a total of 56 household items are split into mutually exclusive sets of 37 train items and 19 test items.

Original: Crucially, for a fair comparison, these test language descriptors are also entirely missing from the predefined CCG
Revised: Crucially, for a fair comparison, these test language descriptors are also entirely missing from the predefined CCG

Original: [Table]
Revised: [Table]

Original: all tasks involve different objects, object properties, or other spatial descriptors.
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: lexicon, and  semantic  on  described in Section 
Revised: lexicon, and so semantic goal parsing must rely on the probabilistic approach described in Section 3.1.

Original: Our results show that our model effectively grounds object concepts described in language with the visual scene, and with the exception of the align-rope task, outperforms our baselines across the benchmark (Table 1).
Revised: Our results show that our model effectively grounds object concepts described in language with the visual scene, and with the exception of the align-rope task, outperforms our baselines across the benchmark (Table 1).

Original: Align-rope is challenging for PROGPORT because it is hard for the CLIP model to ground abstract spatial concepts used in this task (e.g. front and back ).
Revised: Align-rope is challenging for PROGPORT because it is hard for the CLIP model to ground abstract spatial concepts used in this task (e.g. front and back ).

Original: Data efficiency .
Revised: Data efficiency .

Original: Not only do we outperform the baselines by a large margin, but on many tasks, our method does significantly better with 10 × fewer demonstrations.
Revised: Not only do we outperform the baselines by a large margin, but on many tasks, our method does significantly better with 10 × fewer demonstrations.

Original: For example, on the “packingshapes” task, our best-performing model trained on 100 demonstrations outperforms its corresponding baseline trained on 1000 demonstrations by 25.0 points.
Revised: For example, on the “packingshapes” task, our best-performing model trained on 100 demonstrations outperforms its corresponding baseline trained on 1000 demonstrations by 25.0 points.

Original: This data efficiency is derived from our improved grounding properties: by semantically parsing and visually grounding one object concept at a time, we more effectively leverage the pretrained VL model.
Revised: This data efficiency is derived from our improved grounding properties: by semantically parsing and visually grounding one object concept at a time, we more effectively leverage the pretrained VL model.

Original: Moreover, note that the best-performing CLIPORT models for different tasks are mostly the single variants.
Revised: Moreover, note that the best-performing CLIPORT models for different tasks are mostly the single variants.

Original: This suggests the zero-shot performance of CLIPORT relies heavily on overfitting to the actions of the specific task, as opposed to leveraging the general visual grounding enabled by
Revised: This suggests the zero-shot performance of CLIPORT relies heavily on overfitting to the actions of the specific task, as opposed to leveraging the general visual grounding enabled by

Original: CLIP.
Revised: CLIP.

Original: With PROGPORT , the gap between our single and multi model variants is much smaller, and indeed the multi variants are often the stronger manipulators.
Revised: With PROGPORT , the gap between our single and multi model variants is much smaller, and indeed the multi variants are often the stronger manipulators.

Original: PROGPORT thus leverages diverse visual and action information across tasks more effectively, and possesses scalability with respect to both the number of demonstrations as well as the number of tasks.
Revised: PROGPORT thus leverages diverse visual and action information across tasks more effectively, and possesses scalability with respect to both the number of demonstrations as well as the number of tasks.

Original: Improved vision-language grounding .
Revised: Improved vision-language grounding .

Original: Qualitatively, we show more interpretable and robust affordance maps compared to the baseline (Fig. 4).
Revised: Qualitatively, we show more interpretable and robust affordance maps compared to the baseline (Fig. 4).

Original: By grounding a single object or property at a time, and then leveraging our DSL operations to compose together these masks, we ground the different concepts.
Revised: By grounding a single object or property at a time, and then leveraging our DSL operations to compose together these masks, we ground the different concepts.

Original: For T pick , the lion figure is extremely small within the visual scene, and positioned in a crowded area; our visual grounding modules are still able to compose the concepts of figure and lion together in a zero-shot manner, informing an accurate pick affordance.
Revised: For T pick , the lion figure is extremely small within the visual scene, and positioned in a crowded area; our visual grounding modules are still able to compose the concepts of figure and lion together in a zero-shot manner, informing an accurate pick affordance.

Original: Similarly, for T place , the baseline is confused by the visual similarity between the box and the porcelain plate .
Revised: Similarly, for T place , the baseline is confused by the visual similarity between the box and the porcelain plate .

Original: By contrast, our visual grounding modules compose box and brown together to focus on the left box-like object.
Revised: By contrast, our visual grounding modules compose box and brown together to focus on the left box-like object.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: yields near-perfect manipulation.

Original: 4. C OMPOSITIONAL G ENERALIZATION OF V ISUAL G ROUNDING AND B EHAVIORS
Revised: 4. C OMPOSITIONAL G ENERALIZATION OF V ISUAL G ROUNDING AND B EHAVIORS

Original: To evaluate compositional generalization, we introduce 4 new tasks which re-combine basic unary object concepts from the original 10 tasks (Table 2).
Revised: To evaluate compositional generalization, we introduce 4 new tasks which re-combine basic unary object concepts from the original 10 tasks (Table 2).

Original: We do not train directly on these tasks; by evaluating on them only, we assess the ability to reason compositionally about the distinct visual properties, spatial relationships, and manipulation primitives learned at training time.
Revised: We do not train directly on these tasks; by evaluating on them only, we assess the ability to reason compositionally about the distinct visual properties, spatial relationships, and manipulation primitives learned at training time.

Original: For example, the packing-location-box task contains instructions of the form “pack the { shape } into the { loc } box”, where both shape and loc are objects and spatial descriptors seen independently throughout various tasks at training time, but now appear in a test-time context that requires the model to reason semantically over their combination.
Revised: For example, the packing-location-box task contains instructions of the form “pack the { shape } into the { loc } box”, where both shape and loc are objects and spatial descriptors seen independently throughout various tasks at training time, but now appear in a test-time context that requires the model to reason semantically over their combination.

Original: For a full specification of these tasks, please refer Appendix A.2.
Revised: For a full specification of these tasks, please refer Appendix A.2.

Original: Length-based generalization to new action settings .
Revised: Length-based generalization to new action settings .

Original: Our fourth generalization task (pushingshapes) is much more difficult, involving language instructions of the form “push the { color } { shape } into the { location } { color } square”.
Revised: Our fourth generalization task (pushingshapes) is much more difficult, involving language instructions of the form “push the { color } { shape } into the { location } { color } square”.

Original: Note that this task evaluates both compositional generalization and “length generalization”.
Revised: Note that this task evaluates both compositional generalization and “length generalization”.

Original: The latter assesses the ability of the model to generalize beyond the assumption of at most one visual descriptor per object in similar training tasks.
Revised: The latter assesses the ability of the model to generalize beyond the assumption of at most one visual descriptor per object in similar training tasks.

Original: Furthermore, this task requires the model to zero-shot generalize to entirely new actions; only pushing of simple, regularly-structured blocks is learned at training time, whereas shapes are much larger, irregular, and have different friction properties.
Revised: Furthermore, this task requires the model to zero-shot generalize to entirely new actions; only pushing of simple, regularly-structured blocks is learned at training time, whereas shapes are much larger, irregular, and have different friction properties.

Original: PROGPORT exhibits much better performance here (+32.6 mean reward), compared to the CLIPORT baseline, which fails entirely.
Revised: PROGPORT exhibits much better performance here (+32.6 mean reward), compared to the CLIPORT baseline, which fails entirely.

Original: 4. A BLATION : D ISENTANGLEMENT OF A CTION AND P ERCEPTION
Revised: 4. A BLATION : D ISENTANGLEMENT OF A CTION AND P ERCEPTION

Original: To illustrate the disentanglement of action and perception in our modular design, we perform an additional ablation study.
Revised: To illustrate the disentanglement of action and perception in our modular design, we perform an additional ablation study.

Original: Specifically, we train our model normally on all 10 main tasks from Section 4.1, and at test time, we choose 5 diverse tasks out of the 10.
Revised: Specifically, we train our model normally on all 10 main tasks from Section 4.1, and at test time, we choose 5 diverse tasks out of the 10.

Original: For each test episode, we replace the attention maps generated by the semantic modules with ground truth segmentation maps corresponding to the picked objects and place locations.
Revised: For each test episode, we replace the attention maps generated by the semantic modules with ground truth segmentation maps corresponding to the picked objects and place locations.

Original: This effectively poses the question: given a perfect perception of the scene, how would our action modules behave?
Revised: This effectively poses the question: given a perfect perception of the scene, how would our action modules behave?

Original: A properly disentangled model would be able to leverage the noiseless ground truth visual grounding and decide the appropriate actions to perform.
Revised: A properly disentangled model would be able to leverage the noiseless ground truth visual grounding and decide the appropriate actions to perform.

Original: As apparent in Table 3, performance across all tasks improves dramatically when provided the ground truth information.
Revised: As apparent in Table 3, performance across all tasks improves dramatically when provided the ground truth information.

Original: This suggests that our action modules do not overfit to task-specific shortcuts when optimizing the imitation objective, but actually learn to leverage the attention maps provided by the semantic modules in a disentangled manner.
Revised: This suggests that our action modules do not overfit to task-specific shortcuts when optimizing the imitation objective, but actually learn to leverage the attention maps provided by the semantic modules in a disentangled manner.

Original: Such disentanglement has also been translated to improved compositional generalization for our model.
Revised: Such disentanglement has also been translated to improved compositional generalization for our model.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Learning language-guided robot behavior .
Revised: Learning language-guided robot behavior .

Original: Many prior works have explored enhancing robotic behavior through language-conditioned imitation learning (Lynch & Sermanet, 2021; Stengel-Eskin et al., 2022; Stepputtis et al., 2020) and reinforcement learning (Andreas et al., 2018;
Revised: Many prior works have explored enhancing robotic behavior through language-conditioned imitation learning (Lynch & Sermanet, 2021; Stengel-Eskin et al., 2022; Stepputtis et al., 2020) and reinforcement learning (Andreas et al., 2018;

Original: Luketina et al., 2019; Misra et al., 2017).
Revised: Luketina et al., 2019; Misra et al., 2017).

Original: With the rise of powerful transformer-based architectures, recent works have used large-scale pretrained vision-language representations to improve performance for tabletop manipulation (Shridhar et al., 2022a), visual navigation (Khandelwal et al., 2022), and real-world skill learning and planning (Ahn et al., 2022; Huang et al., 2022).
Revised: With the rise of powerful transformer-based architectures, recent works have used large-scale pretrained vision-language representations to improve performance for tabletop manipulation (Shridhar et al., 2022a), visual navigation (Khandelwal et al., 2022), and real-world skill learning and planning (Ahn et al., 2022; Huang et al., 2022).

Original: For example, CLIPORT shows that by integrating embeddings from the CLIP VL model into the Transporter manipulation framework (Zeng et al., 2020), a robot can better generalize to manipulating new objects in a zero-shot manner.
Revised: For example, CLIPORT shows that by integrating embeddings from the CLIP VL model into the Transporter manipulation framework (Zeng et al., 2020), a robot can better generalize to manipulating new objects in a zero-shot manner.

Original: However, in contrast with our method PROGPORT , these works do not leverage the semantic structure in natural language to ground individual visual and action concepts.
Revised: However, in contrast with our method PROGPORT , these works do not leverage the semantic structure in natural language to ground individual visual and action concepts.

Original: In particular, prior works use the pretrained language model as a monolith that consumes the entire language input and outputs a latent embedding.
Revised: In particular, prior works use the pretrained language model as a monolith that consumes the entire language input and outputs a latent embedding.

Original: This embedding is not grounded to individual concepts within the language inputs, which leads to poor generalization when faced with novel (combinations) object attributes.
Revised: This embedding is not grounded to individual concepts within the language inputs, which leads to poor generalization when faced with novel (combinations) object attributes.

Original: Neurosymbolic learning .
Revised: Neurosymbolic learning .

Original: Prior works have shown neurosymbolic learning, which refers to the integration of deep neural networks for pattern recognition and symbolic structures such as programs for reasoning, as an effective mechanism for enforcing VL grounding (Mao et al., 2019; Wu et al., 2019).
Revised: Prior works have shown neurosymbolic learning, which refers to the integration of deep neural networks for pattern recognition and symbolic structures such as programs for reasoning, as an effective mechanism for enforcing VL grounding (Mao et al., 2019; Wu et al., 2019).

Original: Mao et al.
Revised: Mao et al.

Original: (2019) is a representative work that learns to jointly disentangle independent visual concepts and parse natural language sentences, from natural supervision in VQA.
Revised: (2019) is a representative work that learns to jointly disentangle independent visual concepts and parse natural language sentences, from natural supervision in VQA.

Original: The learned concepts can be composed for novel downstream tasks via programmatic (re)composition of neural network modules, enabling zero-shot and compositional generalization.
Revised: The learned concepts can be composed for novel downstream tasks via programmatic (re)composition of neural network modules, enabling zero-shot and compositional generalization.

Original: More generally, a structured approach involving neural perception of vision and language, and higher-order symbolic reasoning has appeared previously (Li et al., 2020; Nottingham et al., 2021; Sun & Alexandre, 2013).
Revised: More generally, a structured approach involving neural perception of vision and language, and higher-order symbolic reasoning has appeared previously (Li et al., 2020; Nottingham et al., 2021; Sun & Alexandre, 2013).

Original: Beyond improved VL-grounding, this approach has also translated to improved performance and data efficiency on many downstream tasks, such as reinforcement learning (Cheng et al., 2019; Verma et al., 2019), scientific discovery (Cranmer et al., 2020; Shah et al., 2020) and robotics (Zellers et al., 2021).
Revised: Beyond improved VL-grounding, this approach has also translated to improved performance and data efficiency on many downstream tasks, such as reinforcement learning (Cheng et al., 2019; Verma et al., 2019), scientific discovery (Cranmer et al., 2020; Shah et al., 2020) and robotics (Zellers et al., 2021).

Original: Our work is similarly inspired to combine structured representations with deep learning, in the new setting of robot manipulation.
Revised: Our work is similarly inspired to combine structured representations with deep learning, in the new setting of robot manipulation.

Original: However, instead of learning a small set of visual concepts, we leverage pretrained VL models to cover a larger, open-ended set of concepts, and improve our zero-shot capabilities.
Revised: However, instead of learning a small set of visual concepts, we leverage pretrained VL models to cover a larger, open-ended set of concepts, and improve our zero-shot capabilities.

Original: Moreover, we ground not just visual concepts with their language descriptors, but also abstract actions for robotic primitives, while preserving compositional generalization capability.
Revised: Moreover, we ground not just visual concepts with their language descriptors, but also abstract actions for robotic primitives, while preserving compositional generalization capability.

Original: Vision-language (VL) grounding .
Revised: Vision-language (VL) grounding .

Original: The goal of VL grounding is to identify correspondences between visual input and descriptors in natural language.
Revised: The goal of VL grounding is to identify correspondences between visual input and descriptors in natural language.

Original: Common tasks include visual question answering (Antol et al., 2015), VL navigation (Anderson et al., 2018), or VL manipulation (Shridhar et al., 2022a;b).
Revised: Common tasks include visual question answering (Antol et al., 2015), VL navigation (Anderson et al., 2018), or VL manipulation (Shridhar et al., 2022a;b).

Original: Recent works show that the CLIP (Radford et al., 2021) model trained on 400M image-caption pairs from the web possesses zero-shot VL grounding capabilities for a wide array of visual concepts (Kim et al., 2021;
Revised: Recent works show that the CLIP (Radford et al., 2021) model trained on 400M image-caption pairs from the web possesses zero-shot VL grounding capabilities for a wide array of visual concepts (Kim et al., 2021;

Original: Shen et al., 2022).
Revised: Shen et al., 2022).

Original: In robotics, this has translated to grounding of robotic affordances with natural language in models such as CLIPORT (Shridhar et al., 2022a).
Revised: In robotics, this has translated to grounding of robotic affordances with natural language in models such as CLIPORT (Shridhar et al., 2022a).

Original: However, we find that the na¨ıve approach taken by these works, which leverage only the frozen, pretrained CLIP embeddings, cannot successfully disentangle task-agnostic visual grounding with task-specific action policies.
Revised: However, we find that the na¨ıve approach taken by these works, which leverage only the frozen, pretrained CLIP embeddings, cannot successfully disentangle task-agnostic visual grounding with task-specific action policies.

Original: Other works show that this na¨ıve integration of pretrained VL models also affects grounding abilities in 3D space (Thomason et al., 2022), or temporal settings such as video understanding (Le & Hoi, 2020).
Revised: Other works show that this na¨ıve integration of pretrained VL models also affects grounding abilities in 3D space (Thomason et al., 2022), or temporal settings such as video understanding (Le & Hoi, 2020).

Original: In particular, VL grounding is often adversely affected by end-to-end training (e.g. via an imitation objective), and the resultant models are unable to reason about novel concepts independently, or compose them de novo for more complex tasks (Lin et al., 2022).
Revised: In particular, VL grounding is often adversely affected by end-to-end training (e.g. via an imitation objective), and the resultant models are unable to reason about novel concepts independently, or compose them de novo for more complex tasks (Lin et al., 2022).

Original: This shortcoming is addressed by PROGPORT through a program-based modular approach.
Revised: This shortcoming is addressed by PROGPORT through a program-based modular approach.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we propose PROGPORT , a program-based modular framework for robotic manipulation.
Revised: In this work, we propose PROGPORT , a program-based modular framework for robotic manipulation.

Original: We leverage the semantic structure of natural language to programmatically ground individual language concepts.
Revised: We leverage the semantic structure of natural language to programmatically ground individual language concepts.

Original: Our programs are executed by a series of neural modules, dedicated to learning either general visual concepts or task-specific manipulation policies.
Revised: Our programs are executed by a series of neural modules, dedicated to learning either general visual concepts or task-specific manipulation policies.

Original: We show that this structured learning approach disentangles general visual grounding from policy learning, leading to improved zero-shot and compositional generalization of learned behaviors.
Revised: We show that this structured learning approach disentangles general visual grounding from policy learning, leading to improved zero-shot and compositional generalization of learned behaviors.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: Our simulated dataset details are provided in the Appendix A.2.
Revised: Our simulated dataset details are provided in the Appendix A.2.

Original: The most salient training details and hyperparameters are presented in Section 4 of the main text.
Revised: The most salient training details and hyperparameters are presented in Section 4 of the main text.

Original: Our code will also be subsequently open-sourced and publicly available.
Revised: Our code will also be subsequently open-sourced and publicly available.

Original:  A A PPENDIX A. D OMAIN -S PECIFIC L ANGUAGE
Revised:  A A PPENDIX A. D OMAIN -S PECIFIC L ANGUAGE

Original: We detail here the operations (Table 4) and types (Table 5) contained within our DSL.
Revised: We detail here the operations (Table 4) and types (Table 5) contained within our DSL.

Original: The implementation of filter , do and goal are given within the main text.
Revised: The implementation of filter , do and goal are given within the main text.

Original: The remaining operations are parameter-free.
Revised: The remaining operations are parameter-free.

Original: In particular, Scene returns a uniform attention map over the visual input; ObjUnion and ActionConcat take the elementwise max of their input grounding or action maps, respectively.
Revised: In particular, Scene returns a uniform attention map over the visual input; ObjUnion and ActionConcat take the elementwise max of their input grounding or action maps, respectively.

Original: Operation
Revised: Operation

Original: Signature
Revised: Signature

Original: Semantics
Revised: Semantics

Original: Scene () —→ Object
Revised: Scene () —→ Object

Original: Return all objects in visual scene.
Revised: Return all objects in visual scene.

Original: Filter (Object, ObjProp) —→ Object
Revised: Filter (Object, ObjProp) —→ Object

Original: Return all objects satisfying a vision-language property.
Revised: Return all objects satisfying a vision-language property.

Original: Relate (Object, Object, ObjRel) —→ Object
Revised: Relate (Object, Object, ObjRel) —→ Object

Original: Return all objects satistfying a relation between target and reference objects.
Revised: Return all objects satistfying a relation between target and reference objects.

Original: Do ( { Goal } , Action) —→ Plan
Revised: Do ( { Goal } , Action) —→ Plan

Original: Perform an action according to a set of goals.
Revised: Perform an action according to a set of goals.

Original: Goal (Object, Object, ObjRel) —→ Goal
Revised: Goal (Object, Object, ObjRel) —→ Goal

Original: Identify a manipulation goal for an object and target relation.
Revised: Identify a manipulation goal for an object and target relation.

Original: ObjUnion (Object, Object) —→ Object
Revised: ObjUnion (Object, Object) —→ Object

Original: Return the union of two (sets) of objects.
Revised: Return the union of two (sets) of objects.

Original: ActionConcat (Plan, Plan) —→ Plan
Revised: ActionConcat (Plan, Plan) —→ Plan

Original: Jointly perform two sets of actions.
Revised: Jointly perform two sets of actions.

Original: [Table]
Revised: [Table]

Original: A. D ATASET D ETAILS
Revised: A. D ATASET D ETAILS

Original: Our dataset is based off the Ravens benchmark proposed in Zeng et al.
Revised: Our dataset is based off the Ravens benchmark proposed in Zeng et al.

Original: (2020), and its adaptation in Shridhar et al.
Revised: (2020), and its adaptation in Shridhar et al.

Original: (2022a).
Revised: (2022a).

Original: In particular, for our manipulatives, we use the same set of 20 synthetic shapes and 56 Google Scanned Objects.
Revised: In particular, for our manipulatives, we use the same set of 20 synthetic shapes and 56 Google Scanned Objects.

Original: For modifying visual appearance, we use the same set ofcolors .
Revised: For modifying visual appearance, we use the same set ofcolors .

Original: Most importantly, we use the same split of “seen” and “unseen” objects and colors, training on the “seen” splits and testing on the “unseen” splits.
Revised: Most importantly, we use the same split of “seen” and “unseen” objects and colors, training on the “seen” splits and testing on the “unseen” splits.

Original: These splits are mutually exclusive for the manipulatives; only ‘red’, ‘green’ and ‘blue’ are shared between splits for the colors.
Revised: These splits are mutually exclusive for the manipulatives; only ‘red’, ‘green’ and ‘blue’ are shared between splits for the colors.

Original: In the remainder of this section, we detail the specific tasks used within our benchmark, including visual scene setup and the success metric used to compute the results in Table 1, Table 2 and Table 3.
Revised: In the remainder of this section, we detail the specific tasks used within our benchmark, including visual scene setup and the success metric used to compute the results in Table 1, Table 2 and Table 3.

Original: A.2. B ASIC T ASKS
Revised: A.2. B ASIC T ASKS

Original: Packing Box Pairs.
Revised: Packing Box Pairs.

Original: Given small boxes of varying colors, and a larger, open-topped brown box, the goal is to tightly pack all small boxes of two specified colors into the large brown box.
Revised: Given small boxes of varying colors, and a larger, open-topped brown box, the goal is to tightly pack all small boxes of two specified colors into the large brown box.

Original: Sizes of all boxes are randomized, and there will be an exact number of the specified small boxes to tightly pack the larger box, in addition to other distractor small boxes.
Revised: Sizes of all boxes are randomized, and there will be an exact number of the specified small boxes to tightly pack the larger box, in addition to other distractor small boxes.

Original: Occasionally, there will be boxes of only one of the two specified colors; the agent must learn to ignore the missing color and pack the boxes with the one present color.
Revised: Occasionally, there will be boxes of only one of the two specified colors; the agent must learn to ignore the missing color and pack the boxes with the one present color.

Original: Success is the fraction of total volume of specified color blocks in the scene that end up packed inside the brown box.
Revised: Success is the fraction of total volume of specified color blocks in the scene that end up packed inside the brown box.

Original: Packing Google Objects Seq.
Revised: Packing Google Objects Seq.

Original: At each timestep , given a number of Google scanned objects and an open-topped brown box, the goal is to place a specified object into the box.
Revised: At each timestep , given a number of Google scanned objects and an open-topped brown box, the goal is to place a specified object into the box.

Original: Each episode is composed of at least 1 and at most 5 timesteps.
Revised: Each episode is composed of at least 1 and at most 5 timesteps.

Original: Success is determined by the total volume of all specified objects within the brown box up to the given timestep, divided by the volume of all specified objects.
Revised: Success is determined by the total volume of all specified objects within the brown box up to the given timestep, divided by the volume of all specified objects.

Original: Packing Google Objects Group.
Revised: Packing Google Objects Group.

Original: Given a number of Google scanned objects from at least two different categories, as well as an open-topped brown box, place all objects of a specified category into the brown box.
Revised: Given a number of Google scanned objects from at least two different categories, as well as an open-topped brown box, place all objects of a specified category into the brown box.

Original: Success is determined by the fraction of volume of specified objects placed within the box,
Revised: Success is determined by the fraction of volume of specified objects placed within the box,

Original: Separating Small Piles.
Revised: Separating Small Piles.

Original: Given 50 small blocks and two square zones of different colors, positions and poses, push all the blocks into the zone with a specified color.
Revised: Given 50 small blocks and two square zones of different colors, positions and poses, push all the blocks into the zone with a specified color.

Original: Success is determined by the fraction of total blocks within the correct zone.
Revised: Success is determined by the fraction of total blocks within the correct zone.

Original: Separating Large Piles.
Revised: Separating Large Piles.

Original: Given 15 large blocks and two square zones of different colors, positions and poses, push all the blocks into the zone with a specified color.
Revised: Given 15 large blocks and two square zones of different colors, positions and poses, push all the blocks into the zone with a specified color.

Original: Beyond size changes, the large blocks also have more mass and lateral friction compared to their small counterparts.
Revised: Beyond size changes, the large blocks also have more mass and lateral friction compared to their small counterparts.

Original: Success is determined by the fraction of total blocks within the correct zone.
Revised: Success is determined by the fraction of total blocks within the correct zone.

Original: Align Rope.
Revised: Align Rope.

Original: Given a soft-body rope formed from 20 articulated beads and the outline of a 3-sided square, the goal is to connect the two endpoints of the rope to two adjacent corners of the square.
Revised: Given a soft-body rope formed from 20 articulated beads and the outline of a 3-sided square, the goal is to connect the two endpoints of the rope to two adjacent corners of the square.

Original: The language instruction can specify one of four possible configurations:
Revised: The language instruction can specify one of four possible configurations:

Original: From front left tip to front right tip 2.
Revised: From front left tip to front right tip 2.

Original: From front right tip to back right corner 3.
Revised: From front right tip to back right corner 3.

Original: From front left tip to back left corner 4.
Revised: From front left tip to back left corner 4.

Original: From back right corner to back left corner
Revised: From back right corner to back left corner

Original: Success is represented by the degree to which the bead poses after manipulation match the line segment between the two adjacent corners specified in the language goal.
Revised: Success is represented by the degree to which the bead poses after manipulation match the line segment between the two adjacent corners specified in the language goal.

Original: Packing Shapes.
Revised: Packing Shapes.

Original: Given 4 distractor shapes and 1 specified shape, as well as a open-topped brown box, the goal is to put the specified shape into the brown box.
Revised: Given 4 distractor shapes and 1 specified shape, as well as a open-topped brown box, the goal is to put the specified shape into the brown box.

Original: Shapes are all unique.
Revised: Shapes are all unique.

Original: Success is binary; 1 if the shape is inside the boundaries of the box, and 0 otherwise.
Revised: Success is binary; 1 if the shape is inside the boundaries of the box, and 0 otherwise.

Original: Assembling Kits.
Revised: Assembling Kits.

Original: Given 5 shapes and a “kit” of 5 similarly-shaped holes of varying poses, the goal is to precisely place each shape into its corresponding hole.
Revised: Given 5 shapes and a “kit” of 5 similarly-shaped holes of varying poses, the goal is to precisely place each shape into its corresponding hole.

Original: Shapes are not unique; in the event that more than one of the same shape exists in the field, the language goal will specify a one-to-one mapping between shapes and holes, e.g. “put the orange star into the left star shape hole.”
Revised: Shapes are not unique; in the event that more than one of the same shape exists in the field, the language goal will specify a one-to-one mapping between shapes and holes, e.g. “put the orange star into the left star shape hole.”

Original: Success is determined by the fraction and degree to which placed shapes match the poses of the shape holes.
Revised: Success is determined by the fraction and degree to which placed shapes match the poses of the shape holes.

Original: Put Blocks in Bowls.
Revised: Put Blocks in Bowls.

Original: Given a number of blocks of varying colors, and a (possibly different) number of bowls of (possible different) varying colors, the goal is to place all blocks of a specified color into some bowl of a specified color.
Revised: Given a number of blocks of varying colors, and a (possibly different) number of bowls of (possible different) varying colors, the goal is to place all blocks of a specified color into some bowl of a specified color.

Original: Note that each bowl will fit at most one block, and it is guaranteed that there are a sufficient number of bowls to achieve the goal.
Revised: Note that each bowl will fit at most one block, and it is guaranteed that there are a sufficient number of bowls to achieve the goal.

Original: Success is determined by the fraction of blocks of the specified color being within any bowl of the specified color.
Revised: Success is determined by the fraction of blocks of the specified color being within any bowl of the specified color.

Original: Towers of Hanoi.
Revised: Towers of Hanoi.

Original: At each timestep , identify and move a ring shape from one of three possible pegs to another specified peg.
Revised: At each timestep , identify and move a ring shape from one of three possible pegs to another specified peg.

Original: Each episode contains 7 such timesteps, which is the ideal solution for 3-ring Towers of Hanoi.
Revised: Each episode contains 7 such timesteps, which is the ideal solution for 3-ring Towers of Hanoi.

Original: The sequence of moves is deterministically composed apriori using a DFS solver.
Revised: The sequence of moves is deterministically composed apriori using a DFS solver.

Original: Success is determined by the fraction of correct ring placements over timesteps.
Revised: Success is determined by the fraction of correct ring placements over timesteps.

Original: Packing Prepositions.
Revised: Packing Prepositions.

Original: Given 4 distractor shapes and 1 specified shape, as well as two brown boxes, the goal is to place the specified shape into the box with the specified spatial relationship.
Revised: Given 4 distractor shapes and 1 specified shape, as well as two brown boxes, the goal is to place the specified shape into the box with the specified spatial relationship.

Original: This spatial relationship is given relative to one of the 4 distractor shapes.
Revised: This spatial relationship is given relative to one of the 4 distractor shapes.

Original: For example, “pack the flower into the brown box left of the star.”
Revised: For example, “pack the flower into the brown box left of the star.”

Original: Success is binary; 1 if the shape is inside the boundaries of the correct box, and 0 otherwise.
Revised: Success is binary; 1 if the shape is inside the boundaries of the correct box, and 0 otherwise.

Original: A.2. C OMPOSITIONAL T ASKS
Revised: A.2. C OMPOSITIONAL T ASKS

Original: After training on basic tasks, we evaluate the baselines and our model on the four compositional tasks described below.
Revised: After training on basic tasks, we evaluate the baselines and our model on the four compositional tasks described below.

Original: In addition to the visual scene setup and success metric, we also describe their compositional nature and from which basic tasks we expect the model to learn the requisite building block behaviors.
Revised: In addition to the visual scene setup and success metric, we also describe their compositional nature and from which basic tasks we expect the model to learn the requisite building block behaviors.

Original: Packing Color Box.
Revised: Packing Color Box.

Original: Given 4 distractor shapes and 1 specified shape, as well as two boxes of different colors, the goal is to place the specified shape into the box with the specified color.
Revised: Given 4 distractor shapes and 1 specified shape, as well as two boxes of different colors, the goal is to place the specified shape into the box with the specified color.

Original: Shapes are still unique.
Revised: Shapes are still unique.

Original: Shapes are seen in the “packing-shapes” task and colors are seen in the “packing-box-pairs” task, as well as the ‘assembling-kits,” “put-blocks-in-bowls,” and “separating-small/large-piles” tasks.
Revised: Shapes are seen in the “packing-shapes” task and colors are seen in the “packing-box-pairs” task, as well as the ‘assembling-kits,” “put-blocks-in-bowls,” and “separating-small/large-piles” tasks.

Original: Note that at training time, the model only ever packs objects into a single brown box.
Revised: Note that at training time, the model only ever packs objects into a single brown box.

Original: Hence, this task assesses the model’s ability to disentangle and separately ground the packing action and the visual properties of the box.
Revised: Hence, this task assesses the model’s ability to disentangle and separately ground the packing action and the visual properties of the box.

Original: Success is binary; 1 if the shape is inside the boundaries of the correct box, and 0 otherwise.
Revised: Success is binary; 1 if the shape is inside the boundaries of the correct box, and 0 otherwise.

Original: Packing Location Box.
Revised: Packing Location Box.

Original: Given 4 distractor shapes and 1 specified shape, as well as two brown boxes, the goal is to place the specified shape into the box with the specified spatial relationship.
Revised: Given 4 distractor shapes and 1 specified shape, as well as two brown boxes, the goal is to place the specified shape into the box with the specified spatial relationship.

Original: For example, “pack the flower into the left brown box.”
Revised: For example, “pack the flower into the left brown box.”

Original: Shapes are still unique.
Revised: Shapes are still unique.

Original: Shapes are seen in the “packing-shapes” task and spatial descriptors are seen in the ‘assembling-kits” task.
Revised: Shapes are seen in the “packing-shapes” task and spatial descriptors are seen in the ‘assembling-kits” task.

Original: Again, the model only ever packs objects into a single brown box at training time.
Revised: Again, the model only ever packs objects into a single brown box at training time.

Original: This task assesses the model’s ability to compose spatial descriptors with the packing action.
Revised: This task assesses the model’s ability to compose spatial descriptors with the packing action.

Original: Success is binary; 1 if the shape is inside the boundaries of the correct box, and 0 otherwise.
Revised: Success is binary; 1 if the shape is inside the boundaries of the correct box, and 0 otherwise.

Original: Separating Location Piles.
Revised: Separating Location Piles.

Original: Given 50 small blocks and two square zones of the same color and size, but different positions and poses, push all the blocks into the zone with a specified spatial relationship.
Revised: Given 50 small blocks and two square zones of the same color and size, but different positions and poses, push all the blocks into the zone with a specified spatial relationship.

Original: For example, “push the pile of purple blocks into the left square.”
Revised: For example, “push the pile of purple blocks into the left square.”

Original: The small blocks and pushing action are present in the “separating-small-piles” task and spatial descriptors are seen in the ‘assembling-kits” task.
Revised: The small blocks and pushing action are present in the “separating-small-piles” task and spatial descriptors are seen in the ‘assembling-kits” task.

Original: Akin to the “packing-location-box” task, this task assesses the model’s ability to compose spatial descriptors with the pushing action.
Revised: Akin to the “packing-location-box” task, this task assesses the model’s ability to compose spatial descriptors with the pushing action.

Original: Success is determined by the fraction of total blocks within the correct zone.
Revised: Success is determined by the fraction of total blocks within the correct zone.

Original: Pushing Shapes.
Revised: Pushing Shapes.

Original: Given 4 distractor shapes and 1 specified shape, and two square zones of the same size, but different colors, positions and poses, the goal is to push the specified shape into the specified zone.
Revised: Given 4 distractor shapes and 1 specified shape, and two square zones of the same size, but different colors, positions and poses, the goal is to push the specified shape into the specified zone.

Original: Zones also have a spatial descriptor; for example, “push the green ring into the left blue square.”
Revised: Zones also have a spatial descriptor; for example, “push the green ring into the left blue square.”

Original: Shapes are no longer unique, but rather combinations of color and shape are unique (e.g. there may be two “ring” shapes in the scene, but only one “green ring” shape.)
Revised: Shapes are no longer unique, but rather combinations of color and shape are unique (e.g. there may be two “ring” shapes in the scene, but only one “green ring” shape.)

Original: The pushing action is present in the “separating-small/large-piles” tasks, spatial descriptors are seen in the ‘assembling-kits” task and shapes are seen within the “packing-shapes” task.
Revised: The pushing action is present in the “separating-small/large-piles” tasks, spatial descriptors are seen in the ‘assembling-kits” task and shapes are seen within the “packing-shapes” task.

Original: This task is highly challenging.
Revised: This task is highly challenging.

Original: The shapes pushed at test time are distinct from those seen at training time in “packing-shapes,” and the pushing action is only learned for blocks in the ‘separatingsmall/large-piles” tasks.
Revised: The shapes pushed at test time are distinct from those seen at training time in “packing-shapes,” and the pushing action is only learned for blocks in the ‘separatingsmall/large-piles” tasks.

Original: Hence, we test for zero-shot and compositional generalization to novel objects and visual properties, but also to novel actions.
Revised: Hence, we test for zero-shot and compositional generalization to novel objects and visual properties, but also to novel actions.

Original: Finally, introducing both spatial and color descriptors to the target square zones also adds a length-based dimension to the generalization properties.
Revised: Finally, introducing both spatial and color descriptors to the target square zones also adds a length-based dimension to the generalization properties.

Original: Packing Nested Prepositions.
Revised: Packing Nested Prepositions.

Original: Given 4 distractor shapes and 1 specified shape, as well as two brown boxes, the goal is to place the specified shape into the box with the specified spatial relationship.
Revised: Given 4 distractor shapes and 1 specified shape, as well as two brown boxes, the goal is to place the specified shape into the box with the specified spatial relationship.

Original: This spatial relationship is given sequentially , relative to 2 of the 4 distractor shapes.
Revised: This spatial relationship is given sequentially , relative to 2 of the 4 distractor shapes.

Original: For example, “pack the flower into the brown box left of the star right of the diamond.”
Revised: For example, “pack the flower into the brown box left of the star right of the diamond.”

Original: Shapes are still unique.
Revised: Shapes are still unique.

Original: Shapes are seen in the “packing-shapes” task and relative spatial descriptors are seen in the ‘packingprepositions” task.
Revised: Shapes are seen in the “packing-shapes” task and relative spatial descriptors are seen in the ‘packingprepositions” task.

Original: Again, the model only ever packs objects into a single brown box at training time.
Revised: Again, the model only ever packs objects into a single brown box at training time.

Original: This task assesses the model’s ability to compose relative spatial descriptors alongwith the packing action.
Revised: This task assesses the model’s ability to compose relative spatial descriptors alongwith the packing action.

Original: Success is binary; 1 if the shape is inside the boundaries of the correct box, and 0 otherwise.
Revised: Success is binary; 1 if the shape is inside the boundaries of the correct box, and 0 otherwise.

Original: A. A DDITIONAL A FFORDANCE V ISUALIZATIONS
Revised: A. A DDITIONAL A FFORDANCE V ISUALIZATIONS

Original: We present qualitative examples of PROGPORT ’s and CLIPORT ’s attention mask across four different tasks in Figures 5, 6, 7, 8.
Revised: We present qualitative examples of PROGPORT ’s and CLIPORT ’s attention mask across four different tasks in Figures 5, 6, 7, 8.

Original: PROGPORT outputs masks over every independent visual concept, while
Revised: PROGPORT outputs masks over every independent visual concept, while

Original: CLIPORT only outputs masks over the final pick and place locations.
Revised: CLIPORT only outputs masks over the final pick and place locations.

Original: For all tasks shown, PROGPORT is able to more accurately localize the target locations compared to CLIPORT .
Revised: For all tasks shown, PROGPORT is able to more accurately localize the target locations compared to CLIPORT .

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: A. M ODEL D ETAILS
Revised: A. M ODEL D ETAILS

Original: In this section, we also illustrate and further describe the architectures of our visual grounding and action modules.
Revised: In this section, we also illustrate and further describe the architectures of our visual grounding and action modules.

Original: [Figure]
Revised: [Figure]

Original: Our visual grounding modules leverage pretrained CLIP embeddings via a layer-weight initialization scheme.
Revised: Our visual grounding modules leverage pretrained CLIP embeddings via a layer-weight initialization scheme.

Original: Formally, CLIP is trained via an embedding-level contrastive objective on 400M imagecaption pairs, which we represent as tuples ( x t , l t ) .
Revised: Formally, CLIP is trained via an embedding-level contrastive objective on 400M imagecaption pairs, which we represent as tuples ( x t , l t ) .

Original: CLIP is comprised of an image encoder (ResNet-50) G and a text encoder (transformer) H , which featurize their respective components in the input, yielding a visual feature map V and a text embedding u .
Revised: CLIP is comprised of an image encoder (ResNet-50) G and a text encoder (transformer) H , which featurize their respective components in the input, yielding a visual feature map V and a text embedding u .

Original: To generate a visual embedding vector v , a spatial weighted-sum of V is performed via qkv-style attention (Vaswani et al., 2017), followed by a linear layer F :
Revised: To generate a visual embedding vector v , a spatial weighted-sum of V is performed via qkv-style attention (Vaswani et al., 2017), followed by a linear layer F :

Original: [Equation]
Revised: [Equation]

Original: where Emb q,k,v (⋅) are linear embedding layers and C is some fixed scalar.
Revised: where Emb q,k,v (⋅) are linear embedding layers and C is some fixed scalar.

Original: Image and text embeddings u , v are then orchestrated to the same representation latent space via CLIP’s contrastive training objective.
Revised: Image and text embeddings u , v are then orchestrated to the same representation latent space via CLIP’s contrastive training objective.

Original: To obtain an attention map over x t representing the language concepts described in l t , we need to maintain the spatiality of the visual feature map V .
Revised: To obtain an attention map over x t representing the language concepts described in l t , we need to maintain the spatiality of the visual feature map V .

Original: To this end, we remove the summation over pixels i,j , and introduce two 1 × 1 convolution layers, C v (initialized with the weights from linear embedding layer Emb v ) and C l (initialized with the weights from the linear layer F ).
Revised: To this end, we remove the summation over pixels i,j , and introduce two 1 × 1 convolution layers, C v (initialized with the weights from linear embedding layer Emb v ) and C l (initialized with the weights from the linear layer F ).

Original: This is illustrated in Fig.
Revised: This is illustrated in Fig.

Original: [Figure]
Revised: [Figure]

Original: The two-stream FCN encoder-decoder architecture used in our action modules borrow heavily from
Revised: The two-stream FCN encoder-decoder architecture used in our action modules borrow heavily from

Original: CLIPort.
Revised: CLIPort.

Original: However, we find that simply treating the pick or place grounding maps as an additional fourth channel to the RGB visual input results in the decoder layers ignoring the visual semantics represented by these grounding maps.
Revised: However, we find that simply treating the pick or place grounding maps as an additional fourth channel to the RGB visual input results in the decoder layers ignoring the visual semantics represented by these grounding maps.

Original: To solve this issue, we first bilinearly upsample or downsample the grounding map to the spatial resolution of different layers of the decoder.
Revised: To solve this issue, we first bilinearly upsample or downsample the grounding map to the spatial resolution of different layers of the decoder.

Original: We then directly multiply these grounding maps with the outputs of the decoder layers, explicitly encoding the notion of attention into these latent features.
Revised: We then directly multiply these grounding maps with the outputs of the decoder layers, explicitly encoding the notion of attention into these latent features.

Revised: A. E XPERIMENTS WITH P REPOSITIONAL

Revised: Task details are available in Appendix A.

Revised: [Table]