Original: StructDiffusion: Object-Centric Diffusion for Semantic Rearrangement of Novel Objects
Revised: StructDiffusion: Object-Centric Diffusion for Semantic Rearrangement of Novel Objects

Original: Anonymous Authors “Make a tower in the center of the table” “Make a short line out of these objects in the center of the table” “Make a small circle out of these objects in the center of the table” 
Revised: “Make a tower in the center of the table” “Make a short line out of these objects in the center of the table” “Make a small circle out of these objects in the center of the table” [Figure]

Original: Abstract —Robots operating in human environments must be able to rearrange objects into semantically-meaningful configurations, even if these objects are previously unseen.
Revised: Abstract —Robots operating in human environments must be able to rearrange objects into semantically-meaningful configurations, even if these objects are previously unseen.

Original: In this work, we focus on the problem of building physicallyvalid structures without step-by-step instructions.
Revised: In this work, we focus on the problem of building physicallyvalid structures without step-by-step instructions.

Original: We propose StructDiffusion, which combines a diffusion model and an object-centric transformer to construct structures out of a single RGB-D image based on high-level language goals, such as “set the table”.
Revised: We propose StructDiffusion, which combines a diffusion model and an object-centric transformer to construct structures out of a single RGB-D image based on high-level language goals, such as “set the table”.

Original: Our method shows how diffusion models can be used for complex multi-step 3D planning tasks.
Revised: Our method shows how diffusion models can be used for complex multi-step 3D planning tasks.

Original: StructDiffusion improves success rate on assembling physically-valid structures out of unseen objects by on average 16% over an existing multimodal transformer model, while allowing us to use one multi- task model to produce a wider range of different structures.
Revised: StructDiffusion improves success rate on assembling physically-valid structures out of unseen objects by on average 16% over an existing multimodal transformer model, while allowing us to use one multi- task model to produce a wider range of different structures.

Original: We show experiments on held-out objects in both simulation and on real-world rearrangement tasks.
Revised: We show experiments on held-out objects in both simulation and on real-world rearrangement tasks.

Revised: For videos and additional results, check out our website: http://weiyuliu.com/ StructDiffusion/ .

Original: I. I NTRODUCTION
Revised: I. I NTRODUCTION

Original: For robots to be successful assistants and collaborators, they must understand object structures.
Revised: For robots to be successful assistants and collaborators, they must understand object structures.

Original: Structures are everywhere in the real world: shelves are stocked, tables set, furniture assembled.
Revised: Structures are everywhere in the real world: shelves are stocked, tables set, furniture assembled.

Original: Capturing all of these relationships requires models that can reason over both object geometry and semantics at once, and also reason about physical validity when dealing with objects that have never been seen before.
Revised: Capturing all of these relationships requires models that can reason over both object geometry and semantics at once, and also reason about physical validity when dealing with objects that have never been seen before.

Original: Affiliation, address, email

Original: Building semantically meaningful structures with unseen objects requires satisfying two different sets of constraints: first, we must place objects in the correct positions to satisfy the desired spatio-semantic relations; second, we must ensure that objects are not colliding and arrangements are structurally sound.
Revised: Building semantically meaningful structures with unseen objects requires satisfying two different sets of constraints: first, we must place objects in the correct positions to satisfy the desired spatio-semantic relations; second, we must ensure that objects are not colliding and arrangements are structurally sound.

Original: We propose StructDiffusion , a single framework that jointly optimizes over these two, sometimes contrasting, constraints.
Revised: We propose StructDiffusion , a single framework that jointly optimizes over these two, sometimes contrasting, constraints.

Original: We hypothesize that by iteratively refining predicted goals subject to these learned constraints, we can better scale and generalize than if we just directly regress to a single solution from a language-conditioned model.
Revised: We hypothesize that by iteratively refining predicted goals subject to these learned constraints, we can better scale and generalize than if we just directly regress to a single solution from a language-conditioned model.

Original: We handle these constraints in two ways: first, we train a language-conditioned object-centric diffusion model from which we can simultaneously sample goal poses for multiple objects; and second, we train a discriminator model that looks at the imagined scenes to reject unrealistic samples.
Revised: We handle these constraints in two ways: first, we train a language-conditioned object-centric diffusion model from which we can simultaneously sample goal poses for multiple objects; and second, we train a discriminator model that looks at the imagined scenes to reject unrealistic samples.

Original: Compared to a language-conditioned multi-task model,
Revised: Compared to a language-conditioned multi-task model,

Original: StructDiffusion achieves a 13 .
Revised: StructDiffusion achieves a 13 .

Original: 9% higher success rate on previously unseen objects.
Revised: 9% higher success rate on previously unseen objects.

Original: Previous work in language-conditioned rearrangement has looked at 2D image representations [1], [2] instead of 3D ones.
Revised: Previous work in language-conditioned rearrangement has looked at 2D image representations [1], [2] instead of 3D ones.

Original: Other prior work which can perform some rearrangement tasks in 3D [3], [4] have limited generalization to novel objects.
Revised: Other prior work which can perform some rearrange- ment tasks in 3D [3], [4] have limited generalization to novel objects.

Original: Others, like StructFormer [5], regress to only a single solution, and cannot handle multiple challenging types of structures in a single model.
Revised: Others, like StructFormer [5], regress to only a single solution, and cannot handle multiple challenging types of structures in a single model.

Original: To our knowledge,
Revised: To our knowledge,

Original: StructDiffusion is the first work that combines a multimodal transformer with a diffusion model to generalize over different objects given high-level language instructions; our method also overcomes limitations of prior techniques regarding voxelization and workspace size [1], [2], [3].
Revised: StructDiffusion is the first work that combines a multimodal transformer with a diffusion model to generalize over different objects given high-level language instructions; our method also overcomes limitations of prior techniques regarding voxelization and workspace size [1], [2], [3].

Original: In our approach, we use unknown object instance segmentation to break our scene up into objects, as per prior work [6], [7], [8], [9].
Revised: In our approach, we use unknown object instance segmentation to break our scene up into objects, as per prior work [6], [7], [8], [9].

Original: Then, we use a multi-modal transformer to combine both word tokens and object encodings from Point Cloud Transformer [10] in order to make 6-DoF goal pose predictions.
Revised: Then, we use a multi-modal transformer to combine both word tokens and object encodings from Point Cloud Transformer [10] in order to make 6-DoF goal pose predictions.

Original: These predictions are both refined iteratively via diffusion and selected with a “discriminator” model that learns to recognize unrealistic samples.
Revised: These predictions are both refined iteratively via diffusion and selected with a “discriminator” model that learns to recognize unrealistic samples.

Original: Our planning-inspired approach has three main benefits:
Revised: Our planning-inspired approach is based on three key ideas:

Original: A object-centric transformer learns how to construct different types of  structures from observations of previously-unseen objects and language instructions; 2) A diffusion model captures a diverse distribution of semantic structures, useful for refinement and planning; 3) A learned discriminator model which dramatically improves performance by rejecting samples violating physical and structural constraints.
Revised: An object-centric transformer learns how to construct different types of multi-object structures from observations of novel objects and language instructions; 2) A diffusion model captures a diverse distribution of semantic structures, useful for refinement and planning; 3) A learned discriminator model which dramatically improves performance by rejecting samples violating physical and structural constraints.

Original: To summarize, we propose the first diffusion-based planner for rearrangement of previously-unseen objects, and demonstrate that it generates realistic samples better than baselines in both the real world andin simulation.
Revised: To summarize, we propose the first diffusion-based planner for rearrangement of previously-unseen objects, and demonstrate that it generates realistic samples better than baselines in both the real world and simulation.

Original: II. R ELATED W ORK
Revised: II. R ELATED W ORK

Original: Language-conditioned robot skill learning is a growing area of research [1], [11], [5], [3], [12].
Revised: Language-conditioned robot skill learning is a growing area of research [1], [11], [5], [3], [12].

Original: Recently.
Revised: Recently.

Original: SayCan [13] showed how a large language model (LLM) can be used to sequence robot skills to respond to a wide range of natural-language queries.
Revised: SayCan [13] showed how a large language model (LLM) can be used to sequence robot skills to respond to a wide range of natural-language queries.

Original: This has been extended to use a map and object-centric representation of the world [14].
Revised: This has been extended to use a map and object-centric representation of the world [14].

Original: A set of other works have used language to build various complex structures [15], [7], [16], [17].
Revised: A set of other works have used language to build various complex structures [15], [7], [16], [17].

Original: One thread of work looks at language-conditioned skills, but these must themselves be sequenced in order to create more complex structures or behaviors [1], [12], [3].
Revised: One thread of work looks at language-conditioned skills, but these must themselves be sequenced in order to create more complex structures or behaviors [1], [12], [3].

Original: Another looks at a purely language-based version of the world [7], [15], [13].
Revised: Another looks at a purely language-based version of the world [7], [15], [13].

Original: ProgPrompt [7] and Code-As-Policies [15] require lots of prompt engineering and object detection capabilities, but they can complete fairly complex structures.
Revised: ProgPrompt [7] and Code-As-Policies [15] require lots of prompt engineering and object detection capabilities, but they can complete fairly complex structures.

Original: Several of these works use an object-centric representation of the world [2], [14], [5], where objects are segmented or detected and encoded separately, but directly, unlike in [15], [7].
Revised: Several of these works use an object-centric representation of the world [2], [14], [5], where objects are segmented or detected and encoded separately.

Original: For example, VIMA [2] used encoded object patches as input to a multimodal transformer.
Revised: For example, VIMA [2] used encoded object patches as input to a multimodal transformer.

Original: None of these works, however, look specifically at how we can ensure we are generating physically realistic structures: in our experiments, we show how these direct-regression-first approaches do not generate the same quality of structures, and that in particular, simply predicting placement poses or actions will lead to more failures.
Revised: None of these works, however, look specifically at how we can ensure we are generating physically realistic structures: in our experiments, we show how these direct-regression-first approaches do not generate the same quality of structures, and that in particular, simply predicting placement poses or actions will lead to more failures.

Original: Several works have looked at planning with unknown objects.
Revised: Several works have looked at planning with unknown objects.

Original: Simeonov et al.
Revised: Simeonov et al.

Original: [18] propose a planning framework for rigid body objects, but they don’t look at creating structures with complex dependencies.
Revised: [18] propose a planning framework for rigid body objects, but they do not create structures with complex dependencies.

Original: Curtis et al.
Revised: Curtis et al.

Original: [19] look at task and motion planning with unknown objects, which relies on a similar segmentation and grasping pipeline to our work, but does not look at learning goals, instead assuming access to a set of predicates which can be evaluated at planning time.
Revised: [19] investigate task and motion planning with unknown objects, which relies on a similar segmentation and grasping pipeline to our work, but does not look at learning goals, instead assuming access to a set of predicates which can be evaluated at planning time.

Original: Finally, there is a set of works which learn object-object relations for planning [6], [20], [21], which is relevant to our method’s refinement process.
Revised: Finally, there is a set of works which learn object-object relations for planning [6], [20], [21], which is relevant to our method’s refinement process.

Original: Many of these do so explicitly.
Revised: Many of these do so explicitly.

Original: In particular, learning object skill preconditions is very useful for sequential manipulation, so some works look at predicting relationships in this context [22], [21], [23].
Revised: In particular, learning object skill preconditions is very useful for sequential manipulation, so some works look at predicting relationships in this context [22], [21], [23].

Original: For example, SORNet [21] learns to predict relations between objects given a canonical image view of the objects;[23] similarly learns predictive models from image inputs  for capturing relationships.
Revised: For example, SORNet [21] learns to predict relations between objects given a canonical image view of the objects; similarly a predictive model from image inputs is learned for capturing relationships [23].

Original: These kinds of relations are an important part of planning sequential manipulation as per StructDiffusion , but we look only at implicitly classifying object relationships as a whole.
Revised: These kinds of relations are an important part of planning sequential manipulation as per StructDiffusion , but we look only at implicitly classifying object relationships as a whole.

Original: Closely related to our use of diffusion model is DALLE-Bot [17], which uses StableDiffusion to generate a goal image for an arrangement of multiple objects, then uses object matching to rearrange these objects.
Revised: Closely related to our use of diffusion model is DALLE-Bot [17], which uses DALL-E to generate a goal image for an arrangement of multiple objects, then uses object matching to rearrange these objects.

Original: This method makes a common assumption [2], [1] which is that there will be an unoccluded top-down view of the scene.
Revised: This method makes a common assumption [2], [1] which is that there will be an unoccluded top-down view of the scene.

Original: This assumption is necessary for their method because they use a combination of image captioning and object recognition to convert between a diffusion image and a motion plan.
Revised: This assumption is necessary for their method because they use a combination of image captioning and object recognition to convert between a diffusion image and a motion plan.

Original: In our case, we are not restricted by which position or angle the image is generated from; we can directly refine object placement poses given point cloud observations.
Revised: In our case, we are not restricted by which position or angle the image is generated from; we can directly refine object placement poses given point cloud observations.

Original: III. P RELIMINARIES
Revised: III. P RELIMINARIES

Original: We provide background information on diffusion models [24], [25] and transformers [26].
Revised: We provide background information on diffusion models [24], [25] and transformers [26].

Original: These two neural network structures provide core components for our approach.
Revised: These two neural network structures provide core components for our approach.

Original: A. Diffusion Models
Revised: A. Diffusion Models

Original: Denoising Diffusion Models are a class of generative models [24], [25].
Revised: Denoising Diffusion Models are a class of generative models [24], [25].

Original: Given a sample x ∼ q ( x 0 ) from the data distribution.
Revised: Given a sample x ∼ q ( x 0 ) from the data distribution.

Original: The forward diffusion process is a Markov chain that creates latent variables x 1 , ..., x T by gradually adding
Revised: The forward diffusion process is a Markov chain that creates latent variables x 1 , ..., x T by gradually adding

Original: Gaussian noise to the sample:
Revised: Gaussian noise to the sample:

Original: [Equation]
Revised: [Equation]

Original: Here β t follows a fixed variance schedule such that the variance at each step is small and the total noise added to the original sample in the chain is large.
Revised: Here β t follows a fixed variance schedule such that the variance at each step is small and the total noise added to the original sample in the chain is large.

Original: These two conditions allows sampling x 0 ∼ p θ ( x 0 ) from a reverse process that starts with a Gaussian noise x T and follows a learned Gaussian posterior
Revised: These two conditions allows sampling x 0 ∼ p θ ( x 0 ) from a reverse process that starts with a Gaussian noise x T and follows a learned Gaussian posterior

Original: [Equation]
Revised: [Equation]

Original: In this work, we adopt the simplified model introduced in [27] that fixes the covariance Σ θ ( x t , t ) to an untrained time-
Revised: In this work, we adopt the simplified model introduced in [27] that fixes the covariance Σ θ ( x t , t ) to an untrained timedependent constant and reparameterize the mean µ θ ( [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: <latexit
Revised: <latexit

Original: [Table]
Revised: [Table]

Original: +53VjCK68hIVRDDQks6Ag5jZIO6vAHjBFCfCJIZgoZm61yQgrTMAUNZfii7k/JFkWSMl1WjJduYvNLJNWrepeVGt355X6dd5aER2hY3SKXHSJ6ugWNVATEaTQM3pFb9aL9W59WJ+z0YKV7xyiOVhfv347ow8=</latexit> ⇠ t <latexit sha1_base64="or/o70P+on8TkMg9ObnsnyLNTrQ=">AAACK3icbVDLSgMxFM34rPU16rKbYBEqSJkp4mNXcKMbqWAf0A4lk6ZtaDIZkoxQhln4Na4E/RZXils/wp2ZdhDbeiBwcu69nHuPHzKqtOO8W0vLK6tr67mN/ObW9s6uvbffUCKSmNSxYEK2fKQIowGpa6oZaYWSIO4z0vRHV2m9+UCkoiK41+OQeBwNAtqnGGkjde1ChyM9xIjFt0nJOYG/35vkuGsXnbIzAVwkbkaKIEOta393egJHnAQaM6RU23VC7cVIaooZSfKdSJEQ4REakLahAeJEefHkiAQeGaUH+0KaF2g4Uf9OxIgrNea+6Ux3VPO1VPyv1o50/8KLaRBGmgR4atSPGNQCponAHpUEazY2BGFJza4QD5FEWJvcZlx8PnNDnHppIZhK8iYrdz6ZRdKolN2zcuXutFi9zFLLgQI4BCXggnNQBdegBuoAg0fwBF7Aq/VsvVkf1ue0dcnKZg7ADKyvH0Ozp/k=</latexit> N (0 , I )
Revised: +53VjCK68hIVRDDQks6Ag5jZIO6vAHjBFCfCJIZgoZm61yQgrTMAUNZfii7k/JFkWSMl1WjJduYvNLJNWrepeVGt355X6dd5aER2hY3SKXHSJ6ugWNVATEaTQM3pFb9aL9W59WJ+z0YKV7xyiOVhfv347ow8=</latexit> ⇠ t <latexit sha1_base64="or/o70P+on8TkMg9ObnsnyLNTrQ=">AAACK3icbVDLSgMxFM34rPU16rKbYBEqSJkp4mNXcKMbqWAf0A4lk6ZtaDIZkoxQhln4Na4E/RZXils/wp2ZdhDbeiBwcu69nHuPHzKqtOO8W0vLK6tr67mN/ObW9s6uvbffUCKSmNSxYEK2fKQIowGpa6oZaYWSIO4z0vRHV2m9+UCkoiK41+OQeBwNAtqnGGkjde1ChyM9xIjFt0nJOYG/35vkuGsXnbIzAVwkbkaKIEOta393egJHnAQaM6RU23VC7cVIaooZSfKdSJEQ4REakLahAeJEefHkiAQeGaUH+0KaF2g4Uf9OxIgrNea+6Ux3VPO1VPyv1o50/8KLaRBGmgR4atSPGNQCponAHpUEazY2BGFJza4QD5FEWJvcZlx8PnNDnHppIZhK8iYrdz6ZRdKolN2zcuXutFi9zFLLgQI4BCXggnNQBdegBuoAg0fwBF7Aq/VsvVkf1ue0dcnKZg7ADKyvH0Ozp/k=</latexit> N ( I )

Original: Samplingdependent constant and reparameterize the mean µ θ ( x t , t ) with a noise term ϵ t .
Revised: Sampling with a noise term ϵ t .

Original: Diffusion models can be trained to minimize the variational lower bound on the negative loglikelihood E [ − log p θ ( x 0 )] .
Revised: Diffusion models can be trained to minimize the variational lower bound on the negative loglikelihood E [ − log p θ ( x 0 )] .

Original: A simplied training objective with the reparameterized mean can be derived as:
Revised: A simplied training objective with the reparameterized mean can be derived as:

Original: [Equation]
Revised: [Equation]

Original: Diffusion models have been used for motion and grasp planning in prior work [28], [29].
Revised: Diffusion models have been used for motion and grasp planning in prior work [28], [29].

Original: However, existing methods require known object models and are not conditioned on flexible language goals.
Revised: However, existing methods require known object models and are not conditioned on flexible language goals.

Original: B. Transformers
Revised: B. Transformers

Original: Transformers were proposed in [26] for modeling sequential data.
Revised: Transformers were proposed in [26] for modeling sequential data.

Original: At the heart of the Transformer architecture is the scaled dot-product attention function, which allows elements in a sequence to attend to other elements.
Revised: At the heart of the Transformer architecture is the scaled dot-product attention function, which allows elements in a sequence to attend to other elements.

Original: Specifically, an attention function takes in an input sequence { x 1 , ..., x n } and outputs a sequence of the same length { y 1 , ..., y n } .
Revised: Specifically, an attention function takes in an input sequence { x 1 , ..., x n } and outputs a sequence of the same length { y 1 , ..., y n } .

Original: Each input x i is linearly projected to a query q i , key k i , and value v i .
Revised: Each input x i is linearly projected to a query q i , key k i , and value v i .

Original: The output y i is computed as a weighted sum of the values, where the weight assigned to each value is based on the compatibility of the query with the corresponding key.
Revised: The output y i is computed as a weighted sum of the values, where the weight assigned to each value is based on the compatibility of the query with the corresponding key.

Original: In this work, we use the encoder layers in the original transformer architecture.
Revised: In this work, we use the encoder layers in the original transformer architecture.

Original: Each encoder layer includes an attention layer and a position-wise fully connected feed forward network.
Revised: Each encoder layer includes an attention layer and a position-wise fully connected feed forward network.

Original: With the use of attention mask, the encoder layer can process sequences with different lengths.
Revised: With the use of attention mask, the encoder layer can process sequences with different lengths.

Original: IV. StructDiffusion FOR O BJECT R EARRANGEMENT
Revised: IV. StructDiffusion FOR O BJECT R EARRANGEMENT

Original: Given a single view of an initial scene containing objects { o 1 , ..., o N } and a language specification containing word tokens { w 1 , ..., w M } , our goal is to rearrange the objects to reach a goal scene that satisfy the language goal.
Revised: Given a single view of an initial scene containing objects { o 1 , ..., o N } and a language specification containing word tokens { w 1 , ..., w M } , our goal is to rearrange the objects to reach a goal scene that satisfy the language goal.

Original: We assume the objects are rigid and we are given a partial-view point cloud of the scene with segment labels for points to identify the objects.
Revised: We assume the objects are rigid and we are given a partial-view point cloud of the scene with segment labels for points to identify the objects.

Original: We can extract the initial poses of the objects { ξ pc 1 , ..., ξ pcN } from the segmented object point clouds { x 1 , ..., x N } by setting the rotation to zero and the position to the centroid of each object point cloud in the world frame.
Revised: We can extract the initial poses of the objects { ξ pc 1 , ..., ξ pcN } from the segmented object point clouds { x 1 , ..., x N } by setting the rotation to zero and the position to the centroid of each object point cloud in the world frame.

Original: To rearrange the objects, the robot needs to move the objects to their respective goal poses ξ goali .
Revised: To rearrange the objects, the robot needs to move the objects to their respective goal poses ξ goali .

Original: In this work, our robot can execute pick and place actions.
Revised: In this work, our robot can execute pick and place actions.

Original: For each object, we can sample a set of stable grasps
Revised: For each object, we can sample a set of stable grasps

Original: G = { g 1 , ..., g M } .
Revised: G = { g 1 , ..., g M } .

Original: Given a target pose for object ξ goali and a stable grasp g j , the robot can move its end effector to ξ eei =
Revised: Given a target pose for object ξ goali and a stable grasp g j , the robot can move its end effector to ξ eei =

Original: ξ goali ( ξ pci ) − 1 g j to place the object at the goal pose.
Revised: ξ goali ( ξ pci ) − 1 g j to place the object at the goal pose.

Original: We only use pick and place actions in our setup to simplify the problem.
Revised: We only use pick and place actions in our setup to simplify the problem.

Original: However, our object-centric actions can be integrated with sampling-based TAMP solutions [19] to also leverage other motion primitives, such as pushing and regrasping, to reach the goal poses predicted by our system.
Revised: However, our object-centric actions can be integrated with sampling-based TAMP solutions [19] to also leverage other motion primitives, such as pushing and regrasping, to reach the goal poses predicted by our system.

Original: Below we describe our approach to sample goal poses for objects based on partial point clouds of the objects and the language goal.
Revised: Below we describe our approach to sample goal poses for objects based on partial point clouds of the objects and the language goal.

Original: Our framework combines a generator based on a diffusion model and a learned discriminator that filters invalid samples.
Revised: Our framework combines a generator based on a diffusion model and a learned discriminator that filters invalid samples.

Original: Our diffusion model is integrated with a transformer model that maintains an individual attention stream for each object.
Revised: Our diffusion model is integrated with a transformer model that maintains an individual attention stream for each object.

Original: This object-centric approach allows us to focus on learning the interactions between objects based on their geometric features as well as the grounding of abstract concepts on spatio-semantic relations between objects (e.g., large, circle, top).
Revised: This object-centric approach allows us to focus on learning the interactions between objects based on their geometric features as well as the grounding of abstract concepts on spatio-semantic relations between objects (e.g., large, circle, top).

Original: The discriminator model operates on the scene level after transforming the objects to their predicted goal poses to further reject invalid samples.
Revised: The discriminator model operates on the scene level after transforming the objects to their predicted goal poses to further reject invalid samples.

Original: A. Encoders
Revised: A. Encoders

Original: We leverage modality-specific encoders to convert the multimodal inputs to latent tokens that are later processed by the transformer network.
Revised: We leverage modality-specific encoders to convert the multimodal inputs to latent tokens that are later processed by the transformer network.

Original: Object encoder.
Revised: Object encoder.

Original: Given the segmented point cloud x i of an object o i , we learn an encoder h
Revised: Given the segmented point cloud x i of an object o i , we learn an encoder h

Original: o ( x i ) , in order to obtain the latent representation of the object.
Revised: o ( x i ) , in order to obtain the latent representation of the object.

Original: This is based on Point
Revised: This is based on Point

Original: Cloud Transformer (PCT) [10], which has been shown to be effective at shape classification and part segmentation.
Revised: Cloud Transformer (PCT) [10], which has been shown to be effective at shape classification and part segmentation.

Original: We process the centered point cloud with PCT and learn a separate multilayer perceptron (MLP) to encode the mean position of the original point cloud.
Revised: We process the centered point cloud with PCT and learn a separate multilayer perceptron (MLP) to encode the mean position of the original point cloud.

Original: Encodings from the two networks are concatenated to give h o ( x i ) .
Revised: Encodings from the two networks are concatenated to give h o ( x i ) .

Original: We rely on this
Revised: We rely on this

Original: [Figure]
Revised: [Figure]

Original: latent representation of objects for semantic, geometric, and spatial reasoning.
Revised: latent representation of objects for semantic, geometric, and spatial reasoning.

Original: Language.
Revised: Language.

Original: To map the language goal to its latent representation, we map each unique word token from the language instructions separately to an embedding with a learned mapping h w ( w i ) .
Revised: To map the language goal to its latent representation, we map each unique word token from the language instructions separately to an embedding with a learned mapping h w ( w i ) .

Original: This method helps establish a finegrained correspondence between each part of the language specification and the respective constraint on the generated structure.
Revised: This method helps establish a finegrained correspondence between each part of the language specification and the respective constraint on the generated structure.

Original: Diffusion encodings.
Revised: Diffusion encodings.

Original: Since the goal poses of objects are iteratively optimized by the diffusion model and need to feed back to the model, we use a MLP to encode the goal poses of the objects h T ( ξ goali ) .
Revised: Since the goal poses of objects are iteratively optimized by the diffusion model and need to feed back to the model, we use a MLP to encode the goal poses of the objects h T ( ξ goali ) .

Original: To compute the time-dependent
Revised: To compute the time-dependent

Original: Gaussian posterior for reverse diffusion, we combine a latent code for t in the feature channel by learning a time embedding h time ( t ) .
Revised: Gaussian posterior for reverse diffusion, we combine a latent code for t in the feature channel by learning a time embedding h time ( t ) .

Original: Positional encoding.
Revised: Positional encoding.

Original: To differentiate the multimodal data, we use a learned position embedding h pos ( i ) to indicate the position of the words and objects in input sequences and a learned type embedding h type ( υ i ) to differentiate object point clouds ( υ i = 1 ) and word tokens ( υ i = 0 ).
Revised: To differentiate the multimodal data, we use a learned position embedding h pos ( i ) to indicate the position of the words and objects in input sequences and a learned type embedding h type ( υ i ) to differentiate object point clouds ( υ i = 1 ) and word tokens ( υ i = 0 ).

Original: B. Conditional Pose Diffusion Model
Revised: B. Conditional Pose Diffusion Model

Original: Combining a diffusion model and an object-centric transformer, StructDiffusion can sample diverse yet realistic object structures while accounting for the complex constraints imposed by the object geometry and language goal.
Revised: Combining a diffusion model and an object-centric transformer, StructDiffusion can sample diverse yet realistic object structures while accounting for the complex constraints imposed by the object geometry and language goal.

Original: The conditional diffusion model predicts the goal poses for the objects ξ 0 = { ξ i } Ni starting from the last time step of the reverse diffusion process ξ T ∼ N (0 , I ) , as illustrated in
Revised: The conditional diffusion model predicts the goal poses for the objects ξ 0 = { ξ i } Ni starting from the last time step of the reverse diffusion process ξ T ∼ N (0 , I ) , as illustrated in

Original: Fig.
Revised: Fig.

Original: We use the bold symbol here because we jointly optimize the poses of all objects.
Revised: We use the bold symbol here because we jointly optimize the poses of all objects.

Original: Different from most existing diffusion models that directly generate goal images and do not explicitly model individual objects [24], [25], [17], we use the transformer model to build an object-centric representation of the scene and reason about the higher-order interactions between multiple objects.
Revised: Different from most existing diffusion models that directly generate goal images and do not explicitly model individual objects [24], [25], [17], we use the transformer model to build an object-centric representation of the scene and reason about the higher-order interactions between multiple objects.

Original: This approach allows us to account for both global constraints and local interactions between objects.
Revised: This approach allows us to account for both global constraints and local interactions between objects.

Original: Leveraging attention masks, a single transformer model can also learn to rearrange different numbers of objects.
Revised: Leveraging attention masks, a single transformer model can also learn to rearrange different numbers of objects.

Original: The use of the diffusion model helps us capture diverse structures since we are sampling from a series of Gaussian noises at different scales when going from ξ T to our goal ξ 0 .
Revised: The use of the diffusion model helps us capture diverse structures since we are sampling from a series of Gaussian noises at different scales when going from ξ T to our goal ξ 0 .

Original: The resulting samples, therefore, can be diversified at different levels of granularity (e.g., different placements of the structures and different orientations of the individual objects).
Revised: The resulting samples, therefore, is diverse at different levels of granularity (e.g., different placements of the structures and different orientations of the individual objects).

Original: The diversity is also crucial when dealing with the inherent ambiguity in language instructions.
Revised: The diversity is also crucial when dealing with the inherent ambiguity in language instructions.

Original: For example, a large circle of plates and a large circle of candles impose different constraints on the sizes of the structures because the objects being arranged have different sizes.
Revised: For example, a large circle of plates and a large circle of candles impose different constraints on the sizes of the structures because the objects being arranged have different sizes.

Original: Combining the advantages of the object-centric transformer and the diffusion model, we propose to model the conditional reverse process as
Revised: Combining the advantages of the object-centric transformer and the diffusion model, we propose to model the conditional reverse process as

Original: [Equation]
Revised: [Equation]

Original: The generation process depends on the point clouds of the objects and language instruction.
Revised: The generation process depends on the point clouds of the objects and language instruction.

Original: As discussed in III-A, we learn the time-dependent noise ϵ t , which can be used to compute ξ t .
Revised: As discussed in III-A, we learn the time-dependent noise ϵ t , which can be used to compute ξ t .

Original: We use the transformer as the backbone to predict the conditional noise ϵ θ ( ξ t , t, { x i } , { w i } ) for each object.
Revised: We use the transformer as the backbone to predict the conditional noise ϵ θ ( ξ t , t, { x i } , { w i } ) for each object.

Original: We obtain the transformer input for the language part and the object part as
Revised: We obtain the transformer input for the language part and the object part as

Original: [Equation]
Revised: [Equation]

Original: where [; ] is the concatenation at the feature dimension.
Revised: where [; ] is the concatenation at the feature dimension.

Original: The model takes in the sequence { c 1 ,t , .., c M,t , e 1 ,t , .. [Equation] , e N,t } and predicts { ϵ 1 ,t , ..., ϵ N,t } for the object poses.
Revised: The model takes in the sequence { c 1 ,t , .., c M,t , e 1 ,t , ..., e N,t } and predicts { ϵ 1 ,t , ..., ϵ N,t } for the object poses.

Original: We parameterize 6-DoF pose target ξ as ( t, R ) ∈ SE (3) .
Revised: We parameterize 6-DoF pose target ξ as ( t, R ) ∈ SE (3) .

Original: We directly predict t ∈ R 3 and predict two vectors a, b ∈ R 3 , which are used to construct the rotation matrix R ∈ SO (3) using a
Revised: We directly predict t ∈ R 3 and predict two vectors a, b ∈ R 3 , which are used to construct the rotation matrix R ∈ SO (3) using a

Original: Gram–Schmidt-like process proposed in [30].
Revised: Gram–Schmidt-like process proposed in [30].

Original: C. Discriminators
Revised: C. Discriminators

Original: Besides the generator, we can also use a learned discriminator model to further filter the predictions for realism.
Revised: Besides the generator, we can also use a learned discriminator model to further filter the predictions for realism.

Original: The discriminator works on imagined scenes, where the point clouds of objects are rigidly transformed to the respective goal poses following x goali =
Revised: The discriminator works on imagined scenes, where the point clouds of objects are rigidly transformed to the respective goal poses following x goali =

Original: ξ goali ( ξ pci ) − 1 x i .
Revised: ξ goali ( ξ pci ) − 1 x i .

Original: Here we also have the opportunity to leverage a spatial abstraction different from the one used by the generator.
Revised: Here we also have the opportunity to leverage a spatial abstraction different from the one used by the generator.

Original: The generator operates the latent object-centric representation that are suitable to imagine possible structures.
Revised: The generator operates the latent object-centric representation that are suitable to imagine possible structures.

Original: For a discriminator, the interactions between the transformed point cloud objectsdo 2: t ∼ ϵ θ ) 3: z 4: ξ β t (cid:80) ts =1 1 − β t t ) + √ β t z 5: can be directly reasoned at the point level.
Revised: For a discriminator, the interactions between the transformed point cloud objects can be directly reasoned at the point level.

Original: To maintain the ability to distinguish each individual objects, we add a one-hot encoding to each point feature.
Revised: To maintain the ability to distinguish each individual objects, we add a one-hot encoding to each point feature.

Original: In our preliminary experiment, we found that the scene-level collision model has more discrimination power than the object-centric model that operates on latent representation of objects.
Revised: In our preliminary experiment, we found that the scene-level collision model has more discrimination power than the object-centric model that operates on latent representation of objects.

Original: We explore two discriminator models.
Revised: We explore two discriminator models.

Original: The first  discriminator is learned to predict pairwise collisions between two objects from their partial point clouds.
Revised: The first collision discriminator is learned to predict pairwise collisions between two objects from their partial point clouds.

Original: The second  discriminator is learned to classify the whole multi-object structure.
Revised: The second structure discriminator is learned to classify the whole multiobject structure.

Original: Similar to the language-conditioned generator, we also condition the  discriminator such that the discriminator can learn structure-specific constraints to score the samples.
Revised: Similar to the language-conditioned generator, we also condition the structure discriminator such that the discriminator can learn structure-specific constraints to score the samples.

Original: We found that the  discriminator works better when it is only required to predict  local constraints are satisfied.
Revised: We found that the structure discriminator works better when it is only required to predict if local constraints are satisfied.

Original: Therefore, we normalize the scene point cloud and drop parts of the language instruction that specify global constraints such as where to place the structure on the table.
Revised: Therefore, we normalize the scene point cloud and drop parts of the language instruction that specify global constraints such as where to place the structure on the table.

Original: D. Planning and Inference
Revised: D. Planning and Inference

Original: In Alg. 1, we show how to combine the different components of our framework to sample object structures.
Revised: In Alg. 1, we show how to combine the different components of our framework to sample object structures.

Original: We first initialize a batch of goal poses R ∈ B × N ×
Revised: We first initialize a batch of goal poses R ∈ B × N ×

Original: (3+3+3) with random noise.
Revised: (3+3+3) with random noise.

Original: We use batch operation on a GPU to perform diffusion and transform point clouds of multiple objects for different samples.
Revised: We use batch operation on a GPU to perform diffusion and transform point clouds of multiple objects for different samples.

Original: For the discriminators, we also generate combined point cloud of objects after the diffusion process and score them in batches.
Revised: For the discriminators, we also generate combined point cloud of objects after the diffusion process and score them in batches.

Original: The ranked samples are returned.
Revised: The ranked samples are returned.

Original: Each sample corresponds to a physically and semantically valid multi-object structure that can be used by other components of the manipulation pipeline for planning.
Revised: Each sample corresponds to a physically and semantically valid multi-object structure that can be used by other components of the manipulation pipeline for planning.

Original: E. Training Details
Revised: E. Training Details

Original: To train our model, we use the dataset from [5] containing tuples ( { x i } , { T goali } ) .
Revised: To train our model, we use the dataset from [5] containing tuples ( { x i } , { T goali } ) .

Original: We train a single model for all structures where the number of examples for different classes of structures are balanced.
Revised: We train a single model for all structures where the number of examples for different classes of structures are balanced.

Original: We use a batch size ofand train the diffusion model on a single RTX3090 GPU for about 12 hours.
Revised: We use a batch size ofand train the diffusion model on a single RTX3090 GPU for about 12 hours.

Original: To train the collision discriminator, we randomly sample 100 , 000 configurations of objects.
Revised: To train the collision discriminator, we randomly sample 100 , 000 configurations of objects.

Original: For the structure discriminator, we generate negative examples by randomly perturbing the ground truth target poses ξ goali .
Revised: For the structure discriminator, we generate negative examples by randomly perturbing the ground truth target poses ξ goali .

Original: For each negative example, we also randomly select a set of objects to perturb so that there are negative examples that have different number of objects out of place.
Revised: For each negative example, we also randomly select a set of objects to perturb so that there are negative examples that have different number of objects out of place.

Original: We augment the data by using point clouds from different time steps of the rearrangement sequence to create the imagined scene as they usually create different occlusions for objects.
Revised: We augment the data by using point clouds from different time steps of the rearrangement sequence to create the imagined scene as they usually create different occlusions for objects.

Original: V. S IMULATION E XPERIMENTS
Revised: V. S IMULATION E XPERIMENTS

Original: We first evaluate our method in simulation.
Revised: We first evaluate our method in simulation.

Original: A. Baselines
Revised: A. Baselines

Original: • StructFormer :
Revised: • StructFormer :

Original: This baseline uses a multimodal and object-centric transformer network to generate multiobject structures based on segmented object point clouds and language instructions [5].
Revised: This baseline uses a multimodal and object-centric transformer network to generate multiobject structures based on segmented object point clouds and language instructions [5].

Original: The transformer network autoregressively predicts the goal poses of each object.
Revised: The transformer network autoregressively predicts the goal poses of each object.

Original: We follow the original work to train a separate model for each class of structure.
Revised: We follow the original work to train a separate model for each class of structure.

Original: • CVAE : This baseline is based on a conditional variational autoencoder (CVAE) model.
Revised: • CVAE : This baseline is based on a conditional variational autoencoder (CVAE) model.

Original: CVAEs have been used to capture the different modes for multi-task learning and language-conditioned manipulation [11], [12].
Revised: CVAEs have been used to capture the different modes for multi-task learning and language-conditioned manipulation [11], [12].

Original: We introduce a CVAE that uses the object-centric transformer backbone as a strong baseline for semantic rearrangement.
Revised: We introduce a CVAE that uses the object-centric transformer backbone as a strong baseline for semantic rearrangement.

Original: To prevent the latent variable being ignored when combining the transformer with CVAE, the transformer network predicts the object goal poses in a single forward pass (i e., not autoregressively).
Revised: To prevent the latent variable being ignored when combining the transformer with CVAE, the transformer network predicts the object goal poses in a single forward pass (i.e., not autoregressively).

Original: A single model is trained for four classes of structures.
Revised: A single model is trained for four classes of structures.

Original: • Optimization with Learned Discriminator : This baseline iteratively optimizes the goal poses of objects with the structure discriminator that is trained to classify valid rearranged scenes and invalid ones.
Revised: • Optimization with Learned Discriminator : This baseline iteratively optimizes the goal poses of objects with the structure discriminator that is trained to classify valid rearranged scenes and invalid ones.

Original: This general approach has been used extensively for learning language-conditioned manipulation from offline data [31], grasping [32], [33], and predicting stable placements of objects [6], but not for language-conditioned multi-object rearrangement.
Revised: This general approach has been used extensively for learning language-conditioned manipulation from offline data [31], grasping [32], [33], and predicting stable placements of objects [6], but not for language-conditioned multi-object rearrangement.

Original: We uses the cross-entropy method for optimization [34].
Revised: We uses the cross-entropy method for optimization [34].

Original: We only optimize the object poses and not the structure pose to simplify the optimization problem.
Revised: We only optimize the object poses and not the structure pose to simplify the optimization problem.

Original: We initialize the samples from the baseline generative models because initializing the variables with random values does not lead to meaningful performance.
Revised: We initialize the samples from the baseline generative models because initializing the variables with random values does not lead to meaningful performance.

Original: [Figure]
Revised: S u cc e ss R a t e 0% 25% 50% 75% 100% Structure Class Circle

Original: StructDi
Revised: Tower Line

Original: [Figure]{+Table Setting 61 73 7851  49  48 50 6148 
 StructFormer StructFormer w/ Collision Discriminator CVAE CVAE w/ Collision Discriminator+}
Revised: {+Table Setting 61 73 7851  49  48 50 6148 
 StructFormer StructFormer w/ Collision Discriminator CVAE CVAE w/ Collision Discriminator+}

Original:  usion w/o Collision Discriminator
Revised: StructDi ﬀ usion w/o Collision Discriminator

Original: Opt with StructFormer Prior
Revised: StructDi ﬀ usion S u cc e ss R a t e 0% 25% 50% 75% 100%

Original: StructFormer w/ Collision
Revised: Structure Class Circle Tower

Original: Discriminator{+Line Table Setting 61 73 7853 64 65 66 
+}
Revised: {+Line Table Setting 61 73 7853 64 65 66 
+}

Original: CVAE w/ Collision Discriminator StructDi
Revised: Opt w/ StructFormer Prior Opt w/ CVAE

Original: [Figure]
Revised: Prior

Original:  usion“small circle in the bottom le ! ” “short line in the middle” “tower in the middle le ! ” “table setting in the middle” Opt with CVAE Prior ✗ ✗ ✗ ✓ ✗ ✗ ✓ ✓ ✗ ✗ ✓ ✗ ✗ ✓ ✓ ✗ ✗ ✗ ✓ ✗ ✗ ✓ ✗ ✗ ✗ ✗ ✗ ✓ ✓ ✗ ✗ ✗
Revised: StructDi ﬀ usion

Original: B. Experimental Setup
Revised: B. Experimental Setup

Original: We evaluate all models in the PyBullet physics simulator [38].
Revised: We evaluate all models in the PyBullet physics simulator [38].

Original: Point cloud observations are rendered with
Revised: Point cloud observations are rendered with

Original: NViSII [39].
Revised: NViSII [39].

Original: We test on novel object models from both known and unknown categories as our goal is to transfer the model learned in simulation directly to real-world objects.
Revised: We test on novel object models from both known and unknown categories as our goal is to transfer the model learned in simulation directly to real-world objects.

Original: Fig.
Revised: Fig.

Original: 4 shows the testing objects, which are collected from
Revised: 4 shows the testing objects, which are collected from

Original: Google Scanned Objects [35], ReplicaCAD dataset [36], and
Revised: Google Scanned Objects [35], ReplicaCAD dataset [36], and

Original: YCB object Set [37].
Revised: YCB object Set [37].

Original: To generate the test scenes, we use the same data collection pipeline that is used to collect groundtruth data from prior work [5].
Revised: To generate the test scenes, we use the same data collection pipeline that is used to collect groundtruth data from prior work [5].

Original: This ensures that a valid rearrangement can be found for each scene.
Revised: This ensures that a valid rearrangement can be found for each scene.

Original: The set of objects and the language goal for each scene are randomly sampled.
Revised: The set of objects and the language goal for each scene are randomly sampled.

Original: Distractor objects are randomly placed in the scene to simulate occlusions.
Revised: Distractor objects are randomly placed in the scene to simulate occlusions.

Original: We report success rate for the rearrangements.
Revised: We report success rate for the rearrangements.

Original: To isolate the pose prediction problem from other components of the system (e.g., grasp sampling and motion planning), we directly place objects 3 cm above the the predicted target poses.
Revised: To isolate the pose prediction problem from other components of the system (e.g., grasp sampling and motion planning), we directly place objects 3 cm above the the predicted target poses.

Original: We checks whether the rearrangement is physically valid by running the simulation loop after placing each object.
Revised: We checks whether the rearrangement is physically valid by running the simulation loop after placing each object.

Original: We check possible collisions and intersections between objects using approximate convex decompositions of the 3D object models.
Revised: We check possible collisions and intersections between objects using approximate convex decompositions of the 3D object models.

Original: We also implement model-based classifiers to evaluate whether the rearrangement satisfy the language goal.
Revised: We also implement model-based classifiers to evaluate whether the rearrangement satisfy the language goal.

Original: For example, we check whether the objects are in a line using the centroids of the models.
Revised: For example, we check whether the objects are in a line using the centroids of the models.

Original: A rearrangement is considered as successful if the placements of objects are not preempted due to physics-related failures and the goal scene satisfies all semantic constraints determined by the given language goal.
Revised: A rearrangement is considered as successful if the placements of objects are not preempted due to physics-related failures and the goal scene satisfies all semantic constraints determined by the given language goal.

Original: On average, there are 5 constraints for different types of structures.
Revised: On average, there are 5 constraints for different types of structures.

Original: C. Comparison with Other Generative Models
Revised: C. Comparison with Other Generative Models

Original: In Fig.
Revised: In Fig.

Original: 6, we compare with other generative models and gain insights into the generator-discriminator design of our model.
Revised: 5, we compare with other generative models and gain insights into the generator-discriminator design of our model.

Original: We see that our complete model, StructDiffusion , drastically outperformed all baselines on the tower, line,and table setting structures and obtained comparable performance on the circle structures. The improvement was S u cc e ss R a t e 0% 25% 50% 75% 100%
Revised: We see that our complete model, StructDiffusion , drastically outperformed all baselines on the tower, line,

Original: Structure Class

Original: [-Circle Tower Line Table Setting 61 73 7851  49  48 50 6148 
-]
Revised: [-Circle Tower Line Table Setting 61 73 7851  49  48 50 6148 
-][Figure]

Original: StructFormer
Revised: StructFormer

Original: StructFormer w/ Collision Discriminator CVAE

Original: CVAE w/ Collision Discriminator StructDi ﬀ usion w/o Collision Discriminator StructDi ﬀ usion S u cc e ss
Revised: [Figure]

Original: R a t e 0% 25% 50% 75% 100% Structure Class Circle Tower
Revised: and table setting structures and obtained comparable performance on the circle structures.

Original: Line

Original: [-Table Setting 61 73 7853 64 65 66 
-]
Revised: [-Table Setting 61 73 7853 64 65 66 
-]

Original: Opt w/ StructFormer Prior Opt w/ CVAE Prior StructDi ﬀ usion most significant for structures that required precise placements of objects and modeling contacts between objects.
Revised: The improvement was most significant for structures that required precise placements of objects and modeling contacts between objects.

Original: The generator-discriminator design was necessary because the diffusion model alone still generated invalid samples, especially for the line structures.
Revised: The generator-discriminator design was necessary because the diffusion model alone still generated invalid samples, especially for the line structures.

Original: The performance difference between StructDiffusion and the ablated model that does not use discriminator supports that our model can leverage the complimentary strengths of object-centric representation and scene-level representation that preserves the point-to-point interactions.
Revised: The performance difference between StructDiffusion and the ablated model that does not use discriminator supports that our model can leverage the complimentary strengths of object-centric representation and scene-level representation that preserves the point-to-point interactions.

Original: Although applying the collision discriminator also improved the performance of StructFormer and CVAE, our diffusion model benefited the most from the addition.
Revised: Although applying the collision discriminator also improved the performance of StructFormer and CVAE, our diffusion model benefited the most from the addition.

Original: We attributes this difference to the different diversities of samples from these three classes of generative models.
Revised: We attributes this difference to the different diversities of samples from these three classes of generative models.

Original: The autoregressive transformer underlying StructFormer does not explicitly model uncertainty, therefore leads to similar samples for each scene.
Revised: The autoregressive transformer underlying StructFormer does not explicitly model uncertainty, therefore leads to similar samples for each scene.

Original: The single source of stochasticity from the latent variable of the CVAE model is also not enough.
Revised: The single source of stochasticity from the latent variable of the CVAE model is also not enough.

Original: As the diffusion model incorporates uncertainties at different scales, it has the ability to both generate different classes of structures but also generate hypotheses of object placements given only partial, and even heavily occluded, point cloud of objects.
Revised: As the diffusion model incorporates uncertainties at different scales, it has the ability to both generate different classes of structures but also generate hypotheses of object placements given only partial, and even heavily occluded, point cloud of objects.

Revised: We provide qualitative comparison in Fig.

Original: D. Comparison with Other Iterative Methods
Revised: D. Comparison with Other Iterative Methods

Original: In Fig.
Revised: In Fig.

Original: [Figure] we compare StructDiffusion with other optimization-based baselines that can take advantage of the additional computational time to iteratively refine the prediction.
Revised: 6, we compare StructDiffusion with other optimization-based baselines that can take advantage of the additional computational time to iteratively refine the prediction.

Original: The result shows that StructDiffusion outperformed the other two baselines.
Revised: The result shows that StructDiffusion outperformed the other two baselines.

Original: Even though strong performance was observed when applying optimization-based method to other manipulation tasks, we do not see significant benefit in our task.
Revised: Even though strong performance was observed when applying optimization-based method to other manipulation tasks, we do not see significant benefit in our task.

Original: Looking more closely, we observe that the challenging cases that are not yet solved by the non-iterative variants are cases where the placements of objects are closely related (e
Revised: Looking more closely, we observe that the challenging cases that are not yet solved by the non-iterative variants are cases where the placements of objects are closely related (e.g., mugs tightly packed in a line without intersection, the third task in Fig. 7). In these cases, the

Original: [Figure]
Revised: [Figure]

Original: g., mugs tightly packed in a line without intersection). In these cases, the guidance from the discriminator can be ambiguous and leads to local minimal without reaching
Revised: CVAE StructDi

Original: [Figure]
Revised: [Figure]

Revised: usion w/o Collision

Original:  StructFormer 
Revised: Discriminator Opt with StructFormer Prior StructFormer w/ Collision Discriminator

Revised: CVAE w/ Collision

Revised: Discriminator StructDi

Original: [Figure]
Revised: [Figure]

Original:  valid solutions.
Revised: usion “small circle in the bottom le ! ” “short line in the middle” “tower in the middle le ! ” “table setting in the middle” Opt with CVAE Prior ✗ ✗ ✗ ✓ ✗ ✗ ✓ ✓ ✗ ✗ ✓ ✗ ✗ ✓ ✓ ✗ ✗ ✗ ✓ ✗ ✗ ✓ ✗ ✗ ✗ ✗ ✗ ✓ ✓ ✗ ✗ ✗ guidance from the discriminator can be ambiguous and leads to local minimal without reaching valid solutions.

Original: We hypothesize the leveraging guidance at different scales is necessary, as studied in a recent work that directly learns to predict scores (i.e., gradients) at different scales for 2D object rearrangement [40].
Revised: We hypothesize the leveraging guidance at different scales is necessary, as studied in a recent work that directly learns to predict scores (i.e., gradients) at different scales for 2D object rearrangement [40].

Original: Score-based method is closely related to the diffusion model  used in the work from Song and colleagues [25].
Revised: Score-based method is closely related to the diffusion model we used in this work, as shown by Song and colleagues [25].

Original: VI. R EAL W ORLD E XPERIMENTS
Revised: VI. R EAL W ORLD E XPERIMENTS

Original: We also performed a set of real world experiments on real data, including testing structure assembly on a robotic manipulation task.
Revised: We also performed a set of real world experiments on real data, including testing structure assembly on a robotic manipulation task.

Original: A. Perception and Hardware
Revised: A. Perception and Hardware

Original: We deployed our system on a 7-DoF JACO arm with an
Revised: We deployed our system on a 7-DoF JACO arm with an

Original: Asus Xtion RGB-D Camera.
Revised: Asus Xtion RGB-D Camera.

Original: We obtained segmented object point clouds by identifying clusters-of-interest through table surface detection and Euclidean distance clustering, using the
Revised: We obtained segmented object point clouds by identifying clusters-of-interest through table surface detection and Euclidean distance clustering, using the

Original: Point Cloud Library [41].
Revised: Point Cloud Library [41].

Original: We calculated antipodal grasps over each object point cloud [42], which are then ordered and executed using pairwise ranking [43].
Revised: We calculated antipodal grasps over each object point cloud [42], which are then ordered and executed using pairwise ranking [43].

Original: We used RRTConnect [44] for motion planning between predicted grasp poses.
Revised: We used RRTConnect [44] for motion planning.

Original: We released each object 3 cm above the predicated pose.
Revised: We released each object 3 cm above the predicated pose.

Original: B. Predictions for Real-World Objects
Revised: B. Predictions for Real-World Objects

Original: We show examples of the predicted structures for realworld objects in Fig.
Revised: We show examples of the predicted structures for realworld objects in Fig.

Original: These examples are created by rigidly transforming the segmented object point clouds from an initial scene with the target poses of the highest ranked structure.
Revised: These examples are created by rigidly transforming the segmented object point clouds from an initial scene with the target poses of the highest ranked structure.

Original: Even though our model is trained only on simulation data, it can be directly used to generate semantically diverse and physically valid structures for real-world objects.
Revised: Even though our model is trained only on simulation data, it can be directly used to generate semantically diverse and physically valid structures for real-world objects.

Original: our model can generate different variations of the same structure type, as shown in (A, B) .
Revised: our model can generate different variations of the same structure type, as shown in (A, B) .

Original: The same set of objects can be arranged into completely different classes of structures conditioning on the language, as shown in (A,
Revised: The same set of objects can be arranged into completely different classes of structures conditioning on the language, as shown in (A,

Original: C) and (D, E) .
Revised: C) and (D, E) .

Original: Besides changing the positions and sizes of the structures, the orientations of the structures can also be specified in language (F, G) and (H, I) .
Revised: Besides changing the positions and sizes of the structures, the orientations of the structures can also be specified in language (F, G) and (H, I) .

Original: Note that eventhough table settings in the training data are only aligned [Table] [Figure]
Revised: Note that even

Original: [Table]
Revised: [Table]

Original: C
Revised: C

Original: D
Revised: D

Original: E
Revised: E

Original: [Table]
Revised: [Table]

Original: We show the
Revised: We show the

Original: [Table]
Revised: [Table]

Original: of
Revised: of

Original: [Table]
Revised: [Table]

Original: thatvalid
Revised: that

Original: [Table]
Revised: [Table]

Revised: and

Original: and motion plans are found and that the plans are executed successfullyby the
Revised: [Table] plans are found and that the plans are [Table] successfully

Original: [Table]
Revised: [Table]

Original:  horizontally as shown in I , the use of language and training on other orientation-specific structures enable compositional generalization to a new orientation shown in H .
Revised: the [Table] though table settings in the training data are only aligned horizontally as shown in I , the use of language and training on other orientation-specific structures enable compositional generalization to a new orientation shown in H .

Original: Finally, we see non-symmetrical object (e.g., mugs, knifes, and spatulas) are correctly aligned in B, D, E, J, H .
Revised: Finally, we see non-symmetrical object (e.g., mugs, knifes, and spatulas) are correctly aligned in B, D, E, J, H .

Original: C. Rearrangement
Revised: C. Rearrangement

Original: To reliably rearrange multiple objects, we combined
Revised: To reliably rearrange multiple objects, we combined

Original: StructDiffusion with grasp and motion planning.
Revised: StructDiffusion with grasp and motion planning.

Original: We performed nested search to find the target structure to execute.
Revised: We performed nested search to find the target structure to execute.

Original: Specifically, we iterate through the generated and ranked structures.
Revised: Specifically, we iterate through the generated and ranked structures.

Original: For each structure, we sample a set of grasp poses for each object and compute corresponding pre-grasp, standoff, and placement poses based on the prediction.
Revised: For each structure, we sample a set of grasp poses for each object and compute corresponding pre-grasp, standoff, and placement poses based on the prediction.

Original: We searched for valid motion plans between these waypoints.
Revised: We searched for valid motion plans between these waypoints.

Original: If all motion plans have been found, we execute on the robot.
Revised: If all motion plans have been found, we execute on the robot.

Original: In Table I, we show success counts and average success rate for trials with different objects and different language goals.
Revised: In Table I, we show success counts and average success rate for trials with different objects and different language goals.

Original: Valid motion and grasp plans can be found most of the time due to the diverse structures generated by
Revised: Valid motion and grasp plans can be found most of the time due to the diverse structures generated by

Original: StructDiffusion .
Revised: StructDiffusion .

Original: We observed that partial point clouds due to noisy real sensor and self-occlusions for large objects led to a small number of invalid structure predictions.
Revised: We observed that partial point clouds due to noisy real sensor and self-occlusions for large objects led to a small number of invalid structure predictions.

Original: While planning, we make the assumption that the objects are rigidly attached to the gripper after grasping without slippage.
Revised: While planning, we make the assumption that the objects are rigidly attached to the gripper after grasping without slippage.

Original: This assumption generally did not hold in the real world and led to occasional failures.
Revised: This assumption generally did not hold in the real world and led to occasional failures.

Original: This assumption can be relaxed by predicting a post-grasp displacement, using learned models such as [45].
Revised: This assumption can be relaxed by predicting a post-grasp displacement, using learned models such as [45].

Original: VII. C ONCLUSIONS
Revised: VII. C ONCLUSIONS

Original: We described StructDiffusion , an approach for creating physically-valid structures using multimodal transformers and diffusion models.
Revised: We described StructDiffusion , an approach for creating physically-valid structures using multimodal transformers and diffusion models.

Original: StructDiffusion operates on point cloud images of previously-unseen objects, and can create structures for a range of language instructions.
Revised: StructDiffusion operates on point cloud images of previously-unseen objects, and can create structures for a range of language instructions.

Original: Specifically, we compared to a number of baselines, including the previous state of the art [5] and to a conditional variational autoencoder.
Revised: Specifically, we compared to a number of baselines, including the previous state of the art [5] and to a conditional variational autoencoder.

Original: End-to-end policies do not perform as well, because they cannot refine placement poses that are nearly correct.
Revised: End-to-end policies do not perform as well, because they cannot refine placement poses that are nearly correct.

Original: Using diffusion models for sampling and a trained discriminator to refine object poses significantly improves performance.
Revised: Using diffusion models for sampling and a trained discriminator to refine object poses significantly improves performance.

Original: In this work, we did not look at optimally planning.
Revised: In this work, we did not look at optimally planning.

Original: In the future, we could look at combining this approach with task and motion planning for unknown objects, as in [19].
Revised: In the future, we could look at combining this approach with task and motion planning for unknown objects, as in [19].

Original: In addition, in the future, we could apply our work to a wider range of structures.
Revised: In addition, in the future, we could apply our work to a wider range of structures.

Revised: ACKNOWLEDGMENT

Revised: We thank Yilun Du for discussions about the use of diffusion models