Original: S EQUENTIAL B RICK A SSEMBLY WITH E FFICIENT C ONSTRAINT S ATISFACTION
Revised: S EQUENTIAL B RICK A SSEMBLY WITH E FFICIENT C ONSTRAINT S ATISFACTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We address the problem of generating a sequence of LEGO brick assembly with high-fidelity structures, satisfying physical constraints between bricks.
Revised: We tackle the problem of assembling LEGO bricks, which can be considered as an instance of combinatorial optimization problems.

Original: The assembly problem is challenging since the number of possible structures increases exponentially with the number of available bricks, complicating the physical constraints to satisfy across bricks.
Revised: Such a problem is challenging, since the number of possible structures increases exponentially with the number of available bricks due to complex physical constraints between bricks.

Original: To tackle this problem, our method performs a brick structureassessment to predict the next brick position and its confidence by employing a U-shaped sparse 3D convolutional network.
Revised: To solve this problem, our method assesses a brick structure to predict the next brick position and its confidence by employing a U-shaped sparse 3D convolutional network.

Original: The convolution filter efficiently validates  physical constraints in a parallelizable and scalable manner, allowing to processof different brick types.
Revised: A convolution filter efficiently validates the physical constraints in a parallelizable and scalable manner and effectively allows us to process different brick types.

Original: To generate a novel structure, we devise a sampling strategy to determine the next brick position by considering attachable positions under  physical constraints.
Revised: To generate a novel structure, we devise a sampling strategy to determine the next brick position by considering attachable positions under the physical constraints.

Original: Instead of using handcrafted brick assembly datasets, our model is trained with a large number of 3D objects that allow to create a new high-fidelity structure.
Revised: Instead of using handcrafted brick assembly datasets, our model is trained with a large number of 3D objects that enable to create a new high-fidelity structure.

Original: We demonstrate that our method successfully generates diverse brick structures while handling two different brick types and outperforms existing methods based on Bayesian optimization,  graph generative model, and reinforcement learning, all of which are limited to a single brick type. 
Revised: We demonstrate that our method successfully generates diverse brick structures while handling two different brick types and outperforms existing methods based on Bayesian optimization, deep graph generative model, and reinforcement learning, all of which consider assembling a single brick type. To show the validity of our method, elaborate studies on various assembly scenarios are also presented.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Most real-world 3D structures are constructed with smaller primitives .
Revised: Most real-world 3D structures are constructed with smaller primitives .

Original: A broad range of studies have tackled an interesting assembly problem such as molecule generation (Ertl et al., 2017; Neil et al., 2018;
Revised: A broad range of studies have tackled an interesting assembly problem such as molecule generation (Ertl et al., 2017; Neil et al., 2018;

Original: You et al., 2018), building construction (Talton et al., 2011; Martinovic & Van Gool, 2013;
Revised: You et al., 2018), building construction (Talton et al., 2011; Martinovic & Van Gool, 2013;

Original: Ritchie et al., 2015), and part assembly generation (Sung et al., 2017; Lee et al., 2021; Jones et al., 2021).
Revised: Ritchie et al., 2015), and part assembly generation (Sung et al., 2017; Lee et al., 2021; Jones et al., 2021).

Original: However, if a primitive unit (e.g., LEGO bricks) is used to construct a 3D structure, it is thought of as an instance of combinatorial optimization problems.
Revised: However, if a primitive unit (e.g., LEGO bricks) is used to construct a 3D structure, it is thought of as an instance of combinatorial optimization problems.

Original: In particular, the search space of the primitive assembly problem increases exponentially as a search depth increases: given n primitives with k possible combinations, the search space increases as O ( n k ) .
Revised: In particular, the search space of the primitive assembly problem increases exponentially as a search depth increases: given n primitives with k possible combinations, the search space increases as O ( n k ) .

Original: In addition, supposing that a constraint between unit primitives should be satisfied, the problem becomes more challenging.
Revised: In addition, supposing that a constraint between unit primitives should be satisfied, the problem becomes more challenging.

Original: In this work, we tackle the problem of assembling LEGO bricks into a 3D structure.
Revised: In this work, we tackle the problem of assembling LEGO bricks into a 3D structure.

Original: The generation problem of brick assembly is defined as a sequential decision-making process that sequentially appends a new brick by determining the brick type, position, and direction of the new brick.
Revised: The generation problem of brick assembly is defined as a sequential decision-making process that sequentially appends a new brick by determining the brick type, position, and direction of the new brick.

Original: Moreover, a brick to assemble has to satisfy the physical constraints related to the disallowance of overlap, no isolated bricks, and LEGO-like brick connections; see Section 2 for the details of such constraints.
Revised: Moreover, a brick to assemble has to satisfy the physical constraints related to the disallowance of overlap, no isolated bricks, and LEGO-like brick connections; see Section 2 for the details of such constraints.

Original: Compared to general 3D generation methods such as previous work (Wu et al., 2016; Gadelha et al., 2018; Achlioptas et al., 2018), the advantages of sequential brick assembly as a generation method are that it can generate a structure in an open space and provide instructions to build the structure.
Revised: Compared to general 3D generation methods such as previous work (Wu et al., 2016; Gadelha et al., 2018; Achlioptas et al., 2018), the advantages of sequential brick assembly as a generation method are that it can generate a structure in an open space and provide instructions to build the structure.

Original: Several attempts have been conducted on the generation of sequential brick assembly by utilizing
Revised: Several attempts have been conducted on the generation of sequential brick assembly by utilizing

Original: Bayesian optimization (Kim et al., 2020), deep graph generative model (Thompson et al., 2020), and reinforcement learning (Chung et al., 2021).
Revised: Bayesian optimization (Kim et al., 2020), deep graph generative model (Thompson et al., 2020), and reinforcement learning (Chung et al., 2021).

Original: Those methods are capable of assembling 2 × 4 LEGO bricks with consideration of the physical constraints.
Revised: Those methods are capable of assembling 2 × 4 LEGO bricks with consideration of the physical constraints.

Original: However, the previous literature has several limitations.
Revised: However, the previous literature has several limitations.

Original: A Bayesian optimization method proposed by Kim et al.
Revised: A Bayesian optimization method proposed by Kim et al.

Original: (2020) requires heavy computations due to their iterative optimization process for each brick position.
Revised: (2020) requires heavy computations due to their iterative optimization process for each brick position.

Original: Thompson et al.
Revised: Thompson et al.

Original: (2020) propose to use masks  along with their method, but the utilization of masks degrades assembly performance.
Revised: (2020) propose to use masks to filter out invalid actions along with their method, but the utilization of masks degrades assembly performance.

Original: To predict a valid action, Chung et al.
Revised: To predict a valid action, Chung et al.

Original: (2021) proposeto employ a neural network,
Revised: (2021) propose

Original: [Table]
Revised: [Table]

Original:  which fails to predict proper moves perfectly.
Revised: to employ a neural network, which fails to predict proper moves perfectly.

Original: More importantly, these methods share common limitations: they only use a single brick type, i.e., 2 × 4 LEGO brick, and inevitably become slower  as the number of bricks increases due toan exponentially increasing search space.
Revised: More importantly, these methods share common limitations: they only consider using a single brick type (i.e., 2 × 4 brick) due to their inherent design choices or technical difficulties, and inevitably become slower in validating the constraints as the number of bricks increases due to exponentially increasing search spaces.

Original: To tackle the limitations above, we devise a novel brick assembly method with a U-shaped sparse 3D convolutional neural network utilizing a convolution filter to validate complex brick constraints efficiently.
Revised: To tackle the limitations above, we devise a novel brick assembly method with a U-shaped sparse 3D convolutional neural network utilizing a convolution filter to validate complex brick constraints efficiently.

Original: By a sampling procedure proposed in this work, our method can create diverse assembly sequences of structures in a training stage, which makes us create training episodes.
Revised: By a sampling procedure proposed in this work, our method can create diverse assembly sequences of structures in a training stage, which makes us create training episodes.

Original: Also, we use the sampling procedure to generate high-fidelity brick structures.
Revised: Also, we use the sampling procedure to generate high-fidelity brick structures.

Original: In addition, a convolution filter to validate the physical constraints enables our method to be easily parallelizable and scalable to processing different brick types.
Revised: In addition, a convolution filter to validate the physical constraints enables our method to be easily parallelizable and scalable to processing different brick types.

Original: Henceforth, we refer to our method as sequential Br ick assembly with E fficient C onstraint S atisfaction (BrECS).
Revised: Henceforth, we refer to our method as sequential Br ick assembly with E fficient C onstraint S atisfaction (BrECS).

Original: We carry out two scenarios of sequential brick assembly: completion and generation tasks.
Revised: We carry out two scenarios of sequential brick assembly: completion and generation tasks.

Revised: A completion task is to assemble bricks from a given partial structure, and a generation task is to create a brick structure from scratch; see Section 5 for their details.

Original: In the completion experiments, BrECS achieves 30.5% higher IoU scores than the best performing baseline on average, while ours is eight times faster than the baseline.
Revised: In the completion task, BrECS achieves 30.5% higher IoU scores than the best performing baseline on average (as reported in Table 2), while ours is eight times faster than the baseline.

Original: Also, in the generation experiments, BrECS performs the best with 117% higher average classification scores than the next best one.
Revised: Also, in the generation task, BrECS performs the best with 117% higher average classification scores than the next best one (as reported in Table 3).

Original: In summary, the contributions of this work are as follows:
Revised: In summary, the contributions of this work are as follows:

Original: • We propose a novel generation model for sequential brick assembly, which validates physical constraints with a convolution filter in parallel and generates high-fidelity 3D structures; • We propose a U-shaped 3D sparse convolutional network that can be trained with a voxel dataset, which enables us to assemble high-fidelity structures by exploiting a popular 3D dataset,
Revised: • We propose a novel generation model for sequential brick assembly, which validates physical constraints with a convolution filter in parallel and generates high-fidelity 3D structures; • We propose a U-shaped 3D sparse convolutional network that can be trained with a voxel dataset, which enables us to assemble high-fidelity structures by exploiting a popular 3D dataset,

Original: i
Revised: i

Original: e., ModelNet40 (Wu et al., 2015); • We show that our model successfully assembles two different brick types,
Revised: e., ModelNet40 (Wu et al., 2015); • We show that our model successfully assembles two different brick types,

Original: i
Revised: i

Original: e., 2 × 4 and 2 × 2 LEGO bricks, in various experiments.
Revised: e., 2 × 4 and 2 × 2 LEGO bricks, in various experiments.

Original: • We will release our codes and datasets upon publication.
Revised: We will release our codes and datasets upon publication.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: 3D Shape Generation.
Revised: 3D Shape Generation.

Original: For 3D shape generation, point cloud and voxel representation are widely used.
Revised: For 3D shape generation, point cloud and voxel representation are widely used.

Original: Various methods have been proposed to generate a point cloud, including methods using variational auto-encoders (Gadelha et al., 2018), generative adversarial networks (Achlioptas et al., 2018), and normalizing flows (Yang et al., 2019).
Revised: Various methods have been proposed to generate a point cloud, including methods using variational auto-encoders (Gadelha et al., 2018), generative adversarial networks (Achlioptas et al., 2018), and normalizing flows (Yang et al., 2019).

Original: On the other hand, voxel-based 3D shape generation has been studied.
Revised: On the other hand, voxel-based 3D shape generation has been studied.

Original: Wu et al.
Revised: Wu et al.

Original: (2016) propose to use a generative adversarial network to generate voxel occupancy.
Revised: (2016) propose to use a generative adversarial network to generate voxel occupancy.

Original: A method, which is suggested by Choy et al.
Revised: A method, which is suggested by Choy et al.

Original: (2019), generates 3D shapes on voxel grids with a fully convolutional neural network that includes efficient sparse convolution.
Revised: (2019), generates 3D shapes on voxel grids with a fully convolutional neural network that includes efficient sparse convolution.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2021) propose a method that generates high-quality voxel-based shapes by iteratively applying 3D convolutional neural networks.
Revised: (2021) propose a method that generates high-quality voxel-based shapes by iteratively applying 3D convolutional neural networks.

Original: A point cloud means that points on the cloud are continuous, borderless, and (possibly) infinite.
Revised: A point cloud means that points on the cloud are continuous, borderless, and (possibly) infinite.

Original: In contrast, voxels are similar to pixels – coordinates are discretized and limited.
Revised: In contrast, voxels are similar to pixels – coordinates are discretized and limited.

Original: Our method employs a voxel-based generation model to generate brick-wise scores since candidates for brick positions are discrete, and the number of bricks is finite.
Revised: Our method employs a voxel-based generation model to generate brick-wise scores since candidates for brick positions are discrete, and the number of bricks is finite.

Original: Sequential Part Assembly.
Revised: Sequential Part Assembly.

Original: Unlike the sequential brick problem, a sequential part assembly problem also shares similar difficulties to the combinatorial optimization problem, i.e., exponentially increasing search spaces.
Revised: Similar to the sequential brick problem, a sequential part assembly problem also shares common difficulties to the combinatorial optimization problem, i.e., exponentially increasing search spaces.

Original: Ghasemipour et al.
Revised: Ghasemipour et al.

Original: (2022) aim to assemble multi-part objects and tackle the problem with large-scale reinforcement learning and graph-based model architecture.
Revised: (2022) aim to assemble multi-part objects and tackle the problem with large-scale reinforcement learning and graph-based model architecture.

Original: Sung et al.
Revised: Sung et al.

Original: (2017) assemble incomplete 3D parts with a part retrieval network and a position prediction network.
Revised: (2017) assemble incomplete 3D parts with a part retrieval network and a position prediction network.

Original: The retrieval model is trained with an embedding network, so corresponding parts have similar representations in a low-dimensional feature space.
Revised: The retrieval model is trained with an embedding network, so corresponding parts have similar representations in a low-dimensional feature space.

Original: Hu et al.
Revised: Hu et al.

Original: (2020) propose a model that sequentially moves a piled box into another container with different shapes.
Revised: (2020) propose a model that sequentially moves a piled box into another container with different shapes.

Original: Their approach has to predict box removal orders and box positions in a new container.
Revised: Their approach has to predict box removal orders and box positions in a new container.

Original: They tackled the problem by representing previous boxes into graphs and reinforcement learning with rewards considering physical constraints in the new container.
Revised: They tackled the problem by representing previous boxes into graphs and reinforcement learning with rewards considering physical constraints in the new container.

Original: In contrast to these methods, we tackle the sequential brick assembly problem, which involves identical bricks to create 3D structures.
Revised: In contrast to these methods, we tackle the sequential brick assembly problem, which involves identical bricks to create 3D structures.

Original: Sequential Brick Assembly.
Revised: Sequential Brick Assembly.

Original: By following previous literature (Kim et al., 2020; Thompson et al., 2020; Chung et al., 2021), we sequentially assemble given LEGO bricks into a 3D object by adding on bricks one by one.
Revised: By following previous literature (Kim et al., 2020; Thompson et al., 2020; Chung et al., 2021), we sequentially assemble given LEGO bricks into a 3D object by adding on bricks one by one.

Original: Unlike generic 3D object generation methods, e.g., previous literature (Wu et al., 2016; Achlioptas et al., 2018), these sequential assembly approaches have to consider particular physical constraints that are introduced by available connections between two adjacent LEGO bricks and the disallowance of brick overlap.
Revised: Unlike generic 3D object generation methods, e.g., previous literature (Wu et al., 2016; Achlioptas et al., 2018), these sequential assembly approaches have to consider particular physical constraints that are introduced by available connections between two adjacent LEGO bricks and the disallowance of brick overlap.

Original: As discussed in the work (Kim et al., 2020), these constraints encourage us to accentuate the nature of combinatorial optimization since there exist a huge number of possible combinations in the presence of complex physical constraints.
Revised: As discussed in the work (Kim et al., 2020), these constraints encourage us to accentuate the nature of combinatorial optimization since there exist a huge number of possible combinations in the presence of complex physical constraints.

Original:  S EQUENTIAL B RICK A SSEMBLY P ROBLEM
Revised:  S EQUENTIAL B RICK A SSEMBLY P ROBLEM

Original: Formulation.
Revised: Formulation.

Original: Similar to the reference by Chung et al.
Revised: Similar to the reference by Chung et al.

Original: (2021), a new brick has to be connected to one or (possibly) more bricks of previously assembled bricks.
Revised: (2021), a new brick has to be connected to one or (possibly) more bricks of previously assembled bricks.

Original: It implies that the position of the next brick can be determined as a relative distance from a certain brick directly connected to the next brick.
Revised: It implies that the position of the next brick can be determined as a relative distance from a certain brick directly connected to the next brick.

Original: We denote this indicator brick as a pivot brick .
Revised: We denote this indicator brick as a pivot brick .

Original: Eventually, after choosing the pivot brick, we determine a relative position from the pivot brick so we can place the new brick on the position determined.
Revised: Eventually, after choosing the pivot brick, we determine a relative position from the pivot brick so we can place the new brick on the position determined.

Original: To choose pivot brick and relative position effectively, we must consider physical constraints and scores for every possible brick position.
Revised: To choose pivot brick and relative position effectively, we must consider physical constraints and scores for every possible brick position.

Original: As the number of bricks increases, determining the pivot brick and the relative position becomes challenging because of exponentially increasing successive brick positions and complex physical constraints.
Revised: As the number of bricks increases, determining the pivot brick and the relative position becomes challenging because of exponentially increasing successive brick positions and complex physical constraints.

Original: Consequently, previous methods struggle to generate fine structures, and they tend to use a smaller number of bricks.
Revised: Consequently, previous methods struggle to generate fine structures, and they tend to use a smaller number of bricks.

Original: Furthermore, while the recent work (Kim et al., 2020; Thompson et al., 2020; Chung et al., 2021) only focuses on sequential assembly with 2 × 4 bricks, we tackle 2 × 4 and 2 × 2 bricks with a more efficient framework.
Revised: Furthermore, while the recent work (Kim et al., 2020; Thompson et al., 2020; Chung et al., 2021) only focuses on sequential assembly with 2 × 4 bricks, we tackle 2 × 4 and 2 × 2 bricks with a more efficient framework.

Original: The summary of each approach is shown in Table 1.
Revised: The summary of each approach is shown in Table 1.

Original: Constraints.
Revised: Constraints.

Original: We should take into account three physical constraints in the brick assembly problem: (i) bricks should not overlap each other ( NO - OVERLAP ); (ii) all bricks of the current structure should be connected to each other, and no isolated bricks exist ( NO - ISOLATION ); (iii) a new brick must be directly attached to the upper or lower position of other bricks ( VERTICAL ASSEMBLE ).
Revised: We should take into account three physical constraints in the brick assembly problem: (i) bricks should not overlap each other ( NO - OVERLAP ); (ii) all bricks of the current structure should be connected to each other, and no isolated bricks exist ( NO - ISOLATION ); (iii) a new brick must be directly attached to the upper or lower position of other bricks ( VERTICAL ASSEMBLE ).

Original: However, validating such constraints is not trivial.
Revised: However, validating such constraints is not trivial.

Original: To overcome these difficulties, as presented in Table 1,
Revised: To overcome these difficulties, as presented in Table 1,

Original: Kim et al.
Revised: Kim et al.

Original: (2020) sample a subset of available next brick positions, Thompson et al.
Revised: (2020) sample a subset of available next brick positions, Thompson et al.

Original: (2020) mask out invalid positions by validating all possible positions, and Chung et al.
Revised: (2020) mask out invalid positions by validating all possible positions, and Chung et al.

Original: (2021) train an auxiliary network that is trained to validate positions.
Revised: (2021) train an auxiliary network that is trained to validate positions.

Original:  P ROPOSED A PPROACH
Revised:  P ROPOSED A PPROACH

Original: We explain four steps of our method, which are illustrated in Figure 1.
Revised: We explain four steps of our method, which are illustrated in Figure 1.

Original: To sum up, we generate a brick structure under physical brick constraints by repeating four steps of the next brick position score computation (Section 4.1.1), invalid position exclusion (Section 4.1.2), pivot brick sampling (Section 4.1.3), and relative brick position determination (Section 4.1.4).
Revised: To sum up, we generate a brick structure under physical brick constraints by repeating four steps of the next brick position score computation (Section 4.1.1), invalid position exclusion (Section 4.1.2), pivot brick sampling (Section 4.1.3), and relative brick position determination (Section 4.1.4).

Original: The neural network for next brick position score computation (utilized in Section 4.1.1) is the only learnable component of our framework.
Revised: The neural network for next brick position score computation (utilized in Section 4.1.1) is the only learnable component of our framework.

Original: The details of the training procedure and loss function for the proposed network are described in Section 4.2.
Revised: The details of the training procedure and loss function for the proposed network are described in Section 4.2.

Original: The details of the procedure are presented in Algorithms 1 and 2, and they can be found in the supplement.
Revised: The details of the procedure are presented in Algorithms 1 and 2.

Original: [Figure]
Revised: [Figure]

Original: 4. E FFICIENT C ONSTRAINT S ATISFACTION
Revised: 4. E FFICIENT C ONSTRAINT S ATISFACTION

Original: In this section, we propose a novel method to tackle the challenge of satisfying the following physical constraints: NO - OVERLAP , which is validated by using convolution operations; NO - ISOLATION and
Revised: In this section, we propose a novel method to tackle the challenge of satisfying the following physical constraints: NO - OVERLAP , which is validated by using convolution operations; NO - ISOLATION and

Original: VERTICAL ASSEMBLE , which are satisfied by following the brick assembly formulation with pivot bricks and relative brick positions, presented in Chung et al. (2021).
Revised: VERTICAL ASSEMBLE , which are satisfied by following the brick assembly formulation with pivot bricks and relative brick positions, presented in Chung et al. (2021).

Original: By borrowing the concept of constraint satisfaction (Tsang, 1993), which is the problem of finding solutions that satisfy a predefined set of constraints, our method is designed to establish an approach to efficient constraint satisfaction for sequential brick assembly.
Revised: By borrowing the concept of constraint satisfaction (Tsang, 1993), which is the problem of finding solutions that satisfy a predefined set of constraints, our method is designed to establish an approach to efficient constraint satisfaction for sequential brick assembly.

Original: 4.1. P REDICTING N EXT B RICK P OSITIONS
Revised: 4.1. P REDICTING N EXT B RICK P OSITIONS

Original: Given a voxel representation of structure at step t , which is denoted as B t ∈ { 0 , 1 } a × a × a where a is the size of 3D space, we first feed the voxel representation B t into a U-shaped sparse 3D convolutional neural network, inspired by the reference (Choy et al., 2019): B ′ t =
Revised: Given a voxel representation of structure at step t , which is denoted as B t ∈ { 0 , 1 } a × a × a where a is the size of 3D space, we first feed the voxel representation B t into a U-shaped sparse 3D convolutional neural network, inspired by the reference (Choy et al., 2019): B ′ t =

Original: U-Net ( B t ) , in order to capture global and local contexts effectively and retain the same dimensionality.
Revised: U-Net ( B t ) , in order to capture global and local contexts effectively and retain the same dimensionality.

Revised: Due to its pyramidal feature extraction structure, the U-Net extracts robust features understanding multidimensional contexts.

Revised: We validate that the U-Net effectively extracts important contexts and thus improves overall performance compared to other neural networks in Table 5.

Original: Moreover, we expect that our neural network produces a likely complete or potentially next-step 3D structure, which is represented by a probability of voxel occupancy.
Revised: Moreover, we expect that our neural network produces a likely complete or potentially next-step 3D structure, which is represented by a probability of voxel occupancy.

Original: Note that the network parameters in this U-Net are only the learnable component in our framework BrECS.
Revised: Note that the network parameters in this U-Net are only the learnable component in our framework BrECS.

Original: The U-Net gives guidance on natural shapes.
Revised: The U-Net gives guidance on natural shapes.

Original: In this section, we assume that we are given pretrained U-Net, and the training procedure is described in Section 4.2.
Revised: In this section, we assume that we are given pretrained U-Net, and the training procedure is described in Section 4.2.

Original: After obtaining B ′ t ∈ R a × a × a , a score for next brick positions A t +1 ∈ R a × a × a is computed by sliding a convolution filter K ∈ R w b × d b × 1 across B ′ t : A t +1 = B ′ t ∗ K , where ∗ is a convolution operation.
Revised: After obtaining B ′ t ∈ R a × a × a , a score for next brick positions A t +1 ∈ R a × a × a is computed by sliding a convolution filter K ∈ R w b × d b × 1 across B ′ t : A t +1 = B ′ t ∗ K , where ∗ is a convolution operation.

Original: Note that we match the size of A t +1 to the size of B t by applying zero padding.
Revised: Note that we match the size of A t +1 to the size of B t by applying zero padding.

Original: In particular, the size of the convolution filter is the same as the brick size we assemble w b × d b where w b and d b are the width and depth of the brick, respectively so that we can determine the scores over all the possible positions of the next brick by aggregating the corresponding voxels where the convolution filter is applied.
Revised: In particular, the size of the convolution filter is the same as the brick size we assemble w b × d b where w b and d b are the width and depth of the brick, respectively so that we can determine the scores over all the possible positions of the next brick by aggregating the corresponding voxels where the convolution filter is applied.

Original: For example, if we use 2 × 4 bricks, the size of the convolution filter is 2 × 4 × 1 .
Revised: For example, if we use 2 × 4 bricks, the size of the convolution filter is 2 × 4 × 1 .

Original: Moreover, K is always initialized as a tensor filled with 1 without updating its values in a training stage to aggregate w b d b voxels equally with a single convolution operation.
Revised: Moreover, K is always initialized as a tensor filled with 1 without updating its values in a training stage to aggregate w b d b voxels equally with a single convolution operation.

Original: In the case of assembling a structure with r brick types, we prepare r convolution filters of the same size with r brick types and repeat the above process for each convolution filter.
Revised: In the case of assembling a structure with r brick types, we use 2 r convolution filters of the same size with r brick types and repeat the above process for each convolution filter.

Revised: In particular, for each brick type, we employ two convolutional filters (i.e., w b × d b × 1 and d b × w b × 1 ) to consider brick rotation.

Original: 4.1. F ILTERING O UT I NVALID B RICK P OSITIONS
Revised: 4.1. F ILTERING O UT I NVALID B RICK P OSITIONS

Original: As shown in Figure 1, to predict the validity of the next brick positions, a brick-sized convolution filter is applied to B t : V t =
Revised: As shown in Figure 1, to predict the validity of the next brick positions, a brick-sized convolution filter is applied to B t : V t =

Original: B t ∗ K .
Revised: B t ∗ K .

Original: The filter K is identical to the filter used in computing the score for the next brick positions.
Revised: The filter K is identical to the filter used in computing the score for the next brick positions.

Original: After applying the convolution filter across B t , all possible brickpositions are determined if the value of the position of interest is zero, which means that no overlap exists within the brick positions. As visually presented in Figure 2, the validity of each position
Revised: After applying the convolution filter across B t , all possible brick

Original: [Figure]
Revised: [Figure]

Revised: positions are determined if the value of the position of interest is zero, which means that no overlap exists within the brick positions.

Original:  in terms of overlap and the number of attachable positions with respect to the brick of interest are readily determined by applying a convolution filter of the corresponding brick type.
Revised: As visually presented in Figure 2, the validity of each position in terms of overlap and the number of attachable positions with respect to the brick of interest are readily determined by applying a convolution filter of the corresponding brick type.

Original: Using these two branches: computing A t +1 with the U-shaped network and V t , we can define a validated score for next brick positions C t +1 .
Revised: Using these two branches: computing A t +1 with the U-shaped network and V t , we can define a validated score for next brick positions C t +1 .

Original: Each entry of C t +1 is equal to its score for the next brick positions filtering out invalid brick positions due to overlaps.
Revised: Each entry of C t +1 is equal to its score for the next brick positions filtering out invalid brick positions due to overlaps.

Original: Formally, we describe C t +1 of assembling a w b × d b brick on a previously assembled brick of size w p × d p as follows:
Revised: Formally, we describe C t +1 of assembling a w b × d b brick on a previously assembled brick of size w p × d p as follows:

Original: [Equation]
Revised: [Equation]

Original:  where 1 [ V t ] ijk =0 ∀ i,j,k ∈ [ a ] is an indicator function, K ∈ R w b × d b × 1 is the same convolution filter as in Section 4.1.1, and ∗ is a convolution operation.
Revised: [Equation] where 1 [ V t ] ijk =0 ∀ i,j,k ∈ [ a ] is an indicator function, K ∈ R w b × d b × 1 is the same convolution filter as in Section 4.1.1, and ∗ is a convolution operation.

Original: Then, we assemble a structure by iteratively placing a brick on one of the positions with non-zero scores; the details of how the position is determined are described in Section 4.1.3.
Revised: Then, we assemble a structure by iteratively placing a brick on one of the positions with non-zero scores; the details of how the position is determined are described in Section 4.1.3.

Original: Since the calculation of the masked score tensor C t + uses convolution operations, we efficiently compute it with modern GPU devices.
Revised: Since the calculation of the masked score tensor C t + uses convolution operations, we efficiently compute it with modern GPU devices.

Original: Specifically, the validity check of (64 , 64 , 64) voxels without parallelization takes 4.7 seconds.
Revised: Specifically, the validity check of (64 , 64 , 64) voxels without parallelization takes 4.7 seconds.

Original: On the other hand, the validity check of (64 , 64 , 64) voxels with our method takes only 0.0021 seconds, which implies that this design serves 2,270 times faster inference on the validity check.
Revised: On the other hand, the validity check of (64 , 64 , 64) voxels with our method takes only 0.0021 seconds, which implies that this design serves 2,270 times faster inference on the validity check.

Original: In the next subsection, we describe the detailed sampling procedure for the next brick positions.
Revised: In the next subsection, we describe the detailed sampling procedure for the next brick positions.

Original: 4.1. S ELECTING P IVOT B RICKS BY S AMPLING
Revised: 4.1. S ELECTING P IVOT B RICKS BY S AMPLING

Original: To place a single brick, we need to choose one of the previously assembled bricks to attach the new brick to make the entire brick assembly connected – as mentioned in Section 2, we refer to it as a pivot brick.
Revised: To place a single brick, we need to choose one of the previously assembled bricks to attach the new brick to make the entire brick assembly connected – as mentioned in Section 2, we refer to it as a pivot brick.

Original: The motivation of our method to select a pivot brick is that a pivot brick with more attachable positions should be more preferable than one with less attachable positions, rather than choosing a pivot brick that is connected to a position with the highest score of C t +1 .
Revised: The motivation of our method to select a pivot brick is that a pivot brick with higher sum of attachable brick scores should be more preferable than one with lower sum of attachable brick scores, rather than choosing a pivot brick that is connected to a position with the highest score of C t +1 .

Original: Since a neural network tends to memorize training samples and their assembly sequences, choosing a position with the highest score fails to create novel structures.
Revised: Since a neural network tends to memorize training samples and their assembly sequences, choosing a position with the highest score fails to create novel structures.

Original: To avoid a deterministic approach based on C t +1 , we alter a method to select a pivot brick into a probabilistic method by sampling.
Revised: To avoid a deterministic approach based on C t +1 , we alter a method to select a pivot brick into a probabilistic method by sampling.

Original: To compare the number of attachable positions, we define a pivot score T ijk of the pivot of ( i, j, k ) to aggregate scores of attachable positions:
Revised: To compare the number of attachable positions, we define a pivot score T ijk of the pivot of ( i, j, k ) to aggregate scores of attachable positions:

Original: [Equation]
Revised: [Equation]

Original: where w ′ =
Revised: where w ′ =

Original: w b + w p, d ′ =
Revised: w b + w p, d ′ =

Original: d b + d p, and w p × d p is the size of a previously assembled brick.
Revised: d b + d p, and w p × d p is the size of a previously assembled brick.

Original: After computing pivot scores, we can determine the most probable pivot brick using arg max operation.
Revised: After computing pivot scores, we can determine the most probable pivot brick using arg max operation.

Original: However, as mentioned above, we employ a sampling strategy to determine a pivot brick:[Equation] [Equation] , c
Revised: However, as mentioned above, we employ a sampling strategy to determine a pivot brick:

Original: [Equation]
Revised: [Equation]

Original: ∼ p , where [ p ] ijk = p
Revised: ∼ p , where [ p ] ijk = p

Original: [Equation]
Revised: [Equation]

Original: Our sequential procedure for pivot brick predictions is inspired by Monte-Carlo tree search (MCTS)
Revised: Our sequential procedure for pivot brick predictions is inspired by Monte-Carlo tree search (MCTS)

Original: (Coulom, 2006).
Revised: (Coulom, 2006).

Original: MCTS evaluates possible actions by expanding a search tree with MonteCarlo simulations and back-ups.
Revised: MCTS evaluates possible actions by expanding a search tree with MonteCarlo simulations and back-ups.

Original: Similar to this, our method also evaluates pivot brick candidates by aggregating their attachable brick scores, which can be understood as a search tree with depth 1.
Revised: Similar to this, our method also evaluates pivot brick candidates by aggregating their attachable brick scores, which can be understood as a search tree with depth 1.

Original: Our method utilizes the U-shaped sparse 3D convolutional network to predict possible brick positions, which is analogous to a policy network of the recent MCTS method (Silver et al., 2016), which employs a neural network to predict prior distributions for search tree expansion.
Revised: Our method utilizes the U-shaped sparse 3D convolutional network to predict possible brick positions, which is analogous to a policy network of the recent MCTS method (Silver et al., 2016), which employs a neural network to predict prior distributions for search tree expansion.

Original: 4.1. D ETERMINING R ELATIVE B RICK P OSITIONS
Revised: 4.1. D ETERMINING R ELATIVE B RICK P OSITIONS

Original: After choosing a pivot brick by following a method described in Section 4.1.3, a relative brick position to the pivot brick is determined to complete the brick placement.
Revised: After choosing a pivot brick by following a method described in Section 4.1.3, a relative brick position to the pivot brick is determined to complete the brick placement.

Original: Possible relative brick positions ( x, y, z ) of assembling a w b × d b brick on a previously assembled brick of size w p × d p are inherently integer-valued positions satisfying following conditions:
Revised: Possible relative brick positions ( x, y, z ) of assembling a w b × d b brick on a previously assembled brick of size w p × d p are inherently integer-valued positions satisfying following conditions:

Original: [Equation]
Revised: [Equation]

Original: By considering the conditions described in Equation 4, we choose the relative brick position ( x, y, z ) with the highest score of C t +1 .
Revised: By considering the conditions described in Equation 4, we choose the relative brick position ( x, y, z ) with the highest score of C t +1 .

Original: A brick must be attached to the pivot brick as we add scores of every attachable bricks in Equation 2.
Revised: A brick must be attached to the pivot brick as we add scores of every attachable bricks in Equation 2.

Original: In this step, we do not employ a sampling based method due to its poor empirical assembly results.
Revised: In this step, we do not employ a sampling based method due to its poor empirical assembly results.

Original: 4. T RAINING THE S CORE F UNCTION
Revised: 4. T RAINING THE S CORE F UNCTION

Original: Similar to language modeling (Mikolov et al., 2010; Sutskever et al., 2014) and reinforcement learning (Sutton & Barto, 2018), our model also predicts a next brick position sequentially, which makes a problem challenging.
Revised: Similar to language modeling (Mikolov et al., 2010; Sutskever et al., 2014) and reinforcement learning (Sutton & Barto, 2018), our model also predicts a next brick position sequentially, which makes a problem challenging.

Original: To train such a prediction model, a pair of ground-truth state transition is required as a training sample.
Revised: To train such a prediction model, a pair of ground-truth state transition is required as a training sample.

Original: However, a final voxel occupancy is only available as ground-truth information.
Revised: However, a final voxel occupancy is only available as ground-truth information.

Original: It implies that we cannot access intermediate states explicitly.
Revised: It implies that we cannot access intermediate states explicitly.

Original: By replacing B ′ t , i.e., the output of the U-Net, with a ground-truth voxel occupancy, we generate an assembly sequence [ (cid:101) B 0 , (cid:101) B 1 , . . .
Revised: By replacing B ′ t , i.e., the output of the U-Net, with a ground-truth voxel occupancy, we generate an assembly sequence [ (cid:101) B 0 , (cid:101) B 1 , . . .

Original: , (cid:101) B T − 1 ] following the procedure of BrECS.
Revised: , (cid:101) B T − 1 ] following the procedure in this section.

Original: Generated sequences are unique and diverse , since the stochasticity is injected from the sampling strategy introduced in Section 4.1.3.
Revised: Generated sequences are unique and diverse , since the stochasticity is injected from the sampling strategy introduced in Section 4.1.3.

Original: Since there exist numerous possible sequences to assemble bricks to a certain 3D structure, a singlestep look-ahead with a pair of contiguous states, i.e., ( (cid:101) B t , (cid:101) B t +1 ) , is not enough to model practical assembly scenarios.
Revised: Since there exist numerous possible sequences to assemble bricks to a certain 3D structure, a singlestep look-ahead with a pair of contiguous states, i.e., ( (cid:101) B t , (cid:101) B t +1 ) , is not enough to model practical assembly scenarios.

Original: In addition, the training becomes unstable even though the training pairs are slightly changed.
Revised: In addition, the training becomes unstable even though the training pairs are slightly changed.

Original: To address these issues, we train our sequential model to predict a k -step lookahead state using pairs of states at step t and step t + k , i.e., ( ˜ B t , ˜ B t + k ) .
Revised: To address these issues, we train our sequential model to predict a k -step lookahead state using pairs of states at step t and step t + k , i.e., ( ˜ B t , ˜ B t + k ) .

Original: From now, we call this technique sequence skipping .
Revised: From now, we call this technique sequence skipping .

Original: We use a voxel-wise binary cross-entropy to train our model.
Revised: We use a voxel-wise binary cross-entropy to train our model.

Original: By  minimizing the voxel-wise binary cross-entropy, our model learns to predict  voxel-wise probabilities of the Bernoulli distribution.
Revised: By restricting the voxel prediction using a sigmoid function and minimizing the voxel-wise binary cross-entropy, our model learns to predict valid voxel-wise probabilities of the Bernoulli distribution.

Original: To sum up, we train our sequential prediction model as follows:
Revised: To sum up, we train our sequential prediction model as follows:

Original: We generate a sequence of brick assembly [ (cid:101) B 0 , (cid:101) B 1 , . . .
Revised: We generate a sequence of brick assembly [ (cid:101) B 0 , (cid:101) B 1 , . . .

Original: , (cid:101) B T ] by running our brick assembly method with a ground-truth voxel occupancy in a training dataset; 2.
Revised: , (cid:101) B T ] by running our brick assembly method with a ground-truth voxel occupancy in a training dataset; 2.

Original: We generate multi-step pairs from [ (cid:101) B 0 , (cid:101) B 1 , . . .
Revised: We generate multi-step pairs from [ (cid:101) B 0 , (cid:101) B 1 , . . .

Original: , (cid:101) B T ] in a sliding-window fashion, i e., { ( ˜ B t , ˜ B t + k ) } T − k t =0 ; 3.
Revised: , (cid:101) B T ] in a sliding-window fashion, i.e., { ( ˜ B t , ˜ B t + k ) } T − k t =0 ; 3.

Original: We train a transition function p θ ( ˜ B t + k | ˜ B t ) with { ( ˜ B t , ˜ B t + k ) } T − k t =0 and the voxel-wise binary cross-entropy.
Revised: We train a transition function p θ ( ˜ B t + k | ˜ B t ) with { ( ˜ B t , ˜ B t + k ) } T − k t =0 and the voxel-wise binary cross-entropy.

Original:  To decorrelate the time steps of training pairs in a batch and shorten training time, we employ a buffer throughout training to store training pairs similar to Zhang et al.
Revised: [Table] To decorrelate the time steps of training pairs in a batch and shorten training time [Table] we employ a buffer throughout training to store training pairs similar to Zhang et al.

Original: The detailed training process is described in Algorithm 2 of the appendix.[Table]
Revised: The detailed training process is described in Algorithm 2 of the appendix.

Original: Inference for Sequential Brick Assembly.
Revised: Inference for Sequential Brick Assembly.

Original: In an inference stage [Table] we determine the next brick positions by following the procedure of BrECS, given B 0 .
Revised: In an inference stage, we determine the next brick positions by following our aforementioned procedure, given B 0 .

Original: We provide a different form of Bdepending on a task, i
Revised: We provide a different form of Bdepending on a task, i

Original: [Table]
Revised: [Table]

Original: e., completion and generation [Table] and sequentially generate B t until a terminal step T .
Revised: e., completion and generation, and sequentially generate B t until a terminal step T .

Original: For a completion task, we use an intermediate state, i
Revised: For a completion task, we use an intermediate state, i

Original: [Table]
Revised: [Table]

Original: e., incomplete brick structure, as B 0 ; see Section 5.1 for the details.
Revised: e., incomplete brick structure, as B 0 .

Original: For a generation task, we sample an initial brick position from a discrete uniform distribution, i [Table] e., ( x, y, z ) ∼ U ( {− 2 , 2 } 3 ) .
Revised: For a generation task, we sample an initial brick position from a discrete uniform distribution, i.e., ( x, y, z ) ∼ U ( {− 2 , 2 } 3 ) .

Original: Then, we assemble bricks on a zerocentered voxel grid of size (64 , 64 , 64) and use the voxel occupancy of the initial brick position sampled as B 0 .
Revised: Then, we assemble bricks on a zero-centered voxel grid of size (64 , 64 , 64) and use the voxel occupancy of the initial brick position sampled as B 0 .

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: We demonstrate that our model generates diverse structures with high fidelity satisfying physical brick constraints in the experiments on the completion of brick structures (see Section 5.1), and the experiments on the generation of brick structures with distinct brick types, e.g., 2 × 4 and 2 × 2 LEGO bricks (see Section 5.2).
Revised: We demonstrate that our model generates diverse structures with high fidelity satisfying physical brick constraints in the experiments on the completion of brick structures (see Section 5.1), and the experiments on the generation of brick structures with distinct brick types, e.g., 2 × 4 and 2 × 2 LEGO bricks (see Section 5.2).

Original: Also, we thoroughly validate each component of BrECS in ablation studies.
Revised: Also, we thoroughly validate each component of BrECS in ablation studies.

Original: Dataset.
Revised: Dataset.

Original: To generate ground-truth assembly sequences and the training pairs based on the groundtruth sequences, we use the ModelNet40 dataset (Wu et al., 2015).
Revised: To generate ground-truth assembly sequences and the training pairs based on the groundtruth sequences, we use the ModelNet40 dataset (Wu et al., 2015).

Original: 3D meshes in the dataset are converted into (64 , 64 , 64) -sized voxel grids, and then they are scaled down to 1/4 of the original size to reduce the number of required bricks.
Revised: 3D meshes in the dataset are converted into (64 , 64 , 64) -sized voxel grids, and then they are scaled down to 1/4 of the original size to reduce the number of required bricks.

Original: Other details are described in the supplement.
Revised: Other details are described in the supplement.

Original: Metric.
Revised: Metric.

Original: For the completion task, we use intersection over union (IoU) to evaluate the performance:
Revised: For the completion task, we use intersection over union (IoU) to evaluate the performance:

Original: [Equation]
Revised: [Equation]

Original: (5) where B (1) , B (2) ∈ R a × a × a .
Revised: (5) where B (1) , B (2) ∈ R a × a × a .

Original: Along with IoU, we measure the ratio of valid bricks in structures of interest.
Revised: Along with IoU, we measure the ratio of valid bricks in structures of interest.

Original: In addition, we utilize a class probability of a target class, which is the softmax output of the target class, in experiments on the generation of brick structures.
Revised: In addition, we utilize a class probability of a target class, which is the softmax output of the target class, in experiments on the generation of brick structures.

Original: The probability of the target class is measured using a pretrained classifier with the ModelNet40 dataset reported.
Revised: The probability of the target class is measured using a pretrained classifier with the ModelNet40 dataset reported.

Original: Baseline Methods.
Revised: Baseline Methods.

Original: We compare the brick assembly performance of our method against Bayesian optimization (Kim et al., 2020), denoted as BayesOpt, Brick-by-Brick (Chung et al., 2021), denoted as BBB, and deep generative model of LEGO graphs (Thompson et al., 2020), denoted as DGMLG, in Tables 2 and 3.
Revised: We compare the brick assembly performance of our method against Bayesian optimization (Kim et al., 2020), denoted as BayesOpt, Brick-by-Brick (Chung et al., 2021), denoted as BBB, and deep generative model of LEGO graphs (Thompson et al., 2020), denoted as DGMLG, in Tables 2 and 3.

Original: BayesOpt optimizes brick positions to maximize intersection over union (IoU) between assembled shapes and target shapes.
Revised: BayesOpt optimizes brick positions to maximize intersection over union (IoU) between assembled shapes and target shapes.

Original: Brick-by-Brick learns to assemble bricks given multiview images of target structures using reinforcement learning.
Revised: BBB learns to assemble bricks given multi-view images of target structures using reinforcement learning.

Original: By following its formulation, we also provide three images (top, left, and front) of target structures in a test dataset.
Revised: Following its formulation, we also provide three images (top, left, and front) of target structures in a test dataset.

Original: DGMLG generates a structure by utilizing the graph representation of brick structures and a deep graph generative model.
Revised: DGMLG generates a structure by utilizing the graph representation of brick structures and a deep graph generative model.

Original: Note that our approach does not provide any guide (image or target shape) to producing a new structure.
Revised: Note that our approach does not provide any guidance (image or target shape) to produce a new structure.

Original: 5. C OMPLETION OF B RICK S TRUCTURES
Revised: 5. C OMPLETION OF B RICK S TRUCTURES

Original: We conduct our method on a completion task for a sequential brick assembly where unseen partial structures are given.
Revised: We conduct our method on a completion task for a sequential brick assembly where unseen partial structures are given.

Original: To measure the completion performance on brick assembly problems, we first  assemble LEGO bricks using voxel occupancy in a test dataset.
Revised: To measure the completion performance on brick assembly problems, we first [Table] [Figure] assemble LEGO bricks using voxel occupancy in a test dataset.

Original: Then, we remove a fraction of [Table]
Revised: Then, we remove a fraction of bricks assembled without losing connectivity between bricks and provide it as a start state B 0 .

Original: [Figure]
Revised: Each model is trained with a training dataset and then complete brick structures from the start states. We compare the completion performance by measuring IoU between ground-truth voxel occupancy and the complete brick structure. In addition, we report valid assembly ratio and inference time.

Original: As shown in Table 2, our method outperforms the other three baseline methods in terms of IoU.
Revised: As shown in Table 2, our method outperforms the other three baseline methods in terms of IoU.

Original: The results show that our method assembles high-fidelity brick structures compared to other methods, despite exhaustive constraint satisfaction.
Revised: The results show that our method assembles high-fidelity brick structures compared to other methods, despite exhaustive constraint satisfaction.

Original: Moreover, our method performs the best in valid ratio alongside BayesOpt, but ours is also the best in inference time.
Revised: Moreover, our method performs the best in valid ratio alongside BayesOpt, but ours is also the best in inference time.

Original: We additionally test our model with distinct brick types by appending 2 × 2 bricks after assembling 2 × 4 brick type first.
Revised: We additionally test our model with distinct brick types by appending 2 × 2 bricks after assembling 2 × 4 brick type first.

Original: The performance of our method is further improved by using two brick types since different brick types can fill brick positions more densely and express the fine aspect of structure in consequence.
Revised: The performance of our method is further improved by using two brick types since different brick types can fill brick positions more densely and express the fine aspect of structure in consequence.

Original: 5. G ENERATION OF B RICK S TRUCTURES
Revised: 5. G ENERATION OF B RICK S TRUCTURES

Original: As our method is a generation model, our brick assembly model can generate a brick structure that belongs to a particular category in a training dataset.
Revised: As our method is a generation model, our brick assembly model can generate a brick structure that belongs to a particular category in a training dataset.

Original: To compare the quality of generated structures semantically, we train a classifier over voxel grids with a small number of 3D convolution layers using the ModelNet40 dataset; the detailed architecture of the classifier is presented in the appendix.
Revised: To compare the quality of generated structures semantically, we train a classifier over voxel grids with a small number of 3D convolution layers using the ModelNet40 dataset; the detailed architecture of the classifier is presented in the appendix.

Original: Given a pretrained classifier over voxel grids, we measure the class probability of generated brick structures for a target class.
Revised: Given a pretrained classifier over voxel grids, we measure the class probability of generated brick structures for a target class.

Original: To feed the voxel grid of generated structure into the classifier, we match the voxel grid size of generated structure with the size of the training dataset for the classifier.
Revised: To feed the voxel grid of generated structure into the classifier, we match the voxel grid size of generated structure with the size of the training dataset for the classifier.

Original: [Table]
Revised: [Table]

Original: The quantitative results are presented in Table 3.
Revised: The quantitative results are presented in Table 3.

Original: Our method achieves the highest score in terms of class probabilities.
Revised: Our method achieves the highest score in terms of class probabilities.

Original: The results indicate that our method generates a semantically meaningful brick structure compared to other methods.
Revised: The results indicate that our method generates a semantically meaningful brick structure compared to other methods.

Original: We also emphasize that the semantic generation quality can be improved using additional 2 × 2 brick types.
Revised: We also emphasize that the semantic generation quality can be improved using additional 2 × 2 brick types.

Original: Brick structures with distinct brick types are generated by assembling 2 × 4 bricks first and then 2 × 2 LEGO bricks.
Revised: Brick structures with distinct brick types are generated by assembling 2 × 4 bricks first and then 2 × 2 LEGO bricks.

Original: This performance gain is led by the additional improvement on structure refinement that cannot be filled with a single brick type.
Revised: This performance gain is led by the additional improvement on structure refinement that cannot be filled with a single brick type.

Original: 5. A BLATION S TUDY
Revised: 5. A BLATION S TUDY

Original: We verify that each of our components in BrECS improves generation performance in the ablation study.
Revised: We verify that each of our components in BrECS improves generation performance in the ablation study.

Original: Specifically, we compare the ablated variants of our method in terms of completion performance.
Revised: Specifically, we compare the ablated variants of our method in terms of completion performance.

Original: Similar to the previous completion experiments, we complete a brick structure from the intermediate state of the unseen structure in a test dataset.
Revised: Similar to the previous completion experiments, we complete a brick structure from the intermediate state of the unseen structure in a test dataset.

Original: The results of the ablation study are presented in Table 4.
Revised: The results of the ablation study are presented in Table 4.

Original: Following the previous experiments, creating brick structures with two brick types improves the performance.
Revised: Following the previous experiments, creating brick structures with two brick types improves the performance.

Original: A validity check using a convolution filter plays a critical role in constraint satisfaction.
Revised: A validity check using a convolution filter plays a critical role in constraint satisfaction.

Original: Interestingly, the validity check also improves generation performance, i.e.,
Revised: Interestingly, the validity check also improves generation performance, i.e.,

Original: IoU, since we train the model with valid assembly sequences.
Revised: IoU, since we train the model with valid assembly sequences.

Original: Moreover, our model performance degrades significantly without sequence skipping as the number of bricks added reduces.
Revised: Moreover, our model performance degrades significantly without sequence skipping as the number of bricks added reduces.

Original: It supports our claim in Section 4.2 that the sequence skipping encourages our model BrECSto predict diverse brick positions. 
Revised: It supports our claim in Section 4.2 that the sequence skipping encourages our model BrECSto predict diverse brick positions. More detailed ablation studies can be found in the appendices.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We proposed a brick assembly method that efficiently generates high-fidelity brick structures satisfying physical constraints.
Revised: We proposed a brick assembly method that efficiently generates high-fidelity brick structures satisfying physical constraints.

Original: Our model sequentially assembles bricks by repeating pivot selection and relative position selection.
Revised: Our model sequentially assembles bricks by repeating pivot selection and relative position selection.

Original: To aggregate brick scores and validate brick positions, we employ a convolution filter.
Revised: To aggregate brick scores and validate brick positions, we employ a convolution filter.

Original: Previous brick assembly methods cannot escape from exponentially increasing search spaces since it is the nature of combinatorial optimization problems.
Revised: Previous brick assembly methods cannot escape from exponentially increasing search spaces since it is the nature of combinatorial optimization problems.

Original: However, our method is efficient where the number of bricks increases as convolution operations are easily parallelizable on
Revised: However, our method is efficient where the number of bricks increases as convolution operations are easily parallelizable on

Original: GPU devices.
Revised: GPU devices.

Original: In addition, we show that our method performs better than prior methods, as shown in quantitative and qualitative results.
Revised: In addition, we show that our method performs better than prior methods, as shown in quantitative and qualitative results.

Original: To our best knowledge, we are the first to attempt to use multiple brick types, i.e., 2 × 4 and 2 × 2 bricks, to generate novel brick structures sequentially.
Revised: To our best knowledge, we are the first to attempt to use multiple brick types, i.e., 2 × 4 and 2 × 2 bricks, to generate novel brick structures sequentially.

Original: Potential Applications.
Revised: Potential Applications.

Original: Our model can be extended to designing a building.
Revised: Our model can be thought of as an instance of combinatorial optimization problems, which is a challenging problem in computer science.

Original: The building design involves developing an overall structure and arranging materials for a building properly, which is exactly what our model is capable of. In addition, our model can be deployed to  industrial robots, enabling the robots to manufacture products without manual control or direct supervision.
Revised: Moreover, our model can be adopted to solve a real-world problem such as building design and product assembly with an industrial robot. Considering such problems, we would like to emphasize that they suffer from coping with invalid actions or physical constraints. Thus, ours can help to solve such problems by utilizing our ideas.

Original: Limitations.
Revised: Limitations.

Original: We utilize only two types of bricks for the demonstration, but we have designed that our model can use any rectangular bricks without an explicit definition of connection rules between different brick types, unlike the work (Kim et al., 2020).
Revised: We utilize only two types of bricks for the demonstration, but we have designed that our model can use any rectangular bricks without an explicit definition of connection rules between different brick types, unlike the work (Kim et al., 2020).

Original: Hence, we can increase the number of brick types for sequential brick assembly.
Revised: Hence, we can increase the number of brick types for sequential brick assembly.

Original: Future work will employ more diverse brick types and even irregular brick types since non-rectangular materials are common in real-world assembly scenarios.
Revised: Future work will employ more diverse brick types and even irregular brick types since non-rectangular materials are common in real-world assembly scenarios.

Original: Another limitation is that our performance degrades when we use two brick types simultaneously.
Revised: Another limitation is that our performance degrades when we use two brick types simultaneously.

Original: In the case of using 2 × 4 and 2 × 2 brick types, smaller bricks are used more frequently compared to a scenario with only 2 × 4 brick type.
Revised: In the case of using 2 × 4 and 2 × 2 brick types, smaller bricks are used more frequently compared to a scenario with only 2 × 4 brick type.

Original: We presume that smaller 2 × 2 bricks in-between reduce attachable positions, resulting in overall performance degradation.
Revised: We presume that smaller 2 × 2 bricks in-between reduce attachable positions, resulting in overall performance degradation.

Original: Also, we would like to consider physical stability, which is essential for real-world applications.
Revised: Also, we would like to consider physical stability, which is essential for real-world applications.

Original: A PPENDIX
Revised: A PPENDIX

Original: In the appendix, we cover the detailed descriptions missing in the main manuscript.
Revised: In the appendix, we cover the detailed descriptions missing in the main manuscript.

Original: A I MPLEMENTATION D ETAILS OF O UR M ETHOD AND B ASELINE M ODELS A. B R ECS
Revised: A I MPLEMENTATION D ETAILS OF O UR M ETHOD AND B ASELINE M ODELS A. B R ECS

Original: To train 3D voxel generation efficiently, we utilize MinkowskiEngine (Choy et al.
Revised: To train 3D voxel generation efficiently, we utilize MinkowskiEngine (Choy et al.

Original: and its 3D sparse convolution operation.
Revised: and its 3D sparse convolution operation.

Original: We train our model with fixed learning rate of 5e-4, Adam optimizer (Kingma & Ba, 2015), batch size of 32, sequence skipping with step size k = 8 , buffer size of 1024, and maximum number of bricks of 150.
Revised: We train our model with fixed learning rate of 5e-4, Adam optimizer (Kingma & Ba, 2015), batch size of 32, sequence skipping with step size k = 8 , buffer size of 1024, and maximum number of bricks of 150.

Original: The input size of our model is (64 , 64 , 64) and output size is (64 , 64 , 64) .
Revised: The input size of our model is (64 , 64 , 64) and output size is (64 , 64 , 64) .

Original: We trained the model until reaching 100k steps.
Revised: We trained the model until reaching 100k steps.

Original: A. B AYESIAN O PTIMIZATION
Revised: A. B AYESIAN O PTIMIZATION

Original: We employ Bayesian optimization to tackle a sequential brick assembly problem.
Revised: We employ Bayesian optimization to tackle a sequential brick assembly problem.

Original: We follow the setup proposed by Kim et al.
Revised: We follow the setup proposed by Kim et al.

Original: The number of bricks are limited to 160 bricks at most.
Revised: The number of bricks are limited to 160 bricks at most.

Original: Every brick position is optimized by maximizing IoU between currently assembled and ground-truth structures.
Revised: Every brick position is optimized by maximizing IoU between currently assembled and ground-truth structures.

Original: We would like to emphasize that we have to provide ground-truth voxel information to this strategy.
Revised: We would like to emphasize that we have to provide ground-truth voxel information to this strategy.

Original: A. B RICK - BY -B RICK
Revised: A. B RICK - BY -B RICK

Original: We use Brick-by-Brick as a baseline method and compare brick assembly performances.
Revised: We use Brick-by-Brick as a baseline method and compare brick assembly performances.

Original: We follow the model architecture and training setup of the model described in (Chung et al., 2021).
Revised: We follow the model architecture and training setup of the model described in (Chung et al., 2021).

Original: We also would like to emphasize that we have to provide partial ground-truth voxel information to this strategy, as the method requires 3 images of ground-truth shape to assemble target shapes.
Revised: We also would like to emphasize that we have to provide partial ground-truth voxel information to this strategy, as the method requires 3 images of ground-truth shape to assemble target shapes.

Original: The number of bricks are limited to 75 bricks at most, which is the same as original setup, due to the excessively increasing memory requirements of the model.
Revised: The number of bricks are limited to 75 bricks at most, which is the same as original setup, due to the excessively increasing memory requirements of the model.

Original: In inference, the sequence of generation is halted when the newly placed brick violates brick constraints, following the environment reset condition of the method.
Revised: In inference, the sequence of generation is halted when the newly placed brick violates brick constraints, following the environment reset condition of the method.

Original: A. D EEP G ENERATIVE M ODEL OF LEGO G RAPHS
Revised: A. D EEP G ENERATIVE M ODEL OF LEGO G RAPHS

Original: We solve brick assembly generation problem using Deep Generative Model of LEGO Graphs and compare the performance against ours.
Revised: We solve brick assembly generation problem using Deep Generative Model of LEGO Graphs and compare the performance against ours.

Original: We follow the model architecture and training setup of the method described in (Thompson et al., 2020).
Revised: We follow the model architecture and training setup of the method described in (Thompson et al., 2020).

Original: To train the graph generation model, we need graphs of target shapes.
Revised: To train the graph generation model, we need graphs of target shapes.

Original: To this end, we assemble target shapes with ground truth voxel shapes.
Revised: To this end, we assemble target shapes with ground truth voxel shapes.

Original: Then we convert the brick assemblies into graphs by representing bricks as nodes and direct connections between bricks as edges, following (Thompson et al., 2020).
Revised: Then we convert the brick assemblies into graphs by representing bricks as nodes and direct connections between bricks as edges, following (Thompson et al., 2020).

Original: We train the model for 200 epochs.
Revised: We train the model for 200 epochs.

Original: B B RICK A SSEMBLY P ROCESS
Revised: B B RICK A SSEMBLY P ROCESS

Original: We describe brick assembly process in Algorithm 1.
Revised: We describe brick assembly process in Algorithm 1.

Revised: C T RAINING

Revised: We describe brick assembly model training procedure in Algorithm 2.

Revised: D D ETAILED A RCHITECTURE OF U-N ET

Revised: We design our score prediction network inspired by Choy et al.

Original:  Input: Voxel of structure at current time step B , assembled brick position list P and brick size w b × d b
Revised: Detailed model architecture is illustrated in Figure 4. Input: Voxel of structure at current time step B , assembled brick position list P and brick size w b × d b

Original: Output: Pivot brick position ( a, b, c ) , relative next brick position ( x, y, z )
Revised: Output: Pivot brick position ( a, b, c ) , relative next brick position ( x, y, z )

Original: Initialize convolution filter K ∈ R w b × d b × 1 with 1, pivot score list T
Revised: Initialize convolution filter K ∈ R w b × d b × 1 with 1, pivot score list T

Original: = ϕ , and possible relative connections N with values satisfying the conditions of Equation
Revised: = ϕ , and possible relative connections N with values satisfying the conditions of Equation

Original: Calculate C using Equation▷ Section 4.1.
Revised: Calculate C using Equation▷ Section 4.1.

Original: for each pivot ( a, b, c ) in P do
Revised: for each pivot ( a, b, c ) in P do

Original: Calculate T abc using Equation
Revised: Calculate T abc using Equation

Original: T ← T ∪ { T abc } end for
Revised: T ← T ∪ { T abc } end for

Original: Sample pivot ( a, b, c ) ∼
Revised: Sample pivot ( a, b, c ) ∼

Original: T abc (cid:80) T ∈T T ▷ Section 4.1.
Revised: T abc (cid:80) T ∈T T ▷ Section 4.1.

Original: Relative next brick position ( x, y, z ) =
Revised: Relative next brick position ( x, y, z ) =

Original: arg max ( a,b,c ) ∈N C a + x,b + y,c
Revised: arg max ( a,b,c ) ∈N C a + x,b + y,c

Original: + z ▷ Section 4.1.
Revised: + z ▷ Section 4.1.

Original: C T RAINING

Original: We describe brick assembly model training procedure in Algorithm 2.

Original: Algorithm

Original: Input: Dataset D , mini-batch size M , sequence skipping value k
Revised: Input: Dataset D , mini-batch size M , sequence skipping value k

Original: Output: Brick assembly model parameter θ
Revised: Output: Brick assembly model parameter θ

Original: Generate ground-truth brick assembly sequences (cid:101) B ( i ) 0: T and store in D s using voxel shapes x i ∈ D
Revised: Generate ground-truth brick assembly sequences (cid:101) B ( i ) 0: T and store in D s using voxel shapes x i ∈ D

Original: Initialize buffer B with ( (cid:101) B ( j ) 0: T , x j , 0) where x j ∼ D , (cid:101) B ( j ) 0:
Revised: Initialize buffer B with ( (cid:101) B ( j ) 0: T , x j , 0) where x j ∼ D , (cid:101) B ( j ) 0:

Original: T ∈ D s repeat
Revised: T ∈ D s repeat

Original: L =
Revised: L =

Original: Sample and remove mini-batch { ( (cid:101) B ( i ) 0: T , x i , t i ) } Mi =1 from B for ( (cid:101) B ( i ) 0: T , x i , t i ) in a mini-batch do
Revised: Sample and remove mini-batch { ( (cid:101) B ( i ) 0: T , x i , t i ) } Mi =1 from B for ( (cid:101) B ( i ) 0: T , x i , t i ) in a mini-batch do

Original: L ← L + log p θ ( (cid:101) B ( i )
Revised: L ← L + log p θ ( (cid:101) B ( i )

Original: t i + k |
Revised: t i + k |

Original: (cid:101) B
Revised: (cid:101) B

Original: ( i ) t i )
Revised: ( i ) t i )

Original: if t i + k = T then
Revised: if t i + k = T then

Original: Push ( (cid:101) B ( j ) 0: T , x j , 0) into B where x j ∼ D , (cid:101) else
Revised: Push ( (cid:101) B ( j ) 0: T , x j , 0) into B where x j ∼ D , (cid:101) B ( j ) 0:

Original:  D D ETAILED A RCHITECTURE OF U-N ET
Revised: T ∈ D s else

Original: We design our score prediction network inspired by Choy et al.{+B ← B ∪ { ( (cid:101) B ( i ) 
+}
Revised: {+B ← B ∪ { ( (cid:101) B ( i ) 
+}

Original: Detailed model architecture is illustrated in Figure 4.
Revised: : T , x i , t i + 1) } end if end for θ ← θ + η ∂ L ∂θ until convergence

Original: E D ETAILED A RCHITECTURE OF V OXEL C LASSIFIER
Revised: E D ETAILED A RCHITECTURE OF V OXEL C LASSIFIER

Original: We train a shallow voxel classification network to compare assembly generation quality semantically.
Revised: We train a shallow voxel classification network to compare assembly generation quality semantically.

Original: We add softmax layer at the end of the network to make output logit to be a distribution.
Revised: We add softmax layer at the end of the network to make output logit to be a distribution.

Original: Detailed network structure is described in Figure 5.
Revised: Detailed network structure is described in Figure 5.

Revised: F A BLATION S TUDY ON M ODEL A RCHITECTURES

Revised: We compare different model architectures for a score prediction model.

Revised: Specifically, we compare

Revised: U-Net and fully convolutional networks.

Original: [Figure]
Revised: For a fair comparison, we use the same number of convolutional filters. We present the results of generation and completion on Table 5. We find that the

Revised: U-Net performs better than fully convolutional network in both generation and completion.

Original:  S p a r s e C o nv 3x3x ,
Revised: This is due to the robust feature extraction from pyramidal structure of U-Net. G M ODEL P ERFORMANCE WITHOUT S TOCHASTICITY We carry out ablation study on the effects of stochasticity in a brick position selection process and compare completion and generation performance against the original model. The quantitative results are presented in Table 6.

Revised: Stochasticity improves generation performance as expected.

Original: S p a r s e C o nv 3x3x ,
Revised: Interestingly, a completion quality is also improved by stochasticity. In the completion task, we remove half of [Figure] model architecture of U-Net.

Original: S p a r s e C o nv 3x3x ,
Revised: S p a r s e C onv 3x3x3 ,

Original: S p a r s e C o nv 3x3x , S p a r s e C o nv 3x3x ,
Revised: S p a r s e C onv 3x3x3 , S p a r s e C onv 3x3x3 , S p a r s e C onv 3x3x3 ,

Original: S p a r s e C o nv 3x3x , S p a r s e C on v 3x3x ,
Revised: S p a r s e C onv 3x3x3 , S p a r s e C onv 3x3x3 , S p a r s e C onv 3x3x3 ,

Original: M a x P oo l
Revised: M a x P oo l

Original: M a x P oo l M a x P oo l
Revised: M a x P oo l M a x P oo l

Original: M a x P oo l M a x P oo l 
Revised: M a x P oo l M a x P oo l A vg P oo l

Revised: C on ca t F C , 512 F C , N s o f t m a x the bricks and the model has to complete partial assembly results, which lose the majority of their information.

Original: A vg
Revised: Stochasticity may help to infer its original shape by making multiple candidates. Wan et al. (2021) also report that stochasticity improves FID score of image completion task if they are masked out large area of original images. H P REVENTION OF S QUEEZING - IN B RICKS

Original: P oo l
Revised: A brick position can be predicted in the middle of bricks. We call this brick as a squeezing-in brick. A Squeezing-in brick is easily prevented with simple modification to our pipeline. As illustrated in

Revised: Figure 6, we employed additional 1 × 1 × 3 convolutional filters after we obtain brick-wise validity.

Revised: Then we check whether the resultant validity score is 2.

Revised: The value 2 means the brick position and one of neighbor brick positions (top or bottom) is empty.

Revised: I A NALYSIS ON D IVERSITY OF G ENERATED S HAPES

Revised: We measure precision and recall to measure the diversity of generated shapes.

Revised: We define precision as how much of real shape distribution can be made using a generated shape distribution and recall as how much of generated shape distribution can be made using a real shape distribution by following the work by Sajjadi et al.

Revised: Distance between distributions is measured in the feature space encoded by a pretrained voxel classifier.

Revised: The results are depicted in Figure 7.

Revised: We find that the diversity is different for each class.

Original: C on ca t
Revised: Airplane class lacks diversity as the class have [Table] [Table] high precision but low recall.

Revised: Table class shows balanced results between diversity and similarity.

Revised: Chair class has diverse shapes as it has high recall but low precision.

Original: F C ,
Revised: Humans assemble bricks in a way that every intermediate structure is stable as depicted in Figure 8. Thus, we conduct brick assembly generation considering intermediate stability. To compare intermediate stability, we introduce a new evaluation metric, called a trajectory stability ratio.

Revised: The metric measures how many intermediate structures are stable by dividing the number of stable structures by the total number of structures.

Original: F C ,
Revised: We consider that a structure is stable when the center point of the structure does not move within given simulation steps. We utilize PyBullet to simulate stability of generated shapes. The quantitative results are presented in Table 7. We measure stability and generation performance of our method and its variants.

Revised: The variants are the models that (i) start assembling from bottom and (ii) consider intermediate stability.

Revised: To consider intermediate stability during inference, we exclude unstable actions by simulating stability of possible outcomes.

Revised: We find that the variant considering intermediate stability shows higher trajectory stability ratio against other variants.

Revised: This scenario is only conducted in sequential brick assembly generation and one step generation method is not appropriate for this case.

Original: N s o f t m a x
Revised: Additional qualitative results of our model is shown in Figure 9. [Figure] [Figure]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]