Fragment Relation Networks for Geometric Shape Assembly
Abstract
A geometric shape is often made of multiple fragments or parts.
Assembling the fragments into the target shape can be viewed as an interesting combinatorial problem with a variety of applications in science and engineering.
Previous related work, however, focuses on tackling limited cases, e.g., primitive fragments of identical shapes or jigsaw-style fragments of textured shapes, which greatly mitigate the combinatorial challenge.
In this work we introduce a challenging problem of shape assembly with textureless fragments of arbitrary shapes and propose a learningbased approach to solving it.
Given a set of candidate fragments and a target shape, the proposed model learns to select one of them and place it into a right place. The proposed model processes the candidate fragments in a permutation-invariant manner and can generalize to cases with an arbitrary number of fragments and even with a different target shape.
We demonstrate the effectiveness of our method on shape assembly tasks with different shapes and assembling scenarios.
Introduction
There have been a large volume of studies [15] on different types of shape assembly or composition in a variety of ﬁelds such as biology [19], earth science [21], archaeology [8] and tiling puzzle [16].
If we focus on shapes and parts without any textures, the problem can also be formulated as a combinatorial optimization problem, which aims to occupy a target space with candidate parts [7]. From this perspective we can view it as the bin packing problem [3], which minimizes the number of bins used.
Johnson [11] has proved the proposition on the guarantee for near-optimal solutions in low-order polynomial time.
As discussed above we interpret that a bin packing problem and a shape assembly problem have the similar aspect at which both can construct a target shape with a set of fragments.
However, prior work on shape assembly usually relies on inherent patterns and textures provided with the fragments that would be assembled.
For example, a jigsaw puzzle problem ﬁnds a correct combination by comparing and matching neighbors of each fragment where the fragments contain visual information [6].
On the other hand, if we assume visual information is not provided (i.e., losing color or discoloring for practical reasons) and a target geometric shape is partitioned at random (i.e., getting divided randomly or split by external forces), assembling geometric fragments becomes challenging.
It is due to its properties derived by the uniqueness of fragments and the combinatorial aspect of fragments [5].
Therefore, we deﬁne a two-dimensional geometric shape assembly problem under the following assumptions.
(i) Stochastic partitioning process [18] splits a given target geometric shape, and the shapes of fragments partitioned are distinct.
(ii) All the fragments of a geometric shape are provided unless otherwise speciﬁed.
For some assembling scenario we will introduce, some fragment can be missing, eroded, or distorted.
(iii) Fragments in a target geometric shape are assembled from bottom to top and from left to right.
This assembling sequence of fragments determined by such a rule makes this problem more combinatorial and more realistic [4].
Under the scenario mentioned above, we can solve the shape assembly problem with one of (i) brute-force approach, (ii) metaheuristic [1], (iii) sequential model-based optimization approach [2], and (iv) learning-based approach [14].
Due to the combinatorial explosion, a brute-force approach is hard to assemble the target shape if the number of fragments increases.
Moreover, both metaheuristic and sequential model-based optimization are not suitable to solve this problem, since such approaches are time-consuming for their inner loop steps.
A learning-based approach is a good alternative, but generic neural networks need to adjust a model conﬁguration if the dimensionalities of inputs and outputs are changed.
For this reason, we propose a permutation-equivariant neural network whose conﬁguration does not depend on the number of fragments and target shapes to resolve the issues.
Problem Deﬁnition
Geometric Shape Assembly Suppose that we are given a set of fragments X = { x j } Nj =1 and a target geometric shape S on a space Φ .
In this work, we assume two-dimensional fragments and a target shape in a two-dimensional space.
We sequentially assemble those fragments into the target shape; at each step, a fragment x i is sampled without replacement and placed on the top of the current space c considering the remaining space S − c .
This procedure is described in
Algorithm 1.
Our goal is to reconstruct the target shape S consuming all the fragments.
a shape c .
At every step, we select which fragment is assembled and where fragment is assembled.
Shape Fragmentation Dataset We create a dataset by partitioning a shape into multiple fragments, which can be used to pose its reverse task, i.e., an assembly problem.
Inspired by binary space partitioning algorithm [20], we randomly split a target shape and create a set of random fragments for each target shape.
The procedure is summarized in Algorithm 2 and some examples are shown in Figure 1.
After K times of binary partitioning, we obtain N = 2 K fragments.
Our Approach
To address the assembly problem, we propose a model that learns to select a fragment from candidates and place it on top of a current shape.
The model takes two inputs: (i) remaining space S − c , which is the result of assembled fragments at previous steps; (ii) fragments X , which consists of all candidates for the next assembly.
It then predicts which x
i ∈
X should be used and where it should be placed on Φ .
We assemble all the fragments by iteratively running our model until no candidate remains.
To make a proper decision, the model needs to extract geometric information of the fragments and the shape, and also understand the relations among them.
We train our model by supervising it with a large number of episodes where a fragment and its position in the next time step is known from a current shape and candidate fragments.
We generate such episodes from the shape fragmentation dataset by creating a sequence of fragments from the bottom left to the top right.
Fragment Relation Networks Our model, fragment relation network (FRN), considers a candidate fragment x
i in the context of the other candidates X \ x i and the remaining space S − c , and produces [Figure] two outputs: (i) selection probability for x i , which means how likely x i is selected.
(ii) placement probability map for x i , which means how likely each position is for x i .
As shown in Figure 3,
FRN contains two branches for the two outputs, fragment selection network (i.e., FRN-Sel) and fragment placement network (i.e., FRN-Loc).
Both networks share many learnable parameters that are parts of two encoders and fragment relation attention module (FRAM).
FRAM, which is inspired by Transformer network [22], captures the relationship between the remaining fragments.
Fragment Selection FRN-Sel (lower part of Figure 3) is a binary classiﬁer taking as input x
i , S − c , and X \ x i : y i =
FRN-Sel ( x i ; S − c , X \ x i ) ∈ R , where x i ∈
X .
Since, in particular, FRN-Sel takes into account fragment relations with FRAM, it can choose the next fragment by considering the remains of fragments.
Furthermore, as will be discussed in the Discussion section, the number of parameters in FRN does not depend on the number of fragments, which implies that the cardinality of X can be varied.
This property helps our network to handle variable-length set of fragments.
Fragment Placement FRN-Loc (upper part of Figure 3) determines the position of x ∈ X by predicting a probability map over pixels: M i = FRN-Loc ( x i ; S − c , X \ x i
) ∈
R w × h .
This network is implemented as an encoder-decoder architecture with skip connections between them, in order to compute pixel-wise probabilities with convolution and transpose convolution operations.
It can reduce the number of learnable parameters due to the absence of the last fully-connected layer.
Fragment Relation Attention Module We suggest an attention-based module FRAM, which considers the high-order relationship between the remaining fragments using multi-head attention networks [22].
First of all, we introduce multi-head attention and scaled dot-product attention.
As proposed by Vaswani et al.
[22], given X ∈ R n 1 × d and Y ∈ R n 2 × d , multi-head attention is composed of multiple scaled dot-product attentions: DP ( Q , K , V ) =
σ (cid:16) QK (cid:62) √ d 1 (cid:17) V and MH ( X , Y , Y )
=
[ DP 1 , . .
. , DP h ] W O , where Q ∈ R n 1 × d 1 , K ∈ R n 2 × d 1 , V ∈ R n 2 × d 2 ,
DP i = DP ( XW Q i , YW K
i , YW V i ) , σ is a softmax function, h is the number of heads, and
W O ∈ R hd 2 ×
d
is the learnable parameters.
To leverage the expression power of this multi-head attention module, X and Y are projected by the different parameter sets W Q ∈ R d × d 1 ,
W K ∈ R d × d 1 , and W V ∈ R d × d 2 .
To express a set of the remaining fragments to a latent representation h , we utilize the multi-head attention: H = MH ( X , X
, X ) ∈ R |X|× d and h = MH ( 1 , H , H )
∈ R d , where 1 ∈ R 1 ×
d is an all-ones matrix.
Without loss of generality, the aforementioned equations can take X and H , respectively.
Moreover, MH ( X , X , X ) can be stacked by feeding H to the multi-head attention as an input, and MH ( 1 , H , H ) can be also stacked in the similar manner.
Note that h becomes R 1 × d , but we straightforwardly employ it as a R d -shaped representation.
FRAM is a generalization of global feature aggregation methods such as average pooling and max pooling across instances, so that it considers high-order interaction between instances and aggregates instance-wise representations (i.e., H ) with learnable parameters.
The ﬁnal output is determined by selecting the fragment with the index of maximum y i : i (cid:63) =
arg max i ∈{ 1 ,..., |X|} y i .
After choosing the fragment, the position with the maximum probability on M i (cid:63) is selected for the placement.
Experimental Results
Experimental Setup As shown in Figure 1, we evaluate our method and the existing methods on three types of target shapes: (i) Square , (ii) Pentagon , and (iii) Hexagon .
Unless otherwise speciﬁed, we use 5,000 examples each of them is partitioned into 8 fragments using binary space partitioning.
We use 64%, 16%, and 20% of the examples for training/validation/test dataset respectively.
Note that there is no prior methods that tackle shape assembly for textureless fragments.
Therefore, we apply two types of classic optimization methods for this task.
Besides, we apply an approach that is based on generative adversarial networks.
Due to a page limit, the architectures of FRN and V-GAN will be presented in the appendices.
We measure an assembling quality via IoU:
IoU ( c , S ) =
area ( c ∩ S ) area ( c ∪ S ) , where c is an assembled shape and S is a target shape.
[Table]
Main Result We compare our approach to other baseline methods on the assembly of different target shapes.
Results on Square , Pentagon , and Hexagon shapes are summarized in Table 1.
V-GAN requires a ﬁxed number of vertices for partitioned fragments, which is only applicable to Square target shape.
We ﬁnd that our method consistently covers much more area than baselines within less time budget.
Both simulated annealing and Bayesian optimization struggle to place fragments in the proper position, resulting in many overlaps between fragments.
They also take much longer time compared to our method in all experimental conditions.
On the other hand,
V-GAN takes much less time than other baseline methods while signiﬁcantly underperforms compared to other methods.
Qualitative results are presented in Figure 4.
Conclusion
In this paper, we solve a two-dimensional shape assembly problem with our proposed neural network
FRN.
It predicts the next fragment and its corresponding position where the fragments that would be assembled are given, by considering fragment relations with an attention-based module FRAM.
We show that our method outperforms other baseline methods such as simulated annealing, Bayesian optimization, and a simple learning-based approach with adversarial training, in the circumstances each of which deals with one of three different target geometric shapes.
Acknowledgments and Disclosure of Funding
This work was supported by Institute for Information & communications Technology Promotion (IITP) grant funded by the Korea government (MSIP) (No.2019-0-01906, Artiﬁcial Intelligence
Graduate School Program (POSTECH)).
Appendices
In this material we describe the detailed contents that supplement our main article.
Appendix A Details of Shape Fragmentation
[Figure]
We divide a target geometric shape into fragments using hyperplanes by a binary space partitioning algorithm [20].
The number of fragments increases exponentially with the number of partitions.
Some partitioning examples are shown in Figure 5.
We create a shape fragmentation dataset under two constraints that prevent generating small fragments.
First, the hyperplane does not cross adjacent edges of the given polygon.
Second, the position that the hyperplane will pass through is randomly selected between the range of ± 25 % of edge length from the center of the edge.
The dataset we used is composed of 5000 samples.
Appendix B Algorithms
The algorithms described in the main article are represented in Algorithm 1 and Algorithm 2.
Appendix C Details of Experiment C.1 Training
We alternately train our neural network with the following objectives.
Since we ﬁrst choose the next fragment, and then place the selected fragment, the gradient updates by two objectives should be separately applied.
The objective for the fragment selection part is
[Equation]
do 8: do 10:
X i ← i
{ x
14: return where t m is a one-hot representation of true fragment, each entry of y m is computed by FRN-Sel, and σ is a softmax function.
Next, the objective for the fragment placement part is
[Equation]
where τ m ∈ R w × h is a one-hot representation of true position, M m is computed by FRN-Loc, and vector is a function for vectorizing a matrix.
Both Eq.
(1) and Eq.
(2) compute the summation of the cross-entropy over outputs that produce through a permutation-equivariant network for logits.
As will be described in the subsequent sections, this property let us examine FRN in various circumstances.
Moreover, since our network does not rely on the sequence (or history) of fragments, we are able to apply a mini-batch training scheme, which enables us to train ofﬂine.
C.2 Baselines
We select the following three baseline methods:
Simulated Annealing (SA) [17] It solves the problem via optimization.
SA computes the overlap to a target shape; Bayesian Optimization (BayesOpt)
[2] A powerful method to optimize a black-box function.
We ﬁnd the position of given fragments to maximize the overlap between each fragment and target shape;
LayoutGAN Modiﬁed for Vertex Inputs (V-GAN) [14]
It is a modiﬁcation of [14].
Since LayoutGAN assumes that fragment shapes are always ﬁxed and only ﬁnds their positions, we modify the generator structure to a neural network that takes fragment vertices and predicts their positions.
This baseline is composed of the generator, and the discriminator that distinguishes real and fake examples is trained via an adversarial training strategy.
C.3 Details of Model Architecture
FRN FRN consists of FRN-Sel that selects next fragment and FRN-Loc that predicts center coordinates of the chosen fragment.
There exist 7 hyperparameters in FRN with 3 speciﬁcally in FRAM.
The default dimension of fully connected layer in the whole pipeline is ﬁxed to 256 throughout the experiments.
For the training, we use batch size of 32 and ADAM optimizer [12] for both
FRN-Sel and FRN-Loc with learning rate of 5 × 10 − 4 and 2 × 10 − 3 respectively.
FRN-Sel is a convolutional neural network (CNN) followed by FRAM and MLP block.
FRN-Loc is convolutional encoder-decoder with MLP block followed by an additional fully-connected layer inserted in between.
Each encoder for FRN-Loc and FRN-Sel do not share the weights, and thus are trained with different loss.
Our FRAM follows similar setting of original Transformer [22] model with 2048 hidden units and 8 attention heads.
V-GAN It follows typical setup of generative adversarial network [9], having both generator and discriminator.
Speciﬁcally, generator takes fragment’s vertex coordinates as input and predicts center coordinates.
Discriminator then makes real or fake decision based on adjusted vertex coordinates.
For training, we use batch size of 128 and ADAM optimizer with learning rate of 1 × 10 − 4 .
Appendix D Discussion and Related Work
In this section, we discuss the properties of our problem and FRN, based on the numerical results.
Moreover, we mention about the previous studies related to our work.
As presented in Table 1, the coverage result that assembles Hexagon is better than the results for
Square and Pentagon .
It implies that if a target shape becomes complicated and contains more information, the problem becomes easily solvable.
From this observation, we can readily induce our formulation to the packing problem with inherent information such as [6] and [16].
Our proposed network FRN can be thought as a permutation-equivariant network, which satisﬁes
[Equation]
where f : R n × d → R n × d (cid:48) is a neural network that outputs a set and π is a permutation function along the ﬁrst dimension.
Guttenberg et al.
[10] suggest a permutational layer to handle the different inputs and not depend on the speciﬁc ordering of inputs.
Zaheer et al.
[23] derive the necessary and sufﬁcient conditions on permutation-equivariant networks.
Lee et al.
[13] propose an attention-based permutation-equivariant block for set operations.
By this deﬁnition of permutation-equivariance, we can describe the following proposition for FRN:
Proposition 1 By the shared architecture over the remaining fragments X , FRN is permutationequivariant.
In particular, the number of learnable parameters does not depend on the dimensionality of logits y and the shape of position probability M i for i ∈ { 1 , . .
. , |X|} .
As computed by FRN-Sel and FRN-Loc, their outputs can handle variable-sized X and the ordering of X affects the ordering of the outputs, satisfying Eq.
The experimental results support that our method can learn this challenging scenario with the permutation-equivariant neural network.