U NSUPERVISED POSE - AWARE PART DECOMPOSITION FOR 3D ARTICULATED OBJECTS
Anonymous authors
Paper under double-blind review
A BSTRACT
Articulated objects exist widely in the real world.
However, previous 3D generative methods for unsupervised part decomposition are unsuitable for such objects, because they assume a spatially ﬁxed part location, resulting in inconsistent part parsing.
In this paper, we propose PPD (unsupervised Pose-aware Part Decomposition) to address a novel setting that explicitly targets man-made articulated objects with mechanical joints, considering the part poses.
We show that categorycommon prior learning for both part shapes and poses facilitates the unsupervised learning of (1) part decomposition with non-primitive-based implicit representation, and (2) part pose as joint parameters under single-frame shape supervision.
We evaluate our method on synthetic and real datasets, and we show that it outperforms previous works in consistent part parsing of the articulated objects based on comparable part pose estimation performance to the supervised baseline.
I NTRODUCTION
Humans are capable of recognizing complex shapes by decomposing them into simpler semantic parts.
Researchers have shown that infants learn to group objects into semantic parts using the location, shape, and kinematics as a cue (Spelke et al., 1995; Slater et al., 1985; Xu & Carey, 1996).
Moreover, even very young infants can learn to reason about kinematics using non-sequential single frames (Shirai & Imura, 2014; Kourtzi & Kanwisher, 2000).
Although humans can naturally achieve such reasoning, it is challenging for machines, particularly in the absence of a rich supervision.
Generative part decomposition and abstraction methods have a long-standing history in computer vision (Roberts, 1963; Binford, 1971).
Learning to reconstruct part shapes from single-frame input has a wide range of applications, such as part-wise shape editing (Funkhouser et al., 2004; Mo et al., 2020) and unsupervised 3D part parsing (Chen et al., 2020;
Paschalidou et al., 2019; Niu et al., 2018; Tulsiani et al., 2017).
However, previous studies have mainly focus on non-articulated objects.
Because they exploit the consistent part location as a cue to group shapes into semantic parts, these approaches are unsuitable for decomposing articulated objects when considering the kinematics of dynamic part locations .
In contrast, there exist discriminative approaches targeting articulated objects for part segmentation, in addition to part pose estimation from single-frame input.
However, they require explicit supervision, such as segmentation labels and joint parameters (Yi et al., 2018;
Xiang et al., 2020; Li et al., 2020).
Removing the need for such expensive supervision has been an important step toward more human-like representation learning (Becker & Hinton, 1992).
In this study, as a novel problem setting, we investigate the generative part decomposition task for man-made articulated objects with mechanical joints, considering part poses as part kinematics, in an unsupervised fashion .
Speciﬁcally, we consider the revolute and prismatic parts with a 1 degreeof-freedom joint state as the part kinematics because they cover most of the kinematic types that common man-made articulated objects have (Xiang et al., 2020; Abbatematteo et al., 2020; Michel et al., 2015).
This task aims to learn consistent part parsing for articulated objects with various part poses from single-frame shape observation.
An overview is shown in Figure 1.
This task expands the target of the current generative part decomposition’s applications to articulated objects in novel ways, such as part shape editing based on part kinematics (part rigging) and part shape transfer between samples with different poses.
However, this task is challenging since the model must consider the kinematics between possibly distant shapes to group them as a single part and
[Figure]
[Table] [Table]
has to disentangle the part poses from shape supervision.
A comparison with previous studies is presented in Table 1.
To address this problem, we propose PPD (unsupervised Pose-aware Part Decomposition) that takes an unsegmented, single-frame point cloud with various underlying part poses as an input.
PPD reconstructs part-wise shapes transformed using the estimated joint parameters as the part poses so that the same semantic parts with different poses are reconstructed as the same part.
We train PPD as an autoencoder using single-frame shape supervision.
To address the problems associated with (1) kinematically learning consistent part parsing and (2) learning part poses with single-frame shape supervision, we demonstrate how the explicit learning of the category-common priors separately from the instance-dependent component for part shapes and poses can effectively addresses these problems.
Furthermore, we employ non-primitive-based part shape representation and utilize deformation by part poses to induce unsupervised part decomposition, in contrast to previous works that employ primitive shapes and rely on its limited expressive power as an inductive bias.
Our contributions are summarized as follows: (1) We propose a novel unsupervised generative part decomposition method for articulated objects based on part kinematics.
(2) We show that the proposed method learns a non-primitive-based implicit ﬁeld as the decomposed part shapes and the joint parameters as the part poses, using single-frame shape supervision.
(3) We also demonstrate that the proposed method outperforms previous generative part decomposition methods in terms of semantic capability and show comparable part pose estimation performance to the supervised baseline.
R ELATED WORKS
Recently, a number of deep generative models have been developed for unsupervised generative part decomposition.
Existing studies assume non-articulated objects in which the part shapes are in a ﬁxed 3D location and induce part decomposition by limiting the expressive power of the shape decoders by employing learnable primitive shapes (Tulsiani et al., 2017;
Niu et al., 2018; Chen et al., 2019b; Paschalidou et al., 2019; Chen et al., 2019b; 2020;
Deng et al., 2020a; Kawana et al., 2020;
Paschalidou et al., 2021).
BAE-Net (Chen et al., 2019b) employs a non-primitive-based implicit ﬁeld as the part shape representation, similar to ours.
However, it still limits the expressive power of the shape decoder using MLP with only three layers.
In contrast, our approach assumes parts to be dynamic with the consistent kinematics and induces part decomposition through rigid transformation of the reconstructed part shapes with the estimated part poses.
A growing number of studies have tackled the reconstruction of category-speciﬁc, natural articulated objects with a particular kinematic structure, such as the human body and animals.
Representative works rely on the use of category-speciﬁc template models as the shape and pose prior (Loper et al., 2015; Zufﬁ et al., 2017; Bogo et al., 2016; Zufﬁ et al., 2019;
Kulkarni et al., 2020).
Another body of works reconstruct target shapes without templates, such as by reconstructing a part wise implicit ﬁeld given a part pose as an input (Deng et al., 2020b) or focusing on non-rigid tracking of the samples (Boˇziˇc et al., 2021).
In contrast, our approach focuses on the man-made articulated object of the general category with various kinematic structures.
Moreover, our approach learns the shape and pose
, without any part information either as is applicable to unseen samples.
In discriminative approaches, a number of studies have focused on the inference of the part segmentation of the input point cloud and part poses as joint parameters (Li et al., 2020; Xiang et al., 2020; Abbatematteo et al., 2020) targeting man-made articulated objects.
These approaches require expensive annotations, such as part labels and ground-truth joint parameters.
Moreover, they require category-speciﬁc prior knowledge of the kinematic structure.
In contrast, our model focuses on generative tasks and is category agnostic.
Moreover, it only requires shape supervision during training.
A very recent work (Huang et al., 2021) assumes an unsupervised setting where multi-frame, complete shape point clouds are available for both input and supervision signals during training and inference.
Whereas our approach assumes a single-frame input and shape supervision, it also works with partial shape input during inference.
Note that, in this study, the purpose of part pose estimation is, as an auxiliary task, to facilitate consistent part parsing.
It is not our focus to outperform the state-of-the-art supervised approaches in part pose estimation.
M ETHODS
In our approach, the goal is to represent an articulated object as a set of semantically consistent part shapes based on their underlying part kinematics.
We represent the target object shape as an implicit ﬁeld that can be evaluated at an arbitrary point x ∈ R 3 in 3D space as O : R 3 → [0 , 1] , where { x ∈ R 3 | O ( x ) = 0 } deﬁnes the outside of the object, { x ∈ R
| O ( x ) = 1 } the inside, and { x ∈ R 3 | O ( x ) = 0 .
5 } the surface.
Given an observation (e.g., point cloud) I ∈
X as an input, we approximate the object shape using a composite implicit ﬁeld ˆ O that is decomposed into a collection of N parts.
The i -th part has an implicit ﬁeld ˆ O i : R 3 × X → [0 , 1] as part shape and part pose
B i ∈ SE (3) .
We ensure that O is approximated as O ( x
) ≈ ˆ O ( x | I, { B i } Ni =1 ) through the losses.
An overview of PPD is shown in Figure 2.
PPD employs an autoencoder architecture, and trained under single category setting.
It consists of an encoder E , category-common pose decoder F p,c , instance-dependent pose decoder F p,z , category-common shape decoders { F s,zi } Ni =1 , and instancedependent shape decoders { F s,ci } Ni =1 .
Given a single-frame point cloud I , the encoder derives the shape latent vector z s and the pose latent vectors z p and z p,c .
Given z p and z p,c , F p,z and F p,c compose homogeneous transformations { B i } Ni =1 as the part poses.
The i -th shape decoders F s,zi and F s,ci decode a part-wise implicit ﬁeld for ˆ
O
i given z s and a transformed 3D coordinate B − 1 i x .
We discuss the details about F s,zi and F s,ci in Section 3.1, and F p,z and F p,c in Section 3.2.
3. P ART SHAPE REPRESENTATION
We propose a non-primitive-based part shape representation that is decomposed into the categorycommon shape prior and instance-dependent shape details.
We employ MLP-based decoders to model a part-wise implicit ﬁeld.
We capture the category-common shape prior using the categorycommon shape decoder F s,ci : R 3 → R .
Because F s,ci does not take a latent vector from the encoder, it learns an input-independent, rest-posed part shape template as the category-common shape prior.
We also employ an instance-dependent shape decoder F s,zi : R 3 × R d
→ R to capture the additional instance-dependent shape details conditioned with the shape prior, where d is the dimension of the shape latent vector z s .
Given F s,ci and F s,zi , we formulate a part-wise implicit ﬁeld ˆ O i as follows:
[Equation]
where σ ( · ) represents the sigmoid function and ˆ O ci ( x )
= σ ( F s,ci ( x )) .
For brevity, we omit I in ˆ
O
i and simply denote it as ˆ O i ( x ) .
Given the part poses { B i } Ni =1 as part-wise locally rigid deformation, we formulate ˆ O as the composition of { ˆ O i } Ni =1 deﬁned as ˆ O ( x | I, B )
=
max i { ˆ O i ( B − 1 i x ) } .
As in the piecewise rigid model of (Deng et al., 2020b), coordinate transformation B − 1 i x realizes locally rigid deformation by B i of the part-wise implicit ﬁeld by querying the rest-posed indicator.
Note that, although we set the maximum number of parts N , the actual number of parts used for reconstruction can change; it is possible that some parts do not contribute to the reconstruction because of the max operation or simply because ˆ O i < 0 .
5 for all 3D locations.
To learn the shape decoders, we minimize the reconstruction loss using the standard binary cross-entropy loss ( BCE ) deﬁned as:
[Equation]
where ˆ O c ( x | B ) = max i { ˆ O c
i ( B − 1 i x ) } , and λ rec and λ crec are the loss weights.
In Equation 1, we experimentally found that conditioning F s,zi by ˆ O ci through multiplication rather than addition effectively prevents F s,zi from deviating largely from F s,ci .
This regularization induces the unsuperivsed part decomposition.
Considering reconstructing the target shape by single i -th part, since the multiplication makes it difﬁcult to output shapes that deviating largely from the category-common prior shape, the large shape variations of target shapes are expressed by B i regarded as the global pose of the reconstructed shape.
However, the datasets’ large shape variations in target shapes are due to the various local poses of multiple part shapes.
Therefore, the large shape variations of target shapes cannot be expressed only by the single part and its part pose B i .
Thus, as an inductive bias of the unsupervised part decomposition, the model is incentivized to use a composition of multiple parts to express the shape variations due to various local part poses.
The details of the part shape deformation by part poses are explained in Section 3.2.
In the learning process, the model ﬁrst tries to reconstruct the target shapes with a single part; then with multiple parts.
Lastly, it starts to deform each part to express the shape variations.
During the learning process, the part poses are disentangled from the shape supervision to transform the part shapes in a way that minimizes the reconstruction loss.
For the visualization of the learning process of part decomposition, see Figure 11 in the Appendix.
In addition, because we consider the locally rigid deformation of the shape, the volumes of the shape before and after the deformation should not be changed by the intersection of parts; we formulate this constraint as follows:
[Equation]
3. P ART POSE REPRESENTATION
Parameterization of the part poses.
We propose to characterize part pose B
i by its part kinematic type y i ∈ { ﬁxed , prismatic , revolute } and joint parameters.
Each y i is manually set as a hyperparameter.
The joint parameters consist of the joint direction u i ∈ R 3 with the unit norm and joint state s i ∈ R + .
Additionally, the ”revolute” part has the pivot point q i ∈ R 3 .
We refer to the joint direction and pivot point as the joint conﬁguration.
For the ”ﬁxed” part, we set
B i as an identity matrix because no transformation is applied.
For the ”prismatic” part, we deﬁne
B i
=
T ( s i u i ) , where T ( · ) represents a homogeneous translation matrix given the translation in R 3 , and s i and u i represent the translation amount and direction, respectively.
For the ”revolute” part, we set B i = T ( q i ) R
( s i , u i ) , where R ( · ) denotes a homogeneous rotation matrix given the rotation representation, and s i and u
i represent the axis-angle rotation around the axis u i by angle s i .
In human shape reconstruction methods using template shape, its pose is initialized to be close to the real distribution to avoid the local minima (Kanazawa et al., 2018;
Kulkarni et al., 2020).
Inspired by these approaches, we parametrize the joint direction as [ u i ; 1] =
R ( r i )
[ e i ; 1] , where e i is a constant directional vector with the unit norm working as the initial joint direction as a hyperparameter and r i ∈ R 3 represents the Euler-angle representation working as a residual from the initial joint direction e i .
This allows us to manually initialize the joint direction in a realistic distribution through e i by initializing r i = 0 .
For the illustration of the geometric relationship of the joint parameters, see
Figure 12 in the Appendix.
Through the observation, we assume that the joint conﬁguration has a category-common bias, while the joint state strongly depends on each instance.
This is because the location of each part and the entire shape of an object can constrain the possible trajectory of the parts, which is deﬁned by the joint conﬁguration.
To illustrate this idea, we propose to decompose the joint conﬁguration into a category-common bias term and an instance-dependent residual term denoted as r i =
r ci + r zi
and q i
= q ci + q zi , respectively.
We employ the category-common pose decoder
F p,c (qt( z p,c )) , which outputs { r ci | i ∈
A p } and { q ci | i ∈
A r } , where A p = { i ∈ [ N ] | y i (cid:54)
= ﬁxed } , A r = {
i ∈
[ N ] | y i = revolute } , z p,c denotes a pose latent vector, and qt( · ) is a latent vector quantization operator following VQ-VAE (Razavi et al., 2019).
The operator qt( · ) outputs the nearest constant vector c p to the input latent vector z p,c among the N qt candidates.
Instead of using a single constant vector, we can switch between multiple constant vectors to capture the discrete, multi-modal category-common biases.
The pose latent vector z p,c is optimized by the loss:
[Equation]
(4) where sg denotes an operator stopping gradient on the backpropagation.
We also employ an instance-dependent pose decoder F p,z ( z p ) that outputs { s i | i ∈
A p } , { r zi | i ∈
A p } , and { q zi | i ∈
A r } .
For q i and r i , to prevent an instance-dependent term from deviating too much from the bias term, we regularize them by the loss:
[Equation]
where N r = | A r | , N p =
| A p | , and λ dev is the loss weight.
Therefore, we limit the possible distribution of the joint conﬁguration around the category-common bias.
This incentivizes the model to reconstruct the instance-dependent shape variation by the joint state, which constrains the part location along the joint direction.
This kinematic constraint biases the model to represent the shapes having the same kinematics with the same part.
Note that, because the previous studies (Kawana et al., 2020; Deng et al., 2020a; Paschalidou et al., 2019) do not impose such a constraint on the part localization, learned part decomposition is not necessarily consistent under different part poses.
Regularization losses for joint parameter learning.
We propose a novel regularization loss that constrains the joint parameters with the implicit ﬁelds.
We assume that the line in 3D space, which consists of the pivot point and joint direction, passes through the reconstructed shape.
The joint should connect at least two parts simultaneously, which means that the joint direction anchored by the pivot point passes through at least two reconstructed parts.
We realize this condition as follows:
[Equation]
(6) where S gt = { x ∈ R 3 |
O
( x ) = 1 } , S i = { x ∈ R 3 |
ˆ O i ( B − 1 i x ) > 0 .
5 } , S i,j = { x ∈
R 3 | ˆ O j ( B − 1 j x ) > 0 .
5 , j ∈
A r \ i } , and λ loc is the loss weight.
Note that L loc is self-regularizing and not supervised by the ground-truth part segmentation.
See Figure 13 in the Appendix for an illustration of L loc .
Moreover, to reﬂect the diverse part poses, we prevent the joint state s i from degenerating into a static state.
In addition, to prevent the degeneration of multiple decomposed parts from representing the same revolute part, we encourage the pivot points to be spread.
We realize these requirements by the loss deﬁned as:
[Equation]
where std B ( · ) denotes the batch statistics of the standard deviation, v is a constant that controls the distance between pivot points, and λ var s and λ var q are the loss weights.
Adversarial losses.
Inspired by human shape reconstruction studies (Chen et al., 2019a; Pavllo et al., 2019), we employ the adversarial losses from WGAN-GP (Gulrajani et al., 2017) to regularize the shape and pose in the realistic distribution.
The losses are deﬁned as:
[Equation]
where D ( · ) is a discriminator;
˜ x is a sample from the reconstructed shapes P g transformed by the estimated joint conﬁguration and randomly sampled joint state
˜ s
i ∼ Uniform(0 , h i ) , with the maximum motion amount h i treated as a hyperparameter; x is a sample from the ground-truth shapes P r ; ˆ x is a sample from P ˆ x , which is a set of randomly and linearly interpolated samples between ˆ x and x ; and λ adv g and λ adv d are the loss weights.
As an input to D , we concatenate the implicit ﬁeld and corresponding 3D points to create a 4D point cloud, following (Kleineberg et al., 2020).
3. I MPLEMENTATION DETAILS
We use the Adam solvers (Kingma & Ba, 2014) with a learning rate of 0 . 0001 to optimize the losses:
L total g =
L rec + L vol + L vq + L dev + L loc + L var + L adv g (sum of Equations 2, 3, 4, 5, 6, 7, and 9) and L total d =
L adv d (Equation 8), with a batch size of 18.
For the input, we use the complete shape point cloud with 4096 points sampled from the surface of the target shape, unless otherwise noted.
For the ground-truth implicit ﬁeld, we use 4096 coordinate points and their corresponding indicator values.
We set the loss weights as follows: λ rec = 0 .
01 , λ crec = 0 .
001 , λ dev = 0 .
1 , λ var s = 0 .
1 , λ loc = 100 , λ var q = 0 .
01 , λ vol = 1000 , λ adv g = 0 .
65 , and λ adv d = 0 .
We set v = 0 .
01 in L var and N qt = 4 for qt( · ) .
For h i in L adv d , we set to π 2 and 0 .
4 the ”revolute” and ”prismatic” parts, respectively.
Note that we experimentally found that it does not constrain the model to predict s i larger than h i to reconstruct the target shape.
Because we do not impose any geometric constraints on the part shapes, we set the number of parts for each part kinematics
y i as its maximum number in the datasets plus an additional one part for over-parameterization.
The detail of the datasets is explained in Section 4.
We set N = 8 , which consists of one ”ﬁxed” part, three ”revolute” parts, and four ”prismatic” parts.
For the initial joint direction e i , for each ”revolute” part, we set it to the ( + z , − z , + y ) directions, and for each ”prismatic” part, we set it to the + x direction.
We use the same hyperparameter for all categories, without assuming the category-speciﬁc knowledge.
We train our network in two stages following (Chen et al., 2020): ﬁrst, we train it on an implicit ﬁeld of 16grids and then on 32 3 grids.
During the training, the max operation is substituted with LogSumExp for gradient propagation to each shape decoder.
See Appendix B for further training details.
Network architecture.
We use the PointNet (Qi et al., 2017)-based architecture from (Mescheder et al., 2019) as an encoder E and the one from (Shu et al., 2019) as a discriminator D .
Our shape decoders { F s,ci } Ni =1 and { F s,zi } Ni =1 are MLP with sine activation (Sitzmann et al., 2020) for a uniform activation magnitude suitable for propagating gradients to each shape decoder.
For the categorycommon pose decoder F p,c , we use two separate networks of MLP, namely, F p,cr and F p,cq .
For the instance-dependent pose decoder F p,z , we employ MLP with a single backbone having multiple output branches.
See Appendix A for further architectural details.
E XPERIMENTS
Datasets.
Following the recent articulated pose estimation study (Li et al., 2020), we evaluate our method on ﬁve categories with various joint conﬁgurations from two synthetic datasets: Motion dataset (Wang et al., 2019) for the oven, eyeglasses, laptop, and washing machine categories, and
SAPIEN dataset (Xiang et al., 2020) for the drawer category.
Each category has a number of parts with the same kinematic structure.
We generate 100 instances with different poses per sample, generating 24k instances in total.
We divide the samples into the training and test sets with a ratio of approximately 8:2.
For further details of the data generation, see Appendix C. To verify the transferability of our approach trained on synthetic data to real data, we use the laptop category from RBO dataset (Mart´ın-Mart´ın et al., 2018) and Articulated Object Dataset (Michel et al., 2015), which is the intersecting category with the synthetic dataset.
Baselines.
We compare our method with the state-of-the-art unsupervised generative part decomposition methods with various characteristics:
BAE-Net (Chen et al., 2020) (non-primitive-based part shape representation), NSD (Kawana et al., 2020)
(primitive-based part shape representation with part localization in R 3 ), and BSP-Net (Chen et al., 2020) (primitive-based part shape representation with part localization by 3D space partitioning).
For BSP-Net, we train up to 32 3 grids of the implicit ﬁeld instead of 64 3 grids in the original implementation to match those used by other methods.
For NSD, we replace its image encoder with the same PointNet-based encoder in our approach.
For the part pose estimation, we use NPCS (Li et al., 2020) as the baseline.
NPCS performs
[Table]
part-based registration by iterative rigid-body transformation, which is a common practice in articulated pose estimation of rigid objects.
Note that NPCS assumes that part segmentation supervision are available during training and part kinematic type per part is known, which we do not assume in both cases.
See Appendix B.1 for further training details of the baselines.
Metrics.
For the quantitative evaluation of the consistent part parsing as a part segmentation task, we use the standard label intersection over union (label IoU), following the previous studies (Chen et al., 2019b; 2020; Deng et al., 2020a; Kawana et al., 2020).
As our method is unsupervised, we follow the standard initial part labeling procedure using a training set to assign each part a groundtruth label for evaluation purposes following (Deng et al., 2020a; Kawana et al., 2020).
A detailed step can be found in Appendix D. For the part pose evaluation, we evaluate the 3D motion ﬂow of the deformation from the canonical pose to the predicted pose measured as the endpoint error (EPE) (Yan & Xiang, 2016), which is a commonly used metric for pose estimation of articulated objects (Wang et al., 2019;
Bo ˇ zi ˇ c et al., 2021).
Finally, we report F-score and Chamfer L1 distance as the surface reconstruction accuracy metrics evaluated on the meshiﬁed implicit ﬁeld sampled on 32grids using marching cubes (Lorensen & Cline, 1987).
4. S EMANTIC CAPABILITY
We evaluate the semantic capability of our approach in part parsing.
As part decomposition approaches aim to learn 3D structure reasoning with as small a number of ground-truth labels as possible , it is preferable to obtain the initial manual annotations with as few numbers of shapes as possible.
This requirement is essential for articulated objects, which have diverse shape variations owing to the different articulations.
As our approach is part pose consistent, we only need a minimal variety of instances for the initial manual labeling.
To verify this, we evaluate the part segmentation performance using only the canonically posed (joint states were all zero) samples in the training set.
See Appendix E.2 for further studies on pose variation for the initial annotation.
The evaluation results are shown in Table 2.
Our approach outperforms all the previous works on average.
The segmentation results are shown in Figure 3.
The same color indicates the same segmentation part.
We also show the number of parts or primitives that each model uses in the last column of the table.
Our model uses a much smaller number of parts than BSP-Net (Chen et al., 2020); however, it still performs the best.
This shows that our model is more parsimonious, and each part has more semantic meaning in part parsing.
For additional visualization of our part segmentation result, see Appendix
E.1.
We also visualize the generated part shapes in Figure 4.
We can see that one dynamic part is successfully reconstructed by a single implicit ﬁeld.
This demonstrates the advantage of using an implicit ﬁeld without any geometric constraint as the part shape representation: we do not have to employ a complicated grouping mechanism of primitive shapes based on part kinematics.
We can also see that our part shapes are more semantic and interpretable than the previous works.
Moreover, our part shape representation exhibits the part shape with disconnected shapes, which the previous single primitive shape cannot express.
Disentanglement between the part shapes and poses.
Because our approach disentangles the part shapes and poses, it allows us to rig a reconstructed shape or generate a novel shape while maintaining the part poses.
We visualize the results of the interpolation of part shapes and joint states in Figure 5.
In the middle row, we show the shape interpolation between the source and the target while ﬁxing the joint state s i of the source to maintain the same part pose.
We can see that the shape is smoothly deformed from the source to the target by maintaining the original pose.
In the bottom row, we interpolate the joint state s i between the source and the target; we can see that the joint state changes from the source to the target while maintaining the shape identity of the source
[Table]
[Figure]
[Table]
shape.
These two disentangled interpolations can only be performed using our approach and not the previous methods, as shown in the top row of the ﬁgure.
4. P ART POSE ESTIMATION
To validate whether the predicted part decomposition is based on the reasonable part pose estimation we quantitatively evaluate the performance.
Because we train our model without specifying a canonically posed shape, we use the estimated deformation between the target instance and the canonically posed instance of the same sample as the estimated part pose to align with the prediction of the supervised baseline.
We present the evaluation results in Table 3.
We show the supervised rigid registration approach NPCS (Li et al., 2020) only as a reference.
Our method is comparable with NPCS, with the same order of performance.
Note again that we are not attempting to outperform supervised pose estimation methods; rather, we aim to show that our unsupervised approach can decompose parts based on reasonable part pose estimation.
See Appendix F for further results.
4. R ECONSTRUCTION
We evaluate the reconstruction performance of our approach to validate whether PPD learns a reasonable shape representation rather than degenerating to ignore the instance-dependent shape details.
The results are presented in Table 4.
We also show the number of learnable parameters of the model to show its capacity.
Note that we show the performance of the baselines only as a reference be-
[Table]
w/o L vol w/o L dev w/o L loc w/o L var w/o L adv g w/o CS w/o CP w/ all
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
[Table]
cause (1) it is not our focus to outperform the state-of-the-art models for structured reconstruction with more learnable parameters that do not consider part kinematics (Kawana et al., 2020; Chen et al., 2020) and (2) the main focus of BAE-Net (Chen et al., 2019b) is part segmentation, rather an accurate reconstruction.
Compared to BAE-Net, PPD shows an improved reconstruction performance, although both the methods employ the same implicit ﬁeld representation.
This is because
PPD enables the use of a deeper network structure of shape decoders for better expressive power, and shape decoders are robust to unseen part poses owing to the disentangled part pose representation.
4. A BLATION STUDIES
We evaluate the effect of the proposed losses and category-common decoders on part segmentation and part pose estimation.
We disable each loss except L rec and L vq one at a time.
We also disable the category-common shape decoders and category-common pose decoder one by one and only use the corresponding instance-dependent decoder(s).
The quantitative results are shown in Table 5.
We see that enabling all losses and the category-common decoders performs the best.
In particular, disabling the category-common shape decoders signiﬁcantly degrades both label IoU and EPE.
This indicates that learning category-common shape prior is essential to perform proper part decomposition and to facilitate part pose learning, which is the core idea of this study.
4. D EPTH MAP INPUT AND REAL DATA
Because the decoders of PPD do not assume a complete shape as an input representation, our method works with depth map input.
Following BSP-Net
(Chen et al., 2020), we train a new encoder that takes a depth map captured from various viewpoints as a partial point cloud and replace the original encoder.
We minimize the mean squared error loss between the output latent vectors of the original encoder and the new encoder so that the latent vectors from the two encoders are close for the same target shape.
The results are shown in Table 6.
The depth map input performs comparably to the complete point cloud input.
We also verify that our model trained on synthetic depth maps reasonably generalizes to real data, as shown in Figure 6.
C ONCLUSION
We propose a novel unsupervised generative part decomposition method, PPD, for articulated objects considering part kinematics.
We show that the proposed method learns the disentangled representation of the part-wise implicit ﬁeld as the decomposed part shapes and the joint parameters of each part as the part poses, using single-frame shape supervision.
We also show that our approach outperforms previous generative part decomposition methods in terms of semantic capability and show comparable part kinematics estimation performance with the supervised baseline.
Finally, we conﬁrm that our model also works on the depth map input and generalizes to real data.
R EPRODUCIBILITY S TATEMENT
For the reproducibility, this paper includes the detailed description of our network architecture in
Appendix A, implementation details on the hyperparameters in Section 3.3 and additional training details in Appendix B including model parameter initialization steps.
Not only our models, but we also describe the training details of the baseline models in Appendix B.1.
We also report the detailed steps of the data preparation process for the synthetic datasets in Appendix C.
We describe the further detail on the data split in Appendix C.1 and data generation steps as well as publicly available source code that we use to generate the data in C.2.
For evaluation, we report the steps for the initial labeling process used to evaluate unsupervised part segmentation results in Appendix D.
A N ETWORK ARCHITECTURE
In this section, we explain the detailed architecture of the proposed network.
The network architectures of the neural networks employed in the proposed method are depicted in Figure 7.
The squircle diagram represents tensors, where the ﬁrst and the second numbers inside the parentheses indicate the channel and the number of points, respectively.
The squircle without the parentheses indicates the scalar value.
For the split operation, N [ X, Y ] denotes the split operation of the input tensor to
N number of sliced tensors with X number of channels with Y points.
For the square diagrams with square brackets, the ﬁrst and the second numbers in the square brackets indicate the input and output channels, respectively.
The green square diagrams indicate multiple identical subnetwork architectures.
P e denotes the number of points in the input point cloud to encoder E .
For the other notations, see Section 3.
We use the simple PointNet architecture in the author-provided code of (Mescheder et al., 2019) as the encoder E .
For the normalization layer in F p,cr and F p,cq , we have experimentally found that using instance normalization (Ulyanov et al., 2016) for F p,cr and layer normalization (Ba et al., 2016) for F p,cq achieves the best performance.
For the joint state s i , we multiply π to { s i | y i = revolute } .
For the discriminator D , we use the architecture based on the PointNet (Qi et al., 2017) implementation in the author-provided code of (Shu et al., 2019).
The weight of each linear layer in our discriminator is normalized using spectral normalization (Miyato et al., 2018) for stable training.
B T RAINING DETAILS
In this section, we explain the implementation and training details of the proposed models.
We train our models per category with the same hyperparameter conﬁguration described in Section 3.3 for all categories.
For the input, we use the point cloud with 4096 points sampled from the surface of the target shape during the training.
Unless otherwise noted, we use the complete shape point cloud.
We use a batch size of 18.
For the ground-truth implicit ﬁeld, for each sample in a batch, we use 4096 3D coordinate points and their corresponding indicator values sampled from either 16 3 or 32 3 grids, depending on the training stage.
This multi-stage training strategy on grids with different resolutions is inspired by (Chen et al., 2020).
We train our network on 16 3 grids in the ﬁrst training stage.
In addition, we set r i =
r c i in the ﬁrst stage.
Then, we set r i =
r c i + r s i in the second stage.
We determine the number of iterations for each stage according to the reconstruction loss and to the visualization of the reconstructed shapes on the validation data.
It takes 2 to 3 days to train one model on a single NVIDIA V100 graphics card with 16 GB of GPU memory.
Model parameter initialization.
We use a sine function as a nonlinear activation function and the weight initialization strategy proposed in (Sitzmann et al., 2020) in our shape decoders, as follows:
[Equation]
where IN is an input channel to a linear layer, U is a uniform distribution, and w is an element of the weight of a linear layer.
For a linear layer that takes 3D coordinates as an input, we do not scale the weight w by 130 .
B. T RAINING OF THE BASELINE MODELS
In this section, we describe the training details of the baseline methods.
We use the author-provided implementations.
[Figure]
BSP-Net (Chen et al., 2020).
Because the models in the author-provided codes of the other part decomposition baselines (BAE-Net (Chen et al., 2019b) and NSD (Kawana et al., 2020)) are trained on 32 3 grids, we also trained BSP-Net on up to 32 3 grids, compared to the 64 3 grids in the original implementation.
For training on the eyeglasses category, we could not successfully train the model even with different random seeds with the provided training script.
After several trials, we experimentally found that scaling ground-truth indicator values by four for the ﬁrst 20,000 iterations produced good initialization of the model.
On the basis of this ﬁnding, we ﬁrst pre-trained the model using the scaled ground-truth indicator values for 20,000 iterations for the eyeglasses category; then, we trained the model with the provided training script.
NSD (Kawana et al., 2020).
The model deﬁned in the author-provided code takes an RGB image as an input, which is a more challenging setting for 3D shape reasoning than 3D shape input.
We replace the image encoder of the original implementation with the same PointNet-based encoder used in our approach for a fair comparison.
NPCS (Xiang et al., 2020).
In the experiment described in Section 4.2, we modiﬁed the original implementation of NPCS to use complete shape point clouds instead of partial point clouds of the depth map as an input with training from scratch, to remove the unnecessary performance degradation caused by pose ambiguity arising from the barely visible articulated part.
C D ATA PREPARATION
In this section, we describe our data preparation approach.
C. D ATA SPLIT
We split our training and test data according to the per-category data split approach introduced in (Li et al., 2020).
We ensure that the test split contains at least six samples per category, except for the laptop category; therefore, the average split ratio is approximately 8:2.
For the laptop category, we use 11 samples in the test split to make the split ratio comparable with those of the other categories.
The number of samples in each split per category is presented in Table 7.
C. G ROUND - TRUTH IMPLICIT FIELD GENERATION
Following (Mescheder et al., 2019), we generate the ground-truth implicit ﬁeld by the volumetric fusion of 100 depth images of a mesh object.
For the mesh object, we sample 100 instances with randomly sampled part poses for each sample.
For the pose sampling, we uniformly sample the rotation amount for each joint for the revolute joints.
For the revolute joints of all categories except the eyeglasses category, we sample the rotation amount between 0 ◦ and 135 ◦ .
For the eyeglasses category, we sample between 0 ◦ and 90 ◦ .
For the prismatic joints of the drawer category, we sample the translation amount between 0 and the maximum amounts of the joints written in the URDF ﬁles of each sample in the SAPIEN dataset (Xiang et al., 2020).
After we sample a part pose for each instance, we articulate the sample in its canonical pose (the rotation amount and translation amount were set to 0 ◦ and 0 , respectively) using the sampled motion amount and ground-truth joint conﬁguration.
The canonically posed shape and the randomly posed shape of the same sample are shown in Figure 8.
Finally, we normalize the size and location of the instances following (Mescheder et al., 2019).
Speciﬁcally, we normalize the instances with the maximum extent collected from the instances generated from the same sample.
D P ART LABELING PROCEDURE FOR EVALUATION
In this section, we explain the labeling procedure using the ground-truth part labels of the training samples to evaluate the part segmentation performance, following the same procedure used in (Kawana et al., 2020; Deng et al., 2020a).
First, for each surface point sampled from the groundtruth part mesh of the instance of the training set, we determine the nearest reconstructed part and vote for the ground-truth part label of that point.
Next, we assign each reconstructed part to the part label that has the highest number of votes.
Finally, for each surface point sampled from the instance in the test split, we determine the nearest reconstructed part surface and assign the part label of the reconstructed part.
E S EMANTIC CAPABILITY E. A DDITIONAL VISUALIZATION OF THE PART SEGMENTATION
We visualize the additional part segmentation results of the proposed approach in Figure 9.
Also, we visualize the part segmentation results given various part poses in Figure 10.
E. P ART SEGMENTATION USING ALL THE TRAINING SAMPLES
In Section 4.1, we show that our method works most efﬁciently by requiring instances with only a limited variety of poses for the initial annotations.
In the experiment discussed in Section 4.1, we use canonically posed shapes, visualized in Figure 8, in the training set for the initial annotations.
This section reports the evaluation setting where annotations of all training instances are available for the initial annotation, which is a favorable setting for the baselines.
However, the annotation cost can be much higher in reality than in the previous setting.
The results are shown in Table 8.
Even under this setting favorable for the previous works, our method performs comparably with the state-of-the-art part decomposition method BSP-Net (Chen et al., 2020) using 256 primitives.
It is not surprising that using many primitives achieves fewer part segmentation errors because, even when one primitive is inconsistently assigned to the ground-truth part, the impact on the label IoU is smaller.
This is because a smaller portion of the evaluation points becomes erroneous compared with the model using fewer parts or primitives.
We also report the evaluation results of BSP-Net using eight primitives, which are comparable with those of other baselines.
In this case, the performance of BSP-Net became comparable to NSD (Kawana et al., 2020) using 10 primitives.
Note that our research focuses on representing ground-truth articulated parts with consistently the same reconstructed parts by considering the part kinematics, unlike BSP-Net and the other baselines, which can
[Figure]
assign different sets of primitives to the same articulated parts without considering the underlying part pose.
To show the effectiveness of considering the part kinematics, we show the performance drop from using all training instances to using only the canonically posed instances in the table under the heading ”Difference.”
We can see that our approach has the least drop, showing that considering the part kinematics contributes to reducing the necessary initial annotation to perform well on the unsupervised part segmentation of articulated objects.
F A DDITIONAL PART POSE EVALUATION
Because we train our model in an unsupervised fashion, through the labeling process described in Appendix D, the part kinematic types of the ground-truth and the assigned reconstructed part do not necessarily match.
Moreover, multiple reconstructed parts may be assigned to one groundtruth part.
Therefore, we choose EPE as the evaluation metric for part pose estimation due to its kinematic type agnostic property and calculation based on point correspondence between prediction and ground-truth, rather than part-level correspondence.
In this section, as an additional part pose evaluation, we evaluate the accuracy of joint parameter estimation for “revolute” and “prismatic” parts.
To avoid the problem of part pose evaluation in unsupervised learning described above, we evaluate the accuracy of joint parameter estimation by considering the prediction is correct when the following three conditions are all satisﬁed.
(1) One reconstructed part is assigned to one ground-truth dynamic part.
The part kinematic type is the same between the ground-truth and the assigned reconstructed part.
The error of the joint parameters against the ground-truth is less than the error threshold.
This evaluation method is more challenging than EPE because of the inﬂuence of (1) and (2) above, besides the prediction error of the joint parameters.
We evaluate joint state accuracy and joint direction accuracy.
Only for the revolute part,
[Figure]
we also evaluate joint axis distance accuracy, deﬁned as the line to line distance between the ground-truth and the predicted line segments consisting of the pivot point and the joint direction.
Figure 14 shows the evaluation results with varying error thresholds.
We show the results of NPCS only as a reference; NPCS is a supervised model and assumes that the part segmentation is available during training and the part kinematic types are also known.
In contrast, our method learns both part segmentation and part kinematic type in an unsupervised fashion.
Since NPCS does not estimate the pivot point, we only show the results of our method for joint axis distance accuracy.
As for the joint state, we see reasonable accuracy of 70.80% for revolute parts on average when the threshold is less than 10 degrees and 79.43% when the threshold is 15 degrees.
For the ”prismatic” part of the drawer, our method outperforms the NPCS when the threshold is less than 0.1.
For joint direction estimation, in three out of ﬁve categories (eyeglasses, laptop, and oven), our method is comparable or outperforming NPCS.
In Table 9, we also show the number of reconstructed parts assigned to the ground-truth parts and the percentage of part kinematic type matches between the ground-truth and the assigned reconstructed parts.
In all categories, the model correctly assigns one part.
Moreover, even without part type supervision, our model successfully predicts correct part types with high accuracy of 91.46%.
Improving the unsupervised learning of joint parameters under shape supervision is an interesting research direction.