Unsupervised Learning of Shape Programs with Repeatable Implicit Parts
Anonymous Author(s)
AffiliationAddress email
Abstract
Shape programs encode shape structures by representing object parts as subroutinesand constructing the overall shape by composing these subroutines.
This usuallyinvolves the reuse of subroutines for repeatable parts, enabling the modeling ofcorrelations among shape elements such as geometric similarity.
However, existing learning-based shape programs suffer from limited representation capacity,because they use coarse geometry representations such as geometric primitivesand low-resolution voxel grids.
Further, their training requires manually annotatedground-truth programs, which are expensive to attain.
We address these limitationsby proposing Shape Programs with Repeatable Implicit Parts (ProGRIP).
Usingimplicit functions to represent parts, ProGRIP greatly boosts the representation capacity of shape programs while preserving the higher-level structure of repetitionsand symmetry.
Meanwhile, we free ProGRIP from any inaccessible supervisedtraining via devising a matching-based unsupervised training objective.
Our empirical studies show that ProGRIP outperforms existing structured representations inboth shape reconstruction fidelity and segmentation accuracy of semantic parts.
Introduction
Representing geometry and structure simultaneously has long been one of the central pursuits of 3Dshape modeling algorithms.
The capability of preserving geometric details indicates the sufficientshape capacity of an algorithm.
Meanwhile, the prediction of plausible structures provides answersto quests for more intelligent shape understanding — What are the parts constituting an object?
Howare they assembled?
What are the relationships among them?
Primitive-based representations have been extensively studied to address such queries.
Specifically,numerous shape primitives are designed to construct objects in a bottom-up manner, includingthe use of oriented boxes [57], ellipsoids [20], convex polytopes [12, 9], superquadrics [43], andinvertible implicit functions [45].
Agnostic to the exact primitive used, these models all predictprimitives independently, where each primitive accounts for a unique part of the object.
Yet, morecommon in real world are recurrences of similar or even identical geometric parts, e.g., legs of a chair.
Consequently, the absence of recurring parts modeling handicaps the advances of these methodstowards human-level understanding of object shapes.
To enable the understanding of shape regularities including the repetition and symmetry of parts, afew recent papers have proposed to cast shape reconstruction as prediction and execution of neural shape programs [55, 25].
The designed domain specific language (DSL) of shape programs supportsthe prediction of repeatable parts.
These methods however suffer from two major limitations.
First,they often fail to achieve accurate geometry reconstruction due to the use of coarse geometricrepresentations, such as primitives or low-resolution voxel grids.
Second, they often rely on handcrafted synthetic datasets with ground-truth programs for training or pre-training [55], which prevents
[Figure]
them from scaling up to various, unseen object categories
— one has to manually annotate shapeprograms for each new category.
In this work, we address these two fundamental limitations by proposing shape programs withrepeatable implicit parts (ProGRIP, Fig. 1), and a method that learns ProGRIP without supervision,illustrated in Fig. 4.
As in earlier studies on shape programs, ProGRIP comprises of repeatable parts,rendering the modeling of recurring structures straightforward; unlike them, we now significantlyenhance the quality of reconstructed shapes by introducing implicit functions as part representations.
Compared with low resolution voxel-based representations, neural implicit representations haverecently demonstrated strikingly high fidelity in shape and part modeling [37, 21].
Reformulatingshape parts in our programs as neural implicit functions leads to detail-preserving reconstruction.
While earlier studies on shape programs require annotated programs for training, we propose anunsupervised learning objective, matching the oriented bounding boxes of predicted repeatable partsto non-repeatable box-based shape decomposition [64], which allows learning from unannotatedshapes.
Unlike the reconstruction loss, which confines each part to fitting local shape in the vicinityof its initialized location, our objective encourages repeatable parts to match to object shape elementswith similar geometry, even if far from their locations.
This leads to unsupervised acquisition ofmeaningful part decomposition, enabling the learning of ProGRIP for any object category.
We conduct extensive experiments on ShapeNet [6] to validate the effectiveness of ProGRIP on shapereconstruction and semantic part segmentation.
Our results demonstrate that ProGRIP consistentlypreserves higher geometric fidelity than state-of-the-art structured representations on all categories.
Meanwhile, it achieves competitive semantic part segmentation accuracy.
In summary, our key contributions are three-fold:
• We propose ProGRIP, introducing implicit functions to represent repeatable parts in shape
programs, greatly enhancing their representation capacity of shape programs
• We develop an unsupervised training objective for learning ProGRIP, relaxing prior requirements of manually designed ground-truth programs.•
We demonstrate the power of ProGRIP representation and learning objective on shapereconstruction, part segmentation, and shape compactness, outperforming state-of-the-artstructured representations on multiple categories.
Related Work
Our method aims to construct primitive-based shape representations with recurring geometric structures.
It builds upon neural shape programs and implements primitives as neural implicit functions.
Thus, we review the relevant literature in the following three fields:
Primitive-based shape modeling .
Most artificial objects naturally exhibit part hierarchy andstructural regularities [40].
Primitive-based shape modeling or abstraction is a decent way of modelingsuch relations.
Some recent works have extended the classic idea of generalized cylinders [48, 3] orgeons [2] by describing shapes as a collection of geometric primitives [57, 66, 49, 55, 25, 44, 32, 31,
[Figure]
In particular, Tulsiani et al.
[57] and Zou et al.
[66] abstracted shapesby minimizing the difference between raw shapes and reconstructions using pre-defined primitives,in an unsupervised fashion.
In Sharma et al.
[49] and Du et al.
[15], 3D CAD programs are inferredfrom the perceptual input of shapes.
In addition to part decomposition, inter-part relations are alsowidely discussed, Tian et al.
[55] defined programs in terms of semantically meaningful parts; Moet al.
[39] built relational graphs upon parts to represent shape structures; Xu et al.
[62] and Ritchieet al.
[47] studied constructive shape modeling processes.
However, these methods usually requiredetailed structural annotations on real or synthetic data, preventing them from being applied to variousunlabeled shape categories.
Visual program induction .
Programs have been commonly used to capture regularities (e.g. representing repetitions by loops) and have been combined with deep models to describe various visualdata, such as 2D graphics [16, 49, 35], 3D shapes [55, 49, 25, 27], scenes [34, 36, 33, 18], andvideos [30].
Most of these works [16, 55, 18, 17, 27, 59] follow the paradigm of analysis by synthesis:a recognition network takes as input visual data and outputs programs, which are rendered backto visual data to compare with the original input; such reconstruction error is back-propagated tooptimize the recognition network.
The rendering can be implemented as either a graphics engine [16]or learned neural models [55].
Our model shares similar spirits with these previous arts, but learns aneural implicit occupancy decoder to render programs into 3D shapes.
Additionally, ShapeMOD [26]assumes shape programs as input and proposes to discover macro ‘options’ to compress them; incontrast, our model aims to infer shape programs and learn to encode parts simultaneously usingimplicit functions.
Also relevant is DeepCAD [61] which utilizes predefined CAD commands tocompose 3D shape programs while our model is not restricted to any predefined shape elements andlearns the programs from scratch.
Neural implicit representations .
Neural implicit representations define surfaces of geometry as zerolevel-sets of volumetric fields parameterized by neural networks [42, 37, 7, 23].
Acquirable fromraw shape data [1, 22, 11], neural implicits show excellent performances in capturing high-frequencydetails at any level [38, 51, 53], even for large-scale scenes [51, 24, 5].
With latent space encodingconditioned on local features [10, 24, 46, 5], poses [14], or parts [13, 41], neural implicits cannaturally incorporate geometric and structural priors into its shape representation.
A few recent worksalso reveal category-level structural similarities through implicit reconstruction, such as Genovaet al.
[20, 21] (via patch-correspondences) and Chen et al.
[8] (via part (co-)segmentation), or crosscategory geometric similarities [56], with zero or minimal supervision.
Recently, Neural Parts [45]also models 3D shapes based on implicit shape elements.
Different from our model, Neural Partstreats each element independently hence ignores essential geometric self-similarties in the modeling.
We summarize the key differences between ProGRIP and most relevant prior methods below andempirically compare ProGRIP against them in Sec. 5.
• CubeSeg [64] is a part-based representation that models shapes as a collection of oriented
boxes.
Each box in the collection is regarded as an independent element with its own shape
and pose.
Contrarily, ProGRIP predicts shape and pose separately and allows the same shape
scale x Query pointx' x
R t inverse transformation implicit part decoder x' o ✔ 🗙 / o existence scaling multiply
Query point canonicalization
Occupancy decoder x' part latent
Occupancyprobability
• Shape2Prog [55] predicts programs with repeatable shapes as voxel grids.
ProGRIP differs
[Figure] from Shape2Prog in using a higher fidelity neural implicit shape representation and learning [Figure]
programs without supervised pretraining.
• BSP-Net [9] is an implicit representation targeting structured shape understanding.
Similar to
CubeSeg, it outputs independent shape elements, while ProGRIP recognizes shape similarity
among elements.
ProGRIP
We model shapes using shape programs with repeatable implicit parts, or ProGRIP (Fig. 1).
Given anobject, we first extract its structural regularity especially repeating geometry via a shape programgenerator.
The generator outputs shape programs with two levels of permutation invariant predictions,parts at the geometry level and poses at the pose level (Sec. 3.1 and Fig. 2) .
The compound setof these predictions are posed parts, which we use to construct object shapes.
Each posed part isessentially a posed implicit function (Sec. 3.2 and Fig. 3).
Composing all posed parts gives us theoverall object shape, which is essentially a spatial occupancy function O : R 3 → [0 , 1] with itssurface boundary defined as ∂ O = { x ∈ R
| O ( x ) = 0 . 5 } .
3.1 Generating ProGRIP
Unlike the sequential programs in Tian et al.
[55], we introduce parallelism into our program, allowingrepeatable parts to be executed simultaneously without a specified order.
Specifically, our programgenerator takes as input a point cloud X ∈ R P × 3 with P points and interprets it as a ProGRIP usingan auto-encoding architecture (Fig. 2).
A ProGRIP consists of a two-level hierarchy.
The top level,namely geometry level, defines a set of repeatable parts as shape building blocks, e.g. the shape of aleg on a chair.
The bottom level, namely pose level, predicts a set of poses for each repeatable partfrom the geometry level, e.g., four copies of the leg shape in four different poses on a chair.
Withinone pose set are the locations and orientations of all occurrences of the corresponding part in theobject (Fig. 1).
All predictions at the pose level, joint by the prediction from its parent part, specifyall element posed parts (e.g., four legs of a table) that constitute the object shape.
Formally, by aslight abuse of notations, we define the hierarchy:
Geometry Level : { ( s i , z i ) | i = 1 , . . .
, N } , where s i ∈ R 3 describes the axis-aligned
scales of the i -th repeatable part, z i ∈ R d encodes the d -dimensional deep latent of this part,and N is the number of parts at this level.
Pose Level : { ( t i,j , R i,j , δ i,j ) | j = 1 , . . .
, M } for the i -th part, where t i,j , R i,j , and δ i,jare the translation, rotation, and existence probability of the j -th posed occurrence of this
part.
Note that during program generation, we predict continuous probability as δ i,j to
allow back-propagation of gradients, while during program execution, we binarise δ i,j into
ˆ δ i,j = 1 ( δ i,j ≥ 0 . 5) and remove
[Figure]
In adaptation to the program parallelism, the predictions at both the geometry level and the poselevel are orderless sets .
We employ two nested transformers [58] in our architecture to facilitatepermutation invariance at both prediction levels.
In particular, the geometry transformer takes in theencoded global feature for the object and predicts the the parts including the scale s i and a deep partlatent z i .
Then, we feed each z i to pose transformer to predict the ( R i,j , t i,j , δ i,j ) triplets for each occurrence.
Combining the pose predictions and their parent part prediction produces a set of{ ( j j
3.2 Executing ProGRIP
We execute ProGRIP by executing the predicted posed parts as posed implicit functions [37, 42, 7].
Instead of simple polygonal primitives with only a few shape parameters, we propose to use animplicit function P ( · | z ) : R 3 → [0 , 1] to represent a part.
It takes in any point query x
and predictsthe likelihood of x being inside of a part.
P is the occupancy decoder parameterized by a deep neuralnetwork and shared by all parts.
z is the part latent from the geometry level.
We condition P onz to obtain a specific implicit function for a part.
For a posed part parameterised by the 5 -tuple( s i , z i , t i,j , R i,j , δ i,j ) , we construct its posed implicit function o i,j as
[Equation]
Note that we binarize δ i,j to ˆ δ i,j .
As in Fig.
3, given a query point x , we first canonicalize itscoordinates to x ′ =
( R − 1 i,j ( x − t
i,j )) / s i .
Then we feed x ′ to P together with the shape code z i .
Thebinarized existence probability ˆ δ i,j is finally multiplied to the predicted occupancy, masking out thenon-existing primitives.
The eventual output of the program is an object shape as the union of allposed parts.
The object occupancy function can therefore be written as
[Equation]
Unsupervised Learning of ProGRIP
As our shape modeling consists of two steps, i.e., the generation and execution of ProGRIP, we firstlearn the ProGRIP generator and then optimize our posed implicit functions.
Both optimizations aredone without any supervision.
4.1 Learning to Generate ProGRIP
Prior works on generating shape programs often rely on hand-crafted synthetic programs to bootstrapthe learning [55].
As manually creating programs for numerous object categories is prohibitive, thissignificantly restricts large-scale applications of these works.
To circumvent this limitation, one wouldresort to unsupervised training objectives.
A trivial design of such objectives is reconstruction losses.
Particularly, we only supervise the composition of all elements by the overall reconstruction quality.
Despite its decent effectiveness in prior works [12, 9, 57, 64], we show in one demonstrative case(Fig. 4) that it’s not applicable when we have repeatable parts.
Reconstruction loss encourages eachposed part to evolve to the local geometric structure closest to its initialized location.
When parts are repeatable, different copies of the same part may appear in irrelevant locations, e.g., one near a chairleg while another close to the chair seat.
The local minimal of reconstruction loss consequently trapsthese copies to its initially assigned neighborhood and suppresses the transformation that would leadto more meaningful assignment of these copies.
To address this issue, we devise a matching-basedunsupervised objective that supervises a posed part using a local geometry of the object with asimilar shape, even if it’s far from the current location of this posed part.
Practically, we match ourpredicted posed repeatable parts to a non-repeatable primitive decomposition of the object, whereeach primitive serves as a local geometry descriptor.
We acquire the non-repeatable primitive decomposition using the unsupervised box-based algorithmin Yang and Chen [64].
Namely, given pointcloud X , it predicts a set of independent orientedbounding boxes { ( s k , t k , R k , δ k ) | k
= 1 , . .
. , NM } .
Here δ k is a binary value indicating if the k -thbox exists ( 1 for present and 0 for absent).
If the number of valid boxes is less than NM (i.e., thenumber of posed parts predicted by our ProGRIP), we pad the remaining entries with all zero.
As thesubset 4 -tuple { ( s k , t k , R k , δ k ) } from the 5 -tuple specifying a posed part also defines its orientedbounding box, we optimize our predicted primitives by fitting these boxes to { ( s k , t k , R k , δ k ) } .
Specifically, we use a box scale loss L s , as well as two auxiliary losses for the box vertices ( L v )and existence ( L e ) to supervise the fitting of boxes.
The overall loss is a weighted combination
L m of these three losses.
We name it matching loss.
Inspired by Carion et al.
[4], we base thecomputation of all loss terms on a permutation ρ of NM elements, where ρ ( k ) is the k -th element inthe permutation.
Without loss of generality, we define L s , L v , and L e for the k -th predicted posedpart as the following:
Scale loss L s .
We first supervise the shape of the posed part using the target local geometry as weseek to find the repeated shape elements in an object.
In the context of oriented boxes, we fit theaxis-aligned scales of our predicted boxes to their assigned target boxes from the non-repeatabledecomposition.
We implement it as the complementary of Intersection-over-Union between twoaxis-aligned origin-centered boxes defined by scales s k and s ρ ( k ) :
[Equation]
Vertices loss L v .
We also require the matched predicted box and the target box to have the samelocation and orientation by minimizing the l 1 -distance between the box vertices after posing.
Formally,we define the box vertices generated using ( R k , t k , s k ) as B k and similarly B ρ ( k ) for ( R k , t k , s k ) .
Consequently,
[Equation]
Existence loss L e .
As we are matching two sets with possibly different number of elements, wesynchronize the existence of each element appearing in the two sets.
To enforce such a coherency, weminimize the cross-entropy for the existence probabilities between each pair of matched predictionand target bounding boxes:
[Equation]
Therefore the matching loss L m of an object is defined as
[Equation]
During optimization, we select the optimal permutation ρ ∗ among all permutations S NM to compute
L m and apply gradient descent.
Particularly, at each step, we search for ρ ∗ =
arg min ρ ∈ S NM L musing Hungarian algorithm
In our experiments, we use λ s = 1 , λ v = 0 . 2 , and λ e = 0 . 8 for allcategories.
Here, we provide an intuitive understanding of our optimization process.
First, our objective anchorsone copy of an initialized shape to a nearby part on the object with similar scales, e.g., a leg of achair, due to the joint impact of L s and L v .
Then, as the gradient on scales will be propagated toother copies of the same shape, these copies will converge to leg shapes as well even if they are notmatched to legs initially.
In later rounds of optimization, these copies’ poses will also converge tothe nearest legs on the chair because of the higher weight on L s .
In the end, ProGRIP learns therecurring leg shapes on a chair that are posed correctly.
[Figure]
4.2 Learning to Execute ProGRIP
[Figure]
After the optimization in Sec. 4.1, ProGRIP can predict the repeatable oriented bounding boxes ofeach shape element.
We further train ProGRIP to refine the shape element within each box as posedimplicit functions.
As defined in Sec. 3.2, the outcome of executing ProGRIP is an object occupancyfunction O .
To optimize this function, we uniformly sample points x within the padded boundingbox of an object and collect the ground truth occupancy for each points O ( x ) , where 1 representsinside points and 0 outside.
We further augment the samples by sampling points near the surfaceand sampling the point cloud of the object.
Note that we define the occupancy of on-surface pointsx ∈
∂ O as O ( x ) = 0 .
The loss L o for a set of C samples from an object is averaged cross entropy:
[Equation]
Experiments
We conduct all our experiments using the ShapeNet [6] dataset following ShapeNet Terms of Use.
We quantitatively evaluate the performance of our model on the reconstruction (Sec. 5.1) and partsegmentation (Sec. 5.2) tasks.
Additionally, we dive deep into our method via investigating the impactof our unsupervised matching loss (Sec. 5.3).
We also showcase applications of ProGRIP in shapeediting and compact shape representations in our supplementary material , along with failure cases.
Metrics .
We quantitatively evaluate our method on its reconstruction quality and segmentationaccuracy.
For reconstruction, we use two quantitative criteria: 1 ⃝ The volume-based Intersection-overUnion (IoU), where we uniformly sample points and evaluate the IoU of our predicted occupancies
[Table]
and the ground truth occupancies.
Note that this metric is focused on the accuracy of the volumereconstruction.
2 ⃝ The surface-based F-Score, where we follow the protocol defined in Tatarchenkoet al.
[54] to compute the “percentage of surface that is correctly reconstructed”.
It is claimed to bethe best metric to assess the fidelity of the reconstruction, with a concentration on the surface.
Forsegmentation, we use the mean per-label Intersection-over-Union (mIoU) to measure the alignmentof our predicted segmentation and the ground truth labels.
As our segmentation task is unsupervised,to compute the mIoU during evaluation, we assign the ground truth label to each predicted label by amajority voting, i.e., picking the ground truth label with the highest correlation in the training split.
Baselines .
We compare against representative state-of-the-art methods from relevant domains asdiscussed in Sec. 2.
Specifically, we select CubeSeg [64] for primitive-based shape representation,
Shape2Prog [55] for shape programs, and BSP-Net [9] for implicit or hybrid shape representations.
Note that while LDIF [21] is also an implicit part-based representation, it strives for high fidelityreconstruction rather than better structural understanding.
We don’t directly compare to it due todifferent encoding schemes, i.e. local feature encoding (LDIF) vs. global feature encoding (ProGRIP).
Implementation details .
We use an auto-encoding architecture for our model.
The encodingarchitecture is identical to Yang and Chen [64] and the programs are predicted by 2 levels of
Transformers [58].
We first train our ProGRIP generator (Fig. 2) using the decomposition from Yangand Chen [64] and L m .
Then we train the ProGRIP execution module (Fig. 3) as well as fine-tune theprogram generator by fitting occupancy predictions to ground truth occupancies of samples.
Pleasesee our supplementary materials for more implementation details.
5.1 Shape Reconstruction
As we extend independent non-repeatable primitive modeling to repeatable geometry modeling, weboost the efficiency of geometry learning — the same repeatable part can receive gradients fromits copies at various local neighborhoods of the object.
Additionally, we employ implicit shaperepresentations as our neural implicit primitive to greatly enrich the capacity of our primitive shapespace.
We validate these two improvements on the point cloud-to-mesh 3D reconstruction task.
Inthis task, each model takes as input a point cloud with 4096 points sampled from the object surfaceand strives for reconstructing the object surface with high fidelity.
We compare against baseline
[Figure]
methods quantitatively in Tab. 1, Reconstruction and display a few qualitative visualizations in Fig.
We find that our method consistently outperforms other primitive-based baselines in reconstructionquality.
Meanwhile, our predicted repeatable parts are reasonably posed to fit local geometry withsimilar or identical shapes, such as the four legs of a chair.
5.2 Unsupervised Part Segmentation
We further quantitatively study the primitive assignment in our shape programs on the unsupervisedpart segmentation task.
In this task, we aim to learn to segment each object into semantic parts withoutany segmentation supervision.
Specifically, we provide the point cloud of an object to each methodand ask for per-point semantic part segmentation labels.
Each method is trained on reconstructiontasks only and we use their part decomposition at test time to evaluate the segmentation quality.
In
Tab. 1, Unsupervised Part Segmentation, we show the per-label mIoU of different primitive-basedmethods and find that our ProGRIP achieves competitive performance in comparison to state-of-theart unsupervised methods, such as CubeSeg and BSP-Net; it even outperforms Shape2Prog, whichuses synthetic semantic structure ground truth for pretraining.
5.3 Analysis: Impact of Matching Loss
We present the matching loss L m in Sec.
4.1 as a better fit for identifying and learning repeatableparts than the commonly used reconstruction loss.
Here we provide an experimental study to validateour design choice.
We visualize the results in Fig.
In this study, we train the same ProGRIP modelusing either our matching loss L m on abstracted non-repeatable boxes or the reconstruction lossdirectly from the ground-truth shape as defined in Yang and Chen [64].
We find that ProGRIP learnedfrom the matching loss produces both decent quality of reconstruction and meaningful repeatableparts arrangement.
This can be observed in the symmetric arms of the chair that is fitted by twocopies of the same repeatable part and also in the four repeated legs.
In contrast, the part allocation of
ProGRIP learned directly from the reconstruction loss appears to be of poor quality elucidating thedifficulty of learning part decomposition directly from reconstruction.
The reconstruction itself isalso crippled by the unwise part assignment as each repeatable part is receiving conflicting gradients.
Conclusion
We propose to represent objects as shape programs with repeatable implicit parts, or ProGRIP, astructured high fidelity shape representation that can be learned without structure annotations.
Whileprior works on shape programs suffer from poor fidelity due to the limited representation capacityof their coarse shape primitives, we significantly augment the reconstruction quality by introducingimplicit functions to implement the predicted parts.
Additionally, we design an unsupervised trainingobjective to facilitate the learning of ProGRIP on any object category, largely enhancing the scalabilityof shape programs compared to prior works that only work with few categories with ground truthsemantic structures.
On the shape reconstruction task, our ProGRIP champions on all categoriesamong structured representations.
It also achieves top-tier part segmentation accuracy and is shownto be an accurate yet highly compact representation capturing repetitions and symmetries of objectparts, enabling efficient shape editing.