V OLUMETRIC C ONVOLUTION : A UTOMATIC R EPRESENTATION L EARNING IN U NIT B ALL
Anonymous authors
Paper under double-blind review
A BSTRACT
Convolution is an efﬁcient technique to obtain abstract feature representations using hierarchical layers in deep networks.
Although performing convolution in
Euclidean geometries is fairly straightforward, its extension to other topological spaces—such as a sphere ( S 2 ) or a unit ball ( B 3 )—entails unique challenges.
In this work we propose a novel ‘ volumetric convolution ’ operation that can effectively convolve arbitrary functions in B 3 .
We develop a theoretical framework for volumetric convolution based on Zernike polynomials and efﬁciently implement it as a differentiable and an easily pluggable layer for deep networks.
Furthermore, our formulation leads to derivation of a novel formula to measure the symmetry of a function in B 3 around an arbitrary axis, that is useful in 3D shape analysis tasks.
We demonstrate the efﬁcacy of proposed volumetric convolution operation on a possible use-case i.e., 3D object recognition task.
I NTRODUCTION
Convolution-based deep neural networks have performed exceedingly well on 2D representation learning tasks (Krizhevsky et al., 2012;
He et al., 2016).
The convolution layers perform parameter sharing to learn repetitive features across the spatial domain while having lower computational cost by using local neuron connectivity.
However, most state-of-the-art convolutional networks can only work on Euclidean geometries and their extension to other topological spaces e.g., spheres, is an open research problem.
Remarkably, the adaptation of convolutional networks to spherical domain can advance key application areas such as robotics, geoscience and medical imaging.
Some recent efforts have been reported in the literature that aim to extend convolutional networks to spherical signals.
Initial progress was made by Boomsma & Frellsen (2017), who performed conventional planar convolution with a careful padding on a spherical-polar representation and its cube-sphere transformation (Ronchi et al., 1996).
A recent pioneering contribution by Cohen et al.
(2018) used harmonic analysis to perform efﬁcient convolution on the surface of the sphere ( S 2 ) to achieve rotational equivariance.
These works, however, do not systematically consider radial information in a 3D shape and the feature representations are learned at speciﬁed radii.
Speciﬁcally,
Cohen et al.
(2018) estimated similarity between spherical surface and convolutional ﬁlter in S 2 , where the kernel can be translated in S 2 .
Furthermore, Weiler et al.
(2018) recently solved the more general problem of SE(3) equivariance by modeling 3D data as dense vector ﬁelds in 3D Euclidean space.
In this work however, we focus on B 3 to achieve the equivariance to SO(3).
In this paper, we propose a novel approach to perform volumetric convolutions inside unit ball ( B 3 ) that explicitly learns representations across the radial axis.
Although we derive generic formulas to convolve functions in B 3 , we experiment on one possible use case in this work, i.e., 3D shape recognition.
In comparison to closely related spherical convolution approaches, modeling and convolving 3D shapes in B 3 entails two key advantages: ‘ volumetric convolution ’ can capture both 2D texture and 3D shape features and can handle non-polar 3D shapes.
We develop the theory of volumetric convolution using orthogonal Zernike polynomials (Canterakis, 1999), and use careful approximations to efﬁciently implement it using low computational-cost matrix multiplications.
Our experimental results demonstrate signiﬁcant boost over spherical convolution and that conﬁrm the high discriminative ability of features learned through volumetric convolution.
Furthermore, we derive an explicit formula based on Zernike Polynomials to measure the axial symmetry of a function in B 3 , around an arbitrary axis.
While this formula can be useful in many function analysis tasks, here we demonstrate one particular use-case with relevance to 3D shape recognition.
Speciﬁcally, we use the the derived formula to propose a hand-crafted descriptor that accurately encodes the axial symmetry of a 3D shape.
Moreover, we decompose the implementation of both volumetric convolution and axial symmetry measurement into differentiable steps, which enables them to be integrated to any end-to-end architecture.
Finally, we propose an experimental architecture to demonstrate the practical usefulness of proposed operations.
We use a capsule network after the convolution layer as it allows us to directly compare feature discriminability of spherical convolution and volumetric convolution without any bias.
In other words, the optimum deep architecture for spherical convolution may not be the same for volumetric convolution.
Capsules, however, do not deteriorate extracted features and the ﬁnal accuracy only depends on the richness of input shape features.
Therefore, a fair comparison between spherical and volumetric convolutions can be done by simply replacing the convolution layer.
It is worth pointing out that the proposed experimental architecture is only a one possible example out of many possible architectures, and is primarily focused on three factors: 1) Capture useful features with a relatively shallow network compared to state-of-the-art. 2)
Show richness of computed features through clear improvements over spherical convolution.
3) Demonstrate the usefulness of the volumetric convolution and axial symmetry feature layers as fully differentiable and easily pluggable layers, which can be used as building blocks for end-to-end deep architectures.
The main contributions of this work include:
• Development of the theory for volumetric convolution that can efﬁciently model functions in B 3 .
• Implementation of the proposed volumetric convolution as a fully differentiable module that can be plugged into any end-to-end deep learning framework.
The ﬁrst approach to perform volumetric convolution on 3D objects that can simultaneously model 2D (appearance) and 3D (shape) features.
• A novel formula to measure the axial symmetry of a function deﬁned in B 3 , around an arbitrary axis using Zernike polynomials.
• An experimental end-to-end trainable framework that combines hand-crafted feature representation with automatically learned representations to obtain rich 3D shape descriptors.
The rest of the paper is structured as follows.
In Sec.
2 we introduce the overall problem and our proposed solution.
Sec.
3 presents an overview of 3D Zernike polynomials.
Then, in Sec. 4 and
Sec.
5 we derive the proposed volumetric convolution and axial symmetry measurement formula respectively.
Sec.
6.2 presents our experimental architecture, and in Sec.
7 we show the effectiveness of the derived operators through extensive experiments.
Finally, we conclude the paper in Sec. 8.
P ROBLEM DEFINITION
Convolution is an effective method to capture useful features from uniformly spaced grids in R n , within each dimension of n , such as gray scale images ( R 2 ), RGB images ( R 3 ), spatio-temporal data ( R 3 ) and stacked planar feature maps ( R n ).
In such cases, uniformity of the grid within each dimension ensures the translation equivariance of the convolution.
However, for topological spaces such as S 2 and B 3 , it is not possible to construct such a grid due to non-linearity.
A naive approach to perform convolution in B 3 would be to create a uniformly spaced three dimensional grid in ( r, θ, φ ) coordinates (with necessary padding) and perform 3D convolution.
However, the spaces between adjacent points in each axis are dependant on their absolute position and hence, modeling such a space as a uniformly spaced grid is not accurate.
To overcome these limitations, we propose a novel volumetric convolution operation which can effectively perform convolution on functions in B 3 .
It is important to note that ideally, the convolution in B 3 should be a signal on both 3D rotation group and 3D translation.
However, since Zernike polynomials do not have the necessary properties to automatically achieve translation equivariance, we stick to 3D rotation group in this work and refer to this operation as convolution from here onwards.
Fig.
1 shows the analogy between planar convolution and volumetric convolution.
In
Sec. 3, we present an overview of 3D Zernike polynomials that will be later used in Sec.
4 to develop volumetric convolution operator.
3D Z ERNIKE P OLYNOMIALS
3D Zernike polynomials are a complete and orthogonal set of basis functions in B 3 , that exhibits a ‘ form invariance ’ property under 3D rotation (Canterakis, 1999).
A ( n, l, m ) th order 3D Zernike basis function is deﬁned as,
[Equation]
where R n,l is the Zernike radial polynomial (Appendix D.3), Y l,m ( θ, φ ) is the spherical harmonics function (Appendix D.1), n ∈ Z + , l ∈ [0 , n ] , m ∈ [ − l, l ] and n − l is even.
Since 3D Zernike polynomials are orthogonal and complete in B 3 , an arbitrary function f ( r, θ, φ ) in B 3 can be approximated using Zernike polynomials as follows.
[Equation]
where Ω n,l,m ( f ) could be obtained using,
[Equation] [Equation]
where † denotes the complex conjugate.
In Sec. 4, we will derive the proposed volumetric convolution.
V OLUMETRIC CONVOLUTION OF FUNCTIONS IN B
When performing convolution in B 3 , a critical problem which arises is that several rotation operations exist for mapping a point p to a particular point p (cid:48) .
For example, using Euler angles, we can decompose a rotation into three rotation operations R ( θ, φ ) =
R ( θ ) y R ( φ )
z R ( θ ) y , and the ﬁrst rotation R ( θ ) y can differ while mapping p to p (cid:48)
(if y is the north pole).
However, if we enforce the kernel function to be symmetric around y , the function of the kernel after rotation would only depend on p and p (cid:48) .
This observation is important for our next derivations because we can then uniquely deﬁne a 3D rotation on kernel in terms of azimuth and polar angles.
Let the kernel be symmetric around y and f ( θ, φ, r ) , g ( θ, φ, r ) be the functions of object and kernel respectively.
Then we deﬁne volumetric convolution as,
[Equation]
where τ ( α,β ) is an arbitrary rotation, that aligns the north pole with the axis towards ( α, β ) direction ( α and β are azimuth and polar angles respectively).
Eq.
4 is able to capture more complex patterns compared to spherical convolution due to two reasons: 1) the inner product integrates along the radius and 2) the projection onto spherical harmonics forces the function into a polar function, that can result in information loss.
In Sec.
4.1 we derive differentiable relations to compute 3D Zernike moments for functions in B 3 .
S HAPE MODELING OF FUNCTIONS IN B 3 USING 3D Z ERNIKE POLYNOMIALS
Instead of using Eq.
3, we derive an alternative method to obtain the set { Ω n,l,m } .
The motivations are two fold: 1) ease of computation and 2) the completeness property of 3D Zernike Polynomials ensures
that lim n
[Equation]
Z
[Equation]
,l,m (cid:13)(cid:13)
=
for any arbitrary function f .
However, since n should be ﬁnite in the implementation, aforementioned property may not hold, leading to increased distance between the Zernike representation and the original shape.
Therefore, minimizing the reconstruction error (cid:80) ( θ,φ,r ) ∈ S 3 (cid:12)(cid:12)
¯ f ( θ, φ, r ) − f ( θ, φ, r ) (cid:12)(cid:12) where ¯ f ( θ, φ, r )
= (cid:80) Nn (cid:80) l (cid:80) m Ω n,l,m Z n,l,m , pushes the set { Ω n,l,m } inside frequency space, where { Ω n,l,m } has a closer resemblance to the corresponding shape.
Following this conclusion, we derive the following method to obtain { Ω n,l,m } .
Since Y l,m ( θ, φ )
=
( − 1) m (cid:113) 2 l +14 π ( l − m )!
( l + m )!
P ml ( cosφ ) e imθ , where P ml ( · ) is the associated Legendre function (Appendix D.2), it can be deduced that, Y l, − m ( θ, φ ) =
( − 1)
m Y † l,m ( θ, φ ) .
Using this relationship we obtain Z n,l, − m ( θ, φ ) =
( − 1)
m Z † n,l,m ( θ, φ ) and hence approximate Eq.
2 as,
[Equation]
where Re { Z n,l,m } and Img { Z n,l,m } are real and imaginary components of Z n,l,m respectively.
In matrix form, this can be rewritten as,
[Equation]
where c is the set of 3D Zernike moments Ω n,l,m .
Eq.
6 can be interpreted as an overdetermined linear system, with the set Ω n,l,m as the solution.
To ﬁnd the least squared error solution to the Eq.we use the pseudo inverse of X .
Since this operation has to be differentiable to train the model end-to-end, a common approach like singular value decomposition cannot be used here.
Instead, we use an iterative method to calculate the pseudo inverse of a matrix (Li et al., 2011).
It has been shown that V n converges to A + where A + is the Moore-Penrose pseudo inverse of A if,
[Equation]
for a suitable initial approximation V 0 .
They also showed that a suitable initial approximation would be V 0 = αA T with 0 < α < 2 /ρ
( AA T ) , where ρ ( · ) denotes the spectral radius.
Empirically, we choose α = 0 .
001 in our experiments.
Next, we derive the theory of volumetric convolution within the unit ball.
4. C ONVOLUTION IN B 3 USING 3D Z ERNIKE POLYNOMIALS
We formally present our derivation of volumetric convolution using the following theorem.
A short version of the proof is then provided.
Please see Appendix A for the complete derivation.
Theorem 1: Suppose f, g : X −→ R 3 are square integrable complex functions deﬁned in Bso
that (cid:104) f, f (cid:105) < ∞ and (cid:104) g, g (cid:105) < ∞ .
Further, suppose g is symmetric around north pole and τ ( α, β )
=
R y
( α ) R z ( β ) where R ∈ SO (3) .
Then,
[Equation]
8) where Ω n,l,m ( f ) , Ω n,l, 0 ( g ) and Y l,m ( θ, φ ) are ( n, l, m )
th 3D Zernike moment of f , ( n, l, 0)
th 3D
Zernike moment of g , and spherical harmonics function respectively.
Proof: Completeness property of 3D Zernike Polynomials ensures that it can approximate an arbitrary function in B 3 , as shown in Eq. 2.
Leveraging this property, Eq.
4 can be rewritten as,
[Equation]
However, since g ( θ, φ, r ) is symmetric around y , the rotation around y should not change the function.
This ensures,
[Equation]
and hence,
[Equation]
This is true,
if and only if m (cid:48) = 0 .
Therefore, a symmetric function around y , deﬁned inside the unit sphere can be rewritten as,
[Equation]
which simpliﬁes Eq.
9 to,
[Equation]
Using the properties of inner product, Eq. 13 can be rearranged as,
[Equation]
Using the rotational properties of Zernike polynomials, we obtain (see Appendix A for our full derivation),
[Equation]
Since we can calculate Ω n,l,m ( f ) and Ω n,l, 0 ( g ) easily using Eq. 6, f ∗ g ( θ, φ ) can be found using a simple matrix multiplication.
It is interesting to note that, since the convolution kernel does not translate, the convolution produces a polar shape, which can be further convolved–if needed–using the relationship
[Equation]
, φ ) where, ˆ f ( l, m ) and ˆ g ( l, m ) are the ( l, m ) th frequency components of f and g in spherical harmonics space.
Next, we present a theorem to show the equivariance of volumetric convolution with respect to 3D rotation group.
4. E QUIVARIANCE TO 3D ROTATION GROUP
One key property of the proposed volumetric convolution is its equivariance to 3D rotation group.
To demonstrate this, we present the following theorem.
Theorem 1: Suppose f, g : X −→ R 3 are square integrable complex functions deﬁned in B 3 so that (cid:104) f, f (cid:105) < ∞ and (cid:104) g, g (cid:105) < ∞ .
Also, let η α,β,γ be a 3D rotation operator that can be decomposed into three Eular rotations R y ( α ) R z ( β ) R y ( γ ) and τ α,β another rotation operator that can be decomposed into R y
( α ) R z ( β ) .
Suppose η α,β,γ ( g ) =
τ α,β ( g ) .
Then, η ( α,β,γ ) ( f ) ∗ g ( θ, φ ) =
τ ( α,β ) ( f ∗ g )( θ, φ ) , where ∗ is the volumetric convolution operator.
The proof to our theorem can be found in Appendix B.
The intuition behind the theorem is that if a 3D rotation is applied to a function deﬁned in B 3 Hilbert space, the output feature map after volumetric convolution exhibits the same rotation.
The output feature map however, is symmetric around north pole, hence the rotation can be uniquely deﬁned in terms of azimuth and polar angles.
A XIAL S YMMETRY OF FUNCTIONS IN B
In this section we present the following proposition to obtain the axial symmetry measure of a function in B 3 , around an arbitrary axis using 3D Zernike polynomials.
Proposition: Suppose g : X −→ R 3 is a square integrable complex function deﬁned in B 3 such that (cid:104) g, g (cid:105) < ∞ .
Then, the power of projection of g in to S =
{ Z i } where S is the set of Zernike basis functions that are symmetric around an axis towards ( α, β ) direction is given by,
[Equation]
where α and β are azimuth and polar angles
The proof to our proposition is given in Appendix C.
A C ASE S TUDY : 3D O BJECT R ECOGNITION 6. 3D OBJECTS AS FUNCTIONS IN B
A 2D image is a function on Cartesian plane, where a unique value exists for any ( x, y ) coordinate.
Similarly, a polar 3D object can be expressed as a function on the surface of the sphere, where any direction vector ( θ, φ ) has a unique value.
To be precise, a 3D polar object has a boundary function in the form of f : S 2 → [0 , ∞ ] .
Translation of the convolution kernel on ( x, y ) plane in 2D case, extends to movements on the surface of the sphere in S 2 .
If both the object and the kernel have polar shapes, this task can be tackled by projecting both the kernel and the object onto spherical harmonic functions (Appendix E).
However, this technique suffers from two drawbacks.
1) Since spherical harmonics are deﬁned on the surface of the unit sphere, projection of a 3D shape function into spherical harmonics approximates the object to a polar shape, which can cause critical loss of information for non-polar 3D shapes.
This is frequently the case in realistic scenarios.
The integration happens over the surface of the sphere, which is unable to capture patterns across radius.
These limitations can be addressed by representing and convolving the shape function inside the unit ball ( B 3 ).
Representing the object function inside B 3 allows the function to keep its complex shape information without any deterioration since each point is mapped to unique coordinates ( r, θ, φ ) , where r is the radial distance, θ and φ are azimuth and polar angles respectively.
Additionally, it allows encoding of 2D texture information simultaneously.
Figure 2 compares volumetric convolution and spherical convolution.
Since we conduct experiments only on 3D objects with uniform surface values, in this work we use the following transformation to apply a simple surface function f
( θ, φ, r ) to the 3D objects:
[Equation]
6. A N EXPERIMENTAL ARCHITECTURE
We implement an experimental architecture to demonstrate the usefulness of the proposed operations.
While these operations can be used as building-tools to construct any deep network, we focus on three key factors while developing the presented experimental architecture: 1) Shallowness: Volumetric convolution should be able to capture useful features compared to other methodologies with less number of layers.
2) Modularity: The architecture should have a modular nature so that a fair comparison can be made between volumetric and spherical convolution.
We use a capsule network after the convolution layer for this purpose.
3) Flexibility: It should clearly exhibit the usefulness
[Figure]
of axial symmetry features as a hand-crafted and fully differentiable layer.
The motivation is to demonstrate one possible use case of axial symmetry measurements in 3D shape analysis.
The proposed architecture consists of four components.
First , we obtain three view angles, and later generate features for each view angle separately.
We optimize the view angles to capture complimentary shape details such that the total information content is maximized.
For each viewing angle ‘ k ’, we obtain two point sets P + k and P − k consisting of tuples denoted as:
[Equation]
[Equation]
such that y denotes the horizontal axis.
Second , the six point sets are volumetrically convolved with kernels to capture local patterns of the object.
The generated features for each point set are then combined using compact bilinear pooling.
Third , we use axial symmetry measurements to generate additional features.
The features that represent each point set are then combined using compact bilinear pooling.
Fourth , we feed features from second and third components of the overall architecture to two independent capsule networks and combine the outputs at decision level to obtain the ﬁnal prediction.
The overall architecture of the proposed scheme is shown in Fig.
6. O PTIMUM VIEW ANGLES
We use three view angles to generate features for better representation of the object.
First, we translate the center of mass of the set of ( x, y, z ) points to the origin.
The goal of this step is to achieve a general translational invariance, which allows us to free the convolution operation from the burden of detecting translated local patterns.
Subsequently, the point set is rearranged as an ordered set on x and z and a 1D convolution net is applied on y values of the points.
Here, the objective is to capture local variations of points along the y axis, since later we analyze point sets P + and P − independently.
The trained ﬁlters can be assumed to capture properties similar to ∂ n y/∂x n and ∂ n y/∂z n , where n is the order of derivative.
The output of the 1D convolution net is rotation parameters represented by a 1 × 9 vector (cid:126)r = { r 1 , r 2 , · · · , r 9 } .
Then, we compute
R 1 =
R x ( r 1 ) R y ( r 2 )
R z ( r 3 ) , R 2 =
R x ( r 4 ) R y ( r 5 ) R z ( r 6 ) and R 3 =
R x ( r 7 ) R y ( r 8 )
R z ( r 9 ) where
R 1 , R 2 and R 3 are the rotations that map the points to three different view angles.
After mapping the original point set to three view angles, we extract the P + k and P − k point sets from each angle k that gives us six point sets.
These sets are then fed to the volumetric convolution layer to obtain feature maps for each point set.
We then measure the symmetry around four equi-angular axes using Eq. 16, and concatenate these measurement values to form a feature vector for the same point sets.
6. F EATURE FUSION USING COMPACT BILINEAR POOLING
Compact bilinear pooling (CBP) provides a compact representation of the full bilinear representation, but has the same discriminative power.
The key advantage of compact bilinear pooling is the signiﬁcantly reduced dimensionality of the pooled feature vector.
We ﬁrst concatenate the obtained volumetric convolution features of the three angles, for P + and P − separately to establish two feature vectors.
These two features are then fused using compact bilinear pooling (Gao et al., 2016).
The same approach is used to combine the axial symmetry features.
These fused vectors are fed to two independent capsule nets.
Furthermore, we experiment with several other feature fusion techniques and present results in
Sec. 7.2.
6. C APSULE N ETWORK
Capsule Network (CapsNet) (Sabour et al., 2017) brings a new paradigm to deep learning by modeling input domain variations through vector based representations.
CapsNets are inspired by so-called inverse graphics , i.e., the opposite operation of image rendering.
Given a feature representation,
CapsNets attempt to generate the corresponding geometrical representation.
The motivation for using CapsNets in the network are twofold: 1) CapsNet promotes a dynamic ‘routing-by-agreement’ approach where only the features that are in agreement with high-level detectors are routed forward.
This property of CapsNets does not deteriorate extracted features and the ﬁnal accuracy only depends on the richness of original shape features.
It allows us to directly compare feature discriminability of spherical and volumetric convolution without any bias.
For example, using multiple layers of volumetric or spherical convolution hampers a fair comparison since it can be argued that the optimum architecture may vary for two different operations.
2) CapsNet provides an ideal mechanism for disentangling 3D shape features through pose and view equivariance while maintaining an intrinsic co-ordinate frame where mutual relationships between object parts are preserved.
Inspired by these intuitions, we employ two independent CapsNets in our network for volumetric convolution features and axial symmetry features.
In this layer, we rearrange the input feature vectors as two sets of primary capsules—for each capsule net—and use the dynamic routing technique proposed by Sabour et al.
(2017) to predict the classiﬁcation results.
The outputs are then combined using max-pooling, to obtain the ﬁnal classiﬁcation result.
For volumetric convolution features, our architecture uses 1000 primary capsules with 10 dimensions each.
For axial symmetry features, we use 2500 capsules, each with 10 dimensions.
In both networks, decision layer consist ofdimensional capsules.
6. H YPERPARAMETERS
We use n = 5 to implement Eq.
15 and three iterations to calculate the Moore-Penrose pseudo inverse using Eq.
We use a decaying learning rate lr = 0 .
1 × 0 .
9 gstep 3000 , where g step is incremented by one per each iteration.
For training, we use the Adam optimizer with β 1 = 0 .
9 , β 2 = 0 .
999 , (cid:15) = 1 × 10 −where parameters refer to the usual notation.
All these values are chosen empirically.
Since we have decomposed the theoretical derivations into sets of low-cost matrix multiplications, speciﬁcally aiming to reduce the computational complexity, the GPU implementation is highly efﬁcient.
For example, the model takes less than 15 minutes for an epoch during the training phase for ModelNet10, with a batchsize 2, on a single GTX 1080Ti GPU.
E XPERIMENTS
In this section, we discuss and evaluate the performance of the proposed approach.
We ﬁrst compare the accuracy of our model with relevant state-of-the-art work, and then present a thorough ablation study of our model, that highlights the importance of several architectural aspects.
We use ModelNet10 and ModelNet40 datasets in our experiments.
Next, we evaluate the robustness of our approach against loss of information and ﬁnally show that the proposed approach for computing 3D Zernike moments produce richer representations of 3D shapes, compared to the conventional approach.
7. C OMPARISON WITH THE STATE - OF - THE - ART
Table 1 illustrates the performance comparison of our model with state-of-the-art.
The model attains an overall accuracy of 92 .
17% on ModelNet10 and 86 .
5% accuracy on ModelNet40, which is on par with state-of-the-art.
We do not compare with other recent work, such as Kanezaki et al.
; Qi et al.
; Sedaghat et al. (2016); Wu et al.
; Qi et al.
; Bai et al.
Maturana & Scherer (2015) that show impressive performance on
[Table]
ModelNet10 and ModelNet40.
These are not comparable with our proposed approach, as we propose a shallow, single model without any data augmentation, with a relatively low number of parameters.
Furthermore, our model reports these results by using only a single volumetric convolution layer for learning features.
Fig.
4 demonstrates effectiveness of our architecture by comparing accuracy against the number of trainable parameters in state-of-the-art models
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
7. A BLATION STUDY
Table 2 depicts the performance comparison between several variants of our model.
To highlight the effectiveness of the learned optimum view points, we replace the optimum view point layer with three ﬁxed orthogonal view points.
This modiﬁcation causes an accuracy drop of 6.57%, emphasizing that the optimum view points indeed depends on the shape.
Another interesting— perhaps the most important—aspect to study is the performance of the proposed volumetric convolution against spherical convolution.
To this end, we replace the volumetric convolution layer of our model with spherical convolution and compare the results.
It can be seen that our volumetric convolution scheme outperforms spherical convolution by a signiﬁcant margin of 12.56%, indicating that volumetric convolution captures shape properties more effectively.
[Figure]
Furthermore, using mean-pooling instead of maxpooling, at the decision layer drops the accuracy to 87 .
We also evaluate performance of using a single capsule net.
In this scenario, we combine axial symmetry features with volumetric convolution features using compact bilinear pooling (CBP), and feed it a single capsule network.
This variant achieves an overall accuracy of 86 .
53% , is a 5 .
64% reduction in accuracy compared to the model with two capsule networks.
Moreover, we compare the performance of two feature categories—volumetric convolution features and axial symmetry features—individually.
Axial symmetry features alone are able to obtain an accuracy of 66 .
73% , while volumetric convolution features reach a signiﬁcant 85 .
3% accuracy.
On the contrary, spherical convolution attains an accuracy of 71 .
6% , which again highlights the effectiveness of volumetric convolution.
Then we compare between different choices that can be applied to the experimental architecture.
We ﬁrst replace the capsule network with a fully connected layer and achieve an accuracy of 87.3%.
This is perhaps because capsules are superior to a simple fully connected layer in modeling view-
[Figure]
0 .0 .0 .n
R ec o n s t r u c t i o n e rr o r
ProposedapproachConventionalapproach
Our Zernike frequencies computation approach has far less error than the conventional approach.
point invariant representations. Then we try different substitutions for compact bilinear pooling and achieve 90.7%, 90.3% and 85.3% accuracies respectively for feature concatenation, max-pooling and average-pooling.
This justiﬁes the choice of compact bilinear pooling as a feature fusion tool.
However, it should be noted that these choices may differ depending on the architecture.
7. R OBUSTNESS AGAINST INFORMATION LOSS
One critical requirement of a 3D object classiﬁcation task is to be robust against various information loss.
To demonstrate the effectiveness of our proposed features in this aspect, we randomly remove data points from the objects in validation set, and evaluate model performance.
The results are illustrated in Fig.
The model shows no performance loss until 20% of the data is lost, and only gradually drops to an accuracy level of 66 .
5 at a 50% data loss, which implies strong robustness against random information loss.
7. E FFECTIVENESS OF THE PROPOSED METHOD FOR CALCULATING 3D Z ERNIKE MOMENTS
In Sec. 4.1, we proposed an alternative method to calculate 3D Zernike moments (Eq. 5, 6), instead of the conventional approach (Eq. 3).
We hypothesized that moments obtained using the former has a closer resemblance to the original shape, due to the impact of ﬁnite number of frequency terms.
In this section, we demonstrate the validity of our hypothesis through experiments.
To this end, we compute moments for the shapes in the validation set of ModelNet10 using both approaches, and compare the mean reconstruction error deﬁned as: 1 T
[Equation]
where T is the total number of points and t ∈ S 3 .
Fig.
6 shows the results.
In both approaches, the mean reconstruction error decreases as n increases.
However, our approach shows a signiﬁcantly low mean reconstruction error of 0 .
0467% at n = 5 compared to the conventional approach, which has a mean reconstruction error of 0 .
56% at same n .
This result also justiﬁes the utility of Zernike moments for modeling complex 3D shapes.
C ONCLUSION
In this work, we derive a novel ‘ volumetric convolution ’ using 3D Zernike polynomials, which can learn feature representations in B 3 .
We develop the underlying theoretical foundations for volumetric convolution and demonstrate how it can be efﬁciently computed and implemented using low-cost matrix multiplications.
Furthermore, we propose a novel, fully differentiable method to measure the axial symmetry of a function in B 3 around an arbitrary axis, using 3D Zernike polynomials.
Finally, using these operations as building tools, we propose an experimental architecture, that gives competitive results to state-of-the-art with a relatively shallow network, in 3D object recognition task.
An immediate extension to this work would be to explore weight sharing along the radius of the sphere.
A C ONVOLUTION WITHIN UNIT SPHERE USING 3D Z ERNIKE POLYNOMIALS
Theorem 1: Suppose f, g : X −→ R 3 are square integrable complex functions deﬁned in Bso
that (cid:104) f, f (cid:105) < ∞ and (cid:104) g, g (cid:105) < ∞ .
Further, suppose g is symmetric around north pole and τ ( α, β )
=
R y
( α ) R z ( β ) where R ∈ SO (3) .
Then,
[Equation] [Equation]
where Ω n,l,m ( f ) , Ω n,l, 0 ( g ) and Y l,m ( θ, φ ) are ( n, l, m )
th 3D Zernike moment of f , ( n, l, 0)
th 3D
Zernike moment of g , and spherical harmonics function respectively.
Proof: Since 3D Zernike polynomials are orthogonal and complete in B 3 , an arbitrary function f ( r, θ, φ ) in B 3 can be approximated using Zernike polynomials as follows.
[Equation]
where Ω n,l,m ( f ) could be obtained using,
[Equation]
where † denotes the complex conjugate.
Leveraging this property (Eq. 20) of 3D Zernike polynomials Eq.
4 can be rewritten as,
[Equation]
But since g ( θ, φ, r ) is symmetric around y , the rotation around y should not change the function.
Which ensures,
[Equation]
(23) and hence, ∞ (cid:88)
[Equation]
This is true,
if and only if m (cid:48) = 0 .
Therefore, a symmetric function around y , deﬁned inside the unit sphere can be rewritten as,
[Equation]
which simpliﬁes Eq.
22 to,
[Equation]
Using the properties of inner product, Eq. 26 can be rearranged as,
[Equation]
Consider the term τ ( θ,φ ) ( Z n (cid:48) ,l (cid:48) , 0 ) .
Then,
[Equation] [Equation]
where D lm,m (cid:48) is the Wigner-D matrix.
But we know that D l
[Equation]
0 ( θ, φ ) =
Y
[Equation]
m
[Equation]
( θ, φ ) .
Then Eq.becomes,
[Equation]
B E QUIVARIANCE OF VOLUMETRIC CONVOLUTION TO 3D ROTATION GROUP
Theorem 1: Suppose f, g : X −→ R 3 are square integrable complex functions deﬁned in B 3 so that (cid:104) f, f (cid:105) < ∞ and (cid:104) g, g (cid:105) < ∞ .
Also, let η α,β,γ be a 3D rotation operator that can be decomposed into three Eular rotations R y ( α ) R z ( β ) R y ( γ ) and τ α,β another rotation operator that can be decomposed into R y
( α ) R z ( β ) .
Suppose η α,β,γ ( g ) =
τ α,β ( g ) .
Then,
[Equation]
) where ∗ is the volumetric convolution operator.
Proof: Since η ( α,β,γ ) ∈
SO (3) , we know that η ( α,β,γ ) ( f ( x )) =
f ( η − 1 ( α,β,γ ) ( x )) .
Also we know that η ( α,β,γ ) : R 3 → R 3 is an isometry.
Deﬁne,
[Equation]
Consider the Lebesgue measure λ ( S 3 )
= (cid:82) S 3 dx .
It can be proven that a lebesgue measure invariant under the isometries, which gives us dx =
dη ( α,β,γ ) ( x ) = dη − 1 ( α,β,γ ) ( x ) , ∀ x ∈ S 3 .
Therefore, < η ( α,β,γ ) f, η ( α,β,γ ) g >
[Equation]
Let f ( θ, φ, r ) and g ( θ, φ, r ) be the object function and kernel function (symmetric around north pole) respectively.
Then volumetric convolution is deﬁned as,
[Equation]
Applying the rotation η ( α,β,γ ) to f , we get,
[Equation]
By the result 33, we have,
[Equation]
However, since η α,β,γ ( g ) = τ α,β ( g ) we get,
[Equation]
We know that,
[Equation]
Then,
[Equation]
Hence, we achieve equivariance over 3D rotations.
C A XIAL SYMMETRY MEASURE OF A FUNCTION IN B 3 AROUND AN ARBITRARY AXIS .
Proposition: Suppose g : X −→ R 3 is a square integrable complex function deﬁned in B 3 such that (cid:104) g, g (cid:105) < ∞ .
Then, the power of projection of g in to S =
{ Z i } where S is the set of Zernike basis functions that are symmetric around an axis towards ( α, β ) direction is given by,
[Equation]
where α and β are azimuth and polar angles respectively .
Proof: The subset of complex functions which are symmetric around north pole is S =
{ Z n,l, 0 } .
Therefore, projection of a function into S gives,
[Equation]
[Equation]
To obtain the symmetry function around any axis which is deﬁned by ( α, β ) , we rotate the function by ( − α, − β ) , project into S , and ﬁnal compute the power of the projection.
[Equation]
For any rotation operator U
[Equation]
and for any two points deﬁned on a complex Hilbert space
[Equation]
x and y
[Equation] [Equation]
Applying this property to 44 gives,
[Equation]
Using Eq.
20 we get
[Equation]
Using properties of inner product Eq.
47 further simpliﬁes to,
[Equation]
Using the same derivation as in 28, sym
[Equation]
Since 3D Zernike Polynomials are orthogonal we get,
[Equation]
[Equation]
In signal theory the power of a function is taken as the integral of the squared function divided by the size of its domain.
Following this we get,
|| sym
( α,β ) ||
= (cid:104) ( (cid:88) n
[Equation]
We drop the constants here since they do not depend on the frequency.
Simplifying Eq.
51 gives,
[Equation]
which leads to,
[Equation]
D F UNCTION DEFINITIONS D. S PHERICAL HARMONICS
Spherical harmonics are a set of complete orthogonal functions, which are deﬁned on S 2 .
[Equation]
54) where l is an integer, m is an integer, | m | < l , and P ml ( · ) is the associated Legendre function (see appendix D.2).
D. A SSOCIATED L EGENDRE FUNCTION
Associated Legendre function P ml ( x ) is deﬁned as,
[Equation]
55) where l is an integer, m is an integer, | m | < l , and x is a real number.
D. Z ERNIKE RADIAL POLYNOMIAL
[Equation]
E S PHERICAL CONVOLUTION ON S
Let the f and g be the shape functions of the object and kernel respectively.
Then f and g can be expressed as,
[Equation]
where Y l,m is the ( l, m ) th spherical harmonics function and ˆ f ( l, m ) and ˆ g ( l, m ) are ( l, m ) th frequency components of f and g respectively.
Then the frequency components of convolution f ∗ g can be easily calculated as,
[Equation]
where † denotes the complex conjugate.