D EEP L EARNING 3D S HAPES U SING A LT - AZ A NISOTROPIC 2-S PHERE C ONVOLUTION
Anonymous authors
Paper under double-blind review
A BSTRACT
The ground-breaking performance obtained by deep convolutional neural networks (CNNs) for image processing tasks is inspiring research efforts attempting to extend it for 3D geometric tasks.
One of the main challenge in applying
CNNs to 3D shape analysis is how to deﬁne a natural convolution operator on noneuclidean surfaces.
In this paper, we present a method for applying deep learning to 3D surfaces using their spherical descriptors and alt-az anisotropic convolution on 2-sphere.
A cascade set of geodesic disk ﬁlters rotate on the 2-sphere and collect spherical patterns and so to extract geometric features for various 3D shape analysis tasks.
We demonstrate theoretically and experimentally that our proposed method has the possibility to bridge the gap between 2D images and 3D shapes with the desired rotation equivariance/invariance, and its effectiveness is evaluated in applications of non-rigid/ rigid shape classiﬁcation and shape retrieval
I NTRODUCTION
A recent research effort in computer vision and geometric processing communities is towards replicating the incredible success of deep convolutional neural networks (CNNs) from the image analysis to 3D shape analysis.
A straightforward extension is to treat a 3D shape as a voxel grid (Wu et al.
(2015); Maturana & Scherer (2015); Song & Xiao (2016); Wang et al.
; Riegler et al.
Alternative methods include encoding a 3D shape as a collection of 2D renderings from multiple cameras (Qi et al.
; Su et al.
; Bai et al.
or projecting a 3D object onto geometric entities which can be ﬂattened as 2D images (Shi et al.
; Cao et al.
Sﬁkas et al.
All these methods convert a 3D shape into an Euclidean grid structure which supports shift (translational) equivariance/invariance, such that conventional CNNs can work out-of-the box.
Although embedded in R 3 , 3D shapes are typically represented as manifold surfaces.
Recent research has particularly focused on convolutional networks for non-Euclidean domains such as manifolds or graphs.
One of the main difﬁculties of adopting CNNs and similar methods in these nonEuclidean domains is the lack of shift-invariance on surfaces or graphs (Masci et al. (2015).)
Our motivation comes from the representation of 3D shapes as functions on spheres.
We transfer the problem of manifold surface convolution into spherical convolution with the primary beneﬁt of rotation invariance .
Although shift-invariance is hard to achieve on general surfaces, by replacing ﬁlter translations with ﬁlter rotations, rotation equivariance/invariance can be obtained on the 2sphere.
Furthermore, spherical descriptors of 3D shapes are compact and require a network of lower capacity, compared to voxel or multi-view representations.
In this work, we are primarily interested in analyzing 3D geometric data using a speciﬁc type of spherical convolution either for classiﬁcation or retrieval tasks.
R ELATED W ORK
Surface convolution One approach to shift-invariance on surfaces is using re-parameterization methods.
Geodesic CNN (Masci et al.
; Bronstein et al.
; Monti et al.
(2017)) uses local geodesic polar coordinates to parameterize a surface patch locally around a point.
An angular max-pooling layer or local coordinate frame alignment are proposed to account for the ﬁlter’s local rotational degree of freedom.
Spherical parameterization methods (Peng & Timalsena (2016); Praun & Hoppe (2003); Gu et al.
(2004)) map a genus-0 3D shape onto a sphere bijectively which provides a global framework for the spherical convolution.
Sinha et al.
(2016; 2017) transfer a genus-0 3D shape into a parameterized spherical image, and then ﬂatten it into a planar geometry image.
Data augmentation is necessary for geometry images in order to account for inconsistent cut positions and orientations.
Toric covers (Maron et al.
(2017)) is a seamless representation which stitches four copies of a genus-0 surface and globally maps them onto a planar ﬂat torus.
Spectral methods perform convolution on the spectral domain using the graph Laplacian and its eigen space decomposition (Yi et al.
; Bruna et al.
This method can efﬁciently address the shift-invariance problem, however, it suffers the difﬁculty with cross-shape learning since the spectral decomposition of each shape can be inconsistent.
Spherical convolution Spherical representation of 3D shapes have been used for shape matching (Kazhdan et al.
Frome et al.
; Makadia & Daniilidis (2010)), remeshing (Praun &
Hoppe (2003)), medical imaging (Shen & Makedon (2006)) and other tasks before the deep learning era.
Recently, researchers have started to explore deep spherical convolutional neural networks for tasks such as molecular modeling (Boomsma & Frellsen (2017)), ominidirectional vision (Su & Grauman (2017)) and 3D shape recognition (Cohen et al.
; Esteves et al.
Su &
Grauman (2017) discretized a spherical image using a lat-lon grid (see Fig.3(a)) and ﬂattened it through equirectangular projection.
A variable ﬁlter size is proposed to compensate for the imbalanced sampling along longitudinal direction.
In Boomsma & Frellsen (2017), a cubed-sphere grid (see Fig.3(b)) is investigated in addition to a lat-lon grid, to achieve relatively more uniform grid on spheres.
The work of Cohen et al.
(2018) generalizes the spherical convolution with the full three rotational degrees of freedom in the 3D space, and it maps a spherical image to features on
SO(3) using generalized Fourier transform.
A similar work is done in Esteves et al.
(2018) with azimuthally symmetric ﬁlters.
In this paper, we propose an alt-az anisotropic spherical convolutional neural network (or a 3 SCNN for short) for various rigid and non-rigid shape analysis tasks.
Fig.
1 gives an overview of our method.
A 3D shape is represented as a set of spherical images using spherical parameterization (for non-rigid shapes) or spherical projection (for rigid shapes, not shown in the ﬁgure).
An icosahedron based spherical grid is used as the discrete representation of the spherical images.
The convolution is applied directly on the spherical representation of the shape using a geodesic disc shape of ﬁlter.
The proposed deep a 3 SCNN has multiple sequential convolutions followed by a nonlinearity such as ReLU and Spherical (max or average)
Pooling, all conducted on the spherical domain.
Output is a set of spherical images which capture high-level shape feature descriptors.
Following are the main contributions of our paper:
(1) theoretical analysis of the relationship between various deﬁnition of convolutions for functions deﬁned on the 2-sphere and a novel convolutional neural network using alt-az anisotropic spherical convolutions that emulates most aspects of standard convolutional networks in R 2 ;
(2) an efﬁcient geodesic grid data structure to support fast computation of the spherical convolution with locally-supported geodesic disc ﬁlters; (3) an empirical demonstration of the utility of a 3 SCNN with 3D shape learning problems.
ALT - AZ C ONVOLUTION ON 2-S PHERE
We now describe the main contribution of our paper, i.e., the alt-az convolution on 2-sphere.
We ﬁrst describe the notations and then describe the alt-az convolution on context of other convolutions for spherical functions.
3. N OTATIONS AND PRELIMINARIES
2-sphere or unit sphere S 2 can be regarded as the set of points u ∈ R 3 with norm one.
The 2sphere is a 2-manifold on which any point ˆ u is a unit vector.
The ˆ u can be parametrized by spherical coordinates ( θ, φ ) ∈ [0 , π ] × [0 , 2 π ] such that ˆ u ( θ, φ )
= (sin θ cos φ, sin θ sin φ, cos θ ) .
A regular region r on S 2 has a positive area
A = (cid:82) r ds (ˆ u ) , where ds (ˆ u ) = sin θdθdφ .
A special region on the 2-sphere is called polar cap region R θ 0 , around the north pole, ˆ η (0 , 0 , 1) , which is azimuthally symmetric and is parameterized by a maximum colatitude angle θ 0 :
[Equation]
(1) 3D Rotations
The set of rotations in three dimensions is called “special orthogonal group” SO(3) .
SO(3) is a 3-manifold on which any rotation R ∈ SO(3) can be represented as a 3 × 3 matrix.
Each rotation R is associated with three independent parameters, we use the right hand rule zyz -Euler angles ϕ ∈ [0 , 2 π ] , ϑ ∈ [0 , π ] , and ω ∈ [0 , 2 π ] , i.e.
R ≡
[Equation]
If we ﬁx the third rotation angle ω to zero, SO(3) is reduced into a subset A with two independent parameters.
Any rotation R ∈ A can be described as an alt-az rotation 1 :
[Equation]
An alt-az rotation can be considered as a composition of an altitude rotation R ( y ) ϑ ∈ SO(2) and a azimuth rotation R ( z ) ϕ ∈ SO(2) .
Rotation operator We deﬁne the effect of general rotation on spherical functions as an operator D R ( ϕ, ϑ, ω ) which corresponds to the rotation matrix R deﬁned in Eqn.
The effect of
D R ( ϕ, ϑ, ω ) on the spherical image f can be realized through an inverse rotation R − 1 of the coordinate system.
That is,
[Equation]
3. C ONVOLUTION ON THE 2- SPHERE
The convolution operator in n dimensional Euclidean space R n is given by:
[Equation]
The above equation is used as a reference to develop different notions of convolution on the 2-sphere.
Unlike conventional Euclidean domain signal, for spherical functions there is no standard convolution operators deﬁned.
Two competing deﬁnitions exist in literature:
Type I: General anisotropic convolution: This convolution operator on 2-sphere tries to emulate the convolution in Euclidean spaces by replacing translations with full rotation in SO(3) and integrating over all possible rotations.
This gives the most general deﬁnition of spherical convolution.
Given a spherical ﬁlter h and spherical image f evaluated at a point ˆ u ∈ S 2 , general anisotropic convolution on S 2 is deﬁned:
[Equation]
Note that the output function g is not deﬁned on the original S 2 .
Instead, it is a function of three
Euler angles ( ϕ, ϑ, ω ) and is therefore deﬁned on the 3-manifold SO(3) (please see Cohen et al.
(2018) for detail.)
Type II: Azimuthally isotropic convolution:
This spherical convolution outputs a function deﬁned on S 2 using an azimuthally symmetric ﬁlter h 0 (ˆ u ) (Esteves et al.
(2018); Driscoll & Healy (1994)):
[Equation]
Referring to Eqn.
(9), we see that an arbitrary ﬁlter h is essentially transformed into a rotationally symmetric ﬁlter h 0 through circular “averaging”.
Type II spherical convolution zeros the contribution of angular variations from a ﬁlter, and hence, is considered restrictive for pattern matching purpose in spherical image processing.
Towards developing a spherical convolution which respects some important properties of standard convolutions deﬁned in R 2 , we propose to use alt-az spherical convolution.
In R 2 , the two spatial translations are isometric mappings and are directly convolved, whereas the isometry corresponding to a rotation in SO(2) is generally not convolved.
Several works on the rotation equivariant/invariant
CNNs have been proposed (Weiler et al.
; Qiu et al.
; Kondor & Trivedi (2018),) and are proven to be effective; but they typically incur a signiﬁcant increase in the number of parameters and computational load.
Similarly, in the spherical domain, the two degrees of freedom in altaz rotation are the direct analogs of two spatial translations in R 2 (“shifting on the sphere”), and the third rotation R ( z ) ω , emulating the non-rotatable ﬁlters in R 2 , can be ﬁxed and treated with data augmentation.
Intuitively, we want to shift a spherical disc ﬁlter on the 2-sphere without self rotating the ﬁlter.
We now formally deﬁne our alt-az spherical convolutional operator.
Type III: alt-az anisotropic spherical convolution ( a 3 SConv): Constraining the rotation of ﬁlter within alt-az rotation set A , a ﬁlter h spans the altitude change by ϑ and azimuth change by ϕ , and is convolved with the spherical signal f .
Mathematically, a 3 SConv is deﬁned as:
[Equation]
a 3 SConv operator has the following desirable properties:
• Domain consistency : It takes two functions in L 2 ( S 2 ) and generates a function back in L 2 ( S 2 ) , such that cascaded layers of spherical convolutions can be utilized to extract hierarchical spherical patterns; • Azimuth rotation equivariance : An map (cid:96) is rotation equivariant if (cid:96) ◦ D Q = D Q ◦ (cid:96) . In general cases, a 3 SConv is not equivariant to an arbitrary rotation in SO(3) . If Q is an azimuth rotation, a 3 SConv has the equivariance property 2 .
I.e. for an azimuth rotation D Q (0 , 0 , ω ) ,
[Equation]
Convolution with locally-supported ﬁlters Traditional CNNs are efﬁcient due to the use of locally-supported ﬁlters and weight sharing.
The metric on the Euclidean grid allows compactly supported ﬁlters, whose sizes are typically much smaller than the size of the input signal.
On the 2-sphere, we propose to use locally-supported geodesic disc ﬁlters in the form of polar caps.
Mathematically, a locally-supported ﬁlter is deﬁned as a space limited spherical function belonging to the follow subspace:
[Equation]
where R r 0 is the polar cap region on which the geodesic disc ﬁlter is deﬁned, and r 0 deﬁnes the size of a ﬁlter.
Fig.
2 shows a locally-supported geodesic disc ﬁlter undergoing different types of rotation.
S PHERICAL C ONVOLUTIONAL N EURAL NETWORK
Using the notion of alt-az anisotropic spherical convolution, we are now ready to extend CNNs to the 2-sphere. our a 3 SCNN consists of several layers that are applied subsequently (see Fig. 1). Besides the a 3 SConv layer described, we further discuss the following two speciﬁc types of layers deﬁned on the 2-sphere.
A local max pooling (LMP) layer replaces a spherical image f
in at any point ˆ u 0 ( θ 0 , φ 0 ) with the maximum function value in its geodesic disc neighborhood, i.e.,
[Equation]
where ˆ u ( θ, φ ) is a neighboring point of ˆ u 0 , and | .
| denotes the geodesic distance between them.
A global spherical max pooling (GMP) layer operates on a spherical image f in with k channels and outputs a k dimensional vector in R k .
For each channel f ini ( i = 1 , 2 , ..., k ), a GMP layer outputs a single value represent the most salient feature.
I.e.,
[Equation]
Notice a GMP layer is invariant to any rotation D R ( ϕ, ϑ, ω ) of the input spherical image f :
GMP( D R f ) =
GMP( f ) .
Data augmentation After going through a set of a 3 SConv layers, the global azimuth rotation of an input spherical image f will be transformed into the same rotation of the extracted spherical descriptors (see Eqn.
With a GMP layer followed, the extract feature vector will be invariant to the azimuth rotation of f .
For arbitrary rotation of f in SO(3) , our a 3 SConv layer does not have the equivariance property.
This means data rotation augmentation is theoretically required to recognize f in random orientations.
In appendix B, we show that, an a 3 SCNN network constructed by several a 3 SConv layers together with a GMP layer can generalize to arbitrary unseen orientations with SO(2) rotation augmentation about any axis which is not parallel to y or z axis.
N UMERICAL COMPUTATION OF ALT - AZ ANISOTROPIC CONVOLUTION
From a computational point of view, implementing the spherical convolution deﬁned above in Eqns.
(8-10) is difﬁcult because it is not possible to uniformly discretize the surface of the sphere such that each sample point shares the same neighborhood.
Therefore, a popular method of performing spherical convolution is to project the discretized spherical functions and ﬁlters onto the span of
Wigner D functions for type I spherical convolution (see Cohen et al.
(2018),) or spherical harmonics for type II spherical convolution (see Esteves et al.
They then perform the convolution in the
Fourier domain via point-wise multiplications.
The lack of locality support in the spherical Fourier transform inhibits us from using this method.
Locally-supported ﬁlters belong to a subspace of space limited signals which is by nature inﬁnite-dimensional.
No non-trivial local ﬁlters can have a ﬁnite representation in the spectral domain.
Esteves et al.
(2018) use spectral smoothness to enforce a spatial decay in ﬁlters, and hence, achieve locality.
However, the ﬁlters are still deﬁned on the whole spherical domain which is memory inefﬁcient.
In this paper, we propose an alternate method which performs direct spherical convolution using geodesic grid discretization.
Geodesic grid discretization Uniform geodesic grid cannot be achieved on the sphere except for the projections of ﬁve platonic polyhedra - tetrahedron, cube, octahedron, dodecahedon and icosahedron.
These polyhedra can be further subdivided into different frequencies to obtain ﬁner approximation of a sphere (Fig. 3(b) shows a subdivision of the projection of a cube on a sphere.)
Among the ﬁve platonic polyhedra, the icosahedron is most similar to the sphere (Schr ¨ oder & Sweldens (1995)).
After the subdivision, the resulting triangulation has the least imbalance in area between its constituent triangle (Fig.3(c)).
Most of the vertices have six direct neighbors except for the original 12 vertices of the icosahedron.
This makes the icosahedron-based geodesic grid discretization most suitable for the discrete spherical convolution.
We call this type of geodesic grid an icosahedronsphere grid.
The total number of grid vertices are N = f 2 × 10 + 2 , where f is the subdivision frequency.
Considering the structure of the icosahedron-sphere grid, in order to obtain a multi-level spherical feature map, the stride of a convolution or pooling layer has to be a multiple of 2 n .
The stride is applied accordingly to the subdivision frequency f .
Fig. 3(c) shows the icosahedron in different subdivision frequencies 1, 2, 4 and 8.
A natural shape of the locally-supported ﬁlter correlating with the icosahedron-sphere grid, is a geodesic disc which can be discretized as a hexagonal grid of different ring sizes.
Fig. 4(c) shows two examples of such ﬁlters.
The same shape of geodesic disc and discretized hexagonal grid is used for local spherical max pooling LMP layers.
Efﬁcient data structure
The icosahedron-sphere grid data structure is self-sufﬁcient to support spherical convolution, pooling and other CNN operators.
However the linked data structure (vertexedge-face and link data for topology) is not space efﬁcient and is time consuming, in order to ﬁnd the neighbors of a vertex and shift a ﬁlter on the sphere during the convolution.
In this work, we use a rectilinear data structure to enable efﬁcient spherical convolution and pooling.
The icosahedronbased spherical mesh can be opened into 2D plane and represented as a grid structure as shown in
Fig.4.
The cut is along eleven edges of the icosahedron as shown in Fig. 4(a) and (b).
By rotating the u and v axes in Fig. 4(b) into orthogonal axes, we obtain ﬁve rectangular 2D patches to store all the vertices of the icosahedron-sphere grid, as illustrated in Fig. 4(c).
This construction has two main advances: (1) within each patch, shifting ﬁlters on the 2-sphere is approximately equivariant to translation in u and v and (2) features on the geodesic grid can naturally be expressed using tensors, which means that the spherical convolution can be efﬁciently implemented on a GPU.
When implementing spherical convolutions and pooling operations for the icosahedron-sphere grid, one has to be careful in padding each patch with the contents of the other two neighboring patches.
If a point is on the colored cut-lines as shown in Fig. 4(c), then its k -ring hexagon neighbors are retrieved across the boundary of matrix.
that by using padding strategy, the rectilinear data structure realizes a seamless geodesic grid representation of the
E XPERIMENTS 6. S PHERICAL D ESCRIPTORS OF 3D SHAPES
As a pre-processing step for all experiments, we ﬁrst need to convert 3D shapes to functions on the 2-sphere.
Two different methods were employed to do this conversion: spherical projection for rigid shapes and spherical parameterization for non-rigid shapes.
Spherical projection For rigid shapes, we project a 3D shape onto an enclosing sphere using a straightforward ray casting scheme and we collect the following two types of spherical descriptors.
(1) Spherical Extent Function ( SEF ):
This function describes a surface by associating each ray from the origin to the distance of the last point of intersection of the model with the ray.
(2) Normal Deviation Function ( NDF ): This function describes the surface using the cosine angle between surface normal at the last intersection point and the ray direction (see Fig.
5 left for example).
Ignoring high [Table] non-convexity of surfaces, we assume the projections capture sufﬁcient information of the shape to be useful for rigid shape analysis.
Spherical parametrization Spherical projection produces extrinsic shape descriptors which are not suitable for non-rigid shape analysis.
To handle deformable shapes, we use the authalic spherical parametrization method (Sinha et al.
to obtain an area-preserving bijective spherical map and use the following intrinsic shape descriptors: (1) Principal Curvatures : the two principal curvature k min and k max measure the degree to which the surface bends in orthogonal directions at a point.
(2) Average Geodesic Distance (AGD): this measures the centerness of a point on the surface.
Heat kernel signature (HKS) (Sun et al.
(2009)): this measures the amount of untransferred heat after time t , assuming an unit heat source is added on each point of the surface (see Fig.
5 right for example).
In all of the following experiments, spherical functions are discretized using icosahedron-sphere grid with subdivision frequency f = 32 .
This will generate ﬁve patches of size 33 × 65 , by stacking them one above the other.
The input size to all networks are 165 × 65 × K , where K is the number of input channels.
For the 12 valence-5 vertices in the icosahedron, we apply the shared hexagon ﬁlter by computing the center point twice.
Since it affects a small number of vertices, we empirically validate that the effect can be ignored.
6. N ON - RIGID SHAPE CLASSIFICATION
We ﬁrst conduct experiments on SHREC’11 non-rigid shape classiﬁcation, and we compare three types of spherical functions: (i) Intrinsic-2 contains the two principal curvatures, (ii) Intrinsic-3 adds
AGD to intrinsic-2 , and (iii) Intrinsic-8 adds ﬁve HKS sampled at 5 logarithmic time scales on top of Intrinsic-3 .
And we compare ﬁve modes of experiments: (a) trained with original data without data augmentation (NA), (b) trained with 36 azimuth rotation augmentation (AZ) by sampling ω per 10 degrees, (c) trained with 36 rotation augmentation (SO(2)(x)) by rotating about x -axis per 10 degrees, (d) trained with 72 alt-az rotation augmentation (Alt-AZ) by sampling θ and φ perdegrees, and (e) trained with arbitrary 128 rotations ( SO(3) ).
In each category, 16 objects are used for training and 4 objects are used for testing.
Architecture and hyper parameters Our network contains ﬁve a 3 SConv-dropout-ReLU-LMP blocks.
A 20% dropout is added right after each spherical convolution layer for regularization.
The resulting spherical functions are pooled using a global max pooling (GMP) layer followed by two fully connected layers for the ﬁnal classiﬁcation.
A 50% dropout layer is inserted in between the last two fully connected layers.
We use 32, 64, 64, 128, 128 features for the a 3 SConv layers, and 512 features are output from the GMP and fed into the ﬁrst fully connected layer.
Each ﬁlter on Shas kernel size ring-2, stride 1 and each LMP layer has size ring-2 and stride 2.
Results Table 1 shows the performance of these intrinsic descriptors for non-rigid shape classiﬁcation under different augmentation modes.
Notice that the original testing data are randomly posed.
In spite the small training data, our network achieves good classiﬁcation accuracy for Intrinsiceven without data augmentation.
We attribute the capability to generalize to random perturbed data to the use of LMP layers, which allows a certain amount SO(3) rotation invariance. Comparing the four types of data augmentation strategies, our experimental result conﬁrms that SO(2)(x) aug-
[Table]
mentation performs better than the original training data, AZ type augmentation and SO(3) random augmentation. It is predictable that alt-az augmentation performs even better with more augmented data. Theoretically, our network is invariant to azimuth rotation except for the two poles. Due to the approximation error introduced in the implementation, we see AZ type data augmentation can compensate those equivariance errors.
Compare to other deep learning based non-rigid shape analysis, the geometry image method (Sinha et al.
(2016)) is most similar to ours.
Their reported classiﬁcation accuracy of 96.6% is also based on an alt-az rotation augmentation.
Our method outperforms the state-of-the-art approach by about 3% margin even by using two principal curvature ( Intrinsic-2 ) as inputs.
6. R IGID SHAPE CLASSIFICATION
We further experiment on ModelNet10 and ModelNet40 rigid shape databases, and we use the model trained and tested on aligned data as baseline, and we explicitly test the equivariance/invariance property of our learned shape representations by perturbing the testing data in different modes.
Architecture and experiment setup We experiment with four types of perturbations: (a) test with original aligned data (NR), (b) test with azimuthal rotation perturbations (AZ), (c) test with alt-az ﬁeld rotation perturbations (Alt-AZ) and (d) test with random SO(3) rotations.
We also randomly perturb the training data and test it with randomly perturbed testing data.
In these experiments, two channels of spherical functions: SEF and NDF are used as the input and we use the same network structure as we use in SHREC’11, except that in the ﬁve cascaded a 3 Sconv layers, 32,64,128,256,512 ﬁlters are used, and in the ﬁrst fully connected layer, 1024 features are generated for classiﬁcation.
Results Table 2 summarizes the performance of a 3 S CNN for classifying rigid objects for unseen orientations.
The column NR/NR shows the classiﬁcation accuracy for aligned training and testing data.
As expected, aligned data gives the best classiﬁcation performance and it serves as the baseline for evaluating the rotation equivariance/invariance of the learned models.
Comparing the four types of perturbing modes, the learned representation has the best classiﬁcation accuracy for testing data perturbed with alt-az rotation.
For random SO(3) rotation, whether it is with perturbed testing data only (see column NR/ SO(3) ) or with both testing and training data perturbed (column SO(3) / SO(3) ), our network still performs well, showing that a 3 S CNN may generalize to unseen orientations even without data augmentation.
It is counter-intuitive that azimuthal type of perturbation gives slightly worse performance (see column NR/AZ)
since in theory, our network is azimuthal rotation invariant. We interpret this as the result of the equivariance error coming from icosahedron-sphere grid tessellation and singularities at poles, while alt-az and SO(3) perturbation might compensate this with perturbations which can be better treated by the LMP and GMP layers.
Discussion It is difﬁcult to provide quantitative comparison because little research has been done on learning rotation invariant shape descriptors.
The state-of-the-art methods such as, volumetric methods and point cloud based method report very high classiﬁcation accuracies on ModelNet, but none of them can generalize to unseen orientations.
Given the rather task agnostic architecture of our model and the lossy but compact input representation we use, we interpret our models performance as strong empirical support of the effectiveness of learning alt-az rotation invariant shape descriptors using alt-az spherical convolution operators. [Table]
6. 3D SHAPE RETRIEVAL
We evaluate shape retrieval performance on the challenging SHREC’11 non-rigid dataset.
Intrinsic8 is used as our input spherical images and we extract the output 512 features of the fully connected layer as the shape descriptors.
Our approach signiﬁcantly outperforms all other methods with 0.
mAP retrieval performance.
Fig.6 uses the dimensionality reduction method t-SNE (van der Maaten & Hinton (2008)) to plot the rotation invariant feature descriptors extracted.
It shows that our learned descriptors successfully disentangle the original 3D object space and exhibit a clustered behavior in the feature vector space.
Finally, we run shape retrieval experiments on ShapeNet Core55, following rules of the SHREC’ 3D shape contest (Savva et al.
There is a aligned regular dataset and a version in which all models are perturbed by rotations.
We concentrate on the perturbed version to test the quality of our learned shape descriptors to unseen orientations for large scale rigid 3D object retrieval task.
An SO(2) rotation augmentation is performed by rotating each training data per 60 degrees about a random axis.
The same architecture and same type and size of input that we used for ModelNet classiﬁcation problem is used in this experiment.
The learned model from ModelNet40 was transferred and ﬁnetuned for SHREC’17 feature extraction.
Our trained model obtains a 83% classiﬁcation accuracy using the validation subset and we use the 1024 features extracted from the ﬁrst fully connected layer as the invariant shape descriptors to perform the shape similarity calculation using cosine distance.
We then evaluated our trained descriptors using the ofﬁcial metrics and compared to the top four competitors, which includes the other two spherical convolution based methods.
As shown in Table.
3, all three spherical convolution based methods (Cohen et al.
Esteves et al.
(2018) and ours) preform slightly below the current best, we believe that this is due to the information loss caused by projecting 3D shapes onto the 2-sphere.
To our surprise, all the three spherical convolution based methods report very similar performance, ours is slightly below Cohen et al.
(2018) and slightly above Esteves et al.
Both of the other two spherical convolution methods utilize Fast Fourier Transform (FFT) to compute the convolution which does not support local ﬁlters.
Ours offers an alternative method which complete the current work while offers multi-level feature extraction capabilities.
C ONCLUSION
In this paper we presented and analyzed a convolutional neural network based on alt-az anisotropic spherical convolution operator which is different from the existing types of networks.
Numerically, we implemented an efﬁcient algorithm for computing spherical convolution with locally-supported geodesic ﬁlters using icosahedron-sphere grid.
We demonstrated the efﬁcacy of our approach for non-rigid/ rigid shape classiﬁcation and retrieval and showed that it compares favorably to competing methods.
Furthermore, we have shown that the proposed method can effectively generalize across rotations, and achieve state-of-the-art results on competitive 3D shape recognition tasks, without excessive data augmentation, feature engineering and task-tuning. APPENDIX A A ZIMUTH ROTATION EQUIVARIANCE
Under the deﬁnition of alt-azimuth anisotropic convolution, for an alt-az rotation D R ( ϕ, ϑ, 0) , and a general rotation D Q ( ϕ 1 , ϑ 1 , ω 1 ) , (assume the number of channels K = 1 for simplicity,) we have:
[Equation]
[Equation]
Q − 1 R is in general a rotation in SO(3) , but when Q is an azimuth rotation, Q − 1 R =
R ( z ) − ϕ
[Equation]
( z ) ϕ R ( y ) ϑ
= R ( z ) ϕ − ϕ 1 R ( y )
ϑ is an alt-az rotation such that,
[Equation]
B SO(2) ROTATION AUGMENTATION FOR GENERAL ORIENTATIONS
Since composite a 3 SConv layers will not affect rotation equivariance property, without losing generality, we assume our network consists of one a 3 SConv and one GMP layer.
For any feature output from a GMP layer, suppose it is activated at point ˆ u 0 ( θ 0 , φ 0 ) .
It corresponds to a maximum correlation between the learned ﬁlter h and the input image f
(when h is “alt-az” rotated onto ˆ u o ).
Any
SO(3) rotation of f which ﬁrst rotates the point
ˆ u o back to the north pole ˆ η (0 , 0 , 1) , followed by an arbitrary alt-az rotation D ( φ (cid:48) , θ (cid:48) , 0) to new point ˆ u 1 , will be invariant to the network.
When h is convolved at point ˆ u 1 .
We deﬁne this set of rotation as “alt-az shift rotation”(see Fig.7(a)).
Alt-az shift rotation rotation causes no relative angular change of the geodesic disc centered at ˆ u o with respect to the h ’s convolving.
For a direction ˆ n ( ˆ n is not along z -axis or y -axis), if one rotates f about ˆ n an arbitrary angle which move the original salient point ˆ u o to ˆ u 1 , this rotation is in general not an alt-az shift rotation and will change the correlation between h and f when h is convolved at ˆ u 1 (See Fig.7(b)).
If one rotates f about ˆ n a full round, the geodesic disc centered at ˆ u o , will go through a relative self rotation fromto 360 degrees, using the alt-az rotation of h to each corresponding point as the direction references.
Therefore, by augmenting f using SO(2) rotation about an arbitrary axis ˆ n , any feature of f , after
SO(3) random rotation, can always alt-az shift rotate to the same feature in one of f ’s augmented copies.