N EURALLY BOOSTED SUPERVISED SPECTRAL CLUSTERING
Anonymous authors
Paper under double-blind review
A BSTRACT
Network embedding methods compute geometric representations of graphs that render various prediction problems amenable to machine learning techniques.
Spectral network embeddings are based on eigenvectors of a normalized graph Laplacian.
When coupled with standard classiﬁers, spectral embeddings yield good baseline performance in node classiﬁcation tasks, but they are widely considered to have been amply surpassed by newer embedding methods and Graph Convolutional Networks.
In this work, we provide a theory-informed implementation of supervised spectral clustering and maximize its performance with the use of a carefully designed R ADIAL classiﬁer that leverages fundamental geometric properties of spectral embeddings. R ADIAL raises the performance of supervised spectrally clustering to previously unobserved levels and renders it directly competitive to newer methods in the context of transductive node classiﬁcation on social networks.
We also observe that that features learned in an inner layer of R ADIAL can signiﬁcantly enhance the performance of standard Graph Convolutional Networks.
I NTRODUCTION
Spectral clustering is a classical unsupervised algorithm for clustering in graphs.
It gained prominence in data mining and machine learning approximately two decades ago (Shi & Malik, 2000; Ng et al.) and it has since found many applications in multiple domains.
While originally conceived as an unsupervised method, spectral clustering has also been employed in transductive learning, and in particular in semi-supervised node classiﬁcation that is the focus of this work.
Background.
The work of Tang & Liu (2011) was likely what initiated a long line of work on multi-label classiﬁcation with experimental focus on social network benchmarks.
They described a general framework for node classiﬁcation consisting of two steps: (i) computing a network embedding, (ii) using the embedding as input to a standard classiﬁer.
They instantiated step (i) with a spectral embedding, giving thus rise to what can be viewed as a supervised spectral clustering algorithm.
Albeit simple, the algorithm proved to be superior relative to the then-available baselines.
The thread continued with the very prominent D EEP W ALK embedding algorithm (Perozzi et al., 2014), that claimed superiority over spectral clustering which however remained competitive.
The success of D EEP W ALK unleashed a wave of research works on network embeddings, including (Grover & Leskovec, 2016;
Tang et al., 2015; Tsitsulin et al., 2018; Qiu et al., 2018a), and a race for higher classiﬁcation accuracy 1 .
More recently, network embedding methods were succeeded by graph neural networks (GNNs) (e.g. see (Kipf & Welling, 2017; Velickovic et al., 2018)) that have dominated the competition.
Since the publication of D EEP W ALK , supervised spectral clustering was at-best considered a baseline, or entirely vanished from consideration.
However it has recently made a comeback with the work of Huang et al.
(Huang et al., 2021) that described a simple ‘correction and smooth’ (C&S) procedure that can enhance the output of other models.
Some of the state-of-the-art results in in the Open
Graph Benchmark leaderboard (Hu et al., 2021) are now based on variants of supervised spectral clustering in combination with the C&S process.
These easily surpass previous embedding methods and even exceed methods based on GNNs using only a small fraction of their number of trainable parameters.
A Critical Look.
This work originates from the observation that previous literature has treated spectral clustering somewhat casually, and largely ignoring recent major advances in its theoretical understanding.
As discussed above, supervised spectral clustering consists of two steps: (i)
Computing a spectral embedding, and (ii) Giving the embedding as input to a multi-label classiﬁer.
In the literature, the term ‘spectral clustering’ is used for multiple different algorithms that implement steps (i) and (ii) in various and sometimes not adequately described ways.
For example, for step (i), (Tang & Liu, 2011) and (Perozzi et al., 2014) use eigenvectors of the normalized graph Laplacian as proposed in (Ng et al.)
but appear to omit a normalization step from that algorithm, (Huang et al., 2021) also use the normalized graph Laplacian but with an additional ‘rotation’ transformation, while other works use the random walk matrix (i.e. the non-symmetric normalization of the Laplacian) which in fact is a more accurate relaxation of the underlying NCut problem (Shi & Malik, 2000).
Classiﬁers used for step (ii) include an One-vs-All SVM-based model (Tang & Liu, 2011), an One-vs-Rest logistic regression model (Perozzi et al., 2014), a softmax regression model, and a three-layered neural network (Huang et al., 2021).
Adding to the confusion, the computation of eigenvectors has been viewed both as computationally expensive to the point of preventing its application on larger networks (Perozzi et al., 2014), or as able to scale up to very large networks via standard iterative eigensolvers whose fast convergence cannot however be guaranteed (Huang et al., 2021).
Goal.
These considerations provide the motivation of our work.
Concretely, we seek to provide a theoryinformed implementation of supervised spectral clustering, maximize its performance with the use of a carefully designed neural model and study its practical performance in the context of multi-label classiﬁcation on social networks.
Contributions. Our R ADIAL classiﬁer can be conceptually viewed as a ‘neural’ implementation of the seminal theoretical work of Lee et al.
(2014b) on multi-way Cheeger inequalities for spectral clustering.
These theoretical guarantees are based on a variant of spectral clustering that is not well-known or widely used, yet it can be theoretically analyzed without assumptions about the underlying graphs.
The analysis of Lee et al.
applies to unsupervised clustering.
Thus a key contribution of our work is to transfer the geometric intuition developed in (Lee et al., 2014b) to the supervised setting with the design of custom-made neural classiﬁers that respect and exploit the geometric facts.
Indeed, R ADIAL raises the performance of supervised spectral clustering on transductive node classiﬁcation to previously unobserved levels, sometimes more than 15% higher than vanilla classiﬁers. As our ablation study conﬁrms, this performance is achieved by introducing certain modiﬁcations and enhancements to the basic unsupervised algorithm of (Lee et al., 2014b), and other training tricks. Within our experimental context, when only graphic data are taken as input, we show that when training nodes are scarce, supervised spectral clustering is directly competitive to prominent alternative methods that were thought to be vastly superior.
On the other hand, we observe that non-parametric diffusion-based methods become progressively more effective when training points are abundant, thus offering an empirical explanation of the success of the C&S algorithm (Huang et al., 2021). Moreover, based on intuition derived in section 3.4, we develop a method of enhancing the performance of standard GCNs (Kipf & Welling, 2017). More speciﬁcally, in section 4, we show that when node features are available, the performance of GCNs can be signiﬁcantly enhanced by feeding as additional input features learned by our R ADIAL classiﬁer.
S PECTRAL C LUSTERING WITH S UPERVISION : T HE ALGORITHM 2. S PECTRAL E MBEDDING AND G EOMETRIC I NTUITION
We begin with a review of the spectral embedding algorithm.
Along it we give concrete details on how it can be implemented to run in near-linear time with strong convergence guarantees.
In Section 2.1.3 we discuss the geometric intuition that underlies the theoretical analysis in (Lee et al., 2014b).
We note that the material in this subsection is not new.
We present it for completeness and in particular in order to discuss the geometric intuition which plays a key role in the construction of our classiﬁers.
2.1. D EFINITIONS AND TERMINOLOGY
Let G =
( V, E, w ) be a weighted undirected graph.
Deﬁnition 1 ( Cut and Volume ) .
If S ⊂ V , we deﬁne the cut of S to be
[Equation]
We also deﬁne the volume of S to be
[Equation]
cut
[Equation]
We will often use the term degree for the cut of a single node.
Deﬁnition 2 ( Conductance ) .
The conductance of a cluster S ⊂ V is deﬁned by
[Equation]
We denote by D the diagonal matrix of the degrees, by d the vector of the node degrees, and by L = D −
A the graph Laplacian, where A is the adjacency matrix of the graph.
2.1. S PECTRAL EMBEDDING AND THE UNSUPERVISED ALGORITHM
The combinatorial idea that underlies spectral clustering is that good clusters should have a relatively small volume to their exterior, or more speciﬁcally a low conductance.
More concretely, the goal is to ﬁnd k disjoint clusters { S 1 , S 2 , ...
S k } such that max i φ ( S i ) is as small as possible.
In this paper we will be looking for k clusters that also provide a partitioning of the vertex set, i.e. we have (cid:83) i S i
= V .
The partitioning problem is less well-understood (Louis et al.), but its simpler non-partitioning variant has been studied extensively from a theoretical perspective (Lee et al., 2014b), and approximation guarantees are known for it.
The spectral clustering algorithm is broadly understood as a relaxation of the discrete clustering problem (e.g. see (Cucuringu et al.) for a complete discussion), and a number of variants have been proposed (von
Luxburg, 2007).
In its simplest form, the variant that is analyzed in (Lee et al., 2014b) consists of three steps, summarized here:
(a) Spectral Embedding.
Compute the eigenvectors x j corresponding to the k smallest non-zero eigenvalues of the generalized problem Lx = λDx , under the constraint that x T d = 0 .
The eigenvectors are normalized so that x Tj Dx j = 1 .
Let X ∈ R n × k be the matrix whose columns are the k eigenvectors 2 .
(b) Orthogonal Projection.
Let X (cid:48) =
XR , where R (cid:48) ∈ R k × k (cid:48) a random orthogonal projection matrix, where k (cid:48) ≤ k . discuss (c) Radial Projection.
Let Y be the matrix resulting by dividing each row
of
X by its Euclidean norm, i.e. by ‘radially projecting’ each point onto the unit sphere in R k .
Each row of Y becomes the embedding of the corresponding node.
(d) Geometric Partitioning.
Use some unsupervised geometric clustering algorithm on the embedding Y .
(e.g. k-means, or the provable algorithms presented in (Lee et al., 2014b)).
We remark that the two hyperparameters k and k (cid:48) are further speciﬁed in sections 2.2 and 3.1.
T HE GEOMETRIC INSIGHT
The radial projection step reﬂects the geometric intuition that the eigenvectors place the nodes into k directions in the embedding space.
Consider the eigenvector matrix X ∈ R n × k .
These generalized eigenvectors are D -orthogonal (Stewart &
Sun, 1990), i.e. we have x Ti Dx j = 0 , for each i (cid:54) =
j .
Given that the eigenvectors are also normalized, we have X T DX =
I .
Using a well known identity we have
[Equation]
Summing up the n diagonal entries of DXX T we thus get n (cid:88)
[Equation]
Now pick an arbitrary direction in R k , i.e. a unit vector z .
Let w =
Xz .
Since X T DX =
I , and z T z
=we have
[Equation]
Let us now take a closer look at the above equations.
From equation 1 we get that that the d -weighted mass of the n embedding points is k .
On the other hand, note that w 2 j is the norm of the projection of X [ j ] onto z .
Thus, equation 2 shows that the d -weighted mass of the projections of the n points onto z is always equal to 1 , for any direction z .
A further interpretation of the above equations is that any given direction z captures only a 1 /k fraction of the d -weighted sum of norms of the n embedding points.
Hence the d -mass of the points must concentrate in at least k different directions.
While ﬁnding such directions is not straightforward, (Lee et al., 2014b) proves that radially projecting the points onto the unit sphere has the following property: if S is a set of points in Y that are in a ball of a small diameter in R k , then S also cannot concentrate a large fraction of the d -mass of the points, and thus the points in Y are expected to be in k well-separated spheres in R k .
This speciﬁc geometric arrangement facilitates the design of an unsupervised geometric clustering algorithm on the points of Y in (Lee et al., 2014b), but it will also guide the design of our supervised geometric clustering model.
2. N EURAL C LASSIFIERS
We present three types of neural classiﬁers that reﬂect our discussion in section 2.1.
These are different but they share certain layers and so they are presented in a compact way in Figure 1.
Below we make our main points on the design of these classiﬁers.
Details of the algorithm are discussed in section 2.2.
Section 2.3 considers initialization and training tricks.
We note that k is the embedding dimension, K is the number of channels, and nc is the number of classes.
Step (ii) in the unsupervised algorithm described in section 2.1.2 is a random orthogonal projection.
In analogy, we introduce a ‘Reduce Layer’ which in our case is entirely trainable .
The Reduce and Radial
Projection layers are reproduced independently in K independent channels, where K is a hyperparameter.
Each of these channels has its own operator R i .
We propose three different types of classiﬁers: Conic , Linear , and Spherical .
These share the same parameters W and β .
The outer dimension of W is proportional to the number of channels, but the inner dimension of both W and β are equal to the number of classes.
What differentiates these three types of classiﬁers is the role of W and β .
Their names reﬂect the corresponding geometric shape of the decision boundaries.
Linear:
Each of the nc rows of W deﬁnes a hyperplane in R K × nc as its normal vector.
The decision boundary for row W j is the hyperplane yW + β .
Conic: Row W j represents represents a direction in R K × nc .
The decision boundary is given by equation yW j − || W j || 2 cos ( β ) , which is well-understood to describe the surface of a cone with axis W j and aperture β , given that y has unit norm (line 12).
We note that the decision boundary can be rewritten as yW j + β (cid:48) , where β (cid:48)
=
||
W j || 2 cos ( β ) .
Thus it can be seen that Conic is a restricted or ‘regularized’ version of Conic because it enforces β (cid:48) ≤
|| W j || 2 .
Spherical: Row W j represents a point in R K × nc , which is the center of the sphere with of radius β .
The decision boundary is the sphere β 2 −
|| W j − y
|| 22 .
2. I NITIALIZATION AND T RAINING
The discussion in previous sections allows us certain considerations on the initialization of the trainable parameters.
In the unsupervised algorithm, the Reduce Layer is taken to be a random orthogonal projection, and we can thus initialize it as such.
Moreover, according to the discussion in section 2.1.3, the node embeddings given by XR are expected to concentrate in at least nc different directions/geometric clusters after the radial projection step.
Approximate directions or ‘centers’ for these clusters can be found using the spectral embeddings of the training points.
Speciﬁcally, in order to compute such approximate centers or directions, we take the radial projection of the training points for each cluster separately, compute their average, and then we radially project it.
These initializations can be carried out independently for each of the K channels.
In our experiments we have used such initializations; they do not affect accuracy, but seem to improve convergence speed (see Appendix C.1).
We also explore a pre-training strategy for K -channel models.
We ﬁrst train K single-channel models with different random initializations.
The Reduce layers from these K models are used to initialize the corresponding layers of the K -channel model, while the last layer (parameter matrix W ∈ R K · nc × nc ) is initialized randomly.
E XPERIMENTS
This section summarizes our experiments.
Our comments on the implications of these experiments are given as bullet-points.
3. E XPERIMENTAL S ETUP
Datasets, Hyperparameters, Variance.
We use well-known benchmarks and follow standard practices.
We refer the reader to Appendix A for detailed information.
Embedding Dimension.
We uniformly set the embedding dimension (parameter k in Figure 1) to k = 2 · nc .
We note that the performance of all methods can be slightly raised with a choice of higher dimension, but we stick with a choice proportional to the number of classes in order to conform with the underlying theory.
Evaluation Method.
We measure the accuracy of the various methods trained with randomly selected training sets at label rates: 2 j % , for j = 0 , . . .
For each label rate, we pick 5 different random training sets.
For each of these training sets we train each classiﬁer for 5 different random initializations of the trainable weights.
For each such experiment, we perform 3-fold cross validation where each validation set is selected randomly and consists of 20% of the training set (i.e. 20% of the label rate).
We ensure that every class is represented in our training set by forcing one random node from each class in the training set.
Validation is used for R ADIAL in order to select the best-performing model over the course of training.
3. A BLATION S TUDY
In Figure 2, we present an ablation study on the layers of the R ADIAL classiﬁer. and a shorter ablation study on multiple channels can be found in Appendix E.1.
Classiﬁer N O -R ADIAL in the Linear branch of R ADIAL , excluding the Reduce and Radial Projection layers.
Classiﬁer N O -R EDUCE runs the Linear branch excluding the Reduce layer.
The results are similar when Conic or Spherical are used instead. Classiﬁers C ONIC and S PHERICAL include all layers of the R ADIAL algorithm in Figure 1.
• We note that despite its very different and non-standard mathematical form, the performance of S PHERICAL is comparable to C ONIC , and potentially better. However we have found that it requires more training, and the reported results use 4K epochs of training for S PHERICAL , and 2K epochs of training for S PHERICAL . For that reason, we stick with C ONIC for the multi-channel experiments. • Overall, Figure 2 justiﬁes the layers of R ADIAL , especially for larger datasets, and demonstrates the extra power of multiple channels and appropriate pre-training strategies.
The C ORA dataset is a clear exception that we discuss separately in section 3.4.
Further ablation studies are reported in section E.1.
3. R ADIAL VS S TANDARD C LASSIFIERS
In Figure 3 we present a comparison of C ONIC -2C and C ONIC -2C- PT with standard classiﬁers.
As input to these other classiﬁers, we use the radially projected spectral embedding, as it maximizes their performance.
We compare with a simple one-vs-rest logistic regression classiﬁer.
Speciﬁcally, we use the L IBLINEAR library, with its default regularization (L IBLINEAR -R) and no regularization (L IBLINEAR -NR).
We also compare with a kNN classiﬁer, for k = 11 that we found to be a good choice.
We have also performed experiments with a standard SVM classiﬁer which did not perform as well.
We note that these classiﬁers do not use validation; the validation sets are added to the training sets.
• L IBLINEAR -R was the classiﬁer of choice in D EEP W ALK (Perozzi et al., 2014), where it was also used in combination with spectral embeddings.
We see that R ADIAL classiﬁers increase accuracy, sometimes by more than 15% relative to this previously used standard classiﬁer 3 . • We also observe that the use of regularization is detrimental to the accuracy of L IBLINEAR , which corroborates the theoretical insight on the value of the radial projection step.
However, C ITESEER and C ORA are exceptions, indicating that overﬁtting may be an issue for these smaller datasets.
• Overall the results of this section show that R ADIAL classiﬁers perform better than standard classiﬁers, with the notable exception of C ORA , that we discuss separately, in section 3.4. 3. R ADIAL S PECTRAL C LUSTERING VS O THER M ETHODS
In this section we compare the R ADIAL classiﬁer with other methods.
The goal of this section is not to claim that R ADIAL classiﬁers are superior, but to compare them directly with some widely acclaimed methods.
Hence, the list is not exhaustive by any means.
time consuming Speciﬁcally we consider these algorithms: DeepWalk (Perozzi et al., 2014), NetMF (Qiu et al., 2018b), a standard GCN model 4 (Kipf & Welling, 2017) and the R ANDOM W ALKER algorithm (harmonic label interpolation) (Grady, 2006).
R ANDOM W ALKER is a non-parameteric algorithm and it is much faster than all other methods. On the other hand, D EEP W ALK and N ET MF are computationally much more expensive than R ADIAL .
We observe that: • Standard GCNs are less competitive in the context of graphic data. Thus we can conjecture that GCNs make sub-optimal use of graphic features.
• Due to the approximation loss reﬂected in standard Cheeger inequalities, eigenvectors of the normalized Laplacian do not always yield good approximations of low-conductance clusters. According to theory developed in (Koutis & Le, 2019), in such cases, a graph modiﬁcation can alleviate the problem.
C ORA -mod is such a modiﬁed version of the C ORA graph 5 . Thus, the fact that C ORA -mod gives rise to a much better spectral embedding explains the failure observed for C ORA in previous sections.
• R ANDOM W ALKER has always a very strong performance in higher training ratios, but it can be very erratic in lower training ratios, where non-trainable graph embeddings do better. In general, R ADIAL is directly competitive to D EEP W ALK and N ET MF, for a small fraction of runtime. • R ANDOM W ALKER is based on a harmonic interpolation objective similar to that used in the Correction & Smooth (C&S) procedures in (Huang et al., 2021).
Indeed, in Figure 9 in the Appendix, we observe that the accuracy enhancement added by C&S correlates strongly with the training ratio.
Figure 9 also shows that an improved ‘base’ generally improves the ﬁnal output of C&S.  H ANDLING N ODE F EATURES
In this section we go beyond graphic features, and include existing node features in R ADIAL . To enable our R ADIAL classiﬁers to work with features we incorporate one extra channel to the architecture of Figure 1. The features are standardized to have zero mean and constant standard deviation, and the rest of the layers are kept in place, including the Radial Projection layer.
We ﬁnd that this simple change raises the accuracy of our model in all cases, except the C ITESEER data set. These results are reported in Figure 5. B OOSTING GCN S WITH LEARNED GRAPHIC FEATURES We have found that GCNs make a better use of node features and in fact produce higher accuracy relative to that reported in Figure 5 for C ONIC -2-FT.
Thus GCNs remain very strong when node features are available.
Given that R ADIAL performs better than GCNs in the graphic setting, we thus propose to use the learned features of our C LASSIFIER . Speciﬁcally we use the activations of the Concatenation layer of the model learned in Figure 5, and feed them as additional input to the standard GCN. The outcome of this approach for the A RXIV data set is shown in Figure 6. A thorough exploration of the power of these learned graphic features in the broader context of GNNs is not in the indented scope of this work, and it is left open for future work.
C ONCLUSION
Research on transductive node classiﬁcation tasks has recently been dominated by Graph Neural Networks and the evolution of progressively more elaborate and highly-parameterized neural architectures that yield end-to-end trainable network embeddings.
This is in stark contrast to the pre-GNN period, when the collective focus was on the design of network embedding methods, with apparently little research devoted to the downstream classiﬁcation task.
In this work we argued that developing a better theoretical understanding of the geometric embedding opens up the fundamental research direction of designing better geometry-aware classiﬁers.
This direction can lead to surprising substantial improvements, even for older algorithms, such as spectral clustering.
In principle, this reasoning can be applied to other existing network embedding methods with the potential to –more broadly– inform the design of improved graph neural architectures.
A D ATASETS , H YPERPARAMETERS , V ARIANCE .
Datasets.
We use 6 labeled datasets that have been used widely as benchmarks.
These datasets have been processed to remove small connected components because spectral clustering naturally works on a connected graph 6 ; self-loops are also removed.
For all datasets, we treat the graphs as undirected and only consider the largest connected component.
Table 1 summarizes the number of vertices, edges and classes of these largest connected components.
[Table]
Code and Experiments. All experiments were performed on Google Colab. The ﬁnal paper will disseminate all code, including the models and the experimental setting.
Hyperparameter Settings.
In RCS-C LASSIFY (algorithm 1), we take k = 2 · nc , where nc is the number of classes.
In non-reported experiments we have found that further increasing the number of dimensions can slightly increase accuracy, but we take a theory-consistent decision to make the dimension proportional to number of clusters.
The Random Walker algorithm does not have any hyperparameters or learnable parameters.
For the GCN model we used the recommended settings, using the identity matrix in the input into the GCN.
For NetMF we followed the recommended default settings, using an embedding of
dimensions.
The dimension for DeepWalk is set to be 2 · nc ; this deviates from the recommendation but it is added here for direct comparison with the setting for R ADIAL .
Variance.
We note that we omit information on the standard deviation of accuracy over random splits and random initializations. This is in general low ( < 0.5%) with the exception of 1-2% training ratio for the three smaller datasets, where standard deviation can be as high as 3%.
B F URTHER D ISCUSSION
• We have also considered the applications of Support Vector Machines on the radially projected spectral embedding.
The results were not competitive, and thus they are not included here.
• The R ANDOM W ALKER method has rarely -if ever- been used as benchmark for supervised node classiﬁcation on social graphs.
However the success of C&S, a closely related algorithm, motivates the experimental study of the R ANDOM W ALKER algorithm, the strengths of which include the fact that it does not have any hyperparameters or trainable parameters.
We see that in multiple cases R ANDOM W ALKER is a viable alternative to other models, especially considering that with a proper implementation using Laplacian solvers, R ANDOM W ALKER is by far faster than all other methods in Table ?? .
The C&S process is not applicable to the output of R ANDOM W ALKER for certain concrete technical reasons, but a worthy research topic it to modify R ANDOM W ALKER in a way that would enable the application of C&S, in order to enhance its performance. C E FFICIENT COMPUTATION
We discuss the computation of eigenvectors of the generalized eigenvalue problem Lx = λDx .
Several works compute eigenvectors using implementations of A RPACK (Lehoucq et al., 1997) in libraries for sparse matrix algebra, in Matlab, Python, or Julia (e.g. (Huang et al., 2021)).
Social networks usually have eigenvalue distributions that are ‘friendly’ to power methods (that are used in A RPACK ) in the sense that they can converge in a reasonable amount of time.
However that is not guaranteed, and in practice convergence can be much slower than possible.
Fast convergence can presently be guaranteed only via using inverse-power algorithms, i.e. algorithms that use approximations of the inverse of the Laplacian, often used as preconditioners. In this work we use the preconditioned eigensolver LOBPCG (Knyazev, 2001) with the CMG preconditioner (Koutis et al., 2011). Besides its stronger theoretical properties, the particular implementation is also signiﬁcantly faster in practice, even on social networks. C. I NITIALIZATION AND C ONVERGENCE
D T HE ALGORITHM
do
K 11:
y =
xR
i
12: y i = y/ ||
y ||
14: y =
[ y 1 ; . . .
; y K ]
15: if then 16: w nrm = norm (
17: y = yW − w nrm .
∗ cos ( β ) (cid:46) pointwise multiplication 18: else if type = ‘Linear’ then 19: y
= yW + β 20: else if type = ‘Spherical’ then 21: y = β 2 − ( W − 1 K · nc × 1 y )
22: end if 23:
y = softmax ( y ) 24: end function
E M ISC E XPERIMENTS E. A BLATION EXPERIMENTS
We have performed these additional ablation experiments:
• We added bias to the Reduce layer.
This makes the models more general. However, in practice, it has a negative impact on accuracy. • We made the Reduce layer non-trainable and orthogonal (as suggested in Lee et al. (2014a)).
This has a signiﬁcantly negative impact on accuracy.
• We have added a regularization term || R ∗ R T −
I || F that penalizes the Reduce layer for deviating from orthogonality.
This has a negative impact on accuracy.
We also experimented with adding more channels to the R ADIAL classiﬁers, which seems to have a positive impact on accuracy .
On a speciﬁc ﬁxed training set for the A RXIV recorded accuracies are shown in
Figure 8.
F A LL R ESULTS WITH C ORRECTION AND S MOOTH