Spherization Layer: Representation Using Only Angles
Abstract
In neural network literature, angular similarity between feature vectors is frequentlyused for interpreting or re-using learned representations.
However, the innerproduct in neural networks partially disperses information over the scales andangles of the involved input vectors and weight vectors.
Therefore, when using onlyangular similarity on representations trained with the inner product, informationloss occurs in downstream methods, which limits their performance.
In this study,we proposed spherization layer to represent all information on angular similarity.
The layer 1) maps the pre-activations of input vectors into the specific range ofangles, 2) converts the angular coordinates of the vectors to Cartesian coordinateswith an additional dimension, and 3) trains decision boundaries from hyperplanes,without bias parameters, passing through the origin.
This approach guarantees thatrepresentation learning always occurs on the hyperspherical surface without theloss of any information unlike other projection-based methods.
Furthermore, thismethod can be applied to any network by replacing an existing layer.
We validatethe functional correctness of the proposed method in a toy task, retention abilityin well-known image classification tasks, and effectiveness in few-shot learning,word analogy test, and hyperspherical learning.
Introduction
The inner product is a key element constituting layers in deep neural networks with the nonlinear activation function.
The inner product with the Euclidean norms and the angle, that is, ∥ w 𝑖 ∥∥ x 𝑗 ∥ cos 𝛉 𝑖𝑗 ,has been analyzed in terms of the norms ∥
w 𝑖 ∥∥ x 𝑗 ∥ and the angle cos 𝛉 𝑖𝑗 , independently [14, 15, 37].
All factors of the inner products learn distinct information.
Therefore, using only one factor results ininformation loss when re-using or understanding the information in downstream tasks.
This problemis termed as dispersion problem .
The angular similarity between features is frequently used.
However,this technique causes a dispersion problem in advanced methods in neural network literature such asdecoupled network [15], representation learning [6, 7, 23, 24, 34], metric learning [35], zero-shotlearning [26] and generative model [3, 27].
To mitigate the dispersion problem, numerous anglebased learning approaches [1, 12, 13, 16, 18, 17, 35, 36, 38] have been proposed.
However, thesestudies are based on projection onto the hyperspherical surface.
In this projection method, distinctionby the scale of features is ignored.
Therefore, information loss occurs.
We proposed the spherization layer as an explicit solution for the dispersion to completely eliminate the interference of the norms in training without drawbacks.
This layer is used to locate all represen-
The spherization layer consists of three main components: (1) Angularization function that convertsthe pre-activations from the previous hidden layer to angles; (2) Conversion from spherical coordinatesto Cartesian coordinates located on the hyperspherical surface; (3) No-bias layer, a hidden layerwithout bias parameter, that determines decision boundaries by using only the angles.
Figure 1illustrates the design of the spherization layer.
Through spherization , representations are locatedon the hyperspherical surface and the effect of the norms in representation learning is completelyeliminated.
Therefore, neural networks are enforced to express all representations differently byusing only the angles.
We experimentally verified the functional correctness of the spherization layer in a toy task and itsapplicability to feedforward and convolutional neural networks by evaluating performance on imageclassification tasks.
The results reveal that the training ability of original networks is preserved afterapplying the spherization layer.
Furthermore, we analyzed the sensitivity to width and depth.
Finally,we investigated the effect of the methods in downstream tasks through visualization, word analogytest, and few-shot learning.
We compared our methods to an angle-based approach using directprojection [15, 19, 21].
The contributions of this paper are as follows:
• To address the dispersion problem, we propose the Spherization layer to represent all featurevectors on the hyperspherical surface and learn the representations with only the angles.• We validate the wide-applicability and scalability of the spherization layer without any loss of
performance through experiments on various well-known networks.
• We empirically show that the spherization layer can be used in many applications in which
angular similarity is an critical metric in the task.
Background
Conversion Spherical to the Cartesian Coordinate System
In most neural networks, all inputsamples on the n -dimensional space are represented as Cartesian coordinates, in which i -th columnvalue denotes the distance from the origin along the i -th axis, and neural networks trains them by usingneurons in which the inner product between the input and weight vector occurred.
The pre-activationsfrom neurons are used to determine whether neurons should be activated by using the followingactivation function.
In this process, the pre-activations are calculated by using weight and biasparameters, represented as Cartesian coordinates, which can be converted to spherical coordinates.
Given a vector s represented as Cartesian coordinates on the n -dimensional space, s can be definedas spherical coordinates 𝛉 =
[ 𝑟, 𝛗 ] , composed of a radial coordinate r and n -1 angular coordinates𝛗 =
[ φ 1 , φ 2 , ..., φ 𝑛 − 1 ] .
In this case, the k -th axis of s can be computed from 𝛉 with Eq.
[Equation]
, where 𝑠 1 <𝑘<𝑛 = 𝑟 cos φ 𝑘 (cid:206)
𝑘 − 1 𝑖 = 1 sin φ 𝑖 .
Generally, the spherical coordinate system is a 3-dimensional version of the polar coordinate system.
However, the spherical coordinate system in the followings indicates all n -dimensional version of thepolar coordinate system, where n is greater than or equals 2.
Spherization Layer
We proposed spherization layer as displayed in Figure 1 to locate feature vectors on the hypersphericalsurface without any loss of information and learn hyperplanes by using only the angles.
In thismethod, a layer of an original network is selected to learn representations by angular similarity.
Next,the layer is replaced to a spherization layer through three sequential stages: (1) Angularization , (2)
Conversion to Cartesian , and (3) No-bias training notated as 𝑓 , 𝑔 , and ℎ functions, respectively.
Thefinal form of spherization layer 𝑓 𝑠𝑝ℎ is expressed as Eq. 2.
[Equation]
After training with the layer, we can obtain the representations on the ( 𝑛 + 1 ) -spherical surface,namely spherized representations, as the outputs of ( 𝑔 ◦ 𝑓 ) operations.
The common goal of all stages is to preserve the training ability of the original network while thespherization layer trains all information on the ( 𝑛 + 1 ) -spherical surface.
The angularization locatesall pre-activations on the safe spherical surface.
The conversion to Cartesian results in the generationof compatible representations to the ordinary layer.
No-bias training enforces training by only theangles.
In the followings, we elaborate each stages more detail and only annotate the ( 𝑙 − 2 ) th and( 𝑙 ) th layer as [ 𝑙 − 2 ] and [ 𝑙 ] , respectively, for the simplicity.
3.1 Angularization
[Equation]
Angularization is the stage to map a pre-activation vector z , passed from ( 𝑙 − 1 ) th layer, to angularcoordinates 𝛗 .
The 𝑛 indicates the dimension of the pre-activation vector.
The role of this stage is toconfigure the shape of the mapped region on the ( 𝑛 + 1 ) -spherical surface for resolving training andcomputational difficulty.
Angularization 𝑓 is implemented by applying the following element-wise function to all coordinatesof z as an activation function, and 𝑓 is illustrated as Eq.4.
[Equation]
, where the terms and form are used by three following motivations.
Converting Pre-Activation to the Angular Coordinate The first step is to convert the input vectorinto angular coordinates.
To ensure conversion as bijective mapping, we restricts the range of thefunction as [ 0 , 𝜋 2 ] .
To allow a unrestricted input representation on the real-valued domain, the sigmoidfunction 𝜎 (·) is used with weight 𝜋 2 for the range setting.
As the sigmoid function used, the inputvector should be pre-activations, not activations because when activations from ReLU or sigmoidare passed to the angularization, inefficient use of the spherical surface or gradient amplification,respectively, may occur.
After converting pre-activations to angular coordinates, the representationson the hyperspherical surface in the same range were located by setting a consistent radius over allinputs.
This radius scale is controlled in the conversion-to-Cartesian stage.
Tailoring Angular Representation Space
In the conversion from angular to Cartesian coordinates,the last coordinate can be an extremely small value because trigonometric values in [0, 1] aremultiplied many times.
This scale descent can map all values in the axis to only a single value by thelimit of the floating point data type.
To reduce this effect in the angularization, we introduce a lowerbound 𝜑 𝐿 of angles to guarantee distinguishable values in its corresponding converted Cartesiancoordinates as follows:
[Equation]
, where 𝛿 is a minimal trigonometric value to guarantee the distinguishable representations.
Thedetails are presented in Appendix A. We set 𝛿 to the empirically obtained proper value 10 − 6 , for allexperiments.
Scaling Pre-Activations In angularization, the activations are concentrated onto the small regionbecause of the lower bound.
This concentration renders training difficult with the decrease in thevariance.
To reduce the effect, we set a learnable parameter 𝛼 as a weight of z , which controls thevariance of z .
Using this scale factor, the generated angular representations become abundant.
3.2 Conversion-to-Cartesian
[Equation]
In the Conversion-to-Cartesian stage, the angular coordinates of the previous stage are convertedto Cartesian coordinates on the ( 𝑛 + 1 ) -spherical surface.
This conversion ensures the consistencybetween the output of angularization (polar coordinate system) and the input of the following no-biaslayer (Cartesian coordinate system), and enables the layer to be trained in the same way as generalneural networks.
This implementation is based on Eq.
1 with the modified range of angles as thefollowing equation.
[Equation]
Calculation Trick Implementation of Eq.
7 as a tensor operation requires the trick defined in thefollowing equation:
[Equation]
, where 𝛟 is a dimension-expended vector in R 𝑛 + 1 and 𝜙 𝑛 + 1 = 𝜙 𝑛 , and 𝑟 is a constant to controlradius.
Here, W 𝜑 , W 𝜙 , and b 𝜙 are constant matrices and vector in R 𝑛 ×( 𝑛 + 1 ) , R ( 𝑛 + 1 )×( 𝑛 + 1 ) , and R ( 𝑛 + 1 ) ,respectively.
W 𝜑 =
[ I 𝑛 ; v ] , where v = [ 0 ; 1 ] ⊺ ∈ R 𝑛 , is used for matching the dimension between 𝛗and s .
The other constants are used for calculating logarithm trigonometric values from expandedangular coordinates in the way of matrix multiplication, where W 𝜙 is an upper triangular matrix inwhich all diagonals are zero and ( W 𝜙 ) 𝑛,𝑛 + 1 is also zero, and b 𝜙 = [ 0 ; − 𝜋 2 ] ⊺ .
See Appendix B fordetail hyperparameters and process.
3.3 No-bias Training
[Equation]
Through angularization and conversion-to-Cartesian, all representations are located on the ( 𝑛 + 1 )-spherical surface.
In this case, any update to the representations is determined by the change of angularsimilarity.
However, hyperplanes determined by the next layer on the ( 𝑛 + 1 ) -dimensional space maynot use only the angular similarity, which may assign semantic information to the Euclidean norm ofthe spherized representation.
To synchronize these parameters, we used no-bias training using onlythe weight parameters W [ 𝑙 ] as illustrated in Eq. 10.
[Equation]
Effect of Bias Elimination to Training In the ordinary layer, the problem of bias elimination is thathyperplanes passing through the origin cannot be shifted to another parallel hyperplanes.
However,the problem disappears when all feature vectors are located on the ( 𝑛 + 1 ) -spherical surface becausethe decision boundary can be shifted by only the angle changes of ( 𝑛 + 1 ) -dimensional hyperplaneseven though they pass through the origin.
3.4 Optimization with Overall Process
A training loss 𝐿 is calculated in the same manner as the original network.
The gradient of 𝐿 for thespherization layer is calculated by multiplying the following partial derivative 𝜕𝐿𝜕 W to the originalbackpropagation step from the ( 𝑙 − 2 ) th to ( 𝑙 ) th layer.
The first term 𝜕𝐿𝜕 z [ 𝑙 ] is calculated by thesubsequent layers in the same way of the original network.
[Equation]
Experiments
[Figure]
First, we define two terms for simplicity on indicating two networks: one is the network beforeapplying the spherization layer, called original network , and the other is the network after substitutingan ordinary layer with the spherization layer, called spherized network .
In the followings, we usedthese two terms consistently.
All experiments were performed five times with random seeds and theirtraining and test accuracy were evaluated except word analogy test and few-shot learning.
The mean𝜇 and standard deviation 𝜎 of accuracy are represented as 𝜇 ± 𝜎 in each table.
Experimental Settings We verified the spherization layer for learningdecision boundaries on a simple binary classification task.
We set upthe simple binary classification task: given ( x 𝑖 , 𝑦 𝑖 ) pairs, where x 𝑖 is thei -th input sample in R 2 and 𝑦 𝑖 ∈ { 0 , 1 } is the label of x 𝑖 .
We randomlygenerated 100 input samples located around (0, 0) for the label 0, and theother 100 samples for the label 1 around (1, 1), as displayed in Figure 2.
We set a 2-layer neural network as the original network, and trained itwith the softmax function, cross-entropy, and SGD at a learning rate of0.01.
We applied the proposed method to the original network by replacing the last fully connectedlayer with the spherization layer.
For comparison of representations in the same dimensional space,we set the dimension of the spherization layer to 1, where it is 2 in the original network.
The othersettings are identical to the original network.
We trained both networks on the input samples for 100epochs with 16 mini-batches, where the networks converged and achieved 100% training accuracy.
Distribution of Representations Initially, all representations are randomly distributed into twogroups on a 2-dimensional space.
After convergence, the feature vectors are divided into twodisjointed groups, as displayed in Figure 2.
This means spherized network locates all representationson the 2-spherical surface in both the initial and final epoch, whereas the original network spreadsthem out.
Hyperplanes Hyperplanes and decision boundary on the hidden vector space are defined as followswith the parameters of the output layer.
[Equation]
, where W 𝑖 is the hyperplane determined by weight parameter w 𝑖 and bias b 𝑖 , and D 𝑖𝑗 is a lineardecision boundary whose points satisfy 0.5 confidence for both labels.
In the spherized network, allbias terms are eliminated.
In Figure 3, the hyperplanes are illustrated as yellow lines and the decisionboundary as a gray line.
In the spherization layer, hyperplanes and decision boundary pass throughthe origin from the initial to the last epoch, unlike the original network.
The results imply that thespherization layer can learn the correct decision boundary by changing only angles of hyperplanespassing through the origin.
4.2 Retention of Training Ability on Image Classification Benchmarks
Environmental Settings In this task, we empirically verified that replacing an existing layer to aspherization layer still maintains the training ability of the original networks on well-known imageclassification tasks.
See Appendix C for the detail about networks and datasets.
Results and Analysis
The accuracy results of original and spherized networks are compared in
Table 1.
The reference and reproduced results of each setting are similar.
The accuracy results ofthe spherized network implemented on the reproduced code are similar or slightly higher on bothtraining and test data than that of the original network.
These results imply that the spherization layermaintains the training ability of the original network.
In Table 2, the accuracy results of originaland spherized networks are displayed in various width and depth settings.
The spherized networkconsistently exhibits similar or higher test accuracy over all width and depth settings.
The resultsreveal that the spherization layer again preserves the training ability of the original networks.
4.3 Analysis on the Effect of Projection
Environmental Settings We compared the method with another angle-based approach to analyzethe effect of reducing information loss by projection.
For comparison, we used 9-layer CNN, namely
CNN-9, with the same experimental settings [19] but reproduced in PyTorch.
[Table]
[Table] [Table] [Table]
[Table]
Results and Analysis To apply a spherization layer, we changed the last hidden layer FC-10 in eachmodel.
Table 3 presents the comparison of the performance of the spherization layer with angle-basedoperators, as called SphereConv operators [19].
In direct projection, the angle-based operatorsconsistently reduce performance, which project input vectors onto the hyperspherical surface forangle-based learning.
In indirect projection as W-Softmax variants omitting feature normalization,the decreased performance of the angle-based operators recovers to the score but still worse thanthe spherization layer.
These results imply that guaranteeing spherized representations improvesperformance by resolving the information loss of projection (See Appendix E).
4.4 Downstream Tasks: Visualization
Environmental Settings To investigate the spherization layer locates feature vectors on the ( 𝑛 +1 ) -spherical surface, we implemented a simple CNN to learn 3-dimensional feature vectors forvisualization.
This technique is called CNN-Vis3D, which generates 3-dimensional feature vectorsand shows those representations as a graph.
Detailed configurations are presented in Appendix C.
Results and Analysis
The results of visualization are depicted in Figure 6.
In the 2D and 3D featurevisualization results of the original network, the representations are distributed over wide rangesof both scale and angles.
As displayed in decoupled network [15], the angle accounts for semantic
[Figure]
[Table] [Table]
difference and the feature norm accounts for intra-class variation.
In the spherized network, all theroles are expressed by the angle change on the hyperspherical surface.
Thus, in the spherization layer,all trained information can be used in angular similarity-based interpretation.
4.5 Downstream Tasks: Word Analogy Test
Environmental Settings We used BERT [5] and RoBERTa [20] to conduct the word analogy test,in which angular similarity is used to predict a relation type between words.
The settings wereidentical with [31].
We applied a spherizaton layer to the last hidden layer of the encoder in eachmodel.
Next, we trained them on WikiText [22] for 3 epochs with 8 mini-batchs, the softmax functionfollowing cross-entropy, SGD at a learning rate 10 − 4 on masked-language modeling.
Results and Analysis Table 4 presents the performance evaluated with three metrics [31].
Applyingthe spherization layer to BERT and RoBERTa improves most average scores.
The improvementimplies that the spherized representations provide accurate information for angle-based distinction ofword relations.
4.6 Downstream Tasks: Few-shot Learning
Environmental Settings We used ProtoNet [29] with ConvNet and ResNet for fewshot learningon Mini-ImageNet [32], in which several feature vectors are compared with the feature vector of aninput image by using distance metric.
This task was performed according to the guidelines in [2].
Results and Analysis Table 5 details the performance results in few-shot learning [2].
Generally,the Euclidean distance is used to discriminate feature vectors.
However, the Eucliean distance alsohas the dispersion problem.
To focus on only the angles, we trained the models with cosine similarityand compared the performances.
As shown in Table 5, all spherized models with cosine similarityoutperform the baseline models.
This improvement indicates the spherized representations are usefulfor the angle-based metric such as cosine similarity.
Related Works
Semantic Analysis on the Inner Product The inner product is a crucial operator in current neuralnetworks, in which the distance between input vector x and weight vector w is encoded.
At thedecoupled networks [15], the inner product is reparametrized with the norms and the angle, and theintra-class variation and the semantic difference are modeled in neural networks by decoupling them.
Furthermore, the substitutes of the inner product have been proposed, where the direction of gradientor the similarity between kernels are used as the key factor instead of the inner product [14, 37].
Thespherization layer can be considered to be the substitute of the inner product, which normalizes theinput vectors by locating them on the hyperspherical surface.
However, the spherization layer is adirect and specific method to convert feature vectors focused on the angles without information loss.
Angle-based Approach The semantic analysis on the inner product has revealed that the streamsfocuses on the information in the angles.
The angle is a crucial factor, in which the most abundantand discriminative information is preserved [1, 12, 13, 18].
In SphereFace variants [13, 16, 17, 38],the angular softmax that enables CNNs to learn discriminative features on angular separability wasused.
Furthermore, some of this angular information have been used for regularization [35, 36].
In most angle-based studies, the angular information was used indirectly by the objective functionor regularization, whereas the spherization layer ensures the model directly learns the angularinformation on the hyperspherical surface.
Hyperspherical Representation Learning In some angle-based approaches, input vectors weredirectly projected onto the hyperphserical surface [15, 19, 21].
These hyperspherical representationlearning methods normalized the input vectors to ensure models are dependent only on the angle.
However, this normalization is the projection onto the hyperspherical surface, and it can be lessdiscriminative when some points overlap after the projection.
The spherization layer locates the inputvectors on the hyperspherical surface without the overlap problem through the spherization.
Conclusion
We addressed the dispersion problem of trained information to the Euclidean norm and angleon representations.
To address the dispersion problem, we proposed spherization layer to learnrepresentations by using only the angles without information loss.
We used the angularizationfor using pre-activations as angular coordinates, conversion-to-Cartesian for locating them on the( 𝑛 + 1 ) -spherical surface, and no-bias training to learn representations by using only the angles.
Inthe experiments on toy, image classification benchmarks, few-shot learning, and word analogy test,the method achieved accurate learning of the decision boundary, retention of the original trainingability, and improved performance in downstream tasks using angle-based information re-used orinterpreted.
Compared with the other projection-based approach, the method achieved accurateand robust performance.
This method can be applied to numerous network layers and downstreamapplications.
A limit of this approach is that the spherization layer should be inserted to a networkin a training step to fully utilize its advantage, which restricts the use of pre-trained models trainedwithout it.
Recovering trained information on representations with sampling should be investigatedin the future.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] See Abstract and Introduction (Section 1)
(b) Did you describe the limitations of your work?
[Yes] See Conclusion (Section 6) and
Appendix D
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Background
(Section 2) and Spherizaton Layer (Section 3)
(b) Did you include complete proofs of all theoretical results?
[Yes] See Spherization
(Section 3), Functional Correctness Test on a Toy Task in Experiments (Section 4), and
Appendix A and B
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] See thesupplemental material (.zip file)(b)
Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Experimental Settings in each experiments (Section 4) and
Appendix C
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes] See Tables in each experiments (Section 4)(d)
Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix and the supplemental
material (.zip file)
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cited them byReference.
The detail about the citation is covered at the supplemental material (.zip
file)
(b) Did you mention the license of the assets?
[Yes] See the supplemental material (.zip
file)
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]See the supplemental material (.zip file)
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A] We used open datasets.
When using curated open datasets, we
cited it by Reference
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A] We used open datasets.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A](c)
Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
A Lower bound 𝜙 𝐿
To prevent the last coordinate from an extremely small value, we set a lower bound 𝜑 𝐿 of angles.
When we assume that the trigonometric value of all angular coordinates is 𝛼 , sin 𝜙 = 𝛼 .
Based onthis assumption, the lower bound 𝜑 𝐿 can be obtained as illustrated in Eq.
[Equation]
B Calculation Trick
To convert the angular coordinates 𝛗 ∈ R 𝑛 to Cartesian coordinates s ∈ R 𝑛 + 1 , first, expand thedimension of 𝛗 by using the constant matrix W 𝜑
=
[ I 𝑛 ; v ] ∈ R 𝑛 ×( 𝑛 + 1 ) , where v = [ 0 ; 1 ] ⊺ ∈ R 𝑛 .
Then, we can get the expanded angular vector 𝛟 as illustrated in Eq. 8.
When we define s ′ as illustrated in Eq.
14 from Eq.
8, we calculate the 𝑖 -th coordinate s 𝑖 as shown in
Eq. 15 , where W 𝜙 ∈ R ( 𝑛 + 1 )×( 𝑛 + 1 ) is an upper triangular matrix in which all diagonals are zero and( W 𝜙 ) 𝑛,𝑛 + 1 is also zero, and b 𝜙 = [ 0 ; − 𝜋 2 ] ⊺ ∈ R 𝑛 + 1 .
[Equation]
[Equation]
Finally, we get the Cartesian coordinates s from this calculation trick as illustrated in Eq.
[Equation]
C Experimental Details
Image Classification We conducted image classification experiments on MNIST [4], FashionMNIST [40], CIFAR10 and CIFAR100 [10].
For MNIST, we used a 3-layer neural network [8] and
LeNet-5 [11].
We followed the settings in [8] and in [11], and the architectures of them are illustratedin Table 6 SimpleFNN and LeNet-5, respectively.
Unlike [8], we used 256 mini-batches, the Adamat a learning rate of 0.001 and weight decay of 5e-5 at the 3-layer neural network.
At LeNet-5, we set128 mini-batches, Adam at a learning rate of 0.001, and the other settings are same with [11].
For
Fashion-MNIST, CIFAR10, and CIFAR100, we used VGG variants [28] with batch normalization [9]denoted as VGG- N .
Batch normalization, ReLU, 128 mini-batches, and SGD with momentum of0.9 and with weight decay of 5e-4 were used as default in three datasets.
The number of epochs wasset to 100, 300, and 200, respectively.
For Fashion-MNIST, the learning rate was initially 0.01 andit was divided by 2 at every 20 epochs.
For CIFAR10, we set the initial learning rate as 0.05 andreduced it to half at every 30 epochs.
For CIFAR100, the learning rate was initially 0.1 and dividedby 5 at 60th, 120th, and 160th epochs.
A spherization layer replaced the last fully connected layerof the backbone network for each task.
As pre-activations used for generating angular coordinates,the range of angles was reduced when dropout.
To resolve this problem, we removed the dropout infront of the spherization layer.
For analysis of robustness of the spherization layer to width and depthconfigurations, we used the VGG backbone again and changed the convolutional configuration of iton CIFAR100.
To generate variations on width, we set the number of channels at Conv1.x, Conv2.x,
Conv3.x, and Conv4.x to 16/32/64/128, 32/64/128/256, 64/128/256/512, 128/256/512/1024 and256/512/1024/1024, respectively.
The filter in Conv5.x has same number with the filter in Conv4.x(see Table 6).
To give variations on depth, we used VGG-11, VGG-16, and VGG-19.
[Table]
Visualization CNN-Vis3D is composed of three convolution layer blocks whose kernel size are32, 64, and 128, as shown in Table 6.
To obtain 3-dimensional feature vectors, we set the number ofneurons at the second fully connected layer, which locates before the last fully connected layer, as 3in the original network.
For a fair comparison, we replaced this layer with the spherization layer andchanged the number of neurons to 2, and visualized 2-dimensional feature vectors from the originalnetwork, in which the number of neurons at the second fully connected layer is 2.
In the CNN-Vis3Dfor 2D and 3D, named Original 2D and Original 3D respectively, batch normalization, ReLU, 64mini-batches, cross-entropy, and Adam at a learning rate 0.001 were used as default.
Unlike theoriginal network, we set 32 mini-batches in the spherized CNN-Vis3D, called Spherization 3D. Wetrained original 2D, original 3D, and Spherization 3D for 20 epochs, and they converged and achievedthe training accuracies(%) 98.87, 99.48, and 99.13, respectively.
D Effect of Fine-tuning
In neural networks, the fine-tuning technique is used in natural language processing [25, 39] andcomputer vision [30, 41].
In this method, the pre-trained model is used and re-trained on a newtarget domain.
Fine-tuning can achieve excellent performance.
However, for spherization, a layer inthe architecture should be replaced with the spherization layer, and this spherized network shouldbe trained from scratch.
If not, a mismatch between the learned representations and spherizationunexpectedly affect the performance of the fine-tuned model.
To verify this phenomenon, weinvestigated fine-tuning on few-shot learning and the word analogy test.
We used the pre-trainedweight from [2] for ConvNet and ResNet, and from [39] for BERT and RoBERTa.
For synchronizingwith spherized features, we loaded the pre-trained models and fine-tuned them on Mini-ImageNet [32]and WikiText dataset [22], respectively.
See the experimental details and results on followingsubsections.
[Table] [Table]
D.1 Word Analogy Test
Experimental Settings Unlike the few-shot learning, there are no data to re-train the pre-trainedweights in word analogy test because they use language models to evaluate the performance on wordanalogy test.
To resolve this problem, we used WikiText [22] to make fine-tuned language models.
We loaded pre-trained weights BERT and RoBERTa, respectively, and trained them on WikiText for3 epochs, 8 mini-batches, softmax function following cross-entropy, SGD at a learning rate 10 − 4 .
Forspherization, we replaced the last output layer of encoder in each network with spherization layer.
Result and Analysis
The results of fine-tuning on word analogy test are illustrated in Table 7.
In the original networks, the performances of both are obviously increased.
The spherized BERTalso have improvements but not much as the original network, while the spherized RoBERTa showsgreater improvement.
This result means that the angular information in the pre-trained representationsmight be more helpful enough to overcome the information collapse.
D.2 Few-shot Learning
Experimental Settings
The experimental settings were used according to the method presentedin Downstream Task:
Few-shot Learning in Experiments Section.
For fine-tuning, we obtained thepre-trained weights [2] for ConvNet and ResNet, respectively.
When loading these weights, the partof last fully connected layer was substituted with the spherization layer.
Thus, the pre-trained weightsof this layer were not necessary anymore.
Result and Analysis
The results of fine-tuning on few-shot learning are displayed in Table 8.
In the original networks, the performances increase considerably.
The spherized networks exhibitimprovements but not as much as the original network.
This result indicates that there are someinformation collapse when training the spherization layer on pre-trained representations but it is notthat fine-tuning is not helpful at all.
To the best of our knowledge, the angular information in thepre-trained representations might be helpful to train spherized representations.
E Analysis on the Effect of Projection
[Table]
Environmental Settings We compared the method with another angle-based approach to analyze [Table] the effect of reducing information loss by projection.
For comparison, we used 9-layer CNN, namely
CNN-9.
To apply a spherization layer, we changed the last hidden layer FC-10 or the second hiddenlayer FC-256, which is located at the previous of the last hidden layer, in each model.
Results and Analysis After training models, we extracted representations from the target layerand analyzed the effect of projection.
First, we count the number of representations overlapping atleast one another representation.
Then, we get the total errors caused by the incorrectly classified andoverlapping representations.
Finally, we calculate the ratio of the incorrectly classified representationsto the overlapping representations.
The results of them are shown in Table 9: # ovlp.
, # err. , and(ratio) , respectively.
As the result, large proportion of representations suffer the overlap problemfrom projection approach, and most significant errors are caused by the overlapping representations.
Furthermore, representation learning relying on bias parameters (direct projection case) causes moresignificant errors than no-bias layer (indirection projection case) because the representations aremore difficult to be distinguished than those of no-bias layer.
In comparison, the spherization layerremoves the errors and decreases the upperbound of errors measured by the number of overlappingrepresentations. F Gradient Flow
[Figure]
[Figure]