Spherization Layer: Representation Using Only Angles
Abstract
In neural network literature, angular similarity between feature vectors is frequently used for interpreting or re-using learned representations.
However, the inner product in neural networks partially disperses information over the scales and angles of the involved input vectors and weight vectors.
Therefore, when using only angular similarity on representations trained with the inner product, information loss occurs in downstream methods, which limits their performance.
In this paper, we proposed the spherization layer to represent all information on angular similarity.
The layer 1) maps the pre-activations of input vectors into the specific range of angles, 2) converts the angular coordinates of the vectors to Cartesian coordinates with an additional dimension, and 3) trains decision boundaries from hyperplanes, without bias parameters, passing through the origin.
This approach guarantees that representation learning always occurs on the hyperspherical surface without the loss of any information unlike other projection-based methods.
Furthermore, this method can be applied to any network by replacing an existing layer.
We validate the functional correctness of the proposed method in a toy task, retention ability in well-known image classification tasks, and effectiveness in word analogy test and few-shot learning.
Code is publicly available at https://github.com/GIST-IRR/spherization_layer
Introduction
The inner product is a key element constituting layers in deep neural networks with a nonlinear activation function.
The inner product with the Euclidean norms and the angle, that is, ∥ w 𝑖 ∥∥ x 𝑗 ∥ cos θ 𝑖𝑗 , has been analyzed in terms of the norms ∥ w 𝑖 ∥∥ x 𝑗 ∥ and the angle cos θ 𝑖𝑗 , independently [11, 12, 31].
All factors of the inner product learn distinct information.
Therefore, using only one factor results in information loss when re-using or understanding the information in downstream tasks.
This problem is termed as dispersion problem .
The angular similarity between features is frequently used.
However, this technique causes the dispersion problem in advanced methods in neural network literature such as decoupled network [12], representation learning [5, 6, 20, 21, 28], regularization [29, 30], zero-shot learning [22], and generative model [3, 23].
To mitigate the dispersion problem, numerous angle-based learning approaches [1, 9, 10, 13, 15, 14, 29, 30, 32] have been proposed.
However, these studies are based on projection onto the hyperspherical surface.
In this projection method, distinction by the scale of features is ignored.
Therefore, information loss occurs.
We proposed the spherization layer as an explicit solution for the dispersion to completely eliminate the interference of the norms in training without drawbacks.
This layer is used to locate all represen-
The spherization layer consists of three main components: Angularization function that converts the pre-activations from the previous hidden layer to angles; Conversion from spherical coordinates to
Cartesian coordinates located on the hyperspherical surface; No-bias layer , a hidden layer without bias parameters, that determines decision boundaries by using only the angles.
Figure 1 illustrates the design of the spherization layer.
Through spherization , representations are located on the hyperspherical surface and the effect of the norms in representation learning is completely eliminated.
Thus, neural networks are enforced to express all representations differently by using only the angles.
We experimentally verified the functional correctness of the spherization layer in a toy task and its applicability to feedforward and convolutional neural networks by evaluating performance on image classification tasks.
The results reveal that the training ability of original networks is preserved after applying the spherization layer.
Furthermore, we analyzed the sensitivity to width and depth, the effect of projection in the spherization layer, and the influence of the spherization layer on the gradient flows in training.
Finally, we investigated the effect of the proposed method in downstream tasks through visualization, word analogy test, and few-shot learning.
In summary, our contributions are three-fold:
• To address the dispersion problem, we propose the spherization layer to represent all feature vectors on the hyperspherical surface and learn the representations with only the angles.
• We validate the wide-applicability and scalability of the spherization layer without any loss of performance through experiments on various well-known networks.
• We empirically show that the spherization layer can be used in many applications in which angular similarity is a critical metric.
Background
Conversion Spherical to the Cartesian Coordinate System
In most neural networks, all input samples on the n -dimensional space are represented as Cartesian coordinates, in which i -th column value denotes the distance from the origin along the i -th axis, and neural networks train them by using neurons in which the inner product between the input and weight vector occurred.
The pre-activations from neurons are used to determine whether neurons should be activated by using the following activation function.
In this process, the pre-activations are calculated by using weight and bias parameters, represented as Cartesian coordinates.
These Cartesian coordinates can be converted from spherical coordinates.
Given a vector s represented as Cartesian coordinates on the n -dimensional space, s can be defined as spherical coordinates 𝛉
=
[ 𝑟, 𝛗 ] , composed of a radial coordinate r and n -1 angular coordinates 𝛗 =
[ φ 1 , φ 2 , ..., φ 𝑛 − 1 ] .
In this case, the k -th axis of s can be computed from 𝛉 with Eq.
[Equation]
, where 𝑠 1 <𝑘<𝑛 = 𝑟 cos φ 𝑘 (cid:206)
𝑘 − 1 𝑖 = 1 sin φ 𝑖 .
Generally, the spherical coordinate system is a 3-dimensional version of the polar coordinate system.
However, the spherical coordinate system in the followings indicates all n -dimensional versions of the polar coordinate system, where n is greater than or equal to 2.
Spherization Layer
We proposed the spherization layer as shown in Figure 1 to locate feature vectors on the hyperspherical surface without any loss of information and learn hyperplanes by using only the angles.
In this method, a layer of an original network is selected to learn representations by angular similarity.
Next, the layer is replaced to a spherization layer through three sequential stages: Angularization ,
Conversion to Cartesian , and No-bias training notated as 𝑓 , 𝑔 , and ℎ functions, respectively.
The final form of spherization layer 𝑓 𝑠𝑝ℎ is expressed as Eq. 2.
[Equation]
After training with the layer, we can obtain the representations on the ( 𝑛 + 1 ) -spherical surface, namely spherized representations, as the outputs of ( 𝑔 ◦ 𝑓 ) operations.
The common goal of all stages is to preserve the training ability of the original network while the spherization layer trains all information on the ( 𝑛 + 1 ) -spherical surface.
The angularization locates all pre-activations on the safe spherical surface.
The conversion to Cartesian results in the generation of compatible representations to the ordinary layer.
No-bias training enforces training by only the angles.
In the followings, we elaborate each stages more detail and only annotate the ( 𝑙 − 2 ) th and ( 𝑙 ) th layer as [ 𝑙 − 2 ] and [ 𝑙 ] , respectively, for the simplicity.
3.1 Angularization
[Equation]
Angularization is the stage to map a pre-activation vector z , passed from ( 𝑙 − 1 ) th layer, to angular coordinates 𝛗 .
The 𝑛 indicates the dimension of the pre-activation vector.
The role of this stage is to configure the shape of the mapped region on the ( 𝑛 + 1 ) -spherical surface for resolving training and computational difficulty.
Angularization 𝑓 is implemented by applying the following element-wise function to all coordinates of z as an activation function, and 𝑓 is illustrated as Eq. 4.
[Equation]
, where the terms and form are used by three following motivations.
Converting Pre-Activation to the Angular Coordinate The first step is to convert the input vector into angular coordinates.
To ensure the conversion as bijective mapping, we restrict the range of the function as [ 0 , 𝜋 2 ] .
To allow unrestricted input representations on the real-valued domain, the sigmoid function 𝜎 (·) is used with weight 𝜋 2 for the range setting.
As the sigmoid function used, the input vector should be pre-activations, not activations because when activations from ReLU or another sigmoid are passed to the angularization, inefficient use of the spherical surface or gradient amplification, respectively, may occur.
After converting pre-activations to angular coordinates, the representations on the hyperspherical surface in the same range were located by setting a consistent radius over all inputs.
This radius scale is controlled in the conversion-to-Cartesian stage.
Tailoring Angular Representation Space
In the conversion from angular to Cartesian coordinates, the last coordinate can be an extremely small value because trigonometric values in [0, 1] are multiplied many times.
This scale descent can map all values in the axis to only a single value by the limit of the floating point data type.
To reduce this effect in the angularization, we introduce a lower bound 𝜑 𝐿 of angles to guarantee distinguishable values in its corresponding converted Cartesian coordinates as the following equation (Eq. 5):
[Equation]
, where 𝛿 is a minimal trigonometric value to guarantee the distinguishable representations.
The details are presented in Appendix A. We set 𝛿 to the empirically obtained proper value 10 − 6 , for all experiments.
Scaling Pre-Activations In angularization, the activations are concentrated onto the small region because of the lower bound.
This concentration renders training difficult with the decrease in the variance.
To reduce the effect, we set a learnable parameter 𝛼 as a weight of z , which controls the variance of z .
Using this scale factor, the generated angular representations become abundant.
3.2 Conversion-to-Cartesian
[Equation]
In the Conversion-to-Cartesian stage, the angular coordinates of the previous stage are converted to Cartesian coordinates on the ( 𝑛 + 1 ) -spherical surface.
This conversion ensures the consistency between the output of angularization (polar coordinate system) and the input of the following no-bias layer (Cartesian coordinate system), and enables the layer to be trained in the same way as general neural networks.
Furthermore, an additional dimension makes the spherization layer have enough capacity to be compatible with the ordinary layer.
This implementation is based on Eq.
1 with the modified range of angles as the following equation (Eq. 7).
[Equation]
Calculation Trick Implementation of Eq.
7 as a tensor operation requires the trick defined in the following equation (Eq.
[Equation]
, where 𝛟 is a dimension-expended vector in R 𝑛 + 1 and 𝜙 𝑛 + 1 = 𝜙 𝑛 , and 𝑟 is a constant to control radius.
Here, W 𝜑 , W 𝜙 , and b 𝜙 are constant matrices and vector in R 𝑛 × ( 𝑛 + 1 ) , R ( 𝑛 + 1 ) × ( 𝑛 + 1 ) , and R ( 𝑛 + 1 ) , respectively.
W 𝜑 =
[ I 𝑛 ; v ] , where v = [ 0 ; 1 ] ⊺ ∈ R 𝑛 , is used for matching the dimension between 𝛗 and s .
The other constants are used for calculating logarithm trigonometric values from expanded angular coordinates in the way of matrix multiplication, where W 𝜙 is an upper triangular matrix in which all diagonals are zero and ( W 𝜙 ) 𝑛,𝑛 + 1 is also zero, and b 𝜙 = [ 0 ; − 𝜋 2 ] ⊺ .
See Appendix B for detail hyperparameters and process.
3.3 No-bias Training
[Equation]
Through angularization and conversion-to-Cartesian, all representations are located on the ( 𝑛 + 1 )- spherical surface.
In this case, any update to the representations is determined by the change of angular similarity.
However, hyperplanes on the ( 𝑛 + 1 ) -dimensional space from the next layer may not use only the angular similarity, which may assign semantic information to the Euclidean norm of the spherized representation.
To synchronize these parameters, we used no-bias training using only the weight parameters W [ 𝑙 ] as illustrated in Eq. 10.
[Equation]
Effect of No-Bias on Training In the ordinary layer, the problem of no-bias is that hyperplanes passing through the origin cannot be shifted to another parallel hyperplanes.
However, the problem disappears when all feature vectors are located on the ( 𝑛 + 1 ) -spherical surface because the decision boundary can be shifted by only the angle changes of ( 𝑛 + 1 )
-dimensional hyperplanes even though they pass through the origin.
3.4 Optimization with Overall Process
A training loss 𝐿 is calculated in the same manner as the original network.
The gradient of 𝐿 for the spherization layer is calculated by multiplying the following partial derivative 𝜕𝐿𝜕 W to the original backpropagation step from the ( 𝑙 − 2 ) th to ( 𝑙 ) th layer, as illustrated in Eq.
The first term 𝜕𝐿𝜕 z [ 𝑙 ] is calculated by the subsequent layers in the same way of the original network.
[Equation]
Experiments
First, we define two terms for simplicity on indicating two networks: one is the network before applying the spherization layer, called original network , and the other is the network after substituting an ordinary layer with the spherization layer, called spherized network .
In the followings, we used these two terms consistently.
All experiments were performed five times with random seeds and their training and test accuracy were evaluated except word analogy test and few-shot learning.
The mean 𝜇 and standard deviation 𝜎 of accuracy are represented as 𝜇 ± 𝜎 in each table.
4.1 Functional Correctness Test on a Toy Task
task
Implementation Details We verified the spherization layer for learning decision boundaries on a simple binary classification task.
We set up the simple binary classification task: given ( x 𝑖 , 𝑦 𝑖 ) pairs, where x 𝑖 is the i -th input sample in R 2 and 𝑦 𝑖 ∈ { 0 , 1 } is the label of x 𝑖 , we randomly generated 100 input samples located around (0, 0) for the label 0, and the other 100 samples for the label 1 around (1, 1), as shown in Figure 2.
We set a 2-layer neural network as the original network, and trained it with the softmax function, cross-entropy, and SGD at a learning rate of 0.01.
We applied the proposed method to the original network by replacing the last fully connected layer with the spherization layer.
For comparison of representations in the same dimensional space, we set the dimension of the spherization layer to 1, where it is 2 in the original network.
The other settings are identical to the original network.
We trained both networks on the input samples for 100 epochs with 16 mini-batches, where the networks converged and achieved 100% training accuracy.
Distribution of Representations Initially, all representations are randomly distributed into two groups on a 2-dimensional space.
After convergence, the feature vectors are divided into two disjointed groups, as illustrated in Figure 3.
This means the spherized network locates all representations on the 2-spherical surface in both the initial and final epoch, whereas the original network spreads them out.
[Table]
[Table]
Hyperplanes Hyperplanes and decision boundary in the feature space are defined as follows with the parameters of the output layer:
[Equation]
, where W 𝑖 is the hyperplane determined by weight parameter w 𝑖 and bias b 𝑖 , and D 𝑖𝑗 is a linear decision boundary whose points satisfy 0.5 confidence for both labels.
In the spherized network, all bias terms are eliminated.
In Figure 3, the hyperplanes are illustrated as yellow lines and the decision boundary as a gray line.
In the spherized network, hyperplanes and decision boundary pass through the origin from the initial to the last epoch, unlike the original network.
The results imply that the spherization layer can learn the correct decision boundary by changing only the angles of hyperplanes passing through the origin.
4.2 Retention of Training Ability on Image Classification Benchmarks
Implementation Details
In this task, we empirically verified that replacing an existing layer to a spherization layer still maintains the training ability of the original networks on well-known image classification tasks.
We reproduced all networks and their performance on the image classification tasks with each dataset.
Then, we validated our proposed method on the same settings with them, where only the last fully connected layer replaced by the spherization layer.
See Appendix C for the detail about networks and datasets.
[Table]
Results and Analysis
The accuracy results of original and spherized networks are compared in
Table 1.
The reference and reproduced results of each setting are similar.
The accuracy results of the spherized networks implemented on the reproduced code are similar or slightly higher on both training and test data than those of the original networks.
These results imply that the spherization layer maintains the training ability of the original network.
In Table 2, the accuracy results of original and spherized networks are shown in various width and depth settings.
The spherized network consistently exhibits similar or higher test accuracy over all width and depth settings.
The results reveal that the spherization layer again preserves the training ability of the original network.
4.3 Analysis: Effect of Projection
Implementation Details We compared the method with another angle-based approach to analyze the effect of reducing information loss by projection.
For comparison, we used 9-layer CNN, namely CNN-9, with the same experimental setup of [16] but reproduced in PyTorch.
To apply a spherization layer, we changed the last fully connected layer in each model.
We considered samples overlapped when their cosine similarity is greater than or equal to ( 1 − 10 − 6 ) .
Results and Analysis After training, we extracted representations from the target layer and analyzed the effect of projection.
First, we count the number of representations overlapping at least one another representation.
Then, we get the total errors caused by the incorrectly classified and overlapping representations.
Finally, we calculate the ratio of the incorrectly classified representations to the overlapping representations.
The results of them are shown in Table 3: # ovlp.
, # err. , and (ratio) , respectively.
As the result, large proportion of representations suffer the overlap problem from projection approach, and most significant errors are caused by the overlapping representations.
Furthermore, representation learning relying on bias parameters (direct projection case) causes more significant errors than no-bias layer (indirection projection case) because the representations are more difficult to be distinguished than those of no-bias layer.
In comparison, the spherization layer removes the errors and decreases the upperbound of errors measured by the number of overlapping representations.
4.4 Analysis: Gradient Flows
Implementation Details
The replacement of a hidden layer and the operations in the spherization layer, such as angularization function and conversion-to-Cartesian, might make the gradient flow unstable in the original networks.
To empirically verify how the spherization layer affects to the gradients during training, we qualitatively compare the gradient flows of original and spherized
VGG-11.
They were trained on the image classification with CIFAR100.
We used the same setting with Section 4.2.
Results and Analysis As shown in Figure 4, the average of absolute gradients in the original and spherized network are not very different during the whole training.
Furthermore, the results show similar flows not only at the last fully connected layer ( fc3 or sph_fc3 ) but at the previous layers.
This result implies the spherization layer does not destroy the gradient flows.
[Figure] [Figure]
4.5 Downstream Tasks: Visualization
Implementation Details To investigate the spherization layer locates feature vectors on the ( 𝑛 + 1 ) -spherical surface we implemented a simple CNN to learn 3-dimensional feature vectors for visualization.
This technique is called CNN-Vis3D which generates 3-dimensional feature vectors and shows those representations as a graph.
Detailed configurations are presented in Appendix C.
In the spherized network the second fully connected layer was replaced by the spherization layer, which is located at the previous of the last fully connected layer.
After training, the pre-activations before the no-bias layer were used to visualization.
Results and Analysis
The results of visualization are illustrated in Figure 5.
In the 2D and 3D feature visualization results of the original network, the representations are distributed over wide ranges of both scale and angles, as shown in Figure 5a and 5b.
As illustrated in decoupled network [12], the angle accounts for semantic difference and the Euclidean norm accounts for intraclass variation.
In the spherized network, all representations are placed on the 3-spherical surface, as shown in Figure 5c.
Based on these spherized representations and the following no-bias layer, all the roles are expressed by the angle change on the hyperspherical surface.
Thus, in the spherization layer, all trained information can be used in angular similarity-based interpretation.
[Table] [Table]
4.6 Downstream Tasks: Word Analogy Test
Implementation Details We used BERT [4] and RoBERTa [17] to conduct the word analogy test, in which angular similarity is used to predict a relation type between words.
The settings were identical with [25].
We applied the spherization layer to the last fully connected layer of the encoder in each model.
Next, we trained them on WikiText [19] for 3 epochs with 8 mini-batches, the softmax function following cross-entropy, SGD at a learning rate 0.0001 on masked-language modeling.
Results and Analysis Table 4 presents the performance evaluated with three metrics [25].
Applying the spherization layer to BERT and RoBERTa improves most average scores.
The improvement implies that the spherized representations provide accurate information for angle-based distinction of word relations.
4.7 Downstream Tasks: Few-shot Learning
Implementation Details We used ProtoNet [24] with ConvNet and ResNet for few-shot learning on Mini-ImageNet [26], in which several feature vectors are compared with the feature vector of an input image by using distance metric.
This task was performed according to the guidelines in [2].
Results and Analysis Table 5 details the performance results in few-shot learning [2].
Generally, the Euclidean distance is used to discriminate feature vectors.
However, the Eucliean distance also has the dispersion problem.
To focus on only the angles, we trained the models with cosine similarity and compared the performances.
As shown in Table 5, all spherized models with cosine similarity outperform the other models.
This improvement indicates the spherized representations are useful for the angle-based metric such as cosine similarity.
Related Works
Semantic Analysis on the Inner Product The inner product is a crucial operator in current neural networks, in which the distance between input vector x and weight vector w is encoded.
At the decoupled networks [12], the inner product is reparametrized with the norms and the angle, and the intra-class variation and the semantic difference are modeled in neural networks by decoupling them.
Furthermore, the substitutes of the inner product have been proposed, where the direction of gradient or the similarity between kernels are used as the key factor instead of the inner product [11, 31].
The spherization layer can be considered to be the substitute of the inner product, which normalizes the input vectors by locating them on the hyperspherical surface.
However, the spherization layer is a direct and specific method to convert feature vectors focused on the angles without information loss.
Angle-based Approach The semantic analysis on the inner product has revealed that the streams focuses on the information in the angles.
The angle is a crucial factor, in which the most abundant and discriminative information is preserved [1, 9, 10, 15].
In SphereFace variants [10, 13, 14, 32], the angular softmax that enables CNNs to learn discriminative features on angular separability was used.
Furthermore, some of this angular information have been used for regularization [29, 30].
In most angle-based studies, the angular information was used indirectly by the objective function or regularization.
In contrast, the spherization layer ensures the model directly learns the angular information on the hyperspherical surface.
Hyperspherical Representation Learning In some angle-based approaches, input vectors were directly projected onto the hyperphserical surface [12, 16, 18].
These hyperspherical representation learning methods normalized the input vectors to ensure models are dependent on only the angles.
However, this normalization is the projection onto the hyperspherical surface, and it can be less discriminative when some points overlap after the projection.
The spherization layer locates the input vectors on the hyperspherical surface without the overlap problem through the spherization.
Conclusion
We introduced the dispersion problem of trained information to the Euclidean norm and angle on representations.
To address the dispersion problem, we proposed the spherization layer to learn representations by using only the angles without information loss.
We used the angularization for using pre-activations as angular coordinates, conversion-to-Cartesian for locating them on the ( 𝑛 + 1 ) -spherical surface, and no-bias training to learn representations by using only the angles.
In the experiments on toy, image classification benchmarks, few-shot learning, and word analogy test, the proposed method achieved accurate learning of the decision boundary and retention of the original training ability, and improved performance in downstream tasks using angle-based information reused or interpreted.
The proposed method can be applied to numerous network layers and downstream applications.
A limit of this approach is that the spherization layer should be inserted to networks in a training step to fully utilize its advantage, which restricts the use of pre-trained models trained without it.
Recovering trained information on representations with sampling should be investigated in the future.
Acknowledgments and Disclosure of Funding
This work was partially supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (2022R1A2C2012054), and by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0-01842, Artificial Intelligence Graduate School Program (GIST))
[-For Fashion-MNIST, the learning rate was initially 0.
-]