A
G RADIENT -B ASED
S ELF -O RGANIZING
N EURAL
N ETWORK
Anonymous authors
Paper under double-blind review
A BSTRACT
Designing the architecture of deep neural networks (DNNs) is a cumbersome task that requires signiﬁcant human expertise.
There have been attempts to automatize this task, mainly by considering DNN architecture parameters such as the number of layers, the number of neurons and the activation function of each layer as hyper-parameters, and using an external method for optimizing them.
This framework is computationally too expensive since each hyper-parameter optimization iteration required a whole model training.
Here we propose Farfalle Neural Networks, a novel recurrent structure in which neurons connect each other through their trainable embeddings.
Hence, important architecture features such as the number of neurons in each layer and the wiring among the neurons are automatically learned during the training process.
We theoretically prove that the proposed model can replace a stack of dense layers, which is used as a part of many DNN architectures.
By comparing the performance of dense networks and FNNs we show FNNs achieve comparable or higher accuracy using signiﬁcantly fewer parameters.
I NTRODUCTION
During the last few years, deep neural networks (DNNs) have played an incomparable role in addressing challenging problems of different areas, most impressively computer vision (Simonyan &
Zisserman (2014); He et al.
However, well-designing the architecture of a DNN requires a lot of expertise and is usually a cumbersome and time-consuming task, since many possible conﬁgurations are assessed before the best one is found.
Additionally, the performance of different architectures heavily relies on the nature of the task and the data.
There are several studies (Zagoruyko & Komodakis (2016); Szegedy et al.
providing insight on how to tune models with state of the art performance for different tasks.
Moreover, many studies (Zoph & Le (2016);
Cai et al.
Rohekar et al.
; Zhong et al.
(2018)) have been accomplished recently to avoid architecture engineering.
These methods are based on network architecture search over the space of possible architectures (Elsken et al.
However, these approaches control the structure externally, hence adding a computational overhead when training the model.
On the other hand, some studies (Li et al.
; Han et al.
(2015)) show that it is possible to reduce the number of network parameters and still achieve comparable performance.
These smaller networks have the beneﬁt of requiring less storage space and less computational power during inference.
These ﬁndings also show that there is still much room for improvement in designing efﬁcient neural network architectures.
In this paper, we propose a new neural network model, called Farfalle Neural Network (FNN) in which the trainable parameters are not the weights on the connections between neurons.
Instead, the network learns embedding vectors for neurons and uses these vectors to determine the weights of neural connections.
More importantly, in the proposed method, instead of hand-crafting the network architecture, it is learned during the training process.
The connections between neurons are indirectly speciﬁed according to neuron embeddings.
Therefore, the proposed network conﬁgures its structure itself during the training process given solely the number of nodes and an upper bound on the network depth.
We also establish the effectiveness of our models through various experiments.
In particular, we show that our model is able to replace fully connected networks achieving higher performance with a 70% reduction in the number of parameters.
R ELATED W ORK
Recent proposed methods for automatic design of neural networks are commonly focused on treating the architecture decisions as hyper-parameters.
These methods use either a supervised or an unsupervised approach to optimize these hyper-parameters.
There are several approaches (Zoph &
Le (2016); Baker et al.
; Zhong et al.
that utilize reinforcement learning for effectively searching the design space.
These approaches usually have a lot of computational overhead because of their need to compute the model’s accuracy during the search in the space of architectures
Elsken et al.
To circumvent this issue, Smithson et al.
(2016) uses another neural network to estimate the trained model’s accuracy.
However, training the estimator network is itself a computationally expensive task.
Rohekar et al.
(2018) propose a lightweight unsupervised approach using
Bayesian network structure learning.
Using this approach, they replace fully connected layers at the end of known networks such as VGGNet with smaller models while still showing a comparable performance in accuracy.
Note that while the reported results show the effectiveness of this method, it still optimizes the structure externally.
Hence, it requires an additional environment setup and have an external overhead in the learning process, though the latter is reported to be reasonably small.
One of the key ideas in our proposed method is to assign embedding vectors to neurons of the network and use the attention mechanism to relate them.
Similar ideas appear in the Transformer network (Vaswani et al.
and CapsNet (Sabour et al.
In Transformer networks (Vaswani et al.
(2017)) input words and their positional information are embedded in a low-dimensional space.
However, they utilized a speciﬁc case of attention called self-attention to relate different parts of the sequence.
In addition, the embeddings used in that architecture are not trained for the purpose of structure learning.
CapsNet (Sabour et al.
(2017)) considers an output vector for each capsule and routes the outputs from one capsule to the next layer’s capsules according to its ability to predict the output vector of those capsules.
However, CapsNets still use weight matrices between neurons and also are not able to self conﬁgure their structure.
F ARFALLE N EURAL N ETWORKS
In traditional neural networks, the number of neurons in each layer and the arrangement of the neurons is ﬁxed.
This rigid conﬁguration prevents straightforward optimization of the network structure during the training process.
Therefore, ﬁnding a network with proper structure requires testing a lot of conﬁgurations.
In contrast, FNNs utilize a new type of neurons which can ﬂoat and ﬁnd the most suitable neurons to obtain information from them.
The connections between these ﬂoating neurons are deﬁned based on their relative similarities.
Thus, during the training process, relevant neurons move toward each other to strengthen their connection.
Figure 1 shows how these neurons ﬂoat to obtain more relevant information.
3. F LOATING N EURONS
A ﬂoating neuron gathers information from relevant neurons at its input, transforms it with a trainable transformation, and emits the result at its output.
In order to avoid confusion, we might refer to a neuron’s input as its head.
Similarly, we sometimes refer to its output as the neuron’s tail.
Inputs and outputs of these neurons are embedded in a d -dimensional space.
These embedding vectors regulate the weights connecting relevant neurons and are updated during the training process.
Figure 2 shows a schematic representation of a ﬂoating neuron and its connections.
Speciﬁcally, a ﬂoating neuron v consists of three parts:
• Input embedding: A trainable d -dimensional vector I v which indicates the coordinates of the neuron’s head.
The similarity between this vector and output embedding of other neurons determines the connection weights between this neuron and other neurons using the attention mechanism.
• Transformation function: A trainable nonlinear function F v that transforms the gathered information.
The transformation used in this study is of the form F v ( x ) =
ReLU ( a v · x + b v ) where a v and b v are neuron-speciﬁc trainable parameters.
• Output embedding: A trainable d -dimensional vector O v which indicates the coordinates of the neuron’s tail.
In addition to normal ﬂoating neurons, there are two custom types of ﬂoating neurons: Input neurons, which receive the input of the whole network and output neurons, which provide the processed data to the outside.
Consequently, input neurons do not have input embedding and output neurons do not have output embedding.
3. C ONSTRUCTION OF M ULTI -L AYER F LOATING N EURAL N ETWORKS
Before introducing FNNs, we discuss how to employ ﬂoating neurons in a layered structure.
In order to form such a network, ﬂoating neurons are grouped in layers.
The neurons at each layer obtain their values from neurons at the previous layer.
The connections between neurons of two consecutive layers are deﬁned based on the attention mechanism.
Formally, suppose v is a neuron in the layer i + 1 and u 1 , u 2 , . .
, u M are neurons of the previous layer, i.e. layer i .
The weights connecting v to related neurons is deﬁned by
[Equation]
where I v is the input embedding of neuron v , O u
i is the output embedding of neuron u i , and N is a normalization function.
For normalization, one can choose softmax function to force each neuron’s input to be a convex combination of the outputs of the neurons in the previous layer.
We found l 2 normalization to work best in our experiments and thus the following function is used for normalization
[Equation]
Utilizing these weights, given y 1 , y 2 , . .
, y M as the values of neurons u 1 , u 2 , . . .
, u M , the output value of neuron v will be F v ( (cid:80) m
w i y i ) where F v is the transformation function of neuron v .
To efﬁciently connect neurons v 1 , v 2 , . .
. , v N of the layer i + 1 to neurons u 1 , u 2 , . . .
, u M of the layer i , let I =
[ I v 1 |
I v 2 | · · · | I v N ] be the concatenation of input embedding vectors of the layer i +
and
O =
[ O u 1 | O u 2 | . . .
| O u M ] be the concatenation of output embedding vectors of the layer i .
Then, the weights connecting neurons of these two layers are determined by
[Equation]
where (cid:101) N is a function normalizing each row of its input matrix according to the normalization function N .
Finally, the output of layer i + 1 will be
[Equation]
where Y is the vector containing output values of layer i and (cid:101)
F i = F v i .
Using this model, the number of parameters required to connect a layer of size M to a layer of size N is of the order O
( d ( M + N )) .
In contrast, in dense models the number of parameters for connecting two such layers would be of the order O ( MN ) .
Hence, by using this model one of the limitations in designing neural networks, that is the huge number of parameters in the weight matrices, is resolved.
3. C ONSTRUCTION OF F ARFALLE N EURAL N ETWORKS
An FNN is a recurrent network of ﬂoating neurons.
Hence neurons in this architecture are used in iterations.
In each iteration, ﬂoating neurons receive the output of the previous iteration along with the input.
Figure 3 represents the architecture of FNNs.
The expanded architecture is available in
Supplementary Figure 7.
This recurrent structure allows neurons to process high-level information along with low-level features.
Also, the ﬂoating nature of these neurons allows the network to balance the number of neurons employed in different levels of abstraction.
Supplementary Figure 8 illustrates how this recurrent structure allows the network to combine information from different levels of abstraction.
This ﬂexibility allows the architecture to evolve during the training.
Before describing the data ﬂow of FNNs, let’s deﬁne an auxiliary function.
For two groups of neurons V and U , equation (3) deﬁnes the weights connecting neurons in V to neurons in U .
In this equation O
and I are the concatenation of output embeddings of neurons in U and the concatenation of input embeddings of neurons in V respectively.
Given Y as the values of neurons in U , equation (4) describes the output values of neurons in V .
Combining these equations results in
[Equation]
where (cid:101) N is a function normalizing each row of its input matrix and (cid:101) F transforms values of each ﬂoating neuron using its own transformation function.
The function Step can be seen as the combination of attention and neuronal transformation.
Constructing an FNN for the input size of R and output size of S needs R input ﬂoating neurons for feeding data to the network, N ﬂoating neurons for processing data in k iterations, and S output ﬂoating neurons for the ﬁnal deduction from hidden neurons.
Here, N and k are the hyperparameters of the network.
Let’s call the input neurons
I = i 1 , i 2 , . . .
, i R , the hidden neurons
V = v 1 , v 2 , . .
. , v N , and the output neurons O =
o 1 , o 2 , . . .
, o S .
The following procedure describes the ﬂow of the network:
Each input neuron assigns an embedding vector to its input variable after applying its transformation function,
i
e. given input x = ( x 1 , x 2 , . . .
, x R ) , input neurons will provide values Y
=
F i 1 ( x 1 ) , F i 2 ( x 2 ) , . . .
, F i R ( x R ) at locations O i 1 , O i 2 , . . .
, O i R .
In the ﬁrst step of the iterative part, hidden neurons process the data provided by input neurons.
Thus, the resulting values of this step is Y 1 =
Step ( V , I , Y 0 ) .
In iteration 1 < j ≤ k , each hidden neuron process Y 0 along with the outputs of all hidden neurons in the previous iteration ( Y j − 1 ) and produce Y j .
Indeed, given Y 0 and Y j − 1 hidden ﬂoating neurons will provide Y j
= Step ( V , [ I|V ] , [ Y 0 , Y j − 1 ]) .
Finally, utilizing output neurons the ﬁnal output will be Step ( O , V , Y k ) .
The following theorem shows how an FNN with its recurrent structure can model a multi-layer ﬂoating neural network.
Theorem 1.
Every multi-layer ﬂoating neural network with l layers, total of N ﬂoating neurons, and embedding dimension d can be modeled by an FNN containing N + 1 ﬂoating neurons with embedding dimension of l · d which iterates for l iterations.
Proof.
Suppose a multi-layer ﬂoating neural network N 1 with layer sizes of n 1 , n 2 , . .
. , n l is given.
Let i 1 , i 2 , . .
, i R be the input ﬂoating neurons, v j 1 , v j 2 , . .
, v jn j be the ﬂoating neurons in layer j , and o 1 , o 2 , . . .
, o S be the output ﬂoating neurons of the network.
To construct an FNN N 2 with the same functionality, deﬁne a network with input neurons i (cid:48) 1 , i (cid:48) 2 , . .
, i (cid:48) R , output neurons o (cid:48) 1 , o (cid:48) 2 , . . .
, o (cid:48) S , and hidden neurons v (cid:48) 0 , v (cid:48) 11 , v (cid:48) 12 , . . .
, v (cid:48)
ln l such that v (cid:48)
ji correspond to hidden neuron v ji of N 1 .
Let all ﬂoating neurons in N 2 have the same transformation function as their corresponding neuron in N 1 and v (cid:48)
be a neuron with F v (cid:48)
= 0 · x and
I v (cid:48)
= O v (cid:48)
= ( (cid:15), . . .
, (cid:15) ) where (cid:15) is a negligible values.
Before deﬁning neural embeddings of N 2 , for 0 < j ≤ l deﬁne
[Equation]
(cid:123)
[Equation] [Equation]
( l
[Equation]
(cid:122)
(cid:125)(cid:124) (cid:123)
[Equation]
This implies that
[Equation]
Now, deﬁne all remaining embedding vectors of N 2 by
[Equation]
These deﬁnitions along Eq.
(6) result that in the ﬁrst iteration, the only neurons which have a non-zero connection with input neurons are v (cid:48) 11 , v (cid:48) 12 , . .
, v (cid:48)
1 n 1 .
In addition, the weights of these connections are the same as the connections in the ﬁrst layer of N 1 .
Similarly, by mathematical induction over j we can state that ﬂoating neurons v (cid:48)
j 1 , v (cid:48)
j 2 , . .
, v (cid:48)
jn
j can only connect to v (cid:48)
j − 1, v (cid:48)
j − 1, . .
, v (cid:48)
j − 1 n j − 1 , the connection weights are the same as those in layer j of N 1 , and iteration j is the ﬁrst time v (cid:48)
j 1 , v (cid:48)
j 2 , . .
, v (cid:48)
jn j can get nonzero values.
This implies that the connections of hidden ﬂoating neurons of N 2 in l iterations, form a chain similar to the architecture of N 1 .
Finally,
I o (cid:48) s is just connected to v (cid:48)
l 1 , v (cid:48)
l 2 , . .
, v (cid:48)
ln l with the same weights as in N 1 .
This completes the proof.
It is worth mentioning that neuron v (cid:48) 0 is deﬁned to eliminate the division by zero occurred in Eq.
(2) when a neuron does not have a non-zero dot product to any input neuron.
Note that by choosing a right value for d , traditional neural networks can be replaced with multi-layer ﬂoating neural networks introduced in this paper.
Additionally, Theorem 1 implies that the farfalle neural networks are more general than multi-layer ﬂoating neural networks.
Thus, as a corollary we note that it is possible to replace traditional multi-layer neural networks with the recurrent structure of FNNs.
Furthermore, there is a repeated observation that replacing weights of a neural network with their low-rank approximations gives a comparable (or even improved) performance (Sainath et al.
Denil et al.
; Denton et al.
Such approximation allows us to replace dense networks with FFNs with small values of d and thus reasonable number of parameters.
Utilizing this model, more ﬂexibility in specifying the layer sizes is provided accordingly.
Furthermore, since we do not need to have weight parameters explicitly, we can consider the whole network as a fully connected structure in which each (hidden) neuron can be potentially connected to all other neurons and even itself.
3. S CALABILITY & P RODUCTION
During training, the weight matrix W needs to be computed to apply the normalization function.
Since the number of elements in this matrix is quadratic in the number of neurons, it is possible that this matrix becomes quite big.
However, after the training, it is possible to workaround the normalization step by updating the matrix I .
Speciﬁcally, using the same notation as the last section, it is enough to replace I v with
[Equation]
I
Tv
O ui ) 2 .
This simpliﬁcation signiﬁcantly reduces the required memory space during inference since the dimension of the embedding space is usually much smaller than the number of neurons.
Furthermore, although the need to compute the weight matrix during the training imposes a practical limit on the maximum number of neurons in FNN, the upper bound is still very large.
Additionally, it is possible to stack FNNs similar to normal layers to employ more ﬂoating neurons.
Such structure does not allow the use of information from all deeper layers but is still much more ﬂexible than commonly used dense layers.
R ESULTS & D ISCUSSION
In the following subsections, we present comparison results of our model with other DNN architectures on CIFAR (Krizhevsky et al.
(2009)), a widely-used image classiﬁcation dataset.
First, we compare FNNs with a dense model and show that our model can outperform them.
Then we discuss some characteristics of FNNs, such as their ability to learn locality.
In the ﬁnal section, we discuss how our model can be integrated with existing convolutional neural networks.
4. C OMPARISON WITH D ENSE M ODELS
Although dense models are not among the state of the art models for neither datasets, the goal of this section is to establish the effectiveness of our model in comparison with dense models.
We do not claim that our model, in any way, can directly outperform highly specialized models such as convolutional neural networks.
Instead, we demonstrate how our model may be used in conjunction with CNNs in subsequent sections.
In order to obtain a proper baseline for CIFAR, we used HyperOpt Bergstra et al.
(2013) to search among fully connected layers with up to 4 hidden layers.
Number of neurons in each layer was chosen from { 500 , 1000 , 1500 , 2000 } .
The top performing model was a fully connected network with 2 hidden layers of 2000 and 500 neurons, in order from input to output.
We trained a multi-layer ﬂoating network with the same structure and embedding size 256 .
We also trained a FNN the same number of neurons and embedding size.
The number of iterations was set to 2 .
Furthermore, we trained a smaller fully connected network with the same number of parameters as our models by reducing the number of neurons in the ﬁrst hidden layer of baseline to 850 .
We trained all four models using Adam (Kingma & Ba (2015)) algorithm for 200 epochs.
In all models we applied 0 .
1 dropout (Srivastava et al.
(2014)) rate of the input and used ReLU (Glorot et al.
(2011)) for activation function.
Summarized results are shown in Table 1.
It is evident that our models signiﬁcantly outperform the fully connected networks.
This is especially signiﬁcant since there is more than 70% reduction in number of parameters in multi-layer ﬂoating network with respect to the fully connected network of the same architecture.
The performance gap is worsened when comparing with the smaller base-line.
The results show that FNNs are able to obtain comparable results.
The slightly better performance of multi-layer ﬂoating network perform with respect to FNN is not unexpected.
The structure for the multi-layer ﬂoating network is the same as the baseline which was chosen by searching through the architecture space.
Moreover, the FNN was trained with almost no hyper-parameter tuning.
4. A NALYSIS OF L EARNED E MBEDDINGS
We analyzed the embeddings of an FNN with 1024 neurons, 256 embedding size, and 1 iteration trained on MNIST (LeCun et al.
(1998a)).
The training setting was the same as the previous section.
[Table]
In Figure 4, each cell is colored according to the l 2 -norm of its corresponding neuron’s output embeddings.
It can be seen that marginal neurons have much lower l 2 -norms.
This means that the embeddings are much smaller for these neurons, and so they have little effect on the model’s output.
Note that this is expected since the marginal pixels in MNIST images seldom provide useful information.
Figure 5 depicts the projection of the learned embeddings to 2D space.
The projection is performed using Uniform Manifold Approximation and Projection (UMAP, McInnes et al.
Neurons corresponding to adjacent cells are connected with a line.
Marginal neurons are excluded in order to obtain a better projection of the embeddings.
It is apparent in the ﬁgure that the learned embeddings respect the locality of pixels, so a pair of pixels close to each other have similar embeddings.
Hence, it can be seen that FNN is able to assign meaningful embeddings to the neurons.
4. I NTEGRATION WITH CNN S
Convolutional neural networks (LeCun et al.
(1998b)) are widely used in image classiﬁcation tasks and have been able to produce state of the art results.
Commonly in such networks, convolutional layers are employed for feature extraction.
The extracted features are then fed into several fully connected layers for classiﬁcation.
We propose that FNNs can be used to replace these fully connected layers.
To test this, we compared FNN with a baseline model on CIFAR10 and CIFAR100 datasets.
We use a slightly modiﬁed version of VGG16 Simonyan & Zisserman (2014) which is suitable for CIFAR as our baseline.
In this version, all layers after the last max-pooling layer are replaced with fully connected network with one hidden layer of 512 neurons.
Batch normalization and a dropout rate of 0 . 5 is applied after the hidden layer.
The same dropout rate is also applied before the hidden layer.
We used ReLU as the activation function.
A similar model has been used as a baseline in other studies (Rohekar et al.
; Li et al.
[Table]
We compare this baseline with an alternative architecture consisting of an FNN with 1024 neurons after the last max-pooling layer.
The number of iterations was set to 3 .
We used Stochastic Gradient Descent (SGD) with 0 .
9 momentum Rumelhart et al.
(1988) for training and employed 0 .weight decay regularization.
The maximum test accuracy of both models are presented in Table 2.
It is evident that our model outperforms the dense layers on both datasets.
F UTURE W ORK
We established that FNNs are able to replace and outperform fully connected layers.
FNNs are an extreme case of sharing embeddings between layers of a multi-layer ﬂoating neural network.
We note that it is possible to use different patterns of sharing.
Exploring this area might specially be interesting in the case of auto-encoders.
We conjecture that some of the neuron embeddings trained to solve one task can be used to solve similar tasks.
The input neuron embeddings projection in Figure 5 supports this conjecture.
However, this task is not trivial, and we leave it as a future work.
Another interesting direction is to assign different activation functions to different neurons in an
FNN.
This feature allows a combination of activation functions to be used together which is, to the best of our knowledge, something less explored in hand-crafted architectures.
C ONCLUSION
In this paper, we introduced a method to learn the network structure internally during training.
This was done mainly based on the new approach of assigning parameters to the neurons instead of the connection between them.
Using this approach, we introduced a novel neural network structure called Farfalle Neural Network.
We established through experiments that this new structure can outperform dense layers in various scenarios while even sometimes using signiﬁcantly ( 70% ) lower number of parameters.
We also discussed how this approach could signiﬁcantly reduce the memory requirements during the inference process
A A BLATION S TUDY
In this section, we investigate the effect of three hyper-parameters, speciﬁcally number of iterations, embedding size, and normalization function, on the performance of FNNs.
To assess the effect of each hyper-parameter, we set all other hyper-parameters to a default value and evaluate model performance for various values of this hyper-parameter.
The default values are 3 for the number of iterations, 256 for the embedding size, and l 2 for the normalization function.
In all experiments, the
FNN consists of 1024 neurons.
The results are presented in Figure 6. [Figure]
B I NTEGRATION WITH OTHER CNN S
In this section we present results of using FNNs instead of the fully connected layers at the tail of other convolutional networks, speciﬁcally Residual Networks He et al. We compare performance of our models with baseline models on CIFAR100 dataset. We train ResNet18 as our baseline and compare it with a variant of ResNet18 with an FNN instead of the tail fully connected layer.
The FNN has 512 neurons with 64 embedding size.
The number of iterations is set to 1 so the network is equivalent with a multi-layer ﬂoating network with one hidden layer.
For this task, we ﬁxed the transformation function of the input neurons to identity.
The results are presented in Table 3.
[Table]
S UPPLEMENTARY F IGURES