A N ADAPTIVE HOMEOSTATIC ALGORITHM FOR THE UNSUPERVISED LEARNING OF VISUAL FEATURES
Anonymous authors
Paper under double-blind review
A BSTRACT
The formation of structure in the brain, that is, of the connections between cells within neural populations, is by large an unsupervised learning process: the emergence of this architecture is mostly self-organized.
In the primary visual cortex of mammals, for example, one may observe during development the formation of cells selective to localized, oriented features.
This leads to the development of a rough representation of contours of the retinal image in area V1.
We modeled these mechanisms using sparse Hebbian learning algorithms.
These algorithms alternate a coding step to encode the information with a learning step to ﬁnd the proper encoder.
A major difﬁculty faced by these algorithms is to deduce a good representation while knowing immature encoders, and to learn good encoders with a non-optimal representation.
To address this problem, we propose to introduce a new regulation process between learning and coding, called homeostasis.
Our homeostasis is compatible with a neuro-mimetic architecture and allows for the fast emergence of localized ﬁlters sensitive to orientation.
The key to this algorithm lies in a simple adaptation mechanism based on non-linear functions that reconciles the antagonistic processes that occur at the coding and learning time scales.
We tested this unsupervised algorithm with this homeostasis rule for a range of existing unsupervised learning algorithms coupled with different neural coding algorithms.
In addition, we propose a simpliﬁcation of this optimal homeostasis rule by implementing a simple heuristic on the probability of activation of neurons.
Compared to the optimal homeostasis rule, we show that this heuristic allows to implement a more rapid unsupervised learning algorithm while keeping a large part of its effectiveness.
These results demonstrate the potential application of such a strategy in machine learning and we illustrate this with one result in a convolutional neural network
I NTRODUCTION
The neural architecture is a complex dynamic system that operates at different time scales.
In particular, one of its properties is to succeed in representing quickly information (the coding step) while optimizing in the long term its encoding (the learning step).
In the case of the mammalian primary visual cortex (V1) for instance, this rapid coding operation, of the order of 50 milliseconds in humans, is the key to the results of Hubel & Wiesel (1968), who showed that some cells of V have relatively localized receptive ﬁelds which are predominantly selective at different orientations.
As such, one can consider the rapid coding of the retinal image as a process of transforming the raw visual information into a rough “sketch” that represents the outlines of objects in the image by using elementary edge-like features.
This internal representation and the visual information share the same property of being sparse : for most natural images, only a relatively small number of features are necessary to describe the input.
Thus, the coding step consists in choosing the right encoder that selects as few features (called atoms) as possible among a collection of them (called the dictionary ).
Amazingly, Olshausen & Field (1996) have shown that when enforcing a sparse prior on the encoding step, such edge-like ﬁlters are emerging using a simple Hebbian unsupervised learning strategy.
Additionally, recent advances in machine learning, and especially on unsupervised learning have shed new light on the functioning of the underlying biological neural processes.
By deﬁnition, unsupervised learning aims at learning the best dictionary to represent the input image autonomously, that is, without using other external knowledge such as in supervised or reinforcement learning.
Algorithms that combines such learning as the input to classical, supervised deep-learning show great success in tasks like image denoising (Vincent et al., 2008) or classiﬁcation (Sulam et al., 2017).
A variant consists in forcing the generated representation to be sparsely encoded (Makhzani &
Frey, 2013), whether by adding a penalty term to the optimized cost function or by encoding each intermediate representation by a pursuit algorithm (Papyan et al., 2016).
Interestingly, (Papyan et al., 2016) proposes a model of Convolutional Sparse Coding (CSC) tightly connected with Convolutional
Neural Network (CNN), so much that the forward pass of the CNN is equivalent to a CSC with a thresholding pursuit algorithm.
These unsupervised algorithms are equivalent to a gradient descent optimization over an informational-type coding cost (Kingma & Welling, 2013).
This cost makes it then possible to quantitatively evaluate the joint exploration of new learning or coding strategies.
As such, this remark shows us that unsupervised learning consists of two antagonistic mechanisms, a long time scale that corresponds to the learning and exploration of new components and a faster scale that corresponds to coding.
In particular, an aspect often ignored in this type of learning is the set of homeostasis mechanisms that control the average activity of neurons within a population.
Indeed, there is an intrinsic complexity in unsupervised dictionary learning algorithms: how to adapt the regularization parameter of each atom to make sure no atoms are wasted because of improper regularization settings?
In the original algorithms of sparse unsupervised learning (Olshausen & Field, 1997), homeostasis is implemented as a heuristic that prevents the algorithm from diverging.
In most unsupervised learning algorithms it takes the form of a normalization, that is, an equalization of the energy of each atom in the dictionary (Mairal et al., 2014).
However, the neural mechanisms of homeostasis are at work in many components of the neural code and are essential to the overall transduction of neural information.
For example, the sub-networks of glutamate and GABA-type neurons may regulate the overall activity of neural populations (Marder & Goaillard, 2006).
In particular, such mechanisms could be tuned to balance the contribution of the excitatory populations with respect to that in inhibitory populations.
As a consequence, this creates a so-called balanced network which may explain many facets of the properties of the primary visual cortex (Hansel & van Vreeswijk, 2012).
At the modelling level, these mechanisms are often implemented in the form of normalization rules (Schwartz & Simoncelli, 2001) which are considered as the basis of a normative theory to explain the function of the primary visual cortex (Carandini & Heeger, 2012).
However, when extending such model using unsupervised learning, most modelling effort is rather intended to show that the cells’ selectivity that emerges have the same characteristics than those observed in neuro-physiology (Ringach, 2002; Rehn & Sommer, 2007; Loxley, 2017).
Other algorithms use non-linearities that implicitly implement homeostatic rules in neuro-mimetic algorithms (Brito & Gerstner, 2016).
These non-linearities are mainly used in the output of successive layers of deep learning networks that are nowadays widely used for image classiﬁcation or artiﬁcial intelligence.
However most of these non-linear normalization rules are based on heuristics mimicking neural mechanisms but are not justiﬁed as part of the global problem underlying unsupervised learning.
Framing this problem in a probabilistic framework allows to consider in addition to coding and learning the intermediate time scale of homeostasis and allows us also to associate it to an adaptation mechanisms (Rao & Ballard, 1999).
Our main argument is that by optimizing unsupervised learning at different time scales, we allow for the implementation of fast algorithms compatible with the performance of biological networks and in comparison with classical (Olshausen & Field, 1997) or Deep Learning approaches.
In this paper, we will ﬁrst deﬁne a simple algorithm for controlling the selection of coefﬁcients in sparse coding algorithms based on a set of non-linear functions similar to a generic neural gain normalization mechanisms.
Such functions will be used to implement an homeostasis mechanism based on histogram equalization by progressively adapting these non-linear functions.
In particular, this algorithm will extend an already existing algorithm of unsupervised sparse learning (Perrinet, 2010) to a more general setting.
In particular, we will show quantitative results of this optimal algorithm by applying it to different pairs of coding and learning algorithms.
Second, we will propose a simpliﬁcation of this homeostasis algorithm based on the activation probability of each neuron and show that it yields similar quantitative results as the full homeostasis algorithm and that it converges more rapidly than classical methods (Olshausen & Field, 1997; Sandin & Martin-del Campo, 2017).
In particular, we focused in our architecture to be able to quantitatively cross-validate for every single hyper-parameters and all these scripts are available at https://github.com/XXX/ZZZ .
Finally, we will conclude by showing an application of such an adaptive algorithm to CNNs and discuss on its development in real-world architectures. [Figure] o n e O L S
H E
[Figure]
[Figure] None [Figure] The compared algorithms are : None normalization method of (Olshausen & Field, 1997)) using the optimal homeostasis described in this
441 ﬁlters image patches (
18 = 324 ) fact that without proper homeostasis inhomogeneous representations
(B) Evolution of cost F (in bits, see Eq. 4) as a function of the number of iterations and cross-validated over 10 runs.
While OLS provides a similar convergence than None , the HEH method provides a better ﬁnal convergence.
U NSUPERVISED LEARNING AND THE OPTIMAL REPRESENTATION OF IMAGES
Visual items composing natural images are often sparse, such that knowing a model for the generation of images, the brain may use this property to reconstruct images using only a few of these items.
In the context of the representation of natural images 1 y =
( y k )
Kk =1 ∈ R K
× M represented in a matrix as a set of K vector samples (herein, we will use a batch size of K = 256 ) as images raveled along M pixels (each y k,j ∈ R are the corresponding luminance values), let us assume the generic Generative
Linear Model, such that for any sample k
the image was generated as y k =
Φ T a k + (cid:15) , where by deﬁnition, the coefﬁcients are denoted by a k = ( a k,i ) Ni =1 ∈ R N and the dictionary by Φ ∈ R N × M .
Finally, (cid:15) ∈ R M is a Gaussian iid noise which is Normal without loss of generality by scaling the norm of the dictionary’s rows.
Knowing this model, unsupervised learning aims at ﬁnding the least surprising causes (the parameters ˆ a k and Φ ) for the data y k .
In particular, the cost may be formalized in a probabilistic terms as (Olshausen & Field, 1997): [Equation]
[Equation]
Such hypothesis allows to retrieve the cost that is optimized in most of existing models of unsupervised learning.
Explicitly, the representation is optimized by minimizing a cost deﬁned on prior assumptions on representation’s sparseness, that is on log p
Φ ( a k ) .
For instance, learning is accomplished in
S PARSE N ET (Olshausen & Field, 1997) by deﬁning a sparse prior probability distribution function for each coefﬁcients in the factorial form log p Φ ( a k ) ∼ − β (cid:80)
i log(1 + a 2 i σ 2 ) where β corresponds to the steepness of the prior and σ to its scaling (see Figure 13.2 from (Olshausen, 2002)).
Then, knowing this sparse solution, learning is deﬁned as slowly changing the dictionary using Hebbian learning.
Indeed, to compute the partial derivate of F with respect to Φ , we have ∀ i :
[Equation]
This allows to deﬁne unsupervised learning as the gradient descent using this equation.
Similarly to Eq.
17 in (Olshausen & Field, 1997) or to Eq.
2 in (Smith & Lewicki, 2006)
, the relation is a linear “Hebbian” rule (Hebb, 1949) since it enhances the weight of neurons proportionally to the activity (coefﬁcients) between pre- and post-synaptic neurons.
Note that there is no learning for non-activated coefﬁcients and also that we used a (classical) scheduling of the learning rate and a proper initialization of the weights (see Annex 2.5 & 2.6).
The novelty of this formulation compared to other linear Hebbian learning rule such as (Oja, 1982) is to take advantage of the sparse (non-linear) representation, hence the name Sparse Hebbian Learning (SHL).
In general, the parameterization of the prior in Eq. 1 has major impacts on results of the sparse coding and thus on the emergence of edge-like receptive ﬁelds and requires proper tuning.
For instance, a L2-norm penalty term (that is, a Gaussian prior on the coefﬁcients) corresponds to Tikhonov regularization (Tikhonov, 1977) and a L1-norm term (that is, an exponential prior for the coefﬁcients) corresponds to the convex cost which is optimized by least-angle regression (LARS)
(Efron et al., 2004) or FISTA (Beck & Teboulle, 2009).
2. A LGORITHM : S PARSE C ODING WITH A CONTROL MECHANISM FOR THE SELECTION OF ATOMS
Concerning the choice of a proper prior distribution, the spiking nature of neural information demonstrates that the transition from an inactive to an active state is far more signiﬁcant at the coding time scale than smooth changes of the ﬁring rate.
This is for instance perfectly illustrated by the binary nature of the neural code in the auditory cortex of rats (DeWeese et al., 2003).
Binary codes also emerge as optimal neural codes for rapid signal transmission (Bethge et al., 2003).
This is also relevant for neuromorphic systems which transmit discrete, asynchronous events (such as a network packet).
With a binary event-based code, the cost is only incremented when a new neuron gets active, regardless to its (analog) value.
Stating that an active neuron carries a bounded amount of information of λ bits, an upper bound for the representation cost of neural activity on the receiver end is proportional to the count of active neurons, that is, to the (cid:96) 0 pseudo-norm
||
a k ||
= |{ i, a k,i (cid:54) = 0 }| :
[Equation]
This cost is similar with information criteria such as the Akaike Information Criteria (Akaike, 1974) or distortion rate (Mallat, 1998, p. 571).
For λ = log 2 N , it gives the total information (in bits) to code for the residual (using entropic coding) and the list of spikes’ addresses.
In general, the high inter-connectivity of neurons (on average approximately 10000 synapses per neurons) justiﬁes such an informational perspective with respect to the analog quantization of information in the point-topoint transfer of information between neurons.
However, Eq.
3 deﬁnes a harder cost to optimize (in comparison to convex formulations in Equation 1 for instance) since the hard (cid:96) 0 pseudo-norm sparseness leads to a non-convex optimization problem which is NP-complete with respect to the dimension M of the dictionary (Mallat, 1998, p. 418).
Still, there are many solutions to this optimization problem and here, we will use a generalized version of the Matching Pursuit (MP)
algorithm
(Mallat, 1998, p. 422).
A crucial aspect of this algorithm is the arg max function as it produces at each step a competition among N neurons.
For this reason, we will introduce a mechanism to tune this competition.
For any signal y k drawn from the database, we get the coefﬁcients a k = S
( y k ; Ψ = { Φ , z, N 0 } ) thanks to Algorithm 1.
The parameter N 0 >controls the amount of sparsity that we impose to the coding.
The novelty of this generalization of
MP lies in the scalar functions z
= { z i } i =1 ...N which control the competition for the best match across atoms.
While an identical symmetric function is chosen in the original MP algorithm (that is, ∀ i, z i ( a k ) = | a k | ), we will deﬁne these at a ﬁrst attempt as the rescaled non-linear rectiﬁed linear unit (ReLU) with gain γ
i
∀
i, z i ( a k,i ) =
γ i ∗ a k,i ∗ δ ( a k,i > 0) where δ is Kronecker’s indicator function.
We found as in (Rehn & Sommer, 2007)
that by using an algorithm like Matching Pursuit (that is using the symmetric function or setting ∀ i, γ i = 1 as in (Mairal et al., 2014) for instance),
the
Sparse Hebbian Learning algorithm could provide results similar to S PARSE N ET . An advantage is [Figure]
[Figure]
) on the ﬁnal cost as computed for the same learning algorithms but with different homeostatic strategies ( None , OLS or HEH ).
Parameters were explored around a default value, on a logarithmic scale and over 4 octaves.
This shows that HEH is robust across a wide range of parameters. the non-parametric assumption on the prior based on this more generic (cid:96) 0 pseudo-norm sparseness.
However, we observed that this class of algorithms could lead to solutions corresponding to a local minimum of the full objective function: Some solutions seem as efﬁcient as others for representing the signal but do not represent edge-like features homogeneously (Figure 1-A, None ).
Moreover, using other sparse coding algorithms which are implemented in the sklearn library, we compared the convergence of the learning with different sparse coding algorithms.
In particular, we compared the learning as implemented with matching pursuit to that with orthogonal matching pursuit (OMP) (Pati et al., 1993), LARS or FISTA (see Annex 2.4).
For all these sparse coding algorithms, during the early learning step, some cells may learn “faster” than others.
In particular, these cells have more peaked distributions of their activity and tend to be selected more often.
There is the need for a homeostasis mechanism that will ensure convergence of learning.
The goal of this work is to study the speciﬁc role of homeostasis in learning sparse representations and to propose a homeostasis mechanism based on the functions
z i which optimizes the learning of an efﬁcient representation.
2. A LGORITHM : H ISTOGRAM E QUALIZATION H OMEOSTASIS
Knowing a dictionary and a sparse coding algorithm, we may transform any data sample y k into a set of sparse coefﬁcients using the above algorithm: a k = S ( y k ;
Ψ = { Φ , z, N 0 } ) (see Algorithm 1).
In particular, at any step during learning, dictionaries may not have been homogeneously learned and may exhibit different distributions.
However, this would not be taken into account in the original cost (see Eq. 3) as we assumed as in Olshausen & Field (1997) that p Φ ( ˆ a k ) is factorized, that is, that the components of the sparse vector are independent.
As a consequence, we may use a deviation to this hypothesis as an additional component to the cost:
[Equation]
Where we used the mutual information MI as a proxy to measure the dependence between the components of the sparse vector.
Indeed, as information is coded in the address of neurons, information transfer as computed through Shannon entropy, is optimized when the activity within the neural population is uniformly balanced, that is when each neuron is a priori selected with the same probability.
In particular, a necessary (yet not sufﬁcient) condition for minimizing this cost is that the prior probability of selecting coefﬁcients are identical ∀ ( i, j ) , q Ψ ( a k,i ) =
q Ψ ( a k,j ) to ensure the optimality of the choice of the (cid:96) 0 pseudo-norm and compare it to the representation in the primary visual cortex.
As we have seen, we may use different transformation functions z to inﬂuence the choice of coefﬁcients such that we may use these functions to optimize the objective cost deﬁned by Eq. 4.
To achieve this uniformity, we may deﬁne an homeostatic gain control mechanism based on histogram equalization, that is, by transforming coefﬁcients in terms of z-scores, that is, by setting ∀ i, z i ( · ) =
P ( · > a i ) .
Such a transform is similar to the inverse transform sampling which is used to optimize representation in auto-encoders (Doersch, 2016) and can be considered as a non-parametric extension of the “re-normalization trick” used in variational auto-encoders (Kingma & Welling, 2013).
Moreover, it has been found that such an adaptation mechanism is observed in the response of the retina to various contrast distributions (Laughlin, 1981).
However, an important point to note is that this joint optimization problem between coding and homeostasis is circular as we can not access the true posterior p
Φ ( a ) :
Indeed, the coefﬁcients depend on non-linear coefﬁcients through a k = S
( y k ; Ψ = { Φ , z i , N 0 } ) , while the non-linear functions depend on the (cumulative) distribution of the coefﬁcients.
We will make the assumption that such a problem can be solved iteratively by slowly learning the non-linear functions.
Starting with an initial set of non-linear functions as in
None , we will derive an approximation for the sparse coefﬁcients.
Then, the function z i for each coefﬁcient of the sparse vector is calculated using an iterative moving average scheme (parameterized by time constant 1 /η h ) to smooth its evolution during learning.
At the coding level, this z-score function is incorporated in the matching step of the matching pursuit algorithm, to modulate the choice of the most probable as that with the maximal z-score: i ∗ = arg max i z
i ( a i ) (see Algorithm 1).
The rest of the algorithm Sparse Hebbian Learning algorithm is left unchanged (see Algorithm 2).
We will coin this algorithm variant as Histogram Equalization Homeostasis ( HEH ).
Furthermore, as we adapt the dictionaries progressively during Sparse Hebbian Learning, we may incorporate this HEH homeostasis during learning by choosing an appropriate learning rate η h .
To recapitulate the different choices we made from the learning to the coding and the homeostasis, the unsupervised learning can be summarized using the following steps:
Algorithm 2 Homeostatic Unsupervised Learning of Kernels: Φ = H ( y ; η, η h , N 0 ) 1: Initialize the point non-linear gain functions
z i to similar cumulative distribution functions, 2: initialize atoms Φ i to random points on the K -unit sphere, 3: for T epochs do : 4: draw a new batch y from the database of natural images, 5: for each data point y k do : 6: compute the sparse representation vector a = S ( y k ; Ψ = { Φ , z, N 0 } ) (see Algorithm 1), 7: modify dictionary: ∀ i, Φ i ← Φ i + η · a i · ( y k − Φ a ) , 8: normalize dictionary: ∀ i, Φ i ← Φ i / || Φ i
|| , 9: update homeostasis functions: ∀ i, z i ( · ) ← (1 − η h ) · z i ( · ) + η h · δ ( a i ≤ · ) .
We compared qualitatively the set Φ of receptive ﬁlters generated with different homeostasis algorithms (see Fig.
1-A).
A more quantitative study of the coding is shown by comparing the decrease of the cost as a function of the iteration step (see Fig.
1-B).
This demonstrate that forcing the learning activity to be uniformly spread among all receptive ﬁelds results in a faster convergence of the representation error as represented by the decrease of the cost F .
H E
H
E M
P
H A
P
[Figure]
[Figure] step
[Figure] [Figure]
OLS
HEH
[Figure]
2. R ESULTS : F AST UNSUPERVISED LEARNING USING HOMEOSTASIS
We have shown above that we can ﬁnd an exact solution to the problem of homeostasis during
Sparse Hebbian Learning.
However, this solution has several drawbacks.
First, it is computationally intensive on a conventional computer as it necessitates to store each z i function to store the cumulative distribution of each coefﬁcient.
More importantly, it seems that biological neurons seem to rather use a simple gain control mechanism.
This can be implemented by modifying the gain γ i of the slope of the ReLu function to operate a gradient descent on the cost based on the distribution of each coefﬁcients.
Such strategy can be included in the SHL algorithm by replacing line 8 in Algorithm 2.
For instance, the strategy of (Olshausen & Field, 1997) assumes a cost on the difference between the observed variance of coefﬁcients
V i as computed over a set of samples compared to a desired value σ 2 g (and assuming a multiplicative noise parameterized by α ) :
[Equation]
and γ i ←
[Equation] i [Equation] This is similar to the mechanisms of gain normalization proposed by Schwartz & Simoncelli (2001) and which were recently shown to provide efﬁcient coding mechanisms by Simoncelli (2017).
However, compared to these methods which manipulate the gain of dictionaries based on the energy of coefﬁcients, we propose to rather use a methodology based on the probability of activation.
Indeed, the main distortion that occurs during learning is on higher statistical moments rather than variance, for instance when an atom is winning more at the earlier iterations, its pdf will typically be more kurtotic than a ﬁlter that has learned less.
Recently, such an approach was proposed by Sandin & Martin-del Campo (2017).
Based on the same observations, the authors propose to optimize the coding during learning by modulating the gain of each dictionary element based on the recent activation history.
They base their Equalitarian
Matching Pursuit ( EMP ) algorithm on a heuristics which cancels the activation of any ﬁlter that was more often activated than a given threshold probability (parameterized by 1 + α h ).
In our setting, we may compute a similar algorithm using an evaluation of probability of activation followed by binary gates:
[Equation]
Interestingly, they reported that such a simple heuristic could improve the learning, deriving a similar result as we have shown in Figure 1 and Figure 2.
Again, such strategy can be included in Algorithm 2.
Similarly, we may derive an approximate homeostasis algorithm based on the current activation probability but using a gradient descent approach on gain modulation.
Ideally, this corresponds to ﬁnding γ i such that we minimize the entropy (cid:80) i =1 ··· N p i log p i .
However, the sparse coding function S ( y k ; Ψ = { Φ , z, N 0 } ) is not differentiable.
One possible heuristic is then to derivate the change of modulation gain that would be necessary to achieve an equiprobable probability, that is when
[Equation]
We will coin this variant of the algorithm Homeostasis on Activation Probability ( HAP ).
Following these derivations, we quantitatively compared OLS , EMP and HAP to HEH (see Figure 3).
This shows that while EMP slightly outperforms OLS (which itself is more efﬁcient than None , see Figure 2-B),
HAP proves to be closer to the optimal solution given by HEH .
In particular, we replicated in HAP the result of Sandin & Martin-del Campo (2017) that while homeostasis was essential in improving unsupervised learning, the coding algorithm (MP versus OMP) mattered relatively little (see Annex 2.4).
Also, we veriﬁed the dependence of this efﬁciency with respect to different hyperparameters (as we did in Figure 2-B).
These quantitative results show that the HEH algorithm could be replaced by a simpler and more rapid heuristic, HAP , which is based on activation probability.
This would generate similar efﬁciency for the coding of patches from natural images.
D ISCUSSION AND CONCLUSION
One core advantage of sparse representations is the efﬁcient coding of complex signals using compact codes.
Inputs are thus represented as combination of few elements drawn from a large dictionary of atoms.
As a consequence, a common design for unsupervised learning rules relies on a gradient descent over a cost measuring representation quality with respect to sparseness.
This constraint introduces a competition between atoms.
In the context of the efﬁcient processing of natural images, we proposed here that such strategies can be optimized by including a proper homeostatic regulation enforcing a fair competition between the elements of the dictionary.
We implemented this rule by introducing a non-linear gain normalization similar to what is observed in biological neural networks.
We validated this theoretical insight by challenging this adaptive unsupervised learning algorithm with alternate heuristics for homeostasis.
Simulations show that at convergence, while the coding accuracy did not vary much, including homeostasis changed qualitatively the learned features.
In particular, homeostasis results in a more homogeneous set of orientation selective ﬁlters, which is closer to what is found in the visual cortex of mammals (Ringach, 2002; Rehn & Sommer, 2007; Loxley, 2017).
To further validate these results, we quantitatively compared the efﬁciency of the different variants of the algorithms, both at the level of homeostasis (homeostatic learning rate, parameters of the heuristics), but also to the coding (by changing M , N or N 0 ) and to the learning (by changing the learning rate, the scheduling or M ).
This demonstrated that overall, this neuro-inspired homeostatic algorithm provided with the best compromise between efﬁciency and computational cost.
In summary, this biologically-inspired learning rule demonstrates that principles observed in neural computations can help improve real-life machine learning algorithms.
Indeed, by developing this fast learning algorithm, we hope for its rapid application in artiﬁcial intelligence algorithms.
This type of architecture is economical, efﬁcient and fast.
It makes it possible to be transferred to most deep learning algorithms.
Along with this, we hope that this new type of rapid unsupervised learning algorithm can provide a normative theory for the coding of information in low-level sensory processing, whether it is visual or auditory, for example.
Moreover, by its nature, this algorithm can easily be extended to convolutional networks such as those used in deep learning neural networks.
This extension is possible by extending the ﬁlter dictionary by the hypothesis of invariances to the translation of representations.
Our results on different databases show the stable and rapid emergence of characteristic ﬁlters on these different bases (see Figure 4 and Annex 3.1).
This result shows a probable prospect of extending this representation and for which we hope to obtain classiﬁcation results superior to the algorithms existing in the state-of-the-art.
As such, empirical evaluations of the proposed algorithms should be extended.
For instance, it would be very useful to test for image classiﬁcation results on standard benchmark datasets.
[Figure]