A correlation game for unsupervised learning yields computational interpretations of Hebbian excitation, anti-Hebbian inhibition, and synapse elimination

Much has been learned about plasticity of biological synapses from empirical studies. Hebbian plasticity is driven by correlated activity of presynaptic and postsynaptic neurons. Synapses that converge onto the same neuron often behave as if they compete for a fixed resource; some survive the competition while others are eliminated. To provide computational interpretations of these aspects of synaptic plasticity, we formulate unsupervised learning as a zero-sum game between Hebbian excitation and anti-Hebbian inhibition in a neural network model. The game formalizes the intuition that Hebbian excitation tries to maximize correlations of neurons with their inputs, while anti-Hebbian inhibition tries to decorrelate neurons from each other. We further include a model of synaptic competition, which enables a neuron to eliminate all connections except those from its most strongly correlated inputs. Through empirical studies, we show that this facilitates the learning of sensory features that resemble parts of objects. Much has been learned about plasticity of biological synapses from empirical studies. One line of research has explored Hebbian plasticity, which is triggered by correlated presynaptic and postsynaptic activity. Many kinds of excitatory synapses are strengthened by correlated activity, and are said to be Hebbian [Bi and Poo, 2001]. Strengthening of inhibitory synapses by correlated activity has also been observed [Gaiarsa et al., 2002]. The phenomenon is sometimes said to be anti-Hebbian, because strengthening an inhibitory synapse is like making a negative number more negative. Another line of research has shown that synapses converging onto the same neuron often behave as if they compete for shares of a fixed “pie”; some survive the competition while others are eliminated [Lichtman and Colman, 2000]. According to this viewpoint, Hebbian plasticity does not increase or decrease the pie of synaptic resources; it only allocates resources across convergent synapses [Miller, 1996] Theoretical neuroscientists have proposed a number of computational functions for Hebbian excitation, anti-Hebbian inhibition, and synaptic competition/elimination. Hebbian excitation has long been invoked as a mechanism for learning features from sensory input [Von der Malsburg, 1973]. Lateral inhibition has been used to sparsen neural activity, thereby facilitating Hebbian feature learning [Von der Malsburg, 1973, Fukushima, 1980, Rumelhart and Zipser, 1985, Kohonen, 1990]. Endowing the lateral inhibition with anti-Hebbian plasticity can give more robust control over sparseness of activity. Földiák [1990] demonstrated this with numerical experiments, but did not provide an interpretation in terms of an optimization principle. Less relevant here are antiHebbian models without reference to sparse activity in linear [Foldiak, 1989, Rubner and Tavan, 1989, Rubner and Schulten, 1990] and nonlinear [Carlson, 1990, Girolami and Fyfe, 1997] networks. Also less relevant is the application of anti-Hebbian plasticity to feedforward rather than lateral connections [Hyvärinen and Oja, 1998]. 1 ar X iv :1 70 4. 00 64 6v 1 [ cs .N E ] 3 A pr 2 01 7 Leen [1991] performed a stability analysis for a linear network with Hebbian feedforward connections and anti-Hebbian lateral connections. Plumbley [1993] derived a linear network with antiHebbian lateral inhibition (but no plasticity of feedforward connections) from the principle of information maximization with a power constraint. Pehlevan et al. [2015] showed that a linear network with Hebbian feedforward connections and anti-Hebbian lateral inhibition can be interpreted as online gradient optimization of a “similarity matching” cost function. Pehlevan and Chklovskii [2014] and Hu et al. [2014] went on to extend the similarity matching principle to derive nonlinear neural networks for unsupervised learning. Synaptic competition and elimination have been studied in models of cortical development, and have been shown to play an important role in the emergence of feature selectivity [Miller, 1996]. The subject of the present work is a mathematical formalism that provides computational interpretations of Hebbian excitation, anti-Hebbian inhibition, and synaptic competition/elimination in nonlinear neural networks. We start by formulating unsupervised learning as the maximization of output-input correlations subject to upper bound constraints on output-output correlations. We motivate our formulation by describing its relation to previous theoretical frameworks for unsupervised learning, such as maximization [Linsker, 1988, Atick and Redlich, 1990, Plumbley, 1993, Bell and Sejnowski, 1995] or minimization of mutual information [Hyvärinen and Oja, 2000], (2) projection onto a subspace that maximizes a moment-based statistic such as variance [Oja, 1982, Linsker, 1988] or kurtosis [Huber, 1985], and the (3) similarity matching principle [Pehlevan et al., 2015]. To solve our constrained maximization problem, we introduce Lagrange multipliers. This Lagrangian dual formulation of unsupervised learning can in turn be solved by a nonlinear neural network with Hebbian excitation and anti-Hebbian inhibition. The network is very similar to the original model of Földiák [1990], differing mainly by its use of rectification rather than sigmoidal nonlinearity. (The latter can also be handled by our formalism, as shown in Appendix B.) Lagrange multipliers were also used to study anti-Hebbian plasticity by Plumbley [1993], but only for linear networks. Effectively, excitation and inhibition behave like players in a game, and the inhibitory connections can be interpreted as Lagrange multipliers. The game is zero-sum, in that excitation tries to maximize a payoff function and inhibition tries to minimize exactly the same payoff function. Roughly speaking, however, one could say that excitation aims to maximize the correlation of each output neuron with its inputs, while inhibition aims to decorrelate the output neurons from each other. Our term “correlation game” is derived from this intuitive picture. Within our mathematical formalism, we also consider a dynamics of synaptic competition and elimination that is drawn from models of cortical development [Miller and MacKay, 1994]. Competition between the excitatory synapses convergent on a single neuron is capable of driving the strengths of some synapses to zero. In numerical experiments with the MNIST dataset, we show that synapse elimination has the computational function of facilitating the learning of features that resemble “parts” of objects. Theoretical analysis shows that the surviving synapses converging onto an output neuron come from its most strongly correlated inputs; synapses from weakly correlated inputs are eliminated. Our correlation game is closely related to the similarity matching principle of Pehlevan and Chklovskii [2014] and Pehlevan et al. [2015]. The major novelty is the introduction of decorrelation as a constraint for the optimization. Paralleling our work, Pehlevan et al. [2017] have shown that the similarity matching principle leads to a game theoretic formulation through Hubbard-Stratonovich duality. Again our novelty is the use of decorrelation as a constraint, which leads to our correlation game through Lagrangian duality. Our model of synaptic competition and elimination was borrowed with slight modification from the literature on modeling neural development [Miller and MacKay, 1994]. It can be viewed as a more biologically plausible alternative to previous unsupervised learning algorithms that sparsen features. For example, Hoyer [2004] is similar to ours because it can be interpreted as independently sparsening each set of convergent synapses, rather than applying a global L1 regularizer to all synapses.