We show that associative networks of threshold linear units endowed with Hebbian learning  can operate closer to the Gardner optimal storage capacity than their binary counterparts and even surpass this bound . This is largely achieved through a sparsification of the retrieved patterns, which we analyze for theoretical and empirical distributions of activity. As reaching the optimal capacity via non-local learning rules like back-propagation requires slow and neurally implausible training procedures, our results indicate that one-shot URLanized Hebbian learning can be just as efficient.
We derive the Gardner storage capacity for associative networks of threshold linear units , and show that with Hebbian learning they can operate closer to such Gardner bound than binary networks, and even surpass it . This is largely achieved through a sparsification of the retrieved patterns, which we analyze for theoretical and empirical distributions of activity. As reaching the optimal capacity via non-local learning rules like backpropagation requires slow and neurally implausible training procedures, our results indicate that one-shot URLanized Hebbian learning can be just as efficient.
R
show that
derive the Gardner storage capacity for
R
endowed
, and show that
A
they
R
the Gardner optimal storage capacity than their binary counterparts
such Gardner bound than binary networks,
R
this bound
it
R
back-propagation
backpropagation