Augmenting Supervised Learning by Meta-learning Unsupervised Local Rules
Abstract
The brain performs unsupervised learning and (perhaps) simultaneous supervised learning.
This raises the question as to whether a hybrid of supervised and unsupervised methods will produce better learning.
Inspired by the rich space of
Hebbian learning rules, we set out to directly learn the unsupervised learning rule on local information that best augments a supervised signal.
We present the
Hebbian-augmented training algorithm ( HAT ) for combining gradient-based learning with an unsupervised rule on pre-synpatic activity, post-synaptic activities, and current weights.
We test HAT ’s effect on a simple problem (Fashion-MNIST) and ﬁnd consistently higher performance than supervised learning alone.
This ﬁnding provides empirical evidence that unsupervised learning on synaptic activities provides a strong signal that can be used to augment gradient-based methods.
We further ﬁnd that the meta-learned update rule is a time-varying function; thus, it is difﬁcult to pinpoint an interpretable Hebbian update rule that aids in training.
We do ﬁnd that the meta-learner eventually degenerates into a non-Hebbian rule that preserves important weights so as not to disturb the learner’s convergence.
Prior Work and the Meta-Learning Setting
Gradient descent achieves great performance as in neural net optimization but might not be biologically plausible because most problems are not explicitly phrased as classiﬁcation with true labels and because neurons only know local signals (e g. synaptic density, ACh levels, current).
These local signals likely do not encode a global loss.
In contrast, local learning computes unsupervised weight updates using only synaptic information.
Consider two adjacent neurons i, j with weight w ij .
Formally, given an impulse traversing i, j with activations v i , v j , a local learning rule computes updates ∆ w ij using local data v i , w ij , v j .
Many neuroscientists have hypothesized the brain’s true local learning rule.
Most such rules involve using the correlation of activations as part of the update rule.
Examples include Hebb’s Rule [3],
Oja’s Rule [6], the Generalized Hebbian Algorithm [2], and nonlineear Hebbian rules [4].
It is not obvious which of these rules (if any) describe the true behavior of neurons.
We employ meta-learning (learning how to learn) as an investigative tool.
In the meta-learning framework, one model A learns a task (e.g. Fashion-MNIST) while another model B learns how to optimize A .
Meta-learning has been applied to learning gradient-based optimization ( B learns to update A using A ’s gradients)
[1] and to to learning gradient-free optimization ( B learns to update A using only the sequence of A ’s losses).
Finally, Metz et al.
demonstrated a fully differentiable architecture for learning to learn unsupervised local rules and demonstrate better-than-random performance on a few-shot basis.
We wish to extend Metz’s approach to learning semi-supervised local rules not only to improve performance but also to investigate the functional form of the meta-learned update rule.
The Hebbian-Augmented Training Algorithm
We create 2 multilayer perceptrons: a learner L ( · | φ L ) with parameters φ L and a meta-learner
M ( v i , v j , w ij | φ M ) with parameters φ M , which takes inputs v i , w ij , v j and returns ∆ w ij .
The Hebbian-Augmented Training algorithm ( HAT ) takes as input learner L , meta-learner M , optimizer A , and training point ( (cid:126)x, (cid:126)y ) .
We train L without supervision using M and (cid:126)x .
We then simultaneously train L and M with supervision using A and (cid:126)y .
On the forward pass, we compute activations for each layer.
We then convolve the meta-learner M over the stacked tensor of the inputs, weights, and activations to produce unsupervised updates to the weights.
On the backward pass, we backpropagate.
Since the outputs of each layer are downstream of the outputs of M in the computational graph, we can compute gradients with respect to parameters in both L and M .
Given a gradient ∇ p for each parameter p in L and M , we then perform a supervised update p ← p + A ( p, ∇ p ) .
The key insight is that the convolution of the meta-learner over the weights of the learner forms a fully differentiable framework M (cid:32) L (cid:32) (cid:126)y .
1: procedure T RAIN -E XAMPLE ( L, M, A, (cid:126)x, (cid:126)y ) 2: (cid:126)v 0 ← (cid:126)x (cid:46) Let (cid:126)v i represent the impulse in layer i 3: for weights W (cid:96) , (cid:96)
∈
[1 ... | L | ] do (cid:46)
Forward pass 4: ˆ v (cid:96) +
= σ ( W (cid:96) × (cid:126)v (cid:96) + b (cid:96) )
(cid:46) ˆ v (cid:96) +1 is a placeholder output as input to M 5:
W (cid:96) ← W (cid:96) + M ( (cid:126)v (cid:96) , W (cid:96) ,(cid:126)v (cid:96) +1 ) (cid:46) Updates weight using local rule M 6: (cid:126)v (cid:96) +1 ← σ ( W (cid:96) × (cid:126)v (cid:96) + b (cid:96) )
(cid:46)
Propagate ˆ v (cid:96) +1 as actual layer output 7:
Backpropagate loss H ( (cid:126)v | L | , (cid:126)y ) .
8: for layer weight W (cid:96) in L and M do (cid:46) Backward pass 9:
W (cid:96) ← A (cid:16) ∂H∂W
(cid:96) (cid:17) (cid:46)
Apply gradient update using optimizer A 10: return L, M (cid:46) Return updated learner and updated meta-learner
Results: HAT Improves Performance on Fashion-MNIST
We ﬁnd that the effects of HAT on training are clearly positive.
The median accuracy of the neural nets trained by HAT is clearly increased along the learning curve, and the HAT -group neural nets reach a higher asymptotic value than the control group.
We do note that the two learning curves seem to inﬂect around the same point – HAT does not seem to cause a faster convergence, just a better one.
We attribute this to the meta-learner’s convergence; it may take the meta-learner up to 0.5 epochs to start to have positive effects.
One potential concern with adding unsupervised meta-learner updates is that after the convergence of the base learner L , the meta-learner’s continued output of non-zero updates might “bounce” the base learner out of an optimum.
Remarkably, we see in the above plot that the performance of the
HAT -trained neural nets is quite stable for the entire 18 epochs of post-convergence duration.
To our surprise, we ﬁnd that HAT is more effective when there are more labels, even though the self-supervised component of the algorithm is designed to take advantage of scarce labels.
We attribute this to slow convergence of the meta-learner M – when labels are scarce, the meta-learner may actually converge slower than the learner and thus provide bad update suggestions.
The Behavior of the Meta-Learned Update Rule
We would like insight into why HAT improves the training of neural nets over vanilla gradient descent.
Thus, we will analyze the functional form of the learned update rule M after it has fully converged.
Recall the setting from experiments 1 and 2: we generate 100 pairs of learners and meta-learners: ( L i , M i ) for i ∈ { 1 , ..., 100 } .
We then investigate the pointwise mean M of these meta-learners.
We ﬁrst visualize the dependence of the function M on its inputs ( v i , v j , w ij ) .
We ﬁnd that a remarkably linear dependence on v j explains almost all of the variance in the outputs of the meta-learned update rule.
This indicates that the rule is a “rich-get-richer” scheme: neurons that already ﬁred with high magnitude will experience larger incoming weights and thus be encouraged to ﬁre with high activation in the future.
This linear dependence is surprising since all of the hypothesized rules in neuroscience have a dependence on v i · v j .
As a sanity check, we attempted to directly apply this update rule ( ∆ w ij ≈ 2 · v j ) without meta-learning to see if we can replicate HAT ’s performance improvement.
However, the results were decisively negative – HAT improves performance, but the a priori application of HAT ’s update rule decreases it.
We present three hypotheses:
• Perhaps M learns a good update rule while L is training, then learns a degenerate rule once L has converged.
The sole purpose of this degenerate rule would be to not un-learn the important weights that have already converged (thus explaining the rich-gets-richer behavior of the rule f ( · ) = 2 v j ).
Thus, analyzing the black-box function at epoch 20 is merely the wrong time – perhaps observing the meta-learned rule at epoch 1 would be more insightful and useful.
• Perhaps M learns a good update rule in each run, and these update rules are all complex functions with no good low-order polynomial approximations; however, their pointwise mean (which is itself not a good local update rule) happens to be linear.
Thus, M is the wrong object to analyze and presents behaviors that are not indicative of the results of experiments 1 and 2.
• Perhaps the learning of M is extremely transient.
For any given point in time, there is a different optimal learning rule, and our exercise in ﬁnding a ﬁxed local, unsupervised update rule that is universal across training is futile.
Conclusion
The HAT algorithm demonstrates that local, unsupervised signals can provide performance-improving weight updates.
Neural nets under HAT converge to better asymptotic losses as long as there is sufﬁcient time ( > 0 .
5 epochs) and a sufﬁcient number of labels ( > 20% of the data is labeled).
The latter ﬁnding is surprising since the addition of an unsupervised learning algorithm depends on the presence of labels in order to deliver marginal beneﬁts over gradient descent.
The underlying form of the learned rule that makes HAT successful is still a mystery; we ﬁnd that while the meta-learner may learn a useful update rule during training, the meta-learner does not converge to this useful rule in the long run and instead devolves into a linear function ConvergedRule .
This converged function preserves fully-converged weights by reinforcing incoming weights for neurons with high activations.
5.1 Future Work
The discovery that HAT does not stably converge to a function makes analysis quite difﬁcult.
However, there is potential for future work to do more subtle analyses.
A follow-up to this thesis might be to observe the function f not as a 3-dimensional function in ( v i , w ij , v j ) but rather as a 4-dimensional function in ( v i , w ij , v j , t ) .
Observing the function along the t -axis and checking for phase changes would shed light on whether a single useful update rule is learned during training or whether HAT ’s learning is truly transient and continuous.
Furthermore, extracting the local rules from multiple domains could either ﬁnd that HAT learns a universal rule or that HAT provides an opportunity for transfer learning in unique domains.