Self-Referential Meta Learning
Abstract
Meta Learning automates the search for learning algorithms.
At the same time, it creates a dependency on human engineering on the metalevel, where meta learning algorithms need to be designed.
In this paper, we investigate selfreferential meta learning systems that modify themselves without the need for explicit meta optimization.
We discuss the relationship of such systems to memory-based meta learning and show that self-referential neural networks require functionality to be reused in the form of parameter sharing.
Finally, we propose Fitness Monotonic
Execution (FME), a simple approach to avoid explicit meta optimization.
A neural network selfmodifies to solve bandit and classic control tasks, improves its self-modifications, and learns how to learn, purely by assigning more computational resources to better performing solutions.
1. Introduction
Machine learning is the process of deriving models and behavior from data or environment interaction using humanengineered learning algorithms.
Meta learning takes this process to the meta-level: Its goal is to derive the learning algorithms themselves automatically as well (Schmidhuber, 1987;
Hochreiter et al., 2001; Wang et al., 2016; Duan et al., 2016; Finn et al., 2017; Flennerhag et al., 2019; Kirsch et al., 2019; Kirsch & Schmidhuber, 2020).
Unfortunately, this creates a dependency on human engineering on the metalevel, where researchers now have to design meta learning algorithms.
In this paper, we investigate methods for neural self-referential meta learning (Schmidhuber, 1993b;
Schmidhuber et al., 1997).
In particular, we seek a process of self-improvement that reduces our reliance on human engineering to the largest extent possible.
A central piece in the discussion of self-referential meta learning is the self-referential neural architecture (Schmidhuber, 1993b).
Such an architecture allows the modification not just of some memory to improve future behavior, but also to modify all its own neural weights.
This enables self-modification and self-improvement allowing the architecture to learn, meta-learn, meta-meta-learn, etc.
We show that (1) in order to construct systems that can change all their parameters (or variables more generally), parts of the computational graph need to be reused.
This is done in the form of parameter (variable) sharing.
(2) We discover that memory-based architectures are capable of similar selfimprovement. There is a representational equivalence between neural networks with memory and self-referential architectures. Despite this, we show that self-referential architectures are useful in the absence of meta optimization.
Finally, we propose Fitness Monotonic Execution (FME), a simple approach to avoid explicit meta optimization.
Instead of proposing changes to the model or learning algorithm explicitly, all changes to the model are self-modifications and the resulting solutions are selected for execution more frequently the better their performance.
We empirically demonstrate FME with a neural network that self-modifies to solve bandit and classic control tasks, improves its selfmodifications, and learns how to learn.
2. Background 2.1. Self-referential Architectures
A key requirement for a self-improving system is that it can make self-modifications such that it can change its behavior and learning arbitrarily.
One previously suggested way of achieving this is to bring all variables in a neural network under control of the network itself (Schmidhuber, 1993b).
This is referred to as a self-referential neural architecture.
Compare this to a conventional neural network where there is a subset of variables (called the weights or parameters) that are only updated by a fixed learning algorithm (such as backpropagation).
This entails that part of the (meta- )learning behavior is fixed and needs to be defined by the researcher.
In contrast, self-referential architectures control all variables.
This includes activations (conventionally updated by the neural network itself), weights (conventionally updated by a learning algorithm), meta weights, etc.
In this section, we discuss necessary conditions on such a self-referential architecture and possible implementations.
Notation In the remainder of this paper, we denote external inputs to the neural network as x ∈ R N x (such as observations and rewards in Reinforcement Learning, or error signals in supervised learning), outputs as y ∈ R N y
(e.g. actions in Reinforcement Learning), and the parameters of the neural network as θ .
Further, we denote time-varying variables (memory) as h ∈ R N h (such as the hidden state of an RNN).
We summarize all variables in a neural network as ϕ
= { θ, h, y } .
Necessary Conditions A self-referential architecture ϕ ← g ϕ ( x ) is described by a connected computational graph for the function g that has variables ϕ
= { θ, h, y } (one node per scalar).
The network controls all of its variables in the sense that any element(s) of ϕ can be changed through network actions at every iteration.
This blurs the distinction between activations (memory) h and weights θ .
Computational graphs that fulfill this definition are required to have a certain structure.
At every iteration at least some variables need to be reused in multiple operations (node out-degree > 1 ).
We refer to this as variable sharing, a generalization of weight sharing (Fukushima, 1979) extending beyond classical weights.
Consider a square dense weight matrix.
It consists of Nweights and N activations.
While the activations are timevarying, the weights are source nodes in the computational graph and cannot be directly updated by actions of the network itself.
To change that, we need to derive N 2 variables from N time-varying variables.
This can only be done by reusing some of the same N time-varying variables in multiple operations generating the N 2 variables.
Observation 2.1 .
Variable sharing in self-referential systems.
Assuming a connected computational graph, an architecture that updates all its variables ϕ ∈ R N ϕ in iteration t needs to reuse elements of ϕ multiple times in the computational graph to generate
ϕ t +1 ∈ R N ϕ from ϕ t ∈ R N ϕ .
Proof.
As there are no more elements in ϕ t than there are in ϕ t +1 , any operation generating an element in ϕ t +1 that makes use of more than one element in ϕ t needs to reuse an element already in use by a different operation.
Implementations Under the previous constraints, various implementations for self-referential neural architectures are conceivable.
Schmidhuber (1993b) assigns an address to each weight such that the network outputs can be used to attend to weights and both read and write their values.
Instead of updating one weight at a time, the fast weight programmers of 1992-93 (Schmidhuber, 1992; 1993a) are networks that learn to generate key and value patterns to rapidly change many fast weights simultaneously.
Outer products between activations (a type of sharing) are used to derive M ∗ N variables, M, N ∈ N , from M + N variables.
This allows updating all the weights of a neural network layer by its own activations (Irie et al., 2021).
Alternatively, a coordinate-wise mechanism may generate all updates continuously as a function of the weight address (D’Ambrosio & Stanley, 2007).
Other works have used multiple RNNs with shared weights and messaging passing between those to increase the number of time-varying variables h arbitrarily while keeping the number of parameters θ constant (Rosa et al., 2019; Kirsch & Schmidhuber, 2020).
This can be made self-referential by using a subset of h to update parameters θ .
2.2. Expressivity of Memory and Self-referential Architectures
In this section, we show that self-referential architectures do not have a representational advantage over memorybased architectures when the free (initial) variables are metaoptimized using a human engineered learning algorithm.
We defined self-referential architectures ϕ, y ← g ϕ ( x ) as those that can update all their variables ϕ in the computational graph.
Compare this to a memory architecture such as a recurrent neural network h, y ← f θ ( h, x ) parameterized by θ where h corresponds to its hidden state (memory).
Can the self-referential architecture represent any functions that the memory architecture can not?
A commonly used intuition (Schmidhuber, 1993b) is that self-referential architectures are self-modifying, in that they change their own weights, affecting not only their outputs and current weights but also future weight changes through g ϕ .
These architectures can thus not only learn, but also meta-learn, meta-meta-learn, etc.
While memory architectures do not update their weights, they are also self-modifying.
Changes in memory h affect the output directly, but also the effective function f θ by modifying its input h , in turn determining future changes to h .
Observation 2.2 .
For any self-referential architecture ϕ, y ← g ϕ ( x ) and some initial ϕ
we can find a memory architecture h, y ← f θ ( h, x ) , θ , and initial h 0 such that for any sequence of x 1: T we have ˆ f ( x 1: T )
=
ˆ g ( x 1: T ) where ˆ f and ˆ g are the unrolls returning y 1: T of f and g respectively.
Proof.
We construct an emulator f θ (a sufficiently large neural network parameterized by θ ) that stores ϕ in h and set θ, h 0 such that at each step t ∈ N it performs the same computation as g ϕ by taking ϕ t from h t , computing ϕ t +1 , and storing it in h t +1 .
Furthermore, any memory-based architecture can be represented by a self-referential architecture where a subset of variables is updated by the identity function.
In conclusion, the function class that can be represented by self-referential architectures is equivalent to memory architectures, given sufficiently rich parameterizations.
For both memory architectures and self-referential architectures the same question arises: How do we set the free variables θ , h 0 , or ϕ 0 ?
If these free variables are directly optimized (eg by following the gradient of some objective), from a representability perspective there is no advantage of self-referential architectures.
Orthogonal to this, the chosen architecture (whether memory or self-referential) may have varying optimizability or regularizing benefits due to e.g. sparsity in the computational graph, multiplicative interactions, or variable sharing.
In the following, we discuss how self-referential architectures are relevant in the absence of direct meta-optimization.
3. Method: Fitness Monotonic Execution
Both in the case of self-referential and memory architectures the free (initial) variables need to be found.
Here we propose a method, Fitness Monotonic Execution (FME), that avoids explicit meta optimization of these free variables.
Instead of modifying ϕ directly using a human-engineered learning algorithm, we simply select between different configurations of ϕ that are generated using self-modifications.
In particular, through interactions with the environment we continuously add new solutions to a set of Φ = { ϕ i } .
Computation time is distributed across solutions monotonic in their performance, i.e. better performing solutions are executed longer (or are selected for execution more frequently).
This can be formalized as a pmf p ( ϕ ) that assigns each solution ϕ
∈
Φ a probability for being executed at any given time-step based on its average reward R ( ϕ ) ∆ t relative to other solutions (where ∆ t is the solution’s total lifetime).
As a special case, p may put all probability mass on the current best solution, greedily selecting for improvement.
See Algorithm 1 for a full description.
Note that in this scheme, memory architectures are now inadequate.
If there were any variables that are not modifiable, their value could not be determined through selfmodifications.
As we do not use any human-engineered optimization process, their value would be undefined.
Thus, self-referential architectures are required.
Algorithm
Require: Initial solution(s) Φ =
{ ϕ i } , self-referential architecture g ϕ , probability p ( ϕ ) , an RL environment E while forever do ϕ ∼ p ( ϕ ) where ϕ ∈ Φ ▷ Sample next solution to execute, monotonic in its performance ϕ, y 1: L ← g Lϕ ( x 1: L ) ▷ Execute g for L steps with x 1: L from the environment E including a feedback signal Φ ← Φ ∪ { ϕ } ▷ Add new ϕ to Φ
Least-recently-used Buffer To limit the number of solutions we need to store, we implement Fitness Monotonic
Execution with a least-recently-used (LRU) buffer.
It consists of m buckets where each bucket holds recent solutions in a specific performance range.
Solutions from buckets with higher performance are sampled exponentially more frequently.
Outer-product-based
Architecture For the selfreferential neural network architecture, we chose an outer-product mechanism adapted from prior work (Irie et al., 2021).
By applying a weight matrix W t − 1 ∈ R N x × ( N y +2 N x +4) to some input x t ∈ R N x we generate the output y t ∈ R N y , key k t ∈ R N x , query q t ∈ R N x , and a learning rate β t ∈ R 4 .
Using an outer-product, the key and query generate an update to the weight matrix W t − 1 , obtaining W t : y t , k t , q t , β t = W t
[Equation]
ψ (
[Equation]
[Equation] [Equation] [Equation] where ψ is the tanh activation, σ is the sigmoid function, and ⊗ is the outer product.
The learning rate β t ∈ R 4 controls the rate of update to the four parts generating y, k, q, β .
We stack multiple such self-referential layers.
4. Experiments
We empirically investigate several questions: Firstly, starting with a randomly initialized solution, can the network modify itself to solve a bandit task?
We compare this to a hill climbing strategy.
Secondly, how do self-modifications compare when solving a markov decision process?
Thirdly, given a bandit task that is non-stationary, can the network learn to modify itself based on the reward it receives as input? [Figure]
[Figure] Learning a Bandit Policy The first question we investigate is whether a randomly initialized self-referential architecture is capable of making self-modifications that lead to a useful policy for a given task.
We test this on a simple 2-armed bandit where one arm gives payouts (rewards) ofand the other 0 .
From Figure 1 (left) we observe that after around 40 self-modifications and selections a solution is found that always selects the arm with a higher payoff.
We compare this to hill climbing with a variance-tuned Gaussian noise on the network parameters.
We observe that in this simple environment, Fitness Monotonic Execution is as effective as hill climbing to find an optimal solution to this bandit problem
[Figure]
Cartpole Next, we increase the difficulty of the policy to be found by running a self-referential network on the
Cartpole task (Figure 1, right).
We observe that reaching a good performing policy takes significantly more selfmodifications and selections.
At the same time, a simple hill climbing strategy (with tuned noise) fails at improving the policy at the same rate as the self-modifying architecture.
This suggests that we are not only selecting for good policies but also strategies for self-modification that lead to policy improvement in the future.
Meta Learning a Bandit Learning Algorithm Given a non-stationary task, a good policy can not exhibit a fixed behavior but must adapt to changing rewards (learn).
We test the capabilities of Fitness Monotonic Execution to adapt to a changing bandit task.
In Figure 2 we swap the good and bad arm at random intervals.
We further feed the reward as an input to the policy such that it can adapt its behavior based on the reward.
We observe that Fitness Monotonic
Execution leads to self-modifying policies that change their action (learn) in response to the reward they previously received.
In contrast, if this reward is not fed as an input, the policy fails to adapt.
5. Discussion
Limitations This paper represents an initial discussion of self-referential systems that do not rely on fixed humanengineered meta-optimization.
The empirical evaluation is still minimalistic at this time but should be a good starting point for larger experiments and improvements.
Our intention is to incite further interest in this research direction.
Semantically, it is an open question whether FME should be called a (minimalistic) meta optimizer after all.
Usually, optimizers define how solutions are modified based on a feedback signal.
In the case of FME, these modifications are self-generated.
Broader impact This paper is not directly concerned with applications of machine learning.
Nevertheless, meta learning methods may pose additional challenges in the future.
For instance, learning algorithms are now also subject to learning which means that learning itself becomes more difficult to interpret and to monitor for biases extracted from data.
Self-modifying architecture
We described a meta learner that can self-modify all its variables including those that define the self-modifications, but its architecture is still hard-coded.
In fitness monotonic execution, the selfmodifications do not require differentiability.
Thus, selfmodifications can be extended to include architecture modifications ϕ, y, g ← g ϕ ( x ) , such as adding or removing neurons and weights, changing operations, and (un-)sharing variables.
6. Conclusion
In this paper, we discussed self-referential systems that exhibit self-improvement while reducing the reliance on human engineering to the largest extent possible.
In particular this means avoiding the use of human engineered learning algorithms on the meta level.
We showed that in order to construct systems that can change all their parameters (or variables more generally), functionality needs to be reused.
This is done in the form of parameter (variable) sharing.
We further demonstrated the representational equivalence between neural networks with memory and self-referential architectures while highlighting the benefit of self-referential architectures in the absence of meta optimization.
Finally, we proposed Fitness Monotonic Execution (FME), a simple approach to avoid explicit meta optimization.
A neural network self-modifies to solve bandit and classic control tasks, improves its self-modifications, and learns how to learn, purely by assigning more computational resources to better performing solutions. Acknowledgements
This work was supported by the ERC Advanced Grant (no: 742870) and computational resources by the Swiss National
Supercomputing Centre (CSCS, projects s978, s and s1127).
We also thank NVIDIA Corporation for donating several DGX machines as part of the Pioneers of AI Research Award, IBM for lending a Minsky machine and weights & biases (Biewald, 2020) for their great experiment tracking software and support.
A. Implementation Details
Least-recently-used Buffer We initialize a least-recently-used (LRU) buffer with a single randomly initialized neural network.
The m = 100 buckets evenly cover the entire current performance range and each hold the 100 most recent solutions.
Solutions from buckets with higher performance are sampled exponentially more frequently.
We use an exponential base of e 20 .
All layers are initialized from a truncated normal with a standard deviation of σ = 1 √ N x .
Architecture We stack three self-referential layers with 32 hidden units.
Sources of Randomness To create a temporal tree of self-modifying solutions, randomness must be injected into the system.
This randomness originates from the policy action sampling, non-deterministic environment steps, and potential external noise injection as an input to the policy.
We found external noise injection not to improve the agent’s performance when sufficient randomness originates from the policy and environment.