Original: M ETA -L EARNING THE I NDUCTIVE B IASES OF S IMPLE N EURAL C IRCUITS
Revised: M ETA -L EARNING THE I NDUCTIVE B IASES OF S IMPLE N EURAL C IRCUITS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Animals receive noisy and incomplete information, from which we must learn how to react in novel situations.
Revised: Animals receive noisy and incomplete information, from which we must learn how to react in novel situations.

Original: A fundamental problem is that training data is always finite, making it unclear how to generalise to unseen data.
Revised: A fundamental problem is that training data is always finite, making it unclear how to generalise to unseen data.

Original: But, animals do react appropriately to unseen data, wielding Occam’s razor to select a parsimonious explanation of the observations.
Revised: But, animals do react appropriately to unseen data, wielding Occam’s razor to select a parsimonious explanation of the observations.

Original: How they do this is called their inductive bias, and it is implicitly built into the operation of animals’ neural circuits.
Revised: How they do this is called their inductive bias, and it is implicitly built into the operation of animals’ neural circuits.

Original: This relationship between an observed circuit and its inductive bias is a useful explanatory window for neuroscience, allowing design choices to be understood normatively.
Revised: This relationship between an observed circuit and its inductive bias is a useful explanatory window for neuroscience, allowing design choices to be understood normatively.

Original: However, it is generally very difficult to map circuit structure to inductive bias.
Revised: However, it is generally very difficult to map circuit structure to inductive bias.

Original: In this work we present a neural network tool to bridge this gap.
Revised: In this work we present a neural network tool to bridge this gap.

Original: The tool allows us to meta-learn the inductive bias of neural circuits by learning functions that a neural circuit finds easy to generalise, since easy-to-generalise functions are exactly those the circuit chooses to explain incomplete data.
Revised: The tool allows us to meta-learn the inductive bias of neural circuits by learning functions that a neural circuit finds easy to generalise, since easy-to-generalise functions are exactly those the circuit chooses to explain incomplete data.

Original: We show that in systems where the inductive bias is known analytically, i.e. linear and kernel regression, our tool recovers it.
Revised: We show that in systems where the inductive bias is known analytically, i.e. linear and kernel regression, our tool recovers it.

Original: Then, we show it is able to flexibly extract inductive biases from differentiable circuits, including spiking neural networks.
Revised: Then, we show it is able to flexibly extract inductive biases from differentiable circuits, including spiking neural networks.

Original: This illustrates the intended usecase of our tool: understanding the role of otherwise opaque pieces of neural functionality, such as non-linearities, learning rules, or connectomic data, through the inductive bias they induce.
Revised: This illustrates the intended use of our tool: understanding the role of otherwise opaque pieces of neural functionality, such as non-linearities, learning rules, or connectomic data, through the inductive bias they induce.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Generalising to unseen data is a fundamental problem for animals and machines: you receive a set of noisy training data, say an assignment of valence to the activity of a sensory neuron, and must fill in the gaps to predictthe valence at other activity values, figure 1A. This is hard because, without prior assumptions, it is completely underconstrained.
Revised: Generalising to unseen data is a fundamental problem for animals and machines: you receive a set of noisy training data, say an assignment of valence to the activity of a sensory neuron, and must fill in the gaps to predict valence from activity, Fig. 1A. This is hard since, without prior assumptions, it is completely underconstrained.

Original: Many explanations or hypothesescan perfectly fit any set of data (Hume, 1748),Fig 1A, but different choices will lead to wildly different outcomes.
Revised: Many explanations or hypotheses perfectly fit any dataset (Hume, 1748), but different choices will lead to wildly different outcomes.

Original: Further, the training datathat you receive is likely noisy; the vital task of sifting the signal from the noisein a training set can heavily influence generalisation to unseen data, figure 1B.
Revised: Further, the training data is likely noisy; how you choose to sift the signal from the noise can heavily influence generalisation, Fig. 1B.

Original: Generalising requires prior assumptions aboutthe likely explanations of the data.
Revised: Generalising requires prior assumptions about likely explanations of the data.

Original: For example, prior belief that small changes inthe neural activityshould lead to correspondingly small changes in valence would bias you towards smoother explanations of the data, breaking the tie between options 1 and 2 in figure 1A. It is a circuit’s inductive bias that leads it to choose certain, otherwise similarly well-fitting, explanationsof the data, over others.
Revised: For example, prior belief that small changes in activity lead to correspondingly small changes in valence would bias you towards smoother explanations, breaking the tie between options 1 and 2 in Fig. 1A. It is a learner’s inductive bias that chooses certain, otherwise similarly well-fitting, explanations over others.

Original: The inductive bias of a learning algorithm, such as a neural network, can be a powerful route to understanding in both Machine Learning and Neuroscience.
Revised: The inductive bias of a learning algorithm, such as a neural network, can be a powerful route to understanding in both Machine Learning and Neuroscience.

Original: Classically, the success of convolutional neural networks can be attributed to their explicit inductive bias towards translation-invariant classifications (LeCun et al., 1998), and these ideas have since been very successfully extended to networks with a range of structural biases (Bronstein et al., 2021).
Revised: Classically, the success of convolutional neural networks can be attributed to their explicit inductive bias towards translation-invariant classifications (LeCun et al., 1998), and these ideas have since been very successfully extended to networks with a range of structural biases (Bronstein et al., 2021).

Original: Further, a wide variety of network features have been linked to implicit regularisation of the network, such as: the stochasticity of SGD (Mandt et al., 2017), parameter initialisation (Glorot & Bengio, 2010), early stopping (Hardt et al., 2016),  low rank biases of gradient descent Gunasekar et al.(2017), or the discretisation of gradient steps (Barrett & Dherin, 2020).
Revised: Further, many network features have been linked to implicit regularisation of the network, such as the stochasticity of SGD (Mandt et al., 2017), parameter initialisation (Glorot & Bengio, 2010), early stopping (Hardt et al., 2016), or low rank biases of gradient descent Gunasekar et al.

Original: In neuroscience, the inductive bias has been used to give normative explanations for representational or structural choices via their effect on generalisation.
Revised: In neuroscience, the inductive bias has been used to assign normative roles to representational or structural choices via their effect on generalisation.

Original: For example, there has been recent work on neural network models of the cerebellumthat has  shown that the threshold of the network’s nonlinearity has a strong effect on its ability to generalise functions with different frequency content (Xie et al., 2022).
Revised: For example, the non-linearity in neural network models of the cerebellum has been shown to have a strong effect on the network’s ability to generalise functions with different frequency content (Xie et al., 2022).

Original: It is experimentally observed that these network properties vary across the cerebellum,and hence this work suggests that each part of the cerebellum may be tuned to tasks with particular smoothness properties.
Revised: Experimentally, these network properties vary across the cerebellum, hence this work suggests that each part of the cerebellum may be tuned to tasks with particular smoothness properties.

Original: This is exemplary of a spate of recent papers that have applied similar techniques to visual representations (Bordelon et al., 2020; Pandey et al., 2021), mechanosensory representations (Pandey et al., 2021), and the olfactory system (Harris, 2019).
Revised: This is exemplary of a spate of recent papers applying similar techniques to visual representations (Bordelon et al., 2020; Pandey et al., 2021), mechanosensory representations (Pandey et al., 2021), and olfaction (Harris, 2019).

Original: Despite the potentialutility of  inductive biasas a way to understand neural circuits, the approach is limited, since mapping between learning algorithms and their inductive bias is highly non-trivial.
Revised: Despite the potential of using inductive bias to understand neural circuits, the approach is limited, since mapping from learning algorithms to their inductive bias is highly non-trivial.

Original: Numerousaspects of a circuit  (learning rules, architecture, non-linearities, etc.)will influence the way it chooses to generalise.
Revised: Numerous circuit features (learning rules, architecture, non-linearities, etc.) influence generalisation.

Original: For example, training two simple ReLU networks of different depth to classify three data points leads tofundamentally different generalisations for non-obvious reasons, figure 1C.
Revised: For example, training two simple ReLU networks of different depth to classify three data points leads to different generalisations for non-obvious reasons, Fig. 1C.

Original: Ina few constrained cases analytic bridges have been built that map a learning algorithm to its inductive bias.
Revised: In constrained cases analytic bridges have mapped learning algorithms to their inductive bias.

Original: Most notably, work has characterised the inductive bias of kernel regression, an algorithm that maps data points to a feature space in which linear regression to labels is then performed (Sollich, 1998; Bordelon et al., 2020;
Revised: In particular, the study of kernel regression, an algorithm that maps data points to a feature space in which linear regression to labels is then performed (Sollich, 1998; Bordelon et al., 2020;

Original: Simon et al., 2021). This approach has been influential; all the cited examples of understanding neuroscientific findings viatheir induced inductive bias have used thisanalytic bridge.
Revised: Simon et al., 2021), has been influential: all the cited examples of understanding in neuroscience via inductive bias have used this bridge.

Original: However, it severely limits theapplicability of this approach: most biological circuits cannot be well approximated as performing a fixed feature map then linearly regressing to labels!
Revised: However, it severely limits the approach: most biological circuits cannot be well approximated as performing a fixed feature map then linearly regressing to labels!

Original: Here, we develop a flexible neural network approach that is able to meta-learn the inductive bias of essentially any differentiable supervised learning algorithm.
Revised: Here, we develop a flexible neural network approach that is able to meta-learn the inductive bias of essentially any differentiable supervised learning algorithm.

Original: It follows a meta-learning framework (Vanschoren, 2019): an outer neural network (the meta-learner) assigns labels to a dataset, this labelled dataset is then used in the inner optimisation to train the inner neural network (the learner).
Revised: It follows a meta-learning framework (Vanschoren, 2019): an outer neural network (the meta-learner) assigns labels to a dataset, this labelled dataset is then used in the inner optimisation to train the inner neural network (the learner).

Original: The meta-learner is then trained on a meta-loss which measures the generalisation error of the learner to unseen data.
Revised: The meta-learner is then trained on a meta-loss which measures the generalisation error of the learner to unseen data.

Original: Through gradient descent on the meta-loss, the meta-learner meta-learns to label data in a way that the learner finds easy to generalise.
Revised: Through gradient descent on the meta-loss, the meta-learner meta-learns to label data in a way that the learner finds easy to generalise.

Original: These easy-to-generalise functions form a description of  inductive bias.
Revised: These easy-to-generalise functions form a description of the inductive bias.

Original: If a function is easy to generalise that implies that when the network receives a few training points labelled according to this function, it chooses to generalise them in the appropriate manner. They are therefore functions the network would regularly use  to explain finite datasets, i.e. if function is easy-to-generalise for a given network then the network is inductively biased towards that function.
Revised: In other words, if the network receives a few training points from this function it will generalise appropriately, and generally the network will regularly use this function to explain finite datasets.

Original: To our knowledge, the most related work is Li et al.
Revised: To our knowledge, the most related work is Li et al.

Original: Li et al.
Revised: Li et al.

Original: create a set of neural networks, trained or untrained, then attempt to summarise this set via a distribution over labels as a function of input.
Revised: view sets of neural networks, trained or untrained, as a distribution over the mapping from input to labels.

Original: To do this, they meta-learn the parameters of a gaussian process, which assigns a  distributionof labels to each input.
Revised: They fit this distribution by meta-learning the parameters of a gaussian process which assigns a label distribution to each input.

Original: This provides an interpretable way to understand the behaviour of  sets of networks, rather than our work which characterises the inductive bias of learning algorithms through the functions they find easy to generalise.
Revised: This provides an interpretable summary of fixed sets of network. In our work we do something very different: rather than focusing on a fixed, static set of networks, we find the inductive biases of learning algorithms via meta-learning easily learnt functions.

Original: We recommend potential users to consider both approaches depending on their application.

Original: In the following sections we describe our scheme, and verify that itworks by comparing to the known inductive biases of linear and kernel regression.
Revised: In the following sections we describe our scheme, and validate it by comparing to the known inductive biases of linear and kernel regression.

Original: We then extend it in several ways.
Revised: We then extend it in several ways.

Original: First, networks are inductively biased towards areas of function space, not single functions.
Revised: First, networks are inductively biased towards areas of function space, not single functions.

Original: To explore this, we learn a set of orthogonal functions that a given network finds easy to learn, hence providing a richer characterisation of the inductive bias.
Revised: Therefore we learn a set of orthogonal functions that a learner finds easy to generalise, providing a richer characterisation of the inductive bias.

Original: Second, we introduce a setup for gaining insight into the way a given design choice (architecture, learning rule, non-linearity) effects the inductive bias.
Revised: Second, we introduce a framework that asks how a given design choice (architecture, learning rule, non-linearity) effects the inductive bias.

Original: We do this by assembling two networks that differ only by the design choice in question; then we meta-learn a function that one network finds much easier to generalise than the other, which might allow a user to answer why a particularfeature of the circuit  is present.
Revised: To do that, we assemble two networks that differ only by the design choice in question, then we meta-learn a function that one network finds much easier to generalise than the other. This can be used to explain why a particular circuit feature is present.

Original: Wewill again verify both of these techniques against algorithms with known inductive biases.
Revised: We again validate both schemes against linear and kernel regression.

Original: Finally we will apply our approach to more adventurous settings. Simple networks trained by gradient descent; and spiking neural networks, trained using the surrogate gradient method (Neftci et al., 2019; Zenke & Vogels, 2021), a recent major advance in gradient-based training of spiking networks. In both cases we show our tool can be used to examine the networks’ inductive biases.
Revised: Finally we show our tool’s flexibility in a series of more adventurous examples: we validate it on a challenging differentiable learner (a spiking neural network); we show it works in high-dimensions by meta-learning MNIST labels; and we highlight its explanatory power for neuroscience by using it to normatively explain patterns in recent connectomic data via their inductive bias.

Original:  A N EURAL N ETWORK TO M ETA -L EARN I NDUCTIVE B IAS
Revised:  A N EURAL N ETWORK TO M ETA -L EARN I NDUCTIVE B IASES

Original: Our main contribution is a meta-learning framework for extracting the inductive bias of differentiable learning algorithms, figure 2A, that wewill describeand test in this section.
Revised: Our main contribution is a meta-learning framework for extracting the inductive bias of differentiable learning algorithms, Fig. 2A, that we describe in this section.

Original: The outer-loopis a neural network, the meta-learner,that assigns labels to inputs sampled from some distribution, hence creating the real-world function that our circuit of interest will try to learn.
Revised: In the outer-loop a neural network, the meta-learner, assigns labels to input sampled from some distribution, hence creating the real-world function that our circuit of interest will try to learn.

Original: The inner-loop learning algorithm, the learner, is the circuit whose inductive bias we would like to extract, for example a model of a biological sensory processing circuit that assigns valences to inputs.
Revised: The inner-loop learning algorithm, the learner, is the circuit whose inductive bias we want to extract; for example, a biological sensory processing circuit that assigns valences to inputs.

Original: When provided with a training dataset of inputs and labels the learner adjusts its parameters according to its internal learning rules.
Revised: When provided with a training dataset of inputs and labels the learner adjusts its parameters according to its internal learning rules.

Original: Then the generalisation error of the trained learner is measured on a held-out test set, and this is used as the meta-loss to train the meta-learner.
Revised: Then the generalisation error of the trained learner is measured on a held-out test set, and this is used as the meta-loss to train the meta-learner.

Original: This process repeats, retraining the learner at every iteration and iteratively developing the weights of the meta-learner, until the meta-learner isperforming a labellingfor which the learner has a low generalisation error after training on a small number of datapoints (we used around 30).
Revised: This process repeats, retraining the learner at every iteration and iteratively developing the meta-learner’s weights, until the meta-learner is labelling the data in a way that the learner finds easy to generalise after training on a few datapoints (we used around 30).

Original: Thus, the meta-learner has extracted a function towards which the learner is inductively biased.
Revised: Thus, the meta-learner has extracted a function towards which the learner is inductively biased.

Original: As just outlined, the meta-learner will find the function that the learner finds easiest to generalise, which in most cases is the trivial function where all inputsare assigned the same label (similar to GAN mode collapse (Lala et al., 2018)).
Revised: As just outlined, the meta-learner will find the easiest-to-generalise function, usually the one that assigns all inputs the same label.

Original: To avoid this, wewill introduce a term in the metaloss that forces the distribution of labels to take a particular (non-constant) form.
Revised: To avoid this trivial function, we introduce a term in the meta-loss that forces the distribution of labels to take a particular (non-constant) form.

Original: Specifically, we introduce a term in the meta-loss that measures the Sinkhorn divergence between the assigned label distribution and a uniform distribution between − 1 and 1 .
Revised: Specifically, it measures the Sinkhorn divergence between the meta-learner’s label distribution and a uniform distribution from − 1 to 1 (other divergences also work, Appendix B).

Original: The fullalgorithm pseudocode isshown in Algorithm 1.
Revised: The full pseudocode is in Algorithm 1.

Original: 1 Initialise metalearner: f θ ( x ) 2 while Step count < Total do Generate dataset from input distribution: x ∼ p ( x ) Label using metalearner: z = f θ ( x )
Revised: to be uniform from − 1 to 1 .

Original: Split inputs and labels into test and train datasets: D Tr & D Te Train leaner using D Tr giving trained learner network: g ( x ) Label D Te using trained learner: ˆ z = g x ) Compute the generalisation error the
Revised: B: The meta-learner learns a linearly separable labelling of data sampled from a circle for a ridge regression learner.

Original: : L g i
Revised: C:

Original: Compute the Sinkhorn Divergence of metalearner’s labels from 1 , 1] : L d Take θ gradient step on meta-loss: L = L g + L d 11 end
Revised: For a kernel regression learner and data sampled from the surface of a sphere, the meta-learner’s labelling is very close to the predicted spherical harmonic ( 99% of norm within first order harmonics).

Original: We now perform a couple of tests to show this schemeworks by meta-learning sensible functions for linear and kernel regression, whose inductive biases are understood.
Revised: We validate our scheme by meta-learning sensible functions for linear and kernel learners, whose inductive biases are known.

Original: First, for ridge regression on data sampled from a 2D circle the meta-learner assigns linearly separable labels to the data, figure 2B; exactly the labelling to which the simple linear circuit can easily generalise.
Revised: First, for ridge regression on data sampled from a 2D circle the metalearner assigns linearly separable labels, Fig. 2B; exactly the labels linear circuits easily generalise.

Original: Next, we meta-learn kernel ridge regression’s  bias.
Revised: Next, we meta-learn kernel ridge regression’s inductive bias.

Original: Kernel regression involves projecting the input data through some fixed mapping to a feature space (e.g. the last hidden layer of a fixed neural network) and performing linear regressionto the labels fromthe feature space.
Revised: Kernel regression involves projecting the input data through a fixed mapping to a feature space (e.g. the last hidden layer of a fixed neural network) and performing linear regression from feature space to labels.

Original: Bordelon et al.
Revised: Bordelon et al.

Original: (2020) have shown that the inductive bias of kernel regressionalgorithms can be understood through the kernel eigenfunctions ( { v i ( x ) } with eigenvalue { λ i } ).
Revised: (2020) show that the inductive bias of kernel regression can be understood through the kernel eigenfunctions ( { v i ( x ) } with eigenvalue { λ i } ).

Original: These are defined onan input distribution p ( x ) via a kernelfunction k ( x , x ′ ) that measures  similarity  in features space as:
Revised: These are defined on input distribution p ( x ) via a kernel k ( x , x ′ ) that measures the similarity of two inputs in feature space:

Original: [Equation]
Revised: [Equation]

Original: Thehigher the eigenvalue the more inductively biased the kernel regression algorithm is  towards this eigenfunction, i.e. the fewer training points  needed to reach a given generalisation error when trained to fit this eigenfunction.
Revised: The algorithm is inductively biased towards higher eigenvalue eigenfunctions; i.e., fewer training points are needed to reach a given generalisation error when fitting high vs. low eigenvalue eigenfunctions.

Original: More general functions can be understood by how much they project onto high and low eigenvalue eigenfunctions.
Revised: General functions can be understood by projecting onto the eigenbasis.

Original: This tells us that our meta-learner, in searching forthe nonconstant function that kernel regression finds easiest to generalise, should pick out theeigenfunction with highest eigenvalue.
Revised: Hence our meta-learner, in searching for kernel regression’s easiest-to-generalise non-constant function, should choose the highest eigenvalue eigenfunction.

Original: To test this we meta-learn the inductive bias of a two-layer neural network with fixed first layer weights.
Revised: To test this, we meta-learn the inductive bias of a two-layer neural network with fixed first layer weights.

Original: We sample data uniformly from the sphere and randomly connect a large hidden layer of
Revised: We sample data uniformly from the sphere and randomly connect a large hidden layer of

Original: ReLU neurons to the three input neurons.
Revised: ReLU neurons to the three input neurons.

Original: The elements of this random weight matrix are drawn iid from a standard normal, and the learning algorithmthen performs ridge regression on the hidden layer activities.
Revised: The elements of this random weight matrix are drawn iid from a standard normal, and the learning algorithm performs ridge regression on the hidden layer activities.

Original: Previous work has analytically derived the kernel for this network and  it’s eigenfunctions (Cho & Saul, 2009; Mairal & Vert, 2018), which are spherical harmonics.
Revised: Previous work has analytically derived the kernel for this network, and computed it’s eigenfunctions (Cho & Saul, 2009; Mairal & Vert, 2018), which are spherical harmonics.

Original: The higher the frequency of the spherical harmonics the lower its eigenvalue.
Revised: The higher the frequency of the spherical harmonics the lower its eigenvalue.

Original: Matching this, our network metalearns one of the set of lowest frequency spherical harmonics, figure 2C.
Revised: Matching this, our network meta-learns one of the set of lowest frequency spherical harmonics, Fig. 2C.

Original:  M ETA -L EARNING MULTIPLE FUNCTIONS
Revised:  M ETA -L EARNING A REAS OF F UNCTION S PACE

Original: Having shown our toolworks on some simple test cases, we now extend it to find a richer characterisation of the inductive bias.
Revised: Having validated our tool on some simple test cases, we now extend it to find a richer characterisation of the inductive bias.

Original: A given learning algorithm is inductively biased towards areas of function space, not just one particular function.
Revised: A given learning algorithm is inductively biased towards areas of function space, not just one particular function.

Original: In order to gain access to this larger space we learn a series of meta-learners.
Revised: To gain access to this larger space, we learn a series of metalearners.

Original: The first of these is exactly as described above, it is trained with a meta-loss that measures the generalisation error and the sinkhorn divergence of the label distribution from uniform. Then we iteratively introduce additional meta-learners.
Revised: The first of these is exactly as described above, then we iteratively introduce additional meta-learners.

Original: To ensure each meta-learner learns a new aspect of the inductive bias we add a term to the meta-loss that penalises the square of the dot product between the current meta-learner’s labelling and that of all the previously trained meta-learners, figure 3A. I.e. for the orthogonality loss between f θ ( x ) and a previously trained meta-learner h ( x ) we sample a set of data and compute,
Revised: To ensure each meta-learner learns a new aspect of the inductive bias we add a term to the meta-loss that penalises the square of the dot product between the current meta-learner’s labelling and that of all the previously trained meta-learners, Fig. 3A. On a dataset { x n } :

Original: [Equation]
Revised: [Equation]

Original: This forces eachsubsequent learner to discover a function that is orthogonal to all previous functions, but is still relatively easy for the learner to generalise.
Revised: for each previous meta-learners f θ i ( x ) and the current meta-learner f θ ′ ( x ) . Thus, each learner must discover an easy-to-generalise function that is orthogonal to all previous functions.

Original: We again check that this scheme is working by comparing to learning algorithms with known inductive biases.
Revised: We again validate this scheme against linear and kernel regression.

Original: When tested on linear regression of 2D data the meta-learners learn two orthogonal linearly separable sets of labels, thenfinds a third orthogonal function that the learner struggles to generalise, as expected, figure 3B. We then test on the same kernel regression network we described previously.
Revised: When tested on linear regression of 2D data the meta-learners learn two orthogonal linearly separable labellings, then a third orthogonal function that the learner struggles to generalise, as expected, Fig. 3B. We then test on the same kernel regression network we described previously.

Original: Theory predicts that the meta-learners should learn the eigenfunctions in decreasing order of their eigenvalue, and we find that this is true to a good approximation, figure 3C, learning approximations to the three first order spherical harmonics, and then three approximations to second order spherical harmonics.
Revised: Theory predicts that the meta-learners should learn the eigenfunctions in decreasing order of their eigenvalue, and we find that this is true to a good approximation, Fig. 3C, learning approximations to the three first order spherical harmonics, and then three approximations to second order spherical harmonics.

Original: For linear classifications (e.g. linear and kernel regression) the full set of orthogonal functions explains the entire inductive bias, since the average generalisation error of any target function is the sum of the generalisation errors of each the kernel eigenfunctions weighted by projection of the target function onto that eigenfunction.
Revised: For linear classifiers (e.g. linear and kernel regression), the full set of orthogonal functions explains the entire inductive bias, since the average generalisation error of any target function is the sum of the generalisation errors of each the kernel eigenfunctions weighted by projection of the target function onto that eigenfunction.

Original: This will not in general be true, since for a non-linear classifier the generalisation error on a function f ( x ) =
Revised: This will not in general be true, since for a non-linear classifier the generalisation error on a function f ( x ) =

Original: f a ( x ) + f b ( x ) does not equal the generalisation error on f a ( x ) plus that on f b ( x ) .
Revised: f a ( x ) + f b ( x ) does not equal the generalisation error on f a ( x ) plus that on f b ( x ) .

Original: Nonetheless, we expect the set of orthogonal functions will still be a helpful guide to a network’s inductive bias, even for non-linear classifiers.
Revised: Nonetheless, we expect the set of orthogonal functions will still be a helpful guide to a network’s inductive bias, even for non-linear classifiers.

Original:  F INDING THE E FFECT OF D ESIGN C HOICES ON THE I NDUCTIVE B IAS
Revised:  F INDING THE E FFECT OF D ESIGN C HOICES ON THE I NDUCTIVE B IAS

Original: Our work is motivated by the desire to understand how design choices in learning algorithmssuch as architecture, learning rule, and non-linearities - lead to downstream generalisation effects, particularly in biologicalneural networks.
Revised: Our work is motivated by the desire to understand how design choices in learning algorithmssuch as architecture, learning rule and non-linearities - lead to downstream generalisation effects, particularly in biological networks.

Original: The ideas we have presented are already capable of answering many of these questions.

Original: However, one additional setting which we have found useful is to compare two networks with some architectural difference between them, and learn functions that one of the networks finds much easier to generalise than the other.
Revised: One additional setting which we have found useful is to compare two networks with some architectural difference between them, and learn functions that one of the networks finds much easier to generalise than the other.

Original: In this way, you can build intuition for the impact of design features on the inductive bias of the network.
Revised: In this way, we can build intuition for the impact of design features on the inductive bias.

Original: For example, you may find a biological network with a particular type of non-linearity not seen elsewhere in the brain.

Original: By finding a function that a network with this non-linearity learns much better than one without, you may gain insight into the role of the non-linearity.

Original: To examine this we again create a meta-learner that labels data, but this time the labels are used to train two learners.
Revised: To illustrate this we again create a meta-learner that labels data but this time the labels are used to train two learners.

Original: We then train the meta-learner so that one learner (the chosen student) is much better at generalising than the other (neglected student).
Revised: We then train the meta-learner so that one learner (the chosen student) is much better at generalising than the other (neglected student).

Original: This is done by minimising the generalisation errors of the chosen student minus the neglected student, figure 4A.
Revised: This is done by minimising the generalisation errors of the chosen student minus the neglected student, Fig. 4A.

Original: We again test this approach on well understood algorithms, and show that it can find functions that a kernel regression algorithm is able to learn better than linear regression, figure 4B, i.e. ahigher frequency, non-linearly seperable function.
Revised: Validating this approach on well understood algorithms, we show that it can find functions that a kernel regression algorithm is able to learn better than linear regression, Fig. 4B, i.e. a non-linearly separable function.

Original: This illustrates some of the games that could be played in this setting.
Revised: This illustrates some of the games that can be played in this setting.

Original: For example, you could play a co-operative game, in which you meta-learn a function that a set of learners all find easy to generalise, and each learner could have different connectivity matrices to match the distribution in real animals, to ensure the tool does not over-fit to some specific details.
Revised: For example, you could play a co-operative game, in which you meta-learn a function that a set of learners all find easy to generalise, and each learner could have different connectivity matrices to match the distribution in real animals, ensuring the tool does not over-fit to some specific details.

Original: However as the losses become more complex training becomes harder, for example this adversarial setting between chosen and neglected student is hard to make robust if the two learners are relatively similar.
Revised: However as the losses become more complex training becomes harder, for example this adversarial setting between chosen and neglected student is hard to make robust if the two learners are relatively similar.

Original:  M ETA -L EARNING A PPLIED TO M ORE C OMPLEX L EARNING A LGORITHMS
Revised:  M ETA -L EARNING A PPLIED TO M ORE C OMPLEX L EARNING A LGORITHMS

Original: In the previous sections we have developed and tested a suite of ways to meta-learn the inductive bias ofsimple learning algorithms.
Revised: So far we have developed and tested a suite of tools for extracting the inductive bias of learning algorithms.

Original: We now apply our tools to networks whose inductive bias cannot be understood analytically.
Revised: We now apply our tools to networks whose inductive bias cannot be understood analytically.

Original: By taking gradients through the training procedure we can meta-learn inductive biases for networks trained using PyTorch, which includes standard networks trained via backpropagation, or, thanks to recent advances in the surrogate gradient method (Neftci et al., 2019), spiking neural networks.
Revised: Specifically: we show our method works on a challenging differentiable learner, a spiking neural network; we validate our method on a high-dimensional MNIST example; and we illustrate how our tool can give normative explanations for biological circuit features, by meta-learning the impact of connectivity structures on the generalisation of a model of the fly mushroom body. Our tool is flexible: by taking gradients through the training procedure we can meta-learn inductive biases for networks trained using PyTorch.

Original: We will provide code on github that producesall our figures,  which should be easily adapted to networks of interest.
Revised: We will provide code on github that produces our figures, including a basic ReLU network (Appendix A) which should be easily adapted to networks of interest.

Original: 5. M ETA -L EARNING FOR B ACKPROP T RAINED N ETWORKS
Revised: 5. S PIKING N EURAL N ETWORK

Original: We begin by testing our framework on a simple feedfoward ReLU network with 2 hidden layers, learnt using gradient descent. While the functions this network finds easy to generalise cannot be extracted analytically, our tool finds that, unsurprisingly, these networks are also inductively biased towards smooth explanations of the data, learning six smooth orthogonal classifications that increase in frequency and mean squared error 5B. We include this example as the simplest PyTorch implementation of learning the inductive bias of a network trained by gradient descent, in the hope that the code can be easily adapted for future use.
Revised: The brain, unlike artificial neural networks, computes using spikes.

Original: S PIKING N EURAL N ETWORK
Revised: How is an open question.

Original: Surrogate gradient methods allow gradient based training of spiking neural networks, (Neftci et al., 2019; Zenke & Vogels, 2021).[-The key difficult of gradient based approaches to training spiking networks is that the gradient is discontinuous: 
-]
Revised: A recent exciting advance in this area is the surrogate gradient method, which permits gradient based training of spiking neural networks by smoothing the discontinuous gradient (Neftci et al., 2019; Zenke & Vogels, 2021).[-The key difficult of gradient based approaches to training spiking networks is that the gradient is discontinuous: 
-]

Original: at all times until a spike arrives, then infinite. Surrogate gradient methods circumvent this by smoothing the gradient, providing a signal for training. We make use of this development to meta-learn functions that a spiking network finds easy to generalise.
Revised: We make use of this development to meta-learn the inductive bias of a spiking network, providing a challenging test case for our method. ϕ , are sampled uniformly and used to generate spike train by sampling from a poisson process with the following rates: for half the neurons r n = r max 2 (1 + sin( t + θ n )) 2 , where n is a neuron index and θ n are uniformly sampled offsets; for the other half we add a phase shift: r n = r max 2 (1 + sin( t + θ n + ∆ ϕ ))

Original: The network we studyis a modification of a model developed for a recent tutorial (Goodman et al., 2022; Zenke, 2019), which is trained to assign a label to an incoming spike train.
Revised: These populations sensory We study a modification of a model developed for a recent tutorial (Goodman et al., 2022; Zenke, 2019), which is trained to assign a label to an incoming spike train.

Original: The network is a model of an interaural phase difference detection circuit.
Revised: The network is a model of an interaural phase difference detection circuit.

Original: The input spike train is parameterised by a phase difference, ∆ ϕ , that generates two sets of spike trains, one in each ear, figure 6A. These spikes arethen processed through a hidden layer of linear-integrate-and-fire neurons (LIF), before reaching a classification layer.
Revised: The input spike train is parameterised by a phase difference, ∆ ϕ , that generates two sets of spike trains, one in each ear, Fig. 5A. These spikes are processed through a hidden layer of linear-integrate-and-fire neurons (LIF), before reaching a classification layer.

Original: The valenceassigned by the circuit is a real number, calculated by summing the output neurons activity over thelength of the trial.
Revised: A real-valued valence is assigned by summing the output neuron’s activity over the trial.

Original: Then the meta-learning framework is as before: the meta-learner assigns valences to input phase differences, these labels are used to train the spikingneural network by surrogate gradient descent, then thegeneralisation error is computed on heldout data and the meta-learner is trained to minimise this error and a distribution loss. 
Revised: The meta-learning framework is as before: the meta-learner assigns valences to input phase differences, these labels are used to train the spiking network by surrogate gradient descent, then the meta-learner is trained to minimise the learner’s generalisation error and a distribution loss. Our method works well, finding a simple smoothness prior, Fig. 5B.

Original: Our resuilts, figure 6B, show that  the inductive bias is relatively simple.
Revised: Next, we test out method on a high-dimensional input dataset. Thus far, to visualise our results, we have only considered low dimensional input data. We demonstrate that our method continues to work in high-dimensions by applying it to a dataset made of the 0 and 1 MNIST digits (LeCun, 1998).

Original: We learn the three orthogonal easiest-to-generalise labellings, using the same method as in section 3, and find thatthey express a simple smoothness prior: low frequency labellings are easy to learn.
Revised: We meta-learn a labelling of this dataset that a simple convolutional neural network finds easy to generalise. Our meta-learner’s architecture is also a convolutional neural network whose outputs are bounded between 0 and 1, and the meta-learner must learn an easy-to-generalise labelling with high variance. We find that the meta-learner consistently rediscovers the MNIST digits within the dataset, separating each digit into its own class, figure 6. We return to the important question of understanding high-dimensional inductive biases in the discussion.

Original: D ISCUSSION
Revised: B 5. I NTERPRETING C ONNECTIVITY P ATTERNS THROUGH THEIR I NDUCED I NDUCTIVE B IAS A large maturing source of neuroscience data is (a list of which neurons connect to one another). However, there is currently a dearth of methods for interpreting this data (Litwin-Kumar & Turaga, 2019).

Revised: In this section, we show our tool can be used to give normative roles to connectomic patterns through their induced inductive bias.

Revised: We study a model of the fly mushroom body, a beautiful circuit that fruit flies use to assign valence to odours (Aso et al., 2014; Hige, 2018), for which connectomic data has recently become available (Zheng et al., 2018; 2022).

Revised: Odorants trigger a subset of the fly’s olfactory receptors.

Revised: These activations are represented in a small glomerular population (input neurons), projected to a large layer of Kenyon cells (hidden neurons), then onwards to output neurons that signal various dimensions of the odour’s valence, Fig.

Revised: 7A.

Revised: An error signal is provided if the fly misclassifies a good odour as bad, or vice versa, allowing the fly to update its weights and learn appropriate responses.

Revised: Classically, the input-to-hidden connectivity was assumed random; i.e., each hidden neuron connects to a few randomly selected input neurons.

Revised: However, connectomic data has shown that hidden neurons preferentially connect to some inputs, and there are input groupings - if a hidden neuron connects to one member of a group it likely connects to many, Fig 7D (Zheng et al., 2018; 2022).

Revised: Zavitz et al.

Revised: (2021) tested networks with this connectivity on a battery of tasks and found that, compared to random, (1) they were better at identifying odours that activated over-connected inputs, and (2) they generalised assigned valence across a group (i.e. if you assign high valence to the activation of one neuron, you do the same for other neurons in the same group).

Revised: We used our tool to verify and develop these findings by examining the effect of different connectivity patterns on the inductive bias of a sparsely-connected model of the mushroom body, Fig.

Revised: 7A. As a baseline, fully connected networks are biased towards smooth functions, appendix A, the simplest being those that assign valence based on one direction in the input space: high at one end, low at the other, like in Fig.

Revised: 2B - C.

Revised: However, which direction is unimportant; they’re all equally easy to learn.

Revised: Sparsity breaks this degeneracy, aligning the easiest to learn functions with the input neuron basis, figure 7B.

Revised: As such, sparse connectivity, which is ubiquitous in neuroscience, ensures the fly is best at assigning labels based on the activity of small collections of neurons.

Revised: Next, we introduced the observed connectomic structure.

Revised: Biasing the connectivity broke the degeneracy amongst neuron axes.

Revised: The networks were, fairly intuitively, best at generalising functions that depended on the activity of overconnected inputs, figure 7C, matching Zavitz et al.

Revised: Finally we introduce connectivity groups, figure 7D. Without additional changes this does little, the neuron basis is still preferred and, unlike Zavitz et al.

Revised: (2021), generalisation across inputs is not observed, figure 7E.

Revised: Only when we additionally constrain the input-to-hidden connections to be excitatory (i.e. positive) do we see that the circuit becomes inductively biased towards functions that generalise across groups of inputs, figure 7F. In retrospect this can be understood intuitively: positive weights and grouped connectivity ensure that a hidden neuron that is activated by one input will also be activated by other group members, encouraging generalisation.

Revised: This effect is removed by permitting negative weights, which let members of the same group excite or inhibit the same hidden neuron.

Revised: Thus, we verify the findings of Zavitz et al.

Revised: without needing to presuppose a battery of tasks.

Revised: In doing so we highlight how our tool can be used to gain insight into the role of circuit design choices, in particular, the importance of the neuron basis for sparsely connected networks.

Revised:  D ISCUSSION & C ONCLUSIONS

Original: Here we have presented a meta-learning approach to extract the inductive bias of differentiable supervised learning algorithms, which we hope will be useful in normatively interpreting the effect ofparticular features of biological networks.
Revised: We presented a meta-learning approach to extract the inductive bias of differentiable supervised learning algorithms, which we hope will be useful in normatively interpreting the role of features of biological networks.

Original: This approach required few assumptionson the biological network beyond those that make the inductive bias an interesting way to conceptualise the circuit in the first place.
Revised: This approach required few assumptions beyond those that make the inductive bias an interesting way to conceptualise a circuit in the first place.

Original: We required, first, the circuit must be interpretable as performingsome supervised learning mapping.
Revised: We required, first, the circuit must be interpretable as performing supervised learning.

Original: Second,you must specify the input data.
Revised: Second, the input data must be specified.

Original: And, third, you must specify, and be able to take gradients through, the way in which the circuit learns to classify data.
Revised: And, third, you must specify the way the circuit learns, and be able to take gradients through this learning process.

Original: We will discuss each of these three limitations and waysin which they could be relaxed. Regardless, it is heartening that any circuit satisfying these will, in principle, suffice.
Revised: We will discuss each of these requirements and ways they could be relaxed; regardless, it is heartening that any circuit satisfying these will, in principle, suffice.

Original: Thepreviously discussed analytic bridge between kernel regression and its inductive bias (Bordelon et al., 2020; Simon et al., 2021) has already found multiple uses in biology in just a few years (Bordelon & Pehlevan, 2021; Pandey et al., 2021; Harris, 2019;
Revised: The analytic bridge between kernel regression and its inductive bias (Bordelon et al., 2020; Simon et al., 2021) has already found multiple uses in biology in just a few years (Bordelon & Pehlevan, 2021; Pandey et al., 2021; Harris, 2019;

Original: Xie et al., 2022), despite its stringent assumptions.
Revised: Xie et al., 2022), despite its stringent assumptions.

Original: We hope thatby relaxingmany of those assumptions we can offer a route to allow these ideas to be applied more broadly.
Revised: We hope that relaxing those assumptions will offer a route to allow these ideas to be applied more broadly.

Original: Despite our optimism for this broad approach, there remain many challenges.
Revised: The first requirement is that the learner performs supervised learning. This is often reasonable. Some circuits contain explicit supervision or error signals, like the fly mushroom body or the cerebellum (Shadmehr, 2020), and generally brain areas that make predictions (i.e., all internal models), can use their prediction errors as a learning signal. Alternatively, some circuits are well modelled as one area providing a supervisory signal for another, as in classic systems consolidation (McClelland et al., 1995), or receiving supervision from a past version of themselves through replay (van de Ven et al., 2020).

Original: Most fundamentally, sets of functions that a given circuit finds easy to learn from few training examples are still relatively hard to interpret.
Revised: Nevertheless, modelling circuits as performing supervised learning will always be an approximation, most simply due to unmodelled effects such as neuromodulation.

Original: In this work we have stuck to low dimensional spaces in which the resulting functions can be interpreted by eye. In this vein, building low-dimensional toy models of biological systems from which inductive biases can be extracted does seem a plausible way to make progress insome cases, but will not be universal. To make the concept of inductive bias more helpful, more tools are needed to interpret the resulting functions.
Revised: As an illustration of how our framework could be extended beyond supervised learning, we consider how neuromodulation could be incorporated. There exist models of how neuromodulation influences circuit function, for example by sharpening neural non-linearities (Ferguson & Cardin, 2020; Aston-Jones & Cohen, 2005), and these could be included in the learner model.

Original: Studying the inducitve bias on established datasets, such as MNIST, seems productive, since there are some ground truth labellings to compare to. Other simple ideas like looking at the frequency spectrum, or projecting onto some other function basis, might show some details, but not all. Our framework maps algorithms to their inductive bias, more ideas are needed to turn those extracted biases into interpretable findings.
Revised: The meta-learner could then meta-learn two outputs, one label and one neuromodulator. It’s goal would be to meta-learn functions that the learner finds easy to generalise with limited quantities of neuromodulatory attention applied to specific, exemplar, training points.

Original: The first requirement for our tool tobe applicable was an inputdata distribution, which  often will not be easy to guess.
Revised: Next, we could relax our second assumption, access to an input distribution, which is often lacking.

Original: This can be avoided by, for example, using real neural data as the input.
Revised: This can be avoided by using real neural data as the input.

Original: Alternatively, ifthere is a limited set of neural data, generative modellingtechniques could be used to fit thedistribution of neural data  and new samples drawn from that distribution.
Revised: Or, if neural data is limited, generative modelling could be used to fit the neural data distribution and new samples drawn from that distribution.

Original: Finally, you could imagine a single meta-learner that creates not only the label, but also the datapoint that a learner finds easy to fit.
Revised: Finally, one could imagine a single meta-learner that creates not only the label, but also the data.

Original: I.e. the meta-learner could generate the entire dataset by transforming a noise sample into an input-output pair.
Revised: That is, the meta-learner could generate the entire dataset by transforming a noise sample into an input-output pair.

Original: This would have to be carefully regularisedin order to avoid trivial input distributions, but could in principle learn the input statistics that particular networks are tuned to process.
Revised: This would have to be carefully regularised to avoid trivial input distributions, but could in principle learn the input statistics that particular networks are tuned to process.

Original: Next, we could imagine relaxing our second assumption, that the network performs simple supervised learning.

Original: Extracting a circuit from the brain and conceptualising it as performing supervised learning will always be an approximation.

Original: Most simply, there will almost always be effects elsewhere in the brain, like attentional focus, that will have an unmodelled impact on the function of the circuit.

Original: As an illustration of how these could be included in our framework, attention is often thought to happen through neuromodulation (Eldar et al., 2013).

Original: There exist models of how neuromodulation influences circuit function, for example by sharpening neural non-linearities (Ferguson & Cardin, 2020; Aston-Jones & Cohen, 2005), and these could be included in the learner model.

Original: The meta-learner could then meta-learn two outputs, one label and one neuromodulator.

Original: It’s goal would be to meta-learn functions that the learner finds easy to learn with limited quantities of neuromodulatory attention applied to specific, exemplar, training points.

Original: Finally, in a slightly cooky way, we could avoid even needing to specify the form of the learning algorithm, by interfacing with an animal directly!
Revised: Finally, in a slightly kooky way, we could avoid specifying the learning algorithm by interfacing with an animal directly!

Original: Animals’ inductive biases are objects of interest in their own right, but could also give insight into the underlying neural processing.
Revised: Animals’ inductive biases are objects of interest in their own right, but can also give insight into the underlying neural processing.

Original: These biases could be studied by replacing the inner learner with a real animal that is trained on a labelled dataset from the meta-learner, then tested on new datapoints.
Revised: These biases could be studied by replacing the inner learner with a real animal that is trained on a labelled dataset from the meta-learner, then tested on new datapoints.

Original: Since you cannot compute gradients through the computations of a living animal (I think...), the meta-learnerwould have to use signals computed using only meta-loss evaluations. This could be done using black-box optimisation procedures, like the Nelder-Mead method (Singer & Nelder, 2009). 
Revised: Since we cannot compute gradients through the computations of a living animal, the meta-learner could be optimised using black-box optimisation procedures that rely only on meta-loss evaluations, like the Nelder-Mead method (Singer & Nelder, 2009). Despite our optimism for this approach, there remain challenges.

Original: The meta-learner would then meta-learn labellings that the real animal is proficient at generalising!
Revised: Most fundamentally, sets of functions that a learner easily generalises are still hard to interpret.

Original: C ONCLUSION
Revised: We have shown how our tool can provide insight for low-dimensional inputs (Fig. 2 - 5), by comparing to ground truth labels (Fig. 6), or by projecting the learnt functions onto an appropriate basis (Fig. 7). However, to make the concept of inductive bias more powerful, more tools are needed to interpret the resulting functions.

Original: The inductive bias is a promising angle from which to understand pattern classifiers.
Revised: To conclude, the inductive bias is a promising angle from which to understand learning algorithms.

Original: Analytic bridges between circuit design and inductive bias ‘explain’ the presence of aspects of the circuit through their effect on the network’s generalisation properties. This has aided understanding in both artificial (Canatar et al., 2021;
Revised: Analytic bridges between circuit design and inductive bias have already ‘explained’ the presence of aspects of the circuit through their effect on the network’s generalisation properties in both artificial (Canatar et al., 2021;

Original: Bahri et al., 2021) and biological (Bordelon & Pehlevan, 2021; 
Revised: Bahri et al., 2021) and biological (Bordelon & Pehlevan, 2021; Pandey et al., 2021; Harris, 2019;

Original: Pandey et al., 2021; Harris, 2019; Xie et al., 2022) networks, however these techniques require very constraining assumptions.
Revised: Xie et al., 2022) networks. However, these techniques require very constraining assumptions.

Original: Here we have presented and tested a meta-learning approach to flexibly extract the inductive bias from neural circuits.
Revised: We have dramatically loosened these assumptions and shown our meta-learning approach can flexibly extract the inductive bias of neural circuits.

Revised: We have shown its utility in interpreting connectomic data, and we believe it will prove useful on other datasets and problems.

Revised: A M ETA -L EARNING A S IMPLE B ACKPROP T RAINED R E LU N ETWORK

Original: We have shown its broad applicability by applying it to simple  networks  trained by backprop and spiking neural networks.
Revised: A simple test of our framework is a feedfoward ReLU network with 2 hidden layers, learnt using gradient descent. While the functions this network finds easy to generalise cannot be extracted analytically, our tool finds that, unsurprisingly, these networks are biased towards smooth explanations of the data, learning six smooth orthogonal classifications that increase in frequency and mean squared error 8B. We include this example as the simplest PyTorch implementation of learning the inductive bias of a network trained by gradient descent, in the hope that the code can be easily adapted for future use. B U SING D IFFERENT D IVERGENCE M EASURES

Original: As new and large neuroscientific datasets mature, such as connectomics (Zheng et al., 2018), methods to interpret the results will increasingly be needed.
Revised: To persuade the meta-learner to find non-trivial functions, we include a divergence loss that forces the meta-learner’s label distribution to take a particular form: uniform between -1 and 1. In this section we show that the particular divergence that we use has little impact on the solutions we find for learning the easiest-to-generalise function of the kernel learner, figure 2C. Figure 9 shows that a variety of divergence metrics can be used, sinkhorn, an energy statistic from Sz ´ ekely & Rizzo (2013), and the maximum mean discrepancy from Gretton et al. (2012) (implementations from Djolonga (2020)). In each case the learnt function has over 99% norm in the space of first order spherical harmonics, demonstrating that the meta-learner has learnt appropriately.

Revised: We different divergence metrics to penalise the meta-learner’s label for deviating from uniform between -1 and 1.

Revised: We add these to the meta-loss, along with the generalisation error of the simple kernel learner introduced in section 2.

Revised: For all three divergence metrics the meta-learner learns a very close approximation to a first order spherical harmonic, as predicted

Revised: C R ANDOM R E -S EEDING CANNOT R EPLACE O RTHOGONALISATION

Original: By building a link between structural and functional domains, we hope this tool will allow otherwise opaque features of biological neural networks to be interpreted and understood, through their impact on the network’s inductive bias.
Revised: We introduced the orthogonalisation procedure in section 3 so that successive meta-learners have to explore larger areas of function space. It is legitimate to wonder whether simply re-running the optimisation would have had the same effect. Here we show it does not, and orthogonalisation is needed to explore the learner’s inductive bias fully. We re-run the meta-learner optimisation without any orthogonality terms for the kernel learner described in section 2.

Revised: In figure 10 we find that the meta-learners find different functions, but only approximations to the first order spherical harmonics.

Revised: This makes sense, the meta-learner is tasked with finding the easiest-to-generalise non-constant function.

Revised: For this particular learner there is a degenerate space of such functions and so re-running the meta-learner simply draws another sample from this space of functions.

Revised: However, to access the second order spherical harmonics that this learner still learns, just worse, we need something like the orthogonality constraint.