DevFly: Bio-Inspired Development of Binary Connections for Locality Preserving Sparse Codes
Anonymous Author(s)
AffiliationAddress email
Abstract
Neural circuits undergo developmental processes which can be influenced byexperience.
Here we explore a bio-inspired development process to form theconnections in a network used for locality sensitive hashing.
The network is asimplified model of the insect mushroom body, which has sparse connections fromthe input layer to a second layer of higher dimension, forming a sparse code.
Inprevious versions of this model, connectivity between the layers is random.
Weinvestigate whether the performance of the hash, evaluated in nearest neighbourquery tasks, can be improved by process of developing the connections, in whichthe strongest input dimensions in successive samples are wired to each successivecoding dimension.
Experiments show that, the accuracy of searching for nearestneighbours is improved, although performance is dependent on the parametervalues and datasets used.
Our approach is also much faster than alternative methodsthat have been proposed for training the connections in this model.
Importantly, thedevelopment process does not impact connections built at an earlier stage, whichshould provide stable coding results for simultaneous learning in a downstreamnetwork.
Introduction
The insect mushroom body (MB) is a learning centre in the insect brain, and its function is of interestas a neural architecture for efficient association of arbitrary sensory patterns to actions.
The MB is ashallow network, with an input layer of projection neurons (PNs) that sparsely connect to a largernumber of Kenyon cells (KCs), which are fully connected to a small number of MB output neurons(MBONs).
In this paper we focus on the PN to KC connectivity, and consider how the activity patternin KCs could provide an efficient code for the input that enhances the appropriate generalisation frompreviously experienced patterns to new inputs.
In the MB, only a small portion of KCs is activatedat any time (Honegger et al., 2011), which is known as sparse coding (Olshausen & Field, 2004).
Sparse coding facilitates learning by letting each KC represent a more specific pattern of sensoryinputs, which reduces unintended association with a different pattern during learning (Treves & Rolls,1991).
Ideally, similarity in sensory input space should map to similarity in the space of sparse codes.
One way to interpret the function of the mapping from PN to KC is treating the mapping as alocality-sensitive hash (Dasgupta et al., 2017).
Ideally, a locality-sensitive hash (LSH) function canpreserve relative distances between data samples in the corresponding hash coding results (Hutchins,1999), such that the codes can be used to efficiently locate close samples.
Compared to a typical LSH,the MB-inspired hash algorithm (Figure 1) proposed by Dasgupta et al.
(2017), called FlyLSH, mapsinputs (with a dimension d ) to a higher dimensional space (with a dimension m ), using sparse binaryconnections, and then creates a sparse binary code by choosing the top k -values and setting them to 1.
FlyLSH achieved better performance than a standard LSH algorithm (if the hash length of the latter is scaled down so that the computational complexity to calculate each form of hash is similar).
Thisin turn allows FlyLSH to attain higher precision than LSH algorithms that use a dense connectivitymatrix.
It has inspired some variations including deep nets that use FlyLSH as initial layers.
Forexample, work by Chancan et al.
combined a FlyLSH based network with a one-dimensionalcontinuous attractor neural network and achieved a better performance than other methods in visualplace recognition tasks.
FlyLSH uses fixed random connections between the PN and KC layers, a feature inspired byprevious accounts of this connectivity in the MB (Caron et al., 2013; Masuda-Nakagawa et al., 2005).
However, more recent biological results suggest that the formation of connections between PNs and
KCs can be experience-dependent, and when a KC connects to PNs depends on (1) when the KC isdevelopmentally generated and (2) the existing connections (Eichler et al., 2017).
Like other neuralcircuits, the MB develops during the animal’s maturation from a few neurons to thousands of neurons,and new connections are built during this process.
In the fly, this happens during embryo, larvae,pupae, and early adult stages (Figure 2) (Lee et al., 1999), with and without external stimulus.
Thereis also evidence suggesting that the distribution of the connections cannot be simply explained asrandom, and this is supported by both experimental (Eichler et al., 2017; Elkahlah et al., 2020) andtheoretical (Litwin-Kumar et al., 2017) studies.
In this paper, we propose a bio-inspired method that develops the binary connections between input
PNs and the coding KCs.
The connections to each coding dimension are built according to strongestdimensions of a selected sample, and once built, they will not be impacted by later development, sothat reliable coding results are preserved during the development process.
We then test this methodagainst FlyLSH in a nearest neighbours search task using several datasets as inputs.
Related works
Models of the MB have been presented by several different groups mostly in the context of associativelearning or classification tasks, such as (Huerta et al., 2004; Smith et al., 2008; Mosqueiro & Huerta,2014; Bennett et al., 2021; Wessnitzer et al., 2012; Ardin et al., 2016; Zhu et al., 2021).
Most ofthese studies make the assumption that the connections between PNs and KCs are randomised.
Theeffect of non-random PN to KC connections has been explored for the biological task of odourdiscrimination in a model by Zavitz et al.
They find the effective dimensionality of the
KC layer is reduced, the response to odours potentially of ethological significance to the fly areheightened and the generalisation between odours is improved.
There are also recent models thatinclude plasticity between PNs and KCs, such as MothNet (Delahunt & Kutz, 2019) and a modelby Peng & Chittka (2017) which adjusts the PN-KC synapses according to reward modulation andcoactivation of connected PN and KC, showing that this alters the generalisation/discriminationtradeoff for KC encoding of sensory patterns.
A number of papers have explored the effects of altering the connection scheme from PN to KCfrom the FlyLSH default, which is for each KC to receive inputs from n randomly selected PNswhere n is some fraction of the input dimension.
Xu & Qiao (2018) explored the use of a variable n ,centred on the same mean, and reported improved performance over FlyLSH for nearest neighboursearch in CIFAR-10, as well as better performance than several data dependent (but unsupervised)standard hash methods including PCA-hash, Sperical Hash and ITQ.
Data-dependent methods for forming the sparse connections should be expected to do better than random connections for the samen .
Ma et al.
(2020) proposed an algorithm to iteratively construct the projection matrix to producethe winner-take-all (WTA) output that maximises the distance between codes for the whole dataset,subject to the limit n on the number of connections per KC.
Ryali et al.
(2020) proposed a methodsimilar to self-organising mapping to train a modified version of FlyLSH, which uses continuousvalues instead of binary values for mapping, noting that this is equivalent to k-means clusteringwith some additional constraints.
Both methods outperform FlyLSH, but are relatively data greedybecause they use all images in the datasets for learning the connections.
In Pehlevan et al.
(2017) aneurally plausible method for online k-means clustering is proposed and related to known PN-KCconnectivity properties, although the method is not explicitly tested as a hashing code.
Preissner & Herbelot (2019) took a contrasting approach when applying the FlyLSH method to alanguage corpora.
In their work the PN layer is incrementally expanded, and the KC connectionsrandomly redistributed, while preserving equal n , as words are encountered in new contexts.
Aninteresting point made in this approach is that the resulting sparse KC code is interpretable, forexample, the set of PNs connecting to one KC tend to refer to related or frequently co-occuring wordsin the input data.
However, like the previous methods, it requires multiple updates for every KCduring the training process.
There is relevant earlier research on growing ANNs (also described as resource allocation networks(Platt, 1991)) where neurons and/or connections are gradually added to the network (see review by
Macleod & Maxwell (2001) and more recently (Mixter & Akoglu, 2020)).
In this approach, new unitsare added when the existing network does not meet some criteria with respect to newly presented data,and then all the connection weights are retrained using gradient descent or similar standard methods.
These methods might also include pruning, to remove units that are not contributing to successfulperformance, e.g., (Yingwei et al., 1998).
In either case the aim is to produce a more compact networksize.
By contrast, our developmental algorithm assumes a target total number of neurons, and tries toconnect each successive one to represent the data efficiently with respect to previous connections,but without changing any existing connections.
Interestingly, our approach (and the MB structure)can be more closely compared to early work on ‘prototype’ learning in the form of RCE networks(Reilly et al., 1982), where input samples are used to set the weights of successive nodes in the firstlayer, with new connections to the next node formed if the existing node activations cannot supportsuccessful classification in the following layer, but no further changes made to existing weights.
RCEnetworks have also been used in an unsupervised form (Marsland et al., 2002) where nodes are addedwhen none of the current nodes are sufficiently activated, and this form of network can be related toradial basis functions.
Methods
Input :
Data
Samples
S =
{ s 1 , s 2 , . . .
s q , . . .
s u } .
Parameter : code dimension m , number of connections per KC n .
Output :
KC weights
R = { r 1 , r 2 , . . .
r j , . . .
r m } .
1: j :=2: # Randomly choose m samples from
S .
3: for s q in m samples from S do 4: # Set the weights of a KC according to a sample.
Find the indexes i ′ of top n values in the sample.
6: r i ′ ∈
i ′ :
= 1 , where r i ∈
r j .
7: j++ 8: end for 9: return R
The aim of the algorithm presented here is to produce alocality sensitive hash code such that the distance in thesample space is positively correlated to the distance inthe code space.
In the following, the distance in samplespace is measured as Euclidean distance, and the distancein the code space is measured as Hamming distance forthe binary coding (Sharma & Navlakha, 2018).
Hashing by FlyLSH or its variations consists of two steps(Figure 1), a linear mapping from the input (with dimension d ) to the code space (with dimension m ), and binarisation of the resulting activity pattern.
Note that weassume the input data has been normalised, however, thishas limited impact on the performance, because the connection building given a sample and binarisation is onlydetermined by the top values.
FlyLSH uses a “centeringthe mean” pre-processing step (Dasgupta et al., 2017),which shifts the samples so the sum of all the samplesare zeros.
However, we did not notice any difference inperformance without the shift in FlyLSH, and our methodsshould not be sensitive to this again because they build the connections according to the most intensive pixels, irrespective of the average intensity. Thus thisstep, which requires advance access to the dataset, is not critical and could be removed for onlinelearning.
In FlyLSH, the linear mapping step uses a binary matrix R generated by randomly choosing n =
⌊ αd ⌋indexes from the input dimensions for each code dimension, where 0 < α < 1 , typically α ∼ 0 .
1 ,and ⌊·⌋ is the floor operation.
Thus, for each row of R , n elements are randomly set to 1, and theremainder are 0.
After all of the coding dimensions are computed, they are binarised by choosing thetop k values and setting them to 1 , with all remaining values set to 0.
Note that although in the fly, and in contrast to standard LSH, m >> d , in practice FlyLSH is oftentested with high dimensional inputs resulting in m < = d .
The key properties preserved from the flyare the use of a low sampling ratio α rather than full connectivity, and a sparse code k << m .
Forclarity, we have not followed Dasgupta et al.
(2017) in referring to k as the hash length, as in practicethe full code of length m is used to calculate Hamming distance, but see section 4.1 below.
The method we propose (which we call DevFLY) uses bioinspired development processes to replacethe random choice that generates matrix R .
They mimic the biological observation that in the MB,individual KCs mature at different times and each will make connections to PNs as they mature, thatis, there is no single stage at which all PN-KC wiring occurs.
We assume that, in the MB, when somecriteria is triggered, synapses are built from those PNs that have strongest activities to an immature
KC; subsequently the connections for this now mature KC remain fixed.
Correspondingly, in DevFly,when some criteria is triggered, the connections are built from strongest input dimensions to a newcode dimension.
We propose three alternative criteria (from hereon referred to as Method 1,2 or 3).
Input : Data Samples S =
{ s 1 , s 2 , . . .
s q , . . .
s u } .
Parameter : threshold θ , threshold increasing rate β , code dimension m , number of connections per KC n .
Output : KC weights R = { r 1 , r 2 , . . .
r j , . . .
r m } .
1: Let R := 0 , j := 1 , q := 1 .
2: while j ≤ m do 3: # Iterate for next unfamiliar sample s q .
4: while y ′ > θ ∃ y ′ ∈ { y ′ j | y ′ j
= r j s q } , j ′ ∈ { 1 , 2 , . . . , j } do 5: q + + 6: if q > u then 7: θ := βθ 8: q :=9: end if 10: end while 11: # Set the weights of a KC according to a sample.
12: if Method 2 then 13:
Find the indexes i ′ of top n values in the sample.
14: else if Method 3 then 15:
Randomly choose n indexes i ′ using the normalised sample as probability.
16: end if 17: r i ∈ i ′ :
= 1 , where r i ∈
r j .
18: j + + .
19: end while 20: return R
according to randomly picked samples.
The first method is designedto test the importance of connectingfrom the strongest input dimensions,by simply setting the criteria to be randomness: (1) Randomly choose datasamples, as many as the number ofcode dimensions.
(2) Given a sample,connect the strongest n input dimensions to a code dimension.
(3) Repeatthe previous step until all code dimensions are connected.
according to unfamiliar samples Inthe second method, we set the criteria so as to build new connectionsonly when a sample input is not already well connected.
That is, given asample, an unconnected code dimension will be connected to the strongestinput connections only if none ofthe previously connected code dimensions have an output stronger than athreshold.
The threshold can be interpreted as a means of recognising“familiar" samples.
(1) Set a thresholdfor when to connect.
(2) Initialise theconnections to the first code dimension with the first sample accordingto the strongest n input dimensions.(3)
For the next sample, check the response of connected code dimensions.
If all them are weaker than the threshold, connect an unconnected code dimension according to thestrongest n input dimensions.(4)
Repeat the previous step until all code dimensions are connected.
If all samples have been examined but some code dimensions remain unconnected, increase thethreshold, and then loop through the samples again.
Method 3: connection development randomly according to unfamiliar samples We propose anadditional variation on Method 2 by introducing noise when building the connections.
Instead ofconnecting the input dimensions and code dimensions strictly according to the strongest input dimensions, the connection is built by randomly selecting input dimensions according to the distributionof the input dimensions.
This seems biologically more realistic, but we also believe that with thismodification, the new connections could have better generalisation while capturing the features.
Method 1 is shown in Algorithm 1.
Method 2 and 3 are shown in Algorithm 2. The code is availablein the supplementary materials.
Experiments
We applied the three connection development methods to FlyLSH, and tested them on four differentdatasets for the precision in finding nearest neighbours.
The datasets are MNIST (CC BY-SA 3.0),
CIFAR-10 (MIT), SIFT10M and GloVe (Apache-2.0).
MNIST (LeCun et al., 1998) is a dataset withimages of handwriting digits from 0 to 9.
Each of the images has a size of 28 × 28 and is in greyscale.
CIFAR-10 (Krizhevsky, 2009) is a dataset with images from ten classes of objects.
Each of theimages has a size of 32 × 32 for each RGB channel.
SIFT
M (Jegou et al., 2010) also containsvectors representing images but has a smaller dimension (128) and is specifically designed for nearestneighbours search.
The GloVe dataset (Pennington et al., 2014) contains pre-trained vectors with adimension of 300 for word representation, and the vectors have linear substructures in their space.
These dataset do not include personally identifiable information or offensive content.
Following the protocol in the experiments by Dasgupta et al.
(2017) and Sharma & Navlakha (2018),for each of the datasets, we chose 10,000 samples as an experiment dataset for the nearest neighbourssearch.
In each experiment, m samples from a dataset were used to train the model, then 1000 sampleswere randomly chosen for querying.
For each of the query sample, we listed the 200 samples with thesmallest hamming distances to the query sample, then compared the list with the true neighbours listfound according to their Euclidean distances.
We report the results for the three DevFly methods withdifferent hyperparameters, that is, varying k , m and α .
Each of the tests with a specific combinationof hyperparameters was repeated 10 times for statistical results.
For comparison, we also run therandom FlyLSH described by Dasgupta et al.
(2017) on the same datasets.
Because the originalcode by Dasgupta et al.
(2017) is not available, we used code published by the same research group(Sharma & Navlakha, 2018) to implement these methods.
We noticed that the performances reportedfor FlyLSH are not consistent in those two papers.
Hence, comparisons are based on our experimentalresults only.
For reference, both Fly methods consistently outperform a traditional LSH (Hutchins,1999) if its hash length is set to k .
Note that in some cases, hashing methods include additional steps to increase performance.
Forexample, multi-bin search by Kamel et al.
(2015), a two-step evaluation is used to find the nearestneighbours: (1) the hash is used to find the bins that contain similar samples, (2) nearest neighboursare determined according to other information, which can be the raw data. The second step cansignificantly improve the accuracy. Other methods compute several hash tables and use them tocompare across different results for the same query. The models used here only focus on the first step,and were trained only by learning some samples once, so the precision is expected to be lower thanstate-of-the-art hashing.
Adding additional steps is still feasible to further improve the performance.
An LSH using a sparse binary code based on WTA has an advantage in query time.
Because thenumber of activating coding dimensions is fixed, to carry out a nearest neighbour search query fora given sample it is not necessary to compare all of the coding dimensions.
Instead, it is sufficientto just compare those coding dimensions which are activated (coded as 1) by the query sample.
Hence, the query time will scale with k rather than with m .
This provides a potential advantage over DenseFly (Sharma & Navlakha, 2018).
DenseFly varies from FlyLSH by not using top- k WTA butinstead using a threshold to set code elements to 1 or 0 .
As we show in figure 3 the mean averagequery time for Method 1 using a fixed sparseness, i.e., k = m/ 20 is significantly lower. The same [Figure]
advantage applies to our other developmental methods (and to the original FlyLSH) as they also usetop- k WTA for binarisation.
4.2 Search accuracy
DevFly methods are tested on four datasets and the mean Average Precision (mAP) for nearestneighbour query is shown is Figure 4.
In these experiments, α = 0 . 1 , sparseness m/k = but kvaries.
We also tested the effects of reducing α or increasing sparseness by using setting m/k = 40 .
The results are shown in supplementary Figures S.1 to S.4.
All the developmental methods show a similar performance on MNIST, and perform better than
FlyLSH for all k .
For MNIST, the relative improvements range from 9% for k = 32 to 106% fork = 2 , even with Method 1, in which the connections are made without any criteria for choosing thesamples used to determine the connections.
For the other datasets, at least one developmental method outperforms FlyLSH, although whichmethod performs best varies with the dataset.
Notably for GloVe, the randomness introducedin Method 3 produces no improvement on performance over FlyLSH.
For CIFAR-10, both theimprovement with DevFLY methods and the overall performance are relatively low.
This mightbe caused by the different complexity of this dataset compared to MNIST.
CIFAR-10 has largerdimension, d = 32 × 32 × 3 = 3072 , than MNIST, d = 28 × 28 = 784 .
More crucially, for MNIST,the information for distinguishing the digits is clearly given by the highest intensity pixels, so ourmethod which forms connections based on intensity will correspond to the key distinguishing features
[Figure] of the data. For illustration, we can visualise the weights for 100 KCs formed by Method 2 after 320steps of development which clearly reflect the structure of the data (Figure S.6).
Images in CIFAR-10are photos of real objects, which have richer and more complex details than images in MNIST, andthe pixels with the strongest intensities are not necessarily the most informative, e.g. they might bepart of the background, corresponding to sky.
Higher performance of the models on MNIST and SIFT10M than on GloVe is due to the data beingmore clustered in MNIST and [-SIFT
 M lying on a lower dimensional manifold, relative to theirinput dimensions, than GloVE, and hence requiring fewer samples-]SIFT10M, making it easier to separate the nearest neighbours, where theseparation is more arbitrary the more distributed the samples are in space.
Indeed, none of the modelswere able to find more than 10% of the nearest neighbours when tested on a randomly generateddataset with an uniform distribution (see Section S.4).
We analysed the principal components of thesefive datasets, and found that the datasets requiring fewer components to capture a high percentageof the the data generally resulted in better performances (Figure 5A). However, for CIFAR-10 theproblem appears not to be lack of clustering but rather to a high variance in magnitudes across thesamples (Figure 5B).
This results in lower performance following the WTA operation, which is notsensitive to differences in magnitudes.
This means that samples that vary only in magnitude will bemapped to the same hash, which will distort the original topology of the samples in the input space. Therefore, LSH methods would perform best on datasets that are clustered and samples that lie on ahypersphere.
4.3 Learning efficiency
There are many existing methods for tuning the projection matrix in LSH, and the methods can becategorised into three groups, unsupervised, supervised, and semisupervised.
These methods can alsobe further categorised according to the levels of supervised information, linearity, one-shot/multipleshot.
For a comprehensive review, see the survey by Wang et al.
Among them, we have notfound a development based method.
Another alternative approach to optimising PN-KC connections for an MB inspired hashing model is
BioHash (Ryali et al., 2020).
BioHash proposed a method similar to self-organising mapping to traina modified version of FlyLSH, which uses a dense connection matrix and continuous values insteadof binary values for mapping.
We modified BioHash in order to have the same number of active KCsas DevFly and FlyLSH.
We find that BioHash takes much more time to reach the same precisionlevel as either Method 1 or Method 2 (Figure 6).
Only 320 iterations were needed for DevFly versus10,000 iterations for BioHash to reach the same performance level.
In application, fast convergenceof the model helps for timeliness.
For huge multi-dimensional datasets, the bigger the dataset, thestricter the computational requirements are, and thus DevFly can be used in cases where BioHash istoo costly.
This also means that DevFly is more suitable for cases where the available training data isscarce. [Figure] g [Figure]
P
[Figure]
c i
[Figure]
i o [Figure] [Figure] 4.4 Insensitivity to sample order
We also tested DevFLY with ordered samples to check if training with non-random samples wouldimpact performance.
Because the original MNIST dataset is in a random order, we sorted 10000samples according to their labels.
Then, in Method 1, m samples are randomly chosen from theordered data and used to form the successive connections to KCs while maintaining their labellingorder.
For Methods 2 and 3, in training, all samples labelled 0 were fed to the model, then all sampleslabelled 1 were fed, etc.
The threshold for choosing if a sample is used to make new connectionsmeans that Methods 2 and 3 will typically pass through all samples several times before all KCs areconnected.
The results suggest that for all three methods, the order does not impact the performance(Figure S.5).
Hence, training with a development-like approach, which only builds new connectionsand keeps the existing connections, helps to reduce the sensitivity to the order of samples.
Note that FlyLSH results are not impacted by the order of samples simply because it does not use the dataset atall in initialising the matrix R .
Discussion
In this paper, we proposed a method to improve the performance of FlyLSH by developmentalinitialisation of its binary connections from input dimensions to code dimensions.
This was inspiredby biological data showing that in the insect mushroom body (the neural circuit that inspired FlyLSH)the connections are not random but there is a process of development in which PNs connect to KCs.
We implement this as a simple wiring process driven by the highest values in data samples.
We showthis improves the performance of FlyLSH in a range of nearest neighbour search tasks.
Essentially,
FlyLSH detects features of the input space and represents them in a sparse high dimensional code.
Using random connections can result in detectors for features that do not exist in the dataset, whereas
DevFLY constructs the feature detectors from samples in the dataset, hence improves the efficiencyof the coding for non-uniform dataset.
The advantage provided by DevFLY diminish as the featurespace gets larger (Table S.3, S.3, and S.4) as the random method covers more of the space.
There are many known methods for optimising the (sparse) connectivity between layers to preservelocality (or other desired) properties of the data.
However most of these require multiple passesthrough a large and representative sample from the dataset and are significantly more costly tocompute.
The method we use here is aimed at scenarios - such as real brains - where ‘rewiring’ iscostly, and a method that requires all weights to be adjusted for each new sample is expensive, aswell as potentially disruptive as each new input changes the existing encoding.
The improvement in performance of DevFly over FlyLSH is due to an overlap between the regionswith highest variance and those with highest intensity (Figure S.7).
Essentially, our method assumesthat the strongest signals in the input dimensions correspond to the most relevant signals for similarity(here defined as Euclidean distance in the input data).
In CIFAR-10, the dimensions containing thehighest intensity pixels don’t overlap with the dimensions that vary the most (Figure S.7).
By relyingon the highest pixel activations to produce the hash, the most important dimensions in CIFAR-10 willbe overlooked, leading to poor results.
For the other datasets, where high intensity pixels coincidewith high variance, DevFly has an advantage because it disregards those areas with low variance, whereas FlyLSH sampling randomly will weight them equally.
Hence a possible way to improveperformance in this case would be to pre-process the data to extract the features relevant to similarity- in a sense this is what we see already for SIFT10M.
Our results also show this method is better suited to datasets that are clustered, have samples with arelatively constant norm, and in general have higher values in those input dimensions that are relevantto the similarity measure.
For consistency with previous work we have used Euclidean distance in theinput space as the ‘ground truth’ for similarity, but note this may not be the best measure for datasetssuch as CIFAR-10.
Of interest is whether our method could be adapted to develop wiring from theinput dimensions that show most correlated variability to the code dimensions.
For example, wiringfrom the changing pixels of a security camera video.
The fact that a weight in DevFly methods only updates once for the entire life of the model facilitatesthe implementation of this model to hardware.
For example, programmable ROM, which permitsdata to be written to memory only once, can be used to save the weights while learning.
It is alsoeasy to be implemented on an integrated circuit with minimum die size usage.
The implementationof Method 3 on hardware does not necessarily need a random number generator.
In fact, Method 3suggests that for datasets like CIFAR-10, variation that naturally results from unreliable hardwarecan be an advantage for better generalisation, providing higher fault tolerance for hardware.
In the insect MB and the mammal cerebellum, sparse coding is used to extract and separate low-levelfeatures across different neurons (Olshausen & Field, 2004; Farris, 2011).
Hence, DevFLY shouldalso be suitable to improve the detection of low-level features in shallow layers of a deep net, such asthe deep net used by Chancan et al.
(2020) for place recognition.
Error propagation-based approacheshave difficulties to train shallow layers due to averaging of the gradient during propagation.
Althoughthere are some tricks like ResNet to convey more information by shortcut connections, it is worthconsidering the use of an unsupervised approach to update shallower layers directly according to datasamples.
Importantly, a developmental training approach provides more stable signals for deeperlayers during training as it does not alter existing connections.
A particular application of this model of interest to us in future is reinforcement learning (RL).
Anapproach to improve exploration efficiency in RL is to count visited states for computing the novelty.
For continuous tasks or tasks with high-dimensional observations, using the raw observation as astate to count is not feasible.
Tang et al.
(2017) proposed a model that uses LSHs instead of theraw observation for state counting which achieved “surprisingly good results".
They claimed thatstatic hash methods such as SimHash perform worse than methods that can learn from data, suchas autoencoders.
However, they also argued that it is important to keep the mapping from states tocodes relatively consistent over time, otherwise the same state could be mapped to different codes bytraining.
DevFly suits this application because each weight only updates once so the learned codingdimension never changes, i.e., improving coding quality while minimising unnecessary change.
This work emphasises the development of connections and can belong to a special case of optimisation,which does not have a direct negative societal impact.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] See Section 1.
(b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeSection 5.(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See Section
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 4.
(c) Did you report error bars (e
g., with respect to the random seed after running ex-
periments multiple times)?
[Yes] See Figure 4 and other figures in supplementary
materials.
(d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See the caption of Figure 3.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 4.
(b) Did you mention the license of the assets?
[Yes] See Section 4.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
There is no new asset in this work.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes] See Section 4.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]