B IO -I NSPIRED
H ASHING
FOR
U NSUPERVISED
S IMILARITY S EARCH
Anonymous authors
Paper under double-blind review
A BSTRACT
The fruit ﬂy Drosophila’s olfactory circuit has inspired a new locality sensitive hashing (LSH) algorithm, FlyHash .
In contrast with classical LSH algorithms that produce low dimensional hash codes, FlyHash produces sparse high-dimensional hash codes and has also been shown to have superior empirical performance compared to classical LSH algorithms in similarity search.
However, FlyHash uses random projections and cannot learn from data.
Building on inspiration from
FlyHash and the ubiquity of sparse expansive representations in neurobiology, our work proposes a novel hashing algorithm BioHash that produces sparse high dimensional hash codes in a data-driven manner.
We show that BioHash outperforms previously published benchmarks for various hashing methods.
Since our learning algorithm is based on a local and biologically plausible synaptic plasticity rule, our work provides evidence for the proposal that LSH might be a computational reason for the abundance of sparse expansive motifs in a variety of biological systems.
We also propose a convolutional variant BioConvHash that further improves performance.
From the perspective of computer science, BioHash and
BioConvHash are fast , scalable and yield compressed binary representations that are useful for similarity search.
I NTRODUCTION
Sparse expansive representations are ubiquitous in neurobiology.
Expansion means that a highdimensional input is mapped to an even higher dimensional secondary representation.
Such expansion is often accompanied by a sparsiﬁcation of the activations: dense input data is mapped into a sparse code, where only a small number of secondary neurons respond to a given stimulus.
A classical example of the sparse expansive motif is the Drosophila fruit ﬂy olfactory system.
In this case, approximately 50 projection neurons send their activities to about 2500 Kenyon cells (Turner et al., 2008), thus accomplishing an approximately 50 x expansion.
An input stimulus typically activates approximately 50% of projection neurons, and less than 10% Kenyon cells (Turner et al., 2008), providing an example of signiﬁcant sparsiﬁcation of the expanded codes.
Another example is the rodent olfactory circuit.
In this system, dense input from the olfactory bulb is projected into piriform cortex, which has 1000 x more neurons than the number of glomeruli in the olfactory bulb.
Only about 10% of those neurons respond to a given stimulus (Mombaerts et al., 1996).
A similar motif is found in rat’s cerebellum and hippocampus (Dasgupta et al., 2017).
From the computational perspective, expansion is helpful for increasing the number of classiﬁcation decision boundaries by a simple perceptron (Cover, 1965) or increasing memory storage capacity in models of associative memory (Hopﬁeld, 1982).
Additionally, sparse expansive representations have been shown to reduce intrastimulus variability and the overlaps between representations induced by distinct stimuli (Sompolinsky, 2014).
Sparseness has also been shown to increase the capacity of models of associative memory (Tsodyks & Feigelman, 1988).
The goal of the present work is to use this “biological” inspiration about sparse expansive motifs, as well as local Hebbian learning, for designing a novel hashing algorithm BioHash that can be used in similarity search.
Below we describe the task, the algorithm, and demonstrate that BioHash improves retrieval performance on common benchmark datasets.
Similarity search and LSH.
In similarity search, given a query q ∈ R d , a similarity measure sim ( q, x ) , and a database X ∈ R n × d containing n items, the objective is to retrieve a ranked list of R items from the database most similar to q .
When data is high-dimensional (e.g. images/documents) and the databases are large (millions or billions items), this is a computationally challenging problem.
However, approximate solutions are generally acceptable, with Locality Sensitive Hashing (LSH) being one such approach (Wang et al., 2014).
Similarity search approaches maybe unsupervised or supervised.
Since labelled information for extremely large datasets is infeasible to obtain, our work focuses on the unsupervised setting.
In LSH (Indyk & Motwani, 1998; Charikar, 2002), the idea is to encode each database entry x
(and query q ) with a binary representation h ( x ) ( h ( q ) respectively) and to retrieve R entries with smallest Hamming distances d H ( h ( x ) , h ( q )) .
Intuitively, (see (Charikar, 2002), for a formal deﬁnition), a hash function h : R d −→ {− 1 , 1 } m is said to be locality sensitive , if similar (dissimilar) items x 1 and x 2 are close by (far apart) in Hamming distance d H ( h ( x 1 ) , h ( x 2 )) .
LSH algorithms are of fundamental importance in computer science, with applications in similarity search, data compression and machine learning (Andoni & Indyk, 2008).
Drosophila olfactory circuit and FlyHash.
In classical LSH approaches, the data dimensionality d is much larger than the embedding space dimension m , resulting in low-dimensional hash codes (Wang et al., 2014; Indyk & Motwani, 1998; Charikar, 2002).
In contrast, a new family of hashing algorithms has been proposed (Dasgupta et al., 2017) where m (cid:29) d , but the secondary representation is highly sparse with only a small number k of m units being active, see Figure 1.
We call this algorithm FlyHash in this paper, since it is motivated by the computation carried out by the ﬂy’s olfactory circuit.
The expansion from the d dimensional input space into an m dimensional secondary representation is carried out using a random set of weights W
(Dasgupta et al., 2017; Caron et al., 2013).
The resulting high dimensional representation is sparsiﬁed by k -Winner-Take-All ( k -WTA) feedback inhibition in the hidden layer resulting in top ∼ 5% of units staying active (Lin et al., 2014;
Stevens, 2016).
While FlyHash uses random synaptic weights, sparse expansive representations are not necessarily random (Sompolinsky, 2014), perhaps not even in the case of Drosophila (Gruntman & Turner, 2013;
Zheng et al., 2018).
Moreover, using synaptic weights that are learned from data might help to further improve the locality sensitivity property of FlyHash .
Thus, it is important to investigate the role of learned synapses on the hashing performance.
A recent work SOLHash (Li et al., 2018), takes inspiration from FlyHash and attempts to adapt the synapses to data, demonstrating improved performance over FlyHash .
However, every learning update step in SOLHash invokes a constrained linear program and also requires computing pairwise inner-products between all training points, making it very time consuming and limiting its scalability to datasets of even modest size.
These limitations restrict SOLHash to training only on a small fraction of the data (Li et al., 2018).
Additionally, SOLHash is biologically implausible (for an extended discussion, see Sec. 5).
BioHash also takes inspiration from FlyHash and demonstrates improved performance compared to random weights used in FlyHash , but it is fast, online, scalable and, importantly, BioHash is neurologically plausible.
[Equation]
[Table]
EPI1AIZfMmJbnJhhkTKPgEsYFPzWQMD5gPWhZqlgEJsimH4zpqVU6tBtrWwrpVP09kbHImFEU2s6IYd8sehPxP6+VYvcqyIRKUgTFZ4u6qaQY00kctCM0cJQjSxjXwt5KeZ9pG4YNrWBD8BZfXib187Lnlr27i1Lleh5HnhyTE3JGPHJJKuSWVEmNcPJInskreXOenBfn3fmYteac+cwh+QPn8wcJlpdB</latexit>
Not only "biological" inspiration can lead to improving hashing techniques, but the opposite might also be true.
One of the statements of the present paper is that BioHash satisﬁes locality sensitive property, and, at the same time, utilizes a biologically plausible learning rule for synaptic weights (Krotov & Hopﬁeld, 2019).
This provides evidence toward the proposal that the reason why sparse expansive representations are so common in biological organisms is because they perform locality sensitive hashing.
In other words, they cluster similar stimuli together and push distinct stimuli far apart.
Thus, our work provides evidence toward the proposal that LSH might be a fundamental computational principle utilized by the sparse expansive circuits Fig.
1 (right).
Importantly, learning of synapses must be neurobiologically plausible (the synaptic plasticity rule should be local).
Contributions.
Building on inspiration from FlyHash and more broadly the ubiquity of sparse, expansive representations in neurobiology, our work proposes a novel hashing algorithm BioHash that produces sparse high dimensional hash codes in a data-driven (as opposed to random synapses in
FlyHash ) manner and with learning of synapses in a neurobiologically plausible way , we provide an existence proof for the proposal that LSH maybe a fundamental computational principle in neural circuits (Dasgupta et al., 2017) in the context of learned synapses.
We incorporated convolutional structure into BioHash , resulting in a hashing with improved performance compared to previously published benchmarks.
From the perspective of computer science, we show that BioHash is simple, scalable to large datasets and demonstrates good performance for similarity search.
Interestingly,
BioHash outperforms a number of recent SOTA deep hashing methods trained via backpropogation.
A PPROXIMATE S IMILARITY S EARCH VIA B IO H ASHING
Formally, if we denote a data point as x ∈ R d , we seek a binary hash code y ∈ {− 1 , 1 } m .
We deﬁne the hash length of a binary code as k , if the exact Hamming distance computation is O ( k ) .
Below we present our bio-inspired hashing algorithm.
2. B IO - INSPIRED H ASHING (B IO H ASH )
We adopt a biologically plausible unsupervised algorithm for representation learning from Krotov &
Hopﬁeld (2019).
Denote the synapses from the input layer to the hash layer as W ∈ R m × d .
The learning dynamics for the synapses of an individual neuron µ , denoted by W µi , is given by
[Equation]
where W µ =
( W µ 1 , W µ 2 ...W µd ) , and g [ µ ] =   1 , µ =− ∆ , µ = r 0 , otherwise (2) and (cid:104) x, y (cid:105) µ = (cid:80) i,j η µi,j x i y j , with η µi,j = | W µi | p − 2 δ ij , where δ ij is Kronecker delta and τ is the time scale of the learning dynamics.
The Rank operation in equation (1) sorts the inner products from the largest ( µ = 1 ) to the smallest ( µ = m ).
It can be shown that the synapses converge to a unit ( p − norm) sphere (Krotov & Hopﬁeld, 2019).
The training dynamics can be shown to minimize the following energy function
[Equation]
where A indexes the training example.
Note that the training dynamics do not perform gradient descent, i.e ˙ W µ (cid:54) =
∇ W µ E .
However, time derivative of the energy function under dynamics (1) is always negative (we show this for the case ∆ = 0 below),
[Equation]
dE
[Equation]
where Cauchy-Schwartz inequality is used.
For every training example A the index of the activated hidden unit is deﬁned as
[Equation]
[Table]
/IHz+QMQCI2m</latexit>
y 3 <latexit sha1_base64="I1Yqf/dyfMgBYEmmGxGuyvCmwQ4=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0laQY9FLx4r2lpoQ9lsJ+3SzSbsboQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IBFcG9f9dgpr6xubW8Xt0s7u3v5B+fCoreNUMWyxWMSqE1CNgktsGW4EdhKFNAoEPgbjm5n/+IRK81g+mEmCfkSHkoecUWOl+0m/3i9X3Ko7B1klXk4qkKPZL3/1BjFLI5SGCap113MT42dUGc4ETku9VGNC2ZgOsWuppBFqP5ufOiVnVhmQMFa2pCFz9fdERiOtJ1FgOyNqRnrZm4n/ed3UhFd+xmWSGpRssShMBTExmf1NBlwhM2JiCWWK21sJG1FFmbHplGwI3vLLq6Rdq3r1au3uotK4zuMowgmcwjl4cAkNuIUmtIDBEJ7hFd4c4bw4787HorXg5DPH8AfO5w8RjI2n</latexit>
Thus, the energy function decreases during learning.
A similar result can be shown for ∆ (cid:54) = 0 .
After the learning-phase is complete, the hash code is generated, as in FlyHash , via WTA sparsiﬁcation: for a given query x we generate a hash code y ∈ {− 1 , 1 } m as y µ
[Equation]
(cid:26)
[Equation]
Thus, the hyperparameters of the method are p, r, m and ∆ .
Note that the synapses are updated based only on pre- and post-synaptic activations resulting in Hebbian or anti-Hebbian updates.
Many "unsupervised" learning to hash approaches provide a sort of "weak supervision" in the form of similarities evaluated in the feature space of deep CNNs trained on ImageNet (Jin et al., 2019) to achieve good performance.
BioHash does not assume such information is provided and is completely unsupervised.
2. I NTUITION BEHIND THE LEARNING ALGORITHM
An intuitive way to think about the learning algorithm is to view the hidden units as particles that are attracted to local peaks of the density of the data, and that simultaneously repel each other.
To demonstrate this, it is convenient to think about input data as randomly sampled points from a continuous distribution.
Consider the case when p = 2 and ∆ = 0 .
In this case, the energy function can be written as (since for p =
the inner product does not depend on the weights, we drop the subscript µ of the inner product)
[Equation]
where we introduced a continuous density of data ρ ( v ) .
Furthermore, consider the case of d = 2 , and imagine that the data lies on a unit circle.
In this case the density of data can be parametrized by a single angle ϕ .
Thus, the energy function can be written as
[Equation]
where
[Equation]
It is instructive to solve a simple case when the data follows an exponential distribution concentrated around zero angle with the decay length σ ( α is a normalization constant),
[Equation]
In this case, the energy (8) can be calculated exactly for any number of hidden units m .
However, minimizing over the position of hidden units cannot be done analytically for general m .
To further simplify the problem consider the case when the number of hidden units m is small.
For m = 2 the energy is equal to
[Equation]
Thus, in this simple case the energy is minimized when
[Equation]
In the limit when the density of data is concentrated around zero angle ( σ → 0 ) the hidden units are attracted to the origin and | ϕ 1 , 2 | ≈ σ .
In the opposite limit ( σ → ∞ ) the data points are uniformly distributed on the circle.
The resulting hidden units are then organized to be on the opposite sides of the circle | ϕ 1 , 2 | = π 2 , due to mutual repulsion.
Another limit when the problem can be solved analytically is the uniform density of the data ρ = 1 / (2 π ) for arbitrary number m of hidden units.
In this case the hidden units span the entire circle homogeneously - the angle between two consecutive hidden units is ∆ ϕ
= 2 π/m .
These results are summarized in an intuitive cartoon in Figure 2, panel A.
After learning is complete, the hidden units, denoted by circles, are localized in the vicinity of local maxima of the probability density of the data.
At the same time, repulsive force between the hidden units prevents them from collapsing onto the exact position of the local maximum.
Thus the concentration of the hidden units near the local maxima becomes high, but, at the same time, they span the entire support (area where there is non-zero density) of the data distribution.
For hashing purposes, trying to ﬁnd a data point x “closest” to some new query q requires a deﬁnition of “distance”.
Since this measure is wanted only for nearby locations q and x , it need not be accurate for long distances.
If we pick a set of m reference points in the space, then the location of point x can be speciﬁed by noting the few reference points it is closest to, producing a sparse and useful local representation.
Uniformly tiling a high dimensional space is not a computationally useful approach.
Reference points are needed only where there is data, and high resolution is needed only where there is high data density.
The learning dynamics in 1 distributes m reference vectors by an iterative procedure such that their density is high where the data density is high, and low where the data density is low.
This is exactly what is needed for a good hash code.
The case of uniform density on a circle is illustrated in Figure 2, panel B. After learning is complete the hidden units homogeneously span the entire circle.
For hash length k = 2 , any given data point activates two closest hidden units.
If two data points are located between two neighboring hidden units (like x and y 1 ) they produce exactly identical hash codes with hamming distance zero between them (black and red active units).
If two data points are slightly farther apart, like x and y 2 , they produce hash codes that are slightly different (black and green circles, hamming distance is equal toin this case).
If the two data points are even farther, like x and y 3 , their hash codes are not overlapping at all (black and magenta circles, hamming distance is equal to 4 ).
Thus, intuitively similar data activate similar hidden units, resulting in similar representations, while dissimilar data result in very different hash codes.
2. C OMPUTATIONAL C OMPLEXITY AND M ETABOLIC C OST .
In classical LSH algorithms (Charikar, 2002; Indyk & Motwani, 1998), typically, k = m and m (cid:28) d , entailing a storage cost of k bits per database entry and O ( k ) computational cost to compute Hamming distance.
In BioHash (and in FlyHash ), typically m (cid:29) k and m > d entailing storage cost of k log 2 m bits per database entry and O ( k ) 1 computational cost to compute Hamming distance.
Note that while there is additional storage/lookup overhead over classical LSH in maintaining pointers, this is not unlike the storage/lookup overhead incurred by quantization methods like Product Quantization (PQ) (Jégou et al., 2011), which stores a lookup table of distances between every pair of codewords for each product space.
From a neurobiological perspective, a highly sparse representation such as the one produced by BioHash keeps the same metabolic cost (Levy & Baxter, 1996) as a dense low-dimensional ( m (cid:28) d ) representation, such as in classical LSH methods.
At the same time, as we empirically show below, it better preserves similarity information.
[Figure]
[Table]
2. C ONVOLUTIONAL B IO H ASH
In order to take advantage of the spatial statistical structure present in images, we use the dynamics in 1 to learn convolutional ﬁlters by training on image patches as in Grinberg et al.
Convolutions in this case are unusual since the patches of the images are normalized to be unit vectors before calculating the inner product with the ﬁlters.
Differently from Grinberg et al.
(2019), we use cross channel inhibition to suppress the activities of the hidden units that are weakly activated.
Speciﬁcally, if there are F convolutional ﬁlters, then only the top k CI of F activations are kept active per spatial location.
Patch normalization is reminiscent of the canonical computation of divisive normalization (Carandini & Heeger, 2011) and performs local intensity normalization.
This is not unlike divisive normalization in the fruit ﬂy’s projection neurons.
Divisive normalization has also been found to be beneﬁcial (Ren et al., 2016) in CNNs trained end-to-end by the backpropogation algorithm on a supervised task.
We found that the cross channel inhibition is important for a good hashing performance.
Post cross-channel inhibition, we use a max-pooling layer, followed by a BioHash layer as in Sec. 2.1.
[Table]
S IMILARITY S EARCH
In this section
[Table]
we empirically evaluate BioHash , investigate the role of sparsity in the latent space
[Table]
and compare our results with previously published benchmarks.
We consider two settings for evaluation: a) the training set contains unlabeled data, and the labels are only used for the evaluation of the performance of the hashing algorithm and b) where supervised pretraining on a different dataset is permissible.
Features extracted from this pretraining are then used for hashing.
In both settings
BioHash outperforms previously published benchmarks for various hashing methods.
3. E VALUATION M ETRIC
Following previous work (Dasgupta et al., 2017; Li et al., 2018;
Su et al., 2018), we use Mean
Average Precision (mAP) as the evaluation metric, a measure that averages precision over different recall.
Speciﬁcally, given a query set Q , we evaluate mAP @
R as
[Equation]
where R is number of retrievals, Rel ( l ) = 1 ( document l is relevant ) and Precision ( l ) is the precision of the top l retrievals.
Notation: when R is equal to size of the entire database, i.e a ranking of the entire database is desired, we use the notation mAP@All or simply mAP, dropping the reference to
R .
3. D ATASETS AND P ROTOCOL
To make our work comparable with recent related work, we used common benchmark datasets: a) MNIST (Lecun et al., 1998), a dataset of 70k grey-scale images (size 28 x 28) of hand-written digits with 10 classes of digits ranging from "0" to "9", b) CIFAR-10 (Krizhevsky, 2009), a dataset containing 60k images (size 32x32x3) from 10 classes (e.g: car, bird).
Following the protocol in Lu et al.
; Chen et al.
(2018), on MNIST we randomly sampleimages from each class to form a query set of 1000 images.
We use the rest of the 69k images as the training set for BioHash as well as the database for retrieval post training.
Similarly, on CIFAR-10, following previous work (Su et al., 2018; Chen et al., 2018; Jin, 2018), we randomly sampledimages per class to create a query set containing 10k images.
The remaining 50k images were used for both training as well as the database for retrieval as in the case of MNIST.
Ground truth relevance for both dataset is based on class labels.
Following previous work (Chen et al., 2018; Lin et al., 2015; Jin et al., 2019), we use mAP@1000 for CIFAR-10 and mAP@All for MNIST.
It is common to benchmark the performance of hashing methods at hash lengths k ∈ { 16 , 32 , 64 } .
However, it was observed in Dasgupta et al.
(2017) that the regime in which FlyHash outperformed LSH was in the regime of low hash lengths k ∈ { 2 , 4 , 8 , 16 , 32 } .
Accordingly, we evaluate performance for k ∈ { 2 , 4 , 8 , 16 , 32 , 64 } .
[Figure]
3. B ASELINES
As baselines we include random hashing methods FlyHash (Dasgupta et al., 2017), classical LSH ( LSH (Charikar, 2002)), and data-driven hashing methods PCAHash
(Gong & Lazebnik, 2011),
Spectral Hashing ( SH (Weiss et al., 2009)), Iterative Quantization ( ITQ (Gong & Lazebnik, 2011)).
As in Dasgupta et al.
(2017), for FlyHash we set the sampling rating from PNs to KCs to be 0 .and m
= 10 d .
Additionally, where appropriate, we also compare performance of BioHash to deep hashing methods: DeepBit (Lin et al., 2015), DH (Lu et al., 2017), USDH (Jin et al., 2019),
UH-BNN (Do et al., 2016), SAH (Do et al., 2017) and GreedyHash (Su et al., 2018).
As previously discussed, in nearly all similarity search methods, a hash length of k entails a dense representation using k units.
In order to clearly demonstrate the utility of sparse expansion in BioHash , we include a baseline (termed " NaiveBioHash "), which uses the learning dynamics in 1 but without sparse expansion, i.e the input data is projected into a dense latent representation with k hidden units.
The activations of those hidden units are then binarized to generate a hash code of length k .
3. R ESULTS AND D ISCUSSION
The performance of BioHash on MNIST is shown in Table 1.
BioHash demonstrates the best retrieval performance, substantially outperforming other methods, including deep hashing methods
DH and UH-BNN , especially at small k .
Indeed, even at a very short hash length of k = 2 , the performance of BioHash is comparable to or better than DH for k ∈ { 16 , 32 } , while at k = 4 , the performance of BioHash is better than the DH and UH-BNN for k ∈ { 16 , 32 , 64 } .
The performance of BioHash saturates around k = 16 , showing only a small improvement from k = 8 to k = 16 and an even smaller improvement from k = 16 to k = 32 ; accordingly, we do not evaluate performance at k = 64 .
We note that while SOLHash also evaluated retrieval performance on MNIST and is a data-driven hashing method inspired by Drosophila’s olfactory circuit, the ground truth in their experiment was top 100 nearest neighbors of a query in the database, based on Euclidean distance between pairs of images in pixel space and thus cannot be directly compared 2 .
Nevertheless, we adopt that protocol (Li et al., 2018) and show that BioHash substantially outperforms SOLHash in
Sec 5.
The performance of BioHash on CIFAR-10 is shown in Table 2.
Similar to the case of MNIST,
BioHash demonstrates the best retrieval performance, substantially outperforming other methods, especially at small k .
Even at k ∈ { 2 , 4 } , the performance of BioHash is comparable to other methods with k ∈ { 16 , 32 , 64 } .
This suggests that BioHash is a particularly good choice when short hash lengths are required.
Effect of sparsity For a given hash length k , we parametrize the total number of neurons m in the hash layer as m × a = k , where a is the activity i.e the fraction of active neurons.
For each hash length k , we varied % of active neurons and evaluated the performance on a validation set
Activity (%): 1.
Activity (%): 5. Activity (%): 20.
Activity (%): 70.
[Table]
(see appendix for details), see Figure 4.
There is an optimal level of activity for each dataset.
For
MNIST and CIFAR-10, a was set to 0 .
05 and 0 .
005 respectively for all experiments.
We visualize tSNE embeddings for different settings of activity levels in Figure 5.
Interestingly, at lower sparsity levels, dissimilar images may become nearest neighbors though highly dissimilar images stay apart.
This is reminiscent of an experimental ﬁnding (Lin et al., 2014) in Drosophila.
Sparsiﬁcation of
Kenyon cells in Drosophila is controlled by feedback inhibition from the anterior paired lateral neuron.
Disrupting this feedback inhibition leads to denser representations, resulting in fruit ﬂies being able to discriminate between dissimilar odors but not similar odors.
Convolutional BioHash
In the case of MNIST, we trained 500 convolutional ﬁlters (as described in Sec. 2.4) of kernel sizes K = 3 , 4 .
In the case of CIFAR-10, we trained 400 convolutional ﬁlters of kernel sizes K = 3 , 4 and 10 .
The convolutional variant of BioHash , which we call
BioConvHash shows further improvement over BioHash on MNIST as well as CIFAR-10, with even small hash lengths k ∈ { 2 , 4 } substantially outperforming other methods at larger hash lengths.
Channel Inhibition is critical for performance of BioConvHash across both datasets, see Table 3.
A high amount of sparsity is essential for good performance.
As discussed previously discussed, convolutions in our network are atypical in yet another way, due to patch normalization.
We show in
Sec 6, that patch normalization results in robustness of BioConvHash to "shadows", a robustness also characteristic of biological vision.
Hashing using deep CNN features State-of-the-art hashing methods generally adapt deep CNNs trained on ImageNet (Su et al., 2018; Jin et al., 2019; Chen et al., 2018; Lin et al., 2017).
These approaches derive large performance beneﬁts from the semantic information learned in pursuit of the classiﬁcation goal on ImageNet (Deng et al., 2009).
To make a fair comparison with our work, we trained BioHash on features extracted from fc7 layer of VGG16 (Simonyan & Zisserman, 2014), since previous work (Su et al., 2018; Lin et al., 2015; Chen et al., 2018) has often adapted this pre-trained network.
BioHash demonstrates substantially improved performance over recent deep unsupervised hashing methods with mAP @ 1000 of 63.47 for k = 16 ; example retrievals are shown in Figure 3.
Even at very small hash lengths of k ∈ { 2 , 4 } , BioHash outperforms other methods at k ∈ { 16 , 32 , 64 } .
For performance of other methods and performance at varying hash lengths see Table 4.
[Table]
It is worth remembering that while exact Hamming distance computation is O ( k ) for all the methods under consideration, unlike classical hashing methods, BioHash (and also FlyHash ) incurs a storage cost of k log 2 m instead of k per database entry.
In the case of MNIST (CIFAR-10), BioHash at k = 2 corresponds to m = 40 ( m = 400 ) entailing a storage cost of 12 ( 18 ) bits respectively.
Even in scenarios where storage is a limiting factor, BioHash at k = 2 compares favorably to other methods at k = 16 , yet Hamming distance computation remains cheaper for BioHash .
C ONCLUSIONS , D ISCUSSION , AND F UTURE W ORK
Inspired by the recurring motif of sparse expansive representations in neural circuits, we introduced a new hashing algorithm, BioHash .
In contrast with previous work (Dasgupta et al., 2017; Li et al., 2018), BioHash is both a data-driven algorithm and has a reasonable degree of biological plausibility.
BioHash demonstrates strong empirical results outperforming recent unsupervised deep hashing methods.
The biological plausibility of our work provides support toward proposal that
LSH might be a general computational function (Valiant, 2014) of the neural circuits featuring sparse expansive representations.
From the perspective of computer science BioHash is easy to train and scalable to large datasets.
Compressed sensing/sparse coding have also been suggested as computational roles of sparse expansive representations in biology (Ganguli & Sompolinsky, 2012).
These ideas, however, require that the input be reconstructable from the sparse latent code.
This is a much stronger assumption than
LSH - downstream tasks might not require such detailed information about the inputs, e.g: novelty detection (Dasgupta et al., 2018).
In this work, we limited ourselves to linear scan using fast Hamming distance computation for image retrieval, like much of the relevant literature (Dasgupta et al., 2017; Su et al., 2018; Lin et al., 2015;
Jin, 2018).
Yet, there is potential for improvement.
One line of future inquiry would be to speed up retrieval using multi-probe methods, perhaps via psuedo-hashes (Sharma & Navlakha, 2018).
Another line of inquiry would be to adapt BioHash for Maximum Inner Product Search (Shrivastava & Li, 2014; Neyshabur & Srebro, 2015).
S UPPLEMENTARY M ATERIAL
We expand on the discussion of related work in section 5.
We also include here some additional results.
In section 6, we show that BioConvHash enjoys robustness to intensity variations.
In section 7, we show that the strong empirical performance of BioHash is not speciﬁc to the choice of VGG16.
Finally, we include technical details about implementation and architecture in section 8.
A PPENDIX A: R ELATED W ORK
Sparse High-Dimensional Representations in Neuroscience.
Previous work has explored the nature of sparse high-dimensional representations through the lens of sparse coding and compressed sensing (Ganguli & Sompolinsky, 2012).
Additionally, Sompolinsky (2014) has examined the computational role of sparse expansive representations in the context of categorization of stimuli as appetitive or aversive.
They studied the case of random projections as well as learned/"structured" projections.
However, structured synapses were formed by a Hebbian-like association between each cluster center and a corresponding ﬁxed, randomly selected pattern from the cortical layer; knowledge of cluster centers provides a strong form a "supervision"/additional information, while BioHash does not assume access to such information.
To the best of our knowledge no previous work has systematically examined the proposal that LSH maybe a computational principle in the brain in the context of structured synapses learned in a biologically plausible manner.
Classical LSH.
A classic LSH algorithm for angular similarity is SimHash (Charikar, 2002), which produces hash codes by h ( x ) = sign ( W (cid:124) x ) , where entries of W ∈ R m × d are i.i.d from a standard normal distribution and sign () is element-wise.
While LSH is a property and consequently is sometimes used to refer to hashing methods in general, when the context is clear we refer to
SimHash as LSH following previous literature.
Fruit Fly inspired LSH.
The fruit ﬂy’s olfactory circuit has inspired research into new families (Dasgupta et al., 2018; Sharma & Navlakha, 2018;
Li et al., 2018) of Locality Sensitive Hashing (LSH) algorithms.
Of these, FlyHash (Dasgupta et al., 2017) and DenseFly (Sharma & Navlakha, 2018) are based on random projections and cannot learn from data.
Sparse Optimal Lifting ( SOLHash ) (Li et al., 2018) is based on learned projections and results in improvements in hashing performance.
SOLHash attempts to learn a sparse binary representation Y ∈ R n × m , by optimizing
[Equation]
13) e m is an all 1 ’s vector of size m .
Note the relaxation from a binary Y ∈ {− 1 , 1 } n × m to continuous Y ∈ [ − 1 , 1] n × m .
After obtaining a Y , queries are hashed by learning a linear map from X to Y by minimizing
[Equation]
Here, c is the # of synapses with weight 1 ; the rest are − 1 .
To optimize this objective, Li et al.
(2018) resorts to Franke-Wolfe optimization, wherein every learning update involves solving a constrained linear program involving all of the training data, which is biologically unrealistic.
In contrast, BioHash is neurobiologically plausible involving only Hebbian/Anti-Hebbian updates and inhibition.
From a computer science perspective, the scalability of SOLHash is highly limited; not only does every update step invoke a constrained linear program but the program involves pairwise similarity matrices, which can become intractably large for datasets of even modest size.
This issue is further exacerbated by the fact that m (cid:29) d and Y Y (cid:124) is recomputed at every step (Li et al., 2018).
Indeed, though Li et al.
(2018) uses the SIFT1M dataset, the discussed limitations limit training to only 5% of the training data.
Nevertheless, we make a comparison to SOLHash in Table 5 and see that
BioHash results in substantially improved performance.
Deep LSH.
A number of state-of-the-art approaches (Su et al., 2018; Jin, 2018; Do et al., 2017; Lin et al., 2015) to unsupervised hashing for image retrieval are perhaps unsurprisingly, based on deep
CNNs trained on ImageNet Deng et al.
A common approach (Su et al., 2018) is to adopt a pretrained DCNN as a backbone, replace the last layer with a custom hash layer and objective [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
SOLHash
[Table]
[Table]
[Table]
[Table]
[Table]
function and to train the network by backpropogation.
Some other approaches (Yang et al., 2018), use DCNNs as feature extractors or to compute a measure of similarity in it’s feature space, which is then used as a training signal.
While deep hashing methods are not the purpose of our work, we include them here for completeness.
A PPENDIX B: R OBUSTNESS OF B IO C ONV H ASH TO VARIATIONS IN INTENSITY
Patch normalization is reminiscent of the canonical neural computation of divisive normalization (Carandini & Heeger, 2011) and performs local intensity normalization.
This makes BioConvHash robust to variations in light intensity.
To test this idea, we modiﬁed the intensities in the query set of CIFAR-10 by multiplying 80% of each image by a factor of 0.3; such images largely remain discriminable to human perception, see Figure 6.
We evaluated the retrieval performance of this query set with "shadows", while the database (and synapses) remain unmodiﬁed.
We ﬁnd that
BioConvHash performs best at small hash lengths, while the performance of other methods except
GreedyHash is almost at chance.
These results suggest that it maybe beneﬁcial to incorporate divisive normalization into DCNNs architectures to increase robustness to intensity variations.
[Table]
A PPENDIX C: E VALUATION USING VGG16BN AND A LEX N ET
The strong empirical performance of BioHash using features extracted from VGG16 fc7 is not speciﬁc to choice of VGG16.
To demonstrate this, we evaluated the performance of BioHash using
VGG16 with batch normalization (BN) (Ioffe & Szegedy, 2015) as well as AlexNet (Krizhevsky et al., 2012).
Consistent with the evaluation using VGG16 reported in the main paper, BioHash consistently demonstrates the best retrieval performance, especially at small k .
[Table] [Table]
A PPENDIX D: I MPLEMENTATION DETAILS
• BioHash : The training /retrieval database was centered.
Queries were also centered using mean computed on the training set.
Weights were initialized by sampling from the standard normal distribution.
For simplicity we used p = 2 , ∆ = 0 .
We set initial learning rate (cid:15)
= 2 × 10 − 2 , which was decayed as (cid:15) t
= (cid:15)
(1 − tT max ) , where t is epoch number and T max is maximum number of epochs.
We used T max = 100 and a mini-batch size of 100.
The criterion for convergence was average norm of synapses was < 1 . 06 .
Convergence usually took < 20 epochs.
In order to set the activity level, we performed cross-validation.
In the case of MNIST, we separated 1k random samples (100 from each class) from the training set, to create a training set of size 68k and validation set of 1k images.
Activity level with highest mAP@All on the validation set was determined to be 5%, see Figure 4.
We then retrained BioHash on the whole training data of size 69k and reported the performance on the query set.
Similarly for CIFAR-10, we separated 1k samples (100 images per class to
create
a training
set of size 49k and validation set of 1k
We set
the activity level
to
be 0.
see
Figure 4.
We
then retrained BioHash on the whole training data of size 50k and reported the performance on the query set.
• BioConvHash A convolutional ﬁlter of kernel size K is learned by dividing the training set into patches of sizes K × K and applying the learning dynamics 1.
In the case of
MNIST, we trained 500 ﬁlters of kernel sizes K = 3 , 4 .
In the case of CIFAR-10, we used
K = 3 , 4 , 10 .
For both datasets, we used a stride of 1 in the convolutional layers.
We set k CI = 10 for MNIST and k CI = 1 for CIFAR-10 , by cross-validation in a procedure similar
to how sparsity was set.
The effect of channel inhibition is shown in Table 3.
k CI = 1 means that only the largest activation across channels per spatial location was kept, while the rest are set to 0.
This was followed by 2d max-pooling with a stride of 2 and kernel size of 7.
This was followed by a fully connected layer (the "hash" layer).
FlyHash
Following Dasgupta et al.
(2017), we set m = 10 d for all hash lengths k and each neuron in the hashing layer ("Kenyon" cell) sampled from 0 .
1 dimensions of input data (Projection neurons).
Following Gong & Lazebnik (2011), ITQ employed 50 iterations.
• To extract representations from VGG16 fc7, CIFAR-10 images were resized to 224 ×and normalized using default values: [0 . 485 , 0 . 456 , 0 . 406] , [0 . 229 , 0 . 224 , 0 . 225] .
To make a fair comparison we used the pre-trained VGG16 model (without BN), since this model is frequently employed by deep hashing methods.
We also evaluated the performance using VGG16 with BN and also using AlexNet (Krizhevsky et al., 2012), see Tables 7, 8. • GreedyHash replaces the softmax layer of VGG16 with a hash layer and is trained endto-end via backpropogation using a custom objective function, see Su et al.
(2018) for more details.
We use the code 3 provided by the authors to measure performance at k = 2 , 4 , 8 , since these numbers were not reported in Su et al.
We used the default parameters: mini-batch size of 32, learning rate of 1 × 10 − 4 and trained for 60 epochs.
A PPENDIX E: T RAINING T IME
Here we report the training times for the best performing (having the highest corresponding mAP@R) variant of our algorithm: BioHash , BioConvHash , or BioHash on top of VGG16 representations.
For the case of MNIST, the best performing variant is BioConvHash , and for CIFAR-10 it is
BioHash on top of VGG16 representations.
We also report the training time of the next best method for each dataset.
This is GreedyHash in the case of CIFAR-10, and BioHash in the case of MNIST.
In the case of MNIST, the best method that is not a variant of BioHash is UH-BNN .
Training time for UH-BNN is unavailable, since it is not reported in literature.
All experiments were run on a single V100 GPU to make a fair comparison.
[Table] [Table]