Paper: SoftHash: High-dimensional Hashing with A Soft Winner-Take-All Mechanism (ICLR-2024)

Reviewer comment (Writing):
There are many places that were not clearly explained, such as, what is ConvHash? what is the difference between SoftHash-1 and Soft-Hash-2?

Author response:
Thanks for your detailed comments. In this paper, ConvHash is the classical LSH method, it generates the hash code of the input data sample $x$ according to $y = sgn(Mx)$, where $\textit{M} \in \mathbb{R}^{m \times k}$ is a random matrix. $k$ is the hash length. We have changed 'ConvHash' to 'LSH' for simplicity. SoftHash-1 and SoftHash-2 have different weight initialization methods, uniform initialization for SoftHash-1, and Gaussian initialization for SoftHash-2. We have added a description of them in Table 2 of the paper.

Reviewer comment (Experiments):
Why not just choose SH and ITQ? there are more similar papers such as SGH [1], DGH [2], COSDISH [3]?

Author response:
Thanks for your comments and recommendation. In this paper, we extend our comparison beyond conventional methods like SH and ITQ, as our proposed SoftHash is an unsupervised bio-inspired hashing method. Therefore, our experiment includes a comparison with other bio-inspired hashing methods. SGH, DGH and COSDISH are well-known hashing methods. We are conducting experiments to evaluate the performance of SGH on image retrieval. Other works have been concluded in the introduction and related works of the paper. Currently, experimental results are shown in the Table R-W2.

Table R-W2

|        |        | Fashion-MNIST |        |        | CIFAR10 |        |        | CIFAR100 |        |
| :----: | :----: | :-----------: | :----: | :----: | :-----: | :----: | :----: | :------: | :----: |
| Method | 32     | 64            | 128    | 32     | 64      | 128    | 32     | 64       | 128    |
| LSH    | 0.1734 | 0.2506   | 0.3348 | 0.0710 | 0.1054  | 0.1446 | 0.0683 | 0.0998   | 0.1375 |
| SH     | 0.3464 | 0.4223   | 0.4675 | 0.1310 | 0.1471  | 0.1819 | 0.1096 | 0.1557   | 0.1870 |
| ITQ    | 0.3133 | 0.4159   | 0.4913 | 0.1763 | 0.2209  | 0.2605 | 0.1672 | 0.2128   | 0.2552 |
| SGH    | 0.3095 | 0.3488   | 0.3941 | 0.1133 | 0.1362  | 0.1690 | 0.1245 | 0.1560   | 0.1869 |

Reviewer comment (Novelty):
What is the main differences between the bio-inspired hashing methods with sparse-and-high-dimensional {0,1}-vectors and the conventional hashing methods with dense-and-low-dimensional {0,1}-vectors?

Author response:
The main differences are
(1) the bio-inspired hashing methods project input data into high-dimensional representations, whereas conventional hashing methods map the data into low-dimensional representations;
(2) the bio-inspired hashing methods sparsify the high-dimensional representation using a winner-take-all mechanism, whereas conventional hashing methods preserve a dense representation.
In general, the high-dimensional sparse representations are always more robust to encoding noisy input data [1].
With the help of the bio-winner-take-all mechanism, the representations can also achieve a good performance in semantic clustering.

[1] How can we be so dense? The Benefits of Using Highly Sparse Representations

Reviewer comment (Theory):
How does the bio-inspired hashing codes capture the data samples' semantics?

Author response:
Most of the bio-inspired hashing codes are generated with the winner-take-all mechanism. 
The kernel assumption of the winner-take-all mechanism is that with an input vector $x$, if the semantic information of attribute $x_i$ is dominant over some other attribute, $x_i$ has stronger discriminative powers to represent $x$ and cluster $x$ [1]. This allows the most semantical features can be more utilized than other noisy features in the hash mapping. With different initialization of synaptic weights and different hash functions, capturing different dominant semantic information, can be learned with the bio-inspired learning rule. In this way, the data samples’ semantics are kept in the hash codes. 
We visualize the semantic capturing capabilities of bio-inspired hashing codes by comparing them to traditional hashing algorithms in Figure 3 in the paper. It is obvious that the cluster structure obtained by $\textit{SoftHash}$ is more compact compared to traditional $\textit{LSH}$ when using the same hash length. It indicates that the hash codes generated by $\textit{SoftHash}$ effectively capture more semantic information of data samples than those of traditional LSH.

[1] Revisiting winner-take-all all (WTA) Hashing for Sparse Datasets

Reviewer comment (Evaluation):
With respect to the storage and computation, how does the bio-inspired hashing approaches realize economic memory and fast retrieval/semantic computing?

Author response:
As the same as most traditional LSH methods, bio-inspired hashing methods construct only a binary hash index for the nearest neighbor search. 
No other additional storage is required, and the binary form of the hash codes is very friendly to storage and search, bringing about an economic memory for semantic similarity computing.  
With respect to the index data structure, compared to traditional LSH methods with dense hash codes, bio-inspired methods use sparse hash codes to perform similarity search. 
In a hamming space, sparse vectors can be computed much faster than dense vectors.
What's more, as introduced in the previous response, the winner-take-all mechanism ensures the output representations of bio-inspired hashing methods convey more semantic information. 
Therefore, with the above, bio-inspired hashing methods can implement fast retrieval/semantic computing.

Reviewer comment (Novelty):
Some similar contents have already appeared in other works. For example, the paragraph saying "$w_i$ is implicitly normalized" is similar to the text around Eq (3) in the BioHash paper (except that $Topk(y)$ is replaced by $y$). Eq (7) also has similar form in the BioHash paper. A clear comparison should be made in the paper regarding the difference and connections.

Author response:
R-W1&W2 Our algorithm design considers a lot about "winner-take-all" dilemma [1]. 
Many winner-take-all hashing methods like BioHash may fall into the dilemma,
where the semantic representation of the learned hash codes may be dominated by the majority class, leading to the semantics loss of the minor classes, which greatly limits its ability to catch the semantic information of the input data. 

In the energy function of BioHash (Eq.(10)), it is clear that each input sample belongs to only one neuron. 
It means only the output neuron with the largest value can be updated during the learning process, leading to the dilemma.
For our hashing method, to alleviate this dilemma, we introduce a soft winner-take-all mechanism for learning the hash function. 
In contrast to the mentioned methods using a "hard" winner-take-all mechanism, our method makes non-winning classes not fully suppressed in the learning process.
By assigning a credit score $\frac{e^{u_j+b_{j}}}{\sum_{l=1}^m e^{u_l+b_{l}}}$ in Eq (5) to each class, our method enables the minor classes to have a chance to influence the representative learning, making the learned hash codes improved with more semantic information. We have a detailed description of the connections and differences between SoftHash and BioHash in Appendix B.

$\frac{e^{u_j+b_{j}}}{\sum_{l=1}^m e^{u_l+b_{l}}}$ in Eq (5) is in the form of softmax function.
The reason why we use softmax here is to assign each class with a normalized credit score.
Compared to other normalization functions, softmax is more general and smooth.
The biological plausibility of "softmax" sources from the lateral inhibition between neurons, is mentioned in [2,3,4].

[1] Domain Adaptive Semantic Segmentation without Source Data

[2] SoftHebb: Bayesian inference in unsupervised Hebbian soft winner-take-all networks

[3] Bayesian computation emerges in generic cortical microcircuits through spike-timing-dependent plasticity

[4] Optimality of short-term synaptic plasticity in modeling certain dynamic environments

Reviewer comment (Experiments):
SoftHash-1 and SoftHash-2 are not explained in the main paper, please revise. From the appendix, it seems that they refer to 2 different weight initialiation methods (uniform and Gaussian). Why is the performance gap so big given that this approach is learning-based? This is confusing to me. Also, why is the batch size set as 3584? Is that carefully picked or just a random number?

Author response:
R-W3: 
(1) SoftHash-1 and SoftHash-2 have different weight initialization methods, uniform initialization for SoftHash-1, and Gaussian initialization for SoftHash-2.
The reason we believe that SoftHash-2 performs better is that Gaussian initialization is more effective in breaking symmetry compared to uniform initialization, leading to more representative feature combinations at the algorithm's beginning.
In Gaussian initialization, weights are randomly sampled from a normal distribution, whereas in uniform initialization, weights are drawn from a uniform distribution. 
Gaussian initialization ensures that each neuron learns distinct features from the beginning, and improves the network's ability to capture input data characteristics during the learning process, enabling the learnable synaptic weights to generate more representative hash codes. 
We have added an ablation study in section 4.2 and descriptions of SoftHash-1 (SoftHash-2) in Table 2.
We are sorry for the typo. 
In our study, the batch size is 1024 for all three datasets.
It is determined by maximizing the usage of GPU memory.
(2) For ease of comparison, we repeat the parameter setting of the previous work[1] for BioHash.
For SoftHash, we have tuned the temperature parameter $T$ with \{5,10, 15, 20\} and set it to 10 according to its best performance.

[1] Unsupervised Learning by Competing Hidden Units

Reviewer comment (Experiments):
For BioHash, why you used $p=4$ and $\triangle=0.4$ and $0.3$? For SoftHash, how is the temperature parameter $T$ chosen? Did you tune these parameters?

Author response:
R-W3: 
(1) SoftHash-1 and SoftHash-2 have different weight initialization methods, uniform initialization for SoftHash-1, and Gaussian initialization for SoftHash-2.
The reason we believe that SoftHash-2 performs better is that Gaussian initialization is more effective in breaking symmetry compared to uniform initialization, leading to more representative feature combinations at the algorithm's beginning.
In Gaussian initialization, weights are randomly sampled from a normal distribution, whereas in uniform initialization, weights are drawn from a uniform distribution. 
Gaussian initialization ensures that each neuron learns distinct features from the beginning, and improves the network's ability to capture input data characteristics during the learning process, enabling the learnable synaptic weights to generate more representative hash codes. 
We have added an ablation study in section 4.2 and descriptions of SoftHash-1 (SoftHash-2) in Table 2.
We are sorry for the typo. 
In our study, the batch size is 1024 for all three datasets.
It is determined by maximizing the usage of GPU memory.
(2) For ease of comparison, we repeat the parameter setting of the previous work[1] for BioHash.
For SoftHash, we have tuned the temperature parameter $T$ with \{5,10, 15, 20\} and set it to 10 according to its best performance.

[1] Unsupervised Learning by Competing Hidden Units

Reviewer comment (Experiments):
Does BioHash have the bias term $b$ in the model? How important is this bias term? Any empirical results for illustration?

Author response:
R-Q1: BioHash does not have the bias term. 
From the view of neuroscience, neuron activation occurs when the input signals reach a certain threshold, which is regarded as the neuron bias.
Different neurons have different thresholds.
Therefore, in order to precisely model the activities of neurons, the bias term should be seriously considered in the model. 
In practice, without the bias term, the learning algorithm may find it hard to fit the input samples, especially when the diversity of input features is large.
As a result, the learned hash codes may lose effectiveness in finding similar data samples as shown in Table R-Q1.

|            |        | Fashion-MNIST |        |        | CIFAR10 |        |        | CIFAR100 |        |
| :--------: | :----: | :-----------: | :----: | :----: | :-----: | :----: | :----: | :------: | :----: |
| Method     | 32     | 64            | 128    | 32     | 64      | 128    | 32     | 64       | 128    |
| SoftHash_with_bias | 0.5080 | 0.6003        | 0.6712 | 0.2632 | 0.3570  | 0.4681 | 0.2872 | 0.4038   | 0.5163 |
| SoftHash_without_bias | 0.5062 | 0.5952        | 0.6690 | 0.2217 | 0.2722  | 0.3104 | 0.2400 | 0.2899   | 0.3262 |

Reviewer comment (Writing):
I think "ConvHash" is usually referred to as "SimHash" or simply "LSH". Perhaps changing the name would be better.

Author response:
R-Q2: We have changed "ConvHash" to "LSH" in the paper.

Reviewer comment (Experiments):
In Table 2, what are the metrics when the code length is larger (256, 512)?

Author response:
R-Q4: In Table 2, we use mean average precision (MAP) for evaluation when the code length is larger. In the image retrieval task, we use MAP to measure the hamming ranking quality of various code lengths.

R-Q4: You are right. In our work, longer codes (k=256, 512) show better retrieval performance on three datasets (Appendix D.2). We will add k=1024 to make the results more complete in the final version.

Reviewer comment (Theory):
Is there an interpretation of Oja's rule in terms of gradient-based optimization? It seems that Eq (2) is the gradient of $||x-wy||^2$? Is it related to the clustering formulation you mentioned?

Author response:
R-Q5: Your observation is keen, which enlightens us a lot. 
Next, we will try to explain Oja's rule from the view of gradient descent. 
First, let us look at the following equation
\begin{equation}
  \frac{\partial\|(x_i - w_{j}^iy_j)\|^2}{\partial w_j^i } = -2y_j (x_i - w_{j}^iy_j)
\end{equation}
It is interesting to find that the gradient of $\|(x_i - w_{j}^iy_j)\|^2$ is the same as the Oja rule (Eq (2)).

The $w_{j}^{i}$ in the numerator is just a regularization term that controls $w_{j}^{i}$ not to grow too big. 
If we ignore this term, the update rule of Oja can be rewritten to minimize the loss function as 
\begin{equation}
\underset{w_j^i}{\text{minimize}} \sum_{i=1}^d \|(x_i-y_j)\|^2 
\end{equation}
where $y_j=\sum_i x_i w_{j}^i$. 
It is easy to see the loss function expects the output neuron ($y_j$) can act similarly as most of the input neurons ($x_i$s) as possible. 
Therefore, at the end of this unsupervised learning, $y_j$ will be determined by the most dominant correlated $x_i$s. 
In other words, the input samples with similar values on the dominant semantic features will have similar hashing values and be hashed into the same bucket. 

This finding is very interesting.

We will add it to our final paper.

Thanks for your inspiration.

Reviewer comment (Experiments):
There is a recent paper [SignRFF: Sign Random Fourier Features, NeurIPS 2023] that extends SimHash to non-linear features. It may perform better than SimHash on some datasets. Please consider adding this baseline as well as deep hashing methods to make the experiments stronger.

Author response:
R-Q6: We have added SignRFF as an important reference in our section of related work. Next, we are going to do our best to implement SignRFF as a comparative baseline of our experiment in our final version. 

Thank you again for your valuable feedback.

Reviewer comment (Reproducibility):
The experiment part was not clear to me. Many important details are in the appendix or are missing, such as the train/test set split.

Author response:
We have added the description of the train/test set split in the $\textit{Datasets}$ part of section 4.1 and rewritten the experiment settings of section 4.1 and section 5.1 in $\textcolor{blue}{BLUE}$ color.

Reviewer comment (Experiments):
Appendix D.1 mentioned that for test: "10 query per class for cifar100, 100 for cifar10". The test size seems small to me. In BioHash, they used 1000 queries per class for cifar10. Any reason why you chose a smaller query size?

Author response:
To further illustrate the effectiveness of our algorithm, in Table R-Q1, we add the experimental results of "10,000 images with 10 classes". It shows that when the query size becomes large, SoftHash still outperforms other hashing methods with different hash lengths. 

 Table R-Q1
|          |        | Fashion-MNIST |        |        | CIFAR10 |        |        | CIFAR100 |        |
| :------: | :----: | :-----------: | :----: | :----: | :-----: | :----: | :----: | :------: | :----: |
| Method   | 32     | 64            | 128    | 32     | 64      | 128    | 32     | 64       | 128    |
| SH       | 0.3492 | 0.4301        | 0.4743 | 0.1369 | 0.1518  | 0.1905 | 0.1136 | 0.1569   | 0.1962 |
| ITQ      | 0.3513 | 0.4305        | 0.4949 | 0.1901 | 0.2337  | 0.2685 | 0.1963 | 0.2418   | 0.2769 |
| BioHash  | 0.4830 | 0.5270        | 0.5529 | 0.2139 | 0.2806  | 0.3289 | 0.2422 | 0.3066   | 0.3501 |
| SoftHash | 0.5205 | 0.6072        | 0.6786 | 0.2791 | 0.3748  | 0.4840 | 0.3021 | 0.4003   | 0.5064 |

Reviewer comment (Evaluation):
Appendix D.1 mentioned that "Ground truth is the top 1000 nearest neighbors of a query in the database, based on Euclidean distance between pairs of images in pixel space." whereas, in BioHash, the ground truth is based on class labels. For me, it makes more sense to use class labels. Is there any reason for using Euclidean distance?

Author response:
In this paper, we focus on designing a data-dependent hashing algorithm $\textit{SoftHash}$ to encode dense continuous features into sparse binary codes for fast searching, meanwhile, adapting semantic knowledge from input data features to output hash codes for semantic clustering.
Considering the above, we do not choose class-label-based evaluation, which is a bit coarse to reflect the effect of semantic clustering of our algorithm. On the other hand, Euclidean-distance-based evaluation is selected, which is more soft and refined to assess the semantic representativeness of output hash codes. Moreover, some datasets like Glove naturally have no class labels, which can not be conducted with class-label-based evaluation.

Reviewer comment (Theory):
Appendix D.1 mentioned that "Output dim is 2000". It is smaller than the 3072-dim input in cifar10/cifar100. Can you explain if this still satisfies high-dimensional hash?

Author response:
The reason we set the output dim as 2000 is motivated by the fact that in the factory circuit of Fruit Fly there are approximately 2000 Kenyon cells for information encoding [1]. Considering this biological basis,  2000 is used as the output dim in the experiments.  "2000 is high" does not mean that the output dimension is higher compared to the input dimension. Here, we say "2000 is high" which means that compared to the traditional LSH methods like [2,3] hashing data into dense low-dimensional(d < 2000) space, our method's 2000 output dimension is a higher number. The main objective of the experiments is to show the superiority of our sparse high-dimensional hash representation over other dense low-dimensional hash representations. 

[1] A neural algorithm for a fundamental computing problem; [2] Spectral hashing; [3] Iterative quantization: A procrustean approach to learning binary codes for large-scale image retrieval.

Reviewer comment (Reproducibility):
I couldn't find train/test split and output dimensions for word search experiments.

Author response:
Thank you for your comments. We have rewritten the description of the experiment settings in Section 4.1 and Section 5.1, including train/test split in the $\textit{Datasets}$ part of Section 4.1. The output dimensions for the word search experiments were set to 2,000. It has been added in Appendix E.1.

Reviewer comment (Presentation):
The reference for "Can a fruit fly learn word embeddings?" needs to be updated: not arxiv preprint; it was published at iclr 2021.

Author response:
Thank you for your correction. We have made the modifications in the references.

Reviewer comment (Experiments):
It would be better to include more baselines including Mongoose paper's learnable hash functions[1].

Author response:
Thank you for pointing out this excellent reference. We are in the process of reimplementing MONGOOSE and testing it on the evaluation benchmarks we have used. We will add the experimental results to our paper once it is available.

Reviewer comment (Theory):
Maybe the authors could justify more on the theoretical analysis of the motivation of Softhash. Similar studies would be [2]

Author response:
Thank you for your comments. The key idea of SoftHash originates from Drosophila’s olfactory circuit [3], which utilizes sparse spike codes to convey the information of dense sensory signals. In particular, Hebbian-like rules are often used to learn such sparse codes. In this paper, our Hebbian-like learning rule is designed as $y_jx_i-y_ju_jw_j^i$, which works similarly to the Oja rule [4]. If the input and output are highly correlated, the first term will enhance the connection weight between the input and output neuron; The second term will give a punishment update when the weight is already large enough. The item $y_j$ is computed according to the Bayesian posterior probability, which allows every possible hash class to be able to influence the update at each iteration. In this way, our hash function can be trained in a soft manner.

Unlike randomized hash functions, the theoretical bounds for learning-based hashing methods are often lacking [5]. Although recently [2] proves a correctness bound for its tree-learning-based hashing method, its theoretical analysis cannot be easily applied to other learning-based methods, such as neural-network-based hashing methods. The reason is that [2] is a white-box algorithm, in which independence and randomness are introduced by the authors’ delicate design. With the well-designed randomness setting, the theoretical bound can be achieved as expected. However, neural-network-based hashing methods like ours are black-box algorithms, in which the function weight may converge to any saddle point with unknown probability. Therefore, it is challenging to provide a theoretical guarantee for our method. However, from the view of machine learning, our algorithm can be treated as an energy-based model as described in Equation (7) in the paper. We have proved that our learning rule can consistently reduce the energy value in Appendix A, which ensures the convergence of our method. Furthermore, we also provided an analysis on the time complexity of the algorithm.

Reviewer comment (Theory):
How to provide search quality guarantees for Softhash in retrieval?

Author response:
Thank you for your comments. The key idea of SoftHash originates from Drosophila’s olfactory circuit [3], which utilizes sparse spike codes to convey the information of dense sensory signals. In particular, Hebbian-like rules are often used to learn such sparse codes. In this paper, our Hebbian-like learning rule is designed as $y_jx_i-y_ju_jw_j^i$, which works similarly to the Oja rule [4]. If the input and output are highly correlated, the first term will enhance the connection weight between the input and output neuron; The second term will give a punishment update when the weight is already large enough. The item $y_j$ is computed according to the Bayesian posterior probability, which allows every possible hash class to be able to influence the update at each iteration. In this way, our hash function can be trained in a soft manner.

Unlike randomized hash functions, the theoretical bounds for learning-based hashing methods are often lacking [5]. Although recently [2] proves a correctness bound for its tree-learning-based hashing method, its theoretical analysis cannot be easily applied to other learning-based methods, such as neural-network-based hashing methods. The reason is that [2] is a white-box algorithm, in which independence and randomness are introduced by the authors’ delicate design. With the well-designed randomness setting, the theoretical bound can be achieved as expected. However, neural-network-based hashing methods like ours are black-box algorithms, in which the function weight may converge to any saddle point with unknown probability. Therefore, it is challenging to provide a theoretical guarantee for our method. However, from the view of machine learning, our algorithm can be treated as an energy-based model as described in Equation (7) in the paper. We have proved that our learning rule can consistently reduce the energy value in Appendix A, which ensures the convergence of our method. Furthermore, we also provided an analysis on the time complexity of the algorithm.