Original: L EARNING H ASH C ODES VIA H AMMING D ISTANCE T ARGETS
Revised: L EARNING H ASH C ODES VIA H AMMING D ISTANCE T ARGETS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We present a powerful new loss function and training scheme for learning binary hash codes with any differentiable model and similarity function.
Revised: We present a powerful new loss function and training scheme for learning binary hash codes with any differentiable model and similarity function.

Original: Our loss function improves over prior methods by using log likelihood loss on top of an accurate approximation for the probability that two inputs fall within a Hamming distance target.
Revised: Our loss function improves over prior methods by using log likelihood loss on top of an accurate approximation for the probability that two inputs fall within a Hamming distance target.

Original: Our novel training scheme obtains a good estimate of the true gradient by better sampling inputs and evaluating loss terms between all pairs of inputs in each minibatch.
Revised: Our novel training scheme obtains a good estimate of the true gradient by better sampling inputs and evaluating loss terms between all pairs of inputs in each minibatch.

Original: To fully leverage the resulting hashes, we use multi-indexing.
Revised: To fully leverage the resulting hashes, we use multi-indexing.

Original: We demonstrate that these techniques provide large improvements to a similarity search tasks.
Revised: We demonstrate that these techniques provide large improvements to a similarity search tasks.

Original: We report the best results to date on competitive information retrieval tasks for ImageNet and SIFT 1M, improving MAP from 73% to 84% and reducing query cost by a factor of 2-8, respectively
Revised: We report the best results to date on competitive information retrieval tasks for ImageNet and SIFT 1M, improving MAP from 73% to 85% and reducing query cost by a factor of 2-8, respectively

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Many information retrieval tasks rely on searching high-dimensional datasets for results similar to a query.
Revised: Many information retrieval tasks rely on searching high-dimensional datasets for results similar to a query.

Original: Recent research has ﬂourished on these topics due to enormous growth in data volume and industry applications Wang et al.
Revised: Recent research has ﬂourished on these topics due to enormous growth in data volume and industry applications Wang et al.

Original: These problems are typically solved in either two steps by computing an embedding and then doing lookup in the embedding space, or in one step by learning a hash function.
Revised: These problems are typically solved in either two steps by computing an embedding and then doing lookup in the embedding space, or in one step by learning a hash function.

Original: We call these three problems the data-to-embedding problem, the embedding-toresults problem, and the data-to-results problem.
Revised: We call these three problems the data-to-embedding problem, the embedding-toresults problem, and the data-to-results problem.

Original: There exists an array of solutions for each one.
Revised: There exists an array of solutions for each one.

Original: Models that solve data-to-embedding problems aim to embed the input data in a space where proximity corresponds to similarity.
Revised: Models that solve data-to-embedding problems aim to embed the input data in a space where proximity corresponds to similarity.

Original: The most commonly chosen embedding space is R n , in order to leverage lookup methods that assume Euclidean distance.
Revised: The most commonly chosen embedding space is R n , in order to leverage lookup methods that assume Euclidean distance.

Original: Recent methods employ neural network architectures for embeddings in speciﬁc domains, such as facial recognition and sentiment analysis
Revised: Recent methods employ neural network architectures for embeddings in speciﬁc domains, such as facial recognition and sentiment analysis

Original: Schroff et al.
Revised: Schroff et al.

Original: (2015); Mikolov et al.
Revised: (2015); Mikolov et al.

Original: Once the data-to-embedding problem is solved, numerous embedding-to-results strategies exist for similarity search in a metric space.
Revised: Once the data-to-embedding problem is solved, numerous embedding-to-results strategies exist for similarity search in a metric space.

Original: For this step, the main challenge is achieving high recall with low query cost.
Revised: For this step, the main challenge is achieving high recall with low query cost.

Original: Exact k -nearest neighbors (KNN) algorithms achieve 100% recall, ﬁnding the k closest items to the query in the dataset, but they can be prohibitively slow.
Revised: Exact k -nearest neighbors (KNN) algorithms achieve 100% recall, ﬁnding the k closest items to the query in the dataset, but they can be prohibitively slow.

Original: Brute force algorithms that compare distance to every other element of the dataset are often the most viable KNN methods, even with large datasets.
Revised: Brute force algorithms that compare distance to every other element of the dataset are often the most viable KNN methods, even with large datasets.

Original: Recent research has enabled exact KNN on surprisingly large datasets with low latency Johnson et al.
Revised: Recent research has enabled exact KNN on surprisingly large datasets with low latency Johnson et al.

Original: However, the compute resources required are still large.
Revised: However, the compute resources required are still large.

Original: Alternatives exist that can reduce query costs in some cases, but increase insertion time.
Revised: Alternatives exist that can reduce query costs in some cases, but increase insertion time.

Original: For instance, k -d trees require O
Revised: For instance, k -d trees require O

Original: (log N ) search time on average with a high constant, but also require O (log N ) insertion time on average.
Revised: (log N ) search time on average with a high constant, but also require O (log N ) insertion time on average.

Original: Approximate nearest neighbors algorithms solve the embedding-to-results problem by ﬁnding results that are likely, but not guaranteed to be among the k closest.
Revised: Approximate nearest neighbors algorithms solve the embedding-to-results problem by ﬁnding results that are likely, but not guaranteed to be among the k closest.

Original: Similarly, approximate nearneighbor algorithms aim to ﬁnd most of the results that fall within a speciﬁc distance of the query’s embedding.
Revised: Similarly, approximate nearneighbor algorithms aim to ﬁnd most of the results that fall within a speciﬁc distance of the query’s embedding.

Original: These tasks (ANN) are generally achieved by hashing the query embedding, then looking up and comparing results under hashes close to that hash.
Revised: These tasks (ANN) are generally achieved by hashing the query embedding, then looking up and comparing results under hashes close to that hash.

Original: Approximate methods can be highly advantageous by providing orders of magnitude faster queries with constant insertion time.
Revised: Approximate methods can be highly advantageous by providing orders of magnitude faster queries with constant insertion time.

Original: Localitysensitive hashing (LSH) is one such method that works by generating multiple, randomly-chosen hash functions for each input.
Revised: Localitysensitive hashing (LSH) is one such method that works by generating multiple, randomly-chosen hash functions for each input.

Original: Each element of the dataset is inserted into multiple hash tables, one for each hash function.
Revised: Each element of the dataset is inserted into multiple hash tables, one for each hash function.

Original: Queries can then be made by checking all hash tables for similar results.
Revised: Queries can then be made by checking all hash tables for similar results.

Original: Another approach is quantization, which solves ANN problems by partitioning the space of inputs into buckets.
Revised: Another approach is quantization, which solves ANN problems by partitioning the space of inputs into buckets.

Original: Each element of the dataset is inserted into its bucket, and queries are made by selecting from multiple buckets close to the query.
Revised: Each element of the dataset is inserted into its bucket, and queries are made by selecting from multiple buckets close to the query.

Original: Data-to-results methods determine similarity between inputs and provide an efﬁcient lookup mechanism in one step.
Revised: Data-to-results methods determine similarity between inputs and provide an efﬁcient lookup mechanism in one step.

Original: These methods directly compute a hash for each input, showing promise of simplicity and efﬁciency.
Revised: These methods directly compute a hash for each input, showing promise of simplicity and efﬁciency.

Original: Additionally, machine learning methods in this category train end-to-end, by which they can reduce inefﬁciencies in the embedding step.
Revised: Additionally, machine learning methods in this category train end-to-end, by which they can reduce inefﬁciencies in the embedding step.

Original: There has been a great deal of recent research into these methods in topics such as content-based image retrieval (CBIR).
Revised: There has been a great deal of recent research into these methods in topics such as content-based image retrieval (CBIR).

Original: In other topics such as automated scene matching, hand-chosen hash functions are common Ansari & Mohammed (2015).
Revised: In other topics such as automated scene matching, hand-chosen hash functions are common Ansari & Mohammed (2015).

Original: But despite recent focus, data-to-results methods have had mixed results in comparison to data-to-embedding methods paired with embedding-to-results lookup Wang et al.
Revised: But despite recent focus, data-to-results methods have had mixed results in comparison to data-to-embedding methods paired with embedding-to-results lookup Wang et al.

Original: ; Klein &
Revised: ; Klein &

Original: Wolf (2017).
Revised: Wolf (2017).

Original: We assert the main reason data-to-results methods have sometimes underperformed is that training methods have not adequately expressed the model’s loss.
Revised: We assert the main reason data-to-results methods have sometimes underperformed is that training methods have not adequately expressed the model’s loss.

Original: Our proposed approach trains neural networks to produce binary hash codes for fast retrieval of results within a Hamming distance target.
Revised: Our proposed approach trains neural networks to produce binary hash codes for fast retrieval of results within a Hamming distance target.

Original: These hash codes can be efﬁciently queried within the same Hamming distance by multi-indexing
Revised: These hash codes can be efﬁciently queried within the same Hamming distance by multi-indexing

Original: Norouzi et al.
Revised: Norouzi et al.

Original: 1. R ELATED W ORK
Revised: 1. R ELATED W ORK

Original: Additional context in quantization and learning to hash is important to our work.
Revised: Additional context in quantization and learning to hash is important to our work.

Original: Quantization is considered state-of-the-art in ANN tasks Wang et al.
Revised: Quantization is considered state-of-the-art in ANN tasks Wang et al.

Original: There are many quantization approaches, but two are particularly noteworthy: iterative quantization (ITQ)
Revised: There are many quantization approaches, but three are particularly noteworthy: iterative quantization (ITQ)

Original: Gong & Lazebnik (2013) and product quantization (PQ) J ´ egou et al. 
Revised: Gong & Lazebnik (2013), product quantization (PQ) J ´ egou et al. (2011), and multi-scale quantization (MSQ) Wu et al.

Original: Iterative quantization learns to produce binary hashes by ﬁrst reducing dimensionality and then minimizing a quantization loss term, a measure of the amount of information lost by quantizing.
Revised: Iterative quantization learns to produce binary hashes by ﬁrst reducing dimensionality and then minimizing a quantization loss term, a measure of the amount of information lost by quantizing.

Original: ITQ uses principal component analysis for dimensionality reduction and || sgn ( v ) − v || 2 for a quantization loss term, where v is the pre-binarized output and sgn ( v ) is the quantized hash.
Revised: ITQ uses principal component analysis for dimensionality reduction and || sgn ( v ) − v || 2 for a quantization loss term, where v is the pre-binarized output and sgn ( v ) is the quantized hash.

Original: It then minimizes quantization loss by alternately updating an offset and then a rotation matrix for the embedding.
Revised: It then minimizes quantization loss by alternately updating an offset and then a rotation matrix for the embedding.

Original: PQ is a generally more powerful quantization method that splits the embedding space R n into R n/M × R n/M × . . .
Revised: PQ is a generally more powerful quantization method that splits the embedding space R n into R n/M × R n/M × . . .

Original: R n/M .
Revised: R n/M .

Original: A k -means algorithm is run on the embedding constrained to each R n/M subspace, giving k Voronoi cells in each subspace for a total of k m hash buckets.
Revised: A k -means algorithm is run on the embedding constrained to each R n/M subspace, giving k Voronoi cells in each subspace for a total of k m hash buckets.

Revised: MSQ builds on PQ by separately quantizing the magnitude and directions of each vector, breaking R n into R × S n − 1 .

Original: Recent methods that learn to hash end-to-end draw from a few families of loss terms to train binary codes Wang et al.
Revised: Recent methods that learn to hash end-to-end draw from a few families of loss terms to train binary codes Wang et al.

Original: These include terms for supervised softmax cross entropy between codes
Revised: These include terms for supervised softmax cross entropy between codes

Original: Jain et al.
Revised: Jain et al.

Original: (2017), supervised Euclidean distance between codes Liu et al.
Revised: (2017), supervised Euclidean distance between codes Liu et al.

Original: (2016), and quantization loss terms Zhou et al.
Revised: (2016), and quantization loss terms Zhou et al.

Original: Softmax cross entropy and Euclidean distance losses assume that
Revised: Softmax cross entropy and Euclidean distance losses assume that

Original: Hamming distance corresponds to Euclidean distance in the pre-binarized outputs.
Revised: Hamming distance corresponds to Euclidean distance in the pre-binarized outputs.

Original: Some papers try to enforce that assumption in a few different ways.
Revised: Some papers try to enforce that assumption in a few different ways.

Original: For instance, quantization loss terms aim to make that assumption more true by penalizing networks for producing outputs far from ± 1 .
Revised: For instance, quantization loss terms aim to make that assumption more true by penalizing networks for producing outputs far from ± 1 .

Original: Alternative methods to force outputs close to ± 1 exist, such as HashNet, which gradually sharpens sigmoid functions on the pre-binarized outputs.
Revised: Alternative methods to force outputs close to ± 1 exist, such as HashNet, which gradually sharpens sigmoid functions on the pre-binarized outputs.

Original: Another family of methods ﬁrst learns a target hash code for each class, then minimizes distance between each embedding and its target hash code Xia et al.
Revised: Another family of methods ﬁrst learns a target hash code for each class, then minimizes distance between each embedding and its target hash code Xia et al.

Original: ; Lu et al.
Revised: ; Lu et al.

Original: We observed four main shortcomings of existing methods that learn to hash end-to-end.
Revised: We observed four main shortcomings of existing methods that learn to hash end-to-end.

Original: First, cross entropy and Euclidean distance between pre-binarized outputs does not correspond to Hamming distance under almost any circumstances.
Revised: First, cross entropy and Euclidean distance between pre-binarized outputs does not correspond to Hamming distance under almost any circumstances.

Original: Second, quantization loss and learning by continuation cause gradients to shrink during training, dissuading the model from changing the sign of any output.
Revised: Second, quantization loss and learning by continuation cause gradients to shrink during training, dissuading the model from changing the sign of any output.

Original: Third, methods using target hash codes are limited to classiﬁcation tasks, and have no obvious extension to applications with non-transitive similarity.
Revised: Third, methods using target hash codes are limited to classiﬁcation tasks, and have no obvious extension to applications with non-transitive similarity.

Original: Finally, various multi-step training methods, including target hash codes, forfeit the beneﬁt of training end-to-end.
Revised: Finally, various multi-step training methods, including target hash codes, forfeit the beneﬁt of training end-to-end.

Original: 1. M ULTI - INDEXING
Revised: 1. M ULTI - INDEXING

Original: Multi-indexing enables search within a Hamming radius r by splitting an n -bit binary hash into m substrings of length n/m Norouzi et al.
Revised: Multi-indexing enables search within a Hamming radius r by splitting an n -bit binary hash into m substrings of length n/m Norouzi et al.

Original: Technically, it is possible to use any m ∈{ 1 , . . .
Revised: Technically, it is possible to use any m ∈

Original: r + 1 } , but in most practical scenarios the best choice is m = r + 1 .
Revised: r + 1 } , but in most practical scenarios the best choice is m = r + 1 .

Original: We consider only this case 1 .
Revised: We consider only this case 1 .

Original: Each of these r + 1 substrings is inserted into its own reverse index, pointing back to the content  (Algorithm 1).
Revised: Each of these r + 1 substrings is inserted into its own reverse index, pointing back to the content and full hash (Algorithm 1).

Original: Insertion runtime is therefore proportional to r + 1 , the number of multi-indices.
Revised: Insertion runtime is therefore proportional to r + 1 , the number of multi-indices.

Original: Lookup is performed by taking the union of all results for each substring, then ﬁltering down to results within the Hamming radius r (Algorithm 2).
Revised: Lookup is performed by taking the union of all results for each substring, then ﬁltering down to results within the Hamming radius r (Algorithm 2).

Original: This enables lookup within a Hamming radius of r by querying each substring in its corresponding index.
Revised: This enables lookup within a Hamming radius of r by querying each substring in its corresponding index.

Original: Any result within r will match on at least one of the r + 1 substrings by pigeonhole principle. 
Revised: Any result within r will match on at least one of the r + 1 substrings by pigeonhole principle. Algorithm

Original: Input: binary hash h and corresponding data D
Revised: Input: binary hash h and corresponding data D

Original: Split h into substrings h 1 , . . .
Revised: Split h into substrings h 1 , . . .

Original: h r + for i = 1 to r + 1 do
Revised: h r + for i = 1 to r + 1 do

Original: Add row with key h i and data  D  to the i th index end for
Revised: Add row with key h i and data ( h, D ) to the i th index end for

Original: Algorithm 
Revised: Algorithm 2 Lookup in a multi-index system

Original: Input Initialize do
Revised: Input: binary hash h

Original:  Add  end 
Revised: Split h into substrings h 1 , . . . h r + Initialize empty set S D for i = 1 to r + 1 do Add exact matches for h i in the i th index to S D end for

Revised: Filter results with Hamming distance greater than r out of S D

Original: Return S D
Revised: Return S D

Original: With a well-distributed hash function, the average runtime of a lookup is proportional to the number of queries times the number of rows returned per query.
Revised: With a well-distributed hash function, the average runtime of a lookup is proportional to the number of queries times the number of rows returned per query.

Original: Norouzi et al.
Revised: Norouzi et al.

Original: treat the time to compare
Revised: treat the time to compare

Original: Hamming distance between codes as constant 2 , giving us a query cost of cost ∼ ( r + 1)
Revised: Hamming distance between codes as constant 2 , giving us a query cost of cost ∼ ( r + 1)

Original: N 2 n/ ( r +1) where N is the total number of n -bit hashes in the database.
Revised: N 2 n/ ( r +1) where N is the total number of n -bit hashes in the database.

Original: Like Norouzi et al.
Revised: Like Norouzi et al.

Original: , we recommend choosing r such that n/ ( r + 1)
Revised: , we recommend choosing r such that n/ ( r + 1)

Original: ≈ log 2 N , providing a runtime of cost ∼ n log 2 N
Revised: ≈ log 2 N , providing a runtime of cost ∼ n log 2 N

Revised: Space cost to store the dataset is Nn ( r + 1) , since each substring must point back to its full hash.

Revised: However, since n is only a very small bit length, this is quite manageable.

Original: We build on this technique in 2.3.
Revised: We build on this technique in 2.3.

Original:  M ETHOD
Revised:  M ETHOD

Original: We propose a method of Hamming distance targets (HDT) that can be used to train any differentiable, black box model to hash.
Revised: We propose a method of Hamming distance targets (HDT) that can be used to train any differentiable, black box model to hash.

Original: We will focus on its application to deep convolutional neural nets trained using stochastic gradient descent.
Revised: We will focus on its application to deep convolutional neural nets trained using stochastic gradient descent.

Original: Our loss function’s foundation is a statistical model relating pairs of embeddings to Hamming distances.
Revised: Our loss function’s foundation is a statistical model relating pairs of embeddings to Hamming distances.

Original: 2. L OSS F UNCTION 2.1. M OTIVATION
Revised: 2. L OSS F UNCTION 2.1. M OTIVATION

Original: Let y ( x ) =
Revised: Let y ( x ) =

Original: ( y 1 ( x ) , . . .
Revised: ( y 1 ( x ) , . . .

Original: y n ( x )) be the model’s embedding for an input x , and let X be the distribution of inputs to consider.
Revised: y n ( x )) be the model’s embedding for an input x , and let X be the distribution of inputs to consider.

Original: We motivate our loss function with the following assumptions:
Revised: We motivate our loss function with the following assumptions:

Original: • If x ∼ X is a random input, then y i ( x ) ∼ N (0 , 1) .
Revised: • If x ∼ X is a random input, then y i ( x ) ∼ N (0 , 1) .

Original: We partially enforce this assumption via batch normalization of y i with mean 0 and variance 1.
Revised: We partially enforce this assumption via batch normalization of y i with mean 0 and variance 1.

Original: • y
Revised: • y

Original: i is independent of other y j .
Revised: i is independent of other y j .

Original: Let z ( x ) =
Revised: Let z ( x ) =

Original: y ( x ) /
Revised: y ( x ) /

Original: || y ( x ) || 2 be the L 2 -normalized output vector.
Revised: || y ( x ) || 2 be the L 2 -normalized output vector.

Original: Since y ( x ) is a vector of n independent random normal variables, z ( x ) is a random variable distributed uniformly on the hypersphere.
Revised: Since y ( x ) is a vector of n independent random normal variables, z ( x ) is a random variable distributed uniformly on the hypersphere.

Original: This L 2 -normalization is the same as SphereNorm Liu et al.
Revised: This L 2 -normalization is the same as SphereNorm Liu et al.

Original: (2017) and similar to Riemannian Batch
Revised: (2017) and similar to Riemannian Batch

Original: Normalization Cho & Lee (2017).
Revised: Normalization Cho & Lee (2017).

Original: Liu et al. posed the question of why this technique works better in conjunction with batch norm than either approach alone, and our work bridges that gap.
Revised: Liu et al. posed the question of why this technique works better in conjunction with batch norm than either approach alone, and our work bridges that gap.

Original: An L 2normalized vector of IID random normal variables forms a uniform distribution on a hypersphere, whereas most other distributions would not.
Revised: An L 2normalized vector of IID random normal variables forms a uniform distribution on a hypersphere, whereas most other distributions would not.

Original: An uneven distribution would limit the regions on the hypersphere where learning can happen and leave room for internal covariate shift toward different, unknown regions of the hypersphere.
Revised: An uneven distribution would limit the regions on the hypersphere where learning can happen and leave room for internal covariate shift toward different, unknown regions of the hypersphere.

Original: To avoid the assumption that Euclidean distance translates to Hamming distance, we further study the distribution of Hamming distance given these L 2 -normalized vectors.
Revised: To avoid the assumption that Euclidean distance translates to Hamming distance, we further study the distribution of Hamming distance given these L 2 -normalized vectors.

Original: We craft a good approximation for the probability that two bits match, given two uniformly random points z i , z j on the hypersphere, conditioned on the angle θ between them.
Revised: We craft a good approximation for the probability that two bits match, given two uniformly random points z i , z j on the hypersphere, conditioned on the angle θ between them.

Original: [Equation]
Revised: [Equation]

Original: between z i and z j .
Revised: between z i and z j .

Original: We know that z i · z j = cos( θ ) , so the arc length of the path on the unit hypersphere between them is arccos( z i · z j ) .
Revised: We know that z i · z j = cos( θ ) , so the arc length of the path on the unit hypersphere between them is arccos( z i · z j ) .

Original: A half loop around the unit hypersphere would cross each of the n axis hyperplanes (i.e. z k = 0 ) once, so a randomly positioned arc of length θ crosses nθ/π axis hyperplanes on average (Figure 1).
Revised: A half loop around the unit hypersphere would cross each of the n axis hyperplanes (i.e. z k = 0 ) once, so a randomly positioned arc of length θ crosses nθ/π axis hyperplanes on average (Figure 1).

Original: Each axis hyperplane crossed corresponds to a bit ﬂipped, so the probability that a random bit differs between these vectors is
Revised: Each axis hyperplane crossed corresponds to a bit ﬂipped, so the probability that a random bit differs between these vectors is

Original: [Equation]
Revised: [Equation]

Original: Given this exact probability, we estimate the distribution of Hamming distance between sgn ( y i ) and sgn ( y j ) by making the approximation that each bit position between the two vectors differs independently from the others with probability P ij .
Revised: Given this exact probability, we estimate the distribution of Hamming distance between sgn ( y i ) and sgn ( y j ) by making the approximation that each bit position between the two vectors differs independently from the others with probability P ij .

Original: Therefore, the probability of Hamming distance being within r is approximately F ( r ; n, P ij ) where F is the binomial CDF.
Revised: Therefore, the probability of Hamming distance being within r is approximately F ( r ; n, P ij ) where F is the binomial CDF.

Original: This approximation proves to be very close for large n (Figure 2.1.1).
Revised: This approximation proves to be very close for large n (Figure 2).

Original: Prior hashing research has made inroads with a similar observation, but applied it in the limited context of choosing vectors to project an embedding onto for binarization Ji et al.
Revised: Prior hashing research has made inroads with a similar observation, but applied it in the limited context of choosing vectors to project an embedding onto for binarization Ji et al.

Revised: Prior quantization research has used the geometry of the hypersphere before, but to choose a projection that minimizes quantization loss Gong et al.

Original: We apply this idea directly in network training. 
Revised: Instead, we apply this idea directly in network training. [Figure]

Original: 2.1. F ORMULATION
Revised: 2.1. F ORMULATION

Original: With batch size b , let Y =
Revised: With batch size b , let Y =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Original: y b ) T be our batch-normalized logit layer for a batch of inputs ( x 1 , . . .
Revised: y b ) T be our batch-normalized logit layer for a batch of inputs ( x 1 , . . .

Original: x b ) and Z = ( z 1 , . . .
Revised: x b ) and Z = ( z 1 , . . .

Original: z b ) T be the b × n L 2 -row-normalized version of Y ; that is, z i=
Revised: z b ) T be the b × n L 2 -row-normalized version of Y ; that is, z i

Original: [Figure]
Revised: =

Original: y i / ||
Revised: y i / ||

Original: y
Revised: y

Original: i ||
Revised: i ||

Original: Let P = arccos ( Z T Z )
Revised: Let P = arccos ( Z T Z )

Original: π .Let
Revised: π .Let

Original: w be the vector of all our model’s learnable weights.
Revised: w be the vector of all our model’s learnable weights.

Original: Let S be a b × b similarity matrix such that S ij = 1 if inputs x i and x j are similar and 0 otherwise.
Revised: Let S be a b × b similarity matrix such that S ij = 1 if inputs x i and x j are similar and 0 otherwise.

Original: Deﬁne ◦ to be the Hammard product, or pointwise multiplication.
Revised: Deﬁne ◦ to be the Hammard product, or pointwise multiplication.

Original: Our loss function is
Revised: Our loss function is

Original: [Equation]
Revised: [Equation]

Original: with
Revised: with

Original: • J 1 =
Revised: • J 1 =

Original: Avg [ S ◦ log F ( r ; n, P )] , the average log likelihood of each similar pair of inputs to be within Hamming distance r .
Revised: Avg [ S ◦ log F ( r ; n, P )] , the average log likelihood of each similar pair of inputs to be within Hamming distance r .

Original: • J 2 =
Revised: • J 2 =

Original: Avg [(1 − S ) ◦ log F ( n − r − 1; n, 1 − P )] , the average log likelihood of each dissimilar pair of inputs to be outside Hamming distance r .
Revised: Avg [(1 − S ) ◦ log F ( n − r − 1; n, 1 − P )] , the average log likelihood of each dissimilar pair of inputs to be outside Hamming distance r .

Original: • J 3 = ||
Revised: • J 3 = ||

Original: w || 22 , a regularization term on the model’s learnable weights to minimize overﬁtting.
Revised: w || 22 , a regularization term on the model’s learnable weights to minimize overﬁtting.

Original: Note that terms J 1 and J 2 , work on all pairwise combinations of images in the batch, providing us with a very accurate estimate of the true gradient.
Revised: Note that terms J 1 and J 2 , work on all pairwise combinations of images in the batch, providing us with a very accurate estimate of the true gradient.

Original: While most machine learning frameworks do not currently have a binomial CDF operation, many (e.g., Tensorﬂow and Torch) support a differentiable operation for a beta distribution’s CDF.
Revised: While most machine learning frameworks do not currently have a binomial CDF operation, many (e.g., Tensorﬂow and Torch) support a differentiable operation for a beta distribution’s CDF.

Original: This can be used instead via the well-known relation between the binomial CDF and the beta CDF I :
Revised: This can be used instead via the well-known relation between the binomial CDF and the beta CDF I :

Original: [Equation]
Revised: [Equation]

Original: For values of p that are too low, this quantity underﬂows ﬂoating point numbers.
Revised: For values of p that are too low, this quantity underﬂows ﬂoating point numbers.

Original: This issue can be addressed by a linear extrapolation of log likelihood for p < p 0 .
Revised: This issue can be addressed by a linear extrapolation of log likelihood for p < p 0 .

Original: An exact formula exists, but a simpler approximation sufﬁces, using the fact that I ( p ; α, β ) ∝ p α for small p :
Revised: An exact formula exists, but a simpler approximation sufﬁces, using the fact that I ( p ; α, β ) ∝ p α for small p :

Original: [Equation]
Revised: [Equation]

Original: 2. T RAINING S CHEME
Revised: 2. T RAINING S CHEME

Original: We construct training batches in a way that ensures every input has another input in the batch it is similar to.
Revised: We construct training batches in a way that ensures every input has another input in the batch it is similar to.

Original: Speciﬁcally, each batch is composed of groups of g inputs, where each group has one randomly selected marker input and g − 1 random inputs similar to the marker.
Revised: Speciﬁcally, each batch is composed of groups of g inputs, where each group has one randomly selected marker input and g − 1 random inputs similar to the marker.

Original: We then choose b/g random groups to form.
Revised: We then choose b/g random groups to form.

Original: During training, similarity between inputs is determined dynamically, such that if two inputs from different groups happen to be similar, they are treated as such.
Revised: During training, similarity between inputs is determined dynamically, such that if two inputs from different groups happen to be similar, they are treated as such.

Original: This method ensures that each loss term is well-deﬁned, since there will be both similar and dissimilar inputs in each batch.
Revised: This method ensures that each loss term is well-deﬁned, since there will be both similar and dissimilar inputs in each batch.

Original: Additionally, it provides a better estimate of the true gradient by balancing the huge class of dissimilar inputs with the small class of similar inputs.
Revised: Additionally, it provides a better estimate of the true gradient by balancing the huge class of dissimilar inputs with the small class of similar inputs.

Original: 2. M ULTI - INDEXING WITH E MBEDDINGS
Revised: 2. M ULTI - INDEXING WITH E MBEDDINGS

Original: For additional recall on ANN tasks, we store our model’s embedding in each row of the multi-index.
Revised: For additional recall on ANN tasks, we store our model’s embedding in each row of the multi-index.

Original: We use this to rank results better, returning the closest l of them to the query embedding.
Revised: We use this to rank results better, returning the closest l of them to the query embedding.

Original: This adds to query cost, since evaluating the Euclidean distance between the query’s embedding scales with the hash size n and obtaining the top l elements is O (log l ) per result.
Revised: This adds to query cost, since evaluating the Euclidean distance between the query’s embedding scales with the hash size n and obtaining the top l elements is O (log l ) per result.

Original: The heightened query cost allows us to compare query cost against quantization methods, which do the same ranking of ﬁnal results by embedding distance.
Revised: The heightened query cost allows us to compare query cost against quantization methods, which do the same ranking of ﬁnal results by embedding distance.

Original: When using embeddings to better rank results in this way, we call our method HDT-E.
Revised: When using embeddings to better rank results in this way, we call our method HDT-E.

Original:  R ESULTS 3. I MAGE N ET
Revised:  R ESULTS 3. I MAGE N ET

Original: We compared HDT against reported numbers for other machine learning approaches to similar image retrieval on ImageNet.
Revised: We compared HDT against reported numbers for other machine learning approaches to similar image retrieval on ImageNet.

Original: We followed the same methodology as Cao et al.
Revised: We followed the same methodology as Cao et al.

Original: , using the same training and test sets drawn from 100 ImageNet classes and starting from a pre-trained [-Resnet V2 
 He-] et al.
Revised: , using the same training and test sets drawn from 100 ImageNet classes and starting from a pre-trained [-Resnet V2 
 He-]Inception V3 Szegedy et al.

Original: ImageNet checkpoint accepting 224 × 224 images.
Revised: ImageNet checkpoint accepting 224 × 224 images.

Original: Fine tuning each model took 5 hours on a single Titan Xp GPU.
Revised: Fine tuning each model took 5 hours on a single Titan Xp GPU.

Original: Following convention, we computed mean average precision (MAP) for the ﬁrst 1000 results by Hamming distance as our evaluation criterion.
Revised: Following convention, we computed mean average precision (MAP) for the ﬁrst 1000 results by Hamming distance as our evaluation criterion.

Original: We also study our model’s precision and recall at different Hamming distances (Figure 3.1).
Revised: We also study our model’s precision and recall at different Hamming distances (Figure 3).

Original: We highlight 5 comparator models: DBR-v3 Lu et al.
Revised: We highlight 5 comparator models: DBR-v3 Lu et al.

Original: (2017), HashNet Cao et al.
Revised: (2017), HashNet Cao et al.

Original: (2017), Deep hashing network for efﬁcient similarity retrieval (DHN)
Revised: (2017), Deep hashing network for efﬁcient similarity retrieval (DHN)

Original: Zhu et al.
Revised: Zhu et al.

Original: (2016), Iterative Quantization (ITQ)
Revised: (2016), Iterative Quantization (ITQ)

Original: Gong & Lazebnik (2013), and LSH Gionis et al.
Revised: Gong & Lazebnik (2013), and LSH Gionis et al.

Original: DBR-v3 learns by ﬁrst choosing a target hash code for each class to maximize Hamming distance between other target hash codes, then minimizing distance between each image’s embedding and target hash code.
Revised: DBR-v3 learns by ﬁrst choosing a target hash code for each class to maximize Hamming distance between other target hash codes, then minimizing distance between each image’s embedding and target hash code.

Original: To the best of our knowledge, it has the highest reported MAP on the ImageNet image retrieval task until this work.
Revised: To the best of our knowledge, it has the highest reported MAP on the ImageNet image retrieval task until this work, partially due to using the Inception V3 architecture whereas previous methods used Alexnet Krizhevsky et al.

Original: HashNet trains a neural network to hash with a supervised cross entropy loss function by gradually sharpening a sigmoid function of its last layer until the outputs are all close to ± 1 .
Revised: HashNet trains a neural network to hash with a supervised cross entropy loss function by gradually sharpening a sigmoid function of its last layer until the outputs are all close to ± 1 .

Original: DHN similarly trains a neural network with supervised cross entropy loss, but with an added binarization loss term to coerce outputs close to ± 1 instead of sharpening a sigmoid.
Revised: DHN similarly trains a neural network with supervised cross entropy loss, but with an added binarization loss term to coerce outputs close to ± 1 instead of sharpening a sigmoid.

Original: Using λ = 2000 and r = 2 , our method achieved 81.2-83.8% MAP for hash bit lengths from 16 to 64 (Table 1), a 4.3-10.5% absolute improvement over the next best method. 
Revised: We trained a 16-bit model with r = 2 , λ = 2000 , a 32-bit model with r = 2 , λ = 3000 , and a 64-bit model with r = 3 , λ = 3500 . Our method achieved 85.1 to 86.1% MAP (Table 1), a 8.2 to 12.0% absolute improvement over the next best method. [Table] [Figure]

Revised: r

Original:  Most interestingly, HDT performed better  on shorter bit lengths.
Revised: [Figure] n [Figure] Most interestingly, HDT performed better worse on 64-bit hashes than it did on 32-bit hashes.

Original: A shorter hash should be strictly worse, since it can be padded with constant bits to a longer hash.
Revised: A shorter hash should be strictly worse, since it can be padded with constant bits to a longer hash.

Original: Our result may reﬂect a capacity for the model to overﬁt slightly with larger bit lengths, an increased difﬁculty to train a larger model, or a need to better tune parameters.
Revised: Our result may reﬂect a capacity for the model to overﬁt slightly with larger bit lengths, an increased difﬁculty to train a larger model, or a need to better tune parameters.

Original: In any case, the clear implication is that16 bits are enough to encode 100 ImageNet classes.
Revised: In any case, the clear implication is that 100 ImageNet classes can be encoded in a small number of bits.

Original: [Table] [Figure]
Revised: Even 16-bit binary hashes offer 2 16 / 100 ≈ 655 possibilities per ImageNet class used, generally enough room for each class to own all 137 hashes within a Hamming radius of 2 around its centroid.

Original: 3. SIFT 1M
Revised: 3. SIFT 1M

Original: We compared HDT against the state-of-the-art embedding-to-results method of Product Quantization on the SIFT 1M dataset, which consists of 10 6 dataset vectors, 10 5 training vectors, and 10query vectors in R 128 .
Revised: We compared HDT against the state-of-the-art embedding-to-results method of Product Quantization on the SIFT 1M dataset, which consists of 10 6 dataset vectors, 10 5 training vectors, and 10query vectors in R 128 .

Original: We trained HDT from scratch using a simple 3-layer Densenet Huang et al.
Revised: We trained HDT from scratch using a simple 3-layer Densenet Huang et al.

Original: (2017) with 256 reluactivated batch-normalized units per layer.
Revised: (2017) with 256 reluactivated batch-normalized units per layer.

Original: During training, we deﬁned input x i to be similar to x j
Revised: During training, we deﬁned input x i to be similar to x j

Original: if x j is among the 10 nearest neighbors to x i .
Revised: if x j is among the 10 nearest neighbors to x i .

Original: Training each model took 75 minutes on a single
Revised: Training each model took 75 minutes on a single

Original: Geforce 1080 GPU.
Revised: Geforce 1080 GPU.

Original: We compared the recall-query cost tradeoff at different values of n , r , and λ (Table 2).
Revised: We compared the recall-query cost tradeoff at different values of n , r , and λ (Table 2).

Original: We used the standard recall metric for this dataset of recall@100, where recall@ k is the proportion of queries whose single nearest neighbor is in the top k results.
Revised: We used the standard recall metric for this dataset of recall@100, where recall@ k is the proportion of queries whose single nearest neighbor is in the top k results.

Original: HDT-E deﬁed even our expectations by providing higher recall than reported numbers for PQ while requiring fewer distance comparisons (Figure 3.2).
Revised: HDT-E deﬁed even our expectations by providing higher recall than reported numbers for PQ while requiring fewer distance comparisons (Figure 4).

Original: This implies that even on embedding-to-result tasks, HDT-E can be implemented to provide better results than PQ with faster query speeds.
Revised: This implies that even on embedding-to-result tasks, HDT-E can be implemented to provide better results than PQ with faster query speeds.

Original: The improvement is particularly great in the high-recall regime.
Revised: The improvement is particularly great in the high-recall regime.

Original: Notably, HDT-E gets 78.1% recall with an average of 12,709 distance comparisons, whereas PQ gets only 74.4% recall with 101, comparisons. 
Revised: Notably, HDT-E gets 78.1% recall with an average of 12,709 distance comparisons, whereas PQ gets only 74.4% recall with 101, comparisons. [Figure]

Original: [Table]
Revised: [Table]

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: Our novel method of Hamming distance targets vastly improved recall and query speed in competitive benchmarks for both data-to-results tasks and embedding-to-results tasks.
Revised: Our novel method of Hamming distance targets vastly improved recall and query speed in competitive benchmarks for both data-to-results tasks and embedding-to-results tasks.

Original: HDT is also general enough to use any differentiable model and similarity criterion, with applications in image, video, audio, and text retrieval.[Figure]
Revised: HDT is also general enough to use any differentiable model and similarity criterion, with applications in image, video, audio, and text retrieval.

Original: We developed a sound statistical model as the foundation of HDT’s loss function.
Revised: We developed a sound statistical model as the foundation of HDT’s loss function.

Original: We also shed light on why L 2 -normalization of layer outputs improves learning in conjunction with batch norm.
Revised: We also shed light on why L 2 -normalization of layer outputs improves learning in conjunction with batch norm.

Original: For future study, we are interested in better understanding the theoretical distribution of Hamming distances between points on a sphere separated by a ﬁxed angle
Revised: For future study, we are interested in better understanding the theoretical distribution of Hamming distances between points on a sphere separated by a ﬁxed angle.