AHash: A Load-Balanced One Permutation Hash
Anonymous authors
Paper under double-blind review
Abstract
Minwise Hashing (MinHash) is a fundamental method to compute set similarities and compact high-dimensional data for eﬃcient learning and searching.
The bottleneck of MinHash is computing k (usually hundreds) MinHash values.
One Permutation Hashing (OPH) only requires one permutation (hash function) to get k MinHash values by dividing elements into k bins.
One drawback of OPH is that the load of the bins (the number of elements in a bin) could be unbalanced, which leads to the existence of empty bins and false similarity computation.
Several strategies for densiﬁcation, that is, ﬁlling empty bins, have been proposed.
However, the densiﬁcation is just a remedial strategy and cannot eliminate the error incurred by the unbalanced load.
Unlike the densiﬁcation to ﬁll the empty bins after they undesirably occur, our design goal is to balance the load so as to reduce the empty bins in advance.
In this paper, we propose a load-balanced hashing, Amortization Hashing (AHash) , which can generate as few empty bins as possible.
Therefore, AHash is more load-balanced and accurate without hurting runtime eﬃciency compared with OPH and densiﬁcation strategies.
Our experiments on real datasets validate the claim.
All source codes and datasets have been released on GitHub anonymously 1 .
Introduction 1. Background
MinHash (Broder et al., 2000; Broder, 1997) is a powerful tool in processing high-dimensional data ( e.g. , texts) that is often viewed as a set.
For a set, MinHash produces k minimum hash values, which are called MinHash values .
MinHash values can support similarity computing (Broder, 1997; Chien & Immorlica, 2005; Henzinger, 2006b; Bayardo et al., 2007), large-scale linear learning (Li & K¨onig, 2010; Li et al., 2011; 2012; Yu et al., 2012), fast near neighbour searching (Shrivastava & Li, 2014a; Li et al., 2010; Indyk & Motwani, 1998; Shrivastava & Li, 2012) and so on.
Due to the importance of MinHash, many recent works strive to improve its performance such as One Permutation Hashing (OPH) (Li et al., 2012), Densiﬁed OPH (DOPH) (Shrivastava & Li, 2014a), and Optimal OPH (OOPH) (Shrivastava, 2017).
1. Prior Art and Their Limitations
MinHash (Broder et al., 2000; Broder, 1997) is the most classical method but has unacceptable computation cost.
Given a set S ∈ U ( U is the universe of all elements), MinHash applies k random permutations (hash functions)
π
i : U → U , on all elements in S (see Figure 1).
For each hash function, it computes | S | hash values but only maintains the minimum one.
Then we get the k MinHash values .
In practice, to achieve high accuracy, users need to compute a large number of MinHash values, i.e. , k needs to be very large.
Unfortunately, computing k hash functions for each element is a computational and resource bottleneck when k is large.
It is showed that some large-scale learning applications need to compute more than 4000 MinHash values, which takes a non-negligible portion of the total computation time (Fernandez et al., 2019; Li, 2015).
One Permutation Hash (OPH) (Li et al., 2012) overcomes the drawback of MinHash, but the produced MinHash values may be less than the demand.
To overcome the drawback of MinHash, OPH reduces the number of hash computations per element from k to 1.
As shown in Figure 1, the key idea of OPH is that all elements are divided into k bins by hashing, and each bin maintains the minimum hash value respectively.
Ideally, each bin will produce a MinHash value.
Unfortunately, if the data is skewed or sparse, some bins could have many elements, but some bins could be empty.
As a result, the number of produced MinHash values could be smaller than k .
Existence of empty bins is fatal for some applications.
To compute similarities, it is known that the existence of empty bins leads to the false similarity estimation (see detailed reasons in Section 2) (Shrivastava, 2017; Shrivastava & Li, 2014a; Fernandez et al., 2019).
To search near neighbours, using the empty bins as indexes leads to the false positives (see detailed reasons in Section 2) (Shrivastava & Li, 2012; 2014a).
Therefore, every empty bin needs a MinHash value, and how to appropriately ﬁll empty bins attracts research interests in recent years (Li et al., 2012; Shrivastava, 2017; Shrivastava & Li, 2014a;b).
Densiﬁcation is helpful for ﬁlling empty bins but does not fairly use the original data.
Densiﬁcation, proposed by Shrivastava & Li (2014a), is to ﬁll the empty bins by reusing the minimum values of the non-empty bins to ﬁx the drawback of OPH.
Many densiﬁcation strategies have been proposed (Li et al., 2012; Shrivastava & Li, 2014a;b; Shrivastava, 2017).
Among all densiﬁcation strategies, OOPH (Shrivastava, 2017) achieves the smallest variance.
It hashes IDs of empty bins to select a non-empty bin to reuse (See Figure 1).
Although it achieves the smallest variance, it is often impossible to choose the optimal value to ﬁll each empty bin.
As shown as Figure 1, OOPH just reuses the MinHash values of non-empty bins.
Because OPH only involves three elements { A, D, C } to compute MinHash values, it places the constrains on OOPH which can only reuse these three elements.
Such a reuse is unfair: the Minhash values could be reused multiple times, but other elements ( { B, E } ) will never be reused.
Unfairness probably incurs error.
1. Our Solution: Amortization Hashing
Instead of ﬁlling empty bins, this paper proposes, Amortization Hashing (AHash), which can produce fewer empty bins, so as to achieve high accuracy.
As shown in Figure 1, to minimize the unfairness of the densiﬁcation, we aim to design a load-balanced hash (AHash) which produces as few empty bins as possible.
After using Amortization Hashing, there could be still empty bins for very sparse sets, and we use OOPH to ﬁll the remaining empty bins.
We have proved (see Theorem 3) that the lower the number of empty bins is, the higher accuracy will be.
This paper is the ﬁrst attempt to design a load-balanced hash for OPH.
Intuitively, if a bin holds too many elements, it is wasted .
In contrast, if a bin holds no element, it is starved .
Therefore, if we can pair up a wasted bin and a starved bin, and amortize the elements between them, we can eliminate the unfairness of using densiﬁcation.
The details of amortization hashing are provided in Section 3.
AHash is a novel solution which is orthogonal with densiﬁcation, and can improve accuracy of all densiﬁcation strategies.
For densiﬁcation strategies, ﬁlling each empty bin could incur small error.
Filling more empty bins will incur larger error.
To minimize the error of ﬁlling empty bins, we are expected to minimize the number of empty bins during dividing elements into bins.
It is worth mentioning that AHash does not incur additional computation cost.
1. Key Contributions
1) We propose a load-balanced hashing, Amortization Hashing, which can improve accuracy while retaining the same runtime eﬃciency.
(Section 3) 2)
We conduct theoretical analysis and extensive experiments, and results show that AHash can achieve higher accuracy.
(Section 4) 3)
We apply AHash for two data mining tasks, linear SVM, and near neighbour search, and results show AHash signiﬁcantly outperforms the state-of-the-art in practical applications.
(Section
Preliminaries 2. Formal Defnitions of MinHash, OPH and OOPH
Formal deﬁnitions of MinHash, OPH and OOPH are as follows:
• MinHash (Broder, 1997; Broder et al., 2000): the i th MinHash value of S is deﬁned as:
[Equation]
where π i ( S ) denotes the i th hashing on S .
• One Permutation Hashing (Li et al., 2012): the i th MinHash value of S is deﬁned as:
[Equation]
where π ( S ) denotes the hashing on S and Ω i denotes i th partition of the rang space of π ( S ).
• One Permutation Hashing with Optimal Densiﬁcation (Shrivastava, 2017): the i th MinHash value of S is deﬁned as:
[Equation]
where
h univ i ) denotes 2-universal hashing (Shrivastava, 2017) and j = h univ ( i ).
2. Use MinHash Values for Computing Similarities
Computing similarities is a key step for many applications like duplicate detection (Broder, 1997; Henzinger, 2006a), semantic similarity estimation (Chien & Immorlica, 2005), frequent pattern mining (Buehrer & Chellapilla, 2008; Chierichetti et al., 2009) and more.
In the applications, the data ( e.g. , texts) can be viewed as 0/1 binary data ( e.g. , the absence/presence of a word), which is equivalent to a set.
Given two sets, S 1 and S 2 , the similarity is usually measured by Jaccard Similarity , which is deﬁned as J ( S 1 , S 2 ) = | S 1 ∩ S 2 | / | S 1 ∪ S 2 | .
For Sand S 2 with the same hashing, the probability of the two minimum hash values being the same is equal to Jaccard Similarity of S 1 and S 2 , which is formally shown as follow:
[Equation]
Such a property is called Locality Sensitive Hash (LSH) Property (Indyk & Motwani, 1998;
Charikar, 2002). Note that, if h ( S 1 ) and h ( S 2 ) are simultaneously empty, they are certain to collide, which violates the LSH Property.
So we must handle empty bins.
Given MinHash values of S 1 and S 2 , the J ( S 1 , S 2 ) can be approximated as:
[Equation]
with 1 { x } being the indicator function that takes value 1 when x is true and otherwise 0.
2. Use MinHash Values for Large-scale Learning
Large-scale linear learning like training SVM (Fan et al., 2008; Hsieh et al., 2008; Joachims, 2006) is faced with extremely high-dimensional data, which emphasizes the application of hashing algorithms.
Given a dataset { x i , y i } li =1 , x i ∈ R n , y i ∈ {− 1 , +1 } , L2-SVM solves the following unconstrained optimization problem:
[Equation]
where C > 0 is a penalty parameter.
MinHash can be used to compact the feature vectors to reduce the feature dimensionality if the dataset is binary ( i.e. , each feature vector consists of only 0s and 1s)
(Li et al., 2011).
Actually, each feature vector can be viewed as a set: if the i -th element of the vector is 1, then the i -th element is in the set.
Thus we can use MinHash values to represent a feature vector.
We use one-hot encoding (Coates & Ng, 2011; Buckman et al., 2018) for each hash value and concatenate these k values to get the new feature vector.
To further reduce the dimension, Li et al. (2010) proposes that we can use only the lowest b ( e.g. ,
b =8) bits of each hash value (usually 64 bits).
Thus the new feature vector is only 2 b × k -bit long, regardless of the dimensionality of the original data.
2. Use MinHash Values for Fast Near Neighbour Search
Fast near neighbour search is important in many areas like databases (Broder, 1997; Friedman et al., 1975) and machine learning (Shrivastava & Li, 2012), especially applications with high-dimensional data.
Given a query set S , near neighbour search is to return other sets whose similarities with S are more than a threshold.
Shrivastava & Li (2012) proposed MinHash values can be used in near neighbour search in sub-linear time complexity ( i.e. , without scanning the whole dataset).
Speciﬁcally, a signature for a set is generated by concatenating k MinHash values like this:
[Equation]
In this way, those similar sets are more likely to have the same signature (It is not true if empty bins occur).
So we can build hash tables by using the signatures as indexes and the sets as the values of the hash table entries.
Moreover, to reduce the number of hash table entries, we only concatenate the lowest b bits of k hashed values to generate signatures (Li et al., 2010).
To improve the recall of the query results, we can calculate L diﬀerent signatures for each set and build L hash tables, and return the union of entries from these L hash tables as the query result.
For diﬀerent hash tables, we should use independent hash functions to compute MinHash values.
Parameters b , L , and k can be used to control the threshold of near neighbor search.
Using L hash tables, the processing cost of MinHash for one query set is O ( nkL ) ( n is the size of the query set).
Shrivastava & Li (2014a) proposes that OPH can generate k × L MinHash values only by one hashing with k × L bins, so the processing cost of OPH is O ( n + kL ).
( O ( n ) is for hashing on n elements and O ( kL ) is for ﬁlling empty bins).
Algorithm 3. Amortization Hashing
Input: k bins B [ . ], a hash function h ( . ), a set S 1 Ω ← output range of h ( . ) 2 Initialize B[.].EvenMin=+ ∞ B[.].OddMin=+ ∞ 3 for each element e in S do V ← h ( e ) i ← h ( e ) /
(Ω /k )
if V%2=1 then B[i].OddMin = min( V , B[i].OddMin) if V%2=0 then
B[i].EvenMin = min( V , B[i].EvenMin) Input: k bins B [ . ]
after Insertion 1 for i = 0 ; i < k ; i += 2 do if B[i] is empty and B[i+1] is non-empty then B[i].OddMin = B[i+1].EvenMin if B[i] is non-empty and B[i+1] is empty then B[i+1].EvenMin = B[i].OddMin 6 If empty bins still exist:
Densiﬁcation
To generate k MinHash values, Amortization Hashing (AHash) has two key steps: Insertion and Amortization .
Insertion : As shown in Algorithm 1 and Figure 2, AHash divides all elements into k bins by hashing and each bin maintains one even min and one odd min .
Even min refers to the minimum one among all even hash values.
Odd min refers to the minimum one among all odd hash values.
Amortization : As shown in Algorithm 2 and Figure 2, AHash pairs up k bins as follows:
i th and ( i + 1) th bins are a pair ( i = 0 , 2 , .., (cid:98)
k
− 22 (cid:99) ).
For a pair, the i th bin is called even bin and the ( i + 1) th bin is called odd bin .
For the pairs with only one empty bin, there are two cases:
• If the even bin B e is empty, we reassign B e .EvenMin with the even min of the odd bin.
• If the odd bin B o is empty, we reassign B o .OddMin with the odd min of the even bin.
For the pairs with two simultaneously non-empty or empty bins, we keep them unchanged.
After the amortization, we collect k MinHash values from k bins and there are two cases (See Figure 2):
• For each even bin with both even and odd mins, we delete the odd min. • For each odd bin with both even and odd mins, we delete the even min.
If there are still empty bins, we conduct densifying like OOPH, which rarely happens because
AHash signiﬁcantly reduces the empty bins.
Formally, for even bins,
[Equation]
where π e ( S ) ( π o ( S )) denotes the even (odd) hash values of S and Ω i denotes i -th partition of the range space of π ( S ).
The formula for odd bins is similar.
3. Time and Memory Overhead
For time overhead, AHash keeps comparable runtime eﬃciency with OPH, the fastest variant of MinHash.
Although each bin maintains one more hash values than OPH, the additional runtime cost is negligible because of the memory access locality of the insertion (two values in a bin).
Note that AHash has much less empty bins which require densiﬁcation compared with OOPH, which can save the time cost.
For memory overhead, AHash also provides k MinHash values, which is the same as MinHash and OOPH.
During the Insertion , the cost of storing two hash values in a bin is acceptable.
For the common setting (hundreds of bins and 64-bit MinHash values), the vector of bins with two values per bin can be totally ﬁtted into L1 cache (usually larger than 32 KB), the fastest memory.
[-=
-]
[-Theorem 1 proves ( E ( ˆ J AHash )) 
-]
[-E (( 1 k (cid:80) k − 1 j =
-]
[-( I emp,j =
-]
Experiments and Applications 4. Setup
We use three publicly available datasets 2 :
a) RCV1: The dataset is a collection for text categorization.
It has 20 , 242 sets and the size of the set is on average 73.
The dimensionality (range space) of elements is 47 , 236.
b) NEWS20:
The dataset is a collection of newsgroup documents.
It has 19 , 996 sets and the size of the set is on average 402.
The dimensionality (range space) of elements is 1 , 355 , 191.
c) URL:
The dataset is a collection for identifying suspicious URLs.
It has 100 , 000 sets and the size of the set is on average 115.
The dimensionality (range space) of elements is 3 , 231 , 961.
We use two metrics to measure the performance:
a) Mean Square Error (MSE) is deﬁned as E [( ˆ J − J ) 2 ].
We use MSE to measure the accuracy of computing the Jaccard Similarity .
b) F1-score is deﬁned as 2 P · R ( P + R ) , where P is the precision rate and R is the recall rate.
We use F 1 - score to evaluate the performance of fast near neighbour search.
We implemented all algorithms in C++ which are publicly available 3 on GitHub.
All experiments are conducted on laptop with 2.9 GHZ Intel Core i7 CPU.
4. Accuracy and Speed
[Figure] [Table]
[Figure]
[Table]
[Table]
[Figure]
Results are averaged over 2000 repetitions.
To measure the accuracy of AHash, we generate 6 pairs of sets from the data RCV1, with varying similarities.
Because it is diﬃcult to use original sets to form the pairs with high similarity, we unite some original sets as a new set to form the pairs.
Detailed statistics is shown in Table 1.
Figure 3 presents the results about the accuracy of computing similarities with varying k (the number of Minhash values).
Regardless of the similarity level, AHash is more accurate than OOPH and MinHash ( e.g. , at k = 512, the MSE of AHash is lower than OOPH by 13 .
For Figure 3, the absolute values of MSE are too small, so it’s a little hard to see the diﬀerence in the values.
But the improvements are signiﬁcant (up to 29.021%), e.g., the MSE of AHash is 29% lower than OOPH with k = 512 in Figure 3(b).Note that the gain of AHash is more signiﬁcant when the number of bins is appropriate (not extremely small or large).
The reason is that if k is extremely small, all bins are non-empty.
Therefore, AHash has no need for amortization.
To the other extreme, if k is extremely large, only a small portion of bins are non-empty and the amortization plays a limited role.
Fortunately, with a practical and commonly used conﬁguration, AHash presents signiﬁcant gains.
Table 2 shows that the speed of computing k MinHash values of AHash is comparable with OOPH, which is two orders of magnitude faster than MinHash.
AHash does not hurt the runtime eﬃciency of OOPH.
AHash and OOPH both use much less runtime to more accurately compute the similarity.
4. Linear Learning
[Figure]
We use LIBLINEAR (Fan et al., 2008) to train a L 2 - nomarlized SVM to measure the eﬀectiveness of AHash in reducing the dimensionality of the training data.
We experimented with varing penalty parameter C , which follows other works (Li et al., 2011; Yu et al., 2012) [Figure]
In this way [Figure] it is more easy to reproduce our experiments.
Figure 4 shows that AHash can achieve 99% test accuracies and outperforms OOPH.
Compared with training with the original high-dimensional data, the training based on the new feature vectors built by AHash is usually two orders of magnitude faster.
Figure 4 shows AHash can further reduce the error, e [Figure] g., the error of AHash on average is 46.81% lower than OOPH in Figure 4(b).
AHash can achieve the satisﬁed accuracy, close to 100% [Figure]
4. Near Neighbour Search
[Figure]
We apply AHash with the optimization of b -bit hashing (Li et al., 2010) for fast near neighbour search.
Given a query set, threshold of the similarity is 0.5, that is, near neighbour search should return sets whose similarity with the query set is more than 0.5.
It is important to balance the precision and recall to measure the eﬀectiveness on near neighbour search.
Therefore, we use F 1 -score as the metric.
Figure 5 shows that the F 1 -score of AHash is signiﬁcantly higher than MinHash and OOPH on all three datasets.
Note that the trend of F 1 -score with varying L is not monotonous.
With L increasing, the number of reported elements and the number of reported near neighbours both increases, but their growth trends are of randomness.
In spite of this, AHash can outperform OOPH and MinHash under most of parameter settings.
Conclusion
We propose Amortization Hashing which can improve the accuracy of One Permutation
Hashing and densiﬁcation strategies without loss in runtime eﬃciency.
AHash outperforms the state-of-the-art OOPH in similarity estimating, large-scale learning and fast near neighbour searching. A Theoretical Analysis
Theorem 1 Unbiased Estimator
[Equation]
where ˆ J AHash denotes the estimator given by AHash.
Theorem 1 is to show the estimator of AHash is following the LSH property and unbiased.
Proof A.1 For convenience, we deﬁne some Boolean values:
[Equation]
1 For both i th bins of S 1 and S 2 , no elements falls in them
[Equation]
otherwise
[Equation]
[Equation]
1 Before Amortization, the values of the i th bins of S 1 and S 2 are the same 0 otherwise
[Equation]
[Equation]
Obviously, for simultaneously non-empty bins, AHash is following the LSH property:
[Equation]
Next, we prove that the Amortization for simultaneously empty bins is unbiased:
[Equation]
i
[Equation]
and I Aemp
[Equation]
If I Aemp,i = 1 , i.e., there is still the pair of bins which are simultaneously empty after Amortization, we use unbiased Optimal Densiﬁcation (Shrivastava, 2017) to ﬁll such bins.
Theorem 2 Reducing Empty Bins
[Equation]
where N emp = (cid:80) ki
{+=
+}
I iemp is the number of simultaneously empty bins.
Proof A.2 (Li et al., 2012) proves the expectation of the number of simultaneously empty bins of OPH is:
[Equation]
where f is | S 1 ∪ S 2 | and D is the dimensionality of the universal hashing function used.
The process of OPH can be viewed as randomly throwing f balls into k bins.
For AHash, because k bins are paired up, the process of AHash can be viewed as randomly throwing f balls into k 2 bins.
Obviously, N emp of AHash is smaller than OPH:
[Equation]
Theorem 3 Improving Variance
[Equation]
{+Proof A.
+}
We deﬁne two Boolean values with 1 { x } being the indicator function that takes 1 when x is true and otherwise 0.
[Equation]
The estimator given by AHash and the variance of AHash can be rewritten as follows using these two values.
[Equation]
{+Theorem 1 proves ( E ( ˆ J AHash )) 
+}
=
J 2 .
Deﬁne two notations:
[Equation]
OOPH uses conditional expectation to simplify variance analysis.
To conveniently compare with OOPH, we follow its method where E ( .
| m ) means E ( .
| k − N AHashemp
= m ) .
We compute f ( m ) =
{+E (( 1 k (cid:80) k − 1 j =
+}
[ M Ej + M Nj ]) 2 | m )
By expanding,
[Equation]
[Equation]
Because M Ai and M Ei are Boolean values,
[Equation]
To analyze another three terms, we provide four probability equations for bins before the amortiza- tion.
[Equation]
I
[Equation]
j =
[Equation]
and the mins from two j-th bins have one and only one position
[Equation]
and the
[Equation]
from two j-th bins have
[Equation]
positions
[Equation]
Deﬁne one notation:
[Equation]
The values of following three terms is proved by classifying diﬀerent combinations of bins and applying probability equations and the property of indicator functions.
E
[Equation]
i
[Equation]
[Equation]
Hence
[Equation]
we get that
[Equation]
For OOPH, the variance is
[Equation]
For OOPH, the conditional expectation is
[Equation]
Because AHash reduces N emp , AHash using Optimal Densiﬁcation (Shrivastava, 2017) can achieve lower variance compared with OOPH.