Scaling Hierarchical Coreference with Homomorphic Compression
Abstract
Locality sensitive hashing schemes such as simhash provide compact representations of multisets from which similarity can be estimated.
However, in certain applications, we need to estimate the similarity of dynamically changing sets.
In this case, we need the representation to be a homomorphism so that the hash of unions and diﬀerences of sets can be computed directly from the hashes of operands.
We propose two representations that have this property for cosine similarity (an extension of simhash and angle-preserving random projections), and make substantial progress on a third representation for Jaccard similarity (an extension of minhash).
We employ these hashes to compress the suﬃcient statistics of a conditional random ﬁeld (CRF) coreference model and study how this compression aﬀects our ability to compute similarities as entities are split and merged during inference.
We also provide novel statistical analysis of simhash to help justify it as an estimator inside a CRF, showing that the bias and variance reduce quickly with the number of hash bits.
On a problem of author coreference, we ﬁnd that our simhash scheme allows scaling the hierarchical coreference algorithm by an order of magnitude without degrading its statistical performance or the model’s coreference accuracy, as long as we employ at least 128 orbit hashes.
Angle-preserving random projections further improve the coreference quality, potentially allowing even fewer dimensions to be used.
1. Introduction
Probabilistic models in machine learning, such as conditional random ﬁelds (CRFs), are widely successful at modeling many problems at the heart of knowledge base construction, including those in natural language processing, information extraction and data integration.
However, when dealing with natural language data, the underlying feature representations are often sparse, high-dimensional and dynamic ( i e., they change during inference).
In this paper we consider the task of coreference resolution, in which the goal is to partition a set of mentions into the entities to which they refer.
We might represent each mention with a feature vector in which each dimension corresponds to a word or n -gram.
Since only a small subset of the vocabulary is observed per mention, most elements of the vector are zero.
Given the model and these representations, inference entails making decisions about whether two entities should be coreferent.
To make such decisions, the model estimates a probability that involves computing the similarities between the aggregate feature representations of the two entities’ mentions.
Since all the vectors are both sparse and high-dimensional, these similarity operations are computationally expensive because the sparse data structures supplant the dense arrays that would otherwise support fast look-ups.
Moreover, as the inference algorithm makes decisions about whether or not two entities are coreferent, we may have to split or merge the entities and thus we must update the feature vector to reﬂect these changes.
Maintaining such sparse-vector representations in the inner-loop of probabilistic inference is expensive, especially as the entities grow in size.
In order to cope with the computational problems associated with sparse, high dimensional dynamic feature representations, we propose using homomorphic compression , in which the compressed representations of intermediate inference results can be computed directly from their operands, allowing inference to run directly on the compressed representations of the data even as they change.
In this paper, we consider several such schemes to scale hierarchical coreference.
First, we propose a novel homomorphic cosine-preserving hashing scheme based on simhash [Charikar, 2002] that also supports addition and subtraction to more eﬃciently represent the data and the evolving intermediate results of probabilistic inference.
Second, because various linear norm-preserving random projections also preserve angles [Magen, 2002], we can directly compute cosine similarity on projected data – linearity of the projections means that they too can be updated dynamically.
The resulting angle estimates are superior to the homomorphic simhash representation, at the cost of reduced eﬃciency for certain operations.
Third, we develop a homomorphic version of minhash [Broder, 1997b] to support
Jaccard similarity.
Our current algorithm is biased, but the bias appears small in practice for the situations we have considered.
Although the minhash based set representations are not currently employed in hierarchical coreference, they might be useful in other models or applications that involve binary features over changing sets [Culotta et al., 2007b].
We provide error analysis for all three schemes, collating and extending known results, and in the case of simhash, we provide novel statistical analysis for its use as a direct estimator for cos( θ ) that shows the bias and variance decrease rapidly with the number of bits, helping to justify its use in a CRF for a task like coreference.
On a hierarchical model for coreference resolution, the proposed simhash scheme improves the speed of probabilistic inference by an order of magnitude while having little eﬀect on model quality.
Moreover, we ﬁnd that the underlying random projection representation can provide even better cosine estimates than simhash, at the cost of not being able to use certain fast bitwise-operations to compute similarities.
Finally, we brieﬂy evaluate homomorphic minhash and show that even though there are possible pathological cases, as long as we employ enough hash functions, the estimates are reasonably close to the true Jaccard, albeit, biased.
2. Coreference Resolution: Background and Challenges at Scale Coreference resolution
Coreference resolution is the problem of determining whether diﬀerent mentions refer to the same underlying entity [Getoor and Machanavajjhala, 2012].
For example, in the sentence “In a few years [McDaniels] would replace [Belichick] as the forty-two year old [quarterback], [Tom Brady] retires.”
coreference must correctly determine that “quarterback” refers to Tom Brady and not Belichick or McDaniels.
This type of coreference is sometimes referred to as noun-phrase or within document coreference and often relies upon linguistic and discourse motivated features [Raghunathan et al., 2010].
Coreference resolution arises in many other situations; for example, when merging two or more databases together it is desirable to remove duplicates that result from the merge, a problem sometimes termed record linkage or deduplication [Newcombe et al., 1959].
Coreference is also foundational to knowledge base construction which requires that we combine information about entities of interest from multiple sources that might mention them in diﬀerent contexts.
For example, if we were to build a knowledge base of all scientists in the world — similar to Google scholar — we would need to perform the task of author coreference to determine who authored what papers [Han et al., 2004, Culotta et al., 2007a].
Are the following two mentions of “J Smith” the same author?
Although generally this is a diﬃcult problem, it can be solved with machine learning since features of the mentions such as the words in the title (both have “Boson” in common), the topic of the title (both are about a similar subﬁeld of physics), the journal (both are physics journals) and the co-authors (there appears to be a co-author in common) provide some evidence about whether or not the two “J Smith’s” might be the same person.
In order to solve the problem, it is thus common to extract such contextual features about each mention, such as in the above example, features from the title, co-author list, venue, year and author-name and employ them in a probabilistic model.
These features are typically the raw words, character-ngrams and normalized variants thereof, sometimes with positive real-valued weights to indicate the importance (e.g., via TFIDF) of each feature.
Then, given such features, a coreference model measures the similarities between mentions via functions such as cosine-similarity.
In contrast to within document coreference discussed earlier, this type of coreference resolution problem is better suited for similarity based models, such as the ones we will use in the following.
Moreover, since coreference decisions are not restricted by document-boundaries, the entities can grow unbounded in size, making compact representations of their growing feature sets especially important.
Typically, the model is a discriminative conditional random ﬁeld (CRF) that measure the probability of an assignment of mentions to entities conditioned on the observed features [McCallum and Wellner, 2003].
The model factorizes into potentials that score local coreference decisions.
Local search procedures such as greedy-agglomerative clustering or
Markov-chain Monte Carlo (MCMC) ﬁnd the most likely assignment of mentions to entities [McCallum and Wellner, 2003, Culotta et al., 2007a, Wick et al., 2012].
In pairwise models, potential functions measure the compatibility of two mentions being in the same cluster.
An issue with such models is that the possible pairwise comparisons scales quadratically with the number of mentions.
An alternative class of models that avoids this quadratic blow-up are entity-based , in which entities are treated as ﬁrst-class variables with their own set of inferred features, and potentials measure compatibility between mentions and entities.
However, entity-based models come with their own scalability challenges.
To illustrate the problem (and our solution), we focus on an entity-based model called hierarchical coreference , which recently won a challenge to disambiguate inventor names for the USPTO, due to its accuracy and scalability [Uni, 2016, Monath and McCallum, 2016].
Hierarchical Coreference
In the hierarchical coreference model, mentions are organized into latent tree structures [Wick et al., 2012].
There is one tree per entity with mentions at the leaves and intermediate nodes as “subentities” that organize subsets of the entity’s mentions.
Rather than modeling interactions between mention-pairs, the potential functions measure compatibility between child and parent nodes in the tree.
The score of a given assignment of mentions into latent trees is the product of all model potentials which includes these child-parent compatibility scores as well as some additional priors on tree-shape and entities.
These compatibility scores are parametrized cosine functions.
Each mention is endowed with multiple feature variables that each capture a subset of the total features.
For example, in author coreference, one feature variable might capture features of the author’s name and another might capture features of the title and venue of the paper.
Colloquially, we refer to these feature variables as “bags” since they inherit the usual bags-of-words assumption.
We distinguish between diﬀerent “bag-types” which each capture diﬀerent types of features (e.g., the author name bag, title words bag, co-author bag, etc).
The other nodes in the tree also contain feature variables (bags), but the values of these variables are determined by the current assignment of children to that parent node.
In particular, a parent’s bag is the sum of all its children’s bags.
In order to maintain this invariant, the bag representations must be updated to reﬂect the current state of inference, and hence for eﬃciency reasons, representations must be homomorphic with respect to operations that will be performed on the bags.
Interpreting bags as vectors, the cosine distance between them is used to calculate their compatibility.
The primary potential functions measure the compatibility between each child’s bag and its parent’s bag.
There is one potential for each bag-type.
For example, to measure the compatibility between a node z i and z j , let y ij be the binary variable that isif and only if z j is the parent of z i , and let b (0)
i and b (0)
j be a bag for z
i and z j respectively, then the potential ψ (0) for “bag 0” scores a coreference decision as:
[Equation]
where w is a real-valued weight and t is a real-valued translation parameter for potential ψ (0) .
The potentials for each bag-type have parameters w, t that we can ﬁt to data.
Because only a small handful of features are ever observed for a given mention, typical implementations of hierarchical coreference employ sparse-vector representations to represent bags ( e.g., the implementation found in FACTORIE [McCallum et al., 2008, 2009]).
However, a key disadvantage of sparse vector representations is that they must store the indices and weights of the non-zero elements, which means that the data-structures must dynamically change in size as MCMC splits and merges entities.
As the sizes of the entities grow, these operations become increasingly expensive.
Similar issues arise in other entity-based models where features of entities are aggregated from those of mentions.
Thus, while entity-based models avoid the quadratic comparison issue of pairwise models, a straight-forward sparse representation of their feature vectors is no longer eﬃcient.
Is there an alternative representation of feature vectors which (a) allows fast evaluation of cosine-similarity, and (b) can be eﬃciently dynamically updated?
As we describe in the next section, the simhash hashing function [Charikar, 2002] provides a representation with property (a).
However, the standard simhash cannot be updated as vectors are modiﬁed.
We therefore develop a variant which we call homomorphic simhash, which has both properties (a) and (b).
We also identify two other schemes that support these properties.
3. Homomorphic Representations for Measuring Similarity
We now discuss three diﬀerent homomorphic representations that support addition and subtraction in the compressed space, while preserving similarity estimates.
We propose homomorphic simhash and a related random projection for cosine similarity of high-dimensional vectors and multisets; and homomorphic minhash for Jacard similarity of sets.
3.1 Homomorphic simhash and its Statistical Properties
Background: simhash
A locality-sensitive hash function for a distance metric d on a set of objects S is a function H such that given x, y ∈ S , we can estimate d ( x, y ) from the hashed representations H ( x ) and H ( y ).
Simhash [Charikar, 2002] is a locality sensitive hash function for cosine similarity.
To understand simhash, it is ﬁrst helpful to consider the following randomized process:
Imagine that we have two vectors a and b on the unit hypersphere in the Euclidean space R d with angle θ between them, and we want to produce an estimate of cos( θ ).
Select a random d -dimensional vector u by sampling each of its coordinates independently from N (0 , 1).
Let the random variable X have value 1 if a and b are on diﬀerent sides of the hyperplane orthogonal to u , and 0 otherwise.
Then X is a Bernoulli random variable with expectation:
[Equation]
Let X 1 , ..., X n be the result of independently repeating this process several times, and set
X n = 1 n n (cid:80) i =
X i .
Then E [ X n ] = 1 − θπ , and hence E [ π (1 − X n )]
= θ , so that we can usecos( π (1 − X n )) as an estimator of cos( θ ).
The idea behind simhash is to come up with a hash representation which lets us reproduce this randomized estimator: to construct a function H which produces n -bit hashes, we ﬁrst randomly sample n vectors u 1 , . .
u n as above.
Then, given a vector a , the hash H ( a ) is the length n bit sequence in which the i th bit is 1 if the sign of a · u i is positive and otherwise.
Now, from two hashed representations H ( a ) and H ( b ), if the i th bit in the two hashes disagree, this is equivalent to X i in the randomized process above being equal to 1.
Thus, by counting the number of positions where the hashes are distinct and dividing by n , we get X n , thereby producing an estimate of cos( θ ).
Rather than constructing the hash function H by sampling the u 1 , . . .
, u n vectors from the d -dimensional unit sphere uniformly, a common optimization is to instead sample them from {− 1 , 1 } d .
This has two advantages.
First, it is faster to compute the dot product since no ﬂoating point multiplication is involved.
Second, rather than having to explicitly sample and store each u i as a vector, we can replace each u i by a 1-bit feature hash function
h i : the “value” of the vector represented by h i is 1 at coordinate j if h i ( j )
=
and is − 1 if h i ( j ) = 0.
We write a · h i for the dot product of a with the vector corresponding to h i .
By restricting only to test vectors with coordinates of the form 1 and − 1, the corresponding expected value of π (1 − X n ) is no longer exactly θ (see [Leskovec et al., 2014, Section 3.7.3] for an example), but for high-dimensional spaces, this approximation is known to be eﬀective in practice [Henzinger, 2006].
Homomorphic simhash
If we want to use simhash as a representation of feature vectors for entities in coreference resolution, then we need to be able to update the simhash representation as entities are merged and split.
In particular, if we join nodes with feature vectors a and b , then the vector of their new parent will be a + b .
However, if we only store
H ( a ) and H ( b ), rather than the vectors a and b themselves, we cannot compute H ( a + b ): the i th bit of H ( a ) and H ( b ) just records the sign of a · h i and b · h i , and if these are diﬀerent, we do not know what the sign of ( a + b ) · h i should be.
A similar problem occurs when we split a child with vector b from a parent with vector a , since the updated parent’s hash should be H ( a − b ).
Our solution is instead to store the actual dot product of a · h i in the hash of a , rather than just the sign.
That is, H ( a ) is now an array of length n instead of an n -bit sequence.
And since (
[Equation]
+
[Equation]
h i and
[Equation]
we can compute H ( a + b ) by adding component-wise the arrays for H ( a ) and H ( b ), and similarly for H ( a − b ).
Finally, we can still eﬃciently compute the cosine distance between two vectors a and b by examining the signs of the entries of H ( a ) and H ( b ).
We call this representation homomorphic because H is a homomorphism with respect to the additive group structure on vectors.
Of course, storing each dot product instead of just the signs increases the size of our hashes.
However, they are still small compared to the feature vectors and, more importantly, their sizes are ﬁxed.
We can also store both the dot product and the signs as a bit vector, making sure to update the sign vector after each operation based on the dot product.
By storing the sign vector separately we can quickly count the signs in common between two vectors using bitwise operations.
Statistical Properties
Recall that since E [ π (1 − X n )]
= θ , we can derive a plausible estimate of cos( θ ) from X n .
In particular, let g ( x ) = cos( π (1 − x )) and consider the estimator
C n = g ( X n ).
We now describe some statistical properties of C n .
Our emphasis here is somewhat diﬀerent from the standard analyses found in related work.
The reason is that
LSHs like simhash are most commonly used for duplicate detection [Henzinger, 2006] and for approximate nearest neighbor search [Leskovec et al., 2014], which is quite diﬀerent from our use case.
In those settings, one wants to show that if two items x and y are very similar, then the distance estimated from h ( x ) and h ( y ) will very likely be quite small, and conversely if x and y are very diﬀerent, then their estimated distances will be large.
In such cases, the linear approximation to cosine X n is suﬃcient.
However, since we want to use the cosine distance estimates C n as part of the potential function in our CRF, we are interested in additional statistical properties of the estimator.
Lemma 3.1.
C n is consistent.
In particular, C n a.s. −−→ cos( θ ) .
Proof.
By the strong law of large numbers, we have that X n a.s. −−→ 1 − θπ .
Since g is continuous, by the continuous mapping theorem [Van Der Vaart, 1998],
[Equation]
Lemma 3.2.
E [ C n ] = cos( θ ) + E n , where | E n | ≤ π 2 8 n
Proof Sketch.
Set µ = E [ X n ] = 1 − θπ .
The ﬁrst degree Taylor series for g ( x ) about µ is:
[Equation]
where R is the remainder term.
We have then that:
[Equation]
Thus it suﬃces to bound | E
[ R ( X n )] | , which can be done using Lagrange’s remainder formula (see appendix).
Lemma 3.3.
V [ C n ] = π 2 sin 2 ( θ ) n
[Equation]
Proof Sketch.
For intuition, note that the Taylor series above for g shows us that g ( x ) ≈ cos( θ )
+ π sin( θ )( x − µ ).
So, recalling that V [ C n ] =
V [ g ( X n )]
=
E [ g ( X n ) 2 ] − E [ g ( X n )] 2 , and plugging in the approximation we get:
[Equation]
To obtain the actual error bound, we carry out the same process but without dropping R ( x ) from the Taylor approximation for g , and then once again use Lagrange’s remainder formula to bound the remainder (see appendix).
Finally, since C n is equal to a Lipschitz continuous function of n independent Bernoulli random variables, we can use the the method of bounded diﬀerences [Dubhashi and Panconesi, 2009, Corollary 5.2], to derive the following Chernoﬀ-like tail bound:
Lemma 3.4.
P
[Equation]
3.2 Angle Preseving Random Projections for Cosine
The statistics underlying simhash, whether the hyperplanes are drawn from Gaussians or the d -dimensional hypercube (i.e., Rademacher distributions), are actually random projections for which the Johnson-Lidenstrauss lemma applies [Johnson and Lindenstrauss, 1982, Indyk and Motwani, 1998, Achlioptas, 2003].
The lemma states that any set of m points in ddimensional Euclidean space can be embedded into n -dimensional Euclidean space, where n is logarithmic in m and independent of d, n
= O ( (cid:15) − 2 log m ), such that all pairwise distances are maintained within an arbitrarily small factor 1 ± (cid:15) , where 0 < (cid:15) < 1.
Since the projections are linear, they are homomorphic with respect to addition and subtraction.
Moreover, previous work shows that the norm-preserving property of a linear random projection A implies an angle-preserving property [Magen, 2002]; that is, for vectors u, v , let θ =
(cid:94) ( v, u ) and ˆ θ = (cid:94) ( Av, Au ).
The result is the following:
[Equation]
where (cid:15) ≤ 13 , (cid:15) (cid:48)
= 8 π √ (cid:15) and n ≥ 60 (cid:15) − 2 log m .
Therefore, we are justiﬁed in using the statistics underlying simhash to directly estimate the cosine similarity; viz.,
cos θ ≈ cos ˆ θ .
More precisely, using a Taylor expansion, |
[Equation]
Although we lose the bit-representation that supports the fast bit-operations that makes simhash so eﬃcient in applications such as webpage deduplication that employ the linear estimator, we potentially gain an improvement in the quality of the cosine estimate.
Certainly, the estimate will be smoother since simhash essentially quantizes the angles into a ﬁnite number of possibilities equal to the number of bits.
Since the same representation supports both types of estimates, we could even alternate between the estimation strategies as dictated by the particular situation: if a large number of similarity comparisons are performed for each entity then simhash makes more sense, otherwise a direct computation of cosine makes more sense.
Regardless, as we will later see, both schemes provide suﬃciently fast and accurate cosine estimates for coreference resolution.
3.3 Homomorphic Minhash for Jaccard Similarity
Minhash [Broder, 1997a, Broder et al., 2000] is a locality-sensitive hash function for Jaccard similarity, deﬁned for binary vectors (encoding sets).
The Jaccard similarity between two sets S 1 , S 2 ∈ Ω is J = | S 1 ∩ S 2 | | S 1 ∪ S 2 | .
Minhash applies a random permutation (which in practice is accomplished using a random hash function, such as seeded 64-bit murmur hash)
π : Ω → Ω, on a given set S ⊂ Ω
and then extracts the minimum value
h π ( S ) =
min( π ( S )).
The probability that the minhash function for a random permutation of a set computes the same value for two sets is equal to the Jaccard similarity of the two sets [Rajaraman and Ullman, 2010].
In practice multiple ( n ) hash functions are employed to reduce the variance, resulting in a vector v S
= (cid:104)
h π 0 ( S ) , · · · , h π n − 1 ( S ) (cid:105) of n minimum values, one per hash function.
There are three methods we need to design for homomorphic minhash: union , difference and score (to compute the Jaccard estimate).
However, unlike simhash for cosine similarity, the operations underlying the statistics for minhash are non-linear due to the elementwise minimum operation that produces the underlying vector of hash values.
Moreover, the set semantics on which the minhash score method relies are problematic because the union and difference methods need to maintain the invariance that a parent set is equal to the union of a collection of child sets: when we perform a diﬀerence operation between a parent set and a child set we cannot simply remove all of the child’s elements from the parent since a sibling might also (redundantly) contribute some of those elements to the parent.
We sketch a solution to these problems and include details in Appendix E.
However, we note that our solution is not complete because there are several corner cases we do not yet handle.
Nevertheless, we ﬁnd that the representation works well in practice.
First, we handle the problem of set-semantics by augmenting the n -dimensional minhash representation v S of each set S with an n -dimensional vector of counts c S , in which each dimension corresponds to a minimum hash value (essentially embracing multiset semantics, but in the hash space).
The count indicates the number of child sets that contribute the associated hash value.
For the union of two sets S = S 1 ∪ S 2 , we either keep the count associated with the smaller hash value if they are diﬀerent (that is, since we set v S = v S (cid:63) we set c Si = c S (cid:63) i where S (cid:63) =
argmin S j ∈{ S 1 ,S 2 } ( v S j i )), or sum the counts if they are the same (that is, c
Si = c S 1 i + c S 2 i ).
For diﬀerence we employ the same strategy except we subtract the counts instead of add them.
The counts are appropriately ignored when estimating the
Jaccard since we want the estimate to reﬂect sets rather than multisets.
Second, we must also address the fact that the minimum is a non-linear operator.
The problem arises when the count associated with a hash value becomes zero.
Then, how do we recompute what the new minimum value should be?
Recomputing the minimum from scratch is a nonstarter because it would require keeping around the original sparse vector representations that we are trying to supplant.
Rewriting the diﬀerence in terms of unions is also computationally expensive since it would require traversing the entire tree to check what the new minimum value should be.
Instead, noting that minhash typically has hundreds of hash functions (e.g., n = 256) for each set, we propose to ignore the hash values associated with zero counts, and employ the remaining hashes to compute the Jaccard.
This strategy has consequences for both bias and variance.
First, since fewer hashes are employed, the variance increases, and if left unchecked may culiminate in a worst case in which all counts are zero and Jaccard can no longer be estimated.
However, we can periodically refresh the counts by traversing the trees and hopefully we do not have to do such refresh operations too often in practice.
Second, since the hashes associated with zero counts are correlated, the estimate is no longer unbiased.
Therefore, as described in Appendix E, we modify the
Jaccard estimate to make better use of the zero-counts rather than simply ignore them, and this eliminates the bias in some cases.
However, we do not have a solution that works in every case.
Finally, there is also the question of how to perform union and diﬀerence for cases in which the count is zero.
For now, we ignore the zero counts during these operations, but are currently exploring strategies for incorporating them to further reduce bias and variance.
3.4 Additional Compression Schemes
Hierarchical coref involves other features, such as entropy and complexity-based penalties on the bag of words representations of context and topics associated with the entities.
Although not a focus of this current study, we note that some of these representations depend on the ratios of p -norms that can be estimated with Johnson-Lindenstrauss style representations.
Moreover, there exist hashing schemes to enable fast estimation of entropy.
We save the homomorphic versions of these hashes for future work.
4. Experiments
In this section we empirically study the two cosine-preserving homomorphic compression schemes, simhash and its related random projection, on a conditional random ﬁeld (CRF) model of author coreference resolution, the problem introduced in Section 2.
First we study simhash and we hypothesize that the this representation will substantially improve the speed of inference for the reasons outlined earlier, but it is less clear how the simhash representation will aﬀect the quality of the model.
On the one hand, our theoretical analysis shows that the variance and bias reduce rapidly with the number of bits, but on the other hand, the sheer number of vector comparisons that take place during inference is substantial, making it likely for errors to occur anyway.
With this in mind, we study how simhash aﬀects the model quality in our ﬁrst set of experiments.
In our second set of experiments, we study the extent to which it improves inference speed.
In a third set of experiments, we compare simhash with the random projection method.
Finally, we present initial results for homomorphic minhash to empirically assess how it compares to exact Jaccard, which is important given the increased bias and variance of our method.
Data
We employ the REXA author coreference dataset, 2 which comprises 1404 author mentions of 289 authors with ambiguous ﬁrst-initial last-name combinations: D. Allen , A.
Blum , S. Jones , H Robinson , S. Young , L. Lee , J. McGuire , A. Moore .
We split the data such that training set contains mentions of the ﬁrst ﬁve ambiguous names (950 mentions) while the testing set comprises the remaining three ambiguous names (454 mentions).
The dataset is highly ambiguous since there are multiple entities with each name.
In addition, we also employ the DBLP dataset which contains over one million paper citations from which we extract about ﬁve million unlabeled author mentions [Ley, 2002].
Model
We investigate homomorphic simhash in the context of the hierarchical coreference model presented in Section 2.
We employ two types of feature variables, a “name bag” that represents the features of the author’s name and a “context bag” that represents the remaining features in the citation from which the author mention is extracted (co-authors, title, venue, topics, keywords).
For more details about the features please see Appendix B.
We employ the implementation of hierarchical coreference available in the FACTORIE toolkit, using FACTORIE’s implementation of the variables, the model and the inference algorithm
[McCallum et al., 2008].
We additionally implement the simhash variables and potential functions inside this framework.
We employ FACTORIE’s default inference algorithm for hierarchical coreference which is essentially a greedy variant of multi-try
Metropolis-Hastings in which the proposals make modiﬁcations to the sub-trees (e.g., move a subtree from one entity to another, or merge two trees under a common root node).
More details are in previous work, and the implementation is publicly available in FACTORIE
[McCallum et al., 2009, Wick et al., 2012].
We estimate the parameters with hyper-parameter search on the training-set (Appendix B).
[Figure]
l
[Figure]
( w i t h
s i m h c
[Figure]
i n
[Figure]
Experiment 1: Simhash Estimation Quality
Here we study the quality of models with simhash representations by comparing them to models with exact representations.
First, we compare how these models evaluate and score the intermediate results of MCMC inference.
We run MCMC for 100,000 steps to optimize simhash-based models on the REXA test set (with either 256, 128, 64 and 32 bit hashes).
The chains begin with the singleton conﬁguration (all mentions in their own entity tree of size one), and at each step proposes changes to the current state which the model decides to either accept or reject.
This process gradually produces larger and larger trees.
For each proposed state change (sample), we record the log model ratio of both the simhash model and the exact model.
We present every 10th sample in a scatter plot in Figure 1.
The closer the points are to the identity reference line y
= x , the more accurate the simhash model is for those points.
Varying the number of bits has a pronounced eﬀect on the model’s ability to judge MCMC states.
For each step, we also record if the simhash model and exact model agree upon whether to accept the stochastic proposal (blue points) or not (red points).
The agreement rates are 97.8, 97.0, 93.6, 88.6 percent (respectively 256, 128, 64, 32 bits).
We also plot the decision boundaries for this agreement as dotted lines.
The upper-left and lower-right quadrants contain all the points for which the two models disagree, while the other two quadrants contain points for which they agree.
In particular, the upper-right quadrants contain the points that both the simhash model and exact model believes should be accepted (true positives), while the lower-right quadrants contain the points that both models think should be rejected (true negatives).
Most points lie in this quadrant since the chance of proposing a fruitful move is relatively low.
Visually, there appears to be a qualitative gap betweenbits and 128 bits on this data, leading to a recommendation of using at least 128 bits.
We also compare the models in terms of their ﬁnal coreference performance (according to B-cubed F1 [Bagga and Baldwin, 1998]).
The exact model achieves an F1 of 78.6, while the simhash variants achieve F1 scores of 77.6, 75.6, 62.8, 55.6 for 256, 128, 64, 32 bits respectively.
For more detailed results, with precision, recall and other types of F1, see
Table 1 in the appendix.
Overall, the accuracy of the 128 and 256-bit models are reasonable with 256 being competitive with the performance of the exact model.
When using fewer bits, again, the performance decreases precipitously.
Experiment 2: Simhash Speed
In this experiment we study the extent to which the compressed simhash representation improves inference speed.
As described before, we tune the models on the REXA training set.
Then, to test the models on a larger dataset, we supplement the 454 labeled REXA test mentions with ﬁve million unlabeled mentions from DBLP.
We run each model on this combined dataset, initializing to the singleton conﬁguration and then running one billion samples.
We record coreference quality on the labeled subset every 10,000 samples and plot how it changes over time in Figures 2(a),2(b).
Although Experiment 1 shows that the simhash models are slightly worse in terms of their ﬁnal F1 accuracy on the REXA test set, we see a striking computational advantage for simhash.
For each F1 value, we compute how much faster the simhash model achieves that value than the exact model and plot this speed-improvement as a function of F1-level in
Figures 2(c),2(d).
As we can see, the speed improvement is more than a base-ten order of magnitude for most accuracy levels, and the speed diﬀerence increases with accuracy.
This
[Figure]
is because the exact representation slows down over time as the number of features in the representation grows during inference.
Indeed if we look at the sampling rate over time for each model, we ﬁnd that the simhash models run at about 20,000-25,000 samples per second the entire time, while the model with the exact representation starts at a rate of 3000-4000 samples per second, but then drops to under 1000 samples per second as the size of the entities get larger.
This raises the question: can we improve the speed of the exact model by reducing the number of features?
We address this question in Appendix C.2, but summarize here brieﬂy: it is possible to improve the speed and reduce the gap, but simhash is still faster and there is a trade-oﬀ with coreference quality.
In addition, whether feature ablation meaningfully improves performance depends on particular aspects of the data set, whereas the simhash representation can be applied generically.
Experiment 3: JL Random Projections
In this experiment, we compare simhash with an approach that directly computes cosine on the statistics that underly the bit representation.
As argued previously in Section 3.2, this approach is justiﬁed because the statistics are random projections that are homomorphic and cosine-preserving.
Intuitively, we expect this approach to work even better because simhash further compresses these statistics into single bits, whereas in the random projection approach, we compute cosine directly on the real-valued statistics.
However, our current theoretical analysis does not allow us to compare one to another; therefore, we turn to the empirical studies in this section.
We perform an experiment analogous to Experiment 1, except we employ the exact model to make decisions about what MCMC proposals to accept.
This allows us to compare both approximation schemes on the same set of samples.
For each accepted sample, we again ask the question how the approximate methods, simhash and JL, would have judged them.
We ﬁnd that JL does indeed perform better than simhash for the same number of bits.
In particuar, the Spearman’s rho for JL increases from 93.2 to 97.2 over simhash when employing 256 bits (dimensions).
For all but one case (128 bits), we ﬁnd a similar improvement.
Moreover, the coreference accuracy also increases in each case; for example, from 77.6 B3 F1 to 78.3.
More detailed results are in Table 3 in Appendix D, along with the associated scatter plots (Figure 6).
5. Experiment 4: MinHash
We also investigate our homomorphic MinHash representation for Jaccard similarity.
For lack of space, and because Jaccard is not employed by the hierarchical coreference, we relegate most of the evaluation to Appendix E.2.
To brieﬂy summarize, we ﬁnd thatand 256 hash functions are suﬃcient for reasonable estimates and that the representation rarely needs to be refreshed to ensure that all the counts are above zero, enabling the computational eﬃciency we desire.
However, there is room for improvement because only 16% of the hash functions on average have a non-zero entry after 100,000 samples.
Thus, after 100,000 samples, a 256 hash version will begin to behave like a 40 hash version.
6. Related Work
Homomorphic representations have many possible uses in machine learning; for example, homomorphic encryption allows models to run directly on encrypted data for cases in which privacy is a concern [Graepel et al., 2012, Dowlin et al., 206].
Instead, our method is similar to homomorphic compression [McGregor, 2013] which allows our model to run directly on compressed data for cases in which computational eﬃciency is a concern.
Our approach is based on simhash, a locality-sensitive hash function.
Locality sensitive hashes such as simhash and minhash are useful in large scale streaming settings; for example, to detect duplicate webpages for web-crawlers or in nearest neighbor search [Broder, 1997b, Manku et al., 2007, Leskovec et al., 2014].
They are sometimes employed in search and machinelearning applications including coreference for “blocking” the data in order to reduce the search space [Getoor and Machanavajjhala, 2012].
Note that this application of locality sensitive hash functions for coreference is complementary to ours, and does not require it to be homomorphic.
Other hashing and sketching algorithms are also employed as a strategy to compress the suﬃcient statistics in Bayesian models so that they can scale better as the number of parameters increase with the dataset.
For example, feature hashing, count-min sketches and approximate counters have been employed in scaling latent Dirichlet categorical models such as LDA by compressing, for example, the counts that assign latent variables to mixture components [Zaheer et al., 2016, Tassarotti et al., 2018].
Our paper focuses on three homomorphic compression schemes including one for minhash.
Our solution of furnishing the minhash values with counts resembles a strategy that similarly employs counts in a k minimum values (KMV) sketch for estimating the number of distinct values in a set [Beyer et al., 2007, 2009].
A key diﬀerence is that that work develops an unbiased estimator for the number of distinct values that explicitly involves the counts as part of the estimate itself, whereas we develop a biased estimator that directly estimates
Jaccard similarity and that employs the counts to bound our knowledge about possible minimum hash values when they vanish during diﬀerence operations.
Our schemes are related to random projections, which are commonly used to improve computational eﬃciency in machine learning.
Examples include feature-hashing [Weinberger et al., 2009], fast matrix decomposition [Halko et al., 2009] and fast kernel computations
[Rahimi and Recht, 2007].
However, while some of these random projections happen to be homomorphic, this property is often not exploited.
The reason is that they are typically used to compress static objects that remain ﬁxed, such as the Gram matrix for kernel methods.
In contrast, our setting requires compressing dynamic sets that change during inference.
Word embeddings [Mikolov et al., 2013] also provide low-dimensional dense representations that are useful for capturing context, but in practice, these embeddings are too smooth to be useful as the sole representation for disambiguating the names of peoples, places or organizations, as is necessary in coreference (e.g., the names “Brady” and “Belichick” would be highly similar according to a word-to-vec style embedding, even though they are unlikely to be coreferent).
Deep learning might allow for more suitable representations to be learnt and its application to coreference is promising.
However, the current research focus is on within-document noun-phrase coreference and entity linking, while emphasizing improvements in accuracy rather than inference speed and scalability [Globerson et al., 2016,
Wiseman et al., 2016, Lee et al., 2017, Raiman and Raiman, 2018].
Combining deep learning and conditional random ﬁelds for hierarchical coreference remains an open problem.
Finally, in addition to the work mentioned throughout the paper, there is an abundance of related work on coreference resolution including noun-phrase coreference, cross-document coreference, record linking, entity linking and author coreference.
While not possible to cover the breadth of the space here, we refer the reader to a few useful surveys and tutorials on the subject [Ng, 2010, Getoor and Machanavajjhala, 2012].
More recently, as mentioned in the foregoing paragraph, deep learning approaches are beginning to show promise.
There has also been promising recent work on scalable hierarchical clustering known as PERCH [Kobren et al., 2017].
Since PERCH employs Euclidean distance rather than cosine similarity, it currently falls outside the purview of our study.
However, the Johnson-Lindenstrauss applies to Euclidean distance making random projections a possibility for PERCH.
7. Conclusion
In this paper we presented several homomorphic compression schemes for representing the sparse, high dimensional features in a graphical model for coreference resolution, even as these features change during inference.
Our primary concern was cosine similarity for which we investigated simhash and angle-preserving random projections.
We also proposed a homomorphic version of minhash for Jaccard similarity. In the particular case of simhash, we presented a non-linear variant of the original estimator and analyzed its statistical properties — including variance, bias and consistency — to help justify its use in a probabilistic model. We found that both simhash and angle-preserving random projections were suﬃciently accurate, given enough dimensions, to represent features for coreference, and that the random projections produce slightly better estimates.
Moreover, these representations were an order of magnitude faster than conventional sparse data structures, laying the foundation for greater scalability.
Appendix A. Properties of SimHash
Let a and b be vectors on the unit hypersphere in the Euclidean space R d with angle θ between them, and let u be a random d -dimensional vector whose coordinates are sampled independently from N (0 , 1).
Note that the probability that the two vectors end up on a diﬀerent side of the hyperplane is
[Equation]
Let the random variable X have value 1 if a and b are on diﬀerent sides of u , and 0 otherwise.
Then:
[Equation]
If we let the family of random variables X 1 , ..., X n be the result of repeating this process several times with independently randomly chosen hyperplanes, then we have:
[Equation]
Set X n = 1 n (cid:80) i =
1 X i .
Then we have that E [ π (1 − X n )]
= θ , suggesting that we use cos( π (1 − X n )) as an estimator for cos( θ ).
Let g ( x ) = cos( π (1 − x )) and C n = g ( X n ) be this estimator.
We now explore some properties of C n .
Lemma A.1.
C n is consistent.
In particular, C n a.s. −−→ cos( θ ) .
Proof.
By the strong law of large numbers, we have that X n a.s. −−→ 1 − θπ .
Since g is continuous, by the continuous mapping theorem [Van Der Vaart, 1998],
[Equation]
Lemma A.2.
E [ C n ] = cos( θ ) + E n , where | E n | ≤ π 2 8 n
Proof.
Set µ = E [ X n ] = 1 − θπ .
The ﬁrst degree Taylor series for g ( x ) about µ is:
[Equation]
where R is the remainder term.
Applying Lagrange’s remainder formula, we have that for each x
, there exists some h ( x ) lying between µ and x such that:
[Equation]
We have then that:
[Equation]
Thus it suﬃces to bound | E
[ R ( X n )] | .
We have:
[Equation]
Lemma A.3.
V [ C n ] = π 2 sin
[Equation]
Proof.
Before delving into the details, let us sketch the idea.
The Taylor approximation for g shows us that g ( x ) ≈ cos( θ )
+ π sin( θ )( x − µ ).
So, recalling that V [ C n ] =
V [ g ( X n )] =
E [ g ( X n ) 2 ] − E [ g ( X n )] 2 , and plugging in the approximation we get:
[Equation]
[Equation] [Equation]
Now, to get the actual error term, we carry out the same process without dropping the remainder term R ( x ) from the Taylor expansion.
We have:
[Equation]
It suﬃces to bound the last three terms.
From the previous result, we know that the ﬁrst of these three terms is the square of something which is O (1 /n ).
For the second,
[Equation]
Let p = θ/π .
Then
[Equation]
so that
[Equation]
For the ﬁnal term, we have:
[Equation]
which is O ( n − 3 / 2 )
Lemma A.4.
P ( | C n − cos( θ ) | > δ + π 2 8 n ) ≤ 2 e − 2 nδ 2 /π
Proof.
Deﬁne f n : [0 , 1] n → R by f n ( x 1 , . . . , x n ) = cos( π (1 − 1 n (cid:80) ni
=1 x i )).
Then f n ( X 1 , . . . , X n ) =
C n .
We now show that for all i , and x 1 , . . .
, x n ∈ [0 , 1] n and x
[Equation]
Fix i and x 1 , . . .
, x i − 1 , x i +1 , . . .
, x n , and consider the function
[Equation]
Then, by the mean value theorem, for all x i , x (cid:48) i ∈ [0 , 1], there exists some x ∗ ∈ [0 , 1] such that:
[Equation]
Hence | h ( x i ) − h
( x (cid:48) i ) | ≤ πn | x i − x (cid:48)
i | .
Since the function f n is Lipschitz continuous in each coordinate with parameter πn , by the method of bounded diﬀerences [Dubhashi and Panconesi, 2009, Corollary 5.2], we have:
[Equation]
Moreover, from above we have that | E [ C n ] − cos( θ ) | < π 2 8 n .
Hence,
[Equation]
The above bound is uniform over θ , but stronger bounds for very small or large values of θ can be obtained as follows:
Set p = 1 − θ/π and apply standard Chernoﬀ bounds to the binomial variable X n .
Then observe that if X n deviates from its mean by no more than δ with high probability, then g ( X n ) is within | sup p − δ ≤ t ≤ p + δ g (cid:48)
( t ) | δ around cos( θ ) with at least the same high probability.
Appendix B. Model and parameter estimation Features
Here we provide more details about the features.
First recall that we employ two types of feature variables, a “name bag” that represents the features of the authors name and a “context bag” that represents the remaining features in the citation from which the author mention is extracted.
We populate the “name” bag of each mention with the full name, the ﬁrst-initial last name, and character tri-grams of the author’s name string as it appears in the mention.
We populate the “context” features of each mention with the title and venue, the co-authors with whom the paper is authored, author-provided keywords for the paper.
For the title we employ white-space tokenization, character 3-grams and 4-grams, for venue we employ white-space tokenization as well as the entire string and for co-authors we employ the ﬁrst-initial last-name of all co-authors.
Finally for topics, we take the top 3 topics for each citation as determined by a 200-topic LDA model trained on all of DBLP and REXA combined.
We employ the training data to experiment with several feature combinations, including using all 200-topics for each citation, but found that this combination of features works best.
For the exact sparse-vector representations, we employ the default implementation in
FACTORIE, which employs hashmaps to store the non-zero features and their corresponding weights.
For the simhash representation we employ a java implementation of murmurhash to hash each feature to construct the initial dense bit vectors that underly the homomorphic simhash representation of the mentions.
These vectors are then added and subtracted during inference to produce the underlying vectors of the entities and subentities on which the actual bits are computed.
Parameter estimation
Following previous work, we manually tune the model with exact feature representations on the training data with the help of hyper-parameter search.
We use identical parameters for the simhash model, except we lower the translation value for the context bag to compensate for the extra variance that can increase the chance of false-positives.
More details follow below.
[Table]
The model comprises only a small handful of parameters: a weight and translation parameter for each bag, and a prior on the number of entities and subentities.
Since the prior is somewhat redundant to the translation parameters, we ﬁx the prior to and do not it.
We also ﬁx the parameter for the number of subentities to -0.5 to penalize deeper
Then, we tune the remaining parameters manually with the assistance of hyper-parameter search.
We explore weight values in the set { 0 . 0 , 1 . 0 , 2 . 0 , 4 . 0 } and translation values in the set {− 0 .
75 , − 0 .
5 , − 0 .
For the model that employs exact feature representations, we found a weight of 4.0 and translation of -0.5 works best on the training data for the model that employs exact feature representations.
Since the variance of the Simhash estimator increases with the angles, false-positives become a potential issue.
Therefore, to compensate for this extra variance, we set the translation for the context bag of the simhash model more aggresively to − 0 .
6 via binary search with the training data and 128-bit model (the other simhash models were not further tuned and employ the same parameters as the 128-bit model).
Appendix C. Detailed coreference results C.1 Evaluation on REXA
Table 1 contains a more detailed set of results on the REXA author coreference data.
We evaluate with both B-cubed (B3) and MUC precision, recall and F1.
The results in this table employ all the features of the author mention.
For feature ablations, see Table 2 in the sequel.
C.2 Feature ablations
As we speculate in Section 4, we might also be able to improve the speed of the exact model by including fewer features since this would reduce the cost of dynamically updating the sparse-vector representations.
To this end, we perform some feature ablations by trying various subsets of the six feature-types: name, topic, co-authors, venue, keywords, and title (with and without n-grams), re-tuning the model on the training set and then reporting the accuracy on the test-set.
Note that for the exact model, changing the number of features
[Figure]
[Table]
aﬀects both the quality of the model and its speed, but for the simhash model, it only aﬀects the quality since the representation is ﬁxed in dimension.
Therefore, in the spirit of the tradeoﬀ, we include all features for simhash and instead vary the number of bits.
For the feature ablation study, we record the samples per second and the B-cubed F on the REXA test set and plot these points in Figure 5 with speed (samples per second) on the x-axis and F1 on the y-axis.
Points in the upper-left correspond to models that are accurate, but slow, while points in the lower-right correspond to models that are fast, but inaccurate.
Points in the upper-right correspond to models that are both fast and accurate.
As we can see, varying the number of features for the exact model indeed improves the performance, but it often comes at the expense of accuracy.
The simhash models on the other hand (indicated by red triangles) are able to achieve both speed and accuracy, given enough bits.
This scatter plot is intended to understand the trade-oﬀ space and we omit the labels on the points to reduce clutter.
Instead, we provide these details in a separate table (Table 2).
From these ablations we can see that the best speed-accuracy trade-oﬀ for the exact model employs all the features except for the title n-grams.
Indeed, the title n-grams are the main culprit for slowing down the exact model since they explode the number of features.
Thus, we repeat the experiment for Figure 2 (Experiment 2 in Section 4), with this reduced set of features and manage to improve the running time of the exact model to the point at which the simhash model is only 2-3 times faster.
In these experiments, the exact model is much faster than when it had to cope with the entire set of features, and achieves around 7000 samples per second before eventually falling to 5000 as the entities get larger.
While reducing the features is a reasonable strategy for improving the inference speed without resorting to simhash, it also has several drawbacks.
First, reducing the number of features can have a negative aﬀect on the accuracy, as we see in Figure 5.
When run
[Figure]
exclusively on the labeled data, the ﬁnal F1 drops to 75.9, which is lower than the 256-bit simhash model (77.6 F1), while also being twice as slow.
In addition, this strategy of reducing the feature space will not work as well in other coreference domains.
For example, in author coreference, the true size of author entities is limited by the number of papers the author publishes.
It is unlikely that many authors publish more than 1000 papers, and the average is likely to be much smaller.
Thus, we can reasonably manage the size of the entity and sub-entity bags simply by reducing the number of features in each mention.
In contrast, consider the problem of cross-document coreference on newswire data.
The entity “United
States” might be mentioned more than a million times in vastly diﬀerent contexts.
Thus, even if we limited the feature-space in the beginning, it is likely that these large entities will still accumulate a large number of features anyway.
Models based on simhash will not succumb to this problem.
Finally, the simhash representation is appealing because it allows a practitioner to focus on achieving high accuracy with the full ﬂexibility and freedom when engineering features, without having to worry about the running-time performance (an experiment like the type we did in Experiment 1 would help the practitioner set the number of bits without the need for labeled data).
C.3 Note on Memory Use
Our implementation employs 32-bit ﬂoats to represent the homomorphic simhash statistics, which results in a memory use of k ∗ 32, or between 1kb-8kb per node depending on the number of dimensions.
This could be substantially reduced using integer representations
[Figure]
with fewer bits or approximate counters.
Although this would still require more memory than non-homomorphic simhash, in the context of hierarchical coreference, the memory use is still small, and often better than the original representation.
Appendix D. Johnson-Lindenstrauss Results
Here we include the details results associated with Experiment 3 in Section 4 in the main paper.
In Figure 6, we compare simhash to the random projection approach, which we abbreviate as JL (for Johnson-Lindenstrauss, since it falls under the purvue of these lemmas).
The experiment is analogous to Experiment 1, except that when we run MCMC inference to obtain the data points, we employ the exact model to evaluate the MCMC moves and make acceptance decisions in order to allow for both approximation schemes to be compared on the same set of samples.
For accepted sample, we can again ask the question how the approximate methods, simhash and JL, would have judged the MCMC sample.
We display the results in a scatter plot with exact model judgements on the x-axis and the approximate method on the y-axis.
In the ﬁgure, there are eight subplots.
The left column contains the simhash comparison and the right column contains the JL comparison.
The number of bits increases with each row (32, 54, 128, 256).
In Table 3 we report a numerical comparison to corroborate the scatter plots.
In particular, for a ﬁxed number of dimensions, we compare the two methods in terms of the coreference F1 (B-cubed and MUC), their correlation (Spearman’s rho) with the exact model, and their accuracy with respect to exact model’s decisions in accepting and rejecting
[Table]
MCMC proposals.
We ﬁnd that a model based on JL cosine estimates are better correlated with the exact model, and this translates into small improvements in coreference F1.
Appendix E. MinHash E.1 Homomorphic MinHash
Background: MinHash Minhash [Broder, 1997a, Broder et al., 2000] is a locality-sensitive hash function for Jaccard similarity, deﬁned for binary vectors.
Note that binary vectors can also represent sets such that, the non-zero entries correspond to elements in the set.
For example, a binary vector in R d is equivalent to a set Ω = { 0 , 1 , 2 , . . .
, d − 1 } .
The Jaccard similarity is a measure of resemblance between two sets S 1 , S 2 ∈ Ω, deﬁned as J = | S 1 ∩ S 2 | | S 1 ∪ S 2 | .
Minhash applies a random permutation (which can be accomplished using a random hash function e.g., seeded 64-bit murmur hash)
π : Ω → Ω, on a given set S ⊂ Ω
and then extracts the minimum value
h π ( S ) =
min( π ( S )).
The probability that the minhash function for a random permutation of a set computes the same value for two sets is equal to the Jaccard similarity of the two sets ([Rajaraman and Ullman, 2010] illustrates an intuitive derivation).
Formally,
[Equation]
The Jaccard estimate is then given by ˆ J = 1 n (cid:80) ni
=1 1 (cid:2)
h π i ( S 1 ) =
h π i ( S 2 ) (cid:3) .
The variance of this estimate is Var( ˆ J ) =
1 n J (1 − J ).
Homomorphic MinHash: Unlike simhash for cosine similarity, the operations underlying the statistics for minhash are non-linear due to the element-wise minimum operation that produces the underlying vector of hash values.
Moreover, the set semantics on which minhash relies are problematic because we need to maintain the invariance that a parent set is equal to the union of a collection of child sets: when we perform a diﬀerence operation between a parent set and a child set we cannot simply remove all of the child’s elements from the parent since a sibling might also (redundantly) contribute some of those elements to the parent.
[Figure]
l s c o r e ( w i t
h s i m h
c o s i n
[Figure] [Figure]
( w i t h
i m c
[Figure]
[Figure] [Figure]
(e)
[Figure]
o d e l s c o r e ( w i t
h
s i m h
c o s i n
[Figure]
g
[Figure] [Figure]
bit
[Figure]
To address this situation, we introduce additional non-negative integer variables as counts that indicate the number of child sets that contribute the associated hash value.
We augment the n -dimensional minhash representation v S of each set S with an n -dimensional vector of counts c S , in which each dimension corresponds to a minimum hash value.
This representation is similar to multiset semantics, but is in the hash space instead.
We observe that given the minhash values of two sets, it is relatively straightforward to compute the minhash value of the union which is simply the minimum of the minhash values of the two nodes being merged,
i.e., h π i ( S 1 ∪ S 2 ) = min( h π i ( S 1 ) , h π i ( S 2 )).
To obtain the corresponding counts, we either keep the count associated with the smaller hash value if they are diﬀerent (that is, we set c S 1 ∪ S 2 i
[Equation]
where S (cid:63) =
argmin S j ∈{ S 1 ,S 2 } ( h π
i
S j )), or sum the counts if they are the same (that is, c S 1 ∪ S 2 i =
c S 1 i + c S 2 i ).
In case of set diﬀerence however, the computation is not trivial (and as we mentioned earlier, the operation is not the standard set diﬀerence either).
We consider the following possible cases to illustrate our approach.
(a) h
π
i
S 1 )
<
h π i ( S 2 ) implies that in the original feature space
there is an element e
such
that e ∈ S 1 and e / ∈ S 2 .
In this case, h π i ( S 1 \ S 2 ) =
h π i ( S 1 ) and the corresponding count is equal to c S 1 i .
(b)
h π i ( S 1 ) >
h π i ( S 2 ) implies that in the original feature space, there is an element e such that e ∈ S 2 and e / ∈ S 1 .
Again in this case, h π i ( S 1 \ S 2 ) =
h π i ( S 1 ) and the corresponding count is equal to c S 1 i .
Note that since we assume that a node being subtracted was previously added to current node at some point, we can eliminate this case.
(c)
h π i ( S 1 ) =
h π i ( S 2 )
i
e., the minhash values are equal.
In this case, if c S 1 i > c S 2 i , h π i ( S 1 \ S 2 )
=
h π i ( S 1 ) and the count is c S
i − c S 2 i .
We do not consider the case c S 1 i < c S 2 i because of the same assumption as above,
i
e., a node being subtracted was previously added to current node.
However, when c S 1 i =
c S 2 i , removing S 2 from S 1 leads to loss of information that can not be retrieved without referring back to the original feature space.
Recomputing the minimum from scratch is a nonstarter because it would require keeping around the original sparse vector representations that we are trying to supplant.
Rewriting the diﬀerence in terms of unions is also computationally expensive since it would require traversing the entire tree to check what the new minimum value should be.
Instead, noting that minhash typically has hundreds of hash functions (e.g., n = 256) for each set, we propose to ignore the hash values associated with zero counts, and employ the remaining hashes to compute the Jaccard.
This strategy of ignoring hash values that have zero counts has consequences for both bias and variance.
First, since fewer hashes are employed, the variance increases, and if left unchecked may culminate in a worst case in which all counts are zero and Jaccard can no longer be estimated.
However, we can periodically refresh the counts by traversing the trees and hopefully we do not have to do such refresh operations too often in practice.
Second, since the hashes associated with zero counts are correlated, the estimate is no longer unbiased.
For example, consider the case when we merge the node S 1 = { mustard } with the node S 2 = { hot-dog, mustard } .
Now, if we remove S 2 from S 1 ∪ S 2 , the hash values corresponding to the element “mustard” become zero.
Consequently, the diﬀerentiating information between ( S 1 ∪ S 2 ) \ S 2 and S 2 is lost resulting in a biased Jaccard estimate, i.e., ˆ J (( S 1 ∪ S 2 ) \ S 2 , S 2 )
= 1.
To address this problem, we modify the Jaccard estimate to make better use of the zero-counts rather than simply ignore them, and this eliminates the bias in some cases.
However, we do not have a solution that works in every case.
Finally, there is also the question of how to perform union and diﬀerence for cases in which the count is zero.
For now, we ignore the zero counts during these operations, but are currently exploring strategies for incorporating them to further reduce bias and variance.
Our approach is summarized in Algorithm 1 for the union and diﬀerence operations as well as the Jaccard estimate computation (the function score ).
Algorithm function Union ( S 1 , S 2 )
h π i ( S 1 ∪ S 2 ) =
min( h π i ( S 1 ) ,
h π i ( S 2 ))
if c S 1 i = c S 2 i then c S 1 ∪ S 2 i =
c S 1 i + c S 2 i else c S 1 ∪ S 2 i =
c S (cid:63) i where S (cid:63) =
argmin S j ∈{ S 1 ,S 2 } ( h π
i
S j ))
function Difference ( S 1 , S 2 )
if h π i ( S 1 ) (cid:54) =
h π i ( S 2 ) then
h π
i ( S 1 \ S 2 ) =
h ( S 1 ) , c S 1 \ S 2 i =
c S 1 i else if h π
i ( S 1 ) =
h π i ( S 2 )
then if c S 1 i > c S 2 i then
h π
i ( S 1 \ S 2 ) =
h ( S 1 ) , c S 1 \ S 2 i =
c S
i − c S 2 i else
[Equation]
function Score ( S 1 , S 2 )
[Equation]
s
[Equation]
t
[Equation]
and c S 2 i >ˆ n is the number of hash function pairs with non-zero counts.
E.2 Homomorphic MinHash Results
Recall that our homomorphic minhash algorithm associates a count with each minimum hash value that indicates the number of times that hash value was a unioned into the set as a minimum.
During the diﬀerence operation that can occur due to a split proposal in hierarchical coreference, it is possible that these counts can become zero.
Then, information is lost because it is not possible to recompute the new minimum value without traversing the entire tree, an operation that is expensive to perform in the inner loop of MCMC inference.
Therefore, as a possible remedy, we propose to ignore hash values that have a count of zero in certain situations.
In certain situations, this is unbiased 5 A primary concern though, is that as the number of samples increases, the number of entries with a zero count will increase and (a) the variance of the estimate might increase and (b) the estimate might be impossible to compute if all counts become zero.
We indirectly study (a) by comparing our homomorphic minhash estimates to the exact
Jaccard estimate in much the same we did for simhash (Figure 7).
Each point in the plot represents an MCMC sample, which we gather by running the exact model with no compressive data structures as before.
The x-axis is the diﬀerence in the exact Jaccard similarity estimates of the context bags and the y-axis is the diﬀerence in exact Jaccard similarity between the homomorphic minhash representations of these bags.
Unlike simhash, minhash employs multiple hash functions; therefore, we ﬁx the number of bits to 64 and vary the number of hash functions instead.
We also ﬁt a linear model to the points in order to help assess the bias (as seen by how far oﬀ it is from the reference line), and ﬁnd that indeed our method introduces some bias.
We further examine the percentage of hash functions have a non-zero count.
In particular, after running hierarchical coref in the previous experiment for 100,000 samples, we look at the root level entities and measure what percentage of their hash functions have counts that have gone to zero).
The reason we look at root entities only is that they are the most likely to have had diﬀerence operations applied to them since every time MCMC proposes to split a sub-tree, the root must undergo a diﬀerence operation.
We ﬁnd that on average about 16% of these hash-values have a non-zero count, and ﬁnd that 78% or these representations have fewer than 25% non-zeros, 15% have between 25-50% non-zeros, 4% have between 50-75% and the rest have above 75%.
As for (b), we check after each sample if it results in a hash representation for which all its counts are zero, thus making Jaccard estimates impossible.
6 We record for each of the 100,000 samples whether or not the sample results in a hash for which all counts are zero.
We ﬁnd that this only happens for the 32 and 64 hash function variants, and even in these cases, the percentage is small (1.4 and 1.7% respectively), and they are responsible for the regimentation of points along the x-axis in Figures 7(a)–7(b).
For 128 and 256 hash function variants, this situation never occurs.
This is encouraging as it means, from the perspective of being able to compute an estimate, that we rarely need to refresh the values