Neural Embeddings for Nearest Neighbor Search Under Edit Distance
Anonymous authors
Paper under double-blind review
Abstract
The edit distance between two sequences is an important metric with many applications.
The drawback, however, is the high computational cost of many basic problems involving this notion, such as the nearest neighbor search.
A natural approach to overcoming this issue is to embed the sequences into a vector space such that the geometric distance in the target space approximates the edit distance in the original space.
However, the known edit distance embedding algorithms, such as Chakraborty et al.
(2016), construct embeddings that are data-independent, i e., do not exploit any structure of embedded sets of strings.
In this paper we propose an alternative approach, which learns the embedding function according to the data distribution.
Our experiments show that the new algorithm has much better empirical performance than prior data-independent methods.
Introduction
The edit distance is a popular metric for computing the dissimilarity between two strings.
Given strings x and y , the edit distance d E ( x, y ) between them is equal to the minimum number of character insertions, deletions and substitutions needed to transform x into y .
The metric and its many variants have been deﬁned and studied since the 1960s (Levenshtein, 1966; Wagner and Fischer, 1974).
It has found many applications in computational biology (to measure the similarity between DNA/RNA sequences), natural language processing, information theory and other ﬁelds.
From the algorithmic perspective, the main drawback of edit distance is its computational complexity.
The best known algorithm for computing the distance between two strings of length n takes time that is roughly quadratic in n (Masek and Paterson, 1980); there is also evidence that this runtime is essentially optimal (Backurs and Indyk, 2015).
Furthermore, many applications require ﬁnding the closest string in a large collection of (say) N strings.
Thus, even if we could estimate the edit distance in time O ( n ), computing the distances to all strings in the collection would take O ( Nn ) time, which is infeasible for large data sets.
A natural approach to resolving both issues relies on low-distortion embeddings .
The idea is to construct a mapping f that maps strings into a d -dimensional space (cid:60) d , such that the edit distance d E ( x, y ) is approximated by the distance (cid:107)
f
( x ) − f ( y ) (cid:107) between the images of the strings, up to some factor D called distortion .
If the mapping f can be evaluated eﬃciently, this immediately implies an eﬃcient way to compute the edit distance.
Furthermore, embedding strings into space equipped with a norm (cid:107) · (cid:107) makes it possible to use nearest neighbor search algorithms designed for that norm.
For many normed spaces, e.g.
Minkowski norms (cid:96) dp , many eﬃcient approximate nearest neighbor search algorithms are available.
This dual goal has motivated a long line of research focused on designing eﬃcient low-distortion embeddings of variants of edit distance into normed spaces (Muthukrishnan and Sahinalp, 2000; Cormode et al., 2001; Ostrovsky and Rabani, 2005;
Batu et al., 2006;
Jowhari, 2012; Andoni and Onak, 2012; Chakraborty et al., 2016).
The method given in the last paper (Chakraborty et al., 2016) was shown to yield good results in practice, in the context of large scale similarity joins, i.e., identifying all pairs of similar sequences (Zhang and Zhang, 2017).
A weakness of the aforementioned algorithms is that they are data-independent , i.e., they do not exploit any structure of embedded sets of strings.
In contrast, over the last decade, a large body of research on “learning to hash” showed that data-dependent methods often yield (binary) embeddings with a signiﬁcantly lower distortion than their data-independent counterparts (Wang et al., 2018).
Such methods, however, are typically tailored to obtaining binary representations of vectors that already live in a geometric (typically Hilbert) space.
It is thus natural to investigate whether similar beneﬁts of data-dependent methods can be obtained for edit distance embeddings as well.
In this paper we investigate whether the “learning to hash” approach can yield lower distortion embeddings than those obtained using data-independent methods.
Since our goal is to obtain distance-preserving mappings of sequences, we use Recurrent Neural Networks as a model implementing those mappings.
Speciﬁcally, we leverage a two-layer GRU network, depicted in Figure 2 and described in more details in Section 4.
The network inputs a sequence of characters (e.g., A, G, T or C for DNA sequences), and outputs a vector of symbols, such that the distance between the vectors approximates the edit distance between the original sequences.
The standard loss function for optimizing the network is the triplet loss (see Figure 1), which has two problems in our scenario.
The ﬁrst problem is distance scaling: triplet loss only controls the relative distance ordering among data points after embedding, rather than the absolute embedding distance between any two data points.
This prevents us from getting accurate edit distance estimation from the embedding.
To alleviate this issue, we designed a warm-start phase (Phase 1) before optimizing triplet loss (Phase 2), which initializes the embedding so that the discrepancy between the original edit distance and the embedding distance is minimized.
The second problem is worst case performance of our model.
Due to the nature of learning, the network we learned ﬁts the training data better, and may not do well on unseen hard examples.
To alleviate this issue, we append a back up phase (Phase 3), which deals with hard cases.
We emphasize that all the three phases are optimizing the same network .
We run Adam algorithm for each phase until convergence, and then use its ﬁnal state as the initial state for the next phase.
Although in principle one may use a single phase with combination of diﬀerent loss functions to train our model, this requires additional hyperparameter adjustment for the combining ratios.
Our current three phase training does not need additional hyperparameters, and therefore can easily ﬁt diﬀerent scenarios.
The Phase 3 design is non-trivial.
Our starting observation is that Recurrent Neural Networks that we employ are not “compatible” with the data-independent mappings of Chakraborty et al.
(2016)(which we call CGK from now on).
This is because CGK can generate diﬀerent number of copies for every input character, which is a discrete decision that is hard to be learned by neural networks.
We show, however, that this “extra power” of the algorithm is not necessary.
In particular, we introduce a “truncated” version of the
CGK algorithm (referred to as CGK’), that outputs only 1 or 2 symbols per round.
We then show that CGK’ has the same guarantees as the original CGK algorithm.
Based on this observation, we design our Phase 3 to be an optimized version of CGK’.
Speciﬁcally, since each round there will be either 1 or 2 symbols in CGK’, we can directly compare the two cases and use the diﬀerence as approximation to the gradients to train our model.
Empirically, Phase 3 does not aﬀect the recall on ordinary datasets much, but greatly improves performance on hard instances.
To evaluate the embeddings, we employ them for nearest neighbor search in a collection of sequences, and compare their performance to CGK.
As in Zhang and Zhang (2017) we use three data sets: Gen50ks (containing sequences from the human genome project),
UniRef (protein sequence dataset from the UniProt project) and Trec (containing textual information from Medline, an online medical information database).
After learning the embeddings, we generate a set of queries, and for each query we identify C sequences in the data set (called candidates ) that are closest to the query in the embedded space .
We then measure “top- K recall”, i.e., the fraction of the actual top K nearest neighbors (computed using the exact edit distance) returned as one of the C candidates.
Our results demonstrate that, over a wide range of parameters K and C , our learned embedding consistently achieves higher recalls than the algorithm of CGK, often by a factor of 3 or more.
Similarly, our method often obtains identical recall to that of CGK while using an order of magnitude fewer candidates.
This means that in a typical nearest neighbor search “ﬁltering” pipeline, where the candidates are quickly identiﬁed in the embedded space and then the exact edit distance is computed between the candidates and queries to identify the ﬁnal answers, our method reduces the number of exact edit distance computations by up to one order of magnitude.
Moreover, our learned embedding also has much more organized structure than CGK, as shown in Appendix A.4.
Related work 2. Edit Distance Computing
Exact and approximate algorithms for edit distance computation have been broadly studied, see Navarro (2001) for an overview.
For exact algorithms, Wagner and Fischer (1974) gave
O ( n 2 )-time algorithm using dynamic programming.
This was later improved to O ( n 2 / log n ) by Masek and Paterson (1980).
Faster approximation algorithms were studied as well, see e.g., Landau et al.
; Batu et al.
Bar-Yossef et al.
; Batu et al.
Andoni et al.
; Andoni and Onak (2012).
The current state of the art consists of a near-linear algorithm of Andoni et al.
(2010) guaranteeing (log n ) c
-approximation (with a trade-oﬀ between the running time and the exponent c ), and a constant factor approximation
O ( n 2 − 2 / 7 )-time algorithm of Chakraborty et al.
To the best of our knowledge, neither algorithm has been implemented.
2. Edit Distance Embedding
As mentioned in the introduction, there has been a long line of research focused on designing embeddings of various variants of the edit distance into geometric space.
For the edit distance as deﬁned in the introduction, the best known algorithm is due to Ostrovsky and Rabani (2005), who designed an embedding into the Hamming space with distortion 2 O ( √ log n loglog n ) .
This algorithm was quite complex, and to the best of our knowledge has never been implemented.
Recently, Chakraborty et al.
(2016) gave a substantially diﬀerent embedding with distortion O ( k ) where k is the edit distance between the original strings.
The empirical evaluation of the algorithm, given in Zhang and Zhang (2017), demonstrated that the embedding yields good results in practice, and that the empirical distortion of this embedding is typically much lower than the worst case bound.
Zhang and Zhang (2019) also proposed a novel string partition based algorithm for edit similarity join, without generating embeddings.
Gomez-Bigorda et al.
(2017) presented a novel approach for word embedding in Levenshtein space to improve downstream handwritten word spotting.
Their method is similar to our Phase 1, but our model deals with longer sequences instead of words, and also adopts triplet loss training to capture sequence similarity (Phase 2) and CGK based algorithm to learn sequence alignment (Phase 3).
2. Similarity Capturing
Neural models are powerful to learn embeddings that capture latent similarity.
For example, word2vec (Mikolov et al., 2013) learns semantic and syntactic similarity of a word from its
GRU
GRU GRU
GRU GRU
GRU GRU
GRU x [    · · · ]
Linear Layer f ( x ) [
A
B
C
D · · · ] · · · · · · linguistic context.
The idea has been extended to learn sentence or document similarity (Le and Mikolov, 2014), biological sequence similarity (Asgari and Mofrad, 2015; Ng, 2017).
The aforementioned methods characterize input representations based on their context.
A diﬀerent approach aims to learn a distance metric, see Kulis (2013) for an overview.
Among them, triplet network (Weinberger and Saul, 2009; Hoﬀer and Ailon, 2015; Schroﬀ et al., 2015) uses triplet loss to learn similarity over triplets.
Part of our model gets inspiration from triplet network.
Preliminaries
We use D to denote the alphabet, i.e., the set of symbols used in the strings.
For any two strings x, y ∈
D ∗ we use d E ( x, y ) to denote the edit distance between x and y , i.e., the minimum number of symbol insertions, deletions or substitutions needed to transform x into y .
We deﬁne the distance metric d ( x, y ) =
|| x − y ||
If the strings have diﬀerent lengths, we pad the shorter string to make their lengths equal.
In the binary case, d ( x, y ) corresponds to the Hamming distance between x and y .
The CGK algorithm is described below.
Its input consists of a string x of length n (to be embedded), and a sequence of 6 n random bits r , which are interpreted as a sequence of 3 n random hash functions h 1 , · · · , h 3 n : { 0 , 1 } → { 0 , 1 } .
It produces a sequence f
( x, r ), obtained by replicating each symbol in x at least once (the actual number is a random variable depending on x and r ).
Input: x ∈ { 0 , 1 } n , and a random string r ∈ { 0 , 1 } 6 n
Output: f ( x, r ) ∈ { 0 , 1 } 3 n
Interpret r as the description of h 1 , · · · , h 3 n : { 0 , 1 } → { 0 , 1 } .
Initialization: i = 1, a = (0 , · · · , 0) ∈ { 0 , 1 } 3 n for j = 1 , 2 , · · · , 3 n do a j = x i ; i = i
+
h j
( x i ); end for
Set f
( x, r ) = a.
Learning-based Embedding Method
Recurrent Neural Network (RNN) models proved eﬀective for sequence embedding tasks like language modeling (Mikolov et al., 2011), image captioning (Mao et al., 2015), protein sequence embedding (Bepler and Berger, 2019), etc.
Long Short Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997) and Gated Recurrent Unit (GRU) (Cho et al., 2014) utilize gates to overcome the long-term dependency problem and often yield improved empirical results.
Our learning-based embedding model leverages two-layer GRU depicted in
Figure 2.
The input is a sequence of n characters in one-hot encoding, x = { x 1 , x 2 , ..., x n } .
We feed the input sequence x into two-layer GRU and a linear layer to get the embedding sequence y =
f ( x ).
Here δ i represents the dropout rate in GRU.
We use Adam optimizer and 4:1 train:validation split for cross-validation to tune the hyper-parameters.
Phasex y
Embed
Embed f 1 ( x
[Equation]
f 1 ( y ) | d E − d |
Phasep a n
Embed Embed
Embed f 2 ( p ) f 2 ( a ) f 2 ( n ) tripletloss
Phasef 3 ( x ) [ A
B
C · · · ] f (cid:48) 3 ( x )
[ A
B
C · · · ] A
B
C
Add minor?
[Equation]
4. Phase 1: Absolute Loss as Warm Start
We use Phase 1 to tune the embedding such that the absolute diﬀerence between the edit distance and the embedding distance is small.
In every iteration, we take a pair of input strings x and y , and pass it through two embedding networks denoted as f 1 , which share weights.
The loss is deﬁned as:
[Equation]
4. Phase 2: Triplet Loss for Similarity Capturing
Phase 2 initializes the network with weights computed in Phase 1.
It leverages triplet loss (Weinberger and Saul, 2009) to capture the dependencies between string similarities.
In each iteration we randomly choose an original string as the anchor and deﬁne positive and negative strings with respect to their edit distances from the anchor.
More speciﬁcally, We randomly sample the positive pairs from top-5 edit distance and sample negative pairs as the top-30 distance in the embedded space, which is a fairly standard approach for triplet loss (Weinberger and Saul, 2009; Sablayrolles et al., 2019).
The anchor, the positive and the negative string form a triplet .
The triplets are passed through three networks, denoted by f 2 , which share common weights.
For an anchor a , a positive string p and a negative string n , we formulate the triplet loss as:
[Equation]
Further, we set margin to be d E ( a, n ) − d E ( a, p ).
Intuitively, we hope that the diﬀerence between the hamming distance between the positive and the negative pairs is larger than that of the edit distance, namely d ( f 2 ( a ) , f 2 ( n ))
− d ( f 2 ( a ) , f 2 ( p )) ≥ d E ( a, n ) − d E ( a, p ).
See
Figure 1 for illustration.
4. Phase 3: Insertion Loss for Hard Cases
Phase 3 is initialized using weights computed in Phase 2.
During each iteration a pair of input strings is passed through two embedding networks denoted as f 3 , which share weights.
Let x
and y represent the original input strings and f 3 ( x ) and f 3 ( y ) denote their embeddings.
Denote the CGK’ (a variant of CGK, see Algorithm 3 in the appendix) with random initialization as g , and we apply it on f 3 ( x ), f 3 ( y ) and get f (cid:48) 3 ( x )
=
g ( f 3 ( x )), f (cid:48)
( y ) =
g ( f 3 ( x )).
f (cid:48) 3 is our ﬁnal embedding function, which is essentially a concatenation of an embedding network f and the CGK’ algorithm g .
Optimization in Phase 3 has two diﬀerent losses: the absolute loss and the insertion loss.
The absolute loss ( L 3 ) ensures that the distance of the ﬁnal embedding is close to the edit distance of x and y .
The insertion loss is deﬁned as follows.
For each character c in f 3 ( x ), we compare the two cases that with or without duplication of c to optimize the hash functions and the threshold.
See Appendix A.1 for more details.
[Equation]
As we mentioned before, we use CGK’ instead of CGK because it restricts the number of replications of each symbol to be at most 2, therefore we can compare the two cases to get an approximation to the gradient, and then optimize the parameters in CGK’.
In what follows we show that this modiﬁcation does not aﬀect the worst-case guarantees of the method (i.e.,
CGK’ with random initialization).
See Appendix A.2 for the proof.
Theorem 1.
The mapping f : { 0 , 1 } n × { 0 , 1 } 4 n → { 0 , 1 } 2 n computed by Algorithmsatisﬁes the following properties:
(1) Given r ∈ { 0 , 1 } 4 n and f ( x, r ) ∈ { 0 , 1 } 2 n as the input, we can always decode x . (2)
For every x, y ∈ { 0 , 1 } n , d E ( x, y ) / 2 ≤ d ( f ( x, r ) , f ( y, r )) .
(3) There exists a constant c 0 such that for every positive constant c ≥ c 0 and every x, y ∈ { 0 , 1 } n , d ( f ( x, r ) , f ( y, r )) ≤ c · d E ( x, y ) 2 with probability at least 1 / 2 .
Experiments 5. Datasets
We evaluate our neural embedding with three datasets as in Zhang and Zhang (2017).
Speciﬁcally, we use Gen50ks : a genetic sequence dataset from the human genome project;
UniRef : UniRef90 protein sequence dataset from the UniProt project; and Trec : a reference dataset from Medline (an online medical information database).
See Table 1 for their properties.
Furthermore, following Zhang and Zhang (2017) we truncate some input strings to avoid dealing with very long outlier strings.
For Gen50ks dataset, the string lengths sharply concentrate around 5000, so we set the truncation length to be 5000 and thresholding aﬀects the distances only minimally.
For UniRef and Trec dataset, string lengths vary drastically.
In order to avoid biasing the original data distribution, we set a rather large threshold(1500 and 2500) so that only a small proportion(1.16% and 0.97%) of the strings exceed the threshold.
For Gen50ks, UniRef and Trec dataset, we train our model with 1000, 2000, 2000 strings (including base and query) respectively, which means that we train with less than 2% data points in the whole dataset.
[Table]
5. Nearest Neighbor Search 5.2. Exact Nearest Neighbor Search
We evaluate our model in the context of nearest neighbour search.
We randomly selectqueries and use the remainder of the dataset as the base set.
Our goal is to ﬁnd strings in the base set most similar to the query strings.
To this end we use linear scan to identify the
C sequences in the data set (called candidates ) that are closest to the query in the embedded space .
We compare embeddings generated by our neural model, CGK and its close variant
CGK’, as well as the identity embedding (which is optimal if no insertions or deletions are needed).
Since CGK and its variants are randomized, we measure the average of ﬁve independent runs.
For all three datasets we observe that our neural model oﬀers signiﬁcant performance gains compared to the other embedding methods.
[Figure] [Figure]
[Figure]
5.2. LSH-based Approximate Nearest Neighbor Search
In the previous section we used linear scan to ﬁnd C nearest neighbors in the embedded space.
Although this task can be often accomplished eﬃciently (e g., using GPUs), in many scenarios more eﬃcient approximate nearest neighbor search algorithms need to be employed.
In this section we investigate one such instantiation employing Locality-Sensitive
Hashing (LSH).
We note that the same framework was used in Zhang and Zhang (2017) to evaluate CGK.
We use the cross-polytope LSH family under the Euclidean distance with multi-probe LSH
(Andoni et al., 2015).
For genetic dataset Gen50ks we build 50 hash tables and 10-bit hash functions(We vary the number of hash bits and ﬁnd 10-bit hash functions give the best performance)
[Table] [Figure]
[Table]
[Table]
[Table]
[Table]
5. String Similarity Join
Zhang and Zhang (2017) adopts the vanilla bit-sampling LSH for hamming distance [Table] which does not suit our embedding as the neural embedded strings are continuous values instead of discrete characters.
So for our string similarity join experiments, we continue to use the cross-polytope LSH family under the Euclidean distance with multi-probe LSH.
The LSH table parameters are the same with the approximate nearest neighbor setting.
We run PassJoin (Li et al., 2011) to evaluate our join results.
5. Ablation Study
We further do ablation study to evaluate the function of three phases in our model.
We notice a performance drop by removing Phase 1 or Phase 2.
We explain this as Phase 1 provides good initialization which makes similarity learning easier while Phase 2 learns string similarity.
Removing phase 3 generally does not lower the performance of top-1 recall much.
This might be that these datasets do not contain many hard instances(like coexisting small shift and small substitution).
Phase 3 does show its eﬀectiveness in our synthetic hard case dataset, see Appendix A.3.
We further show the standard deviation by testing with ﬁve diﬀerent query sets, each of which contains 100 randomly selected strings.
Conclusion
In this paper, we introduce the idea of using neural embedding for computing nearest neighbor search under edit distance.
Our method has three phases, i.e., the warm-start phase for generating an initial network with the correct scale, the triplet loss phase for learning relative distance among data points, and the back up phase for dealing with hard cases.
Our last phase is non-trivial, because we designed a truncated version of CGK algorithm (aka
CGK’)
and proved that it has worst case performance guarantee.
Our phase 3 directly optimizes CGK’ and easily solves hard cases.
Overall, our method gives much better empirical results than the existing methods for nearest neighbor search.
A Appendix A. CGK’ algorithm and Phase
In Algorithm 2, we present the truncated version of CGK that we use in the paper.
Compared with Algorithm 1, the main diﬀerence is in CGK’, every character will be duplicated for at most twice (with probability 0 . 5).
Our Phase 3 is an optimized version of CGK’.
We use Algorithm 3 in Phase 3.
The main diﬀerence between Algorithm 2 and 3 is that we are using a threshold variable thres in the if-statement, to decide whether to duplicate the character or not.
Moreover, the if-condition is evaluated based on the inner product between hash function h j and a one hot vector x i .
Hence, Algorithm 3 is just a more general version of Algorithm 2.
Then how do we optimize the parameters in Algorithm 3(E.g., thres and h )?
Since Algorithm 3 involves discrete decisions with if-statements, we manually deﬁne the gradients in
Algorithm 4, by comparing the case whether the character is duplicated or not and calculate how it aﬀects the loss.
Input: x ∈ { 0 , 1 } n , and a random string r ∈ { 0 , 1 } 4 n
Output: f ( x, r ) ∈ { 0 , 1 } 2 n
Interpret r as the description of h 1 , · · · , h 2 n : { 0 , 1 } → { 0 , 1 } .
Initialization: j = 1, a = (0 , · · · , 0) ∈ { 0 , 1 } 2 n for i = 1 , 2 , · · · , n do if h j ( x i ) then a j = x i ; j = j + 1; else a j +1 =
a j = x i ; j = j + 2; end if end for
Set f
( x, r ) = a.
Algorithm
Input: x where x i is a one-hot vector, H is a 2 × 2 n matrix interpreted as h 1 , ..., h 2 n , thres ∈ R after optimization
Output: f (cid:48) ( x ), set Duplicate which records the duplicated bits in f (cid:48)
( x )
Initialization: j = 1 , a = 0 2 × 2 n , Duplicate = ∅ for i = 1 , 2 , · · · , n do if h j · x i < thres then a j = x i ; j = j + 1; else a j
; end if end for
Set f (cid:48)
( x ) = a.
A. Proof of the Main Theorem
For simplicity, denote the two embedded strings f ( x, r ) , f ( y, r ) as f x , f y .
Below we always use i to denote the index for the input strings, and use j to denote the index for the embedded strings.
We use idx( x, j ) to denote the corresponding index i for producing the
Input: x, y, f (cid:48) ( x ) , f (cid:48) ( y ), H interpreted as h 1 , ..., h 2 n , thres , set Duplicate in Algorithm 3, learning parameters (cid:15), η
Output: H and thres after optimization
Initialization j =
L = | d H ( f (cid:48)
( x ) , f (cid:48) ( y ))
− d E ( x, y ) | ; for i = 1 , 2 , · · · , n do g ( x ) =
f (cid:48)
( x ) if j ∈ Duplicate then
Remove g ( x ) j +1 from g ( x );
L (cid:48)
=
| d H ( g ( x ) , f (cid:48) ( y ))
− d E ( x, y ) | ; if L (cid:48) <
L then thres = thres + (cid:15) ;
h j = h j − η · x i ; end if j = j + 2; else
Duplicate g ( x ) j in g ( x );
L (cid:48)
=
| d H ( g ( x ) , f (cid:48) ( y ))
− d E ( x, y ) | ; if L (cid:48) <
L then thres = thres − (cid:15) ; h j = h j + η · x i ; end if j = j + 1; end if end for j -th index of output f ( x, r ).
Deﬁne ∆ j (cid:44) idx( x, j ) − idx( y, j ) as the input index diﬀerence between x and y at embedding index j .
In Algorithm 2, given the input x and index i , we may append one or two copies of x i into the output.
We call the ﬁrst copy the major copy, and the second copy the minor copy.
The major copy always exists in the output, but the minor copy exists with probability 1 / 2, depending on the hash function h j .
For any index j , if f xj and f yj are both major copies or both minor copies, we say j is aligned .
Otherwise, j is not aligned .
Initially, f x 1 and f yare major copies, therefore index 1 is aligned.
See ﬁgure 7 for illustration.
We scan f x and f y from left to right, and for every index j such that j +1 is not aligned but j is aligned, we ﬁnd the next aligned index j (cid:48) , and group all the indices j +1 , j +2 , · · · , j (cid:48) −as a sub-index string for the aligned index j .
For the corner case that j (cid:48) does not exist, we pick j (cid:48) as the minimum ﬁnal j value for computing f x and f y
(So starting from j (cid:48) , f x or f y will be padded with zeros).
Now we focus on the aligned indices.
For an aligned index j without a sub-index string, we know index j +1 is also aligned, otherwise j +1 will be inside the sub-index string for index j .
In this case, we can show ∆ j +1 − ∆ j = 0 by discussing whether index j + 1 has major or minor copies.
On the other hand, consider an aligned index j with a sub-index string.
In this case, we know both f xj and f yj are major copies.
Without loss of generality, we may assume f xj +1 is a major copy and f yj +1 is a minor copy (the other case is symmetric).
We have the following lemma for a few facts related to the sub-index string.
Lemma 2.
Denote the next aligned index by j (cid:48) >
j + 1 .
Assume the sub-index string has length t (cid:44) j (cid:48)
− j − 1 > 0 .
Given T such that T/ 2 ∈ Z
+ , we have ( f xj (cid:48) and f yj (cid:48) are major copies,
t
≤
T holds with probability 1 − 2 − T ,
A B B C C D E E
B B C C D D E E j j (cid:48)
A B B C C D E F
B B C C D E E F j j (cid:48)
A : Major
A : Minor : Aligned : Not Aligned
(3) Conditioned on
[Equation]
Proof.
Since j (cid:48) is the ﬁrst aligned index after j , we know that for any j < j o < j (cid:48) , j o is not aligned, therefore one of f xj o and f yj o is major and the other one is minor.
By Algorithm 2, for the string with the minor copy, with probability 1, the next copy on j o + 1 is a major copy; and for the string with the major copy, the next copy on j o + 1 is a major copy or minor copy with equal probability.
However, if both copies on j o + 1 is major, we know j o + 1 is aligned, and thus j (cid:48) =
j o + 1.
In other words, this case only happens for the index j
o = j (cid:48) − 1, and for other indices, the two strings are not aligned, and have major copies in turn.
That already proves (1).
Since the two strings have major copies in turn, and only major copies will increment the value of idx( x, j ), we know
[Equation]
See Figure 7 for the two possible cases.
Moreover, every step from j + 1 to j (cid:48) holds with probability 1 / 2, so the probability that the sub-index string has length t is 2 − t .
By summing over the probabilities of t from 1 to T , we proved (2).
Finally, (3) holds by comparing the probabilities of t ≤ T being odd and being even, which is exactly 2 : 1.
Remark 1.
Based on Lemma 2(2), if there are l sub-index strings, and for a given T (to be determined later), the probability that all sub-index strings are within length T is bounded by 1 − l · 2 − T using union bound.
Below we discuss the case conditioned on this event.
Lemma 2(3) states that if an aligned index j has a sub-index string, and f xj +1 is a major copy, then ∆ j will be incremented by 1 with probability 2 / 3, and stay unchanged with probability 1 / 3.
Since f x j , f y j are major copies, we know that this case happens with probability 1 / 4.
The symmetric case where f xj +1 is a minor copy and f y j +1 is a major copy also happens with probability 1 / 4.
Combine the two possible cases for aligned index j (with or without sub-index string), and assuming the next aligned major index is j (cid:48) .
If f xj and f yj are minor copies, or are the same, we have ∆ j (cid:48) − ∆ j = 0 by simple case analysis.
Otherwise, we have:
[Equation]
This is because for the two aligned major copies at index j , with probability 1 / 2, the next index j + 1 is still aligned, and as a result ∆ j does not change.
With probability 1 / 2, there exists a sub-index string for index j , but conditioned on this, the probability that ∆ j does not change is 1 / 3.
So the total probability for being unchanged is 1 / 2+1 / 6 = 2 / 3.
The other two cases are symmetric, so each has probability (1 − 2 / 3) / 2 = 1 / 6.
This deﬁnes a random walk process for ∆ j , and the remaining analysis is similar to the analysis in Chakraborty et al.
Assume that d E ( x, y )
=
k .
Let x = x (0) , x (1) , · · · , x ( k ) =
y be a series of strings such that d E ( x ( l − 1) , x ( l ) )
= 1.
Let i l be the ﬁrst index on which x ( l − 1) and x ( l ) diﬀer.
For simplicity we assume that i 1 < · · · < i k
(and we will later show this is without loss of generality).
Let d j be the diﬀerence between deleted and inserted bits between x and y before the index j .
Therefore, d 0 = 0, and
[Equation]
=
[Equation]
l
[Equation]
x ( l ) is obtained by deletion
[Equation]
idx
[Equation]
idx
[Equation]
x
[Equation]
(cid:54) =
i l
[Equation]
x ( l ) is obtained by insertion d
[Equation]
otherwise , (9) d j denotes the required length of “shift” between f x and f y .
In other words, for j such that idx( x, j ) ∈ ( i l , i l +1 ), and idx( x, j ) − idx( y, j ) =
∆ j = d j , we know f xj = f yj .
Moreover, for any j (cid:48) ≥ j such that idx( x, j (cid:48) ) ∈ [idx( x, j ) , i l +1 ), we know that f xj (cid:48) =
f yj (cid:48) holds as well, because the same hash function will be applied to both strings after idx( x, j ).
By modeling ∆ j and d j as a chasing game between a cat and a dog with a kennel, the following lemma was proved in Chakraborty et al.
Lemma 3 (Lemma 4.4 in Chakraborty et al.
If d E ( x, y ) =
k , then
Pr( |{ j : f xj (cid:54) =
f yj , j is an aligned major index }| ≤ l ) ≥ l (cid:88) t =
q ( t, k )
Where q ( t, l ) denotes the probability that the random walk process deﬁned by (8) starting at the origin, reaches the point l at time t for the ﬁrst time.
As discussed in Chakraborty et al.
(2016), the assumption that i 1 , · · · , i k are strictly monotone is without loss of generality, because it suﬃces to modify the updating rule of d j in (9) such that d j and d j − 1 can be diﬀered by more than 1 unit.
Once d j is properly deﬁned,
Lemma 3 still applies.
The sum (cid:80) lt =0 q ( t, k ) in Lemma 3 is easy to bound:
Lemma 4.
For any k, l ∈ N , with probability more than 1 − e − l 24 , it holds that
[Equation]
In particular, (cid:80) 7776 k 2 t =
q ( t, k ) ≥ 23 .
Proof.
Below is a well known fact about random walks that can be found e.g. (Theorem 2.17 in Levin and Peres (2017)).
For any k, l ∈ N , it holds that:
[Equation]
where r ( t, k ) denotes the probability a simple random walk starting at the origin reaches point k for the ﬁrst time in t steps.
Let m be the number of non-lazy steps during a t-step random walk deﬁned in (8).
By applying Chernoﬀ Bounds , for any 0 < δ < 1 we have:
[Equation]
Therefore, with probability more than 1 − e − δ
[Equation]
t , we are performing more than (1 − δ ) 13 t -steps simple random walk.
Setting δ = 12 gives us the result in Lemma 4.
Now we are ready to prove our Theorem 1.
Proof of Theorem 1.
(1) is trivially true, because we can simulate Algorithm 2 using the random string r to decide for each index of f x whether it is a major copy or not.
To see (2) is true, we need to do case analysis.
We scan f x and f y from left to right, and skip all indices that are the same.
For the ﬁrst diﬀerent index j , there are three cases:
j is aligned major, and j + 1 is aligned minor. 2.
j is aligned major, and j + 1 is also aligned major.
j is aligned major, and j + 1 is not aligned.
For the ﬁrst two cases, it suﬃces to replace the bit at idx( x, j ) of x , and we can keep scanning starting from the next aligned major.
For the third case, we can do induction on the length t of the sub-index string for j , to show that it suﬃces to make at most s edits if there are s diﬀerent bits from j to j + t .
Without loss of generality, we assume that the index j + 1 of f x is a major copy while j + 1 of f y is a minor copy.
• If t = 1, which means j + 2 is aligned major.
In this case, if bits at index j + 1 of f x and f y are matched, it suﬃces to edit the bit at index j .
Otherwise, it suﬃces to delete idx( x, j ) of x , and replace the bit at idx( x, j + 1) of x to match y .
• If t = 2, which means j + 3 is aligned major, and the index j of f y is a major copy.
In this case, we check whether the bits at j +
2 for f x , f y are the same.
– If yes, it means, idx( y, j + 2) of y and idx( x, j + 2) of x are the same.
In this case, it suﬃces to replace idx( x, j ) of x to match y .
– If no, it suﬃces to replace both idx( x, j ) , idx( x, j + 1) of x to match y .
• If t > 2, we also check whether the bits at j + t for f x , f y are the same.
– If yes, it means, idx( y, j + t ) of y and idx( x, j + t ) of x are the same.
Then we may reduce the analysis to the t − 2 case.
By induction, we know that from j to j + t − 2, we can make the proper edit to match x and y .
– If no, it means idx( y, j + t ) of y and idx( x, j + t ) of x are diﬀerent.
We make one edit on idx( x, j + t ) to match idx( y, j + t ) of y , and then reduce the analysis to the t − 2 case.
Notice that in the above two sub-cases, we did not analyze the case at index j + t − 1.
This is because this index is not aligned, and the minor copy is included in the analysis of index j to j + t − 2, while the major copy is included in the analysis of index j + t .
The above analysis shows that it suﬃces to make at most d ( f x , f y ) edits to match x and y until one of f x or f y starts to pad zeros.
Since there are at most d ( f x , f y ) deletions or insertions on x , and originally x and y have the same length, we know that there are at most d ( f x , f y ) padded zeros.
Combine the two cases together, we have d E ( f x , f y ) / 2 ≤ d ( f x , f y ).
(3) Apply Lemma 3, we know that with probability (1 − e − l 24 ) · (1 − 12 k (cid:113) 6 l ), there are at most l aligned major indices that satisﬁes f xj (cid:54) =
f yj .
Notice that, if for any indices, if f xj = f yj , it does not aﬀect d ( f x , f y ).
Hence below we consider other cases when f xj (cid:54) =
f yj :
• If j is an aligned minor, we know that j − 1 is aligned major such that f xj (cid:54) =
f yj , and also j + 1 is an aligned major index.
Thus the number of such indices is at most l .
• If j is not aligned, we know that j belongs to a sub-index string for an aligned major index j (cid:48) <
j . Notice that f xj (cid:54) =
f yj , otherwise j (cid:48) won’t have sub-index string.
We bound the number of such indices below.
To recap, we need to bound the expected sum of lengths of sub-index strings that start from an aligned major index j such that f xj (cid:54) = f yj .
Since the number of such strings is l , and the expected length of each string is at most 2, it follows that the expected sum of lengths is at most 2 l .
We can thus use Markov inequality to show that this quantity is at most 20 l with probability at least 9 / 10.
We note that in Lemma 3, we did not consider the comparison between f x and f y when one of them has padded zeros.
However, it is easy to see that in this case every edit operation can only incur one unit of cost in d ( f x , f y ), which is not the worst case.
Therefore, with probability at least (1 − e − l 24 ) · (1 − 12 k (cid:113) 6 l ) − 0 .
1, the hamming distance is at most 20 l .
Let l = ck 2 20 , we know the probability is at least
[Equation]
The last inequality holds because k ≥ 1, otherwise the hamming distance is 0.
Therefore it suﬃces pick c 0 = 50000, and the expression above is larger than 0 .
Notice that we did not carefully tune the constants here.
[Figure]
A. Synthetic Data Set
Although Phase 1+2 has good performance for most data sets, we can easily construct synthetic data sets for which it will fail.
For example, given a string x , applying 1-bit shift to the right gives rise to a large hamming distance but the edit distance is only two.
If this 1-bit shift coexists with a 3-bit substitution (so that the hamming distance and edit distance are both equal to three), it would be quite “hard” for Phase 1+2 to ﬁnd nearest neighbors under edit distance.
However, adding Phase 3 easily solves the problem, and gives improved accuracy on the challenging hard cases.
[Table]
We have generated a synthetic data set to illustrate the algorithm eﬀectiveness.
First we have chosen some preﬁx strings from the Gen50ks dataset as the query set.
Then, for each query, we generate its small shifts (one or two) and small substitution strings (three or four), which are used as the base set.
For each query we want to ﬁnd the closest edit distance string in the base set.
The performance of all studied algorithms on this data set are shown in Table 6.
It can be seen that adding Phase 3 improves the performance of our method.
A. Visualization
We visualize the embedding results for both CGK and our model using t-SNE (van der
Maaten and Hinton, 2008).
For CGK embedding, the points distribute over the 2D space randomly no matter what hyper-parameters are used.
For neural embedding, on the other hand, points are well clustered.
Figure 8 shows 2D t-SNE projections with perplexity 8 and learning rate 10.