S EARCH DATA STRUCTURE LEARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
In our modern world, an enormous amount of data surrounds us, and we are rarely interested in more than a handful of data points at once.
It is like searching for needles in a haystack, and in many cases, there is no better algorithm than a random search, which might not be viable.
Previously proposed algorithms for efﬁcient database access are made for particular applications such as ﬁnding the min/max, ﬁnding all points within a range or ﬁnding the k-nearest neighbours.
Consequently, there is a lack of versatility concerning what we can search when it comes to a gigantic database.
In this work, we propose Search Data Structure
Learning (SDSL), a generalization of the standard Search Data Structure (SDS) in which the machine has to learn how to search in the database.
To evaluate approaches in this ﬁeld, we propose a novel metric called Sequential Search Work
Ratio (SSWR), a natural way of measuring a search’s efﬁciency and quality.
Finally, we inaugurate the ﬁeld with the Efﬁcient Learnable Binary Access (ELBA), a family of models for Search Data Structure Learning.
It requires a means to train two parametric functions and a search data structure for binary codes.
For the training, we developed a novel loss function, the F-beta Loss.
For the SDS, we describe the Multi-Bernoulli Search (MBS), a novel approach for probabilistic binary codes.
Finally, we exhibit the F-beta Loss and the MBS synergy by experimentally showing that it is at least twice as better than using the alternative loss functions of MIHash and HashNet and twenty times better than with another SDS based on the Hamming radius.
I NTRODUCTION
In many applications, the machines need to perform many searches in a gigantic database where the number of relevant documents is minuscule, e.g. ten in a billion.
It is like searching for some needles in a haystack.
In those cases, considering every document is extremely inefﬁcient.
For productivity, the search should not consider the whole database.
Traditionally, this is accomplished by building a search data structure and seeking within it.
Those data structures can take many forms.
For example, there are tree-based structures such as the B-Tree (Bayer & McCreight, 1970), the k-d tree (Friedman et al., 1977), the R-Tree (Guttman, 1984) or the M-Tree (Ciaccia et al., 1997) to name a few.
In addition to trees, KNNG (Paredes & Ch´avez, 2005) build a graph designed for the k-nearest neighbour search.
Later approaches improve on KNNG, both for construction and search time and for the search quality itself.
In those lines, there is Efanna (Fu & Cai, 2016), HNSW (Malkov &
Yashunin, 2018) and ONNG (Iwasaki & Miyazaki, 2018).
One of the most common types of search data structures is the hash table.
It is so useful that it is implemented natively in programming languages such as Python (with the dictionary type).
Hash table is often the main tool an application will use for efﬁciency.
For example, from a short and noisy song sample, Shazam (Wang et al., 2003) can retrieve the whole song by using hash tables ﬁlled with well-designed ﬁngerprints of each song.
Traditionally, the design of a search data structure was for a particular type of search.
For example, hash tables can retrieve documents very quickly, even in gigantic databases.
However, the query must be equal to the key.
This requirement makes the hash table not always applicable.
For instance, if the database is indexed by date and time and we seek all documents from a speciﬁc day, then it might not be optimal to query every second of that day with an equality search.
B-Tree (Bayer &
McCreight, 1970) was precisely introduced for applications where a range search is preferable (and faster insertion than dichotomic search is needed).
Equality and range are far from being the only types of search.
For instance, the k-nearest neighbours is another well-studied type a search.
Also, the subset search is a more exotic example that occurs when every queries and documents are sets and when a document is relevant if and only if it is a subset of the query.
As a ﬁnal example, the auto-complete function often uses a Trie data structure (De La Briandais, 1959) to suggest the end of each word.
It is easy not to realize how the problem of efﬁciently ﬁnding needles in a haystack was solved multiple times for speciﬁc applications.
This is the ascertainment that make Search Data Structure Learning (SDSL) a signiﬁcant subject.
Machine Learning has been a very ﬂexible paradigm, whether by solving multiple NLP (Natural Language Processing) tasks with a unique Transformer (Vaswani et al., 2017) or solving most Atari games with Reinforcement Learning (Mnih et al., 2013), the capacity of a single learning algorithm to perform on multiple tasks is outstanding.
Search
Data Structure Learning aims at developing generic learning algorithms meant for multiple types of search.
Furthermore, what makes a document relevant need not to be described formally or even understood by a human.
It might be k-nearest neighbour with a complex metric or something else altogether, the only thing we need for learning is a dataset.
While we use the term ”Search Data
Structure Learning” for the ﬁrst time, algorithms that fall into its paradigm already exist.
The large video-hosting platform YouTube implements an SDSL algorithm (Covington et al., 2016) for its recommendation system (the user being the query and the videos being the documents).
Not having a formalized deﬁnition of what makes a document relevant and relying on Machine
Learning has its challenges, the most important being the evaluation.
Traditional search data structure, such as the hash table, the Trie, the B-Tree, are exact, meaning the documents retrieved contains all and uniquely the relevant documents.
For comparing those exact search data structures, when possible, the comparison between two exact search data structures is made with the asymptotic time complexity (the big-O notation).
However, when the search is not exact, it is unclear how to compare structures with different efﬁciency and exactitude.
The precision at Hamming distance ofis an attempt to unify those two properties into a single measure speciﬁc to the context of binary encoding.
However, as described below, it fails in many aspects.
It might seem like it is up to the programmer to decide what is more important between the speed and the quality of the retrieved documents.
For example, the recall-queries per second (Aum ¨ uller et al., 2017) helps to visually understand the trade-off between speed and quality.
In section 3, we describe a reliable measure to evaluate the efﬁciency and quality simultaneously of any search data structure.
This metric solidiﬁes the Machine Learning subﬁeld of Search Data Structure Learning.
This article presents the SDSL framework that brings two crucial generalization w.r.t.
its predecessors (Li et al., 2011; Cayton & Dasgupta, 2008).
First, it allows for dynamic databases, i.e. databases that might change or evolve after training.
For example, it is plausible that a company wants to design and train a search engine ready for distribution to multiple clients without further training on each client’s database. The current mindset is to retrain each time a new database is given; however, this is not feasible in many cases. Hopefully, this article motivates the research towards models that can generalize to never seen databases. Secondly, the previous framework does not support relative relations, i.e. when the relevance of a document w.r.t.
a query depends on the other documents in the database.
The most studied relative relations is probably the KNN task, which is relative since it is impossible to know if a document is in the k-nearest neighbour of a query without knowing the other documents.
In contrast, radius search is an example of what we call an absolute relation because it is possible to know if a document is relevant to a query only by looking at the query-document pair.
In this work, however, we did not introduce relative relations only for KNN.
Many interesting relative relation tasks exist; for example, another rather exciting example of relative relation is the multiple supporting facts:
A harder task is to answer questions where two supporting statements have to be chained to answer the question [...] where to answer the question “Where is the football?”
one has to combine information from two sentences “John is in the playground” and “John picked up the football”.
(Weston et al., 2015).
In this work, we ﬁrst introduce a general framework to formalize the SDSL task in which we present a novel metric to simultaneously evaluate the efﬁciency and quality of the search.
Then, we inaugurate the ﬁeld of SDSL with Efﬁcient Learning Binary Access (ELBA)
(Section 4) that describes a family of models that use a traditional search data structure and parametric functions (e.g. neural networks) to create a discrete binary code(s) for both the queries and documents.
A reader familiar with the ﬁeld must appreciate the difﬁculty that has to be overcome when dealing with (semi)- discrete structure.
To instantiate ELBA, we concocted the F-beta Loss used for the training and Multi-Bernoulli Search (MBS), a novel SDS technique designed for probabilistic binary codes.
Finally, for comparisons, we will instantiate ELBA with other loss functions and another SDS, namely the MIHash’s loss (Cakir et al., 2017), the HashNet’s loss (Cao et al., 2017), and the Hamming Radius Search C.4.
We will then experimentally show the F-beta Loss and MBS’s advantage by putting in evidence their synergy.
R ELATED WORK
In data structure terminology, the concepts of dynamic and static structures describe whether or not the structure can change via insertion, deletion or merge.
In SDSL, if the database(s) used for training are not the same as the one(s) used for evaluation, then the structure has to search for documents only seen once at insertion.
From a Machine Learning perspective, this is known as a One-Shot Learning task.
For example, Matching Network (Vinyals et al., 2016) tries to match never seen elements together.
However, applying their technique would require a database scan.
Hence it is incompatible with a gigantic database.
In the same vein, soft addressing (or attention) is a differentiable mechanism to select an element from many, thus compatible with gradient descent.
Memory Network (Kumar et al., 2016), Neural-Turing Machine (Graves et al., 2014) and
Transformer (Vaswani et al., 2017)
all use some kind of soft addressing.
It is interesting for training our models but cannot be used alone in SDSL.
For the same reason as above, it would require considering the whole database.
Finding the k-nearest neighbour is trivial with unlimited resources.
In this ﬁeld, the research focuses mainly on the efﬁciency of both the search and the structure’s construction.
The exact algorithms are as efﬁcient in higher dimensions than a random search due to the curse of dimensionality.
Consequently, the focus has recently been on approximate k-nearest neighbour.
The search data structure developed are mostly tree-based, such as the k-d tree (Friedman et al., 1977) or the K-Means tree (Nister & Stewenius, 2006), and graph-based, such as the KNNG (Paredes & Ch ´ avez, 2005), Efanna (Fu & Cai, 2016), HNSW (Malkov & Yashunin, 2018) or ONNG (Iwasaki & Miyazaki, 2018) just to name a few.
A good resource for comparing those approaches is the ann-benchmark Aum¨uller et al.
In this work, we generalize the problem to conceive algorithms able to learn what to search efﬁciently.
Efﬁcient Learnable Binary Access, described below, encodes queries and documents into binary vectors.
In this work, we will use neural networks as the encoders.
Such encoders already exists in the literature.
For example, CNNH (Xia et al., 2014), DPSH (Li et al., 2015), DHN (Zhu et al., 2016), GreedyHash (Su et al., 2018), PGDH (Yuan et al., 2018), HashGan (Cao et al., 2018), ADSH (Jiang & Li, 2018) or JMLH (Shen et al., 2019), just to name a few.
Below we compare different loss functions, the F-beta Loss 4 the MIHash’s loss Cakir et al.
(2017) and HashNet’s loss
Cao et al.
Graph learning, introduced in Zhu et al.
(2003) for semi-supervised learning, is a type of data structure learning that has shown experimentally to be a strong idea.
Those models learn to do inference from graphs, sometime by generating them ﬁrst.
Some approaches work with static graphs (static structures) (Zhu et al., 2003; Perozzi et al., 2014; Scarselli et al., 2008; Bruna et al., 2013) while other work with dynamic graphs (dynamic structures)
(Narayan & Roe, 2018; Manessi et al., 2020).
While this literature does not focus on retrieval, they learn to compute using a data structure.
To put SDSL in contrast with the Learning to Search framework (Li et al., 2011).
As mentioned in the introduction, it does not support dynamic databases and relative relation. It is possible to update the framework to deal with dynamic databases by taking an expectation over the databases in the retrieval quality Q ( T ) and computational cost C ( T ) . However, it is not clear how to deal with relative relations because the selection function T ( q, x ) is a ”matching function” that does not exist for relative tasks. Generalizing the selection function by allowing it to consider the whole database (i.e. with T ( q, X ) ) does not work because T ( q, X ) could use the ranking function s ( x, q ) on every document and nothing would penalize such exhaustive strategies since the computational cost is the number of candidates.
Nevertheless, this is not the main issue. As with the framework proposed in Cayton & Dasgupta (2008), the computational cost does not consider the retrieval cost but only the size of the candidates set (divided by the number of documents in the database for the latter framework). Those frameworks fail to quantify the work needed to retrieve the candidates. For example, while proposing the Learning to Search framework, the authors relied on timing to evaluate their model.
The SDSL framework, proposed below, provides a unique quantity that quantiﬁes both the cost of retrieval and the candidates’ quality simultaneously.
Finally, while not introduced as such, an SDSL algorithm is used in NLP. In this ﬁeld, many articles attempt to accelerate the training and inference of the neural network based models, in which the main bottleneck is the normalization over a large vocabulary.
Morin & Bengio (2005) use a precomputed tree and train their model to travel from the root to a leaf, where each leaf corresponds to a word.
Doing so accelerates both training and inference.
Latter, Mnih & Hinton (2009) proposed a way to learn the structure of the tree.
F ORMALISATION OF THE PROBLEM
Let Q be the query universe, let U be the document universe, and let D be the set of databases, i.e. the set of all ﬁnite sets of documents.
The task is formulated with a set of relations corresponding to each database R =
{R D : Q → 2 D | D ∈ D } .
This is to allow the general case where the relation is relative.
Deﬁnition 3.1.
The relation set R is absolute if there is a match function M : Q × U → { True , False } s.t. ∀ q ∈ Q , D ∈ D , d ∈ D, M ( q, d ) ⇔ d ∈ R D ( q ) otherwise, we say it is relative .
This deﬁnition can easily be generalized to the cases where each R D is probabilistic map by using probability instead of a truth value.
For the F-beta loss to be deﬁned in Section 4, we restrict ourselves to absolute relation sets and thus, only the query-document pair determines if the document is relevant.
However, the rest of this formalization is for both relative and absolute relation sets.
The mAP is a widely used metric in information retrieval.
However, it does not consider the work done to perform the ranking. An SDS could compare the query to every document in the database and have a good mAP. In SDSL, we want to monitor the quality as well as the efﬁciency of retrieval. The Recall Query per second (RQPS) (Aum ¨ uller et al., 2017) is also used in the ANN literature.
However, it is not suitable for theoretical analysis since the results depend on the implementation and the hardware.
It is possible to generalize the RQPS by changing what quantiﬁes the amount of work done for retrieval (to something else than the number of seconds).
Nevertheless, the RQPS has a parameter (k) that limits the number of candidates to generate.
This parameter prevents a model from generating all documents in the database as its candidates, which would give 100% recall without any computation and, consequently, having an excellent score doing nothing.
Ultimately, the parameter k is a ﬁx to the ﬂaw that the RQPS does not consider the precision.
In SDSL, we want to legitimately compare models that might not generate the same number of candidates.
To the best of our knowledge, the precision at a Hamming distance of two (p@2) is the only proposal in the literature to consolidate the search’s efﬁciency and quality without relying on the hardware and the implementation.
Obviously, this metric has many limitations.
First, it is only relevant when the model transforms the queries and documents into binary codes.
More importantly, it does not consider the recall quality.
For example, if a query should return several relevant documents but only one relevant document (and no irrelevant ones) are within a Hamming distance of two according to the model, the p@2 score would be maximal for this query even though the system has a very poor recall.
Another signiﬁcant limitation is the fact that it does not weight the score w.r.t.
the distance.
In many contexts, the amount of work increases a thousandfold when comparing a perfect match (Hamming distance 0) and distance 2.
In this work, we present a generic metric for any SDSL task.
We grounded the metric on a very pragmatic standpoint by asking what kind of strategy a programmer would use to ﬁnd relevant documents in a database quickly.
At ﬁrst, one might consider a random search with a good matching function (e.g. a neural network).
However, if the database is enormous, this strategy will give poor performances.
One could then consider ﬁltering a signiﬁcant portion of the database using a search data structure, but the retrieved documents might contain multiple false positives, decreasing the precision.
We believe we can have the best of both solutions by combining them, ﬁrst ﬁltering a large portion of the database with a search data structure and then using a good matching function to ﬁlter the retrieved documents.
Finally, to evaluate if the search data structure is useful, the programmer could consider the cost of searching in the structure plus the cost of a random search in the retrieved documents versus the cost of a random search in the whole database.
This is the central idea of the
Search Work Ratio, the precursor of the Sequential Search Work Ratio, both deﬁned below.
Deﬁnition 3.2.
A relevance oracle is an oracle capable of computing if a document is relevant.
Deﬁnition 3.3.
The relevance oracle cost , noted C ( N, K, k ) , is the expected number of oracle’s calls needed to ﬁnd k relevant documents within a set containing N documents where K are relevant, when doing a random search without replacement.
Lemma 3.1.
C ( N, K, k ) =
k ( N + 1) /
( K + 1) .
(Proof in appendix A.)
As mentioned previously, we do not intend for the search data structure to produce the ﬁnal results.
Instead, another entity (e.g. a program or a human) should reﬁne the retrieved documents.
In many applications, mainly when the relevance function is absolute, it is conceivable that this entity is nearly perfect or, at least, signiﬁcantly more precise than the SDS.
Consequently, we deﬁne the cost of ﬁnding k relevant documents in a set of size N containing K relevant documents as the relevance oracle cost C ( N, K, k ) .
As a generalization, we can weigh differently when the oracle receives a relevant document versus when it does not.
The weighting would consider a real reﬁnement entity’s potential errors and put different values to the false positives and false negatives.
For simplicity, in this work, we will not weight the calls to the oracle.
The Search Work Ratio (SWR) is the ratio between the work done using the SDS versus the work done without using the SDS.
Consequently, an SWR score of less than 1 implies that it is less costly to use the SDS and vice-versa.
Furthermore, the SWR has a simple interpretation.
For example, an
SWR of 1/2 implies that using the SDS reduces the cost by a factor of two.
Deﬁnition 3.4.
Let D ∈ D , R ∈ 2 D , k ∈ N , ω 0 ∈ R and δ 0 ∈ 2 D , then the Search Work Ratio is
[Equation]
where D is a database, R is the relevant documents in this database, k is the minimum number of documents we want to ﬁnd, ω 0 ∈ R + is the cost of searching with the SDS, and δ 0 is the candidates retrieved by the SDS.
The cost could be any complexity measure, e.g. time or space.
In this work, since we will work only with hash tables, ω 0 will be the number of hashes computed.
We assume that using the oracle has the same cost as computing a hash.
The SWR has a signiﬁcant ﬂaw; it requires that the SDS ﬁnd enough relevant documents.
Otherwise, it is not deﬁned.
We will now slightly generalize this deﬁnition using a relevance generator to avoid this issue.
It is not rare that an SDS can be slightly modiﬁed to produce a sequence of sets of candidates..
For example, an approximate tree or graph search often employs a limit of nodes in the exploration.
It is possible to modify those algorithms to generate candidates with an increasing number of nodes to explore.
Deﬁnition 3.5.
Let D ∈ D , R ∈ 2 D , k ∈ N , ω ∈ R N and δ ∈ (2 D ) N s.t.
T = min { t s.t. t ∈ N and | ∪ ti =
δ i ∩ R | ≥ k } exists, then the Sequential Search Work Ratio is
[Equation]
with H =
∪ T − 1 i =
δ i if T > 0 and H = ∅ otherwise.
The SSWR’s numerator corresponds to a random search with the relevance oracle on the last generated candidates set plus the cost of considering all previously generated candidates plus the amount of work for computing all candidates sets up to T . The SSWR uses the relevance oracle cost only over the last sets of candidates because the generator did not found enough relevant documents before generating the last sets of candidates.
Consequently, an exhaustive search with the oracle in the previous sets of candidates was performed before asking the generator to yield more candidates.
The SSWR account for this exhaustive search with the | H | term.
Finally, the sets of candidates are intended to be mutually exclusive because this will reduce the relevance oracle cost computed over the last sets of candidates and give a better SSWR.
However, it is not mandatory. Deﬁnition 3.6. The Search Data Structure Learning (SDSL) framework consists of minimizing the expected SSWR over generators (of sets) of candidates w.r.t.
database-query pairs. Formally, given a work function W : G D , q (cid:55)→ ( ω 0 , ω 1 , . . . )
∈ R N where q ∈ Q and G D is a generators w.r.t.
a database D ∈ D the goal of SDSL is to minimize, min G [Equation] D [Equation]
[Equation] q ))] . We minimize the expectation over all databases to ensure the generator’s quality even if the database changes, i.e. for dynamic databases. By letting the distribution over D to be deterministic, we fall into the framework with a static database.
E FFICIENT LEARNABLE BINARY ACCESS
This section describes a family of models to tackle SDSL tasks: the Efﬁcient Learnable Binary
Access (ELBA).
It consists of two parametric families of function F Q and F U (e.g. neural networks) called the queries and documents encoders, and a Multi-Bernoulli Search (MBS) data structure S that will be made explicit later.
Any function from F Q and F U must have their domain on Q and U , respectively, and their image in [0 , 1] n to be interpreted as the parameters of a Multi-Bernoulli 1 (in its canonical form).
Precisely, ELBA is speciﬁed by the following triplet ELBA =
( F Q , F U , S ) with F Q = { f θ : Q → [0 , 1] n | θ ∈ Θ Q } and F U =
{ f θ : U → [0 , 1] n | θ ∈ Θ U } .
Note that in the particular case where Q = U , it is possible to use the same function for the queries and the documents ( ELBA
=
( F , S ) ).
We call this the shared variant of ELBA.
As opposed to the unshared variant where the parametric families might be the same, but the parameters are free to be different (e.g. the same neural network but with different parameters).
Multi-Bernoulli Search (MBS) data structure is a key-value based data structure that implements insert and search .
This data structure uses M back-end key-value based data structures compatible with binary vectors keys.
The back-end data structures S 1 , S 2 , . .
. , S M , must also implement insert( S, key , value) where S ∈ { S 1 , S 2 , . .
. , S M } is the data structure into which we insert the value w.r.t the key .
Similarly, the back-end structures must implement search( S, key) , which must return the appropriate set of values.
While the key given to the back-end data structures are binary vectors, the key given the MBS must be the parameters of a Multi-Bernoulli distribution of dimension n , i.e.
[Equation]
For insertion, the MBS computes the M most probable outcomes of the Multi-Bernoulli (which might not be unique) and uses them as the keys for inserting in a back-end data structures.
For searching, the MBS computes the T most probable outcomes of the Multi-Bernoulli (which, again, might not be unique) and uses them as the keys for searching in each back-end data structures.
Consequently, the search performs TM back-end searches.
The pseudo-code is in the appendix
B.2.
Note that the insert method does not require T .
Consequently, we can make the insertions and then choose T. This fact makes possible the modiﬁcation of the search to generate candidates every time it searches in a back-end structure.
Finally, to conform with the SDSL’s framework, it is possible to generate candidates by yielding candidates each time we search in a back-end structure.
Computing efﬁciently the top-k most probable outcomes of a Multi-Bernoulli is not trivial.
In the appendix B, we describe how to do it.
Throughout this work, we will use the Hashing MultiBernoulli Search (HMBS), an implementation of the MBS that uses hash tables as its back-end data structures.
An example of how inserting, searching and generating can be found in the appendix B.3.
To implement ELBA, we need a means to select a function from each parametric family.
As the parametric families, we consider neural networks, which we aim to train with gradient descent.
Consequently, we need a loss function.
Thus in this section, we will describe the F-beta Loss, a novel loss function design to perform well with MBS.
We will restrain ourselves to absolute relevance function.
For this reason, we will use a dataset of the form { ( q i , d i , r i ) } Ni =
= ( q i , d i , M ( q i , d i )) }
Ni =1 .
The model will try to predict the Matching function M .
We will denote the matching prediction function (cid:99)
M θ .
Since the Multi-Bernoulli Search requires the (canonical) parameters of the Multi-Bernoulli representation of the query { π qi } ni =1 for search and of the document { π di } ni =1 for insert (with n the chosen number of Bernoulli in the Multi-Bernoulli random variables), it is primordial that the model provides such quantity in its computational pipeline.
Let f Q θ ( q )
=
π q and f U θ ( d ) =
π d be the parametric functions for the queries and the documents, both implemented with a neural network ending with a sigmoid.
Note that, depending on the case, the two neural networks might or might not share parameters.
Finally, since we want to create a synergy with Hashing Multi-Bernoulli Search, we need the bits to be all equal if and only if the matching prediction function is one.
Thus we deﬁne,
[Equation]
which is the probability of both Multi-Bernoulli random variables are equal according to the distributions { π q i } n i =1 and { π d i } n i =1 .
We deﬁne three essential quantities: the recall, the fallout, and the predicted matching marginal (pmm for short), respectively:
[Equation]
We can compute empirical averages to produce unbiased estimators of those quantities.
[Equation]
where I + and I − are the sets of indexes for query-document pairs in the dataset that match and do not match, respectively, and with q being the probability of having a matching pair.
It is also possible to derive an estimator for the precision p θ =
qr θ m θ , with ˆ p θ =
q ˆ r θ ˆ m θ .
However, it is biased.
For numerical stability and because the gradients w.r.t r θ , s θ and m θ are near zero when the training starts, we need to consider their logarithm for the loss function.
Maximizing precision gives a model capable of discriminating between positive and negative pairs, but it leaves the recall untouched, and having a high recall is vital for the HMBS to ﬁnd the relevant documents.
Furthermore, only maximizing the recall induce the model towards a constant Multi-Bernoulli distribution with zero entropy, i.e., independent of the input and where all the probabilities are near 0 or 1.
We tried maximizing the recall while minimizing the fallout, i.e. with a loss similar to max θ log( r θ ) − λ log( s θ ) .
However, we found it extremely hard to optimize — there was no sweet spot for lambda.
When the model’s recall was sufﬁcient, it was because it collapsed to a constant function.
Scheduling λ to alternate between a small value and a relatively high value has shown limited experimental success.
In the end, we were looking for a tradeoff between the precision and the recall.
The F-beta came naturally.
However, since our precision estimator is biased, it is simpler to reparameterize the standard F-beta with the pmm using p θ =
qr θ m θ given us,
[Equation]
For the above reasons, we considered the logarithm of the F-beta,
[Equation]
β [Equation] R [Equation]
However, if we replace the recall term by its estimator directly we will get the LogSumExp ( LSE ) w.r.t.
log (cid:99) M θ ( q i , d i ) for i ∈
I + .
i.e., log(ˆ r θ )
= LSE { log (cid:99) M θ ( q i , d i ) | i ∈
I + } which is known to act as a soft maximum (not to be confused with its gradient, the Softmax).
Doing this will yield a near-zero gradient for the matching pairs with the lowest predicted matching value.
It would be problematic since those pairs are the ones that need the most gradient.
[Equation]
with β ∈ R + and c = log(1 + β 2 )
+ log q . Note that it is simple to compute with numerical stability the logarithm of the sigmoid function.
Most, if not all, Machine Learning libraries natively deﬁne this function.
E XPERIMENTS AND RESULTS
We performed all experiments on a dataset build from MNIST (LeCun et al., 1998), which we call
NoisyMnist .
We do not intend NoisyMnist to be a challenging task but rather a tool to analyze the convergence properties and draw comparisons between each models’ qualities.
In this dataset, the document and query are MNIST images with value ranging from 0 to 1 with additive Gaussian noise (the value can consequently go below 0 and above 1).
The mean and std of the noise isand 0.2, respectively.
The relevance function of NoisyMnist is absolute, and we deﬁne the matching function as follows: a query match with a document if and only if their original MNIST image was the same before we added the noise.
In ﬁgure 1, there are 6 examples of queries-documents pairs.
For evaluation, We build a ﬁxed database with 10K different images from MNIST not accessible while training.
From those 10K, we randomly selected 1K to create the queries.
Finally, we added the noise on each image (the query and their corresponding document do not share the same noise).
In this evaluation database, there is a unique document relevant for each query.
It is one in ten thousand, making it a proper database for SDSL.
We considered two alternatives for the F-beta Loss and one alternative to the MBS.
For the F-beta
Loss, we selected the loss function of MIHash (Cakir et al., 2017) and the loss function of HashNet (Cao et al., 2017) because of their compatibility with with ELBA.
More speciﬁcally, they both produce quantities that can be interpreted as the parameters of a Multi-Bernoulli, and they both can be trivially generalized to the unshared case.
Furthermore, as an alternative to MBS, we choose the
Hamming Radius Search (HRS) described in appendix C.4.
Combining the three losses with the two data structures creates six Efﬁcient Learnable Binary Access models.
We deployed each model in both the shared and unshared categories, for a total of twelve scenarios.
Training was run 5 times for each model, and the top 5 sets of parameters w.r.t the SSWR were selected for a total of 25 sets of parameters for each of the twelve models.
All the values provided are the average of those 25 points.
Each training consisted of 100K batches of size 32, which was plenty for all models to converge.
At each 500 batches, we performed an evaluation giving ussets of parameters to select from.
We performed all evaluations using the same ﬁxed set of 10K validation documents and a corresponding ﬁxed set of 1K validation query described above.
Those documents and queries were never seen while training.
All networks are ResNet18 (He et al., 2016) adapted to MNIST, i.e. the ﬁrst convolution takes from one channel with no stride and the last linear layer output a vector of size 64 (for 64 bits).
The hyperparameters and training schedule can be found in the appendix C. Finally, we use halting for all six models as is describe below. [Table]
When using generators iterating over binary codes, such as HRS and MBS, it is crucial to halt the iteration before it ﬁnishes.
For example, for 64 bits, there are 2 64 possible hashes, which is certainly larger than any database.
In those cases, it does not make sense to compute every possible hash.
Halting is the mechanism that decides when to stop a relevance generator and produce the database’s remaining document as the ﬁnal candidates set.
Note that, when halting, the SSWR is always greater than one.
In all experiments with HRS, we used a halting of 2081.
It corresponds to generating every document with binary codes within a radius of two from the query’s code (forbits).
In this case, halting afterward is arbitrary since the next 41664 codes (distance of 3) come in no particular order.
Finally, in all experiments with HMBS, we use a halting of 5001, which corresponds to stopping when the amount of work done exceeds the expected amount of work without an SDS, i.e. C (10000 , 1 , 1) .
Table 1 shows that F-beta (from this work) outperforms MIHash and HashNet in both unshared and shared categories.
Noteworthily, MIHash and HashNet fail in the unshared categories.
While HashNet successfully produces binary codes with lower Hamming distance for positive than negative pairs, the distance is way too high to be used with hash tables.
MIHash, on the other hand, only push towards increasing the mutual information between the Hamming distance and whether or not the pair matches.
It implies that there is no (explicit) pressure towards having a small Hamming distance.
The synergy between the loss function and shared parameters is why MIHash and HashNet produce low Hamming distances for positive pairs for the shared problem.
The found solutions for the unshared problem are not available when the queries and documents networks are the same.
Parameter sharing acts as a colossal regularization.
Those poor results suggest that both MIHash and
HashNet are constrained to similarity search.
Making F-beta the superior loss function for ELBA with hash table based SDS.
On the other hand, HMBS’s results are far superior to those of HRS.
In the case of shared F-beta, they are 46 times better.
The only case for which it is not true is for shared HashNet.
However, both of those models are not competitive.
The fat tail of its Hamming distances distribution is at cause.
In our experiments, we noted that varying the halting did not seem to change the results drastically.
Additional ﬁgures provided in the appendix D might convey a more intuitive understanding of the contrast between the different scenarios.
C ONCLUSION AND FUTURE WORK
In this article, we proposed a novel and practical ﬁeld of Machine Learning called Search Data
Structure Learning, for which we propose a natural metric, the SSWR.
We inaugurated this ﬁeld with a new family of models, the Efﬁcient Learnable Binary Access, which we instantiated with the F-beta Loss and the MBS that outperformed multiple alternatives, reducing by at least a factor of two the SSWR.
We cannot overstate the importance of F-beta in this project.
The capacity to obtain convergence on discrete output without being caught in a local minimum is a powerful tool.
In NLP, the F-beta led us to several exciting breakthroughs when learning semantically rich discrete embedding for words.
The fact that studying SDSL led to breakthroughs in other domains is, for us, an attestation of its signiﬁcance.
Furthermore, we plan to extend the formalization to consider insertion, deletion, dependent queries, and dependent retrieved documents in future work.
Such generalization could be useful for tasks like dialogue modelling or question answering.
Also, in this work, the halting procedure was simplistic.
In the future, we are interested in working with models that can decide when to stop based on the query and the retrieved documents.
Finally, we are eager to work with non-hashing-based approaches, such as trees or graphs.
A T HE RELEVANCE ORACLE COST
Formally let S be a set of N documents containing K ≤ N relevant documents.
From this K elements we want to ﬁnd at least k ≤ K elements.
We want to compute how many calls in expectation are needed to ﬁnd those k elements if we sample from S uniformly without replacement.
We will denote this expectation C ( N, K, k ) .
Let G ( k ; N, K, n ) be the Hypergeometric Distribution with parameters N , K , n .
This distribution gives the probability that from a set with N documents from which K are relevant, we sample exactly k relevant documents in n uniform trials without replacement.
[Equation]
Let P ( n ; N, K, k ) be probability distribution, deﬁned below, with parameters N , K , k .
This gives the probability that from a set with N documents from which K are relevant, it takes n uniform trials without replacement to get precisely k relevant documents.
We have,
[Equation]
which is the probability that we have k − 1 relevant documents in n − 1 trials multiplied by the probability that we sample a relevant document in a set with N − ( n − 1) documents from which
K − ( k − 1) are relevant.
Finally, the expectation of P ( n ; N, K, k ) yields the wanted measure.
I.e. if X ∼ P ( · ; N, K, k ) then,
[Equation]
with the last equality shown below A.1.
Lemma A.1.
If X ∼ P ( · ; N, K, k ) then,
[Equation]
Proof.
with A ≥ a , B ≥ b and A ≥ B all natural numbers.
E
[Equation]
] =
[Equation]
=
[Equation]
remove zeros =
[Equation]
n
[Equation]
N
[Equation]
n
[Equation] [Equation]
[Equation]
1) N − ( n − 1) =
N − ( K − k ) (cid:88) n = k n (cid:0)
Kk − 1 (cid:1)(cid:0) N − K n − k (cid:1) (cid:0) Nn − 1 (cid:1)
K − ( k − 1)
N − ( n − 1) =
N − ( K − k ) (cid:88) n = k n k K − k +1 (cid:0)
Kk (cid:1)(cid:0)
N − K n − k (cid:1) n N − n +1 (cid:0) Nn (cid:1)
K − ( k − 1)
N − ( n − 1) with (cid:18)
A a −(cid:19) =
a
A − a +(cid:18)
A a (cid:19) =
N − ( K − k ) (cid:88) n = k k (cid:0)
Kk (cid:1)(cid:0)
N − K n − k (cid:1) (cid:0) Nn (cid:1) =
N − ( K − k ) (cid:88) n = k k (cid:0)
nk (cid:1)(cid:0)
N − n K − k (cid:1) (cid:0) NK (cid:1) with (cid:0) Bb (cid:1)(cid:0)
A − B a − b (cid:1)
[Equation]
n
[Equation]
a variant of Vandermonde’s identity
[Equation]
with (cid:18) A
+a +(cid:19)
=
A
[Equation]
a
[Equation]
(cid:18)
A a (cid:19)
[Equation]
k
B T OP - K MAXIMAL MULTI - BERNOULLI OUTCOMES
Let b i be n independent and non-identical Bernoulli random variables with probability π i to be one.
Let z i be the most probable outcome of b i and let p i be the probability that b i be the most probable outcome ( b i =
z i ) i.e. z i = (cid:26)
if π i > 1 /0 otherwise p i = (cid:26) π i if π i > 1 /1 − π
i otherwise note that p i is always greater or equal than 1 / 2 .
The theorem B.2 gives us the following result.
[Equation]
− p i )
This implies that ﬁnding the Top-K Minimal Subset Sum of A = { a 1 , a 2 , . . .
, a n } with a i = log( p i ) − log(1 − p i ) ≥ 0 will yield the Top-K Maximal Multi-Bernoulli Outcomes.
The Top-K
Minimal Subset Sum problem can be solve using standard programming techniques.
The pseudocode for the reduction is in the appendix B.2.
B. R EDUCTION TO T OP -K M INIMAL S UBSET S UM
Lemma B.1.
Let p i ∈
[0 , 1] , a i ∈ { 0 , 1 } and b i ∈ { 0 , 1 } , for i ∈ { 0 , 1 , . . . , n } then n (cid:89) i =
[Equation]
<
[Equation]
⇐⇒
[Equation]
Proof.
Let I +
= { i | i ∈ N , 1 ≤ i ≤ n, a i =
b i } and let I +
= { i | i ∈ N , 1 ≤ i ≤ n, a i (cid:54) =
b i } we have, n (cid:89) i
= p a i i (
1 − p i ) 1 − a i < n (cid:89) i =
p b i i (1 − p i )
1 − b i ⇐⇒ (cid:89) i ∈ I
+ p a
i i (1 − p i ) 1 − a i (cid:89) i ∈
I − p a
i i (1 − p i ) 1 − a i < (cid:89) i ∈ I
+
p b i i (
1 − p i ) 1 − b i (cid:89) i ∈
I − p
b i i (
1 − p i )
1 − b i ⇐⇒ (cid:89) i ∈
I − p a
i i (1 − p i ) 1 − a i < (cid:89) i ∈
I − p
b i i (
1 − p i )
1 − b i ⇐⇒ (cid:89) i ∈
I − p 1 − b i i (1 − p i )
b i < (cid:89) i ∈
I − p 1 − a
i
i (1 − p i ) a i since ∀ i ∈
I − , a i = 1 − b i ⇐⇒ (cid:89) i ∈ I
+ p 1 − b i
i (1 − p i )
b i (cid:89) i ∈
I − p 1 − b i i (1 − p i )
b i < (cid:89) i ∈ I + p 1 − a
i i (1 − p i ) a i (cid:89) i ∈
I − p 1 − a
i
i (1 − p i ) a i ⇐⇒ n (cid:89) i
= p 1 − b i i (1 − p i )
b i < n (cid:89) i
= p 1 − a
i
i (1 − p i ) a
i
Theorem B.2.
Let b be a Multi-Bernoulli of parameter π ∈ ]0 , 1[ n and let z i = (cid:26)
if π i > 1 /0 otherwise p i = (cid:26) π i if π i > 1 /1 − π
i otherwise then, for any x ∈ { 0 , 1 } n and y ∈ { 0 , 1 } n we have
P ( b = x ) > P ( b = y )
⇐⇒ n (cid:88) i =
1 s.t. x
i (cid:54) =
z i log( p i )
− log(1 − p i ) <
n (cid:88) i =1 s.t.
y i (cid:54) =
z i log( p i )
− log(1 − p i )
Proof.
For notational simplicity, let ¯ x i
= x i ⊕
z
i and let ¯ y i
=
y i ⊕
z
i (with ⊕ being the exclusive or)
P
[Equation] [Equation]
P
[Equation]
b = y )
⇐⇒
[Equation]
y i ) since the Bernoulli are independent ⇐⇒ n
[Equation]
n
[Equation]
⇐⇒ n
[Equation]
(1 − p
i
[Equation]
y i proved by cases ( π i > 1 / 2 and π i ≤ 1 / 2 )
⇐⇒ n (cid:89) i
= p
[Equation]
n (cid:89) i = p
[Equation]
(1 − p i ) 1 − ¯ y i by lemma B.
⇐⇒ n (cid:88) i
= ¯ x i log( p i )
+ (1 − ¯ x i ) log(
− p i ) < n (cid:88) i =
¯
y
i log( p i )
+ (1 − ¯ y i ) log(1 − p i )
⇐⇒ n (cid:88) i
= ¯ x i (log( p i ) − log(1 − p i ))
+ log(1 − p i ) < n (cid:88) i =
¯ y i (log( p i ) − log(1 − p i ))
+ log(1 − p i )
⇐⇒ n (cid:88) i
= ¯ x i (log( p i ) − log(1 − p i ))
< n (cid:88)
i =
¯ y i (log( p i ) − log(1 − p i ))
⇐⇒ n (cid:88) i =
1 s.t.
¯ x i = log( p i )
− log(1 − p i ) <
n (cid:88) i =1 s.t.
¯ y
i = log( p i )
− log(1 − p i )
Remark.
The case where π i is exactly one or zero for some i can be trivially taken into account by setting the bit to the only possible value.
All outcomes that are not generated will have probability zero.
B. A LGORITHMS
Require: Int n , Int M , SDSArray[M] S , RealArray[n] key , Object value outcomes ← Top-K Maximal Multi-Bernoulli Outcomes ( M, n, key) for i ∈ { 1 , . . .
, M } do insert( S i , outcomes i , value) end for
Require: Int n , Int M , Int T , SDSArray[M] S , RealArray[n] key values ← set() outcomes ← Top-K Maximal Multi-Bernoulli Outcomes ( T, n, key) for j ∈ { 1 , . . .
, T } do for i ∈ { 1 , . . .
, M } do values ← values ∪ search( S i , outcomes j ) end for end for return values
Require:
Int k , Int n , RealArray[ n ] π 1: z ← BinaryArray[ n ] 2: p ← RealArray[ n ] 3: a ← RealArray[ n ] 4: for i = 1 , . . . , n do 5: if π i > 1 / 2 then 6: z i ←7: p i ← π i 8: else 9: z i ←10: p i ← 1 − π i 11: end if 12: a i ← log( p i ) − log(1 − p i ) 13: end for 14: outcomes ← BinaryArray[ k × n ] 15: index ← TopKMinimalSubsetSumIndexes( k, n, a ) 16: for j = 1 , . . .
, k do 17: for i = 1 , . . .
, n do 18: if i ∈ indexes j then 19: outcomes ji ← 1 − z i 20: else 21: outcomes ji ← z
i 22: end if 23: end for 24: end for 25: return outcomes
B. HMBS EXAMPLE
In this example, the number of bits in the Multi-Bernoulli is n = 3 , the number of back-end data structures is M = 3 , and the number of search keys is T = 2 .
Since this is a Hashing MultiBernoulli Search data structure, the M = 3 back-end structures will be hash tables; we will call them H 1 , H 2 and H 3 .
Let say we want to insert a document with the following key, π 1 = (0 . 3 , 0 . 1 , 0 . We ﬁrst need to compute its M = 3 most probable outcomes. Here they are in order: (0 , 0 , 1) , (1 , 0 , 1) , (0 , 0 , 0) then we will insert the document in H 1 using the most probable outcome as the key, insert the document in H 2 with the second most probable outcome as the key and insert the document in Hwith the third most probable outcome as the key. Now, let say we have two other documents to insert with the following keys, respectively:
[Equation]
Here are their M = 3 most probable outcomes, respectively: (1 , 1 , 0) , (0 , 1 , 0) , (1 , 1 , 1) and (0 , 0 , 0) , (1 , 0 , 0) , (0 , 1 , 0) .
We will then insert both of them in the M = 3 hash tables like we did for the ﬁrst document.
To search in the HMBS given a query π = (0 . 1 , 0 . 6 , 0 . 2) , we ﬁrst need to compute its T = 2 most probable outcomes: (0 , 1 , 0) , (0 , 0 , 0) .
We will then search in the M =
3 hash tables with these T
= 2 , doing a total of TM = 6 search in the back-end structures.
These searches will ﬁnd all of the three documents since the ﬁrst document is in H 3 with the key (0 , 0 , 0) , the second document is in H 2 with the key (0 , 1 , 0) and the third document is both in H 1 with the key (0 , 0 , 0) and in H 3 with the key (0 , 1 , 0) .
To generate documents as in the SDSL’s framework, we do not need the parameter T .
However, we might want to halt before considering all possible outcomes of the query.
With the same query as above, say we want to generate document but halt after the ﬁfth hashes, i.e. after doing 5 searches in the back-end structures.
We will ﬁrst generate the most probable outcome (0 , 1 , 0) and search in H 1 ; however, we will ﬁnd nothing.
We will then try the same outcome in H 2 and ﬁnd the second document, which we will yield.
Then we will try the same outcome in H 3 and ﬁnd the third document, which we will yield.
After this, we will compute the second most probable outcome (0 , 0 , 0) and search in H 1 to ﬁnd the third document again; thus, we will do nothing.
After we will try the same outcome in H 2 and ﬁnd nothing.
Finally, we will halt since we computedhashes, ultimately never ﬁnding the ﬁrst documents.
Note that if there were multiple documents simultaneously, we would have yielded them together as a set.
For the F-beta model 4 we use ramping for the β hyperparameter using this equation, log 2 β i = (cid:26) i [Equation] 8 10 K −if i < 10 K −otherwise for each batch i = 0 , . . .
, 100 K .
C.
M
IHASH
MIHash Cakir et al.
(2017) is based on the mutual information
[Equation]
(cid:19) and a generalization of the Hamming distance,
[Equation]
for x and y in R n .
Note that if x and y are in { 0 , 1 } n , d (2 x − 1 , 2 y − 1) is their Hamming distance.
Lets use the above notation, i.e. let f Q θ ( q )
=
π q ∈ [0 , 1] n and f U θ ( d ) =
π d ∈ [0 , 1] n be the parametric query and document functions (in the original article, they uses the same function for the queries and the documents).
Let X and Y be two Multi-Bernoulli of dimension n with parameters π q and π d respectively.
Finally, with H = d (2 X − 1 , 2 Y − 1) , they aim to maximize,
I ( H, M ( q, d ))
To allow gradient descent, they use differentiable histogram binning, i.e. they approximate
P ( H = k | M ( q, d )=1) with
P ( H = k | M ( q, d )=1) =
| I + | (cid:88) i ∈ I
+ δ i,k with δ i,k = (cid:40)
d i − ( k − 1) if d i ∈ [ k − 1 , k ] ( k + 1) −
d
i if d i ∈
[ k, k
+ 1] otherwise and d
i =
d (2 π q i − 1 , 2 π d i − 1) .
Similarly they estimate P ( H = k | M ( q, d )=0) and P ( H = k ) to compute the mutual information.
[Equation]
HashNet Cao et al.
(2017) optimize an increasingly closer to discrete sequence of tasks to alleviate the challenge of solving a discrete task with differentiable methods.
It is possible to frame their approach entirely with sigmoids but it is is simpler to use the tanh function as in the original article.
Let Net Q θ ( q ) =
logits q ∈ R n
and Net U θ ( d ) =
logits d ∈ R n be the parametric query and document functions before activation.
Let g Q θ ( q )
=
tanh (cid:0) β Net Q θ ( q ) (cid:1) and g U θ ( d ) = tanh (cid:0) β
Net U θ ( d ) (cid:1) be the activated functions.
Note that,
[Equation]
For simplicity, let g Q i =
g Q θ ( q i ) and g U i =
g U θ ( d i ) .
In their work they modelize the matching random variable with,
[Equation]
This gives,
[Equation]
Finally, they train the model with the (weighted) negative log-likelihood.
[Equation]
with w i a positive real number useful when match and non-match are unbalanced.
In the following experiment we ignored this term since the task is way to unbalanced and adding a weighting term would break the loss function.
The β term, in the tanh, is ﬁrst set to 1 and then increased when a convergence criteria is obtain.
This process repeats ten times, creating a sequence of of increasingly harder optimization which, if repeated inﬁnitely, would converge to a discrete optimization.
[Equation]
In all experiments, we use α = 0 .
2 for the sigmoid to have enough signal in the range [ − 32 , 32] .
C. H AMMING RADIUS SEARCH
The Hamming Radius Search (HRS) is a naive approach to quickly ﬁnd documents indexed with a binary code which have low Hamming distance r (the radius) to a particular binary code (the query).
For insertion, we map each document’s binary code the the document using a hash table.
For searching, we compute all binary codes at distance than 0 (i.e. only the query) and use the hash table, then we do the same for distance up to r, the radius.
The number a binary codes to consider grow very quickly w.r.t to the radius.
For example, at radius 2 with 64 bits codes, the number of codes to consider is 2081.
[Equation]
and for the radius 3 we there is 43745 codes to consider.
Radius 3 would not make sense for a database of 10K elements as the last 41664 elements comes in no particular order.
This is why we consider for the following experiments 2081 hash before halting for Hamming Radius Search.
D S UPPLEMENTARY F IGURES
[Table]