zhiminy/EvalEng huggingface dataset:
Repository: erikbern/ann-benchmarks
Issue: Consider adding a sentence embedding dataset
Root cause: Unimplemented Feature Gap

In addition to word embedding models like GloVe, there are now text embedding models like BERT and Universal Sentence Encoder that work at the level of sentences. These embedding models take an entire sentence as input and output a single vector representation. Sentence embedding models have started to be used alongside kNN to power text search applications.

It would be great to have a dataset based on a sentence embedding model. A sentence vector dataset can show different properties from the set of GloVe word vectors, and kNN algorithms may show a difference in performance between the two.

As an example, a sentence vector dataset could be created by gathering a set of short questions from [Stack Exchange](https://archive.org/details/stackexchange), then running each question through an embedding model like [Universal Sentence Encoder](https://tfhub.dev/google/universal-sentence-encoder).

That makes sense. It would be great if there's already some kind of popular dataset that we can rely on – I don't want to make it too "arbitrary". If we have to build a custom pipeline then I'd love for it to be very trivial (like 10-20 lines at most to generate the data). Any thoughts?

> It would be great if there's already some kind of popular dataset that we can rely on – I don't want to make it too "arbitrary"

I was also wondering about this point -- I haven't come across a commonly used dataset of sentence embedding vectors and would be curious if others had ideas.

To add some context, the idea of using StackExchange was motivated by the fact that it has a friendly license, and that it has formed the basis for some research around question retrieval + duplicate question detection (including the dataset [CQADupStack](https://people.eng.unimelb.edu.au/tbaldwin/pubs/adcs2015.pdf)). In a [comparison of text embedding models](https://arxiv.org/abs/1806.06259), Universal Sentence Encoder showed the best performance on semantic similarity tasks. It's available in a pre-trained form and can be downloaded from [TensorFlow Hub](https://tfhub.dev/google/universal-sentence-encoder). (It's worth noting that text embedding is an active area, and new models/ versions seem to come out pretty often).

i'm definitely open to adding this provided it can use as many standard datasets and algorithms as possible, and it can be reasonably concise.

https://github.com/erikbern/ann-benchmarks/blob/master/ann_benchmarks/datasets.py is where all the datasets are defined so far. of course, most people can ignore that file and just download the hdf5 files.
Repository: erikbern/ann-benchmarks
Issue: Clone the latest version of NGT
Root cause: Configuration Error

[NGT](https://github.com/yahoojapan/NGT) was updated to improve the search performance especially for glove-25. When you run your benchmarks, please clone the latest version (v1.9.0) of NGT.

it pulls the latest version when it installs – https://github.com/erikbern/ann-benchmarks/blob/master/install/Dockerfile.ngt

My understanding is that when a git repository is cached in docker, it is not cloned, even if the repository is updated.

right but i rebuild all containers from scratch every time i run the benchmarks so it's not a problem

it's been a while since last time so it's probably time soon again
Repository: erikbern/ann-benchmarks
Issue: Sklearn pairwise? 
Root cause: Documentation Deficiency

Forgive me for my ignorance, but for a low throughput scenario (i.e. ~10-20 items per second) we used [pairwise cosine](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.pairwise.cosine_similarity.html) from sklearn which does not suffer from problems that approximate search libraries have (i.e. rebuilding index, adding to index, sharing the index to multiple nodes, etc etc).  

With this you can just minimize your vectors, maybe lower precision and it works decently. The major limitation - you cannot really go beyond 10k - 100k items in the database without search becoming unreasonably slow. The major advantage - you can just store your database as serialized numpy array and add items to it easily.

So my question is, this method was not included because of limited scaling / throughput? Combined with domain heuristics (i.e. you can a priori split your database into 10 chunks) this is a very practical solution.





Hi! Sorry for the super-late reply!

I think our bruteforce implementation here https://github.com/erikbern/ann-benchmarks/blob/master/ann_benchmarks/algorithms/bruteforce.py should basically provide identical performance.
Repository: erikbern/ann-benchmarks
Issue: performance on low dimensional datasets
Root cause: Algorithmic Error

I have tried this algorithm and flann on low dimensional dataset(3-d point cloud).
Here are 800000+ map points, and 300 query points, and find the query points' k neighbor with this algorithm and flann.
the setup of hnswlib is ef_construction = 40, M = 6.
neighbor number k = 10
the performance of two algorithm as follows:
index build time : hnswlib 2249.92 ms flann 207.523 ms
search time : hnswlib 1.3619 ms flann 0.5652 ms
and there are some difference between the search results of these two algorithm, the performance of hnswlib is worse than flann.
Is this reasonable? I'm looking forward to some advice.
here are my dataset and search result.
[dataset.tar.gz](https://github.com/erikbern/ann-benchmarks/files/4385743/dataset.tar.gz)
[flann_result.txt](https://github.com/erikbern/ann-benchmarks/files/4385744/flann_result.txt)
[hnswlib_result.txt](https://github.com/erikbern/ann-benchmarks/files/4385745/hnswlib_result.txt)




I think this is rather a question for the hnswlib guys (@yurymalkov?), but it seems more than likely to me that for 3 dimensions standard tree approaches are much more efficient. (I ran some experiments for a two-dimensional dataset back in the days and `kd` was by far the fastest implementation.) 

There are some experiments on low-dimensional datasets in the HNSW lib paper https://arxiv.org/ftp/arxiv/papers/1603/1603.09320.pdf, maybe there is a hint how to set parameters in there.





Thanks a lot for your reply! Could you give me some advice that which algorithms are better for low dimensional dataset.

might be interesting to add a very low dimensional dataset (3-10 dims) for benchmarks

As said earlier, please give the standard kd tree a try, it's `--algorithm kd` in the benchmark.

With regard to low-dimensional datasets: Would be cool, but I don't know of any standard dataset :-) 

one idea is to just take some free data set on world cities lat/lon and project it into 3d space. i have some code for this lying around already.

Thanks for your reply. I will try it.
Repository: erikbern/ann-benchmarks
Issue: Did this benchmark consider the GPU option ?
Root cause: Documentation Deficiency

Hi,

Since some libraries, like FAISS, have a GPU version, I'm not sure whether the test use any GPU hardware.

Thanks.

You can run it with GPU enabled, but the charts shown right are for CPU only

We report on runs for FAISS gpu implementations on SIFT at http://ann-benchmarks.com/ at the bottom ('batched queries'). 

> We report on runs for FAISS gpu implementations on SIFT at http://ann-benchmarks.com/ at the bottom ('batched queries').

Awesome, thanks for your work. 

@maumueller 
Where can I find the GPU hardware spec? 
Thanks.
Repository: erikbern/ann-benchmarks
Issue: Benchmarks for binary vectors?
Root cause: Unimplemented Feature Gap

[FAISS](https://github.com/facebookresearch/faiss/wiki/Binary-indexes) and I assume some other libraries include optimized algorithms for binary features (both exact like [Multi-Index Hashing](http://www.cs.toronto.edu/~norouzi/research/papers/multi_index_hashing.pdf) and approximate).

I was wondering if there are any plans to include such specializations? I assume SIFT features would be good candidates for data.

This already exists with two datasets at http://ann-benchmarks.com/ (Distance: Hamming). In particular, MIH is supported already.

Would be nice to have the FAISS binary variants, though. Feel free to add it via a PR. :-)

Gave it a shot today, but I think I want to clarify a couple of things. Is it OK to open a WIP PR to ask for help fixing a couple of potential issues?

Sorry for the late answer. Please submit a [WIP] PR, yes. 

> This already exists with two datasets at http://ann-benchmarks.com/ (Distance: Hamming)

@maumueller i couldn't find it when i looked there today! Were binary benchmarks excluded again?

Yes, I think the website currently reflects only the latest run of @erikbern. We could put it into the results manually, but this will mix up different versions of implementations. :-(

I can include it next time I run the benchmarks!

@erikbern that would be great, thanks! :+1: 

@erikbern Any progress on this?

Sorry, haven't had time to work on it. I hope to have plenty of free time in March

Thanks for the feedback @erikbern :) Great to here, that it's not forgotten!

Any new regarding this? Would appreciate having binary vectors benchmarks :)
Repository: erikbern/ann-benchmarks
Issue: Can't check in custom branch

To whom it may concern,
My name is Braden Riggs and I'm an Intern at GSI Technology. I maybe be interested in submitting a pull request with some modifications and improvements to the ann-benchmarks. Unfortunately when I try to submit a branch I get this error.


braden@apu2:~/ann-benchmarks$ git push origin BR/apu2_modifications
Username for 'https://github.com': Briggs599
Password for 'https://Briggs599@github.com':
remote: Permission to erikbern/ann-benchmarks.git denied to Briggs599.
fatal: unable to access 'https://github.com/erikbern/ann-benchmarks/': The requested URL returned error: 403

Thanks, Braden.

Hi Braden. 

That's by design :-) Please fork the repository, make your changes there, and then open a pull request with your changes so we can discuss them. 

You can find a detailed description of this workflow here: https://help.github.com/en/github/collaborating-with-issues-and-pull-requests

Best
Martin
Repository: erikbern/ann-benchmarks
Issue: Deep1b training size only ~10 Million
Root cause: Unimplemented Feature Gap

Just wanted to ask a quick question about the Deep1b data set. I've been interning with GSI Technology and we have been benchmarking algorithms on the Deep1b data set (see attached image for an example) and are interested in seeing how our hardware performs on very large data sets. I just wanted to ask why the Deep1b data set only has a training size of 9,990,000 instead of the full billion. I'm assuming its just so people don't accidentally try to benchmark such a large data set but I wanted to confirm with you guys first. 

Thanks for the quick response on my last question :),
Braden.

![deep-image-96-angular](https://user-images.githubusercontent.com/48164242/85183997-f23ea980-b242-11ea-8fa2-81d072ad4667.png)


We only expose the first 1% of the dataset for now. Maybe at some other stage we'll do the full billion!

I just wanted to follow up and ask if you have the full deep1b dataset all prepared and just haven't implemented it or if the 10-million subset is all that is prepared. We are interested in running the full dataset on some more powerful machines. Sorry to keep bothering you :). 

no we don't have it prepared

Okay, we are interested in doing some true billion-scale benchmarks, if you could point us towards the code for the dataset prep we would be happy to share the code if we make changes ect. Or if you just tailored the code of the 10M set we would appreciate that code as well and any guidance you could provide. If we get it working we would love to share it with your GitHub.

shouldn't be super hard to do using https://github.com/erikbern/ann-benchmarks/blob/master/ann_benchmarks/datasets.py#L197

you might have to rewrite it a bit so it doesn't brute force the solutions though and instead it just uses the provided ground truth!
Repository: erikbern/ann-benchmarks
Issue: Falconn not included?

Is there a specific reason why Falconn is not included?

https://github.com/falconn-lib/falconn


It is included – see https://github.com/erikbern/ann-benchmarks/blob/master/ann_benchmarks/__init__.py#L21

I just haven't run the benchmark in a while. Planning to re-run it in a couple of weeks


actually let me re-run it now. should be done in a day or so :)


Great, thanks!


added some intermediate results, will push more results in a day or two
Repository: erikbern/ann-benchmarks
Issue: Faiss-lsh

We've gotten faiss-ivf working on a really large dataset (54 million vecs). Is there a reason faiss-lsh is disabled in algos.yaml? Should we not even bother using it?

Sorry missed this issue. Looks like it happened in this PR: https://github.com/erikbern/ann-benchmarks/pull/74

I believe I did it because the performance was quite bad

All good ended up seeing that outcome reflected in our results. Cheers for getting back to me though!
Repository: erikbern/ann-benchmarks
Issue: Benchmark of libnabo
Root cause: Unimplemented Feature Gap

Hi,
Thank you for sharing this really cool work with the community.

I was wondering if you had plans of benchmarking [libnabo](https://github.com/ethz-asl/libnabo). This KNN library is widely used for associating data features for Simultaneous Localization And Mapping (SLAM) by many developers, and it's mentioned in papers about visual localization with inverted search indices by Google.

It would be super useful to gather performance insights as the ones you have extracted from other algorithms.

Best,
Yoshua Nava

I haven't seen that one but it looks cool! If you're interested in getting it added to ann-benchmarks, feel free to put together a pull request!
Repository: erikbern/ann-benchmarks
Issue: ScaNN-only rerun?
Root cause: Unimplemented Feature Gap

The ScaNN results currently shown on the website reflect configurations that were only tuned for Glove-100; its performance on Glove-25 and LastFM aren't really reflective of its true, properly tuned performance. I later tuned for these datasets in #172. Would it be possible to update ann-benchmarks.com to reflect these new tunings? I regret not taking more time in polishing the original submission, and I understand you plan on doing a full re-run in a few months, but in the mean time we'd prefer our Glove-25 and LastFM numbers to be more representative of our performance. I don't want every minor tweak to trigger a re-run, but 2-4x more QPS at the same recall is quite significant, and ScaNN trains quite fast so the whole thing should be doable in an hour.

I might give it a month? I think the Milvus team is also eager to submit an updated implementation (see #167)

Btw would love to also feature ScaNN in the Euclidean benchmarks

Ok, sounds good!
Repository: erikbern/ann-benchmarks
Issue: Question: Why ann-benchmarks use distance to compute recall rather than id ? 

I noticed that the procedure compute the distance between result vectors and query vector, and use these distances to compute recall value. However, the query results could contain vectors id, which can also be used to compute recall value. Why ann-benchmarks chose the former way?

Because data points can have the same distance to the query, so "the nearest neighbor" isn't necessarily well defined. (This is particularly easy to imagine for Hamming distance and Jaccard similarity.)

Thanks @maumueller , I see. I found that there are two dataset `distances` and `neighbors` in data file '*.hdf5', the shape of each dataset is (10000, 100). Is there consistent one-to-one match between those two dataset ?  For example, distance between first query vector and vector which id is `neighbors[0, 10]` is equal to `distances[0, 10]`

Yes, `neighbors[0][10]` is the point with `distances[0][10]` from the first query point.

I guess this can be closed?
Repository: erikbern/ann-benchmarks
Issue: Is it possible to plot recall vs. latency?
Root cause: Unimplemented Feature Gap

I want to specify latency as x-axis and recall as y-axis when `python plot.py --x-axis --y-axis`.

But I checked metrics.py, there are no metric of latency.
https://github.com/erikbern/ann-benchmarks/blob/af634f77d0f84a0f377ca079a2197dfe80d90a91/ann_benchmarks/plotting/metrics.py#L100-L152

Anybody has any advice?

Latency is 1/QPS so just use QPS. The reason I like QPS more is that it makes the metrics consistent in the sense that higher values are better.

What is the definition of latency in your case? Is it the average time it took to answer a query? In that case, it seems analogous to QPS. In any case, you could just add it to `metrics.py` and compute latency (say in milliseconds) from QPS by returning `1000/QPS`. 

Haha, I should keep my answers shorter :-P

> What is the definition of latency in your case? Is it the average time it took to answer a query? In that case, it seems analogous to QPS. In any case, you could just add it to `metrics.py` and compute latency (say in milliseconds) from QPS by returning `1000/QPS`.

OK I'll take your advice, add new metric to metrics.py. Thanks!
Repository: erikbern/ann-benchmarks
Issue: [Question] Are there any ANN methods for sparse data?

I'm wondering if there exist any ANN method suitable for datasets with high number of dimensions but where the dimensions are sparsely populated.

I can see that the typical suggestion is to do dimensionality reduction, which I might end up doing, but before I go that route I was just wondering whether there are any alternatives.

My specific dataset are multiple sequence aligments in bioinformatics. So my data are 3D (n x L x 21). where n is the number of sequences/data points (typically between 50k-10M), L is the length of the sequences (typically somewhere around 80-500) and 21 are the different types of amino acids that each sequence can have.

So in my case I flatten the data to 2d and get (n x 21L) dimensions. However only 1 out of every 21 entries will be nonzero in this case. I'm wondering whether I can use that to my advantage somehow when computing ANN?

You can use nmslib (https://github.com/nmslib/nmslib), it supports sparse distances.

I'm seeing speedups of around 2-3 times with that one, so that seems to work great (not sure how much of a speedup I should expect)

Thank you very much!

+1 for dimensionality reduction as you mentioned
Repository: erikbern/ann-benchmarks
Issue: What does the parameter 'False' mean?
Root cause: Documentation Deficiency

Check
https://github.com/erikbern/ann-benchmarks/blob/9511d6a7544564ab4c356115a4354fd456de9a8c/algos.yaml#L367-L369
Comparably there isn't 'False' in the following snippet.
https://github.com/erikbern/ann-benchmarks/blob/9511d6a7544564ab4c356115a4354fd456de9a8c/algos.yaml#L588-L589

Makes any difference?

@searchivarius probably knows.

@maumueller not sure. Which parameter is it mapped to?

I took a quick look and the missing `False` is actually a bug. 

The `False` will map to `query_param` which will tell the implementation that the query parameters are set later
https://github.com/erikbern/ann-benchmarks/blob/master/ann_benchmarks/algorithms/nmslib.py#L13, which seems to mainly affect the name that is given for the run.

It's actually a bit annoying that the query parameter is not included in the string representation. Because of that, you can't see which query parameter is used when you hover over the data points in the interactive plots, e.g., http://ann-benchmarks.com/glove-100-angular_10_angular.html. 

This seems to be true for most of the "top performing" algorithms :-)
Repository: erikbern/ann-benchmarks
Issue: Installation fails on Ubuntu 18.04
Root cause: External Dependency Breakage

Installing via python3 fails running installation.py, because the request 'https://raw.githubusercontent.com/milvus-io/docs/v0.6.0/assets/server_config.yaml'' returns 404. The repo structure of milvus-io/docs has changed, URL should be changed to 'https://raw.githubusercontent.com/milvus-io/docs/master/v0.6.0/assets/server_config.yaml'. Possibly that once this is fixed, some other URLs from the same repo might be broken.

ok, i think the milvus setup is pretty old and we should update it to the latest version

Apparently, after the milvus link was updated, it still fails on Ubuntu at pynndescent. I'll push a fix later.

I'm also thinking of updating `install.py` so that if a particular algo fails, the installation still continues and just logs which algos failed and which passed. In this way, we won't face a completely failed installation if just one or two package fails. what do you think?

I think that makes sense. Also the Milvus implementation right now is pretty bad. It's a really old version and it's a bit fragile how it's set up. I'm OK disabling/removing it for now.
Repository: erikbern/ann-benchmarks
Issue: Two issues while installing current master branch and their workarounds
Root cause: Environment Incompatibility

Thank you for the excellent package. The current master branch is having some issues installing. Here are the issues and their workarounds. I already saw a PR for `pynndescent`. I'm not sure if `milvus` workaround is ok. I'll report back if I encounter any other issue. Admin, pls close this issue once the master has been updated to handle these two. Thanks.

**Issue 1:**
`pynndescent` dockerfile requirements are outdated.

**Workaround:**
Update `install/Dockerfile.pynndescent` as below

```
FROM ann-benchmarks

RUN pip3 install 'numba==0.46' 'llvmlite==0.33.0' 'numpy==1.17' scikit-learn
RUN pip3 install pynndescent
RUN python3 -c 'import pynndescent'
```

**Issue 2:**
milvus `server_config.yaml` link is broken

**Workaround:**
Update install/Dockerfile.milvus as below

```
# Install Milvus
FROM milvusdb/milvus:0.6.0-cpu-d120719-2b40dd as milvus
RUN apt-get update
RUN apt-get install -y wget
RUN wget https://raw.githubusercontent.com/milvus-io/docs/master/v0.6.0/assets/server_config.yaml
RUN sed -i 's/cpu_cache_capacity: 16/cpu_cache_capacity: 4/' server_config.yaml  # otherwise my Docker blows up
RUN mv server_config.yaml /var/lib/milvus/conf/server_config.yaml

# Switch back to ANN-benchmarks base image and copy all files
FROM ann-benchmarks
COPY --from=milvus /var/lib/milvus /var/lib/milvus
ENV LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/var/lib/milvus/lib"
RUN apt-get update
RUN apt-get install -y libmysqlclient-dev

# Python client
RUN pip3 install pymilvus==0.2.7

# Fixing some version incompatibility thing
RUN pip3 install numpy==1.18 scipy==1.1.0 scikit-learn==0.21

# Dumb entrypoint thing that runs the daemon as well
RUN echo '#!/bin/bash' >> entrypoint.sh
RUN echo '/var/lib/milvus/bin/milvus_server -d -c /var/lib/milvus/conf/server_config.yaml -l /var/lib/milvus/conf/log_config.conf' >> entrypoint.sh
RUN echo 'sleep 5' >> entrypoint.sh
RUN echo 'python3 run_algorithm.py "$@"' >> entrypoint.sh
RUN chmod u+x entrypoint.sh
ENTRYPOINT ["/home/app/entrypoint.sh"]
```

Hi – this is great! Would you mind submitting a pull request so that we can update the code?

Submitted!
Repository: erikbern/ann-benchmarks
Issue: Consider HTML results output
Root cause: Documentation Deficiency

Several graphing libraries in python (e.g. Plotly, Vega) support outputting to interactive HTML that would make it easier for users to understand the data.

Also, you may want to consider making the output results consistent by displaying but graying out results from algorithms that either failed to return outputs or don't support a given data set. For example, `bruteforce-blas` and `scann` are not available on `sift-128-euclidean`.

Thanks for this tool!

`create_website.py` generates interactive plots. See the website for examples, e.g., http://ann-benchmarks.com/glove-100-angular_10_angular.html. 

Ah, so sorry!