zhiminy/EvalEng huggingface dataset:
Repository: erikbern/ann-benchmarks
Issue: how to use  --local arg ?
Root cause: Environment Incompatibility

when i try to use --local for below command:
sudo python3 run.py --dataset gist-960-euclidean --algorithm kgraph --force --runs 1 --count 10 --local
report errors:
xxxxxx:~/ann/ann-benchmarks$ sudo python3 run.py --dataset gist-960-euclidean --algorithm kgraph --force --runs 1 --count 10 --local
2021-09-15 04:58:17,994 - annb - INFO - running only kgraph
2021-09-15 04:58:1**7,994 - root - WARNING - ann_benchmarks.algorithms.kgraph.KGraph(['euclidean', {'reverse': -1}, Tru**e]): the module 'ann_benchmarks.algorithms.kgraph' could not be loaded; skipping
Traceback (most recent call last):
  File "run.py", line 6, in <module>
    main()
  File "/home/xxx/ann/ann-benchmarks/ann_benchmarks/main.py", line 234, in main
    raise Exception('Nothing to run')
Exception: Nothing to run
:: it seems script can't parse --local correctly


Does `sudo python3 -c 'import pykgraph'` work on your setup?

i tried this but it seems no this module:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'pykgraph'
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 72, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 12, in <module>
    import subprocess, tempfile, os.path, re, pwd, grp, os, time, io
  File "/home/lnic/ann/ann-benchmarks/ann_benchmarks/algorithms/subprocess.py", line 7, in <module>
    from ann_benchmarks.data import \
ModuleNotFoundError: No module named 'ann_benchmarks'

Original exception was:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'pykgraph'


i found a content in Dockerfile.kgraph. it seems we need install kgraph seperately
FROM ann-benchmarks

RUN apt-get update && apt-get install -y libboost-timer-dev libboost-chrono-dev libboost-program-options-dev libboost-system-dev libboost-python-dev
RUN git clone https://github.com/aaalgo/kgraph
RUN cd kgraph && python3 setup.py build && python3 setup.py install
RUN python3 -c 'import pykgraph'

Yes, before running locally you have to make sure that the libraries are actually accessible from your Python installation.
Repository: erikbern/ann-benchmarks
Issue: what's the physical memory limit ?
Root cause: Documentation Deficiency

hi,  do we have minimum memory limit for ann workload ?
  we run ann workload with "python3 run.py --dataset gist-960-euclidean --algorithm kgraph --force --runs 1 --count 10" and use cgroup to limit its physical memory 60% of peak memory. we found swapfile hang, ann thread becomes zombie, cpu is 100%. we must reboot our machine. 



For GIST I was experiencing that you need around 16 GB of RAM to carry out all the implementations.

we are doing experiment to test physical memory to swapfile ratio. we keep total memory equals peak memory. but we tune the ratio of physical memory and try to find the minimal physical memory demand with enough swapfile
Repository: erikbern/ann-benchmarks
Issue: question about metric of recall at K used on the paper

I saw recall at k in the ann-benchmarks paper. When I look at the formula, it looks like the same formula as precision at k, but it is expressed as recall at k. Is there any reason why it is expressed as recall rather than precision?

In our very particular setting, precision (#true positives/#returned elements) and recall (#true positives/#positives) are the same, since we ask the algorithms to always return exactly k nearest neighbors. So, #returned elements = #positives = k.  I think different communities are used to different words.

Thanks for the quick reply! I will close this issue. Thank you.

I came across this recall metric problem recently. I'm wondering what if k is larger than #positives for some queries, the recall defined in the paper can never be 1.0

@zhiqihuang I'm not sure what you mean by "some queries". As soon as there are more than k vectors in the dataset, you always have enough "positives"?
Repository: erikbern/ann-benchmarks
Issue: ScaNN Docker image build fails
Root cause: Environment Incompatibility

When trying to install the scann Dockerfile, it fails with this error:

```
Step 5/14 : RUN apt-get update && apt-get install -y bazel-3.4.1
 ---> Running in 7ea9c7b3145e
Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease
Err:3 https://storage.googleapis.com/bazel-apt stable InRelease
  403  Forbidden [IP: 199.36.153.4 443]
Get:4 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:5 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [614 kB]
Get:6 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Get:7 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [2372 kB]
Get:8 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1433 kB]
Reading package lists...
E: Failed to fetch https://storage.googleapis.com/bazel-apt/dists/stable/InRelease  403  Forbidden [IP: 199.36.153.4 443]
E: The repository 'https://storage.googleapis.com/bazel-apt stable InRelease' is not signed.
The command '/bin/sh -c apt-get update && apt-get install -y bazel-3.4.1' returned a non-zero code: 100


Install Status:
{'scann': 'fail'}
```

This seems to be an issue with Ubuntu PPA key(?) but looks like it's being installed correctly in the DockerFile by first updating keyring:

`RUN curl https://bazel.build/bazel-release.pub.gpg | apt-key add -` 

and then `RUN apt-get update && apt-get install -y bazel-3.4.1` 

Not sure if anyone else faced this issue?

Want to create a pull request with the fix? Thanks!

Sure. Would give it a try. Thanks!

@erikbern here you go.  https://github.com/erikbern/ann-benchmarks/pull/279. It's a pretty small fix. I tested it and running `python3 install --algorithm scann` returned 
```
Install Status:
{'scann': 'success'}
```
Repository: erikbern/ann-benchmarks
Issue: Which parameters were used for your published results?
Root cause: Documentation Deficiency

Hey! 

Is there a way to know which parameters were used for your published results? I noticed that in the algos.yaml some approaches like n2 have multiple parameter. Which one lead to the best results using SIFT-128 for example?

We just run all of them and then remove the results that aren't on the "efficient frontier". We don't track which parameter was the best – different parameter combinations may win at different parts of the tradeoff curve

@Gian-Alessio: You can read off this information from the interactive plots. See http://ann-benchmarks.com/glove-100-angular_10_angular.html as an example. However, over the data points shows the parameters used to obtain the recall/QPS tradeoff.
Repository: erikbern/ann-benchmarks
Issue: Building image for faiss algorithm fails
Root cause: Configuration Error

`python install.py --algorithm faiss` fails with the error:

> ERROR: Cannot uninstall 'PyYAML'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
> The command '/bin/sh -c python3 -m pip install -r requirements_py38.txt' returned a non-zero code: 1

When installing the requirements at line: [RUN python3 -m pip install -r requirements_py38.txt](https://github.com/erikbern/ann-benchmarks/blob/9966b9020eacd432b89f41a13f0c5d82dd6870e4/install/Dockerfile.faiss#L12)

It fails when building the faiss image. Tried also building it manually without install.py with same error. Assuming, this is due to the switch to conda. Probably has something to do with the following commit: 
https://github.com/erikbern/ann-benchmarks/commit/dfa2abc81bcc868013bcfce8e434a225f9ea709a#diff-d6599cc06b437f681f7fbce3207c0f5d9347cfea4d5b52ea81bb40e55fccdec0

Mixing conda and pip is tough :P

Thanks @carlryn (👋 fellow CPHer)

This happens because conda installs pyyaml==5.3.1 and we try to raise it to 5.4 a couple of commits ago. I'l take a look at it in #281 

Great!

I found https://github.com/harsha-simhadri/big-ann-benchmarks/blob/main/install/requirements_conda.txt
which at leasts builds the requirements and I can start executing. 
My faiss dockerfile became:
```
FROM ann-benchmarks

RUN apt update && apt install -y wget
RUN wget https://repo.anaconda.com/archive/Anaconda3-2020.11-Linux-x86_64.sh
RUN bash Anaconda3-2020.11-Linux-x86_64.sh -b

ENV PATH /root/anaconda3/bin:$PATH
RUN conda install -c pytorch faiss-cpu
COPY conda_requirements.txt .
# RUN conda install -c anaconda pandas
RUN python3 -m pip install -r conda_requirements.txt 
# conda doesn't like some of our packages, use pip
# https://developpaper.com/a-pit-of-mkl-library-in-linux-anaconda/
ENV LD_PRELOAD /root/anaconda3/lib/libmkl_core.so:/root/anaconda3/lib/libmkl_sequential.so 

RUN python3 -c 'import faiss; print(faiss.IndexFlatL2)'

```

But yea, it installs pyyaml 5.3.1






Yes, but I did this unpinned version to fix another problem. I think the fix in #281 is better, it explicitly installs the packages missing in conda.
Repository: erikbern/ann-benchmarks
Issue: Container exit code is not an integer, but a dictionary
Root cause: Interface Contract Mismatch

Hej,

Was executing the faiss-ivf and error after what seems to be a succesfull benchmarking run.

> 2021-12-02 15:45:57,143 - annb.0e88ef45e8 - ERROR - Container.wait for container 0e88ef45e8 failed with exception
> Traceback (most recent call last):
>   File "/home/ubuntu/git/ann-benchmarks/ann_benchmarks/runner.py", line 269, in run_docker
>     logger.error('Child process for container %s raised exception %d' % (container.short_id, exit_code))
> TypeError: %d format: a number is required, not dict

I changed the line providing the error message to:
`logger.error(f'Child process for container {container.short_id} raised exception {exit_code}')`

This gives the following output:

> 2021-12-02 15:54:36,931 - annb.a277890f6b - ERROR - Child process for container a277890f6b raised exception {'Error': None, 'StatusCode': 0}

Now, changing the line https://github.com/erikbern/ann-benchmarks/blob/b95dda65146156ee93c52e0265c8ee5b62bd11dd/ann_benchmarks/runner.py#L265
to: `if exit_code["StatusCode"] != 0:` fixes it.

nvm I can see a pull request that actually fixes this...
https://github.com/erikbern/ann-benchmarks/pull/278

Actually the pull request should be accepted. 

And how do I open this, it should not be closed until the pull request is merged in


I think it's a problem where different versions of Docker returns different things. If you fix it, can you handle both the dict case and the integer case? Thanks!

I'll fix it! :-)
Repository: erikbern/ann-benchmarks
Issue: Format of dataset, queries and output
Root cause: Documentation Deficiency

Hello,
I'm trying to include my own algorithm to benchmark it. However, I'm not sure in which format (type and shape) the dataset, single and batch queries are supplied to the algorithms. In which format should the algorithm return the result? As a list of the indices of the nearest neighbors?
Thanks for your help!

There's plenty of examples if you look under https://github.com/erikbern/ann-benchmarks/tree/master/ann_benchmarks/algorithms
Repository: erikbern/ann-benchmarks
Issue: Evaluate TorchPQ
Root cause: Unimplemented Feature Gap

Hi,

Thanks for the great work !

I found out TorchPQ recently : https://github.com/DeMoriarty/TorchPQ

> TorchPQ is a python library for Approximate Nearest Neighbor Search (ANNS) and Maximum Inner Product Search (MIPS) on GPU using Product Quantization (PQ) algorithm. TorchPQ is implemented mainly with PyTorch, with some extra CUDA kernels to accelerate clustering, indexing and searching.

It could be nice to be added to the list (I'm not the author).

Thanks!

interesting – yeah seems like it would be a good addition

Seems indeed interesting, but our current version doesn't work nicely with GPUs. It might be a bit difficult to get this included and requires some changes in our docker setup. 

If you have the resources @ljak, it would be great if you could give it a try. I'd be happy to help.
Repository: erikbern/ann-benchmarks
Issue: spatial search on large data set

I am seeking for pythonic libraries that could be used for spatial searching with good performance using large data set. I am very eager to know if this library can satisfy my needs in terms of execution runtimes and performance.

**_My problem is:_**

I have lots of spheres and want to find corresponding spheres in a distance of each sphere as fast as it is possible because I must call this function many times during my simulation. I have written a python code using numpy and scipy in [one of issues in stack overflow](https://stackoverflow.com/questions/71104627/how-could-i-speed-up-my-written-python-code-spheres-contact-detection-using-spa). But, as it is mentioned in that issue, it is not appropriate for large data set. Is there any prepared example code or any done time-consuming evaluations to investigate capability of this library on this subject? Does this library can perform this spatial searching for 100 nearest spheres on 500000 spheres less than a second or around it? How much time will be your estimation on this subject?

I would appreciate any explanation or for referring to any prepared code to evaluate runtime by this library.

pretty much all libraries in ann-benchmarks should handle this use case!

> pretty much all libraries in ann-benchmarks should handle this use case!

Thanks for the rapid response.

I have considered annoy and flann for test. I would be appreciated if you would suggest the other ones, based on your experiences, you think will be among the bests on this subject.

best regards
Repository: erikbern/ann-benchmarks
Issue: Evaluate Elasticsearch approximate kNN API
Root cause: Unimplemented Feature Gap

This was recently released in Elasticsearch 8.0+ and the API is documented [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/knn-search-api.html)

Elasticsearch is distinct in both implementation and product from OpenSearch (Amazon maintained fork), and I expect that the two APIs will have different results. Both are HNSW based.

Thanks for maintaining this repo! 

Feel free to submit a pull request!
Repository: erikbern/ann-benchmarks
Issue: Wrong x-axis label values (recall) in README

I just went through the ann-benchmarks [paper](https://arxiv.org/pdf/1807.05614.pdf). Now I am wondering whether the values on the x-axis in the README plots should be reversed, going from zero to one and not from one to zero. All figures in the paper (and on ann-benchmarks.com) have x-values going from zero to one with a high number of queries per second when recall is closer to zero.

Shouldn't the recall value be low when there are lots of queries per second and higher when there are less queries per second?

Hi Marc, 

I find the logit-scale a bit difficult to read as well, but they seem correct to me. The `x` values are going from 0 to 1, the values close to one are just written as 1-10^-1 (0.9),1-10^{-2} (0.99), ...  

Hi @maumueller,

sure that makes sense. Didn't see the `1-` there, thanks for pointing that out.
Guess this issue can now be seen as feedback, from what I can tell this scale isn't used elsewhere with ann-benchmarks which makes it confusing.
Repository: erikbern/ann-benchmarks
Issue: Relative error metric
Root cause: Documentation Deficiency

I was wondering if there is a way of defining what is a “reasonable” relative error, something similar to recall 0.9, 0.95, 0.99.
Furthermore, how much is this metric is effected by the distance metric?
If you have any experience with this metric I’ll be happy to here about it.
Thanks in advance,
Guy
Repository: erikbern/ann-benchmarks
Issue: category filter
Root cause: Unimplemented Feature Gap

Would it be in the spirit of this benchmarks to add a second benchmark category for ANN in conjunction with categorial filters?

Most real-world applications of ANN will required category filtering e.g. when searching for cloths in an e-commerce scenario via ANN one might filter by gender (categorial) or availability (categorial).

There are several software products which allow combining ANN and category filters e.g. Apache Solr, Elasticsearch, Vertex AI Matching Engine, weaviate, qdrant. However, they mainly differ to this benchmarks in that they are managed services or just services but not embeddable libraries.

In addition to recall vs. queries per second there should be a view which filters to a fraction of the date vs. recall vs. queries per second. For the proprietary managed services, also a cost dimension might be useful.

I have found the following blog articles covering the topic:
* https://blog.vasnetsov.com/posts/categorical-hnsw/
* https://towardsdatascience.com/effects-of-filtered-hnsw-searches-on-recall-and-latency-434becf8041c
* https://towardsdatascience.com/using-approximate-nearest-neighbor-search-in-real-world-applications-a75c351445d

Given that this would be very useful for practical implementations but also the fact that it significantly complicates the benchmarks I would be interested in your opinion and/or how I could help with it. Also I would be great to know if someone has already done such benchmarks.

I think that would be interesting! I think the downside is
1. Would make it more complex
2. Not sure if there's any obvious public datasets for this?

> I think that would be interesting! I think the downside is
> 
> 1. Would make it more complex

Fully agree, probably the key reason not to do it.

> 2. Not sure if there's any obvious public datasets for this?

Datasets from the existing benchmark could be reused if an additional artificial, categorial random variable is introduced, allowing to filter to fractions of the original dataset between 0-100%. The approach is described here: https://towardsdatascience.com/effects-of-filtered-hnsw-searches-on-recall-and-latency-434becf8041c


> if an additional artificial, categorial random variable is introduced, allowing to filter to fractions of the original dataset between 0-100%

I think that makes sense, but it would be nice if there's some more natural way to do it. Eg for the MNIST dataset, filtering by digit 0-9 could be nice.
Repository: erikbern/ann-benchmarks
Issue: Include Solr9 vector search in the benchmark
Root cause: Unimplemented Feature Gap

https://solr.apache.org/guide/solr/latest/query-guide/dense-vector-search.html

new HNSW implementation in Solr9 

cool, might be worth adding!

Hey @vedina is there a reason why the existing Lucene benchmark doesn't provide the necessary information?

I would image Solr (like Elasticsearch) has network communication overhead.

Lucene is nearing around a 1/5th-1/7th a millisecond per query served. At these latency numbers I would expect network serialization overhead to wash out any algorithmic improvements or changes.
Repository: erikbern/ann-benchmarks
Issue: Can we use ann-benchmarks to benchmark the All-Nearest-Neighbors problem?
Root cause: Unimplemented Feature Gap

The All-Nearest-Neighbors problem (All-NN) asks us to find all pairs of nearest neighbors in a set of n points in d-dimensions. Classically, there is a [deterministic O(n log n) algorithm](https://link.springer.com/content/pdf/10.1007/BF02187718.pdf), as well as [stochastic O(n sqrt(n))](http://sharif.edu/~zarrabi/papers/cccg-19/ann.pdf) algorithm.

I would like to test the All-NN performance of algorithms like Faiss. According to a google insider, the closed version of faiss does have a mode that solve All-NN. However, I don't know if the public version does.

I suspect it may, and I suspect that ann-benchmarks has a mode that can effectively test All-NN performance via its batch mode. However I don't understand the terminology well enough to determine of batch mode is equivalent to solving All-NN.  Can it?

@maxsu With some manual intervention, this is rather easy to support. (Just change `datasets.py` to not split up train/test but use the dataset for both.) Practically, the dataset files will become really large (because of the n x 1000 groundtruth data), and our brute-force approach should probably be replaced by faiss to compute the groundtruth quicker. From there, it's indeed just using batch mode. 

@Cecca and I did these changes at some point for [this paper](https://itu.dk/people/maau/additional/benchmarking-local-dimensionality-measures.pdf). The code is [available here](https://github.com/Cecca/role-of-dimensionality), but we didn't intend to merge something back.
Repository: erikbern/ann-benchmarks
Issue: More/better Algorithm implementation documentation

I see that elastic search and solr both use hnsw algorithms. I see that there are 4 hnsw algorithms on the website. I see that there are elastic search specific implementations in the code base, but I can't tell which of the 4 hnsw algorithms is used by either db. I also happen to know that SCANN is a google algrothim and annoy is from spotify, but those details are not on the site. 

It would be very helpful to be to know how algorithms relate to companies or database implementations. Perhaps a section could be added to the algorithm page citing the paper the algorithm was defined in and/or the library versions used to run tests?

There is a ton of configuration going into each algorithm, so it's not always trivial to summarize. I think the easiest is actually to just look in the configuration – in particular the [algos.yaml file](https://github.com/erikbern/ann-benchmarks/blob/master/algos.yaml). Hope that makes sense!