zhiminy/EvalEng huggingface dataset:
Repository: erikbern/ann-benchmarks
Issue: Does the ann-benchmarks homepage have plans to update the data?
Root cause: Documentation Deficiency

Hello, the data on the ann-benchmarks homepage is already two years ago. I would like to ask if there is any plan to update the data in the near future?
Our Milvus database has been upgraded to the V2.* version, if the data will be updated, we are happy to update our code to the latest version.
Looking forward to your reply, thank you very much!

@erikbern: What do you think? I agree that it's about time to re-run the benchmark. I'll run the benchmark locally on a server before an official run to identify issues and see if we waste time with some implementations. 

Yeah agree. Let me do it! I'll use the latest code in the repo, so @wangting0128 feel free to update the Milvus code if you want me to use the latest. I can do some quick sanity check that we're not too far behind the latest version on other ones too. 

@erikbern @maumueller 
Really thank you  for your reply!
We will update our latest code before March 3rd, I wonder if we can keep up with the plan?

Hi, I still have a few questions I would like to ask.

1. What scenarios will the re-run benchmark mainly cover, in addition to those shown in [ann-benchmarks.com](http://ann-benchmarks.com/index.html)
2. Will new server deployment scenarios be added? Similar to distributed server, we are interested in this

Looking forward to your reply, thank you very much
@erikbern @maumueller 

I'm just planning to run the benchmarks as they are in a repo – you can take a look at algos.yml to see.

It's possible I'll run it twice with a few weeks in between – just so that there's an opportunity for people to fix any issues that come up in the first run.

Milvus updated version: https://github.com/erikbern/ann-benchmarks/pull/333
Please review and let me know if you need to modify it, thanks a lot.

Great! I'm planning to go through all algos in the next couple of weeks

It's really exciting that a new round of benchmarks is about to begin.
If there are any problems during the operation, please synchronize with us, and we will actively correct the problems.
We have also run the benchmark locally. If the first round of benchmark is over, is there a way for us to read the benchmark result?

I'll probably run two rounds with a few weeks in between, just in case there's any issues with the benchmarks.

pgvector is mentioned in the readme but not in any of the results, so I hope to see them in the next run.

Currently running it – will be updating the stats in the next few days!

Will it include updates to Pgvector?

I updated the charts in the repo yesterday but haven't updated the website yet. Planning to do that in a few days!

Bump to please update the charts. They are so much more readable when they are interactive graphs on the website and I can filter out algorithms to only the few in consideration

The website has the latest data now!
Repository: erikbern/ann-benchmarks
Issue: Add LOPQ
Root cause: Documentation Deficiency

https://github.com/yahoo/lopq

writeup from Flickr: http://code.flickr.net/2017/03/07/introducing-similarity-search-at-flickr/

Hi. I took a look into adding lopq (https://github.com/yahoo/lopq) , however currently it only supports python 2.7. I did some PR on the library but it seems it has no active maintainer for now. When someone merges my PR, I'll port it to Python 3.6 and add it to this benchmark. 

@erikbern  Is it possible to use the current python 2.7 version of lopq within the benchmark framework? 
Thanks.

@mohamed-ali That's no problem. In fact, faiss is still run on Python 2.7. Just see https://github.com/erikbern/ann-benchmarks/blob/master/install/Dockerfile.faiss. (Nothing has to be done on the algorithm wrapper put in `ann-benchmarks/algorithms/...`.) 

@maumueller Thanks for clarifying that (it saves me some time). I'll attempt to add LOPQ in here then. 
 

@erikbern  @maumueller  In readme, section **running** there is only an example on how to run the benchmark in a production setup (if that's correct it to say), however I don't see anything about running for local testing/development. It would be useful to have also an example on how to run when developing(integrating a new algorithm for instance). 

Regarding adding LOPQ, I made some progress, however, it seems there is no clear way to run the tests locally. For example, trying to use the docker `ann-benchmarks` doesn't work because it not accessible:

```
$ docker build -f install/Dockerfile.lopq -t ann-lopq . 
Sending build context to Docker daemon  16.07MB
Step 1/6 : FROM ann-benchmarks
pull access denied for ann-benchmarks, repository does not exist or may require 'docker login'
```

I can keep debugging, however, if you have quick guidelines on how to run locally that would save me some time. 
Thanks.  

Please use `python3 install.py --algorithm lopq` in the root. This script takes care of building the base image and `lopq`.

if my memory serves me right, i believe you can just do 

```
python install.py --algorithm lopq
python run.py --algorithm lopq
```

does that work? we can add it to the documentation
Repository: erikbern/ann-benchmarks
Issue: Ad: SISAP'23 Indexing challenge

Since ann-benchmarks is **the meeting place** for people working in approximate nearest neighbor search, I'd like to use the chance to advertise an indexing challenge that provides a different angle on ANN: https://sisap-challenges.github.io/ (Disclaimer: I provided some consultation on the challenge design) 

All solutions that we benchmarked in ann-benchmarks (with fine-tuning for scale) should be eligible for the track A challenge, but track B and track C are concerned with finding binary embeddings and indexing these embeddings. I hope that some of you find this an interesting perspective and consider participating in the challenge.
Repository: erikbern/ann-benchmarks
Issue: Which libraries support incremental item additions?

Thanks for putting this project together. Is there any way a table or small list could be included to describe which libraries support incremental item additions? I imagine for many this is an important criteria in real-time or near-real-time applications where new content is continuously added and needs to be indexed for look-ups.

i don't think any of them do, unfortunately :(

Last time I checked (many years ago), FLANN's C++ API supported insertion.  In theory it's also easy for all LSH-based methods to support insertion.  The Python API might not have such functionality exposed though.  


@aaalgo Yes from the FLANN website:

> (14 December 2012) Version 1.8.0 is out bringing incremental addition/removal of points to/from indexes

Though I'm still trying to figure out how to leverage this via Python.

@erikbern Do you have any recommendations for "online" NN-search from your experience? One approach I'm considering is to maintain an indexed lookup (e.g. Annoy) for the majority of my items as well as a brute-force lookup (e.g. sklearn) for the newest items. Then at some threshold, rebuild the index to include the newest points.

I don't have any great recommendations, but your idea makes sense though. You could even formalize it and have indexes that have size even powers of two, and any time you have two indexes of the same size, you merge them. That would give you log(n) amortized runtime, although worst case is linear.

@alexklibisz Both hnsw and sw-graph are built incrementally, and thus naturally support incremental item addition. Hovewer, nmslib does not have a python interface for that, only c++.

@erikbern @alexklibisz I think you are talking about Bentley-Saxe method. http://www.sciencedirect.com/science/article/pii/S030643791300104X 

@yurymalkov yeah i've never heard of it, but think that's what i meant. basically play 2048 with the indexes :)

For anyone interested, I built a project that supports the feature from my original question on top of Elasticsearch: https://github.com/alexklibisz/elastik-nearest-neighbors

@alexklibisz Cool
@erikbern There are now several algorithms that support updates: flann, hnswlib, faiss (batched updates). Probably, this should be noted in readme somehow.
Repository: erikbern/ann-benchmarks
Issue: Zero recall for angular datasets
Root cause: Algorithmic Error

Hi, it looks like recall is always zero for angular datasets like `random-xs-20-angular` and `glove-25-angular` after 399c708544acf169904d67085977022bc31d3d0d.

```sh
git checkout 399c708544acf169904d67085977022bc31d3d0d
export DATASET=random-xs-20-angular
export LIBRARY=annoy
python3 run.py --docker-tag ann-benchmarks-${LIBRARY} --max-n-algorithms 3 --runs 2 --dataset $DATASET --run-disabled --timeout 300
python3 plot.py --dataset $DATASET --output plot.png
```

huh weird, i re-ran the benchmarks after that fix and i didn't run into that. will check!

Explanation of issue is here: https://github.com/erikbern/ann-benchmarks/issues/363#issuecomment-1503160543

Fix: #372
Extra tests: #373 

Thanks, working great now!
Repository: erikbern/ann-benchmarks
Issue: relax the timeout setting to 24 hours for better qps-recall performance
Root cause: Architectural Constraint

@erikbern , would you please consider relaxing the timeout setting to 24 hours? We found that for some datasets, some algorithms (such as NGTqg and qsgNGT) cannot finish the index building stage within 2 hours, but when the timeout is set to 24 hours, they dould get very good qps-recall performance. Of course, these algorithms’ disadvantage in building time will be reflected in the Recall-build time performance. 24 hours of construction time is indeed a bit long, but for some offline construction applications, it is acceptable to trade construction time for qps-recall performance.

I don't think it's worth doing since

1. IMO this seems a bit impractical – if it takes 24h to build an index, it's only going to be useful in very limited applications, and I think it would create a misrepresentation when presenting the results
2. It would cause the benchmarks to take a lot longer to run. I was able to run all benchmarks in about 12h on an r6.16xlarge at a cost of about $50 (for one dataset). But if we increase the timeouts to 24h then I expect the total runtime of the benchmarks to go up to several days or maybe a week – it will be much harder to run everything and it might cost $500 instead of $50
3. Related to the previous point, there have been some misbehaving algorithms that crash into an infinite loop and just get stuck. So the 2h time limit makes sure we're not wasting time waiting for an algorithm that will never finish.

2h is obviously a completely arbitrary limit, but I think it's reasonable in practice – it keeps the cost of benchmarks below $100 per dataset, and it makes it practical to run in 6-12h on a big machine.

I understand your concern,thanks for your patient answer
Repository: erikbern/ann-benchmarks
Issue: Use of multiple CPUs for index building
Root cause: Unimplemented Feature Gap

Thank you for such excellent benchmarks. 
As for #374, I agree that the 2 h timeout is reasonable in practical use. However, I don't think that using only one CPU for building indices is reasonable in practical use. Would it be possible to use multiple CPUs for building them like [big-ann-benchmarks](https://github.com/harsha-simhadri/big-ann-benchmarks)?

Yeah I think this would be more practical, but the reason I don't know if it makes sense to add it is:
* It wouldn't improve the total time of all benchmarks – I've been running a lot of benchmarks in parallel (i.e. running 31 benchmarks in parallel on a VM with 31 physical cores)
* Since the queries are single-threaded, we would have to change the number of cores dynamically for a container. I don't think this is possible with Docker (could be wrong)

I second Erik's thoughts on this. I was in favor of such a setup as well, but I like the current way of parallelizing running the benchmark. We do it differently in the big-ann-benchmarks project because it is completely unfeasible to build single-threaded on billion-scale.

To add a thought on the docker core limitations: I think in most application we would be able to set/reset the number of threads used internally, but then all trust is put into the implementation and it might be difficult to audit. 

The question of threads keeps coming up so I might add a section in the readme to explain the rationale for single threaded benchmarks

Thank you for your answers. I understand why you want to finish the benchmark as soon as possible because it costs a lot of money. When I want the result of a benchmark with a longer timeout, I will try to run it myself and publish the result.
Repository: erikbern/ann-benchmarks
Issue: Thinking about restructuring the code a bit

I think it would be nice to keep each algorithm together. I'm thinking about doing something like this

`ann_benchmarks/algorithms/xyz/module.py` – the python code for each one
`ann_benchmarks/algorithms/xyz/Dockerfile` – installation
`ann_benchmarks/algorithms/xyz/config.yml` – what's currently in `algos.yml`, but broken up by algorithm

This would replace `install/` and `algos.yml` that we have today

lmk if you have any strong objections to this? I think it would simplify the structure a bit.

I like it. This would also let some algos simplify their Dockerfiles. Some algos currently create config files "in-line" in the dockerfile, via `cat`, `echo`, etc., which is pretty brittle, e.g.,  https://github.com/erikbern/ann-benchmarks/blob/main/install/Dockerfile.opensearchknn#L30-L37 Much cleaner to just keep that config as a file in the directory and `ADD` it into the docker image.

yeah good point, that would work too

Great idea. Took me a while, the first time, to figure out all the files I needed to edit.

I did a quick speed run of this refactor. A couple of thoughts I had:
1. Many algorithms use the same Dockerfile. `ann_benchmarks/algorithms/xyz/Dockerfile` does not seem intuitive in these cases. These cases do not account for the majority, and could be satisfied with the following options:
   1. No dockerfile in `ann_benchmarks/algorithms/xyz`, but have xyz's config.yml point to abc's dockerfile (however we want to remove this dependency between docker tag and config.yml).
   2. A simple one liner Dockerfile, `FROM ann-benchmarks-abc`. I haven't entirely thought through the subprocess building implications of this (i.e. if `FROM ann-benchmarks-abc` is not local, it will have a failed pull).
   3. Dockerfile duplication, will just required contributors to remember to update each, but I don't think this is a problem (a contributor changing a common dockerfile should exist as a standalone PR, anyhow).
2. Having a separate config.yml per algorithm (instead of a single algos.yml) requires N yaml file reads when parsing which algorithms to use. 

In sum, however, I think the refactor still makes sense. 

I can have a PR up soon for perusal.

I think it's probably fine if most `Dockerfile` files end up just being something like 
```
FROM ann-benchmarks
pip install xyz
```
Repository: erikbern/ann-benchmarks
Issue: Idea: Cohere Wikipedia Dataset
Root cause: Unimplemented Feature Gap

I believe the recently released [Cohere's Wikipedia Embedding Archives ](https://txt.cohere.com/embedding-archives-wikipedia/)could be a good addition to the benchmarks dataset.

It's note worth the multi language nature of the dataset.

<p><br></p>

Wikipedia | Number of vectors / embedded passages
-- | --
English | 35 million
German | 15 million
French | 13 million
Spanish | 10 million
Italian | 8 million
Japanese | 5 million
Arabic | 3 million
Chinese (Simplified) | 2 million
Korean | 1 million
Simple English | 486 Thousand
Hindi | 432 Thousand
**Total** | **94 Million**



Good idea! Do you want to add it to https://github.com/erikbern/ann-benchmarks/blob/main/ann_benchmarks/datasets.py (for English)? I'm about to run a new round of benchmarks so we could include that as one dataset.

I'm pretty new to this, so would probably take some time before getting it to work 😬 

I may give it a try next week, if nobody does it.


Ok no rush, I can also take a look at it. But you're very welcome to look at it too, if I don't have time to!
Repository: erikbern/ann-benchmarks
Issue: Why is `lastfm-64-dot` called `-dot`?
Root cause: Documentation Deficiency

From the path http://ann-benchmarks.com/lastfm-64-dot_10_angular.html it seems that this dataset is actually angular.
But the name indicates dot-product, which many of the algorithms don't natively support.

yeah this is very confusing – I think it's a mistake. https://github.com/erikbern/ann-benchmarks/blob/main/ann_benchmarks/datasets.py#L427 indicates it's angular (cosine) distance too.

Maybe let's remove this dataset from the benchmarks for now.

@benfred should be able to shed some light on this. 

The original intent was to test out inner-product distance (dot), not angular distance: https://github.com/erikbern/ann-benchmarks/pull/91 . 

IIRC, the rationale was that certain algorithms either didn't support IP distance - or didn't have good performance when applying transforms like https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/XboxInnerProduct.pdf to convert IP distance to a cosine space


I think it's nice to have a dataset for dot products. But I'll fix that after I'm done with this run.

Hi, just a follow-up to the old question. May I ask why lastfm-64-dot has 65 dimensions while there is a '64' in its name? Is there any transformation applied on this dataset? @maumueller @erikbern 

May I assume the dataset is transformed based on https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/XboxInnerProduct.pdf? (So it actually does Maximum-Inner-Product Search during Euclidean-based ANN Search)
Repository: erikbern/ann-benchmarks
Issue: Split up building and query parameters
Root cause: Unimplemented Feature Gap

Currently, we spend a lot of time basically building the same data structure over and over again. What we could do instead, is split up _build_ and _query_ parameters. For example, Annoy is currently configured by

```
    annoy:
      constructor: Annoy
      base-args: ["@metric"]
      run-groups:
        annoy:
          args: [[100, 200, 400], [100, 200, 400, 1000, 2000, 4000, 10000,
                20000, 40000, 100000, 200000, 400000]] # number of trees x number of points to gather
```

It would save a lot of time, if one were to separate query and build parameters in the following way: 
```
    annoy:
      constructor: Annoy
      base-args: ["@metric"]
      run-groups:
        annoy:
          build-args: [[100, 200, 400]] # number of trees 
          query-args: [[100, 200, 400, 1000, 2000, 4000, 10000,
                20000, 40000, 100000, 200000, 400000]] # number of points to gather during query
```
The idea would now be to build the data structure for each `build-args` group, and query it for all (combinations of) `query-args`. This should be rather straight-forward to do.

Should be labeled as "enhancement". 

i like the idea – actually thought about something similar too. would help with the slow nmslib index building time as well

This has been added, closing.
Repository: erikbern/ann-benchmarks
Issue: Feature / Idea: `pip install` for reuse/standardisation
Root cause: Unimplemented Feature Gap

## Overview
[erikbern/ann-benchmarks](https://github.com/erikbern/ann-benchmarks) is not only an invaluable open-source comparison of popular ANN method (and, I guess, now ANN databases), but it could also provides a solid framework for performance testing/reporting. 

## Motivation
I am looking to build performance testing/benchmarks into [pgvector](https://github.com/pgvector/pgvector) (see [Issue #16](https://github.com/pgvector/pgvector/issues/116)), which currently reports results to ann-benchmarks. It seems superfluous to reinvent the wheel (i.e. dataset process, test execution, metric computation, etc), when pgvector will always want to report to ann-benchmarks too. 

## Scope
The most important functionality for ann-benchmarks to be useful as an import are in `ann_benchmarks/*.py` and the base algorithm implementation [class](https://github.com/erikbern/ann-benchmarks/blob/main/ann_benchmarks/algorithms/base.py#L6). 
```shell
pip install ann-benchmarks

# For algorithms, all or specific
pip install ann-benchmarks[algorithms]
pip install ann-benchmarks[annoy]

# For datasets (may be preferable to just download and cache)
pip install ann-benchmarks[DEEP1B]
```
This may work better/well with the code restructuring mentioned in #383 .  

Most/all of these algorithms require custom binary stuff though, so that's why we use Docker. Using pip for particular algorithms would just get us 30% of the way there!

That being said, it probably makes sense to turn ann-benchmarks into a bit more of a library. There's a lot of refactoring worth doing in general. I'm hoping to find a bit more time but random stuff keeps coming in between (starting a startup etc). But definitely want to clean up a bunch of stuff and I'll see if we can try to turn it into a library slowly.

Good point, I didn't know of a use case for `pip install ann-benchmarks[algorithms]` off the top of my head. Maybe even `pip install ann-benchmarks[annoy]` is excessive. I think the library user could be responsible for both 1) starting the local binary/docker container, then 2) starting their ann-benchmarks pointing (either via docker id of URI) to their algorithm

I'm happy to help out with some of the refactoring. If i understand correctly, #383 is the main issue discussing future refactors?
Repository: erikbern/ann-benchmarks
Issue: python install.py syntax error
Root cause: Environment Incompatibility

I cloned the ann-benchmarks project on Friday and ran into the following issue when trying to install it.  I am not a python person, but it seems likely that there is a problem getting the arguments resulting in a syntax error during evaluation.

sudo python install.py
```
Traceback (most recent call last):
  File "install.py", line 7, in <module>
    from ann_benchmarks.main import positive_int
  File "/home/ec2-user/ann-benchmarks/ann_benchmarks/main.py", line 152
    logger.info(f"running only {args.docker_tag}")
                                                ^
SyntaxError: invalid syntax
```
python --version
```Python 3.10.6```

Ah. There were a few issues.  When using sudo, a different python version was in use by root.  When using the current used, ```python install.py``` I was getting some permissions problems.  ```newgrp docker``` unblocked those.
Repository: erikbern/ann-benchmarks
Issue: Benchmarking index "drift" as new vectors are added?
Root cause: Unimplemented Feature Gap

It's my understanding that as more data pts are vectorized after the initial index is built, many of these algorithms start to perform worse. In situations where you have a regularly updating dataset (e.g. Twitter) that you'd like to perform semantic search over, it would be nice to understand how these different algorithms stack up against each other in terms of index drift without having to entirely rebuild the index which could be prohibitively expensive.

The benchmark right now just adds all vectors and then build the index.

I agree it would be nice to have a benchmark that adds / queries sequentially. But that would change the nature of the benchmarks quite a bit. Maybe later!
Repository: erikbern/ann-benchmarks
Issue: Hamming distance graphs seem to be empty
Root cause: Algorithmic Error

https://ann-benchmarks.com/sift-256-hamming_10_hamming.html
<img width="905" alt="Screenshot 2023-05-26 at 9 36 48" src="https://github.com/erikbern/ann-benchmarks/assets/123553/00e8cd8b-619c-4220-a68f-fcaf554b3bcb">

Same concerns Recall / Build time, Recall / Index size, ..

The only populated graph is "Relative Error/Queries per second (1/s)".