zhiminy/EvalEng huggingface dataset:
Repository: erikbern/ann-benchmarks
Issue: All the integrity checks failed to run...
Root cause: Environment Incompatibility

<img width="962" alt="Image" src="https://github.com/user-attachments/assets/a9998ab9-c046-4962-b697-1f8459dd99d6" />
@erikbern 

Yes, we test each different implementation and some implementations are usually run in CI instances that do not support the necessary CPU instructions. 

I can see that some fail for other reasons. Feel free to add a pull request that fixes some of these issues.

> Yes, we test each different implementation and some implementations are usually run in CI instances that do not support the necessary CPU instructions.
> 
> I can see that some fail for other reasons. Feel free to add a pull request that fixes some of these issues.

Hmm...I have no idea...how to fix it. Anyone in charge of this project now?
Repository: erikbern/ann-benchmarks
Issue: Unable to download the Glove-25 training, test and ground truth from the Download link provided
Root cause: Documentation Deficiency

Hello,

I am trying to download the Glove-25 training, test and ground truth in the HDF5 format from the 'Download' link provided in the github page.

Can you please point me in the right direction.

-Best,
Santhosh.
Repository: erikbern/ann-benchmarks
Issue: Why cosine distance is refered to as angular distance?
Root cause: Documentation Deficiency

Hi,

Thanks for the great project!

I noticed that in the [ann_benchmarks/distance.py](https://github.com/erikbern/ann-benchmarks/blob/main/ann_benchmarks/distance.py) file, there is a distance called angular that uses the formula for cosine distance.

![Image](https://github.com/user-attachments/assets/647e2bdc-573c-4568-ac09-f83fb86c752e)

Why is it called angular when it uses the cosine distance formula? To my knowledge, angular distance has a [different formula](https://en.wikipedia.org/wiki/Angular_distance).
Repository: erikbern/ann-benchmarks
Issue: Memory usage tracking is possibly not so accurate (especially with memory-mapped files)
Root cause: Algorithmic Error

The current implementation of `get_memory_usage()` in `BaseANN` uses `psutil.Process().memory_info().rss` to measure memory usage, which may not accurately represent the actual memory consumption when using memory-mapped indexing. We are testing our soon-to-be-released new algorithm, and `ann-benchmarks` showed the same memory usage for both In-Memory and On-Disk implementations, which prompted further investigation.

The RSS value includes both private and shared memory pages. So when using memory-mapped indexing, RSS value can report significantly high RAM footprint. However, the system can reclaim the memory space used by memory-mapped files, so the process does not truly own this memory, though it's included in RSS measurements. 

One possible fix would be to subtract shared space from RSS, or better yet, use USS (Unique Set Size) from `memory_full_info()`. This will be more accurate in both cases, in-memory or on-disk indexing. 

psutils man page: https://psutil.readthedocs.io/en/latest/#psutil.Process.memory_full_info

There is a blog post referenced in the psutils man page: https://gmpy.dev/blog/2016/real-process-memory-and-environ-in-python

From the blog: 

> "Determining how much memory a process really uses is not an easy matter. RSS (Resident Set Size), which is what most people usually rely on, is misleading because it includes both the memory which is unique to the process and the memory shared with other processes. What would be more interesting in terms of profiling is the memory which would be freed if the process was terminated right now. In the Linux world this is called USS (Unique Set Size), and this is the major feature which was introduced in psutil 4.0.0 (not only for Linux but also for Windows and OSX)."


Thank you for looking into this issue.
Repository: erikbern/ann-benchmarks
Issue: When the test results will be updated?

Hi,
Could you please inform me about the frequency and schedule for updating the test results?

I have some result in #582 ready to be published.

Could you please let us know the expected date of the next result update? We regret that we were unable to catch this update in time. We sincerely hope to integrate support for the openGauss database before that update. Thank you for your consideration. 

I don't have anything planned, but I might update it in a few weeks – there's a few changes I want to make
Repository: erikbern/ann-benchmarks
Issue: Missing configuration for Milvus (Knowhere) on GitHub repo
Root cause: Configuration Error

When running benchmarks locally, we noticed a discrepancy between how Milvus is represented on the ann-benchmarks website versus what's produced when running the benchmark code from the repository.

The ann-benchmarks website shows a single entry labeled `Milvus (Knowhere)` in the graphs, but when running the benchmark code locally, it produces multiple separate entries like `milvus-flat`, `milvus-ivf`, etc., instead of consolidating them under a single `Milvus (Knowhere)` label.

Is there a specific configuration needed to produce the consolidated representation shown on the website? The `config.yml` in the `algorithm/milvus` folder in the GitHub report does not have a `Knowhere` configuration. 

Any help on how to achieve the same consolidated representation would be appreciated. 



looks like there is a change of the Milvus algorithm. Milvus(Knowhere) is replaced by milvus standalone deployment
https://github.com/erikbern/ann-benchmarks/pull/517
Repository: erikbern/ann-benchmarks
Issue: Slow search queries when running pgvector with external database
Root cause: Environment Incompatibility

Hello everyone,
I am running ann-benchmarks using pgvector algorithm/extension on different datasets provided by this repository and with different parameters for the hnsw index. At first, I have tested it on my local machine, running the **run.py** script. But then, i have tested it on a external database instead of initializing and creating a postgreSQL database which runs on the docker container initialized by the script mentioned above.
I have enhanced some of the postgreSQL external database configurations, as you can see below (the comparison between the database created by the ann-benchmark script and my external database): 

<img width="695" alt="Image" src="https://github.com/user-attachments/assets/c91020ee-1391-4d28-b470-452fe460d457" />

What I am facing is that the _**qps**_ metric on my external database is way worse(100~ times slower) than the one I get when running the database locally, even though the external database configuration is more robust.
Any tips on what could it be, or rather, what configuration could i change to improve queries speed?



It might be a network-related issue.

Could you try running ann-benchmark on the server that hosts the external database directly, to see if the problem persists?

@SunnyBo0302 The problem was gone, it was a network issue indeed. 
Thank you
Repository: erikbern/ann-benchmarks
Issue: How to run algorithm with --local and --parallelism >1?
Root cause: Validation Gap

Hi, Erik
Here are several issues are seen when I run algorithm with --local and --parallelism > 1, could you please take a look?
1. Milvus is failed to create index with pymilvus.exceptions.MilvusException: <MilvusException: (code=65535, message=CreateIndex failed: at most one distinct index is allowed per field)>. And seems that the data are inserted several times.
2. Although log shows "Terminating 31 workers", the QPS for pgvector seems to be only for one process, not the QPS for all parallel processes, right?
Thanks.
J
# Running command:
python run.py --dataset fashion-mnist-784-euclidean --local --parallelism 31 --algorithm pgvector --force
python run.py --dataset fashion-mnist-784-euclidean --local --parallelism 31 --algorithm milvus-hnsw --force
Repository: erikbern/ann-benchmarks
Issue: Submitted new vector search algorithms to ann-benchmarks achieving 99.0% Recall@10 + 27K QPS with IP-protected Docker approach
Root cause: Others

## TL;DR
- 🏆 **99.0% Recall@10** + **27,857 QPS** achieved
- 📊 **Beat industry standards** by 10-40% across all metrics  
- 🔒 **IP protected** with Docker blackbox (no source code exposed)
- ✅ **Fully reproducible** via ann-benchmarks framework
- 🔗 **PR submitted**: https://github.com/erikbern/ann-benchmarks/pull/596

## What we built
Quark Platform algorithms (quark-hnsw, quark-ivf, quark-binary) that significantly outperform existing solutions:

| Algorithm | Recall@10 | QPS | Use Case |
|-----------|-----------|-----|----------|
| **Quark HNSW** | **99.0%** | 5,033 | High accuracy |
| **Quark IVF** | 70.5% | **27,857** | Ultra speed |
| **Balance** | **98.1%** | 6,119 | Most practical |

## Innovation: Docker Blackbox Approach
- ✅ Complete IP protection (compiled libraries only)
- ✅ Full reproducibility (anyone can test)
- ✅ Standard compliance (BaseANN interface)
- ✅ Community verification ready

## Technical Details
- **Dataset**: SIFT-1M (200K base, 2K queries)
- **Verification**: Independent brute-force ground truth
- **Environment**: CPU-only, conservative parameters
- **Libraries**: Both FAISS and hnswlib compared

## Call for Testing
Docker image ready for community testing:
```bash
docker pull quarkplatform/ann-benchmarks:v1.0.0
python -m ann_benchmarks --dataset sift-128-euclidean --algorithm quark-hnsw-high1
```

Curious about the community's thoughts on this approach!

contact: angelon000@gmail.com
Repository: erikbern/ann-benchmarks
Issue: Upgrade the Elasticsearch version
Root cause: External Dependency Breakage

The current version of Elasticsearch used in this benchmark is version 8.7.0. While there are many enhancements to functionality in more modern versions, there are specific improvements to low-level performance of vector comparison operations that greatly affect the performance of vector search - namely the use of SIMD to optimise the vector ops.

At time of writing, the most recent version is:
https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-9.0.2-linux-x86_64.tar.gz

Care should be taken in the docker file when customising the JVM options, since replacing it may drastically affect how Elasticsearch performs. It is best to add custom setting in `jvm.options.d` (rather than `jvm.options`).
https://www.elastic.co/docs/reference/elasticsearch/jvm-settings#set-jvm-options

I'm happy to help with this.
Repository: erikbern/ann-benchmarks
Issue: Upgrade the OpenSearch version
Root cause: Unimplemented Feature Gap

The current version of OpenSearch used in this benchmark is version 2.6.0.  
https://github.com/opensearch-project/OpenSearch/releases 
However, opensearch has been updated to 2.19. x and 3.1.x.

OpenSearch has made many optimizations in vector search in the past two years. Can we update the version to obtain more objective evaluation metrics. Thank you.

Additionally, JVM suggests adjusting to the upper limit of 31GB.This is really important for Elasticsearch and OpenSearch.
Repository: erikbern/ann-benchmarks
Issue: Can we just use "euclidean " instead of "angular" to gain some datasets ?

Hi authors,

It's very nice of you to notice this issue. My question is whether we can just use euclidean instead of angular to gain some trainsets and testsets ,such as glove and deep ?

Best wishes.(●'◡'●)

I'm confused what's the ask is? Can you clarify? Closing this for now
Repository: erikbern/ann-benchmarks
Issue: Specification of Milvus(Knowhere)
Root cause: Documentation Deficiency

I'm wondering about which index did Milvus(Knowhere) use. I didn't find any "Knowhere" configs in the code. Thanks!

looks like there is a change of the Milvus algorithm. Milvus(Knowhere) is replaced by milvus standalone deployment
https://github.com/erikbern/ann-benchmarks/pull/517
Repository: erikbern/ann-benchmarks
Issue: Are the dimensions correct for the jaccard-based datasets
Root cause: Documentation Deficiency

I noticed that datasets using Jaccard similarity were recently added, namely movie lens and kosarak. I downloaded the HDF5 and extracted the 'train' split for kosarak. However, the shape I get doesn't seem correct. Am I missing something? 

```python 
>>> kosarak = np.load("kosarak-jaccard.train.npy")
>>> kosarak.shape
(4167103,)
>>> 
```

This certainly is very different from what the readme says (74962 vectors each of dimensionality 27983). Would appreciate clarification on this.
Repository: erikbern/ann-benchmarks
Issue: Remove FALCONN from benchmarks
Root cause: Unimplemented Feature Gap

Please, remove FALCONN from the benchmarks. It's not properly tuned for the new settings (100 neighbors instead of 10 etc), and the results are confusing.

We might later add it back (the new, faster version), but meanwhile I don't want the results to be here.

Can you submit a pull request? Thanks

There is no way for me to do that without re-running everything, since it does not seem like the numerical results are available.

I will re-run the results in a couple of weeks, so feel free to make any changes (eg disabling FALCONN) until then. Unfortunately I also don't have the results available anymore
Repository: erikbern/ann-benchmarks
Issue: Incorrect handling of dimension=None default parameter of train_test_split() in datasets.py
Root cause: Validation Gap

Location:
datasets.py
train_test_split()
line 206

Current: dimension = dimension if not None else X.shape[1]
Correct: dimension = dimension if dimension is not None else X.shape[1]

Since None is always False, the current behavior will leave dimension as None, rather than changing it to X.shape[1] when dimension=None as I assume was intended.
Repository: erikbern/ann-benchmarks
Issue: Add switch and metadata to separate algorithm approach types
Root cause: Unimplemented Feature Gap

Algorithms are currently not grouped by their underlying approach, which makes comparisons between fundamentally different methods unreliable. For example, comparing hnswlib with Glass is misleading since one is graph-based and the other combines graph and quantization.

Possible Solution:
* Require each author to specify the underlying algorithm type when submitting a method (e.g., graph-based, quantized, tree-based, hashing, neural, hybrid, etc.).
* Add this classification as metadata to the benchmark configuration.
* Extend the website visualization with a selector to filter and compare results by approach type.

This would support more transparent evaluations, fairer comparisons, and clearer research interpretation.
Repository: erikbern/ann-benchmarks
Issue: Are numerical results available as well?
Root cause: Unimplemented Feature Gap

Right now, I can see only plots in the repo, which are pretty useless for the "external" comparison as such.

In particular, is HNSW still using hours for preprocessing? :) There does not seem to be a way to check now.

Information about build time and other metrics will be included on the website. (And the website should be available once #57 is done.) I think we should also make it possible to access the data that the plots are based on. What do you think, Erik? (I could host it.)

I think without the data, 99% of the benchmarks' value is lost.

You mean the data in `results/` right? Or do you mean the granular metrics (i.e. the stuff that's plotted just in numerical form)?

Can make that available although I'm a bit confused why the benchmarks are useless until then. Everything is containerized so it's very easy to re-run in case you want to run your own experiments

The granular metrics.

Ok, sounds good...as @maumueller pointed out they should be published with #57
Repository: erikbern/ann-benchmarks
Issue: disable nearpy, panns, dolphinnpy
Root cause: Configuration Error

They seem pretty bad. I don't mind keeping the code, but probably makes sense to disable them by default. They add a bunch of noise to the plots and also take a disproportionate amount of time during benchmarks

I would add faiss-lsh to this list as well. I've never seen it outperforming faiss-ivf. I second leaving the code in, we could just comment out their entries from `algos.yaml`. 

closing this in favor of https://github.com/erikbern/ann-benchmarks/pull/74
Repository: erikbern/ann-benchmarks
Issue: Batch queries and single-threaded mode
Root cause: Unimplemented Feature Gap

There are currently two relicts in the code which are not accessible since the arguments for them have been removed. What shall we do with them? 

1) Running an algorithm with `runner.py` has a `force_single` which avoids the thread-pooling mechanism altogether, independent of the `use_threads` method of an algorithm. I find it nice for checking whether the ThreadPool decreases performance. 

2) Running an algorithm also has support for batch mode and each algorithm already implements `batch_query`. However, only `faiss_gpu.py` currently implements an actual batch mode. However, HNSW and faiss implementations actually have good batch query implementations and they can easily be included to the benchmark in a different section.

All of this now seems to work. Closing it.
Repository: erikbern/ann-benchmarks
Issue: Zero vectors in nytimes-256-angular
Root cause: Validation Gap

I found that nytimes-256-angular and nytimes-16-angular datasets have hundreds of zero-valued vectors.
The problem is that the cosine distance to such vectors is undefined, making the search an ill-defined task.
Probably, such vectors should be removed. Another option is using unnormalized inner product distance instead of cosine.

I don't have super strong feelings about this, but think this is arguably a "feature" not a bug. Vectors that are zero should simply be ignored and never returned. We should never *query* for zero vectors though (there should be no zero vectors in the query set).

If it is a feature, it's probably a not very good design choice.
Cosine similarity for null vectors is fundamentally undefined, some discussion can be found here https://stackoverflow.com/questions/26699659/cosine-similarity-when-one-of-vectors-is-all-zeros . 
So one can expect general-purpose search algorithms to rightfully show different behavior on such data.


yeah i don't have super strong feelings, might make sense to remove them.

feel free to submit a pull request if you want to. otherwise i can do it but it might take longer.

Ok. I am busy now with another pull request to this repo (adding structure reusing to some algorithms). Maybe I'll do it later.

ok – sounds great! i'll keep an eye out for your other PR. and regarding removing null vectors i may or may not do it in the next couple of weeks

i'm eager to kick off another round of benchmarks very soon so do you think you can wrap up your changes in the next week or two?
Repository: erikbern/ann-benchmarks
Issue: New run May 2018
Root cause: Configuration Error

Have been running it on a c5.4xlarge for about 36h. It's still on glove-100-angular and it's mostly done:

![image](https://user-images.githubusercontent.com/1027979/40232816-58d44558-5a6e-11e8-8980-0a7fa04aa454.png)

Not sure what's up with NGT... think maybe it timed out and we should increase timeout a bit. Especially since we're now running many searches in the same container and CPU is limited to 1. I'll double it from 2h to 4h I think.

Hm...  It seems that kgraph and nmslib also didn't pass the time limit. I am not sure that 4h limit would be enough. Building a high quality nmslib hnsw index may take an hour on a 6-core machine. So probably it may take 5 hours on a single core.

I'll start with 4h and will increase it further if it's not enough. c5.4xlarge are pretty fast machines

Cool! 
Will there be a simple way to see what indexes failed to build in 4 hours?

yeah will get that for you later

The NGT construction was adjusted for many cores. I'll provide NGT for single core.

Here's the latest numbers:

![image](https://user-images.githubusercontent.com/1027979/40521895-e755494e-5f9b-11e8-8489-39baeecf2283.png)

This took almost a week to run... the problem is that there's so many algorithms now. The plot above is almost 600 different runs vs I think it used to be around 150. I just removed a lot of combinations from algos.yml and will rerun. I also decreased the number of trials from 3 to 2.

MRPT keeps crashing for glove-100-angular also, not a single combination of parameters succeeded

@erikbern Thanks! I would note that hnsw query time runs are pretty cheap (you can tell it just by looking at the graph), removing them should not save much time.


Does it make sense to keep both hnswlib and hnsw(nmslib) running? They seem to perform rather identical.

@erikbern You could also plot with `--raw` or use `create_website.py` with the `--scatter` option to get a better impression of which parameter configurations can be removed.

@maumueller hnsw(nmslib) and hnswlib are different implementations with somewhat different functionality (i.e. exotic spaces+slightly faster at query time vs incremental index, less memory footprint and less build time). 
I am actually not sure if they perform that similar on other datasets.

Np keeping both

Closing this in favor of #82 which I'm merging now
Repository: erikbern/ann-benchmarks
Issue: Noob question: OS you used and requirements
Root cause: Documentation Deficiency

Hello @erikbern and team,

First of all, let me congratulate you for your fine job, it is amazing :D 

I'm trying to run your benchmark for K = 1 (seems like performance changes). I'm using Ubuntu 16.04 and python3.5.

While I was installing (sudo python3 install.py) I noticed it installs lots of dependencies from bionic beaver. Is Bionic Beaver (18.04) or any other OS a requirement?


I actually ran this on Ubuntu 16... it should be working. When you say "lots of dependencies" what in particular are you referring to?

Yep, just finished running run.py with MNIST dataset and K=1 :) Seems ok, except for some red text on the console (include screenshot)
![image](https://user-images.githubusercontent.com/15041034/42704855-13247870-8697-11e8-9b7c-6508346a8ced.png)
... I have also saved all console output to a text file if you'd like to see it
[2018_0612_ann_bench_run_mnist.txt](https://github.com/erikbern/ann-benchmarks/files/2193616/2018_0612_ann_bench_run_mnist.txt)


I mean all the packages that install from docker (all the lines that appear at the terminal while running install.py)... I have no screenshot or txt for this.



weird, does it write results though? are you able to plot?

Yep, but my results are pretty similar to yours. I'm still waiting for the run on sift-128-euclidean to end, but this is my result for mnist-784-euclidean:
![test](https://user-images.githubusercontent.com/15041034/42709307-b7779278-86a5-11e8-89ec-9d6844804258.png)
and Y-Log version:
![mnist-count1-ylog](https://user-images.githubusercontent.com/15041034/42709311-b9506a34-86a5-11e8-851b-3fa6d6fef701.png)


ok cool!

seems like things are working so i'm going to close this!
Repository: erikbern/ann-benchmarks
Issue: Website Title Isn't Very Descriptive

Just a suggestion, I think having a more descriptive `<title>` tag on the website's homepage would be helpful. Right now, it's simply `ANN-Benchmarks`. It might be better as "ANN-Benchmarks: Approximate Nearest-Neighbors Algorithm Benchmarks" or something like that.

Nice suggestion @xaprb, I've updated the title.