zhiminy/EvalEng huggingface dataset:
Repository: erikbern/ann-benchmarks
Issue: Question about recall calculation
Root cause: Validation Gap

Hello,
Could you clarify please how recall is calculated in the case when an algorithm search returns 10 equal vectors each of which is the closest to the request? Is it possible that in such case the recall (from get_recall_values function?) will be 100%?

We look at the distances of the reported points. Every point that is within the distance (+ small epsilon) of the k-th nearest neighbor is considered a correct result.

Suppose for k=10 an algorithm returns 10 copies of the same point which is the closest to a query. Will the recall be 100% or 10% in this case?

Now I get what you mean. As far as I can see, we do not check whether an algorithm returns duplicated points. It would get 100% recall by just returning the nearest neighbor 10 times. 

OK, thanks. But now it seems this is a loophole for ANNS algorithms developers :). If someone fix it, the results may change a little.
Repository: erikbern/ann-benchmarks
Issue: Is it really possible to test milvus
Root cause: Documentation Deficiency

I could test pgvector and other db successfully, while could never connect to mivus. and I couldn't find the package pyknowhere in anywhere.(only package "knowhere" could be found)
Thanks for your help, I was so confuse TT 
![image](https://github.com/erikbern/ann-benchmarks/assets/32962590/2cbc0f0a-9b85-42f4-969d-af6492c272a2)
![image](https://github.com/erikbern/ann-benchmarks/assets/32962590/afdc649e-f062-4390-ad2c-f1bd6789ea98)


You might want to consider trying the latest code.

> You might want to consider trying the latest code.

I'd love to. Is it support to run locally? (adding the --local param) or just support to run in docker? @cococo2000 

> > You might want to consider trying the latest code.
> 
> I'd love to. Is it support to run locally? (adding the --local param) or just support to run in docker? @cococo2000

The current code does not support running locally; it only supports running in Docker. Additionally, the Milvus service also needs to be used in Docker.

Thanks for your quick reply @cococo2000 ! I have install milvus2.3.0-beta in docker， is the new code support this version to test?



> Thanks for your quick reply @cococo2000 ! I have install milvus2.3.0-beta in docker， is the new code support this version to test?

The code uses the latest version of [Milvus, v2.4.1](https://github.com/erikbern/ann-benchmarks/blob/1a171c552874aeb8e303e00c5482bb454f91974a/ann_benchmarks/algorithms/milvus/Dockerfile#L23), and uses Docker Compose to manage the services according to [Install Milvus Standalone with Docker Compose](https://milvus.io/docs/install_standalone-docker-compose.md). 
And Milvus service will be automatically started and stopped by the code.

Thanks a lot! I will try this @cococo2000
Repository: erikbern/ann-benchmarks
Issue: [Notice] "Not supported URL scheme http+docker" error with requests==2.32.x
Root cause: External Dependency Breakage

https://github.com/docker/docker-py/issues/3256

## Temporary Fix: Pin Requests Library to Version 2.31.0

Before the current issue was addressed, we need to temporarily point the requests library to version 2.31.0. This can be done by adding `requests==2.31.0` to [requirements.txt](https://github.com/erikbern/ann-benchmarks/blob/main/requirements.txt).
Repository: erikbern/ann-benchmarks
Issue: Not able to read Kosarak dataset in a vector format.
Root cause: Documentation Deficiency

When I'm reading .hdf5 file, getting train or test output as a number. For below example, output is 3954. For other dataset (Ex- fashionmnist, lastfm etc), I'm getting train, test data in a vector format for a given id. Could anyone help here to read the train, test data in vector dimension format for kosarak?


```
import h5py
import csv
file_path = "kosarak-jaccard.hdf5"
file_path_csv = "kosarak_train.csv"
with h5py.File(file_path, "r") as hdf_file:
    print(hdf_file['train'][100])

```
Repository: erikbern/ann-benchmarks
Issue: Not able to run some algorithms or plots
Root cause: Documentation Deficiency

Good afternoon, I have been trying to use the platform because of its multiple options but in some things I get errors in executions for which I cannot find explanations. You probably have something configured wrong or you are running parameters that don't make sense, but I found this site that usually answers questions. I appreciate any response in advance. When executing some algorithms I get an error: "Nothing to run", specifically it happened when executing _python3 run.py --algorithm balltree_. On the other hand, when plotting some combinations I also get the same error, for example, having executed _python3 run.py --dataset fashion-mnist-784-euclidean --algorithm annoy_ I try to see the corresponding plots with _python3 plot.py - -fashion-mnist-784-euclidean_. I have verified that the original data is in _data_ folder and that the results are in _results_ folder. As I said before, it is probably due to a silly mistake due to lack of knowledge of the platform.
Thank you so much.

Hi @alexlopexrodri8, have you ever solved this? I have the same problem. I'm running `python3 run.py --algorithm faiss` and it says "Nothing to run". However, if I run `python3 run.py` without specifying any parameter, it runs on faiss correctly, but only on the glove-100-angular dataset. Thanks.
Repository: erikbern/ann-benchmarks
Issue: Failed to connect to Elasticsearch
Root cause: Environment Incompatibility

when I run ann-benchmark for elasticsearch
`python run.py --dataset glove-100-angular -k 10 --runs 1  --algorithm elasticsearch`
![Snipaste_2024-05-26_16-42-33](https://github.com/erikbern/ann-benchmarks/assets/78813459/8f70b63f-edaf-4076-825f-8ae5e8cd444f)

but I cannot establish a connection 
I enter the Docker container where Elasticsearch is running using the exec command， and run lsof -i :9200 and lsof -i :9300,and there were no programs running on either of these two ports.


Hi, @ccq1 did you find any solution to this problem?


> Hi, @ccq1 did you find any solution to this problem?

no, I gave up that

Ah okay. Thanks for the update.

@smeesamnaqvi 
In my case, Increasing `wait_seconds` of `_wait_for_health_status` worked well

@MarsMan13 thanks, I will give it a go.
Repository: erikbern/ann-benchmarks
Issue: 【Why is the throughput of testing the same algorithm with raft ann benchmark one order of magnitude higher than using ann benchmark】
Root cause: Documentation Deficiency

Hello, Everyone! Recently, I tested the performance of hnswlib using [raft-ann-benchmark](https://github.com/rapidsai/raft/tree/branch-24.08) and ann-benchmarks, and found that under the same recall rate, the QPS of raft-ann-benchmark is about an order of magnitude higher than that of ann-benchmarks. Is this due to the overhead of Python calling the cpp library when calculating time in ann-benchmarks?

environment: x86_64,CentOS7,96core
raft-ann-benchmark command
`
python -m raft-ann-bench.run --force --dataset glove-100-inner --algorithms hnswlib -m throughput
`

ann-benchmarks command
`
python run.py --local --algorithm hnswlib --batch
`
Repository: erikbern/ann-benchmarks
Issue: Can we run profiler and export the profiler data while running benchmarks?

Hello,
I am trying to run profiler while running ann-benchmarks using the following code:
`from multiprocessing import freeze_support
import cProfile
import pstats
from ann_benchmarks.main import main
import yappi



if __name__ == "__main__":
    profiler = cProfile.Profile()
    profiler.enable()

    freeze_support()
    main()  # The function that runs the benchmark

    profiler.disable()
    stats = pstats.Stats(profiler).sort_stats('cumtime')
    stats.print_stats(10)  # Print top 10 results
    stats.dump_stats('profile_results.prof')
`
Then run` python run.py --dataset fashion-mnist-784-euclidean --algorithm milvus-ivfflat --runs 1`
I am able to get the profiler data. But i am not sure if this is the correct way to profile for Milvus. 
It would be great if someone who have already done profiling for Milvus database share their opinion.
Repository: erikbern/ann-benchmarks
Issue: Benchmark pgvectorscale
Root cause: Unimplemented Feature Gap

[pgvectorscale](https://github.com/timescale/pgvectorscale/) is a postgres extension that builds on top of pgvector to add a few capabilities to make it more performant. Their claims have been quite good: https://www.timescale.com/blog/pgvector-is-now-as-fast-as-pinecone-at-75-less-cost/ , however it would be good to have it run through a more neutral, unbiased benchmarking system to validate it's performance


https://github.com/erikbern/ann-benchmarks/pull/539, I added a PR pending review to include pgvectorscale

Hey, are there any updates on when this pr is expected to be done? Would really love to see the comparison with other methods 

> Hey, are there any updates on when this pr is expected to be done? Would really love to see the comparison with other methods

Still pending review... Would love to see results also. 

Hey, I see that pr was merged recently, but I don't see pgvecote scale on the plots. Do you know when are they going to be updated? Or am I missing it? 

Same - where can I see pgvectorscale performance plots?
Repository: erikbern/ann-benchmarks
Issue: How to run faiss on AMD
Root cause: Environment Incompatibility

We found that faiss container uses conda/mkl and couldnt force it to use AOCL.
We also found that on faiss standalone app compiled from sources and usses AOCL's libblis-mt , the performance jumps
about 400%, however we couldn't create a container which runs python to perform as good.

You will find that most implementations in this benchmark are targeting an Intel architecture, sometimes even requiring AVX512.

Thanks
Repository: erikbern/ann-benchmarks
Issue: Use angular datasets for inner product?
Root cause: Documentation Deficiency

# Question

Can I use the "_ground truth_" part of angular datasets for inner product, after normalizing the "_data_" part. If not, are there any datasets for inner product? I have found [music-100](https://github.com/stanis-morozov/ip-nsw), but it seems unavailable now :(

We are computing similarity by the inner product, so yes, you can do that for the datasets in this repo. 

The billion-scale ann benchmarks project contains additional datasets: https://github.com/harsha-simhadri/big-ann-benchmarks for inner product similarity.
Repository: erikbern/ann-benchmarks
Issue: Incorrect Recall Rate with SIFT 50M Dataset in Ann-Benchmarks using PGVector IVF
Root cause: Validation Gap

Dear Ann-Benchmarks team,

I am currently attempting to test an algorithm using the SIFT 50M dataset from the following link: http://corpus-texmex.irisa.fr/. I have downloaded the Base set and Query set from the provided FTP links:

- Base set: ftp://ftp.irisa.fr/local/texmex/corpus/bigann_base.bvecs.gz
- Query set: ftp://ftp.irisa.fr/local/texmex/corpus/bigann_query.bvec

I have extracted the first 50 million entries from the Base set and combined them into an HDF5 file following the structure outlined in datasets.py's write_output function. However, when performing the recall calculation with the PGVector IVF, I am encountering an issue where the recall rate seems incorrect.

Could you please assist me in identifying potential causes for this discrepancy? I have ensured that the data preprocessing and the recall calculation steps are followed as per the guidelines.

I would appreciate any guidance or insights you could provide to resolve this issue.

Best regards

@erikbern  I really need your help. Please

@HungryFour Please provide more details in which sense the recall seems wrong. I suppose it would be easiest to share the hdf5 files of a single result, the groundtruth nearest neighbors calculated by ann-benchmarks, and a script that shows your way of computing recall. 

Note that we compute recall not by comparing to the groundtruth labels, but rather look at the fraction of points that are within the k-NN distance of the groundtruth. 

Thanks for the reply, the issue has been fixed. After reading the bvecs in Python, the query was not converted to numpy.float32, which led to a deviation in the distance calculation
Repository: erikbern/ann-benchmarks
Issue: Variability in test size
Root cause: Documentation Deficiency

I’m trying to understand the variability in the Test Size of the datasets provided and if that has an impact on the QPS. The test sizes for [datasets](https://github.com/erikbern/ann-benchmarks?tab=readme-ov-file#data-sets) listed vary from 500 to 50,000. I had a few of questions:

- Is there any reason for the test size selection?
- Will smaller test size impact recall? 
- In case we iterate more than once on this dataset for querying, i.e. `run_count > 1`, databases might have cached the results from earlier runs. In which case the impact on smaller test size should be more pronounced.
Repository: erikbern/ann-benchmarks
Issue: Custome data for bench marking
Root cause: Documentation Deficiency

Hey @erikbern and ANN Benchmarking team!,

Thanks for adding this clean implementation.

i have two query -
1) not a easy way to run test with custom data, i have data in .pkl format.
2) i dont see oracle 23AI  added in engine. 

Could you provide help with these 2 ?

Hi @praweenkumar,

1. You will have to write some custom code to transform your dataset to the hdf5 file format that we are using. See the examples in https://github.com/erikbern/ann-benchmarks/blob/main/ann_benchmarks/datasets.py for inspiration on how to do that. 

2. Feel free to contribute the dataset yourself as a PR. We are primarily interested in datasets that add some diversity to the existing ones, so a plot with the performance curves of some of our implementations would be appreciated to judge that. 

Best,
Martin
Repository: erikbern/ann-benchmarks
Issue: Evaluate JVector
Root cause: Unimplemented Feature Gap

JVector is available [here](https://github.com/jbellis/jvector) and it would be awesome to have it in the benchmark comparisons. This library powers some of the famours vector databases out there such as but not limited to [DataStax Astra DB](https://db.new), [Apache Cassandra](https://cassandra.apache.org), [Upstash](https://upstash.com/blog/introducing-vector-database), etc.,

This would indeed be interesting, I'd be happy to see support for this. The only problem is that our setup assumes a Python wrapper to be available (or some client/server architecture), and I'm not sure how this would work for their Java-based solution.
Repository: erikbern/ann-benchmarks
Issue: Can you provide an available Ubuntu 22.04 basic image?
Root cause: Environment Incompatibility

As the title, I have many problems with the dependency of the Ubuntu basic image pulled by Docker? Can you provide a downloadable image that I can use directly?
```
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 dpkg-dev : Depends: bzip2 but it is not installable
            Recommends: gnupg but it is not installable or
                        gnupg2 but it is not installable
 gcc-7 : Depends: libgcc-7-dev (= 7.5.0-3ubuntu1~18.04) but it is not installable
 libc6-dev : Depends: libc6 (= 2.27-3ubuntu1.6) but 2.35-0ubuntu3.8 is to be installed
             Depends: libc-dev-bin (= 2.27-3ubuntu1.6) but it is not going to be installed
 libstdc++-7-dev : Depends: libgcc-7-dev (= 7.5.0-3ubuntu1~18.04) but it is not installable
 perl : Depends: perl-base (= 5.26.1-6ubuntu0.7) but 5.34.0-3ubuntu1.3 is to be installed
        Recommends: netbase but it is not going to be installed
 perl-base : Breaks: perl (< 5.34.0~) but 5.26.1-6ubuntu0.7 is to be installed
E: Error, pkgProblemResolver::Resolve generated breaks, this may be caused by held packages.
The command '/bin/sh -c apt-get install -y python python3-numpy python3-scipy python3-pip build-essential git' returned a non-zero code: 100
Traceback (most recent call last):
  File "/home/li/python3.10/bin/ann-benchmarks/install.py", line 39, in <module>
    subprocess.check_call(
  File "/home/li/python3.10/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'docker build          --rm -t ann-benchmarks -f ann_benchmarks/algorithms/base/Dockerfile .' returned non-zero exit status 100.
```

By the way, is the ubuntu 22.04 image used in your tests x86 architecture? My test machine is aarch64 platform.
Repository: erikbern/ann-benchmarks
Issue: Minor Enhancements for Plot Selection
Root cause: Unimplemented Feature Gap

I like to suggest the following enhancements

1) Select/Unselect All Plots
Add an option to easily select or unselect all plots at once. This will allow users to start with all plots unselected and then add only the specific charts they need.

2) Improved Indicator for Disabled Plots
Currently, disabled plot names are struck through, which makes them harder to read. I instead suggest greying them out or use another visual indicator that maintains readability.



Also, selection/removal of plots should apply to all plots. Currently, the selection needs to be repeated for all the plots.
Repository: erikbern/ann-benchmarks
Issue: "Nothing to run" when specifying algorithm parameter
Root cause: Validation Gap

Hi, whenever I try to specify an algorithm parameter I get the following error:

```
$ python3 run.py --algorithm faiss_hnsw --dataset mnist-784-euclidean
2024-11-13 13:13:19,992 - annb - INFO - running only faiss_hnsw
Traceback (most recent call last):
  File "/home/test/ann-benchmarks/run.py", line 7, in <module>
    main()
  File "/home/test/ann-benchmarks/ann_benchmarks/main.py", line 344, in main
    raise Exception("Nothing to run")
Exception: Nothing to run
$ python3 run.py --algorithm faiss_hnsw
2024-11-13 13:13:26,071 - annb - INFO - running only faiss_hnsw
Traceback (most recent call last):
  File "/home/test/ann-benchmarks/run.py", line 7, in <module>
    main()
  File "/home/test/ann-benchmarks/ann_benchmarks/main.py", line 344, in main
    raise Exception("Nothing to run")
Exception: Nothing to run
```

However, if I disable the algorithms I don't want to test in the `config.yml` files and then I just run `$ python3 run.py --dataset mnist-784-euclidean` or `$ python3 run.py`, that works. So I think there's a bug; I just thought it may be useful to report it. 

Thanks @Hillier98, I can reproduce this and will look into it. 

I am facing the same issue.

any updates please

a gentle reminder to fix

Sorry, this took me a while. The reason for it not working was more subtle and boiled down to a difference between the folder name and the config name. Since one implementation (e.g., faiss) might support different data structures, these are sometimes only loosely connected. 

`--list-algorithms` with now identify the correct algorithm name, which was `hnsw(faiss)` in your example.
Repository: erikbern/ann-benchmarks
Issue: Illegal instruction (core dumped) when importing vespa_ann_benchmark in Vespa container
Root cause: Environment Incompatibility

Firstly, the specification of my host system is as follows:

OS: RockyLinux 7
CPU: Intel(R) Xeon(R) CPU E5-2690


When I run the following command:

`python3 run.py --dataset nytimes-256-angular --algorithm 'hnsw(vespa)' --parallelism 1 --runs 3`
An error occurs.

Upon debugging, I discovered that the issue arises inside the Vespa container when attempting to import vespa_ann_benchmark (at `/ann_benchmarks/vespa/module.py:1`)
Specifically, the error message is: `Illegal instruction (core dumped)`

Could you provide guidance on resolving this issue?

Thank you!

It seems that this requires a CPU that supports AVX-512. This is not in our hands, you will have to ask the vespa team directly.

@maumueller 
Thank you for your helpful answer!
Repository: erikbern/ann-benchmarks
Issue: 【question】Regarding the update of the latest assessment results issue
Root cause: Unimplemented Feature Gap

1. I saw from the official website that the latest results available are from 2023. I would like to ask if there are any plans to update the latest evaluation results?
2. If we want to submit an algorithm and update the latest results display, what do we need to provide?

Yeah I'm planning to re-run all results, let me find time in the next few weeks!

What is the update frequency for the subsequent plans?
Repository: erikbern/ann-benchmarks
Issue: RuntimeWarning: invalid value encountered in divide when create nytimes-256-angular dataset
Root cause: Validation Gap

```
(base) jiangyinzuo@kiwi:/usr3/jiangyinzuo_data/ann-benchmarks$ python3 create_dataset.py --dataset nytimes-256-angular
building matrix...
normalizing matrix entries with tfidf...
reducing dimensionality...
Splitting 300000*None into train/test
train size: 290000 * 256
test size:  10000 * 256
/usr3/jiangyinzuo_data/ann-benchmarks/ann_benchmarks/algorithms/bruteforce/module.py:47: RuntimeWarning: invalid value encountered in divide
  X /= numpy.sqrt(lens)[..., numpy.newaxis]
0/10000...
/usr3/jiangyinzuo_data/ann-benchmarks/ann_benchmarks/distance.py:42: RuntimeWarning: invalid value encountered in scalar divide
  distance=lambda a, b: 1 - np.dot(a, b) / (norm(a) * norm(b)),
1000/10000...
2000/10000...
3000/10000...
4000/10000...
5000/10000...
6000/10000...
7000/10000...
8000/10000...
9000/10000...
```

- python version: 3.12
- numpy version: 1.26.3
Repository: erikbern/ann-benchmarks
Issue: HubNSW/FlatNav benchmark--new HNSW-like NNS library
Root cause: Unimplemented Feature Gap

Dear Ann-benchmark team, 

I am wondering whether the recently developed FlatNav (https://github.com/BlaiseMuhirwa/flatnav, paper here: https://arxiv.org/abs/2412.01940) or simple HubNSW, which use hubs in the dataset instead of hierarchy to guide navigation. According to my test, it is more accurate without losing speed and can be equally applied to datasets with more than 32 dimensions. They have python bindings so I think it is very easy to add to Ann-benchmark.

Let me know your thoughts for this.  


Best,
Jianshu
Repository: erikbern/ann-benchmarks
Issue: I think DEEP should be Euclidean distance?
Root cause: Configuration Error

Thank you for providing a common format and benchmark suite for many standard datasets.

# Issue:

I believe the original DEEP dataset is using Euclidean distance, not Angular as you have it.
Since, the vectors are l2-normalized, the two distances are highly correlated but not the same, so you might not notice immediately from QPS-Recall.

The only reason I am not certain and have a question mark in the title, is that based on #145, your download source is different and on another format from the following sources (.fvecs vs .ibin). 
## Sources:
I'm looking at big-ann-benchmarks regarding this issue, since the author of the original paper for DEEP is listed one of the organizers of the original '21 challenge (Artem Babenko). I've also consistently seen deep mentioned for euclidean distance on research papers, which makes sense as, to the best of my knowledge, that's more common for images, and IP/angular is more common for text data.
- "Yandex DEEP":  https://big-ann-benchmarks.com/neurips21.html 
- metric: https://github.com/harsha-simhadri/big-ann-benchmarks/blob/main/benchmark/datasets.py#L364 
- their source: https://github.com/harsha-simhadri/big-ann-benchmarks/blob/main/benchmark/datasets.py#L354
- your source from (#145) : https://disk.yandex.ru/d/11eDCm7Dsn9GA