zhiminy/EvalEng huggingface dataset:
Repository: erikbern/ann-benchmarks
Issue: Installation of Milvus
Root cause: Environment Incompatibility

When I try to build the image for Milvus, the following error occurs:

`Step 6/7 : RUN python3 -c 'import pyknowhere'
 ---> Running in e84180a73171
Illegal instruction (core dumped)
The command '/bin/sh -c python3 -c 'import pyknowhere'' returned a non-zero code: 132`

are you running on an x86?

Also encountered this problem.
```
Python 3.10.6
```
```
Linux version 3.10.0-1160.71.1.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC)) #1 SMP Tue Jun 28 15:37:28 UTC 2022
```




Also encountered this problem.
```
Python 3.10.6
```
```
Linux version 3.10.0-1160.71.1.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC)) #1 SMP Tue Jun 28 15:37:28 UTC 2022
```




It's supposed to be run on an AVX supported machine. 

@hhy3   It's supposed  AVX.
![image](https://github.com/erikbern/ann-benchmarks/assets/8122332/4c5a50f4-e6d4-4d1d-bf3d-61d0c97e9d5f)


> are you running on an x86?

yes

same issue here

I'm wondering if changing the base image from `milvusdb/knowhere-env` to `milvusdb/milvus` might work.
(I've only glanced at it, but `milvusdb/knowhere-env` seems quite internal and is known only by zilliz employees)

https://github.com/milvus-io/knowhere.  knowhere need to install

can anyone able to connect to Milvus server from ann benchmark client ? I don't see connection specification exist inside the module.py file ? 

> can anyone able to connect to Milvus server from ann benchmark client ? I don't see connection specification exist inside the module.py file ?

The benchmark does not evaluate Milvus as a complete engine, only [Knowhere](https://github.com/zilliztech/Knowhere) - a library written in C++ that acts as Milvus's internal core. So, there is no connection (see [ann_benchmarks/algorithms/milvus/module.py](https://github.com/erikbern/ann-benchmarks/blob/main/ann_benchmarks/algorithms/milvus/module.py))

It also requires avx-512 for the latest version. Environment Checklist for Milvus on Kubernetes Milvus documentation https://milvus.io/docs/prerequisite-helm.md

For cloud provider, you're gonna need Cascade Lake or newer series. Take AWS  as example. You'll need at least the 6th generation ec2s, like m6, r6.

it is supported in python 3.10

> When I try to build the image for Milvus, the following error occurs:
> 
> `Step 6/7 : RUN python3 -c 'import pyknowhere' ---> Running in e84180a73171 Illegal instruction (core dumped) The command '/bin/sh -c python3 -c 'import pyknowhere'' returned a non-zero code: 132`

Has this issue been resolved?

As @haorenfsa pointed out, it requires a machine supporting avx-512.
Repository: erikbern/ann-benchmarks
Issue: Run milvus raise
Root cause: Configuration Error

Hello, I got an error when I ran milvus.
![image](https://github.com/erikbern/ann-benchmarks/assets/8122332/f3fba6d3-0b74-4c4f-a4ab-2a7034772fa8)


--algorithm "Milvus(Knowhere)"  it's worked
Repository: erikbern/ann-benchmarks
Issue: What's the exact instance type for benchmarks?
Root cause: Documentation Deficiency

I'd like to reproduce some of the benchmarks. The readme says "r6.16xlarge", but there are a few flavors of r6: r6g, r6i, r6in, r6a (https://aws.amazon.com/ec2/instance-types/). @erikbern Do you remember precisely which one of these you used? Thanks!

Pretty sure it's r6i.x16xlarge – i stands for intel, since benchmarks aren't supported on ARM at this point
Repository: erikbern/ann-benchmarks
Issue: Program error when using --local flag
Root cause: Validation Gap

Hello,

I encountered a problem while using the --local feature. I ran the following command to test faiss-ivf:
```
python run.py --dataset random-s-100-euclidean --runs 1 --local --algorithm=faiss-ivf
```
However, the program threw the following error:
```
Traceback (most recent call last):
  File "run.py", line 7, in <module>
    main()
  File "/root/ann-benchmark/ann_benchmarks/main.py", line 185, in main
    definitions = [d for d in definitions if _test(d)]
  File "/root/ann-benchmark/ann_benchmarks/main.py", line 185, in <listcomp>
    definitions = [d for d in definitions if _test(d)]
  File "/root/ann-benchmark/ann_benchmarks/main.py", line 168, in _test
    raise Exception(
Exception: ann_benchmarks.algorithms.faiss.FaissIVF(['euclidean', 2048]): error: the module 'ann_benchmarks.algorithms.faiss' does not expose the named constructor
```
I found that the code is checking the validity of the constructor at this line:
https://github.com/erikbern/ann-benchmarks/blob/main/ann_benchmarks/definitions.py#L36
It imports the module using definition.module, but I noticed that the actual directory of the constructor is f"{definition.module}.module. Is this a bug? 
Thank you.
Repository: erikbern/ann-benchmarks
Issue: Random datasets too easy
Root cause: Unimplemented Feature Gap

The random blobs created from sklearn seem to be far too easy to solve, see screenshot below. I propose to bring back the old random datasets that use https://github.com/maumueller/random-inputs/. For me the easiest way would be to create them, put them on a website, add a function to `datasets.py` that fetches it and transforms it correctly. 

![image](https://user-images.githubusercontent.com/6311646/33171566-e48b0ec6-d04c-11e7-9afc-39f2afa75a06.png)


ok – i can also try to increase the number of blobs?

i think everything else equals, it's good to have a reproducible pipeline. if we end up using your algorithm that's fine, but is it hard to reimplement in python/numpy?

thanks for running this experiment btw!!
Repository: erikbern/ann-benchmarks
Issue: Standardize parameters across hnsw implementations
Root cause: Architectural Constraint

It looks like some of the hnsw implementations are run with very different parameters which makes it hard to compare apples to apples in the results.

I count seven hnsw implementations that should probably be run with similar parameters (the ones labeled hnsw, plus lucene and weaviate).  Without looking at the code I'm not sure if there's more.  

e.g. weaviate has M=8..72 and efConstruction=64..512, lucene has M=4..96 and efConstruction=500

@jbellis Our philosophy is that it's up to the authors contributing the implementation to fix the hyper-parameter settings.
Repository: erikbern/ann-benchmarks
Issue: Include number of vectors in the db for benchmarking
Root cause: Unimplemented Feature Gap

Hi @erikbern, We recently benchmarked the performance of two popular open-source databases: Qdrant and Postgres (Pgvector) using the ann-benchmarks repo. It became quite popular on [Twitter](https://twitter.com/NirantK/status/1674110063286571008?s=20). We think the number of vectors can in the DB is also an important factor when comparing performance. For example, have a look at our [results](https://docs.google.com/spreadsheets/d/1t2-tXID2LJCXdLv1JTPQaYhmMs6woOnK7W7nkEuDsUc)

I did the benchmark against  100k, 200k, ..., and 1M vectors in the DB but had to split the 1M dataset into separate datasets to make it work with the repo (See [this](https://github.com/KShivendu/ann-benchmarks/blob/openai-experiments/ann_benchmarks/datasets.py#L619)) and then I created a [script](https://github.com/KShivendu/ann-benchmarks/blob/openai-experiments/experiment.py#L25) to iterate through them and run them one by one. 

I feel these changes are a bit hacky and we can make them fit more naturally with the code so that it's reproducible and easier to work with. I'd love to hear your thoughts if you think there's a cleaner way to implement this.  Happy to raise a PR for the final design that we agree upon :)

Thanks! 

We have a few different datasets with different number of vectors so maybe that covers the variation to some extent? There's a lot of parameters that affect the performance – eg dimensionality, how the data is distributed, etc

Yeah indeed. So should I [create multiple splits from the same dbpedia-entities-1M dataset](https://github.com/KShivendu/ann-benchmarks/blob/openai-experiments/ann_benchmarks/datasets.py#L620) so that we have variations for 100k to 1M (each step being 100k)? I'll update the https://github.com/erikbern/ann-benchmarks/pull/434 PR accordingly. 

Updated the #434 PR to have multiple splits since the glove-angular dataset was already doing the [same](https://github.com/erikbern/ann-benchmarks/pull/434/files#diff-6a33a2ec7e432a5d84bba3fee0f59ec57fc44f3786a8fe1c75b6a3f617270ee2R499-R502)

Tbh instead of varying this parameter, I would rather just see a larger diversity of datasets. I think that would create more "independent" data points which seems useful.
Repository: erikbern/ann-benchmarks
Issue: How to build Docker image?
Root cause: Documentation Deficiency

How does one build docker image? If I try to run:

```
% docker build . -t ann-benchmarks
```

in `ann_benchmarks/algorithms/base` I get this error:

```
Dockerfile:9
--------------------
   7 |
   8 |     WORKDIR /home/app
   9 | >>> COPY requirements.txt run_algorithm.py ./
  10 |     RUN pip3 install -r requirements.txt
  11 |
--------------------
ERROR: failed to solve: failed to compute cache key: failed to calculate checksum of ref 4457b4d5-e62c-4a43-9663-bac9c67e36c8::v8jmvuy2c455r2fz9iowil5zp: "/requirements.txt": not found
```

you can run `python install.py`
Repository: erikbern/ann-benchmarks
Issue: Could not find a version that satisfies the requirement scann (from versions: none)
Root cause: Environment Incompatibility

I am trying to build a docker image for the `scann` algorithm like this on a **Mac**:

```
% docker build . -f ann_benchmarks/algorithms/scann/Dockerfile -t ann-benchmarks/scann
```

I understand the `scann` library only works on Linux (why?) - that's why I am using docker for mac which spins up a linux VM. i see this in docker output:

```
[+] Building 1.2s (6/7)                                           docker:desktop-linux
```

but then i get this error:

```
 > [3/4] RUN pip3 install scann:
0.389 ERROR: Could not find a version that satisfies the requirement scann (from versions: none)
0.389 ERROR: No matching distribution found for scann
------
Dockerfile:4
--------------------
   2 |
   3 |     RUN pip3 install --upgrade pip
   4 | >>> RUN pip3 install scann
   5 |     RUN python3 -c 'import scann'
   6 |
```
Repository: erikbern/ann-benchmarks
Issue: New Benchmarking Dataset Enquiry
Root cause: Unimplemented Feature Gap

Hey @erikbern and ANN Benchmarking team!

Thanks for creating and maintaining a clean and sophisticated repository for the community.

I am Anubhav, an NLP Researcher at Advanced Symbolics Inc. Me and my company work in the space of probabilistic sampling of social media data and mining people's opinions out of it (https://www.advancedsymbolics.com). 

I recently went through all the 13 benchmarking datasets included in the repository and discovered that although the datasets belong to a variety of categories (Computer Vision, Recommender System, Movie Ratings etc), there was a lack of datasets from the NLP Domain (only GloVe). GloVe embeddings have been pretty old and there are better and much more sophisticated textual embeddings present in the industry now.

I would like to ask you if you would allow me to contribute a benchmarking dataset of tweets ranging from 1M - 10M. These tweets would be probabilistically sampled and would represent an actual world text/NLP dataset. The test set would contain a set of "n" queries which would be search engine queries from users (for eg. "Top SUVs under $50K ?"). We would then use the latest language models to convert all of the data to embeddings. Finally, we can run the experiments for all the indices on this dataset as well.

**Why should we consider adding this dataset?**
There has been a recent boom of LLMs and vector databases, and many people refer to Ann-benchmarks for it. We also referred to ANN benchmarks to decide which index to choose. But sadly there was no up to date NLP use case included in the repo. 

Please let me know If this plan sounds good to you!

Thanks and Regards,
Anubhav Chhabra

Hey @erikbern @maumueller @ale-f any thoughts on this? If you would accept the contributions I would start working on them right away!

Hi @anubhav562. Could you please provide some results on the suggested dataset? I like the application, but I think it should add some diversity in terms of the performance that we see. 

Makes sense! I will work that out and post the results here. Thanks for your input @maumueller!
Repository: erikbern/ann-benchmarks
Issue: Data behind the plots as a CSV
Root cause: Unimplemented Feature Gap

It'd be useful to provide csvs in the results folder with the raw data in tabular form for each plot.  This would make it easier to modify the plots, e.g. to zero in on a subset of the data, etc.

@erikbern  I think that would be excellent: You can just run the `data_export.py` script and upload the resulting csv. In the same way, it would be great to have access to the actual raw data. 

I agree, making the raw data available would be great.
Repository: erikbern/ann-benchmarks
Issue: The `base` image fails to build
Root cause: Validation Gap

The output from docker:
```
Building base image...
[+] Building 1.1s (9/12)                                                                                          docker:desktop-linux
 => [internal] load .dockerignore                                                                                                 0.0s
 => => transferring context: 90B                                                                                                  0.0s
 => [internal] load build definition from Dockerfile                                                                              0.0s
 => => transferring dockerfile: 414B                                                                                              0.0s
 => [internal] load metadata for docker.io/library/ubuntu:22.04                                                                   1.0s
 => [1/8] FROM docker.io/library/ubuntu:22.04@sha256:0bced47fffa3361afa981854fcabcd4577cd43cebbb808cea2b1f33a3dd7f508             0.0s
 => [internal] load build context                                                                                                 0.0s
 => => transferring context: 147B                                                                                                 0.0s
 => CACHED [2/8] RUN apt-get update                                                                                               0.0s
 => CACHED [3/8] RUN apt-get install -y python3-numpy python3-scipy python3-pip build-essential git                               0.0s
 => CACHED [4/8] RUN pip3 install -U pip                                                                                          0.0s
 => ERROR [5/8] RUN python3 --version | grep 'Python 3.10.6'                                                                      0.1s
------
 > [5/8] RUN python3 --version | grep 'Python 3.10.6':
------
Dockerfile:6
--------------------
   4 |     RUN apt-get install -y python3-numpy python3-scipy python3-pip build-essential git
   5 |     RUN pip3 install -U pip
   6 | >>> RUN python3 --version | grep 'Python 3.10.6'
   7 |
   8 |     WORKDIR /home/app
--------------------
ERROR: failed to solve: process "/bin/sh -c python3 --version | grep 'Python 3.10.6'" did not complete successfully: exit code: 1
```

The actual output of `python3 --version` is `Python 3.10.12`.

weird, we can probably change it to 3.10. though – feel free to submit a pull request!

OK, I'll submit a PR for that pretty soon!
Repository: erikbern/ann-benchmarks
Issue: Query About Calculation of Recall!

Hey ANN Benchmarking Team!

I am deep inside the code and am trying to understand the metrics, recall to be precise. 

Can some one explain me what are dataset_distances, run_distances, count, threshold and epsilon in the below function. 


> def get_recall_values(dataset_distances, run_distances, count, threshold, epsilon=1e-3):
>     recalls = np.zeros(len(run_distances))
>     for i in range(len(run_distances)):
>         t = threshold(dataset_distances[i], count, epsilon)
>         actual = 0
>         for d in run_distances[i][:count]:
>             if d <= t:
>                 actual += 1
>         recalls[i] = actual
>     return (np.mean(recalls) / float(count), np.std(recalls) / float(count), recalls)

    
    
I have some inherent understanding that we are comparing the approximate neighbours with the actual neighbours at a particular threshold. It would be great if someone could explain in plain English how exactly we calculate the recall. 

I think it's better to go to the source code for the truth – I don't have detailed memory of how it works since it's been like 5 years. But we spent a lot of time on it so I'm pretty confident it works!
Repository: erikbern/ann-benchmarks
Issue: Some datasets are not available (403 Forbidden)
Root cause: External Dependency Breakage

The following datasets are not available, indicates `403 Forbidden`:
- Kosarak (http://ann-benchmarks.com/kosarak-jaccard.hdf5)
- MovieLens-10M (http://ann-benchmarks.com/movielens10m-jaccard.hdf5)
- Last.fm (http://ann-benchmarks.com/lastfm-64-dot.hdf5)

Also seeing this with the `dbpedia-openai-*k-angular` datasets, fwiw

Ping @erikbern

I'm not able to access those files, and it seems like the script to build the datasets have broken.

```
ubuntu@ip-172-31-3-226:~/ann-benchmarks$ python create_dataset.py --dataset kosarak-jaccard
Traceback (most recent call last):
  File "/home/ubuntu/ann-benchmarks/create_dataset.py", line 10, in <module>
    DATASETS[args.dataset](fn)
  File "/home/ubuntu/ann-benchmarks/ann_benchmarks/datasets.py", line 597, in <lambda>
    "kosarak-jaccard": lambda out_fn: kosarak(out_fn),
  File "/home/ubuntu/ann-benchmarks/ann_benchmarks/datasets.py", line 452, in kosarak
    X_train, X_test = train_test_split(numpy.array(X), test_size=500, dimension=dimension)
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (75462,) + inhomogeneous part.
```

Not sure how to fix, does anyone know?

Also not able to regenerate `lastfm`:

```
ubuntu@ip-172-31-3-226:~/ann-benchmarks$ python create_dataset.py --dataset lastfm-64-dot
Traceback (most recent call last):
  File "/home/ubuntu/ann-benchmarks/create_dataset.py", line 10, in <module>
    DATASETS[args.dataset](fn)
  File "/home/ubuntu/ann-benchmarks/ann_benchmarks/datasets.py", line 595, in <lambda>
    "lastfm-64-dot": lambda out_fn: lastfm(out_fn, 64),
  File "/home/ubuntu/ann-benchmarks/ann_benchmarks/datasets.py", line 486, in lastfm
    from implicit.approximate_als import augment_inner_product_matrix
ImportError: cannot import name 'augment_inner_product_matrix' from 'implicit.approximate_als' (/home/ubuntu/.local/lib/python3.10/site-packages/implicit/approximate_als.py)
```

Do you know what's the issue @benfred?

NVM I figured out how to fix the original permission issue.

However we should probably fix the issue with the datasets – it would be good to have a way to regenerate them from scratch
Repository: erikbern/ann-benchmarks
Issue: Add FLASH
Root cause: Unimplemented Feature Gap

https://github.com/RUSH-LAB/Flash

https://arxiv.org/pdf/1709.01190.pdf

<img width="557" alt="screen shot 2017-12-26 at 03 41 38" src="https://user-images.githubusercontent.com/596262/34342645-3aff0e98-e9f1-11e7-812e-f2397f22abfb.png">


Looks like it's more for sparse vectors?

I actually contacted the authors of Flash earlier this year. They confirmed that their algorithm is tuned for much sparser datasets and probably won‘t perform well in our benchmark. 

Furthermore, they will contact us if the have some python wrapper that we can easily use.

thanks for the update @maumueller !
Repository: erikbern/ann-benchmarks
Issue: Publish 1M embeddings benchmark

Hello again @erikbern, I hope you're doing well. 

I was curious to know the progress on releasing the benchmarks on the 1M OpenAI embeddings dataset that I created for https://github.com/erikbern/ann-benchmarks/pull/434

We can use this issue to track the same. Let me know if I can help in any way :)

https://github.com/erikbern/ann-benchmarks/issues/460 will also fix this. So closing this issue. Thanks :)

@erikbern @maumueller Is there anything I can contribute to publish the 1M benchmark sooner? It'd really help my friends to see a larger dataset benchmark. I'm happy to help with the running and handling any errors for the benchmarks as well.

Hi – planning to rerun all benchmarks at some point soon.

That being said, is the OpenAI dataset significantly different than previous datasets? I'm somewhat hesitant to use too many similar datasets – we already have a few ones that are similar size.

It's different in some specific ways:

1. 1536 embeddings — most other datasets are at 384-512-768 embedding.
2. 1M records is among the largest
3. OpenAI embeddings are one of the most popular ones at the moment. Benchmarks for that will make ann-benchmarks.com even more useful for the community. 

Hi @erikbern I hope you're doing well

I noticed that [ann-benchmarks.com](http://ann-benchmarks.com/) was last updated in Dec 2021 (2+ years). A lot has changed since then. I'm pretty sure there's a lot of value for the community if we update the website. I'm happy to spend some time running the benchmarks for you. Let me know if I can help :)
Repository: erikbern/ann-benchmarks
Issue: Kosarak dataset - ground truth
Root cause: Documentation Deficiency

I have found results for the kosarak-jaccard dataset on this website, but there are no details in the github repository. The downloaded dataset (https://www.cs.cornell.edu/~arb/data/uchoice-Kosarak/ or http://fimi.uantwerpen.be/data/) does not contain any.

How can I compare my metric index to your results on this dataset? How do you evaluate recall here?

Thank you.

Hello,
[the processed dataset (kosarak-jaccard.hdf5)](http://ann-benchmarks.com/kosarak-jaccard.hdf5), containing only sets with at least 20 elements, includes a ground truth for 100 nearest neighbors.