zhiminy/EvalEng huggingface dataset:
Repository: erikbern/ann-benchmarks
Issue: Milvus algorithm couldn't find the connection options inside ../ann_benchmarks/algorithms/milvus
Root cause: Documentation Deficiency

hi , 
I couldn't find how to add connections options under module.py or config.yaml .
I have ann benchmark in one node and vector db (milvus) in different host . 

While looking at this : https://milvus.io/docs/example_code.md
I am looking for a connection entry inside ANN benchmark for milvus but couldn't locate how to connect remotely to the vector DB server node . any help ?

connections.connect("default", host="localhost", port="19530")


@wangting0128 added the Milvus implementation, so he probably knows best. 

As far as I understand, the version that is used here is not using a client/server architecture.

if that is the case then I need to know how and which setup on the ANN benchmark code I should be using as client connecting to any Vector DB server please ? thanks !

You can find one example here: https://github.com/erikbern/ann-benchmarks/tree/main/ann_benchmarks/algorithms/elasticsearch The dockerfile takes care of starting the server, the connections are made inside `module.py`. 

hi thanks . Elasticseach has the connection entry but Milvus doesn't . Could you please help adding them ? https://github.com/erikbern/ann-benchmarks/tree/main/ann_benchmarks/algorithms/milvus

I can't see it inside module.py 

@maumueller   I would appreciate NOT to close the issue until addressed : "Milvus algorithm couldn't find the connection options" inside module.py .

I want to inject the below code in the module.py and what is right place to have them added on ANN benchmark please ? 

from pymilvus import connections
connections.connect(
  alias="default",
  user='username',
  password='password',
  host='localhost',
  port='19530'
)

@wangting0128  do you know why I am getting below exceptions ? 

~/ann-benchmarks$ sudo python3 run.py --dataset glove-25-angular --algorithm milvus --runs=1
2023-09-29 22:37:12,919 - annb - INFO - running only milvus
Traceback (most recent call last):
  File "/home/ubuntu/ann-benchmarks/run.py", line 7, in <module>
    main()
  File "/home/ubuntu/ann-benchmarks/ann_benchmarks/main.py", line 341, in main
    raise Exception("Nothing to run")
Exception: Nothing to run
Repository: erikbern/ann-benchmarks
Issue: how to clean the older benchmark runs

I do see bunch of png and html file generated and also the previous runs keep in history .
How to purge the older runs and make sure any benchmark starts clean ? 
Also any way to make sure the html and png generated into a dedicated folder rather messing up the root /ann_benchmark folder ?
thanks


You probably see these because of something that you did. It would have been helpful to know which commands you invoked. 

You can use `--force` while invoking `run.py` to overwrite the results files, even easier would be to just remove the files the are present in `results/`. The "bunch of png and html file" is probably the result of invoking `create_website.py`. You can specify an `--outputdir`, as mentioned in the README. 

great input thank you @maumueller  . found them and working
Repository: erikbern/ann-benchmarks
Issue: Milvus  benchmarking run issue
Root cause: External Dependency Breakage

Opening new issue as: 
Any pointer what is wrong and what code should be used to connect Milvus database with connection info ? 

~/ann-benchmarks$ sudo python3 run.py --dataset glove-25-angular --algorithm milvus --runs=1
2023-09-29 22:37:12,919 - annb - INFO - running only milvus
Traceback (most recent call last):
File "/home/ubuntu/ann-benchmarks/run.py", line 7, in
main()
File "/home/ubuntu/ann-benchmarks/ann_benchmarks/main.py", line 341, in main
raise Exception("Nothing to run")
Exception: Nothing to run

You can try executing the following command to see if it works:
`sudo python3 run.py --dataset glove-25-angular --algorithm 'Milvus(Knowhere)'`

@wangting0128  do you think this code can be reused and replace on module.py to make pymilvus works ? 
https://gitcode.net/mirrors/milvus-io/ann-benchmarks/-/blob/master/ann_benchmarks/algorithms/milvus.py

> @wangting0128 do you think this code can be reused and replace on module.py to make pymilvus works ? https://gitcode.net/mirrors/milvus-io/ann-benchmarks/-/blob/master/ann_benchmarks/algorithms/milvus.py

This is milvus1.0, which is no longer maintained.

yes I tried to reproduce and failed on : import milvus part of the code even if I downgraded pymilvus to older version on python 3.11 . its weird and not sure why its not working !

> yes I tried to reproduce and failed on : import milvus part of the code even if I downgraded pymilvus to older version on python 3.11 . its weird and not sure why its not working !

pymilvus is working for milvus version >= 2.0

@pauldintel hi, you can try the latest code. https://github.com/erikbern/ann-benchmarks/pull/517 has updated the code for milvus.
Repository: erikbern/ann-benchmarks
Issue: [Dataset] The train and test sections of some datasets have only 1 dimension for each row
Root cause: Documentation Deficiency

Hi,

I try to read data from the hdf5 input dataset, I found for `Kosarak` and `[MovieLens-10M]`, they have only 1 element for each row? Is there any problem with these datasets?
![image](https://github.com/erikbern/ann-benchmarks/assets/104770161/194a6b2a-7eda-4c24-95c0-c3d40affb3f7)


These datasets are sparse and use somewhat like the scipy csr format for packing them efficiently. Please see https://github.com/erikbern/ann-benchmarks/blob/main/ann_benchmarks/distance.py#L104 as an example of how you can work with them. 

So, when I use this distance function to get train and test objects which are basically list object, right? np.ndarray.

Then I found that len of each row is not the same, so it still stored in compact style, right?

## I guess that does each element in the rows represents the index of the non-zero element in the orignal sparse vector?

```
f = h5py.File(h5_file, 'r')
train, test = dataset_transform(f)
print(type(train))
print(len(train))
for i in train[0]:
      print(str(i), end=' ')
print()
```
![image](https://github.com/erikbern/ann-benchmarks/assets/104770161/5b64d486-3fa7-48d9-afc9-37757ec3498f)



Yes, `train[0]` is the first sparse vector, it has 123 non-zeros, at the positions listed in your example.
Repository: erikbern/ann-benchmarks
Issue: How the sparse datasets are stored?
Root cause: Documentation Deficiency

Hi professor @maumueller ,

As you mentioned the `Kosarak and MovieLens-10M` are sparse dataset and they are packed like a scipy csr format,

So, when I use this distance function in https://github.com/erikbern/ann-benchmarks/blob/main/ann_benchmarks/distance.py#L104 
 to get train and test objects which are basically list object, right? np.ndarray.

Then I found that len of each row is not the same, so it still stored in compact style, right?

## I guess that does each element in the rows represents the index of the non-zero element in the orignal sparse vector?

```
f = h5py.File(h5_file, 'r')
train, test = dataset_transform(f)
print(type(train))
print(len(train))
for i in train[0]:
      print(str(i), end=' ')
print()
```
![image](https://github.com/erikbern/ann-benchmarks/assets/104770161/5b64d486-3fa7-48d9-afc9-37757ec3498f)
Repository: erikbern/ann-benchmarks
Issue: Adding Spotify's new Voyager Algorithm(Library)
Root cause: Unimplemented Feature Gap

Hello @maumueller  and @wangting0128 can we try to include spotify's recent release for ANN, i.e. Voyager into it.
The python implementation seems to be fairly easy to use and it will keep the benchmark updated with latest results.

Here is he python api reference for the same:
https://spotify.github.io/voyager/python/reference.html

and below is the GitHub repo of the same:
https://github.com/spotify/voyager

Closing as it it is already done.
Apologies for the disturbance!
Repository: erikbern/ann-benchmarks
Issue: i need some help about connect my database,thanks!!
Root cause: Documentation Deficiency

I have a question. I want to ask someone who can answer me. I want to deploy the database locally on Linux and use ann-benchmarks to test my self-developed vector database. But I don’t know how to connect ann-benmarks to my relational vector database. , where should I configure it in the python code? Maybe I should write some code to connect, right?
I'm new to databases, so my question may seem a bit silly, but I'm hoping someone can shed some light on my confusion, thank you!

Hi @frg01. You question is a bit open to give a specific answer, but I suggest that you take a look at the code for elasticsearch https://github.com/erikbern/ann-benchmarks/blob/main/ann_benchmarks/algorithms/elasticsearch/module.py. You will have to write all the code to setup the connection in your own `module.py` file, setting up the connection in the `init`, building the index in `fit` and search for vectors in `query`.  It's usually easiest to write the code in local mode without the docker interface. Use `python run.py --local ...` instead of `python run.py ...`.

Hope that helps.

Thank you very much for your guidance.

   I have another question. I deployed the Ann-Benchmarks project on the Ubantu system. When processing the fit() function, I don’t know how to handle the passed X (numpy.ndarray type) parameter.
   When I connect to my database and insert data individually（ cur.execute()）, the project can run successfully, but when I insert data in batches（cur.executemany()）, errors always occur. I'm really about to collapse. I really don't know how to use the numpy package.
# The following code can be executed successfully

```
import psycopg2
text_data = []
for i, x in enumerate(X):
    c = x.tolist()
    em = json.dumps(c)
    ems = "\'" + em + "\'"
    id = json.dumps(i)
    try:
        cur.execute(f"insert into items (id,embedding) values ({id},{ems})")
        conn.commit()
    except Exception as e:                                                                                                                                      
        print("Insert failed: ", e)
```

# The following code doesn't work

```
res = []
sql = "insert into items (id,embedding) values ( %s , %s )"
for i, x in enumerate(X):
    id = json.dumps(i)
    c = x.tolist()
    em = json.dumps(c)
    ems = "\'" + em + "\'"
    temp = (id,x)
    res.append(temp)

    if int(i) % 4 == 0 and int(i) >= 3:
        try:
            cur.executemany(sql,res)
        except Exception as e:
            print(e, "------")
        finally:
            conn.commit()
            res = []
cur.executemany(sql,res)
conn.commit()
```


   I try to use many function ,but don't know how to slove it. For example : numpy.ndarray:tostring() tolist()    json:dumps()....
How do I do this? Help me plzz.
Repository: erikbern/ann-benchmarks
Issue: looking for the results data

I'm developing my own implementation of a knn search algorithm, and  I'm trying to compare with the other implementations . 
However, my hardware is very limited so I'm having a hard time trying to replicate your results. I was wondering if I could get the results data (the once are stored in /results) so I can make the comparison. If so, I appreciate if you could provide a link to it.
Thanks in advance

why you want replicate author's result ,I mean ,it's just a reference result for us. You can run all algorithm on you machine.  Or ,you can see offical result in https://ann-benchmarks.com/;
Repository: erikbern/ann-benchmarks
Issue: Wrong result in using FASHION-Mnist
Root cause: Documentation Deficiency

Hi Team,

I am testing postgres:pgvector vector database. I am running fashionmnist dataset. At the 1753rd vector in test, the nearest neighbour algorithms is giving me wrong result.

From db: {6349,35759,28985,6773,47304,20541,42928,1917,37047,25325,29116,59764,47933,2860,54623,55051,27275,52975,17235,25513,55102,12008,49533,46245,26659,53822,38326,4236,27257,3023,52362,4253,46072,48465,59565,58853,33506,12176,8910,53075,30478,50572,615,24559,21951,50392,41150,35174,55308,40291,17818,58653,578,16808,47401,36499,24629,21902,41678,46517,11300,29654,31262,1689,20176,13866,54595,53476,54953,3835,32542,40057,16698,25292,58483,40747,7271,20580,51551,45222,14984,37766,11633,49348,49519,35873,46319,23766,35257,57740,4595,36427,14214,38592,27046,43771,35610,54373,25315,**2583**}

Actual: {6349,35759,28985,6773,47304,20541,42928,1917,37047,25325,29116,59764,47933,2860,54623,55051,27275,52975,17235,25513,55102,12008,49533,46245,26659,53822,38326,4236,27257,3023,52362,4253,46072,48465,59565,58853,33506,12176,8910,53075,30478,50572,615,24559,21951,50392,41150,35174,55308,40291,17818,58653,578,16808,47401,36499,24629,21902,41678,46517,11300,29654,31262,1689,20176,13866,54595,53476,54953,3835,32542,40057,16698,25292,58483,40747,7271,20580,51551,45222,14984,37766,11633,49348,49519,35873,46319,23766,35257,57740,4595,36427,14214,38592,27046,43771,35610,54373,25315,**32897**}


You can see the result differ at the last value. Is this expected

Thanks,
Mithun

pgvector is approximate and not guaranteed to return the correct nearest neighbors
Repository: erikbern/ann-benchmarks
Issue: pykgraph unavailable for kgraph
Root cause: External Dependency Breakage

Hi all,

I am not able to install kgraph algorithm using this command -

`sudo python3 install.py --algorithm kgraph
`
I am unable to find pykgraph.

![junk](https://github.com/erikbern/ann-benchmarks/assets/153271464/521f156f-0729-4ed0-bdda-561bc790163a)


I am unable to install kgraph with this command 

`sudo python3 run.py --algorithm kgraph`

![image](https://github.com/erikbern/ann-benchmarks/assets/153271464/c19fce17-6a07-4a19-af18-6c74e5949690)
Repository: erikbern/ann-benchmarks
Issue: Running benchmark with custom dataset
Root cause: Documentation Deficiency

Hello,
Currently the repo has pre-defined dataset (glove, SIFT, NYTIMES).
Can I use my own custom dataset(*.csv) instead of the above and run the benchmark ?

Thanks in advance.

You won't be able to run directly on your csv file. You will first have to add some wrapper code to https://github.com/erikbern/ann-benchmarks/blob/main/ann_benchmarks/datasets.py (see the `DATASETS` dictionary in the very bottom). Usually, this involves parsing the csv into a numpy array, splitting the dataset into train/test, and using `write_output(...)`. Hopefully, you find the many examples which are already present useful.
Repository: erikbern/ann-benchmarks
Issue: time consumed by run.py

Hi all,

please let me know how much time it takes for run.py code to finish the execution ?

Thanks.

For all the benchmarks? It should take a few days to run all of them

@erikbern 
i ran the run.py day before yesterday. it has reached upto the logs as shown in screenshot.
please give some clarity that how much time it might take more to complete the execution ? and how can we see/analyse which DB is performing better ?
![Screenshot from 2024-03-07 10-32-58](https://github.com/erikbern/ann-benchmarks/assets/52161957/6f680800-eb4d-4057-9156-da2ab909b3d3)


Thanks in advance.
Repository: erikbern/ann-benchmarks
Issue: @erikbern
Root cause: Documentation Deficiency

              @erikbern 
i ran the run.py day before yesterday. it has reached upto the logs as shown in screenshot.
please give some clarity that how much time it might take more to complete the execution ? and how can we see/analyse which DB is performing better ?
![Screenshot from 2024-03-07 10-32-58](https://github.com/erikbern/ann-benchmarks/assets/52161957/6f680800-eb4d-4057-9156-da2ab909b3d3)


Thanks in advance.

_Originally posted by @dipshirajput in https://github.com/erikbern/ann-benchmarks/issues/492#issuecomment-1982359858_
            

It's going to take several days to run it. You need to create plots afterwards!

@erikbern 
ok. currently i am running the test using,
`sudo python3 run.py --dataset glove-100-angular`
any approximation if you can mention about the number of days it might take ?
Repository: erikbern/ann-benchmarks
Issue: @erikbern

              @erikbern 
ok. currently i am running the test using,
`sudo python3 run.py --dataset glove-100-angular`
any approximation if you can mention about the number of days it might take ?

_Originally posted by @dipshirajput in https://github.com/erikbern/ann-benchmarks/issues/493#issuecomment-1993701274_
Repository: erikbern/ann-benchmarks
Issue: db running before running the benchmark
Root cause: Documentation Deficiency

HI @erikbern ,
do we need to run the databases seperately before following the steps in the doc for running the benchmark? i mean the steps mentioned in this repo are only responsible for running the benchmark or it runs the databases also ?


The contributors of the specific implementation made sure that it installs and runs the database. Note that some implementations like milvus have a standalone version for this benchmark that avoids the client/server architecture.
Repository: erikbern/ann-benchmarks
Issue: time taken for benchmarking to finish the execution
Root cause: Documentation Deficiency

Hi all,

i wanted to test the benchmark performance of Qdrant and Milvus and so i ran the run.py script. but it has been 5 days approx and the benchmark is still running. 
please let me know how much more time it might take for the benchmarking to finish ?

Also, please let me know if there is any way to minimize the timing by restricting the dataset or specifying the only 2 db , qdrant and milvus for the benchmark execution.

Thanks in advance.

A small example to run only these two algorithms on the smallest real-world dataset available is:

`for algo in milvus qdrant; do python3 run.py --algorithm $algo --dataset fashion-mnist-784-euclidean --runs 1 --run-disabled; done`

This assumes that you already built the container for these two using the `install.py` script.


@maumueller  
`python3 run.py --algorithm $algo --dataset fashion-mnist-784-euclidean --runs 1 --run-disabled; done`

this is the command , is it ? also, which part of this command specifies milvus and qdrant ?

This uses bash to run both implementations. If the above doesn't make sense to you, you can execute: 

`python3 run.py --algorithm milvus --dataset fashion-mnist-784-euclidean --runs 1 --run-disabled`

wait until it has finished, and then run 

`python3 run.py --algorithm qdrant --dataset fashion-mnist-784-euclidean --runs 1 --run-disabled`

ohk. any approximation of how much time individually these can take ?

It seems to me that qdrant searches through a lot of hyperparameters, so I'm not sure. If it's more than a day on this tiny dataset, their parameter search is clearly too exhaustive. You can further restrict the number of hyper-paramters by setting `--max-n-algorithms ` to a small number. 

ohk, i will try with this. 
also, do we need to run the databases seperately or run.py already gets the databases into picture for performance benchmarking ?
Repository: erikbern/ann-benchmarks
Issue: Consistently prewarming database
Root cause: Documentation Deficiency

pg_embedding client currently does pre-warming of database while the same is not happening in other databases, including pgvector. Does this effect the comparision in terms of QPS? And should this be standardized across the board? 

Hi @wahajali, for pgvector specifically, the data should already be in shared buffers, so prewarming shouldn't make a difference.

Hi @ankane could you please elaborate why that is the case? Does the shared_buffer get populated on insertion?

Yes, and when the index is created. You can run the benchmarks with and without `pg_prewarm` to compare.
Repository: erikbern/ann-benchmarks
Issue: benchmark stopped with exit code 0 
Root cause: Documentation Deficiency

@maumueller @erikbern  
Hi all,
while i was running the benchmark from 10days approx, it stopped with an exit code 0.
1. can you tell me if its the completion of benchmark performance run or there could be some issue ?
2. also, how can we analyse the results after graph plotting has been done ?

![remmina_10 168 1 45:1_10 168 1 45:1_2024325-62740 858156](https://github.com/erikbern/ann-benchmarks/assets/52161957/d1494443-e36a-4687-a650-19c7031662a0)

Thanks in advance .

I think that means it's done!
Repository: erikbern/ann-benchmarks
Issue: qdrant  disabled in the benchmark 
Root cause: Configuration Error

Hi all,

i ran the benchmark test and found out that qdrant folder is not even created in the result directory. 
i found out that qdrant was disabled in its configuration file . other than that config file, do we need to enable it anywhere else ?

Thanks in advance.

It's probably fine to enable it in the config file
Repository: erikbern/ann-benchmarks
Issue: analysing the output after plot is generated for the databases
Root cause: Documentation Deficiency

Hi all,

I was working with benchmarking Milvus and qdrant . i am able to generate the plots for both the db.
please let me know how can we analyse the graph and conclude their performance comparison ?

1. how can we analyse the following plots ? please help me with this.
2. also which plot should we go through in order to conclude the performance ?

![fashion-mnist-784-euclidean_10_euclidean](https://github.com/erikbern/ann-benchmarks/assets/52161957/42867bd9-eba5-4d33-a678-745bfaaef518)
![glove-100-angular_10_angular](https://github.com/erikbern/ann-benchmarks/assets/52161957/30e79a9a-8dd2-41e7-b213-bdd1592dab99)


Thanks in advance.

HI, @dipshirajput ,could you pls tell me how you successfully run the milvus test? It keeps reminding missing docker images while idk how to find this image.Didn't dound in internet too. Thanks a lot for your help!
![image](https://github.com/erikbern/ann-benchmarks/assets/32962590/b2a03a8d-d862-4a76-9267-b8cff3070eb2)
Repository: erikbern/ann-benchmarks
Issue: Request: Provide nearest neighbors for training data
Root cause: Unimplemented Feature Gap

For datasets like SIFT 1M, it would be **great** if nearest neighbors were provided for training data as well. From the perspective of computational cost, the test data is very small and users can create its metrics themsleves. But if you could provide the NNs for 1M training points that would help those who want to use the larger datasets.

Feel free to submit a pull request! Should be quite easy – here's the relevant code: https://github.com/erikbern/ann-benchmarks/blob/master/ann_benchmarks/datasets.py#L36

I can re-run and re-upload datasets

Thanks for pointing me to the right place. I have just submitted a PR.

Small change. On line 69, please replace `res.remove(x)` with `res.remove((x,0.))`. I forgot this was a list of tuples.

@dataplayer12 Can you explain the significance of testing on the "train" data? 
I think any hash table with precomputed nearest neighbors for each dataset point can solve this task perfectly. 
The test results on train data will be significantly different in terms of both speed and accuracy. i.e. k-nn graph algorithms store precomputed nearest neighbors for each of the dataset point and show way better performance on "train" data.

I am not sure that the division of 'train' and 'test' data is very strict here. Unless one is using a neural network or some other supervised learning technique, hashing is mostly an unsupervised task. Having 1Mx100 nearest neighbors for training dataset will make the dataset more useful for the community by allowing it to be used in contexts other than the framework and evaluation method defined in this repo.  Hope that helps.

closing this – i remember trying at some point to compute nearest neighbors for the training set but it's brutally slow

Can't argue about it being slow. Thank you for trying.