zhiminy/EvalEng huggingface dataset:
Repository: erikbern/ann-benchmarks
Issue: Default Colors are a bit hard to see on certain algorithms
Root cause: Configuration Error

The default plot colors are too light to clearly see on a white background (especially for faiss-ivf, hnsw(nmslib). I propose a slight modification to the colors to make it more readable in my opinion. What do you folk think?

Code change to ann-benchmarks/plotting/utils.py:
```
def generate_n_colors(n):
    vs = numpy.linspace(0.3, 0.9, 7)
```

Defaults:
![image](https://user-images.githubusercontent.com/35026793/100823467-38a16100-348f-11eb-84ae-91452c903b88.png)


Updates:
![image](https://user-images.githubusercontent.com/35026793/100823520-51aa1200-348f-11eb-9948-71565d21b7db.png)


sure, feel free to submit a PR and I'll merge!
Repository: erikbern/ann-benchmarks
Issue: float32 vs float64
Root cause: Configuration Error

It appears that the benchmark has switched from float32 to float64 quite a while ago, and this somehow contributed to a substantial performance regression of KGraph, which is only optimized for float32 although float64 also runs.

Below are my justifications for using float32 in the context of approximate K-NN search:
- The search process is usually approximate, and the feature data are usually noisy by nature, so typically the precision of float64 is not necessary.
- The necessity of an index to speed up search presumes that data is big.  So a factor of 2x in the cost just to store the data in memory is significant.
- For some algorithm/implementation, using float32 means more data can be fit into cache and the code being much faster.

I believe the performance behavior of some library will be very different for float32 and float64 (though I believe the relative ranking of different libraries should roughly stay the same).  If only one can be chosen, I feel it should be float32 so that the benchmark represents the typical use case.  

Is there a particular reason to keep float64, or it's better supported by some libraries in the benchmark?
(There's another reason why KGraph is slow which I'm still investigating.)


I didn't realize that we have switched to float64. Annoy uses only float32 for instance. If it's just a matter of setting `dtype=numpy.float32` then I can go ahead and fix that. I agree that float32 is more realistic since it's rarely useful to use 64 bit precision. In fact I even experimented a bit with float16 and float8 for Annoy to see if it would be even faster (it was not)


I checked the implementation all contenders, and found the following all use float32 internally: NMSLib, Falconn, Annoy, FLANN and KGraph.  NMSLib, Annoy and FLANN do automatic conversion on the C/C++ side.  Falconn and KGraph does explicit conversion in the benchmark code.  So I think the benchmark is totally fair and reasonable as far as the above methods are concerned.  

Among the remaining ones, I think the sklearn-based ones use float64 internally, but I'm not very sure.     RPForest uses float64 internally, and probably does not support automatic conversion from float32.  PANNS is implemented purely in python and uses numpy, and by default, the index is constructed with float32, as in the benchmark, but it is configurable.  NearPy also uses numpy, and computation is done in the type that is passed in, so it should be float64 in the benchmark.

It appears to me that the overhead of type conversion is minimal.  I think it's OK to leave the choice of type to the library implementation.    I'm closing this issue.
Repository: erikbern/ann-benchmarks
Issue: Graphs hard to parse for color blind people
Root cause: Documentation Deficiency

Hey very cool project! I find the performance graphs really hard to understand though because of my red-green color blindness. I can imagine many others with worse color blindness having even worse issues. Is there a textual comparison table available? This would help a lot.

Another option could be to make the diagrams interactive and highlight the current selected algorithms performance in some way.

There's actually interactive graphs available here! http://ann-benchmarks.com/glove-100-angular_10_angular.html

I'm not colorblind but honestly I find it hard to parse too. Maybe we could limit it to a smaller number of algorithms

Cool looks like I made a good choice using https://github.com/nmslib/hnswlib/ for my project.

Yes maybe reduce to the top 10 and the interactive graph is awesome, even for me! I suggest linking to it more prominently in the top section of the README as I was not aware of it after spending some time looking for it.

@erikbern any plans to use interactive plotting like [Altair](https://altair-viz.github.io/) in python. Would help with more algorithms and still all pythonic without much overhead 

I'd be open to doing that. If you have time, feel free to give it a shot and submit a pull request!
Repository: erikbern/ann-benchmarks
Issue: Different result from the website
Root cause: Configuration Error

I run faiss-ivf and hnsw(faiss) on glove-25-angular dataset in batch mode, and then plot the result:
![glove-25-angular-batch](https://user-images.githubusercontent.com/45357009/108654274-4a7d2800-7503-11eb-8897-5d9cf76c82d7.png)
While the result post on the [interactive website](http://ann-benchmarks.com/glove-25-angular_10_angular.html) like this:
<img width="1064" alt="截屏2021-02-22 上午11 45 31" src="https://user-images.githubusercontent.com/45357009/108654368-7b5d5d00-7503-11eb-9102-c43e95a05755.png">
HNSW result seems reasonable, but the trend of faiss-ivf are totally different, is that normal?

Maybe due to the batch mode. close
Repository: erikbern/ann-benchmarks
Issue: index construction time plotting
Root cause: Documentation Deficiency

I plotted index construction time using data saved in glove.txt and see a spread of almost 2 orders of magnitude (hnsw(nmslib) index construction might have been parallelized according to code).  I'm most interested in the performance of sw-graph vs hnsw.  It's very hard to say which one is better because one might be able to improve the search performance of sw-graph by building more aggressive indices.  I collaborated with Princeton folks on developing a hierarchical version of the algorithm behind KGraph and the results were negative.  Initially there was noticeable speedup at the same accuracy, but after carefully controlling index construction setup, the speedup was gone.

I also suggest that we establish a standard hardware configuration to run the benchmark.  Preferrably some type of AWS instance which can be easily reproduced.  I think something close to your current setup should be good.

![build](https://cloud.githubusercontent.com/assets/5427210/15939002/da24341e-2e43-11e6-9dbc-3944177274e7.png)
![throughput](https://cloud.githubusercontent.com/assets/5427210/15939003/da294b5c-2e43-11e6-940a-2b0b70c000a6.png)


This is great – feel free to send me a PR and I can include those benchmarks in the README.

The standard right now is using a c2.xlarge on AWS. I think it could be interesting also to look at bigger data sets in which case we should use a machine type with more RAM


Btw in the precision-performance plot I filter out any points that are not on the Pareto frontier (meaning there is another point that dominates it both in terms of precision and performance)


The construction time plot uses exactly the same Pareto frontier points. 

```
diff plot-time.py plot.py
51c51
<                 ys.append(t[-3]) #1.0 / t[-2])    # this is indexing time
---
>                 ys.append(1.0 / t[-2])
```

I don't see c2.xlarge instances on EC2.  They might be no longer available.
Will come back to this when I have run through everything on my machine.


Sorry I meant c4.2xlarge – it says that in the readme


@aaalgo heres my vote to see this re-run for latest results and incorporated via PR this is an important facet of overall performance that needs to be part of the discussion.
Repository: erikbern/ann-benchmarks
Issue: python3 create_website.py to generate website， ERROR 'Permission denied'
Root cause: Environment Incompatibility

Should we execute "chmod -R 777 results" after writing out the result folder every time, otherwise we will not have read permission and we cannot use create_website.py to generate a website？

The idea is to to run `create_website.py` with `sudo`. The docker container writes the result files as `root`, and `create_website.py` will cache some computations and needs write access to these files. @erikbern Maybe we should run the docker images as the actual user. 

I think Docker generally likes to run as the superuser on a system, and I'm not sure if we have control over that, but I could be wrong?

My preference here would be to not mess around with permissions if possible. Maybe it's possible to write the website to a different directory or to mount a subdirectory of the results/ directory?

I think it's pretty straight-forward to do, see e.g., https://github.com/lemire/docker_programming_station/blob/master/Dockerfile#L21-L27, from https://lemire.me/blog/2020/05/22/programming-inside-a-container/. 

The problem is not with the website creation, but with the `hdf5` files being owned by `root`, and the metrics scripts trying to cache metrics by writing them into the `hdf5` file. 

If it can be fixed in the Dockerfile, let's definitely do that. I didn't realize it was possible.
Repository: erikbern/ann-benchmarks
Issue: Question:   The difference between the memory of the container and the index process.
Root cause: Documentation Deficiency

I use the Scann algorithm in the ann-benchmarks framework to run my own data set, the size of the data set is 50M, and the startup command is:
```bash
python3 run.py --algorithm scann --dataset data-50M --batch --timeout 18000
```
Docker container information is:
```
Created container d04dcfa2d5: CPU limit 0-79, mem limit 530102299392, timeout 18000, command ['--dataset', 'data-50M', '--algorithm', 'scann', '--module', 'ann_benchmarks.algorithms.scann', '--constructor', 'Scann', '--runs', '5', '--count', '10', '--batch', '[2000, 0.2, 2, "dot_product"]', '[[1, 30]]', '[[2, 30]]', '[[4, 30]]', '[[8, 30]]', '[[30, 120]]', '[[35, 100]]', '[[40, 80]]', '[[45, 80]]', '[[50, 80]]', '[[55, 95]]', '[[60, 110]]', '[[65, 110]]', '[[75, 110]]', '[[90, 110]]', '[[110, 120]]', '[[130, 150]]', '[[150, 200]]', '[[170, 200]]', '[[200, 300]]', '[[220, 500]]', '[[250, 500]]', '[[310, 300]]', '[[400, 300]]', '[[500, 500]]', '[[800, 1000]]']
```
After the program is finished running, I use grafana to check the container memory. The peak memory is 238G.
![image](https://user-images.githubusercontent.com/38530078/112310313-654be380-8cdf-11eb-9f00-9fb0d7eeadec.png)

 From the standard output log, the memory used by the indexing process is 54141892.0K ~ 54G. 
![image](https://user-images.githubusercontent.com/38530078/112309982-fff7f280-8cde-11eb-80f8-b03539cc75c9.png)

Why is there such a big gap? 

The way we measure memory is to just take a snapshot before and after indexing. I think this is probably the last spike from 186 to 233gb that you see. There are other parts that we don't measure, for example the pure size of the dataset and copies of that, which might be the first spikes that you get?


Thank you for responding so quickly, I understand.
Repository: erikbern/ann-benchmarks
Issue: Nearpy distance is Cosine by default - Euclidean not applied
Root cause: Configuration Error

Hi, 

I was checking some libraries and realized that in the case of Nearpy, the euclidean distance is not applied in the benchmark. As can be seen in the [docs](https://github.com/pixelogik/NearPy#engine) of Nearpy, the default is the cosine but the euclidean one is not applied. I will PR changes
thanks for the bench, it's great
Repository: erikbern/ann-benchmarks
Issue: How do you know the interface of faiss python?

I want to add some faiss algorithms on this project, but I can't find description of faiss python interface. How do you get these information?

I am not completely sure what you mean, but you can just take a look how we implemented IVF/LSH/HNSW in https://github.com/erikbern/ann-benchmarks/tree/master/ann_benchmarks/algorithms. I've implemented these following https://github.com/facebookresearch/faiss/wiki/Faiss-indexes if I remember correctly.

> I am not completely sure what you mean, but you can just take a look how we implemented IVF/LSH/HNSW in https://github.com/erikbern/ann-benchmarks/tree/master/ann_benchmarks/algorithms. I've implemented these following https://github.com/facebookresearch/faiss/wiki/Faiss-indexes if I remember correctly.

I had seen IVF/LSH/HNSW, they are fantastic implementation. I just wondering how do you know the argument of faiss function, such as search(), is that same as c++?

I think I took it from the examples the pasted on the wiki. You can always take a look at https://github.com/facebookresearch/faiss/blob/master/faiss/python/__init__.py, or just do `help(...)` on the index object to see its methods.

> I think I took it from the examples the pasted on the wiki. You can always take a look at https://github.com/facebookresearch/faiss/blob/master/faiss/python/__init__.py, or just do `help(...)` on the index object to see its methods.

Thanks! That solved my question.
Repository: erikbern/ann-benchmarks
Issue: Running with Parallelism on AWS C5.4x Large Instance Leads to OOM Errors
Root cause: Resource Mishandling

Hey everyone,

This is an issue I faced while trying to generate the QPS vs Recall plots for `ann-benchmarks`. The issue is that when building the algorithms with `parallelism = 3`, the available memory gets divided equally between the docker containers running the three algorithms.

A typical `AWS C5.4x large` instance comes with 30 GBs of total RAM available. However, when building on top of the ann-benchmarks infrastructure, the maximum RAM availability that I've seen is close to 20 GBs. Hence 20 / 3 = 6.67GBs is the final RAM available while building and running to an algorithm.

This leads to an Out-Of-Memory (OOM) error, wherein the `docker` daemon kills the container running the algorithm build if the required memory exceeds the available memory. Hence certain configurations of a heavy algorithm never build to completion, and as such the QPS vs Recall plot is significantly different from the scenario where every algorithm gets to use the full 20 GBs of RAM (in simpler words, everything is run sequentially).

For this reason, I believe the QPS vs Recall plots are slightly misleading in their current state as certain build configs wouldn't even run to completion, messing up the final pareto-optimal plot. I believe this also explains why so many dataset plots don't contain results from the full set of algorithms (one example being [this](https://github.com/erikbern/ann-benchmarks/blob/master/results/gist-960-euclidean.png)). Attaching screenshot for reference.

<img width="960" alt="OOM Killed Containers" src="https://user-images.githubusercontent.com/19722966/115124343-4a6b4700-9fdf-11eb-995c-5b758e52dd16.png">


Exit Code 137 refers to OOM killed containers. Command: `sudo python3 run.py --dataset glove-100-angular --parallelism 3`



Thanks so much @ShikharJ . I think that this is indeed a huge problem. What do you think, @erikbern ? It seems the `r6g` type instances might be better for running the benchmark in the future?

Last time I ran the benchmarks, some fraction of containers crashed because of OOM issues, and I personally think it's not a super big deal. I prefer if we're more liberal with the algo definitions and you can include a larger range of parameter combinations.

But if 50% of all algos crash then maybe we can run on a bigger instance type next time, I'm open to that.
Repository: erikbern/ann-benchmarks
Issue: big-ann-benchmarks.com

Just saw http://big-ann-benchmarks.com

I'm a bit surprised the authors of that competition didn't mention anything to me or the other people working on this benchmark suite? Feels like there's a lot of overlap and I wish we could have worked together on it.

cc @maumueller 

Actually just realized @maumueller is in the list of organizers... are you involved? Curious about your thoughts on whether it could be merged into this repo. I think there's a lot of value in having reproducible containers and everything open source.

Yes, I'm involved. You were also part of the initial email exchange as far as I remember?

The framework is following ann-benchmarks (and proper credit will be given), but there are some interesting changes with regard to the API and dataset/groundtruth handling. We plan to release the framework in about two weeks, I'll let you know!

Sorry I must have missed that email or forgotten about it. Looking forward to see the code, hopefully there's an opportunity to collaborate!

Hello! Is there some progress on releasing the new framework? Specifically I'm interested to see the I/O libraries to handle loading of 100M or 1B vectors. When using for instance HNSW directly (code taken from the creators of it), data load becomes a bottlneck -- and this is not an issue of HNSW, but more a pure I/O slowness. I've been using this code implemented by Yandex (as part of open sourcing their 1B dataset): https://github.com/DmitryKey/bert-solr-search/blob/feature/hnswlib/src/util/utils.py#L57

We are in the final phase, so hopefully the framework will be released in a week or two. We memory-map binary files, roughly like https://github.com/facebookresearch/faiss/blob/master/contrib/datasets.py#L194-L199. 

sounds great! Looking forward to test it.
Repository: erikbern/ann-benchmarks
Issue: Using custom metrics

Hello,

I wanted to use NN approximation for a problem and I need to use a custom metric for distance calculation between 2 data points. As per my understanding, I would need to implement the algorithm on my own and no library provides such support for adding my own distance score? My problem is related to genomes and the distance between 2 genomes is calculated by an in-house program. Also, to find the best algorithm would require me to test them out rather than only choosing say top 3 because as per my understanding NN search algorithm performance varies with datasets and dimensions. Any help/advice would be appreciated.

We benchmark NN approaches and do not develope them, but my suggestion is to look at some of the graph-based approaches. Personally, I think https://github.com/lmcinnes/pynndescent is a good starting point, see e.g., https://github.com/lmcinnes/pynndescent/commit/56a16eb6bd5ea954c5aa7906a1587b386d29e5d0 for adding a new distance.

It will be quite some work to "test out multiple different implementations" because you need to customize the individual implementations. I would rather start with one and see whether it's performance/quality is sufficient in your use case. 

Adding @searchivarius because he probably has much more to say about that :-) 

Closing this for now as it doesn't seem related to ann-benchmarks per se!
Repository: erikbern/ann-benchmarks
Issue: Regarding a clarification

From the [site](https://awesomeopensource.com/project/erikbern/ann-benchmarks):

> Note that we consider that set similarity datasets are sparse and thus we pass a sorted array of integers to algorithms to represent the set of each user.

Could you expand a bit more on this preferably with a small example w.r.t any of the benchmarking datasets provided? What is sorted and how exactly it is sorted? 

Hi @sayakpaul 

It means that three sets `[{1,3,2}, {1,3}, {1,5}]` are represented by the array `[1,2,3,1,3,1,5]` with an accompanying `size` attribute that would be `[3,2,2]`. See https://github.com/erikbern/ann-benchmarks/blob/18ba39b6a4af1243b26bf5fedc6c5c7ced791d70/ann_benchmarks/distance.py#L38-L45 for details.

Thanks much. Appreciate it.
Repository: erikbern/ann-benchmarks
Issue: benchmark running for too long
Root cause: Configuration Error

I've been running the benchmark on a machine that shouldn't be much slower than c4.2xlarge (core i7-2600k @ 3.4GHz).  It's been 5 days and I only see 195 lines in sift.txt (the plot looks good though).  There are 321 lines in the standard glove.txt, and 82 lines in the standard sift.txt which I think is incomplete.  So I estimate it will take 15 - 20 days for me to finish running everything.  Is it supposed to be so?

I'm attaching the SIFT plot I produced with the data I have so far, which is coherent to the Glove plot I see from the project page.  (KGraph bug fixed.)

![xx](https://cloud.githubusercontent.com/assets/5427210/16066621/e962de4c-3282-11e6-8b7c-e2e6b8abeb2b.png)


i think i takes something like 5-7 days to run everything

you don't have to rerun all results though – nearpy, lshf, panns etc haven't changed much

good to see that kgraph is back!


@aaalgo please, uncomment the line where SW-graph and HNSW save indices. This will make things faster:
https://github.com/erikbern/ann-benchmarks/blob/master/ann_benchmarks/__init__.py#L281


Is there any reason to disable saving index on disk?  I thought I was following NMSLib's example to save the index, but didn't realize the saving's actually disabled by a comment.  I think without saving index, the computation time has become almost impractical to me.


@aaalgo I think @erikbern had some problems with disk space. 


That's legitimate.  I'll leave @erikbern to decide whether or not to comment out the line for KGraph:
https://github.com/erikbern/ann-benchmarks/blob/master/ann_benchmarks/__init__.py#L238


@aaalgo I made the benchmark save indices by default (for nmslib & kgraph). One should be able to cancel saving by --no_save_index 


I think saving index is a good enough solution.
Repository: erikbern/ann-benchmarks
Issue: Update benchmark results
Root cause: Documentation Deficiency

It has been a while since the last time the benchmark results were updated. Do you have any plan to rerun the benchmark and update the README?

Yeah that would be a good idea! Maybe I'll set aside some time and do it soon.

@erikbern  What's the status? I could probably also do it soon. 

Sure, we can do it. Any final updates to the set of algorithms before I kick it off? Thinking if there's any new contenders we should add or if we should purge any old ones.
Repository: erikbern/ann-benchmarks
Issue: Support for an common JSON benchmark results file
Root cause: Unimplemented Feature Gap

Apart from being able to generate the results website, I believe that for a continuous persistency of results/sharing/post-processing/visualization, being able to export them as a common JSON benchmark schema would be extremely beneficial. 


I worked on something like that but it will take a while before being ready. I agree that it would be nice to have :-) 

@maumueller I'm bringing this issue back online. Does the option that we (myself and @filipecosta90) will open a PR for that is valid?
Repository: erikbern/ann-benchmarks
Issue: ERROR - Child process for container raised exception 137 : Annoy
Root cause: Environment Incompatibility

Using Python v3.6.10
Followed all the instructions of readme.

Command `python run.py --dataset glove-25-angular --docker-tag ann-benchmarks-nmslib` worked perfectly and generated the ouput graph for nmslib on glove-25-angular dataset.

While running Annoy on dataset glove-25-angular using command `python run.py --dataset glove-25-angular --docker-tag ann-benchmarks-annoy` I get error
ERROR - Child process for container dd46e7851b raised exception 137
What could be the reason?

The install status for annoy was 'success'

How much RAM do you have available on your setup? It might run out of memory.

I am using mac with 16gb memory. 

You are right ! The docker preferences was set to 4 GB of RAM only. Maybe, thats why the error was occuring.
Increasing it to 16GB seems to be working. 

Thanks for the help !
Repository: erikbern/ann-benchmarks
Issue: What is meant by setting "parallelism" to 3?
Root cause: Documentation Deficiency

Is "parallelism" a setting for the AWS instance or is it an argument that should be included when running python install.py or python run.py?
![image](https://user-images.githubusercontent.com/20480027/132387320-b83a7546-28e3-4b36-a486-03cb7777352c.png)