zhiminy/EvalEng huggingface dataset:
Repository: dezoito/ollama-grid-search
Issue: [ollama-grid-search_0.9.0_aarch64.dmg] is damaged
Root cause: Environment Incompatibility

I get a damaged error trying to download any of the latest apple-arch disk images.

[ollama-grid-search_0.9.0_aarch64.dmg](https://github.com/dezoito/ollama-grid-search/releases/download/v0.9.0/ollama-grid-search_0.9.0_aarch64.dmg)
“ollama-grid-search” is damaged and can’t be opened. You should eject the disk image.

Version 0.8.0 and 0.7.0 also give the same error for me.

I'm Running Mac OS 15.1.1 on an m1 max. 

Hey @awlego, sorry you are having issues with the release.

I have little to no control over the output generated by the github action that builds the releases (and not having a new Mac where I can test them doesn't help either). Looks like something was recently changed on that pipeline that might be affecting the generated artifacts (or maybe MacOS now hates unsigned releases even more).

Once I update ollama-rs I'll let you know so you can try again with the new version.



aye aye thanks! I'll close the issue to not pollute your issues. Appreciate your work and the project.
Repository: dezoito/ollama-grid-search
Issue: [ollama-grid-search_0.9.1_aarch64.dmg] is damaged
Root cause: Environment Incompatibility

This is basically the same issue as [this one](https://github.com/dezoito/ollama-grid-search/issues/69), but for the latest release.

I have access to a couple of M-series Macs, if you need someone to help with testing.

Thank you!
I really appreciate the offer.

I'm finally getting one next week which will help, but this is hard to fix since the release building is dependent on external scripts and GitHub actions. 

Providing the release for all platforms, I think, has been one of the best "features", since not everyone has Rust knowledge, and I intend to keep it. 


Can't run on OS X (Error msg : Apple can't verify...)

Experiencing same issue on an M4.

Version 0.9.2 (hopefully) solves this. 
Please give it a try and LMK.
Repository: dezoito/ollama-grid-search
Issue: ollama-grid-search seems disconnected with the ollama server
Root cause: Documentation Deficiency

Hi, I attempt to install `Ollama.Grid.Search_0.9.1_x64-setup.exe` on my local Windows 11 laptop and it raised the following error after launch:
<img width="1616" height="1112" alt="Image" src="https://github.com/user-attachments/assets/41b3d7bc-4598-4068-8022-263a780b0f0f" />

Hey there!

The title for this issue is misleading, since you were able to install the application and it's running.

You are having network issues that prevent the app from connecting to your Ollama instance.
I had the same problem running on my Windows machine in a corporate environment, where even using "127.0.0.1" for the server address would not work.

Some things you can try:

1- Make sure you have the "Expose Ollama to the network" option selected in Ollama settings.
2- Make sure you don't have any old environment versions from previous installations of Ollama.
3- Try different addresses for the Ollama Server URL (in Ollama Grid Search settings). 
4- Are you using the correct port for your Ollama server? (the default is 11434, but it might be different in your system).
5- Make sure that Ollama Grid Search is not being blocked by a local firewall or network configuration.

Hope you can find a solution (and if  you  do, please post it here).


Thanks, it seems to be working now!

BTW, it is better to mention the tool setup in the readme as well.
Repository: dezoito/ollama-grid-search
Issue: Error: Dependency not satisfiable when trying to install .deb package
Root cause: External Dependency Breakage

Hi,
when trying to install ollama grid search via the .deb package either for 0.9.2 or the prior version, I get an error: Dependency Not satisfiable libwebkit2gtk-4.0-37.

I'm running Linux Mint 22.2 (Ubuntu 24.04 LTS based).

Doing a cursory apt search for the dependency shows 4.1-0 is available but 4.0-37 is not, seemingly because it is deprecated. 

It seems that Tauri v2 has updated the dependency to address the lack of availability but Tauri v1 is no longer updated or supported by Tauri.

Are there any plans to migrate to Tauri v2 or an alternative?

Thanks.

Webkit compatibility has been an occasional pain in the ass (long time Mint user here too).

> Are there any plans to migrate to Tauri v2 or an alternative?

Not for the moment, sorry.

Can you try the [ollama-grid-search_0.9.2_amd64.AppImage](https://github.com/dezoito/ollama-grid-search/releases/download/v0.9.2/ollama-grid-search_0.9.2_amd64.AppImage) version? 

It usually works well.

> Webkit compatibility has been an occasional pain in the ass (long time Mint user here too).
> 
> > Are there any plans to migrate to Tauri v2 or an alternative?
> 
> Not for the moment, sorry.
> 
> Can you try the [ollama-grid-search_0.9.2_amd64.AppImage](https://github.com/dezoito/ollama-grid-search/releases/download/v0.9.2/ollama-grid-search_0.9.2_amd64.AppImage) version?
> 
> It usually works well.

The AppImage seems to work for now.
Repository: dezoito/ollama-grid-search
Issue: Issues with Llama 3
Root cause: Configuration Error

If you are having issues with Llama3 or any of its versions, here's the most straightforward to get it running:

1) Use this as your model file for Llama3 (make sure to use the name for the version you've pulled):

```
FROM llama3:8b
TEMPLATE """{{ if .System }}
system


{{ .System }}
{{ end }}{{ if .Prompt }}
user


{{ .Prompt }}
{{ end }}
assistant


{{ .Response }}
"""
PARAMETER stop "
"
PARAMETER stop "
"
PARAMETER stop "
"
PARAMETER stop "
"
PARAMETER stop "assistant"
PARAMETER stop ".assistant"
PARAMETER stop "!assistant"
PARAMETER stop "?assistant"
```
2) On the `settings` model in Ollama Grid Search, make sure to **completely** remove the `stop` parameter from the "default options" textarea:

![image](https://github.com/dezoito/ollama-grid-search/assets/6494010/5db22760-563a-465d-bdbb-45204c100c28)
(The above is what I am currently using)


**But why is this necessary?**

As far as I can understand, Llama3 requires these specific stop parameters, but there were some issues or misspellings with the models provided from Ollama's library (maybe an upstream issue).

If those are not setup correctly (and you don't specify a `num_predict`  option) , the model will not stop generating tokens (and thus run indefinitely or timeout).

If you pass the API anything for the `stop` parameter, it will try to use that, instead of what's in the `modelfile`, and run into the same previous issues.

That was not a requirement for **any** of the previous models I've tested (and that's a long list), but currently this is the best way to deal with it.


@zedmango
Repository: erikbern/ann-benchmarks
Issue: for help: code raise a exception
Root cause: Environment Incompatibility

when runing install.py this line of code raise a exception，
subprocess.check_call( 'docker build \   --rm -t ann-benchmarks -f install/Dockerfile .', shell=True) 
Exception is：
subprocess.CalledProcessError: Command 'docker build         --rm -t ann-benchmarks -f install/Dockerfile .' returned non-zero exit status 1.
I hope you can help me

Is there any solution to this problem? I am facing the same on Ubuntu 16.04.

Does it print a bunch of stuff to standard out? Can you share?

I finally resolved this issue by ensuring the docker daemon was running correctly before I ran this code. Thanks.

Hello, I too am facing the same issue once I try to run the install.py script, Is there a solution for the same?

What does it say if you run ` docker build --rm -t ann-benchmarks -f install/Dockerfile .` in your terminal (in the ann-benchmarks directory)?

I also has experienced errors with the docker build, probably due to ubuntu update.
Changing the first line of the docker file to `FROM ubuntu:18.04` helped me.

> What does it say if you run ` docker build --rm -t ann-benchmarks -f install/Dockerfile .` in your terminal (in the ann-benchmarks directory)?

Hello Sir, I tried the suggestion by Yury Malkov, it now passed the step6/7 of  RUN pip3 install -rrequirements.txt :) 


> I also has experienced errors with the docker build, probably due to ubuntu update.
> Changing the first line of the docker file to `FROM ubuntu:18.04` helped me.

Hello Sir, this helped pass the step where I had got stuck. Thank you :)
Repository: erikbern/ann-benchmarks
Issue: ivf-hnsw
Root cause: Documentation Deficiency

I stumbled over https://github.com/dbaranchuk/ivf-hnsw

Did anyone tried to benchmark their implementation?
Is there conzeptual difference between ivf-hnsw and hnsw from hnswlib or nswlib?

I suppose @yurymalkov can say more about it. :-) 

@godofdream,
Yes, there is a conceptual difference. 
ivf-hnsw uses hnswlib as a part and it targets billion scale scenario with low memory footprint (e.g. 1B index in 20 Gb RAM). 
So, it should not be as fast as hnswlib, but it is much more memory efficient. 
Also, @dbaranchuk is the right person to ask about it. :)


great if anyone wants to add it!

@yurymalkov : any comments on the quality of the results if the index could be possibly stored in hnswlib as well

@prateekpatelsc AFAIK it is using a fork of hnswlib, which should have a similar performance (although I haven't tested).

@yurymalkov : sorry may be my question was not clear.
I meant to ask that hnsw-ivf uses quantizers (from a quick look at code), i was curious if that causes any lower recall quality for example for a dataset of size 1 million (so could have used either hnswlib or hsnw-ivf index approach).  I was more curious about how much perf degrades in terms of quality and latency/throughput if i decide to use hnsw-ivf as opposed to hnswlib . 

Also i do see https://arxiv.org/pdf/1812.09162.pdf , which tries to further reduce memory footprints and optimize on performance on top of hnsw-ivf indexes, wondering if you had a chance to look at this as well ?
Repository: erikbern/ann-benchmarks
Issue: Add PQT
Root cause: Unimplemented Feature Gap

https://github.com/cgtuebingen/Product-Quantization-Tree

https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Wieschollek_Efficient_Large-Scale_Approximate_CVPR_2016_paper.pdf

sure – want to give it a shot?
Repository: erikbern/ann-benchmarks
Issue: Add SPTAG
Root cause: Unimplemented Feature Gap

This looks interesting: https://github.com/microsoft/SPTAG

Bump, came here to say the same thing. Microsoft just announced this. 

looks pretty cool!

I haven't try yet but it looks like it provides "update" function too. 

They even called it "A library for **fast** approximate nearest neighbor search".

@maumueller @erikbern do we have results yet? Thanks!

@searchivarius i'll re-run the benchmark in the next couple of months

@erikbern Any reason to not just rerun SPTAG on an ec2 instance to have some results in? (I can do that.)

@maumueller for methods that are far from the top, there's certainly little need for re-running, but I would think one needs to re-run a few top methods on the same machine as the difference from 1 year ago in hardware might matter a bit.

there's probably updates to a lot of the libraries as well, would make sense to re-run all of them

SPTAG was added in #119 so i'll close this

Thank you @erikbern 

Hello, since SPTAG has been added into this library, but why the figure in readme does not show the performance of SPTAG?

@chasingegg it's just been a long time since the last time i ran a benchmark. probably worth doing again
Repository: erikbern/ann-benchmarks
Issue: About SPTAG test results
Root cause: Documentation Deficiency

I tested it with a GIST data set，The size of the query vector is (1000*960)，Return 100 neighbors，I used the official default parameters，The time I built the index is 20562.084 s，The query time is 35.082 s，recall is 0.92669，The time to add 100 vectors is 22.281 s. I feel this time is very long, am I using it correctly? Its index construction and query time are longer than NSG. Can you give SPTAG test reference results?
my machine:
Intel(R) Xeon(R) Silver 4116 CPU @ 2.10GHz
Ubuntu 18.04
python3

yeah that sounds unreasonable. i'll look at it in a few months when i re-run the results

Hi @MrHwc 

How you talk about the measurements, it doesn't seem that you run it in ann-benchmarks, right?

I just had a quick go on fashion-mnist (fewer points, but also high dimensional).

```
Parameters                                 Recall      Queries per Second
Sptag(metric=L2, algo=BKT, check=4000)        0.998      343.924
Sptag(metric=L2, algo=BKT, check=50)        0.903      977.190
Sptag(metric=L2, algo=BKT, check=200)        0.933      901.778
Sptag(metric=L2, algo=KDT, check=400)        0.951     1597.451
Sptag(metric=L2, algo=BKT, check=10)        0.903      972.951
Sptag(metric=L2, algo=KDT, check=4000)        0.952     1177.086
Sptag(metric=L2, algo=BKT, check=400)        0.966      756.975
Sptag(metric=L2, algo=KDT, check=5)        0.913     1335.098
Sptag(metric=L2, algo=KDT, check=200)        0.932     1767.731
Sptag(metric=L2, algo=BKT, check=100)        0.905     1009.242
Sptag(metric=L2, algo=KDT, check=2000)        0.952     1438.305
Sptag(metric=L2, algo=KDT, check=50)        0.913     1241.687
Sptag(metric=L2, algo=BKT, check=1000)        0.988      610.652
Sptag(metric=L2, algo=BKT, check=2000)        0.995      464.747
Sptag(metric=L2, algo=KDT, check=10)        0.913     1364.505
Sptag(metric=L2, algo=BKT, check=5)        0.903      907.239
Sptag(metric=L2, algo=KDT, check=1000)        0.952     1600.346
Sptag(metric=L2, algo=KDT, check=100)        0.912     1933.498
```

This looks reasonable good, given the high recall it achieves. (The influence of the `maxCheck` value doesn't really match what is written in https://github.com/microsoft/SPTAG/blob/master/docs/Parameters.md, though.) 

Index building is indeed very slow. (I ran the benchmark with `--batch` to allow multi-threaded index building.)
Repository: erikbern/ann-benchmarks
Issue: Colors
Root cause: Documentation Deficiency

@erikbern Your choices of colors make the charts really hard to read... and I am cannot fathom what it would be for a color blind person. It would be great if you can pick something with more contrast and possibly avoid shades of blue/green/orange. Thanks!


sure makes sense. i agree. i can fix


@erikbern Thank you my lord! 


Btw the data and the plot script is in the repo, so feel free to update it yourself :) Would love a PR!


I will try, though that removes a bit of the charm of having you handle all the setup ;)
And http://matplotlib.org/users/colormaps.html seems to have too many choices... 


you don't need any setup for the plotting, i think cloning the repo and running plot.py should be enough :)

but i can take a look at this too


better now? :)


Thank you++. I can see clearly now.
Repository: erikbern/ann-benchmarks
Issue: Add N2 ?
Root cause: Unimplemented Feature Gap

This one is promising:
https://github.com/kakao/n2

just glancing at the readme, definitely seems interesting.

@ummae – would you be interested in submitting a pull request to add n2 to the benchmarks?


@erikbern Okay, I will.

great @ummae ! no rush

Ping @ummae do you want to contribute with it? I'd love to re-run the benchmarks soon

@erikbern Sorry for late. This work is being delayed as we are preparing major update of N2. It will be updated in next 3-4 weeks (I hope so). It would be nice if anyone could contribute before then.
Repository: erikbern/ann-benchmarks
Issue:  Is there any plan to compare with Rayuela.jl ??

 Is there any plan to compare with Rayuela.jl ??

https://github.com/una-dinosauria/Rayuela.jl
This is the code for the paper
Julieta Martinez, Shobhit Zakhmi, Holger H. Hoos, James J. Little. LSQ++: Lower running time and higher recall in multi-codebook quantization. In ECCV 2018. [CVF].

Rayuela.jl is a package that implements non-orthogonal multi-codebook quantization methods (MCQ). These methods are useful for fast search of high-dimensional (d=100s-1000s) dense vectors. Rayuela is written in Julia.


@zsz00 i would love to add it. maybe the authors can take a look. @una-dinosauria ?

Given that it requires a GPU and doesn't have a Python wrapper, this would be a nice test case for our wrapper protocol https://github.com/erikbern/ann-benchmarks/tree/master/protocol :-)

@una-dinosauria I would be happy to help with that.

Hello everyone!

I'm super glad to see that you'd like to give my code a try. That said, I think it would be a bit hard for it to achieve competitive performance in this benchmark, as I have not implemented an IVF or any other shortlisting data structure -- In our ECCV18 paper, we assumed exhaustive search, as we compared only the encoding quality.

There would be two ways to get around this:

1. Implement a quick and dirty IVF into Rayuela (apparently someone implemented this in Julia already: https://github.com/una-dinosauria/Rayuela.jl/issues/37), or 
2. Port our method into a library that already has shortlisting structures, such as FAISS (IMO, preferrable, since IIUC they have implemented more recent structures too). Rayuela has a C++ version of the code update method, and the codebook update should be easy to port as it only calls BLAS a few times.

Let me know if anyone is interested in pursuing this, and I'd be happy to provide more pointers.
Repository: erikbern/ann-benchmarks
Issue: Redirect http://www.itu.dk/people/pagh/SSS/ann-benchmarks/ to ann-benchmarks.com

There was a hacker news thread linking to the itu.dk page: https://news.ycombinator.com/item?id=21374400

Probably makes sense to remove that page and redirect to ann-benchmarks.com or what do you think @maumueller @ale-f ?

Reminds me it's probably time to run another benchmark soon. There's been some additions lately.

Sure, made a redirect.

Awesome thanks!
Repository: erikbern/ann-benchmarks
Issue: Add Granne
Root cause: Unimplemented Feature Gap

https://github.com/granne/granne

Saw this from https://0x65.dev/blog/2019-12-07/indexing-billions-of-text-vectors.html

It optimizes for storage space: _Even though there are ANN libraries producing reasonable recall in less than 1 ms, for our use case we can accept sacrificing some of that speed for reduced server costs._

It seems that it only fits in their own business scenario, not in a general approximate nearest neighbor search.

@chasingegg curious if you can elaborate. what's different with the scenario they are trying to optimize for?
Repository: erikbern/ann-benchmarks
Issue: Compare both Index Build time and Query Time using some sort of combined metric?
Root cause: Unimplemented Feature Gap

Is there a known metric to combine both build time and query time into a metric, for those that wish to do approximate k nearest neighbor within a different dataset each time.

Hi

it would be trivial to add such a measure by adding something like <build time> + <time to answer all queries> to https://github.com/erikbern/ann-benchmarks/blob/master/ann_benchmarks/plotting/metrics.py and plotting this against recall in https://github.com/erikbern/ann-benchmarks/blob/master/ann_benchmarks/plotting/plot_variants.py. 

However, I'm not sure if it provides much more information than inspecting the recall/build time plot on the web site (e.g., http://ann-benchmarks.com/glove-100-angular_10_angular.html).

Martin
Repository: erikbern/ann-benchmarks
Issue: plot.py does not plot 
Root cause: Validation Gap

I used python plot.py --dataset=glove-100-angular to plot the results after the simulations are done. The results are in results/glove-100-angular/10. 
But it does not plot. Instead, it says  Exception: Nothing to plot
I tracked into compute_metrics in utils.py and found that the res is empty.  
Can you help me out?  


Could you try to run `plot.py` with `sudo`? I think the plotting script writes to the result files for caching and might run into permission errors. 

Thanks a lot for your very quick response. Yes, that is cause. I can either run with a sudo or remove the "+" in the following line in results.py 

f = h5py.File(os.path.join(root, fn), 'r+')




Another problem.  I guess this may be some env setup issue 

Traceback (most recent call last):
  File "plot.py", line 139, in <module>
    linestyles, args.batch)
  File "plot.py", line 28, in create_plot
    for algo in sorted(all_data.keys(), key=lambda x: x.lower()):
TypeError: '<' not supported between instances of 'str' and 'bytes'





Later on I found I should not remove "+" in that line. 
Anyway, all problem resolved now.
Repository: erikbern/ann-benchmarks
Issue: Add Milvus
Root cause: Unimplemented Feature Gap

https://github.com/milvus-io/milvus

"Milvus -- the world's fastest vector search engine." would love to verify that claim :)

Hey Erik, 

I am from Milvus team:) Big fan. Much appriaciate your work in ANN, really helped us a lot when we build Milvus. We are honored that you noticed our work and look forward to your verification. 

We'd love to talk more and assist the test if needed, let us know:)

Regards,
Ryan



@RyanWei take a look at #149!
Repository: erikbern/ann-benchmarks
Issue: Add vearch
Root cause: Unimplemented Feature Gap

"Vearch is a scalable distributed system for efficient similarity search of deep learning vectors."

Jie Li, Haifeng Liu, Chuanghua Gui, Jianyu chen, Zhenyun Ni, Ning Wang, Yuan Chen. The Design and Implementation of a Real Time Visual Search System on JD E-commerce Platform. In the 19th International ACM Middleware Conference, December 10–14, 2018, Rennes, France. https://arxiv.org/abs/1908.07389

https://github.com/vearch/vearch
Repository: erikbern/ann-benchmarks
Issue: Consider adding a 'deep image descriptor' dataset
Root cause: Unimplemented Feature Gap

In the past few years, it seems the computer vision community has expanded out from SIFT descriptors and started working with 'deep descriptors' of images from convolutional neural nets. It would be helpful to include a dataset of deep image descriptors to more closely represent this approach to image similarity search.

One idea is to take a sample from [this large dataset](http://sites.skoltech.ru/compvision/noimi/), which consists of images from ImageNet run through the a CNN to produce vectors. Some prominent kNN papers report results against this dataset, including [Billion-scale similarity search with GPUs](https://arxiv.org/abs/1702.08734) and [Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://arxiv.org/abs/1603.09320).


That's a good idea. I think the full billion is probably too much, but subsampling a few million sounds like a great idea. Maybe 10M of them (10M*96*4 = ~4GB, which should fit well into memory)

I would be happy to open a PR to help get this started, aiming for a 10M vector subsample.

sure, would definitely appreciate it

I think this would be an excellent addition!

@erikbern We might want to rethink allow multi-threaded index building for such data set sizes. None of the graph-based indexes will complete within the 6 hour preprocessing time limit.

@maumueller i think the plan was to use a subsample of a few million, not the full billion dataset

but sure, i'm ok enabling multithreading. let me think about how that would affect single threaded libraries during searching. i believe we used to have a thread pool to saturate all cpus

I was referring to the idea of having 10M. (I'm currently experimenting with 3M 300d and it takes about 20 hours to build an HNSW index for some of its parameter choices.)

we should also consider some reasonable timeout, like maybe 1-3h.