Repository: erikbern/ann-benchmarks
Issue: dbpedia-openai-*k-angular datasets return 403
Root cause: External Dependency Breakage

Hi, it looks like the `dbpedia-openai-*k-angular` datasets are still returning 403 (#457).

```text
downloading http://ann-benchmarks.com/dbpedia-openai-100k-angular.hdf5 -> data/dbpedia-openai-100k-angular.hdf5...
Cannot download http://ann-benchmarks.com/dbpedia-openai-100k-angular.hdf5
```

Ok, will fix

Thanks @erikbern. Also found an issue with the build script, which is fixed in #462.

@maumueller @erikbern is there any blocker for releasing the dataset? If yes, is there anything I can do to help? :)

Thanks @maumueller for your quick response :smile: 

Curious to know what's the process for:
- Making the dataset available as a .hdf5 file on ann-benchmarks.com (i.e. resolving this issue)
- Re-running the benchmark at 1M scale with all the algorithms and publishing on the website. 1M is a good number for most practical purposes so this will be a great value addition to the community.

Let me know if you want me to do something to save you time and effort :)

@KShivendu Did you do a run on your machine to see if the results diverge from the current trends that we have observed? Also, is there a empirical evidence for having 10 different versions of the dataset around? We don't do it for the other datasets, only for the synthetic test datasets.

@erikbern I think we should only release the 1M dataset, could you put that on s3? 

  > Did you do a run on your machine to see if the results diverge from the current trends that we have observed?
  
I recently benchmarked Qdrant and Pgvector, you can find the results [here](https://nirantk.com/writing/pgvector-vs-qdrant/). We found some interesting results.
  
  > Also, is there empirical evidence for having 10 different versions of the dataset around?
  
The idea was to understand when different algorithms tend to plateau or go down when using OpenAI embeddings.  We can add another set of graphs for QPS/Recall vs. the number of vectors. However, that would require more changes. So we can do it in the next release and publish the results with the 1M dataset in the meantime.
  
What do you think? :)  @maumueller 

> > Did you do a run on your machine to see if the results diverge from the current trends that we have observed?

> I recently benchmarked Qdrant and Pgvector, you can find the results [here](https://nirantk.com/writing/pgvector-vs-qdrant/). We found some interesting results.

Ok, I agree that this could be an interesting use case, but it's typically not the type of experiment we publish results on. @ankane  provided pgvector with hnsw as a PR, so the results look probably different now.  :-) 

(On a more general note, it's a bit odd that IVF would provide linear slow-down from 100k->1M, I wonder which parameters were used for this experiment. Were it the pre-hnsw pgvector ones from ann-benchmark?)

I'd like to go forward with keeping all versions in the codebase, and publishing results for 1M. 

> Were it the pre-hnsw pgvector ones from ann-benchmark

Yes. 

> I'd like to go forward with keeping all versions in the codebase, and publishing results for 1M.

Sure :)

> publishing results for 1M.

Great. Excited to see how it goes :D

I don't think the dbpedia files were ever present in ann-benchmarks.com?