Repository: erikbern/ann-benchmarks
Issue: Any plan to test even higher dimensional data?
Root cause: Unimplemented Feature Gap

It'll be very interesting to see the comparisons between different suitable methods for higher dimensional data (e.g., several thousands or even larger).

Thank you.


might actually run a test with 1000D... was thinking about that. would be fun


Why not trying sparse TF*IDF?


Does any of the libraries support sparse data?

Btw I tried running a 1000D dataset (word2vec on freebase) yesterday but the code isn't very memory efficient so I had to rent a r3.8xlarge (244GB)... was super slow and unfortunately my spot instance was killed later. I think I need to fix up the scripts before I rerun the experiments.


I am not sure how many libraries support sparse data. Anyways, if you are interested in more dimensions, here's probably a good data set:
http://disa.fi.muni.cz/profiset/
Disclaimer: I didn't use it yet, but looks promising.


I think higher-dimensional dense data (say around 1000 dimensions) and very high-dimensional sparse data (say 10k - 100k dimensions) would both be very interesting benchmarks.

FALCONN (added in #16 ) supports sparse data. If other libraries also support sparse data, @ilyaraz and I can help with setting up a benchmark.


@ludwigschmidt AFAIK, currently only two libraries support sparse spaces.


But I would be all for some more tests with weird spaces at least l1 or lp with the fractional p < 1.


We should test for 1000D at least. But need to improve the script a bit first.


I think it would be also be interesting to consider a 1000D dataset without increasing the available memory by 10x. That would test the libraries in the regime where less memory is available.


Sure. If someone has time, would really appreciate it if you can make the scripts work with larger data set. Right now it's quite inefficient – it reads the entire data set into memory (using double precision) – then uses `sklearn.cross_validation` to split it up. It's essentially using 5x more memory than what should be needed (2x because of double precision, 2x because we store the original data and the train/test split at the same time, then another 1x because we feed it into the algorithm). It would be much better if we could just build up the index incrementally for instance.


@ludwigschmidt while measuring memory footprint is important (and we do it automatically in NMSLIB), I would be against arbitrary memory constraints. 

For one thing, this may become an exercise in data compression for methods that have small memory footprint already (e.g., for some still good parameters SW-graph uses 20-40% of what the data set uses, even less for some high dimensional data sets). And for another, you have to consider how quickly retrieval time grows with the number of points. One may have a large index, but the method may scale better with a data set size. There may be other subtleties that I don't remember off the top of my head.

Taking all such factors into account, makes testing too complicated (but you would surely need to do something like this, if you plan to deploy your method on 1000 machines). That said, I think that testing more diverse data is a valuable exercise. 


The benchmark now includes results for a dataset with 960 dimensions, so I'll close this!