Repository: erikbern/ann-benchmarks
Issue: Support 'logit' scale for x-axis
Root cause: Unimplemented Feature Gap

The matplotlib `logit` scale is very nice when plotting values between 0 and 1. In particular, this fixes the current problem of plots being incomprehensible in the "high recall" regime.

I suggest changing the `--x-log` and `--y-log` parameters from `plot.py` to `--x-scale` and `--y-scale` taking parameters for plugging directly into `matplotlib`. This will also support the `symlog` scale.

I have a commit with this change here: https://github.com/thomasahle/ann-benchmarks/commit/3bf07f41715a8cfe64b4c7c7bfdaa46027b2892b

But will that work when the accuracy is exactly 1 or 0? Won't those be projected to +-infinity?

Do you have a screenshot you can show?

I haven't run all the algos, but I do think the logit plot makes things easier to read:
![glove-100-angular](https://user-images.githubusercontent.com/946355/104250962-02cca080-546f-11eb-9698-16b52cd32a8a.png)
In particular, I think for a plot like http://ann-benchmarks.com/glove-25-angular_10_angular.html that is currently incomprehensible.

You are right that the brute force algorithm with its recall=1 is a problem...
Perhaps one could make a custom xscale that adds 0 and 1 to logit by surgery...

I think something that warps [0, 1] using roughly `x^10` might also be OK. That would project 0.9 to 0.35 and 0.99 to 0.9

Not sure how hard it is to implement a custom scale. I looked at it a while ago and it seemed like a fair amount of boiler plate

It turns out to be [pretty easy in matplotlib 3.3](https://mpltest2.readthedocs.io/en/stable/users/next_whats_new/2018-11-25-JMK.html) - and upgrading doesn't break anything in ann-benchmarks.

    alpha = 3
    ax.set_xscale('function', functions=(
        (lambda x: 1-(1-x)**(1/alpha)),
        (lambda x: 1-(1-x)**alpha)))

This produces the following output:

![glove-100-angular-logit](https://user-images.githubusercontent.com/946355/104489017-75549200-55cf-11eb-8365-929b17871129.png)

Varying alpha can "stretch" the high-recall regime more or less.

It might be nice to fix the ticks as well, so there isn't a big blank space.

This is the standard linear scale for comparison.

![glove-100-angular](https://user-images.githubusercontent.com/946355/104518323-ddb76980-55f7-11eb-9859-9c4196a153e2.png)


Nice! I think we might want to throw in minor gridlines, but looks pretty good!!

This is alpha=2.

With minor grid lines:

![glove-100-angular-logit](https://user-images.githubusercontent.com/946355/104589321-f7919480-5669-11eb-8bb5-4623ebbe1114.png)

With evenly spaced major gridlines:

![glove-100-angular-logit](https://user-images.githubusercontent.com/946355/104589250-daf55c80-5669-11eb-8244-cbf246c7c41c.png)

(not sure the minor gridlines make sense here, since you can't guess what values they represent.)

Are your result hdf5 files available for download? Then I could check what looks better for the full plot (without having to run days of benchmarks myself :))

Hi Thomas

You can access the groundtruth for glove-100-angular via ` aws s3 cp --recursive s3://ann-benchmarks.com/results/2020-07-13/glove-100-angular/ .`. 

I would also like to see it with a bit more algorithms. I honestly don't know how important it is to put that much focus on the high recall regime. I think the linear-scale is still easier to interpret, but I'm clearly biased after having looked at these plots for years :-) 

Hi Martin, I get `fatal error: Unable to locate credentials`.

I see, I hope this will work: http://ann-benchmarks.com/results/glove-100-angular.zip

It somehow doesn't find those files. It seems to be because they don't have the suffix hdf5. Maybe my ann-benchmarks version is too new?
E.g. in the `glove-100-angular.zip` zip file I find `glove-100-angular/10/puffinn/angular_1073741824_fht_crosspolytope_0_1`, but when I run ann-benchmarks myself, it generates files like `glove-100-angular-mine/10/scann/1000_0_2_1_1_10.hdf5`.

Oh, right. That was a recent change. Just rename all of them to contain the suffix. 

Alright, here you go.
Certainly a bit more uniform in terms of where "the action" is in the plot.
If you upload the data, I can test glove-25 too, on which the effect should be even stronger.

![glove-100-angular-logit](https://user-images.githubusercontent.com/946355/104648154-094b5a00-56b3-11eb-86b4-6c73419ffeca.png)


I can do that tomorrow (or Erik might be able to tell you how to get s3 to work ;-)) 

Alright, here is alpha 3, 2, 1 (same as linear) and logit for glove-25.

![glove-25-angular_a3](https://user-images.githubusercontent.com/946355/104755311-8e3d7e80-575a-11eb-848c-d324ff1b5063.png)
![glove-25-angular_a2](https://user-images.githubusercontent.com/946355/104755317-909fd880-575a-11eb-9f95-aaf646a2946f.png)
![glove-25-angular_alpha=1](https://user-images.githubusercontent.com/946355/104755319-91386f00-575a-11eb-80c3-9e3350b1ba12.png)
![glove-25-angular_logit](https://user-images.githubusercontent.com/946355/104755321-91d10580-575a-11eb-9db4-b2b451bc1503.png)

The plots make it clear that there is a lot of difference between the algorithms in the high recall regime. 
I probably prefer alpha=3 here, but alpha=2 may be the best compromise for all the datasets.
One could also consider having it as a user-controlled option in the interactive plots.
For now I'm just suggesting adding the option to the `plot.py` script.

I also tried doing alpha=4 with axis labels looking more like the logit plot:

![glove-25-angular](https://user-images.githubusercontent.com/946355/104757735-b9759d00-575d-11eb-8c86-e80c73c2b9d5.png)


Nice, I think this looks really great. Happy to merge!

Any chance of getting this actually on annbenchmarks.com? :-)

Sure, please update `create_website.py` to use these plots as well.

Would be great to have good defaults in the website create script.

I guess it's time to rerun the benchmarks soon!