Repository: erikbern/ann-benchmarks
Issue: Can't you just provide ranking in a table?
Root cause: Unimplemented Feature Gap

This project is awesome and great but I was wondering couldn't you also provide a table with final rankings? Use whatever criteria you like but would be good to have a tabular scorecard.

on another note, it seems the best performing algorithm is qsgngt which has a mere 7 stars on github. how does one explain that?

It's sort of hard to rank though – you would have to make some choices about what recall is most important etc

i understand that Erik but if you look anywhere, ultimately people doing
benchmarking provide a leaderboard which acts as Tl;DR e.g., see hugging
face leaderboard, GPU leaderboard, big-ann-benchmarking leaderboard etc.
that's why i said choose whatever criteria you like but provide a
leaderboard with unambiguous ranking. anyway just my 2c.

On Wed, Jul 5, 2023 at 9:34 PM Erik Bernhardsson ***@***.***>
wrote:

> It's sort of hard to rank though – you would have to make some choices
> about what recall is most important etc
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/erikbern/ann-benchmarks/issues/437#issuecomment-1622961981>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/A6NWEK5EMI4VLJJYG5JCSGTXOY56FANCNFSM6AAAAAAZ7TH3DY>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>


the other reason i asked for leaderboard is that i tried looking at the
graphs. there is so much overlap amongst the curves that its hard to read
the graphs and make out which model is better.

On Thu, Jul 6, 2023 at 9:54 AM Siddharth Jain ***@***.***> wrote:

> i understand that Erik but if you look anywhere, ultimately people doing
> benchmarking provide a leaderboard which acts as Tl;DR e.g., see hugging
> face leaderboard, GPU leaderboard, big-ann-benchmarking leaderboard etc.
> that's why i said choose whatever criteria you like but provide a
> leaderboard with unambiguous ranking. anyway just my 2c.
>
> On Wed, Jul 5, 2023 at 9:34 PM Erik Bernhardsson ***@***.***>
> wrote:
>
>> It's sort of hard to rank though – you would have to make some choices
>> about what recall is most important etc
>>
>> —
>> Reply to this email directly, view it on GitHub
>> <https://github.com/erikbern/ann-benchmarks/issues/437#issuecomment-1622961981>,
>> or unsubscribe
>> <https://github.com/notifications/unsubscribe-auth/A6NWEK5EMI4VLJJYG5JCSGTXOY56FANCNFSM6AAAAAAZ7TH3DY>
>> .
>> You are receiving this because you authored the thread.Message ID:
>> ***@***.***>
>>
>


yeah I hear you – I've been thinking about this for a long time too, I just can't come up with something obvious. But if you have any thoughts let me know!

Separately agree on the graphs being too messy. I'll try to clean that up

one possibility is to compute the AUC - area under the curve - and use that
as the metric for the leaderboard. we can have 2 leaderboards - one for
angular and another for euclidean. choose any dataset you like.

one thing i don't understand is https://github.com/WPJiang/HWTL_SDU-ANNS
library comes out at the top in the graphs - however it has only 7 github
stars and it itself relies on ngt/qbg libraries to do the work. so why
don't ngt/qbg rank equal or better than
https://github.com/WPJiang/HWTL_SDU-ANNS? is it because of parameter tuning?

i just came across this benchmarking suite yesterday so pardon if i am
asking noob questions.

On Thu, Jul 6, 2023 at 10:17 AM Erik Bernhardsson ***@***.***>
wrote:

> yeah I hear you – I've been thinking about this for a long time too, I
> just can't come up with something obvious. But if you have any thoughts let
> me know!
>
> Separately agree on the graphs being too messy. I'll try to clean that up
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/erikbern/ann-benchmarks/issues/437#issuecomment-1624034707>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/A6NWEK3JHZWMFECJWBVGWR3XO3XJTANCNFSM6AAAAAAZ7TH3DY>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>


ok, I didn't realize that – maybe we should remove HWTL_SDU-ANNS then? we definitely shouldn't include wrappers. I've been quite liberal accepting contributions