Spaces:

autoevaluate

/

leaderboards

like 137

Running

Yes that could be cool. What do you think about this:

A button that says "subscribe to this leaderboard". When the button is clicked, it asks you to log in if you are not logged in. If you are logged in, then it will subscribe you to an email system that emails people when a new model moves to the top, for any metric. Users can then unsubscribe by returning to the leaderboard where the "subscribe" button will be replaced with an "unsubscribe" button.

There would likely be a decent amount of noise (e.g., if someone briefly enters self reported results that are incorrect and top the leaderboard, everyone will be notified), but people can always unsubscribe.

Edit: I've found out that it's not possible to get account info from spaces, so I'll have to have an input field for people to enter their emails directly.

Perhaps it would be nice to allow options for the frequency of notifications: 1/day, 1/week, 1/month

Oh yeah that is a great idea. Also, if we are going to use a free email api at first to see how things go, they tend to have limits of a few hundred emails a day.

Could we use the same email service that sends out discussion notification emails?

notifications@huggingface.co? Do you know who I reach out to about this?

I do not. Sorry

Thanks for sharing this feature request @nbroad !

Since this is something that might eventually be implemented on the AutoTrain side and there doesn't seem to be a strong need from the community (yet), I suggest we first focus our efforts on improving the leaderboards in more impactful ways (e.g. grouping the results by task).

WDYT @Tristan ?

I'm down to go with what you think is best @lewtun ! Regarding doing this on the AutoTrain side, though: if we did that, we might have to duplicate some of the leaderboards logic in AutoTrain to see if a model's results top the leaderboard for any metric.