Apply for community grant: Academic project (gpu)
Dear Huggingface team,
We are building since one year, a new benchmarking framework called "clembench" and rank LLMs based on their performance, see https://clembench.github.io/leaderboard
Different than ChatArena our framework is built on the idea of making LLMs play dialogue games so we don't require any human annotation.
We are comparing open-weight LLMs on HuggingFace with commercial ones via APIs.
I'd like to build a space on HF that would enable a user (restricted to someone from our team, https://huggingface.co/colab-potsdam) to select a model and run the benchmark with it. So we need usually A100 GPUs to run the benchmark and it is getting much difficult to acquire such resources as interesting models drop almost every week.
We have already built our Multimodal Clem Leaderboard and will continue benchmarking interesting text-only and multimodal models on HuggingFace.
We will gladly acknowledge any support or grant on our website as well as in future publications.
Kind regards,
Sherzod Hakimov
Hi Sherzod,
For leaderboards, we usually suggest finding a partner which will be able to sponsor the compute - we can do a one month grant (on a smaller GPU) so you get the time to setup your backend, but we won't sponsor for longer than this.