|
from dataclasses import dataclass |
|
from enum import Enum |
|
|
|
@dataclass |
|
class Task: |
|
benchmark: str |
|
metric: str |
|
col_name: str |
|
|
|
|
|
|
|
|
|
class Tasks(Enum): |
|
|
|
task0 = Task("mmlu_it", "acc", "MMLU_IT") |
|
task1 = Task("arc_it", "acc_norm", "ARC_IT") |
|
task2 = Task("hellaswag_it", "acc_norm", "HELLASWAG_IT") |
|
|
|
NUM_FEWSHOT = 0 |
|
|
|
|
|
|
|
|
|
|
|
TITLE = """<h1 align="center" id="space-title">🚀 Classifica generale degli LLM italiani 🚀</h1>""" |
|
|
|
|
|
INTRODUCTION_TEXT = """ |
|
Benvenuti nella pagina della Open ITA LLM Leaderboard! |
|
In questa dashboard potrete trovare tutti i risultati delle performance dei Large Language Models in lingua italiana sui principali test di valutazione, effettuati grazie alla [Eleuther AI Language Model Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness) |
|
Maggiori informazioni sono disponibili nella sezione "About". |
|
|
|
La classifica è completamente open source e chiunque può contribuire aggiungendo il proprio modello tramite l'apposita sezione "Submit". |
|
Le valutazioni sono automatizzate e i risultati saranno disponibili entro un paio d'ore. Questo è reso possibile grazie alla GPU A100 di [Seeweb](https://www.seeweb.it/prodotti/cloud-server-gpu). |
|
Per gli LLM di dimensioni superiori a 35B, si prega di inviare la richiesta tramite questo [form](https://forms.gle/Gc9Dfu52xSBhQPpAA). |
|
Se avete idee, miglioramenti o suggerimenti, non esitate a [contattarmi](https://www.linkedin.com/in/samuele-colombo-ml/) oppure trovatemi sul [server Discord della community](https://discord.gg/kc97Zwc4ze). |
|
""" |
|
|
|
|
|
LLM_BENCHMARKS_TEXT = f""" |
|
## Come funziona |
|
Valutiamo i modelli tramite <a href="https://github.com/EleutherAI/lm-evaluation-harness" target="_blank"> Eleuther AI Language Model Evaluation Harness </a>, il framework più utilizzato dalla community internazionale per l'evaluation dei modelli |
|
Nella classifica troverete i dataset di benchmark più famosi, adatti alla lingua italiana. I task sono: |
|
- <a href="https://huggingface.co/datasets/alexandrainst/m_hellaswag" target="_blank"> hellaswag_it </a> |
|
- <a href="https://huggingface.co/datasets/alexandrainst/m_arc" target="_blank"> arc_it </a> |
|
- <a href="https://huggingface.co/datasets/alexandrainst/m_mmlu" target="_blank"> m_mmlu_it </a> (5 shots) |
|
|
|
Per tutti questi task, a un punteggio migliore corrisponde una performance maggiore |
|
|
|
|
|
## Reproducibility |
|
Per riprodurre i risultati scaricate la <a href="https://github.com/EleutherAI/lm-evaluation-harness" target="_blank"> Eleuther AI Language Model Evaluation Harness </a> ed eseguite: |
|
* lm-eval --model hf --model_args pretrained=<vostro modello> --tasks hellaswag_it,arc_it --device cuda:0 --batch_size auto:2; |
|
* lm-eval --model hf --model_args pretrained=<vostro modello>, --tasks m_mmlu_it --num_fewshot 5 --device cuda:0 --batch_size auto:2 |
|
""" |
|
|
|
EVALUATION_QUEUE_TEXT = """ |
|
## Some good practices before submitting a model |
|
|
|
### 1) Make sure you can load your model and tokenizer using AutoClasses: |
|
```python |
|
from transformers import AutoConfig, AutoModel, AutoTokenizer |
|
config = AutoConfig.from_pretrained("your model name", revision=revision) |
|
model = AutoModel.from_pretrained("your model name", revision=revision) |
|
tokenizer = AutoTokenizer.from_pretrained("your model name", revision=revision) |
|
``` |
|
If this step fails, follow the error messages to debug your model before submitting it. It's likely your model has been improperly uploaded. |
|
|
|
Note: make sure your model is public! |
|
Note: if your model needs `use_remote_code=True`, we do not support this option yet but we are working on adding it, stay posted! |
|
|
|
### 2) Convert your model weights to [safetensors](https://huggingface.co/docs/safetensors/index) |
|
It's a new format for storing weights which is safer and faster to load and use. It will also allow us to add the number of parameters of your model to the `Extended Viewer`! |
|
|
|
### 3) Make sure your model has an open license! |
|
This is a leaderboard for Open LLMs, and we'd love for as many people as possible to know they can use your model 🤗 |
|
|
|
### 4) Fill up your model card |
|
When we add extra information about models to the leaderboard, it will be automatically taken from the model card |
|
|
|
## In case of model failure |
|
If your model is displayed in the `FAILED` category, its execution stopped. |
|
Make sure you have followed the above steps first. |
|
If everything is done, check you can launch the EleutherAIHarness on your model locally, using the above command without modifications (you can add `--limit` to limit the number of examples per task). |
|
""" |
|
|
|
CITATION_BUTTON_LABEL = "Copy the following snippet to cite these results" |
|
CITATION_BUTTON_TEXT = r""" |
|
Paper coming soon! |
|
Pls dateci credits se usate i nostri benchmarks :) |
|
""" |
|
|