Repository: huggingface/lighteval
Issue: finetasks for thai language not found
Root cause: Documentation Deficiency

Hi,

I want to use lighteval to evaluate my model on the same task as fine tasks used with the Thai language

```
# General Knowledge (GK)
lighteval|meta_mmlu_tha_mcf|5|1
lighteval|m3exams_tha_mcf|5|1

# Reading Comprehension (RC)
lighteval|belebele_tha_Thai_mcf|5|1
lighteval|thaiqa_tha|5|1
lighteval|xquad_tha|5|1

# Natural Language Understanding (NLU)
lighteval|community_hellaswag_tha_mcf|5|1
lighteval|xnli2.0_tha_mcf|5|1
```

Unfortunately, it seems that none of this has been registered to the task list, which is very strange given that there is this blog 
https://huggingface.co/spaces/HuggingFaceFW/blogpost-fine-tasks

Here is my command
`export CUDA_VISIBLE_DEVICES="1" 
lighteval accelerate "pretrained=wannaphong/mark3" "lighteval|meta_mmlu_tha_mcf|5|1" --save-details`

Here is the error
`ValueError: Cannot find tasks lighteval|meta_mmlu_tha_mcf in task list or in custom task registry)`

I used lighteval 0.7.0, PyTorch version 2.2.2+cu121


Hi!
If you want to evaluate your models on the same task as the blog, you need to use the same command:
```
lighteval accelerate\
    --model_args vllm,pretrained=model_name,pairwise_tokenization=True \
    --custom_task lighteval.tasks.multilingual.tasks \
    --tasks 'examples/tasks/finetasks/{cf,mcf}/tha' \
    --max_samples '1000'
```

Hi, thanks for the answer.
Before I wrote this issue, I had tried to command your suggestion and It also did not work.
And I think there are no `--model_args` and `--tasks` flags according to `--help`

Can we reopen this issue?

Ha, you are right, we need to update the blog, it should be
```
lighteval accelerate\
     vllm,pretrained=model_name,pairwise_tokenization=True \
     'examples/tasks/finetasks/{cf,mcf}/tha' \
    --custom_task lighteval.tasks.multilingual.tasks \
    --max_samples '1000'
```
now, cc @hynky1999 

What was the issue you had with the above command?

Thanks for your suggestion

I don't think this version is correct either

> Ha, you are right, we need to update the blog, it should be
> 
> ```
> lighteval accelerate\
>      vllm,pretrained=model_name,pairwise_tokenization=True \
>      'examples/tasks/finetasks/{cf,mcf}/tha' \
>     --custom_task lighteval.tasks.multilingual.tasks \
>     --max_samples '1000'
> ```
> 
> now, cc [@hynky1999](https://github.com/hynky1999)

I got this error 
`ValueError: Cannot get task info from examples/tasks/fine_tasks/mcf/tha.txt. correct format is suite|task|few_shot|truncate_few_shots`

Maybe this one is close to correct
```
lighteval accelerate "pretrained=wannaphong/mark3,dtype=bfloat16" "examples/tasks/fine_tasks/mcf/th.txt" --custom-tasks "src/lighteval/tasks/multilingual/tasks.py" --output-dir "./demorun/"
```

But I still got error from this command

`DatasetNotFoundError: Dataset 'meta-llama/Meta-Llama-3.1-8B-Instruct-evals' is a gated dataset on the Hub. Visit the dataset page at `


FYI, I didn't use pip to install lighteval. I clone the repo to my local and to `pip install -e .[dev]`

It's because the Meta MMLU requires you to request access to the above dataset which is gated

Yes, I requested the token permitted to use that dataset from my mentor. I don't get that error now but I don't know if the command is still able to run

I use this version now 
```
export CUDA_VISIBLE_DEVICES="1"
lighteval accelerate \
    "pretrained=wannaphong/mark3,dtype=bfloat16" \
    "examples/tasks/fine_tasks/mcf/th.txt" \
    --custom-tasks "src/lighteval/tasks/multilingual/tasks.py" \
    --output-dir "./demorun/"
```

this is the output from my terminal right now

```
(lighteval) root@1ef399243353:/workspace/lighteval# bash demorun.sh
[2025-01-23 08:44:13,713] [    INFO]: PyTorch version 2.2.2+cu121 available. (config.py:54)
[2025-01-23 08:44:19,722] [ WARNING]: Detected kernel version 3.10.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher. (other.py:335)
[2025-01-23 08:44:19,728] [ WARNING]: Detected kernel version 3.10.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher. (other.py:335)
[2025-01-23 08:44:19,728] [    INFO]: Test gather tensor (parallelism.py:133)
[2025-01-23 08:44:21,394] [    INFO]: gathered_tensor tensor([0], device='cuda:0'), should be [0] (parallelism.py:136)
[2025-01-23 08:44:21,395] [    INFO]: --- LOADING MODEL --- (pipeline.py:168)
[2025-01-23 08:44:22,134] [    INFO]: Tokenizer truncation and padding size set to the left side. (transformers_model.py:539)
[2025-01-23 08:44:22,136] [    INFO]: Setting model parallel to False since the number of local processes is 1 and the number of GPUs is 1 (transformers_model.py:402)
[2025-01-23 08:44:22,136] [    INFO]: Model parallel was set to False, max memory set to None and device map to None (transformers_model.py:423)
[2025-01-23 08:44:23,397] [    INFO]: Using Data Parallelism, putting model on device cuda (transformers_model.py:275)
[2025-01-23 08:44:23,801] [    INFO]: --- LOADING TASKS --- (pipeline.py:195)
[2025-01-23 08:44:23,976] [ WARNING]: No word tokenizer found for language Language.ASSAMESE, will split on spaces. (linguistic_tokenizers.py:276)
[2025-01-23 08:44:23,977] [ WARNING]: No word tokenizer found for language Language.ORIYA, will split on spaces. (linguistic_tokenizers.py:276)
[2025-01-23 08:44:23,977] [ WARNING]: No word tokenizer found for language Language.PUNJABI, will split on spaces. (linguistic_tokenizers.py:276)
[2025-01-23 08:44:25,723] [ WARNING]: No word tokenizer found for language Language.KHMER, will split on spaces. (linguistic_tokenizers.py:276)
[2025-01-23 08:44:25,807] [ WARNING]: If you want to use extended_tasks, make sure you installed their dependencies using `pip install -e .[extended_tasks]`. (registry.py:136)
[2025-01-23 08:44:25,808] [    INFO]: Found 34113 custom tasks in /root/.cache/huggingface/modules/datasets_modules/datasets/tasks/c5d5805545a70cdcae6337a7808e296324fd87f8c43d8dacbd23315d3f363f13/tasks.py (registry.py:141)
[2025-01-23 08:44:25,912] [    INFO]: facebook/belebele tha_Thai (lighteval_task.py:187)
[2025-01-23 08:44:25,912] [ WARNING]: Careful, the task lighteval|belebele_tha_Thai_mcf is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,912] [    INFO]: lighteval/hellaswag_thai default (lighteval_task.py:187)
[2025-01-23 08:44:25,912] [    INFO]: chiayewken/m3exam thai (lighteval_task.py:187)
[2025-01-23 08:44:25,912] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,912] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:abstract_algebra is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,912] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,912] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:anatomy is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,912] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,913] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:astronomy is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,913] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,913] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:business_ethics is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,913] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,913] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:clinical_knowledge is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,913] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,913] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:college_biology is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,913] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,913] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:college_chemistry is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,913] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,913] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:college_computer_science is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,913] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,914] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:college_mathematics is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,914] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,914] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:college_medicine is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,914] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,914] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:college_physics is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,914] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,914] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:computer_security is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,914] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,914] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:conceptual_physics is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,914] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,914] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:econometrics is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,914] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,914] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:electrical_engineering is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,915] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,915] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:elementary_mathematics is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,915] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,915] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:formal_logic is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,915] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,915] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:global_facts is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,915] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,915] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:high_school_biology is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,915] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,915] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:high_school_chemistry is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,915] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,915] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:high_school_computer_science is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,915] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,916] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:high_school_european_history is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,916] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,916] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:high_school_geography is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,916] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,916] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:high_school_government_and_politics is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,916] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,916] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:high_school_macroeconomics is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,916] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,916] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:high_school_mathematics is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,916] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,916] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:high_school_microeconomics is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,916] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,917] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:high_school_physics is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,917] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,917] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:high_school_psychology is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,917] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,917] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:high_school_statistics is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,917] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,917] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:high_school_us_history is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,917] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,917] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:high_school_world_history is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,917] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,917] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:human_aging is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,918] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,918] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:human_sexuality is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,918] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,918] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:international_law is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,918] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,918] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:jurisprudence is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,918] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,918] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:logical_fallacies is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,918] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,918] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:machine_learning is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,918] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,918] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:management is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,918] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,919] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:marketing is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,919] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,919] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:medical_genetics is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,919] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,919] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:miscellaneous is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,919] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,919] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:moral_disputes is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,919] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,919] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:moral_scenarios is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,919] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,919] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:nutrition is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,919] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,919] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:philosophy is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,920] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,920] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:prehistory is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,920] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,920] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:professional_accounting is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,920] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,920] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:professional_law is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,920] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,920] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:professional_medicine is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,920] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,920] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:professional_psychology is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,920] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,920] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:public_relations is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,920] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,920] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:security_studies is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,921] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,921] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:sociology is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,921] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,921] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:us_foreign_policy is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,921] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,921] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:virology is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,921] [    INFO]: meta-llama/Meta-Llama-3.1-8B-Instruct-evals Llama-3.1-8B-Instruct-evals__multilingual_mmlu_th__details (lighteval_task.py:187)
[2025-01-23 08:44:25,921] [ WARNING]: Careful, the task lighteval|meta_mmlu_tha_mcf:world_religions is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,921] [    INFO]: lighteval/thaiqa_squad_fixed default (lighteval_task.py:187)
[2025-01-23 08:44:25,921] [    INFO]: Harsit/xnli2.0_train_thai default (lighteval_task.py:187)
[2025-01-23 08:44:25,921] [ WARNING]: Careful, the task lighteval|xnli2.0_tha_mcf is using evaluation data to build the few shot examples. (lighteval_task.py:261)
[2025-01-23 08:44:25,921] [    INFO]: google/xquad xquad.th (lighteval_task.py:187)
(…)_2024-07-22T22-35-33.566039.parquet.gzip: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 10.8M/10.8M [00:00<00:00, 24.6MB/s]
Generating latest split: 14042 examples [00:00, 33553.72 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12151.24 examples/s]
Filter: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 9160.64 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12225.81 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12145.21 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12204.30 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12366.43 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12348.11 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12119.81 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12172.91 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12099.65 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12237.48 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12169.27 examples/s]
Filter: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 8141.09 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12177.41 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12257.76 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12096.77 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12142.72 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12332.57 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12111.44 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12291.62 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12231.83 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12226.39 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12339.79 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12259.71 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12399.20 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12331.06 examples/s]
Filter: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 8325.80 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12232.56 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12337.34 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12261.35 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12303.58 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12334.32 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12310.85 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12069.07 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12074.29 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12310.54 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12147.31 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12386.47 examples/s]
Filter: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 9290.15 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12350.34 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12337.84 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12224.90 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12125.04 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12180.90 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12295.80 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12292.06 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12391.17 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12345.15 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12247.33 examples/s]
Filter: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 9433.65 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12531.00 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12256.23 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12444.89 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12340.60 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12268.51 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12247.24 examples/s]
Filter: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 12259.54 examples/s]
README.md: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 683/683 [00:00<00:00, 5.85MB/s]
train-00000-of-00001.parquet: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9.96M/9.96M [00:00<00:00, 22.4MB/s]
validation-00000-of-00001.parquet: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 284k/284k [00:00<00:00, 96.4MB/s]
Generating train split: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4000/4000 [00:00<00:00, 24948.91 examples/s]
Generating validation split: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 74/74 [00:00<00:00, 9825.52 examples/s]
thai_train.csv: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 164M/164M [00:06<00:00, 24.4MB/s]
Generating train split: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 392702/392702 [00:02<00:00, 143122.95 examples/s]
Filter: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 392702/392702 [00:02<00:00, 176309.61 examples/s]
README.md: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15.6k/15.6k [00:00<00:00, 115MB/s]
validation-00000-of-00001.parquet: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 337k/337k [00:00<00:00, 75.0MB/s]
Generating validation split: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1190/1190 [00:00<00:00, 117054.92 examples/s]
Token indices sequence length is longer than the specified maximum sequence length for this model (1797 > 1024). Running this sequence through the model will result in indexing errors
```

> this is the output from my terminal right now

Yeah that looks good 😉 as long as it's not stuck.

Yes, Thanks to all of you for sticking with me. I think my command can run now.

So the steps are 


1.

```
git clone https://github.com/huggingface/lighteval.git
cd lighteval
```

The reason for using git clone instead of pip is step NO.3. To specify the path to finetask dir and custom task path.
I don't know if this is the case though. I haven't tried using lighteval from pip yet.
Also, ensure cuda version and torch.
Actually, I have a problem with torch and lighteval. Apparently, I can not use torch 2.4.1 or 2.5 with lighteval. Right now I use torch 2.2.2 with cuda 12.1 and the program works.

2.
```
pip install -e .[dev]
```
maybe use other group for vllm or nanotron


3.
```
export CUDA_VISIBLE_DEVICES="1"
lighteval accelerate \
    "pretrained=<model_name>,dtype=bfloat16,max_length=1024" \
    "examples/tasks/fine_tasks/mcf/th.txt" \
    --custom-tasks "src/lighteval/tasks/multilingual/tasks.py" \
```

And don't forget to request permission for meta mmlu


I will update the blog with the new command. Thanks for debugging through :)

Hi guys, sorry to bother you.

I need a little help.

 So, I just finished evaluating my model on Thai finetasks. And I would like to compare my model with this table

<img width="655" alt="Image" src="https://github.com/user-attachments/assets/9e5527f4-8927-490f-83af-e7f339a3207c" />

I am unsure how to compute the score in the score column.
Here is the result of one of my model

```
{'all': {'acc_': 0.29229183088464955,
         'acc__stderr': 0.032172784772745364,
         'exact_match_tha_prefix': 0.0005451680672268908,
         'exact_match_tha_prefix_stderr': 0.0005451680672268886,
         'f1_tha': 0.009631290097624132,
         'f1_tha_stderr': 0.0016258163303400992},
 'lighteval|belebele_tha_Thai_mcf|5': {'acc_': 0.26555555555555554,
                                       'acc__stderr': 0.014729137787585106},
 'lighteval|community_hellaswag_tha_mcf|5': {'acc_': 0.2514524750174297,
                                             'acc__stderr': 0.006614582593648662},
 'lighteval|m3exams_tha_mcf|5': {'acc_': 0.20525830258302583,
                                 'acc__stderr': 0.008676287024499635},
 'lighteval|meta_mmlu_tha_mcf:_average|5': {'acc_': 0.29135859401393976,
                                            'acc__stderr': 0.03388671336376532},
 'lighteval|meta_mmlu_tha_mcf:abstract_algebra|5': {'acc_': 0.31,
                                                    'acc__stderr': 0.04648231987117316},
 'lighteval|meta_mmlu_tha_mcf:anatomy|5': {'acc_': 0.3111111111111111,
                                           'acc__stderr': 0.03999262876617721},
 'lighteval|meta_mmlu_tha_mcf:astronomy|5': {'acc_': 0.2631578947368421,
                                             'acc__stderr': 0.035834961763610645},
 'lighteval|meta_mmlu_tha_mcf:business_ethics|5': {'acc_': 0.24,
                                                   'acc__stderr': 0.042923469599092816},
 'lighteval|meta_mmlu_tha_mcf:clinical_knowledge|5': {'acc_': 0.27547169811320754,
                                                      'acc__stderr': 0.027495663683724057},
 'lighteval|meta_mmlu_tha_mcf:college_biology|5': {'acc_': 0.2361111111111111,
                                                   'acc__stderr': 0.03551446610810826},
 'lighteval|meta_mmlu_tha_mcf:college_chemistry|5': {'acc_': 0.35,
                                                     'acc__stderr': 0.0479372485441102},
 'lighteval|meta_mmlu_tha_mcf:college_computer_science|5': {'acc_': 0.3,
                                                            'acc__stderr': 0.046056618647183814},
 'lighteval|meta_mmlu_tha_mcf:college_mathematics|5': {'acc_': 0.24,
                                                       'acc__stderr': 0.04292346959909284},
 'lighteval|meta_mmlu_tha_mcf:college_medicine|5': {'acc_': 0.3352601156069364,
                                                    'acc__stderr': 0.03599586301247078},
 'lighteval|meta_mmlu_tha_mcf:college_physics|5': {'acc_': 0.27450980392156865,
                                                   'acc__stderr': 0.044405219061793275},
 'lighteval|meta_mmlu_tha_mcf:computer_security|5': {'acc_': 0.24,
                                                     'acc__stderr': 0.04292346959909283},
 'lighteval|meta_mmlu_tha_mcf:conceptual_physics|5': {'acc_': 0.28085106382978725,
                                                      'acc__stderr': 0.029379170464124815},
 'lighteval|meta_mmlu_tha_mcf:econometrics|5': {'acc_': 0.23684210526315788,
                                                'acc__stderr': 0.03999423879281337},
 'lighteval|meta_mmlu_tha_mcf:electrical_engineering|5': {'acc_': 0.2827586206896552,
                                                          'acc__stderr': 0.03752833958003336},
 'lighteval|meta_mmlu_tha_mcf:elementary_mathematics|5': {'acc_': 0.2751322751322751,
                                                          'acc__stderr': 0.023000086859068656},
 'lighteval|meta_mmlu_tha_mcf:formal_logic|5': {'acc_': 0.25396825396825395,
                                                'acc__stderr': 0.038932596106046734},
 'lighteval|meta_mmlu_tha_mcf:global_facts|5': {'acc_': 0.32,
                                                'acc__stderr': 0.046882617226215034},
 'lighteval|meta_mmlu_tha_mcf:high_school_biology|5': {'acc_': 0.29354838709677417,
                                                       'acc__stderr': 0.02590608702131929},
 'lighteval|meta_mmlu_tha_mcf:high_school_chemistry|5': {'acc_': 0.30049261083743845,
                                                         'acc__stderr': 0.03225799476233485},
 'lighteval|meta_mmlu_tha_mcf:high_school_computer_science|5': {'acc_': 0.27,
                                                                'acc__stderr': 0.0446196043338474},
 'lighteval|meta_mmlu_tha_mcf:high_school_european_history|5': {'acc_': 0.2545454545454545,
                                                                'acc__stderr': 0.03401506715249039},
 'lighteval|meta_mmlu_tha_mcf:high_school_geography|5': {'acc_': 0.35353535353535354,
                                                         'acc__stderr': 0.03406086723547153},
 'lighteval|meta_mmlu_tha_mcf:high_school_government_and_politics|5': {'acc_': 0.3626943005181347,
                                                                       'acc__stderr': 0.034697137917043715},
 'lighteval|meta_mmlu_tha_mcf:high_school_macroeconomics|5': {'acc_': 0.3,
                                                              'acc__stderr': 0.023234581088428484},
 'lighteval|meta_mmlu_tha_mcf:high_school_mathematics|5': {'acc_': 0.24444444444444444,
                                                           'acc__stderr': 0.02620276653465215},
 'lighteval|meta_mmlu_tha_mcf:high_school_microeconomics|5': {'acc_': 0.3067226890756303,
                                                              'acc__stderr': 0.029953823891887048},
 'lighteval|meta_mmlu_tha_mcf:high_school_physics|5': {'acc_': 0.2582781456953642,
                                                       'acc__stderr': 0.035737053147634576},
 'lighteval|meta_mmlu_tha_mcf:high_school_psychology|5': {'acc_': 0.3119266055045872,
                                                          'acc__stderr': 0.019862967976707245},
 'lighteval|meta_mmlu_tha_mcf:high_school_statistics|5': {'acc_': 0.39814814814814814,
                                                          'acc__stderr': 0.033384734032074016},
 'lighteval|meta_mmlu_tha_mcf:high_school_us_history|5': {'acc_': 0.3137254901960784,
                                                          'acc__stderr': 0.032566854844603886},
 'lighteval|meta_mmlu_tha_mcf:high_school_world_history|5': {'acc_': 0.25738396624472576,
                                                             'acc__stderr': 0.028458820991460305},
 'lighteval|meta_mmlu_tha_mcf:human_aging|5': {'acc_': 0.3273542600896861,
                                               'acc__stderr': 0.03149384670994132},
 'lighteval|meta_mmlu_tha_mcf:human_sexuality|5': {'acc_': 0.2824427480916031,
                                                   'acc__stderr': 0.03948406125768361},
 'lighteval|meta_mmlu_tha_mcf:international_law|5': {'acc_': 0.256198347107438,
                                                     'acc__stderr': 0.03984979653302872},
 'lighteval|meta_mmlu_tha_mcf:jurisprudence|5': {'acc_': 0.25925925925925924,
                                                 'acc__stderr': 0.04236511258094632},
 'lighteval|meta_mmlu_tha_mcf:logical_fallacies|5': {'acc_': 0.24539877300613497,
                                                     'acc__stderr': 0.03380939813943354},
 'lighteval|meta_mmlu_tha_mcf:machine_learning|5': {'acc_': 0.30357142857142855,
                                                    'acc__stderr': 0.04364226155841044},
 'lighteval|meta_mmlu_tha_mcf:management|5': {'acc_': 0.36893203883495146,
                                              'acc__stderr': 0.047776151811567386},
 'lighteval|meta_mmlu_tha_mcf:marketing|5': {'acc_': 0.31196581196581197,
                                             'acc__stderr': 0.03035152732334495},
 'lighteval|meta_mmlu_tha_mcf:medical_genetics|5': {'acc_': 0.29,
                                                    'acc__stderr': 0.045604802157206845},
 'lighteval|meta_mmlu_tha_mcf:miscellaneous|5': {'acc_': 0.2554278416347382,
                                                 'acc__stderr': 0.015594955384455772},
 'lighteval|meta_mmlu_tha_mcf:moral_disputes|5': {'acc_': 0.25722543352601157,
                                                  'acc__stderr': 0.023532925431044262},
 'lighteval|meta_mmlu_tha_mcf:moral_scenarios|5': {'acc_': 0.2335195530726257,
                                                   'acc__stderr': 0.014149575348976266},
 'lighteval|meta_mmlu_tha_mcf:nutrition|5': {'acc_': 0.2908496732026144,
                                             'acc__stderr': 0.026004800363952113},
 'lighteval|meta_mmlu_tha_mcf:philosophy|5': {'acc_': 0.24758842443729903,
                                              'acc__stderr': 0.024513879973621967},
 'lighteval|meta_mmlu_tha_mcf:prehistory|5': {'acc_': 0.3055555555555556,
                                              'acc__stderr': 0.025630824975621355},
 'lighteval|meta_mmlu_tha_mcf:professional_accounting|5': {'acc_': 0.2907801418439716,
                                                           'acc__stderr': 0.027090664368353178},
 'lighteval|meta_mmlu_tha_mcf:professional_law|5': {'acc_': 0.2646675358539765,
                                                    'acc__stderr': 0.011267332992845533},
 'lighteval|meta_mmlu_tha_mcf:professional_medicine|5': {'acc_': 0.4485294117647059,
                                                         'acc__stderr': 0.030211479609121596},
 'lighteval|meta_mmlu_tha_mcf:professional_psychology|5': {'acc_': 0.2679738562091503,
                                                           'acc__stderr': 0.017917974069594726},
 'lighteval|meta_mmlu_tha_mcf:public_relations|5': {'acc_': 0.32727272727272727,
                                                    'acc__stderr': 0.04494290866252089},
 'lighteval|meta_mmlu_tha_mcf:security_studies|5': {'acc_': 0.3877551020408163,
                                                    'acc__stderr': 0.031192230726795656},
 'lighteval|meta_mmlu_tha_mcf:sociology|5': {'acc_': 0.2835820895522388,
                                             'acc__stderr': 0.03187187537919798},
 'lighteval|meta_mmlu_tha_mcf:us_foreign_policy|5': {'acc_': 0.24,
                                                     'acc__stderr': 0.04292346959909284},
 'lighteval|meta_mmlu_tha_mcf:virology|5': {'acc_': 0.29518072289156627,
                                            'acc__stderr': 0.03550920185689629},
 'lighteval|meta_mmlu_tha_mcf:world_religions|5': {'acc_': 0.3157894736842105,
                                                   'acc__stderr': 0.03565079670708312},
 'lighteval|thaiqa_tha|5': {'exact_match_tha_prefix': 0.00025,
                            'exact_match_tha_prefix_stderr': 0.00024999999999999713,
                            'f1_tha': 0.0069117424242424235,
                            'f1_tha_stderr': 0.0009052938149068328},
 'lighteval|xnli2.0_tha_mcf|5': {'acc_': 0.500095492013048,
                                 'acc__stderr': 0.0009772019971102728},
 'lighteval|xquad_tha|5': {'exact_match_tha_prefix': 0.0008403361344537816,
                           'exact_match_tha_prefix_stderr': 0.00084033613445378,
                           'f1_tha': 0.012350837771005839,
                           'f1_tha_stderr': 0.0023463388457733657}}

```

I wonder if I could use the `all` for this purpose.
Any ideas?

THANKS!

Hey I am a bit bussy right now, I will send it on Wednesday :)

Sorry, but what do you mean by "send it on Wednesday" ?

I just want to ask if my understanding is correct or not about how to compare my score to the leaderboard score. 

Thanks

Was thinking too far ahead, to get the same results as on the leaderboard you have to rescale the scores as in https://huggingface.co/docs/leaderboards/open_llm_leaderboard/normalization. I have it in a notebook, can send it on wednesday. So no using all is not possible because it does no rescalling

> Was thinking too far ahead, to get the same results as on the leaderboard you have to rescale the scores as in https://huggingface.co/docs/leaderboards/open_llm_leaderboard/normalization. I have it in a notebook, can send it on wednesday. So no using all is not possible because it does no rescalling

Thank you so much

> Was thinking too far ahead, to get the same results as on the leaderboard you have to rescale the scores as in https://huggingface.co/docs/leaderboards/open_llm_leaderboard/normalization. I have it in a notebook, can send it on wednesday. So no using all is not possible because it does no rescalling

@hynky1999 Could you please send me the file?

> Was thinking too far ahead, to get the same results as on the leaderboard you have to rescale the scores as in https://huggingface.co/docs/leaderboards/open_llm_leaderboard/normalization. I have it in a notebook, can send it on wednesday. So no using all is not possible because it does no rescalling

HI, just wanna check-up. Any update, sir?

Sorry completely forgot about it:

To review what needs to be done to get to a final score for one language:
1) Rescale each of the tasks (even subtasks!!!!). By rescaling we mean (score - rand_baseline) / (1 - rand_baseline)
2) Because we rescaled even subtasks, we then create new average for tasks that have susbtasks (e.g mmlu)
3) Group each task based on predefined categories and compute mean in each of the category
4) Finaly compute the mean over categories == your final score

Steps 3 and 4 are kinda optional, you don't have to do them, since individual scores are provided for the models too! But to me it certainly makes sense to do them.

To do 1 you need random baseline, we have a way to compute that in lighteval (see baseline mode) but I added the csv with precomputed baselines

Here is roughly the code we used:
```
import pandas as pd
import numpy as np
from typing import List, Dict, Literal
from collections import defaultdict

# Define task categories from task_type_mapping.py
TASK_TYPE = Literal['GK', 'RC', 'RES', 'NLU', 'SAFE', 'MATH', 'SENT']

task_mapping: dict[TASK_TYPE, list[str]] = {
    'GK': ['.*_arc_', 'hi-arc', '.*mmlu', 'mkqa', '.*exams', "mintaka", 'chegeka', '.*triviaqa', '.*acva', '.*boolq', '.*m3exam', 'acva', 'agieval', 'ceval', '.*sciqa', '.*worldtree', '.*truthfulqa'],
    'RC': ['belebele', 'alghafa_meta_dialects' ,'.*soqal', '.*facts' ,'tydiqa', 'xquad', 'fquad', 'indicqa', 'cmrc', 'thaiqa', 'tquadv2', '.*mlqa', '.*_race_', 'c3', 'arcd', 'chai', '.*boolq', 'lambada', 'kenswquad', 'chinese_squad', 'sber_squad', 'cmrc2018'],
    'RES': ['xcodah', '.*xcopa', 'parus', 'mathlogic_qa', '.*openbookqa', '.*_piqa_', 'xcsqa'],
    'NLU': ['.*xnli', 'cmnli', 'wsci', '.*hellaswag', '.*winograd', 'rcb', 'ocnli', 'indicnxnli', 'pawsx', 'xstory_cloze'],
    'SAFE': ['toxigen'],
    'MATH': ['cmath', '.*mgsm', 'msvamp', 'cmm_math', 'math23k', 'tal_scq5k', '.*math', "hawp", "mwp"],
    'SENT': ['.*sentiment'],
}

def get_task_category(task_name: str) -> TASK_TYPE:
    """
    Determines the category of a task based on task_mapping patterns.
    """
    import re
    for category, patterns in task_mapping.items():
        for pattern in patterns:
            if re.match(pattern, task_name):
                return category
    return None

def rescale_scores(df: pd.DataFrame, 
                  baseline_runs: list[str],
                  metric_columns: list[str]) -> pd.DataFrame:
    """
    Rescales scores relative to a baseline performance.
    """
    df = df.copy()
    
    # Calculate mean baseline performance
    baseline_mask = df['runname'].isin(baseline_runs)
    baseline = df[baseline_mask].groupby('steps')[metric_columns].mean()
    
    # Reindex to match all steps and interpolate missing values
    baseline = baseline.reindex(df['steps'].unique()).interpolate()
    
    # Merge baseline scores with main df
    df_with_baseline = df.merge(
        baseline.reset_index(),
        on=['steps'],
        how='left',
        suffixes=('', '_baseline')
    ).fillna(0)
    
    # Rescale each metric column
    for col in metric_cols:
        baseline_col = f"{col}_baseline"
        # Skip if baseline is 0 or 1
        if (df_with_baseline[baseline_col] == 0).all() or (df_with_baseline[baseline_col] == 1).all():
            continue
            
        # Rescale: (score - baseline) / (1 - baseline)
        df[col] = (df[col] - df_with_baseline[baseline_col]) / (1 - df_with_baseline[baseline_col])
        
    return df

def group_tasks_by_category(df: pd.DataFrame) -> pd.DataFrame:
    """
    Groups tasks first by category (GK, RC, etc.) then averages across categories.
    """
    df = df.copy()
    
    # Get all metric columns (containing 'lighteval|')
    metric_cols = [col for col in df.columns 
                  if 'lighteval|' in col 
                  and ':_average' not in col  # Exclude existing averages
                  and ('/acc_' in col or '/f1' in col)]  # Include only metric columns
    
    # First, group by categories
    category_groups = defaultdict(list)
    for col in metric_cols:
        task_name = col.split('|')[1].split('_')[0]  # Get base task name
        category = get_task_category(task_name)
        if category:
            category_groups[category].append(col)
    
    # Calculate mean for each category
    for category, cols in category_groups.items():
        df[f"category_{category}"] = df[cols].mean(axis=1)
    
    # Calculate macro score (average across categories)
    category_cols = [col for col in df.columns if col.startswith('category_')]
    df['agg_score_macro'] = df[category_cols].mean(axis=1)
    
    return df

def process_results(csv_path: str, 
                   baseline_runs: List[str]) -> pd.DataFrame:
    """
    Load results, rescale scores, and group by categories.
    
    Args:
        csv_path: Path to CSV file
        baseline_runs: List of baseline run names
    """
    # Load data
    df = pd.read_csv(csv_path)
    
    # Get metric columns
    metric_cols = [col for col in df.columns 
                  if 'lighteval|' in col 
                  and ('/acc_' in col or '/f1' in col)]
    
    # Rescale scores if needed
    if baseline_runs:
        df = rescale_scores(df, baseline_runs, metric_cols)
    
    # Group tasks by category and compute macro score
    df = group_tasks_by_category(df)
    
    return df

if __name__ == "__main__":
    # Example usage
    results = process_results(
        csv_path='results_tr.csv',
        baseline_runs=['baseline-tr-']
    )
    
    # Sort by macro score
    results_sorted = results.sort_values('agg_score_macro', ascending=False)
    
    # Display results
    print("\nTop models by category and macro scores:")
    display_cols = ['runname', 'agg_score_macro'] + [
        col for col in results_sorted.columns 
        if col.startswith('category_')
    ]
    print(results_sorted[display_cols].head())
```

[lighteval_baseline.csv](https://github.com/user-attachments/files/18697439/lighteval_baseline.csv)

Each column contains the baseline score (most are 0.25 cause of most of mcq have 4 options)

MANY THANKS!!

@hynky1999 

Thanks for the code. Could you please help me a little more

I really don't understand this funtion

```
def rescale_scores(df: pd.DataFrame, baseline_runs: list[str], metric_columns: list[str]) -> pd.DataFrame:
    """
    Rescales scores relative to a baseline performance.
    """
    df = df.copy()

    # Calculate mean baseline performance
    baseline_mask = df["runname"].isin(baseline_runs)
    baseline = df[baseline_mask].groupby("steps")[metric_columns].mean()

    # Reindex to match all steps and interpolate missing values
    baseline = baseline.reindex(df["steps"].unique()).interpolate()

    # Merge baseline scores with main df
    df_with_baseline = df.merge(baseline.reset_index(), on=["steps"], how="left", suffixes=("", "_baseline")).fillna(0)

    # Rescale each metric column
    for col in metric_cols:
        baseline_col = f"{col}_baseline"
        # Skip if baseline is 0 or 1
        if (df_with_baseline[baseline_col] == 0).all() or (df_with_baseline[baseline_col] == 1).all():
            continue

        # Rescale: (score - baseline) / (1 - baseline)
        df[col] = (df[col] - df_with_baseline[baseline_col]) / (1 - df_with_baseline[baseline_col])

    return df

```

I know that this is the function that do the normalization. But what I am struck is the column `runname` and `baseline_runs`
What is this column? Because if I am not mistaken, the result of `lighteval accelerate` command is json file and I can not find any field named `runname` at all. The same goes for `baseline_runs`


I think I can use this code to format my json result for using with your code like this

```
path = "./results_2025-01-27T02-49-24.720725.json"
with open(path, "r") as f:
    r = json.load(f)

result = r["results"]
# result
result_tr = {}
for task, item in result.items():
    for k,v in item.items():
        new_key = f"{task}/{k}"
        result_tr[new_key] = v
tmp_r_df = pd.DataFrame([result_tr])
```

But I can not figure out how o use these 2 things.

For reference, I attach one of my json result here

[results_2025-01-27T02-49-24.720725.json](https://github.com/user-attachments/files/18703743/results_2025-01-27T02-49-24.720725.json)








The input that we have our results in is following:
```
runname,steps,...score_columns
```
So you have to convert it to that format

Yes, I understand that but what I means is there are on fields named `runname` and `step` in json output from command line. So I am wondering how did you get them?

Ahhh I see, so we have a code, that will translate the folder structure of the lighteval outputs e.g results/model_name_provider/model/results into this table.

If you only have a single model, just use your model_name as runname and step can be whatever, so go with 0

Thanks for the clarification. Is it possible for me to have that code? Actually, I have 8 models right now