Repository: huggingface/lighteval
Issue: [BUG] DatasetNotFoundError: Dataset 'asas-ai/AraTrust-categorized' doesn't exist on the Hub or cannot be accessed.
Root cause: Environment Incompatibility

It seems that the asas-ai/AraTrust-categorized dataset does not exist on hugging face. can you guys fix it?@alielfilali01
## Describe the bug
When trying to run lighteval with custom Arabic evaluation tasks, it fails with a DatasetNotFoundError, indicating that the dataset 'asas-ai/AraTrust-categorized' cannot be found on the Hugging Face Hub.

## To Reproduce
1. Set up the conda environment with Python 3.10
2. Install required packages
3. Run the following command:
```bash
accelerate launch --multi_gpu --num_processes=8 -m lighteval \
accelerate "pretrained=/mnt/sg_nas/liheng/Marco_checkpoint/Qwen2-7B-mmmlu-latest/checkpoint-1150,dtype=bfloat16,max_length=16384" \
"examples/tasks/OALL_v2_tasks.txt" \
--custom-tasks "community_tasks/arabic_evals.py" \
--output-dir="./evals/"
```

## Full Error Message

```bash
[rank3]: DatasetNotFoundError: Dataset 'asas-ai/AraTrust-categorized' doesn't exist on the Hub or cannot be accessed.
```

## Version info
- **Operating System**: (Linux)
- **Python Version**: Python 3.10

---

Thanks for the issue! It would help a lot if you could follow the template, and provide, each time you report a bug:
- the command you run 
- the full stack trace of the error

> Thanks for the issue! It would help a lot if you could follow the template, and provide, each time you report a bug:
> 
> * the command you run
> * the full stack trace of the error

sry, my bad bro. I submitted the detailed error information above.

Hey @BobTsang1995 & @clefourrier 
Plz feel free to close this issue since it was not a bug.

Dataset is public again (someone switched it to private by mistake). Find [here](https://huggingface.co/datasets/asas-ai/AraTrust-categorized)
Now the task should run as expected.

@BobTsang1995 can you plz keep commenting here even after the issue get closed if you face any further issues ? Usually they are just minor things otherwise if it's something really wrong we can open a separate issue for it and adress it properly.
Thank you

@BobTsang1995 thanks a lot for the update, much better! :)

> Hey @BobTsang1995 & @clefourrier Plz feel free to close this issue since it was not a bug.
> 
> Dataset is public again (someone switched it to private by mistake). Find [here](https://huggingface.co/datasets/asas-ai/AraTrust-categorized) Now the task should run as expected.
> 
> @BobTsang1995 can you plz keep commenting here even after the issue get closed if you face any further issues ? Usually they are just minor things otherwise if it's something really wrong we can open a separate issue for it and adress it properly. Thank you

thx a lot, I will try it again, appreciate for your repeat

One last thing.
I hope the name `Qwen2-7B-mmmlu-latest` doesn't mean that is trained on `mmmlu` >_<
Have a good day/night and plz keep the comments coming !

> One last thing. I hope the name `Qwen2-7B-mmmlu-latest` doesn't mean that is trained on `mmmlu` >_< Have a good day/night and plz keep the comments coming !

LOL! This is my unique naming method. We have indeed done a lot of work on the mmmlu benchmark. Our multilingual model will be coming soon, so you can look forward to it. 

another error occurred. it seems that some entries do not have these keys KeyError: 'sol3' @alielfilali01 
![image](https://github.com/user-attachments/assets/dbcb9cbf-6a8f-4ee4-badf-68209933a8ca)



Oh shot ! That's because of the last PR #440 !
AlGhafa prompt function is applied on all 9 subsets of AlGhafa Native ([here](https://huggingface.co/datasets/OALL/AlGhafa-Arabic-LLM-Benchmark-Native)) but not all of the subsets share the same columns ! 
This PR #442 should fix it ounce merged (make sure to make the changes on your local machine so you don't wait for the PR to be merged) 

Last PR's fixed the issue that @BobTsang1995 had on the previous subset, because this line caused an error. Please find a way to explicitly provide the columns (for example, providing the full list of all allowed columns, instead of excluding some)

Oh sorry ! got mixed between the issues ! Ok i will investigate this further and update the associated PR

> Oh sorry ! got mixed between the issues ! Ok i will investigate this further and update the associated PR

There is still a problem with PR #444. The task_name passed to the alghafa_pfn() function has a prefix of community|xxx:, and the key cannot be found in the dictionary. I think maybe you should run the code before PR.

at the same time, It seems that you added an undefined parameter when inheriting the Doc class, resulting in an error: TypeError: Doc.__init__() got an unexpected keyword argument 'target_for_fewshot_sorting'

Hey @BobTsang1995 tnx for the highlight.

Well, the suite "community" is already part of the config so naturally it shouldn't be part of the task_name + the main task "alghafa:" is already defined in the tasks list which goes later to the final (end of the script) tasks table. So in theory it should be fine ! 

But you are correct i haven't tested the code before pushing as i was leaving desk actually before i checked the notif 🥲 

I'll give it a look tomorrow and try to fix it asap. Thanks again for your feedback 

> Hey @BobTsang1995 tnx for the highlight.
> 
> Well, the suite "community" is already part of the config so naturally it shouldn't be part of the task_name + the main task "alghafa:" is already defined in the tasks list which goes later to the final (end of the script) tasks table. So in theory it should be fine !
> 
> But you are correct i haven't tested the code before pushing as i was leaving desk actually before i checked the notif 🥲
> 
> I'll give it a look tomorrow and try to fix it asap. Thanks again for your feedback

Sorry to bother，Do you have any plans to fix this issue today?

It is fixed (test went well) and waiting for the PR #444 to be merged.
In the meantime feel free to run using this [fork](https://github.com/alielfilali01/lighteval)
Thanks @BobTsang1995 for your feedback !

> It is fixed (test went well) and waiting for the PR #444 to be merged. In the meantime feel free to run using this [fork](https://github.com/alielfilali01/lighteval) Thanks @BobTsang1995 for your feedback !

@alielfilali01 another question, bro now I want to eval 72B model, but accelerate always oom when i use model parallel，I'm wondering how you guys eval big size model when use lighteval.

`accelerate launch --multi_gpu --num_processes=8 -m lighteval accelerate "pretrained=/mnt/sg_nas/liheng/Marco_checkpoint/Qwen2-72B,dtype=bfloat16,max_length=2048,model_parallel=True" --override-batch-size 1  "examples/tasks/OALL_v1_tasks.txt" --custom-tasks "community_tasks/arabic_evals.py" --output-dir="./evals/"`

Hey @BobTsang1995 
You are already using DP by setting `--multi_gpu --num_processes=8` and no GPUs left for PP.
Consider setting `--num_processes=4` for DP so you can still be able to shard the model on 2 GPUs with PP

Run this instead:
```shell
accelerate launch --multi_gpu --num_processes=4 -m lighteval accelerate "pretrained=/mnt/sg_nas/liheng/Marco_checkpoint/Qwen2-72B,dtype=bfloat16,max_length=2048,model_parallel=True" --override-batch-size 1  "examples/tasks/OALL_v1_tasks.txt" --custom-tasks "community_tasks/arabic_evals.py" --output-dir="./evals/"
```

PS: plz consider closing this issue.

> 嘿@BobTsang1995 您已通过设置使用 DP `--multi_gpu --num_processes=8`，并且没有剩余 GPU 用于 PP。 请考虑设置`--num_processes=4`DP，以便您仍能使用 PP 在 2 个 GPU 上分片模型
> 
> 运行这个：
> 
> ```shell
> accelerate launch --multi_gpu --num_processes=4 -m lighteval accelerate "pretrained=/mnt/sg_nas/liheng/Marco_checkpoint/Qwen2-72B,dtype=bfloat16,max_length=2048,model_parallel=True" --override-batch-size 1  "examples/tasks/OALL_v1_tasks.txt" --custom-tasks "community_tasks/arabic_evals.py" --output-dir="./evals/"
> ```
> 
> PS：请考虑关闭此问题。

thx, bro have a good day~