Repository: huggingface/lighteval
Issue:  ValueError: Cannot find tasks community|alghafa:mcq_exams_test_ar in task list or in custom task registry)
Root cause: Interface Contract Mismatch

This FrameWork is a great job！but it seems you guys didn't say how to eval the community benchmark？I am wandering what should I do to eval the list of benchmark in examples/tasks/OALL_v2_tasks.txt

Hi @BobTsang1995 !
You'll need to use the --custom-tasks argument and point to the correct file. @alielfilali01 , want to share your usual command line to launch the evals? 

> Hi @BobTsang1995 ! You'll need to use the --custom-tasks argument and point to the correct file. @alielfilali01 , want to share your usual command line to launch the evals?
I try to use this command,but failed.
accelerate launch --multi_gpu --num_processes=8 -m lighteval accelerate "pretrained=/mnt/Qwen2-7B-CT-latest/checkpoint-1150,dtype=bfloat16,max_length=16384" "community|alghafa:mcq_exams_test_ar|0|0" --custom-tasks "community_tasks/arabic_evals.py" --output-dir="./evals/"
it reminds me two type error below:
TypeError: LightevalTaskConfig.__init__() got an unexpected keyword argument 'output_regex'
TypeError: LightevalTaskConfig.__init__() got an unexpected keyword argument 'frozen'

so i comment out the code "output_regex" and "frozen"
It may be because I commented out the above two parameters, which caused another error to occur at the same time.

 /mnt/sg_nas/liheng/lighteval/src/lighteval/models/abstract_model.py:200 in tok_encode_pair       │
[rank0]: │                                                                                                  │
[rank0]: │   197 │   │   │                                                                                  │
[rank0]: │   198 │   │   │   return context_enc, continuation_enc                                           │
[rank0]: │   199 │   │                                                                                      │
[rank0]: │ ❱ 200 │   │   whole_enc = self.tok_encode(context + continuation)                                │
[rank0]: │   201 │   │   context_enc = self.tok_encode(context)                                             │
[rank0]: │   202 │   │   context_enc_len = len(context_enc)                                                 │
[rank0]: │   203 │   │   # In case continuation tokens merge with context tokens we use the merged token    │
[rank0]: │                                                                                                  │
[rank0]: │ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
[rank0]: │ │      context = 'الأسئلة التالية هي أسئلة متعددة الإختيارات مع الجواب الصحيح\n\nالسؤال: صل    │ │
[rank0]: │ │                أولى دول'+98                                                                  │ │
[rank0]: │ │ continuation = 245                                                                           │ │
[rank0]: │ │     n_spaces = 0                                                                             │ │
[rank0]: │ │     pairwise = False                                                                         │ │
[rank0]: │ │         self = <lighteval.models.transformers.base_model.BaseModel object at 0x7fc53c0ea680> │ │
[rank0]: │ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
[rank0]: ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
[rank0]: TypeError: can only concatenate str (not "int") to str



Hi! Are you using the latest version of lighteval on main?

> Hi! Are you using the latest version of lighteval on main?

yeah，I'm pretty sure

Hm, I'll take some time to investigate today, thanks a lot for the report then :)

It seems that the parent class `LightevalTaskConfig` does not have the parameters `output_regex` and `frozen`.

Yep, these params have been removed in a PR yesterday, but I'm more concerned about the second error you're getting

Hi! Can you test if the code in this PR: https://github.com/huggingface/lighteval/pull/440 fixes your issue?


> Hi! Can you test if the code in this PR: #440 fixes your issue?
Of course I'd love to test this fork, just give me a few minutes


thanks god it works, Merci, mon ami français


bro, something went wrong...
I found that the asas-ai/AraTrust-categorized dataset does not exist on hugging face

Hi @BobTsang1995 ! Could you open a new issue, since it's unrelated, and tag @alielfilali01, the community contributor who added the arabic evals? And please follow the bug template :)

Hey @BobTsang1995 someone in the AsasAI HF org switched it to private ! It is now public and i will send a msg to the group to avoid this issue in the future