Repository: EleutherAI/lm-evaluation-harness
Issue: Traceback "TypeError: 'NoneType' " when running the task "MMLU" 
Root cause: Configuration Error

Hi! 
I'm testing my model in the "MMLU" task with lm-eval and had the traceback "TypeError: 'NoneType' object is not callable"  like below:
![84eac3bc-39bf-4005-9352-45fbcf8cbcab](https://github.com/user-attachments/assets/e14c97f5-4d78-474a-afc4-0738490a18ab)

Here's my command: 
'''
lm_eval --model hf
             --model_args pretrained=/home/fangce/model/zephyrRMU,device_map=auto,parallelize=True,trust_remote_code=True
             --tasks mmlu
             --batch_size 16
             --output_path /home/fangce/code/WMDP/eval_res/mmlu
'''
I've considered the network and set the proxy with "www.hf-mirror.com". It worked when I tested the task “wmdp”, while the "mmlu" still had the "TypeError".
I also downloaded all files of "hails/mmlu_no_train", and replace the code "dataset_path: hails/mmlu_no_train" with local file path in the all mmlu yaml.  It has the different traceback "ValueError: BuilderConfig 'world_religions' not found. Available: ['all']"  
![2](https://github.com/user-attachments/assets/041e76e5-76d9-4546-b534-ed5dac693951)

please tell me the solution, looking forward to your reply! Thanks!



I find a solution here https://github.com/EleutherAI/lm-evaluation-harness/issues/1659#issuecomment-2126421186
First, download the python script `mmlu_no_train.py` in hails/mmlu_no_train and store in local dict. 
Then,  change the `dataset_path: hails/mmlu_no_train`  to `dataset_path: /local_path` in `_default_template_yaml` which contains the python script.
Finally,  change `_URL = "https://huggingface.co/datasets/cais/mmlu/resolve/main/data.tar"` to `_URL = "https://hf-mirror.com/datasets/cais/mmlu/resolve/main/data.tar"`  in  local file `mmlu_no_train.py` 

> I find a solution here [#1659 (comment)](https://github.com/EleutherAI/lm-evaluation-harness/issues/1659#issuecomment-2126421186) First, download the python script `mmlu_no_train.py` in hails/mmlu_no_train and store in local dict. Then, change the `dataset_path: hails/mmlu_no_train` to `dataset_path: /local_path` in `_default_template_yaml` which contains the python script. Finally, change `_URL = "https://huggingface.co/datasets/cais/mmlu/resolve/main/data.tar"` to `_URL = "https://hf-mirror.com/datasets/cais/mmlu/resolve/main/data.tar"` in local file `mmlu_no_train.py`

Thanks a lot for your sharing. However, I did as you told: put `mmlu_no_train.py` in ` /local_path` and changed the `dataset_path` and `_URL` accordingly, but I still got the error "ValueError: BuilderConfig 'world_religions' not found. Available: ['all']".  Would you mind please telling me how to solve this?

Hi! @JiahaoWuGit I'm sorry to hear that. Please check the `dataset_path` in `mmlu/default/_default_template_yaml` is correct or try to use the absolute path. Meanwhile, do you set the `export HF_ENDPOINT=https://hf-mirror.com` in `~/.bashrc` or run `export HF_ENDPOINT=https://hf-mirror.com` before lm_eval?

> Hi! [@JiahaoWuGit](https://github.com/JiahaoWuGit) I'm sorry to hear that. Please check the `dataset_path` in `mmlu/default/_default_template_yaml` is correct or try to use the absolute path. Meanwhile, do you set the `export HF_ENDPOINT=https://hf-mirror.com` in `~/.bashrc` or run `export HF_ENDPOINT=https://hf-mirror.com` before lm_eval?

Yes! I have set `export HF_ENDPOINT=https://hf-mirror.com` before lm_eval. However, I still got the same error.

> Hi! [@JiahaoWuGit](https://github.com/JiahaoWuGit) I'm sorry to hear that. Please check the `dataset_path` in `mmlu/default/_default_template_yaml` is correct or try to use the absolute path. Meanwhile, do you set the `export HF_ENDPOINT=https://hf-mirror.com` in `~/.bashrc` or run `export HF_ENDPOINT=https://hf-mirror.com` before lm_eval?

Here is my settings:
dataset_path: /data/Backdoor-Enhanced-Alignment/lm_eval_dir #a copy of `cais/mmlu` with no auxiliary_train split
test_split: test
fewshot_split: dev
fewshot_config:
  sampler: first_n
output_type: multiple_choice
doc_to_text: "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:"
doc_to_choice: ["A", "B", "C", "D"]
doc_to_target: answer
metric_list:
  - metric: acc
    aggregation: mean
    higher_is_better: true
metadata:
  version: 1.0
# dataset_kwargs:


> > Hi! [@JiahaoWuGit](https://github.com/JiahaoWuGit) I'm sorry to hear that. Please check the `dataset_path` in `mmlu/default/_default_template_yaml` is correct or try to use the absolute path. Meanwhile, do you set the `export HF_ENDPOINT=https://hf-mirror.com` in `~/.bashrc` or run `export HF_ENDPOINT=https://hf-mirror.com` before lm_eval?
> 
> Here is my settings: dataset_path: /data/Backdoor-Enhanced-Alignment/lm_eval_dir #a copy of `cais/mmlu` with no auxiliary_train split test_split: test fewshot_split: dev fewshot_config: sampler: first_n output_type: multiple_choice doc_to_text: "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:" doc_to_choice: ["A", "B", "C", "D"] doc_to_target: answer metric_list:
> 
> * metric: acc
>   aggregation: mean
>   higher_is_better: true
>   metadata:
>   version: 1.0
> 
> # dataset_kwargs:
If the `mmlu_no_train.py` in the directory `/data/Backdoor-Enhanced-Alignment/lm_eval_dir`, change the dataset_path to `/data/Backdoor-Enhanced-Alignment/lm_eval_dir/mmlu_no_train`

> > > Hi! [@JiahaoWuGit](https://github.com/JiahaoWuGit) I'm sorry to hear that. Please check the `dataset_path` in `mmlu/default/_default_template_yaml` is correct or try to use the absolute path. Meanwhile, do you set the `export HF_ENDPOINT=https://hf-mirror.com` in `~/.bashrc` or run `export HF_ENDPOINT=https://hf-mirror.com` before lm_eval?
> > 
> > 
> > Here is my settings: dataset_path: /data/Backdoor-Enhanced-Alignment/lm_eval_dir #a copy of `cais/mmlu` with no auxiliary_train split test_split: test fewshot_split: dev fewshot_config: sampler: first_n output_type: multiple_choice doc_to_text: "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:" doc_to_choice: ["A", "B", "C", "D"] doc_to_target: answer metric_list:
> > 
> > * metric: acc
> >   aggregation: mean
> >   higher_is_better: true
> >   metadata:
> >   version: 1.0
> > 
> > # dataset_kwargs:
> 
> If the `mmlu_no_train.py` in the directory `/data/Backdoor-Enhanced-Alignment/lm_eval_dir`, change the dataset_path to `/data/Backdoor-Enhanced-Alignment/lm_eval_dir/mmlu_no_train`

Thanks! As you suggested, I eliminated the previous error. However, I got a new error as follows, and the screenshot of the  directory is also attached afterwards.

  File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 398module>
    cli_evaluate()
  File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/__main__.py", line 382, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/l    results = evaluator.simple_evaluate(
  File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/utils.py", line 402, in _wrapper
    return fn(*args, **kwargs)
  File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/evaluator.py", line 235, in simple_evaluate
    task_dict = get_task_dict(tasks, task_manager)
  File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 618, in get_task_dict
    task_name_from_string_dict = task_manager.load_task_or_group(
  File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 414, in load_task_or_group
m_eval/tasks/__init__.py", line 414, in load_task_or_group
  File "/root/anaconda3/envs/self-instruct/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/__main__.py", line 461, in <module>
    cli_evaluate()
  File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/__main__.py", line 382, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/utils.py", line 402, in _wrapper
    return fn(*args, **kwargs)
  File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/evaluator.py", line 235, in simple_evaluate
    task_dict = get_task_dict(tasks, task_manager)
  File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 618, in get_task_dict
    task_name_from_string_dict = task_manager.load_task_or_group(
  File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 414, in load_task_or_group
    collections.ChainMap(*map(self._load_individual_task_or_group, task_list))
  File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 398, in _load_individual_task_or_group
    group_name: dict(collections.ChainMap(*map(fn, reversed(subtask_list))))
  File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 398, in _load_individual_task_or_group
    group_name: dict(collections.ChainMap(*map(fn, reversed(subtask_list))))
  File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 314, in _load_individual_task_or_group
    return _load_task(task_config, task=name_or_config)
  File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 280, in _load_task
    task_object = ConfigurableTask(config=config)
  File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/api/task.py", line 820, in __init__
    self.download(self.config.dataset_kwargs)
  File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/api/task.py", line 928, in download
    self.dataset = datasets.load_dataset(
  File "/root/anaconda3/envs/self-instruct/lib/python3.10/site-packages/datasets/load.py", line 2129, in load_dataset
    builder_instance = load_dataset_builder(
  File "/root/anaconda3/envs/self-instruct/lib/python3.10/site-packages/datasets/load.py", line 1849, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/root/anaconda3/envs/self-instruct/lib/python3.10/site-packages/datasets/load.py", line 1737, in dataset_module_factory
    raise FileNotFoundError(f"Couldn't find any data file at {relative_to_absolute_path(path)}.")
FileNotFoundError: Couldn't find any data file at /data/Backdoor-Enhanced-Alignment/lm_eval_dir/mmlu_no_train.


![Image](https://github.com/user-attachments/assets/714a00a8-c038-4f5b-954d-49e8305d9bfa)
![Image](https://github.com/user-attachments/assets/e6a16b25-6972-47f5-a026-53d2bb8aa356)

> > > > Hi! [@JiahaoWuGit](https://github.com/JiahaoWuGit) I'm sorry to hear that. Please check the `dataset_path` in `mmlu/default/_default_template_yaml` is correct or try to use the absolute path. Meanwhile, do you set the `export HF_ENDPOINT=https://hf-mirror.com` in `~/.bashrc` or run `export HF_ENDPOINT=https://hf-mirror.com` before lm_eval?
> > > 
> > > 
> > > Here is my settings: dataset_path: /data/Backdoor-Enhanced-Alignment/lm_eval_dir #a copy of `cais/mmlu` with no auxiliary_train split test_split: test fewshot_split: dev fewshot_config: sampler: first_n output_type: multiple_choice doc_to_text: "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:" doc_to_choice: ["A", "B", "C", "D"] doc_to_target: answer metric_list:
> > > 
> > > * metric: acc
> > >   aggregation: mean
> > >   higher_is_better: true
> > >   metadata:
> > >   version: 1.0
> > > 
> > > # dataset_kwargs:
> > 
> > 
> > If the `mmlu_no_train.py` in the directory `/data/Backdoor-Enhanced-Alignment/lm_eval_dir`, change the dataset_path to `/data/Backdoor-Enhanced-Alignment/lm_eval_dir/mmlu_no_train`
> 
> Thanks! As you suggested, I eliminated the previous error. However, I got a new error as follows, and the screenshot of the directory is also attached afterwards.
> 
> File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/**init**.py", line 398module> cli_evaluate() File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/**main**.py", line 382, in cli_evaluate results = evaluator.simple_evaluate( File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/l results = evaluator.simple_evaluate( File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/utils.py", line 402, in _wrapper return fn(*args, **kwargs) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/evaluator.py", line 235, in simple_evaluate task_dict = get_task_dict(tasks, task_manager) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/**init**.py", line 618, in get_task_dict task_name_from_string_dict = task_manager.load_task_or_group( File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/**init**.py", line 414, in load_task_or_group m_eval/tasks/**init**.py", line 414, in load_task_or_group File "/root/anaconda3/envs/self-instruct/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/**main**.py", line 461, in cli_evaluate() File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/**main**.py", line 382, in cli_evaluate results = evaluator.simple_evaluate( File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/utils.py", line 402, in _wrapper return fn(*args, **kwargs) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/evaluator.py", line 235, in simple_evaluate task_dict = get_task_dict(tasks, task_manager) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/**init**.py", line 618, in get_task_dict task_name_from_string_dict = task_manager.load_task_or_group( File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/**init**.py", line 414, in load_task_or_group collections.ChainMap(*map(self._load_individual_task_or_group, task_list)) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/**init**.py", line 398, in _load_individual_task_or_group group_name: dict(collections.ChainMap(*map(fn, reversed(subtask_list)))) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/**init**.py", line 398, in _load_individual_task_or_group group_name: dict(collections.ChainMap(*map(fn, reversed(subtask_list)))) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/**init**.py", line 314, in _load_individual_task_or_group return _load_task(task_config, task=name_or_config) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/**init**.py", line 280, in _load_task task_object = ConfigurableTask(config=config) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/api/task.py", line 820, in **init** self.download(self.config.dataset_kwargs) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/api/task.py", line 928, in download self.dataset = datasets.load_dataset( File "/root/anaconda3/envs/self-instruct/lib/python3.10/site-packages/datasets/load.py", line 2129, in load_dataset builder_instance = load_dataset_builder( File "/root/anaconda3/envs/self-instruct/lib/python3.10/site-packages/datasets/load.py", line 1849, in load_dataset_builder dataset_module = dataset_module_factory( File "/root/anaconda3/envs/self-instruct/lib/python3.10/site-packages/datasets/load.py", line 1737, in dataset_module_factory raise FileNotFoundError(f"Couldn't find any data file at {relative_to_absolute_path(path)}.") FileNotFoundError: Couldn't find any data file at /data/Backdoor-Enhanced-Alignment/lm_eval_dir/mmlu_no_train.
> 
> ![Image](https://github.com/user-attachments/assets/714a00a8-c038-4f5b-954d-49e8305d9bfa) ![Image](https://github.com/user-attachments/assets/e6a16b25-6972-47f5-a026-53d2bb8aa356)

sorry, I didn't encounter this ever

> > > > > Hi! [@JiahaoWuGit](https://github.com/JiahaoWuGit) I'm sorry to hear that. Please check the `dataset_path` in `mmlu/default/_default_template_yaml` is correct or try to use the absolute path. Meanwhile, do you set the `export HF_ENDPOINT=https://hf-mirror.com` in `~/.bashrc` or run `export HF_ENDPOINT=https://hf-mirror.com` before lm_eval?
> > > > 
> > > > 
> > > > Here is my settings: dataset_path: /data/Backdoor-Enhanced-Alignment/lm_eval_dir #a copy of `cais/mmlu` with no auxiliary_train split test_split: test fewshot_split: dev fewshot_config: sampler: first_n output_type: multiple_choice doc_to_text: "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:" doc_to_choice: ["A", "B", "C", "D"] doc_to_target: answer metric_list:
> > > > 
> > > > * metric: acc
> > > >   aggregation: mean
> > > >   higher_is_better: true
> > > >   metadata:
> > > >   version: 1.0
> > > > 
> > > > # dataset_kwargs:
> > > 
> > > 
> > > If the `mmlu_no_train.py` in the directory `/data/Backdoor-Enhanced-Alignment/lm_eval_dir`, change the dataset_path to `/data/Backdoor-Enhanced-Alignment/lm_eval_dir/mmlu_no_train`
> > 
> > 
> > Thanks! As you suggested, I eliminated the previous error. However, I got a new error as follows, and the screenshot of the directory is also attached afterwards.
> > File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/**init**.py", line 398module> cli_evaluate() File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/**main**.py", line 382, in cli_evaluate results = evaluator.simple_evaluate( File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/l results = evaluator.simple_evaluate( File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/utils.py", line 402, in _wrapper return fn(*args, **kwargs) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/evaluator.py", line 235, in simple_evaluate task_dict = get_task_dict(tasks, task_manager) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/**init**.py", line 618, in get_task_dict task_name_from_string_dict = task_manager.load_task_or_group( File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/**init**.py", line 414, in load_task_or_group m_eval/tasks/**init**.py", line 414, in load_task_or_group File "/root/anaconda3/envs/self-instruct/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/**main**.py", line 461, in cli_evaluate() File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/**main**.py", line 382, in cli_evaluate results = evaluator.simple_evaluate( File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/utils.py", line 402, in _wrapper return fn(*args, **kwargs) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/evaluator.py", line 235, in simple_evaluate task_dict = get_task_dict(tasks, task_manager) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/**init**.py", line 618, in get_task_dict task_name_from_string_dict = task_manager.load_task_or_group( File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/**init**.py", line 414, in load_task_or_group collections.ChainMap(*map(self._load_individual_task_or_group, task_list)) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/**init**.py", line 398, in _load_individual_task_or_group group_name: dict(collections.ChainMap(*map(fn, reversed(subtask_list)))) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/**init**.py", line 398, in _load_individual_task_or_group group_name: dict(collections.ChainMap(*map(fn, reversed(subtask_list)))) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/**init**.py", line 314, in _load_individual_task_or_group return _load_task(task_config, task=name_or_config) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/**init**.py", line 280, in _load_task task_object = ConfigurableTask(config=config) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/api/task.py", line 820, in **init** self.download(self.config.dataset_kwargs) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/api/task.py", line 928, in download self.dataset = datasets.load_dataset( File "/root/anaconda3/envs/self-instruct/lib/python3.10/site-packages/datasets/load.py", line 2129, in load_dataset builder_instance = load_dataset_builder( File "/root/anaconda3/envs/self-instruct/lib/python3.10/site-packages/datasets/load.py", line 1849, in load_dataset_builder dataset_module = dataset_module_factory( File "/root/anaconda3/envs/self-instruct/lib/python3.10/site-packages/datasets/load.py", line 1737, in dataset_module_factory raise FileNotFoundError(f"Couldn't find any data file at {relative_to_absolute_path(path)}.") FileNotFoundError: Couldn't find any data file at /data/Backdoor-Enhanced-Alignment/lm_eval_dir/mmlu_no_train.
> > ![Image](https://github.com/user-attachments/assets/714a00a8-c038-4f5b-954d-49e8305d9bfa) ![Image](https://private-user-images.githubusercontent.com/44497691/406990994-e6a16b25-6972-47f5-a026-53d2bb8aa356.jpg?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3Mzc5OTk3NjUsIm5iZiI6MTczNzk5OTQ2NSwicGF0aCI6Ii80NDQ5NzY5MS80MDY5OTA5OTQtZTZhMTZiMjUtNjk3Mi00N2Y1LWEwMjYtNTNkMmJiOGFhMzU2LmpwZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAxMjclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMTI3VDE3Mzc0NVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWYwYzhlMzZjMGMwMGMwNGM4ZmUyOGI2MWU0OTA0NDQxOWZkZGU0ZjUxMDdjYzEwMTI0MTdkNTlkZDIyNmJmMzUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.9X6-V94W9gvyKSqMr9xSgVkEm3b_D9B9PXJ3FSFeyZU)
> 
> sorry, I didn't encounter this ever

Thanks anyway

> I find a solution here [#1659 (comment)](https://github.com/EleutherAI/lm-evaluation-harness/issues/1659#issuecomment-2126421186) First, download the python script `mmlu_no_train.py` in hails/mmlu_no_train and store in local dict. Then, change the `dataset_path: hails/mmlu_no_train` to `dataset_path: /local_path` in `_default_template_yaml` which contains the python script. Finally, change `_URL = "https://huggingface.co/datasets/cais/mmlu/resolve/main/data.tar"` to `_URL = "https://hf-mirror.com/datasets/cais/mmlu/resolve/main/data.tar"` in local file `mmlu_no_train.py`

You save me time and time again, thank you.

> I find a solution here [#1659 (comment)](https://github.com/EleutherAI/lm-evaluation-harness/issues/1659#issuecomment-2126421186) First, download the python script `mmlu_no_train.py` in hails/mmlu_no_train and store in local dict. Then, change the `dataset_path: hails/mmlu_no_train` to `dataset_path: /local_path` in `_default_template_yaml` which contains the python script. Finally, change `_URL = "https://huggingface.co/datasets/cais/mmlu/resolve/main/data.tar"` to `_URL = "https://hf-mirror.com/datasets/cais/mmlu/resolve/main/data.tar"` in local file `mmlu_no_train.py`我在这里找到了一个解决办法。首先，在 hails/mmlu_no_train 中下载 python 脚本，并将其存储在本地目录中。然后，在包含 Python 脚本的 中将 改为 。最后，在本地文件中将 更改为

it really works!

Thx！

Get Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: xinmin ***@***.***>
Sent: Tuesday, April 15, 2025 2:12:57 PM
To: EleutherAI/lm-evaluation-harness ***@***.***>
Cc: JiahaoWuGit ***@***.***>; Mention ***@***.***>
Subject: Re: [EleutherAI/lm-evaluation-harness] Traceback "TypeError: 'NoneType' " when running the task "MMLU" (Issue #2608)


I find a solution here #1659 (comment)<https://github.com/EleutherAI/lm-evaluation-harness/issues/1659#issuecomment-2126421186> First, download the python script mmlu_no_train.py in hails/mmlu_no_train and store in local dict. Then, change the dataset_path: hails/mmlu_no_train to dataset_path: /local_path in _default_template_yaml which contains the python script. Finally, change _URL = "https://huggingface.co/datasets/cais/mmlu/resolve/main/data.tar" to _URL = "https://hf-mirror.com/datasets/cais/mmlu/resolve/main/data.tar" in local file mmlu_no_train.py我在这里找到了一个解决办法。首先，在 hails/mmlu_no_train 中下载 python 脚本，并将其存储在本地目录中。然后，在包含 Python 脚本的 中将 改为 。最后，在本地文件中将 更改为

it really works!

―
Reply to this email directly, view it on GitHub<https://github.com/EleutherAI/lm-evaluation-harness/issues/2608#issuecomment-2803925878>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AKTPWG7ANGPO2YFAU4LAFD32ZSPOTAVCNFSM6AAAAABUSIOWVGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQMBTHEZDKOBXHA>.
You are receiving this because you were mentioned.Message ID: ***@***.***>

[https://avatars.githubusercontent.com/u/131684511?s=20&v=4]hhlxm left a comment (EleutherAI/lm-evaluation-harness#2608)<https://github.com/EleutherAI/lm-evaluation-harness/issues/2608#issuecomment-2803925878>

I find a solution here #1659 (comment)<https://github.com/EleutherAI/lm-evaluation-harness/issues/1659#issuecomment-2126421186> First, download the python script mmlu_no_train.py in hails/mmlu_no_train and store in local dict. Then, change the dataset_path: hails/mmlu_no_train to dataset_path: /local_path in _default_template_yaml which contains the python script. Finally, change _URL = "https://huggingface.co/datasets/cais/mmlu/resolve/main/data.tar" to _URL = "https://hf-mirror.com/datasets/cais/mmlu/resolve/main/data.tar" in local file mmlu_no_train.py我在这里找到了一个解决办法。首先，在 hails/mmlu_no_train 中下载 python 脚本，并将其存储在本地目录中。然后，在包含 Python 脚本的 中将 改为 。最后，在本地文件中将 更改为

it really works!

―
Reply to this email directly, view it on GitHub<https://github.com/EleutherAI/lm-evaluation-harness/issues/2608#issuecomment-2803925878>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AKTPWG7ANGPO2YFAU4LAFD32ZSPOTAVCNFSM6AAAAABUSIOWVGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQMBTHEZDKOBXHA>.
You are receiving this because you were mentioned.Message ID: ***@***.***>


> > > > > > Hi! [@JiahaoWuGit](https://github.com/JiahaoWuGit) I'm sorry to hear that. Please check the `dataset_path` in `mmlu/default/_default_template_yaml` is correct or try to use the absolute path. Meanwhile, do you set the `export HF_ENDPOINT=https://hf-mirror.com` in `~/.bashrc` or run `export HF_ENDPOINT=https://hf-mirror.com` before lm_eval?
> > > > > 
> > > > > 
> > > > > Here is my settings: dataset_path: /data/Backdoor-Enhanced-Alignment/lm_eval_dir #a copy of `cais/mmlu` with no auxiliary_train split test_split: test fewshot_split: dev fewshot_config: sampler: first_n output_type: multiple_choice doc_to_text: "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:" doc_to_choice: ["A", "B", "C", "D"] doc_to_target: answer metric_list:
> > > > > 
> > > > > * metric: acc
> > > > >   aggregation: mean
> > > > >   higher_is_better: true
> > > > >   metadata:
> > > > >   version: 1.0
> > > > > 
> > > > > # dataset_kwargs:
> > > > 
> > > > 
> > > > If the `mmlu_no_train.py` in the directory `/data/Backdoor-Enhanced-Alignment/lm_eval_dir`, change the dataset_path to `/data/Backdoor-Enhanced-Alignment/lm_eval_dir/mmlu_no_train`
> > > 
> > > 
> > > Thanks! As you suggested, I eliminated the previous error. However, I got a new error as follows, and the screenshot of the directory is also attached afterwards.
> > > File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/**init**.py", line 398module> cli_evaluate() File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/**main**.py", line 382, in cli_evaluate results = evaluator.simple_evaluate( File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/l results = evaluator.simple_evaluate( File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/utils.py", line 402, in _wrapper return fn(*args, **kwargs) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/evaluator.py", line 235, in simple_evaluate task_dict = get_task_dict(tasks, task_manager) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/**init**.py", line 618, in get_task_dict task_name_from_string_dict = task_manager.load_task_or_group( File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/**init**.py", line 414, in load_task_or_group m_eval/tasks/**init**.py", line 414, in load_task_or_group File "/root/anaconda3/envs/self-instruct/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/**main**.py", line 461, in cli_evaluate() File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/**main**.py", line 382, in cli_evaluate results = evaluator.simple_evaluate( File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/utils.py", line 402, in _wrapper return fn(*args, **kwargs) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/evaluator.py", line 235, in simple_evaluate task_dict = get_task_dict(tasks, task_manager) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/**init**.py", line 618, in get_task_dict task_name_from_string_dict = task_manager.load_task_or_group( File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/**init**.py", line 414, in load_task_or_group collections.ChainMap(*map(self._load_individual_task_or_group, task_list)) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/**init**.py", line 398, in _load_individual_task_or_group group_name: dict(collections.ChainMap(*map(fn, reversed(subtask_list)))) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/**init**.py", line 398, in _load_individual_task_or_group group_name: dict(collections.ChainMap(*map(fn, reversed(subtask_list)))) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/**init**.py", line 314, in _load_individual_task_or_group return _load_task(task_config, task=name_or_config) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/**init**.py", line 280, in _load_task task_object = ConfigurableTask(config=config) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/api/task.py", line 820, in **init** self.download(self.config.dataset_kwargs) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/api/task.py", line 928, in download self.dataset = datasets.load_dataset( File "/root/anaconda3/envs/self-instruct/lib/python3.10/site-packages/datasets/load.py", line 2129, in load_dataset builder_instance = load_dataset_builder( File "/root/anaconda3/envs/self-instruct/lib/python3.10/site-packages/datasets/load.py", line 1849, in load_dataset_builder dataset_module = dataset_module_factory( File "/root/anaconda3/envs/self-instruct/lib/python3.10/site-packages/datasets/load.py", line 1737, in dataset_module_factory raise FileNotFoundError(f"Couldn't find any data file at {relative_to_absolute_path(path)}.") FileNotFoundError: Couldn't find any data file at /data/Backdoor-Enhanced-Alignment/lm_eval_dir/mmlu_no_train.
> > > ![Image](https://github.com/user-attachments/assets/714a00a8-c038-4f5b-954d-49e8305d9bfa) ![Image](https://private-user-images.githubusercontent.com/44497691/406990994-e6a16b25-6972-47f5-a026-53d2bb8aa356.jpg?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3Mzc5OTk3NjUsIm5iZiI6MTczNzk5OTQ2NSwicGF0aCI6Ii80NDQ5NzY5MS80MDY5OTA5OTQtZTZhMTZiMjUtNjk3Mi00N2Y1LWEwMjYtNTNkMmJiOGFhMzU2LmpwZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAxMjclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMTI3VDE3Mzc0NVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWYwYzhlMzZjMGMwMGMwNGM4ZmUyOGI2MWU0OTA0NDQxOWZkZGU0ZjUxMDdjYzEwMTI0MTdkNTlkZDIyNmJmMzUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.9X6-V94W9gvyKSqMr9xSgVkEm3b_D9B9PXJ3FSFeyZU)
> > 
> > 
> > sorry, I didn't encounter this ever
> 
> Thanks anyway

I have met the same situation like "ValueError: BuilderConfig 'world_religions' not found. Available: ['default']", and I find it should ensure the `dataset_path` end of "hails/mmlu_no_train" rather your `lm_eval_dir` here. The dataset path should contain subdir `all` including the parquet files and the `mmlu_no_train.py`

thanks for your supplement!




------------------&nbsp;原始邮件&nbsp;------------------
发件人:                                                                                                                        "EleutherAI/lm-evaluation-harness"                                                                                    ***@***.***&gt;;
发送时间:&nbsp;2025年4月17日(星期四) 下午5:30
***@***.***&gt;;
***@***.***&gt;;"State ***@***.***&gt;;
主题:&nbsp;Re: [EleutherAI/lm-evaluation-harness] Traceback "TypeError: 'NoneType' " when running the task "MMLU"  (Issue #2608)



         
Hi! @JiahaoWuGit I'm sorry to hear that. Please check the dataset_path in mmlu/default/_default_template_yaml is correct or try to use the absolute path. Meanwhile, do you set the export HF_ENDPOINT=https://hf-mirror.com in ~/.bashrc or run export HF_ENDPOINT=https://hf-mirror.com before lm_eval?
  
Here is my settings: dataset_path: /data/Backdoor-Enhanced-Alignment/lm_eval_dir #a copy of cais/mmlu with no auxiliary_train split test_split: test fewshot_split: dev fewshot_config: sampler: first_n output_type: multiple_choice doc_to_text: "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:" doc_to_choice: ["A", "B", "C", "D"] doc_to_target: answer metric_list:
  
metric: acc
 aggregation: mean
 higher_is_better: true
 metadata:
 version: 1.0
  
dataset_kwargs:
  
If the mmlu_no_train.py in the directory /data/Backdoor-Enhanced-Alignment/lm_eval_dir, change the dataset_path to /data/Backdoor-Enhanced-Alignment/lm_eval_dir/mmlu_no_train
  
Thanks! As you suggested, I eliminated the previous error. However, I got a new error as follows, and the screenshot of the directory is also attached afterwards.
 File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/init.py", line 398module&gt; cli_evaluate() File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/main.py", line 382, in cli_evaluate results = evaluator.simple_evaluate( File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/l results = evaluator.simple_evaluate( File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/utils.py", line 402, in _wrapper return fn(*args, **kwargs) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/evaluator.py", line 235, in simple_evaluate task_dict = get_task_dict(tasks, task_manager) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/init.py", line 618, in get_task_dict task_name_from_string_dict = task_manager.load_task_or_group( File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/init.py", line 414, in load_task_or_group m_eval/tasks/init.py", line 414, in load_task_or_group File "/root/anaconda3/envs/self-instruct/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/main.py", line 461, in cli_evaluate() File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/main.py", line 382, in cli_evaluate results = evaluator.simple_evaluate( File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/utils.py", line 402, in _wrapper return fn(*args, **kwargs) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/evaluator.py", line 235, in simple_evaluate task_dict = get_task_dict(tasks, task_manager) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/init.py", line 618, in get_task_dict task_name_from_string_dict = task_manager.load_task_or_group( File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/init.py", line 414, in load_task_or_group collections.ChainMap(*map(self._load_individual_task_or_group, task_list)) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/init.py", line 398, in _load_individual_task_or_group group_name: dict(collections.ChainMap(*map(fn, reversed(subtask_list)))) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/init.py", line 398, in _load_individual_task_or_group group_name: dict(collections.ChainMap(*map(fn, reversed(subtask_list)))) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/init.py", line 314, in _load_individual_task_or_group return _load_task(task_config, task=name_or_config) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/init.py", line 280, in _load_task task_object = ConfigurableTask(config=config) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/api/task.py", line 820, in init self.download(self.config.dataset_kwargs) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/api/task.py", line 928, in download self.dataset = datasets.load_dataset( File "/root/anaconda3/envs/self-instruct/lib/python3.10/site-packages/datasets/load.py", line 2129, in load_dataset builder_instance = load_dataset_builder( File "/root/anaconda3/envs/self-instruct/lib/python3.10/site-packages/datasets/load.py", line 1849, in load_dataset_builder dataset_module = dataset_module_factory( File "/root/anaconda3/envs/self-instruct/lib/python3.10/site-packages/datasets/load.py", line 1737, in dataset_module_factory raise FileNotFoundError(f"Couldn't find any data file at {relative_to_absolute_path(path)}.") FileNotFoundError: Couldn't find any data file at /data/Backdoor-Enhanced-Alignment/lm_eval_dir/mmlu_no_train.
 56091737993173_.pic.jpg (view on web) 
  
sorry, I didn't encounter this ever
  
Thanks anyway
  
I have met the same situation like "ValueError: BuilderConfig 'world_religions' not found. Available: ['default']", and I find it should ensure the dataset_path end of "hails/mmlu_no_train" rather your lm_eval_dir here. The dataset path should contain subdir all including the parquet files and the mmlu_no_train.py

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you modified the open/close state.Message ID: ***@***.***&gt;
  GeekOrangeLuYao left a comment (EleutherAI/lm-evaluation-harness#2608)
       
Hi! @JiahaoWuGit I'm sorry to hear that. Please check the dataset_path in mmlu/default/_default_template_yaml is correct or try to use the absolute path. Meanwhile, do you set the export HF_ENDPOINT=https://hf-mirror.com in ~/.bashrc or run export HF_ENDPOINT=https://hf-mirror.com before lm_eval?
  
Here is my settings: dataset_path: /data/Backdoor-Enhanced-Alignment/lm_eval_dir #a copy of cais/mmlu with no auxiliary_train split test_split: test fewshot_split: dev fewshot_config: sampler: first_n output_type: multiple_choice doc_to_text: "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:" doc_to_choice: ["A", "B", "C", "D"] doc_to_target: answer metric_list:
  
metric: acc
 aggregation: mean
 higher_is_better: true
 metadata:
 version: 1.0
  
dataset_kwargs:
  
If the mmlu_no_train.py in the directory /data/Backdoor-Enhanced-Alignment/lm_eval_dir, change the dataset_path to /data/Backdoor-Enhanced-Alignment/lm_eval_dir/mmlu_no_train
  
Thanks! As you suggested, I eliminated the previous error. However, I got a new error as follows, and the screenshot of the directory is also attached afterwards.
 File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/init.py", line 398module&gt; cli_evaluate() File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/main.py", line 382, in cli_evaluate results = evaluator.simple_evaluate( File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/l results = evaluator.simple_evaluate( File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/utils.py", line 402, in _wrapper return fn(*args, **kwargs) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/evaluator.py", line 235, in simple_evaluate task_dict = get_task_dict(tasks, task_manager) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/init.py", line 618, in get_task_dict task_name_from_string_dict = task_manager.load_task_or_group( File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/init.py", line 414, in load_task_or_group m_eval/tasks/init.py", line 414, in load_task_or_group File "/root/anaconda3/envs/self-instruct/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/main.py", line 461, in cli_evaluate() File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/main.py", line 382, in cli_evaluate results = evaluator.simple_evaluate( File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/utils.py", line 402, in _wrapper return fn(*args, **kwargs) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/evaluator.py", line 235, in simple_evaluate task_dict = get_task_dict(tasks, task_manager) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/init.py", line 618, in get_task_dict task_name_from_string_dict = task_manager.load_task_or_group( File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/init.py", line 414, in load_task_or_group collections.ChainMap(*map(self._load_individual_task_or_group, task_list)) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/init.py", line 398, in _load_individual_task_or_group group_name: dict(collections.ChainMap(*map(fn, reversed(subtask_list)))) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/init.py", line 398, in _load_individual_task_or_group group_name: dict(collections.ChainMap(*map(fn, reversed(subtask_list)))) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/init.py", line 314, in _load_individual_task_or_group return _load_task(task_config, task=name_or_config) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/tasks/init.py", line 280, in _load_task task_object = ConfigurableTask(config=config) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/api/task.py", line 820, in init self.download(self.config.dataset_kwargs) File "/data/Backdoor-Enhanced-Alignment/data_generation/AGen/utility_eval/lm-evaluation-harness/lm_eval/api/task.py", line 928, in download self.dataset = datasets.load_dataset( File "/root/anaconda3/envs/self-instruct/lib/python3.10/site-packages/datasets/load.py", line 2129, in load_dataset builder_instance = load_dataset_builder( File "/root/anaconda3/envs/self-instruct/lib/python3.10/site-packages/datasets/load.py", line 1849, in load_dataset_builder dataset_module = dataset_module_factory( File "/root/anaconda3/envs/self-instruct/lib/python3.10/site-packages/datasets/load.py", line 1737, in dataset_module_factory raise FileNotFoundError(f"Couldn't find any data file at {relative_to_absolute_path(path)}.") FileNotFoundError: Couldn't find any data file at /data/Backdoor-Enhanced-Alignment/lm_eval_dir/mmlu_no_train.
 56091737993173_.pic.jpg (view on web) 
  
sorry, I didn't encounter this ever
  
Thanks anyway
  
I have met the same situation like "ValueError: BuilderConfig 'world_religions' not found. Available: ['default']", and I find it should ensure the dataset_path end of "hails/mmlu_no_train" rather your lm_eval_dir here. The dataset path should contain subdir all including the parquet files and the mmlu_no_train.py
 
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you modified the open/close state.Message ID: ***@***.***&gt;