Repository: EleutherAI/lm-evaluation-harness
Issue: Error loading MMLU 'prehistory' config: BuilderConfig not found (available: ['default'])
Root cause: Configuration Error

(eval) [ruioliao@gpuh8001 ruioliao]$ bash /data/work/ruioliao/topo_agent/evaluate/benchmark/slurm_script/test_normal.sh
2025-02-27:14:06:39,371 INFO     [utils.py:146] Note: detected 192 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
2025-02-27:14:06:39,371 INFO     [utils.py:149] Note: NumExpr detected 192 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16.
2025-02-27:14:06:39,371 INFO     [utils.py:162] NumExpr defaulting to 16 threads.
2025-02-27:14:06:42,396 INFO     [config.py:58] PyTorch version 2.4.0+cu121 available.
2025-02-27:14:07:16,913 INFO     [__main__.py:156] Verbosity set to INFO
2025-02-27:14:07:39,888 WARNING  [__init__.py:194] Some tasks could not be loaded due to missing dependencies. Run with `--verbosity DEBUG` for full details.
2025-02-27:14:07:45,802 WARNING  [__init__.py:194] Some tasks could not be loaded due to missing dependencies. Run with `--verbosity DEBUG` for full details.
2025-02-27:14:07:45,805 INFO     [__main__.py:229] Selected Tasks: ['mmlu']
2025-02-27:14:07:45,891 WARNING  [evaluator.py:89] generation_kwargs specified through cli, these settings will be used over set parameters in yaml tasks.
2025-02-27:14:07:49,557 WARNING  [other.py:335] Detected kernel version 3.10.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
2025-02-27:14:07:49,558 INFO     [huggingface.py:148] Using device 'cuda'
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:23<00:00,  5.82s/it]
Resolving data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 174/174 [00:00<00:00, 7248.37it/s]
Resolving data files: 100%|███████████████████████████████████████████████████████████████████████████████████████| 116/116 [00:00<00:00, 106720.61it/s]
Resolving data files: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 58/58 [00:00<00:00, 9363.01it/s]
Traceback (most recent call last):
  File "/data/home/ruioliao/anaconda3/envs/eval/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/data/home/ruioliao/lm-evaluation-harness-main/lm_eval/__main__.py", line 231, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/data/home/ruioliao/lm-evaluation-harness-main/lm_eval/utils.py", line 415, in _wrapper
    return fn(*args, **kwargs)
  File "/data/home/ruioliao/lm-evaluation-harness-main/lm_eval/evaluator.py", line 122, in simple_evaluate
    task_dict = lm_eval.tasks.get_task_dict(tasks)
  File "/data/home/ruioliao/lm-evaluation-harness-main/lm_eval/tasks/__init__.py", line 255, in get_task_dict
    task_obj = get_task_dict(task_name)
  File "/data/home/ruioliao/lm-evaluation-harness-main/lm_eval/tasks/__init__.py", line 255, in get_task_dict
    task_obj = get_task_dict(task_name)
  File "/data/home/ruioliao/lm-evaluation-harness-main/lm_eval/tasks/__init__.py", line 275, in get_task_dict
    task_name: get_task(task_name=task_element, config=config),
  File "/data/home/ruioliao/lm-evaluation-harness-main/lm_eval/tasks/__init__.py", line 217, in get_task
    return TASK_REGISTRY[task_name](config=config)
  File "/data/home/ruioliao/lm-evaluation-harness-main/lm_eval/api/task.py", line 622, in __init__
    self.download(self.config.dataset_kwargs)
  File "/data/home/ruioliao/lm-evaluation-harness-main/lm_eval/api/task.py", line 716, in download
    self.dataset = datasets.load_dataset(
  File "/data/home/ruioliao/anaconda3/envs/eval/lib/python3.10/site-packages/datasets/load.py", line 2587, in load_dataset
    builder_instance = load_dataset_builder(
  File "/data/home/ruioliao/anaconda3/envs/eval/lib/python3.10/site-packages/datasets/load.py", line 2296, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
  File "/data/home/ruioliao/anaconda3/envs/eval/lib/python3.10/site-packages/datasets/builder.py", line 374, in __init__
    self.config, self.config_id = self._create_builder_config(
  File "/data/home/ruioliao/anaconda3/envs/eval/lib/python3.10/site-packages/datasets/builder.py", line 599, in _create_builder_config
    raise ValueError(
ValueError: BuilderConfig 'prehistory' not found. Available: ['default']   




Hi! I can't reproduce this. notebook [here](https://colab.research.google.com/drive/1HaPp6ZBM6o1jfFxjRg3ezG5GRsb_dohQ?usp=sharing)

> Hi! I can't reproduce this. notebook [here](https://colab.research.google.com/drive/1HaPp6ZBM6o1jfFxjRg3ezG5GRsb_dohQ?usp=sharing)

Thanks for your reply. I want to load the MMLU dataset offline because our machine doesn’t have internet access. I downloaded the dataset following the same approach as in data_loader.py and updated the default YAML configuration, setting the dataset path to the cache directory where I stored the downloaded data. However, I still encountered this issue.

> > Hi! I can't reproduce this. notebook [here](https://colab.research.google.com/drive/1HaPp6ZBM6o1jfFxjRg3ezG5GRsb_dohQ?usp=sharing)
> 
> Thanks for your reply. I want to load the MMLU dataset offline because our machine doesn’t have internet access. I downloaded the dataset following the same approach as in data_loader.py and updated the default YAML configuration, setting the dataset path to the cache directory where I stored the downloaded data. However, I still encountered this issue.

@ruio248  Hi, I encountered similar error as you when I try to evaluate some hf models on the XQuAD benchmark. Have you ever figured out any approach to solve the problem?

Closing as there's now a meta-issue about this.