Repository: EleutherAI/lm-evaluation-harness
Issue: Load dataset error
Root cause: External Dependency Breakage

I use the following command to run lm-eval:
```
accelerate launch --multi-gpu --num_processes 2 \
    -m lm_eval --model hf \
    --model_args pretrained=${local_model_path} \
    --tasks mmlu \
    --batch_size 8 \
    --log_samples \
    --output_path ${output_path} \
    --trust_remote_code
```

but meet the problem:
```
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/__main__.py", line 461, in <module>
    cli_evaluate()
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/__main__.py", line 382, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/evaluator.py", line 235, in simple_evaluate
    task_dict = get_task_dict(tasks, task_manager)
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/tasks/__init__.py", line 618, in get_task_dict
    task_name_from_string_dict = task_manager.load_task_or_group(
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/tasks/__init__.py", line 414, in load_task_or_group
    collections.ChainMap(*map(self._load_individual_task_or_group, task_list))
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/tasks/__init__.py", line 398, in _load_individual_task_or_group
    group_name: dict(collections.ChainMap(*map(fn, reversed(subtask_list))))
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/tasks/__init__.py", line 398, in _load_individual_task_or_group
    group_name: dict(collections.ChainMap(*map(fn, reversed(subtask_list))))
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/tasks/__init__.py", line 314, in _load_individual_task_or_group
    return _load_task(task_config, task=name_or_config)
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/tasks/__init__.py", line 280, in _load_task
    task_object = ConfigurableTask(config=config)
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/api/task.py", line 818, in __init__
    self.download(self.config.dataset_kwargs)
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/api/task.py", line 925, in download
    self.dataset = datasets.load_dataset(
  File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 2132, in load_dataset
    builder_instance = load_dataset_builder(
  File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 1890, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
TypeError: 'NoneType' object is not callable
```

`lm-eval==0.4.5` and `datasets==3.1.0`. 

I also tried installing different versions of lm-eval and datasets, but it didn't work. Do you have any suggestions for solving the problem?

same error here, have you solved it?

You can download datasets with VPN or replace the .py file from the huggingface repo with the local empty one.

> You can download datasets with VPN or replace the .py file from the huggingface repo with the local empty one.

Hi thx for the reply, the download env is in US already, but the error is still there. By saying huggingface repo, do you mean the huggingface source code in the python env folder?

By downgrading the package datasets version from 3.1.0 to 2.16.0, the problem is solved.

It seems downgrade datasets still does not work for me.

> It seems downgrade datasets still does not work for me.

+1, it doesn’t work for me either.


Or guys could consider using this repo as an alternative: [meta's evaluation repo](https://github.com/meta-llama/llama-recipes/tree/main/tools/benchmarks/llm_eval_harness/meta_eval.)