Repository: EleutherAI/lm-evaluation-harness
Issue: Error in loading from HF datasets
Root cause: External Dependency Breakage

Multiple users are reporting problems loading datasets (refs: #2743, #2793, #2752), particularly those behind restrictive firewalls (potentially China). I haven't been able to reproduce these issues in my environment. This could act as a meta issue to consolidate all these related reports and find a common solution.

My understanding was setting `HF_ENDPOINT` environment variable for example:

```export HF_ENDPOINT="https://hf-mirror.com"```,

would fix things, but apparently it's not working for everyone.

As a diagnostic step, check if you can load datasets directly using the HF datasets library. For example:

```python
import datasets
df = datasets.load_dataset("ceval/ceval-exam", "accountant")
```
If this works (either from HF remote or locally), then our library should also work since we use `load_dataset` internally. See our [Dataset configuration options](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/task_guide.md#configurations) documentation for details on how to pass these and other parameters from task configs.

Some users have had success in downloading the dataset and loading it locally, though this becomes more complicated when the dataset has multiple subsets. cc: @lhoestq 

Let's use this thread to collect any successful solutions or workarounds.

Hi ! Some of these datasets are script-based datasets. Maybe the issue comes from here ? I see some of them have hardcoded HF URLs in their scripts:

- https://huggingface.co/datasets/ceval/ceval-exam/blob/main/ceval-exam.py
- https://huggingface.co/datasets/Muennighoff/xwinograd/blob/main/xwinograd.py

We could try to make Parquet versions of those datasets and ask users to try them and see if it fixes the issue ? Script-based datasets are not ideal because they require `trust_remote_code=True` and can contain unexpected code/behavior.

@baberabb Hi! I think I've solved the problem under your guidance. Thanks!
However, I notice that there still exists some error with the benchmark XWinograd and I suppose the issue may be consistent with what @lhoestq said because I could make it on XQuAD, XNLI,  XCopa but only fail on XWinograd. And I also fail to run the test scripts that you've given normally (By the way, I think the correct method should be "load_dataset" instead of "load_datasets"). Wish these information could help.

Thanks @lhoestq! Appreciate you taking the time to root out the potential issue. And generally moving the datasets to parquet sounds like a general best-practice. I'll try converting them as they are identified.

@baberabb Hi, sorry to disturb you amidst your busy schedule but I wonder if you have any advice or plan to fix this kind of issue? I found the evaluation of many other benchmarks also influenced by this issue such as iwslt2017, XStoryCloze and flores.

> [@baberabb](https://github.com/baberabb) Hi, sorry to disturb you amidst your busy schedule but I wonder if you have any advice or plan to fix this kind of issue? I found the evaluation of many other benchmarks also influenced by this issue such as iwslt2017, XStoryCloze and flores.

The core issue identified above, the use of scripts that hardcode a path to a hf page that may not be accessible to you, is not something we can solve. In such cases the dataset that is currently hosted on HF cannot be loaded. We can make progress on a case-by-case basis by creating, uploading, and linking to parquet files instead, but that's time consuming and not future-proof.

The HF datasets loader breaks in these cases, and the ideal solution would be on the HF side to enable such users. I don't see anything we can do to effect widespread improvement in this regard. At the end of the day, if the dataset can't be loaded we can't use it for evaluations.

Unless you've discovered something further @lhoestq? 

@StellaAthena Thanks for the response! Actually, I've figured out a easier method to solve this kind of issue even though I'm not so sure if it's a universal way. I'll take the XWinograd benchmark as an example:
1. Download the dataset from huggingface to the local. Make sure the local script (xwinograd.py) is not empty, otherwise we should copy the original code from the script on huggingface hub. (I found that the script I downloaded is empty maybe due to some setting of huggingface-cli tools)
2. Modify the _URL variable in the xwinograd.py into the local path or directory.
For example, I notice that the data on "https://huggingface.co/datasets/Muennighoff/xwinograd/raw/main/test/{lang}.jsonl" is accessible exactly in the directory "xwinograd/test" so I change the _URL into "./test/{lang}.jsonl"
![Image](https://github.com/user-attachments/assets/2dd60e29-e2ef-4ab4-a210-525d7c32d87e)
3. Modify the dataset_path of xwinograd_common_yaml into that local path of XWinograd directory.
![Image](https://github.com/user-attachments/assets/b8f598e3-2d4d-49f4-ba12-3aa52bee4405)
4. Run the lm_eval command line and remember the args "--trust_remote_code" 
![Image](https://github.com/user-attachments/assets/30bf644c-9961-4bdc-b2fe-381d14ed0561)

I'm not sure how it works but it could. Wish these information could help for you.

@baberabb Hi, I set the environment variable `HF_ENDPOINT` with `export HF_ENDPOINT="https://hf-mirror.com"`, and problem like 'NoneType object is not callable' is still existing with only `load_dataset`
I found the reason may be that the script(e.g. `super_glue.py`) it downloads is empty, this can also occur when using `huggingface-cli` to download the datasets.
An easy way for users to avoid this problem is just to clone the datasets directly with git and set the path to the local path. This method is at least useful for me.

> We can make progress on a case-by-case basis by creating, uploading, and linking to parquet files instead, but that's time consuming and not future-proof.

There aren't that many datasets that are scripts based actually ! I opened PRs to fix the reported ones:

- https://huggingface.co/datasets/Muennighoff/xwinograd/discussions/4 (edit: merged)
- https://huggingface.co/datasets/aps/super_glue/discussions/16 (edit: merged)
- https://huggingface.co/datasets/ceval/ceval-exam/discussions/6 (edit: merged)

(feel free to comment / thumb up there !)

If you want to open more PRs on other datasets, feel free to use the CLI. For example I used

```
datasets-cli convert_to_parquet --trust_remote_code Muennighoff/xwinograd
```

(btw in the meantime you can use the parquet data from those PRs instead of the script on the `main` branch of the datasets by passing e.g. `revision="refs/pr/4"` to `load_dataset()` in the lm_eval `Task.download` function)


> [@StellaAthena](https://github.com/StellaAthena) Thanks for the response! Actually, I've figured out a easier method to solve this kind of issue even though I'm not so sure if it's a universal way. I'll take the XWinograd benchmark as an example:
> 
> [...]
> 
> I'm not sure how it works but it could. Wish these information could help for you.

I'm pretty surprised that this works since I was tinkering with the `transformers` library and couldn't get it to run a model that requires `true_remote_code` without an internet connection. Maybe I was doing it wrong, but it didn't seem like I could stop it from trying to pull the script off the internet.

> > We can make progress on a case-by-case basis by creating, uploading, and linking to parquet files instead, but that's time consuming and not future-proof.
> 
> There aren't that many datasets that are scripts based actually ! I opened PRs to fix the reported ones:
> 
> * https://huggingface.co/datasets/Muennighoff/xwinograd/discussions/4
> * https://huggingface.co/datasets/aps/super_glue/discussions/16
> * https://huggingface.co/datasets/ceval/ceval-exam/discussions/6 (edit: already merged)
> 
> (feel free to comment / thumb up there !)
> 
> If you want to open more PRs on other datasets, feel free to use the CLI. For example I used
> 
> ```
> datasets-cli convert_to_parquet --trust_remote_code Muennighoff/xwinograd
> ```
> 
> (btw in the meantime you can use the parquet data from those PRs instead of the script on the `main` branch of the datasets by passing e.g. `revision="refs/pr/4"` to `load_dataset()` in the lm_eval `Task.download` function)

Hello, I'm not good at calling apis. I want to run mmlu test, and I passed the parameter `revision="refs/convert/parquet"` to
```
def download(self, dataset_kwargs: Optional[Dict[str, Any]] = None) -> None:
        print(self.DATASET_PATH, self.DATASET_NAME, dataset_kwargs)
        self.dataset = datasets.load_dataset(
            path=self.DATASET_PATH,
            name=self.DATASET_NAME,
            revision="refs/convert/parquet"
            **dataset_kwargs if dataset_kwargs is not None else {},
        )
```
But it reports
```
  File "/home/zjnyly/miniconda3/envs/py310/lib/python3.10/site-packages/lm_eval/tasks/__init__.py", line 280, in _load_task
    task_object = ConfigurableTask(config=config)
  File "/home/zjnyly/miniconda3/envs/py310/lib/python3.10/site-packages/lm_eval/api/task.py", line 819, in __init__
    self.download(self.config.dataset_kwargs)
  File "/home/zjnyly/miniconda3/envs/py310/lib/python3.10/site-packages/lm_eval/api/task.py", line 930, in download
    revision="refs/convert/parquet"
TypeError: unsupported operand type(s) for ** or pow(): 'str' and 'dict'
```
It seems a version is needed(`revision: Optional[Union[str, Version]] = None,`).  When I changed it to `revision= {"refs/convert/parquet":datasets.Version("1.0.0")}`, it still reports 
```
  File "/home/zjnyly/miniconda3/envs/py310/lib/python3.10/site-packages/lm_eval/api/task.py", line 819, in __init__
    self.download(self.config.dataset_kwargs)
  File "/home/zjnyly/miniconda3/envs/py310/lib/python3.10/site-packages/lm_eval/api/task.py", line 927, in download
    self.dataset = datasets.load_dataset(
  File "/home/zjnyly/miniconda3/envs/py310/lib/python3.10/site-packages/datasets/load.py", line 2129, in load_dataset
    builder_instance = load_dataset_builder(
  File "/home/zjnyly/miniconda3/envs/py310/lib/python3.10/site-packages/datasets/load.py", line 1886, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
TypeError: 'NoneType' object is not callable
```
 What should I do? Thanks! 

Hi ! it looks like you're missing a comma:

```diff
def download(self, dataset_kwargs: Optional[Dict[str, Any]] = None) -> None:
        print(self.DATASET_PATH, self.DATASET_NAME, dataset_kwargs)
        self.dataset = datasets.load_dataset(
            path=self.DATASET_PATH,
            name=self.DATASET_NAME,
-           revision="refs/convert/parquet"
+           revision="refs/convert/parquet",
            **dataset_kwargs if dataset_kwargs is not None else {},
        )
```

Which dataset are you trying to load ?

> Hi ! it looks like you're missing a comma:
> 
> def download(self, dataset_kwargs: Optional[Dict[str, Any]] = None) -> None:
>         print(self.DATASET_PATH, self.DATASET_NAME, dataset_kwargs)
>         self.dataset = datasets.load_dataset(
>             path=self.DATASET_PATH,
>             name=self.DATASET_NAME,
> -           revision="refs/convert/parquet"
> +           revision="refs/convert/parquet",
>             **dataset_kwargs if dataset_kwargs is not None else {},
>         )
> Which dataset are you trying to load ?

Thanks for your quick reply!
I tried to evaluate the MMLU task, and the corresponding dataset required by lm_eval is super_glue.
I think I found out why. In my case, the dataset script was not fully downloaded.
```
(py310) zjnyly@ric-MS-7D25:~$ ls ~/.cache/huggingface/hub/datasets--super_glue/snapshots/b051de3f07b5fd5ab80398a4836458db56234e24/
README.md  super_glue.py
``` 
I mean, the super_glue.py file was empty!
I'm not sure why this happened, but that's the reason it returned a NoneType.
When I manually filled the empty file with a copy from the webpage, it simply worked.
Hope this helps others as well.
And thanks again for your help!

Hi ! quick update: all the reported datasets have been converted to parquet, so it should be all good now :)

edit: update from july 2025 - `datasets` 4 dropped support for `trust_remote_code` completely

@lhoestq Thank you!

> Hi ! Some of these datasets are script-based datasets. Maybe the issue comes from here ? I see some of them have hardcoded HF URLs in their scripts:
> 
> * https://huggingface.co/datasets/ceval/ceval-exam/blob/main/ceval-exam.py
> * https://huggingface.co/datasets/Muennighoff/xwinograd/blob/main/xwinograd.py
> 
> We could try to make Parquet versions of those datasets and ask users to try them and see if it fixes the issue ? Script-based datasets are not ideal because they require `trust_remote_code=True` and can contain unexpected code/behavior.

@lhoestq The link has been 404, any new now?

The scripts have been removed (and `datasets` 4 dropped support for scripts anyway), now all the datasets are only Parquet files :)

Though if you were curious about the scripts contents, you can retrieve them from the Git history. For example here if the link in the Git history:
- https://huggingface.co/datasets/ceval/ceval-exam/blob/3923b519fd180e689d0961bf3a032ece929742f3/ceval-exam.py
- https://huggingface.co/datasets/Muennighoff/xwinograd/blob/e98b9216f60fc8dbabfe766e014534a08ff01949/xwinograd.py