Repository: EleutherAI/lm-evaluation-harness
Issue: pubmedqa task data fails to download
Root cause: External Dependency Breakage

using `lm-eval==0.2.0`:

```
python ./tasks/eval_harness/download.py --task_list pubmedqa
Downloading and preparing dataset pubmed_qa/pqa_labeled (download: 656.02 MiB, generated: 1.99 MiB, post-processed: Unknown size, total: 658.01 MiB) to /gpfswork/rech/six/commun/datasets/pubmed_qa/pqa_labeled/1.0.0/2e65addecca4197502cd10ab8ef1919a47c28672f62d7abac7cc9afdcf24fb2d...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 4495.50it/s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 671.73it/s]
Traceback (most recent call last):
  File "./tasks/eval_harness/download.py", line 20, in <module>
    main()
  File "./tasks/eval_harness/download.py", line 17, in main
    tasks.get_task_dict(task_list)
  File "/gpfswork/rech/six/commun/conda/py38-pt111/lib/python3.8/site-packages/lm_eval/tasks/__init__.py", line 325, in get_task_dict
    task_name_dict = {
  File "/gpfswork/rech/six/commun/conda/py38-pt111/lib/python3.8/site-packages/lm_eval/tasks/__init__.py", line 326, in <dictcomp>
    task_name: get_task(task_name)()
  File "/gpfswork/rech/six/commun/conda/py38-pt111/lib/python3.8/site-packages/lm_eval/tasks/common.py", line 11, in __init__
    super().__init__()
  File "/gpfswork/rech/six/commun/conda/py38-pt111/lib/python3.8/site-packages/lm_eval/base.py", line 350, in __init__
    self.download()
  File "/gpfswork/rech/six/commun/conda/py38-pt111/lib/python3.8/site-packages/lm_eval/tasks/common.py", line 14, in download
    self.data = datasets.load_dataset(path=self.DATASET_PATH, name=self.DATASET_NAME)
  File "/gpfswork/rech/six/commun/conda/py38-pt111/lib/python3.8/site-packages/datasets/load.py", line 1632, in load_dataset
    builder_instance.download_and_prepare(
  File "/gpfswork/rech/six/commun/conda/py38-pt111/lib/python3.8/site-packages/datasets/builder.py", line 607, in download_and_prepare
    self._download_and_prepare(
  File "/gpfswork/rech/six/commun/conda/py38-pt111/lib/python3.8/site-packages/datasets/builder.py", line 679, in _download_and_prepare
    verify_checksums(
  File "/gpfswork/rech/six/commun/conda/py38-pt111/lib/python3.8/site-packages/datasets/utils/info_utils.py", line 40, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=1RsGLINVce-0GsDkCLDuLZmoLuzfmoCuQ', 'https://drive.google.com/uc?export=download&id=15v1x6aQDlZymaHGP7cZJZZYFfeJt2NdS']
```

probably those files got updated and require new checksums perhaps?

I checked the files are there.

Thanks.




Hey @stas00 Thanks for reporting. This was a problem we ran into with earlier versions of HuggingFace `datasets`. It should work with an upgrade to `datatsets>=2.0.0`. Let us know!

Well, I updated to `datasets==0.2.1` now it fails with:

```
$ python ./tasks/eval_harness/download.py --task_list pubmedqa
Traceback (most recent call last):
  File "./tasks/eval_harness/download.py", line 5, in <module>
    from lm_eval import tasks
  File "/gpfswork/rech/six/commun/conda/py38-pt111/lib/python3.8/site-packages/lm_eval/tasks/__init__.py", line 11, in <module>
    from . import race
  File "/gpfswork/rech/six/commun/conda/py38-pt111/lib/python3.8/site-packages/lm_eval/tasks/race.py", line 38, in <module>
    class RACE(HFTask):
  File "/gpfswork/rech/six/commun/conda/py38-pt111/lib/python3.8/site-packages/lm_eval/tasks/race.py", line 46, in RACE
    assert datasets.__version__ == "1.15.1", "RACE requires datasets==1.15.1!"
AssertionError: RACE requires datasets==1.15.1!
```



~~That assertion should not happen with `lm-eval==0.2.0` as it's been removed. Could you double-check your lm-eval version, please?~~ Sorry, this fix on the repo head.

the HEAD is broken too:

```
$ python ./tasks/eval_harness/download.py --task_list arc_challenge
Traceback (most recent call last):
  File "./tasks/eval_harness/download.py", line 5, in <module>
    from lm_eval import tasks
  File "/gpfswork/rech/six/commun/conda/tr11-176B-ml/lib/python3.8/site-packages/lm_eval/tasks/__init__.py", line 10, in <module>
    from . import coqa
  File "/gpfswork/rech/six/commun/conda/tr11-176B-ml/lib/python3.8/site-packages/lm_eval/tasks/coqa.py", line 14, in <module>
    import lm_eval.datasets.coqa.coqa
ModuleNotFoundError: No module named 'lm_eval.datasets.coqa'
```

Okay, that's related to #311 (which you previously detected). Sorry about that. We have a fix in a PR awaiting approval.

cc @leogao2 or @StellaAthena: can you please review #309 to resolve this pressing issue?  (Note you may have to revert all commits after [`c901185943eccd94b6dbc242b58fbadb58ef916c`](https://github.com/EleutherAI/lm-evaluation-harness/pull/309/commits/c901185943eccd94b6dbc242b58fbadb58ef916c))


Update: We've merged #309 which should fix [this]( https://github.com/EleutherAI/lm-evaluation-harness/issues/312#issuecomment-1112572575) error.

Doesn't work with HEAD and `datasets==2.0.0` (which is forced by lm-eval)

```
$ pip install git+https://github.com/EleutherAI/lm-evaluation-harness
Collecting git+https://github.com/EleutherAI/lm-evaluation-harness
  Cloning https://github.com/EleutherAI/lm-evaluation-harness to /tmp/pip-req-build-roim88hh
  Running command git clone -q https://github.com/EleutherAI/lm-evaluation-harness /tmp/pip-req-build-roim88hh
  Resolved https://github.com/EleutherAI/lm-evaluation-harness to commit 14b8ec7ec3996a7c7906e2a2691bcfd369e54e1b
[...]
Successfully installed datasets-2.0.0 lm-eval-0.2.0

$ pip install datasets 
Requirement already satisfied: datasets in .../conda/py38-pt111/lib/python3.8/site-packages (2.0.0)
[...]

$ python ./tasks/eval_harness/download.py --task_list pubmedqa
Downloading and preparing dataset pubmed_qa/pqa_labeled (download: 656.02 MiB, generated: 1.99 MiB, post-processed: Unknown size, total: 658.01 MiB) to /gpfswork/rech/six/commun/datasets/pubmed_qa/pqa_labeled/1.0.0/2e65addecca4197502cd10ab8ef1919a47c28672f62d7abac7cc9afdcf24fb2d...
Downloading data files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 3870.47it/s]
Extracting data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 705.44it/s]
Traceback (most recent call last):
  File "./tasks/eval_harness/download.py", line 20, in <module>
    main()
  File "./tasks/eval_harness/download.py", line 17, in main
    tasks.get_task_dict(task_list)
  File "/gpfswork/rech/six/commun/conda/py38-pt111/lib/python3.8/site-packages/lm_eval/tasks/__init__.py", line 332, in get_task_dict
    task_name_dict = {
  File "/gpfswork/rech/six/commun/conda/py38-pt111/lib/python3.8/site-packages/lm_eval/tasks/__init__.py", line 333, in <dictcomp>
    task_name: get_task(task_name)()
  File "/gpfswork/rech/six/commun/conda/py38-pt111/lib/python3.8/site-packages/lm_eval/base.py", line 381, in __init__
    self.download(data_dir, cache_dir, download_mode)
  File "/gpfswork/rech/six/commun/conda/py38-pt111/lib/python3.8/site-packages/lm_eval/base.py", line 410, in download
    self.dataset = datasets.load_dataset(
  File "/gpfswork/rech/six/commun/conda/py38-pt111/lib/python3.8/site-packages/datasets/load.py", line 1687, in load_dataset
    builder_instance.download_and_prepare(
  File "/gpfswork/rech/six/commun/conda/py38-pt111/lib/python3.8/site-packages/datasets/builder.py", line 605, in download_and_prepare
    self._download_and_prepare(
  File "/gpfswork/rech/six/commun/conda/py38-pt111/lib/python3.8/site-packages/datasets/builder.py", line 1104, in _download_and_prepare
    super()._download_and_prepare(dl_manager, verify_infos, check_duplicate_keys=verify_infos)
  File "/gpfswork/rech/six/commun/conda/py38-pt111/lib/python3.8/site-packages/datasets/builder.py", line 676, in _download_and_prepare
    verify_checksums(
  File "/gpfswork/rech/six/commun/conda/py38-pt111/lib/python3.8/site-packages/datasets/utils/info_utils.py", line 40, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=1RsGLINVce-0GsDkCLDuLZmoLuzfmoCuQ', 'https://drive.google.com/uc?export=download&id=15v1x6aQDlZymaHGP7cZJZZYFfeJt2NdS']

```

same error with `datasets==2.1.0` even if I override the forcing of 2.0.0

it's pretty clear this is not a `datasets` version issue but most likely the source has changed and the checksum was not updated in the repo. i.e. the problem is in the `pubmed_qa` repo.

```
python -c 'from datasets import load_dataset; dataset = load_dataset("pubmed_qa", "pqa_artificial")'
Downloading: 11.0kB [00:00, 15.7MB/s]
Downloading: 12.4kB [00:00, 17.7MB/s]
Downloading and preparing dataset pubmed_qa/pqa_artificial (download: 656.02 MiB, generated: 422.96 MiB, post-processed: Unknown size, total: 1.05 GiB) to /home/stas/.cache/huggingface/datasets/pubmed_qa/pqa_artificial/1.0.0/2e65addecca4197502cd10ab8ef1919a47c28672f62d7abac7cc9afdcf24fb2d...
Downloading: 2.58MB [00:00, 45.3MB/s]
Downloading: 2.22kB [00:00, 1.06MB/s]████████                                                                                | 1/3 [00:01<00:02,  1.11s/it]
Downloading: 2.22kB [00:00, 1.33MB/s]████████████████████████████████████████████████                                        | 2/3 [00:01<00:00,  1.48it/s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:01<00:00,  1.61it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 4225.29it/s]
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/load.py", line 1632, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/builder.py", line 607, in download_and_prepare
    self._download_and_prepare(
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/builder.py", line 679, in _download_and_prepare
    verify_checksums(
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/utils/info_utils.py", line 40, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=1RsGLINVce-0GsDkCLDuLZmoLuzfmoCuQ', 'https://drive.google.com/uc?export=download&id=15v1x6aQDlZymaHGP7cZJZZYFfeJt2NdS']
```

I manually checked that the files downloaded manually via `gdown` have the correct checksum - so they haven't been updated.

Let me see if I can fix the repo.

I was not able to reproduce this issue 😕.  See: https://colab.research.google.com/drive/16RC5PBLyBIglwucZEGFtjpTD5nVVXSOp?usp=sharing 

so one can download these manually via `gdown` or browser, but not via datasets - apparently due to reaching some traffic limit.

we will have to move the data to a different storage that doesn't have such caps.

In any case the issue is not of `lm-eval`.

I will close it once we get that fixed on the hub.

Sounds good. Thanks for letting us know!

@jon-tow, so the hub version has been updated to not use GDrive for this dataset, but since at the moment by default the non-hub info files are used and you have a forced fixed `datasets` version, `lm-eval` needs to be updated to use: `revision="master"` when downloading this dataset, e.g. you can try:

```
python -c 'from datasets import load_dataset; \
dataset = load_dataset("pubmed_qa", "pqa_artificial", revision="master")'
```

This tells `datasets` to use the info files from the hub's repo.

Would it be possible for you to add that to `lm-eval`?

Thank you!


@stas00 We've loosened the `datasets` dependency to `datasets>=2.0.0`. 
Let me know if this solves your issues and we can close this issue.

Then it will have to wait till a new `datasets` release is made. Which is a possibility as well. I will ask the datasets team to do so.

@jon-tow,

so `datasets==2.2.0` is out

you said you fixed the deps but it doesn't appear to be the case:

```
$ pip install datasets -U # gets 2.2.0 as of this writing
$ pip install git+https://github.com/EleutherAI/lm-evaluation-harness
$ python ./tasks/eval_harness/download.py --task_list pubmedqa
Traceback (most recent call last):
  File "./tasks/eval_harness/download.py", line 5, in <module>
    from lm_eval import tasks
  File "/gpfswork/rech/six/commun/conda/py38-pt111/lib/python3.8/site-packages/lm_eval/tasks/__init__.py", line 11, in <module>
    from . import race
  File "/gpfswork/rech/six/commun/conda/py38-pt111/lib/python3.8/site-packages/lm_eval/tasks/race.py", line 38, in <module>
    class RACE(HFTask):
  File "/gpfswork/rech/six/commun/conda/py38-pt111/lib/python3.8/site-packages/lm_eval/tasks/race.py", line 46, in RACE
    assert datasets.__version__ == "1.15.1", "RACE requires datasets==1.15.1!"
AssertionError: RACE requires datasets==1.15.1!
```

If it's really needed for RACE, perhaps it shouldn't be loaded on init and assert impacting all other tasks?

Thanks


This problem was addressed a few days after the `0.2.0` release (the version I assume you're using). I'll put together a new `lm_eval` package update as soon as possible.  If you cannot wait a few hours, please pip install from the master branch. 

Sorry about that.

No problem at all, @jon-tow - let's sort it out.

If you look at the log of my last comment I was using lm-evaluation-harness@master - hence my message. i.e. a new package won't help.

I'm not sure why you're getting that assertion error because it's been removed for nearly 2 months now (see the file [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/master/lm_eval/tasks/race.py)) 😅 .

Could you try this in a fresh environment?


I get this error in 2 unrelated envs. Perhaps I need to uninstall lm-eval first - sometimes pip packages get installed multiple times.


Yup, I had to explicitly uninstall `lm-eval` before installing from master. and then it was all working.
```
$ pip uninstall lm-eval
Found existing installation: lm-eval 0.2.0
Uninstalling lm-eval-0.2.0:
  Would remove:
    /mnt/nvme0/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/lm_eval-0.2.0.dist-info/*
    /mnt/nvme0/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/lm_eval/*
    /mnt/nvme0/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/scripts/*
Proceed (Y/n)? y
  Successfully uninstalled lm-eval-0.2.0

$ pip install git+https://github.com/EleutherAI/lm-evaluation-harness
[...]
  Successfully installed datasets-2.2.0 lm-eval-0.2.0
  
$ python ./tasks/eval_harness/download.py --task_list pubmedqa
Downloading builder script: 11.1kB [00:00, 15.3MB/s]
Downloading metadata: 12.7kB [00:00, 15.1MB/s]
Downloading and preparing dataset pubmed_qa/pqa_labeled (download: 656.02 MiB, generated: 1.99 MiB, post-processed: Unknown size, total: 658.01 MiB) to /home/stas/.cache/huggingface/datasets/pubmed_qa/pqa_labeled/1.0.0/dd4c39f031a958c7e782595fa4dd1b1330484e8bbadd4d9212e5046f27e68924...
Downloading data files: 100%|████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 19152.07it/s]
Extracting data files: 100%|██████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 4000.93it/s]
Dataset pubmed_qa downloaded and prepared to /home/stas/.cache/huggingface/datasets/pubmed_qa/pqa_labeled/1.0.0/dd4c39f031a958c7e782595fa4dd1b1330484e8bbadd4d9212e5046f27e68924. Subsequent calls will reuse this data.
```

No idea why it wasn't updating `lm-eval`

Awesome.  Thank you for your help, @jon-tow 

Closing this ticket.

Glad to hear it worked out! Thanks for reporting everything :)