Repository: EleutherAI/lm-evaluation-harness
Issue: Hellaswag filenotfound error
Root cause: Environment Incompatibility

When trying to load your Hellaswag dataset with this script:

```
from lm_eval import evaluator


# Define tasks for zero-shot and for GSM8K 8-shot evaluation
tasks_zero = ["hellaswag", "mmlu"]
tasks_8shot = ["gsm8k"]

# Evaluate LLaMA‑3.1‑8B for zero-shot tasks
llama_model_name = "meta-llama/Llama-3.1-8B"
llama_results_zero = evaluator.simple_evaluate(
    model="hf",
    model_args=f"pretrained={llama_model_name},dtype=float16,device_map=auto",
    tasks=tasks_zero,
    num_fewshot=0,  # zero-shot evaluation for these tasks
    batch_size=8
)

```

I get this error:

```
Traceback (most recent call last):
  File "/home/dusan001/llama_cad/.venv/lib/python3.10/site-packages/aiohttp/connector.py", line 1122, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs, sock=sock)
  File "/home/dusan001/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/python3.10/asyncio/base_events.py", line 1103, in create_connection
    transport, protocol = await self._create_connection_transport(
  File "/home/dusan001/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/python3.10/asyncio/base_events.py", line 1133, in _create_connection_transport
    await waiter
  File "/home/dusan001/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/python3.10/asyncio/sslproto.py", line 534, in data_received
    ssldata, appdata = self._sslpipe.feed_ssldata(data)
  File "/home/dusan001/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/python3.10/asyncio/sslproto.py", line 188, in feed_ssldata
    self._sslobj.do_handshake()
  File "/home/dusan001/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/python3.10/ssl.py", line 975, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/dusan001/llama_cad/.venv/lib/python3.10/site-packages/fsspec/implementations/http.py", line 427, in _info
    await _file_info(
  File "/home/dusan001/llama_cad/.venv/lib/python3.10/site-packages/fsspec/implementations/http.py", line 815, in _file_info
    r = await session.get(url, allow_redirects=ar, **kwargs)
  File "/home/dusan001/llama_cad/.venv/lib/python3.10/site-packages/aiohttp/client.py", line 703, in _request
    conn = await self._connector.connect(
  File "/home/dusan001/llama_cad/.venv/lib/python3.10/site-packages/aiohttp/connector.py", line 548, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/home/dusan001/llama_cad/.venv/lib/python3.10/site-packages/aiohttp/connector.py", line 1056, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/home/dusan001/llama_cad/.venv/lib/python3.10/site-packages/aiohttp/connector.py", line 1400, in _create_direct_connection
    raise last_exc
  File "/home/dusan001/llama_cad/.venv/lib/python3.10/site-packages/aiohttp/connector.py", line 1369, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
  File "/home/dusan001/llama_cad/.venv/lib/python3.10/site-packages/aiohttp/connector.py", line 1124, in _wrap_create_connection
    raise ClientConnectorCertificateError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host raw.githubusercontent.com:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)')]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/dusan001/llama_cad/run_evals_multiple.py", line 14, in <module>
    llama_results_zero = evaluator.simple_evaluate(
  File "/home/dusan001/lm-evaluation-harness/lm_eval/utils.py", line 439, in _wrapper
    return fn(*args, **kwargs)
  File "/home/dusan001/lm-evaluation-harness/lm_eval/evaluator.py", line 264, in simple_evaluate
    task_dict = get_task_dict(
  File "/home/dusan001/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 635, in get_task_dict
    task_name_from_string_dict = task_manager.load_task_or_group(
  File "/home/dusan001/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 426, in load_task_or_group
    collections.ChainMap(
  File "/home/dusan001/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 428, in <lambda>
    lambda task: self._load_individual_task_or_group(task),
  File "/home/dusan001/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 326, in _load_individual_task_or_group
    return _load_task(task_config, task=name_or_config)
  File "/home/dusan001/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 286, in _load_task
    task_object = ConfigurableTask(config=config)
  File "/home/dusan001/lm-evaluation-harness/lm_eval/api/task.py", line 856, in __init__
    self.download(self.config.dataset_kwargs)
  File "/home/dusan001/lm-evaluation-harness/lm_eval/api/task.py", line 978, in download
    self.dataset = datasets.load_dataset(
  File "/home/dusan001/llama_cad/.venv/lib/python3.10/site-packages/datasets/load.py", line 2084, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/dusan001/llama_cad/.venv/lib/python3.10/site-packages/datasets/builder.py", line 925, in download_and_prepare
    self._download_and_prepare(
  File "/home/dusan001/llama_cad/.venv/lib/python3.10/site-packages/datasets/builder.py", line 1649, in _download_and_prepare
    super()._download_and_prepare(
  File "/home/dusan001/llama_cad/.venv/lib/python3.10/site-packages/datasets/builder.py", line 979, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/dusan001/.cache/huggingface/modules/datasets_modules/datasets/hellaswag/512a66dd8b1b1643ab4a48aa4f150d04c91680da6a4096498a5e5f799623d5ae/hellaswag.py", line 72, in _split_generators
    dl_dir = dl_manager.download_and_extract(urls_to_download)
  File "/home/dusan001/llama_cad/.venv/lib/python3.10/site-packages/datasets/download/download_manager.py", line 326, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/home/dusan001/llama_cad/.venv/lib/python3.10/site-packages/datasets/download/download_manager.py", line 159, in download
    downloaded_path_or_paths = map_nested(
  File "/home/dusan001/llama_cad/.venv/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 513, in map_nested
    mapped = [
  File "/home/dusan001/llama_cad/.venv/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 514, in <listcomp>
    _single_map_nested((function, obj, batched, batch_size, types, None, True, None))
  File "/home/dusan001/llama_cad/.venv/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 382, in _single_map_nested
    return [mapped_item for batch in iter_batched(data_struct, batch_size) for mapped_item in function(batch)]
  File "/home/dusan001/llama_cad/.venv/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 382, in <listcomp>
    return [mapped_item for batch in iter_batched(data_struct, batch_size) for mapped_item in function(batch)]
  File "/home/dusan001/llama_cad/.venv/lib/python3.10/site-packages/datasets/download/download_manager.py", line 219, in _download_batched
    return [
  File "/home/dusan001/llama_cad/.venv/lib/python3.10/site-packages/datasets/download/download_manager.py", line 220, in <listcomp>
    self._download_single(url_or_filename, download_config=download_config)
  File "/home/dusan001/llama_cad/.venv/lib/python3.10/site-packages/datasets/download/download_manager.py", line 229, in _download_single
    out = cached_path(url_or_filename, download_config=download_config)
  File "/home/dusan001/llama_cad/.venv/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 206, in cached_path
    output_path = get_from_cache(
  File "/home/dusan001/llama_cad/.venv/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 383, in get_from_cache
    response = fsspec_head(url, storage_options=storage_options)
  File "/home/dusan001/llama_cad/.venv/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 297, in fsspec_head
    return fs.info(path)
  File "/home/dusan001/llama_cad/.venv/lib/python3.10/site-packages/fsspec/asyn.py", line 118, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/home/dusan001/llama_cad/.venv/lib/python3.10/site-packages/fsspec/asyn.py", line 103, in sync
    raise return_result
  File "/home/dusan001/llama_cad/.venv/lib/python3.10/site-packages/fsspec/asyn.py", line 56, in _runner
    result[0] = await coro
  File "/home/dusan001/llama_cad/.venv/lib/python3.10/site-packages/fsspec/implementations/http.py", line 440, in _info
    raise FileNotFoundError(url) from exc
FileNotFoundError: https://raw.githubusercontent.com/rowanz/hellaswag/master/data/hellaswag_train.jsonl

```


MMLU and gsm8k work fine.


same

I have code which also uses `evaluator.simple_evaluate()` and it does not run into issues with hellaswag.

This could be an issue with your environment. Are you running from inside a container such as docker/apptainer/singularity or are you running barebones? Does the issue persist when running from the CLI rather than from a python script? And have you checked separately if you are able to download hellaswag using the `datasets.load_dataset` method from HF datasets?

Thanks for explaining! I am running this from the latest HPC cluster in our environment, with `lm-eval` installed from source. I have attempted to load this dataset with `datasets.load_dataset` and was unable to do so. But some other datasets like PIQA and MMLU which also failed on `datasets` worked well on `lm-eval`, and that's the reasons why I have moved to `lm-eval` instead of staying on 'datasets'. Hope this helps!



I have the same problem , it resolved with ssl