Repository: EleutherAI/lm-evaluation-harness
Issue: Running lm-evaluation-harness offline
Root cause: Documentation Deficiency

Hi!

I am trying to run lm_eval in a docker container without internet connection. But I run into errors with lm_eval trying to download files at run time. How can i make sure that all the datasets and necessary files are downloaded ahead of time, so that the eval can be run without internet connection?

More context:
I first ran into issue #1071, but fixed it by adding the exact_match folder into the root. However now when trying to run:
```python
    subprocess.check_call(
        [
            "lm_eval",
            "--model",
            "hf",
            "--model_args",
            f"pretrained={local_model}",
            "--tasks",
            "mnli,qnli,rte,coqa,record,drop,copa,wsc,multirc,mathqa,logiqa",
            "--device",
            "cpu",
            "--batch_size",
            "2",
            "--limit",
            "0.2",
            "--output_path",
            "./benchmark",
            "--verbosity",
            "CRITICAL",
        ]
    )
```
I get the error:
```
Traceback (most recent call last):
  File "/usr/local/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
             ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/lm_eval/__main__.py", line 342, in cli_evaluate
    results = evaluator.simple_evaluate(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/lm_eval/utils.py", line 288, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/lm_eval/evaluator.py", line 192, in simple_evaluate
    task_dict = get_task_dict(tasks, task_manager)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/lm_eval/tasks/__init__.py", line 420, in get_task_dict
    task_name_from_string_dict = task_manager.load_task_or_group(
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/lm_eval/tasks/__init__.py", line 270, in load_task_or_group
    collections.ChainMap(*map(self._load_individual_task_or_group, task_list))
  File "/usr/local/lib/python3.11/site-packages/lm_eval/tasks/__init__.py", line 161, in _load_individual_task_or_group
    return load_task(task_config, task=name_or_config, group=parent_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/lm_eval/tasks/__init__.py", line 150, in load_task
    task_object = ConfigurableTask(config=config)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/lm_eval/api/task.py", line 782, in __init__
    self.download(self.config.dataset_kwargs)
  File "/usr/local/lib/python3.11/site-packages/lm_eval/api/task.py", line 871, in download
    self.dataset = datasets.load_dataset(
                   ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/datasets/load.py", line 2556, in load_dataset
    builder_instance = load_dataset_builder(
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/datasets/load.py", line 2228, in load_dataset_builder
    dataset_module = dataset_module_factory(
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/datasets/load.py", line 1879, in dataset_module_factory
    raise e1 from None
  File "/usr/local/lib/python3.11/site-packages/datasets/load.py", line 1812, in dataset_module_factory
    raise ConnectionError(f"Couldn't reach '{path}' on the Hub ({type(e).__name__})")
ConnectionError: Couldn't reach 'super_glue' on the Hub (ConnectionError)
```


Set the environment variable HF_DATASETS_OFFLINE="1"

setting the env variable Stella described, and also setting HF_HOME to a cache folder where you have predownloaded the necessary models and datasets should work. let us know if it doesn't, although this may be a problem best addressed or answered by HuggingFace.