Repository: EleutherAI/lm-evaluation-harness
Issue: ConnectionError: Couldn't reach 'truthful_qa' on the Hub (ProxyError)
Root cause: Environment Incompatibility

I try to reproduce the results in [open LLM leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) and I go to test the truthful_qa testset with the following script:
```
TASK=truthfulqa_mc
FEWSHOT=0
python=python3

$python ${LM_EVAL_PATH}/main.py \
    --model=hf-causal \
    --model_args="pretrained=${model_path}" \
    --tasks=${TASK} \
    --num_fewshot=${FEWSHOT} \
    --batch_size=2 \
    --output_path=${output_path}
```
Then, it raise this error: `ConnectionError: Couldn't reach 'truthful_qa' on the Hub (ProxyError)`
```
Selected Tasks: ['truthfulqa_mc']
Using device 'cuda'
Traceback (most recent call last):
  File "/mmlu/lm-evaluation-harness/main.py", line 93, in <module>
    main()
  File "/mmlu/lm-evaluation-harness/main.py", line 59, in main
    results = evaluator.simple_evaluate(
  File "/mmlu/lm-evaluation-harness/lm_eval/utils.py", line 243, in _wrapper
    return fn(*args, **kwargs)
  File "/mmlu/lm-evaluation-harness/lm_eval/evaluator.py", line 100, in simple_evaluate
    task_dict = lm_eval.tasks.get_task_dict(tasks)
  File "/mmlu/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 411, in get_task_dict
    task_name_dict = {
  File "/mmlu/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 412, in <dictcomp>
    task_name: get_task(task_name)()
  File "/mmlu/lm-evaluation-harness/lm_eval/base.py", line 511, in __init__
    self.download(data_dir, cache_dir, download_mode)
  File "/mmlu/lm-evaluation-harness/lm_eval/base.py", line 540, in download
    self.dataset = datasets.load_dataset(
  File "/miniconda2/envs/py39/lib/python3.9/site-packages/datasets/load.py", line 2112, in load_dataset
    builder_instance = load_dataset_builder(
  File "/miniconda2/envs/py39/lib/python3.9/site-packages/datasets/load.py", line 1798, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/miniconda2/envs/py39/lib/python3.9/site-packages/datasets/load.py", line 1495, in dataset_module_factory
    raise e1 from None
  File "/miniconda2/envs/py39/lib/python3.9/site-packages/datasets/load.py", line 1451, in dataset_module_factory
    raise ConnectionError(f"Couldn't reach '{path}' on the Hub ({type(e).__name__})")
ConnectionError: Couldn't reach 'truthful_qa' on the Hub (ProxyError)
```
It seems the DATASET_NAME is wrong with `truthful_qa` ?

Is there any way to use this toolkit in an offline way?

Hi, I've confirmed that `truthful_qa` is the correct dataset name on the HF hub. I believe this is due to not having internet connection.

What can be done to use the library offline is pre-download the datasets and models you require via HF, and then use these cached files offline.

@SefaZeng have you resolve it using the cache without connect hf?

> @SefaZeng have you resolve it using the cache without connect hf?

Yes, you can download the dataset before and change the path where lm-eval finds the dataset.

> > @SefaZeng have you resolve it using the cache without connect hf?
> 
> Yes, you can download the dataset before and change the path where lm-eval finds the dataset.

I try to set `DATASET_PATH = inspect.getfile(lm_eval.datasets.mmlu.mmlu)` in hendrycks_test.py to load off-load mmlu dataset, but still ConnectionError. How should I set the data set path correctly?

> > > @SefaZeng have you resolve it using the cache without connect hf?
> > 
> > 
> > Yes, you can download the dataset before and change the path where lm-eval finds the dataset.
> 
> I try to set `DATASET_PATH = inspect.getfile(lm_eval.datasets.mmlu.mmlu)` in hendrycks_test.py to load off-load mmlu dataset, but still ConnectionError. How should I set the data set path correctly?

@Davidwhw Hi, I also encountered the same problem about local dataset loading, did you solve it?

> > > > @SefaZeng have you resolve it using the cache without connect hf?
> > > 
> > > 
> > > Yes, you can download the dataset before and change the path where lm-eval finds the dataset.
> > 
> > 
> > I try to set `DATASET_PATH = inspect.getfile(lm_eval.datasets.mmlu.mmlu)` in hendrycks_test.py to load off-load mmlu dataset, but still ConnectionError. How should I set the data set path correctly?
> 
> @Davidwhw Hi, I also encountered the same problem about local dataset loading, did you solve it?

*I am using the master branch of the repository, you may need to find the corresponding path of the branch you are using.*
In addition to changing the `DATASET_PATH` I mentioned above, you should also prepare the corresponding data set on the corresponding path.  And change the data load path in the dataset folder, such as `archive = dl_manager.download(_URL)` in the `lm_eval/datasets/mmlu/mmlu.py` I used, to your local path.

I found a solution to the issue of not being able to connect to huggingface.co without needing to download the dataset locally. You just need to modify the corresponding _URL link in the dataset's Python script to a mirror site. 
For example, if the error message is:
```
File "YourPath/mmlu_no_train.py", line 129, in _split_generators
    archive = dl_manager.download(_URL)
```
Then reach to mmlu_no_train.py, and change 
```
_URL = "https://huggingface.co/datasets/cais/mmlu/resolve/main/data.tar"
```
 to 
```
_URL = "https://hf-mirror.com/datasets/cais/mmlu/resolve/main/data.tar"
```
The same applies to other datasets.