Title: Couldn't reach certain URLs and for the ones that can be reached, code just blocks after downloading.
Author: sudarshan85 (NONE)
State: closed
Created: 2020-09-04T21:46:25Z

Hi,

I'm following the [quick tour](https://huggingface.co/nlp/quicktour.html) and tried to load the glue dataset:
```
>>> from nlp import load_dataset
>>> dataset = load_dataset('glue', 'mrpc', split='train')
```

However, this ran into a `ConnectionError` saying it could not reach the URL (just pasting the last few lines):
```

/net/vaosl01/opt/NFS/su0/miniconda3/envs/hf/lib/python3.7/site-packages/nlp/utils/file_utils.py in get_from_cache(url, cache_dir, force_download, proxies, etag_timeout, resume_download, user_agent, local_files_only)
    354                 " to False."
    355             )
--> 356         raise ConnectionError("Couldn't reach {}".format(url))
    357 
    358     # From now on, connected is True.

ConnectionError: Couldn't reach https://firebasestorage.googleapis.com/v0/b/mtl-sentence-representations.appspot.com/o/data%2Fmrpc_dev_ids.tsv?alt=media&token=ec5c0836-31d5-48f4-b431-7480817f1adc
```

I tried glue with cola and sst2. I got the same error, just instead of mrpc in the URL, it was replaced with cola and sst2.

Since this was not working, I thought I'll try another dataset. So I tried downloading the imdb dataset:
```
ds = load_dataset('imdb', split='train')
```
This downloads the data, but it just blocks after that:
```
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4.56k/4.56k [00:00<00:00, 1.38MB/s]
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2.07k/2.07k [00:00<00:00, 1.15MB/s]
Downloading and preparing dataset imdb/plain_text (download: 80.23 MiB, generated: 127.06 MiB, post-processed: Unknown sizetotal: 207.28 MiB) to /net/vaosl01/opt/NFS/su0/huggingface/datasets/imdb/plain_text/1.0.0/76cdbd7249ea3548c928bbf304258dab44d09cd3638d9da8d42480d1d1be3743...
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 84.1M/84.1M [00:07<00:00, 11.1MB/s]
```

I checked the folder `$HF_HOME/datasets/downloads/extracted/<id>/aclImdb`. This folder is constantly growing in size. When I navigated to the train folder within, there was no file. However, the test folder seemed to be populating. The last time I checked it was 327M. I thought the Imdb dataset was smaller than that. My questions are:
1. Why is it still blocking? Is it still downloading?
2. I specified split as train, so why is the test folder being populated?
3. I read somewhere that after downloading, `nlp` converts the text files into some sort of `arrow` files, which will also take a while. Is this also happening here?

Thanks.

Comments:
Update:

The imdb download completed after a long time (about 45 mins). Ofcourse once download loading was instantaneous. Also, the loaded object was of type `arrow_dataset`. 

The urls for glue still doesn't work though.

Thanks for the report, I'll give a look!

I am also seeing a similar error when running the following:

```
import nlp
dataset = load_dataset('cola')
```
Error:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/js11133/.conda/envs/jiant/lib/python3.8/site-packages/nlp/load.py", line 509, in load_dataset
    module_path = prepare_module(path, download_config=download_config, dataset=True)
  File "/home/js11133/.conda/envs/jiant/lib/python3.8/site-packages/nlp/load.py", line 248, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "/home/js11133/.conda/envs/jiant/lib/python3.8/site-packages/nlp/utils/file_utils.py", line 191, in cached_path
    output_path = get_from_cache(
  File "/home/js11133/.conda/envs/jiant/lib/python3.8/site-packages/nlp/utils/file_utils.py", line 356, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/cola/cola.py
```

@jeswan `"cola"` is not a valid dataset identifier (you can check the up-to-date list on https://huggingface.co/datasets) but you can find cola inside glue.

Ah right. Thanks!

Hi. Closing this one since #626 updated the glue urls.

> 1. Why is it still blocking? Is it still downloading?

After downloading it generates the arrow file by iterating through the examples.
The number of examples processed by second is shown during the processing (not sure why it was not the case for you)

> 2. I specified split as train, so why is the test folder being populated?

It downloads every split