Title: Loading wikipedia 20200501.en throws pyarrow related error
Author: TomPyonsuke (NONE)
State: open
Created: 2021-03-30T10:38:31Z

**Problem description**
I am getting the following error when trying to load wikipedia/20200501.en dataset.

**Error log**
Downloading and preparing dataset wikipedia/20200501.en (download: 16.99 GiB, generated: 17.07 GiB, post-processed: Unknown size, total: 34.06 GiB) to /usr/local/workspace/NAS_NLP/cache/wikipedia/20200501.en/1.0.0/50aa706aa417bb77d910ad61211cc672c0ef3e0f224225a5e0a18277ade8b931...
Downloading: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14.6k/14.6k [00:00<00:00, 5.41MB/s]
Downloading:  59%|███████████████████████████████████████████████████████████████████████████████████████▊                                                              | 10.7G/18.3G [11:30<08:08, 15.5MB/s]
Dataset wikipedia downloaded and prepared to /usr/local/workspace/NAS_NLP/cache/wikipedia/20200501.en/1.0.0/50aa706aa417bb77d910ad61211cc672c0ef3e0f224225a5e0a18277ade8b931. Subsequent calls will reuse this data.
Traceback (most recent call last):
  File "load_wiki.py", line 2, in <module>
    ds = load_dataset('wikipedia', '20200501.en', cache_dir='/usr/local/workspace/NAS_NLP/cache')
  File "/usr/local/lib/python3.6/dist-packages/datasets/load.py", line 751, in load_dataset
    ds = builder_instance.as_dataset(split=split, ignore_verifications=ignore_verifications, in_memory=keep_in_memory)
  File "/usr/local/lib/python3.6/dist-packages/datasets/builder.py", line 746, in as_dataset
    map_tuple=True,
  File "/usr/local/lib/python3.6/dist-packages/datasets/utils/py_utils.py", line 204, in map_nested
    _single_map_nested((function, obj, types, None, True)) for obj in tqdm(iterable, disable=disable_tqdm)
  File "/usr/local/lib/python3.6/dist-packages/datasets/utils/py_utils.py", line 204, in <listcomp>
    _single_map_nested((function, obj, types, None, True)) for obj in tqdm(iterable, disable=disable_tqdm)
  File "/usr/local/lib/python3.6/dist-packages/datasets/utils/py_utils.py", line 142, in _single_map_nested
    return function(data_struct)
  File "/usr/local/lib/python3.6/dist-packages/datasets/builder.py", line 763, in _build_single_dataset
    in_memory=in_memory,
  File "/usr/local/lib/python3.6/dist-packages/datasets/builder.py", line 835, in _as_dataset
    in_memory=in_memory,
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_reader.py", line 215, in read
    return self.read_files(files=files, original_instructions=instructions, in_memory=in_memory)
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_reader.py", line 236, in read_files
    pa_table = self._read_files(files, in_memory=in_memory)
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_reader.py", line 171, in _read_files
    pa_table: pa.Table = self._get_dataset_from_filename(f_dict, in_memory=in_memory)
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_reader.py", line 302, in _get_dataset_from_filename
    pa_table = ArrowReader.read_table(filename, in_memory=in_memory)
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_reader.py", line 324, in read_table
    pa_table = f.read_all()
  File "pyarrow/ipc.pxi", line 544, in pyarrow.lib.RecordBatchReader.read_all
  File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
OSError: Expected to be able to read 9176784 bytes for message body, got 4918712

**Detailed version info**
datasets==1.5.0
  - dataclasses [required: Any, installed: 0.8]
  - dill [required: Any, installed: 0.3.3]
  - fsspec [required: Any, installed: 0.8.7]
    - importlib-metadata [required: Any, installed: 1.7.0]
      - zipp [required: >=0.5, installed: 3.1.0]
  - huggingface-hub [required: <0.1.0, installed: 0.0.7]
    - filelock [required: Any, installed: 3.0.12]
    - importlib-metadata [required: Any, installed: 1.7.0]
      - zipp [required: >=0.5, installed: 3.1.0]
    - requests [required: Any, installed: 2.24.0]
      - certifi [required: >=2017.4.17, installed: 2020.6.20]
      - chardet [required: >=3.0.2,<4, installed: 3.0.4]
      - idna [required: >=2.5,<3, installed: 2.6]
      - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.10]
    - tqdm [required: Any, installed: 4.49.0]
  - importlib-metadata [required: Any, installed: 1.7.0]
    - zipp [required: >=0.5, installed: 3.1.0]
  - multiprocess [required: Any, installed: 0.70.11.1]
    - dill [required: >=0.3.3, installed: 0.3.3]
  - numpy [required: >=1.17, installed: 1.17.0]
  - pandas [required: Any, installed: 1.1.5]
    - numpy [required: >=1.15.4, installed: 1.17.0]
    - python-dateutil [required: >=2.7.3, installed: 2.8.0]
      - six [required: >=1.5, installed: 1.15.0]
    - pytz [required: >=2017.2, installed: 2020.1]
  - pyarrow [required: >=0.17.1, installed: 3.0.0]
    - numpy [required: >=1.16.6, installed: 1.17.0]
  - requests [required: >=2.19.0, installed: 2.24.0]
    - certifi [required: >=2017.4.17, installed: 2020.6.20]
    - chardet [required: >=3.0.2,<4, installed: 3.0.4]
    - idna [required: >=2.5,<3, installed: 2.6]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.10]
  - tqdm [required: >=4.27,<4.50.0, installed: 4.49.0]
  - xxhash [required: Any, installed: 2.0.0]

Comments:
That's how I loaded the dataset
```python
from datasets import load_dataset
ds = load_dataset('wikipedia', '20200501.en', cache_dir='/usr/local/workspace/NAS_NLP/cache')
```

Hi ! It looks like the arrow file in the folder
`/usr/local/workspace/NAS_NLP/cache/wikipedia/20200501.en/1.0.0/50aa706aa417bb77d910ad61211cc672c0ef3e0f224225a5e0a18277ade8b931` is corrupted.

Can you take a look and check that it's 18.3GB ?

If not, then maybe you need to redownload it:
```python
from datasets import load_dataset
ds = load_dataset('wikipedia', '20200501.en', cache_dir='/usr/local/workspace/NAS_NLP/cache', download_mode="force_redownload")
```

> Hi ! It looks like the arrow file in the folder
> `/usr/local/workspace/NAS_NLP/cache/wikipedia/20200501.en/1.0.0/50aa706aa417bb77d910ad61211cc672c0ef3e0f224225a5e0a18277ade8b931` is corrupted.
> 
> Can you take a look and check that it's 18.3GB ?
> 
> If not, then maybe you need to redownload it:
> 
> ```python
> from datasets import load_dataset
> ds = load_dataset('wikipedia', '20200501.en', cache_dir='/usr/local/workspace/NAS_NLP/cache', download_mode="force_redownload")
> ```

Hi Ihoestq, thanks for the reply! Actually i think my issue is i couldn't download the dataset beyond 10.7G. It feels like the whole dataset is split into different volumes and after the first one was downloaded it crashed before proceeding to the next one. I did try 'force_redownload' mode but still got the same issue.

I just tried on my side and got no issues.
When downloading the dataset again, did it crash at 10.7GB as well ?

> I just tried on my side and got no issues.
> When downloading the dataset again, did it crash at 10.7GB as well ?

Yes i have tried it multiple times on different machines. I am wondering if you could share the screenshot of your dependency versions and i will try to make them the same as yours?

I tried using `datasets` from `master` on macos with python 3.7.2
I also have `requests==2.23.0` and `tqdm==4.45.0`.