`load_dataset` does not work with uploaded arrow file
Opened by patrickvonplaten (open)

## Describe the bug

I've preprocessed and uploaded a dataset here: https://huggingface.co/datasets/ami-wav2vec2/ami_headset_single_preprocessed . The dataset is in `.arrow` format.

The dataset can correctly be loaded when doing:

```bash
git lfs install
git clone https://huggingface.co/datasets/ami-wav2vec2/ami_headset_single_preprocessed
```

followed by 

```python
from datasets import load_from_disk

ds = load_from_disk("./ami_headset_single_preprocessed")
```

However when I try to directly download the dataset as follows:

```python
from datasets import load_dataset

ds = load_dataset("ami-wav2vec2/ami_headset_single_preprocessed")
```

the following error occurs:

```bash
/usr/local/lib/python3.7/dist-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, script_version, use_auth_token, task, streaming, **config_kwargs)
   1115         ignore_verifications=ignore_verifications,
   1116         try_from_hf_gcs=try_from_hf_gcs,
-> 1117         use_auth_token=use_auth_token,
   1118     )
   1119 

/usr/local/lib/python3.7/dist-packages/datasets/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    635                     if not downloaded_from_gcs:
    636                         self._download_and_prepare(
--> 637                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    638                         )
    639                     # Sync info

/usr/local/lib/python3.7/dist-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    724             try:
    725                 # Prepare split will record examples associated to the split
--> 726                 self._prepare_split(split_generator, **prepare_split_kwargs)
    727             except OSError as e:
    728                 raise OSError(

/usr/local/lib/python3.7/dist-packages/datasets/builder.py in _prepare_split(self, split_generator)
   1186                 generator, unit=" tables", leave=False, disable=bool(logging.get_verbosity() == logging.NOTSET)
   1187             ):
-> 1188                 writer.write_table(table)
   1189             num_examples, num_bytes = writer.finalize()
   1190 

/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py in write_table(self, pa_table, writer_batch_size)
    424         # reorder the arrays if necessary + cast to self._schema
    425         # we can't simply use .cast here because we may need to change the order of the columns
--> 426         pa_table = pa.Table.from_arrays([pa_table[name] for name in self._schema.names], schema=self._schema)
    427         batches: List[pa.RecordBatch] = pa_table.to_batches(max_chunksize=writer_batch_size)
    428         self._num_bytes += sum(batch.nbytes for batch in batches)

/usr/local/lib/python3.7/dist-packages/pyarrow/table.pxi in pyarrow.lib.Table.from_arrays()

/usr/local/lib/python3.7/dist-packages/pyarrow/table.pxi in pyarrow.lib._sanitize_arrays()

/usr/local/lib/python3.7/dist-packages/pyarrow/array.pxi in pyarrow.lib.asarray()

/usr/local/lib/python3.7/dist-packages/pyarrow/table.pxi in pyarrow.lib.ChunkedArray.cast()

/usr/local/lib/python3.7/dist-packages/pyarrow/compute.py in cast(arr, target_type, safe)
    279     else:
    280         options = CastOptions.unsafe(target_type)
--> 281     return call_function("cast", [arr], options)
    282 
    283 

/usr/local/lib/python3.7/dist-packages/pyarrow/_compute.pyx in pyarrow._compute.call_function()

/usr/local/lib/python3.7/dist-packages/pyarrow/_compute.pyx in pyarrow._compute.Function.call()

/usr/local/lib/python3.7/dist-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

/usr/local/lib/python3.7/dist-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowNotImplementedError: Unsupported cast from struct<train: struct<name: string, num_bytes: int64, num_examples: int64, dataset_name: string>, validation: struct<name: string, num_bytes: int64, num_examples: int64, dataset_name: string>, test: struct<name: string, num_bytes: int64, num_examples: int64, dataset_name: string>> to list using function cast_list
```

## Expected results

The dataset should be correctly loaded with `load_dataset` IMO.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.12.2.dev0
- Platform: Linux-5.11.0-34-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyArrow version: 5.0.0


Comments: Hi ! This is not a bug, this is simply not implemented.
`save_to_disk` is for on-disk serialization and was not made compatible for the Hub.
That being said, I agree we actually should make it work with the Hub x)

cc @LysandreJik maybe we can solve this at the same time as adding `push_to_hub`