Title: Push to hub capabilities for `Dataset` and `DatasetDict`
Author: LysandreJik (MEMBER)
State: closed
Created: 2021-10-17T04:12:44Z

This PR implements a `push_to_hub` method on `Dataset` and `DatasetDict`. This does not currently work in `IterableDatasetDict` nor `IterableDataset` as those are simple dicts and I would like your opinion on how you would like to implement this before going ahead and doing it.

This implementation needs to be used with the following `huggingface_hub` branch in order to work correctly: https://github.com/huggingface/huggingface_hub/pull/415

### Implementation

The `push_to_hub` API is entirely based on HTTP requests rather than a git-based workflow:
- This allows pushing changes without firstly cloning the repository, which reduces the time in half for the `push_to_hub` method.
- Collaboration, as well as the system of branches/merges/rebases is IMO less straightforward than for models and spaces. In the situation where such collaboration is needed, I would *heavily* advocate for the `Repository` helper of the `huggingface_hub` to be used instead of the `push_to_hub` method which will always be, by design, limiting in that regard (even if based on a git-workflow instead of HTTP requests)

In order to overcome the limit of 5GB files set by the HTTP requests, dataset sharding is used.

### Testing

The test suite implemented here makes use of the moon-staging instead of the production setup. As several repositories are created and deleted, it is better to use the staging.

It does not require setting an environment variable or any kind of special attention but introduces a new decorator `with_staging_testing` which patches global variables to use the staging endpoint instead of the production endpoint.

### Examples

The tests cover a lot of examples and behavior.

Comments:
Thank you for your reviews! I should have addressed all of your comments, and I added a test to ensure that `private` datasets work correctly too. I have merged the changes in `huggingface_hub`, so the `main` branch can be installed now; and I will release v0.1.0 soon.

As blockers for this PR:
- It's still waiting for #3027 to be addressed as the folder name will dictate the split name
- The `self.split` name is set to `None` when the dataset dict is instantiated as follows:
```py
ds = Dataset.from_dict({"x": [1, 2, 3], "y": [4, 5, 6]})
local_ds = DatasetDict({"random": ds})

local_ds['random'].split  # returns None
```
In order to remove the `split=key` I would need to know of a different way to test here as it relies on the above as a surefire way of constructing a `DatasetDict`.
- Finally, the `threading` parameter is flaky on moon-staging which results in many errors server side. I propose to leave it as an argument instead of having it having it set to `True` so that users may toggle it according to their wish. 

Currently it looks like it only saves the last split.
Indeed when writing the data of one split, it deletes all the other files from the other splits
```python
>>> dataset.push_to_hub("lhoestq/squad_titles", shard_size=50<<10)           
Pushing split train to the Hub.
Pushing dataset shards to the dataset hub: 100%|█| 31/31 [00:22<00:00,  1.38
Pushing split validation to the Hub.
The repository already exists: the `private` keyword argument will be ignored.
Deleting unused files from dataset repository: 100%|█| 31/31 [00:14<00:00,  
Pushing dataset shards to the dataset hub: 100%|█| 4/4 [00:03<00:00,  1.18it
```
Note the "Deleting" part.

I think this PR should fix #3035, so feel free to link it. 

Thank you for your comments! I have rebased on `master` to have PR #3221. I've updated all tests to reflect the `-` instead of the `_` in the filenames.

@lhoestq, I have fixed the issue with splits and added a corresponding test.

@mariosasko I have not updated the `load_dataset` method to work differently, so I don't expect #3035 to be resolved with `push_to_hub`.

Only remaining issues before merging:
- Take a good look at the `threading` and if that's something we want to keep.
- As mentioned above:
>The self.split name is set to None when the dataset dict is instantiated as follows:
> ```
> ds = Dataset.from_dict({"x": [1, 2, 3], "y": [4, 5, 6]})
> local_ds = DatasetDict({"random": ds})
> 
> local_ds['random'].split  # returns None
> ```
I need to understand how to build a `DatasetDict` from some `Dataset` objects to be able to leverage the `split` parameter in `DatasetDict.push_to_hub`

Cool thanks ! And indeed this won't solve https://github.com/huggingface/datasets/issues/3035 yet

> I need to understand how to build a DatasetDict from some Dataset objects to be able to leverage the split parameter in DatasetDict.push_to_hub

You can use the key in the DatasetDict instead of the `split` attribute

What do you think about bumping the minimum version of pyarrow to 3.0.0 ? This is the minimum required version to write parquet files, which is needed for push_to_hub. That's why our pyarrow 1 CI is failing.

I think it's fine since it's been available for a long time (january 2021) and it's also the version that is installed on Google Colab.

Pushing pyarrow to 3.0.0 is fine for me. I don’t think we need to keep a lot of backward support for pyarrow.

Hi.
I published in the forum about my experience with `DatasetDict.push_to_hub()`: here is my [post.](https://discuss.huggingface.co/t/save-datasetdict-to-huggingface-hub/12075/4)
On my side, there is a problem as my train and validation `Datasets` are concatenated when I do a `load_dataset()` from the `DatasetDict` I pushed to the HF datasets hub.

Hi ! Let me respond here as well in case other people have the same issues and come here:

`push_to_hub` was introduced in `datasets` 1.16, and to be able to properly load a dataset with separated splits you need to have `datasets>=1.16.0` as well. 

Old version of `datasets` used to concatenate everything in the `train` split.