Creating a dummy dataset from a bigger one
Opened by patrickvonplaten (closed)

### Describe the bug

I often want to create a dummy dataset from a bigger dataset for fast iteration when training. However, I'm having a hard time doing this especially when trying to upload the dataset to the Hub.

### Steps to reproduce the bug

```python
from datasets import load_dataset

dataset = load_dataset("lambdalabs/pokemon-blip-captions")
dataset["train"] = dataset["train"].select(range(20))

dataset.push_to_hub("patrickvonplaten/dummy_image_data")
```

gives:

```
~/python_bin/datasets/arrow_dataset.py in _push_parquet_shards_to_hub(self, repo_id, split, private, token, branch, max_shard_size, embed_external_files)                                                                                    
   4003                 base_wait_time=2.0,
   4004                 max_retries=5,
-> 4005                 max_wait_time=20.0,
   4006             )
   4007         return repo_id, split, uploaded_size, dataset_nbytes

~/python_bin/datasets/utils/file_utils.py in _retry(func, func_args, func_kwargs, exceptions, status_codes, max_retries, base_wait_time, max_wait_time)                                                                                      
    328     while True:
    329         try:
--> 330             return func(*func_args, **func_kwargs)
    331         except exceptions as err:
    332             if retry >= max_retries or (status_codes and err.response.status_code not in status_codes):

~/hf/lib/python3.7/site-packages/huggingface_hub/utils/_validators.py in _inner_fn(*args, **kwargs)
    122             )
    123
--> 124         return fn(*args, **kwargs)
    125
    126     return _inner_fn  # type: ignore

TypeError: upload_file() got an unexpected keyword argument 'identical_ok'

In [2]: 
```

### Expected behavior

I would have expected this to work. It's for me the most intuitive way of creating a dummy dataset.

### Environment info

```
- `datasets` version: 2.1.1.dev0
- Platform: Linux-4.19.0-22-cloud-amd64-x86_64-with-debian-10.13
- Python version: 3.7.3
- PyArrow version: 11.0.0
- Pandas version: 1.3.5
```

Comments: Update `datasets` or downgrade `huggingface-hub` ;)

The `huggingface-hub` lib did a breaking change a few months ago, and you're using an old version of `datasets` that does't support it

Awesome thanks a lot! Everything works just fine with `datasets==2.9.0` :-) 

Getting same error with latest versions.


```shell
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[99], line 1
----> 1 dataset.push_to_hub("mirfan899/kids_phoneme_asr")

File /opt/conda/lib/python3.10/site-packages/datasets/arrow_dataset.py:3538, in Dataset.push_to_hub(self, repo_id, split, private, token, branch, shard_size, embed_external_files)
   3493 def push_to_hub(
   3494     self,
   3495     repo_id: str,
   (...)
   3501     embed_external_files: bool = True,
   3502 ):
   3503     """Pushes the dataset to the hub.
   3504     The dataset is pushed using HTTP requests and does not need to have neither git or git-lfs installed.
   3505 
   (...)
   3536     ```
   3537     """
-> 3538     repo_id, split, uploaded_size, dataset_nbytes = self._push_parquet_shards_to_hub(
   3539         repo_id=repo_id,
   3540         split=split,
   3541         private=private,
   3542         token=token,
   3543         branch=branch,
   3544         shard_size=shard_size,
   3545         embed_external_files=embed_external_files,
   3546     )
   3547     organization, dataset_name = repo_id.split("/")
   3548     info_to_dump = self.info.copy()

File /opt/conda/lib/python3.10/site-packages/datasets/arrow_dataset.py:3474, in Dataset._push_parquet_shards_to_hub(self, repo_id, split, private, token, branch, shard_size, embed_external_files)
   3472     shard.to_parquet(buffer)
   3473     uploaded_size += buffer.tell()
-> 3474     _retry(
   3475         api.upload_file,
   3476         func_kwargs=dict(
   3477             path_or_fileobj=buffer.getvalue(),
   3478             path_in_repo=path_in_repo(index),
   3479             repo_id=repo_id,
   3480             token=token,
   3481             repo_type="dataset",
   3482             revision=branch,
   3483             identical_ok=True,
   3484         ),
   3485         exceptions=HTTPError,
   3486         status_codes=[504],
   3487         base_wait_time=2.0,
   3488         max_retries=5,
   3489         max_wait_time=20.0,
   3490     )
   3491 return repo_id, split, uploaded_size, dataset_nbytes

File /opt/conda/lib/python3.10/site-packages/datasets/utils/file_utils.py:330, in _retry(func, func_args, func_kwargs, exceptions, status_codes, max_retries, base_wait_time, max_wait_time)
    328 while True:
    329     try:
--> 330         return func(*func_args, **func_kwargs)
    331     except exceptions as err:
    332         if retry >= max_retries or (status_codes and err.response.status_code not in status_codes):

File /opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py:120, in validate_hf_hub_args.<locals>._inner_fn(*args, **kwargs)
    117 if check_use_auth_token:
    118     kwargs = smoothly_deprecate_use_auth_token(fn_name=fn.__name__, has_token=has_token, kwargs=kwargs)
--> 120 return fn(*args, **kwargs)

TypeError: HfApi.upload_file() got an unexpected keyword argument 'identical_ok'
```

Feel free to update `datasets` and `huggingface-hub`, it should fix it :)

I went ahead and upgraded both datasets and hub and still getting the same error


Which version do you have ? It's been a while since it has been fixed

huggingface                             0.0.1
huggingface-hub                         0.17.1
datasets                                2.14.5

still has the issue!!

I face the same issue even after upgrading :/