Title: `push_to_hub` is not robust to hub closing connection
Author: msis (NONE)
State: closed
Created: 2023-11-08T20:44:53Z

### Describe the bug

Like to #6172, `push_to_hub` will crash if Hub resets the connection and raise the following error:
```
Pushing dataset shards to the dataset hub:  32%|███▏      | 54/171 [06:38<14:23,  7.38s/it]
Traceback (most recent call last):
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/urllib3/connectionpool.py", line 462, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1348, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 316, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 285, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/urllib3/connectionpool.py", line 799, in urlopen
    retries = retries.increment(
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/urllib3/util/retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/urllib3/connectionpool.py", line 462, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1348, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 316, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 285, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/_commit_api.py", line 383, in _wrapped_lfs_upload
    lfs_upload(operation=operation, lfs_batch_action=batch_action, token=token)
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/lfs.py", line 223, in lfs_upload
    _upload_multi_part(operation=operation, header=header, chunk_size=chunk_size, upload_url=upload_action["href"])
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/lfs.py", line 319, in _upload_multi_part
    else _upload_parts_iteratively(operation=operation, sorted_parts_urls=sorted_parts_urls, chunk_size=chunk_size)
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/lfs.py", line 375, in _upload_parts_iteratively
    part_upload_res = http_backoff("PUT", part_upload_url, data=fileobj_slice)
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/utils/_http.py", line 258, in http_backoff
    response = session.request(method=method, url=url, **kwargs)
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/utils/_http.py", line 63, in send
    return super().send(request, *args, **kwargs)
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: (ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')), '(Request ID: 2bab8c06-b701-4266-aead-fe2e0dc0e3ed)')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "convert_to_hf.py", line 116, in <module>
    main()
  File "convert_to_hf.py", line 108, in main
    audio_dataset.push_to_hub(
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/datasets/dataset_dict.py", line 1641, in push_to_hub
    repo_id, split, uploaded_size, dataset_nbytes, _, _ = self[split]._push_parquet_shards_to_hub(
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 5308, in _push_parquet_shards_to_hub
    _retry(
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/datasets/utils/file_utils.py", line 290, in _retry
    return func(*func_args, **func_kwargs)
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/hf_api.py", line 828, in _inner
    return fn(self, *args, **kwargs)
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/hf_api.py", line 3221, in upload_file
    commit_info = self.create_commit(
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/hf_api.py", line 828, in _inner
    return fn(self, *args, **kwargs)
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/hf_api.py", line 2695, in create_commit
    upload_lfs_files(
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/_commit_api.py", line 393, in upload_lfs_files
    _wrapped_lfs_upload(filtered_actions[0])
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/_commit_api.py", line 385, in _wrapped_lfs_upload
    raise RuntimeError(f"Error while uploading '{operation.path_in_repo}' to the Hub.") from exc
RuntimeError: Error while uploading 'batch_19/train-00054-of-00171-932beb4082c034bf.parquet' to the Hub.
```

The function should retry if the operations fails, or at least offer a way to recover after such a failure. 
Right now, calling the function again will start sending all the parquets files leading to duplicates in the repository, with no guarantee that it will actually  be pushed.

Previously, it would crash with an error 400 #4677 .

### Steps to reproduce the bug

Any large dataset pushed the hub:

```py
audio_dataset.push_to_hub(
        repo_id="org/dataset",
    )
```

### Expected behavior

`push_to_hub` should have an option for max retries or resume.

### Environment info

- `datasets` version: 2.14.6
- Platform: Linux-5.15.0-1044-aws-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.16.4
- PyArrow version: 13.0.0
- Pandas version: 2.0.3

Comments:
Hi! We made some improvements to `push_to_hub` to make it more robust a couple of weeks ago but haven't published a release in the meantime, so it would help if you could install `datasets` from `main` (`pip install https://github.com/huggingface/datasets`) and let us know if this improved version of `push_to_hub` resolves the issue (in case the `ConnectionError` happens, re-running `push_to_hub` should be faster now).

Also, note that the previous implementation retries the upload, but sometimes this is not enough, so re-running the op is the only option.

The update helped push more data.
However it still crashed a little later:

```
Traceback (most recent call last):
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/utils/_errors.py", line 270, in hf_raise_for_status
    response.raise_for_status()
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://hf-hub-lfs-us-east-1.s3.us-east-1.amazonaws.com/repos/6c/33/6c33b3be1463a656e43c7a4f2d43c4a1cdae6e9d81fff87f69167ef25ccb1b88/5f53cb57cf2a52ca0d4c2166a69a6714c64fcdbb7cb8936dfa5b11ac60058e5f?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIA2JU7TKAQFN2FTF47%2F20231110%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20231110T011254Z&X-Amz-Expires=86400&X-Amz-Signature=74e3e33c09ac4e7c6ac887aaee8d489f068869abbe1ee6d58a910fb18d0601d4&X-Amz-SignedHeaders=host&partNumber=13&uploadId=kQwunNkunfmT9D8GulQu_ufw1BTZtRA6wEUI4hnYOjytfdf.GKxDETgMr4wm8_0WNF2yGaNco_0h3JAGm4l9KV1N0nqr5XXyUCbs1ROmHP475fn9FIhc1umWQLEDc97V&x-id=UploadPart

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/_commit_api.py", line 391, in _wrapped_lfs_upload
    lfs_upload(operation=operation, lfs_batch_action=batch_action, token=token)
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/lfs.py", line 223, in lfs_upload
    _upload_multi_part(operation=operation, header=header, chunk_size=chunk_size, upload_url=upload_action["href"])
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/lfs.py", line 319, in _upload_multi_part
    else _upload_parts_iteratively(operation=operation, sorted_parts_urls=sorted_parts_urls, chunk_size=chunk_size)
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/lfs.py", line 376, in _upload_parts_iteratively
    hf_raise_for_status(part_upload_res)
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/utils/_errors.py", line 330, in hf_raise_for_status
    raise HfHubHTTPError(str(e), response=response) from e
huggingface_hub.utils._errors.HfHubHTTPError: 500 Server Error: Internal Server Error for url: https://hf-hub-lfs-us-east-1.s3.us-east-1.amazonaws.com/repos/6c/33/6c33b3be1463a656e43c7a4f2d43c4a1cdae6e9d81fff87f69167ef25ccb1b88/5f53cb57cf2a52ca0d4c2166a69a6714c64fcdbb7cb8936dfa5b11ac60058e5f?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIA2JU7TKAQFN2FTF47%2F20231110%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20231110T011254Z&X-Amz-Expires=86400&X-Amz-Signature=74e3e33c09ac4e7c6ac887aaee8d489f068869abbe1ee6d58a910fb18d0601d4&X-Amz-SignedHeaders=host&partNumber=13&uploadId=kQwunNkunfmT9D8GulQu_ufw1BTZtRA6wEUI4hnYOjytfdf.GKxDETgMr4wm8_0WNF2yGaNco_0h3JAGm4l9KV1N0nqr5XXyUCbs1ROmHP475fn9FIhc1umWQLEDc97V&x-id=UploadPart

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "convert_to_hf.py", line 121, in <module>
    main()
  File "convert_to_hf.py", line 109, in main
    audio_dataset.push_to_hub(
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/datasets/dataset_dict.py", line 1699, in push_to_hub
    split_additions, uploaded_size, dataset_nbytes = self[split]._push_parquet_shards_to_hub(
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 5215, in _push_parquet_shards_to_hub
    _retry(
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/datasets/utils/file_utils.py", line 290, in _retry
    return func(*func_args, **func_kwargs)
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/hf_api.py", line 3665, in preupload_lfs_files
    _upload_lfs_files(
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/_commit_api.py", line 401, in _upload_lfs_files
    _wrapped_lfs_upload(filtered_actions[0])
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/_commit_api.py", line 393, in _wrapped_lfs_upload
    raise RuntimeError(f"Error while uploading '{operation.path_in_repo}' to the Hub.") from exc
RuntimeError: Error while uploading 'batch_20/train-00206-of-00261.parquet' to the Hub.
```

I think the previous implementation was actually better: it pushes to the hub every shard. So if it fails, as long as the shards have the same checksum, it will skip the ones that have been pushed.

The implementation in `main` pushes commits at the end, so when it fails, there are no commits and therefore restarts from the beginning every time.

Below is the another error log from another run with `main`. I've reverting back to the current release as it does the job for me.

```
Uploading the dataset shards:  86%|████████▌ | 224/261 [21:46<03:35,  5.83s/it]s]
Traceback (most recent call last):
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/utils/_errors.py", line 270, in hf_raise_for_status
    response.raise_for_status()
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://hf-hub-lfs-us-east-1.s3.us-east-1.amazonaws.com/repos/6c/33/6c33b3be1463a656e43c7a4f2d43c4a1cdae6e9d81fff87f69167ef25ccb1b88/97e68d7a5d4a747ffaa249fc09798e961d621fe4170599e6100197f7733f321d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIA2JU7TKAQFN2FTF47%2F20231110%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20231110T145155Z&X-Amz-Expires=86400&X-Amz-Signature=5341e4b34dc325737f92dc9005c4a31e4d3f9a3d3d853b267e01915260acf629&X-Amz-SignedHeaders=host&partNumber=27&uploadId=NRD0izEWv7MPtC2bYrm5VJ4XgIbHctKNguR7zS1UhGOOrXwBJvigrOywBvQBnS9sxiy0J0ma9sNog8S13nIdTdE9p60MIITTstUFeKvLHSxpU.a527QED1JVYzJ.9xA0&x-id=UploadPart

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/_commit_api.py", line 391, in _wrapped_lfs_upload
    lfs_upload(operation=operation, lfs_batch_action=batch_action, token=token)
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/lfs.py", line 223, in lfs_upload
    _upload_multi_part(operation=operation, header=header, chunk_size=chunk_size, upload_url=upload_action["href"])
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/lfs.py", line 319, in _upload_multi_part
    else _upload_parts_iteratively(operation=operation, sorted_parts_urls=sorted_parts_urls, chunk_size=chunk_size)
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/lfs.py", line 376, in _upload_parts_iteratively
    hf_raise_for_status(part_upload_res)
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/utils/_errors.py", line 330, in hf_raise_for_status
    raise HfHubHTTPError(str(e), response=response) from e
huggingface_hub.utils._errors.HfHubHTTPError: 500 Server Error: Internal Server Error for url: https://hf-hub-lfs-us-east-1.s3.us-east-1.amazonaws.com/repos/6c/33/6c33b3be1463a656e43c7a4f2d43c4a1cdae6e9d81fff87f69167ef25ccb1b88/97e68d7a5d4a747ffaa249fc09798e961d621fe4170599e6100197f7733f321d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIA2JU7TKAQFN2FTF47%2F20231110%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20231110T145155Z&X-Amz-Expires=86400&X-Amz-Signature=5341e4b34dc325737f92dc9005c4a31e4d3f9a3d3d853b267e01915260acf629&X-Amz-SignedHeaders=host&partNumber=27&uploadId=NRD0izEWv7MPtC2bYrm5VJ4XgIbHctKNguR7zS1UhGOOrXwBJvigrOywBvQBnS9sxiy0J0ma9sNog8S13nIdTdE9p60MIITTstUFeKvLHSxpU.a527QED1JVYzJ.9xA0&x-id=UploadPart

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "convert_to_hf.py", line 121, in <module>
    main()
  File "convert_to_hf.py", line 109, in main
    audio_dataset.push_to_hub(
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/datasets/dataset_dict.py", line 1699, in push_to_hub
    p, glob_pattern_to_regex(PUSH_TO_HUB_WITHOUT_METADATA_CONFIGS_SPLIT_PATTERN_SHARDED)
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 5215, in _push_parquet_shards_to_hub
    token = token if token is not None else HfFolder.get_token()
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/datasets/utils/file_utils.py", line 290, in _retry
    return func(*func_args, **func_kwargs)
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/hf_api.py", line 3665, in preupload_lfs_files
    _upload_lfs_files(
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/_commit_api.py", line 401, in _upload_lfs_files
    _wrapped_lfs_upload(filtered_actions[0])
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/_commit_api.py", line 393, in _wrapped_lfs_upload
    raise RuntimeError(f"Error while uploading '{operation.path_in_repo}' to the Hub.") from exc
RuntimeError: Error while uploading 'batch_20/train-00224-of-00261.parquet' to the Hub.
```

There's a new error from the hub now:
```
Pushing dataset shards to the dataset hub:  49%|████▉     | 128/261 [11:38<12:05,  5.45s/it]
Traceback (most recent call last):
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/utils/_errors.py", line 270, in hf_raise_for_status
    response.raise_for_status()
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://huggingface.co/api/datasets/tarteel-ai/tawseem/commit/main

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "convert_to_hf.py", line 121, in <module>
    main()
  File "convert_to_hf.py", line 109, in main
    audio_dataset.push_to_hub(
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/datasets/dataset_dict.py", line 1641, in push_to_hub
    repo_id, split, uploaded_size, dataset_nbytes, _, _ = self[split]._push_parquet_shards_to_hub(
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 5308, in _push_parquet_shards_to_hub
    _retry(
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/datasets/utils/file_utils.py", line 293, in _retry
    raise err
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/datasets/utils/file_utils.py", line 290, in _retry
    return func(*func_args, **func_kwargs)
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/hf_api.py", line 1045, in _inner
    return fn(self, *args, **kwargs)
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/hf_api.py", line 3850, in upload_file
    commit_info = self.create_commit(
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/hf_api.py", line 1045, in _inner
    return fn(self, *args, **kwargs)
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/hf_api.py", line 3237, in create_commit
    hf_raise_for_status(commit_resp, endpoint_name="commit")
  File "/admin/home-piraka9011/.virtualenvs/w2v2/lib/python3.8/site-packages/huggingface_hub/utils/_errors.py", line 330, in hf_raise_for_status
    raise HfHubHTTPError(str(e), response=response) from e
huggingface_hub.utils._errors.HfHubHTTPError: 429 Client Error: Too Many Requests for url: https://huggingface.co/api/datasets/tarteel-ai/tawseem/commit/main (Request ID: Root=1-654e48e6-598511b14413bb293fa67084;783522b4-66f9-4f8a-8a74-2accf7cabd17)

You have exceeded our hourly quotas for action: commit. We invite you to retry later.
```

At least this is more explicit from the server side.

>  think the previous implementation was actually better: it pushes to the hub every shard. So if it fails, as long as the shards have the same checksum, it will skip the ones that have been pushed.
>
>The implementation in main pushes commits at the end, so when it fails, there are no commits and therefore restarts from the beginning every time.
>
>Below is the another error log from another run with main. I've reverting back to the current release as it does the job for me.

The `preupload` step is instant for the already uploaded shards, so only the Parquet conversion is repeated without uploading the actual Parquet data (only to check the SHAs). The previous implementation manually checks the Parquet shard's fingerprint to resume uploading, so the current implementation is cleaner.

> You have exceeded our hourly quotas for action: commit. We invite you to retry later.

This is the problem with the previous implementation. If the number of shards is large, it creates too many commits for the Hub in a short period.

But I agree that the `500 Server Error` returned by the Hub is annoying. Earlier today, I also got it on a small 5GB dataset (with 500 MB shards).

@Wauplin @julien-c Is there something we can do about this?

@mariosasko can't do much if AWS raises a HTTP 500 unfortunately (we are simply pushing data to a S3 bucket).
What we can do is to add a retry mechanism in the multi-part upload logic here: https://github.com/huggingface/huggingface_hub/blob/c972cba1fecb456a7b3325cdd1fdbcc425f21f94/src/huggingface_hub/lfs.py#L370 :confused: 

@Wauplin That code already retries the request using `http_backoff`, no?

> That code already retries the request using http_backoff, no?

Currently only on HTTP 503 by default. We should add 500 as well (and hope it is a transient error from AWS)

Opened a PR to retry in case S3 raises HTTP 500. Will also retry on any `ConnectionError` (connection reset by peer, connection lost,...). Hopefully this should make the upload process more robust to transient errors.

I still get the same error, using `push_to_hub`. Using `git lfs` and pushing the files solved it for me.

@BEpresent the fix has not been released yet. You can expect a release of `huggingface_hub` (with this fix) today or tomorrow :)