Title: Dataset.push_to_hub of a canonical dataset creates an additional dataset under the user namespace
Author: albertvillanova (MEMBER)
State: closed
Created: 2024-01-16T11:27:07Z
Labels: bug

While using `Dataset.push_to_hub` of a canonical dataset, an additional dataset was created under my user namespace.

## Steps to reproduce the bug
The command:
```python
commit_info = ds.push_to_hub(
    "caner",
    config_name="default",
    commit_message="Convert dataset to Parquet",
    commit_description="Convert dataset to Parquet.",
    create_pr=True,
    token=token,
)
```
creates the additional dataset `albertvillanova/caner`.

Comments:
It is caused by these code lines: https://github.com/huggingface/datasets/blob/9d6d16117a30ba345b0236407975f701c5b288d4/src/datasets/dataset_dict.py#L1688-L1694

Also note the information in the docstring: https://github.com/huggingface/datasets/blob/9d6d16117a30ba345b0236407975f701c5b288d4/src/datasets/dataset_dict.py#L1582-L1585

> Also accepts `<dataset_name>`, which will default to the namespace of the logged-in user.

This behavior was "reverted" by the PR: 
- #6519

We have therefore contradictory requirements. We should decide:
- whether to support passing dataset_namespace without user/org that defaults to the logged-in user (and not support canonical datasets)
- or vice-versa, to support canonical datasets and not support passing only dataset_name

As canonical datasets are "deprecated" (and will eventually disappear), I would choose the first option. However, if so, the Space to convert datasets to Parquet will not work for canonical datasets: https://huggingface.co/spaces/albertvillanova/convert-dataset-to-parquet

IIUC, this could also be "fixed" by `create_repo("dataset_name")` not defaulting to `create_repo("user/dataset_name")` (when the user's token is available), which would be consistent with the rest of the `HfApi` ops used in the `push_to_hub` implementation. This is a (small) breaking change for `huggingface_hub`, but justified to make the API more consistent.

I tag @Wauplin to have his opinion as well.

Hmm, creating repo with implicit namespace (e.g. `create_repo("dataset_name")`) is a convenient feature used in a lot of integrations. It is not consistent with other HfApi methods specifically because it is the method to create repos. Once the repo is created, the return value provides the explicit repo_id (`namespace/repo_name`) that has to be passed to every `HfApi` method. Otherwise, libraries/scripts would often need to do a `whoami` call to get the namespace before creating a repo.

 Another solution for https://github.com/huggingface/datasets/issues/6597#issuecomment-1893746690 could be that implicit namespace is allowed (same as today) except if the `repo_id` is in a hard-coded list of canonical datasets. This list can be maintained automatically and should be slowly decreasing. **Caveat:** as a normal user I wouldn't be able to implicitly push to `imagenet-1k` if I wanted to push to `Wauplin/imagenet-1k`. Shouldn't be too problematic, no? Worse case, would need to add a `whoami` call and allow implicit-canonical-name for non-HF users for instance (a bit too over-engineered IMO but doable). 

As canonical datasets are going to disappear in the following couple of months, I would not make any effort on their support.

I propose reverting #6519, so that the behavior of `push_to_hub` is aligned with the one described in its dosctring: "Also accepts `<dataset_name>`, which will default to the namespace of the logged-in user."

I'm opening a PR.