Title: add draft of registering function for pandas
Author: lvwerra (MEMBER)
State: closed
Created: 2022-03-14T17:54:29Z

This PR adds a register function for `pandas`. It allows to directly push `DataFrame` objects to the hub and in return loading datasets on the hub from `DataFrame`. The motivation for this integration is to enable the vast number of `pandas` users to be able to easily push `DataFrames` to the hub. 

Here is an example:

```python
import pandas as pd
from datasets import register_pandas

register_pandas()

# push to hub
df = pd.DataFrame.from_dict({"test": [1,2,3]})
df.push_to_hub("my_test")

# load from hub
df_retrieved = pd.DataFrame.load_from_hub("lvwerra/my_test")
```

It follows a similar philosophy as the `tqdm` [integration](https://github.com/tqdm/tqdm#pandas-integration). Also see [this issue](https://github.com/pandas-dev/pandas/issues/46000) on the `pandas` repository.

This is just a rough draft of what such integration could look like but I would like appreciate some feedback on this: is this something you would like to add the library and is this the way to go? cc @lhoestq @albertvillanova @julien-c

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3912). All of your documentation changes will be reflected on that endpoint.

That's cool ! Though I would expect such an integration to only require `huggingface_hub`, not the full `datasets` library. 
 Indeed if users want to use the `datasets` lib they could just to `Dataset.from_pandas(df).push_to_hub()` already. Therefore I would explore something that doesn't not necessarily requires `datasets`.

For other could storage solutions (S3, GCS, etc.), pandas allows users to pass URIs like `s3://bucket-name/path/data.csv` to the `read_xxx` and `to_xxx` (for csv, parquet, json, etc). It also support passing the **root directory** like `s3://bucket-name/dataset-dir` instead of a single file name.

In the Hugging Face Hub case, we have one dataset = one repository. We can enter pandas' paradigm by saying one dataset = one repository = one root directory. Here is what we could have:

### push to Hub:
```python
"""
Demo script for writing a pandas data frame to a CSV file on HF using fsspec-supported pandas APIs
"""
import pandas as pd

HF_USER = os.getenv("HF_USER")
HF_TOKEN = os.getenv("HF_TOKEN")

books_df = pd.DataFrame(
    data={"Title": ["Book I", "Book II", "Book III"], "Price": [56.6, 59.87, 74.54]},
    columns=["Title", "Price"],
)

dataset_name = "books1"

books_df.to_csv(
    f"hf://{HF_USER}/{dataset_name}",
    index=False,
    storage_options={
        "repo_type": "dataset",
        "token": HF_TOKEN,
    },
)

```

### load from Hub:
```python
"""
Demo script for reading a CSV file from HF into a pandas data frame using fsspec-supported pandas
APIs
"""
import pandas as pd

HF_USER = os.getenv("HF_USER")
HF_TOKEN = os.getenv("HF_TOKEN")

dataset_name = "books1"

books_df = pd.read_csv(
    f"hf://{HF_USER}/{dataset_name}",
    storage_options={
        "repo_type": "dataset",
        "token": HF_TOKEN,
    },
)

print(books_df)
```

And you could do the same with Parquet data using `read/to_parquet` or other formats. Formats like CSV, Parquet or JSON Lines would work out of the box with `datasets`. This API would also allow anyone to use Dask with the Hugging Face Hub for example.

What do you think ?

I'm closing this PR as [`hffs`](https://github.com/huggingface/hffs) can now be used for reading/writing data frames from/to the Hub.