Title: Add the 800GB Pile dataset?
Author: lewtun (MEMBER)
State: closed
Created: 2021-01-01T22:58:12Z
Labels: dataset request

## Adding a Dataset
- **Name:** The Pile
- **Description:** The Pile is a 825 GiB diverse, open source language modelling data set that consists of 22 smaller, high-quality datasets combined together. See [here](https://twitter.com/nabla_theta/status/1345130408170541056?s=20) for the Twitter announcement
- **Paper:** https://pile.eleuther.ai/paper.pdf
- **Data:** https://pile.eleuther.ai/
- **Motivation:** Enables hardcore  (GPT-3 scale!) language modelling

## Remarks
Given the extreme size of this dataset, I'm not sure how feasible this will be to include in `datasets` 🤯  . I'm also unsure how many `datasets` users are pretraining LMs, so the usage of this dataset may not warrant the effort to integrate it.

Comments:
The pile dataset would be very nice.
Benchmarks show that pile trained models achieve better results than most of actually trained models

The pile can very easily be added and adapted using this [tfds implementation](https://github.com/EleutherAI/The-Pile/blob/master/the_pile/tfds_pile.py) from the repo. 

However, the question is whether you'd be ok with 800GB+ cached in your local disk, since the tfds implementation was designed to offload the storage to Google Cloud Storage.

With the dataset streaming feature (see #2375) it will be more convenient to play with such big datasets :)
I'm currently adding C4 (see #2511 ) but I can probably start working on this afterwards

Hi folks! Just wanted to follow up on this -- would be really nice to get the Pile on HF Datasets... unclear if it would be easy to also add partitions of the Pile subject to the original 22 datasets used, but that would be nice too!

Hi folks, thanks to some awesome work by @lhoestq and @albertvillanova you can now stream the Pile as follows:

```python
# Install master branch of `datasets`
pip install git+https://github.com/huggingface/datasets.git#egg=datasets[streaming]
pip install zstandard

from datasets import load_dataset

dset = load_dataset("json", data_files="https://the-eye.eu/public/AI/pile/train/00.jsonl.zst", streaming=True, split="train")
next(iter(dset))
# {'meta': {'pile_set_name': 'Pile-CC'},
# 'text': 'It is done, and submitted. You can play “Survival of the Tastiest” on Android, and on the web ... '}
```

Next step is to add the Pile as a "canonical" dataset that can be streamed without specifying the file names explicitly :)

> Hi folks! Just wanted to follow up on this -- would be really nice to get the Pile on HF Datasets... unclear if it would be easy to also add partitions of the Pile subject to the original 22 datasets used, but that would be nice too!

Hi @siddk thanks to a tip from @richarddwang it seems we can access some of the partitions that EleutherAI created for the Pile [here](https://the-eye.eu/public/AI/pile_preliminary_components/). What's missing are links to the preprocessed versions of pre-existing datasets like DeepMind Mathematics and OpenSubtitles, but worst case we do the processing ourselves and host these components on the Hub.

My current idea is that we could provide 23 configs: one for each of the 22 datasets and an `all` config that links to the train / dev / test splits that EleutherAI released [here](https://the-eye.eu/public/AI/pile/), e.g.

```python
from datasets import load_dataset

# Load a single component
youtube_subtitles = load_dataset("the_pile", "youtube_subtitles")
# Load the train / dev / test splits of the whole corpus
dset = load_dataset("the_pile", "all")
```

Ideally we'd like everything to be compatible with the streaming API and there's ongoing work by @albertvillanova to make this happen for the various compression algorithms.

cc @lhoestq 

Ah I just saw that @lhoestq is already thinking about the specifying of one or more subsets in [this PR](https://github.com/huggingface/datasets/pull/2817#issuecomment-901874049) :)