Title: Add CelebA Dataset
Author: gchhablani (CONTRIBUTOR)
State: closed
Created: 2021-02-05T20:20:55Z

Trying to add CelebA Dataset. 
Need help with testing. Loading examples takes a lot of time so I am unable to generate the `dataset_infos.json` and unable to test. Also, need help with creating `dummy_data.zip`.

Additionally, trying to load a few examples using `load_dataset('./datasets/celeb_a',split='train[10:20]')` still loads all the examples (doesn't stop at 10).

Comments:
Hi @gchhablani! Thanks for all the contributions! We definitely want more image datasets, but Face datasets are tricky in general, in this one includes predicting attributes such as Attractiveness, Gender, or Race, which can be pretty problematic.

Would you be up for starting with only object classification or object detection datasets instead? (Your CIFAR-100 contribution will be super useful for example!)

Hi @yjernite, You're welcome. I am enjoying adding new datasets :)
By "pretty problematic", are you referring to the ethical issues? I used TFDS's [CelebA](https://github.com/tensorflow/datasets/blob/5ef7861470896acb6f74dacba85036001e4f1b8c/tensorflow_datasets/image/celeba.py#L91) as a reference. Here they mention in a "Note" that CelebA  "may contain potential bias". Can we not do the same? I skipped the note for now, and we can add it. However, if you feel this isn't the right time, then I won't pursue this further. 

But, can this issue be handled at a later stage? Does this also apply for my Hateful Memes Issue #1810?

Also, how can I 
1. load a part of the dataset? since `load_dataset(<>,split='train[10:20]')` still loads all the examples.
2. make `datasets_infos.json` for huge datasets which have a single configuration?

I will ofcourse be looking for other datasets to add regardless. 


It's definitely a thorny question. The short answer is: Hateful Memes and hate speech detection datasets are different since their use case is specifically to train systems to identify and hopefully remove hateful content, whereas the purpose of a dataset that has an Attractiveness score as output is implicitly to train more models to rate "Attractiveness". 

As far as warning about the "potential biases", I do not think it is quite enough, especially because it is hard to guarantee that every potential user will read the documentation (it is also an insufficient warning.)

Note that we do have higher standards for the dataset cards of hate speech and hateful memes datasets, so if you do choose to add that one yourself we will ask that you summarize the relevant literature in the Social Impact section.

If you really need to add this dataset for your own research for the explicit purpose of studying these biases, you can add it as a community provided dataset following https://huggingface.co/docs/datasets/master/share_dataset.html#sharing-a-community-provided-dataset but I'd recommend just skipping it for now.

So currently you do need to download the whole dataset when using it, we are working on making it easier to stream parts of it from a remote host. You can also use the filesystem integration if local storage is an issue:
https://huggingface.co/docs/datasets/master/filesystems.html


I don't think we have a great solution for `dataset_infos.json` with a single very large config when storage space is an issue, but it should be solved by the same upcoming feature mentioned above

Okay, then I won't pursue this one further. I'll keep this branch on my repository just in case the possibility of adding this dataset comes up in the future.

> So currently you do need to download the whole dataset when using it, we are working on making it easier to stream parts of it from a remote host. You can also use the filesystem integration if local storage is an issue:
> https://huggingface.co/docs/datasets/master/filesystems.html

After downloading the whole dataset (around 1.4GB), it still loads all the examples despite using `split='train[:10%]'` or `split='train[10:20]'`. 

EDIT: I think this would happen only when the examples are generated for the first time and saved to the cache. Streaming parts of the data from a remote host sounds amazing! But, would that also allow for streaming examples of the data from the local cache? (without saving all the examples the first time).

What I used:
`d = load_dataset('./datasets/celeb_a',split='train[:10]')`
Output:
`570 examples [01:33,  6.25 examples/s]` and it keeps going. 

EDIT 2: After a few thousand images, I get the following error:
```python
OSError: [Errno 24] Too many open files: '~/.cache/huggingface/datasets/celeb_a/default/1.1.0/01f9dca66039ab7c40b91b09af47a5fa8c3e49dc8d55df50da55b14116229207.incomplete'
```
I understand this is because of the way I load the images :
```python
Image.open(<path>)
```
What could be better alternative? I am only asking in case I face the same issues in the future.

Just some addition about loading only a subset of the data:
Currently if even you specify `split='train[:10]'`, it downloads and generate the full dataset, so that you can pick another part afterward if you want to. We may change that in the future and use streaming.

And about your open files issue, you can try to close each image file after reading its content.

Hi @lhoestq,
Thanks for your response.

I used `gc.collect()` inside the loop and that worked for me. I think since we are using a generator, and if I have something like `train[100000:100002]`, we will need to generate the first 1000001 examples and store. Ofcourse, this feature isn't a necessity right now, I suppose.

Closing this PR.