t20062443/AT1ds huggingface dataset:
lhoestq (MEMBER) commented on 2021-02-01 17:05:13:
Hi ! Indeed we used to require mapping functions to be picklable with `pickle` or `dill` in order to cache the resulting datasets. And FAISS indexes are not picklable unfortunately.

But since #1703 this is no longer required (the caching will simply be disabled). This change will be available in the next release of `datasets`, or you can also install `datasets` from source.
SBrandeis (CONTRIBUTOR) commented on 2021-02-02 10:56:23:
AFAIK, there is currently no built-in method on the `Dataset` object to do this.
However, a workaround is to directly use the Arrow table backing the dataset, **but it implies loading the whole dataset in memory** (correct me if I'm mistaken @lhoestq).

You can convert the Arrow table to a pandas dataframe to save the data as csv as follows:
```python
arrow_table = dataset.data
dataframe = arrow_table.to_pandas()
dataframe.to_csv("/path/to/file.csv")
```

Similarly, you can convert the dataset to a Python dict and save it as JSON:
```python
import json
arrow_table = dataset.data
py_dict = arrow_table.to_pydict()
with open("/path/to/file.json", "w+") as f:
    json.dump(py_dict, f)
```
lhoestq (MEMBER) commented on 2021-02-02 15:04:08:
Indeed this works as long as you have enough memory.
It would be amazing to have export options like csv, json etc. !

It should be doable to implement something that iterates through the dataset batch by batch to write to csv for example.
There is already an `export` method but currently the only export type that is supported is `tfrecords`.
gchhablani (CONTRIBUTOR) commented on 2021-02-03 16:33:06:
Hi @lhoestq,

I think this happened because of rebasing. I'm unable to remove the duorc commit from the branch.  GEM, Arabic sarcasm datasets are also there. I can't see any merge conflicts, however. Before commiting I always rebase (shouldn't have done that).
Can you explain what is to be done? Should I create a clean PR?
lhoestq (MEMBER) commented on 2021-02-03 16:36:53:
Hi @gchhablani 
I think you can simply create another branch and another PR.

If I understand correctly the github diff is messed up because you rebased instead of merge.
Rebasing is supposed to be used only before pushing the branch the first time, or github messes up the diff.
If you want to include changes from master on a branch that is already push you need to use git merge.
gchhablani (CONTRIBUTOR) commented on 2021-02-03 17:15:05:
@lhoestq Haha, well, it's not as weird as not reading the [instructions](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#open-a-pull-request-on-the-main-huggingface-repo-and-share-your-work).
Also, I'm enjoying adding new datasets so it's all cool :)
aclifton314 (CONTRIBUTOR) commented on 2021-02-03 18:28:33:
To fix the branch issue, I went ahead and made a backup of the dataset then deleted my local copy of my fork of `datasets`. I then followed the [detailed guide](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md) from the beginning to reclone the fork and start a new branch. 

However, when it came time to create the dummy data I got the following error:
```python
aclifton@pop-os:~/data/hf_datasets_sprint/datasets$ datasets-cli dummy_data ./datasets/logiqa_en --auto_generate --n_lines 32
2021-02-03 11:23:23.145885: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2021-02-03 11:23:23.145914: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Using custom data configuration default
Couldn't generate dummy file 'datasets/logiqa_en/dummy/1.1.0/dummy_data/master.zip/LogiQA-dataset-master/README.md'. Ignore that if this file is not useful for dummy data.
Traceback (most recent call last):
  File "/home/aclifton/anaconda3/bin/datasets-cli", line 36, in <module>
    service.run()
  File "/home/aclifton/anaconda3/lib/python3.7/site-packages/datasets/commands/dummy_data.py", line 317, in run
    keep_uncompressed=self._keep_uncompressed,
  File "/home/aclifton/anaconda3/lib/python3.7/site-packages/datasets/commands/dummy_data.py", line 355, in _autogenerate_dummy_data
    dataset_builder._prepare_split(split_generator)
  File "/home/aclifton/anaconda3/lib/python3.7/site-packages/datasets/builder.py", line 905, in _prepare_split
    example = self.info.features.encode_example(record)
  File "/home/aclifton/anaconda3/lib/python3.7/site-packages/datasets/features.py", line 799, in encode_example
    return encode_nested_example(self, example)
  File "/home/aclifton/anaconda3/lib/python3.7/site-packages/datasets/features.py", line 710, in encode_nested_example
    (k, encode_nested_example(sub_schema, sub_obj)) for k, (sub_schema, sub_obj) in utils.zip_dict(schema, obj)
  File "/home/aclifton/anaconda3/lib/python3.7/site-packages/datasets/features.py", line 710, in <genexpr>
    (k, encode_nested_example(sub_schema, sub_obj)) for k, (sub_schema, sub_obj) in utils.zip_dict(schema, obj)
  File "/home/aclifton/anaconda3/lib/python3.7/site-packages/datasets/features.py", line 737, in encode_nested_example
    return schema.encode_example(obj)
  File "/home/aclifton/anaconda3/lib/python3.7/site-packages/datasets/features.py", line 522, in encode_example
    example_data = self.str2int(example_data)
  File "/home/aclifton/anaconda3/lib/python3.7/site-packages/datasets/features.py", line 481, in str2int
    output.append(self._str2int[str(value)])
KeyError: "Some Cantonese don't like chili, so some southerners don't like chili."
```
lhoestq (MEMBER) commented on 2021-02-04 16:44:19:
Hi !

We already have some datasets that can have many many configurations possible.
To be able to support that, we allow to subclass BuilderConfig to add as many additional parameters as you may need.
This way users can load any language they want. For example the [bible_para](https://github.com/huggingface/datasets/blob/master/datasets/bible_para/bible_para.py) dataset is a dataset for translation and therefore users should be able to provide any language pair. You can check how the subclass of BuilderConfig is defined [here](https://github.com/huggingface/datasets/blob/master/datasets/bible_para/bible_para.py#L49).

For testing, only the configurations defined in the `BUILDER_CONFIGS` class attribute are used.
All the other configs combinations are not tested, but they can be used by users. If a config doesn't already exist in `BUILDER_CONFIGS`, then it is created on the fly.
For example in [bible_para](https://github.com/huggingface/datasets/blob/master/datasets/bible_para/bible_para.py#L61), only 6 configs are defined in `BUILDER_CONFIGS`.

So what I would do in your case is have something like
```python

class CCAlignedConfig(datasets.BuilderConfig):
    def __init__(self, *args, documents_or_sentences=None, language_code=None, **kwargs):
        super().__init__(
            *args,
            name=f"{documents_or_sentences}-{language_code}",
            **kwargs,
        )
        self.documents_or_sentences = documents_or_sentences
        self.language_code = language_code
```
And of course, feel free to change/rename things if you want to. In particular I think we can improve the name of the parameter `documents_or_sentences`
albertvillanova (MEMBER) commented on 2021-02-05 07:57:15:
Hi!

Thank you for reporting this issue. I agree that the information about the exception should be more clear and explicit.

I could take on this issue.

On the meantime, as you can see from the exception stack trace, HF Datasets uses pandas to read the CSV files. You can pass arguments to `pandas.read_csv` by passing additional keyword arguments to `load_dataset`. For example, you may find useful this argument:
- `error_bad_lines` : bool, default True
  Lines with too many fields (e.g. a csv line with too many commas) will by default cause an exception to be raised, and no DataFrame will be returned. If False, then these “bad lines” will be dropped from the DataFrame that is returned.

You could try:
```python
datasets = load_dataset("csv", data_files=dict(train=train_files, validation=validation_files), error_bad_lines=False)
```
lhoestq (MEMBER) commented on 2021-02-05 11:03:49:
Hi ! The error happens when the script is verifying that the generated dummy data work fine with the dataset script.
Apparently it fails because the text `"Some Cantonese don't like chili, so some southerners don't like chili."` was given in a field that is a ClassLabel feature (probably the `answer` field), while it actually expects "a", "b", "c" or "d". Can you fix the script so that it returns the expected labels for this field instead of the text ?

Also it would be awesome to rename this field `answerKey` instead of `answer` to have the same column names as the other multiple-choice-QA datasets in the library :)
lhoestq (MEMBER) commented on 2021-02-05 13:18:38:
Hi ! Looks related to #861 

You are right: tokenizing a dataset using map takes a lot of space since it can store `input_ids` but also `token_type_ids`, `attention_mask` and `special_tokens_mask`. Moreover if your tokenization function returns python integers then by default they'll be stored as int64 which can take a lot of space. Padding can also increase the size of the tokenized dataset.

To make things more convenient, we recently added a "lazy map" feature that allows to tokenize each batch at training time as you mentioned. For example you'll be able to do
```python
from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")

def encode(batch):
    return tokenizer(batch["text"], padding="longest", truncation=True, max_length=512, return_tensors="pt")

dataset.set_transform(encode)
print(dataset.format)
# {'type': 'custom', 'format_kwargs': {'transform': <function __main__.encode(batch)>}, 'columns': ['idx', 'label', 'sentence1', 'sentence2'], 'output_all_columns': False}
print(dataset[:2])
# {'input_ids': tensor([[  101,  2572,  3217, ... 102]]), 'token_type_ids': tensor([[0, 0, 0, ... 0]]), 'attention_mask': tensor([[1, 1, 1, ... 1]])}

```
In this example the `encode` transform is applied on-the-fly on the "text" column.

This feature will be available in the next release 2.0 which will happen in a few days.
You can already play with it by installing `datasets` from source if you want :)

Hope that helps !
lhoestq (MEMBER) commented on 2021-02-05 13:23:35:
Hi ! Thanks for reporting. This was indeed a bug introduced when we moved the `json` dataset loader inside the `datasets` package (before that, the `json` loader was fetched online, as all the other dataset scripts).

This should be fixed on master now. Feel free to install `datasets` from source to try it out.
The fix will be available in the next release of `datasets` in a few days
lhoestq (MEMBER) commented on 2021-02-05 14:29:20:
Hi !
The error you have is due to the `input_ids` column not having the same number of examples as the other columns.
Indeed you're concatenating the `input_ids` at this line:

https://github.com/LuCeHe/GenericTools/blob/431835d8e13ec24dceb5ee4dc4ae58f0e873b091/KerasTools/lm_preprocessing.py#L134

However the other columns are kept unchanged, and therefore you end up with an `input_ids` column with 599 elements while the others columns like `attention_mask` have 1500.

To fix that you can instead concatenate them all using
```python
concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}
```

Also you may need to drop the "text" column before applying `group_texts` since strings can't be concatenated with lists. You can drop it at the tokenization step:
```python
dset = dset.map(
    tokenize_function,
    batched=True,
    remove_columns=["text"]
)
```
EvanZ (NONE) commented on 2021-02-05 18:17:46:
How recently was `set_transform` added? I am actually trying to implement it and getting an error:

`AttributeError: 'Dataset' object has no attribute 'set_transform'
`

I'm on v.1.2.1.

EDIT: Oh, wait I see now it's in the v.2.0. Whoops! This should be really useful.
yjernite (MEMBER) commented on 2021-02-05 20:47:28:
Hi @gchhablani! Thanks for all the contributions! We definitely want more image datasets, but Face datasets are tricky in general, in this one includes predicting attributes such as Attractiveness, Gender, or Race, which can be pretty problematic.

Would you be up for starting with only object classification or object detection datasets instead? (Your CIFAR-100 contribution will be super useful for example!)
gchhablani (CONTRIBUTOR) commented on 2021-02-05 21:07:34:
Hi @yjernite, You're welcome. I am enjoying adding new datasets :)
By "pretty problematic", are you referring to the ethical issues? I used TFDS's [CelebA](https://github.com/tensorflow/datasets/blob/5ef7861470896acb6f74dacba85036001e4f1b8c/tensorflow_datasets/image/celeba.py#L91) as a reference. Here they mention in a "Note" that CelebA  "may contain potential bias". Can we not do the same? I skipped the note for now, and we can add it. However, if you feel this isn't the right time, then I won't pursue this further. 

But, can this issue be handled at a later stage? Does this also apply for my Hateful Memes Issue #1810?

Also, how can I 
1. load a part of the dataset? since `load_dataset(<>,split='train[10:20]')` still loads all the examples.
2. make `datasets_infos.json` for huge datasets which have a single configuration?

I will ofcourse be looking for other datasets to add regardless.
yjernite (MEMBER) commented on 2021-02-05 22:06:13:
It's definitely a thorny question. The short answer is: Hateful Memes and hate speech detection datasets are different since their use case is specifically to train systems to identify and hopefully remove hateful content, whereas the purpose of a dataset that has an Attractiveness score as output is implicitly to train more models to rate "Attractiveness". 

As far as warning about the "potential biases", I do not think it is quite enough, especially because it is hard to guarantee that every potential user will read the documentation (it is also an insufficient warning.)

Note that we do have higher standards for the dataset cards of hate speech and hateful memes datasets, so if you do choose to add that one yourself we will ask that you summarize the relevant literature in the Social Impact section.

If you really need to add this dataset for your own research for the explicit purpose of studying these biases, you can add it as a community provided dataset following https://huggingface.co/docs/datasets/master/share_dataset.html#sharing-a-community-provided-dataset but I'd recommend just skipping it for now.
gchhablani (CONTRIBUTOR) commented on 2021-02-06 05:27:56:
Okay, then I won't pursue this one further. I'll keep this branch on my repository just in case the possibility of adding this dataset comes up in the future.

> So currently you do need to download the whole dataset when using it, we are working on making it easier to stream parts of it from a remote host. You can also use the filesystem integration if local storage is an issue:
> https://huggingface.co/docs/datasets/master/filesystems.html

After downloading the whole dataset (around 1.4GB), it still loads all the examples despite using `split='train[:10%]'` or `split='train[10:20]'`. 

EDIT: I think this would happen only when the examples are generated for the first time and saved to the cache. Streaming parts of the data from a remote host sounds amazing! But, would that also allow for streaming examples of the data from the local cache? (without saving all the examples the first time).

What I used:
`d = load_dataset('./datasets/celeb_a',split='train[:10]')`
Output:
`570 examples [01:33,  6.25 examples/s]` and it keeps going. 

EDIT 2: After a few thousand images, I get the following error:
```python
OSError: [Errno 24] Too many open files: '~/.cache/huggingface/datasets/celeb_a/default/1.1.0/01f9dca66039ab7c40b91b09af47a5fa8c3e49dc8d55df50da55b14116229207.incomplete'
```
I understand this is because of the way I load the images :
```python
Image.open(<path>)
```
What could be better alternative? I am only asking in case I face the same issues in the future.
gchhablani (CONTRIBUTOR) commented on 2021-02-06 05:35:28:
Hi @acul3 

Issue #1776 talks about  doing on-the-fly data pre-processing, which I think is solved in the next release as mentioned in the issue #1825. I also look forward to using this feature, though :)

I wanted to ask about on-the-fly data loading from the cache (before pre-processing).
gchhablani (CONTRIBUTOR) commented on 2021-02-06 15:28:31:
Hi @miyamonz,

I tried replicating this issue using the same snippet used by you. I am able to download the dataset without any issues, although I stopped it in the middle because the dataset is huge.

Based on a similar issue [here](https://github.com/google-research/fixmatch/issues/23), it could be related to your environment setup, although I am just guessing here. Can you share these details?
lhoestq (MEMBER) commented on 2021-02-08 11:38:43:
Hi ! Currently when you load a dataset via `load_dataset` for example, then the dataset is memory-mapped from an Arrow file on disk. Therefore there's almost no RAM usage even if your dataset contains TB of data.
Usually at training time only one batch of data at a time is loaded in memory.

Does that answer your question or were you thinking about something else ?
lhoestq (MEMBER) commented on 2021-02-08 12:00:39:
Hi ! The `dl_manager` is a `DownloadManager` object and is responsible for downloading the raw data files.
It is used by dataset builders in their `_split_generators` method to download the raw data files that are necessary to build the datasets splits.

The `Conll2003` class is a dataset builder, and so you can download all the raw data files by calling `_split_generators` with a download manager:
```python
from datasets import DownloadManager
from datasets.load import import_main_class

conll2003_builder = import_main_class(...)

dl_manager = DownloadManager()
splis_generators = conll2003_builder._split_generators(dl_manager)
```

Then you can see what files have been downloaded with
```python
dl_manager.get_recorded_sizes_checksums()
```
It returns a dictionary with the format {url: {num_bytes: int, checksum: str}}

Then you can get the actual location of the downloaded files with
```python
from datasets import cached_path

local_path_to_downloaded_file = cached_path(url)
```

------------------

Note that you can also get the urls from the Dataset object:
```python
from datasets import load_dataset

conll2003 = load_dataset("conll2003")
print(conll2003["train"].download_checksums)
```
It returns the same dictionary with the format {url: {num_bytes: int, checksum: str}}
albertvillanova (MEMBER) commented on 2021-02-08 14:24:16:
I am afraid that there is not a very straightforward way to get that location.

Another option, from _split_generators would be to use:
- `dl_manager._download_config.cache_dir` to get the directory where all the raw downloaded files are:
  ```python
  download_dir = dl_manager._download_config.cache_dir
  ```
- the function `datasets.utils.file_utils.hash_url_to_filename` to get the filenames of the raw downloaded files:
  ```python
  filenames = [hash_url_to_filename(url) for url in urls_to_download.values()]
  ```
Therefore the complete path to the raw downloaded files would be the join of both:
```python
downloaded_paths = [os.path.join(download_dir, filename) for filename in filenames]
```

Maybe it would be interesting to make these paths accessible more easily. I could work on this. What do you think, @lhoestq ?
lhoestq (MEMBER) commented on 2021-02-08 14:53:12:
Sure it would be nice to have an easier access to these paths !
The dataset builder could have a method to return those, what do you think ?
Feel free to work on this @albertvillanova , it would be a nice addition :) 

Your suggestion does work as well @albertvillanova if you complete it by specifying `etag=` to `hash_url_to_filename`.

The ETag is obtained by a HEAD request and is used to know if the file on the remote host has changed. Therefore if a file is updated on the remote host, then the hash returned by `hash_url_to_filename` is different.
patrickvonplaten (CONTRIBUTOR) commented on 2021-02-09 06:38:23:
That's awesome! Actually, I just noticed that this dataset might become a bit too big!

MuST-C is the main dataset used for IWSLT19 and should probably be added as a standalone dataset. Would you be interested also in adding `datasets/MuST-C` instead?

Description: 
_MuST-C is a multilingual speech translation corpus whose size and quality facilitates the training of end-to-end systems for speech translation from English into several languages. For each target language, MuST-C comprises several hundred hours of audio recordings from English TED Talks, which are automatically aligned at the sentence level with their manual transcriptions and translations._

Paper: https://www.aclweb.org/anthology/N19-1202.pdf

Dataset: https://ict.fbk.eu/must-c/ (One needs to fill out a short from to download the data, but it's very easy).

It would be awesome if you're interested in adding this datates. I'm very happy to guide you through the PR! I think the easiest way to start would probably be to read [this README on how to add a dataset](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md) and open a PR. Think you can copy & paste some code from:

- Librispeech_asr: https://github.com/huggingface/datasets/blob/master/datasets/librispeech_asr/librispeech_asr.py
- Flores Translation: https://github.com/huggingface/datasets/blob/master/datasets/flores/flores.py

Think all the rest can be handled on the PR :-)
albertvillanova (MEMBER) commented on 2021-02-09 07:17:02:
Thank you @lhoestq. This logging configuration makes more sense to me.

Once propagation is allowed, the end-user can customize logging behavior and add custom handlers to the proper top logger in the hierarchy.

And I also agree with following the best practices and removing any custom handlers:
- it is the end user who has to implement any custom handlers
- indeed, the previous logging problem with TensorFlow was due to the fact that absl did not follow best practices and had implemented a custom handler

Our errors/warnings will be displayed anyway, even if we do not implement any custom handler. Since Python 3.2, logging has a built-in "default" handler (logging.lastResort) with the expected default behavior (sending error/warning messages to sys.stderr), which is used only if the end user has not configured any custom handler.
lhoestq (MEMBER) commented on 2021-02-09 10:17:39:
Good catch @pjox ! I just checked and this is because the scripts doesn't handle having several blank lines in a row.
Blank lines introduced by deduplication are currently not ignored so we end up with the same number of examples in the dataset as the original version (but with empty examples...)
I fixed that in this [commit](https://github.com/huggingface/datasets/commit/837a152e4724adc5308e2c4481908c00a8d93383). I'm re-running the metadata generation for deduplicated configs.
albertvillanova (MEMBER) commented on 2021-02-10 08:28:51:
Hi @sshleifer.

I am not sure of understanding the need of the `add_item` approach...

By just reading your "Desired API" section, I would say you could (nearly) get it with a 1-column Dataset:
```python
data = {"input_ids": [np.array([4,4,2]), np.array([8,6,5,5,2]), np.array([3,3,31,5])]}
ds = Dataset.from_dict(data)
assert (ds["input_ids"][0] == np.array([4,4,2])).all()
```