t20062443/AT1ds huggingface dataset:
lhoestq (MEMBER) commented on 2020-12-23 16:06:11:
Looks like #1456 added CC100 already.
The difference with your approach is that this implementation uses the `BuilderConfig` parameters to allow the creation of custom configs for all the languages, without having to specify them in the `BUILDER_CONFIGS` class attribute.
For example even if the dataset doesn't have a config for english already, you can still load the english CC100 with
```python
from datasets import load_dataset

load_dataset("cc100", lang="en")
```
zhuzilin (NONE) commented on 2020-12-24 05:11:49:
@lhoestq
I think we first need to save the transformation chain to a list in `self._fingerprint`. Then we can
- decode all the current saved datasets to see if there is already one that is equivalent to the transformation we need now.
- or, calculate all the possible hash value of the current chain for comparison so that we could continue to use hashing.
If we find one, we can adjust the list in `self._fingerprint` to it.

As for the transformation reordering rules, we can just start with some manual rules, like two sort on the same column should merge to one, filter and select can change orders.

And for encoding and decoding, we can just manually specify `sort` is 0, `shuffling` is 2 and create a base-n number or use some general algorithm like `base64.urlsafe_b64encode`.

Because we are not doing lazy evaluation now, we may not be able to normalize the transformation to its minimal form. If we want to support that, we can provde a `Sequential` api and let user input a list or transformation, so that user would not use the intermediate datasets. This would look like tf.data.Dataset.
rabeehkarimimahabadi (NONE) commented on 2020-12-24 16:40:44:
Hi,
thanks, I did remove the cache and still the same error here

```
>>> a = datasets.load_dataset("imdb", split="train")
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets
Downloading and preparing dataset imdb/plain_text (download: 80.23 MiB, generated: 127.06 MiB, post-processed: Unknown size, total: 207.28 MiB) to /idiap/temp/rkarimi/cache_home_1/datasets/imdb/plain_text/1.0.0/90099cb476936b753383ba2ae6ab2eae419b2e87f71cd5189cb9c8e5814d12a3...
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets/downloads
Traceback (most recent call last):        
  File "<stdin>", line 1, in <module>
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/load.py", line 611, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 476, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 558, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/info_utils.py", line 73, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='unsupervised', num_bytes=67125548, num_examples=50000, dataset_name='imdb'), 'recorded': SplitInfo(name='unsupervised', num_bytes=4902716, num_examples=3680, dataset_name='imdb')}]
```

datasets version
```
datasets                  1.1.2                     <pip>
tensorflow-datasets       4.1.0                     <pip>

```
ghost (NONE) commented on 2020-12-24 22:34:27:
Hi @thomwolf 
thank you, I was not aware of this, I was looking into the data viewer linked into readme page. 

This is exactly what I was looking for, but  this does not work currently, please see the attached 
I am selecting to see all nli datasets in english and it retrieves none. thanks

![5tarDHn9CP6ngeM](https://user-images.githubusercontent.com/53898419/103107612-1509aa80-4638-11eb-85b5-0c995a189969.png)
hungluumfc (NONE) commented on 2020-12-26 08:38:10:
I have same issue for other datasets (`myanmar_news` in my case).

A version of `datasets` runs correctly on my local machine (**without GPU**) which looking for the dataset at 
```
https://raw.githubusercontent.com/huggingface/datasets/master/datasets/myanmar_news/myanmar_news.py
```

Meanwhile, other version runs on Colab (**with GPU**) failed to download the dataset. It try to find the dataset at `1.1.3` instead of `master` . If I disable GPU on my Colab, the code can load the dataset without any problem.

Maybe there is some version missmatch with the GPU and CPU version of code for these datasets?
mariosasko (COLLABORATOR) commented on 2020-12-26 18:30:58:
Progress bar can be disabled like this:
```python
from datasets.utils.logging import set_verbosity_error
set_verbosity_error()
```

There is this line in `Dataset.map`:
```python
not_verbose = bool(logger.getEffectiveLevel() > WARNING)
```

So any logging level higher than `WARNING` turns off the progress bar.
amoux (NONE) commented on 2020-12-28 01:59:10:
I have encountered the same error with `v1.0.1` and `v1.0.2` on both Windows and Linux environments. However, cloning the repo and using the path to the dataset's root directory worked for me. Even after having the dataset cached - passing the path is the only way (for now) to load the dataset.

```python
from datasets import load_dataset

dataset = load_dataset("squad")                      # Works
dataset = load_dataset("code_search_net", "python")  # Error
dataset = load_dataset("covid_qa_deepset")           # Error

path = "/huggingface/datasets/datasets/{}/"
dataset = load_dataset(path.format("code_search_net"), "python")  # Works
dataset = load_dataset(path.format("covid_qa_deepset"))           # Works
```
SBrandeis (CONTRIBUTOR) commented on 2020-12-28 04:36:03:
Hi @mrm8488 and @amoux!
 The datasets you are trying to load have been added to the library during the community sprint for v2 last month. They will be available with the v2 release!
For now, there are still a couple of solutions to load the datasets:
1. As suggested by @amoux, you can clone the git repo and pass the local path to the script
2. You can also install the latest (master) version of `datasets` using pip: `pip install git+https://github.com/huggingface/datasets.git@master`
SBrandeis (CONTRIBUTOR) commented on 2020-12-28 04:40:24:
Hi @him1411, the dataset you are trying to load has been added during the community sprint and has not been released yet. It will be available with the v2 of `datasets`.
For now, you should be able to load the datasets after installing the latest (master) version of `datasets` using pip:
`pip install git+https://github.com/huggingface/datasets.git@master`
lhoestq (MEMBER) commented on 2020-12-28 13:53:48:
It looks like they're two different issues

----------

First for `myanmar_news`: 

It must come from the way you installed `datasets`.
If you install `datasets` from source, then the `myanmar_news` script will be loaded from `master`.
However if you install from `pip` it will get it using the version of the lib (here `1.1.3`) and `myanmar_news` is not available in `1.1.3`.

The difference between your GPU and CPU executions must be the environment, one seems to have installed `datasets` from source and not the other.

----------

Then for `winogrande`:

The errors says that the url https://raw.githubusercontent.com/huggingface/datasets/1.1.3/datasets/winogrande/winogrande.py is not reachable.
However it works fine on my side.

Does your machine have an internet connection ? Are connections to github blocked by some sort of proxy ?
Can you also try again in case github had issues when you tried the first time ?
lhoestq (MEMBER) commented on 2020-12-28 13:59:56:
Maybe you can use nltk's treebank detokenizer ?
```python
from nltk.tokenize.treebank import TreebankWordDetokenizer

TreebankWordDetokenizer().detokenize("it 's a charming and often affecting journey . ".split())
# "it's a charming and often affecting journey."
```
bhavitvyamalik (CONTRIBUTOR) commented on 2020-12-29 15:54:44:
Hi @eric-mitchell,
I think the issue might be that this dataset was added during the community sprint and has not been released yet. It will be available with the v2 of `datasets`.
For now, you should be able to load the datasets after installing the latest (master) version of `datasets` using pip:
`pip install git+https://github.com/huggingface/datasets.git@master`
lhoestq (MEMBER) commented on 2020-12-30 17:34:45:
> I'm getting this same issue when loading the `imdb` corpus via `dataset = load_dataset("imdb")`. When I try `ignore_verifications=True`, no examples are read into the `train` portion of the dataset.

When the checksums don't match, it may mean that the file you downloaded is corrupted. In this case you can try to load the dataset again `load_dataset("imdb", download_mode="force_redownload")`

Also I just checked on my side and it worked fine:

```python
from datasets import load_dataset

dataset = load_dataset("imdb")
print(len(dataset["train"]))
# 25000
```

Let me know if redownloading fixes your issue @EmilyAlsentzer .
If not, feel free to open a separate issue.
pjox (CONTRIBUTOR) commented on 2020-12-30 17:48:53:
That TODO is normal, I haven't touched the python script in months (I haven't had the time, sorry), but I guess @lhoestq fixed the paths if he's already working on the metadata. In any case from now on, only the unshuffled versions of OSCAR will be distributed through the hf/datasets library as in any case it is the version most people use to train language models.

If for any reason, you need the shuffled version it will always be available on the [OSCAR website](https://oscar-corpus.com).

Also future versions of OSCAR will be unshuffled only.
rabeehkarimimahabadi (NONE) commented on 2020-12-30 23:06:03:
Also, mayjor issue for me is the format issue, even if I go through changing the whole code to use load_from_disk, then if I do 

d = datasets.load_from_disk("imdb")
d = d["train"][:10] => the format of this is no more in datasets format
this is different from you call load_datasets("train[10]")

could you tell me how I can make the two datastes the same format @lhoestq
thomwolf (MEMBER) commented on 2021-01-01 21:20:05:
Currently this dataset is only available when the library is installed from source since it was added after the last release.

We pin the dataset version with the library version so that people can have a reproducible dataset and processing when pinning the library.

We'll see if we can provide access to newer datasets with a warning that they are newer than your library version, that would help in cases like yours.
SBrandeis (CONTRIBUTOR) commented on 2021-01-03 08:59:30:
Hi @koenvandenberge and @alighofrani95!
The datasets you're experiencing issues with were most likely added recently to the `datasets` library, meaning they have not been released yet. They will be released with the v2 of the library.
Meanwhile, you can still load the datasets using one of the techniques described in this issue: #1641 
Let me know if this helps!
aditya3498 (NONE) commented on 2021-01-03 12:51:37:
Hey, so i tried installing the library from source using the commands : **git clone https://github.com/huggingface/datasets**,  **cd datasets** and then **pip3 install -e .**. But i still am facing the same error that file is not found. Please advise.

The Datasets library version now is 1.1.3 by installing from source as compared to the earlier 1.0.3 that i had loaded using pip command but I am still getting same error

![Error](https://user-images.githubusercontent.com/30871963/103479005-69f3b080-4df0-11eb-83ae-58d7bb56a90e.png)
bhavitvyamalik (CONTRIBUTOR) commented on 2021-01-04 07:07:55:
Hi @KennethEnevoldsen ,
I think the issue might be that this dataset was added during the community sprint and has not been released yet. It will be available with the v2 of datasets.
For now, you should be able to load the datasets after installing the latest (master) version of datasets using pip:
pip install git+https://github.com/huggingface/datasets.git@master
yjernite (MEMBER) commented on 2021-01-04 09:53:32:
We currently provide a wrapper on the TensorFlow implementation: https://huggingface.co/metrics/bleurt

We have long term plans to better handle model-based metrics, but they probably won't be implemented right away

@adamwlev it would still be cool to add the BLEURT checkpoints to the transformers repo if you're interested, but that would best be discussed there :) 

closing for now
lhoestq (MEMBER) commented on 2021-01-04 09:59:50:
> `
requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='s3.amazonaws.com', port=443): Max retries exceeded with url: /datasets.huggingface.co/datasets/datasets/glue/glue.py (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7ff6d6c60a20>, 'Connection to s3.amazonaws.com timed out. (connect timeout=10)'))`

Do you have an internet connection on the machine ? Is there a proxy that might block requests to aws ?

> I tried to do read the data, save it to a path and then set HF_HOME, which does not work and this is still not reading from the old set path, could you assist me how to save the datasets in a path, and let dataset library read from this path to avoid connection issue. thanks

HF_HOME is used to specify the directory for the cache files of this library.
You can use save_to_disk and load_from_disk without changing the HF_HOME:
```python
imdb = datasets.load_dataset("imdb")
imdb.save_to_disk("/idiap/temp/rkarimi/hf_datasets/imdb")
imdb = datasets.load_from_disk("/idiap/temp/rkarimi/hf_datasets/imdb")
```

> could you tell me how I can make the two datastes the same format

Indeed they returns different things:
- `load_dataset` returns a `Dataset` object if the split is specified, or a `DatasetDict` if no split is given. Therefore `load_datasets("imdb", split="train[10]")` returns a `Dataset` object containing 10 elements.
- doing `d["train"][:10]` on a DatasetDict "d" gets the train split `d["train"]` as a `Dataset` object and then gets the first 10 elements as a dictionary
lhoestq (MEMBER) commented on 2021-01-04 13:49:07:
Hi !
The arrow file size is due to the embeddings. Indeed if they're stored as float32 then the total size of the embeddings is

20 000 000 vectors * 768 dimensions * 4 bytes per dimension ~= 60GB

If you want to reduce the size you can consider using quantization for example, or maybe using dimension reduction techniques.
skyprince999 (CONTRIBUTOR) commented on 2021-01-04 15:55:33:
> Nice thank you ! Actually as it is a translation dataset we should probably have one configuration = one language pair no ?
> 
> Could you use the same trick for this dataset ?

I was looking for this input, infact I had written a long post on the Slack channel,...(_but unfortunately due to the holidays didn;t get a respones_). Initially I had tried with language pairs and then with specific language configs. 

I'll have a look at the `opus-gnomes` dataset
lhoestq (MEMBER) commented on 2021-01-04 18:41:38:
cc100 was added recently, that's why it wasn't available yet.

To load it you can just update `datasets`
```
pip install --upgrade datasets
```

and then you can load `cc100` with

```python
from datasets import load_dataset

lang = "en"
dataset = load_dataset("cc100", lang=lang, split="train")
```
lhoestq (MEMBER) commented on 2021-01-04 18:43:06:
The `dane` dataset was added recently, that's why it wasn't available yet. We did an intermediate release today just before the v2.0.

To load it you can just update `datasets`
```
pip install --upgrade datasets
```

and then you can load `dane` with

```python
from datasets import load_dataset

dataset = load_dataset("dane")
```
lhoestq (MEMBER) commented on 2021-01-04 18:48:35:
Muchocine was added recently, that's why it wasn't available yet.

To load it you can just update `datasets`
```
pip install --upgrade datasets
```

and then you can load `muchocine` with

```python
from datasets import load_dataset

dataset = load_dataset("muchocine", split="train")
```
lhoestq (MEMBER) commented on 2021-01-04 18:50:58:
`ade_corpus_v2` was added recently, that's why it wasn't available yet.

To load it you can just update `datasets`
```
pip install --upgrade datasets
```

and then you can load `ade_corpus_v2` with

```python
from datasets import load_dataset

dataset = load_dataset("ade_corpus_v2", "Ade_corpos_v2_drug_ade_relation")
```

(looks like there is a typo in the configuration name, we'll fix it for the v2.0 release of `datasets` soon)
lhoestq (MEMBER) commented on 2021-01-04 18:55:25:
Looks like the mapping function returns a dictionary with a 768-dim array in the `embeddings` field. Since the map is batched, we actually expect the `embeddings` field to be an array of shape (batch_size, 768) to have one embedding per example in the batch.

To fix that can you try to remove one of the `[0]` ? In my opinion you only need one of them, not two.
trisongz (NONE) commented on 2021-01-04 22:26:42:
The pile can very easily be added and adapted using this [tfds implementation](https://github.com/EleutherAI/The-Pile/blob/master/the_pile/tfds_pile.py) from the repo. 

However, the question is whether you'd be ok with 800GB+ cached in your local disk, since the tfds implementation was designed to offload the storage to Google Cloud Storage.
CharleoY (NONE) commented on 2021-01-04 23:39:10:
> Hey guys,
> 
> I was trying to create a new bert model from scratch via _huggingface transformers + tokenizers + dataets_ (actually using this example script by your team: https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py). It was supposed to be a first test with a small 5 GB raw text file but I can't even end the preprocessing handled by datasets because this tiny 5 GB text file becomes more than 1 TB when processing. My system was running out of space and crashed prematurely.
> 
> I've done training from scratch via Google's bert repo in the past and I can remember that the resulting pretraining data can become quite big. But 5 GB becoming 1 TB was never the case. Is this considered normal or is it a bug?
> 
> I've used the following CMD:
> `python xla_spawn.py --num_cores=8 run_mlm.py --model_type bert --config_name config.json --tokenizer_name tokenizer.json --train_file dataset_full.txt --do_train --output_dir out --max_steps 500000 --save_steps 2500 --save_total_limit 2 --prediction_loss_only --line_by_line --max_seq_length 128 --pad_to_max_length --preprocessing_num_workers 16 --per_device_train_batch_size 128 --overwrite_output_dir --debug`

It's actually because of the parameter 'preprocessing_num_worker' when using TPU. 
I am also planning to have my model trained on the google TPU with a 11gb text corpus. With x8 cores enabled, each TPU core has its own dataset.  When not using distributed training, the preprocessed file is about 77gb. On the opposite, if enable xla, the file produced will easily consume all my free space(more than 220gb, I think it will be, in the end, around 600gb ). 
So I think that's maybe where the problem came from. 

Is there any possibility that all of the cores share the same preprocess dataset?

@sgugger @RammMaschine
lhoestq (MEMBER) commented on 2021-01-05 10:17:52:
Looks like the wikipedia dump for hindi at the date of 05/05/2020 is not available anymore.
You can try to load a more recent version of wikipedia
```python
from datasets import load_dataset

d = load_dataset("wikipedia", language="hi", date="20210101", split="train", beam_runner="DirectRunner")
```
bhavitvyamalik (CONTRIBUTOR) commented on 2021-01-05 17:45:22:
@gmihaila, instead of using the current repo you should look into [this](https://github.com/cgpotts/swda). You can use the `csv` files uploaded in this repo (`swda.zip`) to access other fields and include them in this dataset. It has one dependency too, `swda.py`, you can download that separately and include it in your dataset's folder to be imported while reading the `csv` files.

Almost all the attributes of `Transcript` and `Utterance` objects are of the type str, int, or list. As far as `trees` attribute is concerned in utterance object you can simply parse it as string and user can maybe later convert it to nltk.tree object
gmihaila (CONTRIBUTOR) commented on 2021-01-06 13:55:47:
@bhavitvyamalik Thank you for the clarification! 

I didn't use [that](https://github.com/cgpotts/swda) because it doesn't have the splits. I think in combination with [what I used](https://github.com/NathanDuran/Switchboard-Corpus) would help.

Let me know if I can help! I can make those changes if you don't have the time.
lhoestq (MEMBER) commented on 2021-01-06 14:11:05:
Ok I just launched the computation of the dataset_infos.json again by grouping lines in paragraphs.
The new _generate_examples is
```python
    def _generate_examples(self, filepaths):
        """This function returns the examples in the raw (text) form."""
        id_ = 0
        current_lines = []
        for filepath in filepaths:
            logging.info("generating examples from = %s", filepath)
            with gzip.open(filepath, "rt") as f:
                for line in f:
                    if len(line.strip()) > 0:
                        current_lines.append(line)
                    else:
                        feature = id_, {"id": id_, "text": "".join(current_lines)}
                        yield feature
                        id_ += 1
                        current_lines = []
                # last paragraph
                if current_lines:
                    feature = id_, {"id": id_, "text": "".join(current_lines)}
                    yield feature
```
lhoestq (MEMBER) commented on 2021-01-06 14:46:00:
They are raw in the sense that nothing is changed from the raw file for each paragraph.
You can split sentences on new lines `\n` for example.

The first example for the unshuffled deduplicated english is going to be 
> Mtendere Village was inspired by the vision of Chief Napoleon Dzombe, which he shared with John Blanchard during his first visit to Malawi. Chief Napoleon conveyed the desperate need for a program to intervene and care for the orphans and vulnerable children (OVC) in Malawi, and John committed to help.
> Established in honor of John & Lindy’s son, Christopher Blanchard, this particular program is very dear to the Blanchard family. Dana Blanchard, or Mama Dana as she is more commonly referred to at Mtendere, lived on site during the initial development, and she returns each summer to spend the season with her Malawian family. The heart of the program is to be His hands and feet by caring for the children at Mtendere, and meeting their spiritual, physical, academic, and emotional needs.
> [...]
> 100X Development Foundation, Inc. is registered 503 (c)(3) nonprofit organization. Donations are deductable to the full extent allowable under IRS regulations.
huu4ontocord (CONTRIBUTOR) commented on 2021-01-07 03:38:47:
hi @lhoestq, I wanted to push another change to this branch b/c I found a bug in the parsing. I need to swap arg1 and arg2. I tried to git push -u origin nell but it didn't work. So I tried to do git push --force -u origin nell which seems to work, but nothing is happening to this branch. I think this is because it's closed. Do I need to open another PR?

The change should be below in _generate_examples:
                    best_arg1 = row[9]
                    best_arg2 = row[8]
bhavitvyamalik (CONTRIBUTOR) commented on 2021-01-07 09:42:11:
I'm a bit busy for the next 2 weeks. I'll be able to complete it by end of January only. Maybe you can start with it and I'll help you?
Also, I looked into the official train/val/test splits and not all the files are there in the repo I used so I think either we'll have to skip them or put all of that into just train