t20062443/AT1ds huggingface dataset:
abarbosa94 (CONTRIBUTOR) commented on 2020-08-31 19:57:06:
I just cleared the cache an run it again. The error persists ):

```
 nlp (master) $ rm -rf  /Users/abarbosa/.cache/huggingface/
 nlp (master) $ python
Python 3.8.5 (default, Aug  5 2020, 03:39:04)
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import nlp
>>> dataset = nlp.load_dataset("race")
Downloading: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4.39k/4.39k [00:00<00:00, 661kB/s]
Downloading: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.81k/1.81k [00:00<00:00, 644kB/s]
Using custom data configuration default
Downloading and preparing dataset race/default (download: 84.52 MiB, generated: 132.61 MiB, post-processed: Unknown size, total: 217.13 MiB) to /Users/abarbosa/.cache/huggingface/datasets/race/default/0.1.0/5461327f1a83549ca0d845a3159c806d2baf4f8d0d8f7d657157ce7cdf3899c2...
Downloading: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 25.4M/25.4M [01:03<00:00, 401kB/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/abarbosa/Documents/nlp/src/nlp/load.py", line 550, in load_dataset
    builder_instance.download_and_prepare(
  File "/Users/abarbosa/Documents/nlp/src/nlp/builder.py", line 471, in download_and_prepare
    self._download_and_prepare(
  File "/Users/abarbosa/Documents/nlp/src/nlp/builder.py", line 530, in _download_and_prepare
    verify_checksums(
  File "/Users/abarbosa/Documents/nlp/src/nlp/utils/info_utils.py", line 38, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
nlp.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['http://www.cs.cmu.edu/~glai1/data/race/RACE.tar.gz']
>>>
```
abarbosa94 (CONTRIBUTOR) commented on 2020-08-31 20:00:45:
This has fixed #537 at least on my machine hahaha.

Nice point! I think it would totally worth it :) What the best implementation approach would you suggest?

Would it be possible to have `high school`, `middle` and  `all` inside each portion of `train`, `validation` and `test`? Would this make sense?
thomwolf (MEMBER) commented on 2020-09-01 09:24:06:
@agemagician you can give a try at a multithreaded version if you want (currently on the #548).

To test it, you just need to copy the new `text` processing script which is [here](https://github.com/huggingface/nlp/blob/07d92a82b7594498ff702f3cca55c074e2052257/datasets/text/text.py) somewhere on your drive and give it's local path instead of `text` to `load_dataset`. E.g. in your example:
```python
train_files = glob.glob("xxx/*.txt",recursive=True)
random.shuffle(train_files)

print(train_files)

dataset = nlp.load_dataset('./datasets/text.py',   # path to where you've dowloaded the multi-threaded text loading script
                           data_files=train_files,
                           name="customDataset",
                           version="1.0.0",
                           cache_dir="xxx/nlp")
```
agemagician (NONE) commented on 2020-09-01 10:39:04:
I have already generated the dataset, but now I tried to reload it and it is still very slow.

I also have installed your commit and it is slow, even after the dataset was already generated.
`pip install git+https://github.com/huggingface/nlp.git@07d92a82b7594498ff702f3cca55c074e2052257`

It uses only a single thread.

Did I miss something ?
thomwolf (MEMBER) commented on 2020-09-01 11:38:39:
Also, @agemagician you have to follow the step I indicate in my previous message [here](https://github.com/huggingface/nlp/issues/546#issuecomment-684648927) to use the new text loading script.

Just doing `pip install git+https://github.com/huggingface/nlp.git@07d92a82b7594498ff702f3cca55c074e2052257` like you did won't use the new script (they are not inside the library but hosted on our hub).
go-inoue (NONE) commented on 2020-09-01 12:14:56:
I ended up specifying the `cache_file_name` argument when I call `map` function.

```python
dataset = dataset.map(lambda ex: tokenizer(ex["text"], add_special_tokens=True, truncation=True, max_length=args.block_size),
                      batched=True,
                      cache_file_name=cache_file_name)
```

Note:
- `text` dataset in `nlp` does not strip `"\n"`.  If you want the same output as in [`LineByLineTextDataset`](https://github.com/huggingface/transformers/blob/afc4ece462ad83a090af620ff4da099a0272e171/src/transformers/data/datasets/language_modeling.py#L88-L111), you would need to create your own dataset class where you replace `line` to `line.strip()` [here](https://github.com/huggingface/nlp/blob/master/datasets/text/text.py#L35).
orsharir (CONTRIBUTOR) commented on 2020-09-01 13:11:19:
I'm afraid that I don't know how to obtain the original BookCorpus data. I believe this version came from an anonymous Google Drive link posted in another issue.

Going through the raw text in this version, it's apparent that NLTK's TreebankWordTokenizer was applied on it (I gave some examples in my original post), followed by:
`' '.join(tokens)`
You can retrieve the tokenization by splitting on whitespace. You can then "detokenize" it with TreebankWordDetokenizer class of NLTK (though, as I suggested, use the fixed version in my repo). This will bring the text closer to its original form, but some steps of TreebankWordTokenizer are destructive, so it wouldn't be one-to-one. Something along the lines of the following should work:
```
treebank_detokenizer = nltk.tokenize.treebank.TreebankWordDetokenizer()
db = nlp.load_dataset('bookcorpus', split=nlp.Split.TRAIN)
db = db.map(lambda x: treebank_detokenizer.detokenize(x['text'].split()))
```

Regarding other issues beyond the above, I'm afraid that I can't help with that.
danieldk (MEMBER) commented on 2020-09-01 14:24:19:
> If it's fine for you then this is the recommended way to solve this issue.

I don't want to use a virtual environment, because Nix is fully reproducible, and virtual environments are not. And I am the maintainer of the `transformers` in nixpkgs, so sooner or later I will have to package `nlp`, since it is becoming a dependency of `transformers` ;).
ynouri (NONE) commented on 2020-09-01 22:32:46:
From my limited understanding, part of the issue seems related to the `prepare_module`  and `download_and_prepare` functions each handling the case where no config is passed. For example, `prepare_module` does mutate the object passed and forces the flags `extract_compressed_file` and `force_extract` to `True`.

See:
* https://github.com/huggingface/nlp/blob/5fb61e1012bda724a9b6b847307d90a1380abfa5/src/nlp/load.py#L227
* https://github.com/huggingface/nlp/blob/5fb61e1012bda724a9b6b847307d90a1380abfa5/src/nlp/builder.py#L388

Maybe a cleaner solution would be to always instantiate a default `DownloadConfig` object at the top-level, have it as non-optional for the lower-level functions and treat it as immutable.
thomwolf (MEMBER) commented on 2020-09-02 08:01:45:
Ok interesting. We could have another check to see if it's possible to download and import the datasets script at another location than the module path. I think this would probably involve tweaking the python system path dynamically.

I don't know anything about Nix so if you want to give this a try your self we can guide you or you can give us more information on your general project and how this works.

Regarding `nlp` and `transformers`, we are not sure `nlp` will become a required dependency for `transformers`.  It will probably be used a lot in the examples but I think it probably won't be a required dependency for the main package since we try to keep it as light as possible in terms of deps.

Happy to help you make all these things work better for your use-case
lhoestq (MEMBER) commented on 2020-09-02 08:30:03:
Logging looks like

```
Done writing 21900 indices in 3854400 bytes .
Process #0 will write at playground/tmp_00000_of_00004.arrow
Done writing 21900 indices in 3854400 bytes .
Process #1 will write at playground/tmp_00001_of_00004.arrow
Done writing 21900 indices in 3854400 bytes .
Process #2 will write at playground/tmp_00002_of_00004.arrow
Done writing 21899 indices in 3854224 bytes .
Process #3 will write at playground/tmp_00003_of_00004.arrow
Spawning 4 processes
#3: 100%|████████████████████████████████████████████████| 21899/21899 [00:02<00:00, 8027.41ex/s]
#0: 100%|████████████████████████████████████████████████| 21900/21900 [00:02<00:00, 7982.87ex/s]
#1: 100%|████████████████████████████████████████████████| 21900/21900 [00:02<00:00, 7923.89ex/s]
#2: 100%|████████████████████████████████████████████████| 21900/21900 [00:02<00:00, 7920.04ex/s]
Concatenating 4 shards from multiprocessing
```
thomwolf (MEMBER) commented on 2020-09-02 13:08:36:
I agree this fix the problem for the CI where the files are always created in a new and clean temporary directory.

However, in a general setting of a succession of fast distributed operation, the files could already exist from previous metrics runs but one process may still finish before another has even started in which case it would mix results from separate operations.

I feel like the most robust way to solve this is to setup a rendez-vous on the first time we write on files and where each process will test and only finish its operation when it cannot acquire a lock on all the other processes (meaning they all have started).

What do you think?
ynouri (NONE) commented on 2020-09-02 13:28:20:
Thanks. Our use case involves running a training job behind a corporate firewall with no access to any external resources (S3, GCP or other web resources).

I was thinking about a 2-steps process:
1) Download the resources / artifacts using some secure corporate channel, ie run `nlp.load_dataset()` without a specific `DownloadConfig`. After that, collect the files from the `$HF_HOME` folder
2) Copy the `$HF_HOME` folder in the firewalled environment. Run `nlp.load_dataset()` with a custom config `DownloadConfig(local_files_only=True)`

However this ends up a bit clunky in practice, even when solving the `DownloadConfig` issue above. For example, the `filename` hash computed in `get_from_cache()` differs in the `local_files_only=False` vs `local_files_only=True` case (local case defaults `etag` to `None`, which results in a different hash). So effectively step 2) above doesn't work because the hash computed differs from the hash in the cache folder. Some hacks / workaround are possible but this solution becomes very convoluted.
https://github.com/huggingface/nlp/blob/c214aa5a4430c1df1bcd0619fd94d6abdf9d2da7/src/nlp/utils/file_utils.py#L417

Would you recommend a different path?
lhoestq (MEMBER) commented on 2020-09-02 16:41:22:
Thanks for reporting.

Apparently this is a versioning issue: the lib downloaded the `bleurt` script from the master branch where we did this change recently. We'll fix that in a new release this week or early next week. Cc @thomwolf 

Until that, I'd suggest you to download the right bleurt folder from github ([this one](https://github.com/huggingface/nlp/tree/0.4.0/metrics/bleurt)) and do

```python
from nlp import load_metric

bleurt = load_metric("path/to/bleurt/folder")
```

To download it you can either clone the repo or download the `bleurt.py` file and place it in a folder named `bleurt`
julien-c (MEMBER) commented on 2020-09-03 10:00:21:
Hi @shahbazsyed, this dataset _is_ inside the hub but it's namespaced by the organization name `webis`.

You can load it following the steps described in https://huggingface.co/datasets/webis/tl_dr

Here's a Colab showcasing that it works: https://colab.research.google.com/drive/11IrzRVpnMLJZ8_UFFHLR8FhiajjAHRUU?usp=sharing

The reason the code is in S3 and not in this repo is that the dataset is namespaced under the `webis` organization. We don't have a lot of namespaced datasets yet but this should become the main way we add more datasets in the future.
Let us know if that's an issue for you. Thank you!
ghomasHudson (CONTRIBUTOR) commented on 2020-09-03 20:33:31:
> Hi @ghomasHudson, did you get the chance to add the test split and regenerate the dataset_infos.json file ?

Have added the test set code but getting an OverflowError when trying to regen the dataset_infos.json:

---
OverflowError: There was an overflow in the <class 'pyarrow.lib.StructArray'>. Try to reduce writer_batch_size to have batches smaller than 2GB

---
lhoestq (MEMBER) commented on 2020-09-04 08:34:49:
I've added save/load for dataset dicts.

I agree that in the future we should also have a way to save indexes too, and also the in-place history of transforms.

Also I understand that it would be cool to have the load function directly at the root of the library, but I'm not sure this should be inside `load_dataset` that loads dataset scripts and data from the dataset repository. Maybe something like `load_from_disk` ?
JetRunner (CONTRIBUTOR) commented on 2020-09-04 16:29:09:
Thanks, @lhoestq! I've addressed the comments. 
Also, I have tried to use `ClassLabel` [when possible](https://github.com/huggingface/nlp/pull/572/files#diff-1026ac7d7b78bf029cb0ebe63162c77dR297). Is there still somewhere else we can use `ClassLabel`?
orsharir (CONTRIBUTOR) commented on 2020-09-05 12:32:13:
At least for me, the file that was downloaded from CMU isn't the complete dataset, but a small subset of it (~25MB vs ~85MB). I've previously downloaded the dataset directly, so for my personal needs I could just swap out the corrupted file with the correct one. Perhaps you could host it like you do for the Wikipedia and BookCorpus datasets.
kshre (NONE) commented on 2020-09-07 00:14:10:
> At least for me, the file that was downloaded from CMU isn't the complete dataset, but a small subset of it (~25MB vs ~85MB). I've previously downloaded the dataset directly, so for my personal needs I could just swap out the corrupted file with the correct one. Perhaps you could host it like you do for the Wikipedia and BookCorpus datasets.

Could you upload this please?
lhoestq (MEMBER) commented on 2020-09-07 16:35:40:
I think we could have one dataset configuration for `high school`, one for `middle` and one for `all`.
You just need to add
```python
    BUILDER_CONFIGS = [
        nlp.BuilderConfig(
            name="high school",
            description="insert description here",
        ),
        nlp.BuilderConfig(
            name="middle",
            description="insert description here",
        ),
        nlp.BuilderConfig(
            name="all",
            description="insert description here",
        ),
    ]
```
as a class attribute for the `Race` class.

Then in `generate_examples` you can check the value of `self.config.name` and choose which files to include when generating examples.

You can check [mlsum](https://github.com/huggingface/nlp/blob/master/datasets/mlsum/mlsum.py) for example if you want to see how it done in general, it's a dataset that has five configurations, and each config has train/val/test splits.
lhoestq (MEMBER) commented on 2020-09-07 16:52:21:
Some wikipedia languages have already been processed by us and are hosted on our google storage. This is the case for "fr" and "en" for example.

For other smaller languages (in terms of bytes), they are directly downloaded and parsed from the wikipedia dump site.
Parsing can take some time for languages with hundreds of MB of xml.

Let me know if you encounter an error or if you feel that is is taking too long for you.
We could process those that really take too much time
lhoestq (MEMBER) commented on 2020-09-08 09:15:46:
Thanks ! This will be very helpful.

About the date issue, I think it's possible to use another date with

```python
load_dataset("wikipedia", language="es", date="...", beam_runner="...")
```

However we've not processed wikipedia dumps for other dates than 20200501 (yet ?)

One more thing that is specific to 20200501.es: it was available once but the `mwparserfromhell` was not able to parse it for some reason, so we didn't manage to get a processed version of 20200501.es (see #321 )
lhoestq (MEMBER) commented on 2020-09-08 09:30:27:
@danieldk modules are now installed in a different location (by default in the cache directory of the lib, in `~/.cache/huggingface/modules`). You can also change that using the environment variable `HF_MODULES_PATH`

Feel free to play with this change from the master branch for now, and let us know if it sounds good for you :)
We plan to do a release in the next coming days
orsharir (CONTRIBUTOR) commented on 2020-09-08 14:18:31:
The TreebankTokenizer makes some assumptions about whitespace, parentheses, quotation marks, etc. For instance, while tokenizing the following text:
```
Dwayne "The Rock" Johnson
```
will result in:
```
Dwayne `` The Rock '' Johnson
```
where the left and right quotation marks are turned into distinct symbols. Upon reconstruction, we can attach the left part to its token on the right, and respectively for the right part. However, the following texts would be tokenized exactly the same:
```
Dwayne " The Rock " Johnson
Dwayne " The Rock" Johnson
Dwayne     " The Rock" Johnson
...
```
In the above examples, the detokenizer would correct these inputs into the canonical text
```
Dwayne "The Rock" Johnson
```
However, there are cases where there the solution cannot easily be inferred (at least without a true LM - this tokenizer is just a bunch of regexes). For instance, in cases where you have a fragment that contains the end of quote, but not its beginning, plus an accidental space:
```
... and it sounds fantastic, " he said.
```
In the above case, the tokenizer would assume that the quotes refer to the next token, and so upon detokenization it will result in the following mistake:
```
... and it sounds fantastic, "he said.
```

While these are all odd edge cases (the basic assumptions do make sense), in noisy data they can occur, which is why I mentioned that the detokenizer cannot restore the original perfectly.
orsharir (CONTRIBUTOR) commented on 2020-09-08 14:28:57:
> > At least for me, the file that was downloaded from CMU isn't the complete dataset, but a small subset of it (~25MB vs ~85MB). I've previously downloaded the dataset directly, so for my personal needs I could just swap out the corrupted file with the correct one. Perhaps you could host it like you do for the Wikipedia and BookCorpus datasets.
> 
> Could you upload this please?

Not sure if I can upload it according to their license ("You agree not to reproduce, duplicate, copy, sell, trade, resell or exploit for any commercial purpose, any portion of the contexts and any portion of derived data.").
lhoestq (MEMBER) commented on 2020-09-09 13:55:03:
Yes if the file changed, then the checksum verification won't pass as it expects to see the checksum of the old file.
The checksum is computed by hashing the complete file.
You can update the checksum by doing 

```
nlp-cli test ./datasets/compguesswhat --save_infos --all_configs
```
lhoestq (MEMBER) commented on 2020-09-09 17:18:59:
It looks like it's an issue with Pyarrow.
By changing the `DEFAULT_MAX_BATCH_SIZE` to 1000 instead of 10 000 in `arrow_writer.py` I was able to run the command.

Basically it seems that is an Arrow StructArray has more than 1-2GB of data, then it shuffles some of its content.
I can't find any issue on Apache Arrow's JIRA about this problem. It will require more investigation.

Maybe we can simply automatically decrease the writer's batch size when this happens. We can just check if the arrow array is more than a certain amount of bytes.
thomwolf (MEMBER) commented on 2020-09-10 07:07:28:
Still the case on master.
I guess we should have an offset in the multi-procs indeed (hopefully it's enough).

Also, side note is that we should add some logging before the "test" to say we are testing the function otherwise its confusing for the user to see two outputs I think. Proposal (see the "Testing the mapped function outputs:" lines):
```
>>> d.select(range(10)).map(fn, with_indices=True, batched=True, num_proc=2)
Done writing 10 indices in 80 bytes .
Done writing 5 indices in 41 bytes .
Done writing 5 indices in 41 bytes .
Spawning 2 processes
Testing the mapped function outputs:
inds: [0, 1]
inds: [0, 1]
Testing finished, running the mapped function on the dataset:
#0:   0%|                                                                                                                                                                    | 0/1 [00:00<?, ?ba/s]
inds: [0, 1, 2, 3, 4]                                                                                                                                                                             inds: [0, 1, 2, 3, 4]                                                                                                                                                         | 0/1 [00:00<?, ?ba/s]
#0: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1321.04ba/s]
#1: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1841.22ba/s]
Concatenating 2 shards from multiprocessing
Dataset(features: {'text': Value(dtype='string', id=None), 'label': ClassLabel(num_classes=2, names=['neg', 'pos'], names_file=None, id=None)}, num_rows: 10)
```
jeswan (CONTRIBUTOR) commented on 2020-09-10 16:46:40:
I am also seeing a similar error when running the following:

```
import nlp
dataset = load_dataset('cola')
```
Error:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/js11133/.conda/envs/jiant/lib/python3.8/site-packages/nlp/load.py", line 509, in load_dataset
    module_path = prepare_module(path, download_config=download_config, dataset=True)
  File "/home/js11133/.conda/envs/jiant/lib/python3.8/site-packages/nlp/load.py", line 248, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "/home/js11133/.conda/envs/jiant/lib/python3.8/site-packages/nlp/utils/file_utils.py", line 191, in cached_path
    output_path = get_from_cache(
  File "/home/js11133/.conda/envs/jiant/lib/python3.8/site-packages/nlp/utils/file_utils.py", line 356, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/cola/cola.py
```
sangyx (NONE) commented on 2020-09-11 08:34:12:
```
<class 'pandas.core.frame.DataFrame'>
Int64Index: 17136104 entries, 0 to 17136103
Data columns (total 6 columns):
 #   Column        Dtype  
---  ------        -----  
 0   item_id       int64  
 1   item_titl     object 
 2   start_price   float64
 3   shipping_fee  float64
 4   picture_url   object 
 5   embeddings    object 
dtypes: float64(2), int64(1), object(3)
memory usage: 915.2+ MB
```
sangyx (NONE) commented on 2020-09-11 08:41:00:
`embedding` is `np.array` of shape `(128,)`. `picture_url` is url, such as 'https://i.ebayimg.com/00/s/MTE5OVgxNjAw/z/ZOsAAOSwAG9fHQq5/$_12.JPG?set_id=880000500F;https://i.ebayimg.com/00/s/MTE5OVgxNjAw/z/OSgAAOSwokBfHQq8/$_12.JPG?set_id=880000500F'