t20062443/AT1ds huggingface dataset:
cstorm125 (CONTRIBUTOR) commented on 2021-01-07 14:26:36:
> Thanks :)
> 
> Apparently the dummy_data.zip got removed. Is this expected ?
> Also can you remove the `data-pos.conll` file that you added ?

Not expected. I forgot to remove the `dummy_data` folder used to create `dummy_data.zip`. 
Changed to only `dummy_data.zip`.
aclifton314 (CONTRIBUTOR) commented on 2021-01-07 17:51:29:
I'm getting an error when I try to create the dummy data:
```python
aclifton@pop-os:~/data/hf_datasets_sprint/datasets$ python datasets-cli dummy_data ./datasets/logiqa_en/ --auto_generate 
2021-01-07 10:50:12.024791: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2021-01-07 10:50:12.024814: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Using custom data configuration default
Couldn't generate dummy file 'datasets/dummy/1.1.0/dummy_data/master.zip/LogiQA-dataset-master/README.md'. Ignore that if this file is not useful for dummy data.
Dummy data generation done but dummy data test failed since splits ['train', 'test', 'validation'] have 0 examples for config 'default''.
Automatic dummy data generation failed for some configs of './datasets/logiqa_en/'
```
wn1652400018 (NONE) commented on 2021-01-08 07:09:57:
> 
> 
> > hello, how did you solve this problems?
> > > > > Hi !
> > > > > The url works on my side.
> > > > > Is the url working in your navigator ?
> > > > > Are you connected to internet ? Does your network block access to `raw.githubusercontent.com` ?
> > > > 
> > > > 
> > > > I tried another server, it's working now. Thanks a lot.
> > > > And I'm curious about why download things from "github" when I load dataset from local files ? Dose datasets work if my network crashed?
> > > 
> > > 
> > > I download the scripts `https://raw.githubusercontent.com/huggingface/datasets/1.1.2/datasets/csv/csv.py` and move it to the package dir `*/datasets/` solved the problem. Could you please put the file `datasets/datasets/csv/csv.py` to `datasets/src/datasets/`？
> > > Thanks :D
> > 
> > 
> > hello, I tried this. but it still failed. how do you fix this error?
> 
> 你把那个脚本下载到你本地安装目录下，然后 `load_dataset(csv_script_path, data_fiels)`

我照着做了，然后报错。
ValueError: unable to parse C:/Software/Anaconda/envs/ptk_gpu2/Lib/site-packages/datasets\dataset_infos.json as a URL or as a local path

`---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-5-fd2106a3f053> in <module>
----> 1 dataset = load_dataset('C:/Software/Anaconda/envs/ptk_gpu2/Lib/site-packages/datasets/csv.py', data_files='./test.csv', delimiter=',', autogenerate_column_names=False)

C:\Software\Anaconda\envs\ptk_gpu2\lib\site-packages\datasets\load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, save_infos, script_version, **config_kwargs)
    588     # Download/copy dataset processing script
    589     module_path, hash = prepare_module(
--> 590         path, script_version=script_version, download_config=download_config, download_mode=download_mode, dataset=True
    591     )
    592 

C:\Software\Anaconda\envs\ptk_gpu2\lib\site-packages\datasets\load.py in prepare_module(path, script_version, download_config, download_mode, dataset, force_local_path, **download_kwargs)
    296         local_dataset_infos_path = cached_path(
    297             dataset_infos,
--> 298             download_config=download_config,
    299         )
    300     except (FileNotFoundError, ConnectionError):

C:\Software\Anaconda\envs\ptk_gpu2\lib\site-packages\datasets\utils\file_utils.py in cached_path(url_or_filename, download_config, **download_kwargs)
    316     else:
    317         # Something unknown
--> 318         raise ValueError("unable to parse {} as a URL or as a local path".format(url_or_filename))
    319 
    320     if download_config.extract_compressed_file and output_path is not None:

ValueError: unable to parse C:/Software/Anaconda/envs/ptk_gpu2/Lib/site-packages/datasets\dataset_infos.json as a URL or as a local path

`
thomwolf (MEMBER) commented on 2021-01-08 09:37:06:
I few comments here for discussion:
- I'm not convinced yet the end user should really have to understand the difference between "caching" and 'fingerprinting", what do you think? I think fingerprinting should probably stay as an internal thing. Is there a case where we want cahing without fingerprinting or vice-versa?
- while I think the random fingerprint mechanism is smart, I have one question: when we disable caching or fingerprinting we also probably don't want the disk usage to grow so we should then try to keep only one cache file. Is it the case currently?
- the warning should be emitted only once per session if possible (we have a mechanism to do that in transformers, you should ask Lysandre/Sylvain)
lhoestq (MEMBER) commented on 2021-01-08 11:06:19:
About your points:
- Yes I agree, I just wanted to bring the discussion on this point. Until now fingerprinting hasn't been blocking for user experience. I'll probably remove the enable/disable fingerprinting function to keep things simple from the user's perspective.
- Right now every time a not in-place transform (i.e. map, filter) is applied, a new cache file is created. It is the case even if caching is disabled since disabling it only means that the cache file won't be reloaded. Therefore you're right that it might end up filling the disk with files that won't be reused. I like the idea of keeping only one cache file. Currently all the cache files are kept on disk until the user clears the cache. To be able to keep only one, we need to know if a dataset that has been transformed is still loaded or not. For example
```python
# case 1 - keep both cache files (dataset1 and dataset2)
dataset2 = dataset1.map(...)
# case 2 - keep only the new cache file
dataset1 = dataset1.map(...)
```
In python it doesn't seem trivial to detect such changes. One thing that we can actually do on the other hand is store the cache files in a temporary directory that is cleared when the session closes. I think that's a good a simple solution for this problem.
- Yes good idea ! I don't like spam either :)
thomwolf (MEMBER) commented on 2021-01-08 15:04:56:
> * To be able to keep only one, we need to know if a dataset that has been transformed is still loaded or not. For example
> 
> ```python
> # case 1 - keep both cache files (dataset1 and dataset2)
> dataset2 = dataset1.map(...)
> # case 2 - keep only the new cache file
> dataset1 = dataset1.map(...)
> ```

I see what you mean. It's a tricky question. One option would be that if caching is deactivated we have a single memory mapped file and have copy act as a copy by reference instead of a copy by value. We will then probably want a `copy()` or `deepcopy()` functionality. Maybe we should think a little bit about it though.
yjernite (MEMBER) commented on 2021-01-08 16:37:27:
- I like the idea of using a temporary directory per session!
- If the default behavior when caching is disabled is to re-use the same file, I'm a little worried about people making mistakes and having to re-download and process from scratch.
- So we already have a keyword argument for `dataset1 = dataset1.map(..., in_place=True)`?
lhoestq (MEMBER) commented on 2021-01-08 17:29:50:
> * If the default behavior when caching is disabled is to re-use the same file, I'm a little worried about people making mistakes and having to re-download and process from scratch.

We should distinguish between the caching from load_dataset (base dataset cache files) and the caching after dataset transforms such as map or filter (transformed dataset cache files). When disabling caching only the second type (for map and filter) doesn't reload from cache files.
Therefore nothing is re-downloaded. To re-download the dataset entirely the argument `download_mode="force_redownload"` must be used in `load_dataset`.
Do we have to think more about the naming to make things less confusing in your opinion ?

> * So we already have a keyword argument for `dataset1 = dataset1.map(..., in_place=True)`?

There's no such `in_place` parameter in map, what do you mean exactly ?
thomwolf (MEMBER) commented on 2021-01-08 22:08:08:
Oh that's a really cool one, we'll review/merge it soon!

In the meantime, do you have any specific positive/negative feedback on the process of adding a datasets Max?
Did you follow the instruction in the [detailed step-by-step](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md)?
maxbartolo (CONTRIBUTOR) commented on 2021-01-09 00:04:50:
Thanks Thom, been a while, hope all is well!

Yes, I followed the step by step instructions and found them pretty straightforward. The only things I wasn't sure of were what should go into the YAML tags field for the dataset card, and whether there was a list of options somewhere (maybe akin to the metrics?) of the possible supported tasks. I found the rest very intuitive and the automated metadata and dummy data generation very handy. Thanks!
yjernite (MEMBER) commented on 2021-01-11 14:01:46:
@maxbartolo cool addition!

For the YAML tag, you should use the tagging app we provide to choose from a drop-down menu:
https://github.com/huggingface/datasets-tagging

The process is described toward the end of the [step-by-step guide](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#tag-the-dataset-and-write-the-dataset-card), do you have any suggestions for making it easier to find?

Otherwise, the dataset card is really cool, thanks for making it so complete!
maxbartolo (CONTRIBUTOR) commented on 2021-01-11 14:45:50:
@yjernite

Thanks, YAML tags added. I think my main issue was with the flow of the [step-by-step guide](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md). For example, the [card creator](https://huggingface.co/datasets/card-creator/) is introduced in Step 4, right after creating an empty directory for your dataset. The first field it requires are the YAML tags, which (at least for me) was the last step of the process.

I'd suggest having the guide structured in the same order as the creation process. For me it was something like:
- Step 1: Preparing your env
- Step 2: Write the loading/processing code
- Step 3: Automatically generate dummy data and `dataset_infos.json`
- Step 4: Tag the dataset
- Step 5: Write the dataset card using the [card creator](https://huggingface.co/datasets/card-creator/)
- Step 6: Open a Pull Request on the main HuggingFace repo and share your work!!

Thanks again!
Opdoop (NONE) commented on 2021-01-11 15:15:56:
I'm a little confused with the data size.
`sst2` dataset is referenced to `Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank` and the link of the dataset in the paper is https://nlp.stanford.edu/sentiment/index.html which is often shown in GLUE/SST2 reference.
From the original data, the standard train/dev/test splits split is 6920/872/1821 for binary classification. 
Why in GLUE/SST2 the train/dev/test split is 67,349/872/1,821 ?
lhoestq (MEMBER) commented on 2021-01-11 16:33:49:
Thanks for reporting !
I was able to reproduce thanks to your code and find the origin of the bug.
The cache was not reusing the same file because one object was not deterministic. It comes from a conversion from `set` to `list` in the `datasets.arrrow_dataset.transmit_format` function, where the resulting list would not always be in the same order and therefore the function that computes the hash used by the cache would not always return the same result.
I'm opening a PR to fix this.

Also we plan to do a new release in the coming days so you can expect the fix to be available soon.
Note that you can still specify `cache_file_name=` in the second `map()` call to name the cache file yourself if you want to.
ofirzaf (NONE) commented on 2021-01-11 16:59:51:
Thanks for the fast reply, waiting for the fix :)

I tried to use `cache_file_names` and wasn't sure how, I tried to give it the following:
```
tokenized_datasets = tokenized_datasets.map(
    group_texts,
    batched=True,
    num_proc=60,
    load_from_cache_file=True,
    cache_file_names={k: f'.cache/{str(k)}' for k in tokenized_datasets}
)
```

and got an error:
```
multiprocess.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/venv/lib/python3.6/site-packages/multiprocess/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/venv/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 157, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/venv/lib/python3.6/site-packages/datasets/fingerprint.py", line 163, in wrapper
    out = func(self, *args, **kwargs)
  File "/venv/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1491, in _map_single
    tmp_file = tempfile.NamedTemporaryFile("wb", dir=os.path.dirname(cache_file_name), delete=False)
  File "/usr/lib/python3.6/tempfile.py", line 690, in NamedTemporaryFile
    (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
  File "/usr/lib/python3.6/tempfile.py", line 401, in _mkstemp_inner
    fd = _os.open(file, flags, 0o600)
FileNotFoundError: [Errno 2] No such file or directory: '_00000_of_00060.cache/tmpsvszxtop'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "test.py", line 48, in <module>
    cache_file_names={k: f'.cache/{str(k)}' for k in tokenized_datasets}
  File "/venv/lib/python3.6/site-packages/datasets/dataset_dict.py", line 303, in map
    for k, dataset in self.items()
  File "/venv/lib/python3.6/site-packages/datasets/dataset_dict.py", line 303, in <dictcomp>
    for k, dataset in self.items()
  File "/venv/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1317, in map
    transformed_shards = [r.get() for r in results]
  File "/venv/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1317, in <listcomp>
    transformed_shards = [r.get() for r in results]
  File "/venv/lib/python3.6/site-packages/multiprocess/pool.py", line 644, in get
    raise self._value
FileNotFoundError: [Errno 2] No such file or directory: '_00000_of_00060.cache/tmpsvszxtop'
```
lhoestq (MEMBER) commented on 2021-01-11 17:05:32:
The documentation says
```
cache_file_names (`Optional[Dict[str, str]]`, defaults to `None`): Provide the name of a cache file to use to store the
    results of the computation instead of the automatically generated cache file name.
    You have to provide one :obj:`cache_file_name` per dataset in the dataset dictionary.
```
What is expected is simply the name of a file, not a path. The file will be located in the cache directory of the `wikitext` dataset. You can try again with something like
```python
cache_file_names = {k: f'tokenized_and_grouped_{str(k)}' for k in tokenized_datasets}
```
ZizhenWang (NONE) commented on 2021-01-12 02:46:09:
here is my way to load a dataset offline, but it **requires** an online machine
1. (online machine)
```
import datasets
data = datasets.load_dataset(...)
data.save_to_disk(/YOUR/DATASET/DIR)
```
2. copy the dir from online to the offline machine
3. (offline machine)
```
import datasets
data = datasets.load_from_disk(/SAVED/DATA/DIR)
```

HTH.
aballigier (NONE) commented on 2021-01-12 02:47:42:
> here is my way to load a dataset offline, but it **requires** an online machine
> 
> 1. (online machine)
> 
> ```
> 
> import datasets
> 
> data = datasets.load_dataset(...)
> 
> data.save_to_disk(/YOUR/DATASET/DIR)
> 
> ```
> 
> 2. copy the dir from online to the offline machine
> 
> 3. (offline machine)
> 
> ```
> 
> import datasets
> 
> data = datasets.load_from_disk(/SAVED/DATA/DIR)
> 
> ```
> 
> 
> 
> HTH.
SBrandeis (CONTRIBUTOR) commented on 2021-01-12 10:15:48:
Hi @lkcao !
Your issue is indeed related to `datasets`. In addition to installing the package manually, you will need to download the `text.py` script on your server. You'll find it (under `datasets/datasets/text`: https://github.com/huggingface/datasets/blob/master/datasets/text/text.py.
Then you can change the line 221 of `run_mlm_new.py` into:
```python
  datasets = load_dataset('/path/to/text.py', data_files=data_files)
```
Where `/path/to/text.py` is the path on the server where you saved the `text.py` script.
SBrandeis (CONTRIBUTOR) commented on 2021-01-12 10:38:23:
Hi Dave,
You are more than welcome to open a PR to make these changes! 🤗
You will find the relevant information about opening a PR in the [contributing guide](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md) and in the [dataset addition guide](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Pinging also @lhoestq for the Google cloud matter.
lhoestq (MEMBER) commented on 2021-01-12 15:53:47:
I opened a PR that allows to reload modules that have already been loaded once even if there's no internet.

Let me know if you know other ways that can make the offline mode experience better. I'd be happy to add them :) 

I already note the "freeze" modules option, to prevent local modules updates. It would be a cool feature.

----------

> @mandubian's second bullet point suggests that there's a workaround allowing you to use your offline (custom?) dataset with `datasets`. Could you please elaborate on how that should look like?

Indeed `load_dataset` allows to load remote dataset script (squad, glue, etc.) but also you own local ones.
For example if you have a dataset script at `./my_dataset/my_dataset.py` then you can do
```python
load_dataset("./my_dataset")
```
and the dataset script will generate your dataset once and for all.

----------

About I'm looking into having `csv`, `json`, `text`, `pandas` dataset builders already included in the `datasets` package, so that they are available offline by default, as opposed to the other datasets that require the script to be downloaded.
cf #1724
thomwolf (MEMBER) commented on 2021-01-12 16:01:58:
It's maybe a bit annoying to add but could we maybe have as well a version of the local data loading scripts in the package?
The `text`, `json`, `csv`. Thinking about people like in #1725 who are expecting to be able to work with local data without downloading anything.

Maybe we can add them to package_data or something?
SBrandeis (CONTRIBUTOR) commented on 2021-01-13 09:17:08:
Hi @ZurMaD!
Thanks for your interest in 🤗 `datasets`. Support for image datasets is at an early stage, with CIFAR-10 added in #1617 
MNIST is also on the way: #1730 

If you feel like adding another image dataset, I would advise starting by reading the [ADD_NEW_DATASET.md](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md) guide. New datasets are always very much appreciated 🚀
SBrandeis (CONTRIBUTOR) commented on 2021-01-13 09:21:41:
Hi @yangp725,
The SWDA has been added very recently and has not been released yet, thus it is not available in the `1.2.0` version of 🤗`datasets`.
You can still access it by installing the latest version of the library (master branch), by following instructions in [this issue](https://github.com/huggingface/datasets/issues/1641#issuecomment-751571471).
Let me know if this helps !
SBrandeis (CONTRIBUTOR) commented on 2021-01-13 09:29:15:
Hi @ameet-1997,
I think what you are looking for is the `concatenate_datasets` function: https://huggingface.co/docs/datasets/processing.html?highlight=concatenate#concatenate-several-datasets

For your use case, I would use the [`map` method](https://huggingface.co/docs/datasets/processing.html?highlight=concatenate#processing-data-with-map) to transform the SQuAD sentences and the `concatenate` the original and mapped dataset.

Let me know If this helps!
lhoestq (MEMBER) commented on 2021-01-13 09:44:13:
I haven't tested C4 on my side so there so there may be a few bugs in the code/adjustments to make.
Here it looks like in c4.py, line 190 one of the `files_to_download` is `'/'` which is invalid.
Valid files are paths to local files or URLs to remote files.
lhoestq (MEMBER) commented on 2021-01-13 10:35:02:
Hi ! Is this an issue you have with `openwebtext` specifically or also with other datasets ?

It looks like the downloaded file is corrupted and can't be extracted using `tarfile`.
Could you try loading it again with 
```python
import datasets
datasets.load_dataset("openwebtext", download_mode="force_redownload")
```
lhoestq (MEMBER) commented on 2021-01-13 10:37:51:
Indeed that would be cool

Also FYI right now the easiest way to do this is
```python
dataset_dict["train"] = dataset_dict["train"].map(my_transform_for_the_train_set)
dataset_dict["test"] = dataset_dict["test"].map(my_transform_for_the_test_set)
```
lhoestq (MEMBER) commented on 2021-01-13 10:47:53:
We're working on adding on-the-fly transforms in datasets.
Currently the only on-the-fly functions that can be applied are in `set_format` in which we transform the data in either numpy/torch/tf tensors or pandas.
For example
```python
dataset.set_format("torch")
```
applies `torch.Tensor` to the dataset entries on-the-fly.

We plan to extend this to user-defined formatting transforms.
For example
```python
dataset.set_format(transform=tokenize)
```

What do you think ?
lhoestq (MEMBER) commented on 2021-01-13 10:53:28:
Thanks for reporting !

Indeed this is a keyword in the library that is used to encode/decode features to a python dictionary that we can save/load to json.
We can probably change `_type` to something that is less likely to collide with user feature names.
In this case we would want something backward compatible though.

Feel free to try a fix and open a PR, and to ping me if I can help :)
lhoestq (MEMBER) commented on 2021-01-13 11:05:03:
> I'd like to make a few minor changes, including the citation information and the `_URL` from which to download the dataset. Can I submit a PR for this?

Sure ! Also feel free to ping us for reviews or if we can help :)

> It also looks like the dataset is being downloaded directly from Huggingface's Google cloud account rather than via the `_URL` in [scifact.py](https://github.com/huggingface/datasets/blob/master/datasets/scifact/scifact.py). Can you help me update the version on gcloud?

What makes you think that ?
Afaik there's no scifact on our google storage
lhoestq (MEMBER) commented on 2021-01-13 11:18:45:
Hi ! And thanks for the tips :) 

Indeed currently `wiki_dpr` takes some time to be processed.
Multiprocessing for dataset generation is definitely going to speed up things.

Regarding the index note that for the default configurations, the index is downloaded instead of being built, which avoid spending time on constructing the index. However in other cases it would be awesome to make the construction faster.

Any contribution that can help things faster are welcome. In particular in you have some code that can build a wiki_dpr IVF PQ index in a sharded GPU setup and would like to share it, we can add it to an `examples` folder. In particular since faiss is becoming the library of reference for dataset indexing for tasks like Open Domain Question Answering.
skyprince999 (CONTRIBUTOR) commented on 2021-01-13 13:32:28:
Updated it as per the comments. But couldn't figure out why the dummy tests are failing >> 
```
$RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_real_dataset_ted_talks_iwslt
.....
....
tests/test_dataset_common.py:198: in check_load_dataset
    self.parent.assertTrue(len(dataset[split]) > 0)
E   AssertionError: False is not true
```