t20062443/AT1ds huggingface dataset:
yjernite (MEMBER) commented on 2020-12-14 18:24:13:
Looks like the name of the file is the issue here: you have a file called `hindencorp05.plaintext`, but it should be called `hindencorp05.plaintext.gz%3Fsequence%3D3%26isAllowed%3Dy`. You just have to rename it to pass the test:
```
cd datasets/hind_encorp/dummy/0.0.0
rm -rf dummy_data
unzip dummy_data.zip
rm dummy_data.zip
mv dummy_data/hindencorp05.plaintext "dummy_data/hindencorp05.plaintext.gz%3Fsequence%3D3%26isAllowed%3Dy"
zip -r dummy_data.zip dummy_data 
```

Then **once you pass the tests locally** you just have to remember to `make style` and `flake8 datasets` to pass the style checks, and you should be good to go :hugs: 

For reference, here are the instructions given by the tool:
```
$ python datasets-cli dummy_data datasets/hind_encorp/
2020-12-14 13:16:26.824828: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2020-12-14 13:16:26.824846: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Using custom data configuration default

==============================DUMMY DATA INSTRUCTIONS==============================
- In order to create the dummy data for , please go into the folder 'datasets/hind_encorp/dummy/0.0.0' with `cd datasets/hind_encorp/dummy/0.0.0` . 

- Please create the following dummy data files 'dummy_data/hindencorp05.plaintext.gz%3Fsequence%3D3%26isAllowed%3Dy' from the folder 'datasets/hind_encorp/dummy/0.0.0'

- For each of the splits 'train', make sure that one or more of the dummy data files provide at least one example 

- If the method `_generate_examples(...)` includes multiple `open()` statements, you might have to create other files in addition to 'dummy_data/hindencorp05.plaintext.gz%3Fsequence%3D3%26isAllowed%3Dy'. In this case please refer to the `_generate_examples(...)` method 

-After all dummy data files are created, they should be zipped recursively to 'dummy_data.zip' with the command `zip -r dummy_data.zip dummy_data/` 

-You can now delete the folder 'dummy_data' with the command `rm -r dummy_data` 

- To get the folder 'dummy_data' back for further changes to the dummy data, simply unzip dummy_data.zip with the command `unzip dummy_data.zip` 

- Make sure you have created the file 'dummy_data.zip' in 'datasets/hind_encorp/dummy/0.0.0' 
===================================================================================
```
aclifton314 (CONTRIBUTOR) commented on 2020-12-14 18:35:54:
I incorporated all the changes but when I go to rebase I get the following error:
```python
aclifton@pop-os:~/hf_datasets_sprint/datasets$ git rebase upstream/master
error: cannot rebase: You have unstaged changes.
error: Please commit or stash them.
aclifton@pop-os:~/hf_datasets_sprint/datasets$ git stash
Saved working directory and index state WIP on imppres: 51736236 Incorporated secondary sets as configurations instead of splits.
aclifton@pop-os:~/hf_datasets_sprint/datasets$ git rebase upstream/master
CONFLICT (add/add): Merge conflict in datasets/wiki_movies/wiki_movies.py
Auto-merging datasets/wiki_movies/wiki_movies.py
CONFLICT (add/add): Merge conflict in datasets/wiki_movies/dataset_infos.json
Auto-merging datasets/wiki_movies/dataset_infos.json
CONFLICT (add/add): Merge conflict in datasets/wiki_movies/README.md
Auto-merging datasets/wiki_movies/README.md
error: could not apply 04d08587... Created wiki_movies dataset.
Resolve all conflicts manually, mark them as resolved with
"git add/rm <conflicted_files>", then run "git rebase --continue".
You can instead skip this commit: run "git rebase --skip".
To abort and get back to the state before "git rebase", run "git rebase --abort".
Could not apply 04d08587... Created wiki_movies dataset.
aclifton@pop-os:~/hf_datasets_sprint/datasets$ git branch
* (no branch, rebasing imppres)
  imppres
  logiqa_en
  master
  wiki_movies
  wiki_movies_htl
aclifton@pop-os:~/hf_datasets_sprint/datasets$ git checkout imppres 
datasets/wiki_movies/README.md: needs merge
datasets/wiki_movies/dataset_infos.json: needs merge
datasets/wiki_movies/wiki_movies.py: needs merge
error: you need to resolve your current index first
aclifton@pop-os:~/hf_datasets_sprint/datasets$ 
```
vrindaprabhu (CONTRIBUTOR) commented on 2020-12-15 05:05:20:
Changed the pre-processing based on the comments raised in [PR-1482](https://github.com/huggingface/datasets/pull/1482).The below command is passing in my local environment:

`python datasets-cli test datasets/covid_qa_ucsd/ --save_infos --all_configs --data_dir ~/Downloads/Medical-Dialogue-Dataset/CovidDailogue/`
vrindaprabhu (CONTRIBUTOR) commented on 2020-12-15 05:08:30:
[PR-1503](https://github.com/huggingface/datasets/pull/1503) on the COVID dialog dataset from the same University has similar features. I kept it separate because it is only on COVID qa. Also consists of only single files per language. Kindly let me know if it has to be added as two more configurations to this existing dataset itself. If it has to be added, then can I still freeze the file names like in PR-1503?
skyprince999 (CONTRIBUTOR) commented on 2020-12-15 08:04:05:
Hi @abdulelahsm - This is an interesting dataset! But there are multiple issues with the PR. Some of them are listed below: 
-  default builder config is not defined. There should be atleast one builder config 
- URL is incorrectly constructed so the data files are not being downloaded 
- dataset_info.json file was not created

Please have a look at some existing merged datasets to get a reference on building the data loader. If you are still stuck, reach out.
lhoestq (MEMBER) commented on 2020-12-15 10:34:31:
Does your instance have an internet connection ?

If you don't have an internet connection you'll need to have the dataset on the instance disk.
To do so first download the dataset on another machine using `load_dataset` and then you can save it in a folder using `my_dataset.save_to_disk("path/to/folder")`. Once the folder is copied on your instance you can reload the dataset with `datasets.load_from_disk("path/to/folder")`
aclifton314 (CONTRIBUTOR) commented on 2020-12-15 16:11:06:
I get this response when I try to switch to master:
```
aclifton@pop-os:~/hf_datasets_sprint/datasets$ git checkout master
datasets/wiki_movies/README.md: needs merge
datasets/wiki_movies/dataset_infos.json: needs merge
datasets/wiki_movies/wiki_movies.py: needs merge
error: you need to resolve your current index first
```
SBrandeis (CONTRIBUTOR) commented on 2020-12-15 16:45:02:
Many thanks @Nilanshrajput  !
The failing tests on CI are not related to your changes, merging master on your branch should fix them :)
If you're interested in what causes the CI to fail, checkout [this commit](https://github.com/huggingface/datasets/commit/9a0f1e20ca1e783cb14c1ab1cc2f54b0b5b201e8)
2dot71mily (CONTRIBUTOR) commented on 2020-12-15 16:50:32:
@lhoestq, thank you for the great feedback, especially given how busy you guys are now! 

I checked out GitHub release tags and looks cool. I have added the version tag to the url, instead of the commit sha as originally suggested, with the hope that it serves the same purpose of pinning the content to this url. Please let me know if I have misunderstood.

In regard to dynamically changing the number of files downloaded by first downloading a JSON listing the files, I love that idea. But I am a little confused, as I was thinking that any changes to the dataset itself would require a new PR with an updated `dataset_infos.json`, e.g. `num_examples` would increase. 

If the purpose of this is not to permit dynamic (without a PR needed) growth of the number of files, but instead to provide stability to the consumers of the dataset, maybe I continued use the release tags, maintaining access to old releases could serve this purpose? I am still learning about these release tags...
lhoestq (MEMBER) commented on 2020-12-15 16:57:24:
For dynamic datasets, i.e. datasets that evolve over time, we support custom configurations: they are configurations that are not part of the BUILDER_CONFIGS or in the dataset_infos.json

For example for wikipedia, you can use the latest wiki dump by specifying `date=` inside `load_dataset()`. A configuration is created on the fly for this date and is used to build the dataset using the latest data.

Therefore we don't need to have PRs to update the script for each wikipedia release.

One downside though is that we don't have metadata in advance such as the size of the dataset.

I think this could be a nice addition for the youtube caption dataset in the future to be have a system of releases and be able to load the version we want easily. What do you think ?
2dot71mily (CONTRIBUTOR) commented on 2020-12-15 17:28:02:




> For dynamic datasets, i.e. datasets that evolve over time, we support custom configurations: they are configurations that are not part of the BUILDER_CONFIGS or in the dataset_infos.json
> 
 
> I think this could be a nice addition for the youtube caption dataset in the future to be have a system of releases and be able to load the version we want easily. What do you think ?

Thank you for the suggestion! This sounds great! I will take a look at the some datasets that do this, and would love to give it a try in the future, if I continue to grow the captions dataset in a meaningful way. 

Appreciate all the help on this. It has been a really great experience for me. :)
SBrandeis (CONTRIBUTOR) commented on 2020-12-15 17:31:16:
I think the `features` attribute of the dataset object is what you are looking for:
```
>>> dataset.features
{'sentence1': Value(dtype='string', id=None),
 'sentence2': Value(dtype='string', id=None),
 'label': ClassLabel(num_classes=2, names=['not_equivalent', 'equivalent'], names_file=None, id=None),
 'idx': Value(dtype='int32', id=None)
}
>>> dataset.features["label"].names
['not_equivalent', 'equivalent']
```

For reference: https://huggingface.co/docs/datasets/exploring.html
SBrandeis (CONTRIBUTOR) commented on 2020-12-15 17:39:45:
Thanks for pointing this out ! 🕵🏻 
Predictions and references should indeed be swapped in the docstring.
However, the call to `accuracy_score` should not be changed, it [signature](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html#sklearn.metrics.accuracy_score) being:
```
sklearn.metrics.accuracy_score(y_true, y_pred, *, normalize=True, sample_weight=None)
```

Feel free to open a PR if you want to fix this :)
SBrandeis (CONTRIBUTOR) commented on 2020-12-15 17:53:00:
Hello @pietrolesci , I am not sure to understand what you are trying to do here.

If you're looking for ways to save a dataset on disk, you can you the `save_to_disk` method:
```python
>>> import datasets
>>> loaded_dataset = datasets.load("dataset_name")
>>> loaded_dataset.save_to_disk("/path/on/your/disk")
```

The saved dataset can later be retrieved using:
```python
>>> loaded_dataset = datasets.Dataset.load_from_disk("/path/on/your/disk")
```

Also, I'd recommend posting your question directly in the issue section of the [elasticsearch repo](https://github.com/elastic/elasticsearch)
SBrandeis (CONTRIBUTOR) commented on 2020-12-15 18:00:29:
Hello @nataly-obr, the `german_legal_entity_recognition` dataset has not yet been released (it is part of the coming soon v2 release).

You can still access it now if you want, but you will need to install `datasets` via the master branch:
`pip install git+https://github.com/huggingface/datasets.git@master`

Please let me know if it solves the issue :)
tanmoyio (CONTRIBUTOR) commented on 2020-12-15 21:29:12:
don't use the --auto-generate argument and you will get a brief instructions on how to create dummy data for your dataset,
also you dont have to run the pytest for main dataset if your data is needed to be downloaded manually, just run the pytest for dummy dataset, and when you will create the json you need to provide the main data directory path by using this argument --data_dir
LuCeHe (NONE) commented on 2020-12-15 21:46:48:
I think it would be very cool. I'm currently working on a cluster from Compute Canada, and I have internet access only when I'm not in the nodes where I run the scripts. So I was expecting to be able to use the wmt14 dataset until I realized I needed internet connection even if I downloaded the data already. I'm going to try option 2 you mention for now though! Thanks ;)
rahul-art (CONTRIBUTOR) commented on 2020-12-15 23:54:08:
@lhoestq i did all the changes you suggested but at the time of load_dataset it is giving me error
`**`datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=76591256, num_examples=1, dataset_name='hind_encorp'), 'recorded': SplitInfo(name='train', num_bytes=78945714, num_examples=273885, dataset_name='hind_encorp')}]`**`
YipingNUS (NONE) commented on 2020-12-16 04:20:51:
Hi, seems I also can't read csv file. I was trying with a dummy csv with only three rows.

```
text,label
I hate google,negative
I love Microsoft,positive
I don't like you,negative
```
I was using the HuggingFace image in Paperspace Gradient (datasets==1.1.3). The following code doesn't work:

```
from datasets import load_dataset
dataset = load_dataset('csv', script_version="master", data_files=['test_data.csv'], delimiter=",")
```
It outputs the following:
```
Using custom data configuration default
Downloading and preparing dataset csv/default-3b6254ff4dd403e5 (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /root/.cache/huggingface/datasets/csv/default-3b6254ff4dd403e5/0.0.0/2960f95a26e85d40ca41a230ac88787f715ee3003edaacb8b1f0891e9f04dda2...
Dataset csv downloaded and prepared to /root/.cache/huggingface/datasets/csv/default-3b6254ff4dd403e5/0.0.0/2960f95a26e85d40ca41a230ac88787f715ee3003edaacb8b1f0891e9f04dda2. Subsequent calls will reuse this data.
```
But `len(dataset)` gives `1` and I can't access rows with indexing `dataset[0]` (it gives `KeyError: 0`).

However, loading from pandas dataframe is working.
```
from datasets import Dataset
import pandas as pd
df = pd.read_csv('test_data.csv')
dataset = Dataset.from_pandas(df)
```
rsanjaykamath (CONTRIBUTOR) commented on 2020-12-16 06:53:05:
Thanks for your help, @tanmoyio 
I tried with and without --auto_generate flag. 
Here are the issues. 

**With --auto_generate**
`python datasets-cli dummy_data datasets/newsqa/ --auto_generate `
`Traceback (most recent call last):
  File "datasets-cli", line 36, in <module>
    service.run()
  File "/Users/sanjaykamath/Python_Projects/HuggingFace/datasets/src/datasets/commands/dummy_data.py", line 321, in run
    keep_uncompressed=self._keep_uncompressed,
  File "/Users/sanjaykamath/Python_Projects/HuggingFace/datasets/src/datasets/commands/dummy_data.py", line 340, in _autogenerate_dummy_data
    dataset_builder._split_generators(dl_manager)
  File "/Users/sanjaykamath/.cache/huggingface/modules/datasets_modules/datasets/newsqa/7a565b204506c1fd91047290073be54d3ae05fa2b0ab17ae0bc6f709350fcbca/newsqa.py", line 180, in _split_generators
    path_to_manual_folder = os.path.abspath(os.path.expanduser(dl_manager.manual_dir))
  File "/Users/sanjaykamath/anaconda3/envs/huggingface/lib/python3.7/posixpath.py", line 235, in expanduser
    path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not NoneType
`


**Without --auto_generate**
`python datasets-cli dummy_data datasets/newsqa/`
`Dataset  with config BuilderConfig(name='combined-csv', version=1.0.0, data_dir=None, data_files=None, description='This part of the dataset covers the whole dataset in the combined format of CSV as mentioned here: https://github.com/Maluuba/newsqa#csv') seems to already open files in the method `_split_generators(...)`. You might consider to instead only open files in the method `_generate_examples(...)` instead. If this is not possible the dummy data has to be created with less guidance. Make sure you create the file None.
Traceback (most recent call last):
  File "datasets-cli", line 36, in <module>
    service.run()
  File "/Users/sanjaykamath/Python_Projects/HuggingFace/datasets/src/datasets/commands/dummy_data.py", line 326, in run
    dataset_builder=dataset_builder, mock_dl_manager=mock_dl_manager
  File "/Users/sanjaykamath/Python_Projects/HuggingFace/datasets/src/datasets/commands/dummy_data.py", line 406, in _print_dummy_data_instructions
    for split in generator_splits:
UnboundLocalError: local variable 'generator_splits' referenced before assignment
`
skyprince999 (CONTRIBUTOR) commented on 2020-12-16 09:40:20:



I cloned the branch and it seems to work fine at my end. try to clear the cache - 

```
rm -rf  /home/ubuntu/.cache/huggingface/datasets/
rm -rf  /home/ubuntu/.cache/huggingface/modules/datasets_modules//datasets/
```
But the dataset has only one record. Is that correct? 
![image](https://user-images.githubusercontent.com/9033954/102331376-c7929b00-3fb0-11eb-8a6c-81b2cf47bc2a.png)
lhoestq (MEMBER) commented on 2020-12-16 09:48:50:
> @lhoestq i did all the changes you suggested but at the time of load_dataset it is giving me error
> `**`datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=76591256, num_examples=1, dataset_name='hind_encorp'), 'recorded': SplitInfo(name='train', num_bytes=78945714, num_examples=273885, dataset_name='hind_encorp')}]`**`

You can ignore this error by adding `ignore_verifications=True` to `load_dataset`.

This error is raised because you're loading a dataset that you've already loaded once in the past. Therefore the library does some verifications to make sure it's generated the same way. 

However since you've done changes in the dataset script you should ignore these verifications.

You can regenerate the dataset_infos.json with
```
datasets-cli test ./datasets/hindi_encorp --save_infos --all_configs --ignore_verifications
```


> I cloned the branch and it seems to work fine at my end. try to clear the cache -
> 
> ```
> rm -rf  /home/ubuntu/.cache/huggingface/datasets/
> rm -rf  /home/ubuntu/.cache/huggingface/modules/datasets_modules//datasets/
> ```
> 
> But the dataset has only one record. Is that correct?
> ![image](https://user-images.githubusercontent.com/9033954/102331376-c7929b00-3fb0-11eb-8a6c-81b2cf47bc2a.png)

Yes the current parsing is wrong, I've already given @rahul-art some suggestions and it looks like it works way better now (num_examples=273885).

Thanks for fixing the parsing @rahul-art !
Feel free to commit and push your changes once it's ready :)
lhoestq (MEMBER) commented on 2020-12-16 10:35:42:
This is because load_dataset without `split=` returns a dictionary of split names (train/validation/test) to dataset.
You can do
```python
from datasets import load_dataset
dataset = load_dataset('csv', script_version="master", data_files=['test_data.csv'], delimiter=",")
print(dataset["train"][0])
```

Or if you want to directly get the train split:

```python
from datasets import load_dataset
dataset = load_dataset('csv', script_version="master", data_files=['test_data.csv'], delimiter=",", split="train")
print(dataset[0])
```
thomwolf (MEMBER) commented on 2020-12-16 10:42:21:
Good point

Design question for us, though: should `load_dataset` when no split is specified and only one split is present in the dataset (common use case with CSV/text/JSON datasets) return a `Dataset` instead of a `DatsetDict`? I feel like it's often what the user is expecting. I break a bit the paradigm of a unique return type but since this library is designed for widespread DS people more than CS people usage I would tend to think that UX should take precedence over CS reasons. What do you think?
rahul-art (CONTRIBUTOR) commented on 2020-12-16 10:56:20:
i ran the command you provided datasets-cli test ./datasets/hindi_encorp --save_infos --all_configs --ignore_verifications 
but now its giving this error @lhoestq 

FileNotFoundError: Couldn't find file locally at ./datasets/hindi_encorp/hindi_encorp.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/./datasets/hindi_encorp/hindi_encorp.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/./datasets/hindi_encorp/hindi_encorp.py.
If the dataset was added recently, you may need to to pass script_version="master" to find the loading script on the master branch.
lhoestq (MEMBER) commented on 2020-12-16 10:56:38:
In this case the user expects to get only one dataset object instead of the dictionary of datasets since only one csv file was specified without any split specifications.
I'm ok with returning the dataset object if no split specifications are given for text/json/csv/pandas.

For the other datasets ton the other hand the user doesn't know in advance the splits so I would keep the dictionary by default. What do you think ?
katnoria (CONTRIBUTOR) commented on 2020-12-16 11:54:00:
> looks like this PR includes changes about many other files than the ones for wiki_lingua.
> 
> Can you create another branch and another PR ?
> (or you can try to fix this branch with rebase and push force if you're familiar with it)

Easier to create a new branch and submit, I have submitted a new PR #1582
SBrandeis (CONTRIBUTOR) commented on 2020-12-16 16:00:04:
Hi @phtephanx , the `amazon_polarity` dataset has not been released yet. It will be available in the coming soon v2of `datasets` :) 

You can still access it now if you want, but you will need to install datasets via the master branch:
`pip install git+https://github.com/huggingface/datasets.git@master`
bharatr21 (CONTRIBUTOR) commented on 2020-12-16 16:28:11:
> Hi @Bharat123rox did you manage to join the different config into one using the IDs ?
> 
> Feel free to ping me when you're ready for the next review :) 

Not yet @lhoestq still working on this! Meanwhile please review #1507 where I added the SelQA dataset :)
aclifton314 (CONTRIBUTOR) commented on 2020-12-16 16:59:09:
Here is what I get:
```
aclifton@pop-os:~/hf_datasets_sprint/datasets$ git stash
datasets/wiki_movies/README.md: needs merge
datasets/wiki_movies/dataset_infos.json: needs merge
datasets/wiki_movies/wiki_movies.py: needs merge
```
tdiggelm (CONTRIBUTOR) commented on 2020-12-16 18:11:06:
I `git rebase`ed my branch to `upstream/master` as suggested in point 7 of <https://huggingface.co/docs/datasets/share_dataset.html> and subsequently used `git pull` to be able to push to my remote branch. However, I think this messed up the history.

Please let me know if I should create a clean new PR with my changes.

Update: I also fixed the dataset name in the Dataset Card.
aclifton314 (CONTRIBUTOR) commented on 2020-12-16 20:32:46:
So close! I got the new branch made and went through all the tests. When I went to push, I got the following:
```
aclifton@pop-os:~/hf_datasets_sprint/datasets$ git push -u origin imppres
Username for 'https://github.com': aclifton314
Password for 'https://aclifton314@github.com': 
To https://github.com/aclifton314/datasets
 ! [rejected]          imppres -> imppres (non-fast-forward)
error: failed to push some refs to 'https://github.com/aclifton314/datasets'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Integrate the remote changes (e.g.
hint: 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.
```