t20062443/AT1ds huggingface dataset:
yjernite (MEMBER) commented on 2020-12-05 20:30:12:
Hi @ACR0S , thanks for adding the dataset!

It looks like `black` is throwing the code quality error: you need to run `make style` with the latest version of `black` (`black --version` should return 20.8b1)

We also added a requirement to specify encodings when using the python `open` function (line 163 in the current version of your script)

Finally, you will need to add the tags and field descriptions to the README as described here https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#tag-the-dataset-and-write-the-dataset-card

Let us know if you have any further questions!
yjernite (MEMBER) commented on 2020-12-05 20:43:06:
Hi @j-chim ! You're right it does feel redundant: your option works better, but I'd even suggest having the references in a Sequence feature, which you can declare as:
```
	    features=datasets.Features(
                {
                    "sentence": datasets.Value("string"),
                    "corrections": datasets.Sequence(datasets.Value("string")),
                }
            ),
```

To create the dummy data, you just need to tell the generator which files it should use, which you can do with:
`python datasets-cli dummy_data datasets/<your-dataset-folder> --auto_generate --match_text_files "train*,dev*,test*"`
yjernite (MEMBER) commented on 2020-12-05 20:47:57:
Glad to have more Japanese data! Couple of comments:
- the abbreviation might confuse some people as there is also an OPUS BSD corpus, would you mind renaming it as `bsd_ja_en`?
- `flake8` is throwing some errors, you can run it locally (`flake8 datasets`) and fix what it tells you until it's happy :)
- We're not using `os.path.join` for URLs as it's unstable across systems (introduces backslashes on Windows). Can you write the URLs explicitly instead?

Thanks!
thomwolf (MEMBER) commented on 2020-12-05 22:09:29:
Hi @hfawaz, this week we are doing the 🤗 `datasets` sprint (see some details [here](https://discuss.huggingface.co/t/open-to-the-community-one-week-team-effort-to-reach-v2-0-of-hf-datasets-library/2176)).

Nothing more to do on your side but it means that if you register on the thread I linked above, you can have some goodies for the present dataset that you have already added (and a special goodie if you want to spend more time and add 2 other datasets as well).

If you want to join, just tell me (or post on the thread on the HuggingFace forum: https://discuss.huggingface.co/t/open-to-the-community-one-week-team-effort-to-reach-v2-0-of-hf-datasets-library/2176)
yjernite (MEMBER) commented on 2020-12-05 22:27:57:
That's a great dataset to have! We need a couple more things to be good to go:
- you should `make style` and `flake8 datasets` before pushing to make the code quality check happy :) 
- the dataset will need some dummy data which you should be able to auto-generate and test locally: https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#automatically-add-code-metadata
- there's some good information in your current README, but we need the format to follow the template [here](https://github.com/huggingface/datasets/blob/master/templates/README.md) and to have YAML tags at the top, as described in the guide: https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#tag-the-dataset-and-write-the-dataset-card

LEt us know if you need any help!
duttahritwik (CONTRIBUTOR) commented on 2020-12-06 13:37:21:
Hi @yjernite 
I was successfully able to generate the dataset_info.json file using the command 
python datasets-cli test datasets/<your-dataset-folder> --save_infos --all_configs

But unfortunately, could not generate the dummy data

While running the command 
python datasets-cli dummy_data datasets/<your-dataset-folder> --auto_generate
I got an error as 

ValueError: Couldn't parse columns ['0', '1', '2', '3', '4', ......, '9982']. Maybe specify which json field must be used to read the data with --json_field <my_field>.

The thing is the dataset I am trying to upload is of the format 
{
   '0': {'Story_no': 15, 'Sentence': ' गाँठ से साढ़े तीन रुपये लग गये, जो अब पेट में जाकर खनकते भी नहीं! जो तेरी करनी मालिक! ” “इसमें मालिक की क्या करनी है? ”', 'Discourse Mode': 'Dialogue'},
  '1': {'Story_no': story_no, 'Sentence': sentence, 'Discourse Mode': discourse_mode},
   .......,
  '9982': {'Story_no': story_no, 'Sentence': sentence, 'Discourse Mode': discourse_mode}
}

Can you please suggest any errors I am making in the  _generate_examples method?

Thanks!
phtephanx (NONE) commented on 2020-12-06 13:59:49:
I solved it in a hacky way since I don't have so much time for this. As I use preemptible engines anyway, I split my dataset into one chunk per day in the data-loading logic.

You could try to change this [line](https://github.com/pytorch/pytorch/blob/master/torch/utils/data/sampler.py#L125) from ...
```python
yield from torch.randperm(n, generator=self.generator).tolist()
```
to
```python
yield from torch.randperm(n, generator=self.generator)
```
and, thereby, avoid converting to list, then build `pytorch` from source. 
If you use [DistributedSampler](https://github.com/pytorch/pytorch/blob/master/torch/utils/data/distributed.py), then, of course, adapt it there.

As pointed out in this [issue](https://github.com/pytorch/pytorch/issues/45427) not converting to lists saves lots of memory already. Let me know if this works!
yjernite (MEMBER) commented on 2020-12-06 14:57:14:
Hi @reshinthadithyan !

As mentioned on the Slack, it would be better in this case to parse the file lines into the following feature structure:
```python
"ast": datasets.Sequence(
    {
        "type": datasets.Value("string"),
        "value": datasets.Value("string"),
        "children": datasets.Sequence(datasets.Value("int32")),
    },
)
```

Here are a few more things to fix before we can move forward:
- the class name needs to be the CamelCase equivalent of the script name, so here it will have to be `PyAst`
- the `README.md` needs to have the tags at the top
- The homepage/info list at the top should be in the same format as the template (added a suggestion)
- You should add the dataset tags and field description to the README as described here: https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#tag-the-dataset-and-write-the-dataset-card

Good luck, let us know if you need any help!
yjernite (MEMBER) commented on 2020-12-06 18:35:21:
Good progress! Here's what still needs to be done:
- first, you need to rebase to master for the tests to pass :)
- the information in your `Data Fields` paragraph should go into `Data Instances`. Data fields should describe the fields one by one, as in e.g. https://github.com/huggingface/datasets/tree/master/datasets/eli5#data-fields
- you still need to add the YAML tags obtained with the tagging app

Should be good to go after that!
yjernite (MEMBER) commented on 2020-12-07 00:17:47:
Hi @tenjjin! This dataset is still up for grabs! Here's the link with the guide to add it. You should play around with the library first (download and look at a few datasets), then follow the steps here:

https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md
jcblaisecruz02 (NONE) commented on 2020-12-07 02:28:34:
Hi! I'm the author of this paper (surprised to see our datasets have been added already).

That paper link only leads to the conference index, here's a link to the actual paper: https://www.aclweb.org/anthology/2020.lrec-1.316/

Would it be fine if I also edited your gsheet entry to reflect this change?
ghomasHudson (CONTRIBUTOR) commented on 2020-12-07 10:32:01:
> Thanks for adding this @ghomasHudson!
> I added some comments for some of the fields.
> 
> Also, I'm not sure about this since I haven't used the library yet, but maybe it's worth adding the identification and resolution as two separate datasets?

Thanks for replying @yanaiela - I hope this will make your dataset more accessible to a wider audience - I've added the changes to the model card you suggested.

In terms of the identification and resolution tasks, I've currently added them as separate `splits` in huggingface/datasets so you can load identification like this:

```
import datasets
dataset = datasets.load_dataset("numeric_fused_head", "identification")
print(dataset["train"][0])
>> {"tokens": ["The", "quick", "brown", "fox",....], "start_index": 11, "end_index": 12, "label": 0}
```
And resolution like this:

```
import datasets
dataset = datasets.load_dataset("numeric_fused_head", "resolution")
print(dataset["train"][0])
>> {"tokens": ["The", "quick", "brown", "fox",....], "head": ["AGE"], "anchors_indices": [12], ...}
```
ghomasHudson (CONTRIBUTOR) commented on 2020-12-07 11:32:42:
@lhoestq I've finally got round to regenerating the `dataset_infos.json` for this and adding all 3 splits. I've done this and updated for the new version of datasets.

The CI tests still aren't passing though (they pass on my machine). `test_load_dataset_narrativeqa` seems to fail but I have no idea how. Would appreciate if you have any ideas - would be great to finally finish this one!
rahul-art (CONTRIBUTOR) commented on 2020-12-07 14:46:12:
yes it is expected because I  made all my changes in  PR  #1186  then I again run code and open PR #1188 to see if this time test passes or not only so there is no code change from #1186 to #1188  
i tried  --match_text_file "*.plaintext" this time it is also not generating dummy data don't know why
olinguyen (CONTRIBUTOR) commented on 2020-12-07 17:32:18:
> It seems to share the same name as this dataset: https://openreview.net/forum?id=JENSKEEzsoU

you're right it can be confusing. I'll add the organization/research group for clarity: `covid_qa_castorini`. I added the dataset you shared as `covid_qa_deepset` in another PR (#1182)
thomwolf (MEMBER) commented on 2020-12-07 22:19:12:
This is ok now @lhoestq

I've included the tweak to `dummy_data` to only use the data transmitted to `_generate_examples` by default (it only do that if it can find at least one path to an existing file in the `gen_kwargs` and this can be unactivated with a flag).

Should I extract it in another PR or is it ok like this?
ghomasHudson (CONTRIBUTOR) commented on 2020-12-07 22:42:20:
> @ghomasHudson I've tried it but still getting code quality error. I've removed all blank lines, etc. required by flake8. Don't know what else to do

Did you run `make style` before committing? When I run it, it fixes some things (e.g. Splitting line 96 which is currently too long).
leoxzhao (CONTRIBUTOR) commented on 2020-12-08 06:12:34:
Also, I attempted to see if I can get the source Chinese sentences from `wmt17` dataset. But this call `data = load_dataset('wmt17', "zh-en")` failed with this error: `FileNotFoundError: Couldn't find file at https://storage.googleapis.com/tfdataset-data/downloadataset/uncorpus/UNv1.0.en-zh.tar.gz`. I think it should be possible and fairly straightforward to get the pairing source sentences from it. I just can not test it right now.
rabeehk (CONTRIBUTOR) commented on 2020-12-08 09:34:52:
here is the minimal code to reproduce

`datasets>>> datasets.load_dataset("boolq", "train")

the errors

```
`cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets
Using custom data configuration train
Downloading and preparing dataset boolq/train (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /idiap/temp/rkarimi/cache_home_1/datasets/boolq/train/0.1.0/2987db1f15deaa19500ae24de560eabeaf1f8ef51df88c0470beeec72943bf11...
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets/downloads
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/load.py", line 611, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 476, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 531, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File " /idiap/home/rkarimi/.cache/huggingface/modules/datasets_modules/datasets/boolq/2987db1f15deaa19500ae24de560eabeaf1f8ef51df88c0470beeec72943bf11/boolq.py", line 74, in _split_generators
    downloaded_files = dl_manager.download_custom(urls_to_download, tf.io.gfile.copy)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 149, in download_custom
    custom_download(url, path)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/tensorflow/python/lib/io/file_io.py", line 516, in copy_v2
    compat.path_to_bytes(src), compat.path_to_bytes(dst), overwrite)



```
huu4ontocord (CONTRIBUTOR) commented on 2020-12-08 13:55:46:
@lhoestq I'm not sure what I did wrong. What did I push that wasn't conceptnet5? How do I see this?

 did this

make style
flake8 datasets
git add datasets/<your_dataset_name>
git commit
git fetch upstream
git rebase upstream/master
git pull
git push -u origin conceptnet5
lhoestq (MEMBER) commented on 2020-12-08 16:51:41:
Can you run `make style` to fix the code format ?

Also it looks like the file `file_downloaded/TWNERTC_TC_Coarse Grained NER_DomainIndependent_NoiseReduction.zip/TWNERTC_TC_Coarse Grained NER_DomainIndependent_NoiseReduction.DUMP` is missing inside the dummy_data.zip


(note that `TWNERTC_TC_Coarse Grained NER_DomainIndependent_NoiseReduction.zip` is a directory name, not an actual zip file)
nataly-obr (NONE) commented on 2020-12-08 19:45:51:
> Please if you could tell me more about the error?
> 
> 1. Please check the directory you've been working on
> 2. Check for any typos

Error happens during the execution of this line:
dataset = load_dataset("german_legal_entity_recognition")

Also, when I try to open mentioned links via Opera I have errors "404: Not Found" and "This XML file does not appear to have any style information associated with it. The document tree is shown below." respectively.
ZacharySBrown (CONTRIBUTOR) commented on 2020-12-08 23:43:56:
Seeing this in the CircleCI builds, this is what I was originally getting before I started messing around with the download URLS to try to fix this:

`FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpwvji917g/extracted/d6185140afb24ad8fee67392100a478269cba286b0d88915a137fdf88872de14/dummy_data/train__VARIABLE_MISUSE__SStuB.txt-00001-of-00300'`

Could this be because of the files in my `dummy_data.zip`? I had to manually create it, and it looked like the test was looking for the following files, so I created the `.zip` with this structure:

```
Archive:  dummy_data.zip
   creating: dummy_data/
  inflating: dummy_data/blobtest     
  inflating: dummy_data/parsed.jsontrain  
  inflating: dummy_data/parsed.jsonvalidation 
```
lhoestq (MEMBER) commented on 2020-12-09 09:25:42:
Can you run `make style` to format the code for the CI please ?

Also about the dummy data, here is how to generate them:

We need a dummy_data.zip file in ./datasets/hindiDiscourse/dummy/1.0.0 (or replace hindiDiscourse by hindi_discourse since we have to rename the folder anyway)
To create the zip file, first go in this directory and create a folder named dummy_data.
Then inside the dummy_data folder create a file `discourse_dataset.json` and fill it with something like 5 examples.
Finally zip the dummy_data folder to end up with the dummy_data.zip file

Once it's done you can check if the dummy data test passes with 
```
RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_hindi_discourse
```

If it passes you can then remove the dummy_data folder to keep only the dummy_data.zip file
lhoestq (MEMBER) commented on 2020-12-09 10:25:48:
Nice !
Could you add the dummy data generation trick in another PR ?
I think we can also extend it to make it work not only with data files paths but also with data directories (sometimes it's one of the parent directory that is passed to gen_kwargs, not the actual path to the file).
This will help a lot to make the dummy data lighter !
forest1988 (CONTRIBUTOR) commented on 2020-12-09 11:11:08:
@cstorm125 Thank you for the comment! 
As you point out, it seems my code has something wrong about downloading and uncompressing the file.
However, my manually created dummy data seems to contain a file of the required format.

On Colaboratory,
`!unzip /content/datasets/datasets/covid_tweets_japanese/dummy/1.1.0/dummy_data.zip`
returns:

```
Archive:  /content/datasets/datasets/covid_tweets_japanese/dummy/1.1.0/dummy_data.zip
   creating: content/datasets/datasets/covid_tweets_japanese/dummy/1.1.0/dummy_data/
 extracting: content/datasets/datasets/covid_tweets_japanese/dummy/1.1.0/dummy_data/data.csv.bz2  
```

The original data is `data.csv.bz2`, and I had a very hard time dealing with uncompressing bzip2.
I think I could handle it, but there may be problems remain.
lhoestq (MEMBER) commented on 2020-12-09 11:30:31:
I just checked and the wmt16 en-ro doesn't have empty lines
```python
from datasets import load_dataset

d = load_dataset("wmt16", "ro-en", split="train")
len(d)  # 610320
len(d.filter(lambda x: len(x["translation"]["en"].strip()) > 0))  # 610320
len(d.filter(lambda x: len(x["translation"]["ro"].strip()) > 0))  # 610320
# also tested for split="validation" and "test"
```

Can you open an issue on the `transformers` repo ? also cc @sgugger
julien-c (MEMBER) commented on 2020-12-09 11:35:23:
FYI you guys are on GitHub's homepage 😍

<img width="1589" alt="Screenshot 2020-12-09 at 12 34 28" src="https://user-images.githubusercontent.com/326577/101624883-a0ecc700-39e8-11eb-8a97-11af0d036536.png">
rahul-art (CONTRIBUTOR) commented on 2020-12-09 15:11:28:
these are a the which i did in #1186 and same in #1188 
![1](https://user-images.githubusercontent.com/56379013/101646577-b4864500-3a5d-11eb-8a5a-91b1b441040a.png)
![2](https://user-images.githubusercontent.com/56379013/101646965-32e2e700-3a5e-11eb-94d9-276e602c6ded.png)
![4](https://user-images.githubusercontent.com/56379013/101646989-38d8c800-3a5e-11eb-92bb-d9c4cb2c3595.png)
![5](https://user-images.githubusercontent.com/56379013/101647017-41c99980-3a5e-11eb-87cf-5268e79df19d.png)
![6](https://user-images.githubusercontent.com/56379013/101647038-48581100-3a5e-11eb-8d05-f67834fcaa7b.png)

![8](https://user-images.githubusercontent.com/56379013/101647080-55750000-3a5e-11eb-8455-8936a35b35c2.png)
![9](https://user-images.githubusercontent.com/56379013/101647084-55750000-3a5e-11eb-988e-ae87f0b252a0.png)
![10](https://user-images.githubusercontent.com/56379013/101647182-6f164780-3a5e-11eb-8af3-f0b0186483c9.png)
![11](https://user-images.githubusercontent.com/56379013/101647230-7c333680-3a5e-11eb-9aeb-2b4ce65965e0.png)
![13](https://user-images.githubusercontent.com/56379013/101647257-848b7180-3a5e-11eb-871c-2fd77b047320.png)
![14](https://user-images.githubusercontent.com/56379013/101647268-89502580-3a5e-11eb-9e2a-b9f7ff1fc95e.png)
these same codes are in both #1186 and #1188 so because it is already present from PR #1186 because of that it is showing zeor code change in #1188 because it is already present from #1186 how i can show or highlight those changes
lhoestq (MEMBER) commented on 2020-12-09 17:39:46:
It not always practical to use nested `Sequence`. If you have troubles with sequence you can use lists instead. 

For example
```python

features=datasets.Features(
    {
        "dial_id": datasets.Value("string"),
        "doc_id": datasets.Value("string"),
        "domain": datasets.Value("string"),
        "turns": [
            {
                "turn_id": datasets.Value("int32"),
                "role": datasets.Value("string"),
                "da": datasets.Value("string"),
                "reference": [
                    {
                        "keys" : datasets.Value("string"),
                        "values": datasets.Value("string"),   
                    }

                ],
                "utterance": datasets.Value("string"),
            }
        ],
    }
),
```

this way `turns` will be a list of dict, and the "reference" key of `turns` will be a list of dict as well
ghomasHudson (CONTRIBUTOR) commented on 2020-12-09 18:16:47:
> Awesome thank you !
> 
> Could you try to reduce the size of the dummy_data.zip file before we merge ? (it's 300KB right now)
> 
> To do so feel free to take a look inside it and remove all the unnecessary files and chunks of text, to only keep a few examples. The idea is to have a zip file that is only a few KB

Ah, it only contains 1 example for each split. I think the problem is that I include an entire story (like in the full dataset). We can probably get away with a summarised version.
anaerobeth (CONTRIBUTOR) commented on 2020-12-10 04:07:11:
@yjernite @lhoestq Any suggestions on how to get the dummy data generator to recognize the columns? The structure of the json is:
```
{
  "Dataset": "FreebaseQA-eval", 
  "Version": "1.0", 
  "Questions": [
    {
      "Question-ID": "FreebaseQA-eval-0", 
      "RawQuestion": "Who is the female presenter of the Channel 4 quiz show '1001 things you should know'?", 
      "ProcessedQuestion": "who is the female presenter of the channel 4 quiz show '1001 things you should know'", 
      "Parses": [
        {
          "Parse-Id": "FreebaseQA-eval-0.P0", 
          "PotentialTopicEntityMention": "1001 things you should know", 
          "TopicEntityName": "1001 things you should know", 
          "TopicEntityMid": "m.0nd3t34", 
          "InferentialChain": "tv.tv_program.regular_personal_appearances..tv.tv_regular_personal_appearance.person", 
          "Answers": [
            {
              "AnswersMid": "m.0216y_", 
              "AnswersName": [
                "sandi toksvig"
              ]
            }
          ]
        }
      ]
    }, 
   ...
  ]
}
```

Thanks!
jonatasgrosman (CONTRIBUTOR) commented on 2020-12-10 12:18:47:
Hi @thomwolf, for simplification purposes, I think you could remove the "`pip install ...`" steps from this commit, 'cause these deps (black, isort, flake8) are already installed on `pip install -e ".[dev]"` on the [Start by preparing your environment](https://github.com/huggingface/datasets/blob/704107f924e74445f6f0fbde69a218b72178b588/ADD_NEW_DATASET.md#start-by-preparing-your-environment)
ghomasHudson (CONTRIBUTOR) commented on 2020-12-10 15:29:51:
> Nice thank you, can you make it even lighter if possible ? Something round 10KB would be awesone
> We try to keep the repo light so that it doesn't take ages to clone. So we have to make sure the dummy data are as small as possible for every single dataset.

Have trimmed a little more out of each example now.
yavuzKomecoglu (CONTRIBUTOR) commented on 2020-12-10 17:29:41:
@coltekin, we have added the data set that you created an article that says "Turkish Attack Language Community in Social Media", HuggingFace dataset update sprint for you. We added Sprint quickly for a short time. I hope you welcome it too. The dataset is accessible at https://huggingface.co/datasets/offenseval2020_tr.
2dot71mily (CONTRIBUTOR) commented on 2020-12-10 17:33:08:

So... my normal work is with mercurial and seem to have clearly forked this up using git... :(

What I did is after calling:
```
git fetch upstream
git rebase upstream/master
```

I then I attempt to pull in my most recent changes UI commit changes based on @lhoestq's feedback with:
```
git pull
```  
... which I now suspect undid the above fetch and rebase. Will look into fixing later today when I have more time. Sorry!