rIsHu009/github-issues-updated huggingface dataset:
Title: Add quora dataset
Author: ghomasHudson (CONTRIBUTOR)
State: closed
Created: 2020-07-09T10:34:22Z

Added the [Quora question pairs dataset](https://www.quora.com/q/quoradata/First-Quora-Dataset-Release-Question-Pairs).

Implementation Notes:
- I used the original version provided on the quora website. There's also a [Kaggle competition](https://www.kaggle.com/c/quora-question-pairs) which has a nice train/test split but I can't find an easy way to download it.
- I've made the questions into a list:
    ```python
    {
        "questions": [
            {"id":0, "text": "Is this an example question?"},
            {"id":1, "text": "Is this a sample question?"},
        ],
    ...
    }
    ```
    rather than:
    ```python
    {
        "question1": "Is this an example question?",
        "question2": "Is this a sample question?"
        "qid0": 0
        "qid1": 1
    ...
    }
    ```
    Not sure if this was the right call.
- Can't find a good citation for this dataset

Comments:
Tests seem to be failing because of pandas

Kaggle needs authentification to download datasets. We don't have a way to handle that in the lib for now
Title: How to augment data ?
Author: astariul (NONE)
State: closed
Created: 2020-07-09T07:52:37Z

Is there any clean way to augment data ?

For now my work-around is to use batched map, like this :

```python
def aug(samples):
    # Simply copy the existing data to have x2 amount of data
    for k, v in samples.items():
        samples[k].extend(v)
    return samples

dataset = dataset.map(aug, batched=True)
```

Comments:
Using batched map is probably the easiest way at the moment.
What kind of augmentation would you like to do ?

Some samples in the dataset are too long, I want to divide them in several samples.

Using batched map is the way to go then.
We'll make it clearer in the docs that map could be used for augmentation.

Let me know if you think there should be another way to do it. Or feel free to close the issue otherwise.

It just feels awkward to use map to augment data. Also it means it's not possible to augment data in a non-batched way.

But to be honest I have no idea of a good API...

Or for non-batched samples, how about returning a tuple ?

```python
def aug(sample):
    # Simply copy the existing data to have x2 amount of data
    return sample, sample

dataset = dataset.map(aug)
```

It feels really natural and easy, but :

* it means the behavior with batched data is different
* I don't know how doable it is backend-wise

@lhoestq 

As we're working with arrow's columnar format we prefer to play with batches that are dictionaries instead of tuples.
If we have tuple it implies to re-format the data each time we want to write to arrow, which can lower the speed of map for example.

It's also a matter of coherence, as we don't want users to be confused whether they have to return dictionaries for some functions and tuples for others when they're doing batches.
Title: add MS MARCO dataset
Author: mariamabarham (CONTRIBUTOR)
State: closed
Created: 2020-07-09T07:11:19Z

This PR adds the MS MARCO dataset as requested in this issue #336. MS mARCO has multiple task including:

- Passage and Document Retrieval

- Keyphrase Extraction

- QA and NLG

This PR only adds the 2 versions of the  QA and NLG task dataset which was realeased with the original paper here https://arxiv.org/pdf/1611.09268.pdf 

Tests are failing because of the dummy data. I tried to fix it without success. Can you please have a look at it? @patrickvonplaten , @lhoestq

Comments:
The dummy data for v2.1 is missing as far as I can see. I think running the  dummy data command should work correctly here. 

Also, it might be that the structure of the dummy data is wrong - looking at `generate_examples` the structure does not look too easy.

The fact that the dummy data for v2.1 is missing shouldn't make the test fails I think. But as you mention the dummy data structure of v1.1 is wrong. I tried to rename files but it does not solve the issue.

Is MS mARCO added to nlp library?I am not able to view it?

> Is MS mARCO added to nlp library?I am not able to view it?

Hi @parthplc ,the PR is  not merged yet. The dummy data structure is still failing. Maybe @patrickvonplaten can help with it.

Dataset is fixed and should be ready for use. @mariamabarham @lhoestq feel free to merge whenever!

> Dataset is fixed and should be ready for use. @mariamabarham @lhoestq feel free to merge whenever!

thanks
Title: [dateset subset missing]  xtreme paws-x
Author: cosmeowpawlitan (CONTRIBUTOR)
State: closed
Created: 2020-07-09T05:04:54Z

I tried nlp.load_dataset('xtreme', 'PAWS-X.es') but get the value error
It turns out that the subset for Spanish is missing
https://github.com/google-research-datasets/paws/tree/master/pawsx

Comments:
You're right, thanks for pointing it out. We will update it
Title: 🐛 [Metrics] ROUGE is non-deterministic
Author: astariul (NONE)
State: closed
Created: 2020-07-09T04:39:37Z

If I run the ROUGE metric 2 times, with same predictions / references, the scores are slightly different.

Refer to [this Colab notebook](https://colab.research.google.com/drive/1wRssNXgb9ldcp4ulwj-hMJn0ywhDOiDy?usp=sharing) for reproducing the problem.

Example of F-score for ROUGE-1, ROUGE-2, ROUGE-L in 2 differents run :

> ['0.3350', '0.1470', '0.2329']
['0.3358', '0.1451', '0.2332']

---

Why ROUGE is not deterministic ?

Comments:
Hi, can you give a full self-contained example to reproduce this behavior?

> Hi, can you give a full self-contained example to reproduce this behavior?

There is a notebook in the post ;)

> If I run the ROUGE metric 2 times, with same predictions / references, the scores are slightly different.
> 
> Refer to [this Colab notebook](https://colab.research.google.com/drive/1wRssNXgb9ldcp4ulwj-hMJn0ywhDOiDy?usp=sharing) for reproducing the problem.
> 
> Example of F-score for ROUGE-1, ROUGE-2, ROUGE-L in 2 differents run :
> 
> > ['0.3350', '0.1470', '0.2329']
> > ['0.3358', '0.1451', '0.2332']
> 
> Why ROUGE is not deterministic ?

This is because of rouge's `BootstrapAggregator` that uses sampling to get confidence intervals (low, mid, high).
You can get deterministic scores per sentence pair by using
```python
score = rouge.compute(rouge_types=["rouge1", "rouge2", "rougeL"], use_aggregator=False)
```
Or you can set numpy's random seed if you still want to use the aggregator.

Maybe we can set all the random seeds of numpy/torch etc. while running `metric.compute` ?

We should probably indeed!

Now if you re-run the notebook, the two printed results are the same @colanim
```
['0.3356', '0.1466', '0.2318']
['0.3356', '0.1466', '0.2318']
```
However across sessions, the results may change (as numpy's random seed can be different). You can prevent that by setting your seed:
```python
rouge = nlp.load_metric('rouge', seed=42)
```

> 

Minor nit: Note that "aggregator" is misspelled in this command. Should be `use_aggregator=False`.  

Thanks, I fixed the code snippet
Title: Starting to add some real doc
Author: thomwolf (MEMBER)
State: closed
Created: 2020-07-08T22:53:03Z

Adding a lot of documentation for:
- load a dataset
- explore the dataset object
- process data with the dataset
- add a new dataset script
- share a dataset script
- full package reference

This version of the doc can be explored here: https://2219-250213286-gh.circle-artifacts.com/0/docs/_build/html/index.html

Also:
- fix a bug in `train_test_split`
- update the `csv` script
- add a verbose argument to the dataset processing methods

Still missing:
- doc for the metrics
- how to directly upload a community provided dataset with the CLI
- clean up more docstrings
- add the `features` argument to `load_dataset` (should be another PR)

Comments:
Ok this is starting to be really big so it's probably good to merge this first version of the doc and continue in another PR :)

This first version of the doc can be explored here: https://2219-250213286-gh.circle-artifacts.com/0/docs/_build/html/index.html
Title: Add hashes to cnn_dailymail
Author: jbragg (CONTRIBUTOR)
State: closed
Created: 2020-07-08T22:45:21Z

The URL hashes are helpful for comparing results from other sources.

Comments:
Looks you to me :)

Could you also update the json file that goes with the dataset script by doing 
```
nlp-cli test ./datasets/cnn_dailymail --save_infos --all_configs
```
It will update the features metadata and the size of the dataset with your changes.

@lhoestq I ran that command.

Thanks for the helpful repository!
Title: Add text dataset
Author: jarednielsen (CONTRIBUTOR)
State: closed
Created: 2020-07-08T19:21:53Z

Usage:

```python
from nlp import load_dataset
dset = load_dataset("text", data_files="/path/to/file.txt")["train"]
```


I created a dummy_data.zip which contains three files: `train.txt`, `test.txt`, `dev.txt`. Each of these contains two lines. It passes

```bash
RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_text
```

but I would like a second set of eyes to ensure I did it right.

Comments:
Title: can't load SNLI dataset
Author: jxmorris12 (CONTRIBUTOR)
State: closed
Created: 2020-07-08T16:54:14Z

`nlp` seems to load `snli` from some URL based on nlp.stanford.edu. This subdomain is frequently down -- including right now, when I'd like to load `snli` in a Colab notebook, but can't.

Is there a plan to move these datasets to huggingface servers for a more stable solution?

Btw, here's the stack trace:

```
File "/content/nlp/src/nlp/builder.py", line 432, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/content/nlp/src/nlp/builder.py", line 466, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/content/nlp/src/nlp/datasets/snli/e417f6f2e16254938d977a17ed32f3998f5b23e4fcab0f6eb1d28784f23ea60d/snli.py", line 76, in _split_generators
    dl_dir = dl_manager.download_and_extract(_DATA_URL)
  File "/content/nlp/src/nlp/utils/download_manager.py", line 217, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/content/nlp/src/nlp/utils/download_manager.py", line 156, in download
    lambda url: cached_path(url, download_config=self._download_config,), url_or_urls,
  File "/content/nlp/src/nlp/utils/py_utils.py", line 190, in map_nested
    return function(data_struct)
  File "/content/nlp/src/nlp/utils/download_manager.py", line 156, in <lambda>
    lambda url: cached_path(url, download_config=self._download_config,), url_or_urls,
  File "/content/nlp/src/nlp/utils/file_utils.py", line 198, in cached_path
    local_files_only=download_config.local_files_only,
  File "/content/nlp/src/nlp/utils/file_utils.py", line 356, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://nlp.stanford.edu/projects/snli/snli_1.0.zip
```

Comments:
I just added the processed files of `snli` on our google storage, so that when you do `load_dataset` it can download the processed files from there :)

We are thinking about having available those processed files for more datasets in the future, because sometimes files aren't available (like for `snli`), or the download speed is too slow, or sometimes the files take time to be processed.

Closing this one. Feel free to re-open if you have other questions :)

Thank you!
Title: More faiss control
Author: lhoestq (MEMBER)
State: closed
Created: 2020-07-08T14:45:20Z

Allow users to specify a faiss index they created themselves, as sometimes indexes can be composite for examples

Comments:
> Ok, so we're getting rid of the `FaissGpuOptions`?

We support `device=...` because it's simple, but faiss GPU options can be used in so many ways (you can set different gpu options for the different parts of your index for example) that it's probably better to let the user create and configure its index and then use `custom_index=...`
Title: add from_pandas and from_dict
Author: lhoestq (MEMBER)
State: closed
Created: 2020-07-07T15:03:53Z

I added two new methods to the `Dataset` class:
- `from_pandas()` to create a dataset from a pandas dataframe
- `from_dict()` to create a dataset from a dictionary (keys = columns)

It uses the `pa.Table.from_pandas` and `pa.Table.from_pydict` funcitons to do so.
It is also possible to specify the features types via `features=...` if there are ambiguities (null/nan values), otherwise the arrow schema is infered from the data automatically by pyarrow.

One question that I have right now:
+ Should we also add a `save()` method that would write the dataset on the disk ? Right now if we create a `Dataset` using those two new methods, the data are kept in RAM. Then to reload it we can call the `from_file()` method.

Comments:
Title: Hyperpartisan news detection
Author: ghomasHudson (CONTRIBUTOR)
State: closed
Created: 2020-07-07T11:06:37Z

Adding the hyperpartisan news detection dataset from PAN. This contains news article text, labelled with whether they're hyper-partisan and why kinds of biases they display.

Implementation notes:
- As with many PAN tasks, the data is hosted on [Zenodo](https://zenodo.org/record/1489920) and must be requested before use. I've used the manual download stuff for this, although the dataset is provided under a Creative Commons Attribution 4.0 International License, so we could host a version if we wanted to?
- The 'bias' attribute doesn't exist for the 'byarticle' configuration. I've added an empty string to the class labels to deal with this. Is there a more standard value for empty data?
- Should we always subclass `nlp.BuilderConfig`?

Comments:
Thank you so much for working on this! This is awesome!

How much would it help you if we would remove the manual request?

We are naturally interested in getting some broad idea of how many people and who are using our dataset. But if you consider hosting the dataset yourself, I would rather remove this small barrier on our side (so that we then still get the download count from your library).

This is an interesting aspect indeed!
Do you want to send me an email (see my homepage) and I'll invite you on our slack channel to talk about that?
@ghomasHudson wanna reach out to me as well? I tried to find your email to invite you without success.
Title: Search qa
Author: mariamabarham (CONTRIBUTOR)
State: closed
Created: 2020-07-06T12:23:16Z

This PR adds the Search QA dataset used in **SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine**. The dataset has the following config name:

- raw_jeopardy: raw data

- train_test_val:  which is the splitted version

#336

Comments:
Could you rebase from master just to make sure we won't break anything for `fever` pls @mariamabarham ?
Title: Fix nested tensorflow format
Author: lhoestq (MEMBER)
State: closed
Created: 2020-07-06T10:13:45Z

In #339 and #337 we are thinking about adding a way to export datasets to tfrecords.

However I noticed that it was not possible to do `dset.set_format("tensorflow")` on datasets with nested features like `squad`. I fixed that using a nested map operations to convert features to `tf.ragged.constant`.

I also added tests on the `set_format` function.

Comments:
Title: Features should be updated when `map()` changes schema
Author: thomwolf (MEMBER)
State: closed
Created: 2020-07-06T08:03:23Z

`dataset.map()` can change the schema and column names.

We should update the features in this case (with what is possible to infer).

Comments:
`dataset.column_names` are being updated but `dataset.features` aren't indeed...
Title: [Feature request] Export Arrow dataset to TFRecords
Author: jarednielsen (CONTRIBUTOR)
State: closed
Created: 2020-07-02T15:47:12Z

The TFRecord generation process is error-prone and requires complex separate Python scripts to download and preprocess the data. I propose to combine the user-friendly features of `nlp` with the speed and efficiency of TFRecords. Sample API:

```python
# use these existing methods
ds = load_dataset("wikitext", "wikitext-2-raw-v1", split="train")
ds = ds.map(lambda ex: tokenizer(ex))
ds.set_format("tensorflow", columns=["input_ids", "token_type_ids", "attention_mask"])
# then add this method
ds.export(folder="/my/tfrecords", prefix="myrecord", num_shards=8, format="tfrecord")
```
which would create files like so:
```bash
/my/tfrecords/myrecord_1.tfrecord
/my/tfrecords/myrecord_2.tfrecord
...
```

I would be happy to contribute this method. We could use a similar approach for PyTorch. Thoughts?

Comments:
Title: [Dataset requests] New datasets for Open Question Answering
Author: thomwolf (MEMBER)
State: closed
Created: 2020-07-02T13:03:03Z
Labels: help wanted

dataset request

We are still a few datasets missing for Open-Question Answering which is currently a field in strong development.

Namely, it would be really nice to add:
- WebQuestions (Berant et al., 2013) [done]
- CuratedTrec (Baudis et al. 2015) [not open-source]
- MS-MARCO (NGuyen et al. 2016) [done]
- SearchQA (Dunn et al. 2017) [done]
- FEVER (Thorne et al. 2018) - [ done]

 

All these datasets are cited in http://arxiv.org/abs/2005.11401

Comments:
Title: fix variable name typo
Author: stas00 (CONTRIBUTOR)
State: closed
Created: 2020-07-01T19:13:50Z


Comments:
Good catch :)
I think there is another occurence that needs to be fixed in the second gist (line 4924 of the notebook file):
```python
bleu = nlp.load_metric(...)
```

Was fixed in e16f79b5f7fc12a6a30c777722be46897a272e6f
Closing it.
Title: Add wiki_dpr
Author: lhoestq (MEMBER)
State: closed
Created: 2020-07-01T17:12:00Z

Presented in the [Dense Passage Retrieval paper](https://arxiv.org/pdf/2004.04906.pdf), this dataset consists in 21M passages from the english wikipedia along with their 768-dim embeddings computed using DPR's context encoder.

Note on the implementation:
- There are two configs: with and without the embeddings (73GB vs 14GB)
- I used a non-fixed-size sequence of floats to describe the feature format of the embeddings. I wanted to use fixed-size sequences but I had issues with reading the arrow file afterwards (for example `dataset[0]` was crashing)
- I added the case for lists of urls as input of the download_manager

Comments:
The two configurations don't have the same sizes, I may change that so that they both have 21015300 examples for convenience, even though it's supposed to have 21015324 examples in total.

One configuration only has 21015300 examples because it seems that the embeddings of the last 24 examples are missing.

It's ok to merge now imo. I'll make another PR if we find a way to have the missing embeddings
Title: Doc red
Author: ghomasHudson (CONTRIBUTOR)
State: closed
Created: 2020-06-30T22:05:31Z

Adding [DocRED](https://github.com/thunlp/DocRED) - a relation extraction dataset which tests document-level RE. A few implementation notes:

- There are 2 separate versions of the training set - *annotated* and *distant*. Instead of `nlp.Split.Train` I've used the splits `"train_annotated"` and `"train_distant"` to reflect this.
- As well as the relation id, the full relation name is mapped from `rel_info.json`
- I renamed the 'h', 'r', 't' keys to 'head', 'relation' and 'tail' to make them more readable.
- Used the fix from #319 to allow nested sequences of dicts.

Comments:
Title: Add SQuADShifts dataset
Author: millerjohnp (CONTRIBUTOR)
State: closed
Created: 2020-06-29T19:11:16Z

This PR adds the four new variants of the SQuAD dataset used in [The Effect of Natural Distribution Shift on Question Answering Models](https://arxiv.org/abs/2004.14444) to facilitate evaluating model robustness to distribution shift.

Comments:
Very cool to have this dataset, thank you for adding it :)
Title: Add package path to sys when downloading package as github archive
Author: yjernite (MEMBER)
State: closed
Created: 2020-06-29T16:46:01Z

This fixes the `coval.py` metric so that imports within the downloaded module work correctly. We can use a similar trick to add the BLEURT metric (@ankparikh)

@thomwolf not sure how you feel about adding to the `PYTHONPATH` from the script. This is the only way I could make it work with my understanding of `importlib` but there might be a more elegant method.

This PR fixes https://github.com/huggingface/nlp/issues/305

Comments:
Sorry for the long diff, everything after the imports comes from `black` for code quality :/ 

 I think it's fine and I can't think of another way to make the import work anyways.

Maybe we can have the `sys.path` behavior inside `prepare_module` instead ? Currently it seems to come out of nowhere in the code ^^'
We could check if external imports have a `__init__.py` and if it is the case then we can add to directory to the `PYTHONPATH`
Title: output nested dict in get_nearest_examples
Author: lhoestq (MEMBER)
State: closed
Created: 2020-06-29T15:47:47Z

As we are using a columnar format like arrow as the backend for datasets, we expect to have a dictionary of columns when we slice a dataset like in this example:
```python
my_examples = dataset[0:10]
print(type(my_examples))
# >>> dict
print(my_examples["my_column"][0]
# >>> this is the first element of the column 'my_column'
```

Therefore I wanted to keep this logic when calling `get_nearest_examples` that returns the top 10 nearest examples:
```python
dataset.add_faiss_index(column="embeddings")
scores, examples = dataset.get_nearest_examples("embeddings", query=my_numpy_embedding)
print(type(examples))
# >>> dict
```

Previously it was returning a list[dict]. It was the only place that was using this output format.

To make it work I had to implement `__getitem__(key)` where `key` is a list.
This is different from `.select` because `.select` is a dataset transform (it returns a new dataset object) while `__getitem__` is an extraction method (it returns python dictionaries).

Comments:
Title: Blog Authorship Corpus, Non Matching Splits Sizes Error, nlp viewer
Author: mariamabarham (CONTRIBUTOR)
State: closed
Created: 2020-06-29T07:36:35Z
Labels: nlp-viewer

Selecting `blog_authorship_corpus` in the nlp viewer throws the following error: 

```
NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=610252351, num_examples=532812, dataset_name='blog_authorship_corpus'), 'recorded': SplitInfo(name='train', num_bytes=614706451, num_examples=535568, dataset_name='blog_authorship_corpus')}, {'expected': SplitInfo(name='validation', num_bytes=37500394, num_examples=31277, dataset_name='blog_authorship_corpus'), 'recorded': SplitInfo(name='validation', num_bytes=32553710, num_examples=28521, dataset_name='blog_authorship_corpus')}]
Traceback:
File "/home/sasha/streamlit/lib/streamlit/ScriptRunner.py", line 322, in _run_script
    exec(code, module.__dict__)
File "/home/sasha/nlp-viewer/run.py", line 172, in <module>
    dts, fail = get(str(option.id), str(conf_option.name) if conf_option else None)
File "/home/sasha/streamlit/lib/streamlit/caching.py", line 591, in wrapped_func
    return get_or_create_cached_value()
File "/home/sasha/streamlit/lib/streamlit/caching.py", line 575, in get_or_create_cached_value
    return_value = func(*args, **kwargs)
File "/home/sasha/nlp-viewer/run.py", line 132, in get
    builder_instance.download_and_prepare()
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/nlp/builder.py", line 432, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/nlp/builder.py", line 488, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/nlp/utils/info_utils.py", line 70, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
```
@srush @lhoestq

Comments:
I wonder if this means downloading failed? That corpus has a really slow server.

This dataset seems to have a decoding problem that results in inconsistencies in the number of generated examples.
See #215.
That's why we end up with a `NonMatchingSplitsSizesError `.
Title: Nested sequences with dicts
Author: ghomasHudson (CONTRIBUTOR)
State: closed
Created: 2020-06-27T23:45:17Z

Am pretty much finished [adding a dataset](https://github.com/ghomasHudson/nlp/blob/DocRED/datasets/docred/docred.py) for [DocRED](https://github.com/thunlp/DocRED), but am getting an error when trying to add a nested `nlp.features.sequence(nlp.features.sequence({key:value,...}))`. 

The original data is in this format:
```python
{
  'title': "Title of wiki page",
  'vertexSet': [
                  [
                    { 'name': "mention_name", 
                      'sent_id': "mention in which sentence", 
                      'pos': ["postion of mention in a sentence"], 
                      'type': "NER_type"},
                    {another mention}
                  ], 
                  [another entity]
                ]
    ...
}
```
So to represent this I've attempted to write:
```
...
features=nlp.Features({
    "title": nlp.Value("string"),
    "vertexSet": nlp.features.Sequence(nlp.features.Sequence({
        "name": nlp.Value("string"),
        "sent_id": nlp.Value("int32"),
        "pos": nlp.features.Sequence(nlp.Value("int32")),
        "type": nlp.Value("string"),
    })),
    ...
    }),
...
```
This is giving me the error:
```
pyarrow.lib.ArrowTypeError: Could not convert [{'pos': [[0,2], [2,4], [3,5]], "type": ["ORG", "ORG", "ORG"], "name": ["Lark Force", "Lark Force", "Lark Force", "sent_id": [0, 3, 4]}..... with type list: was not a dict, tuple, or recognized null value for conversion to struct type
```
Do we expect the pyarrow stuff to break when doing this deeper nesting? I've checked that it still works when you do `nlp.features.Sequence(nlp.features.Sequence(nlp.Value("string"))` or `nlp.features.Sequence({key:value,...})` just not nested sequences with a dict.

If it's not possible, I can always convert it to a shallower structure. I'd rather not change the DocRED authors' structure if I don't have to though.

Comments:
Oh yes, this is a backward compatibility feature with tensorflow_dataset in which a `Sequence` or `dict` is converted in a `dict` of `lists`, unfortunately it is not very intuitive, see here: https://github.com/huggingface/nlp/blob/master/src/nlp/features.py#L409

To avoid this behavior, you can just define the list in the feature with a simple list or a tuple (which is also simpler to write).
In your case, the features could be as follow:
``` python
...
features=nlp.Features({
    "title": nlp.Value("string"),
    "vertexSet": [[{
        "name": nlp.Value("string"),
        "sent_id": nlp.Value("int32"),
        "pos": nlp.features.Sequence(nlp.Value("int32")),
        "type": nlp.Value("string"),
    }]],
    ...
    }),
...
```
Title: Adding a dataset with multiple subtasks
Author: erickrf (NONE)
State: closed
Created: 2020-06-26T23:14:19Z

I intent to add the datasets of the MT Quality Estimation shared tasks to `nlp`. However, they have different subtasks -- such as word-level, sentence-level and document-level quality estimation, each of which having different language pairs, and some of the data reused in different subtasks.

For example, in [QE 2019,](http://www.statmt.org/wmt19/qe-task.html) we had the same English-Russian and English-German data for word-level and sentence-level QE. 

I suppose these datasets could have both their word and sentence-level labels inside `nlp.Features`; but what about other subtasks? Should they be considered a different dataset altogether?

I read the discussion on #217 but the case of QE seems a lot simpler.

Comments:
For one dataset you can have different configurations that each have their own `nlp.Features`.
We imagine having one configuration per subtask for example.
They are loaded with `nlp.load_dataset("my_dataset", "my_config")`.

For example the `glue` dataset has many configurations. It is a bit different from your case though because each configuration is a dataset by itself (sst2, mnli).
Another example is `wikipedia` that has one configuration per language.
Title: Add MWSC
Author: ghomasHudson (CONTRIBUTOR)
State: closed
Created: 2020-06-25T09:22:02Z

Adding the [Modified Winograd Schema Challenge](https://github.com/salesforce/decaNLP/blob/master/local_data/schema.txt) dataset which formed part of the [decaNLP](http://decanlp.com/) benchmark. Not sure how much use people would find for it it outside of the benchmark, but it is general purpose.

Code is heavily borrowed from the [decaNLP repo](https://github.com/salesforce/decaNLP/blob/1e9605f246b9e05199b28bde2a2093bc49feeeaa/text/torchtext/datasets/generic.py#L773-L877).

There's a few (possibly overly opinionated) design choices I made:

- I used the train/test/dev split [buried in the decaNLP code](https://github.com/salesforce/decaNLP/blob/1e9605f246b9e05199b28bde2a2093bc49feeeaa/text/torchtext/datasets/generic.py#L852-L855)
- I split out each example into the 2 alternatives. Originally the data uses the format:
    ```
    The city councilmen refused the demonstrators a permit because they [feared/advocated] violence. 
    Who [feared/advocated] violence? 
    councilmen/demonstrators
    ```
    I split into the 2 variants:
    ```
    The city councilmen refused the demonstrators a permit because they feared violence. 
    Who feared violence? 
    councilmen/demonstrators
    
    The city councilmen refused the demonstrators a permit because they advocated violence. 
    Who advocated violence? 
    councilmen/demonstrators
    ```
    I can't see any use for having the options combined into a single example (splitting them is [the way decaNLP processes](https://github.com/salesforce/decaNLP/blob/1e9605f246b9e05199b28bde2a2093bc49feeeaa/text/torchtext/datasets/generic.py#L846-L850)) them. You can't train on both versions with them combined, and splitting the examples later would be a pain to do. I think [winogrande.py](https://github.com/huggingface/nlp/blob/master/datasets/winogrande/winogrande.py) presents the data in this way?

- I've not used the decaNLP framing (appending the options to the question e.g. `Who feared violence? 
 -- councilmen or demonstrators?`) but left it more generic by adding the options as a new key: `"options":["councilmen","demonstrators"]` This should be an easy thing to change using `map` if needed by a specific application.

Dataset is working as-is but if anyone has any thoughts/preferences on the design decisions here I'm definitely open to different choices.

Comments:
Looks good to me
Title: [Feature request] Add `shard()` method to dataset
Author: jarednielsen (CONTRIBUTOR)
State: closed
Created: 2020-06-24T22:48:33Z

Currently, to shard a dataset into 10 pieces on different ranks, you can run

```python
rank = 3 # for example
size = 10
dataset = nlp.load_dataset('wikitext', 'wikitext-2-raw-v1', split=f"train[{rank*10}%:{(rank+1)*10}%]")
```

However, this breaks down if you have a number of ranks that doesn't divide cleanly into 100, such as 64 ranks. Is there interest in adding a method shard() that looks like this?

```python
rank = 3
size = 64
dataset = nlp.load_dataset("wikitext", "wikitext-2-raw-v1", split="train").shard(rank=rank, size=size)
```

TensorFlow has a similar API: https://www.tensorflow.org/api_docs/python/tf/data/Dataset#shard. I'd be happy to contribute this code.

Comments:
Hi Jared,
Interesting, thanks for raising this question. You can also do that after loading with `dataset.select()` or `dataset.filter()` which let you keep only a specific subset of rows in a dataset.
What is your use-case for sharding?

Thanks for the pointer to those functions! It's still a little more verbose since you have to manually calculate which ids each rank would keep, but definitely works.

My use case is multi-node, multi-GPU training and avoiding global batches of duplicate elements. I'm using horovod. You can shuffle indices, or set random seeds, but explicitly sharding the dataset up front is the safest and clearest way I've found to do so.
Title: Add qa_zre
Author: ghomasHudson (CONTRIBUTOR)
State: closed
Created: 2020-06-24T22:17:22Z

Adding the QA-ZRE dataset from ["Zero-Shot Relation Extraction via Reading Comprehension"](http://nlp.cs.washington.edu/zeroshot/).

A common processing step seems to be replacing the `XXX` placeholder with the `subject`. I've left this out as it's something you could easily do with `map`.

Comments:
Title: add wikisql
Author: ghomasHudson (CONTRIBUTOR)
State: closed
Created: 2020-06-24T18:00:35Z

Adding the [WikiSQL](https://github.com/salesforce/WikiSQL) dataset.

Interesting things to note:
- Have copied the function (`_convert_to_human_readable`) which converts the SQL query to a human-readable (string) format as this is what most people will want when actually using this dataset for NLP applications.
- `conds` was originally a tuple but is converted to a dictionary to support differing types.

Would be nice to add the logical_form metrics too at some point.

Comments:
That's great work @ghomasHudson !
Title: add pg19 dataset
Author: lucidrains (CONTRIBUTOR)
State: closed
Created: 2020-06-23T22:03:52Z

https://github.com/huggingface/nlp/issues/274

Add functioning PG19 dataset with dummy data

`cos_e.py` was just auto-linted by `make style`

Comments:
@lucidrains - Thanks a lot for making the PR - PG19 is a super important dataset! Thanks for making it. Many people are asking for PG-19, so it would be great to have that in the library as soon as possible @thomwolf .

@mariamabarham yup! around 11GB!

I'm looking forward to our first deep learning written novel already lol. It's definitely happening

Good to merge IMO.

Oh I just noticed but as we changed the urls to download the files, we have to update `dataset_infos.json`.
Could you re-rurn `nlp-cli test ./datasets/pg19 --save_infos` ?

@lhoestq on it!

should be good!

@lhoestq - I think it's good to merge no?

`dataset_infos.json` is still not up to date with the new urls (we can see that there are urls like `gs://deepmind-gutenberg/train/*` instead of `https://storage.googleapis.com/deepmind-gutenberg/train/*` in the json file)

Can you check that you re-ran the command to update the json file, and that you pushed the changes @lucidrains ?

@lhoestq ohhh, I made the change in this commit https://github.com/lucidrains/nlp/commit/f3e23d823ad9942031be80b7c4e4212c592cd90c , that's interesting that the pull request didn't pick it up. maybe it's because I did it on another machine, let me check and get back to you!

@lhoestq wrong branch 😅 thanks for catching! 

Awesome thanks 🎉
Title: Importing downloaded package repository fails
Author: yjernite (MEMBER)
State: closed
Created: 2020-06-23T21:09:05Z
Labels: metric bug

The `get_imports` function in `src/nlp/load.py` has a feature to download a package as a zip archive of the github repository and import functions from the unpacked directory. This is used for example in the `metrics/coval.py` file, and would be useful to add BLEURT (@ankparikh).

Currently however, the code seems to have trouble with imports within the package. For example:
```
import nlp
coval = nlp.load_metric('coval')
```
yields:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/yacine/Code/nlp/src/nlp/load.py", line 432, in load_metric
    metric_cls = import_main_class(module_path, dataset=False)
  File "/home/yacine/Code/nlp/src/nlp/load.py", line 57, in import_main_class
    module = importlib.import_module(module_path)
  File "/home/yacine/anaconda3/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/yacine/Code/nlp/src/nlp/metrics/coval/a78807df33ac45edbb71799caf2b3b47e55df4fd690267808fe963a5e8b30952/coval.py", line 21, in <module>
    from .coval_backend.conll import reader  # From: https://github.com/ns-moosavi/coval
  File "/home/yacine/Code/nlp/src/nlp/metrics/coval/a78807df33ac45edbb71799caf2b3b47e55df4fd690267808fe963a5e8b30952/coval_backend/conll/reader.py", line 2, in <module>
    from conll import mention
ModuleNotFoundError: No module named 'conll'
```

Not sure what the fix would be there.

Comments:
Title: Problem while printing doc string when instantiating multiple metrics.
Author: codehunk628 (CONTRIBUTOR)
State: closed
Created: 2020-06-23T19:32:05Z
Labels: metric bug

When I load more than one metric and try to print doc string of a particular metric,. It shows the doc strings of all imported metric one after the other which looks quite confusing and clumsy.
Attached [Colab](https://colab.research.google.com/drive/13H0ZgyQ2se0mqJ2yyew0bNEgJuHaJ8H3?usp=sharing) Notebook for problem clarification..

Comments:
Title: allow to move files across file systems
Author: lhoestq (MEMBER)
State: closed
Created: 2020-06-23T14:56:08Z

Users are allowed to use the `cache_dir` that they want.
Therefore it can happen that we try to move files across filesystems.
We were using `os.rename` that doesn't allow that, so I changed some of them to `shutil.move`.

This should fix #301

Comments:
Title: Question - Sign Language Datasets
Author: AmitMY (CONTRIBUTOR)
State: closed
Created: 2020-06-23T14:53:40Z
Labels: enhancement

generic discussion

An emerging field in NLP is SLP - sign language processing.

I was wondering about adding datasets here, specifically because it's shaping up to be large and easily usable.
The metrics for sign language to text translation are the same.

So, what do you think about (me, or others) adding datasets here?


An example dataset would be [RWTH-PHOENIX-Weather 2014 T](https://www-i6.informatik.rwth-aachen.de/~koller/RWTH-PHOENIX-2014-T/)
For every item in the dataset, the data object includes:
1. video_path - path to mp4 file
2. pose_path - a path to `.pose` file with human pose landmarks
3. openpose_path - a path to a `.json` file with human pose landmarks
4. gloss - string
5. text - string
6. video_metadata - height, width, frames, framerate


------

To make it a tad more complicated - what if sign language libraries add requirements to `nlp`? for example, sign language is commonly annotated using `ilex`, `eaf`, or `srt` files, which are all loadable as text, but there is no reason for the dataset to parse that file by itself, if libraries exist to do so.

Comments:
Even more complicating - 

As I see it, datasets can have "addons".
For example, the WebNLG dataset is a dataset for data-to-text. However, a work of mine and other works enriched this dataset with text plans / underlying text structures. In that case, I see a need to load the dataset "WebNLG" with "plans" addon.

Same for sign language - if there is a dataset of videos, one addon can be to run OpenPose, another to run ARKit4 pose estimation, and another to run PoseNet, or even just a video embedding addon. (which are expensive to run individually for everyone who wants to use these data)

This is something I dabbled with my own implementation to a [research datasets library](https://github.com/AmitMY/meta-scholar/) and I love to get the discussion going on these topics.

This is a really cool idea !
The example for data objects you gave for the RWTH-PHOENIX-Weather 2014 T dataset can totally fit inside the library.

For your point about formats like `ilex`, `eaf`, or `srt`, it is possible to use any library in your dataset script.
However most user probably won't need these libraries, as most datasets don't need them, and therefore it's unlikely that we will have them in the minimum requirements to use `nlp` (we want to keep it as light-weight as possible). If a user wants to load your dataset and doesn't have the libraries you need, an error is raised asking the user to install them.

More generally, we plan to have something like a `requirements.txt` per dataset. This could also be a place for addons as you said. What do you think ?

Thanks, Quentin, I think a `requirements.txt` per dataset will be a good thing.
I will work on adding this dataset next week, and once we sort all of the kinks, I'll add more.
Title: Setting cache_dir gives error on wikipedia download
Author: hallvagi (NONE)
State: closed
Created: 2020-06-23T11:31:44Z

First of all thank you for a super handy library! I'd like to download large files to a specific drive so I set `cache_dir=my_path`. This works fine with e.g. imdb and squad. But on wikipedia I get an error:
```
nlp.load_dataset('wikipedia', '20200501.de', split = 'train', cache_dir=my_path)
```
```
OSError                                   Traceback (most recent call last)
<ipython-input-2-23551344d7bc> in <module>
      1 import nlp
----> 2 nlp.load_dataset('wikipedia', '20200501.de', split = 'train', cache_dir=path)

~/anaconda3/envs/fastai2/lib/python3.7/site-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    522         download_mode=download_mode,
    523         ignore_verifications=ignore_verifications,
--> 524         save_infos=save_infos,
    525     )
    526 

~/anaconda3/envs/fastai2/lib/python3.7/site-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, try_from_hf_gcs, dl_manager, **download_and_prepare_kwargs)
    385                     with utils.temporary_assignment(self, "_cache_dir", tmp_data_dir):
    386                         reader = ArrowReader(self._cache_dir, self.info)
--> 387                         reader.download_from_hf_gcs(self._cache_dir, self._relative_data_dir(with_version=True))
    388                         downloaded_info = DatasetInfo.from_directory(self._cache_dir)
    389                         self.info.update(downloaded_info)

~/anaconda3/envs/fastai2/lib/python3.7/site-packages/nlp/arrow_reader.py in download_from_hf_gcs(self, cache_dir, relative_data_dir)
    231             remote_dataset_info = os.path.join(remote_cache_dir, "dataset_info.json")
    232             downloaded_dataset_info = cached_path(remote_dataset_info)
--> 233             os.rename(downloaded_dataset_info, os.path.join(cache_dir, "dataset_info.json"))
    234             if self._info is not None:
    235                 self._info.update(self._info.from_directory(cache_dir))

OSError: [Errno 18] Invalid cross-device link: '/home/local/NTU/nn/.cache/huggingface/datasets/025fa4fd4f04aaafc9e939260fbc8f0bb190ce14c61310c8ae1ddd1dcb31f88c.9637f367b6711a79ca478be55fe6989b8aea4941b7ef7adc67b89ff403020947' -> '/data/nn/nlp/wikipedia/20200501.de/1.0.0.incomplete/dataset_info.json'
```

Comments:
Whoops didn't mean to close this one.
I did some changes, could you try to run it from the master branch ?

Now it works, thanks!
Title: Fix bertscore references
Author: lhoestq (MEMBER)
State: closed
Created: 2020-06-23T09:38:59Z

I added some type checking for metrics. There was an issue where a metric could interpret a string a a list. A `ValueError` is raised if a string is given instead of a list.

Moreover I added support for both strings and lists of strings for `references` in `bertscore`, as it is the case in the original code.

Both ways work:
```
import nlp

scorer = nlp.load_metric("bertscore")
with open("pred.txt") as p, open("ref.txt") as g:
    for lp, lg in zip(p, g):
        scorer.add(lp, [lg])
score = scorer.compute(lang="en")
```

```
import nlp

scorer = nlp.load_metric("bertscore")
with open("pred.txt") as p, open("ref.txt") as g:
    for lp, lg in zip(p, g):
        scorer.add(lp, lg)
score = scorer.compute(lang="en")
```

This should fix #295 and #238

Comments:
Title: Error in Demo for Specific Datasets
Author: s-jse (NONE)
State: closed
Created: 2020-06-23T00:38:42Z
Labels: nlp-viewer

Selecting `natural_questions` or `newsroom` dataset in the online demo results in an error similar to the following.

![image](https://user-images.githubusercontent.com/60150701/85347842-ac861900-b4ae-11ea-98c4-a53a00934783.png)

Comments:
Thanks for reporting these errors :)

I can actually see two issues here.

First, datasets like `natural_questions` require apache_beam to be processed. Right now the import is not at the right place so we have this error message. However, even the imports are fixed, the nlp viewer doesn't actually have the resources to process NQ right now so we'll have to wait until we have a version that we've already processed on our google storage (that's what we've done for wikipedia for example).

Second, datasets like `newsroom` require manual downloads as we're not allowed to redistribute the data ourselves (if I'm not wrong). An error message should be displayed saying that we're not allowed to show the dataset.

I can fix the first issue with the imports but for the second one I think we'll have to see with @srush to show a message for datasets that require manual downloads (it can be checked whether a dataset requires manual downloads if `dataset_builder_instance.manual_download_instructions is not None`).



I added apache-beam to the viewer. We can think about how to add newsroom. 

We don't plan to host the source files of newsroom ourselves for now.
You can still get the dataset if you follow the download instructions given by `dataset = load_dataset('newsroom')` though.
The viewer also shows the instructions now.

Closing this one. If you have other questions, feel free to re-open :)
Title: snli -1 labels
Author: jxmorris12 (CONTRIBUTOR)
State: closed
Created: 2020-06-22T23:33:30Z

I'm trying to train a model on the SNLI dataset. Why does it have so many -1 labels?
```
import nlp
from collections import Counter
data = nlp.load_dataset('snli')['train']
print(Counter(data['label']))
Counter({0: 183416, 2: 183187, 1: 182764, -1: 785})
```

Comments:
@jxmorris12  , we use `-1` to label examples for which `gold label`  is missing (`gold label = -` in the original dataset). 

Thanks @mariamabarham! so the original dataset is missing some labels? That is weird. Is standard practice just to discard those examples training/eval?

Yes the original dataset is missing some labels maybe  @sleepinyourhat , @gangeli can correct me if I'm wrong 
For my personal opinion at least if you want your model to learn to predict no answer (-1) you can leave it their but otherwise you can discard them. 

thanks @mariamabarham  :)
Title: Improve input warning for evaluation metrics
Author: Tiiiger (NONE)
State: closed
Created: 2020-06-22T17:28:57Z

Hi, 

I am the author of `bert_score`. Recently, we received [ an issue ](https://github.com/Tiiiger/bert_score/issues/62) reporting a problem in using `bert_score` from the `nlp` package (also see #238 in this repo).  After looking into this, I realized that the problem arises from the format `nlp.Metric` takes input. 

Here is a minimal example:
```python
import nlp

scorer = nlp.load_metric("bertscore")
with open("pred.txt") as p, open("ref.txt") as g:
    for lp, lg in zip(p, g):
        scorer.add(lp, lg)
score = scorer.compute(lang="en")
```

The problem in the above code is that `scorer.add()` expects a list of strings as input for the references. As a result, the `scorer` here would take a list of characters in `lg` to be the references. The correct implementation would be calling
```python
scorer.add(lp, [lg])
```

I just want to raise this issue to you to prevent future user errors of a similar kind. I assume some simple type checking can prevent this from happening?

Thanks!

Comments: