rIsHu009/github-issues-updated huggingface dataset:
Title: cache_dir parameter for load_from_disk ?
Author: chbensch (NONE)
State: closed
Created: 2021-06-10T17:39:36Z
Labels: enhancement

**Is your feature request related to a problem? Please describe.**
When using Google Colab big datasets can be an issue, as they won't fit on the VM's disk. Therefore mounting google drive could be a possible solution. Unfortunatly when loading my own dataset by using the _load_from_disk_ function, the data gets cached to the VM's disk:

`
from datasets import load_from_disk

myPreprocessedData = load_from_disk("/content/gdrive/MyDrive/ASR_data/myPreprocessedData")

`
I know that chaching on google drive could slow down learning. But at least it would run.

**Describe the solution you'd like**
Add cache_Dir parameter to the load_from_disk function.

**Describe alternatives you've considered**
It looks like you could write a custom loading script for the load_dataset function. But this seems to be much too complex for my use case. Is there perhaps a template here that uses the load_from_disk function?

Comments:
Hi ! `load_from_disk` doesn't move the data. If you specify a local path to your mounted drive, then the dataset is going to be loaded directly from the arrow file in this directory. The cache files that result from `map` operations are also stored in the same directory by default.

However note than writing data to your google drive actually fills the VM's disk (see https://github.com/huggingface/datasets/issues/643)

Given that, I don't think that changing the cache directory changes anything.

Let me know what you think

Thanks for your answer! I am a little surprised since I just want to read the dataset.

After debugging a bit, I noticed that the VM’s disk fills up when the tables (generator) are converted to a list:

https://github.com/huggingface/datasets/blob/5ba149773d23369617563d752aca922081277ec2/src/datasets/table.py#L850

If I try to iterate through the table’s generator e.g.: 

`length = sum(1 for x in tables)`

the VM’s disk fills up as well.

I’m running out of Ideas 😄 

Indeed reading the data shouldn't increase the VM's disk. Not sure what google colab does under the hood for that to happen

Apparently, Colab uses a local cache of the data files read/written from Google Drive. See:
- https://github.com/googlecolab/colabtools/issues/2087#issuecomment-860818457
- https://github.com/googlecolab/colabtools/issues/1915#issuecomment-804234540
- https://github.com/googlecolab/colabtools/issues/2147#issuecomment-885052636
Title: Add Disfl-QA
Author: bhavitvyamalik (CONTRIBUTOR)
State: closed
Created: 2021-06-10T16:18:00Z

Dataset: https://github.com/google-research-datasets/disfl-qa

To-Do: Update README.md and add YAML tags

Comments:
Sounds great! It'll make things easier for the user while accessing the dataset. I'll make some changes to the current file then.

I've updated with the suggested changes. Updated the README, YAML tags as well (not sure of Size category tag as I couldn't pass the path of `dataset_infos.json` for this dataset)
Title: Fix automatic generation of Zenodo DOI
Author: albertvillanova (MEMBER)
State: closed
Created: 2021-06-10T15:15:46Z
Labels: bug

After the last release of Datasets (1.8.0), the automatic generation of the Zenodo DOI failed: it appears in yellow as "Received", instead of in green as "Published".

I have contacted Zenodo support to fix this issue.

TODO:
- [x] Check with Zenodo to fix the issue
- [x] Check BibTeX entry is right

Comments:
I have received a reply from Zenodo support:
> We are currently investigating and fixing this issue related to GitHub releases. As soon as we have solved it we will reach back to you.

Other repo maintainers had the same problem with Zenodo. 

There is an open issue on their GitHub repo: zenodo/zenodo#2181

I have received the following request from Zenodo support:
> Could you send us the link to the repository as well as the release tag?

My reply:
> Sure, here it is:
> - Link to the repository: https://github.com/huggingface/datasets
> - Link to the repository at the release tag: https://github.com/huggingface/datasets/releases/tag/1.8.0
> - Release tag: 1.8.0

Zenodo issue has been fixed. The 1.8.0 release DOI can be found here: https://zenodo.org/record/4946100#.YMd6vKj7RPY
Title: Fix PermissionError on Windows when using tqdm >=4.50.0
Author: albertvillanova (MEMBER)
State: closed
Created: 2021-06-10T08:31:49Z
Labels: bug

See: https://app.circleci.com/pipelines/github/huggingface/datasets/235/workflows/cfb6a39f-68eb-4802-8b17-2cd5e8ea7369/jobs/1111

```
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process
```

Comments:
Title: Bump tqdm version
Author: lewtun (MEMBER)
State: closed
Created: 2021-06-09T17:24:40Z


Comments:
i tried both the latest version of `tqdm` and the version required by `autonlp` - no luck with windows 😞 

it's very weird that a progress bar would trigger these kind of errors, so i'll have a look to see if it's something unique to `datasets`

Closing since this is now fixed in #2482
Title: change udpos features structure
Author: cosmeowpawlitan (CONTRIBUTOR)
State: closed
Created: 2021-06-09T08:03:31Z

The structure is change such that each example is a sentence
The change is done for issues:
#2061 
#2444 

Close #2061 , close #2444.

Comments:
Let's add the tags in another PR. Thanks again !

Close #2061 , close #2444.
Title: adding masahaner dataset
Author: dadelani (CONTRIBUTOR)
State: closed
Created: 2021-06-08T21:20:25Z

Adding Masakhane dataset https://github.com/masakhane-io/masakhane-ner 

@lhoestq , can you please review

Comments:
Thank you for the review. 

Thanks a lot for the corrections and comments. 

I have resolved point 2. The make style still throws some errors, please see below

black --line-length 119 --target-version py36 tests src benchmarks datasets/**/*.py metrics
/bin/sh: 1: black: not found
Makefile:13: recipe for target 'style' failed
make: *** [style] Error 127

Can you help to resolve this?

Thank you very much @lhoestq for the help.
Title: fix: adjusting indexing for the labels.
Author: drugilsberg (CONTRIBUTOR)
State: closed
Created: 2021-06-08T20:47:25Z

The labels index were mismatching the actual ones used in the dataset. Specifically `0` is used  for `SUPPORTS` and `1` is used for `REFUTES`
After this change, the `README.md` now reflects the content of `dataset_infos.json`.

Signed-off-by: Matteo Manica <drugilsberg@gmail.com>

Comments:
> Good catch ! Thanks for fixing it

My pleasure🙏
Title: Merge DatasetDict and Dataset
Author: albertvillanova (MEMBER)
State: open
Created: 2021-06-08T19:22:04Z
Labels: enhancement

generic discussion

As discussed in #2424 and #2437 (please see there for detailed conversation):
- It would be desirable to improve UX with respect the confusion between DatasetDict and Dataset.
- The difference between Dataset and DatasetDict is an additional abstraction complexity that confuses "typical" end users. 
- A user expects a "Dataset" (whatever it contains multiple or a single split) and maybe it could be interesting to try to simplify the user-facing API as much as possible to hide this complexity from the end user.

Here is a proposal for discussion and refined (and potential abandon if it's not good enough):
- let's consider that a DatasetDict is also a Dataset with the various split concatenated one after the other
- let's disallow the use of integers in split names (probably not a very big breaking change)
- when you index with integers you access the examples progressively in split after the other is finished (in a deterministic order)
- when you index with strings/split name you have the same behavior as now (full backward compat)
- let's then also have all the methods of a Dataset on the DatasetDict

The end goal would be to merge both Dataset and DatasetDict object in a single object that would be (pretty much totally) backward compatible with both.


There are a few things that we could discuss if we want to merge Dataset and DatasetDict:

1. what happens if you index by a string ? Does it return the column or the split ? We could disallow conflicts between column names and split names to avoid ambiguities. It can be surprising to be able to get a column or a split using the same indexing feature
   ```
   from datasets import load_dataset

   dataset = load_dataset(...)
   dataset["train"]
   dataset["input_ids"]
   ```
2. what happens when you iterate over the object ? I guess it should iterate over the examples as a Dataset object, but a DatasetDict used to iterate over the splits as they are the dictionary keys. This is a breaking change that we can discuss.

Moreover regarding your points:

- integers are not allowed as split names already
- it's definitely doable to have all the methods. Maybe some of them like train_test_split that is currently only available for Dataset can be tweaked to work for a split dataset


cc: @thomwolf @lhoestq

Comments:
Any update on this? @lhoestq 

Unless there is high demande I don't think we will end up implementing this. This is a lot of work with very few advantages
Title: Support sliced list arrays in cast
Author: lhoestq (MEMBER)
State: closed
Created: 2021-06-08T17:38:47Z

There is this issue in pyarrow:
```python
import pyarrow as pa

arr = pa.array([[i * 10] for i in range(4)])
arr.cast(pa.list_(pa.int32()))  # works

arr = arr.slice(1)
arr.cast(pa.list_(pa.int32()))  # fails
# ArrowNotImplementedError("Casting sliced lists (non-zero offset) not yet implemented")
```

However in `Dataset.cast` we slice tables to cast their types (it's memory intensive), so we have the same issue.
Because of this it is currently not possible to cast a Dataset with a Sequence feature type (unless the table is small enough to not be sliced).

In this PR I fixed this by resetting the offset of `pyarrow.ListArray` arrays to zero in the table before casting.
I used `pyarrow.compute.subtract` function to update the offsets of the ListArray.

cc @abhi1thakur @SBrandeis

Comments:
Title: `Proto_qa` hosting seems to be broken
Author: VictorSanh (CONTRIBUTOR)
State: closed
Created: 2021-06-08T16:16:32Z
Labels: bug

## Describe the bug
The hosting (on Github) of the `proto_qa` dataset seems broken. I haven't investigated more yet, just flagging it for now. 

@zaidalyafeai if you want to dive into it, I think it's just a matter of changing the links in `proto_qa.py`

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("proto_qa")
```

## Actual results

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/load.py", line 751, in load_dataset
    use_auth_token=use_auth_token,
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/builder.py", line 575, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/builder.py", line 630, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/hf/.cache/huggingface/modules/datasets_modules/datasets/proto_qa/445346efaad5c5f200ecda4aa7f0fb50ff1b55edde3003be424a2112c3e8102e/proto_qa.py", line 131, in _split_generators
    train_fpath = dl_manager.download(_URLs[self.config.name]["train"])
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 199, in download
    num_proc=download_config.num_proc,
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/utils/py_utils.py", line 195, in map_nested
    return function(data_struct)
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 218, in _download
    return cached_path(url_or_filename, download_config=download_config)
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 291, in cached_path
    use_auth_token=download_config.use_auth_token,
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 621, in get_from_cache
    raise FileNotFoundError("Couldn't find file at {}".format(url))
FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/iesl/protoqa-data/master/data/train/protoqa_train.jsonl
```

Comments:
@VictorSanh , I think @mariosasko is already working on it.
Title: Revert default in-memory for small datasets
Author: albertvillanova (MEMBER)
State: closed
Created: 2021-06-08T15:51:41Z
Labels: enhancement

Users are reporting issues and confusion about setting default in-memory to True for small datasets.

We see 2 clear use cases of Datasets:
- the "canonical" way, where you can work with very large datasets, as they are memory-mapped and cached (after every transformation)
- some edge cases (speed benchmarks, interactive/exploratory analysis,...), where default in-memory can explicitly be enabled, and no caching will be done

After discussing with @lhoestq we have agreed to:
- revert this feature (implemented in #2182)
- explain in the docs how to optimize speed/performance by setting default in-memory

cc: @stas00 https://github.com/huggingface/datasets/pull/2409#issuecomment-856210552

Comments:
cc: @krandiash (pinged in reverted PR).
Title: Add align_labels_with_mapping function
Author: lewtun (MEMBER)
State: closed
Created: 2021-06-08T13:54:00Z

This PR adds a helper function to align the `label2id` mapping between a `datasets.Dataset` and a classifier (e.g. a transformer with a `PretrainedConfig.label2id` dict), with the alignment performed on the dataset itself.

This will help us with the Hub evaluation, where we won't know in advance whether a model that is fine-tuned on say MNLI has the same mappings as the MNLI dataset we load from `datasets`.

An example where this is needed is if we naively try to evaluate `microsoft/deberta-base-mnli` on `mnli` because the model config has the following mappings:

```python
  "id2label": {
    "0": "CONTRADICTION",
    "1": "NEUTRAL",
    "2": "ENTAILMENT"
  },
  "label2id": {
    "CONTRADICTION": 0,
    "ENTAILMENT": 2,
    "NEUTRAL": 1
  }
```

while the `mnli` dataset has the `contradiction` and `neutral` labels swapped:

```python
id2label = {0: 'entailment', 1: 'neutral', 2: 'contradiction'}
label2id = {'contradiction': 2, 'entailment': 0, 'neutral': 1}
```

As a result, we get a much lower accuracy during evaluation:

```python
from datasets import load_dataset
from transformers.trainer_utils import EvalPrediction
from transformers import AutoModelForSequenceClassification, Trainer

# load dataset for evaluation
mnli = load_dataset("glue", "mnli", split="test")
# load model
model_ckpt = "microsoft/deberta-base-mnli"
model = AutoModelForSequenceClassification.from_pretrained(checkpoint)
# preprocess, create trainer ...
mnli_enc = ...
trainer = Trainer(model, args=args, tokenizer=tokenizer)
# generate preds
preds = trainer.predict(mnli_enc)
# preds.label_ids misalinged with model.config => returns wrong accuracy (too low)!
compute_metrics(EvalPrediction(preds.predictions, preds.label_ids))
```

The fix is to use the helper function before running the evaluation to make sure the label IDs are aligned:

```python
mnli_enc_aligned = mnli_enc.align_labels_with_mapping(label2id=config.label2id, label_column="label")
# preds now aligned and everyone is happy :)
preds = trainer.predict(mnli_enc_aligned)
```

cc @thomwolf @lhoestq

Comments:
@lhoestq i think this is ready for another review 🙂 

@lhoestq thanks for the feedback - it's now integrated :) 

i also added a comment about sorting the input label IDs

Created the PR here: https://github.com/huggingface/datasets/pull/2510

> Thanks ! Looks all good now :)
> 
> We will also need to have the `DatasetDict.align_labels_with_mapping` method. Let me quickly add it

thanks a lot! i always forget about `DatasetDict` - will be happy when it's just one "dataset" object :)

So, there seems to be a problem with the function align_labels_with_mapping for models like this: https://huggingface.co/huggingface/distilbert-base-uncased-finetuned-mnli]. At least with this model, but perhaps also with others, the model.config.label2id values are of type str not int, which crashes said function. After manually converting the model.config.label2id values to int, the script runs smoothly.
Title: Update version in xor_tydi_qa.py
Author: changjonathanc (CONTRIBUTOR)
State: closed
Created: 2021-06-08T02:23:45Z

Fix #2449

@lhoestq Should I revert to the old `dummy/1.0.0` or  delete it and keep only `dummy/1.1.0`?

Comments:
Hi ! Thanks for updating the version

> Should I revert to the old dummy/1.0.0 or delete it and keep only dummy/1.1.0?

Feel free to delete the old dummy data files
Title: Keep original features order
Author: albertvillanova (MEMBER)
State: closed
Created: 2021-06-07T16:26:38Z

When loading a Dataset from a JSON file whose column names are not sorted alphabetically, we should get the same column name order, whether we pass features (in the same order as in the file) or not.

I found this issue while working on #2366.

Comments:
The arrow writer was supposing that the columns were always in the sorted order. I just pushed a fix to reorder the arrays accordingly to the schema. It was failing for many datasets like squad

and obviously it broke everything

Feel free to revert my commit. I can investigate this in the coming days

@lhoestq I do not understand when you say:
> It was failing for many datasets like squad

All the tests were green after my last commit.

> All the tests were green after my last commit.

Yes but loading the actual squad dataset was failing :/
Title: BLUE file not found
Author: mirfan899 (NONE)
State: closed
Created: 2021-06-06T17:01:54Z

Hi, I'm having the following issue when I try to load the `blue` metric.

```shell
import datasets
metric = datasets.load_metric('blue')
Traceback (most recent call last):
  File "/home/irfan/environments/Perplexity_Transformers/lib/python3.6/site-packages/datasets/load.py", line 320, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "/home/irfan/environments/Perplexity_Transformers/lib/python3.6/site-packages/datasets/utils/file_utils.py", line 291, in cached_path
    use_auth_token=download_config.use_auth_token,
  File "/home/irfan/environments/Perplexity_Transformers/lib/python3.6/site-packages/datasets/utils/file_utils.py", line 621, in get_from_cache
    raise FileNotFoundError("Couldn't find file at {}".format(url))
FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/1.7.0/metrics/blue/blue.py
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/irfan/environments/Perplexity_Transformers/lib/python3.6/site-packages/datasets/load.py", line 332, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "/home/irfan/environments/Perplexity_Transformers/lib/python3.6/site-packages/datasets/utils/file_utils.py", line 291, in cached_path
    use_auth_token=download_config.use_auth_token,
  File "/home/irfan/environments/Perplexity_Transformers/lib/python3.6/site-packages/datasets/utils/file_utils.py", line 621, in get_from_cache
    raise FileNotFoundError("Couldn't find file at {}".format(url))
FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/master/metrics/blue/blue.py
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/home/irfan/environments/Perplexity_Transformers/lib/python3.6/site-packages/datasets/load.py", line 605, in load_metric
    dataset=False,
  File "/home/irfan/environments/Perplexity_Transformers/lib/python3.6/site-packages/datasets/load.py", line 343, in prepare_module
    combined_path, github_file_path
FileNotFoundError: Couldn't find file locally at blue/blue.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.7.0/metrics/blue/blue.py.
The file is also not present on the master branch on github.
```
Here is dataset installed version info
```shell
pip freeze | grep datasets
datasets==1.7.0
```

Comments:
Hi ! The `blue` metric doesn't exist, but the `bleu` metric does.
You can get the full list of metrics [here](https://github.com/huggingface/datasets/tree/master/metrics) or by running
```python
from datasets import list_metrics

print(list_metrics())
```

Ah, my mistake. Thanks for correcting
Title: Update `xor_tydi_qa` url to v1.1
Author: changjonathanc (CONTRIBUTOR)
State: closed
Created: 2021-06-06T09:44:58Z

The dataset is updated and the old url no longer works. So I updated it.

I faced a bug while trying to fix this. Documenting the solution here. Maybe we can add it to the doc (`CONTRIBUTING.md` and `ADD_NEW_DATASET.md`).
> And to make the command work without the ExpectedMoreDownloadedFiles error, you just need to use the --ignore_verifications flag.
https://github.com/huggingface/datasets/issues/2076#issuecomment-803904366

Comments:
Just noticed while 
```load_dataset('local_path/datastes/xor_tydi_qa')``` works,
```load_dataset('xor_tydi_qa')``` 
outputs an error: 
`
FileNotFoundError: Couldn't find file at https://nlp.cs.washington.edu/xorqa/XORQA_site/data/xor_dev_retrieve_eng_span.jsonl
`
(the old url)

I tired clearing the cache `.cache/huggingface/modules` and `.cache/huggingface/datasets`, didn't work.

Anyone know how to fix this? Thanks.

It seems like the error is not on your end. By default, the lib tries to download the version of the dataset script that matches the version of the lib, and that version of the script is, in your case, broken because the old URL no longer works. Once this PR gets merged, you can wait for the new release or set `script_version` to `"master"` in `load_dataset` to get the fixed version of the script.

@mariosasko Thanks! It works now.

Pasting the docstring here for reference.
```
    script_version (:class:`~utils.Version` or :obj:`str`, optional): Version of the dataset script to load:

        - For canonical datasets in the `huggingface/datasets` library like "squad", the default version of the module is the local version fo the lib.
          You can specify a different version from your local version of the lib (e.g. "master" or "1.2.0") but it might cause compatibility issues.
        - For community provided datasets like "lhoestq/squad" that have their own git repository on the Datasets Hub, the default version "main" corresponds to the "main" branch.
          You can specify a different version that the default "main" by using a commit sha or a git tag of the dataset repository.
```
Branch name didn't work, but commit sha works.

Regarding the issue you mentioned about the `--ignore_verifications` flag, I think we should actually change the current behavior of the `--save_infos` flag to make it ignore the verifications as well, so that you don't need to specific `--ignore_verifications` in this case.

@lhoestq I realized I forgot to change this:

https://github.com/huggingface/datasets/blob/fdbf5a97d3393f4a91e4cddcabe364029508f7ce/datasets/xor_tydi_qa/xor_tydi_qa.py#L72-L73

What should I do?

Oh indeed. Please open a PR to change this. This should be 1.1.0
Title: dataset adversarial_qa has no answers in the "test" set
Author: bjascob (NONE)
State: closed
Created: 2021-06-05T14:57:38Z
Labels: bug

## Describe the bug
When loading the adversarial_qa dataset the 'test' portion has no answers.  Only the 'train' and 'validation' portions do.  This occurs with all four of the configs ('adversarialQA', 'dbidaf', 'dbert', 'droberta')

## Steps to reproduce the bug
```
from   datasets import load_dataset
examples = load_dataset('adversarial_qa', 'adversarialQA', script_version="master")['test']
print('Loaded {:,} examples'.format(len(examples)))
has_answers = 0
for e in examples:
    if e['answers']['text']:
        has_answers += 1
print('{:,} have answers'.format(has_answers))
>>> Loaded 3,000 examples
>>> 0 have answers

examples = load_dataset('adversarial_qa', 'adversarialQA', script_version="master")['validation']
<...code above...>
>>> Loaded 3,000 examples
>>> 3,000 have answers
```

## Expected results
If 'test' is a valid dataset, it should have answers. Also note that all of the 'train' and 'validation' sets have answers, there are no "no answer" questions with this set (not sure if this is correct or not).

## Environment info
- `datasets` version: 1.7.0
- Platform: Linux-5.8.0-53-generic-x86_64-with-glibc2.29
- Python version: 3.8.5
- PyArrow version: 1.0.0

Comments:
Hi ! I'm pretty sure that the answers are not made available for the test set on purpose because it is part of the DynaBench benchmark, for which you can submit your predictions on the website.
In any case we should mention this in the dataset card of this dataset.

Makes sense, but not intuitive for someone searching through the datasets.  Thanks for adding the note to clarify.
Title: `yelp_polarity` is broken
Author: JetRunner (CONTRIBUTOR)
State: closed
Created: 2021-06-04T15:44:29Z

![image](https://user-images.githubusercontent.com/22514219/120828150-c4a35b00-c58e-11eb-8083-a537cee4dbb3.png)

Comments:
```
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/streamlit/script_runner.py", line 332, in _run_script
    exec(code, module.__dict__)
File "/home/sasha/nlp-viewer/run.py", line 233, in <module>
    configs = get_confs(option)
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/streamlit/caching.py", line 604, in wrapped_func
    return get_or_create_cached_value()
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/streamlit/caching.py", line 588, in get_or_create_cached_value
    return_value = func(*args, **kwargs)
File "/home/sasha/nlp-viewer/run.py", line 148, in get_confs
    builder_cls = nlp.load.import_main_class(module_path[0], dataset=True)
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/datasets/load.py", line 85, in import_main_class
    module = importlib.import_module(module_path)
File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/sasha/.cache/huggingface/modules/datasets_modules/datasets/yelp_polarity/a770787b2526bdcbfc29ac2d9beb8e820fbc15a03afd3ebc4fb9d8529de57544/yelp_polarity.py", line 36, in <module>
    from datasets.tasks import TextClassification
```

Solved by updating the `nlpviewer`
Title: Sentence Boundaries missing in Dataset: xtreme / udpos
Author: cosmeowpawlitan (CONTRIBUTOR)
State: closed
Created: 2021-06-04T09:10:26Z
Labels: bug

I was browsing through annotation guidelines, as suggested by the datasets introduction.

The guidlines saids "There must be exactly one blank line after every sentence, including the last sentence in the file. Empty sentences are not allowed." in the [Sentence Boundaries and Comments section](https://universaldependencies.org/format.html#sentence-boundaries-and-comments)

But the sentence boundaries seems not to be represented by huggingface datasets features well. I found out that multiple sentence are concatenated together as a 1D array, without any delimiter.

PAN-x, which is another token classification subset from xtreme do represent the sentence boundary using a 2D array.

You may compare in PAN-x.en and udpos.English in the explorer:
 https://huggingface.co/datasets/viewer/?dataset=xtreme

Comments:
Hi,

This is a known issue. More info on this issue can be found in #2061. If you are looking for an open-source contribution, there are step-by-step instructions in the linked issue that you can follow to fix it.

Closed by #2466.
Title: Some tests hang on Windows
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2021-06-03T00:27:30Z
Labels: bug

Currently, several tests hang on Windows if the max path limit of 260 characters is not disabled. This happens due to the changes introduced by #2223 that cause an infinite loop in `WindowsFileLock` described in #2220.  This can be very tricky to debug, so I think now is a good time to address these issues/PRs. IMO throwing an error is too harsh, but maybe we can emit a warning in the top-level `__init__.py ` on startup if long paths are not enabled.

Comments:
Hi ! That would be nice indeed to at least have a warning, since we don't handle the max path length limit.
Also if we could have an error instead of an infinite loop I'm sure windows users will appreciate that

Unfortunately, I know this problem very well... 😅 

I remember having proposed to throw an error instead of hanging in an infinite loop #2220: 60c7d1b6b71469599a27147a08100f594e7a3f84, 8c8ab60018b00463edf1eca500e434ff061546fc 
but @lhoestq told me:
> Note that the filelock module comes from this project that hasn't changed in years - while still being used by ten of thousands of projects:
https://github.com/benediktschmitt/py-filelock
> 
> Unless we have proper tests for this, I wouldn't recommend to change it

I opened an Issue requesting a warning/error at startup for that case: #2224

@albertvillanova Thanks for additional info on this issue.

Yes, I think the best option is to throw an error instead of suppressing it in a loop. I've considered 2 more options, but I don't really like them:
1. create a temporary file with a filename longer than 255 characters on import; if this fails, long paths are not enabled and raise a warning. I'm not sure about this approach because I don't like the idea of creating a temporary file on import for this purpose.
2. check if long paths are enabled with [this code](https://stackoverflow.com/a/46546731/14095927). As mentioned in the comment, this code relies on an undocumented function and Win10-specific.
Title: add english language tags for ~100 datasets
Author: VictorSanh (CONTRIBUTOR)
State: closed
Created: 2021-06-02T16:24:56Z

As discussed on Slack, I have manually checked for ~100 datasets that they have at least one subset in English. This information was missing so adding into the READMEs.

Note that I didn't check all the subsets so it's possible that some of the datasets have subsets in other languages than English...

Comments:
Fixing the tags of all the datasets is out of scope for this PR so I'm merging even though the CI fails because of the missing tags
Title: Remove `extended` field from dataset tagger
Author: lewtun (MEMBER)
State: closed
Created: 2021-06-01T17:18:42Z
Labels: bug

## Describe the bug
While working on #2435 I used the [dataset tagger](https://huggingface.co/datasets/tagging/) to generate the missing tags for the YAML metadata of each README.md file. However, it seems that our CI raises an error when the `extended` field is included:

```
dataset_name = 'arcd'

    @pytest.mark.parametrize("dataset_name", get_changed_datasets(repo_path))
    def test_changed_dataset_card(dataset_name):
        card_path = repo_path / "datasets" / dataset_name / "README.md"
        assert card_path.exists()
        error_messages = []
        try:
            ReadMe.from_readme(card_path)
        except Exception as readme_error:
            error_messages.append(f"The following issues have been found in the dataset cards:\nREADME:\n{readme_error}")
        try:
            DatasetMetadata.from_readme(card_path)
        except Exception as metadata_error:
            error_messages.append(
                f"The following issues have been found in the dataset cards:\nYAML tags:\n{metadata_error}"
            )
    
        if error_messages:
>           raise ValueError("\n".join(error_messages))
E           ValueError: The following issues have been found in the dataset cards:
E           YAML tags:
E           __init__() got an unexpected keyword argument 'extended'

tests/test_dataset_cards.py:70: ValueError
```

Consider either removing this tag from the tagger or including it as part of the validation step in the CI.

cc @yjernite

Comments:
The tagger also doesn't insert the value for the `size_categories` field automatically, so this should be fixed too

Thanks for reporting. Indeed the `extended` tag doesn't exist. Not sure why we had that in the tagger.
The repo of the tagger is here if someone wants to give this a try: https://github.com/huggingface/datasets-tagging
Otherwise I can probably fix it next week

I've opened a PR on `datasets-tagging` to fix the issue 🚀 

thanks ! this is fixed now
Title: Fix NQ features loading: reorder fields of features to match nested fields order in arrow data
Author: lhoestq (MEMBER)
State: closed
Created: 2021-06-01T16:09:30Z

As mentioned in #2401, there is an issue when loading the features of `natural_questions` since the order of the nested fields in the features don't match. The order is important since it matters for the underlying arrow schema.

To fix that I re-order the features based on the arrow schema:

```python
inferred_features = Features.from_arrow_schema(arrow_table.schema)
self.info.features = self.info.features.reorder_fields_as(inferred_features)
assert self.info.features.type == inferred_features.type
```

The re-ordering is a recursive function. It takes into account that the `Sequence` feature type is a struct of list and not a list of struct.

Now it's possible to load `natural_questions` again :)

Comments:
Title: Update DatasetMetadata and ReadMe
Author: gchhablani (CONTRIBUTOR)
State: closed
Created: 2021-06-01T09:32:37Z

This PR contains the changes discussed in #2395.

**Edit**:
In addition to those changes, I'll be updating the `ReadMe` as follows:

Currently, `Section` has separate parsing and validation error lists. In `.validate()`, we add these lists to the final lists and throw errors.

One way to make `ReadMe` consistent with `DatasetMetadata` and add a separate `.validate()` method is to throw separate parsing and validation errors. 

This way, we don't have to throw validation errors, but only parsing errors in `__init__ ()`. We can have an option in `__init__()` to suppress parsing errors so that an object is created for validation. Doing this will allow the user to get all the errors in one go.

In `test_dataset_cards` , we are already catching error messages and appending to a list. This can be done for `ReadMe()`  for parsing errors, and `ReadMe(...,suppress_errors=True); readme.validate()`  for validation, separately.

**Edit 2**:
The only parsing issue we have as of now is multiple headings at the same level with the same name. I assume this will happen very rarely, but it is still better to throw an error than silently pick one of them. It should be okay to separate it this way. 

Wdyt @lhoestq ?

Comments:
Title: Insert Extractive QA templates for SQuAD-like datasets
Author: lewtun (MEMBER)
State: closed
Created: 2021-05-31T14:09:11Z

This PR adds task templates for 9 SQuAD-like templates with the following properties:

* 1 config
* A schema that matches the `squad` one (i.e. same column names, especially for the nested `answers` column because the current implementation does not support casting with mismatched columns. see #2434)
* Less than 20GB (my laptop can't handle more right now)

The aim of this PR is to provide a few datasets to experiment with the task template integration in other libraries / services. 

PR #2429 should be merged before this one.

cc @abhi1thakur

Comments:
hi @lhoestq @SBrandeis i've now added the missing YAML tags, so this PR should be good to go :)

urgh, the windows tests are failing because of encoding issues 😢 

```
dataset_name = 'squad_kor_v1'

    @pytest.mark.parametrize("dataset_name", get_changed_datasets(repo_path))
    def test_changed_dataset_card(dataset_name):
        card_path = repo_path / "datasets" / dataset_name / "README.md"
        assert card_path.exists()
        error_messages = []
        try:
            ReadMe.from_readme(card_path)
        except Exception as readme_error:
            error_messages.append(f"The following issues have been found in the dataset cards:\nREADME:\n{readme_error}")
        try:
            DatasetMetadata.from_readme(card_path)
        except Exception as metadata_error:
            error_messages.append(
                f"The following issues have been found in the dataset cards:\nYAML tags:\n{metadata_error}"
            )
    
        if error_messages:
>           raise ValueError("\n".join(error_messages))
E           ValueError: The following issues have been found in the dataset cards:
E           README:
E           'charmap' codec can't decode byte 0x90 in position 2283: character maps to <undefined>
E           The following issues have been found in the dataset cards:
E           YAML tags:
E           'charmap' codec can't decode byte 0x90 in position 2283: character maps to <undefined>
```

Seems like the encoding issues on windows is also being tackled in #2418 - will see if this solves the problem in the current PR
Title: Fix CI six installation on linux
Author: lhoestq (MEMBER)
State: closed
Created: 2021-05-31T13:15:36Z

For some reason we end up with this error in the linux CI when running pip install .[tests]
```
pip._vendor.resolvelib.resolvers.InconsistentCandidate: Provided candidate AlreadyInstalledCandidate(six 1.16.0 (/usr/local/lib/python3.6/site-packages)) does not satisfy SpecifierRequirement('six>1.9'), SpecifierRequirement('six>1.9'), SpecifierRequirement('six>=1.11'), SpecifierRequirement('six~=1.15'), SpecifierRequirement('six'), SpecifierRequirement('six>=1.5.2'), SpecifierRequirement('six>=1.9.0'), SpecifierRequirement('six>=1.11.0'), SpecifierRequirement('six'), SpecifierRequirement('six>=1.6.1'), SpecifierRequirement('six>=1.9'), SpecifierRequirement('six>=1.5'), SpecifierRequirement('six<2.0'), SpecifierRequirement('six<2.0'), SpecifierRequirement('six'), SpecifierRequirement('six'), SpecifierRequirement('six~=1.15.0'), SpecifierRequirement('six'), SpecifierRequirement('six<2.0,>=1.6.1'), SpecifierRequirement('six'), SpecifierRequirement('six>=1.5.2'), SpecifierRequirement('six>=1.9.0')
```
example CI failure here:
https://app.circleci.com/pipelines/github/huggingface/datasets/6200/workflows/b64fdec9-f9e6-431c-acd7-e9f2c440c568/jobs/38247

The main version requirement comes from tensorflow: `six~=1.15.0`
So I pinned the six version to this.

Comments:
Title: DuplicatedKeysError when trying to load adversarial_qa
Author: hanss0n (NONE)
State: closed
Created: 2021-05-31T12:11:19Z
Labels: bug

## Describe the bug
A clear and concise description of what the bug is.

## Steps to reproduce the bug
```python
dataset = load_dataset('adversarial_qa', 'adversarialQA')
```

## Expected results
The dataset should be loaded into memory

## Actual results

>DuplicatedKeysError: FAILURE TO GENERATE DATASET !
>Found duplicate Key: 4d3cb5677211ee32895ca9c66dad04d7152254d4
>Keys should be unique and deterministic in nature
>
>
>During handling of the above exception, another exception occurred:
>
>DuplicatedKeysError                       Traceback (most recent call last)
>
>/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py in check_duplicate_keys(self)
>    347         for hash, key in self.hkey_record:
>    348             if hash in tmp_record:
>--> 349                 raise DuplicatedKeysError(key)
>    350             else:
>    351                 tmp_record.add(hash)
>
>DuplicatedKeysError: FAILURE TO GENERATE DATASET !
>Found duplicate Key: 4d3cb5677211ee32895ca9c66dad04d7152254d4
>Keys should be unique and deterministic in nature

## Environment info
- `datasets` version: 1.7.0
- Platform: Linux-5.4.109+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyArrow version: 3.0.0

Comments:
Thanks for reporting !
#2433 fixed the issue, thanks @mariosasko :)

We'll do a patch release soon of the library.
In the meantime, you can use the fixed version of adversarial_qa by adding `script_version="master"` in `load_dataset`
Title: Add version-specific BibTeX
Author: albertvillanova (MEMBER)
State: closed
Created: 2021-05-31T10:05:42Z

As pointed out by @lhoestq in #2411, after the creation of the Zenodo DOI for Datasets, a new BibTeX entry is created with each release.

This PR adds a version-specific BibTeX entry, besides the existing one which is generic for the project.

See version-specific BibTeX entry here: https://zenodo.org/record/4817769/export/hx#.YLSyd6j7RPY

Comments:
Maybe we should only keep one citation ?
cc @thomwolf @yjernite 

For info:
- The one automatically generated by Zenodo is version-specific, and a new one will be generated after each release.
- Zenodo has also generated a project-specific DOI (they call it *Concept DOI* as opposed to *Version DOI*), but currently this only redirects to the DOI page of the latest version.
- All the information automatically generated by Zenodo can be corrected/customized if necessary.
  - If we decide to correct/update metadata, take into account that there are the following fields (among others): Authors, Contributors, Title, Description, Keywords, Additional Notes, License,...

According to Zenodo: https://help.zenodo.org/#versioning
> **Which DOI should I use in citations?**
> 
> You should normally always use the DOI for the specific version of your record in citations. This is to ensure that other researchers can access the exact research artefact you used for reproducibility. By default, Zenodo uses the specific version to generate citations.
> 
> You can use the Concept DOI representing all versions in citations when it is desirable to cite an evolving research artifact, without being specific about the version.

Thanks for the details ! As zenodo says we should probably just show the versioned DOI. And we can remove the old citation.

I have removed the old citation.

What about the new one? Should we customize it? I have fixed some author names (replaced nickname with first and family names). Note that the list of authors is created automatically by Zenodo from this list: https://github.com/huggingface/datasets/graphs/contributors
I do not know if this default automatic list of authors is what we want to show in the citation...
Title: Rename QuestionAnswering template to QuestionAnsweringExtractive
Author: lewtun (MEMBER)
State: closed
Created: 2021-05-31T10:04:42Z

Following the discussion with @thomwolf  in #2255, this PR renames the QA template to distinguish extractive vs abstractive QA. The abstractive template will be added in a future PR.

Comments:
> I like having "extractive" in the name to make things explicit. However this creates an inconsistency with transformers.
> 
> See
> https://huggingface.co/transformers/task_summary.html#extractive-question-answering
> 
> But this is minor IMO and I'm ok with this renaming

yes i chose this convention because it allows us to match the `QuestionAnsweringXxx` naming and i think it's better to have `task_name-subtask_name` should auto-complete ever become part of the Hub :)
Title: Fix Docstring Mistake: dataset vs. metric
Author: PhilipMay (CONTRIBUTOR)
State: closed
Created: 2021-05-29T06:09:53Z

PR to fix #2412

Comments:
IMO this PR is ready for review. I do not know why tests fail...

The CI fail is unrelated to this PR, and it has been fixed on master, merging :)

> I just have one comment: we use rouge, not rogue :p

Oops!

rebased on master
Title: load_from_disk and save_to_disk are not compatible with each other
Author: roholazandie (NONE)
State: closed
Created: 2021-05-28T23:07:10Z

## Describe the bug
load_from_disk and save_to_disk are not compatible. When I use save_to_disk to save a dataset to disk it works perfectly but given the same directory load_from_disk throws an error that it can't find state.json. looks like the load_from_disk only works on one split

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("art")
dataset.save_to_disk("mydir")
d = Dataset.load_from_disk("mydir")
```

## Expected results
It is expected that these two functions be the reverse of each other without more manipulation

## Actual results
FileNotFoundError: [Errno 2] No such file or directory: 'mydir/art/state.json'

## Environment info
- `datasets` version: 1.6.2
- Platform: Linux-5.4.0-73-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyTorch version (GPU?): 1.8.1+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

Comments:
Hi,

`load_dataset` returns an instance of `DatasetDict` if `split` is not specified, so instead of `Dataset.load_from_disk`, use `DatasetDict.load_from_disk` to load the dataset from disk.

Thanks it worked!

Though I see a stream of issues open by people lost between datasets and datasets dicts so maybe there is here something that could be better in terms of UX. Could be better error handling or something else smarter to even avoid said errors but maybe we should think about this. Reopening to use this issue as a discussion place but feel free to open a new open if you prefer @lhoestq @albertvillanova 

We should probably improve the error message indeed.

Also note that there exists a function `load_from_disk` that can load a Dataset or a DatasetDict. Under the hood it calls either `Dataset.load_from_disk` or `DatasetDict.load_from_disk`:


```python
from datasets import load_from_disk

dataset_dict = load_from_disk("path/to/dataset/dict")
single_dataset = load_from_disk("path/to/single/dataset")
```

I just opened #2437 to improve the error message

Superseded by #2462
Title: add `desc` in `map` for `DatasetDict` object
Author: bhavitvyamalik (CONTRIBUTOR)
State: closed
Created: 2021-05-28T19:28:44Z

`desc` in `map` currently only works with `Dataset` objects. This PR adds support for `DatasetDict` objects as well

Comments:
The CI error is unrelated to the PR, merging

@lhoestq, can we release this feature if you guys are planning for any patch release for Datasets. It'll slow down [#11927](https://github.com/huggingface/transformers/pull/11927) otherwise :/ 

Sure definitely, having a discrepancy between Dataset.map and DatasetDict.map is an issue that we should fix and include in a patch release. Will do it in the coming days
Title: Fix save_to_disk nested features order in dataset_info.json
Author: lhoestq (MEMBER)
State: closed
Created: 2021-05-28T15:03:28Z

Fix issue https://github.com/huggingface/datasets/issues/2267

The order of the nested features matters (pyarrow limitation), but the save_to_disk method was saving the features types as JSON with `sort_keys=True`, which was breaking the order of the nested features.

Comments: