rIsHu009/github-issues-updated huggingface dataset:
Title: Add OpenAI's HumanEval dataset
Author: lvwerra (MEMBER)
State: closed
Created: 2021-09-11T09:37:47Z

This PR adds OpenAI's [HumanEval](https://github.com/openai/human-eval) dataset. The dataset consists of 164 handcrafted programming problems with solutions and unittests to verify solution. This dataset is useful to evaluate code generation models.

Comments:
I just fixed the class name, and added `[More Information Needed]` in empty sections in case people want to complete the dataset card :)
Title: add multi-proc in `to_csv`
Author: bhavitvyamalik (CONTRIBUTOR)
State: closed
Created: 2021-09-10T21:35:09Z

This PR extends the multi-proc method used in #2747 for`to_json` to `to_csv` as well. 

Results on my machine post benchmarking on `ascent_kb` dataset (giving ~45% improvement when compared to num_proc = 1):
```
Time taken on 1 num_proc, 10000 batch_size  674.2055702209473
Time taken on 4 num_proc, 10000 batch_size  425.6553490161896

Time taken on 1 num_proc, 50000 batch_size  623.5897650718689
Time taken on 4 num_proc, 50000 batch_size  380.0402421951294

Time taken on 4 num_proc, 100000 batch_size  361.7168130874634
```
This is a WIP as writing tests is pending for this PR. 

I'm also exploring [this](https://arrow.apache.org/docs/python/csv.html#incremental-writing) approach for which I'm using `pyarrow-5.0.0`.

Comments:
I think you can just add a test `test_dataset_to_csv_multiproc` in `tests/io/test_csv.py` and we'll be good

Hi @lhoestq, 
I've added `test_dataset_to_csv` apart from `test_dataset_to_csv_multiproc` as no test was there to check generated CSV file when `num_proc=1`. Please let me know if anything is also required!
Title: Use pyarrow.Table.replace_schema_metadata instead of pyarrow.Table.cast
Author: arsarabi (CONTRIBUTOR)
State: closed
Created: 2021-09-10T17:56:57Z

This PR partially addresses #2252.

``update_metadata_with_features`` uses ``Table.cast`` which slows down ``load_from_disk`` (and possibly other methods that use it) for very large datasets. Since ``update_metadata_with_features`` is only updating the schema metadata, it makes more sense to use ``pyarrow.Table.replace_schema_metadata`` which is much faster. This PR adds a ``replace_schema_metadata`` method to all table classes, and modifies  ``update_metadata_with_features`` to use it instead of ``cast``.

Comments:
Title: add mbpp dataset
Author: lvwerra (MEMBER)
State: closed
Created: 2021-09-10T15:27:30Z

This PR adds the mbpp dataset introduced by Google [here](https://github.com/google-research/google-research/tree/master/mbpp) as mentioned in #2816.

The dataset contain two versions: a full and a sanitized one. They have a slightly different schema and it is current state the loading preserves the original schema. An open question is whether to harmonize the two schemas when loading the dataset or to preserve the original one. Since not all fields are overlapping the schema will not be exactly the same.

Comments:
I think it's fine to have the original schema
Title: Error when encoding a dataset with None objects with a Sequence feature
Author: lhoestq (MEMBER)
State: closed
Created: 2021-09-10T14:11:43Z
Labels: bug

There is an error when encoding a dataset with None objects with a Sequence feature

To reproduce:
```python
from datasets import Dataset, Features, Value, Sequence
data = {"a": [[0], None]}
features = Features({"a": Sequence(Value("int32"))})
dataset = Dataset.from_dict(data, features=features)
```
raises

```python
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-24-40add67f8751> in <module>
      2 data = {"a": [[0], None]}
      3 features = Features({"a": Sequence(Value("int32"))})
----> 4 dataset = Dataset.from_dict(data, features=features)
[...]
~/datasets/features.py in encode_nested_example(schema, obj)
    888         if isinstance(obj, str):  # don't interpret a string as a list
    889             raise ValueError("Got a string but expected a list instead: '{}'".format(obj))
--> 890         return [encode_nested_example(schema.feature, o) for o in obj]
    891     # Object with special encoding:
    892     # ClassLabel will convert from string to int, TranslationVariableLanguages does some checks

TypeError: 'NoneType' object is not iterable
```

Instead, if should run without error, as if the `features` were not passed

Comments:
This has been fixed by https://github.com/huggingface/datasets/pull/2900
We're doing a new release 1.12 today to make the fix available :)
Title: 0x290B112ED1280537B24Ee6C268a004994a16e6CE
Author: rcacho172 (NONE)
State: closed
Created: 2021-09-10T09:51:17Z
Labels: dataset request

## Adding a Dataset
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Comments:
Title: Coc
Author: Bwiggity (NONE)
State: closed
Created: 2021-09-10T07:32:07Z
Labels: dataset request

## Adding a Dataset
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Comments:
Title: v1.11.1 release date
Author: fcakyon (NONE)
State: closed
Created: 2021-09-09T21:53:15Z
Labels: question

Hello, i need to use latest features in one of my packages but there have been no new datasets release since 2 months ago.

When do you plan to publush v1.11.1 release?

Comments:
Hi ! Probably 1.12 on monday :)


@albertvillanova i think this issue is still valid and should not be closed till `>1.11.0` is published :)
Title: Fix data URLs and metadata in DocRED dataset
Author: albertvillanova (MEMBER)
State: closed
Created: 2021-09-09T08:55:34Z

The host of `docred` dataset has updated the `dev` data file. This PR:
- Updates the dev URL
- Updates dataset metadata

This PR also fixes the URL of the `train_distant` split, which was wrong.

Fix #2882.

Comments:
Title: Extend support for streaming datasets that use pathlib.Path stem/suffix
Author: albertvillanova (MEMBER)
State: closed
Created: 2021-09-08T08:42:43Z

This PR extends the support in streaming mode for datasets that use `pathlib`, by patching the properties `pathlib.Path.stem` and `pathlib.Path.suffix`.

Related to #2876, #2874, #2866.

CC: @severo

Comments:
Title: In v1.4.1, all TIMIT train transcripts are "Would such an act of refusal be useful?"
Author: rcgale (NONE)
State: closed
Created: 2021-09-07T18:53:45Z
Labels: bug

## Describe the bug
Using version 1.4.1 of `datasets`, TIMIT transcripts are all the same.

## Steps to reproduce the bug
I was following this tutorial
- https://huggingface.co/blog/fine-tune-wav2vec2-english

But here's a distilled repro:
```python
!pip install datasets==1.4.1
from datasets import load_dataset
timit = load_dataset("timit_asr", cache_dir="./temp")
unique_transcripts = set(timit["train"]["text"])
print(unique_transcripts)
assert len(unique_transcripts) > 1
```
## Expected results
Expected the correct TIMIT data. Or an error saying that this version of `datasets` can't produce it.

## Actual results
Every train transcript was "Would such an act of refusal be useful?" Every test transcript was "The bungalow was pleasantly situated near the shore."

## Environment info
- `datasets` version: 1.4.1
- Platform: Darwin-18.7.0-x86_64-i386-64bit
- Python version: 3.7.9
- PyTorch version (GPU?): 1.9.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: tried both
- Using distributed or parallel set-up in script?: no
-

Comments:
Hi @rcgale, thanks for reporting.

Please note that this bug was fixed on `datasets` version 1.5.0: https://github.com/huggingface/datasets/commit/a23c73e526e1c30263834164f16f1fdf76722c8c#diff-f12a7a42d4673bb6c2ca5a40c92c29eb4fe3475908c84fd4ce4fad5dc2514878

If you update `datasets` version, that should work.

On the other hand, would it be possible for @patrickvonplaten to update the [blog post](https://huggingface.co/blog/fine-tune-wav2vec2-english) with the correct version of `datasets`?

I just proposed a change in the blog post.

I had assumed there was a data format change that broke a previous version of the code, since presumably @patrickvonplaten tested the tutorial with the version they explicitly referenced. But that fix you linked suggests a problem in the code, which surprised me.

I still wonder, though, is there a way for downloads to be invalidated server-side? If the client can announce its version during a download request, perhaps the server could reject known incompatibilities? It would save much valuable time if `datasets` raised an informative error on a known problem ("Error: the requested data set requires `datasets>=1.5.0`."). This kind of API versioning is a prudent move anyhow, as there will surely come a time when you'll need to make a breaking change to data.

Also, thank you for a quick and helpful reply!
Title: Don't keep the dummy data folder or dataset_infos.json when resolving data files
Author: lhoestq (MEMBER)
State: closed
Created: 2021-09-07T14:09:04Z
Labels: enhancement

When there's no dataset script, all the data files of a folder or a repository on the Hub are loaded as data files.

There are already a few exceptions:
- files starting with "." are ignored
- the dataset card "README.md" is ignored
- any file named "config.json" is ignored (currently it isn't used anywhere, but it could be used in the future to define splits or configs for example, but not 100% sure)

However any data files in a folder named "dummy" should be ignored as well as they should only be used to test the dataset.
Same for "dataset_infos.json" which should only be used to get the `dataset.info`

Comments:
Hi @lhoestq I am new to huggingface datasets, I would like to work on this issue!


Thanks for the help :) 

As mentioned in the PR, excluding files named "dummy_data.zip" is actually more general than excluding the files inside a "dummy" folder. I just did the change in the PR, I think we can merge it now
Title: Extend support for streaming datasets that use pathlib.Path.glob
Author: albertvillanova (MEMBER)
State: closed
Created: 2021-09-07T13:43:45Z

This PR extends the support in streaming mode for datasets that use `pathlib`, by patching the method `pathlib.Path.glob`.

Related to #2874, #2866.

CC: @severo

Comments:
I am thinking that ideally we should call `fs.glob()` instead...

Thanks, @lhoestq: the idea of adding the mock filesystem is to avoid network calls and reduce testing time ;) 

I have added `rglob` as well and fixed some bugs.
Title: Add Congolese Swahili speech datasets
Author: osanseviero (CONTRIBUTOR)
State: open
Created: 2021-09-07T12:13:50Z
Labels: dataset request

speech

## Adding a Dataset
- **Name:** Congolese Swahili speech corpora
- **Data:** https://gamayun.translatorswb.org/data/

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Also related: https://mobile.twitter.com/OktemAlp/status/1435196393631764482

Comments:
Title: Support streaming datasets that use pathlib
Author: albertvillanova (MEMBER)
State: closed
Created: 2021-09-07T07:35:49Z

This PR extends the support in streaming mode for datasets that use `pathlib.Path`.

Related to: #2866.
CC: @severo

Comments:
I've tried https://github.com/huggingface/datasets/issues/2866 again, and I get the same error.

```python
import datasets as ds
ds.load_dataset('counter', split="train", streaming=False)
```

@severo Issue #2866 is not fully fixed yet: multiple patches need to be implemented for `pathlib`, as that dataset uses quite a lot of `pathlib` functions... 😅 

No worry and no stress, I just wanted to check for that case :) I'm very happy that you're working on issues I'm interested in!
Title: adding swedish_medical_ner
Author: bwang482 (CONTRIBUTOR)
State: closed
Created: 2021-09-07T04:44:53Z

Adding the Swedish Medical NER dataset, listed in "Biomedical Datasets - BigScience Workshop 2021"

Code refactored

Comments:
Hi, what's the current status of this request? It says Changes requested, but I can't see what changes?

Hi, it looks like this PR includes changes to other files that `swedish_medical_ner`.

Feel free to remove these changes, or simply create a new PR that only contains the addition of the dataset
Title: datasets.config.PYARROW_VERSION has no attribute 'major'
Author: bwang482 (CONTRIBUTOR)
State: closed
Created: 2021-09-06T21:06:57Z
Labels: bug

In the test_dataset_common.py script, line 288-289

```
if datasets.config.PYARROW_VERSION.major < 3:
   packaged_datasets = [pd for pd in packaged_datasets if pd["dataset_name"] != "parquet"]
```

which throws the error below. `datasets.config.PYARROW_VERSION` itself return the string '4.0.1'. I have tested this on both datasets.__version_=='1.11.0' and '1.9.0'. I am using Mac OS.

```
import datasets
datasets.config.PYARROW_VERSION.major
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/var/folders/1f/0wqmlgp90qjd5mpj53fnjq440000gn/T/ipykernel_73361/2547517336.py in <module>
      1 import datasets
----> 2 datasets.config.PYARROW_VERSION.major

AttributeError: 'str' object has no attribute 'major'
```

## Environment info
- `datasets` version: 1.11.0
- Platform: Darwin-20.6.0-x86_64-i386-64bit
- Python version: 3.7.11
- PyArrow version: 4.0.1

Comments:
I have changed line 288 to `if int(datasets.config.PYARROW_VERSION.split(".")[0]) < 3:` just to get around it.

Hi @bwang482,

I'm sorry but I'm not able to reproduce your bug.

Please note that in our current master branch, we made a commit (d03223d4d64b89e76b48b00602aba5aa2f817f1e) that simultaneously modified:
- test_dataset_common.py: https://github.com/huggingface/datasets/commit/d03223d4d64b89e76b48b00602aba5aa2f817f1e#diff-a1bc225bd9a5bade373d1f140e24d09cbbdc97971c2f73bb627daaa803ada002L289 that introduces the usage of `datasets.config.PYARROW_VERSION.major`
- but also changed config.py: https://github.com/huggingface/datasets/commit/d03223d4d64b89e76b48b00602aba5aa2f817f1e#diff-e021fcfc41811fb970fab889b8d245e68382bca8208e63eaafc9a396a336f8f2L40, so that `datasets.config.PYARROW_VERSION.major` exists


Sorted. Thanks!

Reopening this. Although the `test_dataset_common.py` script works fine now.

Has this got something to do with my pull request not passing `ci/circleci: run_dataset_script_tests_pyarrow` tests?

https://github.com/huggingface/datasets/pull/2873

Hi @bwang482,

If you click on `Details` (on the right of your non passing CI test names: `ci/circleci: run_dataset_script_tests_pyarrow`), you can have more information about the non-passing tests.

For example, for ["ci/circleci: run_dataset_script_tests_pyarrow_1" details](https://circleci.com/gh/huggingface/datasets/46324?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link), you can see that the only non-passing test has to do with the dataset card (missing information in the `README.md` file): `test_changed_dataset_card`
```
=========================== short test summary info ============================
FAILED tests/test_dataset_cards.py::test_changed_dataset_card[swedish_medical_ner]
= 1 failed, 3214 passed, 2874 skipped, 2 xfailed, 1 xpassed, 15 warnings in 175.59s (0:02:55) =
```

Therefore, your PR non-passing test has nothing to do with this issue.
Title: Add Common Objects in 3D (CO3D)
Author: nateraw (CONTRIBUTOR)
State: open
Created: 2021-09-02T20:36:12Z
Labels: dataset request

vision

## Adding a Dataset
- **Name:** *Common Objects in 3D (CO3D)*
- **Description:** *See blog post [here](https://ai.facebook.com/blog/common-objects-in-3d-dataset-for-3d-reconstruction)*
- **Paper:** *[link to paper](https://arxiv.org/abs/2109.00512)*
- **Data:** *[link to data](https://ai.facebook.com/datasets/co3d-downloads/)*
- **Motivation:** *excerpt from above blog post:*

> As the first data set of its kind, CO3D will aptly enable reconstruction of real-life 3D objects. Indeed, CO3D already provides training data to enable our NeRFormer to tackle the new-view synthesis (NVS) task. Here, photorealistic NVS is a major step on the path to fully immersive AR/VR effects, where objects can be virtually transported across different environments, which will allow connecting users by sharing or recollecting their experiences.
> 
> Besides practical applications in AR/VR, we hope that the data set will become a standard testbed for the recent proliferation of methods (including NeRFormer, Implicit Differentiable Renderer, NeRF, and others) that reconstruct 3D scenes by means of an implicit shape model.
> 

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Comments:
Title: Add CaSiNo dataset
Author: kushalchawla (CONTRIBUTOR)
State: closed
Created: 2021-09-02T17:06:23Z

Hi. I request you to add our dataset to the repository. 

This data was recently published at NAACL 2021: https://aclanthology.org/2021.naacl-main.254.pdf

Comments:
Hi @lhoestq  

Just a request to look at the dataset. Please let me know if any changes are necessary before merging it into the repo. Thank you.

Hey @lhoestq 

Thanks for merging it. One question: I still cannot find the dataset on https://huggingface.co/datasets. Does it take some time or did I miss something?

Hi ! It takes a few hours or a day for the list of datasets on the website to be updated ;)
Title: Add MultiEURLEX dataset
Author: iliaschalkidis (CONTRIBUTOR)
State: closed
Created: 2021-09-02T09:42:24Z

**Add new MultiEURLEX Dataset**

MultiEURLEX comprises 65k EU laws in 23 official EU languages (some low-ish resource). Each EU law has been annotated with EUROVOC concepts (labels) by the Publication Office of EU. As with the English EURLEX, the goal is to predict the relevant EUROVOC concepts (labels); this is multi-label classification task (given the text, predict multiple labels).

Comments:
Hi @lhoestq, we have this new cool multilingual dataset coming at EMNLP 2021. It would be really nice if we could have it in Hugging Face asap. Thanks! 

Hi @lhoestq, I adopted most of your suggestions:

- Dummy data files reduced, including the 2 smallest documents per subset JSONL.
- README was updated with the publication URL and instructions on how to download and use label descriptors. Excessive newlines were deleted.

I would prefer to keep the label list in a pure format (original ids), to enable people to combine those with more information or possibly in the future explore the dataset, find inconsistencies and fix those to release a new version. 

Thanks for the changes :)

Regarding the labels:

If you use the ClassLabel feature type, the only change is that it will store the ids as integers instead of (currently) string.
The advantage is that if people want to know what id corresponds to which label name, they can use `classlabel.int2str`. It is also the format that helps automate model training for classification in `transformers`.

Let me know if that sounds good to you or if you still want to stick with the labels as they are now.

Hey @lhoestq, thanks for providing this information. This sounds great. I updated my code accordingly to use `ClassLabel`. Could you please provide a minimal example of how `classlabel.int2str` works in practice in my case, where labels are a sequence?

```python
from datasets import load_dataset
dataset = load_dataset('multi_eurlex', 'all_languages')
# Read strs from the labels (list of integers) for the 1st sample of the training split
```

I would like to include this in the README file.

Could you also provide some info on how I could define the supervized key to automate model training, as you said?

Thanks!

Thanks for the update :)

Here is an example of usage:
```python
from datasets import load_dataset
dataset = load_dataset('multi_eurlex', 'all_languages', split='train')
classlabel = dataset.features["labels"].feature
print(dataset[0]["labels"])
# [1, 20, 7, 3, 0]
print(classlabel.int2str(dataset[0]["labels"]))
# ['100160', '100155', '100158', '100147', '100149']
```

The ClassLabel is simply used to define the `id2label` dictionary of classification models, to make the ids match between the model and the dataset. There nothing more to do :p 

I think one last thing to do is just update the `dataset_infos.json` file and we'll be good !

Everything is ready! 👍
Title: Update dataset URL
Author: mrm8488 (CONTRIBUTOR)
State: closed
Created: 2021-09-02T05:22:18Z


Comments:
Superseded by PR #2864.

@mrm8488 next time you would like to work on an issue, you can first self-assign it to you (by writing `#self-assign` in a comment on the issue). That way, other people can see you are already working on it and there are not multiple people working on the same issue. 😉
Title: fix: 🐛 be more specific when catching exceptions
Author: severo (COLLABORATOR)
State: closed
Created: 2021-09-01T12:18:12Z

The same specific exception is catched in other parts of the same
function.

Comments:
To give more context: after our discussion, if I understood properly, you are trying to fix a call to `datasets` that takes 15 minutes: https://github.com/huggingface/datasets-preview-backend/issues/17 Is this right?



Yes, that's it. And to do that I'm trying to use https://pypi.org/project/stopit/, which will raise a stopit.TimeoutException exception. But currently, if this exception is raised, it's caught and considered as a "FileNotFoundError" while it should not be caught. 

And what about passing the `timeout` parameter instead?

It might be a good idea, but I would have to add a timeout argument to several methods, I'm not sure we want that (I want to ensure all my queries in https://github.com/huggingface/datasets-preview-backend/tree/master/src/datasets_preview_backend/queries resolve in a given time, be it with an error in case of timeout, or with the successful response). The methods are `prepare_module`, `import_main_class`, *builder_cls.*`get_all_exported_dataset_infos`, `load_dataset_builder`, and `load_dataset`

I understand, you are trying to find a fix for your use case. OK.

Just note that it is also an issue for `datasets` users. Once #2859 fixed in `datasets`, you will no longer have this issue...

Closing, since 1. my problem is more #2859, and I was asking for that change in order to make a hack work on my side, 2. if we want to change how exceptions are handled, we surely want to do it on all the codebase, not only in this particular case.
Title: Cannot download TOTTO dataset
Author: mrm8488 (CONTRIBUTOR)
State: closed
Created: 2021-09-01T11:04:10Z
Labels: bug

Error: Couldn't find file at https://storage.googleapis.com/totto/totto_data.zip

`datasets version: 1.11.0`
# How to reproduce:

```py
from datasets import load_dataset
dataset = load_dataset('totto')
```

Comments:
Hola @mrm8488, thanks for reporting.

Apparently, the data source host changed their URL one week ago: https://github.com/google-research-datasets/ToTTo/commit/cebeb430ec2a97747e704d16a9354f7d9073ff8f

I'm fixing it.
Title: Loading allenai/c4 in streaming mode does too many HEAD requests
Author: lhoestq (MEMBER)
State: closed
Created: 2021-08-31T21:11:04Z
Labels: enhancement

streaming

This does 60,000+ HEAD requests to get all the ETags of all the data files:
```python
from datasets import load_dataset
load_dataset("allenai/c4", streaming=True)
```
It makes loading the dataset completely impractical.

The ETags are used to compute the config id (it must depend on the data files being used).
Instead of using the ETags, we could simply use the commit hash of the dataset repository on the hub, as well and the glob pattern used to resolve the files (here it's `*` by default, to load all the files of the repository)

Comments:
https://github.com/huggingface/datasets/blob/6c766f9115d686182d76b1b937cb27e099c45d68/src/datasets/builder.py#L179-L186

Thanks a lot!!!
Title: Update: Openwebtext - update size
Author: lhoestq (MEMBER)
State: closed
Created: 2021-08-31T17:11:03Z

Update the size of the Openwebtext dataset

I also regenerated the dataset_infos.json but the data file checksum didn't change, and the number of examples either (8013769 examples)

Close #2839, close #726.

Comments:
merging since the CI error in unrelated to this PR and fixed on master
Title: fix: 🐛 remove URL's query string only if it's ?dl=1
Author: severo (COLLABORATOR)
State: closed
Created: 2021-08-31T13:40:07Z

A lot of URL use the query strings, for example
http://opus.nlpl.eu/download.php?f=Bianet/v1/moses/en-ku.txt.zip, we
must not remove it when trying to detect the protocol. We thus remove it
only in the case of the query string being ?dl=1 which occurs on dropbox
and dl.orangedox.com. Also: add unit tests.

See https://github.com/huggingface/datasets/pull/2843 for the original
discussion.

Comments:
Title: Fix windows CI CondaError
Author: lhoestq (MEMBER)
State: closed
Created: 2021-08-31T13:22:02Z

From this thread: https://github.com/conda/conda/issues/6057

We can fix the conda error
```
CondaError: Cannot link a source that does not exist.
C:\Users\...\Anaconda3\Scripts\conda.exe
```

by doing
```bash
conda update conda
```

before doing any install in the windows CI

Comments:
Title: Fix caching when moving script
Author: lhoestq (MEMBER)
State: closed
Created: 2021-08-31T10:58:35Z

When caching the result of a `map` function, the hash that is computed depends on many properties of this function, such as all the python objects it uses, its code and also the location of this code.

Using the full path of the python script for the location of the code makes the hash change if a script like `run_mlm.py` is moved.

I changed this by simply using the base name of the script instead of the full path.

Note that this change also affects the hash of the code used from imported modules, but I think it's fine. Indeed it hashes the code of the imported modules anyway, so the location of the python files of the imported modules doesn't matter when computing the hash.

Close https://github.com/huggingface/datasets/issues/2825

Comments:
Merging since the CI failure is unrelated to this PR
Title: Add AMI dataset
Author: cahya-wirawan (CONTRIBUTOR)
State: closed
Created: 2021-08-31T10:19:01Z

This is an initial commit for AMI dataset

Comments:
Hey @cahya-wirawan, 

I played around with the dataset a bit and it looks already very good to me! That's exactly how it should be constructed :-) I can help you a bit with defining the config, etc... on Monday!

@lhoestq - I think the dataset is ready to be merged :-) 

At the moment, I don't really see how the failing tests correspond to this PR:
- https://app.circleci.com/pipelines/github/huggingface/datasets/7838/workflows/932a40a2-3e11-48be-84f0-c6434510058e/jobs/48318?invite=true#step-107-18
- https://app.circleci.com/pipelines/github/huggingface/datasets/7838/workflows/932a40a2-3e11-48be-84f0-c6434510058e/jobs/48316?invite=true#step-102-136

could you maybe give it a look? :-)
Title: Wound segmentation datasets
Author: osanseviero (CONTRIBUTOR)
State: open
Created: 2021-08-30T10:44:32Z
Labels: dataset request

vision

## Adding a Dataset
- **Name:** Wound segmentation datasets
- **Description:** annotated wound image dataset 
- **Paper:** https://www.nature.com/articles/s41598-020-78799-w
- **Data:** https://github.com/uwm-bigdata/wound-segmentation
- **Motivation:** Interesting simple image dataset, useful for segmentation, with visibility due to http://www.miccai.org/special-interest-groups/challenges/ and https://fusc.grand-challenge.org/

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Comments:
Title: Add Open Catalyst Project Dataset
Author: osanseviero (CONTRIBUTOR)
State: open
Created: 2021-08-30T10:14:39Z
Labels: dataset request

## Adding a Dataset
- **Name:**  Open Catalyst 2020 (OC20) Dataset
- **Website:** https://opencatalystproject.org/
- **Data:** https://github.com/Open-Catalyst-Project/ocp/blob/master/DATASET.md

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Comments:
Title: Negative timezone
Author: jadermcs (CONTRIBUTOR)
State: closed
Created: 2021-08-27T20:50:33Z
Labels: bug

## Describe the bug
The load_dataset method do not accept a parquet file with a negative timezone, as it has the following regex:
```
"^(s|ms|us|ns),\s*tz=([a-zA-Z0-9/_+:]*)$"
```
So a valid timestap ```timestamp[us, tz=-03:00]``` returns an error when loading parquet files.

## Steps to reproduce the bug
```python
# Where the timestamp column has a tz of -03:00
datasets = load_dataset('parquet', data_files={'train': train_files, 'validation': validation_files,
                                        'test': test_files}, cache_dir="./cache_teste/")
```

## Expected results
The -03:00 is a valid tz so the regex should accept this without raising an error.

## Actual results
As this regex disaproves a valid tz it raises the following error:
```python
raise ValueError(
                f"{datasets_dtype} is not a validly formatted string representation of a pyarrow timestamp."
                f"Examples include timestamp[us] or timestamp[us, tz=America/New_York]"
                f"See: https://arrow.apache.org/docs/python/generated/pyarrow.timestamp.html#pyarrow.timestamp"
            )
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.11.0
- Platform: Ubuntu 20.04
- Python version: 3.8
- PyArrow version: 5.0.0

Comments:
Fixed by #2847.
Title: [feature request] adding easy to remember `datasets.cache_dataset()` + `datasets.is_dataset_cached()`
Author: stas00 (CONTRIBUTOR)
State: open
Created: 2021-08-27T18:21:51Z
Labels: enhancement

Often, there is a need to prepare a dataset but not use it immediately, e.g. think tests suite setup, so it'd be really useful to be able to do:

``` 
if not datasets.is_dataset_cached(ds): datasets.cache_dataset(ds)
```

This can already be done with:
```
builder = load_dataset_builder(ds)
if not os.path.idsir(builder.cache_dir):
    builder.download_and_prepare()
```

but the current way is a way less intuitive and much harder to remember than the proposed API, IMHO. 

One more way is to do:

```
_ = load_dataset(ds)
```
but it wastes resources loading the dataset when it's not needed.

this has been discussed at https://huggingface.slack.com/archives/C01229B19EX/p1630021912025800

Thank you!

@lhoestq

Comments:
Title: Fix: wikicorpus - fix keys
Author: lhoestq (MEMBER)
State: closed
Created: 2021-08-27T15:56:06Z

As mentioned in https://github.com/huggingface/datasets/issues/2552, there is a duplicate keys error in `wikicorpus`.

I fixed that by taking into account the file index in the keys

Comments:
The CI error is unrelated to this PR

... merging !
Title: Fix extraction protocol inference from urls with params
Author: lhoestq (MEMBER)
State: closed
Created: 2021-08-27T14:40:57Z

Previously it was unable to infer the compression protocol for files at URLs like
```
https://foo.bar/train.json.gz?dl=1
```
because of the query parameters.

I fixed that, this should allow 10+ datasets to work in streaming mode:
```
  "discovery",
  "emotion",
  "grail_qa",
  "guardian_authorship",
  "pragmeval",
  "simple_questions_v2",
  "versae/adobo",
  "w-nicole/childes_data",
  "w-nicole/childes_data_no_tags_",
  "w-nicole/childes_data_with_tags",
  "w-nicole/childes_data_with_tags_"
```

cc @severo

Comments:
merging since the windows error is just a CircleCI issue

It works, eg https://observablehq.com/@huggingface/datasets-preview-backend-client#{%22datasetId%22%3A%22discovery%22} and https://datasets-preview.huggingface.tech/rows?dataset=discovery&config=discovery&split=train

Nice !
Title: always requiring the username in the dataset name when there is one
Author: stas00 (CONTRIBUTOR)
State: closed
Created: 2021-08-26T23:31:53Z
Labels: enhancement

Me and now another person have been bitten by the `datasets`'s non-strictness on requiring a dataset creator's username when it's due.

So both of us started with `stas/openwebtext-10k`, somewhere along the lines lost `stas/` and continued using `openwebtext-10k` and it all was good until we published the software and things broke, since there is no `openwebtext-10k`

So this feature request is asking to tighten the checking and not allow dataset loading if it was downloaded with the user prefix, but then attempted to be used w/o it.

The same in code:

```
# first run
python -c "from datasets import load_dataset; load_dataset('stas/openwebtext-10k')"
# now run immediately
python -c "from datasets import load_dataset; load_dataset('openwebtext-10k')"
# the second command should fail, but it doesn't fail now.
```

Please let me know if I explained myself clearly.

Thank you!

Comments:
From what I can understand, you want the saved arrow file directory to have username as well instead of just dataset name if it was downloaded with the user prefix?

I don't think the user cares of how this is done, but the 2nd command should fail, IMHO, as its dataset name is invalid:
```
# first run
python -c "from datasets import load_dataset; load_dataset('stas/openwebtext-10k')"
# now run immediately
python -c "from datasets import load_dataset; load_dataset('openwebtext-10k')"
# the second command should fail, but it doesn't fail now.
```

Moreover, if someone were to create `openwebtext-10k` w/o the prefix, they will now get the wrong dataset, if they previously downloaded `stas/openwebtext-10k`.

And if there are 2 users with the same dataset name `foo/ds` and `bar/ds` - currently this won't work to get the correct dataset.

So really there 3 unrelated issues hiding in the current behavior.

This has been fixed now, and we'll do a new release of the library today.

Now the stas/openwebtext-10k dataset is cached at `.cache/huggingface/datasets/stas___openwebtext10k` and openwebtext-10k would be at `.cache/huggingface/datasets/openwebtext10k`. Since they are different, the cache won't fall back on loading the wrong one anymore.

Same for the python script used to generate the dataset: stas/openwebtext-10k is cached at `.cache/huggingface/modules/datasets_modules/datasets/stas___openwebtext10k` and openwebtext-10k would be at `.cache/huggingface/modules/datasets_modules/datasets/openwebtext10k`

Amazing! Thank you for adding this improvement, @lhoestq!

(can be closed?)

Yes indeed :) thanks
Title: Adding GLUECoS Hinglish and Spanglish code-switching bemchmark
Author: yjernite (MEMBER)
State: open
Created: 2021-08-26T17:47:39Z
Labels: dataset request

## Adding a Dataset
- **Name:** GLUECoS
- **Description:** a Microsoft Benchmark to evaluate code-switching for only two language pairs but a variety of tasks
- **Paper:** https://aclanthology.org/2020.acl-main.329/
- **Data:** https://github.com/microsoft/GLUECoS
- **Motivation:** We currently only have [one other](https://huggingface.co/datasets/lince) dataset for code-switching

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Comments:
Hi @yjernite I am interested in adding this dataset. 
In the repo they have also added a code mixed MT task from English to Hinglish [here](https://github.com/microsoft/GLUECoS#code-mixed-machine-translation-task). I think this could be a good dataset addition in itself and then I can add the rest of the GLUECoS tasks as one dataset. What do you think?
Title: How can I compute BLEU-4 score use `load_metric` ?
Author: Doragd (NONE)
State: closed
Created: 2021-08-26T17:36:37Z

I have found the sacrebleu metric. But, I do not know the difference between it and BLEU-4.
If I want to compute BLEU-4 score, what can i do?

Comments:
Title: OpenWebText: NonMatchingSplitsSizesError
Author: thomasw21 (CONTRIBUTOR)
State: closed
Created: 2021-08-26T13:50:26Z
Labels: bug

## Describe the bug

When downloading `openwebtext`, I'm getting:
```
datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=39769494896, num_examples=8013769, dataset_name='openwebtext'), 'recorded': SplitInfo(name='train', num_bytes=39611023912, num_examples=7982430, dataset_name='openwebtext')}]
```

I suspect that the file we download from has changed since the size doesn't look like to match with documentation

`Downloading:   0%|          | 0.00/12.9G [00:00<?, ?B/s]` This suggest the total size is 12.9GB, whereas the one documented mentions `Size of downloaded dataset files: 12283.35 MB`.

## Steps to reproduce the bug
```python
from datasets import load_dataset

load_dataset("openwebtext", download_mode="force_redownload")
```

## Expected results

Loading is successful

## Actual results

Loading throws above error.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.10.2
- Platform: linux (Redhat version 8.1)
- Python version: 3.8
- PyArrow version: 4.0.1

Comments:
Thanks for reporting, I'm updating the verifications metadata

I just regenerated the verifications metadata and noticed that nothing changed: the data file is fine (the checksum didn't change), and the number of examples is still 8013769. Not sure how you managed to get 7982430 examples.

Can you try to delete your cache ( by default at `~/.cache/huggingface/datasets`) and try again please ?
Also, on which platform are you (linux/macos/windows) ?

I'll try without deleting the whole cache (we have large datasets already stored). I was under the impression that `download_mode="force_redownload"` would bypass cache.
Sorry plateform should be linux (Redhat version 8.1)

Hi @thomasw21 , are you still having this issue after clearing your cache ?

Sorry I haven't had time to work on this. I'll close and re-open if I can't figure out why I'm having this issue. Thanks for taking a look !
Title: prepare_module issue when loading from read-only fs
Author: Dref360 (CONTRIBUTOR)
State: closed
Created: 2021-08-25T15:21:26Z
Labels: bug

## Describe the bug

When we use prepare_module from a readonly file system, we create a FileLock using the `local_path`.
This path is not necessarily writable.

`lock_path = local_path + ".lock"`


## Steps to reproduce the bug

Run `load_dataset` on a readonly python loader file.
```python
ds = load_dataset(
        python_loader, data_files={"train": train_path, "test": test_path}
    )
```

where `python_loader` is a path to a file located in a readonly folder.

## Expected results
This should work I think?

## Actual results

```python
    return load_dataset(
  File "/usr/local/lib/python3.8/dist-packages/datasets/load.py", line 711, in load_dataset
    module_path, hash, resolved_file_path = prepare_module(
  File "/usr/local/lib/python3.8/dist-packages/datasets/load.py", line 465, in prepare_module
    with FileLock(lock_path):
  File "/usr/local/lib/python3.8/dist-packages/datasets/utils/filelock.py", line 314, in __enter__
    self.acquire()
  File "/usr/local/lib/python3.8/dist-packages/datasets/utils/filelock.py", line 263, in acquire
    self._acquire()
  File "/usr/local/lib/python3.8/dist-packages/datasets/utils/filelock.py", line 378, in _acquire
    fd = os.open(self._lock_file, open_mode)
OSError: [Errno 30] Read-only file system: 'YOUR_FILE.py.lock'
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.7.0
- Platform: macOS-10.15.7-x86_64-i386-64bit
- Python version: 3.8.8
- PyArrow version: 3.0.0

Comments:
Hello, I opened #2887 to fix this.
Title: Optimize Dataset.filter to only compute the indices to keep
Author: lhoestq (MEMBER)
State: closed
Created: 2021-08-25T14:41:22Z

Optimize `Dataset.filter` to only compute the indices of the rows to keep, instead of creating a new Arrow table with the rows to keep. Creating a new table was an issue because it could take a lot of disk space.

This will be useful to process audio datasets for example cc @patrickvonplaten

Comments:
Maybe worth updating the docs here as well?

Yup, will do !
Title: Update: timit_asr - make the dataset streamable
Author: lhoestq (MEMBER)
State: closed
Created: 2021-08-25T14:22:49Z

The TIMIT ASR dataset had two issues that was preventing it from being streamable:

1. it was missing a call to `open` before `pd.read_csv`
2. it was using `os.path.dirname` which is not supported for streaming

I made the dataset streamable by using `open` to load the CSV, and by adding the support for `os.path.dirname` in dataset scripts to stream data

You can now do
```python
from datasets import load_dataset

timit_asr = load_dataset("timit_asr", streaming=True)
print(next(iter(timit_asr["train"])))
```
prints:

```json
{"file": "zip://data/TRAIN/DR4/MMDM0/SI681.WAV::https://data.deepai.org/timit.zip",
"phonetic_detail": {"start": [0, 1960, 2466, 3480, 4000, 5960, 7480, 7880, 9400, 9960, 10680, 13480, 15680, 15880, 16920, 18297, 18882, 19480, 21723, 22516, 24040, 25190, 27080, 28160, 28560, 30120, 31832, 33240, 34640, 35968, 37720],
"utterance": ["h#", "w", "ix", "dcl", "s", "ah", "tcl", "ch", "ix", "n", "ae", "kcl", "t", "ix", "v", "r", "ix", "f", "y", "ux", "zh", "el", "bcl", "b", "iy", "y", "ux", "s", "f", "el", "h#"],
"stop": [1960, 2466, 3480, 4000, 5960, 7480, 7880, 9400, 9960, 10680, 13480, 15680, 15880, 16920, 18297, 18882, 19480, 21723, 22516, 24040, 25190, 27080, 28160, 28560, 30120, 31832, 33240, 34640, 35968, 37720, 39920]},
"sentence_type": "SI", "id": "SI681",
"speaker_id": "MMDM0",
"dialect_region": "DR4",
"text": "Would such an act of refusal be useful?",
"word_detail": {
    "start": [1960, 4000, 9400, 10680, 15880, 18297, 27080, 30120],
    "utterance": ["would", "such", "an", "act", "of", "refusal", "be", "useful"],
    "stop": [4000, 9400, 10680, 15880, 18297, 27080, 30120, 37720]
}}
```

cc @patrickvonplaten @vrindaprabhu

Comments:
Title: IndexError when accessing first element of a Dataset if first RecordBatch is empty
Author: lhoestq (MEMBER)
State: closed
Created: 2021-08-24T16:49:20Z

The computation of the offsets of the underlying Table of a Dataset has some issues if the first RecordBatch is empty.

```python
from datasets import Dataset
import pyarrow as pa

pa_table = pa.Table.from_pydict({"a": [1]})
pa_table2 = pa.Table.from_pydict({"a": []}, schema=pa_table.schema)
ds_table = pa.concat_tables([pa_table2, pa_table])

dataset = Dataset(ds_table)

print([len(b) for b in dataset.data._batches])
# [0, 1]

print(dataset.data._offsets)
# [0 0 1] (should be [0, 1])

dataset[0]
```
raises
```python
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/datasets/table.py in _interpolation_search(arr, x)
     90         else:
     91             i, j = i, k
---> 92     raise IndexError(f"Invalid query '{x}' for size {arr[-1] if len(arr) else 'none'}.")
     93 
     94 

IndexError: Invalid query '0' for size 1.
```

This can be fixed by ignoring empty batches when computing `table._batches` and `table._offsets`

cc @SaulLu

Comments:
Title: Logging levels not taken into account
Author: LysandreJik (MEMBER)
State: closed
Created: 2021-08-24T11:50:41Z
Labels: bug

## Describe the bug

The `logging` module isn't working as intended relative to the levels to set.

## Steps to reproduce the bug

```python
from datasets import logging

logging.set_verbosity_debug()
logger = logging.get_logger()

logger.error("ERROR")
logger.warning("WARNING")
logger.info("INFO")
logger.debug("DEBUG"
```

## Expected results

I expect all logs to be output since I'm putting a `debug` level.

## Actual results

Only the two first logs are output.

## Environment info

- `datasets` version: 1.11.0
- Platform: Linux-5.13.9-arch1-1-x86_64-with-glibc2.33
- Python version: 3.9.6
- PyArrow version: 5.0.0

## To go further

This logging issue appears in `datasets` but not in `transformers`. It happens because there is no handler defined for the logger. When no handler is defined, the `logging` library will output a one-off error to stderr, using a `StderrHandler` with level `WARNING`.

`transformers` sets a default `StreamHandler` [here](https://github.com/huggingface/transformers/blob/5c6eca71a983bae2589eed01e5c04fcf88ba5690/src/transformers/utils/logging.py#L86)

Comments:
I just take a look at all the outputs produced by `datasets` using the different log-levels.
As far as i can tell using `datasets==1.17.0` they overall issue seems to be fixed.

However, I noticed that there is one tqdm based progress indicator appearing on STDERR that I can simply not suppress.
```
Resolving data files: 100%|██████████| 652/652 [00:00<00:00, 1604.52it/s]
```

According to _get_origin_metadata_locally_or_by_urls it shold be supressable by using the `NOTSET` log-level
https://github.com/huggingface/datasets/blob/1406a04c3e911cec2680d8bc513653e0cafcaaa4/src/datasets/data_files.py#L491-L501
Sadly when specifiing the log-level `NOTSET` it seems to has no effect.

But appart from it not having any effect I must admit that it seems unintuitive to me.
I would suggest changing this such that it is only shown when the log-level is greater or equal to INFO.

This would conform better to INFO according to the [documentation](https://huggingface.co/docs/datasets/v1.0.0/package_reference/logging_methods.html#datasets.logging.set_verbosity_info).
> This will display most of the logging information and tqdm bars.

Any inputs on this?
I will be happy to supply a PR if desired 👍 

Hi! This should disable the tqdm output:
```python
import datasets
datasets.set_progress_bar_enabled(False)
```

On a side note: I believe the issue with logging (not tqdm) is still relevant on master.