rIsHu009/github-issues-updated huggingface dataset:
Title: Pin torchmetrics to fix the COMET test
Author: lhoestq (MEMBER)
State: closed
Created: 2022-01-18T11:03:49Z

Torchmetrics 0.7.0 got released and has issues with `transformers` (see https://github.com/PyTorchLightning/metrics/issues/770)

I'm pinning it to 0.6.0 in the CI, since 0.7.0 makes the COMET metric test fail. COMET requires torchmetrics==0.6.0 anyway.

Comments:
Title: No module named 'fsspec.archive'
Author: shuuchen (NONE)
State: closed
Created: 2022-01-18T10:17:01Z
Labels: bug

## Describe the bug
Cannot import datasets after installation.

## Steps to reproduce the bug
```shell
$ python
Python 3.9.7 (default, Sep 16 2021, 13:09:58)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import datasets
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/shuchen/miniconda3/envs/hf/lib/python3.9/site-packages/datasets/__init__.py", line 34, in <module>
    from .arrow_dataset import Dataset, concatenate_datasets
  File "/home/shuchen/miniconda3/envs/hf/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 61, in <module>
    from .arrow_writer import ArrowWriter, OptimizedTypedSequence
  File "/home/shuchen/miniconda3/envs/hf/lib/python3.9/site-packages/datasets/arrow_writer.py", line 28, in <module>
    from .features import (
  File "/home/shuchen/miniconda3/envs/hf/lib/python3.9/site-packages/datasets/features/__init__.py", line 2, in <module>
    from .audio import Audio
  File "/home/shuchen/miniconda3/envs/hf/lib/python3.9/site-packages/datasets/features/audio.py", line 7, in <module>
    from ..utils.streaming_download_manager import xopen
  File "/home/shuchen/miniconda3/envs/hf/lib/python3.9/site-packages/datasets/utils/streaming_download_manager.py", line 18, in <module>
    from ..filesystems import COMPRESSION_FILESYSTEMS
  File "/home/shuchen/miniconda3/envs/hf/lib/python3.9/site-packages/datasets/filesystems/__init__.py", line 6, in <module>
    from . import compression
  File "/home/shuchen/miniconda3/envs/hf/lib/python3.9/site-packages/datasets/filesystems/compression.py", line 5, in <module>
    from fsspec.archive import AbstractArchiveFileSystem
ModuleNotFoundError: No module named 'fsspec.archive'
```

Comments:
Title: Revisit `enable/disable_` toggle function prefix
Author: jaketae (CONTRIBUTOR)
State: closed
Created: 2022-01-18T04:09:55Z
Labels: enhancement

As discussed in https://github.com/huggingface/transformers/pull/15167, we should revisit the `enable/disable_` toggle function prefix, potentially in favor of `set_enabled_`. Concretely, this translates to

- De-deprecating `disable_progress_bar()`
- Adding `enable_progress_bar()`
- On the caching side, adding `enable_caching` and `disable_caching`

Additional decisions have to be made with regards to the existing `set_enabled_X` functions; that is, whether to keep them as is or deprecate them in favor of the aforementioned functions. 

cc @mariosasko @lhoestq

Comments:
Title: Datasets streaming + map doesn't work for `Audio`
Author: patrickvonplaten (CONTRIBUTOR)
State: closed
Created: 2022-01-17T12:55:42Z
Labels: bug

duplicate

## Describe the bug
When using audio datasets in streaming mode, applying a `map(...)` before iterating leads to an error as the key `array` does not exist anymore.

## Steps to reproduce the bug

```python
from datasets import load_dataset

ds = load_dataset("common_voice", "en", streaming=True, split="train")


def map_fn(batch):
    print("audio keys", batch["audio"].keys())
    batch["audio"] = batch["audio"]["array"][:100]
    return batch


ds = ds.map(map_fn) 

sample = next(iter(ds))
```

I think the audio is somehow decoded before `.map(...)` is actually called.

## Expected results

IMO, the above code snippet should work.

## Actual results

```bash
audio keys dict_keys(['path', 'bytes'])
Traceback (most recent call last):
  File "./run_audio.py", line 15, in <module>
    sample = next(iter(ds))
  File "/home/patrick/python_bin/datasets/iterable_dataset.py", line 341, in __iter__
    for key, example in self._iter():
  File "/home/patrick/python_bin/datasets/iterable_dataset.py", line 338, in _iter
    yield from ex_iterable
  File "/home/patrick/python_bin/datasets/iterable_dataset.py", line 192, in __iter__
    yield key, self.function(example)
  File "./run_audio.py", line 9, in map_fn
    batch["input"] = batch["audio"]["array"][:100]
KeyError: 'array'
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.17.1.dev0
- Platform: Linux-5.3.0-64-generic-x86_64-with-glibc2.17
- Python version: 3.8.12
- PyArrow version: 6.0.1

Comments:
This seems related to https://github.com/huggingface/datasets/issues/3505.
Title: https://huggingface.co/datasets/huggingface/transformers-metadata
Author: ecankirkic (NONE)
State: closed
Created: 2022-01-17T00:18:14Z
Labels: wontfix

dataset-viewer

## Dataset viewer issue for '*name of the dataset*'

**Link:** *link to the dataset viewer page*

*short description of the issue*

Am I the one who added this dataset ? Yes-No

Comments:
Title: Add The Medical Segmentation Decathlon Dataset
Author: omarespejel (NONE)
State: open
Created: 2022-01-16T21:42:25Z
Labels: dataset request

vision

## Adding a Dataset
- **Name:** *The Medical Segmentation Decathlon Dataset*
- **Description:** The underlying data set was designed to explore the axis of difficulties typically encountered when dealing with medical images, such as small data sets, unbalanced labels, multi-site data, and small objects. 
- **Paper:** [link to the dataset paper if available](https://arxiv.org/abs/2106.05735)
- **Data:** http://medicaldecathlon.com/
- **Motivation:** Hugging Face seeks to democratize ML for society. One of the growing niches within ML is the ML + Medicine community. Key data sets will help increase the supply of HF resources for starting an initial community.

(cc @osanseviero @abidlabs )

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Comments:
Hello! I have recently been involved with a medical image segmentation project myself and was going through the `The Medical Segmentation Decathlon Dataset` as well. 
I haven't yet had experience adding datasets to this repository yet but would love to get started. Should I take this issue?
If yes, I've got two questions -
1. There are 10 different datasets available, so are all datasets to be added in a single PR, or one at a time? 
2. Since it's a competition, masks for the test-set are not available. How is that to be tackled? Sorry if it's a silly question, I have recently started exploring `datasets`.

Hi! Sure, feel free to take this issue. You can self-assign the issue by commenting `#self-assign`.

To answer your questions:
1. It makes the most sense to add each one as a separate config, so one dataset script with 10 configs in a single PR.
2. Just set masks in the test set to `None`.

Note that the images/masks in this dataset are in NIfTI format, which our `Image` feature currently doesn't support, so I think it's best to yield the paths to the images/masks in the script and add a preprocessing section to the card where we explain how to load/process the images/masks with `nibabel` (I can help with that).   



> Note that the images/masks in this dataset are in NIfTI format, which our `Image` feature currently doesn't support, so I think it's best to yield the paths to the images/masks in the script and add a preprocessing section to the card where we explain how to load/process the images/masks with `nibabel` (I can help with that).

Gotcha, thanks. Will start working on the issue and let you know in case of any doubt.

#self-assign

This is great! There is a first model on the HUb that uses this dataset! https://huggingface.co/MONAI/example_spleen_segmentation
Title: label information get lost after parquet serialization
Author: Tudyx (NONE)
State: closed
Created: 2022-01-14T10:10:38Z
Labels: bug

## Describe the bug
In *dataset_info.json* file, information about the label get lost after the dataset serialization.

## Steps to reproduce the bug
```python
from datasets import load_dataset

# normal save
dataset = load_dataset('glue', 'sst2', split='train')
dataset.save_to_disk("normal_save")


# save after parquet serialization
dataset.to_parquet("glue-sst2-train.parquet")
dataset = load_dataset("parquet", data_files='glue-sst2-train.parquet')
dataset.save_to_disk("save_after_parquet")
```

## Expected results
I expected to keep label information in *dataset_info.json* file even after parquet serialization

## Actual results
In the normal serialization i got
```json
"label": {
      "num_classes": 2,
      "names": [
        "negative",
        "positive"
      ],
      "names_file": null,
      "id": null,
      "_type": "ClassLabel"
    },
```
And after parquet serialization i got
```json
"label": {
      "dtype": "int64",
      "id": null,
      "_type": "Value"
    },
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.0
- Platform: ubuntu 20.04
- Python version: 3.8.10
- PyArrow version: 6.0.1

Comments:
Hi ! We did a release of `datasets` today that may fix this issue. Can you try updating `datasets` and trying again ?

EDIT: the issue is still there actually

I think we can fix that by storing the Features in the parquet schema metadata, and then reload them when loading the parquet file

This info is stored in the Parquet schema metadata as of https://github.com/huggingface/datasets/pull/5516
Title: Add The Mexican Emotional Speech Database (MESD)
Author: omarespejel (NONE)
State: open
Created: 2022-01-13T23:49:36Z
Labels: dataset request

speech

## Adding a Dataset
- **Name:** *The Mexican Emotional Speech Database (MESD)*
- **Description:** *Contains 864 voice recordings with six different prosodies: anger, disgust, fear, happiness, neutral, and sadness. Furthermore, three voice categories are included: female adult, male adult, and child. *
- **Paper:** *[Paper](https://ieeexplore.ieee.org/abstract/document/9629934/authors#authors)*
- **Data:** *[link to the Github repository or current dataset location](https://data.mendeley.com/datasets/cy34mh68j9/3)*
- **Motivation:** *Would add Spanish speech data to the HF datasets :) *

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Comments:
Title: Add Mauve metric
Author: jthickstun (CONTRIBUTOR)
State: closed
Created: 2022-01-13T03:52:48Z

Add support for the [Mauve](https://github.com/krishnap25/mauve) metric introduced in this [paper](https://arxiv.org/pdf/2102.01454.pdf) (Neurips, 2021).

Comments:
Hi ! The CI was failing because `mauve-text` wasn't installed. I added it to the CI setup :)

I also did some minor changes to the script itself, especially to remove `**kwargs` and explicitly mentioned all the supported arguments (this way if someone does a typo with some parameters they get an error)
Title: ConnectionError in IndicGLUE dataset
Author: sahoodib (NONE)
State: closed
Created: 2022-01-12T17:59:36Z
Labels: dataset bug

While I am trying to load IndicGLUE dataset (https://huggingface.co/datasets/indic_glue) it is giving me with the error:
```
ConnectionError: Couldn't reach https://storage.googleapis.com/ai4bharat-public-indic-nlp-corpora/evaluations/wikiann-ner.tar.gz (error 403)

Comments:
@sahoodib, thanks for reporting.

Indeed, none of the data links appearing in the IndicGLUE website are working, e.g.: https://storage.googleapis.com/ai4bharat-public-indic-nlp-corpora/evaluations/soham-articles.tar.gz
```
<Error>
<Code>UserProjectAccountProblem</Code>
<Message>User project billing account not in good standing.</Message>
<Details>
The billing account for the owning project is disabled in state delinquent
</Details>
</Error>
```

We have contacted the data owners to inform them about their issue and ask them if they plan to fix it.

Yesterday I resent a reminder email with more AI4Bharat-related people in the loop.

I also opened an issue in their repos:
- https://github.com/AI4Bharat/indicnlp_corpus/issues/14
- https://github.com/AI4Bharat/ai4bharat.org/issues/71

We have received a reply from the authors reporting they have updated the URLs of their data files and opened a PR. See:
- #4978
Title: Add the KMWP dataset (extension of #3564)
Author: sooftware (NONE)
State: closed
Created: 2022-01-12T15:33:08Z
Labels: dataset contribution

New pull request of #3564 (Add the KMWP dataset)

Comments:
Sorry, I'm late to check! I'll send it to you soon!

Thanks for your contribution, @sooftware. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there, under this organization namespace: https://huggingface.co/tunib

Please, feel free to tell us if you need some help.

Close this PR. Thanks!
Title: Add the DKTC dataset (Extension of #3564)
Author: sooftware (NONE)
State: closed
Created: 2022-01-12T15:31:29Z
Labels: dataset contribution

New pull request of #3564. (for DKTC)

Comments:
I reflect your comment! @lhoestq 

Wait, the format of the data just changed, so I'll take it into consideration and commit it.

I update the code according to the dataset structure change.

Thanks ! I think the dummy data are not valid yet - the dummy train.csv file only contains a partial example (the quotes `"` start but never end).

> Thanks ! I think the dummy data are not valid yet - the dummy train.csv file only contains a partial example (the quotes `"` start but never end).

Hi! @lhoestq There is a problem.  
<img src="https://user-images.githubusercontent.com/42150335/149804142-3800e635-f5a0-44d9-9694-0c2b0c05f16b.png" width=500>
  
As shown in the picture above, the conversation is divided into "\n" in the "conversion" column.  
That's why there's an error in the file path that only saved only five lines like below. 

```
'idx', 'class', 'conversation'
'0', '협박 대화', '"지금 너 스스로를 죽여달라고 애원하는 것인가?'
아닙니다. 죄송합니다.'
죽을 거면 혼자 죽지 우리까지 사건에 휘말리게 해? 진짜 죽여버리고 싶게.'
정말 잘못했습니다.
```
   
In fact, these five lines are all one line.  
  



Hi ! I see, in this case ca you make sure that the dummy data has a full sample ?

Feel free to open the dummy train.csv in the dummy_data.zip file and add the missing lines

Sorry, I'm late to check! I'll send it to you soon!

Thanks for your contribution, @sooftware. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there, under this organization namespace: https://huggingface.co/tunib

Please, feel free to tell us if you need some help.

Close this PR. Thanks!
Title: Fix push to hub to allow individual split push
Author: thomasw21 (CONTRIBUTOR)
State: closed
Created: 2022-01-12T12:42:58Z

# Description of the issue

If one decides to push a split on a datasets repo, he uploads the dataset and overrides the config. However previous config splits end up being lost despite still having the dataset necessary.

The new flow is the following:
 - query the old config from the repo
 - update into a new config (add/overwrite new split for example)
 - push the new config

# Side fix

 - `repo_id` in HfFileSystem was wrongly typed.
 - I've added `indent=2` as it becomes much easier to read now.

Comments:
This has been addressed in https://github.com/huggingface/datasets/pull/4415. Closing.
Title: Add initial electricity time series dataset
Author: kashif (CONTRIBUTOR)
State: closed
Created: 2022-01-12T10:21:32Z

Here is an initial prototype time series dataset

Comments:
@kashif Some commits on the PR branch are not authored by you, so could you please open a new PR and not use rebase this time :)? You can copy and paste the dataset dir to the new branch. 



making a new PR
Title: Add parameter `preserve_index` to `from_pandas`
Author: Sorrow321 (CONTRIBUTOR)
State: closed
Created: 2022-01-11T15:26:37Z

Added optional parameter, so that user can get rid of useless index preserving. [Issue](https://github.com/huggingface/datasets/issues/3563)

Comments:
> 

I did `make style` and it affected over 500 files

```
All done! ✨ 🍰 ✨
575 files reformatted, 372 files left unchanged.
isort tests src benchmarks datasets/**/*.py metri
```

(result)
![image](https://user-images.githubusercontent.com/20703486/149166681-2f9d1bc4-116a-4f53-ad42-e54e3b8bd605.png)


Nvm I was using wrong black version
Title:  Add the KMWP & DKTC dataset.
Author: sooftware (NONE)
State: closed
Created: 2022-01-11T14:14:08Z

Add the DKTC dataset.
- https://github.com/tunib-ai/DKTC

Comments:
I reflect your review. cc. @lhoestq 

Ah sorry, I missed KMWP comment, wait.

I request 2 new pull requests. #3569 #3570
Title: Dataset.from_pandas preserves useless index
Author: Sorrow321 (CONTRIBUTOR)
State: closed
Created: 2022-01-11T12:07:07Z
Labels: bug

## Describe the bug
Let's say that you want to create a Dataset object from pandas dataframe. Most likely you will write something like this:

```
import pandas as pd
from datasets import Dataset


df = pd.read_csv('some_dataset.csv')
# Some DataFrame preprocessing code...
dataset = Dataset.from_pandas(df)
```
If your preprocessing code contain indexing operations like this:
```
df = df[df.col1 == some_value]
```
then your df.index can be changed from (default) ```RangeIndex(start=0, stop=16590, step=1)``` to something like this ```Int64Index([    0,     1,     2,     3,     4,     5,     6,     7,     8,
                9,
            ...
            83979, 83980, 83981, 83982, 83983, 83984, 83985, 83986, 83987,
            83988],
           dtype='int64', length=16590)```

In this case, PyArrow (by default) will preserve this non-standard index. In the result, your dataset object will have the extra field that you likely don't want to have: '__index_level_0__'.

You can easily fix this by just adding extra argument ```preserve_index=False``` to call of ```InMemoryTable.from_pandas``` in ```arrow_dataset.py```.

If you approve that this isn't desirable behavior, I can make a PR fixing that.

## Environment info
- `datasets` version: 1.16.1
- Platform: Linux-5.11.0-44-generic-x86_64-with-glibc2.31
- Python version: 3.9.7
- PyArrow version: 6.0.1

Comments:
Hi! That makes sense. Sure, feel free to open a PR! Just a small suggestion: let's make `preserve_index` a parameter of `Dataset.from_pandas` (which we then pass to `InMemoryTable.from_pandas`) with `None` as a default value to not have this as a breaking change.
Title: Cannot load ‘bookcorpusopen’
Author: HUIYINXUE (NONE)
State: closed
Created: 2022-01-10T20:17:18Z
Labels: bug

dataset bug

## Describe the bug
Cannot load 'bookcorpusopen'


## Steps to reproduce the bug
```python
dataset = load_dataset('bookcorpusopen')
```
or
```python
dataset = load_dataset('bookcorpusopen',script_version='master')
```

## Actual results
ConnectionError: Couldn't reach https://the-eye.eu/public/AI/pile_preliminary_components/books1.tar.gz

## Environment info
- `datasets` version: 1.9.0
- Platform: Linux version 3.10.0-1160.45.1.el7.x86_64
- Python version:  3.6.13
- PyArrow version: 6.0.1

Comments:
The host of this copy of the dataset (https://the-eye.eu) is down and has been down for a good amount of time ([potentially months](https://www.reddit.com/r/Roms/comments/q82s15/theeye_downdied/))

Finding this dataset is a little esoteric, as the original authors took down the official BookCorpus dataset some time ago.

There are community-created versions of BookCorpus, such as the files hosted in the link below.
https://battle.shawwn.com/sdb/bookcorpus/

And more discussion here:
https://github.com/soskek/bookcorpus

Do we want to remove this dataset entirely?  There's a fair argument for this, given that the official BookCorpus dataset was taken down by the authors.   If not, perhaps can open a PR with the link to the community-created tar above and updated dataset description.

Hi! The `bookcorpusopen` dataset is not working for the same reason as explained in this comment: https://github.com/huggingface/datasets/issues/3504#issuecomment-1004564980

Hi @HUIYINXUE, it should work now that the data owners created a mirror server with all data, and we updated the URL in our library.
Title: Run pyupgrade for Python 3.6+
Author: bryant1410 (CONTRIBUTOR)
State: closed
Created: 2022-01-10T19:20:53Z

Run the command:

```bash
pyupgrade $(find . -name "*.py" -type f) --py36-plus
```

Which mainly avoids unnecessary lists creations and also removes unnecessary code for Python 3.6+.

It was originally part of #3489.

Tip for reviewing faster: use the CLI (`git diff`) and scroll.

Comments:
Hi ! Thanks for the change :)
Could it be possible to only run it for the code in `src/` ? We try to not change the code in the `datasets/` directory too often since it refreshes the users cache when they upgrade `datasets`.

> Hi ! Thanks for the change :)
> Could it be possible to only run it for the code in `src/` ? We try to not change the code in the `datasets/` directory too often since it refreshes the users cache when they upgrade `datasets`.

I reverted the changes in `datasets/` instead of changing only `src/`. Does it sound good?

I just resolved some conflicts with the master branch. If the CI is green we can merge :)
Title: Integrate Milvus (pymilvus) library
Author: mariosasko (COLLABORATOR)
State: open
Created: 2022-01-10T15:20:29Z
Labels: enhancement

Milvus is a popular open-source vector database. We should add a new vector index to support this project.

Comments:
Hi @mariosasko，Just search randomly and I found this issue~ I'm the tech lead of Milvus and we are looking forward to integrate milvus together with huggingface datasets.

Any suggestion on how we could start?


Feel free to assign to me and we probably need some guide on it

@mariosasko any updates my man?


Hi! For starters, I suggest you take a look at this file: https://github.com/huggingface/datasets/blob/master/src/datasets/search.py, which contains all the code for Faiss/ElasticSearch support. We could set up a Slack channel for additional guidance. Let me know what you prefer.

> Hi! For starters, I suggest you take a look at this file: https://github.com/huggingface/datasets/blob/master/src/datasets/search.py, which contains all the code for Faiss/ElasticSearch support. We could set up a Slack channel for additional guidance. Let me know what you prefer.

Sure, we take a look and do some research
Title: Fix bug in `ImageClassifcation` task template
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2022-01-10T14:09:59Z

Fixes a bug in the `ImageClassification` task template which requires specifying class labels twice in dataset scripts. Additionally, this PR refactors the API around the classification task templates for cleaner `labels` handling.

CC: @lewtun @nateraw

Comments:
The CI failures are unrelated to the changes in this PR.

> The CI failures are unrelated to the changes in this PR.

It seems that some of the failures are due to the tests on the dataset cards (e.g. CIFAR, MNIST, FASHION_MNIST). Perhaps it's worth addressing those in this PR to avoid confusing downstream developers who branch off `master` and suddenly have a failing CI?

@lewtun We only run these tests against the modified datasets on the PR branch, so this will not lead to errors after merging.
Title: Preserve encoding/decoding with features in `Iterable.map` call
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2022-01-10T13:32:20Z

As described in https://github.com/huggingface/datasets/issues/3505#issuecomment-1004755657, this PR uses a generator expression to encode/decode examples with `features` (which are set to None in `map`) before applying a map transform.

Fix #3505

Comments:
Title: DuplicatedKeysError when loading tweet_qa dataset
Author: LeonieWeissweiler (NONE)
State: closed
Created: 2022-01-10T10:53:11Z
Labels: bug

When loading the tweet_qa dataset with `load_dataset('tweet_qa')`, the following error occurs: 

`DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 2a167f9e016ba338e1813fed275a6a1e
Keys should be unique and deterministic in nature
`
Might be related to issues #2433 and #2333

- `datasets` version: 1.17.0
- Python version: 3.8.5

Comments:
Hi, we've just merged the PR with the fix. The fixed version of the dataset can be downloaded as follows:
```python
import datasets
dset = datasets.load_dataset("tweet_qa", revision="master")
```
Title: set_format("np") no longer works for Image data
Author: cgarciae (NONE)
State: closed
Created: 2022-01-09T17:18:13Z
Labels: bug

## Describe the bug
`dataset.set_format("np")` no longer works for image data, previously you could load the MNIST like this:

```python
dataset = load_dataset("mnist")
dataset.set_format("np")
X_train = dataset["train"]["image"][..., None] # <== No longer a numpy array
```
but now it doesn't work, `set_format("np")` seems to have no effect and the dataset just returns a list/array of PIL images instead of numpy arrays as requested.

Comments:
A quick fix for now is doing this:

```python
X_train = np.stack(dataset["train"]["image"])[..., None]

This error also propagates to jax and is even trickier to fix, since `.with_format(type='jax')` will use numpy conversion internally (and fail). For a three line failure:

```python
dataset = datasets.load_dataset("mnist")
dataset.set_format("jax")
X_train = dataset["train"]["image"]
```

Hi! We've recently introduced a new Image feature that yields PIL Images (and caches transforms on them) instead of arrays.

However, this feature requires a custom transform to yield np arrays directly:
```python
ddict = datasets.load_dataset("mnist")

def pil_image_to_array(batch):
     return {"image": [np.array(img) for img in batch["image"]]}  # or jnp.array(img) for Jax

ddict.set_transform(pil_image_to_array, columns="image", output_all_columns=True)
```

[Docs](https://huggingface.co/docs/datasets/master/process.html#format-transform) on `set_transform`.

Also, the approach proposed by @cgarciae is not the best because it loads the entire column in memory.

@albertvillanova @lhoestq WDYT? The Audio and the Image feature currently don't support the TF/Jax/PT Formatters, but for the Numpy Formatter maybe it makes more sense to return np arrays (and not a dict in the case of the Audio feature or a PIL Image object in the case of the Image feature).

Yes I agree it should return arrays and not a PIL image (and possible an array instead of a dict for audio data).
I'm currently finishing some code refactoring of the image and audio and opening a PR today. Maybe we can look into that after the refactoring

This has been fixed in https://github.com/huggingface/datasets/pull/5072, which is included in the latest release of `datasets`.
Title: Add the KMWP & DKTC dataset.
Author: sooftware (NONE)
State: closed
Created: 2022-01-08T17:12:14Z

Add the KMWP & DKTC dataset.
  
Additional notes:
- Both datasets will be released on January 10 through the GitHub link below.
  - https://github.com/tunib-ai/DKTC
  - https://github.com/tunib-ai/KMWP
- So it doesn't work as a link at the moment, but the code will work soon (after it is released on January 10).

Comments:
Title: Add more compression types for `to_json`
Author: bhavitvyamalik (CONTRIBUTOR)
State: closed
Created: 2022-01-07T18:25:02Z

This PR adds `bz2`, `xz`, and `zip` (WIP) for `to_json`. I also plan to add `infer` like how `pandas` does it

Comments:
@lhoestq, I looked into how to compress with `zipfile` for which few methods exist, let me know which one looks good:
1. create the file in normal `wb` mode and then zip it separately
2. use `ZipFile.write_str` to write file into the archive. For this we'll need to change how we're writing files from `_write` method  

How `pandas` handles it is that they have created a wrapper for standard library class `ZipFile` and allow the returned file-like handle to accept byte strings via `write` method instead of `write_str` (purpose was to change the name of function by creating that wrapper)

1. sounds not ideal since it creates an intermediary file.
I like pandas' approach. Is it possible to implement 2. using the pandas class ? Or maybe we can have something similar ?

Definitely, @lhoestq! I've adapted that from original code and turns out it is faster than `gz` compression. Apart from that I've also added `infer` option to automatically infer compression type from `path_or_buf` given

One small thing, currently I'm assuming that user will provide compression extension in `path_or_buf`. Is it this also possible?
`dataset.to_json("from_dataset.json", compression="zip")`? 
Should I put an `assert` to ensure the file name provided always has a compression extension?

Thanks !

> One small thing, currently I'm assuming that user will provide compression extension in path_or_buf. Is it this also possible?
>dataset.to_json("from_dataset.json", compression="zip")?
>Should I put an assert to ensure the file name provided always has a compression extension?

I think it's fine as it is right now :) No need to check the extension of the filename passed to `path_or_buf`.


> turns out it is faster than gz compression

I think the default compression level of `gzip` is 9 in python, which is very slow. Maybe we can switch to compression level 6 instead which is faster, like the `gzip` command on unix

I found that `fsspec` has something that may interest you: [fsspec.open(..., compression=...)](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.open). I don't remember if we've already mentioned it or not

It also has `zip` if I understand correctly ! see https://github.com/fsspec/filesystem_spec/blob/master/fsspec/compression.py#L70

Since `fsspec` is a dependency of `datasets` we can use all this :)

Let me know if you prefer using `fsspec` instead (I haven't tested this yet to write compressed files). IMO it sounds pretty easy to use and it would make the code base simpler

Just tried `fsspec` but I'm not able to write compressed `zip` files :/
`gzip`, `xz`, `bz2` are all working fine and it's really simple (no need for `FileWriteHandler` now!)
Title: Bug in `openbookqa` dataset
Author: lucadiliello (CONTRIBUTOR)
State: closed
Created: 2022-01-07T17:32:57Z
Labels: bug

dataset bug

## Describe the bug
Dataset entries contains a typo.

## Steps to reproduce the bug

```python
>>> from datasets import load_dataset
>>> obqa = load_dataset('openbookqa', 'main')
>>> obqa['train'][0]
```

## Expected results
```python
{'id': '7-980', 'question_stem': 'The sun is responsible for', 'choices': {'text': ['puppies learning new tricks', 'children growing up and getting old', 'flowers wilting in a vase', 'plants sprouting, blooming and wilting'], 'label': ['A', 'B', 'C', 'D']}, 'answerKey': 'D'}
```

## Actual results
```python
{'id': '7-980', 'question_stem': 'The sun is responsible for', 'choices': {'text': ['puppies learning new tricks', 'children growing up and getting old', 'flowers wilting in a vase', 'plants sprouting, blooming and wilting'], 'label': ['puppies learning new tricks', 'children growing up and getting old', 'flowers wilting in a vase', 'plants sprouting, blooming and wilting']}, 'answerKey': 'D'}
```

The bug is present in all configs and all splits.

## Environment info
- `datasets` version: 1.17.0
- Platform: Linux-5.4.0-1057-aws-x86_64-with-glibc2.27
- Python version: 3.9.7
- PyArrow version: 4.0.1

Comments:
Closed by:
- #4259
Title: Fix sem_eval_2018_task_1 download location
Author: maxpel (CONTRIBUTOR)
State: closed
Created: 2022-01-07T15:37:52Z

This changes the download location of sem_eval_2018_task_1 files to include the test set labels as discussed in https://github.com/huggingface/datasets/issues/2745#issuecomment-954588500_ with @lhoestq.

Comments:
Hi ! Thanks for pushing this :)

It seems that you created this PR from an old version of `datasets` that didn't have the sem_eval_2018_task_1.py file.

Can you try merging `master` into your branch ? Or re-create your PR from a branch that comes from a more recent version of `datasets` ?

And sorry for the late response !

Hi! No problem! I made the new branch like you said and opened https://github.com/huggingface/datasets/pull/3643 for it. I will close this one.
Title: Specify the feature types of a dataset on the Hub without needing a dataset script
Author: lhoestq (MEMBER)
State: closed
Created: 2022-01-07T15:17:06Z
Labels: enhancement

**Is your feature request related to a problem? Please describe.**
Currently if I upload a CSV with paths to audio files, the column type is string instead of Audio.

**Describe the solution you'd like**
I'd like to be able to specify the types of the column, so that when loading the dataset I directly get the features types I want.

The feature types could read from the `dataset_infos.json` for example.

**Describe alternatives you've considered**
Create a dataset script to specify the features, but that seems complicated for a simple thing.

cc @abidlabs

Comments:
After looking into this, discovered that this is already supported if the `dataset_infos.json` file is configured correctly! Here is a working example: https://huggingface.co/datasets/abidlabs/test-audio-13

This should be probably be documented, though.
Title: fix: 🐛 pass token when retrieving the split names
Author: severo (COLLABORATOR)
State: closed
Created: 2022-01-07T10:29:22Z


Comments:
Currently, it does not work with https://huggingface.co/datasets/mozilla-foundation/common_voice_7_0/blob/main/common_voice_7_0.py#L146 (which was the goal), because `dl_manager.download_config.use_auth_token` is ignored, and the authentication is required to be use `huggingface-cli login`.
In my use case (dataset viewer), I'd prefer to use a specific "User Token Access", with only the "read" role (https://huggingface.co/settings/token).

See https://github.com/huggingface/datasets-preview-backend/issues/74#issuecomment-1007316853 for the context

> Simply passing download_config is ok :)

hmm, I prefer only passing use_auth_token. But the question is more: is it correct, in the (convoluted) case if `download_config.use_auth_token` exists and is different from `use_auth_token`? Which one should be used?

If both are passed, `use_auth_token` should have the priority (more specific parameters have the higher priority)
Title: Ability to split a dataset in multiple files.
Author: Dref360 (CONTRIBUTOR)
State: open
Created: 2022-01-06T23:02:25Z
Labels: enhancement

Hello,

**Is your feature request related to a problem? Please describe.**

My use case is that I have one writer that adds columns and multiple workers reading the same `Dataset`. Each worker should have access to columns added by the writer when they reload the dataset.

I understand that we shouldn't overwrite an arrow file as this could cause Segfault and so on. Before 1.16, I was able to overwrite the dataset and that would work most of the time with some retries.

**Describe the solution you'd like**

I was thinking that if we could append `Dataset._data_files`, when the workers reload the Dataset, they would get the new columns.


**Describe alternatives you've considered**
I currently need to
1. Save multiple "versions" of the dataset and load the latest.
2. Try working with cache files to get the latest columns.

**Additional context**

I think this would be a great addition to HFDataset as Parquet supports multi-files input out of the box!

I can make a PR myself with some pointers as needed :)

Comments:
Title: Allow loading community metrics from the hub, just like datasets
Author: eladsegal (CONTRIBUTOR)
State: closed
Created: 2022-01-06T11:26:26Z
Labels: enhancement

generic discussion

**Is your feature request related to a problem? Please describe.**
Currently, I can load a metric implemented by me by providing the local path to the file in `load_metric`.
However, there is no option to do it with the metric uploaded to the hub.
This means that if I want to allow other users to use it, they must download it first which makes the usage less smooth.

**Describe the solution you'd like**
Load metrics from the hub just like datasets are loaded.
In order to not break stuff, the convention can be to put the metric file in a "metrics" folder in the hub.

Comments:
Hi ! Thanks for your message :) This is a great idea indeed. We haven't started working on this yet though. For now I guess you can host your metric on the Hub (either with your model or your dataset) and use `hf_hub_download` to download it (docs [here](https://github.com/huggingface/huggingface_hub/blob/main/docs/hub/how-to-downstream.md#cached_download))

This is a great solution in the meantime, thanks!

Here's the code I used, in case it can be of help to someone else:
```python
import os, shutil
from huggingface_hub import hf_hub_download
def download_metric(repo_id, file_path):
    # repo_id: for models "username/model_name", for datasets "datasets/username/model_name"
    local_metric_path = hf_hub_download(repo_id=repo_id, filename=file_path)
    updated_local_metric_path = (os.path.dirname(local_metric_path) + os.path.basename(local_metric_path).replace(".", "_") + ".py")
    shutil.copy(local_metric_path, updated_local_metric_path)
    return updated_local_metric_path

metric = load_metric(download_metric(REPO_ID, FILE_PATH))
```

Solved with https://github.com/huggingface/evaluate 🤗 

Yay!! cc @lvwerra @sashavor @douwekiela 

Please share your feedback @eladsegal =)
Title: Support 7-zip compressed data files
Author: albertvillanova (MEMBER)
State: open
Created: 2022-01-06T07:11:03Z
Labels: enhancement

**Is your feature request related to a problem? Please describe.**

We should support 7-zip compressed data files:
- [x] in `extract`:
  - #4672 
- [ ] in `iter_archive`: for streaming mode

both in streaming and non-streaming modes.

Comments:
This should also resolve: https://github.com/huggingface/datasets/issues/3185.
Title: How to convert  torch.utils.data.Dataset to datasets.arrow_dataset.Dataset?
Author: CindyTing (NONE)
State: open
Created: 2022-01-06T02:13:42Z
Labels: enhancement

Hi,
I use torch.utils.data.Dataset to define my own data, but I need to use the 'map' function of datasets.arrow_dataset.Dataset later, so I hope to convert torch.utils.data.Dataset to datasets.arrow_dataset.Dataset.

Here is an example.
```
from torch.utils.data import Dataset
from datasets.arrow_dataset import Dataset as HFDataset

class ADataset(Dataset):
    def __init__(self, data): 
        super().__init__()
        self.data = data
    def __getitem__(self, index):
        return self.data[index]
    def __len__(self):
        return self.len
class MDataset():
    def __init__(self, tokenizer: AutoTokenizer, data_args, training_args):
        self.train_dataset = ADataset(data_args)
        self.tokenizer = tokenizer
        self.data_args = data_args

        self.train_dataset = self.train_dataset.map(
            self.process_function,
            batched=True,
            remove_columns=column_names,
            load_from_cache_file=True,
            desc="Running tokenizer on train dataset",
             )
    def process_function(self, examples):
        sentences = [" ".join(sample[0][3]) for sample in examples]

        tokenized = self.tokenizer(
            sentences,
            max_length=self.max_seq_len,
            padding=self.padding, 
            truncation=True)
```
But it would raise an ERROR, AttributeError: 'ADataset' object has no attribute 'map'.

so how to convert  torch.utils.data.Dataset to datasets.arrow_dataset.Dataset?

Thanks in advance!

Comments:
Title: Add SVHN dataset
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2022-01-05T18:29:09Z

Add the SVHN dataset. 

Additional notes:
* compared to the TFDS implementation, exposes additional the "full numbers" config
* adds the streaming support for `os.path.splitext` and `scipy.io.loadmat` 
* adds `h5py` to the requirements list for the dummy data test

Comments:
Title: Task search function on hub not working correctly
Author: patrickvonplaten (CONTRIBUTOR)
State: open
Created: 2022-01-05T09:36:30Z
Labels: bug

When I want to look at all datasets of the category: `speech-processing`  *i.e.* https://huggingface.co/datasets?task_categories=task_categories:speech-processing&sort=downloads , then the following dataset doesn't show up for some reason:

- https://huggingface.co/datasets/speech_commands

even thought it's task tags seem correct: 
https://raw.githubusercontent.com/huggingface/datasets/master/datasets/speech_commands/README.md

Comments:
known issue due to https://github.com/huggingface/datasets/pull/2362 (and [internal](https://github.com/huggingface/moon-landing/issues/946)) , will be solved soon

hmm actually i have no recollection of why I said that

Because it has dots in some YAML keys, it can't be parsed and indexed by the back-end
Title: Give clearer instructions to add the YAML tags
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-01-05T06:47:52Z

Fix #3531.

CC: @julien-c @VictorSanh

Comments:
this is great, maybe just put all of it in one line?

> TODO: Add YAML tags here. Copy-paste the tags obtained with the online tagging app: https://huggingface.co/spaces/huggingface/datasets-tagging
Title: Give clearer instructions to add the YAML tags
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-01-05T06:44:20Z
Labels: bug

## Describe the bug
As reported by @julien-c, many community datasets contain the line `YAML tags:` at the top of the YAML section in the header of the README file.  See e.g.: https://huggingface.co/datasets/bigscience/P3/commit/a03bea08cf4d58f268b469593069af6aeb15de32

Maybe we should give clearer instruction/hints in the README template.

Comments:
Title: Update license to bookcorpus dataset card
Author: meg-huggingface (CONTRIBUTOR)
State: closed
Created: 2022-01-04T23:25:23Z
Labels: dataset contribution

Not entirely sure, following the links here, but it seems the relevant license is at https://github.com/soskek/bookcorpus/blob/master/LICENSE

Comments:
The smashwords ToS apply for this dataset, we did the same for https://github.com/huggingface/datasets/pull/3525

_The documentation is not available anymore as the PR was closed or merged._
Title: Adding license information for Openbookcorpus
Author: meg-huggingface (CONTRIBUTOR)
State: closed
Created: 2022-01-04T23:20:36Z

Not entirely sure, following the links here, but it seems the relevant license is at https://github.com/soskek/bookcorpus/blob/master/LICENSE

Comments:
The MIT license seems to be for the crawling code, no ? Then maybe we can also redirect users to the [terms of smashwords.com](https://www.smashwords.com/about/tos) regarding copyrights, in particular the paragraph 10 for end-users. In particular it seems that end users can download and use the content "for their personal enjoyment in any reasonable non-commercial manner in compliance with copyright law" and the smashwords end-users agreement.

It should be the same for https://github.com/huggingface/datasets/pull/3526 as well

May I merge this one ?

_The documentation is not available anymore as the PR was closed or merged._
Title: wmt19 is broken (zh-en)
Author: AjayP13 (NONE)
State: closed
Created: 2022-01-04T22:33:45Z
Labels: bug

dataset bug

## Describe the bug
A clear and concise description of what the bug is.

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("wmt19", 'zh-en')
```

## Expected results
The dataset should download.

## Actual results
`ConnectionError: Couldn't reach ftp://cwmt-wmt:cwmt-wmt@datasets.nju.edu.cn/parallel/casia2015.zip`

## Environment info
- `datasets` version: 1.15.1
- Platform: Linux
- Python version: 3.8

Comments:
This issue is not reproducible.
Title: Audio datacard update - first pass
Author: meg-huggingface (CONTRIBUTOR)
State: closed
Created: 2022-01-04T20:58:25Z

Filling out data card "Personal and Sensitive Information" for speech datasets to note PII concerns

Comments:
I'm not sure that we want to change the tags at the top of the cards by hand. Those are used to create the tags in the hub. Although looking at all the tags now, we might want to normalize the current tags again (hyphens or no, ".0" or no). Maybe we could add a binary tag for public domain or not?

> 

That's a good point, I didn't realize these were auto-populated.
At the same time, some of them are wrong -- how/where are they auto-populated?  Seems like we should fix it at that source for the future.
In the mean time, I see that "cc0-1.0" is the desired tag for public domain, so I will change that for now.
Title: Add PubMed Central Open Access dataset
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-01-04T06:54:35Z
Labels: dataset request

## Adding a Dataset
- **Name:** PubMed Central Open Access
- **Description:** The PMC Open Access Subset includes more than 3.4 million journal articles and preprints that are made available under license terms that allow reuse.
- **Paper:** *link to the dataset paper if available*
- **Data:** https://www.ncbi.nlm.nih.gov/pmc/tools/openftlist/
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Comments:
In the framework of BigScience:
- bigscience-workshop/data_tooling#121

I have created this dataset as a community dataset: https://huggingface.co/datasets/albertvillanova/pmc_open_access

However, I was wondering that it may be more appropriate to move it under an org namespace: `pubmed_central` or `pmc`
This way, we could add other datasets I'm also working on: Author Manuscript Dataset, Historical OCR Dataset, LitArch Open Access Subset.

What do you think? @lhoestq @mariosasko 

Why not ! Having them under such namespaces would also help people searching for this kind of datasets.
We can also invite people from pubmed at one point

DONE: https://huggingface.co/datasets/pmc/open_access
Title: Fix to_tf_dataset references in docs
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2022-01-03T15:31:39Z

Fix the `to_tf_dataset` references in the docs. The currently failing example of usage will be fixed by #3338.

Comments:
The code snippet in [this section](https://huggingface.co/docs/datasets/master/use_dataset.html?highlight=to_tf_dataset#tensorflow) is missing an import (`DataCollatorWithPadding`) and doesn't initialize the TF model before the `model.fit` call.
Title: No Data format found
Author: shazzad47 (NONE)
State: closed
Created: 2022-01-03T09:41:11Z
Labels: dataset-viewer

## Dataset viewer issue for '*name of the dataset*'

**Link:** *link to the dataset viewer page*

*short description of the issue*

Am I the one who added this dataset ? Yes-No

Comments:
Hi, which dataset is giving you an error?
Title: Dataset
Author: MIKURI0114 (NONE)
State: closed
Created: 2022-01-03T02:03:23Z
Labels: dataset-viewer

## Dataset viewer issue for '*name of the dataset*'

**Link:** *link to the dataset viewer page*

*short description of the issue*

Am I the one who added this dataset ? Yes-No

Comments:
Can you reopen with the correct dataset name (if relevant)?

Thanks

The dataset viewer was down tonight. It works again.