rIsHu009/github-issues-updated huggingface dataset:
Title: Error of writing with different schema, due to nonpreservation of nullability
Author: richarddwang (CONTRIBUTOR)
State: closed
Created: 2022-02-24T08:23:07Z

## 1. Case
```
dataset.map(
  batched=True,
  disable_nullable=True,
)
```
will get the following error at here https://github.com/huggingface/datasets/blob/c9967f55626931f8059dc416526c791444cdfdf7/src/datasets/arrow_writer.py#L516

`pyarrow.lib.ArrowInvalid: Tried to write record batch with different schema`

## 2. Debugging
### 2.1 tracing
During `_map_single`, the following are called
https://github.com/huggingface/datasets/blob/c9967f55626931f8059dc416526c791444cdfdf7/src/datasets/arrow_dataset.py#L2523
https://github.com/huggingface/datasets/blob/c9967f55626931f8059dc416526c791444cdfdf7/src/datasets/arrow_writer.py#L511
### 2.2. Observation
The problem is, even after `table_cast`, `pa_table.schema != self._schema`
`pa_table.schema` (before/after `table_cast`)
```
input_ids: list<item: int32>
  child 0, item: int32
```
`self._schema`
```
input_ids: list<item: int32> not null
  child 0, item: int32
```
### 2.3. Reason
https://github.com/huggingface/datasets/blob/c9967f55626931f8059dc416526c791444cdfdf7/src/datasets/table.py#L1121
Here we lose nullability stored in `schema` because it seems that `Features` is always nullable and don't store nullability.
https://github.com/huggingface/datasets/blob/c9967f55626931f8059dc416526c791444cdfdf7/src/datasets/table.py#L1103
So, casting to schema from such `Features` loses nullability, and eventually causes error of writing with different schema

## 3. Solution
1. Let `Features` stores nullability.
2. Directly cast table with original schema but not schema from converted `Features`. (this PR)
3. Don't `cast_table` when `write_table`

Comments:
Hi ! Thanks for reporting, indeed `disable_nullable` doesn't seem to be supported in this case. Maybe at one point we can have `disable_nullable` as a parameter of certain feature types
Title: Reddit dataset card additions
Author: anna-kay (CONTRIBUTOR)
State: closed
Created: 2022-02-23T21:29:16Z

The changes proposed are based on the "TL;DR: Mining Reddit to Learn Automatic Summarization" paper & https://zenodo.org/record/1043504#.YhaKHpbQC38
It is a Reddit dataset indeed, but the name given to the dataset by the authors is Webis-TLDR-17 (corpus), so perhaps it should be modified as well. 
The task at which the dataset is aimed is abstractive summarization.

Comments:
Hello! I added the tags and created a PR. Just to note, regarding the paperswithcode_id tag, that currently has the value "reddit"; the dataset described as reddit in paperswithcode is https://paperswithcode.com/dataset/reddit and it isn't the Webis-tldr-17. I could not find Webis-tldr-17 in paperswithcode neither in the Summarization category nor using the keywords reddit, webis, & tldr. I didn't change this tag.
Title: Not be able to download dataset - "Newsroom"
Author: Darshan2104 (NONE)
State: closed
Created: 2022-02-23T10:15:50Z
Labels: dataset bug

Hello,

I tried to download the **newsroom** dataset but it didn't work out for me. it said me to **download it manually**!

For manually, Link is also didn't work! It is sawing some ad or something!

If anybody has solved this issue please help me out or if somebody has this dataset please share your google drive link, it would be a great help!

Thanks
Darshan Tank

Comments:
Hi @Darshan2104, thanks for reporting.

Please note that at Hugging Face we do not host the data of this dataset, but just a loading script pointing to the host of the data owners.

Apparently the data owners changed their data host server. After googling it, I found their new website at: https://lil.nlp.cornell.edu/newsroom/index.html
- Download page: https://lil.nlp.cornell.edu/newsroom/download/index.html

I'm fixing the link in our Datasets library.

@albertvillanova Thanks for the solution and link you made my day!
Title: Start removing canonical datasets logic
Author: lhoestq (MEMBER)
State: closed
Created: 2022-02-22T18:23:30Z

I updated the source code and the documentation to start removing the "canonical datasets" logic.

Indeed this makes the documentation confusing and we don't want this distinction anymore in the future. Ideally users should share their datasets on the Hub directly.

### Changes

- the documentation about dataset loading mentions the datasets on the Hub (no difference between canonical and community, since they all have their own repository now)
- the documentation about adding a dataset doesn't explain the technical differences between canonical and community anymore, and only presents how to add a community dataset. There is still a small section at the bottom that mentions the datasets that are still on GitHub and redirects to the `ADD_NEW_DATASET.md` guide on GitHub about how to contribute a dataset to the `datasets` library
- the code source doesn't mention "canonical" anymore anywhere. There is still a `GitHubDatasetModuleFactory` class that is left, but I updated the docstring to say that it will be eventually removed in favor of the `HubDatasetModuleFactory` classes that already exist

Would love to have your feedbacks on this !

cc @julien-c @thomwolf @SBrandeis

Comments:
I'm not sure if the documentation explains why the dataset identifiers might have a namespace or not (the user/org): 'glue' vs 'severo/glue'. Do you think we should explain it, and relate it to the GitHub/Hub distinction?

> I'm not sure if the documentation explains why the dataset identifiers might have a namespace or not (the user/org): 'glue' vs 'severo/glue'. Do you think we should explain it, and relate it to the GitHub/Hub distinction?

I added an explanation, let me know if it sounds good to you:

```
Datasets used to be hosted on our GitHub repository, but all datasets have now been migrated to the Hugging Face Hub.
The legacy GitHub datasets were added originally on our GitHub repository and therefore don't have a namespace: "squad", "glue", etc. unlike the other datasets that are named "username/dataset_name" or "org/dataset_name".
```


Thanks for the feedbacks ! Merging this now - if you have some comments I can take care of them in a subsequent PR

I'll also take care of resolving the conflicts with https://github.com/huggingface/datasets/pull/3690
Title: Allow download only some files from the Wikipedia dataset
Author: jvanz (NONE)
State: open
Created: 2022-02-22T13:46:41Z
Labels: enhancement

**Is your feature request related to a problem? Please describe.**
The Wikipedia dataset can be really big. This is a problem if you want to use it locally in a laptop with the Apache Beam `DirectRunner`. Even if your laptop have a considerable amount of memory (e.g. 32gb).


**Describe the solution you'd like**
I would like to use the `data_files` argument in the `load_dataset` function to define which file in the wikipedia dataset I would like to download. Thus, I can work with the dataset in a smaller machine using the Apache Beam `DirectRunner`.

**Describe alternatives you've considered**
I've tried to use the `simple` Wikipedia dataset. But it's in English and I would like to use Portuguese texts in my model.

Comments:
Hi @jvanz, thank you for your proposal.

In fact, we are aware that it is very common the problem you mention. Because of that, we are currently working in implementing a new version of wikipedia on the Hub, with all data preprocessed (no need to use Apache Beam), from where you will be able to use `data_files` to load only a specific subset of the data files.

See:
- #3401
Title: Update gigaword card and info
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2022-02-22T12:27:16Z

Reported on the forum: https://discuss.huggingface.co/t/error-loading-dataset/14999

Comments:
I think it actually comes from an issue here:

https://github.com/huggingface/datasets/blob/810b12f763f5cf02f2e43565b8890d278b7398cd/src/datasets/utils/file_utils.py#L575-L579

and 

https://github.com/huggingface/datasets/blob/810b12f763f5cf02f2e43565b8890d278b7398cd/src/datasets/utils/streaming_download_manager.py#L386-L389

This code doesn't seem to work anymore. This can probably be fixed with

```python
if url.startswith("https://drive.google.com/"): 
    url += "&confirm=t"
    cookies = response.cookies 
```

because Google Drive doesn't return the `download_warning` cookie anymore.

Actually it seems that is has been fixed already in https://github.com/huggingface/datasets/pull/3787 :)

I think it should have fixed the gigaword dataset loading

@lhoestq The linked PR indeed fixes the issue. This PR is still worth merging IMO to update `gigaword`'s card.
Title: DuplicatedKeysError on msr_sqa dataset
Author: kolk (NONE)
State: closed
Created: 2022-02-22T00:43:33Z

### Describe the bug
Failure to generate dataset msr_sqa because of duplicate keys.

### Steps to reproduce the bug
```
from datasets import load_dataset
load_dataset("msr_sqa")
```
### Expected results
The examples keys should be unique.

**Actual results**
```
>>> load_dataset("msr_sqa")
Downloading:
6.72k/? [00:00<00:00, 148kB/s]
Downloading:
2.93k/? [00:00<00:00, 53.8kB/s]
Using custom data configuration default
Downloading and preparing dataset msr_sqa/default (download: 4.57 MiB, generated: 26.25 MiB, post-processed: Unknown size, total: 30.83 MiB) to /root/.cache/huggingface/datasets/msr_sqa/default/0.0.0/70b2a497bd3cc8fc960a3557d2bad1eac5edde824505e15c9c8ebe4c260fd4d1...
Downloading: 100%
4.80M/4.80M [00:00<00:00, 7.49MB/s]
---------------------------------------------------------------------------
DuplicatedKeysError                       Traceback (most recent call last)
[/usr/local/lib/python3.7/dist-packages/datasets/builder.py](https://localhost:8080/#) in _prepare_split(self, split_generator)
   1080                     example = self.info.features.encode_example(record)
-> 1081                     writer.write(example, key)
   1082             finally:

8 frames
DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: nt-639
Keys should be unique and deterministic in nature

During handling of the above exception, another exception occurred:

DuplicatedKeysError                       Traceback (most recent call last)
[/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py](https://localhost:8080/#) in check_duplicate_keys(self)
    449         for hash, key in self.hkey_record:
    450             if hash in tmp_record:
--> 451                 raise DuplicatedKeysError(key)
    452             else:
    453                 tmp_record.add(hash)

DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: nt-639
Keys should be unique and deterministic in nature
```

### Environment info
datasets version: 1.18.3
Platform: Google colab notebook
Python version: 3.7
PyArrow version: 6.0.1

Comments:
Thanks for reporting, @kolk.

We are fixing it.
Title: `dataset = dataset.map()` causes faiss index lost
Author: Oaklight (NONE)
State: open
Created: 2022-02-21T21:59:23Z
Labels: bug

## Describe the bug
assigning the resulted dataset to original dataset causes lost of the faiss index

## Steps to reproduce the bug
`my_dataset` is a regular loaded dataset. It's a part of a customed dataset structure
```python
self.dataset.add_faiss_index('embeddings')
self.dataset.list_indexes()
# ['embeddings']

dataset2 = my_dataset.map(
    lambda x: self._get_nearest_examples_batch(x['text']), batch=True
)

# the unexpected result:
dataset2.list_indexes()
# []

self.dataset.list_indexes()
# ['embeddings']
```

in case something wrong with my `_get_nearest_examples_batch()`, it's like this
```python
def _get_nearest_examples_batch(self, examples, k=5):
    queries = embed(examples)
    scores_batch, retrievals_batch = self.dataset.get_nearest_examples_batch(self.faiss_column, queries, k)
    return {
        'neighbors': [batch['text'] for batch in retrievals_batch],
        'scores': scores_batch
    }
```

## Expected results
`map` shouldn't drop the indexes, in another word, indexes should be carried to the generated dataset

## Actual results
map drops the indexes

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Ubuntu 20.04.3 LTS
- Python version: 3.8.12
- PyArrow version: 7.0.0

Comments:
Hi ! Indeed `map` is dropping the index right now, because one can create a dataset with more or fewer rows using `map` (and therefore the index might not be relevant anymore)

I guess we could check the resulting dataset length, and if the user hasn't changed the dataset size we could keep the index, what do you think ?

doing `.add_column("x",x_data)` also removes the index. the new column might be irrelevant to the index so I don't think it should drop. 

Minimal example

```python
from datasets import load_dataset
import numpy as np

data=load_dataset("ceyda/cats_vs_dogs_sample") #just a test dataset
data=data["train"]
embd_data=data.map(lambda x: {"emb":np.random.uniform(-1,0,50).astype(np.float32)})
embd_data.add_faiss_index(column="emb")
print(embd_data.list_indexes())
embd_data=embd_data.add_column("x",[0]*data.num_rows)
print(embd_data.list_indexes())
```

I agree `add_column` shouldn't drop the index indeed ! Is it something you'd like to contribute ? I think it's just a matter of copying the `self._indexes` dictionary to the output dataset
Title: `Dataset.class_encode` should support custom class names
Author: Dref360 (CONTRIBUTOR)
State: closed
Created: 2022-02-19T21:21:45Z
Labels: enhancement

I can make a PR, just wanted approval before starting.

**Is your feature request related to a problem? Please describe.**

It is often the case that classes are not ordered in alphabetical order. Current `class_encode_column` sort the classes before indexing.
https://github.com/huggingface/datasets/blob/master/src/datasets/arrow_dataset.py#L1235

**Describe the solution you'd like**
I would like to add a **optional** parameter `class_names` to `class_encode_column` that would  be used for the mapping instead of sorting the unique values.

**Describe alternatives you've considered**

One can use map instead. I find it harder to read.
```python
CLASS_NAMES = ['apple', 'orange', 'potato']
ds = ds.map(lambda item: CLASS_NAMES.index(item[label_column]))

# Proposition
ds = ds.class_encode_column(label_column, CLASS_NAMES)
```

**Additional context**

I can make the PR if this feature is accepted.

Comments:
Hi @Dref360, thanks a lot for your proposal.

It totally makes sense to have more flexibility when class encoding, I agree.

You could even further customize the class encoding by passing an instance of `ClassLabel` itself (instead of replicating `ClassLabel` instantiation arguments as `Dataset.class_encode_column` arguments).

And the latter made me think of `Dataset.cast_column`...

Maybe better to have some others' opinions @lhoestq @mariosasko 

Hi @Dref360! You can use [`Dataset.align_labels_with_mapping`](https://huggingface.co/docs/datasets/master/package_reference/main_classes.html#datasets.Dataset.align_labels_with_mapping) after `Dataset.class_encode_column` to assign a different mapping of labels to ids.

@albertvillanova I'd like to avoid adding more complexity to the API where it's not (absolutely) needed, so I don't think introducing a new param in `Dataset.class_encode_column` is a good idea.



I wasn't aware that it existed thank you for the link.

Closing then!
Title: Know your data for HF hub 
Author: Muhtasham (NONE)
State: closed
Created: 2022-02-19T19:48:47Z
Labels: enhancement

**Is your feature request related to a problem? Please describe.**
Would be great to see be able to understand datasets with the goal of improving data quality, and helping mitigate fairness and bias issues.

**Describe the solution you'd like**
Something like https://knowyourdata.withgoogle.com/ for HF hub

Comments:
Hi @Muhtasham you should take a look at https://huggingface.co/blog/data-measurements-tool and accompanying demo app at https://huggingface.co/spaces/huggingface/data-measurements-tool

We would be interested in your feedback. cc @meg-huggingface @sashavor @yjernite
Title: Unable to view the Gradio flagged call back dataset
Author: kingabzpro (NONE)
State: closed
Created: 2022-02-19T17:45:08Z
Labels: dataset-viewer

## Dataset viewer issue for '*savtadepth-flags*'

**Link:** *[savtadepth-flags](https://huggingface.co/datasets/kingabzpro/savtadepth-flags)*

*with the Gradio 2.8.1 the dataset viers stopped working. I tried to add values manually but its not working. The dataset is also not showing the link with the app https://huggingface.co/spaces/kingabzpro/savtadepth.*

Am I the one who added this dataset ? Yes

Comments:
Hi @kingabzpro.

I think you need to create a loading script that creates the dataset from the CSV file and the image paths.

As example, you could have a look at the Food-101 dataset: https://huggingface.co/datasets/food101
- Loading script: https://huggingface.co/datasets/food101/blob/main/food101.py

Once the loading script is created, the viewer will show a previsualization of your dataset. 

@albertvillanova I don't think this is the issue. I have created another dataset with similar files and format and it works. https://huggingface.co/datasets/kingabzpro/savtadepth-flags-V2

Yes, you are right, that was not the issue.

Just take into account that sometimes the viewer can take some time until it shows the preview of the dataset.
After some time, yours is finally properly shown: https://huggingface.co/datasets/kingabzpro/savtadepth-flags

The problem was resolved by deleted the dataset and creating new one with similar name and then clicking on flag button.

I think if you make manual changes to dataset the whole system breaks.
Title: Rename GenerateMode to DownloadMode
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-02-18T16:53:53Z

This PR:
- Renames `GenerateMode` to `DownloadMode`
- Implements `DeprecatedEnum`
- Deprecates `GenerateMode`

Close #769.

Comments:
Thanks! Used here: https://github.com/huggingface/datasets-preview-backend/blob/main/src/datasets_preview_backend/models/dataset.py#L26 :)
Title: head_qa file missing
Author: severo (COLLABORATOR)
State: closed
Created: 2022-02-18T16:32:43Z
Labels: bug

## Describe the bug

A file for the `head_qa` dataset is missing (https://drive.google.com/u/0/uc?export=download&id=1a_95N5zQQoUCq8IBNVZgziHbeM-QxG2t/HEAD_EN/train_HEAD_EN.json)

## Steps to reproduce the bug
```python
>>> from datasets import load_dataset
>>> load_dataset("head_qa", name="en")
```

## Expected results

The dataset should be loaded

## Actual results

```
Downloading and preparing dataset head_qa/en (download: 75.69 MiB, generated: 2.69 MiB, post-processed: Unknown size, total: 78.38 MiB) to /home/slesage/.cache/huggingface/datasets/head_qa/en/1.1.0/583ab408e8baf54aab378c93715fadc4d8aa51b393e27c3484a877e2ac0278e9...
Downloading data: 2.21kB [00:00, 2.05MB/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/load.py", line 1729, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/builder.py", line 594, in download_and_prepare
    self._download_and_prepare(
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/builder.py", line 665, in _download_and_prepare
    verify_checksums(
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/utils/info_utils.py", line 40, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/u/0/uc?export=download&id=1a_95N5zQQoUCq8IBNVZgziHbeM-QxG2t']
```

## Environment info
- `datasets` version: 1.18.4.dev0
- Platform: Linux-5.11.0-1028-aws-x86_64-with-glibc2.31
- Python version: 3.9.6
- PyArrow version: 6.0.1

Comments:
We usually find issues with files hosted at Google Drive...

In this case we download the Google Drive Virus scan warning instead of the data file.

Fixed: https://huggingface.co/datasets/head_qa/viewer/en/train. Thanks

<img width="1551" alt="Capture d’écran 2022-02-28 à 15 29 04" src="https://user-images.githubusercontent.com/1676121/156000224-fd3f62c6-8b54-4df1-8911-bdcb0bac3f1a.png">
Title: Add perplexity to metrics
Author: emibaylor (CONTRIBUTOR)
State: closed
Created: 2022-02-18T15:52:23Z

Adding perplexity metric

This code differs from the code in [this](https://huggingface.co/docs/transformers/perplexity) HF blog post because the blogpost code fails in at least the following circumstances:
- returns nans whenever the stride = 1
- hits a runtime error when the stride is significantly larger than the max model length (e.g. if max_model_length = 512 and stride = 1024)


Note that:
- As it is, it only works for causal models. Pseudoperplexity can be added later as another metric to work with masked language models.
- It takes in a list of strings so that it can be dataset independent. This does mean that it doesn't currently batch inputs, and is therefore relatively slow.
- It overwrites the metrics compute() function for a specific perplexity compute() function. This is because the current general metrics compute() function requires model-generated predictions, which doesn't make sense in the context of perplexity

Comments:
Awesome thank you ! The implementation of the parent `Metric` class was assuming that all metrics were supposed to have references/predictions pairs - I just changed that so you don't have to override `compute()`. I took the liberty of doing the changes directly inside this PR to make sure it works as expected with perplexity.

Other than that it looks in pretty good shape :) I just did minor changes like remove a remaining `print` as well as fixing the `Features` defined in `_info()`. I also renamed `input_text` to `input_texts` since it makes it more obvious that it's a list of strings - let me know if it sounds good to you.

Let me know if you'd like to make other changes or if it's all good for you !

The test with the full test set seems to take too much time in the CI - maybe we can just select `split="test[:few_examples]"` (around 10 maybe ?)
Title: Images get decoded when using `map()` with `input_columns` argument on a dataset
Author: kklemon (NONE)
State: closed
Created: 2022-02-18T15:35:38Z
Labels: bug

## Describe the bug

The `datasets.features.Image` feature class decodes image data by default. Expectedly, when indexing a dataset or using the `map()` method, images are returned as PIL Image instances.

However, when calling `map()` and setting a specific data column with the `input_columns` argument, the image data is passed as raw byte representation to the mapping function.

## Steps to reproduce the bug
```python
from datasets import load_dataset
from torchvision import transforms
from PIL.Image import Image

dataset = load_dataset('mnist', split='train')

def transform_all_columns(example):
    # example['image'] is encoded as PIL Image
    assert isinstance(example['image'], Image)
    return example

def transform_image_column(image):
    # image is decoded here and represented as raw bytes
    assert isinstance(image, Image)
    return image

# single-sample dataset for debugging purposes
dev = dataset.select([0])

dev.map(transform_all_columns)
dev.map(transform_image_column, input_columns='image')
```

## Expected results

Image data should be passed in decoded form, i.e. as PIL Image objects to the mapping function unless the `decode` attribute on the image feature is set to `False`.

## Actual results

The mapping function receives images as raw byte data.

## Environment info
- `datasets` version: 1.18.3
- Platform: Linux-5.11.0-49-generic-x86_64-with-glibc2.32
- Python version: 3.8.0b4
- PyArrow version: 7.0.0

Comments:
Hi! If I'm not mistaken, this behavior is intentional, but I agree it could be more intuitive.

@albertvillanova Do you remember why you decided not to decode columns in the `Audio` feature PR when `input_columns` is not `None`? IMO we should decode those columns, and we don't even have to use lazy structures here because the user explicitly requires them in the map transform. 

cc @lhoestq for visibility

I think I excluded to decorate the function when `input_columns` were passed as a quick fix for some non-passing tests: 
- https://github.com/huggingface/datasets/pull/2324/commits/9d7c3e8fa53e23ec636859b4407eeec904b1b3f9

That PR was quite complex and I decided to focus on the main feature requests, leaving refinements for subsequent PRs.

Note that when `input_columns` are passed, the signature of the function is effectively changed, while the decorated function expects an item (whether an example or a batch) as first arg (which is not the case when passing `input_columns`.

I agree we should consider supporting the case when `input_columns` are passed.
Title: Cannot preview dataset
Author: frascuchon (NONE)
State: closed
Created: 2022-02-18T13:06:45Z
Labels: dataset-viewer

## Dataset viewer issue for '*rubrix/news*'

**Link:https://huggingface.co/datasets/rubrix/news** *link to the dataset viewer page*

Cannot see the dataset preview:

```
Status code:   400
Exception:     Status400Error
Message:       Not found. Cache is waiting to be refreshed.
```
Am I the one who added this dataset ? No

Comments:
Thanks for reporting. The dataset viewer depends on some backend treatments, and for now, they might take some hours to get processed. We're working on improving it.

It has finally been processed. Thanks for the patience.

Thanks for the info @severo !
Title: Overflowing indices in `select`
Author: lvwerra (MEMBER)
State: closed
Created: 2022-02-18T11:30:52Z
Labels: bug

## Describe the bug
The `Dataset.select` function seems to accept indices that are larger than the dataset size and seems to effectively use `index %len(ds)`.

## Steps to reproduce the bug
```python
from datasets import Dataset

ds = Dataset.from_dict({"test": [1,2,3]})
ds = ds.select(range(5))

print(ds)
print()
print(ds["test"])
```
Result:
```python
Dataset({
    features: ['test'],
    num_rows: 5
})

[1, 2, 3, 1, 2]
```

This behaviour is not documented and can lead to unexpected behaviour when for example taking a sample larger than the dataset and thus creating a lot of duplicates.

## Expected results
It think this should throw an error or at least a very big warning:
```python
IndexError: Invalid key: 5 is out of bounds for size 3
```

## Environment info
- `datasets` version: 1.18.3
- Platform: macOS-12.0.1-x86_64-i386-64bit
- Python version: 3.9.10
- PyArrow version: 7.0.0

Comments:
Fixed on master (see https://github.com/huggingface/datasets/pull/3719).

Awesome, I did not find that one! Thanks.
Title: Fix typo in train split name
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-02-18T08:18:04Z

In the README guide (and consequently in many datasets) there was a typo in the train split name:
```
| Tain   | Valid | Test |
```

This PR:
- fixes the typo in the train split name
- fixes the column alignment of the split tables

in the README guide and in all datasets.

Comments:
Title: `NonMatchingSplitsSizesError` for cats_vs_dogs dataset
Author: jaketae (CONTRIBUTOR)
State: closed
Created: 2022-02-18T05:46:39Z
Labels: bug

## Describe the bug

Cannot download cats_vs_dogs dataset due to `NonMatchingSplitsSizesError`.

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("cats_vs_dogs")
```

## Expected results

Loading is successful.

## Actual results

```
NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=7503250, num_examples=23422, dataset_name='cats_vs_dogs'), 'recorded': SplitInfo(name='train', num_bytes=7262410, num_examples=23410, dataset_name='cats_vs_dogs')}]
```

## Environment info

Reproduced on a fresh [Colab notebook](https://colab.research.google.com/drive/13GTvrSJbBGvL2ybDdXCBZwATd6FOkMub?usp=sharing).

## Additional Context

Originally reported in https://github.com/huggingface/transformers/issues/15698.

cc @mariosasko

Comments:
Thnaks for reporting @jaketae. We are fixing it.
Title: Add tqdm arguments
Author: penguinwang96825 (NONE)
State: closed
Created: 2022-02-18T01:34:46Z

In this PR, tqdm arguments can be passed to the map() function and such, in order to be more flexible.

Comments:
Hi ! Thanks this will be very useful :)

It looks like there are some changes in the github diff that are not related to your contribution, can you try fixing this by merging `master` into your PR, or create a new PR from an updated version of `master` ?

I have already solved the conflict on this latest version. This is my first time sending PR, if there's anything I need to adjust just let me know~

Thanks, most changes are gone :)
It still seems to include changes though - do you mind try creating a new branch from upstream/master and create a new PR please ?

Yeah sure, I'll try to send a new PR today!

Please forward to [#3850](https://github.com/huggingface/datasets/pull/3850)

Thanks ! Closing this one in favor of https://github.com/huggingface/datasets/pull/3850/files
Title: Passing invalid subset should throw an error
Author: jxmorris12 (CONTRIBUTOR)
State: open
Created: 2022-02-17T18:16:11Z
Labels: bug

## Describe the bug
Only some datasets have a subset (as in `load_dataset(name, subset)`). If you pass an invalid subset, an error should be thrown.

## Steps to reproduce the bug
```python
import datasets
datasets.load_dataset('rotten_tomatoes', 'asdfasdfa')
```

## Expected results
This should break, since `'asdfasdfa'` isn't a subset of the `rotten_tomatoes` dataset.

## Actual results
This API call silently succeeds.

Comments:
Title: Use the same seed to shuffle shards and metadata in streaming mode
Author: lhoestq (MEMBER)
State: closed
Created: 2022-02-17T17:06:31Z

When shuffling in streaming mode, those two entangled lists are shuffled independently. In this PR I changed this to shuffle the lists of same length with the exact same seed, in order for the files and metadata to still be aligned.
```python
gen_kwargs = {
    "files": [os.path.join(data_dir, filename) for filename in all_files],
    "metadata_files": [all_metadata[filename] for filename in all_files],
}
```
IMO this is important to avoid big but silent issues.

Fix https://github.com/huggingface/datasets/issues/3744

Comments:
Title: Add mIoU metric
Author: NielsRogge (CONTRIBUTOR)
State: closed
Created: 2022-02-17T15:52:17Z

This PR adds the mean Intersection-over-Union metric to the library, useful for tasks like semantic segmentation.

It is entirely based on mmseg's [implementation](https://github.com/open-mmlab/mmsegmentation/blob/master/mmseg/core/evaluation/metrics.py).

I've removed any PyTorch dependency, and rely on Numpy only.

Comments:
Hmm the doctest failed again - maybe the full result needs to be on one single line

cc @lhoestq for the final review

Cool ! Feel free to merge if it's all good for you
Title: Better shards shuffling in streaming mode
Author: lhoestq (MEMBER)
State: closed
Created: 2022-02-17T15:07:21Z
Labels: enhancement

streaming

Sometimes a dataset script has a `_split_generators` that returns several files as well as the corresponding metadata of each file. It often happens that they end up in two separate lists in the `gen_kwargs`:
```python
gen_kwargs = {
    "files": [os.path.join(data_dir, filename) for filename in all_files],
    "metadata_files": [all_metadata[filename] for filename in all_files],
}
```
It happened for Multilingual Spoken Words for example in #3666

However currently **the two lists are shuffled independently** when shuffling the shards in streaming mode. This leads to `_generate_examples` not having the right metadata for each file.

To prevent this issue I suggest that we always shuffle lists of the same length the exact same way to avoid such a big but silent issue.

cc @polinaeterna

Comments:
Title: Fix ValueError message formatting in int2str
Author: aaakulchyk (CONTRIBUTOR)
State: closed
Created: 2022-02-17T10:50:08Z

Hi!

I bumped into this particular `ValueError` during my work (because an instance of `np.int64` was passed instead of regular Python `int`), and so I had to `print(type(values))` myself. Apparently, it's just the missing `f` to make message an f-string.

It ain't much for a contribution, but it's honest work. Hope it spares someone else a few seconds in the future 😃

Comments:
Title: Pubmed dataset does not work in streaming mode
Author: abhi-mosaic (CONTRIBUTOR)
State: closed
Created: 2022-02-16T17:13:37Z
Labels: bug

## Describe the bug
Trying to use the `pubmed` dataset with `streaming=True` fails.

## Steps to reproduce the bug
```python
import datasets
pubmed_train = datasets.load_dataset('pubmed', split='train', streaming=True)
print (next(iter(pubmed_train)))
```

## Expected results
I would expect to see the first training sample from the pubmed dataset. 

## Actual results
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/abhinav/Documents/mosaicml/mosaicml_venv/lib/python3.8/site-packages/datasets/iterable_dataset.py", line 367, in __iter__
    for key, example in self._iter():
  File "/Users/abhinav/Documents/mosaicml/mosaicml_venv/lib/python3.8/site-packages/datasets/iterable_dataset.py", line 364, in _iter
    yield from ex_iterable
  File "/Users/abhinav/Documents/mosaicml/mosaicml_venv/lib/python3.8/site-packages/datasets/iterable_dataset.py", line 79, in __iter__
    for key, example in self.generate_examples_fn(**self.kwargs):
  File "/Users/abhinav/.cache/huggingface/modules/datasets_modules/datasets/pubmed/9715addf10c42a7877a2149ae0c5f2fddabefc775cd1bd9b03ac3f012b86ce46/pubmed.py", line 373, in _generate_examples
    tree = etree.parse(filename)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/xml/etree/ElementTree.py", line 1202, in parse
    tree.parse(source, parser)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/xml/etree/ElementTree.py", line 584, in parse
    source = open(source, "rb")
FileNotFoundError: [Errno 2] No such file or directory: 'gzip://pubmed21n0001.xml::ftp://ftp.ncbi.nlm.nih.gov/pubmed/baseline/pubmed21n0001.xml.gz'
```
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.2
- Platform: macOS-11.4-x86_64-i386-64bit
- Python version: 3.8.2
- PyArrow version: 6.0.0

## Comments
The error looks like an issue with `open` vs. `xopen` inside the `xml` package. It looks like it's trying to open the remote source URL, which has been edited with prefix `gzip://...`. 

Maybe there can be an explicit `xopen` before passing the raw data to `etree`, something like:

```python
# Before
tree = etree.parse(filename)
root = tree.getroot()

# After
with xopen(filename) as f:
  data_str = f.read()
root = etree.fromstring(data_str)
```

Comments:
Thanks for reporting, @abhi-mosaic (related to #3655).

Please note that `xml.etree.ElementTree.parse` already supports streaming:
- #3476

No need to refactor to use `open`/`xopen`. Is is enough with importing the package `as ET` (instead of `as etree`).
Title: Local paths in common voice
Author: lhoestq (MEMBER)
State: closed
Created: 2022-02-16T15:01:29Z

Continuation of https://github.com/huggingface/datasets/pull/3664:

- pass the `streaming` parameter to _split_generator
- update @anton-l's code to use this parameter for `common_voice`
- add a comment to explain why we use `download_and_extract` in non-streaming and `iter_archive` in streaming

Now the `common_voice` dataset has a local path back in `ds["path"]`, and this field is `None` in streaming mode.

cc @patrickvonplaten @anton-l @albertvillanova 

Fix #3663.

Comments:
I just changed to `dl_manager.is_streaming` rather than an additional parameter `streaming` that has to be handled by the DatasetBuilder class - this way the streaming logic doesn't interfere with the base builder's code.

I think it's better this way, but let me know if you preferred the previous way and I can revert

> But on the other hand, IMHO, I think this specific solution adds complexity to handling streaming/non-streaming, and moves this complexity to the loading script and thus to the contributors/users who want to create the loading script for their canonical/community datasets (instead of keeping it hidden form the end users).

I'm down to discuss this more in the future !

@lhoestq good idea: much cleaner this way! That way each class has its own responsibilities without mixing around...
Title: Bugs in NewsQA dataset
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-02-16T13:17:37Z
Labels: bug

## Describe the bug
NewsQA dataset has the following bugs:
- the field `validated_answers` is an exact copy of the field `answers` but with the addition of `'count': [0]` to each dict
- the field `badQuestion` does not appear in `answers` nor `validated_answers`

## Steps to reproduce the bug
By inspecting the dataset script we can see that:
- the parsing of `validated_answers` is a copy-paste of the one for `answers`
- the `badQuestion` field is ignored in the parsing of both `answers` and `validated_answers`

Comments:
Title: Support streaming in size estimation function in `push_to_hub`
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2022-02-16T13:10:48Z

This PR adds the streamable version of `os.path.getsize` (`fsspec` can return `None`, so we fall back to `fs.open` to make it more robust) to account for possible streamable paths in the nested `extra_nbytes_visitor` function inside `push_to_hub`.

Comments:
would this allow to include the size in the dataset info without downloading the files? related to https://github.com/huggingface/datasets/pull/3670

@severo I don't think so. We could use this to get `info.download_checksums["num_bytes"]`, but we must process the files to get the rest of the size info.
Title: Checksum Error when loading multi-news dataset
Author: byw2 (NONE)
State: closed
Created: 2022-02-16T05:11:08Z
Labels: bug

## Describe the bug
When using the load_dataset function from datasets module to load the Multi-News dataset, does not load the dataset but throws Checksum Error instead.

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("multi_news")
```


## Expected results
Should download and load Multi-News dataset.

## Actual results
Throws the following error and cannot load data successfully:

```
NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=1vRY2wM6rlOZrf9exGTm5pXj5ExlVwJ0C']
```

Could this issue please be looked at? Thanks!

Comments:
Thanks for reporting @byw2.
We are fixing it.
In the meantime, you can load the dataset by passing `ignore_verifications=True`:
 ```python
dataset = load_dataset("multi_news", ignore_verifications=True)
Title: Wrong number of examples when loading a text dataset
Author: kg-nlp (NONE)
State: closed
Created: 2022-02-16T01:13:31Z
Labels: bug

## Describe the bug
when I use load_dataset to read a txt file  I find  that the number of the samples is incorrect

## Steps to reproduce the bug
```
fr = open('train.txt','r',encoding='utf-8').readlines()
print(len(fr))  # 1199637

datasets = load_dataset('text', data_files={'train': ['train.txt']}, streaming=False)
print(len(datasets['train']))  # 1199649
```
I also use command line operation to verify it
```
$ wc -l train.txt 
1199637 train.txt
```

## Expected results
please fix that issue 


## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.8.3
- Platform:windows&linux
- Python version:3.7
- PyArrow version:6.0.1

Comments:
Hi @kg-nlp, thanks for reporting.

That is weird... I guess we would need some sample data file where this behavior appears to reproduce the bug for further investigation... 

ok, I found the reason why that two results are not same.
there is /u2029 in the text, the datasets will split sentence according to the /u2029,but when I use open function will not do that .
so I want to know which function shell do that
thanks
Title: VoxPopuli
Author: VictorSanh (CONTRIBUTOR)
State: closed
Created: 2022-02-15T23:04:55Z
Labels: dataset request

## Adding a Dataset
- **Name:** VoxPopuli
- **Description:** A Large-Scale Multilingual Speech Corpus
- **Paper:** https://arxiv.org/pdf/2101.00390.pdf
- **Data:** https://github.com/facebookresearch/voxpopuli
- **Motivation:** one of the largest (if not the largest) multilingual speech corpus: 400K hours of multilingual unlabeled speech + 17k hours of labeled speech

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

👀 @kahne @Molugan

Comments:
duplicate of https://github.com/huggingface/datasets/issues/2300
Title: Patch all module attributes in its namespace
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-02-15T17:12:27Z

When patching module attributes, only those defined in its `__all__` variable were considered by default (only falling back to `__dict__` if `__all__` was None).

However those are only a subset of all the module attributes in its namespace (`__dict__` variable).

This PR fixes the problem of modules that have non-None `__all__` variable, but try to access an attribute present in `__dict__` (and not in `__all__`).

For example, `pandas` has attribute `__version__` only present in `__dict__`.
- Before version 1.4, pandas `__all__` was None, thus all attributes in `__dict__` were patched
- From version 1.4, pandas `__all__` is not None, thus attributes in `__dict__` not present in `__all__` are ignored

Fix #3724.

CC: @severo @lvwerra

Comments:
Title: Bug while streaming CSV dataset with pandas 1.4
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-02-15T15:16:19Z
Labels: bug

## Describe the bug
If we upgrade to pandas `1.4`, the patching of the pandas module is no longer working
```
AttributeError: '_PatchedModuleObj' object has no attribute '__version__'
```

## Steps to reproduce the bug
```
pip install pandas==1.4
```
```python
from datasets import load_dataset
ds = load_dataset("lvwerra/red-wine", split="train", streaming=True)
item = next(iter(ds))
item
```

Comments:
Title: Fix flatten of complex feature types
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2022-02-15T14:45:33Z

Fix `flatten` for the following feature types: Image/Audio, Translation, and  TranslationVariableLanguages.

Inspired by `cast`/`table_cast`, I've introduced a `table_flatten` function to handle the Image/Audio types.

CC: @SBrandeis  

Fix #3686.

Comments:
Apparently the merge brought back some tests that use `flatten_()` that we removed recently

_The documentation is not available anymore as the PR was closed or merged._
Title: Multi-GPU support for `FaissIndex`
Author: rentruewang (CONTRIBUTOR)
State: closed
Created: 2022-02-14T17:26:51Z

Per #3716 , current implementation does not take into consideration that `faiss` can run on multiple GPUs.

In this commit, I provided multi-GPU support for `FaissIndex` by modifying the device management in `IndexableMixin.add_faiss_index` and `FaissIndex.load`.

Now users are able to pass in
1. a positive integer (as usual) to use 1 GPU
2. a negative integer `-1` to use all GPUs
3. a list of integers e.g. `[0, 1]` to run only on those GPUs
4. Of course, passing in nothing still runs on CPU.

This closes: #3716

Comments:
Any love?

Hi, any update?

@albertvillanova Sorry for bothering you again, quick follow up: is there anything else you want me to add / modify?

Hi @rentruewang , we updated the documentation on `master`, could you merge `master` into your branch please ?

@lhoestq I've merge `huggingface/datasets/master` into this PR. Please review. Thanks! 🤗

Edit: Umm... I was experimenting with what renaming a branch would do to a pull request. Please ignore the `closed this PR` down below. 🤗
Title: Fix Evidence Infer Treatment dataset
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-02-14T11:58:07Z

This PR:
- fixes a bug in the script, by removing an unnamed column with the row index: fix KeyError
- fix the metadata JSON, by adding both configurations (1.1 and 2.0): fix ExpectedMoreDownloadedFiles
- updates the dataset card

Fix #3515.

Comments:
Title: wrong condition in `Features ClassLabel encode_example`
Author: Tudyx (NONE)
State: closed
Created: 2022-02-14T11:44:35Z
Labels: bug

## Describe the bug

The `encode_example` function in *features.py* seems to have a wrong condition.

```python
if not -1 <= example_data < self.num_classes:
    raise ValueError(f"Class label {example_data:d} greater than configured num_classes {self.num_classes}")
```

## Expected results

The `not - 1` condition change the result of the condition. For instance, if  `example_data`  equals 4 and ` self.num_classes` equals 4 too, `example_data < self.num_classes` will give `False` as expected  . But if i add the  `not - 1`  condition, `not -1 <= example_data < self.num_classes` will give `True` and raise an exception.

## Environment info

- `datasets` version: 1.18.3
- Python version: 3.8.10
- PyArrow version: 7.00

Comments:
Hi @Tudyx, 

Please note that in Python, the boolean NOT operator (`not`) has lower precedence than comparison operators (`<=`, `<`), thus the expression you mention is equivalent to:
```python
 not (-1 <= example_data < self.num_classes)
```

Also note that as expected, the exception is raised if:
- `example_data < -1`
- or `example_data >= self.num_classes`

The raise of the exception is expected when `example_data` equals 4 and `self.num_classes` equals 4 too.
Title: `FaissIndex` to support multiple GPU and `custom_index`
Author: rentruewang (CONTRIBUTOR)
State: closed
Created: 2022-02-14T06:21:43Z
Labels: enhancement

**Is your feature request related to a problem? Please describe.**
Currently, because `device` is of the type `int | None`, to leverage `faiss-gpu`'s multi-gpu support, you need to create a `custom_index`. However, if using a `custom_index` created by e.g. `faiss.index_cpu_to_all_gpus`, then `FaissIndex.save` does not work properly because it checks the device id (which is an int, so no multiple GPUs).

**Describe the solution you'd like**
I would like `FaissIndex` to support multiple GPUs, by passing in a list to `add_faiss_index`.

**Describe alternatives you've considered**
Alternatively, I would like it to at least provide a warning cause it wasn't the behavior that I expected.

**Additional context**
Relavent source code here:
 https://github.com/huggingface/datasets/blob/6ed6ac9448311930557810383d2cfd4fe6aae269/src/datasets/search.py#L340-L349

Device management needs changing to support multiple GPUs, probably by `isinstance` calls.

I can provide a PR if you like :)

Thanks for reading!

Comments:
Hi @rentruewang, thansk for reporting and for your PR!!! We should definitely support this. 

@albertvillanova Great! :)