sjwsjw/github-issues huggingface dataset:
GitHub issue: Checking that split name is correct happens only after the data is downloaded
State: open

### Describe the bug

Verification of split names (=indexing data by split) happens after downloading the data. So when the split name is incorrect, users learn about that only after the data is fully downloaded, for large datasets it might take a lot of time.

### Steps to reproduce the bug

Load any dataset with random split name, for example:
```python
from datasets import load_dataset

load_dataset("mozilla-foundation/common_voice_11_0", "en", split="blabla")
```
and the download will start smoothly, despite there is no split named "blabla".

### Expected behavior

Raise error when split name is incorrect. 

### Environment info

`datasets==2.9.1.dev0`
GitHub issue: Minor changes in JAX-formatting docstrings & type-hints
State: closed

Hi to whoever is reading this! 🤗

## What's in this PR?

I was exploring the code regarding the `JaxFormatter` implemented in 🤗`datasets`, and found some things that IMO could be changed. Those are mainly regarding the docstrings and the type-hints based on `jax`'s 0.4.1 release where `jax.Array` was introduced as the default type for JAX-arrays (instead of `jnp.DeviceArray`, `jnp.SharedDeviceArray`, and `jnp.GlobalDeviceArray`). Even though `isinstance(..., jax.Array)` also works with lower versions such as e.g. `0.3.25`.

More information about the latter at [`jax` v0.4.1 - Release Notes](https://github.com/google/jax/releases/tag/jax-v0.4.1) and [jax.Array migration - JAX documentation](https://jax.readthedocs.io/en/latest/jax_array_migration.html).

## What's missing?

* Do you want me to write an entry in the documentation on how to use 🤗`datasets` with JAX as https://huggingface.co/docs/datasets/use_with_pytorch with PyTorch?
* Do we need to actually include `pyarrow` under the `TYPE_CHECKING` when needed? I just did it for JAX, but if we are OK with that, I can do that with the rest of the formatters, just LMK.
* Should the License header be included in `datasets.formatting.np_formatter`? If so, do I include the one from 2020 e.g. https://github.com/huggingface/datasets/blob/b065547654efa0ec633cf373ac1512884c68b2e1/src/datasets/formatting/tf_formatter.py#L1-L13
* Is there any reason why `jnp.array` is being used instead of `jnp.asarray`? There's no difference between both, just that `jnp.asarray` has `copy=False` as default, even though `numpy` to `jax.numpy` conversion is not zero-copy, but just asking :)
GitHub issue: ClassLabel.cast_storage raises TypeError when called on an empty IntegerArray
State: closed

### Describe the bug

`ClassLabel.cast_storage` raises `TypeError` when called on an empty `IntegerArray`.

### Steps to reproduce the bug

Minimal steps:
```python
import pyarrow as pa
from datasets import ClassLabel

ClassLabel(names=['foo', 'bar']).cast_storage(pa.array([], pa.int64()))
```

In practice, this bug arises in situations like the one below:
```python
from datasets import ClassLabel, Dataset, Features, Sequence

dataset = Dataset.from_dict({'labels': [[], []]}, features=Features({'labels': Sequence(ClassLabel(names=['foo', 'bar']))}))

# this raises TypeError
dataset.map(batched=True, batch_size=1)
```

### Expected behavior

`ClassLabel.cast_storage` should return an empty Int64Array.

### Environment info

- `datasets` version: 2.9.1.dev0
- Platform: Linux-4.15.0-1032-aws-x86_64-with-glibc2.27
- Python version: 3.10.6
- PyArrow version: 11.0.0
- Pandas version: 1.5.3
GitHub issue: Lint code with `ruff`
State: closed

EDIT:
Use `ruff` for linting instead of `isort` and `flake8` ~~`black`~~ to be consistent with [`transformers`](https://github.com/huggingface/transformers/pull/21480) and [`hfh`](https://github.com/huggingface/huggingface_hub/pull/1323).

TODO:
- [x] ~Merge the community contributors' PR to avoid having to run `make style` on their PR branches~ (we have some new PRs, but fixing those shouldn't be too big of a problem)
GitHub issue: `with_format("numpy")` silently downcasts float64 to float32 features
State: open

### Describe the bug

When I create a dataset with a `float64` feature, then apply numpy formatting the returned numpy arrays are silently downcasted to `float32`.

### Steps to reproduce the bug

```python
import datasets
dataset = datasets.Dataset.from_dict({'a': [1.0, 2.0, 3.0]}).with_format("numpy")
print("feature dtype:", dataset.features['a'].dtype)
print("array dtype:", dataset['a'].dtype)
```
output:
```
feature dtype: float64
array dtype: float32
```

### Expected behavior

```
feature dtype: float64
array dtype: float64
```

### Environment info

- `datasets` version: 2.8.0
- Platform: Linux-5.4.0-135-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 10.0.1
- Pandas version: 1.4.4

### Suggested Fix

Changing [the `_tensorize` function of the numpy formatter](https://github.com/huggingface/datasets/blob/b065547654efa0ec633cf373ac1512884c68b2e1/src/datasets/formatting/np_formatter.py#L32) to 
```python
    def _tensorize(self, value):

        if isinstance(value, (str, bytes, type(None))):
            return value
        elif isinstance(value, (np.character, np.ndarray)) and np.issubdtype(value.dtype, np.character):
            return value
        elif isinstance(value, np.number):
            return value

        return np.asarray(value, **self.np_array_kwargs)
```
fixes this particular issue for me. Not sure if this would break other tests. This should also avoid unnecessary copying of the array.
GitHub issue: Reload features from Parquet metadata
State: closed

Resolves #5482. 

Attaches feature metadata to parquet files serialised using `Dataset.to_parquet`. 

This allows retrieving data with "rich" feature types (e.g., `datasets.features.image.Image` or `datasets.features.audio.Audio`) from parquet files without cumbersome casting (for an example, see #5482).

@lhoestq It seems that it is sufficient to attach metadata to the schema prior to serialising and features are loaded back with correct types afterwards automatically. 

I used the following script to test the implementation:

```python
from pathlib import Path

import datasets

dataset_name = "Maysee/tiny-imagenet"
ds = datasets.load_dataset(dataset_name, split=datasets.Split.TRAIN)
output_directory_path = Path(__file__).parent.joinpath("example_test_outputs", dataset_name.replace("/", "_"))
output_directory_path.mkdir(exist_ok=True, parents=True)
output_filepath = output_directory_path.joinpath("ds.parquet")
ds.to_parquet(str(output_filepath))

reloaded_ds = datasets.load_dataset(str(output_directory_path), split=datasets.Split.TRAIN)

assert ds.features == reloaded_ds.features
```

Prior to the change in this PR this script raises an `AssertionError` and the `Image` features lose their type after serialisation. After the change in this PR, the assertion does not raise an error and manual inspection of the features shows type `Image` for the respective columns of `reloaded_ds `.

Some open questions:
* How/where can I best add new unit tests for this implementation?
* What dataset would I best use in the tests? I chose `Maysee/tiny-imagenet` mainly because it is small and contains an ?Image` feature that can be used to test, but I'd be happy for suggestions on a suitable data source to use.
* Currently I'm calling `datasets.arrow_writer.ArrowWriter._build_metadata` as I need the same logic. However, I'm not happy with the coupling between `datasets.io.parquet` and `datasets.arrow_writer` it leaves me with. Suggest to factor this common logic out into a helper function and reuse it from both of these. Do you agree and if yes, could you please guide me where I would best place this function?

Many thanks in advance and kind regards,
MFreidank
GitHub issue: Improve inconsistency of `Dataset.map` interface for `load_from_cache_file`
State: closed

### Feature request


1. Replace the `load_from_cache_file` default value to `True`. 
2. Remove or alter checks from `is_caching_enabled` logic.

### Motivation

I stumbled over an inconsistency in the `Dataset.map` interface. The documentation (and source) states for the parameter `load_from_cache_file`:
```
load_from_cache_file (`bool`, defaults to `True` if caching is enabled):
    If a cache file storing the current computation from `function`
    can be identified, use it instead of recomputing.
```

1. `load_from_cache_file` default value is `None`, while being annotated as `bool`
2. It is inconsistent with other method signatures like `filter`, that have the default value `True`
3. The logic is inconsistent, as the `map` method checks if caching is enabled through `is_caching_enabled`. This logic is not used for other similar methods.

### Your contribution

I am not fully aware of the logic behind caching checks. If this is just a inconsistency that historically grew, I would suggest to remove the `is_caching_enabled` logic as the "default" logic. Maybe someone can give insights, if environment variables have a higher priority than local variables or vice versa. 

If this is clarified, I could adjust the source  according to the "Feature request" section of this issue.
GitHub issue: Some functions use a param named `type` shouldn't that be avoided since it's a Python reserved name?
State: closed

Hi @mariosasko, @lhoestq, or whoever reads this! :)

After going through `ArrowDataset.set_format` I found out that the `type` param is actually named `type` which is a Python reserved name as you may already know, shouldn't that be renamed to `format_type` before the 3.0.0 is released?

Just wanted to get your input, and if applicable, tackle this issue myself! Thanks 🤗
GitHub issue: Creating a dummy dataset from a bigger one
State: closed

### Describe the bug

I often want to create a dummy dataset from a bigger dataset for fast iteration when training. However, I'm having a hard time doing this especially when trying to upload the dataset to the Hub.

### Steps to reproduce the bug

```python
from datasets import load_dataset

dataset = load_dataset("lambdalabs/pokemon-blip-captions")
dataset["train"] = dataset["train"].select(range(20))

dataset.push_to_hub("patrickvonplaten/dummy_image_data")
```

gives:

```
~/python_bin/datasets/arrow_dataset.py in _push_parquet_shards_to_hub(self, repo_id, split, private, token, branch, max_shard_size, embed_external_files)                                                                                    
   4003                 base_wait_time=2.0,
   4004                 max_retries=5,
-> 4005                 max_wait_time=20.0,
   4006             )
   4007         return repo_id, split, uploaded_size, dataset_nbytes

~/python_bin/datasets/utils/file_utils.py in _retry(func, func_args, func_kwargs, exceptions, status_codes, max_retries, base_wait_time, max_wait_time)                                                                                      
    328     while True:
    329         try:
--> 330             return func(*func_args, **func_kwargs)
    331         except exceptions as err:
    332             if retry >= max_retries or (status_codes and err.response.status_code not in status_codes):

~/hf/lib/python3.7/site-packages/huggingface_hub/utils/_validators.py in _inner_fn(*args, **kwargs)
    122             )
    123
--> 124         return fn(*args, **kwargs)
    125
    126     return _inner_fn  # type: ignore

TypeError: upload_file() got an unexpected keyword argument 'identical_ok'

In [2]: 
```

### Expected behavior

I would have expected this to work. It's for me the most intuitive way of creating a dummy dataset.

### Environment info

```
- `datasets` version: 2.1.1.dev0
- Platform: Linux-4.19.0-22-cloud-amd64-x86_64-with-debian-10.13
- Python version: 3.7.3
- PyArrow version: 11.0.0
- Pandas version: 1.3.5
```
GitHub issue: Add a static `__all__` to `__init__.py` for typecheckers
State: open

This adds a static `__all__` field to `__init__.py`, allowing typecheckers to know which symbols are accessible from `datasets` at runtime. In particular [Pyright](https://github.com/microsoft/pylance-release/issues/2328#issuecomment-1029381258) seems to rely on this. At this point I have added all (modulo oversight) the symbols mentioned in the Reference part of [the docs](https://huggingface.co/docs/datasets), but that could be adjusted. As a side effect, only these symbols will be imported by `from datasets import *`, which may or may not be a good thing (and if it isn't, that's easy to fix).

Another option would be to add a pyi stub, but I think `__all__` should be the most pythonic solution.

This should fix #3841.
GitHub issue: Saving a dataset after setting format to torch doesn't work, but only if filtering
State: closed

### Describe the bug

Saving a dataset after setting format to torch doesn't work, but only if filtering

### Steps to reproduce the bug

```
a = Dataset.from_dict({"b": [1, 2]})
a.set_format('torch')
a.save_to_disk("test_save") # saves successfully
a.filter(None).save_to_disk("test_save_filter") # does not
>> [...] TypeError: Provided `function` which is applied to all elements of table returns a `dict` of types [<class 'torch.Tensor'>]. When using `batched=True`, make sure provided `function` returns a `dict` of types like `(<class 'list'>, <class 'numpy.ndarray'>)`.

# note: skipping the format change to torch lets this work.



### Expected behavior

Saving to work

### Environment info

- `datasets` version: 2.4.0
- Platform: Linux-6.1.9-arch1-1-x86_64-with-glibc2.36
- Python version: 3.10.9
- PyArrow version: 9.0.0
- Pandas version: 1.4.4
GitHub issue: Optimise behaviour in respect to indices mapping
State: open

_Originally [posted](https://huggingface.slack.com/archives/C02V51Q3800/p1675443873878489?thread_ts=1675418893.373479&cid=C02V51Q3800) on Slack_

Considering all this, perhaps for Datasets 3.0, we can do the following:
* [ ] have `continuous=True` by default in `.shard`  (requested in the survey and makes more sense for us since it doesn't create an indices mapping)
* [x] allow calling `save_to_disk` on "unflattened" datasets
* [ ] remove "hidden" expensive calls in `save_to_disk`, `unique`, `concatenate_datasets`, etc. For instance, instead of silently calling `flatten_indices`  where it's needed, it's probably better to be explicit (considering how expensive these ops can be) and raise an error instead
GitHub issue: IterableDataset and Dataset return different batch sizes when using Trainer with multiple GPUs
State: closed

### Describe the bug

I am training a Roberta model using 2 GPUs and the `Trainer` API with a batch size of 256.

Initially I used a standard `Dataset`, but had issues with slow data loading. After reading [this issue](https://github.com/huggingface/datasets/issues/2252), I swapped to loading my dataset as contiguous shards and passing those to an `IterableDataset`. I observed an unexpected drop in GPU memory utilization, and found the batch size returned from the model had been cut in half.

When using `Trainer` with 2 GPUs and a batch size of 256, `Dataset` returns a batch of size 512 (256 per GPU), while `IterableDataset` returns a batch size of 256 (256 total). My guess is `IterableDataset` isn't accounting for multiple cards.


### Steps to reproduce the bug

```python
import datasets
from datasets import IterableDataset

from transformers import RobertaConfig
from transformers import RobertaTokenizerFast
from transformers import RobertaForMaskedLM

from transformers import DataCollatorForLanguageModeling
from transformers import Trainer, TrainingArguments

use_iterable_dataset = True
def gen_from_shards(shards):
    for shard in shards:
        for example in shard:
            yield example

dataset = datasets.load_from_disk('my_dataset.hf')

if use_iterable_dataset:
    n_shards = 100
    shards = [dataset.shard(num_shards=n_shards, index=i) for i in range(n_shards)]
    dataset = IterableDataset.from_generator(gen_from_shards, gen_kwargs={"shards": shards})

tokenizer = RobertaTokenizerFast.from_pretrained("./my_tokenizer", max_len=160, use_fast=True)

config = RobertaConfig(
    vocab_size=8248,
    max_position_embeddings=256,
    num_attention_heads=8,
    num_hidden_layers=6,
    type_vocab_size=1)

model = RobertaForMaskedLM(config=config)

data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=True, mlm_probability=0.15)

training_args = TrainingArguments(
    per_device_train_batch_size=256
    # other args removed for brevity 
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset,
)

trainer.train()
```

### Expected behavior

Expected `Dataset` and `IterableDataset` to have the same batch size behavior. If the current behavior is intentional, the batch size printout at the start of training should be updated. Currently, both dataset classes result in `Trainer` printing the same total batch size, even though the batch size sent to the GPUs are different.

### Environment info

datasets 2.7.1
transformers 4.25.1
GitHub issue: PyTorch BatchSampler still loads from Dataset one-by-one
State: closed

### Describe the bug

In [the docs here](https://huggingface.co/docs/datasets/use_with_pytorch#use-a-batchsampler), it mentions the issue of the Dataset being read one-by-one, then states that using a BatchSampler resolves the issue.

I'm not sure if this is a mistake in the docs or the code, but it seems that the only way for a Dataset to be passed a list of indexes  by PyTorch (instead of one index at a time) is to define a `__getitems__` method (note the plural) on the Dataset object, and since the HF Dataset doesn't have this, PyTorch executes [this line of code](https://github.com/pytorch/pytorch/blob/master/torch/utils/data/_utils/fetch.py#L58), reverting to fetching one-by-one.

### Steps to reproduce the bug

You can put a breakpoint in `Dataset.__getitem__()` or just print the args from there and see that it's called multiple times for a single `next(iter(dataloader))`, even when using the code from the docs:
```py
from torch.utils.data.sampler import BatchSampler, RandomSampler
batch_sampler = BatchSampler(RandomSampler(ds), batch_size=32, drop_last=False)
dataloader = DataLoader(ds, batch_sampler=batch_sampler)
```



### Expected behavior

The expected behaviour would be for it to fetch batches from the dataset, rather than one-by-one.

To demonstrate that there is room for improvement: once I have a HF dataset `ds`, if I just add this line:
```py
ds.__getitems__ = ds.__getitem__
```
...then the time taken to loop over the dataset improves considerably (for wikitext-103, from one minute to 13 seconds with batch size 32). Probably not a big deal in the grand scheme of things, but seems like an easy win.

### Environment info

- `datasets` version: 2.9.0
- Platform: Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.31
- Python version: 3.10.8
- PyArrow version: 10.0.1
- Pandas version: 1.5.3
GitHub issue: `load_dataset` has ~4 seconds of overhead for cached data
State: open

### Feature request

When loading a dataset that has been cached locally, the `load_dataset` function takes a lot longer than it should take to fetch the dataset from disk (or memory).
This is particularly noticeable for smaller datasets. For example, wikitext-2, comparing `load_data` (once cached) and `load_from_disk`, the `load_dataset` method takes 40 times longer.

⏱    4.84s ⮜ load_dataset
⏱    119ms ⮜ load_from_disk

### Motivation

I assume this is doing something like checking for a newer version.

If so, that's an age old problem: do you make the user wait _every single time they load from cache_ or do you do something like load from cache always, _then_ check for a newer version and alert if they have stale data. The decision usually revolves around what percentage of the time the data will have been updated, and how dangerous old data is.

For most datasets it's extremely unlikely that there will be a newer version on any given run, so 99% of the time this is just wasted time.

Maybe you don't want to make that decision for all users, but at least having the _option_ to not wait for checks would be an improvement.

### Your contribution

.
GitHub issue: TypeError: 'bool' object is not iterable when filtering a datasets.arrow_dataset.Dataset
State: closed

### Describe the bug

Hi, 

Thanks for the amazing work on the library!

**Describe the bug**

I think I might have noticed a small bug in the filter method. 
Having loaded a dataset using `load_dataset`,  when I try to filter out empty entries with `batched=True`, I get a TypeError.

### Steps to reproduce the bug

```
train_dataset = train_dataset.filter(
            function=lambda example: example["image"] is not None, 
            batched=True,
            batch_size=10)
```
Error message:

```
File .../lib/python3.9/site-packages/datasets/fingerprint.py:480, in fingerprint_transform.<locals>._fingerprint.<locals>.wrapper(*args, **kwargs)
    476             validate_fingerprint(kwargs[fingerprint_name])
    478 # Call actual function
--> 480 out = func(self, *args, **kwargs)
...
-> 5666 indices_array = [i for i, to_keep in zip(indices, mask) if to_keep]
   5667 if indices_mapping is not None:
   5668     indices_array = pa.array(indices_array, type=pa.uint64())

TypeError: 'bool' object is not iterable
```

**Removing batched=True allows to bypass the issue.** 

### Expected behavior

According to the doc, "[batch_size corresponds to the] number of examples per batch provided to function if batched = True", so we shouldn't need to remove the batchd=True arg?

source: https://huggingface.co/docs/datasets/v2.9.0/en/package_reference/main_classes#datasets.Dataset.filter

### Environment info

- `datasets` version: 2.9.0
- Platform: Linux-5.4.0-122-generic-x86_64-with-glibc2.31
- Python version: 3.9.10
- PyArrow version: 10.0.1
- Pandas version: 1.5.3
GitHub issue: Add a `reduce` method
State: closed

### Feature request

Right now the `Dataset` class implements `map()` and `filter()`, but leaves out the third functional idiom popular among Python users: `reduce`. 

### Motivation

A `reduce` method is often useful when calculating dataset statistics, for example, the occurrence of a particular n-gram or the average line length of a code dataset.  

### Your contribution

I haven't contributed to `datasets` before, but I don't expect this will be too difficult, since the implementation will closely follow that of `map` and `filter`. I could have a crack over the weekend.
GitHub issue: Update audio installation doc page
State: closed

Our [installation documentation page](https://huggingface.co/docs/datasets/installation#audio) says that one can use Datasets for mp3 only with `torchaudio<0.12`. `torchaudio>0.12` is actually supported too but requires a specific version of ffmpeg which is not easily installed on all linux versions but there is a custom ubuntu repo for it, we have insctructions in the code: https://github.com/huggingface/datasets/blob/main/src/datasets/features/audio.py#L327 
So we should update the doc page. But first investigate [this issue](5488).
GitHub issue: Push_to_hub in a pull request
State: closed

Right now `ds.push_to_hub()` can push a dataset on `main` or on a new branch with `branch=`, but there is no way to open a pull request. Even passing `branch=refs/pr/x` doesn't seem to work: it tries to create a branch with that name

cc @nateraw 

It should be possible to tweak the use of `huggingface_hub` in `push_to_hub` to make it open a PR or push to an existing PR
GitHub issue: Do not add index column by default when exporting to CSV
State: closed

As pointed out by @merveenoyan, default behavior of `Dataset.to_csv` adds the index as an additional column without name.

This PR changes the default behavior, so that now the index column is not written.

To add the index column, now you need to pass `index=True` and also `index_label=<name of the index colum>` to name that column.

CC: @merveenoyan
GitHub issue: Pin dill lower version
State: closed

Pin `dill` lower version compatible with `datasets`.

Related to:
- #5487
- #288

Note that the required `dill._dill` module was introduced in dill-2.8.0, however we have heuristically tested that datasets can only be installed with dill>=3.0.0 (otherwise pip hangs indefinitely while preparing metadata for multiprocess-0.70.7)
GitHub issue: Error loading MP3 files from CommonVoice
State: closed

### Describe the bug

When loading a CommonVoice dataset with `datasets==2.9.0` and `torchaudio>=0.12.0`, I get an error reading the audio arrays: 

```python
---------------------------------------------------------------------------
LibsndfileError                           Traceback (most recent call last)
~/.local/lib/python3.8/site-packages/datasets/features/audio.py in _decode_mp3(self, path_or_file)
    310             try:  # try torchaudio anyway because sometimes it works (depending on the os and os packages installed)
--> 311                 array, sampling_rate = self._decode_mp3_torchaudio(path_or_file)
    312             except RuntimeError:

~/.local/lib/python3.8/site-packages/datasets/features/audio.py in _decode_mp3_torchaudio(self, path_or_file)
    351 
--> 352         array, sampling_rate = torchaudio.load(path_or_file, format="mp3")
    353         if self.sampling_rate and self.sampling_rate != sampling_rate:

~/.local/lib/python3.8/site-packages/torchaudio/backend/soundfile_backend.py in load(filepath, frame_offset, num_frames, normalize, channels_first, format)
    204     """
--> 205     with soundfile.SoundFile(filepath, "r") as file_:
    206         if file_.format != "WAV" or normalize:

~/.local/lib/python3.8/site-packages/soundfile.py in __init__(self, file, mode, samplerate, channels, subtype, endian, format, closefd)
    654                                          format, subtype, endian)
--> 655         self._file = self._open(file, mode_int, closefd)
    656         if set(mode).issuperset('r+') and self.seekable():

~/.local/lib/python3.8/site-packages/soundfile.py in _open(self, file, mode_int, closefd)
   1212             err = _snd.sf_error(file_ptr)
-> 1213             raise LibsndfileError(err, prefix="Error opening {0!r}: ".format(self.name))
   1214         if mode_int == _snd.SFM_WRITE:

LibsndfileError: Error opening <_io.BytesIO object at 0x7fa539462090>: File contains data in an unknown format.

```

I assume this is because there's some issue with the mp3 decoding process. I've verified that I have `ffmpeg>=4` (on a Linux distro), which appears to be the fallback backend for `torchaudio,` (at least according to #4889).

### Steps to reproduce the bug

```python
dataset = load_dataset("mozilla-foundation/common_voice_11_0", "be", split="train")
dataset[0]
```

### Expected behavior

Similar behavior to `torchaudio<0.12.0`, which doesn't result in a `LibsndfileError`

### Environment info

- `datasets` version: 2.9.0
- Platform: Linux-5.15.0-52-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 10.0.1
- Pandas version: 1.5.1
GitHub issue: Incorrect filepath for dill module
State: closed

### Describe the bug

I installed the `datasets` package and when I try to `import` it, I get the following error:

```
Traceback (most recent call last):

  File "/var/folders/jt/zw5g74ln6tqfdzsl8tx378j00000gn/T/ipykernel_3805/3458380017.py", line 1, in <module>
    import datasets

  File "/Users/avivbrokman/opt/anaconda3/lib/python3.9/site-packages/datasets/__init__.py", line 43, in <module>
    from .arrow_dataset import Dataset

  File "/Users/avivbrokman/opt/anaconda3/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 66, in <module>
    from .arrow_writer import ArrowWriter, OptimizedTypedSequence

  File "/Users/avivbrokman/opt/anaconda3/lib/python3.9/site-packages/datasets/arrow_writer.py", line 27, in <module>
    from .features import Features, Image, Value

  File "/Users/avivbrokman/opt/anaconda3/lib/python3.9/site-packages/datasets/features/__init__.py", line 17, in <module>
    from .audio import Audio

  File "/Users/avivbrokman/opt/anaconda3/lib/python3.9/site-packages/datasets/features/audio.py", line 12, in <module>
    from ..download.streaming_download_manager import xopen

  File "/Users/avivbrokman/opt/anaconda3/lib/python3.9/site-packages/datasets/download/__init__.py", line 9, in <module>
    from .download_manager import DownloadManager, DownloadMode

  File "/Users/avivbrokman/opt/anaconda3/lib/python3.9/site-packages/datasets/download/download_manager.py", line 36, in <module>
    from ..utils.py_utils import NestedDataStructure, map_nested, size_str

  File "/Users/avivbrokman/opt/anaconda3/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 602, in <module>
    class Pickler(dill.Pickler):

  File "/Users/avivbrokman/opt/anaconda3/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 605, in Pickler
    dispatch = dill._dill.MetaCatchingDict(dill.Pickler.dispatch.copy())

AttributeError: module 'dill' has no attribute '_dill'

```

Looking at the github source code for dill, it appears that `datasets` has a bug or is not compatible with the latest `dill`.  Specifically, rather than `dill._dill.XXXX` it should be `dill.dill._dill.XXXX`.  But given the popularity of `datasets` I feel confused about me being the first person to have this issue, so it makes me wonder if I'm misdiagnosing the issue.

### Steps to reproduce the bug

Install `dill` and `datasets` packages and then `import datasets`

### Expected behavior

I expect `datasets` to import.

### Environment info

- `datasets` version: 2.9.0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.9.13
- PyArrow version: 11.0.0
- Pandas version: 1.4.4
GitHub issue: Adding `sep` to TextConfig 
State: open

I have a local a `.txt` file that follows the `CONLL2003` format which I need to load using `load_script`. However, by using `sample_by='line'`, one can only split the dataset into lines without splitting each line into columns. Would it be reasonable to add a `sep` argument in combination with `sample_by='paragraph'` to parse a paragraph into an array for each column ? If so, I am happy to contribute!

## Environment
* `python 3.8.10`
* `datasets 2.9.0`

## Snippet of `train.txt`

```txt
Distribution	NN	O	O
and	NN	O	O
dynamics	NN	O	O
of	NN	O	O
electron	NN	O	B-RP
complexes	NN	O	I-RP
in	NN	O	O
cyanobacterial	NN	O	B-R
membranes	NN	O	I-R

The	NN	O	O
occurrence	NN	O	O
of	NN	O	O
prostaglandin	NN	O	B-R
F2α	NN	O	I-R
in	NN	O	O
Pharbitis	NN	O	B-R
seedlings	NN	O	I-R
grown	NN	O	O
under	NN	O	O
short	NN	O	B-P
days	NN	O	I-P
or	NN	O	I-P
days	NN	O	I-P

```

## Current Behaviour 
```python
# defining 4 features ['tokens', 'pos_tags', 'chunk_tags', 'ner_tags'] here would fail with `ValueError: Length of names (4) does not match length of arrays (1)`
dataset = datasets.load_dataset(path='text', features=features, data_files={'train': 'train.txt'}, sample_by='line')
dataset['train']['tokens'][0]
>>> 'Distribution\tNN\tO\tO'
```

## Expected Behaviour / Suggestion
```python
# suppose we defined 4 features ['tokens', 'pos_tags', 'chunk_tags', 'ner_tags']
dataset = datasets.load_dataset(path='text', features=features, data_files={'train': 'train.txt'}, sample_by='paragraph', sep='\t')
dataset['train']['tokens'][0]
>>> ['Distribution', 'and', 'dynamics', ... ]

dataset['train']['ner_tags'][0]
>>> ['O', 'O', 'O', ... ]
```
GitHub issue: Update docs for `nyu_depth_v2` dataset
State: closed

This PR will fix the issue mentioned in #5461. Here is brief overview,

## Bug:

Discrepancy between depth map of `nyu_depth_v2` dataset [here](https://huggingface.co/docs/datasets/main/en/depth_estimation) and actual depth map. Depth values somehow got **discretized/clipped** resulting in depth maps that are different from actual ones. Here is a side-by-side comparison,

![image](https://user-images.githubusercontent.com/36858976/214381162-1d9582c2-6750-4114-a01a-61ca1cd5f872.png)


## Fix:

When I first loaded the datasets from HF I noticed it was 30GB but in DenseDepth data is only 4GB with dtype=uint8. This means data from fast-depth (before loading to HF) must have high precision. So when I tried to dig deeper by directly loading depth_map from `h5py`, I found depth_map from `h5py`  came with `float32`. But when the data is processed in HF with `datasets.Image()` it was directly converted to `uint8` from `float32`  hence the **discretized** depth map.
https://github.com/huggingface/datasets/blob/c78559cacbb0ca6e0bc8bfc313cc0359f8c23ead/src/datasets/features/image.py#L91-L93


cc: @sayakpaul @lhoestq
GitHub issue: Reload features from Parquet metadata
State: closed

The idea would be to allow this :
```python 
ds.to_parquet("my_dataset/ds.parquet")
reloaded = load_dataset("my_dataset")
assert ds.features == reloaded.features 
```
And it should also work with Image and Audio types (right now they're reloaded as a dict type)

This can be implemented by storing and reading the feature types in the parquet metadata, as we do for arrow files.
GitHub issue: Dataset scan time is much slower than using native arrow
State: closed

### Describe the bug

I'm basically running the same scanning experiment from the tutorials https://huggingface.co/course/chapter5/4?fw=pt except now I'm comparing to a native pyarrow version.

I'm finding that the native pyarrow approach is much faster (2 orders of magnitude). Is there something I'm missing that explains this phenomenon?

### Steps to reproduce the bug

https://colab.research.google.com/drive/11EtHDaGAf1DKCpvYnAPJUW-LFfAcDzHY?usp=sharing

### Expected behavior

I expect scan times to be on par with using pyarrow directly.

### Environment info

standard colab environment
GitHub issue: Column project operation on `datasets.Dataset`
State: closed

### Feature request

There is no operation to select a subset of columns of original dataset. Expected API follows.
```python
a = Dataset.from_dict({
    'int': [0, 1, 2]
    'char': ['a', 'b', 'c'],
    'none': [None] * 3,
})

b = a.project('int', 'char')  # usually, .select()

print(a.column_names)  # stdout: ['int', 'char', 'none']
print(b.column_names)  # stdout: ['int', 'char']
```
Method project can easily accept not only column names (as a `str)` but univariant function applied to corresponding column as an example. Or keyword arguments can be used in order to rename columns in advance (see `pandas`, `pyspark`, `pyarrow`, and SQL)..

### Motivation

Projection is a typical operation in every data processing library. And it is a basic block of a well-known data manipulation language like SQL. Without this operation `datasets.Dataset` interface is not complete.

### Your contribution

Not sure. Some of my PRs are still open and some do not have any discussions.
GitHub issue: Add num_test_batches option
State: closed

`to_tf_dataset` calls can be very costly because of the number of test batches drawn during `_get_output_signature`. The test batches are draw in order to estimate the shapes when creating the tensorflow dataset. This is necessary when the shapes can be irregular, but not in cases when the tensor shapes are the same across all samples. This PR adds an option to change the number of batches drawn, so the user can speed this conversion up. 

Running the following, and modifying `num_test_batches`
```
import time
from datasets import load_dataset
from transformers import DefaultDataCollator

data_collator = DefaultDataCollator()
dataset = load_dataset("beans")
dataset = dataset["train"].with_format("np")
start = time.time()
dataset = dataset.to_tf_dataset(
    columns=["image"],
    label_cols=["label"],
    batch_size=8,
    collate_fn=data_collator,
    num_test_batches=NUM_TEST_BATCHES,
)
end = time.time()
print(end - start)
```

NUM_TEST_BATCHES=200:  0.8197s
NUM_TEST_BATCHES=50:    0.3070s
NUM_TEST_BATCHES=2:      0.1417s
NUM_TEST_BATCHES=1:      0.1352s
GitHub issue: Allow opposite of remove_columns on Dataset and DatasetDict
State: closed

### Feature request

In this blog post https://huggingface.co/blog/audio-datasets, I noticed the following code:

```python
COLUMNS_TO_KEEP = ["text", "audio"]
all_columns = gigaspeech["train"].column_names
columns_to_remove = set(all_columns) - set(COLUMNS_TO_KEEP)

gigaspeech = gigaspeech.remove_columns(columns_to_remove)
```

This kind of thing happens a lot when you don't need to keep all columns from the dataset. It would be more convenient (and less error prone) if you could just write:

```python
gigaspeech = gigaspeech.keep_columns(["text", "audio"])
```

Internally, `keep_columns` could still call `remove_columns`, but it expresses more clearly what the user's intent is.

### Motivation

Less code to write for the user of the dataset.

### Your contribution

-
GitHub issue: audiofolder creates empty dataset even though the dataset passed in follows the correct structure
State: closed

### Describe the bug

The structure of my dataset folder called "my_dataset" is : data  metadata.csv
The data folder consists of all mp3 files and metadata.csv consist of file locations like 'data/...mp3 and transcriptions. There's 400+ mp3 files and corresponding transcriptions for my dataset.

When I run the following:
ds = load_dataset("audiofolder", data_dir="my_dataset")


I get:
Using custom data configuration default-...
Downloading and preparing dataset audiofolder/default to /...
Downloading data files:   0%|          | 0/2 [00:00<?, ?it/s]
Downloading data files: 0it [00:00, ?it/s]
Extracting data files: 0it [00:00, ?it/s]
Generating train split: 0 examples [00:00, ? examples/s]
Dataset audiofolder downloaded and prepared to /.... Subsequent calls will reuse this data.
  0%|          | 0/1 [00:00<?, ?it/s]
DatasetDict({
    train: Dataset({
        features: ['audio', 'transcription'],
        num_rows: 1
    })
})





### Steps to reproduce the bug

Create a dataset folder called 'my_dataset' with a subfolder called 'data' that has mp3 files. Also, create metadata.csv that has file locations like 'data/...mp3' and their corresponding transcription.

Run:
ds = load_dataset("audiofolder", data_dir="my_dataset")

### Expected behavior

It should generate a dataset with numerous rows. 

### Environment info

Run on Jupyter notebook
GitHub issue: NonMatchingChecksumError for hendrycks_test
State: closed

### Describe the bug

The checksum of the file has likely changed on the remote host.


### Steps to reproduce the bug

`dataset = nlp.load_dataset("hendrycks_test", "anatomy")`


### Expected behavior

no error thrown


### Environment info

- `datasets` version: 2.2.1
- Platform: macOS-13.1-arm64-arm-64bit
- Python version: 3.9.13
- PyArrow version: 9.0.0
- Pandas version: 1.5.1
GitHub issue: Concatenate on axis=1 with misaligned blocks
State: closed

Allow to concatenate on axis 1 two tables made of misaligned blocks.
For example if the first table has 2 row blocks of 3 rows each, and the second table has 3 row blocks or 2 rows each.
To do that, I slice the row blocks to re-align the blocks.

Fix https://github.com/huggingface/datasets/issues/5413
GitHub issue: Discrepancy in `nyu_depth_v2` dataset
State: open

### Describe the bug

I think there is a discrepancy between depth map of `nyu_depth_v2` dataset [here](https://huggingface.co/docs/datasets/main/en/depth_estimation) and actual depth map. Depth values somehow got **discretized/clipped** resulting in depth maps that are different from actual ones. Here is a side-by-side comparison,

![image](https://user-images.githubusercontent.com/36858976/214381162-1d9582c2-6750-4114-a01a-61ca1cd5f872.png)

I tried to find the origin of this issue but sadly as I mentioned in tensorflow/datasets/issues/4674, the download link from `fast-depth` doesn't work anymore hence couldn't verify if the error originated there or during porting data from there to HF.

Hi @sayakpaul, as you worked on huggingface/datasets/issues/5255, if you still have access to that data could you please share the data or perhaps checkout this issue?


### Steps to reproduce the bug

This [notebook](https://colab.research.google.com/drive/1K3ZU8XUPRDOYD38MQS9nreQXJYitlKSW?usp=sharing#scrollTo=UEW7QSh0jf0i) from @sayakpaul  could be used to generate depth maps and actual ground truths could be checked from this  [dataset](https://www.kaggle.com/datasets/awsaf49/nyuv2-bts-dataset) from BTS repo.

> Note: BTS dataset has only 36K data compared to the train-test 50K. They sampled the data as adjacent frames look quite the same

### Expected behavior

Expected depth maps should be smooth rather than discrete/clipped.

### Environment info

- `datasets` version: 2.8.1.dev0
- Platform: Linux-5.10.147+-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 9.0.0
- Pandas version: 1.3.5
GitHub issue: Disable aiohttp requoting of redirection URL
State: closed

The library `aiohttp` performs a requoting of redirection URLs that unquotes the single quotation mark character: `%27` => `'`

This is a problem for our Hugging Face Hub, which requires exact URL from location header.

Specifically, in the query component of the URL (`https://netloc/path?query`), the value for `response-content-disposition` contains `%27`:
```
response-content-disposition=attachment%3B+filename*%3DUTF-8%27%27sample.jsonl.gz%3B+filename%3D%22sample.jsonl.gz%22%3B
```
and after the requoting, the `%27` characters get unquoted to `'`:
```
response-content-disposition=attachment%3B+filename*%3DUTF-8''sample.jsonl.gz%3B+filename%3D%22sample.jsonl.gz%22%3B
```

This PR disables the `aiohttp` requoting of redirection URLs.
GitHub issue: slice split while streaming
State: closed

### Describe the bug

When using the `load_dataset` function with streaming set to True, slicing splits is apparently not supported.

Did I miss this in the documentation?

### Steps to reproduce the bug

`load_dataset("lhoestq/demo1",revision=None, streaming=True, split="train[:3]")`

causes ValueError: Bad split: train[:3]. Available splits: ['train', 'test'] in builder.py, line 1213, in as_streaming_dataset

### Expected behavior

The first 3 entries of the dataset as a stream

### Environment info

- `datasets` version: 2.8.0
- Platform: Windows-10-10.0.19045-SP0
- Python version: 3.10.9
- PyArrow version: 10.0.1
- Pandas version: 1.5.2
GitHub issue: Single TQDM bar in multi-proc map
State: closed

Use the "shard generator approach with periodic progress updates" (used in `save_to_disk` and multi-proc `load_dataset`) in `Dataset.map` to enable having a single TQDM progress bar in the multi-proc mode.

Closes https://github.com/huggingface/datasets/issues/771, closes https://github.com/huggingface/datasets/issues/3177

TODO:
- [x] cleaner refactor of the `_map_single` decorators now that they also have to wrap generator functions (decorate `map` instead of `map_single` with the `transmit_` decorators and predict the shards' fingerprint in `map`)
GitHub issue: Save and resume the state of a DataLoader
State: open

It would be nice when using `datasets` with a PyTorch DataLoader to be able to resume a training from a DataLoader state (e.g. to resume a training that crashed)

What I have in mind (but lmk if you have other ideas or comments):

For map-style datasets, this requires to have a PyTorch Sampler state that can be saved and reloaded per node and worker.

For iterable datasets, this requires to save the state of the dataset iterator, which includes:
- the current shard idx and row position in the current shard
- the epoch number
- the rng state
- the shuffle buffer

Right now you can already resume the data loading of an iterable dataset by using `IterableDataset.skip` but it takes a lot of time because it re-iterates on all the past data until it reaches the resuming point.

cc @stas00 @sgugger
GitHub issue: Fix base directory while extracting insecure TAR files
State: closed

This PR fixes the extraction of insecure TAR files by changing the base path against which TAR members are compared:
- from: "."
- to: `output_path`

This PR also adds tests for extracting insecure TAR files.

Related to:
- #5441
- #5452

@stas00 please note this PR addresses just one of the issues you pointed out: the use of the cwd by the extractor. The other issues (actionable error messages, raise instead of log error) should be addressed in other PRs.
GitHub issue: ImageFolder BadZipFile: Bad offset for central directory
State: closed

### Describe the bug

I'm getting the following exception:
```
lib/python3.10/zipfile.py:1353 in _RealGetContents        │
│                                                                                                  │
│   1350 │   │   # self.start_dir:  Position of start of central directory                         │
│   1351 │   │   self.start_dir = offset_cd + concat                                               │
│   1352 │   │   if self.start_dir < 0:                                                            │
│ ❱ 1353 │   │   │   raise BadZipFile("Bad offset for central directory")                          │
│   1354 │   │   fp.seek(self.start_dir, 0)                                                        │
│   1355 │   │   data = fp.read(size_cd)                                                           │
│   1356 │   │   fp = io.BytesIO(data)                                                             │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
BadZipFile: Bad offset for central directory                                                                            
Extracting data files:  35%|█████████████████▊                                 | 38572/110812 [00:10<00:20, 3576.26it/s]
```



### Steps to reproduce the bug

```
load_dataset(
    args.dataset_name,
    args.dataset_config_name,
    cache_dir=args.cache_dir,
),
```

### Expected behavior

loads the dataset

### Environment info

datasets==2.8.0
Python 3.10.8
Linux 129-146-3-202 5.15.0-52-generic #58~20.04.1-Ubuntu SMP Thu Oct 13 13:09:46 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
GitHub issue: to_tf_dataset with a TF collator causes bizarrely persistent slowdown
State: closed

### Describe the bug

This will make more sense if you take a look at [a Colab notebook that reproduces this issue.](https://colab.research.google.com/drive/1rxyeciQFWJTI0WrZ5aojp4Ls1ut18fNH?usp=sharing) 

Briefly, there are several datasets that, when you iterate over them with `to_tf_dataset` **and** a data collator that returns `tf` tensors, become very slow. We haven't been able to figure this one out - it can be intermittent, and we have no idea what could possibly cause it. The weirdest thing is that **the slowdown affects other attempts to access the underlying dataset**. If you try to iterate over the `tf.data.Dataset`, then interrupt execution, and then try to iterate over the original dataset, the original dataset is now also very slow! This is true even if the dataset format is not set to `tf` - the iteration is slow even though it's not calling TF at all!

There is a simple workaround for this - we can simply get our data collators to return `np` tensors. When we do this, the bug is never triggered and everything is fine. In general, `np` is preferred for this kind of preprocessing work anyway, when the preprocessing is not going to be compiled into a pure `tf.data` pipeline! However, the issue is fascinating, and the TF team were wondering if anyone in datasets (cc @lhoestq @mariosasko) might have an idea of what could cause this.

### Steps to reproduce the bug

Run the attached Colab.

### Expected behavior

The slowdown should go away, or at least not persist after we stop iterating over the `tf.data.Dataset`

### Environment info

The issue occurs on multiple versions of Python and TF, both on local machines and on Colab.

All testing was done using the latest versions of `transformers` and `datasets` from `main`