sjwsjw/github-issues huggingface dataset:
GitHub issue: What is the Right way to use discofuse dataset??
State: closed

[Click here for Dataset link](https://huggingface.co/datasets/discofuse/viewer/discofuse-wikipedia/train?row=6)
**Below is the following way, as per my understanding , Is it correct :question: :question:**

The **columns/features from `DiscoFuse dataset`** that will be the **input to the `encoder` and `decoder`** are:

[Click here for Dataset link](https://huggingface.co/datasets/discofuse/viewer/discofuse-wikipedia/train?row=6)

1. **coherent_first_sentence**

2. **coherent_second_sentence**

3. **incoherent_first_sentence**

4. **incoherent_second_sentence**

[Click here for Dataset link](https://huggingface.co/datasets/discofuse/viewer/discofuse-wikipedia/train?row=6)

The **`encoder` will take these four columns as input and encode them into a sequence of hidden states. The `decoder` will then take these hidden states as input and decode them into a new sentence that fuses the two original sentences together.**

The **discourse type, connective_string, has_coref_type_pronoun, and has_coref_type_nominal columns will not be used as input to the encoder or decoder.** These columns are used to provide additional information about the dataset, but they are not necessary for the task of sentence fusion.

Please correct me if I am wrong; otherwise, if this understanding is right, how shall I implement this task practically?
GitHub issue: Support for data with instance-wise dictionary as features
State: open

### Feature request

I notice that when loading data instances with feature type of python dictionary, the dictionary keys would be broadcast so that every instance has the same set of keys. Please see an example in the Motivation section.

It is possible to avoid this behavior, i.e., load dictionary features as it is and do not broadcast the keys among instances? Please note that these dictionaries would have to be processed dynamically at each training iteration into strings (and tokenized).

### Motivation

I am trying to load a dataset from a json file. Each instance of the dataset has a feature that is a dictionary but its keys depend on the instance. Every two instances may have different keys. For example, imagine a dataset that contains a set of math expressions from a bunch of mutually redundant expressions:
```
{
    "index": 0,
    "feature": {
        "2 * x + y >= 3": ["2 * x + y >= 3", "4 * x + 2 * y >= 6"],
        ...
    }
},
...
{
    "index": 9999,
    "feature": {
        "x >= 6": ["x >= 6", "x >= 0", "x >= -1"],
        ...
    }
},
...
```
When directly loading the dataset using `data = load_dataset("json", data_files=file_paths, split='train')`, each instance would have all the keys from other instances and None as values. That is, instance of index 0 becomes:
```
{
    "index": 0,
    "feature": {
        "2 * x + y >= 3": ["2 * x + y >= 3", "4 * x + 2 * y >= 6"],
        ...
        "x >= 6": None,  # keys from other instances
        ...
    }
},
```
This is not desirable. Moreover, issue would be raised if I attempt to combine two such datasets using `data = concatenate_datasets(multi_datasets)`, perhaps because their dictionary features contain different keys.

A solution I can think of is to store the dictionary features as a long string, and evaluate it later. Please kindly suggest any other solution using existing methods of datasets.

### Your contribution

N/A
GitHub issue: Replace metadata utils with `huggingface_hub`'s RepoCard API
State: closed

Use `huggingface_hub`'s RepoCard API instead of `DatasetMetadata` for modifying the card's YAML, and deprecate `datasets.utils.metadata` and `datasets.utils.readme`.

After removing these modules, we can also delete `datasets.utils.resources` since the moon landing repo now stores its own version of these resources for the metadata UI.

PS: this change requires bumping `huggingface_hub` to 0.13.0 (Transformers requires 0.14.0, so should be ok)
GitHub issue: Return the audio filename when decoding fails due to corrupt files
State: open

### Feature request

Return the audio filename when the audio decoding fails. Although currently there are some checks for mp3 and opus formats with the library version there are still cases when the audio decoding could fail, eg. Corrupt file. 

### Motivation

When you try to load an object file dataset and the decoding fails you can't know which file is corrupt
```

raise LibsndfileError(err, prefix="Error opening {0!r}: ".format(self.name))
soundfile.LibsndfileError: Error opening <_io.BytesIO object at 0x7f5ab7e38290>: Format not recognised.
```

### Your contribution

Make a PR to Add exceptions for LIbsndfileError to return the audio filename or path when soundfile decoding fails.
GitHub issue: Failing to upload dataset to the hub
State: closed

### Describe the bug

Trying to upload a dataset of hundreds of thousands of audio samples (the total volume is not very large, 60 gb) to the hub with push_to_hub, it doesn't work.
From time to time one piece of the data (parquet) gets pushed and then I get RemoteDisconnected even though my internet is stable.
Please help.
I'm trying to upload the dataset for almost a week.
Thanks

### Steps to reproduce the bug

not relevant 

### Expected behavior

Be able to upload thedataset

### Environment info

python: 3.9
GitHub issue: Pass datasets-cli additional args as kwargs to DatasetBuilder in `run_beam.py`
State: open

Hi,

Following this <https://discuss.huggingface.co/t/how-to-preprocess-a-wikipedia-dataset-using-dataflowrunner/41991/3>, here is a simple PR to pass any additional args to datasets-cli as kwargs in the DatasetBuilder in `run_beam.py`.

I also took the liberty to add missing setup steps to the `beam.mdx` docs in order to help everyone.

@lhoestq
GitHub issue: Load Data Sets Too Slow In Train Seq2seq Model
State: closed

### Describe the bug


step 'Generating train split' in load_dataset is too slow：
![image](https://github.com/huggingface/datasets/assets/19569322/d9b08eee-95fe-4741-a346-b70416c948f8)


### Steps to reproduce the bug

Data： own data，16K16B Mono wav
Oficial Script:[ run_speech_recognition_seq2seq.py](https://github.com/huggingface/transformers/blob/main/examples/pytorch/speech-recognition/run_speech_recognition_seq2seq.py)
Add Code：
    if data_args.data_path is not None:
        print(data_args.data_path)
        raw_datasets = load_dataset("audiofolder", data_dir=data_args.data_path, cache_dir=model_args.cache_dir)
        raw_datasets = raw_datasets.cast_column("audio", Audio(sampling_rate=16000))
        raw_datasets = raw_datasets["train"].train_test_split(test_size=0.005, shuffle=True)
 （change cache_dir to other path ，ex:/DATA/cache）

### Expected behavior

load data fast,at least 1000+
`Generating train split: 387875 examples [32:24:45, 1154.83 examples/s]`

### Environment info


- `transformers` version: 4.28.0.dev0
- Platform: Linux-5.4.0-149-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.7.16
- Huggingface_hub version: 0.13.2
- PyTorch version (GPU?): 1.13.1+cu116 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
GitHub issue: Make get_from_cache use custom temp filename that is locked
State: closed

This PR ensures that the temporary filename created is the same as the one that is locked, while writing to the cache.

This PR stops using `tempfile` to generate the temporary filename.

Additionally, the behavior now is aligned for both `resume_download` `True` and `False`.

Refactor temp_file_manager so that it uses the filename that is locked: 
- Use: `cache_path + ".incomplete"`, when the locked one is `cache_path + ".lock"`

Before it was using `tempfile` inside `cache_dir`, which was not locked: although very improbable name collision (8 random characters), this was not impossible when huge number of multiple processes.

Maybe related to "Stale file handle" issues caused by `tempfile`: 
- [ ] https://huggingface.co/datasets/tapaco/discussions/4
- [ ] https://huggingface.co/datasets/xcsr/discussions/1
- [ ] https://huggingface.co/datasets/covost2/discussions/3
```
Error code:   ConfigNamesError
Exception:    OSError
Message:      [Errno 116] Stale file handle
Traceback:    Traceback (most recent call last):
                File "/src/services/worker/src/worker/job_runners/dataset/config_names.py", line 61, in compute_config_names_response
                  for config in sorted(get_dataset_config_names(path=dataset, use_auth_token=use_auth_token))
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 323, in get_dataset_config_names
                  dataset_module = dataset_module_factory(
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 1219, in dataset_module_factory
                  raise e1 from None
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 1188, in dataset_module_factory
                  return HubDatasetModuleFactoryWithScript(
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 907, in get_module
                  dataset_readme_path = self.download_dataset_readme_file()
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 896, in download_dataset_readme_file
                  return cached_path(
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/utils/file_utils.py", line 183, in cached_path
                  output_path = get_from_cache(
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/utils/file_utils.py", line 611, in get_from_cache
                  http_get(
                File "/usr/local/lib/python3.9/tempfile.py", line 496, in __exit__
                  result = self.file.__exit__(exc, value, tb)
              OSError: [Errno 116] Stale file handle
```
- the stale file handle error can be raised when `tempfile` tries to close (when exiting its context manager) a filename that has been already closed by other process
  - note that `tempfile` filenames are randomly generated but not locked in our code

CC: @severo
GitHub issue: `datasets.map` not reusing cached copy by default
State: closed

### Describe the bug

When I load the dataset from local directory, it's cached copy is picked up after first time. However, for `map` operation, the operation is applied again and cached copy is not picked up. Is there any way to pick cached copy instead of processing it again? The only solution I could think of was to use `save_to_disk` after my last transform and then use that in my DataLoader pipeline. Are there any other solutions for the same?

One more thing, my dataset is occupying 6GB storage memory after I use `map`, is there any way I can reduce that memory usage?


### Steps to reproduce the bug

```
# make sure that dataset decodes audio with correct sampling rate
dataset_sampling_rate = next(iter(self.raw_datasets.values())).features["audio"].sampling_rate
if dataset_sampling_rate != self.feature_extractor.sampling_rate:
    self.raw_datasets = self.raw_datasets.cast_column(
        "audio", datasets.features.Audio(sampling_rate=self.feature_extractor.sampling_rate)
    )

vectorized_datasets = self.raw_datasets.map(
    self.prepare_dataset,
    remove_columns=next(iter(self.raw_datasets.values())).column_names,
    num_proc=self.num_workers,
    desc="preprocess datasets",
)
# filter data that is longer than max_input_length
self.vectorized_datasets = vectorized_datasets.filter(
    self.is_audio_in_length_range,
    num_proc=self.num_workers,
    input_columns=["input_length"],
        )

def prepare_dataset(self, batch):
    # load audio
    sample = batch["audio"]
    inputs = self.feature_extractor(sample["array"], sampling_rate=sample["sampling_rate"])
    batch["input_values"] = inputs.input_values[0]
    batch["input_length"] = len(batch["input_values"])

    batch["labels"] = self.tokenizer(batch["target_text"]).input_ids
    return batch

```

### Expected behavior

`map` to use cached copy and if possible an alternative technique to reduce memory usage after using `map`

### Environment info


- `datasets` version: 2.12.0
- Platform: Linux-3.10.0-1160.71.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.16
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.2
GitHub issue: Importing PyTorch reduces multiprocessing performance for map
State: closed

### Describe the bug

I noticed that the performance of my dataset preprocessing with `map(...,num_proc=32)` decreases when PyTorch is imported.

### Steps to reproduce the bug

I created two example scripts to reproduce this behavior:

```
import datasets
datasets.disable_caching()

from datasets import Dataset
import time
    
PROC=32

if __name__ == "__main__":
    dataset = [True] * 10000000
    dataset = Dataset.from_dict({'train': dataset})
    

    start = time.time()
    dataset.map(lambda x: x, num_proc=PROC)
    end = time.time()
    print(end - start)
```
Takes around 4 seconds on my machine.

While the same code, but with an `import torch`:
```
import datasets
datasets.disable_caching()

from datasets import Dataset
import time
import torch
    
PROC=32

if __name__ == "__main__":
    dataset = [True] * 10000000
    dataset = Dataset.from_dict({'train': dataset})
    

    start = time.time()
    dataset.map(lambda x: x, num_proc=PROC)
    end = time.time()
    print(end - start)
```
takes around 22 seconds.



### Expected behavior

I would expect that the import of torch to not have such a significant effect on the performance of map using multiprocessing.


### Environment info

- `datasets` version: 2.12.0
- Platform: Linux-5.15.0-56-generic-x86_64-with-glibc2.35
- Python version: 3.11.3
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.2
- torch: 2.0.1
GitHub issue: Breaking API change in datasets.list_datasets caused by change in HfApi.list_datasets
State: closed

### Describe the bug

Hi all,

after an update of the `datasets` library, we observer crashes in our code. We relied on `datasets.list_datasets` returning a `list`. Now, after the API of the HfApi.list_datasets was changed and it returns a `list` instead of an `Iterable`, the `datasets.list_datasets` now sometimes returns a `list` and somesimes an `Iterable`. 

It would be helpful to indicate that by the return type of the `datasets.list_datasets` function.

Thanks,
Martin 

### Steps to reproduce the bug

Here, the code crashed after we updated the `datasets` library:

```python
# list_datasets no longer returns a list, which leads to an error when one tries to slice it
for datasets.list_datasets(with_details=True)[:limit]:
    ...
```

### Expected behavior

It would be helpful to indicate that by the return type of the `datasets.list_datasets` function.

### Environment info

Ubuntu 22.04
datasets 2.12.0
GitHub issue: Cannot import datasets - ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility
State: closed

### Describe the bug

When trying to import datasets, I get a pyarrow ValueError:

Traceback (most recent call last):
  File "/Users/edward/test/test.py", line 1, in <module>
    import datasets
  File "/Users/edward/opt/anaconda3/envs/cs235/lib/python3.9/site-packages/datasets/__init__.py", line 43, in <module>
    from .arrow_dataset import Dataset
  File "/Users/edward/opt/anaconda3/envs/cs235/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 65, in <module>
    from .arrow_reader import ArrowReader
  File "/Users/edward/opt/anaconda3/envs/cs235/lib/python3.9/site-packages/datasets/arrow_reader.py", line 28, in <module>
    import pyarrow.parquet as pq
  File "/Users/edward/opt/anaconda3/envs/cs235/lib/python3.9/site-packages/pyarrow/parquet/__init__.py", line 20, in <module>
    from .core import *
  File "/Users/edward/opt/anaconda3/envs/cs235/lib/python3.9/site-packages/pyarrow/parquet/core.py", line 45, in <module>
    from pyarrow.fs import (LocalFileSystem, FileSystem, FileType,
  File "/Users/edward/opt/anaconda3/envs/cs235/lib/python3.9/site-packages/pyarrow/fs.py", line 49, in <module>
    from pyarrow._gcsfs import GcsFileSystem  # noqa
  File "pyarrow/_gcsfs.pyx", line 1, in init pyarrow._gcsfs
ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility. Expected 88 from C header, got 72 from PyObject

### Steps to reproduce the bug

`import datasets`

### Expected behavior

Successful import

### Environment info

Conda environment, MacOS
python 3.9.12
datasets 2.12.0
GitHub issue: Length of table does not accurately reflect the split
State: closed

### Describe the bug

I load a Huggingface Dataset and do `train_test_split`. I'm expecting the underlying table for the dataset to also be split, but it's not.

### Steps to reproduce the bug

![image](https://github.com/huggingface/datasets/assets/8068268/83e5768f-8b4c-422a-945c-832a7585afff)


### Expected behavior

The expected behavior is when `len(hf_dataset["train"].data)` should match the length of the train split, and not be the entire unsplit dataset.

### Environment info

datasets 2.10.1
python 3.10.11
GitHub issue: add support for storage_options for load_dataset API
State: closed

to solve the issue in #5880 

1.  add s3 support in the link check step, previous we only check `http` and `https`,

2. change the parameter of `use_auth_token` to `download_config` to support both `storage_options` and `use_auth_token` parameter when trying to handle(list, open, read, etc,.) the remote files.

3.  integrate the check part's duplicate code to make adding or deleting other sources easier.
GitHub issue: File not found for audio dataset
State: open

### Describe the bug

After loading an audio dataset, and looking at a sample entry, the `path` element, which is supposed to be the path to the audio file, doesn't actually exist.


### Steps to reproduce the bug

Run bug.py:

```py
import os.path

from datasets import load_dataset

def run() -> None:
    cv13 = load_dataset(
        "mozilla-foundation/common_voice_13_0",
        "hi",
        split="train",
    )

    print(cv13[0])
    audio_file = cv13[0]["path"]
    if not os.path.exists(audio_file):
        raise ValueError(f'File {audio_file} does not exist.')

if __name__ == "__main__":
    run()
```

The result (on my machine):

```json
{'client_id': '0f018a99663f33afbb7d38aee281fb1afcfd07f9e7acd00383f604e1e17c38d6ed8adf1bd2ccbf927a52c5adefb8ac4b158ce27a7c2ed9581e71202eb302dfb3', 'path': 'C:\\Users\\rober\\.cache\\huggingface\\datasets\\downloads\\extracted\\8d1479bc09b4609bc2675bd02d6869a4d5e09f7e6616f540bd55eacef46c6e2b\\common_voice_hi_26008353.mp3', 'audio': {'path': 'C:\\Users\\rober\\.cache\\huggingface\\datasets\\downloads\\extracted\\8d1479bc09b4609bc2675bd02d6869a4d5e09f7e6616f540bd55eacef46c6e2b\\common_voice_hi_26008353.mp3', 'array': array([ 6.46234854e-26, -1.35709319e-25, -8.07793567e-26, ...,
        1.06425944e-07,  4.46417090e-08,  2.61451660e-09]), 'sampling_rate': 48000}, 'sentence': 'हमने उसका जन्मदिन मनाया।', 'up_votes': 2, 'down_votes': 0, 'age': '', 'gender': '', 'accent': '', 'locale': 'hi', 'segment': '' ', 'variant': ''}
```

```txt
Traceback (most recent call last):
  File "F:\eo-reco\bug.py", line 18, in <module>
    run()
  File "F:\eo-reco\bug.py", line 15, in run
    raise ValueError(f'File {audio_file} does not exist.')
ValueError: File C:\Users\rober\.cache\huggingface\datasets\downloads\extracted\8d1479bc09b4609bc2675bd02d6869a4d5e09f7e6616f540bd55eacef46c6e2b\common_voice_hi_26008353.mp3 does not exist.
```


### Expected behavior

The `path` element points to the correct file, which happens to be:

```
C:\Users\rober\.cache\huggingface\datasets\downloads\extracted\8d1479bc09b4609bc2675bd02d6869a4d5e09f7e6616f540bd55eacef46c6e2b\hi_train_0\common_voice_hi_26008353.mp3
```

That is, there's an extra directory `hi_train_0` that is not in the `path` element.


### Environment info

- `datasets` version: 2.12.0
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.11.3
- Huggingface_hub version: 0.14.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.1
-
GitHub issue: I tried to load a custom dataset using the following statement: dataset = load_dataset('json', data_files=data_files). The dataset contains 50 million text-image pairs, but an error occurred.
State: closed

### Describe the bug

File "/home/kas/.conda/envs/diffusers/lib/python3.7/site-packages/datasets/builder.py", line 1858, in _prepare_split_single
Downloading and preparing dataset json/default to /home/kas/diffusers/examples/dreambooth/cache_data/datasets/json/default-acf423d8c6ef99d0/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4...
Downloading data files: 0%| | 0/1 [00:00<?, ?it/s] Downloading data files: 100%|██████████| 1/1 [00:00<00:00, 84.35it/s]
Extracting data files: 0%| | 0/1 [00:00<?, ?it/s] for _, table in generator:
File "/home/kas/.conda/envs/diffusers/lib/python3.7/site-packages/datasets/packaged_modules/json/json.py", line 114, in _generate_tables
io.BytesIO(batch), read_options=paj.ReadOptions(block_size=block_size)
File "pyarrow/_json.pyx", line 258, in pyarrow._json.read_json
Extracting data files: 100%|██████████| 1/1 [00:00<00:00, 27.72it/s]
Generating train split: 0 examples [00:00, ? examples/s] File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 125, in pyarrow.lib.check_status
pyarrow.lib.ArrowCapacityError: array cannot contain more than 2147483646 bytes, have 2390448764

### Steps to reproduce the bug

1、data_files = ["1.json", "2.json", "3.json"]
2、dataset = load_dataset('json', data_files=data_files)

### Expected behavior

Read the dataset normally.



### Environment info

- `datasets` version: 2.12.0
- Platform: Linux-4.15.0-29-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.16
- Huggingface_hub version: 0.14.1
- PyArrow version: 12.0.0
- Pandas version: 1.3.5
GitHub issue: Missing elements in `map` a batched dataset
State: closed

### Describe the bug

As outlined [here](https://discuss.huggingface.co/t/length-error-using-map-with-datasets/40969/3?u=sachin), the following collate function drops 5 out of possible 6 elements in the batch (it is 6 because out of the eight, two are bad links in laion). A reproducible [kaggle kernel ](https://www.kaggle.com/sachin/laion-hf-dataset/edit) can be found here.

The weirdest part is when inspecting the sizes of the tensors as shown below, both `tokenized_captions["input_ids"]` and `image_features` show the correct shapes. Simply the output only has one element (with the batch dimension squeezed out).

```python
class CollateFn:        
    def get_image(self, url):
        try:
            response = requests.get(url)
            return Image.open(io.BytesIO(response.content)).convert("RGB")
        except PIL.UnidentifiedImageError:
            logger.info(f"Reading error: Could not transform f{url}")
            return None
        except requests.exceptions.ConnectionError:
            logger.info(f"Connection error: Could not transform f{url}")
            return None
        
        
    def __call__(self, batch):
        images = [self.get_image(url) for url in batch["url"]]
        captions = [caption for caption, image in zip(batch["caption"], images) if image is not None]
        images = [image for image in images if image is not None]
        
        tokenized_captions = tokenizer(
            captions,
            padding="max_length",
            truncation=True,
            max_length=tokenizer.model_max_length,
            return_tensors="pt",
        )
        
        image_features = torch.stack([torch.Tensor(feature_extractor(image)["pixel_values"][0]) for image in images])
#         import pdb; pdb.set_trace()
        return {"input_ids": tokenized_captions["input_ids"], "images": image_features}

collate_fn = CollateFn()
laion_ds = datasets.load_dataset("laion/laion400m", split="train", streaming=True)
laion_ds_batched = laion_ds.map(collate_fn, batched=True, batch_size=8, remove_columns=next(iter(laion_ds)).keys())
```

### Steps to reproduce the bug

A reproducible [kaggle kernel ](https://www.kaggle.com/sachin/laion-hf-dataset/edit) can be found here.

### Expected behavior

Would expect `next(iter(laion_ds_batched))` to produce two tensors of shape `(batch_size, 77)` and `batch_size, image_shape`.

### Environment info

datasets==2.12.0
python==3.10
GitHub issue: Cannot use both set_format and set_transform
State: closed

### Describe the bug

I need to process some data using the set_transform method but I also need the data to be formatted for pytorch before processing it.

I don't see anywhere in the documentation something that says that both methods cannot be used at the same time.

### Steps to reproduce the bug

```
from datasets import load_dataset
ds = load_dataset("mnist", split="train")
ds.set_format(type="torch")
def transform(entry):
    return entry["image"].double()
ds.set_transform(transform)

print(ds[0])
```

### Expected behavior

It should print the pytorch tensor image as a double, but it errors because "entry" in the transform function doesn't receive a pytorch tensor to begin with, it receives a PIL Image -> entry.double() errors because entry isn't a pytorch tensor.

### Environment info
Latest versions.


### Note:
It would be at least handy to have access to a function that can do the dataset.set_format in the set_transform function.

Something like:
```
from datasets import load_dataset, do_format
ds = load_dataset("mnist", split="train")
def transform(entry):
    entry = do_format(entry, type="torch")
    return entry["image"].double()
ds.set_transform(transform)

print(ds[0])
```
GitHub issue: Use more efficient and idiomatic way to construct list.
State: closed

Using `*` is ~2X faster according to [benchmark](https://colab.research.google.com/gist/ttsugriy/c964a2604edf70c41911b10335729b6a/for-vs-mult.ipynb) with just 4 patterns. This doesn't matter much since this tiny difference is not going to be noticeable, but why not?
GitHub issue: Unbearably slow sorting on big mapped datasets
State: open

### Describe the bug

For me, with ~40k lines, sorting took 3.5 seconds on a flattened dataset (including the flatten operation) and 22.7 seconds on a mapped dataset (right after sharding), which is about x5 slowdown. Moreover, it seems like it slows down exponentially with bigger datasets (wasn't able to sort 700k lines at all, with flattening takes about a minute).

### Steps to reproduce the bug

```Python
from datasets import load_dataset
import time

dataset = load_dataset("xnli", "en", split="train")

dataset = dataset.shard(10, 0)

print(len(dataset))

t = time.time()

# dataset = dataset.flatten_indices()  # uncomment this line and it's fast

dataset = dataset.sort("label", reverse=True, load_from_cache_file=False)

print(f"finished in {time.time() - t:.4f} seconds")

```

### Expected behavior

Expect sorting to take the same or less time than flattening and then sorting.

### Environment info

- `datasets` version: 2.12.1.dev0 (same with 2.12.0 too)
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.10.10
- Huggingface_hub version: 0.14.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.1
GitHub issue: Could you unpin responses version?
State: closed

### Describe the bug

Could you unpin [this](https://github.com/huggingface/datasets/blob/main/setup.py#L139) or move it to test requirements? This is a testing library and we also use it for our tests as well. We do not want to use a very outdated version.

### Steps to reproduce the bug

could not install this library due to dependency conflict.

### Expected behavior

can install datasets

### Environment info

linux 64
GitHub issue: Offer an alternative to Iterable Dataset that allows lazy loading and processing while skipping batches efficiently
State: open

### Feature request

I would like a way to resume training from a checkpoint without waiting for a very long time when using an iterable dataset.

### Motivation

I am training models on the speech-recognition task. I have very large datasets that I can't comfortably store on a disk and also quite computationally intensive audio processing to do. As a result I want to load data from my remote when it is needed and perform all processing on the fly.

I am currently using the iterable dataset feature of _datasets_. It does everything I need with one exception. My issue is that when resuming training at a step n, we have to download all the data and perform the processing of steps < n, just to get the iterable at the right step. In my case it takes almost as long as training for the same steps, which make resuming training from a checkpoint useless in practice.

I understand that the nature of iterators make it probably nearly impossible to quickly resume training.

I thought about a possible solution nonetheless : 

I could in fact index my large dataset and make it a mapped dataset. Then I could use set_transform to perform the processing on the fly. Finally, if I'm not mistaken, the _accelerate_ package allows to [skip steps efficiently](https://github.com/huggingface/accelerate/blob/a73898027a211c3f6dc4460351b0ec246aa824aa/src/accelerate/data_loader.py#L827) for a mapped dataset.

Is it possible to lazily load samples of a mapped dataset ? I'm used to [dataset scripts](https://huggingface.co/docs/datasets/dataset_script), maybe something can be done there.
If not, I could do it using a plain _Pytorch_ dataset. Then I would need to convert it to a _datasets_' dataset to get all the features of _datasets_. Is it something possible ?

### Your contribution

I could provide a PR to allow lazy loading of mapped dataset or the conversion of a mapped _Pytorch_ dataset into a _Datasets_ dataset if you think it is an useful new feature.
GitHub issue: Validate name parameter in make_file_instructions
State: closed

Validate `name` parameter in `make_file_instructions`.

This way users get more informative error messages, instead of:
```stacktrace
.../huggingface/datasets/src/datasets/arrow_reader.py in make_file_instructions(name, split_infos, instruction, filetype_suffix, prefix_path)
    110     name2len = {info.name: info.num_examples for info in split_infos}
    111     name2shard_lengths = {info.name: info.shard_lengths for info in split_infos}
--> 112     name2filenames = {
    113         info.name: filenames_for_dataset_split(
    114             path=prefix_path,

.../huggingface/datasets/src/datasets/arrow_reader.py in <dictcomp>(.0)
    111     name2shard_lengths = {info.name: info.shard_lengths for info in split_infos}
    112     name2filenames = {
--> 113         info.name: filenames_for_dataset_split(
    114             path=prefix_path,
    115             dataset_name=name,

.../huggingface/datasets/src/datasets/naming.py in filenames_for_dataset_split(path, dataset_name, split, filetype_suffix, shard_lengths)
     68 
     69 def filenames_for_dataset_split(path, dataset_name, split, filetype_suffix=None, shard_lengths=None):
---> 70     prefix = filename_prefix_for_split(dataset_name, split)
     71     prefix = os.path.join(path, prefix)
     72 

.../huggingface/datasets/src/datasets/naming.py in filename_prefix_for_split(name, split)
     52 
     53 def filename_prefix_for_split(name, split):
---> 54     if os.path.basename(name) != name:
     55         raise ValueError(f"Should be a dataset name, not a path: {name}")
     56     if not re.match(_split_re, split):

.../lib/python3.9/posixpath.py in basename(p)
    140 def basename(p):
    141     """Returns the final component of a pathname"""
--> 142     p = os.fspath(p)
    143     sep = _get_sep(p)
    144     i = p.rfind(sep) + 1

TypeError: expected str, bytes or os.PathLike object, not NoneType
```

Related to #5895.
GitHub issue: Relax `ci.yml` trigger for `pull_request` based on modified paths
State: open

## What's in this PR?

As of a previous PR at #5902, I've seen that the CI was automatically trigger on any file, in that case when modifying a Jupyter Notebook (.ipynb), which IMO could be skipped, as the modification on the Jupyter Notebook has no effect/impact on the `ci.yml` outcome. So this PR controls the paths that trigger the `ci.yml` to avoid wasting resources when not needed.

## What's pending in this PR?

I would like to confirm whether this should affect both `push` and `pull_request`, since just modifications in those files won't change the `ci.yml` outcome, so maybe it's worth skipping it too in the `push` trigger.
GitHub issue: Fix `Overview.ipynb` & detach Jupyter Notebooks from `datasets` repository
State: closed

## What's in this PR?

This PR solves #5887 since there was a mismatch between the tokenizer and the model used, since the tokenizer was `bert-base-cased` while the model was `distilbert-base-case` both for the PyTorch and TensorFlow alternatives. Since DistilBERT doesn't use/need the `token_type_ids`, the `**batch` was failing, as the batch contained `input_ids`, `attention_mask`, `token_type_ids`, `start_positions` and `end_positions`, and `token_type_ids` was not required.

Besides that, at the end `seqeval` was being used to evaluate the model predictions, and just `evaluate` was being installed, so I've also included the `seqeval` installation.

Finally, I've re-run everything in Google Colab, and every cell was successfully executed!

## What was done on top of the original PR?

Based on the comments from @mariosasko and @stevhliu, I've updated the contents of this PR to also review the `quickstart.mdx` and update what was needed, besides that, we may eventually move the `Overview.ipynb` dataset to `huggingface/notebooks` following @stevhliu suggestions.
GitHub issue: Make prepare_split more robust if errors in metadata dataset_info splits
State: closed

This PR uses `split_generator.split_info` as default value for `split_info` if any exception is raised while trying to get `split_generator.name` from `self.info.splits` (this may happen if there is any error in the metadata dataset_info splits).

Please note that `split_info` is only used by the logger.

Fix #5895 if passed `verification_mode="no_checks"`:
```python
ds = load_dataset(
    "ArmelR/stack-exchange-instruction", 
    data_dir="data/finetune", 
    split="train", 
    verification_mode="no_checks", 
    revision="c609f1caade5cfbf3b9fe9cfa17d7cb000b457bd",
)
```
GitHub issue: Loading The flores data set for specific language
State: closed

### Describe the bug

I am trying to load the Flores data set

the code which is given is
```

from datasets import load_dataset

dataset = load_dataset("facebook/flores")

```
This gives the error of config name 

""ValueError: Config name is missing"

Now if I add some config it gives me the some error

"HFValidationError: Repo id must use alphanumeric chars or '-', '_', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is 96: 'facebook/flores, 'ace_Arab''.
"

How I can load the data of the specific language ?

Couldn't find any tutorial

any one can help me out?

### Steps to reproduce the bug

step one load the data set 

`from datasets import load_dataset

dataset = load_dataset("facebook/flores")`

it gives the error of config

once config is given 

it gives the error of

"HFValidationError: Repo id must use alphanumeric chars or '-', '_', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is 96: 'facebook/flores, 'ace_Arab''.
"

### Expected behavior

Data set should be loaded but  I am receiving error

### Environment info

Datasets , python ,
GitHub issue: HuggingFace does not cache downloaded files aggressively/early enough
State: closed

### Describe the bug

I wrote the following script:

```
import datasets

dataset = datasets.load.load_dataset("wikipedia", "20220301.en", split="train[:10000]")
```

I ran it and spent 90 minutes downloading a 20GB file. Then I saw:

```
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20.3G/20.3G [1:30:29<00:00, 3.73MB/s]
Traceback (most recent call last):
  File "/home/jack/Code/Projects/Transformers/Codebase/main.py", line 5, in <module>
    dataset = datasets.load.load_dataset("wikipedia", "20220301.en", split="train[:10000]")
  File "/home/jack/.local/lib/python3.10/site-packages/datasets/load.py", line 1782, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/jack/.local/lib/python3.10/site-packages/datasets/builder.py", line 883, in download_and_prepare
    self._save_info()
  File "/home/jack/.local/lib/python3.10/site-packages/datasets/builder.py", line 2037, in _save_info
    import apache_beam as beam
ModuleNotFoundError: No module named 'apache_beam'
```

And the 20GB of data was seemingly instantly gone forever, because when I ran the script again, it had to do the download again.

### Steps to reproduce the bug

See above

### Expected behavior

See above

### Environment info

datasets 2.10.1
Python 3.10
GitHub issue: The dir name and split strings are confused when loading ArmelR/stack-exchange-instruction dataset
State: closed

### Describe the bug

When I load the ArmelR/stack-exchange-instruction dataset, I encounter a bug that may be raised by confusing the dir name string and the split string about the dataset. 

When I use the script "datasets.load_dataset('ArmelR/stack-exchange-instruction', data_dir="data/finetune", split="train", use_auth_token=True)", it fails. But it succeeds when I add the "streaming = True" parameter. 

The website of the dataset is https://huggingface.co/datasets/ArmelR/stack-exchange-instruction/ .

The traceback logs are as below: 

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/xxx/miniconda3/envs/code/lib/python3.9/site-packages/datasets/load.py", line 1797, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/xxx/miniconda3/envs/code/lib/python3.9/site-packages/datasets/builder.py", line 890, in download_and_prepare
    self._download_and_prepare(
  File "/home/xxx/miniconda3/envs/code/lib/python3.9/site-packages/datasets/builder.py", line 985, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/xxx/miniconda3/envs/code/lib/python3.9/site-packages/datasets/builder.py", line 1706, in _prepare_split
    split_info = self.info.splits[split_generator.name]
  File "/home/xxx/miniconda3/envs/code/lib/python3.9/site-packages/datasets/splits.py", line 530, in __getitem__
    instructions = make_file_instructions(
  File "/home/xxx/miniconda3/envs/code/lib/python3.9/site-packages/datasets/arrow_reader.py", line 112, in make_file_instructions
    name2filenames = {
  File "/home/xxx/miniconda3/envs/code/lib/python3.9/site-packages/datasets/arrow_reader.py", line 113, in <dictcomp>
    info.name: filenames_for_dataset_split(
  File "/home/xxx/miniconda3/envs/code/lib/python3.9/site-packages/datasets/naming.py", line 70, in filenames_for_dataset_split
    prefix = filename_prefix_for_split(dataset_name, split)
  File "/home/xxx/miniconda3/envs/code/lib/python3.9/site-packages/datasets/naming.py", line 54, in filename_prefix_for_split
    if os.path.basename(name) != name:
  File "/home/xxx/miniconda3/envs/code/lib/python3.9/posixpath.py", line 142, in basename
    p = os.fspath(p)
TypeError: expected str, bytes or os.PathLike object, not NoneType





### Steps to reproduce the bug

1. import datasets library function: ```from datasets import load_dataset```
2. load dataset: ```ds=load_dataset('ArmelR/stack-exchange-instruction', data_dir="data/finetune", split="train", use_auth_token=True)```

### Expected behavior

The dataset can be loaded successfully without the streaming setting. 

### Environment info

Linux, 
python=3.9
datasets=2.12.0
GitHub issue: User access requests with manual review do not notify the dataset owner
State: closed

### Describe the bug

When a user access requests are enabled, and new requests are set to Manual Review, the dataset owner should be notified of the pending requests. However, instead, currently nothing happens, and so the dataset request can go unanswered for quite some time until the owner happens to check that particular dataset's Settings pane.

### Steps to reproduce the bug

1. Enable a dataset's user access requests
2. Set to Manual Review
3. Ask another HF user to request access to the dataset
4. Dataset owner is not notified

### Expected behavior

The dataset owner should receive some kind of notification, perhaps in their HF site inbox, or by email, when a dataset access request is made and manual review is enabled.

### Environment info

n/a
GitHub issue: Token Alignment for input and output data over train and test batch/dataset.
State: open

`data`
> DatasetDict({
    train: Dataset({
        features: ['input', 'output'],
        num_rows: 4500
    })
    test: Dataset({
        features: ['input', 'output'],
        num_rows: 500
    })
})

**# input (in-correct sentence)**
`data['train'][0]['input']`
**>>** 'We are meet sunday 10am12pmET in Crown Heights Brooklyn New York'
**# output (correct sentence)**
`data['train'][0]['output']`
**>>** 'We meet Sundays 10am-12pmET in Crown Heights, Brooklyn, New York.'

**I Want to align the output tokens with input**

```
`# tokenize both inputs and targets
def tokenize_fn(batch):
    # tokenize the input sequence first
    # this populates input_ids, attention_mask, etc.

    tokenized_inputs = tokenizer(
    batch['input']
    )
    
    labels_batch = tokenizer.tokenize(batch['output']) # original targets

    aligned_labels_batch = []
    for i, labels in enumerate(labels_batch):
        word_ids = tokenized_inputs[i].word_ids()
        aligned_labels_batch.append(align_targets(labels, word_ids))  # align_targets is another user defined function which is been called here

    # recall: the 'target' must be stored in key called 'labels'
    tokenized_inputs['labels'] = aligned_labels_batch

    return tokenized_inputs`
```
```
data.map(
  tokenize_fn,
  batched=True,
  remove_columns=data['train'].column_names,
)
```

When this user defined function is mapped to every records of train and test batch am getting following error:

**1.** **raise DatasetTransformationNotAllowedError(
   3457         "Using `.map` in batched mode on a dataset with attached indexes is allowed only if it doesn't create or remove existing examples. You can first run `.drop_index() to remove your index and then re-add it."**

**2.** **TypeError: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]**
GitHub issue: HuggingsFace dataset example give error
State: closed

### Describe the bug

![image](https://github.com/huggingface/datasets/assets/1328316/1f4f0086-3db9-4c79-906b-05a375357cce)


![image](https://github.com/huggingface/datasets/assets/1328316/733ebd3d-89b9-4ece-b80a-00ab5b0a4122)


### Steps to reproduce the bug

Use link as reference document written https://colab.research.google.com/github/huggingface/datasets/blob/main/notebooks/Overview.ipynb#scrollTo=biqDH9vpvSVz

```python
# Now let's train our model
device = 'cuda' if torch.cuda.is_available() else 'cpu'

model.train().to(device)
for i, batch in enumerate(dataloader):
    batch.to(device)
    outputs = model(**batch)
    loss = outputs.loss
    loss.backward()
    optimizer.step()
    model.zero_grad()
    print(f'Step {i} - loss: {loss:.3}')
    if i > 5:
        break
```

Error

```python
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[<ipython-input-44-7040b885f382>](https://localhost:8080/#) in <cell line: 5>()
      5 for i, batch in enumerate(dataloader):
      6     batch.to(device)
----> 7     outputs = model(**batch)
      8     loss = outputs.loss
      9     loss.backward()

[/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _call_impl(self, *args, **kwargs)
   1499                 or _global_backward_pre_hooks or _global_backward_hooks
   1500                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501             return forward_call(*args, **kwargs)
   1502         # Do not call functions when jit is used
   1503         full_backward_hooks, non_full_backward_hooks = [], []

TypeError: DistilBertForQuestionAnswering.forward() got an unexpected keyword argument 'token_type_ids'
```

https://github.com/huggingface/datasets/assets/1328316/5d8b1d61-9337-4d59-8423-4f37f834c156



### Expected behavior

Run success on Google Colab (free)

### Environment info

Windows 11 x64, Google Colab free (my Google Drive just empty about 200 MB, but I don't think it cause problem)
GitHub issue: Use work-stealing algorithm when parallel computing
State: open

### Feature request

when i used Dataset.map api to process data concurrently, i found that
 it gets slower and slower as it gets closer to completion. Then i read the source code of arrow_dataset.py and found that it shard the dataset and use multiprocessing pool to execute each shard.It may cause the slowest task to drag out the entire program's execution time,especially when processing huge dataset.

### Motivation

using work-stealing algorithm instead of sharding and parallel computing to optimize performance. 

### Your contribution

just an idea.
GitHub issue: A way to upload and visualize .mp4 files (millions of them) as part of a dataset
State: open

**Is your feature request related to a problem? Please describe.**
I recently chose to use huggingface hub as the home for a large multi modal dataset I've been building. https://huggingface.co/datasets/Antreas/TALI

It combines images, text, audio and video. Now, I could very easily upload a dataset made via datasets.Dataset.from_generator, as long as it did not include video files. I found that including .mp4 files in the entries would not auto-upload those files. 

Hence I tried to upload them myself. I quickly found out that uploading many small files is a very bad way to use git lfs, and that it would take ages, so, I resorted to using 7z to pack them all up. But then I had a new problem.

My dataset had a size of 1.9TB. Trying to upload such a large file with the default huggingface_hub API always resulted in time outs etc. So I decided to split the large files into chunks of 5GB each and reupload. 

So, eventually it all worked out. But now the dataset can't be properly and natively used by the datasets API because of all the needed preprocessing -- and furthermore the hub is unable to visualize things. 

**Describe the solution you'd like**
A native way to upload large datasets that include .mp4 or other video types.

**Describe alternatives you've considered**
Already explained earlier

**Additional context**
https://huggingface.co/datasets/Antreas/TALI