`FaissIndex` to support multiple GPU and `custom_index`

enhancement

New feature or request

Hi @rentruewang, thansk for reporting and for your PR!!! We should definitely support this.

**Is your feature request related to a problem? Please describe.**
Currently, because `device` is of the type `int | None`, to leverage `faiss-gpu`'s multi-gpu support, you need to create a `custom_index`. However, if using a `custom_index` created by e.g. `faiss.index_cpu_to_all_gpus`, then `FaissIndex.save` does not work properly because it checks the device id (which is an int, so no multiple GPUs).

**Describe the solution you'd like**
I would like `FaissIndex` to support multiple GPUs, by passing in a list to `add_faiss_index`.

**Describe alternatives you've considered**
Alternatively, I would like it to at least provide a warning cause it wasn't the behavior that I expected.

**Additional context**
Relavent source code here:
 https://github.com/huggingface/datasets/blob/6ed6ac9448311930557810383d2cfd4fe6aae269/src/datasets/search.py#L340-L349

Device management needs changing to support multiple GPUs, probably by `isinstance` calls.

I can provide a PR if you like :)

Thanks for reading!


Fix problems in msr_sqa

It shows below when I run test:

FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_msr_sqa - ValueError: Unknown split "validation". Should be one of ['train', 'test'].

It make no sense for me😂.

@albertvillanova Does this PR has some additional fixes compared to https://github.com/huggingface/datasets/pull/3771 or we can close it?

@mariosasko besides the fix of the DuplicatedKeysError, this PR:
- changes the reading of one of the files: use pandas instead of splitting by comma
- changes the splits: modifying train and adding validation
- adds some extra logic in the processing of the data: adding a new field "question_and_history"

We should decide whether validating these additional changes.
- for example, if we accept as pertinent the addition of the field "question_and_history", this should be added as feature to the info, and the matadata should be regenerated...

Hi guys, anything we can do to fix that bug👀? @mariosasko @albertvillanova @lhoestq

The last version has many problems,

1) Errors in table load-in. Split by a single comma instead of using pandas is wrong.
2) id reduplicated in _generate_examples function.
3) Missing information of history questions which make it hard to use.

I fix it refer to https://github.com/HKUNLP/UnifiedSKG. And we test it to perform normally.


tatoeba_mt: File not found error and key error

dataset-viewer

Related to the dataset viewer on huggingface.co

## Dataset viewer issue for 'tatoeba_mt'

**Link:** https://huggingface.co/datasets/Helsinki-NLP/tatoeba_mt

My data loader script does not seem to work.
The files are part of the local repository but cannot be found. An example where it should work is the subset for "afr-eng".

Another problem is that I do not have validation data for all subsets and I don't know how to properly check whether validation exists in the configuration before I try to download it. An example is the subset for "afr-deu".

Am I the one who added this dataset ? Yes


Rm sphinx doc

Thanks for pushing this :)
One minor comment regarding the PR itself - I noticed that some changes are coming from the upstream master, this might be due to a rebase. Would be nice if this PR doesn't include them for readabily, feel free to open a new one if necessary

Closing in favour https://github.com/huggingface/datasets/pull/3741

Checklist
- [x] Update circle ci yaml
- [x] Delete sphinx static & python files in docs dir
- [x] Update readme in docs dir
- [ ] Update docs config in setup.py


Fix the error of msr_sqa dataset

Fix the error of _load_table_data function in msr_sqa dataset, it is wrong to use comma to split each row.


Fix the error of _load_table_data function in msr_sqa dataset

The _load_table_data function from the last version is wrong, it is wrong to use comma to split each row.


Fix CI code quality issue

Fix CI code quality issue introduced by #3695.


Set base path to hub url for canonical datasets

If we agree to have data files in a dedicated directory "data/" then we should be fine. You're right we should not try to edit a dataset script from the repository directly, but from github, in order to avoid conflicts

This should allow canonical datasets to use relative paths to download data files from the Hub

cc @polinaeterna this will be useful if we have audio datasets that are canonical and for which you'd like to host data files


Loading JSON gets stuck with many workers/threads

bug

Something isn't working

Hi ! Note that it does `block_size *= 2` until `block_size > len(batch)`, so it doesn't loop indefinitely. What do you mean by "get stuck indefinitely" then ? Is this the actual call to `paj.read_json` that hangs ?

> increasing the `chunksize` argument decreases the chance of getting stuck

Could you share the values of chunksize that you're using to observe this ? And maybe the order of magnitude of number of bytes per line of JSON ?

To clarify, I don't think it loops indefinitely but the `paj.read_json` gets stuck after the first try. That's why I think it could be an issue with a lock somewhere.

Using `load_dataset(..., chunksize=40<<20)` worked without errors.

## Describe the bug
Loading a JSON dataset with `load_dataset` can get stuck when running on a machine with many CPUs. This is especially an issue when loading a large dataset on a large machine.

## Steps to reproduce the bug
I originally created the following script to reproduce the issue:
```python
from datasets import load_dataset
from multiprocessing import Process
from tqdm import tqdm
import datasets
from transformers import set_seed

def run_tasks_in_parallel(tasks, ds_list):
    for _ in tqdm(range(1000)):
        print('new batch')
        running_tasks = [Process(target=task, args=(ds, i)) for i, (task, ds) in enumerate(zip(tasks, ds_list))]
        for running_task in running_tasks:
            running_task.start()
        for running_task in running_tasks:
            running_task.join()

def get_dataset():
    dataset_name = 'transformersbook/codeparrot'
    ds = load_dataset(dataset_name+'-train', split="train", streaming=True)
    ds = ds.shuffle(buffer_size=1000, seed=1)
    return iter(ds)

def get_next_element(ds, process_id,  N=10000):
    for _ in range(N):
        _ = next(ds)['content']
    print(f'process {process_id} done')
    return

set_seed(1)
datasets.utils.logging.set_verbosity_debug()

n_processes = 8
tasks = [get_next_element for _ in range(n_processes)]
args = [get_dataset() for _ in range(n_processes)]
run_tasks_in_parallel(tasks, args)
```

Today I noticed that it can happen when running it on a single process on a machine with many cores without streaming. So just `load_dataset("transformersbook/codeparrot-train")` alone might cause the issue after waiting long enough or trying many times. It's a slightly random process which makes it especially hard to track down. When I encountered it today it had already processed 17GB of data (the size of the cache folder when it got stuck) before getting stuck.

Here's my current understanding of the error. As far as I can tell it happens in the following block: https://github.com/huggingface/datasets/blob/be701e9e89ab38022612c7263edc015bc7feaff9/src/datasets/packaged_modules/json/json.py#L119-L139

When the try on line 121 fails and the `block_size` is increased it can happen that it can't read the JSON again and gets stuck indefinitely. A hint that points in that direction is that increasing the `chunksize` argument decreases the chance of getting stuck and vice versa. Maybe it is an issue with a lock on the file that is not properly released.

## Expected results
Read a JSON before the end of the universe.

## Actual results
Read a JSON not before the end of the universe.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Linux-4.19.0-18-cloud-amd64-x86_64-with-glibc2.28
- Python version: 3.9.10
- PyArrow version: 7.0.0

@lhoestq we dicsussed this a while ago. @albertvillanova we discussed this today :)


`.select`: unexpected behavior with `indices`

Something isn't working

Hi! Currently, we compute the final index as `index % len(dset)`. I agree this behavior is somewhat unexpected and that it would be more appropriate to raise an error instead (this is what `df.iloc` in Pandas does, for instance).

@albertvillanova @lhoestq wdyt?

I agree. I think `index % len(dset)` was used to support negative indices.

I think this needs to be fixed in `datasets.formatting.formatting._check_valid_index_key` if I'm not mistaken

## Describe the bug
The `.select` method will not throw when sending `indices` bigger than the dataset length; `indices` will be wrapped instead. This behavior is not documented anywhere, and is not intuitive.

## Steps to reproduce the bug
```python
from datasets import Dataset
ds = Dataset.from_dict({"text": ["d", "e", "f"], "label": [4, 5, 6]})
res1 = ds.select([1, 2, 3])['text']
res2 = ds.select([1000])['text']
```

## Expected results
Both results should throw an `Error`.

## Actual results
`res1` will give `['e', 'f', 'd']`
`res2` will give `['e']`

## Environment info
Bug found from this environment:
- `datasets` version: 1.16.1
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.7
- PyArrow version: 6.0.1

It was also replicated on `master`.


Unable to load dataset 'big_patent'

Something isn't working

Hi @ankitk2109,

Have you tried passing the split name with the keyword `split=`? See e.g. an example in our Quick Start docs: https://huggingface.co/docs/datasets/quickstart.html#load-the-dataset-and-model
```python
 ds = load_dataset("big_patent", "d", split="validation")

Hi @albertvillanova,

Thanks for your response.

Yes, I tried the `split='validation'` as well. But getting the same issue.

I'm sorry, but I can't reproduce your problem:
```python
In [5]: ds = load_dataset("big_patent", "d", split="validation")
Downloading and preparing dataset big_patent/d (download: 6.01 GiB, generated: 169.61 MiB, post-processed: Unknown size, total: 6.17 GiB) to .../.cache/big_patent/d/1.0.0/bdefa7c0b39fba8bba1c6331b70b738e30d63c8ad4567f983ce315a5fef6131c...
Downloading data: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6.45G/6.45G [27:36<00:00, 3.89MB/s]
Extracting data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [03:18<00:00, 66.08s/it]
Dataset big_patent downloaded and prepared to .../.cache/big_patent/d/1.0.0/bdefa7c0b39fba8bba1c6331b70b738e30d63c8ad4567f983ce315a5fef6131c. Subsequent calls will reuse this data.

In [6]: ds
Out[6]:
Dataset({
    features: ['description', 'abstract'],
    num_rows: 565
})

Maybe you had a connection issue while downloading the file and this was corrupted?
Our cache system uses the file you downloaded first time.
If so, you could try forcing redownload of the file with:
```python
ds = load_dataset("big_patent", "d", split="validation", download_mode="force_redownload")

I am able to download the dataset with ``` download_mode="force_redownload"```. As you mentioned it was an issue with the cached version which was failed earlier due to a network issue. I am closing the issue now, once again thank you.

## Describe the bug
Unable to load the "big_patent" dataset

## Steps to reproduce the bug
```python
load_dataset('big_patent', 'd', 'validation')
```

## Expected results
Download big_patents' validation split from the 'd' subset

## Getting an error saying:
{FileNotFoundError}Local file ..\huggingface\datasets\downloads\6159313604f4f2c01e7d1cac52139343b6c07f73f6de348d09be6213478455c5\bigPatentData\train.tar.gz doesn't exist

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:1.18.3
- Platform: Windows
- Python version:3.8
- PyArrow version:7.0.0


Raise informative error when loading a save_to_disk dataset

People recurrently report error when trying to load a dataset (using `load_dataset`) that was previously saved using `save_to_disk`.

This PR raises an informative error message telling them they should use `load_from_disk` instead.

Close #3700.


OSCAR-2109 datasets are misaligned and truncated

Something isn't working

Hi @adrianeboyd, thanks for reporting.

There is indeed a bug in that community dataset:
Line:
```python
metadata_and_text_files = list(zip(metadata_files, text_files))
```
should be replaced with
```python
metadata_and_text_files = list(zip(sorted(metadata_files), sorted(text_files)))
```

I am going to contact their owners (https://huggingface.co/oscar-corpus) in order to inform them about the bug.

I keep you informed.

That fix is part of it, but it's clearly not the only issue.

I also already contacted the OSCAR creators, but I reported it here because it looked like huggingface members were the main authors in the git history. Is there a better place to have reported this?

Hello,

We've had an issue that could be linked to this one here: https://github.com/oscar-corpus/corpus/issues/15.

I have been spot checking the source (`.txt`/`.jsonl`) files for a while, and have not found issues, especially in the start/end of corpora (but I conceed that more integration testing would be necessary on our side).

The text and metadata files are designed to be used in sync (with `lang_part_n.txt` and `lang_meta_part_n.jsonl` working together), while staying independent from part to part, so that anyone could randomly choose a part and work with it.

The fix @albertvillanova proposed should fix the problem, as the parts will be in sync again.

Let me know if you need help or more details, I'd be glad to help!

I'm happy to move the discussion to the other repo!

Merely sorting the files only **maybe** fixes the processing of the first part. If the first part contains non-unix newlines, it will still be misaligned/truncated, and all the following parts will be truncated with incorrect text offsets and metadata due the offset and newline bugs.

Hi @Uinelj, This is a total noobs question but how can I integrate that bugfix into my code? I reinstalled the datasets library this time from source. Should that have fixed the issue? I am still facing the misalignment issue. Do I need to download the dataset from scratch?

Hi, I re-downloaded the dataset and still have the problem. See: https://github.com/oscar-corpus/corpus/issues/18

Sorry @norakassner for the late reply.

There are indeed several issues creating the misalignment, as @adrianeboyd cleverly pointed out:
-  https://huggingface.co/datasets/oscar-corpus/OSCAR-2109/commit/3cd7e95aa1799b73c5ea8afc3989635f3e19b86b fixed one of them
- but there are still others to be fixed

Normally, the issues should be fixed now:
- Fix offset initialization for each file: https://huggingface.co/datasets/oscar-corpus/OSCAR-2109/commit/1ad9b7bfe00798a9258a923b887bb1c8d732b833
- Disable default universal newline support: https://huggingface.co/datasets/oscar-corpus/OSCAR-2109/commit/0c2f307d3167f03632f502af361ac6c3c393f510

Feel free to reopen if you find additional misalignments/truncations.

CC: @adrianeboyd @norakassner @Uinelj

Thanks for the updates!

The purist in me would still like to have the rstrip not strip additional characters from the original text (unicode whitespace mainly in practice, I think), but the differences are extremely small in practice and it doesn't actually matter for my current task:

```python
text = "".join([text_f.readline() for _ in range(meta["nb_sentences"])]).rstrip("\n")
```

## Describe the bug

The `oscar-corpus/OSCAR-2109` data appears to be misaligned and truncated by the dataset builder for subsets that contain more than one part and for cases where the texts contain non-unix newlines.

## Steps to reproduce the bug

A few examples, although I'm not sure how deterministic the particular (mis)alignment is in various configurations:

```python
from datasets import load_dataset
dataset = load_dataset("oscar-corpus/OSCAR-2109", "deduplicated_fi", split="train", use_auth_token=True)
entry = dataset[0]
# entry["text"] is from fi_part_3.txt.gz
# entry["meta"] is from fi_meta_part_2.jsonl.gz

dataset = load_dataset("oscar-corpus/OSCAR-2109", "deduplicated_no", split="train", use_auth_token=True)
entry = dataset[900000]
# entry["text"] is from no_part_3.txt.gz and contains a blank line
# entry["meta"] is from no_meta_part_1.jsonl.gz

dataset = load_dataset("oscar-corpus/OSCAR-2109", "deduplicated_mk", split="train", streaming=True, use_auth_token=True)
# 9088 texts in the dataset are empty
```

For `deduplicated_fi`, all exported raw texts from the dataset are 17GB rather than 20GB as reported in the data splits overview table. The token count with `wc -w` for the raw texts is 2,067,556,874 rather than the expected 2,357,264,196 from the data splits table.

For `deduplicated_no` all exported raw texts contain 624,040,887 rather than the expected 776,354,517 tokens.

For `deduplicated_mk` it is 122,236,936 rather than 134,544,934 tokens.

I'm not expecting the `wc -w` counts to line up exactly with the data splits table, but for comparison the `wc -w` count for `deduplicated_mk` on the raw texts is 134,545,424.

## Issues

* The meta / text files are not paired correctly when loading, so the extracted texts do not have the right offsets, the metadata is not associated with the correct text, and the text files may not be processed to the end or may be processed beyond the end (empty texts).
* The line count offset is not reset per file so the texts aren't aligned to the right offsets in any parts beyond the first part, leading to truncation when in effect blank lines are not skipped.
* Non-unix newline characters are treated as newlines when reading the text files while the metadata only counts unix newlines for its line offsets, leading to further misalignments between the metadata and the extracted texts, and which also results in truncation.

## Expected results

All texts from the OSCAR release are extracted according to the metadata and aligned with the correct metadata.

## Fixes

Not necessarily the exact fixes/checks you may want to use (I didn't test all languages or do any cross-platform testing, I'm not sure all the details are compatible with streaming), however to highlight the issues:

```diff
diff --git a/OSCAR-2109.py b/OSCAR-2109.py
index bbac1076..5eee8de7 100644
--- a/OSCAR-2109.py
+++ b/OSCAR-2109.py
@@ -20,6 +20,7 @@
 import collections
 import gzip
 import json
+import os

 import datasets

@@ -387,9 +388,20 @@ class Oscar2109(datasets.GeneratorBasedBuilder):
         with open(checksum_file, encoding="utf-8") as f:
             data_filenames = [line.split()[1] for line in f if line]
             data_urls = [self.config.base_data_path + data_filename for data_filename in data_filenames]
-        text_files = dl_manager.download([url for url in data_urls if url.endswith(".txt.gz")])
-        metadata_files = dl_manager.download([url for url in data_urls if url.endswith(".jsonl.gz")])
+        # sort filenames so corresponding parts are aligned
+        text_files = sorted(dl_manager.download([url for url in data_urls if url.endswith(".txt.gz")]))
+        metadata_files = sorted(dl_manager.download([url for url in data_urls if url.endswith(".jsonl.gz")]))
+        assert len(text_files) == len(metadata_files)
         metadata_and_text_files = list(zip(metadata_files, text_files))
+        for meta_path, text_path in metadata_and_text_files:
+            # check that meta/text part numbers are the same
+            if "part" in os.path.basename(text_path):
+                assert (
+                    os.path.basename(text_path).replace(".txt.gz", "").split("_")[-1]
+                    == os.path.basename(meta_path).replace(".jsonl.gz", "").split("_")[-1]
+                )
+            else:
+                assert len(metadata_and_text_files) == 1
         return [
             datasets.SplitGenerator(name=datasets.Split.TRAIN, gen_kwargs={"metadata_and_text_files": metadata_and_text_files}),
         ]
@@ -397,10 +409,14 @@ class Oscar2109(datasets.GeneratorBasedBuilder):
     def _generate_examples(self, metadata_and_text_files):
         """This function returns the examples in the raw (text) form by iterating on all the files."""
         id_ = 0
-        offset = 0
         for meta_path, text_path in metadata_and_text_files:
+            # line offsets are per text file
+            offset = 0
             logger.info("generating examples from = %s", text_path)
-            with gzip.open(open(text_path, "rb"), "rt", encoding="utf-8") as text_f:
+            # some texts contain non-Unix newlines that should not be
+            # interpreted as line breaks for the line counts in the metadata
+            # with readline()
+            with gzip.open(open(text_path, "rb"), "rt", encoding="utf-8", newline="\n") as text_f:
                 with gzip.open(open(meta_path, "rb"), "rt", encoding="utf-8") as meta_f:
                     for line in meta_f:
                         # read meta
@@ -411,7 +427,12 @@ class Oscar2109(datasets.GeneratorBasedBuilder):
                             offset += 1
                             text_f.readline()
                         # read text
-                        text = "".join([text_f.readline() for _ in range(meta["nb_sentences"])]).rstrip()
+                        text_lines = [text_f.readline() for _ in range(meta["nb_sentences"])]
+                        # all lines contain text (no blank lines or EOF)
+                        assert all(text_lines)
+                        assert "\n" not in text_lines
                         offset += meta["nb_sentences"]
+                        # only strip the trailing newline
+                        text = "".join(text_lines).rstrip("\n")
                         yield id_, {"id": id_, "text": text, "meta": meta}
                         id_ += 1
```

I've tested this with a number of smaller deduplicated languages with 1-20 parts and the resulting datasets looked correct in terms of word count and size when compared to the data splits table and raw texts, and the text/metadata alignments were correct in all my spot checks. However, there are many many languages I didn't test and I'm not sure that there aren't any texts containing blank lines in the corpus, for instance. For the cases I tested, the assertions related to blank lines and EOF made it easier to verify that the text and metadata were aligned as intended, since there would be little chance of spurious alignments of variable-length texts across so much data.


> Hi! Some of our metrics require additional dependencies to work. In your case, simply installing the `seqeval` package with `pip install seqeval` should resolve the issue.
I installed seqeval, but still reported the same error. That's too bad.

> > Hi! Some of our metrics require additional dependencies to work. In your case, simply installing the `seqeval` package with `pip install seqeval` should resolve the issue.
> > I installed seqeval, but still reported the same error. That's too bad.

Same issue here. What should I do to fix this error? Please help! Thank you.

I tried to install **seqeval** package through anaconda instead of pip:
`conda install -c conda-forge seqeval`
It worked for me!

I can run it through the following steps:
![image](https://user-images.githubusercontent.com/69563759/159264511-1e252a4e-c8c8-44ab-b7bc-b4aac609bd9e.png)
Thank you for answering for me!

hi :
I want to use the seqeval indicator because of direct load_ When metric ('seqeval '), it will prompt that the network connection fails. So I downloaded the seqeval Py to load locally. Loading code: metric = load_ metric(path='mymetric/seqeval/seqeval.py')

But tips:

Traceback (most recent call last):
  File "/home/ubuntu/Python3.6_project/zyf_project/transformers/examples/pytorch/token-classification/run_ner.py", line 604, in <module>
    main()
  File "/home/ubuntu/Python3.6_project/zyf_project/transformers/examples/pytorch/token-classification/run_ner.py", line 481, in main
    metric = load_metric(path='mymetric/seqeval/seqeval.py')
  File "/home/ubuntu/Python3.6_project/zyf_project/transformers_venv_0209/lib/python3.7/site-packages/datasets/load.py", line 610, in load_metric
    dataset=False,
  File "/home/ubuntu/Python3.6_project/zyf_project/transformers_venv_0209/lib/python3.7/site-packages/datasets/load.py", line 450, in prepare_module
    f"To be able to use this {module_type}, you need to install the following dependencies"
ImportError: To be able to use this metric, you need to install the following dependencies['seqeval'] using 'pip install seqeval' for instance'

**What should I do? Please help me, thank you**


Update the address to use https

Hi ! I'm getting some 503 from both the http and https addresses. Do you think we could host this data somewhere else ? (please check if there is a license and if it allows redistribution)

The http address doesn't work anymore


Pin ElasticSearch

Until we manage to support ES 8.0, I'm setting the version to `<8.0.0`

Currently we're getting this error on 8.0:
```python
ValueError: Either 'hosts' or 'cloud_id' must be specified
```
When instantiating a `Elasticsearch()` object


Unable to load a dataset

Something isn't working

Hi! `load_dataset` is intended to be used to load a canonical dataset (`wikipedia`), a packaged dataset (`csv`, `json`, ...) or a dataset hosted on the Hub. For local datasets saved with `save_to_disk("path/to/dataset")`, use `load_from_disk("path/to/dataset")`.

Maybe we should raise an informative error message in this case...

## Describe the bug
Unable to load a dataset from Huggingface that I have just saved.

## Steps to reproduce the bug
On Google colab
`! pip install datasets `
`from datasets import load_dataset`
`my_path = "wiki_dataset"`
`dataset = load_dataset('wikipedia', "20200501.fr")`
`dataset.save_to_disk(my_path)`
`dataset = load_dataset(my_path)`

## Expected results
Loading the dataset

## Actual results
ValueError: Couldn't cast
_data_files: list<item: struct<filename: string>>
  child 0, item: struct<filename: string>
      child 0, filename: string
_fingerprint: string
_format_columns: null
_format_kwargs: struct<>
_format_type: null
_indexes: struct<>
_output_all_columns: bool
_split: string
to
{'builder_name': Value(dtype='string', id=None), 'citation': Value(dtype='string', id=None), 'config_name': Value(dtype='string', id=None), 'dataset_size': Value(dtype='int64', id=None), 'description': Value(dtype='string', id=None), 'download_checksums': {}, 'download_size': Value(dtype='int64', id=None), 'features': {'title': {'dtype': Value(dtype='string', id=None), 'id': Value(dtype='null', id=None), '_type': Value(dtype='string', id=None)}, 'text': {'dtype': Value(dtype='string', id=None), 'id': Value(dtype='null', id=None), '_type': Value(dtype='string', id=None)}}, 'homepage': Value(dtype='string', id=None), 'license': Value(dtype='string', id=None), 'post_processed': Value(dtype='null', id=None), 'post_processing_size': Value(dtype='null', id=None), 'size_in_bytes': Value(dtype='int64', id=None), 'splits': {'train': {'name': Value(dtype='string', id=None), 'num_bytes': Value(dtype='int64', id=None), 'num_examples': Value(dtype='int64', id=None), 'dataset_name': Value(dtype='string', id=None)}}, 'supervised_keys': Value(dtype='null', id=None), 'task_templates': Value(dtype='null', id=None), 'version': {'version_str': Value(dtype='string', id=None), 'description': Value(dtype='string', id=None), 'major': Value(dtype='int64', id=None), 'minor': Value(dtype='int64', id=None), 'patch': Value(dtype='int64', id=None)}}
because column names don't match

## Environment info
- `datasets` version: 1.18.3
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 6.0.1


Add dev-only config to Natural Questions dataset

Great thanks ! I think we can fix the CI by copying the NQ folder on gcs to 0.0.3. Does that sound good ?

I've copied the 0.0.2 folder content to 0.0.3, as suggested.

I'm updating the dataset card...

As suggested by @lhoestq and @thomwolf, a new config has been added to Natural Questions dataset, so that only dev split can be downloaded.

Fix #413.


Add finetune-data CodeFill


Add code-fill datasets for pretraining/finetuning/evaluating

Hi ! Thanks for adding this dataset :)

It looks like your PR contains many changes in files that are unrelated to your changes, I think it might come from running `make style` with an outdated version of `black`. Could you try opening a new PR that only contains your additions ? (or force push to this PR)


Force unique keys in newsqa dataset

Currently, it may raise `DuplicatedKeysError`.

Fix #3630.


Fix ClassLabel to/from dict when passed names_file

Currently, `names_file` is a field of the data class `ClassLabel`, thus appearing when transforming it to dict (when saving infos). Afterwards, when trying to read it from infos, it conflicts with the other field `names`.

This PR, removes `names_file` as a field of the data class `ClassLabel`.
- it is only used at instantiation to generate the `labels` field

Fix #3631.


Standardize to `Example::`


Update data URL in pubmed dataset

- I updated the previous dummy data: I just had to rename the file and its directory
  - the dummy data zip contains only a single file: `pubmed22n0001.xml.gz`

Then I discover it fails: https://app.circleci.com/pipelines/github/huggingface/datasets/9800/workflows/173a4433-8feb-4fc6-ab9e-59762084e3e1/jobs/60437
```
No such file or directory: '.../dummy_data/pubmed22n0002.xml.gz'
```
- it needs dummy data for all the 1114 files:
  `_URLs = [f"ftp://ftp.ncbi.nlm.nih.gov/pubmed/baseline/pubmed22n{i:04d}.xml.gz" for i in range(1, 1115)]`
- this confirms me that it never passed the test: these dummy data files were not present before my PR
- therefore, is it really useful the data test if we just ignore it when it does not pass?

In relation with JSON metadata, I was generating the file for `pubmed` (see above) in a GCP instance: after running during ~12h without finishing, I decided to stop the process.

Hi ! Yes I remembered we hardcoded an exception for this one:
https://github.com/huggingface/datasets/blob/36db39c75179a0a491c69a4491f7ae7e4615e66f/src/datasets/utils/mock_download_manager.py#L174-L176

The exception was used to only require one dummy data file, feel free to update it if you want

Fix #3655.


Upgrade black to version ~=22.0

Upgrades the `datasets` library quality tool `black` to use the first stable release of `black`, version 22.0.


Update docs to new frontend/UI

We can have the docstrings of the properties that are missing docstrings (from discussion [here](https://github.com/huggingface/doc-builder/pull/96)) here by using your new `inject_arrow_table_documentation` onthem as well ?

@sgugger & @lhoestq could you help me with what should the `docs` section in setup.py be changed to [here](https://github.com/huggingface/datasets/blob/master/setup.py#L212-L227) ?

as a reference, here is a transformers setup.py docs [section](https://github.com/huggingface/transformers/blob/master/setup.py#L304-L308)

For now, you can put an empty list. Once the `doc-builder` is in a PyPi package (with the bug we fixed on Datasets but still waiting on the standing PR with the code switch) we can put it there.

None of those dependencies are needed from this list?

```py
    "docs": [
        "docutils==0.16.0",
        "recommonmark",
        "sphinx==3.1.2",
        "sphinx-markdown-tables",
        "sphinx-rtd-theme==0.4.3",
        "sphinxext-opengraph==0.4.1",
        "sphinx-copybutton",
        "fsspec<2021.9.0",
        "s3fs",
        "sphinx-panels",
        "sphinx-inline-tabs",
        "myst-parser",
        "Markdown!=3.3.5",
    ],
```

No, that was all for sphinx. The only thing needed to build the doc is a pip install of `doc-builder` (only from git right now).

@lhoestq feel free to request reviews from other maintainers 😊

Thanks ! @mariosasko and @albertvillanova feel free to take a look :)
I can do a thorough review this afternoon

Cool thanks ! Feel free to merge master into this branch and run `make style` to fix the python code formatting

Love the colorful vibes here!
![Screen Shot 2022-02-22 at 9 54 17 AM](https://user-images.githubusercontent.com/59462357/155193444-45e639dc-79cd-463c-98ad-1d44a6d6d385.png)

I just fixed the conflicts with the `master` branch :)

Could you update preprod please ? Or is there a preview somewhere I can check to make sure everything is ok ?

> Could you update preprod please ? Or is there a preview somewhere I can check to make sure everything is ok ?

I'll let you know once preprod gets updated

@lhoestq @stevhliu updated [preprod](https://moon-preprod.huggingface.co/docs/datasets/index) with the latest; please let e know if you see any errors

One more tiny error that doesn't seem specific to Datasets (Transformers example [here](https://huggingface.co/docs/transformers/multilingual#xlm-language-embeddings)), but apostrophes and symbols aren't properly displayed in the right navbar:

![Screen Shot 2022-03-02 at 8 39 10 AM](https://user-images.githubusercontent.com/59462357/156406988-27e79533-b02a-4fc2-af32-8ad84657488f.png)

In the latest commit https://github.com/huggingface/datasets/pull/3690/commits/20bddf28b22798c309e6eb1198a716f055889e1b, I tried to reflect changes from https://github.com/huggingface/transformers/pull/15903 , however, the gh workflow is not being triggered. @lhoestq do you know why it might be the case?

eve though, we have
https://github.com/huggingface/datasets/blob/20bddf28b22798c309e6eb1198a716f055889e1b/.github/workflows/build_dev_documentation.yml#L3-L7

I removed this line to trigger the job
```
  pull_request:
```

but got this error
```
[Error: .github#L1](https://github.com/huggingface/datasets/commit/033fe623c556b9dbc964708b672ff9bb4896c906#annotation_2897984435)
a step cannot have both the `uses` and `run` keys
```

It seems to be running again, and I re-added the line I removed.

Now the error is
```
> Run cd doc-build-dev && ...
README.md
datasets
transformers
On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean
Error: Process completed with exit code 1.
```

@lhoestq if the CI passes, Im gonna merge this PR
please let me know if that sounds good

### TLDR: Update `datasets` `docs` to the new syntax (markdown and mdx files) & frontend (as how it looks on [hf.co/transformers](https://huggingface.co/docs/transformers/index))

| Light mode                                                                                | Dark mode                                                             |
|-----------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------|
|     |                                      |

## Checklist

- [x] update datasets docs to new syntax (should call `doc-builder convert`) (this PR)
- [x] discuss `@property` methods frontend https://github.com/huggingface/doc-builder/pull/87
- [x] discuss `inject_arrow_table_documentation` (this PR) https://github.com/huggingface/datasets/pull/3690#discussion_r801847860
- [x] update datasets docs path on moon-landing https://github.com/huggingface/moon-landing/pull/2089
- [x] convert pyarrow docstring from Numpydoc style to groups style  https://github.com/huggingface/doc-builder/pull/89(https://stackoverflow.com/a/24385103/6558628)
- [x] handle `Raises` section on frontend and doc-builder https://github.com/huggingface/doc-builder/pull/86
- [x] check imgs path (this PR) (nothing to update here)
- [x] doc exaples block has to follow format `Examples::` https://github.com/huggingface/datasets/pull/3693
- [x] fix [this docstring](https://github.com/huggingface/datasets/blob/6ed6ac9448311930557810383d2cfd4fe6aae269/src/datasets/arrow_dataset.py#L3339) (causing svelte compilation error)
- [x] Delete sphinx related files
- [x] Delete sphinx CI
- [x]  Update docs config in setup.py
- [x] add `versions.yml` in doc-build  https://github.com/huggingface/doc-build/pull/1
- [x] add `versions.yml` in doc-build-dev https://github.com/huggingface/doc-build-dev/pull/1
- [x] https://github.com/huggingface/moon-landing/pull/2089
- [x] format docstrings for example `datasets.DatasetBuilder.download_and_prepare` args format look wrong
- [x] create new github actions. (can probably be in a separate PR) (see the transformers equivalents below)
1. [build_dev_documentation.yml](https://github.com/huggingface/transformers/blob/master/.github/workflows/build_dev_documentation.yml)
2. [build_documentation.yml](https://github.com/huggingface/transformers/blob/master/.github/workflows/build_documentation.yml)
3. [delete_dev_documentation.yml](https://github.com/huggingface/transformers/blob/master/.github/workflows/delete_dev_documentation.yml)

## Note to reviewers

The number of changed files is a lot (100+) because I've converted all `.rst` files to `.mdx` files & they are compiling fine on the svelte side (also, moved all the imgs to to [doc-imgs repo](https://huggingface.co/datasets/huggingface/documentation-images/tree/main/datasets)). Moreover, you should just review them on preprod and see if the rendering look fine.
_Therefore, I'd suggest to focus on the changed_ **`.py`** and **CI files** (github workflows, etc. you can use [this filter here](https://github.com/huggingface/datasets/pull/3690/files?file-filters%5B%5D=.py&file-filters%5B%5D=.yml&show-deleted-files=true&show-viewed-files=true)) during the review & ignore `.mdx` files. (if there's a bug in `.mdx` files, we can always handle it in a separate PR afterwards).


Fix streaming for servers not supporting HTTP range requests

Does it mean that huge files might end up being downloaded? It would go against the purpose of streaming, I think. At least, this fallback should be an option that could be disabled

Yes, it is against the purpose of streaming, but streaming is not possible if the server does not allow HTTP range requests.

We have two options: either we download the file or we throw an error.

I think we simply cannot fallback to downloading the file if streaming fails without the user being aware of it. Some options:
- make the fallback optional (using an env var? or a function param)
- use the fallback only if the dataset size is under some threshold (provided we have the data in the DatasetInfo) -> it's the option I use in `datasets-preview-backend` ([here](https://github.com/huggingface/datasets-preview-backend/blob/48ac19e49c19809763e8d640986bf2c3d792faed/src/datasets_preview_backend/models/typed_row.py#L40) and [here](https://github.com/huggingface/datasets-preview-backend/blob/aa86c5493b275c9e2dbae7dab7bd469da5773a41/src/datasets_preview_backend/models/split.py#L31-L37))
- throw an exception and let the user decide what to do

IMO in general we should throw an exception and ask the user to not use streaming mode in that case.

Your second point is also interesting but I feel like it could be confusing for users sometimes: it doesn't feel natural that the streaming-ability should depend on the size of the file.

Sure, I think we should just throw an exception

Current behavior is already throwing an Exception:
```
ValueError: Cannot seek streaming HTTP file
```

We could customize the exception class and/or the exception message.

I'm not sure we really need to change anything. I opened the issue https://github.com/huggingface/datasets/issues/3677  because discovery was streamable and is not anymore (according to my test suite in https://github.com/huggingface/datasets-preview-backend): I was not sure if it was due to some regression in the library, or to some change in the dataset itself.

I'm wondering why it worked before and it is no longer working...

> We could customize the exception class and/or the exception message.

Yup a message that says that the host doesn't support streaming because it doesn't support HTTP Range requests would be useful !

Some servers do not support HTTP range requests, whereas this is required to stream some file formats (like ZIP).

~~This PR implements a workaround for those cases, by download the files locally in a temporary directory (cleaned up by the OS once the process is finished).~~

This PR raises custom error explaining that streaming is not possible because data host server does not support HTTP range requests.

Fix #3677.


Pyarrow version error

Something isn't working

Hi @Zaker237, thanks for reporting.

This is weird: the error you get is only thrown if the installed pyarrow version is less than 3.0.0.

Could you please check that you install pyarrow in the same Python virtual environment where you installed datasets?

From the Python command line (or terminal) where you get the error, please type:
```
import pyarrow
print(pyarrow.__version__)
import datasets
print(datasets.__version__)
```

hi @albertvillanova  i try yesterday to create a new python environement with python 7 and try it on the environement and it worked. so i think that the error was not the package but may be jupyter notebook on conda. still yet i'm not yet sure but it worked in an environment created with venv

OK, thanks @Zaker237 for your feedback.

I close this issue then. Please, feel free to reopen it if the problem arises again.

## Describe the bug
I installed datasets(version 1.17.0, 1.18.0, 1.18.3) but i'm right now nor able to import it because of pyarrow. when i try to import it, i get the following error:
`To use datasets, the module pyarrow>=3.0.0 is required, and the current version of pyarrow doesn't match this condition`.
i tryed with all version of pyarrow execpt `4.0.0` but still get the same error.

## Steps to reproduce the bug
```python
import datasets
```

## Expected results
A clear and concise description of the expected results.

## Actual results
AttributeError                            Traceback (most recent call last)
<ipython-input-19-652e886d387f> in <module>
----> 1 import datasets

~\AppData\Local\Continuum\anaconda3\lib\site-packages\datasets\__init__.py in <module>
     26
     27
---> 28 if _version.parse(pyarrow.__version__).major < 3:
     29     raise ImportWarning(
     30         "To use `datasets`, the module `pyarrow>=3.0.0` is required, and the current version of `pyarrow` doesn't match this condition.\n"

AttributeError: 'Version' object has no attribute 'major'

## Environment info
Traceback (most recent call last):
  File "c:\users\alex\appdata\local\continuum\anaconda3\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\alex\appdata\local\continuum\anaconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\Alex\AppData\Local\Continuum\anaconda3\Scripts\datasets-cli.exe\__main__.py", line 5, in <module>
  File "c:\users\alex\appdata\local\continuum\anaconda3\lib\site-packages\datasets\__init__.py", line 28, in <module>
    if _version.parse(pyarrow.__version__).major < 3:
AttributeError: 'Version' object has no attribute 'major'

- `datasets` version:
- Platform: Linux(Ubuntu) and Windows: conda on the both
- Python version: 3.7
- PyArrow version: 7.0.0


Can't get the text data when calling to_tf_dataset

You are correct that `to_tf_dataset` only handles numerical columns right now, yes, though this is a limitation we might remove in future! The main reason we do this is that our models mostly do not include the tokenizer as a model layer, because it's very difficult to compile some of them in TF. So the "normal" Huggingface workflow is to first tokenize your dataset, and then pass tokenized tensors to the model.

For your use case, would you prefer to pass strings to the model, and use some text processing layers instead of the built-in tokenizers?

Also tagging @gante just so he's aware, but I can handle this one!

Thanks for the quick follow-up to my issue.

For my use-case, instead of the built-in tokenizers I wanted to use the `TextVectorization` layer to map from strings to integers. To achieve this, I came up with the following solution:

```
from datasets import load_dataset
from transformers import DefaultDataCollator
import tensorflow as tf
import string
import re
from tensorflow.keras.layers.experimental.preprocessing import TextVectorization

#some hyper-parameters for the text-to-integer mapping
max_features = 20000
embedding_dim = 128
sequence_length = 210

data_collator = DefaultDataCollator(return_tensors="tf")
dataset = load_dataset("sst", "default")

#adapt the vectorization layer on train data only
vectorize_layer.adapt(dataset["train"].to_dict(batched=False)["sentence"])

def prepare_features(text, label):
    text = tf.expand_dims(text, -1)
    return {"vectorized_text": vectorize_layer(text)[0], "label": tf.expand_dims(label, axis=-1)}

encoded_dataset = dataset.map(lambda example: prepare_features(example["sentence"], example["label"]), batched=False)

def custom_standardization(input_data):
    lowercase = tf.strings.lower(input_data)
    return tf.strings.regex_replace(
        lowercase, f"[{re.escape(string.punctuation)}]", ""
    )

vectorize_layer = TextVectorization(
    standardize=custom_standardization,
    max_tokens=max_features,
    output_mode="int",
    output_sequence_length=sequence_length,
)

train_dataset = encoded_dataset["train"].to_tf_dataset(columns=['vectorized_text'], label_cols=["label"],
                                                       shuffle=True, batch_size=1, collate_fn=data_collator).unbatch()
#similar for the other sub-sets

```

Since the strings would have been mapped to integers or floats at some point, it's no drawback that this mapping is done early in the process.

For the future, however, it'd be more convenient to get the string data, since I am also inspecting the dataset (longest sentence, shortest sentence), which is more challenging when working with integer or float. For now, this can be done by calling `to_dict`.

> For the future, however, it'd be more convenient to get the string data, since I am also inspecting the dataset (longest sentence, shortest sentence), which is more challenging when working with integer or float.

Yes, I agree, so let's keep this issue open.

I am working with the SST2 dataset, and am using TensorFlow 2.5
I'd like to convert it to a `tf.data.Dataset` by calling the `to_tf_dataset` method.
The following snippet is what I am using to achieve this:

```
from datasets import load_dataset
from transformers import DefaultDataCollator

data_collator = DefaultDataCollator(return_tensors="tf")
dataset = load_dataset("sst")

train_dataset = dataset["train"].to_tf_dataset(columns=['sentence'], label_cols="label", shuffle=True, batch_size=8,collate_fn=data_collator)
```
However, this only gets me the labels; the text--the most important part--is missing:

```
for s in train_dataset.take(1):
  print(s) #prints something like: ({}, <tf.Tensor: shape=(8,), ...>)
```

As you can see, it only returns the label part, not the data, as indicated by the empty dictionary, `{}`. So far, I've played with various settings of the method arguments, but to no avail; I do not want to perform any text processing at this time. On my quest to achieve what I want ( a `tf.data.Dataset`), I've consulted these resources:

[https://www.philschmid.de/huggingface-transformers-keras-tf](https://www.philschmid.de/huggingface-transformers-keras-tf)
[https://huggingface.co/docs/datasets/use_dataset.html?highlight=tensorflow](https://huggingface.co/docs/datasets/use_dataset.html?highlight=tensorflow)

I was surprised to not find more extensive examples on how to transform a Hugginface dataset to one compatible with TensorFlow.

If you could point me to where I am going wrong, please do so.
Thanks in advance for your support.

---
Edit: In the [docs](https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.to_tf_dataset), I found the following description:

_In general, only columns that the model can use as input should be included here (numeric data only)._

Does this imply that no textual, i.e., `string` data can be loaded?


`Translation` features cannot be `flatten`ed

Something isn't working

Thanks for reporting, @SBrandeis! Some additional feature types that don't behave as expected when flattened: `Audio`, `Image` and `TranslationVariableLanguages`

## Describe the bug

(`Dataset.flatten`)[https://github.com/huggingface/datasets/blob/master/src/datasets/arrow_dataset.py#L1265] fails for columns with feature (`Translation`)[https://github.com/huggingface/datasets/blob/3edbeb0ec6519b79f1119adc251a1a6b379a2c12/src/datasets/features/translation.py#L8]

## Steps to reproduce the bug

```python
from datasets import load_dataset

dataset = load_dataset("europa_ecdc_tm", "en2fr", split="train[:10]")
print(dataset.features)
# {'translation': Translation(languages=['en', 'fr'], id=None)}
print(dataset[0])
# {'translation': {'en': 'Vaccination against hepatitis C is not yet available.', 'fr': 'Aucune vaccination contre l’hépatite C n’est encore disponible.'}}

dataset.flatten()
```

## Expected results

`dataset.flatten` should flatten the `Translation` column as if it were a dict of `Value("string")`

```python
dataset[0]
# {'translation.en':  'Vaccination against hepatitis C is not yet available.', 'translation.fr': 'Aucune vaccination contre l’hépatite C n’est encore disponible.' }
dataset.features
# {'translation.en': Value("string"), 'translation.fr': Value("string")}
```

## Actual results

```python
In [31]: dset.flatten()
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-31-bb88eb5276ee> in <module>
----> 1 dset.flatten()

[...]\site-packages\datasets\fingerprint.py in wrapper(*args, **kwargs)
    411             # Call actual function
    412
--> 413             out = func(self, *args, **kwargs)
    414
    415             # Update fingerprint of in-place transforms + update in-place history of transforms

[...]\site-packages\datasets\arrow_dataset.py in flatten(self, new_fingerprint, max_depth)
   1294                 break
   1295         dataset.info.features = self.features.flatten(max_depth=max_depth)
-> 1296         dataset._data = update_metadata_with_features(dataset._data, dataset.features)
   1297         logger.info(f'Flattened dataset from depth {depth} to depth {1 if depth + 1 < max_depth else "unknown"}.')
   1298         dataset._fingerprint = new_fingerprint

[...]\site-packages\datasets\arrow_dataset.py in update_metadata_with_features(table, features)
    534 def update_metadata_with_features(table: Table, features: Features):
    535     """To be used in dataset transforms that modify the features of the dataset, in order to update the features stored in the metadata of its schema."""
--> 536     features = Features({col_name: features[col_name] for col_name in table.column_names})
    537     if table.schema.metadata is None or b"huggingface" not in table.schema.metadata:
    538         pa_metadata = ArrowWriter._build_metadata(DatasetInfo(features=features))

[...]\site-packages\datasets\arrow_dataset.py in <dictcomp>(.0)
    534 def update_metadata_with_features(table: Table, features: Features):
    535     """To be used in dataset transforms that modify the features of the dataset, in order to update the features stored in the metadata of its schema."""
--> 536     features = Features({col_name: features[col_name] for col_name in table.column_names})
    537     if table.schema.metadata is None or b"huggingface" not in table.schema.metadata:
    538         pa_metadata = ArrowWriter._build_metadata(DatasetInfo(features=features))

KeyError: 'translation.en'
```

## Environment info

- `datasets` version: 1.18.3
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.7.10
- PyArrow version: 3.0.0


Add support for `Audio` and `Image` feature in `push_to_hub`

> Cool thanks !
>
> Also cc @patrickvonplaten @anton-l it means that when calling push_to_hub, the audio bytes are embedded in the parquet files (we don't upload the audio files themselves)

Just to verify quickly the size of the dataset doesn't change in this case no? E.g. if a dataset has say 20GB in size when stored in `.mp3` format it could have up to 100GB when stored in WAV. But since we are just taking the bytes here a 20GB .mp3 dataset would also have 20GB when stored in parquet no?

@lhoestq  I've addressed your comments. Additionally, I've modified `cast_storage` to account for possible null (`None`) values.

@patrickvonplaten Yes, the dataset size stays the same (at least because Parquet files are compressed).

Feel free to merge if it's all good to you :)

Add support for the `Audio` and the `Image` feature in `push_to_hub`.

The idea is to remove local path information and store file content under "bytes" in the Arrow table before the push.

My initial approach (https://github.com/huggingface/datasets/commit/34c652afeff9686b6b8bf4e703c84d2205d670aa) was to use a map transform similar to [`decode_nested_example`](https://github.com/huggingface/datasets/blob/5e0f6068741464f833ff1802e24ecc2064aaea9f/src/datasets/features/features.py#L1023-L1056) while having decoding turned off, but I wasn't satisfied with the code quality, so I ended up using the `temporary_assignment` decorator to override `cast_storage`, which allows me to directly modify the underlying storage (the final op is similar to `Dataset.cast`) and results in a much simpler code.

Additionally, I added the `allow_cast` flag that can disable this behavior in the situations where it's not needed (e.g. the dataset is already in the correct format for the Hub, etc.)

EDIT:
`allow_cast` renamed to `embed_external_files`


[fix]: iwslt2017 download urls

Hi ! Thanks for the fix ! Do you know where this new URL comes from ?

Also we try to not use Google Drive if possible, since it has download quota limitations. Do you know if the data is available from another host than Google Drive ?

Oh, I found it just by following the link from the [IWSLT2017 homepage](https://wit3.fbk.eu/2017-01). Not sure if it's available from another host.

Ok cool ! I guess it's ok to use this URL for now, and we can see later if we need to change it.

Before merging, could you update the `dataset_infos.json` file by running this command please ?
```
datasets-cli test ./datasets/iwslt2017 --save_infos --all_configs
```

sure thing. lmk if there's anything else i can do to help.

just checking in. is there anything i can do to help on my end to get this merged? (the dummy data tests are failing due an incorrect path, i think)

Thanks ! I also fixed the dummy data :)

To fix the CI, feel free to merge the `master` branch into your PR.

If you have some time, feel free to also take a look at the missing YAML tags at the top of the README.md file of this dataset:
```
E           ValueError: The following issues have been found in the dataset cards:
E           YAML tags:
E           missing 9 required tags: 'annotations_creators', 'language_creators', 'languages', 'licenses', 'multilinguality', 'size_categories', 'source_datasets', 'task_categories', and 'task_ids'
```
you can use the dataset tagging app here: https://huggingface.co/spaces/huggingface/datasets-tagging

Fixes #2076.


added told-br (brazilian hate speech) dataset

Amazing thank you ! Feel free to regenerate the `dataset_infos.json` to account for the feature type change, and then I think we'll be good to merge :)

Hey,

Adding ToLD-Br. Feel free to ask for modifications.

Thanks!!


adding told-br for toxic/abusive hatespeech detection

Sorry for using multiple github accounts, I didn't notice I was using my professional account to commit/push. Please consider this @JAugusto97 account as the correct one.

Will remake the PR with the correct github account.

Hey,

I'm adding our dataset from our paper published at AACL 2020. Feel free to ask for modifications.

Thanks!


Fix TestCommand to move dataset_infos instead of copying

All the datasets that are loaded normally with `load_dataset`, if `dataset_infos.json` exists, have this file in the importable directory. So it's fine if we copy the file instead of moving it but it's not a big deal.

Any reason to prefer moving it rather than copying it ?

@lvwerra reported than when generating the `dataset_infos.json` for multiple dataset directories containing only JSONL files, subsequent `dataset_infos.json` files contained all previous directories as configs:
- First generate metadata for dataset in dir `dir1`: dataset_infos.json contains one config for `dir1`
- Then generate metadata for dataset in dir `dir2`: dataset_infos.json contains 2 configs, for `dir1` and `dir2`

The reason is that all dataset_infos.json files are first created in the same dir (the one containing the json builder) and then **copied** to the user dir.

Subsequent calls of TestCommand don't replace the dataset_infos.json already present in the dir of the json builder, but append to it.

MAYBE: we should just move for this use case, and copy for the other use cases? See this use case here:
- #3680

@lhoestq aren't you mentioning the case in the else clause?
```python
else:  # in case of a remote dataset
    dataset_dir = None
```

In that case `dataset_infos.json` is not copied: `dataset_dir = None`

When using the JSON loader, calling `get_imported_module_dir()` returns a path inside the pip installed packages, so we shouldn't write files in it anyway, and the dataset_infos.json file should be written directly in the user's directory instead (some users don't have write access to the pip installed packages for example).

Maybe the packaged modules like `json` should override `_save_infos` to save them in the user's directory instead of next to the builder's script. What do you think ?

Anyway as a hotfix we can just add an exception for the `json` builder for now, if the issue has to be fixed soon

Why do we copy instead of moving the file?

CC: @lhoestq @lvwerra


Fix TestCommand to copy dataset_infos to local dir with only data files

Currently this case is missed.

CC: @lvwerra


Download datasets from a private hub

enhancement

New feature or request

Hi ! For information one can set the environment variable `HF_ENDPOINT` (default is `https://huggingface.co`) if they want to use a private hub.

We may need to coordinate with the other libraries to have a consistent way of changing the hub endpoint

Yes, I tested it successfully this morning. Thanks.

In the context of a private hub deployment, customers would like to use load_dataset() to load datasets from their hub, not from the public hub. This doesn't seem to be configurable at the moment and it would be nice to add this feature.

The obvious workaround is to clone the repo first and then load it from local storage, but this adds an extra step. It'd be great to have the same experience regardless of where the hub is hosted.

The same issue exists with the transformers library and the CLI. I'm going to create issues there as well, and I'll reference them below.


Add code example in wikipedia card

Close #3292.


Discovery cannot be streamed anymore

bug

Something isn't working

Seems like a regression from https://github.com/huggingface/datasets/pull/2843

Or maybe it's an issue with the hosting. I don't think so, though, because https://www.dropbox.com/s/aox84z90nyyuikz/discovery.zip seems to work as expected

Hi @severo, thanks for reporting.

Some servers do not support HTTP range requests, and those are required to stream some file formats (like ZIP in this case).

Let me try to propose a workaround.

## Describe the bug
A clear and concise description of what the bug is.

## Steps to reproduce the bug
```python
from datasets import load_dataset
iterable_dataset = load_dataset("discovery", name="discovery", split="train", streaming=True)
list(iterable_dataset.take(1))
```

## Expected results

The first row of the train split.

## Actual results

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 365, in __iter__
    for key, example in self._iter():
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 362, in _iter
    yield from ex_iterable
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 272, in __iter__
    yield from islice(self.ex_iterable, self.n)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 79, in __iter__
    yield from self.generate_examples_fn(**self.kwargs)
  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/discovery/542fab7a9ddc1d9726160355f7baa06a1ccc44c40bc8e12c09e9bc743aca43a2/discovery.py", line 333, in _generate_examples
    with open(data_file, encoding="utf8") as f:
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/streaming.py", line 64, in wrapper
    return function(*args, use_auth_token=use_auth_token, **kwargs)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/utils/streaming_download_manager.py", line 369, in xopen
    file_obj = fsspec.open(file, mode=mode, *args, **kwargs).open()
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/core.py", line 456, in open
    return open_files(
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/core.py", line 288, in open_files
    fs, fs_token, paths = get_fs_token_paths(
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/core.py", line 611, in get_fs_token_paths
    fs = filesystem(protocol, **inkwargs)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/registry.py", line 253, in filesystem
    return cls(**storage_options)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/spec.py", line 68, in __call__
    obj = super().__call__(*args, **kwargs)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/implementations/zip.py", line 57, in __init__
    self.zip = zipfile.ZipFile(self.fo)
  File "/home/slesage/.pyenv/versions/3.9.6/lib/python3.9/zipfile.py", line 1257, in __init__
    self._RealGetContents()
  File "/home/slesage/.pyenv/versions/3.9.6/lib/python3.9/zipfile.py", line 1320, in _RealGetContents
    endrec = _EndRecData(fp)
  File "/home/slesage/.pyenv/versions/3.9.6/lib/python3.9/zipfile.py", line 263, in _EndRecData
    fpin.seek(0, 2)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/implementations/http.py", line 676, in seek
    raise ValueError("Cannot seek streaming HTTP file")
ValueError: Cannot seek streaming HTTP file
```

## Environment info

- `datasets` version: 1.18.3
- Platform: Linux-5.11.0-1027-aws-x86_64-with-glibc2.31
- Python version: 3.9.6
- PyArrow version: 6.0.1


`None` replaced by `[]` after first batch in map

It looks like this is because of this behavior in pyarrow:
```python
import pyarrow as pa

arr = pa.array([None, [0]])
reconstructed_arr = pa.ListArray.from_arrays(arr.offsets, arr.values)
print(reconstructed_arr.to_pylist())
# [[], [0]]
```

It seems that `arr.offsets` can reconstruct the array properly, but an offsets array with null values can:
```python
fixed_offsets = pa.array([None, 0, 1])
fixed_arr = pa.ListArray.from_arrays(fixed_offsets, arr.values)
print(fixed_arr.to_pylist())
# [None, [0]]

print(arr.offsets.to_pylist())
# [0, 0, 1]
print(fixed_offsets.to_pylist())
# [None, 0, 1]
```
EDIT: this is because `arr.offsets` is not enough to reconstruct the array, we also need the validity bitmap

The offsets don't have nulls because they don't include the validity bitmap from `arr.buffers()[0]`, which is used to say which values are null and which values are non-null.

Though the validity bitmap also seems to be wrong:
```python
bin(int(arr.buffers()[0].hex(), 16))
# '0b10'
# it should be 0b110 - 1 corresponds to non-null and 0 corresponds to null, if you take the bits in reverse order
```

So apparently I can't even create the fixed offsets array using this.

If I understand correctly it's always missing the 1 on the left, so I can add it manually as a hack to fix the issue until this is fixed in pyarrow EDIT: actually it may be more complicated than that

EDIT2: actuall it's right, it corresponds to the validity bitmap of the array of logical length 2. So if we use the offsets array, the values array, and this validity bitmap it should be possible to reconstruct the array properly

I created an issue on Apache Arrow's JIRA: https://issues.apache.org/jira/browse/ARROW-15837

And another one: https://issues.apache.org/jira/browse/ARROW-15839

FYI the behavior is the same with:
- `datasets` version: 1.18.3
- Platform: Linux-5.8.0-50-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.7.11
- PyArrow version: 6.0.1

but not with:
- `datasets` version: 1.8.0
- Platform: Linux-4.18.0-305.40.2.el8_4.x86_64-x86_64-with-redhat-8.4-Ootpa
- Python version: 3.7.11
- PyArrow version: 3.0.0

i.e. it outputs:
```py
0  [None, [0]]
1  [None, [0]]
2  [None, [0]]
3  [None, [0]]
```

Thanks for the insights @PaulLerner !

I found a way to workaround this issue for the code example presented in this issue.

Note that empty lists will still appear when you explicitly `cast` a list of lists that contain None values like [None, [0]] to a new feature type (e.g. to change the integer precision). In this case it will show a warning that it happened. If you don't cast anything, then the None values will be kept as expected.

Let me know what you think !

Hi! I feel like I’m missing something in your answer, *what* is the workaround? Is it fixed in some `datasets` version?

`pa.ListArray.from_arrays` returns empty lists instead of None values. The workaround I added inside `datasets` simply consists in not using `pa.ListArray.from_arrays` :)

Once this PR [here ](https://github.com/huggingface/datasets/pull/4282)is merged, we'll release a new version of `datasets` that currectly returns the None values in the case described in this issue

Sometimes `None` can be replaced by `[]` when running map:

```python
from datasets import Dataset

ds = Dataset.from_dict({"a": range(4)})
ds = ds.map(lambda x: {"b": [[None, [0]]]}, batched=True, batch_size=1, remove_columns=["a"])
print(ds.to_pandas())
#              b
# 0  [None, [0]]
# 1    [[], [0]]
# 2    [[], [0]]
# 3    [[], [0]]
```

This issue has been experienced when running the `run_qa.py` example from `transformers` (see issue https://github.com/huggingface/transformers/issues/15401)

This can be due to a bug in when casting `None` in nested lists. Casting only happens after the first batch, since the first batch is used to infer the feature types.

cc @sgugger


Add CodeContests dataset

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** CodeContests
- **Description:** CodeContests is a competitive programming dataset for machine-learning.
- **Paper:**
- **Data:** https://github.com/deepmind/code_contests
- **Motivation:** This dataset was used when training [AlphaCode](https://deepmind.com/blog/article/Competitive-programming-with-AlphaCode).

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Add FrugalScore metric

@lhoestq

The model used by default (`moussaKam/frugalscore_tiny_bert-base_bert-score`) is a tiny model.

I still want to make one modification before merging.
I would like to load the model checkpoint once. Do you think it's a good idea if I load it in `_download_and_prepare`? In this case should the model name be the `self.config_name` or another variable say `self.model_name` ?

OK, I added a commit that loads the checkpoint in `_download_and_prepare`. Please let me know if it looks good.

I triggered the CI and it's failing, can you merge the `master` branch into yours ? It should fix the issues.

Also the doctest apparently raises an error because it outputs `{'scores': [0.6307542, 0.6449357]}` instead of `{'scores': [0.631, 0.645]}` - feel free to edit the code example in the docstring to round the scores, that should fix it

This pull request add FrugalScore metric for NLG systems evaluation.

FrugalScore is a reference-based metric for NLG models evaluation. It is based on a distillation approach that allows to learn a fixed, low cost version of any expensive NLG metric, while retaining most of its original performance.

Paper: https://arxiv.org/abs/2110.08559?context=cs
Github: https://github.com/moussaKam/FrugalScore

@lhoestq


`load_dataset("snli")` is different from dataset viewer

bug

Something isn't working

Yes, we decided to replace the encoded label with the corresponding label when possible in the dataset viewer. But
1. maybe it's the wrong default
2. we could find a way to show both (with a switch, or showing both ie. `0 (neutral)`).

Hi @severo,

Thanks for clarifying.

I think this default is a bit counterintuitive for the user. However, this is a personal opinion that might not be general. I think it is nice to have the actual (non-encoded) labels in the viewer. On the other hand, it would be nice to match what the user sees with what they get when they download a dataset. I don't know - I can see the difficulty of choosing a default :)
Maybe having non-encoded labels as a default can be useful?

Anyway, I think the issue has been addressed. Thanks a lot for your super-quick answer!

Thanks for the 👍  in https://github.com/huggingface/datasets/issues/3673#issuecomment-1029008349 @mariosasko @gary149 @pietrolesci, but as I proposed various solutions, it's not clear to me which you prefer. Could you write your preferences as a comment?

_(note for myself: one idea per comment in the future)_

As I am working with seq2seq, I prefer having the label in string form rather than numeric. So the viewer is fine and the underlying dataset should be "decoded" (from int to str). In this way, the user does not have to search for a mapping `int -> original name` (even though is trivial to find, I reckon). Also, encoding labels is rather easy.

I hope this is useful

I like the idea of "0 (neutral)". The label name can even be greyed to make it clear that it's not part of the actual item in the dataset, it's just the meaning.

I like @lhoestq's idea of having grayed-out labels.

Proposals by @gary149. Which one do you prefer? Please vote with the thumbs

- 👍

    ![image](https://user-images.githubusercontent.com/1676121/152387949-883c7d7e-a9f3-48aa-bff9-11a691555e6e.png)

- 👎

    ![image (1)](https://user-images.githubusercontent.com/1676121/152388061-32d95e42-cade-4ae4-9a77-7365e7b72b8f.png)

I like Option 1 better as it shows clearly what the user is downloading

It's [live](https://huggingface.co/datasets/glue/viewer/cola/train):

<img width="1126" alt="Capture d’écran 2022-02-14 à 10 26 03" src="https://user-images.githubusercontent.com/1676121/153836716-25f6205b-96af-42d8-880a-7c09cb24c420.png">

Thanks all for the help to improve the UI!

## Describe the bug
The dataset that is downloaded from the Hub via `load_dataset("snli")` is different from what is available in the dataset viewer. In the viewer the labels are not encoded (i.e., "neutral", "entailment", "contradiction"), while the downloaded dataset shows the encoded labels (i.e., 0, 1, 2).

Is this expected?

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform: Ubuntu 20.4
- Python version: 3.7


Prioritize `module.builder_kwargs` over defaults in `TestCommand`

This fixes a bug in the `TestCommand` where multiple kwargs for `name` were passed if it was set in both default and `module.builder_kwargs`. Example error:

```Python
Traceback (most recent call last):
  File "create_metadata.py", line 96, in <module>
    main(**vars(args))
  File "create_metadata.py", line 86, in main
    metadata_command.run()
  File "/opt/conda/lib/python3.7/site-packages/datasets/commands/test.py", line 144, in run
    for j, builder in enumerate(get_builders()):
  File "/opt/conda/lib/python3.7/site-packages/datasets/commands/test.py", line 141, in get_builders
    name=name, cache_dir=self._cache_dir, data_dir=self._data_dir, **module.builder_kwargs
TypeError: type object got multiple values for keyword argument 'name'
```

Let me know what you think.


Give an estimate of the dataset size in DatasetInfo

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**

Currently, only part of the datasets provide `dataset_size`, `download_size`, `size_in_bytes` (and `num_bytes` and `num_examples` inside `splits`). I would want to get this information, or an estimation, for all the datasets.

**Describe the solution you'd like**

- get access to the git information for the dataset files hosted on the hub
- look at the [`Content-Length`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Length) for the files served by HTTP


feat: 🎸 generate info if dataset_infos.json does not exist

It's a first attempt at solving https://github.com/huggingface/datasets/issues/3013.

I only kept these ones:
```
    path: str,
    data_files: Optional[Union[Dict, List, str]] = None,
    download_config: Optional[DownloadConfig] = None,
    download_mode: Optional[GenerateMode] = None,
    revision: Optional[Union[str, Version]] = None,
    use_auth_token: Optional[Union[bool, str]] = None,
    **config_kwargs,
```

Let me know if it's better for you now !

(note that there's no breaking change since the ones that are removed can be passed as config_kwargs if you really want)

(https://github.com/huggingface/datasets/pull/3670/commits/5636911880ea4306c27c7f5825fa3f9427ccc2b6 and https://github.com/huggingface/datasets/pull/3670/commits/07c3f0800dd34dfebb9674ad46c67a907b08ded8 -> I has forgotten to update black in my venv)

in get_dataset_infos(). Also: add the `use_auth_token` parameter, and create get_dataset_config_info()

✅ Closes: #3013


Common voice validated partition

Hi @patrickvonplaten - could you please advise whether this would be a welcomed change, and if so, who I consult regarding the unit-tests?

I'd be happy with adding this change. @anton-l @lhoestq - what do you think?

Cool ! I just fixed the tests by adding a dummy `validated.tsv` file in the dummy data archive of common_voice

I wonder if you should separate the train/valid/test configuration from the validated/invalidated configuration of the splits ?
In particular having `validated` along with the train/valid/test splits could be a bit weird since it comprises them. We can do that if you think it makes more sense. Otherwise it's also good as it is right now :)

Thanks! I think that there are 2 cases for using the validated partition: 1) trainset = {validated - dev - test}, dev and test as they come; 2) train, dev, and test sampled from validated manually with the desired ratios.
In either case, I think that it's quite a big change on the HF interface part, so could as well be taken care of in the client code. Or is it not? (In which case, what's the most compact way to implement this?)

What's important IMO is to let the users as much flexibility as they need - so we try to not do too much regarding splits to not constrain users. So I guess the way it is right now is ok. Can you confirm that it's ok @patrickvonplaten and that it won't break some speech training script out there ?

@lhoestq all split names are explicit in our example scripts, so this shouldn't break anything, feel free to merge :)
I'll go ahead and add this to the official `mozilla-foundation` datasets as well

Good for me! This has no real down-sides IMO and surely won't break any training scripts.

This patch adds access to the 'validated' partitions of CommonVoice datasets (provided by the dataset creators but not available in the HuggingFace interface yet).
As 'validated' contains significantly more data than 'train' (although it contains both test and validation, so one needs to be careful there), it can be useful to train better models where no strict comparison with the previous work is intended.


Couldn't cast array of type string error with cast_column

bug

Something isn't working

Hi ! I wasn't able to reproduce the error, are you still experiencing this ? I tried calling `cast_column` on a string column containing paths.

If you manage to share a reproducible code example that would be perfect

Hi,

I think my team mate got this solved. Clolsing it for now and will reopen if I experience this again.
Thanks :)

Hi @R4ZZ3,

If it is not too much of a bother, can you please help me how to resolve this error? I am exactly getting the same error where I am going as per the documentation guideline:

`my_audio_dataset = my_audio_dataset.cast_column("audio_paths", Audio())`

where `"audio_paths"` is a dataset column (feature) having strings of absolute paths to mp3 files of the dataset.

## Describe the bug

In OVH cloud during Huggingface Robust-speech-recognition event on a AI training notebook instance using jupyter lab and running jupyter notebook When using the dataset.cast_column("audio",Audio(sampling_rate=16_000))
method I get error
![image](https://user-images.githubusercontent.com/25264037/152214027-9c42a71a-dd24-463c-a346-57e0287e5a8f.png)

This was working with datasets version 1.17.1.dev0
but now with version 1.18.3 produces the error above.

## Steps to reproduce the bug

load dataset:
![image](https://user-images.githubusercontent.com/25264037/152216145-159553b6-cddc-4f0b-8607-7e76b600e22a.png)

remove columns:
![image](https://user-images.githubusercontent.com/25264037/152214707-7c7e89d1-87d8-4b4f-8cfc-5d7223d35644.png)

run my fix_path function.
This also creates the audio column that is referring to the absolute file path of the audio

![image](https://user-images.githubusercontent.com/25264037/152214773-51f71ccf-d31b-4449-b63a-1af56436e49f.png)

Then I concatenate few other datasets and finally try the cast_column method
![image](https://user-images.githubusercontent.com/25264037/152215032-f341ec86-9d6d-48c9-943b-e2efe37a4d98.png)

but get error:
![image](https://user-images.githubusercontent.com/25264037/152215073-b85bd057-98e8-413c-9b05-51e9805f2c24.png)

## Expected results
A clear and concise description of the expected results.

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform:
OVH Cloud, AI Training section, container for Huggingface Robust Speech Recognition event image(baaastijn/ovh_huggingface)
![image](https://user-images.githubusercontent.com/25264037/152215161-b4ff7bfb-2736-4afb-9223-761a3338d23c.png)

- Python version: 3.8.8
- PyArrow version:
![image](https://user-images.githubusercontent.com/25264037/152215936-4d365760-557e-456b-b5eb-ad1d15cf5073.png)


Process .opus files with torchaudio

> Note that torchaudio is maybe less practical to use for TF or JAX users. This is not in the scope of this PR, but in the future if we manage to find a way to let the user control the decoding it would be nice

@lhoestq so maybe don't do this PR? :) if it doesn't work anyway with an opened file, only with path

Yes as discussed offline there seems to be issues with torchaudio on opened files. Feel free to close this PR if it's better to stick with soundfile because of that

We should be able to remove torchaudio, which has torch as a hard dependency, soon and use only soundfile for decoding: https://github.com/bastibe/python-soundfile/issues/252#issuecomment-1000246773 (opus + mp3 support is on the way).

@anton-l suggested to proccess .opus files with `torchaudio` instead of `soundfile` as it's faster:
![opus](https://user-images.githubusercontent.com/16348744/152177816-2df6076c-f28b-4aef-a08d-b499b921414d.png)

(moreover, I didn't manage to load .opus files with `soundfile` / `librosa` locally on any my machine anyway for some reason, even with `ffmpeg` installed).

For now my current changes work with locally stored file:
```python
# download sample opus file (from MultilingualSpokenWords dataset)
!wget https://huggingface.co/datasets/polinaeterna/test_opus/resolve/main/common_voice_tt_17737010.opus

from datasets import Dataset, Audio

audio_path = "common_voice_tt_17737010.opus"
dataset = Dataset.from_dict({"audio": [audio_path]}).cast_column("audio", Audio(48000))
dataset[0]
# {'audio': {'path': 'common_voice_tt_17737010.opus',
#   'array': array([ 0.0000000e+00,  0.0000000e+00,  3.0517578e-05, ...,
#          -6.1035156e-05,  6.1035156e-05,  0.0000000e+00], dtype=float32),
#   'sampling_rate': 48000}}
```
But it doesn't work when loading inside s dataset from bytes (I checked on [MultilingualSpokenWords](https://github.com/huggingface/datasets/pull/3666), the PR is a draft now, maybe the bug is somewhere there  )

```python
import torchaudio
with open(audio_path, "rb") as b:
    print(torchaudio.load(b))
# RuntimeError: Error loading audio file: failed to open file <in memory buffer>
```


process .opus files (for Multilingual Spoken Words)

@lhoestq  I still have problems with processing `.opus` files with `soundfile` so I actually cannot fully check that it works but it should... Maybe this should be investigated in case of someone else would also have problems with that.

Also, as the data is in a private repo on the hub (before we come to a decision about audio data privacy), the needed checks cannot be done right now.

@lhoestq I check the data redownloading for configs sharing the same languages, you were right: the data is downloaded once for each language. But samples are generated from scratch each time. Is it a supposed behavior?

> But samples are generated from scratch each time. Is it a supposed behavior?

Yea that's the way it works right now, because we generate one arrow file per configuration. Since changing the languages creates a new configuration, then it generates a new arrow file.

Opus files requires `libsndfile>=1.0.30`. Add check for this version and tests.

**outdated:**
Add [Multillingual Spoken Words dataset](https://mlcommons.org/en/multilingual-spoken-words/)

You can specify multiple languages for downloading 😌:
```python
ds = load_dataset("datasets/ml_spoken_words", languages=["ar", "tt"])
```

1. I didn't take into account that each time you pass a set of languages the data for a specific language is downloaded even if it was downloaded before (since these are custom configs like `ar+tt` and `ar+tt+br`. Maybe that wasn't a good idea?
2. The script will have to be slightly changed after merge of https://github.com/huggingface/datasets/pull/3664
2. Just can't figure out what wrong with dummy files... 😞 Maybe we should get rid of them at some point 😁


Fix MP3 resampling when a dataset's audio files have different sampling rates

The resampler needs to be updated if the `orig_freq` doesn't match the audio file sampling rate

Fix https://github.com/huggingface/datasets/issues/3662


[WIP] Return local paths to Common Voice

Cool thanks for giving it a try @anton-l !

Would be very much in favor of having "real" paths to the audio files again for non-streaming use cases. At the same time it would be nice to make the audio data loading script as understandable as possible so that the community can easily add audio datasets in the future by looking at this one as an example. Think if it's clear for a contributor how to add an audio datasets script that works for the standard non-streaming case while it is easy to extend it afterwards to a streaming dataset script, then this would be perfect

> @anton-l @patrickvonplaten @lhoestq Is it possible somehow to provide this logic inside the library instead of a loading script so that we don't need to completely rewrite all the scripts for audio datasets and users don't have to care about two different loading approaches in the same script? thinking

Not sure @lhoestq - what do you think?

Now that we've corrected the previous resampling bug, think this one here is of high importance. @lhoestq - what do you think how we should proceed here?

> @anton-l @patrickvonplaten @lhoestq Is it possible somehow to provide this logic inside the library instead of a loading script so that we don't need to completely rewrite all the scripts for audio datasets and users don't have to care about two different loading approaches in the same script? 🤔

Yes let's do this :)

Maybe we can change the behavior of `DownloadManager.iter_archive` back to extracting the TAR archive locally, and return an iterable of (local path, file obj). And the `StreamingDownloadManager.iter_archive` can return an iterable of (relative path inside the archive, file obj) ?

In this case, a dataset would need to have something like this:
```python
for path, f in files:
    yield id_, {"audio": {"path": path, "bytes": f.read() if not is_local_file(path) else None}}
```

Alternatively, we can allow this if we consider that `Audio.encode_example` sets the "bytes" field to `None` automatically if `path` is a local path:
```python
for path, f in files:
    yield id_, {"audio": {"path": path, "bytes": f.read()}}
```
Note that in this case the file is read for nothing though (maybe it's not a big deal ?)

Let me know if it sounds good to you and what you'd prefer !

@lhoestq I'm very much in favor of your first aproach! With the full paths returned I think we won't even need to mess with `os.path.join` vs `"/".join()"` and other local/streaming differences 👍

@lhoestq I also like the idea and favor your first approach to avoid an unnecessary read and make yielding faster.

Looks cool - thanks for working on this. I just feel strongly about `path` being an absolute `path` that exist and can be inspected in the non-streaming case :-) For streaming=True IMO it's absolutely fine if we only have access to the bytes

Hi ! I started implementing this but I noticed that returning an absolute path is breaking for many datasets that do things like
```python
for path, f in files:
    if path.startswith(data_dir):
        ...
```
so I think I will have to add a parameter to `iter_archive` like `extract_locally=True` to avoid the breaking change, does that sound good to you ?

This makes me also think that in streaming mode it could also return a local path too, if we think that writing and deleting temporary files on-the-fly while iterating over the streaming dataset is ok.

@lhoestq I think it is a good idea to rollback to extracting the archives locally in non-streaming mode, as far as (as you mentioned) we do not store the bytes in the Arrow file for those cases to avoid "doubling" the disk space usage.

On the other hand, I don't like:
- neither the possibility to avoid extracting locally in non-streaming: the behavior should be consistent; thus we always extract in non-streaming
  - which could be the criterium to decide whether an archive should or should not be extracted? Just because I want to make a condition on path.startswith?
- nor the option to download/delete temporary files in streaming (see discussion here: https://github.com/huggingface/datasets/pull/3689#issuecomment-1032858345)

Unfortunately, in order to fix the datasets that are breaking after the rollback, I would suggest fixing their scripts so that the paths are handled more robustly (considering that they can be absolute or relative).

I agree with Albert, fixing all of the audio datasets isn't too big of a deal (yet). I can help with those if needed :)

Ok cool ! I'm completely rolling it back then

Alright I did the rollback and now you can get local paths :)
Feel free to try it out and let me know if it's good for you

Ok according to the CI there around 60+ datasets to fix

> fixing all of the audio datasets isn't too big of a deal (yet). I can help with those if needed :)

I can help with them too :)

Here is the full list to keep track of things:

- [x] air_dialogue
- [x] id_nergrit_corpus
- [ ] id_newspapers_2018
- [x] imdb
- [ ] indic_glue
- [ ] inquisitive_qg
- [x] klue
- [x] lama
- [x] lex_glue
- [ ] lm1b
- [x] amazon_polarity
- [ ] mac_morpho
- [ ] math_dataset
- [ ] md_gender_bias
- [ ] mdd
- [ ] assin
- [ ] atomic
- [ ] babi_qa
- [ ] mlqa
- [ ] mocha
- [ ] blended_skill_talk
- [ ] capes
- [ ] cbt
- [ ] newsgroup
- [ ] cifar10
- [ ] cifar100
- [ ] norec
- [ ] ohsumed
- [ ] code_x_glue_cc_clone_detection_poj104
- [x] openslr
- [ ] orange_sum
- [ ] paws
- [ ] paws-x
- [ ] cppe-5
- [ ] polyglot_ner
- [ ] dbrd
- [ ] empathetic_dialogues
- [ ] eraser_multi_rc
- [ ] flores
- [ ] flue
- [ ] food101
- [ ] py_ast
- [ ] qasc
- [ ] qasper
- [ ] race
- [ ] reuters21578
- [ ] ropes
- [ ] rotten_tomatoes
- [x] vivos
- [ ] wi_locness
- [ ] wiki_movies
- [ ] wikiann
- [ ] wmt20_mlqe_task1
- [ ] wmt20_mlqe_task2
- [ ] wmt20_mlqe_task3
- [ ] scicite
- [ ] xsum
- [ ] scielo
- [ ] scifact
- [ ] setimes
- [ ] social_bias_frames
- [ ] sogou_news
- [x] speech_commands
- [ ] ted_hrlr
- [ ] ted_multi
- [ ] tlc
- [ ] turku_ner_corpus

I'll do my best to fix as many as possible tomorrow :)

the audio datasets are fixed if I didn't forget anything :)

btw what are we gonna do with the community ones that would be broken after the fix?

Closing in favor of https://github.com/huggingface/datasets/pull/3736

Fixes https://github.com/huggingface/datasets/issues/3663

This is a proposed way of returning the old local file-based generator while keeping the new streaming generator intact.

TODO:
- [ ] brainstorm a bit more on https://github.com/huggingface/datasets/issues/3663 to see if we can do better
- [ ] refactor the heck out of this PR to avoid completely copying the logic between the two generators