Error loading C4 realnewslike dataset

bug

Something isn't working

Hi @danshirron,
`c4` was updated few days back by @lhoestq. The new configs are `['en', 'en.noclean', 'en.realnewslike', 'en.webtextlike'].` You'll need to remove any older version of this dataset you previously downloaded and then run `load_dataset` again with new configuration.

@bhavitvyamalik @lhoestq , just tried the above and got:
>>> a=datasets.load_dataset('c4','en.realnewslike')
Downloading: 3.29kB [00:00, 1.66MB/s]
Downloading: 2.40MB [00:00, 12.6MB/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/dshirron/.local/lib/python3.8/site-packages/datasets/load.py", line 819, in load_dataset
    builder_instance = load_dataset_builder(
  File "/home/dshirron/.local/lib/python3.8/site-packages/datasets/load.py", line 701, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
  File "/home/dshirron/.local/lib/python3.8/site-packages/datasets/builder.py", line 1049, in __init__
    super(GeneratorBasedBuilder, self).__init__(*args, **kwargs)
  File "/home/dshirron/.local/lib/python3.8/site-packages/datasets/builder.py", line 268, in __init__
    self.config, self.config_id = self._create_builder_config(
  File "/home/dshirron/.local/lib/python3.8/site-packages/datasets/builder.py", line 360, in _create_builder_config
    raise ValueError(
ValueError: BuilderConfig en.realnewslike not found. Available: ['en', 'realnewslike', 'en.noblocklist', 'en.noclean']
>>>

datasets version is 1.11.0

I think I had an older version of datasets installed and that's why I commented the old configurations in my last comment, my bad! I re-checked and updated it to latest version (`datasets==1.11.0`) and it's showing `available configs: ['en', 'realnewslike', 'en.noblocklist', 'en.noclean']`.

I tried `raw_datasets = load_dataset('c4', 'realnewslike')` and the download started. Make sure you don't have any old copy of this dataset and you download it fresh using the latest version of datasets. Sorry for the mix up!

It works. I probably had some issue with the cache. after cleaning it im able to download the dataset. Thanks

## Describe the bug
Error loading C4 realnewslike dataset. Validation part mismatch

## Steps to reproduce the bug
```python
 raw_datasets = load_dataset('c4', 'realnewslike', cache_dir=model_args.cache_dir)
## Expected results
success on data loading
## Actual results
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15.3M/15.3M [00:00<00:00, 28.1MB/s]Traceback (most recent call last):
  File "run_mlm_tf.py", line 794, in <module>
    main()
  File "run_mlm_tf.py", line 425, in main
    raw_datasets = load_dataset(data_args.dataset_name, data_args.dataset_config_name, cache_dir=model_args.cache_dir)                                                                                File "/home/dshirron/.local/lib/python3.8/site-packages/datasets/load.py", line 843, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/dshirron/.local/lib/python3.8/site-packages/datasets/builder.py", line 608, in download_and_prepare
    self._download_and_prepare(
  File "/home/dshirron/.local/lib/python3.8/site-packages/datasets/builder.py", line 698, in _download_and_prepare                                                                                verify_splits(self.info.splits, split_dict)                                                                                File "/home/dshirron/.local/lib/python3.8/site-packages/datasets/utils/info_utils.py", line 74, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='validation', num_bytes=38165657946, num_examples=13799838, dataset_name='c4'), 'recorded': SplitInfo(name='validation', num_bytes=37875873, num_examples=13863, dataset_name='c4')}]

## Environment info
- `datasets` version: 1.10.2
- Platform: Linux-5.4.0-58-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 4.0.1


Add Nuswide dataset

dataset request

Requesting to add a new dataset

vision

Vision datasets

## Adding a Dataset
- **Name:** *NUSWIDE*
- **Description:** *[A Real-World Web Image Dataset from National University of Singapore](https://lms.comp.nus.edu.sg/wp-content/uploads/2019/research/nuswide/NUS-WIDE.html)*
- **Paper:** *[here](https://lms.comp.nus.edu.sg/wp-content/uploads/2019/research/nuswide/nuswide-civr2009.pdf)*
- **Data:** *[here](https://github.com/wenting-zhao/nuswide)*
- **Motivation:** *This dataset is a benchmark in the Text Retrieval task.*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


the meteor metric seems not consist with the official version

bug

Something isn't working

metric bug

A bug in a metric script

the issue is caused by the differences between varied meteor versions:
meteor1.0 is for https://aclanthology.org/W07-0734.pdf
meteor1.5 is for https://aclanthology.org/W14-3348.pdf

here is a very similar issue in NLTK
https://github.com/nltk/nltk/issues/2655

Hi @jianguda, thanks for reporting.

Currently, at 🤗 `datasets` we are using METEOR 1.0 (indeed using NLTK: `from nltk.translate import meteor_score`): See the [citation here](https://github.com/huggingface/datasets/blob/master/metrics/meteor/meteor.py#L23-L35).

If there is some open source implementation of METEOR 1.5, that could be an interesting contribution! 😉

## Describe the bug
The computed meteor score seems strange because the value is very different from the scores computed by other tools. For example, I use the meteor score computed by [NLGeval](https://github.com/Maluuba/nlg-eval) as the reference (which reuses the official jar file for the computation)

## Steps to reproduce the bug
```python
from datasets import load_metric
from nlgeval import NLGEval, compute_individual_metrics

meteor = load_metric('meteor')
predictions = ["It is a guide to action which ensures that the military always obeys the commands of the party"]
references = ["It is a guide to action that ensures that the military will forever heed Party commands"]
results = meteor.compute(predictions=predictions, references=references)
# print the actual result
print(round(results["meteor"], 4))
metrics_dict = compute_individual_metrics(references, predictions[0])
# print the expected result
print(round(metrics_dict["METEOR"], 4))
```
By the way, you need to install the `nlg-eval` library first. Please check the installation guide [here](https://github.com/Maluuba/nlg-eval#setup), thanks!

## Expected results
`0.4474`

## Actual results
`0.7398`

## Environment info
- `datasets` version: 1.10.2
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.5
- PyArrow version: 4.0.1


Raise ManualDownloadError when loading a dataset that requires previous manual download

This PR implements the raising of a `ManualDownloadError` when loading a dataset that requires previous manual download, and this is missing.

The `ManualDownloadError` is raised whether the dataset is loaded in normal or streaming mode.

Close #2749.

cc: @severo


Unexpected type after `concatenate_datasets`

bug

Something isn't working

Hi @JulesBelveze, thanks for your question.

Note that 🤗  `datasets` internally store their data in Apache Arrow format.

However, when accessing dataset columns, by default they are returned as native Python objects (lists in this case).

If you would like their columns to be returned in a more suitable format for your use case (torch arrays), you can use the method `set_format()`:
```python
concat_dataset.set_format(type="torch")
```

You have detailed information in our docs:
- [Using a Dataset with PyTorch/Tensorflow](https://huggingface.co/docs/datasets/torch_tensorflow.html)
- [Dataset.set_format()](https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.set_format)

Thanks @albertvillanova it indeed did the job 😃
Thanks for your answer!

## Describe the bug
I am trying to concatenate two `Dataset` using `concatenate_datasets` but it turns out that after concatenation the features are casted from `torch.Tensor` to `list`.
It then leads to a weird tensors when trying to convert it to a `DataLoader`. However, if I use each `Dataset` separately everything behave as expected.

## Steps to reproduce the bug
```python
>>> featurized_teacher
Dataset({
    features: ['t_labels', 't_input_ids', 't_token_type_ids', 't_attention_mask'],
    num_rows: 502
})
>>> for f in featurized_teacher.features:
     print(featurized_teacher[f].shape)
torch.Size([502])
torch.Size([502, 300])
torch.Size([502, 300])
torch.Size([502, 300])

>>> featurized_student
Dataset({
    features: ['s_features', 's_labels'],
    num_rows: 502
})
>>> for f in featurized_student.features:
     print(featurized_student[f].shape)
torch.Size([502, 64])
torch.Size([502])
```
The shapes seem alright to me. Then the results after concatenation are as follow:
```python
>>> concat_dataset = datasets.concatenate_datasets([featurized_student, featurized_teacher], axis=1)
>>> type(concat_dataset["t_labels"])
<class 'list'>
```
One would expect to obtain the same type as the one before concatenation.

Am I doing something wrong here? Any idea on how to fix this unexpected behavior?

## Environment info
- `datasets` version: 1.9.0
- Platform: macOS-10.14.6-x86_64-i386-64bit
- Python version: 3.9.5
- PyArrow version: 3.0.0


Fix metadata JSON for ubuntu_dialogs_corpus dataset

Related to #2743.


Fix metadata JSON for turkish_movie_sentiment dataset

Related to #2743.


Generate metadata JSON for telugu_books dataset

Related to #2743.


Generate metadata JSON for reclor dataset

Related to #2743.


Generate metadata JSON for lm1b dataset

Related to #2743.


Update metadata for wikihow dataset

Update metadata for wikihow dataset:
- Remove leading new line character in description and citation
- Update metadata JSON
- Remove no longer necessary `urls_checksums/checksums.txt` file

Related to #2748.


Second concatenation of datasets produces errors

Something isn't working

Hi @Aktsvigun, thanks for reporting.

I'm investigating this.

Hi @albertvillanova ,
any update on this? Can I probably help in some way?

Hi @Aktsvigun! We are planning to address this issue before our next release, in a couple of weeks at most. 😅

In the meantime, if you would like to contribute, feel free to open a Pull Request. You are welcome. Here you can find more information: [How to contribute to Datasets?](CONTRIBUTING.md)

I can't reproduce the bug on master. I believe this issue was fixed by https://github.com/huggingface/datasets/pull/3551.

Hi,

I am need to concatenate my dataset with others several times, and after I concatenate it for the second time, the features of features (e.g. tags names) are collapsed. This hinders, for instance, the usage of tokenize function with `data.map`.

```
from datasets import load_dataset, concatenate_datasets

data = load_dataset('trec')['train']
concatenated = concatenate_datasets([data, data])
concatenated_2 = concatenate_datasets([concatenated, concatenated])
print('True features of features:', concatenated.features)
print('\nProduced features of features:', concatenated_2.features)
```
outputs

```
True features of features: {'label-coarse': ClassLabel(num_classes=6, names=['DESC', 'ENTY', 'ABBR', 'HUM', 'NUM', 'LOC'], names_file=None, id=None), 'label-fine': ClassLabel(num_classes=47, names=['manner', 'cremat', 'animal', 'exp', 'ind', 'gr', 'title', 'def', 'date', 'reason', 'event', 'state', 'desc', 'count', 'other', 'letter', 'religion', 'food', 'country', 'color', 'termeq', 'city', 'body', 'dismed', 'mount', 'money', 'product', 'period', 'substance', 'sport', 'plant', 'techmeth', 'volsize', 'instru', 'abb', 'speed', 'word', 'lang', 'perc', 'code', 'dist', 'temp', 'symbol', 'ord', 'veh', 'weight', 'currency'], names_file=None, id=None), 'text': Value(dtype='string', id=None)}

Produced features of features: {'label-coarse': Value(dtype='int64', id=None), 'label-fine': Value(dtype='int64', id=None), 'text': Value(dtype='string', id=None)}
```

I am using `datasets` v.1.11.0


Raise a proper exception when trying to stream a dataset that requires to manually download files

Something isn't working

Hi @severo, thanks for reporting.

As discussed, datasets requiring manual download should be:
- programmatically identifiable
- properly handled with more clear error message when trying to load them with streaming

In relation with programmatically identifiability, note that for datasets requiring manual download, their builder have a property `manual_download_instructions` which is not None:
```python
# Dataset requiring manual download:
builder.manual_download_instructions is not None
```

## Describe the bug

At least for 'reclor', 'telugu_books', 'turkish_movie_sentiment', 'ubuntu_dialogs_corpus', 'wikihow', trying to `load_dataset` in streaming mode raises a `TypeError` without any detail about why it fails.

## Steps to reproduce the bug

```python
from datasets import load_dataset
dataset = load_dataset("reclor", streaming=True)
```

## Expected results

Ideally: raise a specific exception, something like `ManualDownloadError`.

Or at least give the reason in the message, as when we load in normal mode:

```python
from datasets import load_dataset
dataset = load_dataset("reclor")
```

```
AssertionError: The dataset reclor with config default requires manual data.
 Please follow the manual download instructions:   to use ReClor you need to download it manually. Please go to its homepage (http://whyu.me/reclor/) fill the google
  form and you will receive a download link and a password to extract it.Please extract all files in one folder and use the path folder in datasets.load_dataset('reclor', data_dir='path/to/folder/folder_name')
  .
 Manual data can be loaded with `datasets.load_dataset(reclor, data_dir='<path/to/manual/data>')
```

## Actual results

```
TypeError: expected str, bytes or os.PathLike object, not NoneType
```

## Environment info

- `datasets` version: 1.11.0
- Platform: macOS-11.5-x86_64-i386-64bit
- Python version: 3.8.11
- PyArrow version: 4.0.1


Generate metadata JSON for wikihow dataset

Related to #2743.


add multi-proc in `to_json`

Thank you for working on this, @bhavitvyamalik

10% is not solving the issue, we want 5-10x faster on a machine that has lots of resources, but limited processing time.

So let's benchmark it on an instance with many more cores, I can test with 12 on my dev box and 40 on JZ.

Could you please share the test I could run with both versions?

Should we also test the sharded version I shared in https://github.com/huggingface/datasets/issues/2663#issue-946552273 so optionally 3 versions to test.

Since I was facing `OSError: [Errno 12] Cannot allocate memory` in CircleCI tests, I've added `num_proc` option instead of always using full `cpu_count`. You can test both v1 and v2 through this branch (some redundancy needs to be removed).

Update: I was able to convert into json which took 50% less time as compared to v1 on `ascent_kb` dataset. Will post the benchmarking script with results here.

Here are the benchmarks with the current branch for both v1 and v2 (dataset: `ascent_kb`, 8.9M samples):
| batch_size | time (in sec) | time (in sec) |
|------------|---------------|---------------|
|            | num_proc = 1  | num_proc = 4  |
| 10k        | 185.56        | 170.11        |
| 50k        | 175.79        | 86.84         |
| **100k**       | 191.09        | **78.35**         |
| 125k       | 198.28        | 90.89         |

Increasing the batch size on my machine helped in making v2 around 50% faster as compared to v1. Timings may vary depending on the machine. I'm including the benchmarking script as well. CircleCI errors are unrelated (something related to `bertscore`)
```
import time
from datasets import load_dataset
import pathlib
import os
from pathlib import Path
import shutil
import gc

batch_sizes = [10_000, 50_000, 100_000, 125_000]
num_procs = [1, 4]  # change this according to your machine

SAVE_LOC = "./new_dataset.json"

for batch in batch_sizes:
    for num in num_procs:
        dataset = load_dataset("ascent_kb")

        local_start = time.time()
        ans = dataset['train'].to_json(SAVE_LOC, batch_size=batch, num_proc=num)
        local_end = time.time() - local_start

        print(f"Time taken on {num} num_proc and {batch} batch_size: ", local_end)

        # remove that dataset and its contents from cache and newly generated json
        new_json = pathlib.Path(SAVE_LOC)
        new_json.unlink()

        try:
            shutil.rmtree(os.path.join(str(Path.home()), ".cache", "huggingface"))
        except OSError as e:
            print("Error: %s - %s." % (e.filename, e.strerror))

        gc.collect()
```
This will download the dataset in every iteration and run `to_json`. I didn't do multiple iterations here for `to_json` (for a specific batch_size and num_proc) and took average time as I found that v1 got faster after 1st iteration (maybe it's caching somewhere). Since you'll be doing this operation only once, I thought it'll be better to report how both v1 and v2 performed in single iteration only.

Important: Benchmarking script will delete the newly generated json and `~/.cache/huggingface/` after every iteration so that it doesn't end up using any cached data (just to be on a safe side)

Thank you for sharing the benchmark, @bhavitvyamalik. Your results look promising.

But if I remember correctly the sharded version at https://github.com/huggingface/datasets/issues/2663#issue-946552273 was much faster. So we probably should compare to it as well? And if it's faster than at least document that manual sharding version?

-------

That's a dangerous benchmark as it'd wipe out many other HF things. Why not wipe out:
```
~/.cache/huggingface/datasets/ascent_kb/
```

Running the benchmark now.

Weird, I tried to adapt your benchmark to using shards and the program no longer works. It instead quickly uses up all available RAM and hangs. Has something changed recently in `datasets`? You can try:

```
import time
from datasets import load_dataset
import pathlib
import os
from pathlib import Path
import shutil
import gc
from multiprocessing import cpu_count, Process, Queue

batch_sizes = [10_000, 50_000, 100_000, 125_000]
num_procs = [1, 8]  # change this according to your machine

DATASET_NAME = ("ascent_kb")
num_shards = [1, 8]
for batch in batch_sizes:
    for shards in num_shards:
        dataset = load_dataset(DATASET_NAME)["train"]
        #print(dataset)

        def process_shard(idx):
            print(f"Sharding {idx}")
            ds_shard = dataset.shard(shards, idx, contiguous=True)
            # ds_shard = ds_shard.shuffle() # remove contiguous=True above if shuffling
            print(f"Saving {DATASET_NAME}-{idx}.jsonl")
            ds_shard.to_json(f"{DATASET_NAME}-{idx}.jsonl", orient="records", lines=True, force_ascii=False)

        local_start = time.time()
        queue = Queue()
        processes = [Process(target=process_shard, args=(idx,)) for idx in range(shards)]
        for p in processes:
            p.start()

        for p in processes:
            p.join()
        local_end = time.time() - local_start

        print(f"Time taken on {shards} shards and {batch} batch_size: ", local_end)
```

Just careful, so that it won't crash your compute environment. As it almost crashed mine.

So this part seems to no longer work:
```
        dataset = load_dataset("ascent_kb")["train"]
        ds_shard = dataset.shard(1, 0, contiguous=True)
        ds_shard.to_json("ascent_kb-0.jsonl", orient="records", lines=True, force_ascii=False)
```

If you are using `to_json` without any `num_proc`or `num_proc=1` then essentially it'll fall back to v1 only and I've kept it as it is (the tests were passing as well)

> That's a dangerous benchmark as it'd wipe out many other HF things. Why not wipe out:

That's because some dataset related files were still left inside `~/.cache/huggingface/datasets` folder. You can wipe off datasets folder inside your cache maybe

> dataset = load_dataset("ascent_kb")["train"]
> ds_shard = dataset.shard(1, 0, contiguous=True)
> ds_shard.to_json("ascent_kb-0.jsonl", orient="records", lines=True, force_ascii=False)

I tried this `lama` dataset (1.3M) and it worked fine. Trying it with `ascent_kb` currently, will update it here.

I don't think the issue has anything to do with your work, @bhavitvyamalik. I forgot to mention I tested to see the same problem with the latest datasets release.

Interesting, I tried your suggestion. This:
```
python -c 'import datasets; ds="lama"; dataset = datasets.load_dataset(ds)["train"]; \
dataset.shard(1, 0, contiguous=True).to_json(f"{ds}-0.jsonl", orient="records", lines=True, force_ascii=False)'
```
works fine and takes just a few GBs to complete.

this on the other hand blows up memory-wise:
```
python -c 'import datasets; ds="ascent_kb"; dataset = datasets.load_dataset(ds)["train"]; \
dataset.shard(1, 0, contiguous=True).to_json(f"{ds}-0.jsonl", orient="records", lines=True, force_ascii=False)'
```
and I have to kill it before it uses up all RAM. (I have 128GB of it, so it should be more than enough)

> That's because some dataset related files were still left inside ~/.cache/huggingface/datasets folder. You can wipe off datasets folder inside your cache maybe

I think recent datasets added a method that will print out the path for all the different components for a given dataset, I can't recall the name though. It was when we were discussing a janitor program to clear up space selectively.

> and I have to kill it before it uses up all RAM. (I have 128GB of it, so it should be more than enough)

Same thing just happened on my machine too. Memory leak somewhere maybe? Even if you were to load this dataset in your memory it shouldn't take more than 4GB. You were earlier doing this for `oscar` dataset. Is it working fine for that?

Hmm, looks like `datasets` has changed and won't accept my currently cached oscar-en (crashes), so I'd rather not download 0.5TB again.

Were you able to reproduce the memory blow up with `ascent_kb`? It's should be a much quicker task to verify.

But yes, oscar worked just fine with `.shard()` which is what I used to process it fast.

What I tried is:
```
HF_DATASETS_OFFLINE=1 HF_DATASETS_CACHE=cache python -c 'import datasets; ds="oscar"; \
dataset = datasets.load_dataset(ds, "unshuffled_deduplicated_en")["train"]; \
dataset.shard(1000000, 0, contiguous=True).to_json(f"{ds}-0.jsonl", orient="records", lines=True, force_ascii=False)'
```
and got:
```
Using the latest cached version of the module from /gpfswork/rech/six/commun/modules/datasets_modules/datasets/oscar/e4f06cecc7ae02f7adf85640b4019bf476d44453f251a1d84aebae28b0f8d51d (last modified on Fri Aug  6 01:52:35 2021) since it couldn't be found locally at oscar/oscar.py or remotely (OfflineModeIsEnabled).
Reusing dataset oscar (cache/oscar/unshuffled_deduplicated_en/1.0.0/e4f06cecc7ae02f7adf85640b4019bf476d44453f251a1d84aebae28b0f8d51d)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/load.py", line 755, in load_dataset
    ds = builder_instance.as_dataset(split=split, ignore_verifications=ignore_verifications, in_memory=keep_in_memory)
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/builder.py", line 737, in as_dataset
    datasets = utils.map_nested(
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 203, in map_nested
    mapped = [
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 204, in <listcomp>
    _single_map_nested((function, obj, types, None, True)) for obj in tqdm(iterable, disable=disable_tqdm)
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 142, in _single_map_nested
    return function(data_struct)
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/builder.py", line 764, in _build_single_dataset
    ds = self._as_dataset(
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/builder.py", line 834, in _as_dataset
    dataset_kwargs = ArrowReader(self._cache_dir, self.info).read(
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/arrow_reader.py", line 217, in read
    return self.read_files(files=files, original_instructions=instructions, in_memory=in_memory)
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/arrow_reader.py", line 238, in read_files
    pa_table = self._read_files(files, in_memory=in_memory)
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/arrow_reader.py", line 173, in _read_files
    pa_table: Table = self._get_table_from_filename(f_dict, in_memory=in_memory)
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/arrow_reader.py", line 308, in _get_table_from_filename
    table = ArrowReader.read_table(filename, in_memory=in_memory)
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/arrow_reader.py", line 327, in read_table
    return table_cls.from_file(filename)
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/table.py", line 450, in from_file
    table = _memory_mapped_arrow_table_from_file(filename)
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/table.py", line 43, in _memory_mapped_arrow_table_from_file
    memory_mapped_stream = pa.memory_map(filename)
  File "pyarrow/io.pxi", line 782, in pyarrow.lib.memory_map
  File "pyarrow/io.pxi", line 743, in pyarrow.lib.MemoryMappedFile._open
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
OSError: Memory mapping file failed: Cannot allocate memory
```

> Were you able to reproduce the memory blow up with ascent_kb? It's should be a much quicker task to verify.

Yes, this blows up memory-wise on my machine too.

I found that a [similar error](https://discuss.huggingface.co/t/saving-memory-with-run-mlm-py-with-wikipedia-datasets/4160) was posted on the forum on 5th March. Since you already knew how much time [#2663 comment](https://github.com/huggingface/datasets/issues/2663#issue-946552273) took, can you try benchmarking v1 and v2 for now maybe until we have a fix for this memory blow up?

OK, so I benchmarked using "lama" though it's too small for this kind of test, since the sharding is much slower than one thread here.

Results: https://gist.github.com/stas00/dc1597a1e245c5915cfeefa0eee6902c

So sharding does really bad there, and your json over procs is doing great!

Any suggestions to a somewhat bigger dataset, but not too big? say 10 times of lama?

Looks great! I had a few questions/suggestions related to `benchmark-datasets-to_json.py`:

1. You have used only 10_000 and 100_000 batch size. Including more batch sizes may help you find the perfect batch size for your machine and even give you some extra speed-up.
For eg, I found `load_dataset("cc100", lang="eu")` with batch size 125_000 took less time as compared to batch size 100_000 (71.16 sec v/s 67.26 sec) since this dataset has 2 fields only `['id', 'text']`, so that's why we can go for higher batch size here.

2. Why have you used `num_procs` 1 and 4 only?

You can use:
1. `dataset = load_dataset("cc100", lang="af")`. Even though it has only 2 fields but there are around 9.9 mil samples. (lama had around 1.3 mil samples)
2. `dataset = load_dataset("cc100", lang="eu")` -> 16 mil samples. (if you want something more than 9.9 mil)
3. `dataset = load_dataset("neural_code_search", 'search_corpus')` -> 4.7 mil samples

Thank you, @bhavitvyamalik

My apologies, at the moment I have not found time to do more benchmark with the proposed other datasets. I will try to do it later, but I don't want it to hold your PR, it's definitely a great improvement based on the benchmarks I did run! And the comparison to sharded is really just of interest to me to see if it's on par or slower.

So if other reviewers are happy, this definitely looks like a great improvement to me and addresses the request I made in the first place.

> Why have you used num_procs 1 and 4 only?

Oh, no particular reason, I was just comparing to 4 shards on my desktop. Typically it's sufficient to go from 1 to 2-4 to see whether the distributed approach is faster or not. Once hit larger numbers you often run into bottlenecks like IO, and then numbers can be less representative. I hope it makes sense.

Tested it with a larger dataset (`srwac`) and memory utilisation remained constant with no swap memory used. @lhoestq should I also add test for the same? Last time I tried this, I got `OSError: [Errno 12] Cannot allocate memory` in CircleCI tests

Closes #2663. I've tried adding multiprocessing in `to_json`. Here's some benchmarking I did to compare the timings of current version (say v1) and multi-proc version (say v2). I did this with `cpu_count` 4 (2015 Macbook Air)

1. Dataset name: `ascent_kb` - 8.9M samples (all samples were used, reporting this for a single run)
v1- ~225 seconds for converting whole dataset to json
v2- ~200 seconds for converting whole dataset to json

2. Dataset name: `lama` - 1.3M samples (all samples were used, reporting this for 2 runs)
v1- ~26 seconds for converting whole dataset to json
v2- ~23.6 seconds for converting whole dataset to json

I think it's safe to say that v2 is 10% faster as compared to v1. Timings may improve further with better configuration.

The only bottleneck I feel is writing to file from the output list. If we can improve that aspect then timings may improve further.

Let me know if any changes/improvements can be done in this @stas00, @lhoestq, @albertvillanova. @lhoestq even suggested to extend this work with other export methods as well like `csv` or `parquet`.


Cannot load `few-nerd` dataset

Something isn't working

Hi @Mehrad0711,

I'm afraid there is no "canonical" Hugging Face dataset named "few-nerd".

There are 2 kinds of datasets hosted at the Hugging Face Hub:
- canonical datasets (their identifier contains no slash "/"): we, the Hugging Face team, supervise their implementation and we make sure they work correctly by means of our test suite
- community datasets (their identifier contains a slash "/", where before the slash it is the username or the organization name): those datasets are uploaded to the Hub by the community, and we, the Hugging Face team, do not supervise them; it is the responsibility of the user/organization implementing them properly if they want them to be used by other users.

In this specific case, there is no "canonical" dataset named "few-nerd". On the other hand, there are two "community" datasets named "few-nerd":
- ["nbroad/few-nerd"](https://huggingface.co/datasets/nbroad/few-nerd)
- ["dfki-nlp/few-nerd"](https://huggingface.co/datasets/dfki-nlp/few-nerd)

If they were properly implemented, you should be able to load them this way:
```python
# "nbroad/few-nerd" community dataset
ds = load_dataset("nbroad/few-nerd", "supervised")

# "dfki-nlp/few-nerd" community dataset
ds = load_dataset("dfki-nlp/few-nerd", "supervised")
```

However, they are not correctly implemented and both of them give errors:
- "nbroad/few-nerd":
  ```
  TypeError: expected str, bytes or os.PathLike object, not dict
  ```
- "dfki-nlp/few-nerd":
  ```
  ConnectionError: Couldn't reach https://cloud.tsinghua.edu.cn/f/09265750ae6340429827/?dl=1
  ```

You could try to contact their users/organizations to inform them about their bugs and ask them if they are planning to fix them. Alternatively you could try to implement your own script for this dataset.

Thanks @albertvillanova for your detailed explanation! I will resort to my own scripts for now.

Hello, @Mehrad0711; Hi, @albertvillanova  !
I am the maintainer of the `dfki/few-nerd" dataset script, sorry for the very late reply and hope this message finds you well!
We should use
```
dataset = load_dataset("dfki-nlp/few-nerd", name="supervised")
```
instead of not specifying the "name" argument, where name is from `["supervised", "inter", "intra"]`. Otherwise the method just treats "supervised" as `split`, which we reserve after specifying the name, since for each name, there are three splits: train, dev and test.

Also we use Tsinghua server source to download data files since it is the official source referred in the paper where the dataset is released (even though it is cc-by-sa-4.0 licensed, means we can copy the data anywhere after mentioning the license
). Sometimes the server just runs down due to high pressure, kinda weird (we encountered the same server problem serveral times a month when we conducted experiments on Few-NERD XD). I tried the script just now and it works perfectly!
```
>> dataset
DatasetDict({
    train: Dataset({
        features: ['id', 'tokens', 'ner_tags', 'fine_ner_tags'],
        num_rows: 131767
    })
    validation: Dataset({
        features: ['id', 'tokens', 'ner_tags', 'fine_ner_tags'],
        num_rows: 18824
    })
    test: Dataset({
        features: ['id', 'tokens', 'ner_tags', 'fine_ner_tags'],
        num_rows: 37648
    })
})
>>> dataset["train"]
Dataset({
    features: ['id', 'tokens', 'ner_tags', 'fine_ner_tags'],
    num_rows: 131767
})
>>> dataset["train"][0]
{'id': '0', 'tokens': ['Paul', 'International', 'airport', '.'], 'ner_tags': [0, 0, 0, 0], 'fine_ner_tags': [0, 0, 0, 0]}
```
Anyways if you cannot stand the pain with the server and its slow download speed, you can also download the `dfki/few-nerd.py` script from HF and change the `_URLs` to your personal drive (after you once successfully download the data and upload to your cloud drive), and then load the .py script locally.

Hope this reply can still be any help. If you still have problems with it, feel free to ask here and I am glad to help!
Best wishes.

Hi @chen-yuxuan, thanks for your answer.

Just a few comments:

- Please, note that as we use `datasets.load_dataset` implementation, we can pass the configuration name as the second positional argument (no need to pass explicitly `name=`) and it downloads the 3 splits:
```python
  In [4]: ds = load_dataset("dfki-nlp/few-nerd", "supervised")
Downloading: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 11.5k/11.5k [00:00<00:00, 2.85MB/s]
Downloading and preparing dataset few_nerd/supervised to .cache\huggingface\datasets\dfki-nlp___few_nerd\supervised\0.0.0\e40882b71f037a4a1f232025899170fbe8113cd2f4a26dddd2add7222a077255...
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14.6M/14.6M [01:16<00:00, 190kB/s]
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 11.9M/11.9M [01:14<00:00, 160kB/s]
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12.0M/12.0M [01:04<00:00, 186kB/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [03:58<00:00, 79.45s/it]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00,  3.11it/s]
```

- On the other hand, please note that your script does not work on Windows machines, because you call `open()` without passing the encoding parameter:
```
~\.cache\huggingface\modules\datasets_modules\datasets\dfki-nlp___few-nerd\e40882b71f037a4a1f232025899170fbe8113cd2f4a26dddd2add7222a077255\few-nerd.py in <genexpr>(.0)
    276         assert filepath[-4:] == ".txt"
    277
--> 278         num_lines = sum(1 for _ in open(filepath))
    279         id = 0
    280

.venv\lib\encodings\cp1252.py in decode(self, input, final)
     21 class IncrementalDecoder(codecs.IncrementalDecoder):
     22     def decode(self, input, final=False):
---> 23         return codecs.charmap_decode(input,self.errors,decoding_table)[0]
     24
     25 class StreamWriter(Codec,codecs.StreamWriter):

UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 5238: character maps to <undefined>
```

If you would like your script to be usable on Windows machines, you should pass `encoding="utf-8"` to every `open()` function:
- line 278: `num_lines = sum(1 for _ in open(filepath, encoding="utf-8"))`
- line 281: `with open(filepath, "r", encoding="utf-8")`

Thank you @albertvillanova for your detailed feedback!

> no need to pass explicitly `name=`

Good catch! I thought `split` stands before `name` in the argument list... but now it is all clear to me, sounds cool! Thanks for the explanation.

Anyways in our old code it still looks bit confusing if we only want one split but the function downloads all, so to allow efficient downloading, I optimized the code a bit so that only the specified split data is downloaded. now we get
```
>>> x = load_dataset("dfki-nlp/few-nerd", "supervised")
Downloading and preparing dataset few_nerd/supervised to /home/user/.cache/huggingface/datasets/few_nerd/supervised/0.0.0/8e7ab598946cd5b395dcec6ea239123c8dff5b58b8e1c03b0c595b540248a885...
Downloading: 100%|███████████████████████████████████████████████████████████████████| 14.6M/14.6M [01:01<00:00, 238kB/s]
100%|██████████████████████████████████████████████████████████████████████| 3359329/3359329 [00:12<00:00, 275462.84it/s]
100%|████████████████████████████████████████████████████████████████████████| 482037/482037 [00:01<00:00, 278633.64it/s]
100%|████████████████████████████████████████████████████████████████████████| 958765/958765 [00:03<00:00, 267472.83it/s]
Dataset few_nerd downloaded and prepared to /home/user/.cache/huggingface/datasets/few_nerd/supervised/0.0.0/8e7ab598946cd5b395dcec6ea239123c8dff5b58b8e1c03b0c595b540248a885. Subsequent calls will reuse this data.
```
where only one progress bar indicates downloading, and the three others just indicate pre-processing for the train, dev, test set.

For the encoding issue, I have made corresponding changes for the two lines you pointed out. However, I have no windows machine at hand, I would really appreciate it if you could help test on your end.

All the updates are uploaded to HF under `dfki-nlp` account where I am working for.
Thank you again for your kind help!

Hi @chen-yuxuan,

I have tested on Windows and now it works perfectly, after the fixing of the encoding issue:
```python
In [1]: from datasets import load_dataset

In [2]: ds = load_dataset("dfki-nlp/few-nerd", "supervised")
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 11.5k/11.5k [00:00<?, ?B/s]
Downloading and preparing dataset few_nerd/supervised to C:\Users\username\.cache\huggingface\datasets\dfki-nlp___few_nerd\supervised\0.0.0\e1ceeaee82073fea12206e4461c7cfcd67e68c8f3ebeca179bddcacee00c4511...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3359329/3359329 [00:25<00:00, 129427.23it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 482037/482037 [00:03<00:00, 134513.66it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 958765/958765 [00:06<00:00, 143152.35it/s]
Dataset few_nerd downloaded and prepared to C:\Users\username\.cache\huggingface\datasets\dfki-nlp___few_nerd\supervised\0.0.0\e1ceeaee82073fea12206e4461c7cfcd67e68c8f3ebeca179bddcacee00c4511. Subsequent calls will reuse this data.765 [00:06<00:00, 139045.03it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 174.71it/s]

In [3]: ds
Out[3]:
DatasetDict({
    train: Dataset({
        features: ['id', 'tokens', 'ner_tags', 'fine_ner_tags'],
        num_rows: 131767
    })
    validation: Dataset({
        features: ['id', 'tokens', 'ner_tags', 'fine_ner_tags'],
        num_rows: 18824
    })
    test: Dataset({
        features: ['id', 'tokens', 'ner_tags', 'fine_ner_tags'],
        num_rows: 37648
    })
})
```

## Describe the bug

Cannot load `few-nerd` dataset.

## Steps to reproduce the bug
```python
from datasets import load_dataset
load_dataset('few-nerd', 'supervised')
```

## Actual results

Executing above code will give the following error:

```
Using the latest cached version of the module from /Users/Mehrad/.cache/huggingface/modules/datasets_modules/datasets/few-nerd/62464ace912a40a0f33a11a8310f9041c9dc3590ff2b3c77c14d83ca53cfec53 (last modified on Wed Jun  2 11:34:25 2021) since it couldn't be found locally at /Users/Mehrad/Documents/GitHub/genienlp/few-nerd/few-nerd.py, or remotely (FileNotFoundError).
Downloading and preparing dataset few_nerd/supervised (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /Users/Mehrad/.cache/huggingface/datasets/few_nerd/supervised/0.0.0/62464ace912a40a0f33a11a8310f9041c9dc3590ff2b3c77c14d83ca53cfec53...
Traceback (most recent call last):
  File "/Users/Mehrad/opt/anaconda3/lib/python3.7/site-packages/datasets/builder.py", line 693, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/Users/Mehrad/opt/anaconda3/lib/python3.7/site-packages/datasets/builder.py", line 1107, in _prepare_split
    disable=bool(logging.get_verbosity() == logging.NOTSET),
  File "/Users/Mehrad/opt/anaconda3/lib/python3.7/site-packages/tqdm/std.py", line 1133, in __iter__
    for obj in iterable:
  File "/Users/Mehrad/.cache/huggingface/modules/datasets_modules/datasets/few-nerd/62464ace912a40a0f33a11a8310f9041c9dc3590ff2b3c77c14d83ca53cfec53/few-nerd.py", line 196, in _generate_examples
    with open(filepath, encoding="utf-8") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/Mehrad/.cache/huggingface/datasets/downloads/supervised/train.json'
```
The bug is probably in identifying and downloading the dataset. If I download the json splits directly from [link](https://github.com/nbroad1881/few-nerd/tree/main/uncompressed) and put them under the downloads directory, they will be processed into arrow format correctly.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.11.0
- Python version: 3.8
- PyArrow version: 1.0.1


added semeval18_emotion_classification dataset

For training the multilabel classifier, I would combine the labels into a list, for example for the English dataset:

```
dfpre=pd.read_csv(path+"2018-E-c-En-train.txt",sep="\t")
dfpre['list'] = dfpre[dfpre.columns[2:]].values.tolist()
df = dfpre[['Tweet', 'list']].copy()
df.rename(columns={'list': 'labels'}, inplace=True)
```

Hi @maxpel , have you had a chance to take my comments into account ?

Let me know if you have questions or if I can help :)

Hi @lhoestq ! I did take your comments into account, changed the naming and tried to add dummy data (manually). I am not sure if the dummy data is correct, maybe you can take a look at that.
The model card is still missing as I am currently very busy.

Thanks ! The dummy data looks all good, good job :)

The CI error can be fixed by merging `master` into your branch
```bash
git fetch upstream
git merge upstream/master
```

Hi! I just added the model card and I did the merge you showed above. Should I then add and commit again? The CI error is still there right now.

@lhoestq Unfortunately, I discovered a problem with the test data sets on the competion page (train and dev is fine). They still contain NONE labels for each of the emotions, for example for English: http://saifmohammad.com/WebDocs/AIT-2018/AIT2018-DATA/AIT2018-TEST-DATA/semeval2018englishtestfiles/2018-E-c-En-test.zip
Luckily, a zip file with all data of the competition contains the correct labels also for the test set:
http://saifmohammad.com/WebDocs/AIT-2018/AIT2018-DATA/SemEval2018-Task1-all-data.zip
What's the best way to correct this?

Hi ! I think we can edit the sem_eval_2018_task_1.py file to use this URL instead, and maybe update the `os.path.join` calls to the new paths to the text data in the new ZIP file. Would you like to try to make this work ?

I added the data set of SemEval 2018 Task 1 (Subtask 5) for emotion detection in three languages.

```
datasets-cli test datasets/semeval18_emotion_classification/ --save_infos --all_configs

RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_real_dataset_semeval18_emotion_classification
```
Both commands ran successfully.

I couldn't create the dummy data (the files are tsvs but have .txt ending, maybe that's the problem?) and therefore the test on the dummy data fails, maybe someone can help here.

I also formatted the code:
```
black --line-length 119 --target-version py36 datasets/semeval18_emotion_classification/
isort datasets/semeval18_emotion_classification/
flake8 datasets/semeval18_emotion_classification/
```
That's the publication for reference:

Mohammad, S., Bravo-Marquez, F., Salameh, M., & Kiritchenko, S. (2018). SemEval-2018 task 1: Affect in tweets. Proceedings of the 12th International Workshop on Semantic Evaluation, 1–17. https://doi.org/10.18653/v1/S18-1001


Fix key by recreating metadata JSON for journalists_questions dataset

Close #2743.


Dataset JSON is incorrect

Something isn't working

As discussed, the metadata JSON files must be regenerated because the keys were nor properly generated and they will not be read by the builder:
> Indeed there is some problem/bug while reading the datasets_info.json file: there is a mismatch with the config.name keys in the file...
In the meanwhile, in order to be able to use the datasets_info.json file content, you can create the builder without passing the name :
```
In [25]: builder = datasets.load_dataset_builder("journalists_questions")
In [26]: builder.info.splits
Out[26]: {'train': SplitInfo(name='train', num_bytes=342296, num_examples=10077, dataset_name='journalists_questions')}
```

After regenerating the metadata JSON file for this dataset, I get the right key:
```
{"plain_text": {"description": "The journalists_questions corpus (
```

## Describe the bug

The JSON file generated for https://github.com/huggingface/datasets/blob/573f3d35081cee239d1b962878206e9abe6cde91/datasets/journalists_questions/journalists_questions.py is https://github.com/huggingface/datasets/blob/573f3d35081cee239d1b962878206e9abe6cde91/datasets/journalists_questions/dataset_infos.json.

The only config should be `plain_text`, but the first key in the JSON is `journalists_questions` (the dataset id) instead.

```json
{
  "journalists_questions": {
    "description": "The journalists_questions corpus (version 1.0) is a collection of 10K human-written Arabic\ntweets manually labeled for question identification over Arabic tweets posted by journalists.\n",
    ...
```

## Steps to reproduce the bug

Look at the files.

## Expected results

The first key should be `plain_text`:

```json
{
  "plain_text": {
    "description": "The journalists_questions corpus (version 1.0) is a collection of 10K human-written Arabic\ntweets manually labeled for question identification over Arabic tweets posted by journalists.\n",
    ...
```

## Actual results

```json
{
  "journalists_questions": {
    "description": "The journalists_questions corpus (version 1.0) is a collection of 10K human-written Arabic\ntweets manually labeled for question identification over Arabic tweets posted by journalists.\n",
    ...
```


Improve detection of streamable file types

enhancement

New feature or request

dataset-viewer

Related to the dataset viewer on huggingface.co

maybe we should rather attempt to download a `Range` from the server and see if it works?

**Is your feature request related to a problem? Please describe.**

```python
from datasets import load_dataset_builder
from datasets.utils.streaming_download_manager import StreamingDownloadManager
builder = load_dataset_builder("journalists_questions", name="plain_text")
builder._split_generators(StreamingDownloadManager(base_path=builder.base_path))
```

raises

```
NotImplementedError: Extraction protocol for file at https://drive.google.com/uc?export=download&id=1CBrh-9OrSpKmPQBxTK_ji6mq6WTN_U9U is not implemented yet
```

But the file at https://drive.google.com/uc?export=download&id=1CBrh-9OrSpKmPQBxTK_ji6mq6WTN_U9U is a text file and it can be streamed:

```bash
curl --header "Range: bytes=0-100" -L https://drive.google.com/uc\?export\=download\&id\=1CBrh-9OrSpKmPQBxTK_ji6mq6WTN_U9U
506938088174940160      yes     1
302221719412830209      yes     1
289761704907268096      yes     1
513820885032378369      yes     %
```

Yet, it's wrongly categorized as a file type that cannot be streamed because the test is currently based on 1. the presence of a file extension at the end of the URL (here: no extension), and 2. the inclusion of this extension in a list of supported formats.

**Describe the solution you'd like**

In the case of an URL (instead of a local path), ask for the MIME type, and decide on that value? Note that it would not work in that case, because the value of `content_type` is `text/html; charset=UTF-8`.

**Describe alternatives you've considered**

Add a variable in the dataset script to set the data format by hand.


Add Hypersim dataset

dataset request

Requesting to add a new dataset

vision

Vision datasets

## Adding a Dataset
- **Name:** Hypersim
- **Description:** photorealistic synthetic dataset for holistic indoor scene understanding
- **Paper:** *link to the dataset paper if available*
- **Data:** https://github.com/apple/ml-hypersim

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Update release instructions

Update release instructions.


Pass tokenize to sacrebleu only if explicitly passed by user

Next `sacrebleu` release (v2.0.0) will remove `sacrebleu.DEFAULT_TOKENIZER`: https://github.com/mjpost/sacrebleu/pull/152/files#diff-2553a315bb1f7e68c9c1b00d56eaeb74f5205aeb3a189bc3e527b122c6078795L17-R15

This PR passes `tokenize` to `sacrebleu` only if explicitly passed by the user, otherwise it will not pass it (and `sacrebleu` will use its default, no matter where it is and how it is called).

Close: #2737.


Sunbird AI Ugandan low resource language dataset

Hi @ak3ra , have you had a chance to take my comments into account ?

Let me know if you have questions or if I can help :)

@lhoestq  Working on this, thanks for the detailed review :)

Hi ! Cool thanks :)
Feel free to merge master into your branch to fix the CI issues

Let me know if you have questions or if I can help

Multi-way parallel text corpus of 5 key Ugandan languages for the task of machine translation.


SacreBLEU update

bug

Something isn't working

Hi @devrimcavusoglu,
I tried your code with latest version of `datasets`and `sacrebleu==1.5.1` and it's running fine after changing one small thing:
```
sacrebleu = datasets.load_metric('sacrebleu')
predictions = ["It is a guide to action which ensures that the military always obeys the commands of the party"]
references = [["It is a guide to action that ensures that the military will forever heed Party commands"]]  # double brackets here should do the work
results = sacrebleu.compute(predictions=predictions, references=references)
print(results)
output: {'score': 41.180376356915765, 'counts': [11, 8, 6, 4], 'totals': [18, 17, 16, 15], 'precisions': [61.111111111111114, 47.05882352941177, 37.5, 26.666666666666668], 'bp': 1.0, 'sys_len': 18, 'ref_len': 16}
```

@bhavitvyamalik hmm. I forgot double brackets, but still didn't work when used it with double brackets. It may be an isseu with platform (using win-10 currently), or versions. What is your platform and your version info for datasets, python, and sacrebleu ?

You can check that here, I've reproduced your code in [Google colab](https://colab.research.google.com/drive/1X90fHRgMLKczOVgVk7NDEw_ciZFDjaCM?usp=sharing). Looks like there was some issue in `sacrebleu` which was fixed later from what I've found [here](https://github.com/pytorch/fairseq/issues/2049#issuecomment-622367967). Upgrading `sacrebleu` to latest version should work.

It seems that next release of `sacrebleu` (v2.0.0) will break our `datasets` implementation to compute it. See my Google Colab: https://colab.research.google.com/drive/1SKmvvjQi6k_3OHsX5NPkZdiaJIfXyv9X?usp=sharing

I'm reopening this Issue and making a Pull Request to fix it.

> It seems that next release of `sacrebleu` (v2.0.0) will break our `datasets` implementation to compute it. See my Google Colab: https://colab.research.google.com/drive/1SKmvvjQi6k_3OHsX5NPkZdiaJIfXyv9X?usp=sharing
>
> I'm reopening this Issue and making a Pull Request to fix it.

How did you solve him

With the latest release of [sacrebleu](https://github.com/mjpost/sacrebleu), `datasets.metrics.sacrebleu` is broken, and getting error.

    AttributeError: module 'sacrebleu' has no attribute 'DEFAULT_TOKENIZER'

this happens since in new version of sacrebleu there is no `DEFAULT_TOKENIZER`, but sacrebleu.py tries to import it anyways. This can be fixed currently with fixing `sacrebleu==1.5.0`

## Steps to reproduce the bug
```python
sacrebleu= datasets.load_metric('sacrebleu')
predictions = ["It is a guide to action which ensures that the military always obeys the commands of the party"]
references = ["It is a guide to action that ensures that the military will forever heed Party commands"]
results = sacrebleu.compute(predictions=predictions, references=references)
print(results)
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.11.0
- Platform: Windows-10-10.0.19041-SP0
- Python version: Python 3.8.0
- PyArrow version: 5.0.0


Add Microsoft Building Footprints dataset

dataset request

Requesting to add a new dataset

vision

Vision datasets

Motivation: this can be a useful dataset for researchers working on climate change adaptation, urban studies, geography, etc. I'll see if I can figure out how to add it!

## Adding a Dataset
- **Name:** Microsoft Building Footprints
- **Description:** With the goal to increase the coverage of building footprint data available as open data for OpenStreetMap and humanitarian efforts, we have released millions of building footprints as open data available to download free of charge.
- **Paper:** *link to the dataset paper if available*
- **Data:** https://www.microsoft.com/en-us/maps/building-footprints
- **Motivation:** this can be a useful dataset for researchers working on climate change adaptation, urban studies, geography, etc.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Reported by: @sashavor


Add Open Buildings dataset

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** Open Buildings
- **Description:** A dataset of building footprints to support social good applications.

  Building footprints are useful for a range of important applications, from population estimation, urban planning and humanitarian response, to environmental and climate science. This large-scale open dataset contains the outlines of buildings derived from high-resolution satellite imagery in order to support these types of uses. The project being based in Ghana, the current focus is on the continent of Africa.

  See: "Mapping Africa's Buildings with Satellite Imagery" https://ai.googleblog.com/2021/07/mapping-africas-buildings-with.html
- **Paper:** https://arxiv.org/abs/2107.12283
- **Data:** https://sites.research.google/open-buildings/
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Reported by: @osanseviero


Update BibTeX entry

Update BibTeX entry.


Add missing parquet known extension

This code was failing because the parquet extension wasn't recognized:

```python
from datasets import load_dataset
base_url = "https://storage.googleapis.com/huggingface-nlp/cache/datasets/wikipedia/20200501.en/1.0.0/"
data_files = {"train": base_url + "wikipedia-train.parquet"}
wiki = load_dataset("parquet", data_files=data_files, split="train", streaming=True)
```

It raises
```python
NotImplementedError: Extraction protocol for file at https://storage.googleapis.com/huggingface-nlp/cache/datasets/wikipedia/20200501.en/1.0.0/wikipedia-train.parquet is not implemented yet
```

I added `parquet` to the list of known extensions

EDIT: added pickle, conllu, xml extensions as well


Updated TTC4900 Dataset

@lhoestq, lütfen bu PR'ı gözden geçirebilir misiniz?

> Thanks ! This looks all good now :)

Thanks

- The source address of the TTC4900 dataset of [@savasy](https://github.com/savasy) has been updated for direct download.
- Updated readme.


Adding to_tf_dataset method

This seems to be working reasonably well in testing, and performance is way better. `tf.py_function` has been dropped for an input generator, but I moved as much of the code as possible outside the generator to allow TF to compile it correctly. I also avoid `tf.RaggedTensor` at all costs, and do the shuffle in the dataset followed by accessing sequential chunks, instead of shuffling an index tensor. The combination of all of these gives us a more flexible data loader as well as a ~20X boost in performance compared to the first solution.

I made a change to the `TFFormatter` in this PR that will need some changes to the tests, so I wanted to ping @lhoestq and anyone else before I made those changes.

The key problem is that up until now the `TFFormatter` always returns `RaggedTensor`, created using the very slow `tf.ragged.constant` function. This is a big performance penalty, but it's also (imo) surprising for users - `RaggedTensor` handles tensors where one dimension has variable length. This is a good choice for tokenized datasets with variable sequence length, but it's an odd choice when the non-batch dimensions are constant, such as in image datasets, or in datasets where all samples are padded to the same length (e.g. for TPU training).

The change I made was to try to return standard `Tensor` objects instead of `RaggedTensor` when all the samples in the batch had the same shape, and if that was not the case to fall back to fast `RaggedTensor` creation with `tf.ragged.stack`, and only falling back to the very slow `tf.ragged.constant` function as a last resort. I think this will match user expectations in most cases and greatly improve performance, but it's a (very slightly) breaking change, so any feedback is welcome!

Also I really can't emphasize enough how slow `tf.ragged.constant` is, it's bad enough to create a data pipeline bottleneck in more or less any training setup:
![image](https://user-images.githubusercontent.com/12866554/131121785-4fbe942a-1ca4-4af6-a9da-cd6d5ea67b30.png)

Hi @lhoestq, the tests have been modified and everything is passing. The Windows tests look to be failing for an unrelated reason, but other than that I'm ready to merge if you are!

Hi @Rocketknight1 ! Feel free to merge `master` into this branch to fix and run the full CI :)

@lhoestq rebased onto master and it looks good! I'm doing some testing with new notebook examples, but are you happy to merge if that looks good?

@lhoestq No, I'm happy to merge it as-is and add documentation afterwards!

Oh my **god** do not merge this yet, it's just a draft.

I've added a method (via a mixin) to the `arrow_dataset.Dataset` class that automatically converts our Dataset classes to TF Dataset classes ready for training. It hopefully has most of the features we want, including streaming from disk (no need to load the whole dataset in memory!), correct shuffling, variable-length batches to reduce compute, and correct support for unusual padding. It achieves that by calling the tokenizer `pad` method in the middle of a TF compute graph via a very hacky call to `tf.py_function`, which is heretical but seems to work.

A number of issues need to be resolved before it's ready to merge, though:

1) Is a MixIn the right way to do this? Do other classes besides `arrow_dataset.Dataset` need this method too?
2) Needs an argument to support constant-length batches for TPU training - this is easy to add and I'll do it soon.
3) Needs the user to supply the list of columns to drop from the arrow `Dataset`. Is there some automatic way to get the columns we want, or see which columns were added by the tokenizer?
4) Assumes the label column is always present and always called "label" - this is probably not great, but I'm not sure what the 'correct' thing to do here is.


Update CommonVoice with new release

Requesting to add a new dataset

Does anybody know if there is a bundled link, which would allow direct data download instead of manual?
Something similar to: `https://voice-prod-bundler-ee1969a6ce8178826482b88e843c335139bd3fb4.s3.amazonaws.com/cv-corpus-6.1-2020-12-11/ab.tar.gz` ? cc @patil-suraj

## Adding a Dataset
- **Name:** CommonVoice mid-2021 release
- **Description:** more data in CommonVoice: Languages that have increased the most by percentage are Thai (almost 20x growth, from 12 hours to 250 hours), Luganda (almost 9x growth, from 8 to 80), Esperanto (7x growth, from 100 to 840), and Tamil (almost 8x, from 24 to 220).
- **Paper:** https://discourse.mozilla.org/t/common-voice-2021-mid-year-dataset-release/83812
- **Data:** https://commonvoice.mozilla.org/en/datasets
- **Motivation:** More data and more varied. I think we just need to add configs in the existing dataset script.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Fix IndexError while loading Arabic Billion Words dataset

bug

Something isn't working

Catch `IndexError` and ignore that record.

Close #2727.


Concurrent use of same dataset (already downloaded)

Something isn't working

Launching simultaneous job relying on the same datasets try some writing issue. I guess it is unexpected since I only need to load some already downloaded file.

If i have two jobs that use the same dataset. I got :

  File "compute_measures.py", line 181, in <module>
    train_loader, val_loader, test_loader = get_dataloader(args)
  File "/gpfsdswork/projects/rech/toto/intRAOcular/dataset_utils.py", line 69, in get_dataloader
    dataset_train = load_dataset('paws', "labeled_final", split='train', download_mode="reuse_cache_if_exists")
  File "/gpfslocalsup/pub/anaconda-py3/2020.02/envs/pytorch-gpu-1.7.0/lib/python3.7/site-packages/datasets/load.py", line 748, in load_dataset
    use_auth_token=use_auth_token,
  File "/gpfslocalsup/pub/anaconda-py3/2020.02/envs/pytorch-gpu-1.7.0/lib/python3.7/site-packages/datasets/builder.py", line 582, in download_and_prepare
    self._save_info()
  File "/gpfslocalsup/pub/anaconda-py3/2020.02/envs/pytorch-gpu-1.7.0/lib/python3.7/site-packages/datasets/builder.py", line 690, in _save_info
    self.info.write_to_directory(self._cache_dir)
  File "/gpfslocalsup/pub/anaconda-py3/2020.02/envs/pytorch-gpu-1.7.0/lib/python3.7/site-packages/datasets/info.py", line 195, in write_to_directory
    with open(os.path.join(dataset_info_dir, config.LICENSE_FILENAME), "wb") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/gpfswork/rech/toto/datasets/paws/labeled_final/1.1.0/09d8fae989bb569009a8f5b879ccf2924d3e5cd55bfe2e89e6dab1c0b50ecd34.incomplete/LICENSE'

You can probably have a solution much faster than me (first time I use the library). But I suspect some write function are used when loading the dataset from cache.

I have the same issue:
```
Traceback (most recent call last):
  File "/dccstor/tslm/envs/anaconda3/envs/trf-a100/lib/python3.9/site-packages/datasets/builder.py", line 652, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/dccstor/tslm/envs/anaconda3/envs/trf-a100/lib/python3.9/site-packages/datasets/builder.py", line 1040, in _prepare_split
    with ArrowWriter(features=self.info.features, path=fpath) as writer:
  File "/dccstor/tslm/envs/anaconda3/envs/trf-a100/lib/python3.9/site-packages/datasets/arrow_writer.py", line 192, in __init__
    self.stream = pa.OSFile(self._path, "wb")
  File "pyarrow/io.pxi", line 829, in pyarrow.lib.OSFile.__cinit__
  File "pyarrow/io.pxi", line 844, in pyarrow.lib.OSFile._open_writable
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 97, in pyarrow.lib.check_status
FileNotFoundError: [Errno 2] Failed to open local file '/dccstor/tslm-gen/.cache/csv/default-387f1f95c084d4df/0.0.0/2dc6629a9ff6b5697d82c25b73731dd440507a69cbce8b425db50b751e8fcfd0.incomplete/csv-validation.arrow'. Detail: [errno 2] No such file or directory
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/dccstor/tslm/elron/tslm-gen/train.py", line 510, in <module>
    main()
  File "/dccstor/tslm/elron/tslm-gen/train.py", line 246, in main
    datasets = prepare_dataset(dataset_args, logger)
  File "/dccstor/tslm/elron/tslm-gen/data.py", line 157, in prepare_dataset
    datasets = load_dataset(extension, data_files=data_files, split=dataset_split, cache_dir=dataset_args.dataset_cache_dir, na_filter=False, download_mode=dataset_args.dataset_generate_mode)
  File "/dccstor/tslm/envs/anaconda3/envs/trf-a100/lib/python3.9/site-packages/datasets/load.py", line 742, in load_dataset
    builder_instance.download_and_prepare(
  File "/dccstor/tslm/envs/anaconda3/envs/trf-a100/lib/python3.9/site-packages/datasets/builder.py", line 574, in download_and_prepare
    self._download_and_prepare(
  File "/dccstor/tslm/envs/anaconda3/envs/trf-a100/lib/python3.9/site-packages/datasets/builder.py", line 654, in _download_and_prepare
    raise OSError(
OSError: Cannot find data file.
Original error:
[Errno 2] Failed to open local file '/dccstor/tslm-gen/.cache/csv/default-387f1f95c084d4df/0.0.0/2dc6629a9ff6b5697d82c25b73731dd440507a69cbce8b425db50b751e8fcfd0.incomplete/csv-validation.arrow'. Detail: [errno 2] No such file or directory
```

## Describe the bug
When launching several jobs at the same time loading the same dataset trigger some errors see (last comments).

## Steps to reproduce the bug
export HF_DATASETS_CACHE=/gpfswork/rech/toto/datasets
for MODEL in "bert-base-uncased" "roberta-base" "distilbert-base-cased"; do #  "bert-base-uncased" "bert-large-cased" "roberta-large" "albert-base-v1" "albert-large-v1"; do
  for TASK_NAME in "mrpc" "rte" 'imdb' "paws" "mnli"; do
    export OUTPUT_DIR=${MODEL}_${TASK_NAME}
    sbatch --job-name=${OUTPUT_DIR} \
      --gres=gpu:1 \
      --no-requeue \
      --cpus-per-task=10 \
      --hint=nomultithread \
      --time=1:00:00 \
      --output=jobinfo/${OUTPUT_DIR}_%j.out \
      --error=jobinfo/${OUTPUT_DIR}_%j.err \
      --qos=qos_gpu-t4 \
      --wrap="module purge; module load pytorch-gpu/py3/1.7.0 ; export HF_DATASETS_OFFLINE=1; export HF_DATASETS_CACHE=/gpfswork/rech/toto/datasets;  python compute_measures.py --seed=$SEED --saving_path=results --batch_size=$BATCH_SIZE --task_name=$TASK_NAME --model_name=/gpfswork/rech/toto/transformers_models/$MODEL"

  done
done

```python
# Sample code to reproduce the bug
        dataset_train = load_dataset('imdb', split='train', download_mode="reuse_cache_if_exists")
        dataset_train = dataset_train.map(lambda e: tokenizer(e['text'], truncation=True, padding='max_length'),
                                          batched=True).select(list(range(args.filter)))

        dataset_val = load_dataset('imdb', split='train', download_mode="reuse_cache_if_exists")
        dataset_val = dataset_val.map(lambda e: tokenizer(e['text'], truncation=True, padding='max_length'),
                                      batched=True).select(list(range(args.filter, args.filter + 5000)))

        dataset_test = load_dataset('imdb', split='test', download_mode="reuse_cache_if_exists")
        dataset_test = dataset_test.map(lambda e: tokenizer(e['text'], truncation=True, padding='max_length'),
                                        batched=True)
```

## Expected results
I believe I am doing something wrong with the objects.

## Actual results
Traceback (most recent call last):
  File "/gpfslocalsup/pub/anaconda-py3/2020.02/envs/pytorch-gpu-1.7.0/lib/python3.7/site-packages/datasets/builder.py", line 652, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/gpfslocalsup/pub/anaconda-py3/2020.02/envs/pytorch-gpu-1.7.0/lib/python3.7/site-packages/datasets/builder.py", line 983, in _prepare_split
    check_duplicates=True,
  File "/gpfslocalsup/pub/anaconda-py3/2020.02/envs/pytorch-gpu-1.7.0/lib/python3.7/site-packages/datasets/arrow_writer.py", line 192, in __init__
    self.stream = pa.OSFile(self._path, "wb")
  File "pyarrow/io.pxi", line 829, in pyarrow.lib.OSFile.__cinit__
  File "pyarrow/io.pxi", line 844, in pyarrow.lib.OSFile._open_writable
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 97, in pyarrow.lib.check_status
FileNotFoundError: [Errno 2] Failed to open local file '/gpfswork/rech/tts/unm25jp/datasets/paws/labeled_final/1.1.0/09d8fae989bb569009a8f5b879ccf2924d3e5cd55bfe2e89e6dab1c0b50ecd34.incomplete/paws-test.arrow'. Detail: [errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "compute_measures.py", line 181, in <module>
    train_loader, val_loader, test_loader = get_dataloader(args)
  File "/gpfsdswork/projects/rech/toto/intRAOcular/dataset_utils.py", line 69, in get_dataloader
    dataset_train = load_dataset('paws', "labeled_final", split='train', download_mode="reuse_cache_if_exists")
  File "/gpfslocalsup/pub/anaconda-py3/2020.02/envs/pytorch-gpu-1.7.0/lib/python3.7/site-packages/datasets/load.py", line 748, in load_dataset
    use_auth_token=use_auth_token,
  File "/gpfslocalsup/pub/anaconda-py3/2020.02/envs/pytorch-gpu-1.7.0/lib/python3.7/site-packages/datasets/builder.py", line 575, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/gpfslocalsup/pub/anaconda-py3/2020.02/envs/pytorch-gpu-1.7.0/lib/python3.7/site-packages/datasets/builder.py", line 658, in _download_and_prepare
    + str(e)
OSError: Cannot find data file.
Original error:
[Errno 2] Failed to open local file '/gpfswork/rech/toto/datasets/paws/labeled_final/1.1.0/09d8fae989bb569009a8f5b879ccf2924d3e5cd55bfe2e89e6dab1c0b50ecd34.incomplete/paws-test.arrow'. Detail: [errno 2] No such file or directory

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: datasets==1.8.0
- Platform: linux (jeanzay)
- Python version: pyarrow==2.0.0
- PyArrow version: 3.7.8


Error in loading the Arabic Billion Words Corpus

Something isn't working

I modified the dataset loading script to catch the `IndexError` and inspect the records at which the error is happening, and I found this:
For the `Techreen` config, the error happens in 36 records when trying to find the `Text` or `Dateline` tags. All these 36 records look something like:
```
<Techreen>
 <ID>TRN_ARB_0248167</ID>
 <URL>http://tishreen.news.sy/tishreen/public/read/248240</URL>
 <Headline>Removed, because the original articles was in English</Headline>
</Techreen>
```

and all the 288 faulty records in the `Almustaqbal` config look like:
```
<Almustaqbal>
 <ID>MTL_ARB_0028398</ID>

 <URL>http://www.almustaqbal.com/v4/article.aspx?type=NP&ArticleID=179015</URL>
 <Headline> Removed because it is not available in the original site</Headline>
</Almustaqbal>
```

so the error is happening because the articles were removed and so the associated records lack the `Text` tag.

In this case, I think we just need to catch the `IndexError` and ignore (pass) it.

Thanks @M-Salti for reporting this issue and for your investigation.

Indeed, those `IndexError` should be catched and the corresponding record should be ignored.

I'm opening a Pull Request to fix it.

## Describe the bug
I get `IndexError: list index out of range` when trying to load the `Techreen` and `Almustaqbal` configs of the dataset.

## Steps to reproduce the bug
```python
load_dataset("arabic_billion_words", "Techreen")
load_dataset("arabic_billion_words", "Almustaqbal")
```

## Expected results
The datasets load succefully.

## Actual results
```python
_extract_tags(self, sample, tag)
    139             if len(out) > 0:
    140                 break
--> 141         return out[0]
    142
    143     def _clean_text(self, text):

IndexError: list index out of range
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.10.2
- Platform: Ubuntu 18.04.5 LTS
- Python version: 3.7.11
- PyArrow version: 3.0.0


Typo fix `tokenize_exemple`

There is a small typo in the main README.md


Pass use_auth_token to request_etags

Fix #2724.


404 Error when loading remote data files from private repo

Something isn't working

I guess the issue is when computing the ETags of the remote files. Indeed `use_auth_token` must be passed to `request_etags` here:

https://github.com/huggingface/datasets/blob/35b5e4bc0cb2ed896e40f3eb2a4aa3de1cb1a6c5/src/datasets/builder.py#L160-L160

Yes, I remember having properly implemented that:
- https://github.com/huggingface/datasets/commit/7a9c62f7cef9ecc293f629f859d4375a6bd26dc8#diff-f933ce41f71c6c0d1ce658e27de62cbe0b45d777e9e68056dd012ac3eb9324f7R160
- https://github.com/huggingface/datasets/pull/2628/commits/6350a03b4b830339a745f7b1da46ece784ca734c

But a subsequent refactoring accidentally removed it...

## Describe the bug
When loading remote data files from a private repo, a 404 error is raised.

## Steps to reproduce the bug
```python
url = hf_hub_url("lewtun/asr-preds-test", "preds.jsonl", repo_type="dataset")
dset = load_dataset("json", data_files=url, use_auth_token=True)
# HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/datasets/lewtun/asr-preds-test/resolve/main/preds.jsonl
```

## Expected results
Load dataset.

## Actual results
404 Error.


Fix en subset by modifying dataset_info with correct validation infos

- Related to: #2682

We correct the values of `en` subset concerning the expected validation values (both `num_bytes` and `num_examples`.

Instead of having:

`{"name": "validation", "num_bytes": 828589180707, "num_examples": 364868892, "dataset_name": "c4"}`

We replace with correct values:

`{"name": "validation", "num_bytes": 825767266, "num_examples": 364608, "dataset_name": "c4"}`

There are still issues with validation with other subsets, but I can't download all the files, unzip to check for the correct number of bytes. (If you have a fast way to obtain those values for other subsets, I can do this in this PR ... otherwise I can't spend those resources)


Missing cache file

Something isn't working

This could be solved by going to the glue/ directory and delete sst2 directory, then load the dataset again will help you redownload the dataset.

Hi ! Not sure why this file was missing, but yes the way to fix this is to delete the sst2 directory and to reload the dataset

Strangely missing cache file after I  restart my program again.

`glue_dataset = datasets.load_dataset('glue', 'sst2')`

`FileNotFoundError: [Errno 2] No such file or directory: /Users/chris/.cache/huggingface/datasets/glue/sst2/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96d6053ad/dataset_info.json'`


Deal with the bad check in test_load.py

Hi ! I did a change for this test already in #2662 :

https://github.com/huggingface/datasets/blob/00686c46b7aaf6bfcd4102cec300a3c031284a5a/tests/test_load.py#L312-L316

(though I have to change the variable name `m_combined_path` to `m_url` or something)

I guess it's ok to remove this check for now :)

This PR removes a check that's been added in #2684. My intention with this check was to capture an URL in the error message, but instead, it captures a substring of the previous regex match in the test function. Another option would be to replace this check with:
```python
m_paths = re.findall(r"\S*_dummy/_dummy.py\b", str(exc_info.value))  # on Linux this will match an URL as well as a local_path due to different os.sep, so take the last element (an URL always comes last in the list)
assert len(m_paths) > 0 and is_remote_url(m_paths[-1])  # is_remote_url comes from datasets.utils.file_utils
```

@lhoestq Let me know which one of these two approaches (delete or replace) do you prefer?


fix: 🐛 fix two typos


Use ETag in streaming mode to detect resource updates

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**

I want to cache data I generate from processing a dataset I've loaded in streaming mode, but I've currently no way to know if the remote data has been updated or not, thus I don't know when to invalidate my cache.

**Describe the solution you'd like**

Take the ETag of the data files into account and provide it (directly or through a hash) to give a signal that I can invalidate my cache.

**Describe alternatives you've considered**

None


New documentation structure

I just did some minor changes + added some content in these sections: share, about arrow, about cache

Feel free to mark this PR as ready for review ! :)

I just separated the `Share` How-to page into three pages: share, dataset_script and dataset_card.

This way in the share page we can explain in more details how to share a community or a canonical dataset - focus in their differences and the steps to upload them.

Also given that making a dataset script or a dataset card both require several steps, I feel like it's better to have dedicated pages for them.

Let me know what you think @stevhliu and others. We can still revert this change if you feel like it was better with everything in the same place.

I just added some minor changes to match the style, fix typos, etc. Great work on the conceptual guides, I learned a lot from them and I'm sure they will help a lot of other people too!

I am fine with splitting `Share` into three separate pages. I think this probably makes it easier for users to navigate, instead of having to scroll up and down on a really long single page.

Thanks a lot for all the suggestions ! I'm doing the final changes based on the remaining comments, then we can merge and release v1.12 of `datasets` and the new documentation ^^

Alright I think I took all the suggestions and comments into account :)
Thanks everyone for the help !

Organize Datasets documentation into four documentation types to improve clarity and discoverability of content.

**Content to add in the very short term (feel free to add anything I'm missing):**
- A discussion on why Datasets uses Arrow that includes some context and background about why we use Arrow. Would also be great to talk about Datasets speed and performance here, and if you can share any benchmarking/tests you did, that would be awesome! Finally, a discussion about how memory-mapping frees the user from RAM constraints would be very helpful.
- Explain why you would want to disable or override verifications when loading a dataset.
- If possible, include a code sample of when the number of elements in the field of an output dictionary aren’t the same as the other fields in the output dictionary (taken from the [note](https://huggingface.co/docs/datasets/processing.html#augmenting-the-dataset) here).


Fix shuffle on IterableDataset that disables batching in case any functions were mapped

Made a very minor change to fix the issue#2716. Added the missing argument in the constructor call.

As discussed in the bug report, the change is made to prevent the `shuffle` method call from resetting the value of `batched` attribute in `MappedExamplesIterable`

Fix #2716.


Calling shuffle on IterableDataset will disable batching in case any functions were mapped

bug

Something isn't working

Hi :) Good catch ! Feel free to open a PR if you want to contribute, this would be very welcome ;)

Have raised the PR [here](https://github.com/huggingface/datasets/pull/2717)

When using dataset in streaming mode, if one applies `shuffle` method on the dataset and `map` method for which `batched=True` than the batching operation will not happen, instead `batched` will be set to `False`

I did RCA on the dataset codebase, the problem is emerging from [this line of code](https://github.com/huggingface/datasets/blob/d25a0bf94d9f9a9aa6cabdf5b450b9c327d19729/src/datasets/iterable_dataset.py#L197) here as it is
`self.ex_iterable.shuffle_data_sources(seed), function=self.function, batch_size=self.batch_size`, as one can see it is missing batched argument, which means that the iterator fallsback to default constructor value, which in this case is `False`.
To remedy the problem we can change this line to
`self.ex_iterable.shuffle_data_sources(seed), function=self.function, batched=self.batched, batch_size=self.batch_size`


Update PAN-X data URL in XTREME dataset

Merging since the CI is just about missing infos in the dataset card

Related to #2710, #2691.


add more precise information for size

enhancement

New feature or request

We already have this information in the dataset_infos.json files of each dataset.
Maybe we can parse these files in the backend to return their content with the endpoint at huggingface.co/api/datasets

For now if you want to access this info you have to load the json for each dataset. For example:
- for a dataset on github like `squad`
-   https://raw.githubusercontent.com/huggingface/datasets/master/datasets/squad/dataset_infos.json
- for a community dataset on the hub like `lhoestq/squad`:
  https://huggingface.co/datasets/lhoestq/squad/resolve/main/dataset_infos.json

For the import into ELG, we would like a more precise description of the size of the dataset, instead of the current size categories. The size can be expressed in bytes, or any other preferred size unit. As suggested in the slack channel, perhaps this could be computed with a regex for existing datasets.


Enumerate all ner_tags values in WNUT 17 dataset

This PR does:
- Enumerate all ner_tags in dataset card Data Fields section
- Add all metadata tags to dataset card

Close #2709.


Update WikiANN data URL

We have to update the URL in the XTREME benchmark as well:

https://github.com/huggingface/datasets/blob/0dfc639cec450ed8762a997789a2ed63e63cdcf2/datasets/xtreme/xtreme.py#L411-L411

WikiANN data source URL is no longer accessible: 404 error from Dropbox.

We have decided to host it at Hugging Face. This PR updates the data source URL, the metadata JSON file and the dataset card.

Close #2691.


Missing documentation for wnut_17 (ner_tags)

bug

Something isn't working

Hi @maxpel, thanks for reporting this issue.

Indeed, the documentation in the dataset card is not complete. I’m opening a Pull Request to fix it.

As the paper explains, there are 6 entity types and we have ordered them alphabetically: `corporation`, `creative-work`, `group`, `location`, `person` and `product`.

Each of these entity types has 2 possible IOB2 format tags:
- `B-`: to indicate that the token is the beginning of an entity name, and the
- `I-`: to indicate that the token is inside an entity name.

Additionally, there is the standalone IOB2 tag
- `O`: that indicates that the token belongs to no named entity.

In total there are 13 possible tags, which correspond to the following integer numbers:

0. `O`
1. `B-corporation`
2. `I-corporation`
3. `B-creative-work`
4. `I-creative-work`
5. `B-group`
6. `I-group`
7. `B-location`
8. `I-location`
9. `B-person`
10. `I-person`
11. `B-product`
12. `I-product`

On the info page of the wnut_17 data set (https://huggingface.co/datasets/wnut_17), the model output of ner-tags is only documented for these 5 cases:

`ner_tags: a list of classification labels, with possible values including O (0), B-corporation (1), I-corporation (2), B-creative-work (3), I-creative-work (4).`

I trained a model with the data and it gives me 13 classes:

```
"id2label": {
    "0": 0,
    "1": 1,
    "2": 2,
    "3": 3,
    "4": 4,
    "5": 5,
    "6": 6,
    "7": 7,
    "8": 8,
    "9": 9,
    "10": 10,
    "11": 11,
    "12": 12
  }

  "label2id": {
    "0": 0,
    "1": 1,
    "10": 10,
    "11": 11,
    "12": 12,
    "2": 2,
    "3": 3,
    "4": 4,
    "5": 5,
    "6": 6,
    "7": 7,
    "8": 8,
    "9": 9
  }
```
The paper (https://www.aclweb.org/anthology/W17-4418.pdf) explains those 6 categories, but the ordering does not match:

```
1. person
2. location (including GPE, facility)
3. corporation
4. product (tangible goods, or well-defined
services)
5. creative-work (song, movie, book and
so on)
6. group (subsuming music band, sports team,
and non-corporate organisations)
```
I would be very helpful for me, if somebody could clarify the model ouputs and explain the "B-" and "I-" prefixes to me.

Really great work with that and the other packages, I couldn't believe that training the model with that data was basically a one-liner!


QASC: incomplete training set

Something isn't working

Hi @danyaljj, thanks for reporting.

Unfortunately, I have not been able to reproduce your problem. My train split has 8134 examples:
```ipython
In [10]: ds["train"]
Out[10]:
Dataset({
    features: ['id', 'question', 'choices', 'answerKey', 'fact1', 'fact2', 'combinedfact', 'formatted_question'],
    num_rows: 8134
})

In [11]: ds["train"].shape
Out[11]: (8134, 8)
```
and the content of the last 5 examples is:
```ipython
In [12]: for i in range(8129, 8134):
    ...:     print(json.dumps(ds["train"][i]))
    ...:
{"id": "3KAKFY4PGU1LGXM77JAK2700NGCI3X", "question": "Chitin can be used for protection by whom?", "choices": {"text": ["Fungi", "People", "Man", "Fish", "trees", "Dogs", "animal", "Birds"], "label": ["A", "B",
 "C", "D", "E", "F", "G", "H"]}, "answerKey": "D", "fact1": "scales are used for protection by scaled animals", "fact2": "Fish scales are also composed of chitin.", "combinedfact": "Chitin can be used for prote
ction by fish.", "formatted_question": "Chitin can be used for protection by whom? (A) Fungi (B) People (C) Man (D) Fish (E) trees (F) Dogs (G) animal (H) Birds"}
{"id": "336YQZE83VDAQVZ26HW59X51JZ9M5M", "question": "Which type of animal uses plates for protection?", "choices": {"text": ["squids", "reptiles", "sea urchins", "fish", "amphibians", "Frogs", "mammals", "salm
on"], "label": ["A", "B", "C", "D", "E", "F", "G", "H"]}, "answerKey": "B", "fact1": "scales are used for protection by scaled animals", "fact2": "Reptiles have scales or plates.", "combinedfact": "Reptiles use
 their plates for protection.", "formatted_question": "Which type of animal uses plates for protection? (A) squids (B) reptiles (C) sea urchins (D) fish (E) amphibians (F) Frogs (G) mammals (H) salmon"}
{"id": "3WZ36BJEV3FGS66VGOOUYX0LN8GTBU", "question": "What are used for protection by fish?", "choices": {"text": ["scales", "fins", "streams.", "coral", "gills", "Collagen", "mussels", "whiskers"], "label": ["
A", "B", "C", "D", "E", "F", "G", "H"]}, "answerKey": "A", "fact1": "scales are used for protection by scaled animals", "fact2": "Fish are backboned aquatic animals.", "combinedfact": "scales are used for prote
ction by fish ", "formatted_question": "What are used for protection by fish? (A) scales (B) fins (C) streams. (D) coral (E) gills (F) Collagen (G) mussels (H) whiskers"}
{"id": "3Z2R0DQ0JHDKFAO2706OYIXGNA4E28", "question": "What are pangolins covered in?", "choices": {"text": ["tunicates", "Echinoids", "shells", "exoskeleton", "blastoids", "barrel-shaped", "protection", "white"
], "label": ["A", "B", "C", "D", "E", "F", "G", "H"]}, "answerKey": "G", "fact1": "scales are used for protection by scaled animals", "fact2": "Pangolins have an elongate and tapering body covered above with ov
erlapping scales.", "combinedfact": "Pangolins are covered in overlapping protection.", "formatted_question": "What are pangolins covered in? (A) tunicates (B) Echinoids (C) shells (D) exoskeleton (E) blastoids
 (F) barrel-shaped (G) protection (H) white"}
{"id": "3PMBY0YE272GIWPNWIF8IH5RBHVC9S", "question": "What are covered with protection?", "choices": {"text": ["apples", "trees", "coral", "clams", "roses", "wings", "hats", "fish"], "label": ["A", "B", "C", "D
", "E", "F", "G", "H"]}, "answerKey": "H", "fact1": "scales are used for protection by scaled animals", "fact2": "Fish are covered with scales.", "combinedfact": "Fish are covered with protection", "formatted_q
uestion": "What are covered with protection? (A) apples (B) trees (C) coral (D) clams (E) roses (F) wings (G) hats (H) fish"}
```

Could you please load again your dataset and print its shape, like this:
```python
ds = load_dataset("qasc", split="train)
print(ds.shape)
```
and confirm which is your output?

Hmm .... it must have been a mistake on my side. Sorry for the hassle!

## Describe the bug
The training instances are not loaded properly.

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset("qasc", script_version='1.10.2')

def load_instances(split):
    instances = dataset[split]
    print(f"split: {split} - size: {len(instances)}")
    for x in instances:
        print(json.dumps(x))

load_instances('test')
load_instances('validation')
load_instances('train')
```

##  results
For test and validation, we can see the examples in the output (which is good!):
```
split: test - size: 920
{"answerKey": "", "choices": {"label": ["A", "B", "C", "D", "E", "F", "G", "H"], "text": ["Anthax", "under water", "uterus", "wombs", "two", "moles", "live", "embryo"]}, "combinedfact": "", "fact1": "", "fact2": "", "formatted_question": "What type of birth do therian mammals have? (A) Anthax (B) under water (C) uterus (D) wombs (E) two (F) moles (G) live (H) embryo", "id": "3C44YUNSI1OBFBB8D36GODNOZN9DPA", "question": "What type of birth do therian mammals have?"}
{"answerKey": "", "choices": {"label": ["A", "B", "C", "D", "E", "F", "G", "H"], "text": ["Corvidae", "arthropods", "birds", "backbones", "keratin", "Jurassic", "front paws", "Parakeets."]}, "combinedfact": "", "fact1": "", "fact2": "", "formatted_question": "By what time had mouse-sized viviparous mammals evolved? (A) Corvidae (B) arthropods (C) birds (D) backbones (E) keratin (F) Jurassic (G) front paws (H) Parakeets.", "id": "3B1NLC6UGZVERVLZFT7OUYQLD1SGPZ", "question": "By what time had mouse-sized viviparous mammals evolved?"}
{"answerKey": "", "choices": {"label": ["A", "B", "C", "D", "E", "F", "G", "H"], "text": ["Reduced friction", "causes infection", "vital to a good life", "prevents water loss", "camouflage from consumers", "Protection against predators", "spur the growth of the plant", "a smooth surface"]}, "combinedfact": "", "fact1": "", "fact2": "", "formatted_question": "What does a plant's skin do? (A) Reduced friction (B) causes infection (C) vital to a good life (D) prevents water loss (E) camouflage from consumers (F) Protection against predators (G) spur the growth of the plant (H) a smooth surface", "id": "3QRYMNZ7FYGITFVSJET3PS0F4S0NT9", "question": "What does a plant's skin do?"}
...
```
However, only a few instances are loaded for the training split, which is not correct.

## Environment info
- `datasets` version: '1.10.2'
- Platform: MaxOS
- Python version:3.7
- PyArrow version: 3.0.0


404 Not Found Error when loading LAMA dataset

Hi @dwil2444! I was able to reproduce your error when I downgraded to v1.1.2. Updating to the latest version of Datasets fixed the error for me :)

Hi @dwil2444, thanks for reporting.

Could you please confirm which `datasets` version you were using and if the problem persists after you update it to the latest version: `pip install -U datasets`?

Thanks @stevhliu for the hint to fix this! ;)

@stevhliu @albertvillanova  updating to the latest version of datasets did in fact fix this issue. Thanks a lot for your help!

The [LAMA](https://huggingface.co/datasets/viewer/?dataset=lama) probing dataset is not available for download:

Steps to Reproduce:

1. `from datasets import load_dataset`
2. `dataset = load_dataset('lama', 'trex')`.

Results:
`FileNotFoundError: Couldn't find file locally at lama/lama.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1.2/datasets/lama/lama.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/lama/lama.py`


Update BibTeX entry

Update BibTeX entry.


404 not found error on loading WIKIANN dataset

Something isn't working

Hi @ronbutan, thanks for reporting.

You are right: we have recently found that the link to the original PAN-X dataset (also called WikiANN), hosted at Dropbox, is no longer working.

We have opened an issue in the GitHub repository of the original dataset (afshinrahimi/mmner#4) and we have also contacted the author by email to ask if they are planning to fix this issue. See the details here: https://github.com/huggingface/datasets/issues/2691#issuecomment-885463027

I close this issue because it is the same as in #2691. Feel free to subscribe to that other issue to be informed about any updates.

## Describe the bug
Unable to retreive wikiann English dataset

## Steps to reproduce the bug
```python
from datasets import list_datasets, load_dataset, list_metrics, load_metric
WIKIANN = load_dataset("wikiann","en")
```

## Expected results
Colab notebook should display successful download status

## Actual results
FileNotFoundError: Couldn't find file at https://www.dropbox.com/s/12h3qqog6q4bjve/panx_dataset.tar?dl=1

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.10.1
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.11
- PyArrow version: 3.0.0


Fix pick default config name message

The error message to tell which config name to load is not displayed.

This is because in the code it was considering the config kwargs to be non-empty, which is a special case for custom configs created on the fly. It appears after this change: https://github.com/huggingface/datasets/pull/2659

I fixed that by making the config kwargs empty by default, even if default parameters are passed

Fix https://github.com/huggingface/datasets/issues/2703


Bad message when config name is missing

When loading a dataset that have several configurations, we expect to see an error message if the user doesn't specify a config name.

However in `datasets` 1.10.0 and 1.10.1 it doesn't show the right message:

```python
import datasets

datasets.load_dataset("glue")
```
raises
```python
AttributeError: 'BuilderConfig' object has no attribute 'text_features'
```
instead of
```python
ValueError: Config name is missing.
Please pick one among the available configs: ['cola', 'sst2', 'mrpc', 'qqp', 'stsb', 'mnli', 'mnli_mismatched', 'mnli_matched', 'qnli', 'rte', 'wnli', 'ax']
Example of usage:
        `load_dataset('glue', 'cola')`
```


Fix download_mode docstrings

documentation

Improvements or additions to documentation

Fix `download_mode` docstrings.


from datasets import Dataset is failing

bug

Something isn't working

Hi @kswamy15, thanks for reporting.

We are fixing this critical issue and making an urgent patch release of the `datasets` library today.

In the meantime, you can circumvent this issue by updating the `tqdm` library: `!pip install -U tqdm`

## Describe the bug
A clear and concise description of what the bug is.

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
from datasets import Dataset
```

## Expected results
A clear and concise description of the expected results.

## Actual results
Specify the actual results or traceback.
/usr/local/lib/python3.7/dist-packages/datasets/utils/file_utils.py in <module>()
     25 import posixpath
     26 import requests
---> 27 from tqdm.contrib.concurrent import thread_map
     28
     29 from .. import __version__, config, utils

ModuleNotFoundError: No module named 'tqdm.contrib.concurrent'

---------------------------------------------------------------------------
NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the
"Open Examples" button below.
---------------------------------------------------------------------------

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: latest version as of 07/21/2021
- Platform: Google Colab
- Python version: 3.7
- PyArrow version:


cannot combine splits merging and streaming?

Something isn't working

Hi ! That's missing indeed. We'll try to implement this for the next version :)

I guess we just need to implement #2564 first, and then we should be able to add support for splits combinations

this does not work:
`dataset = datasets.load_dataset('mc4','iw',split='train+validation',streaming=True)`
with error:
`ValueError: Bad split: train+validation. Available splits: ['train', 'validation']`

these work:
`dataset = datasets.load_dataset('mc4','iw',split='train+validation')`
`dataset = datasets.load_dataset('mc4','iw',split='train',streaming=True)`
`dataset = datasets.load_dataset('mc4','iw',split='validation',streaming=True)`

i could not find a reference to this in the documentation and the error message is confusing. also would be nice to allow streaming for the merged splits