rIsHu009/github-issues-updated huggingface dataset:
Title: Pubmed dataset not reachable
Author: abhi-mosaic (CONTRIBUTOR)
State: closed
Created: 2022-01-31T18:45:47Z
Labels: bug

## Describe the bug
Trying to use the `pubmed` dataset fails to reach / download the source files.

## Steps to reproduce the bug
```python
pubmed_train = datasets.load_dataset('pubmed', split='train')
```

## Expected results
Should begin downloading the pubmed dataset.

## Actual results
```
ConnectionError: Couldn't reach ftp://ftp.ncbi.nlm.nih.gov/pubmed/baseline/pubmed21n0865.xml.gz (InvalidSchema("No connection adapters were found for 'ftp://ftp.ncbi.nlm.nih.gov/pubmed/baseline/pubmed21n0865.xml.gz'"))
```

## Environment info
- `datasets` version: 1.18.2
- Platform: macOS-11.4-x86_64-i386-64bit
- Python version: 3.8.2
- PyArrow version: 6.0.0

Comments:
Hi @abhi-mosaic, thanks for reporting.

I'm looking at it... 

also hitting this issue

Hey @albertvillanova, sorry to reopen this... I can confirm that on `master` branch the dataset is downloadable now but it is still broken in streaming mode:

```python
 >>> import datasets
 >>> pubmed_train = datasets.load_dataset('pubmed', split='train', streaming=True)
 >>> next(iter(pubmed_train))
```
```
 No such file or directory: 'gzip://pubmed22n0001.xml::ftp://ftp.ncbi.nlm.nih.gov/pubmed/baseline/pubmed22n0001.xml.gz'
```


Hi @abhi-mosaic, would you mind opening another issue for this new problem?

First issue (already solved) was a ConnectionError due to the yearly update release of PubMed: we fixed it by updating the URLs from year 2021 to year 2022.

However this is another problem: to make pubmed streamable. Please note that NOT all our datastes are streamable: we are making streamable more and more of them... but this is an on-going process...

Thanks.

@albertvillanova 
When I tried below codes, I got the similar error

```

dataset=load_dataset("pubmed",split="train")

Couldn't reach ftp://ftp.ncbi.nlm.nih.gov/pubmed/baseline/pubmed21n0601.xml.gz
```

@y-rok you need to update `datasets`:
```shell
pip install -U datasets
```
Title: Better TQDM output
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2022-01-31T17:22:43Z

This PR does the following:
* if `dataset_infos.json` exists for a dataset, uses `num_examples` to print the total number of examples that needs to be generated (in `builder.py`)
* fixes `tqdm` + multiprocessing in Jupyter Notebook/Colab (the issue stems from this commit in the `tqdm` repo: https://github.com/tqdm/tqdm/commit/f7722edecc3010cb35cc1c923ac4850a76336f82) 
* adds the missing `drop_last_batch` and `with_ranks` params to `DatasetDict.map` 
* correctly computes the number of iterations in `map` and the CSV/JSON loader when `batched=True` to fix `tqdm` progress bars
* removes the `bool(logging.get_verbosity() == logging.NOTSET)` (or simplifies `bool(logging.get_verbosity() == logging.NOTSET) or not utils.is_progress_bar_enabled()` to `not utils.is_progress_bar_enabled()`)  condition and uses `utils.is_progress_bar_enabled` to check if `tqdm` output is enabled (this comment from @stas00 explains why  the  `bool(logging.get_verbosity() == logging.NOTSET)` check is problematic: https://github.com/huggingface/transformers/issues/14889#issue-1087318463)

Fix #2630

Comments:
@lhoestq I've created a notebook for you to see the difference: https://colab.research.google.com/drive/1by3EqnoKvC2p-yKW4lPDGOFOZHyGVyeQ?usp=sharing.

Feel free to suggest better descriptions for the progress bars. 

If everything looks good, think we can merge.
Title: `to_json` in multiprocessing fashion sometimes deadlock
Author: thomasw21 (CONTRIBUTOR)
State: open
Created: 2022-01-31T09:35:07Z
Labels: bug

## Describe the bug

`to_json` in multiprocessing fashion sometimes deadlock, instead of raising exceptions. Temporary solution is to see that it deadlocks, and then reduce the number of processes or batch size in order to reduce the memory footprint.

As @lhoestq pointed out, this might be related to https://bugs.python.org/issue22393#msg315684 where `multiprocessing` fails to raise the OOM exception. One suggested alternative is not use `concurrent.futures` instead.

## Steps to reproduce the bug

## Expected results

Script fails when one worker hits OOM, and raise appropriate error.

## Actual results

Deadlock

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.8.1
- Platform: Linux
- Python version: 3.8
- PyArrow version: 6.0.1

Comments:
Title: sp. Columbia => Colombia
Author: serapio (CONTRIBUTOR)
State: closed
Created: 2022-01-31T00:41:03Z

"Columbia" is various places in North America. The country is "Colombia".

Comments:
The original openslr site mixed both names https://openslr.org/72/ :-)

Yeah, I filed the issue to have it fixed there last year, but it looks like they missed a few.
Title: Update link in wiki_bio dataset
Author: jxmorris12 (CONTRIBUTOR)
State: closed
Created: 2022-01-30T16:28:54Z

Fixes #3580 and makes the wiki_bio dataset work again. I changed the link and some documentation, and all the tests pass. Thanks @lhoestq for uploading the dataset to the HuggingFace data bucket.

@lhoestq -- all the tests pass, but I'm still not able to import the dataset, as the old Google Drive link is cached somewhere:
```python
>>> from datasets import load_dataset
load_dataset("wiki_bio>>> load_dataset("wiki_bio")
Using custom data configuration default
Downloading and preparing dataset wiki_bio/default (download: 318.53 MiB, generated: 736.94 MiB, post-processed: Unknown size, total: 1.03 GiB) to /home/jxm3/.cache/huggingface/datasets/wiki_bio/default/1.1.0/5293ce565954ba965dada626f1e79684e98172d950371d266bf3caaf87e911c9...
Traceback (most recent call last):
  ...
  File "/home/jxm3/random/datasets/src/datasets/utils/file_utils.py", line 612, in get_from_cache
    raise FileNotFoundError(f"Couldn't find file at {url}")
FileNotFoundError: Couldn't find file at https://drive.google.com/uc?export=download&id=1L7aoUXzHPzyzQ0ns4ApBbYepsjFOtXil
```

what do I have to do to invalidate the cache and actually import the dataset? It's clearly set up correctly, since the data is downloaded and processed by the tests.

As an aside, this caching-loading-scripts behavior makes for a really bad developer experience. I just wasted an hour trying to figure out where the caching was happening and how to disable it, and I don't know. All I wanted to do was update the link and submit a pull request! I recommend that you all either change this behavior (i.e. updating the link to a dataset should "just work") or document it, since I couldn't find any information about this in the contributing.md or readme or anywhere else! Thanks!

Comments:
> all the tests pass, but I'm still not able to import the dataset

Since it's not merged on `master` yet, you have to provide the path to your local `wiki_bio.py` to use it.
Indeed the library downloads the dataset files from `master` if you have a dev installation of the library.

I agree it would be nice to change that, and use the local dataset scripts from the `datasets` directory - it feels definitely more natural.

Cool, thanks for your help and I agree!
Title: Allow 'to_json' to run in unordered fashion in order to lower memory footprint
Author: thomasw21 (CONTRIBUTOR)
State: closed
Created: 2022-01-30T13:23:19Z

I'm using `to_json(..., num_proc=num_proc, compressiong='gzip')` with `num_proc>1`. I'm having an issue where things seem to deadlock at some point. Eventually I see OOM. I'm guessing it's an issue where one process starts to take a long time for a specific batch, and so other process keep accumulating their results in memory.

In order to flush memory, I propose we use optional `imap_unordered`. This will prevent one process to block the other ones. The logical thinking is that index are rarily relevant, and in one wants to keep an index, one can still create another column and reconstruct from there.

Comments:
Hi @thomasw21, I remember suggesting `imap_unordered` to @lhoestq at that time to speed up `to_json` further but after trying `pool_imap` on multiple datasets (>9GB) , memory utilisation was almost constant and we decided to go ahead with that only. 

1. Did you try this without `gzip`? Because `gzip`  feature was introduced recently and I didn't check multi_proc thing with `gzip`. One thing I know is that `gzip` is slow in our implementation than `zip` (it's a WIP #3551) 
2. You can try reducing your batch size, this can also help in avoiding OOM errors!

Thanks @bhavitvyamalik ! I see. I'm not sure this PR actually fixes things for me either (I ended up reducing the num_proc/batch_size to lower it). It does allow the process to run for longer, but I think the reason why it was waiting is that one of the process crashes .... Unfortunately I was working on a setup with a low RAM/cpu core ratio. I'm actually very surprised that it doesn't change memory utilization, otherwise I don't see the purpose of `imap_unordered` existing. I think it's main purpose are when you have high variance in samples (in terms of bytes), which causes unecessary accumulation in `imap`
  1. Did not try without `gzip`
  2. Yeah or `num_proc`

Can you please try without `gzip` to see how it performs? If it works fine then we can improve `gzip` from our side (I'm already working on it)

I'll be busy for next few weeks on another project, will do as soon as I have some bandwidth.


Should we close this PR?

Yes we can close this PR if considered unneeded.
Title: Add IGLUE dataset
Author: lewtun (MEMBER)
State: open
Created: 2022-01-28T14:59:41Z
Labels: dataset request

multimodal

## Adding a Dataset
- **Name:** IGLUE
- **Description:** IGLUE brings together 4 vision-and-language tasks across 20 languages (Twitter [thread](https://twitter.com/ebugliarello/status/1487045497583976455?s=20&t=SB4LZGDhhkUW83ugcX_m5w))
- **Paper:** https://arxiv.org/abs/2201.11732
- **Data:** https://github.com/e-bug/iglue
- **Motivation:** This dataset would provide a nice example of combining the text and image features of `datasets` together for multimodal applications.

Note: the data / code are not yet visible on the GitHub repo, so I've pinged the authors for more information.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Comments:
Title: Fix `add_column` on datasets with indices mapping
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2022-01-28T13:06:29Z

My initial idea was to avoid the `flatten_indices` call and reorder a new column instead, but in the end I decided to follow `concatenate_datasets` and use `flatten_indices` to avoid padding when `dataset._indices.num_rows != dataset._data.num_rows`.

Fix #3599

Comments:
Sure, let's include this in today's release.

Cool ! The windows CI should be fixed on master now, feel free to merge :)
Title: Fix streaming datasets that are not reset correctly
Author: lhoestq (MEMBER)
State: closed
Created: 2022-01-27T17:21:02Z

Streaming datasets that use `StreamingDownloadManager.iter_archive` and `StreamingDownloadManager.iter_files` had some issues. Indeed if you try to iterate over such dataset twice, then the second time it will be empty.

This is because the two methods above are generator functions. I fixed this by making them return iterables that are reset properly instead.

Close https://github.com/huggingface/datasets/issues/3645
cc @anton-l

Comments:
Works smoothly with the `transformers.Trainer` class now, thank you!
Title: Streaming dataset based on dl_manager.iter_archive/iter_files are not reset correctly
Author: lhoestq (MEMBER)
State: closed
Created: 2022-01-27T17:17:41Z

Hi ! When iterating over a streaming dataset once, it's not reset correctly because of some issues with `dl_manager.iter_archive` and `dl_manager.iter_files`. Indeed they are generator functions (so the iterator that is returned can be exhausted). They should be iterables instead, and be reset if we do a for loop again:

```python
from datasets import load_dataset
d = load_dataset("common_voice", "ab", split="test", streaming=True)

i = 0
for i, _ in enumerate(d):
    pass
print(i)  # 8
# let's do it again
i = 0
for i, _ in enumerate(d):
    pass
print(i)  # 0
```

Comments:
Title: Fix sem_eval_2018_task_1 download location
Author: maxpel (CONTRIBUTOR)
State: closed
Created: 2022-01-27T15:45:00Z

As discussed with @lhoestq in https://github.com/huggingface/datasets/issues/3549#issuecomment-1020176931_ this is the new pull request to fix the download location.

Comments:
I fixed those two things, the two remaining failing checks seem to be due to some dependency missing in the tests.
Title: Fix numpy rngs when seed is None
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2022-01-27T14:29:09Z

Fixes the NumPy RNG when `seed` is `None`.

The problem becomes obvious after reading the NumPy notes on RNG (returned by `np.random.get_state()`):
> The MT19937 state vector consists of a 624-element array of 32-bit unsigned integers plus a single integer value between 0 and 624 that indexes the current position within the main array.

`The MT19937 state vector`: the seed which we currently index, but this value stays the same for multiple rounds.
`plus a single integer value`: the `pos` value in this PR (is 624 if `seed` is set to a fixed value with `np.random.seed`, so we take the first value in the `seed` array returned by `np.random.get_state()`: https://stackoverflow.com/questions/32172054/how-can-i-retrieve-the-current-seed-of-numpys-random-number-generator)

NumPy notes: https://numpy.org/doc/stable/reference/random/bit_generators/mt19937.html

Fix #3634

Comments:
Title: Issues with custom dataset in Wav2Vec2
Author: peregilk (NONE)
State: closed
Created: 2022-01-27T12:09:05Z
Labels: bug

We are training Vav2Vec using the run_speech_recognition_ctc_bnb.py-script.

This is working fine with Common Voice, however using our custom dataset and data loader at [NbAiLab/NPSC]( https://huggingface.co/datasets/NbAiLab/NPSC) it crashes after roughly 1 epoch with the following stack trace:

![image](https://user-images.githubusercontent.com/9079808/151355893-6d5887cc-ca19-4b12-948a-124eb6dac372.png)


We are able to work around the issue, for instance by adding this check in line#222 in transformers/models/wav2vec2/modeling_wav2vec2.py:
```python
if input_length - (mask_length - 1) < num_masked_span:
           num_masked_span = input_length - (mask_length - 1)
```
Interestingly, these are the variable values before the adjustment:
```
input_length=10
mask_length=10
num_masked_span=2
````
After adjusting num_masked_spin to 1, the training script runs. The issue is also fixed by setting “replace=True” in the same function.

Do you have any  idea what is causing this, and how to fix this error permanently? If you do not think this is an Datasets issue, feel free to move the issue.

Comments:
Closed and moved to transformers.
Title: Adding CC-100: Monolingual Datasets from Web Crawl Data (Datasets links are invalid)
Author: AnzorGozalishvili (CONTRIBUTOR)
State: closed
Created: 2022-01-26T13:35:37Z
Labels: bug

## Describe the bug
The dataset links are no longer valid for CC-100. It seems that the website which was keeping these files are no longer accessible and therefore this dataset became unusable. 
Check out the dataset [homepage](http://data.statmt.org/cc-100/)  which isn't accessible.
Also the URLs for dataset file per language isn't accessible: http://data.statmt.org/cc-100/<language code here>.txt.xz (language codes: am, sr, ka, etc.)

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("cc100", "ka")
```
It throws 503 error.

## Expected results
It should successfully download and load dataset but it throws an exception because the dataset files are no longer accessible.


## Environment info
Run from google colab. Just installed the library using pip:
```!pip install -U datasets```

Comments:
Hi @AnzorGozalishvili,

Maybe their site was temporarily down, but it seems to work fine now.

Could you please try again and confirm if the problem persists? 

Hi @albertvillanova 
I checked and it works. 
It seems that it was really temporarily down.
Thanks!
Title: DuplicatedKeysError of NewsQA dataset
Author: StevenTang1998 (NONE)
State: closed
Created: 2022-01-26T03:05:49Z
Labels: dataset bug

After processing the dataset following official [NewsQA](https://github.com/Maluuba/newsqa), I used datasets to load it:
```
a = load_dataset('newsqa', data_dir='news')
```
and the following error occurred: 
```
Using custom data configuration default-data_dir=news
Downloading and preparing dataset newsqa/default to /root/.cache/huggingface/datasets/newsqa/default-data_dir=news/1.0.0/b0b23e22d94a3d352ad9d75aff2b71375264a122fae301463079ee8595e05ab9...
Traceback (most recent call last):        
  File "/usr/local/lib/python3.8/dist-packages/datasets/builder.py", line 1084, in _prepare_split
    writer.write(example, key)
  File "/usr/local/lib/python3.8/dist-packages/datasets/arrow_writer.py", line 442, in write
    self.check_duplicate_keys()
  File "/usr/local/lib/python3.8/dist-packages/datasets/arrow_writer.py", line 453, in check_duplicate_keys
    raise DuplicatedKeysError(key)
datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: ./cnn/stories/6a0f9c8a5d0c6e8949b37924163c92923fe5770d.story
Keys should be unique and deterministic in nature

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/dist-packages/datasets/load.py", line 1694, in load_dataset
    builder_instance.download_and_prepare(
  File "/usr/local/lib/python3.8/dist-packages/datasets/builder.py", line 595, in download_and_prepare
    self._download_and_prepare(
  File "/usr/local/lib/python3.8/dist-packages/datasets/builder.py", line 684, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/usr/local/lib/python3.8/dist-packages/datasets/builder.py", line 1086, in _prepare_split
    num_examples, num_bytes = writer.finalize()
  File "/usr/local/lib/python3.8/dist-packages/datasets/arrow_writer.py", line 524, in finalize
    self.check_duplicate_keys()
  File "/usr/local/lib/python3.8/dist-packages/datasets/arrow_writer.py", line 453, in check_duplicate_keys
    raise DuplicatedKeysError(key)
datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: ./cnn/stories/6a0f9c8a5d0c6e8949b37924163c92923fe5770d.story
Keys should be unique and deterministic in nature
```

Comments:
Thanks for reporting, @StevenTang1998.

I'm fixing it.
Title: Dataset Card Creator drops information for "Additional Information" Section
Author: dennlinger (NONE)
State: open
Created: 2022-01-25T14:06:17Z
Labels: bug

First of all, the card creator is a great addition and really helpful for streamlining dataset cards!

## Describe the bug
I encountered an inconvenient bug when entering "Additional Information" in the react app, which drops already entered text when switching to a previous section, and then back again to "Additional Information". I was able to reproduce the issue in both Firefox and Chrome, so I suspect a problem with the React logic that doesn't expect users to switch back in the final section.  
Edit: I'm also not sure whether this is the right place to open the bug report on, since it's not clear to me which particular project it belongs to, or where I could find associated source code.

## Steps to reproduce the bug

1. Navigate to the Section "Additional Information" in the [dataset card creator](https://huggingface.co/datasets/card-creator/)
2. Enter text in an arbitrary field, e.g., "Dataset Curators".
3. Switch back to a previous section, like "Dataset Creation".
4. When switching back again to "Additional Information", the text has been deleted.

Notably, this behavior can be reproduced again and again, it's not just problematic for the first "switch-back" from Additional Information.

## Expected results
For step 4, the previously entered information should still be present in the boxes, similar to the behavior to all other sections (switching back there works as expected)

## Actual results
The text boxes are empty again, and previously entered text got deleted.

## Environment info
- `datasets` version: N/A
- Platform: Firefox 96.0 / Chrome 97.0
- Python version: N/A
- PyArrow version: N/A

Comments:
Title: Fix host URL in The Pile datasets
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-01-25T08:11:28Z

This PR fixes the host URL in The Pile datasets, once they have mirrored their data in another server.

Fix #3626.

Comments:
We should also update the `bookcorpusopen` download url (see #3561) , no? 

For `the_pile_openwebtext2` and `the_pile_stack_exchange` I did not regenerate the JSON files, but instead I just changed the download_checksums URL. 

Seems like the mystic URL is now broken and the original should be used. 

Also if I git clone and edit the repo or reset it before this PR it is still trying to pull using mystic? Why is this?
Title: Extend support for streaming datasets that use os.path.relpath
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-01-24T16:00:52Z

This PR extends the support in streaming mode for datasets that use `os.path.relpath`, by patching that function.

This feature will also be useful to yield the relative path of audio or image files, within an archive or parent dir.

Close #3622.

Comments:
Title: Consider adding `ipywidgets` as a dependency.
Author: koaning (NONE)
State: closed
Created: 2022-01-24T14:27:11Z
Labels: bug

When I install `datasets` in a fresh virtualenv with jupyterlab I always see this error. 

```
ImportError: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
```

It's a bit of a nuisance, because I need to run shut down the jupyterlab server in order to install the required dependency. Might it be an option to just include it as a dependency here?

Comments:
Hi! We use `tqdm` to display progress bars, so I suggest you open this issue in their repo.

It depends on how you use `tqdm`, no? 

Doesn't this library import via; 

```
from tqdm.notebook import tqdm
```

Hi! Sorry for the late reply. We import `tqdm` as `from tqdm.auto import tqdm`, which should be equal to `from tqdm.notebook import tqdm` in Jupyter.

Any objection if I make a PR that checks if the widgets library is installed beforehand?
Title: fix meta in mls
Author: polinaeterna (CONTRIBUTOR)
State: closed
Created: 2022-01-24T12:54:38Z

`monolingual` value of `m ultilinguality` param in yaml meta was changed to `multilingual` :)

Comments:
Feel free to merge @polinaeterna as soon as you got an approval from either @lhoestq , @albertvillanova or @mariosasko
Title: PR for the CFPB Consumer Complaints dataset
Author: kayvane1 (CONTRIBUTOR)
State: closed
Created: 2022-01-23T17:47:12Z

Think I followed all the steps but please let me know if anything needs changing or any improvements I can make to the code quality

Comments:
> Nice ! Thanks for adding this dataset :)
> 
> 
> 
> I left a few comments:

Thanks!

I'd be interested in contributing to the core codebase - I had to go down the custom loading approach because I couldn't pull this dataset in using the load_dataset() method. Using either the json or csv files available for this dataset as it was erroring. 

I'll rerun it and share the errors and try debug

Hey @lhoestq ,

When I use this dataset as part of my project, I'm using this method

`text_dataset = text_dataset['train'].train_test_split(test_size=0.2)`

to create a train and test split as this dataset doesn't have one. 

Can I add this directly in the script itself somehow, or is it better to give users the flexibility to slice and split their datasets after loading?

> I'd be interested in contributing to the core codebase - I had to go down the custom loading approach because I couldn't pull this dataset in using the load_dataset() method. Using either the json or csv files available for this dataset as it was erroring.
>
> I'll rerun it and share the errors and try debug

Cool ! Let me know if you have questions or if I can help :)

> Can I add this directly in the script itself somehow, or is it better to give users the flexibility to slice and split their datasets after loading?

Usually we let the users the flexibility to split the datasets themselves (unless the dataset is already split, or if there is already a standard way to split it in the papers that use it)

Thanks Quentin!
All okay to merge now?

Thanks for the feedback Quentin and Mario - implemented all changes  :)
![Screenshot 2022-01-31 at 23 11 20](https://user-images.githubusercontent.com/42403093/151889262-30737feb-ac9c-4c5a-9326-9812db1d05bc.png)


Hey @lhoestq  / @mariosasko 
Any other changes required to merge? 🤗

Hi ! Thanks and sorry for the late response 

It looks very good ! The CI is still failing because it can't file the dummy_data.zip file, you can fix that by moving `datasets/consumer-finance-complaints/dummy/1.0.0/dummy_data.zip` to `datasets/consumer-finance-complaints/dummy/0.0.0/dummy_data.zip` and it should be all good !

@lhoestq - hopefully that should do it!
Title: Dataset BnL Historical Newspapers does not work in streaming mode
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-01-22T14:12:59Z
Labels: bug

## Describe the bug
When trying to load in streaming mode, it "hangs"...

## Steps to reproduce the bug
```python
ds = load_dataset("bnl_newspapers", split="train", streaming=True)
```

## Expected results
The code should be optimized, so that it works fast in streaming mode.

CC: @davanstrien

Comments:
@albertvillanova let me know if there is anything I can do to help with this. I had a quick look at the code again and though I could try the following changes:
- use `download` instead of `download_and_extract`
https://github.com/huggingface/datasets/blob/d3d339fb86d378f4cb3c5d1de423315c07a466c6/datasets/bnl_newspapers/bnl_newspapers.py#L136
- swith to using `iter_archive` to loop through downloaded data to replace
https://github.com/huggingface/datasets/blob/d3d339fb86d378f4cb3c5d1de423315c07a466c6/datasets/bnl_newspapers/bnl_newspapers.py#L159

Let me know if it's useful for me to try and make those changes. 

Thanks @davanstrien.

I have already been working on it so that it can be used in the BigScience workshop.

I agree that the `rglob()` is not efficient in this case.

I tried different solutions without success:
- `iter_archive` cannot be used in this case because it does not support ZIP files yet

Finally I have used `iter_files()`.

I see this is fixed now 🙂.  I also picked up a few other tips from your redactors so hopefully my next attempts will support streaming from the start.
Title: Files not updating in dataset viewer
Author: abidlabs (MEMBER)
State: closed
Created: 2022-01-21T16:47:20Z
Labels: dataset-viewer

## Dataset viewer issue for '*name of the dataset*'

**Link:**
Some examples:
* https://huggingface.co/datasets/abidlabs/crowdsourced-speech4
* https://huggingface.co/datasets/abidlabs/test-audio-13

*short description of the issue*
It seems that the dataset viewer is reading a cached version of the dataset and it is not updating to reflect new files that are added to the dataset. I get this error:

![image](https://user-images.githubusercontent.com/1778297/150566660-30dc0dcd-18fd-4471-b70c-7c4bdc6a23c6.png)


Am I the one who added this dataset? Yes

Comments:
Yes. The jobs queue is full right now, following an upgrade... Back to normality in the next hours hopefully. I'll look at your datasets to be sure the dataset viewer works as expected on them.

Should have been fixed now.
Title: wikifix
Author: apergo-ai (CONTRIBUTOR)
State: closed
Created: 2022-01-21T14:05:11Z

This should get the wikipedia dataloading script back up and running - at least I hope so (tested with language ff and ii)

Comments:
tests fail because of dataset_infos.json isn't updated. Unfortunately, I cannot get the datasets-cli locally to execute without error. Would need to troubleshoot, what's missing. Maybe someone else can pick up the stick. 

Hi ! If we change the default date to the latest one, users won't be able to load the "big" languages like english anymore, because it requires an Apache Beam runtime to process them. On the contrary, the old data 20200501 has been processed by Hugging Face so that users don't need to run Apache Beam stuff.

Therefore I'm in favor of not changing the default date until we have processed the latest versions of wikipedia.

Users that want to load other languages or that can use Apache Beam can still pass the `language` and `date` parameter to `load_dataset` if they want anyway:
```python
load_dataset("wikipedia", language="fr", date="20220120")
```

in that case you can close the PR

Ok thanks !

(oh I I just noticed that the dataset card is missing the documentation regarding the language and date parameters, let me add it)
Title: Indexing bug after dataset.select()
Author: kamalkraj (NONE)
State: closed
Created: 2022-01-21T12:09:30Z
Labels: bug

## Describe the bug
A clear and concise description of what the bug is.
 
Dataset indexing is not working as expected after `dataset.select(range(100))`


## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
import datasets

task_to_keys = {
    "cola": ("sentence", None),
    "mnli": ("premise", "hypothesis"),
    "mrpc": ("sentence1", "sentence2"),
    "qnli": ("question", "sentence"),
    "qqp": ("question1", "question2"),
    "rte": ("sentence1", "sentence2"),
    "sst2": ("sentence", None),
    "stsb": ("sentence1", "sentence2"),
    "wnli": ("sentence1", "sentence2"),
}

task_name = "sst2"
raw_datasets = datasets.load_dataset("glue", task_name)


train_dataset = raw_datasets["train"]

print("before select: ",train_dataset[-2:])
# before select:  {'sentence': ['a patient viewer ', 'this new jangle of noise , mayhem and stupidity must be a serious contender for the title . '], 'label': [1, 0], 'idx': [67347, 67348]}

train_dataset = train_dataset.select(range(100))

print("after select: ",train_dataset[-2:])
# after select:  {'sentence': [], 'label': [], 'idx': []}

```

link to colab: https://colab.research.google.com/drive/1LngeRC9f0jE7eSQ4Kh1cIeb411lRXQD-?usp=sharing

## Expected results
A clear and concise description of the expected results.
showing 98, 99 index data

## Actual results
Specify the actual results or traceback.
empty

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.17.0
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 3.0.0

Comments:
Hi! Thanks for reporting! I've opened a PR with the fix.
Title: Checksum error when trying to load amazon_review dataset
Author: ghost (NONE)
State: closed
Created: 2022-01-20T21:20:32Z
Labels: bug

## Describe the bug
A clear and concise description of what the bug is.

## Steps to reproduce the bug
I am getting the issue when trying to load dataset using
```
dataset = load_dataset("amazon_polarity")
```

## Expected results
dataset loaded

## Actual results
```
---------------------------------------------------------------------------
NonMatchingChecksumError                  Traceback (most recent call last)
<ipython-input-3-b4758ba980ae> in <module>()
----> 1 dataset = load_dataset("amazon_polarity")
      2 dataset.set_format(type='pandas')
      3 content_series = dataset['train']['content']
      4 label_series = dataset['train']['label']
      5 df = pd.concat([content_series, label_series], axis=1)

3 frames
/usr/local/lib/python3.7/dist-packages/datasets/utils/info_utils.py in verify_checksums(expected_checksums, recorded_checksums, verification_name)
     38     if len(bad_urls) > 0:
     39         error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> 40         raise NonMatchingChecksumError(error_msg + str(bad_urls))
     41     logger.info("All the checksums matched successfully" + for_verification_name)
     42 

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/u/0/uc?id=0Bz8a_Dbh9QhbaW12WVVZS2drcnM&export=download']
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.17.0
- Platform: Google colab
- Python version: 3.7.12

Comments:
It is solved now
Title: Fixes to pubmed dataset download function
Author: spacemanidol (NONE)
State: closed
Created: 2022-01-20T17:31:35Z

Pubmed has updated its settings for 2022 and thus existing download script does not work.

Comments:
Hi ! I think we can simply add a new configuration for the 2022 data instead of replacing them.
You can add the new configuration here:
```python
    BUILDER_CONFIGS = [
        datasets.BuilderConfig(name="2021", description="The 2021 annual record", version=datasets.Version("1.0.0")),
        datasets.BuilderConfig(name="2022", description="The 2022 annual record", version=datasets.Version("1.0.0")),
    ]
```

And we can have the URLs for these two versions this way:
```python
_URLs = {
    "2021": f"ftp://ftp.ncbi.nlm.nih.gov/pubmed/baseline/pubmed21n{i:04d}.xml.gz" for i in range(1, 1063)],
    "2022": f"ftp://ftp.ncbi.nlm.nih.gov/pubmed/baseline/pubmed22n{i:04d}.xml.gz" for i in range(1, 1114)]
}
```
and depending on the configuration name (you can get it with `self.config.name`) we can pick the URLs of 2021 or the ones of 2022 and pass them to the `dl_manager` in `_split_generators`

Feel free to ping me if you have questions or if I can help !

Hi @spacemanidol, thanks for your contribution.

The update of the PubMed dataset URL (besides the update of the corresponding metadata and the dummy data) was already merged to master branch in this other PR:
- #3692 

I'm closing this PR then.

@lhoestq please take into account that 2021 data is no longer accessible: every year PubMed releases the baseline data (containing all previous data until that year) and from that on, they release daily updates.  

> @lhoestq please take into account that 2021 data is no longer accessible: every year PubMed releases the baseline data (containing all previous data until that year) and from that on, they release daily updates.

Oh ok I didn't know, thanks
Title: Add support for continuous metrics (RMSE, MAE)
Author: ck37 (NONE)
State: closed
Created: 2022-01-20T13:35:36Z
Labels: enhancement

good first issue

**Is your feature request related to a problem? Please describe.**

I am uploading our dataset and models for the "Constructing interval measures" method we've developed, which uses item response theory to convert multiple discrete labels into a continuous spectrum for hate speech. Once we have this outcome our NLP models conduct regression rather than classification, so binary metrics are not relevant. The only continuous metrics available at https://huggingface.co/metrics are pearson & spearman correlation, which don't ensure that the prediction is on the same scale as the outcome.

**Describe the solution you'd like**
I would like to be able to tag our models on the Hub with the following metrics:
- RMSE
- MAE

**Describe alternatives you've considered**

I don't know if there are any alternatives.

**Additional context**
Our preprint is available here: https://arxiv.org/abs/2009.10277 . We are making it available for use in Jigsaw's Toxic Severity Rating Kaggle competition: https://www.kaggle.com/c/jigsaw-toxic-severity-rating/overview . I have our first model uploaded to the Hub at https://huggingface.co/ucberkeley-dlab/hate-measure-roberta-large

Thanks,
Chris

Comments:
Hey @ck37 

You can always use a custom metric as explained [in this guide from HF](https://huggingface.co/docs/datasets/master/loading_metrics.html#using-a-custom-metric-script).

If this issue needs to be contributed to (for enhancing the metric API) I think [this link](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_error.html) would be helpful for the `MAE` metric.

You can use a local metric script just by providing its path instead of the usual shortcut name 

#self-assign I have starting working on this issue to enhance the metric API.
Title: Adding Turkic X-WMT evaluation set for machine translation
Author: mirzakhalov (CONTRIBUTOR)
State: closed
Created: 2022-01-20T01:40:29Z

This dataset is a human-translated evaluation set for MT crowdsourced and provided by the [Turkic Interlingua ](turkic-interlingua.org) community. It contains eval sets for 8 Turkic languages covering 88 language directions. Languages being covered are:

Azerbaijani (az)
Bashkir (ba)
English (en)
Karakalpak (kaa)
Kazakh (kk)
Kirghiz (ky)
Russian (ru)
Turkish (tr)
Sakha (sah)
Uzbek (uz)

More info about the corpus is here: [https://github.com/turkic-interlingua/til-mt/tree/master/xwmt](https://github.com/turkic-interlingua/til-mt/tree/master/xwmt)

A paper describing the test set is here: [https://arxiv.org/abs/2109.04593](https://arxiv.org/abs/2109.04593)

Comments:
hi! Thank you for all the comments! I believe I addressed them all. Let me know if there is anything else

Hi there! I was wondering if there is anything else to change before this can be merged

@lhoestq Hi! Just a gentle reminder about the steps to merge this one! 

Thanks for the heads up ! I think I fixed the last issue with the YAML tags

The CI failure is unrelated to this PR and fixed on master, let's merge :)

Thanks a lot !
Title: Dataset Viewer not showing Previews for Private Datasets
Author: abidlabs (MEMBER)
State: closed
Created: 2022-01-19T19:29:26Z
Labels: enhancement

dataset-viewer

## Dataset viewer issue for 'abidlabs/test-audio-13'

It seems that the dataset viewer does not show previews for `private` datasets, even for the user who's private dataset it is. See [1] for example. If I change the visibility to public, then it does show, but it would be useful to have the viewer even for private datasets.

![image](https://user-images.githubusercontent.com/1778297/150200515-93ff1545-11fd-4793-be64-6bed3cd895e2.png)

**Link:** 

[1] https://huggingface.co/datasets/abidlabs/test-audio-13

**Am I the one who added this dataset?** 

Yes

Comments:
Sure, it's on the roadmap.

Closing in favor of https://github.com/huggingface/datasets-server/issues/39.
Title: Add British Library books dataset
Author: davanstrien (MEMBER)
State: closed
Created: 2022-01-19T17:53:05Z

This pull request adds a dataset of text from digitised (primarily 19th Century) books from the British Library. This collection has previously been used for training language models, e.g. https://github.com/dbmdz/clef-hipe/blob/main/hlms.md. It would be nice to make this dataset more accessible for others to use through datasets. 

This is still a WIP but I wanted to get some initial feedback in particular; I wanted to check:
- I am handling the use of `iter_archive` correctly - I intend to ensure that `dl_manager.download` gets the complete list of URLs to download upfront, so the progress bar knows how much is left to download and then to pass through the `gen_kwargs` a list of downloaded zip archives wrapped in `iter_archive`. I am unsure if there is a more elegant approach for this?
- the number of configs: I have aimed to keep this limited - there are a lot of URLs covering the entire dataset, but I have tried to base the configs on what I believe the majority of people will want to they are not presented with too many options - I am happy to hear suggestions for changing this

If there are other glaring omissions or mistakes, I'd be happy to hear them. If this approach seems sensible in general, I will finish all the remaining TODOs, generate dummy_data, etc.

Comments:
Thanks for all the help and suggestions

> Since the dataset has a very specific structure it might not be that easy so feel free to ping me if you have questions or if I can help !

I did get a little stuck here! So far I have created directories for each config i.e:

`datasets/datasets/blbooks/dummy/1700_1799/1.0.2/dummy_data.zip` 

I have then added two examples of the `jsonl.gz` files that are in the underlying dataset to each dummy_data directory.This fails the test using local files. 

Since 

```python
def _generate_examples(self, data_dirs):
```

takes as input `data_dirs` which is a list of `iter_dirs` do I need to put the dummy files inside another directory? i.e. 

`datasets/datasets/blbooks/dummy/1700_1799/1.0.2/dummy_data/1700/00.jsonl.gz` 


 

I think I managed to create the dummy data :)

I think everything is good now, if you don't have other changes to do, please mark your PR as "ready for review" and ping me!

> I think I managed to create the dummy data :)

Thanks so much for that!

> I think everything is good now, if you don't have other changes to do, please mark your PR as "ready for review" and ping me!

Think it is ready to merge from my end @lhoestq. 

The CI failure on windows is unrelated to your PR and fixed on `master`, we can ignore it
Title: Update url for conll2003
Author: lhoestq (MEMBER)
State: closed
Created: 2022-01-19T15:35:04Z

Following https://github.com/huggingface/datasets/issues/3582 I'm changing the download URL of the conll2003 data files, since the previous host doesn't have the authorization to redistribute the data

Comments:
Hi. lhoestq 

![image](https://user-images.githubusercontent.com/21982975/150345097-154f2b1a-bb12-47af-bddf-40eec0a0dadb.png)
What is the solution for it?
you can see it is still doesn't work here.
https://colab.research.google.com/drive/1l52FGWuSaOaGYchit4CbmtUSuzNDx_Ok?usp=sharing
Thank you.


For now you can specify `load_dataset(..., revision="master")` to use the fix on `master`.

We'll also do a new release of `datasets` tomorrow I think
Title: Use old url for conll2003
Author: lhoestq (MEMBER)
State: closed
Created: 2022-01-19T13:56:49Z

As reported in https://github.com/huggingface/datasets/issues/3582 the CoNLL2003 data files are not available in the master branch of the repo that used to host them. 

For now we can use the URL from an older commit to access the data files

Comments:
Title: Readme info not being parsed to show on Dataset card page
Author: davidcanovas (NONE)
State: closed
Created: 2022-01-19T13:32:29Z
Labels: bug

## Describe the bug
The info contained in the README.md file is not being shown in the dataset main page. Basic info and table of contents are properly formatted in the README.

## Steps to reproduce the bug

# Sample code to reproduce the bug
The README file is this one: https://huggingface.co/datasets/softcatala/Tilde-MODEL-Catalan/blob/main/README.md

## Expected results
README info should appear in the Dataset card page.

## Actual results
Nothing is shown. However, labels are parsed and shown successfully.

Comments:
i suspect a markdown parsing error, @severo do you want to take a quick look at it when you have some time?

# Problem
The issue seems to coming from the front matter of the README
```---
annotations_creators:
- no-annotation
language_creators:
- machine-generated
languages:
- 'ca'
- 'de'
licenses:
- cc-by-4.0
multilinguality:
- translation
pretty_name: Catalan-German aligned corpora to train NMT systems.
size_categories:
- "1M<n<10M"            
source_datasets:
- extended|tilde_model
task_categories:
- machine-translation
task_ids:
- machine-translation
---
``` 
# Solution
The fix is to correctly style the README as explained [here](https://huggingface.co/docs/datasets/v1.12.0/dataset_card.html). I have also correctly parsed the font matter as shown below:
```
---
annotations_creators: []
language_creators: [machine-generated]
languages: ['ca', 'de']
licenses: []
multilinguality:
- multilingual
pretty_name: 'Catalan-German aligned corpora to train NMT systems.'
size_categories: 
- 1M<n<10M
source_datasets: ['extended|tilde_model']
task_categories: ['machine-translation']
task_ids: ['machine-translation']
---
```
You can find the README for a sample dataset [here](https://huggingface.co/datasets/ritwikraha/Test)

Thank you. It finally worked implementing your changes and leaving a white line between title and text in the description.

Thanks, if this solves your issue, can you please close it?
Title: ERROR: File "setup.py" or "setup.cfg" not found. Directory cannot be installed in editable mode: /content
Author: amitkml (NONE)
State: closed
Created: 2022-01-19T13:19:28Z
Labels: bug

## Bug
The install of streaming dataset is giving following error.

## Steps to reproduce the bug
```python
! git clone https://github.com/huggingface/datasets.git
! cd datasets
! pip install -e ".[streaming]"
```

## Actual results
Cloning into 'datasets'...
remote: Enumerating objects: 50816, done.
remote: Counting objects: 100% (2356/2356), done.
remote: Compressing objects: 100% (1606/1606), done.
remote: Total 50816 (delta 834), reused 1741 (delta 525), pack-reused 48460
Receiving objects: 100% (50816/50816), 72.47 MiB | 27.68 MiB/s, done.
Resolving deltas: 100% (22541/22541), done.
Checking out files: 100% (6722/6722), done.
ERROR: File "setup.py" or "setup.cfg" not found. Directory cannot be installed in editable mode: /content

Comments:
Hi! The `cd` command in Jupyer/Colab needs to start with `%`, so this should work:
```
!git clone https://github.com/huggingface/datasets.git
%cd datasets
!pip install -e ".[streaming]"
```

thanks @mariosasko i had the same mistake and your solution is what was needed
Title: Add ImageNet toy datasets from fastai
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2022-01-18T19:03:35Z
Labels: dataset contribution

Adds the ImageNet toy datasets from FastAI: Imagenette, Imagewoof and Imagewang.

TODOs:
* [ ] add dummy data
* [ ] add dataset card
* [ ] generate `dataset_info.json`

Comments:
Thanks for your contribution, @mariosasko. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
Title: fix multiple language downloading in mC4
Author: polinaeterna (CONTRIBUTOR)
State: closed
Created: 2022-01-18T17:25:19Z

If we try to access multiple languages of the [mC4 dataset](https://github.com/huggingface/datasets/tree/master/datasets/mc4), it will throw an error. For example, if we do 
```python
mc4_subset_two_langs = load_dataset("mc4", languages=["st", "su"])
```
we got
```
FileNotFoundError: Couldn't find file at https://huggingface.co/datasets/allenai/c4/resolve/1ddc917116b730e1859edef32896ec5c16be51d0/multilingual/c4-st+su.tfrecord-00000-of-00002.json.gz
```
Now it should work. Check it (from the root dir of a project): 
```python
mc4_subset_two_langs = load_dataset("./datasets/mc4/", languages=["st", "su"])
```

Comments:
The CI failure is unrelated to your PR and fixed on master, merging :)
Title: Add QuickDraw dataset
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2022-01-18T15:13:39Z

Add the QuickDraw dataset.

TODOs:
* [x] add dummy data
* [x] add dataset card
* [x] generate `dataset_info.json`

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Add support for time, date, duration, and decimal dtypes
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2022-01-18T13:46:05Z

Add support for the pyarrow time (maps to `datetime.time` in python), date (maps to `datetime.time` in python), duration (maps to `datetime.timedelta` in python), and decimal (maps to `decimal.decimal` in python) dtypes. This should be helpful when writing scripts for time-series datasets.

Comments:
Is there a dataset which uses these four datatypes for tests purposes?


@severo Not yet. I'll let you know if that changes.