Title: add Speech commands dataset
Author: polinaeterna (CONTRIBUTOR)
State: closed
Created: 2021-11-29T13:52:47Z

closes  #3283

Comments:
@anton-l ping

@lhoestq 
Hi Quentin! Thank you for your feedback and suggestions! 🤗

Yes, that was actually what I wanted to do next - I mean the steaming stuff :)
Also, I need to make some changes to the readme (to account for the updated features set).

Hopefully, I will be done by tomorrow afternoon if that's ok. 


@lhoestq  Hi Quentin!

I've implemented (hopefully, correctly) the streaming compatibility but the problem with the current approach is that we first need to iterate over the full archive anyway to get the list of filenames for train and validation sets (see [this](https://github.com/huggingface/datasets/pull/3335/files#diff-aeea540d136025e30a842856779e9c6485a5dc6fc9eb7fd6d3be2acd2f49b8e3R186), the same approach is implemented in TFDS version). Only after that, we can generate examples, so we cannot stream the dataset before the first iteration ends and it takes some time. It's probably not the most effective way. 

If the streaming mode is turned off, this approach (with two iterations) is actually slower than the previous implementation (with archive extraction). 

My suggestion is to host separate archives for each split prepared in advance. That way there would be no need for iterating over the common archive to collect train and validation filenames.  @anton-l  suggested to make AWS mirrors for them. I've prepared these archives, for now you can take a look at them [here](https://drive.google.com/drive/folders/1oMrZHzPgHAKprKJuvih91CM8KMSzh_pL?usp=sharing). I simplified their structure a bit so if we switch to using them, the code then should be changed (and simplified) a bit too.


Hi ! Thanks for the changes :)

> My suggestion is to host separate archives for each split prepared in advance. That way there would be no need for iterating over the common archive to collect train and validation filenames. @anton-l suggested to make AWS mirrors for them. I've prepared these archives, for now you can take a look at them here. I simplified their structure a bit so if we switch to using them, the code then should be changed (and simplified) a bit too.

I agree, I just uploaded them on AWS

https://s3.amazonaws.com/datasets.huggingface.co/SpeechCommands/v0.01/v0.01_test.tar.gz
https://s3.amazonaws.com/datasets.huggingface.co/SpeechCommands/v0.01/v0.01_train.tar.gz
https://s3.amazonaws.com/datasets.huggingface.co/SpeechCommands/v0.01/v0.01_validation.tar.gz
https://s3.amazonaws.com/datasets.huggingface.co/SpeechCommands/v0.02/v0.02_test.tar.gz
https://s3.amazonaws.com/datasets.huggingface.co/SpeechCommands/v0.02/v0.02_validation.tar.gz

Note that in the future we can move those files to actual repositories on the Hugging Face Hub, since we are migrating the datasets from this repository to the Hugging Face Hub (as mirrors), to make them more accessible to the community.

@lhoestq Thank you! Gonna look at this tomorrow :)

@lhoestq I've modified the code to fit new data format, now it works for v0.01 but doesn't work for v0.02 as the training archive is missing. Could you please create a mirror for that one too? You can find it [here](https://drive.google.com/file/d/1mPjnVMYb-VhPprGlOX8v9TBT1GT-rtcp/view?usp=sharing)

And when it's done I'll need to regenerate all the meta / dummy stuff, and this version will be ready for a review :)

Here you go :)
https://s3.amazonaws.com/datasets.huggingface.co/SpeechCommands/v0.02/v0.02_train.tar.gz

FYI I juste merged a fix for the Windows CI error on `master`, feel free to merge `master` again into your branch

All green ! I had to fix some minor stuff in the CI but it's good now

Next step is to mark it as ready for review, and I think it's all good so we can merge 🚀 

@lhoestq 🤗

:tada: