Title: Fix `fsspec` download
Author: mariosasko (COLLABORATOR)
State: open
Created: 2023-07-27T18:54:47Z

Testing `ds = load_dataset("audiofolder", data_files="s3://datasets.huggingface.co/SpeechCommands/v0.01/v0.01_test.tar.gz", storage_options={"anon": True})` and trying to fix the issues raised by `fsspec` ...

TODO: fix
```
    self.session = aiobotocore.session.AioSession(**self.kwargs)
TypeError: __init__() got an unexpected keyword argument 'hf'
```
by "preparing `storage_options`" for the `fsspec` head/get

Comments:
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006031 / 0.011353 (-0.005322) | 0.003579 / 0.011008 (-0.007429) | 0.080862 / 0.038508 (0.042354) | 0.056660 / 0.023109 (0.033551) | 0.388285 / 0.275898 (0.112387) | 0.422270 / 0.323480 (0.098790) | 0.004651 / 0.007986 (-0.003335) | 0.002895 / 0.004328 (-0.001433) | 0.062767 / 0.004250 (0.058517) | 0.046491 / 0.037052 (0.009438) | 0.389918 / 0.258489 (0.131428) | 0.434650 / 0.293841 (0.140809) | 0.027265 / 0.128546 (-0.101281) | 0.007946 / 0.075646 (-0.067701) | 0.261207 / 0.419271 (-0.158065) | 0.045057 / 0.043533 (0.001525) | 0.391977 / 0.255139 (0.136838) | 0.418525 / 0.283200 (0.135326) | 0.020705 / 0.141683 (-0.120978) | 1.459271 / 1.452155 (0.007116) | 1.516935 / 1.492716 (0.024218) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.174659 / 0.018006 (0.156653) | 0.429627 / 0.000490 (0.429137) | 0.003714 / 0.000200 (0.003514) | 0.000070 / 0.000054 (0.000016) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023255 / 0.037411 (-0.014156) | 0.073463 / 0.014526 (0.058937) | 0.083000 / 0.176557 (-0.093557) | 0.146704 / 0.737135 (-0.590431) | 0.084419 / 0.296338 (-0.211919) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.392222 / 0.215209 (0.177013) | 3.902620 / 2.077655 (1.824966) | 1.903056 / 1.504120 (0.398936) | 1.753423 / 1.541195 (0.212228) | 1.874547 / 1.468490 (0.406057) | 0.495947 / 4.584777 (-4.088829) | 3.084680 / 3.745712 (-0.661032) | 4.235064 / 5.269862 (-1.034797) | 2.626840 / 4.565676 (-1.938837) | 0.057273 / 0.424275 (-0.367002) | 0.006457 / 0.007607 (-0.001150) | 0.466018 / 0.226044 (0.239974) | 4.648264 / 2.268929 (2.379335) | 2.520293 / 55.444624 (-52.924331) | 2.339393 / 6.876477 (-4.537083) | 2.538848 / 2.142072 (0.396775) | 0.592018 / 4.805227 (-4.213210) | 0.125041 / 6.500664 (-6.375623) | 0.061038 / 0.075469 (-0.014431) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.244285 / 1.841788 (-0.597503) | 18.411576 / 8.074308 (10.337268) | 13.850100 / 10.191392 (3.658708) | 0.131904 / 0.680424 (-0.548520) | 0.016824 / 0.534201 (-0.517377) | 0.328931 / 0.579283 (-0.250352) | 0.364801 / 0.434364 (-0.069563) | 0.376298 / 0.540337 (-0.164039) | 0.525045 / 1.386936 (-0.861891) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006059 / 0.011353 (-0.005294) | 0.003693 / 0.011008 (-0.007315) | 0.062982 / 0.038508 (0.024473) | 0.062155 / 0.023109 (0.039046) | 0.389467 / 0.275898 (0.113568) | 0.437046 / 0.323480 (0.113566) | 0.004823 / 0.007986 (-0.003163) | 0.002935 / 0.004328 (-0.001393) | 0.062679 / 0.004250 (0.058429) | 0.049676 / 0.037052 (0.012623) | 0.418054 / 0.258489 (0.159565) | 0.442467 / 0.293841 (0.148626) | 0.027652 / 0.128546 (-0.100895) | 0.008146 / 0.075646 (-0.067501) | 0.069414 / 0.419271 (-0.349858) | 0.042884 / 0.043533 (-0.000649) | 0.387167 / 0.255139 (0.132028) | 0.418684 / 0.283200 (0.135484) | 0.022419 / 0.141683 (-0.119264) | 1.460606 / 1.452155 (0.008451) | 1.514204 / 1.492716 (0.021487) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.200523 / 0.018006 (0.182517) | 0.415970 / 0.000490 (0.415481) | 0.003202 / 0.000200 (0.003002) | 0.000069 / 0.000054 (0.000014) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025836 / 0.037411 (-0.011575) | 0.078859 / 0.014526 (0.064333) | 0.088523 / 0.176557 (-0.088034) | 0.141572 / 0.737135 (-0.595563) | 0.090258 / 0.296338 (-0.206080) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.416548 / 0.215209 (0.201339) | 4.155278 / 2.077655 (2.077623) | 2.126683 / 1.504120 (0.622563) | 1.963762 / 1.541195 (0.422568) | 2.029018 / 1.468490 (0.560528) | 0.499005 / 4.584777 (-4.085772) | 3.063503 / 3.745712 (-0.682209) | 4.250800 / 5.269862 (-1.019061) | 2.642634 / 4.565676 (-1.923043) | 0.057815 / 0.424275 (-0.366460) | 0.006784 / 0.007607 (-0.000823) | 0.492481 / 0.226044 (0.266437) | 4.914306 / 2.268929 (2.645377) | 2.601582 / 55.444624 (-52.843042) | 2.337863 / 6.876477 (-4.538614) | 2.462854 / 2.142072 (0.320782) | 0.593738 / 4.805227 (-4.211489) | 0.127030 / 6.500664 (-6.373634) | 0.064206 / 0.075469 (-0.011263) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.326919 / 1.841788 (-0.514868) | 18.728929 / 8.074308 (10.654621) | 13.903681 / 10.191392 (3.712289) | 0.162670 / 0.680424 (-0.517754) | 0.016913 / 0.534201 (-0.517288) | 0.337504 / 0.579283 (-0.241779) | 0.339786 / 0.434364 (-0.094577) | 0.384955 / 0.540337 (-0.155383) | 0.514358 / 1.386936 (-0.872578) |

</details>
</details>

![](https://cml.dev/watermark.png#c5c31b492c45e01c6f4593ada2b84517a75a5c7c "CML watermark")


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6085). All of your documentation changes will be reflected on that endpoint.

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007610 / 0.011353 (-0.003743) | 0.004616 / 0.011008 (-0.006392) | 0.100330 / 0.038508 (0.061821) | 0.084450 / 0.023109 (0.061341) | 0.386610 / 0.275898 (0.110712) | 0.418479 / 0.323480 (0.094999) | 0.006085 / 0.007986 (-0.001900) | 0.003800 / 0.004328 (-0.000529) | 0.076248 / 0.004250 (0.071997) | 0.065175 / 0.037052 (0.028122) | 0.387154 / 0.258489 (0.128665) | 0.425484 / 0.293841 (0.131643) | 0.035946 / 0.128546 (-0.092601) | 0.009901 / 0.075646 (-0.065745) | 0.343015 / 0.419271 (-0.076256) | 0.060965 / 0.043533 (0.017432) | 0.390585 / 0.255139 (0.135446) | 0.405873 / 0.283200 (0.122673) | 0.026929 / 0.141683 (-0.114754) | 1.767916 / 1.452155 (0.315761) | 1.893431 / 1.492716 (0.400715) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.237888 / 0.018006 (0.219882) | 0.503949 / 0.000490 (0.503459) | 0.004769 / 0.000200 (0.004570) | 0.000088 / 0.000054 (0.000033) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031553 / 0.037411 (-0.005859) | 0.096950 / 0.014526 (0.082424) | 0.110374 / 0.176557 (-0.066183) | 0.176754 / 0.737135 (-0.560381) | 0.111703 / 0.296338 (-0.184635) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.449232 / 0.215209 (0.234023) | 4.510247 / 2.077655 (2.432592) | 2.188547 / 1.504120 (0.684427) | 2.007530 / 1.541195 (0.466335) | 2.095650 / 1.468490 (0.627160) | 0.563262 / 4.584777 (-4.021515) | 4.062412 / 3.745712 (0.316700) | 6.338350 / 5.269862 (1.068489) | 3.844669 / 4.565676 (-0.721008) | 0.064517 / 0.424275 (-0.359758) | 0.008536 / 0.007607 (0.000929) | 0.553872 / 0.226044 (0.327828) | 5.530311 / 2.268929 (3.261383) | 2.835109 / 55.444624 (-52.609516) | 2.493900 / 6.876477 (-4.382577) | 2.728412 / 2.142072 (0.586340) | 0.680161 / 4.805227 (-4.125066) | 0.155831 / 6.500664 (-6.344833) | 0.070359 / 0.075469 (-0.005110) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.504852 / 1.841788 (-0.336936) | 22.806335 / 8.074308 (14.732027) | 16.598386 / 10.191392 (6.406994) | 0.207857 / 0.680424 (-0.472566) | 0.021425 / 0.534201 (-0.512776) | 0.474069 / 0.579283 (-0.105214) | 0.472263 / 0.434364 (0.037899) | 0.542195 / 0.540337 (0.001858) | 0.782871 / 1.386936 (-0.604065) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007443 / 0.011353 (-0.003910) | 0.004465 / 0.011008 (-0.006544) | 0.076268 / 0.038508 (0.037759) | 0.086607 / 0.023109 (0.063498) | 0.443295 / 0.275898 (0.167397) | 0.472819 / 0.323480 (0.149339) | 0.005841 / 0.007986 (-0.002144) | 0.003727 / 0.004328 (-0.000602) | 0.076015 / 0.004250 (0.071765) | 0.063188 / 0.037052 (0.026136) | 0.450555 / 0.258489 (0.192066) | 0.478532 / 0.293841 (0.184691) | 0.036258 / 0.128546 (-0.092288) | 0.009869 / 0.075646 (-0.065777) | 0.083786 / 0.419271 (-0.335486) | 0.056546 / 0.043533 (0.013013) | 0.449647 / 0.255139 (0.194508) | 0.457588 / 0.283200 (0.174389) | 0.027197 / 0.141683 (-0.114486) | 1.769991 / 1.452155 (0.317836) | 1.859905 / 1.492716 (0.367189) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.268637 / 0.018006 (0.250631) | 0.492860 / 0.000490 (0.492370) | 0.008574 / 0.000200 (0.008374) | 0.000140 / 0.000054 (0.000085) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.037679 / 0.037411 (0.000268) | 0.108258 / 0.014526 (0.093733) | 0.117850 / 0.176557 (-0.058707) | 0.181611 / 0.737135 (-0.555524) | 0.120901 / 0.296338 (-0.175437) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.485780 / 0.215209 (0.270571) | 4.851289 / 2.077655 (2.773635) | 2.486068 / 1.504120 (0.981948) | 2.299417 / 1.541195 (0.758222) | 2.387093 / 1.468490 (0.918603) | 0.568826 / 4.584777 (-4.015951) | 4.163426 / 3.745712 (0.417713) | 6.224964 / 5.269862 (0.955102) | 3.255619 / 4.565676 (-1.310058) | 0.067081 / 0.424275 (-0.357194) | 0.009065 / 0.007607 (0.001458) | 0.580449 / 0.226044 (0.354405) | 5.786394 / 2.268929 (3.517465) | 3.057780 / 55.444624 (-52.386844) | 2.764339 / 6.876477 (-4.112138) | 2.880718 / 2.142072 (0.738645) | 0.681376 / 4.805227 (-4.123851) | 0.157858 / 6.500664 (-6.342806) | 0.072481 / 0.075469 (-0.002988) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.590704 / 1.841788 (-0.251083) | 23.141929 / 8.074308 (15.067620) | 17.001141 / 10.191392 (6.809749) | 0.203790 / 0.680424 (-0.476634) | 0.021766 / 0.534201 (-0.512435) | 0.475309 / 0.579283 (-0.103974) | 0.466448 / 0.434364 (0.032084) | 0.551470 / 0.540337 (0.011132) | 0.727876 / 1.386936 (-0.659060) |

</details>
</details>

![](https://cml.dev/watermark.png#61b19eea7fc5cf484e8cdf41d6ae035f94d8a671 "CML watermark")