Title: unpin-fsspec
Author: lhoestq (MEMBER)
State: closed
Created: 2023-10-23T10:16:46Z

Close #6333.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6336). All of your documentation changes will be reflected on that endpoint.

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006202 / 0.011353 (-0.005151) | 0.003627 / 0.011008 (-0.007381) | 0.080643 / 0.038508 (0.042135) | 0.057135 / 0.023109 (0.034026) | 0.315853 / 0.275898 (0.039955) | 0.348503 / 0.323480 (0.025023) | 0.004762 / 0.007986 (-0.003224) | 0.002884 / 0.004328 (-0.001445) | 0.063208 / 0.004250 (0.058958) | 0.046777 / 0.037052 (0.009725) | 0.321426 / 0.258489 (0.062937) | 0.362128 / 0.293841 (0.068287) | 0.027494 / 0.128546 (-0.101052) | 0.007931 / 0.075646 (-0.067715) | 0.262262 / 0.419271 (-0.157009) | 0.044330 / 0.043533 (0.000797) | 0.310504 / 0.255139 (0.055366) | 0.339409 / 0.283200 (0.056209) | 0.021030 / 0.141683 (-0.120652) | 1.405333 / 1.452155 (-0.046822) | 1.493497 / 1.492716 (0.000781) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.225431 / 0.018006 (0.207425) | 0.451723 / 0.000490 (0.451233) | 0.007763 / 0.000200 (0.007563) | 0.000310 / 0.000054 (0.000256) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023381 / 0.037411 (-0.014031) | 0.074183 / 0.014526 (0.059657) | 0.084003 / 0.176557 (-0.092553) | 0.143628 / 0.737135 (-0.593507) | 0.084543 / 0.296338 (-0.211796) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.393062 / 0.215209 (0.177853) | 3.905649 / 2.077655 (1.827994) | 1.923155 / 1.504120 (0.419035) | 1.751554 / 1.541195 (0.210359) | 1.816141 / 1.468490 (0.347651) | 0.502789 / 4.584777 (-4.081988) | 3.006149 / 3.745712 (-0.739564) | 2.979645 / 5.269862 (-2.290216) | 1.877408 / 4.565676 (-2.688269) | 0.057544 / 0.424275 (-0.366731) | 0.006733 / 0.007607 (-0.000874) | 0.468469 / 0.226044 (0.242425) | 4.695595 / 2.268929 (2.426667) | 2.367238 / 55.444624 (-53.077387) | 2.041035 / 6.876477 (-4.835442) | 2.087396 / 2.142072 (-0.054676) | 0.586866 / 4.805227 (-4.218361) | 0.125616 / 6.500664 (-6.375049) | 0.060535 / 0.075469 (-0.014934) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.244753 / 1.841788 (-0.597035) | 17.652902 / 8.074308 (9.578594) | 13.733195 / 10.191392 (3.541803) | 0.143741 / 0.680424 (-0.536683) | 0.016775 / 0.534201 (-0.517426) | 0.335487 / 0.579283 (-0.243797) | 0.350292 / 0.434364 (-0.084072) | 0.388744 / 0.540337 (-0.151594) | 0.536630 / 1.386936 (-0.850306) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006008 / 0.011353 (-0.005345) | 0.003708 / 0.011008 (-0.007301) | 0.062504 / 0.038508 (0.023996) | 0.058570 / 0.023109 (0.035461) | 0.450549 / 0.275898 (0.174651) | 0.467768 / 0.323480 (0.144288) | 0.004955 / 0.007986 (-0.003031) | 0.002903 / 0.004328 (-0.001426) | 0.062778 / 0.004250 (0.058528) | 0.048750 / 0.037052 (0.011698) | 0.439848 / 0.258489 (0.181359) | 0.471780 / 0.293841 (0.177939) | 0.028472 / 0.128546 (-0.100074) | 0.008221 / 0.075646 (-0.067425) | 0.068325 / 0.419271 (-0.350946) | 0.040612 / 0.043533 (-0.002921) | 0.435530 / 0.255139 (0.180391) | 0.458992 / 0.283200 (0.175792) | 0.020143 / 0.141683 (-0.121539) | 1.479101 / 1.452155 (0.026947) | 1.507408 / 1.492716 (0.014692) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.207723 / 0.018006 (0.189717) | 0.406596 / 0.000490 (0.406106) | 0.004431 / 0.000200 (0.004231) | 0.000078 / 0.000054 (0.000024) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027037 / 0.037411 (-0.010374) | 0.081576 / 0.014526 (0.067050) | 0.091177 / 0.176557 (-0.085379) | 0.146191 / 0.737135 (-0.590944) | 0.092485 / 0.296338 (-0.203854) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.456676 / 0.215209 (0.241467) | 4.556214 / 2.077655 (2.478559) | 2.500146 / 1.504120 (0.996026) | 2.325175 / 1.541195 (0.783981) | 2.421023 / 1.468490 (0.952533) | 0.512135 / 4.584777 (-4.072641) | 3.167070 / 3.745712 (-0.578642) | 2.897697 / 5.269862 (-2.372165) | 1.881974 / 4.565676 (-2.683702) | 0.058453 / 0.424275 (-0.365823) | 0.006515 / 0.007607 (-0.001092) | 0.530742 / 0.226044 (0.304698) | 5.304943 / 2.268929 (3.036014) | 2.928824 / 55.444624 (-52.515800) | 2.598023 / 6.876477 (-4.278454) | 2.758496 / 2.142072 (0.616423) | 0.601777 / 4.805227 (-4.203450) | 0.126701 / 6.500664 (-6.373964) | 0.061808 / 0.075469 (-0.013661) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.357844 / 1.841788 (-0.483943) | 17.887666 / 8.074308 (9.813358) | 14.561904 / 10.191392 (4.370512) | 0.146788 / 0.680424 (-0.533636) | 0.018277 / 0.534201 (-0.515924) | 0.343168 / 0.579283 (-0.236115) | 0.382220 / 0.434364 (-0.052144) | 0.401234 / 0.540337 (-0.139104) | 0.546246 / 1.386936 (-0.840690) |

</details>
</details>

![](https://cml.dev/watermark.png#b0980a74d58098b8b1738e2411f1212161a211b8 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008919 / 0.011353 (-0.002434) | 0.006110 / 0.011008 (-0.004898) | 0.110554 / 0.038508 (0.072046) | 0.075705 / 0.023109 (0.052596) | 0.391235 / 0.275898 (0.115336) | 0.458331 / 0.323480 (0.134851) | 0.007489 / 0.007986 (-0.000497) | 0.003744 / 0.004328 (-0.000585) | 0.078124 / 0.004250 (0.073874) | 0.057244 / 0.037052 (0.020192) | 0.393251 / 0.258489 (0.134762) | 0.460153 / 0.293841 (0.166312) | 0.047245 / 0.128546 (-0.081301) | 0.014086 / 0.075646 (-0.061560) | 0.421272 / 0.419271 (0.002001) | 0.067668 / 0.043533 (0.024135) | 0.397325 / 0.255139 (0.142186) | 0.432683 / 0.283200 (0.149483) | 0.039086 / 0.141683 (-0.102596) | 1.764898 / 1.452155 (0.312744) | 1.848820 / 1.492716 (0.356104) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.258163 / 0.018006 (0.240156) | 0.498655 / 0.000490 (0.498165) | 0.014959 / 0.000200 (0.014759) | 0.000465 / 0.000054 (0.000410) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028889 / 0.037411 (-0.008522) | 0.091568 / 0.014526 (0.077042) | 0.102700 / 0.176557 (-0.073857) | 0.173580 / 0.737135 (-0.563555) | 0.108763 / 0.296338 (-0.187576) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.610147 / 0.215209 (0.394938) | 5.851239 / 2.077655 (3.773584) | 2.467471 / 1.504120 (0.963351) | 2.117189 / 1.541195 (0.575995) | 2.197947 / 1.468490 (0.729457) | 0.851736 / 4.584777 (-3.733041) | 5.163183 / 3.745712 (1.417471) | 5.039564 / 5.269862 (-0.230297) | 3.067215 / 4.565676 (-1.498462) | 0.098593 / 0.424275 (-0.325682) | 0.008646 / 0.007607 (0.001038) | 0.788397 / 0.226044 (0.562352) | 7.340837 / 2.268929 (5.071909) | 3.511611 / 55.444624 (-51.933013) | 2.767479 / 6.876477 (-4.108998) | 2.687368 / 2.142072 (0.545296) | 1.046387 / 4.805227 (-3.758841) | 0.215902 / 6.500664 (-6.284763) | 0.072939 / 0.075469 (-0.002530) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.512795 / 1.841788 (-0.328992) | 22.086131 / 8.074308 (14.011823) | 20.235550 / 10.191392 (10.044158) | 0.240381 / 0.680424 (-0.440043) | 0.029171 / 0.534201 (-0.505030) | 0.465123 / 0.579283 (-0.114160) | 0.569260 / 0.434364 (0.134896) | 0.540967 / 0.540337 (0.000629) | 0.764006 / 1.386936 (-0.622930) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.011024 / 0.011353 (-0.000329) | 0.005915 / 0.011008 (-0.005094) | 0.076455 / 0.038508 (0.037947) | 0.087842 / 0.023109 (0.064733) | 0.471732 / 0.275898 (0.195834) | 0.513666 / 0.323480 (0.190186) | 0.007062 / 0.007986 (-0.000924) | 0.004013 / 0.004328 (-0.000315) | 0.076016 / 0.004250 (0.071766) | 0.061296 / 0.037052 (0.024244) | 0.487277 / 0.258489 (0.228788) | 0.508185 / 0.293841 (0.214344) | 0.049963 / 0.128546 (-0.078583) | 0.013774 / 0.075646 (-0.061873) | 0.089376 / 0.419271 (-0.329895) | 0.067502 / 0.043533 (0.023969) | 0.471283 / 0.255139 (0.216144) | 0.507365 / 0.283200 (0.224165) | 0.033638 / 0.141683 (-0.108045) | 1.785544 / 1.452155 (0.333390) | 1.878765 / 1.492716 (0.386048) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.230462 / 0.018006 (0.212456) | 0.502458 / 0.000490 (0.501968) | 0.005987 / 0.000200 (0.005787) | 0.000114 / 0.000054 (0.000060) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031588 / 0.037411 (-0.005824) | 0.113566 / 0.014526 (0.099040) | 0.115734 / 0.176557 (-0.060822) | 0.174162 / 0.737135 (-0.562974) | 0.121574 / 0.296338 (-0.174764) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.662837 / 0.215209 (0.447628) | 6.420327 / 2.077655 (4.342672) | 3.033522 / 1.504120 (1.529402) | 2.728294 / 1.541195 (1.187099) | 2.790621 / 1.468490 (1.322131) | 0.852478 / 4.584777 (-3.732299) | 5.033637 / 3.745712 (1.287925) | 4.543152 / 5.269862 (-0.726709) | 2.980261 / 4.565676 (-1.585415) | 0.102444 / 0.424275 (-0.321831) | 0.008362 / 0.007607 (0.000755) | 0.786868 / 0.226044 (0.560823) | 7.887665 / 2.268929 (5.618737) | 4.010614 / 55.444624 (-51.434010) | 3.220715 / 6.876477 (-3.655762) | 3.317316 / 2.142072 (1.175244) | 1.098137 / 4.805227 (-3.707090) | 0.218309 / 6.500664 (-6.282355) | 0.078182 / 0.075469 (0.002713) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.696740 / 1.841788 (-0.145047) | 23.762454 / 8.074308 (15.688146) | 21.802645 / 10.191392 (11.611253) | 0.233654 / 0.680424 (-0.446770) | 0.032911 / 0.534201 (-0.501290) | 0.511760 / 0.579283 (-0.067524) | 0.586299 / 0.434364 (0.151935) | 0.583704 / 0.540337 (0.043367) | 0.780762 / 1.386936 (-0.606174) |

</details>
</details>

![](https://cml.dev/watermark.png#0a94aa2f738075bbc08291583f1b153220d5e6e7 "CML watermark")