Title: Add `fsspec` version to the `datasets-cli env` command output
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2023-10-26T17:19:25Z

... to make debugging issues easier, as `fsspec`'s releases often introduce breaking changes.

Comments:
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008775 / 0.011353 (-0.002578) | 0.005304 / 0.011008 (-0.005704) | 0.108912 / 0.038508 (0.070404) | 0.075589 / 0.023109 (0.052479) | 0.456612 / 0.275898 (0.180713) | 0.502303 / 0.323480 (0.178823) | 0.006695 / 0.007986 (-0.001291) | 0.004404 / 0.004328 (0.000076) | 0.084802 / 0.004250 (0.080552) | 0.062711 / 0.037052 (0.025659) | 0.465062 / 0.258489 (0.206573) | 0.505321 / 0.293841 (0.211480) | 0.049401 / 0.128546 (-0.079146) | 0.014784 / 0.075646 (-0.060862) | 0.378202 / 0.419271 (-0.041069) | 0.069826 / 0.043533 (0.026293) | 0.461161 / 0.255139 (0.206022) | 0.484616 / 0.283200 (0.201416) | 0.035998 / 0.141683 (-0.105685) | 1.846343 / 1.452155 (0.394189) | 1.999439 / 1.492716 (0.506723) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.317779 / 0.018006 (0.299773) | 0.605967 / 0.000490 (0.605477) | 0.011412 / 0.000200 (0.011212) | 0.000410 / 0.000054 (0.000356) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031118 / 0.037411 (-0.006293) | 0.095425 / 0.014526 (0.080900) | 0.108002 / 0.176557 (-0.068554) | 0.184625 / 0.737135 (-0.552511) | 0.108180 / 0.296338 (-0.188159) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.587497 / 0.215209 (0.372288) | 5.818632 / 2.077655 (3.740977) | 2.629776 / 1.504120 (1.125656) | 2.266129 / 1.541195 (0.724934) | 2.324618 / 1.468490 (0.856128) | 0.830049 / 4.584777 (-3.754728) | 5.380062 / 3.745712 (1.634350) | 4.808525 / 5.269862 (-0.461336) | 2.960368 / 4.565676 (-1.605309) | 0.093637 / 0.424275 (-0.330638) | 0.009187 / 0.007607 (0.001580) | 0.703468 / 0.226044 (0.477424) | 6.924509 / 2.268929 (4.655580) | 3.380582 / 55.444624 (-52.064043) | 2.689118 / 6.876477 (-4.187358) | 2.712418 / 2.142072 (0.570345) | 1.017144 / 4.805227 (-3.788084) | 0.212874 / 6.500664 (-6.287791) | 0.080053 / 0.075469 (0.004584) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.623663 / 1.841788 (-0.218125) | 23.668872 / 8.074308 (15.594564) | 20.245972 / 10.191392 (10.054580) | 0.236448 / 0.680424 (-0.443976) | 0.029730 / 0.534201 (-0.504470) | 0.491525 / 0.579283 (-0.087758) | 0.593780 / 0.434364 (0.159416) | 0.548776 / 0.540337 (0.008438) | 0.799370 / 1.386936 (-0.587566) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009714 / 0.011353 (-0.001639) | 0.005328 / 0.011008 (-0.005681) | 0.078460 / 0.038508 (0.039952) | 0.077791 / 0.023109 (0.054682) | 0.510124 / 0.275898 (0.234226) | 0.547769 / 0.323480 (0.224289) | 0.006868 / 0.007986 (-0.001118) | 0.004145 / 0.004328 (-0.000183) | 0.088696 / 0.004250 (0.084445) | 0.072387 / 0.037052 (0.035334) | 0.527373 / 0.258489 (0.268884) | 0.561948 / 0.293841 (0.268107) | 0.049769 / 0.128546 (-0.078777) | 0.014401 / 0.075646 (-0.061246) | 0.097541 / 0.419271 (-0.321731) | 0.062237 / 0.043533 (0.018705) | 0.531001 / 0.255139 (0.275862) | 0.561797 / 0.283200 (0.278597) | 0.038482 / 0.141683 (-0.103201) | 1.783558 / 1.452155 (0.331404) | 1.864339 / 1.492716 (0.371622) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.289389 / 0.018006 (0.271383) | 0.595326 / 0.000490 (0.594836) | 0.004583 / 0.000200 (0.004383) | 0.000114 / 0.000054 (0.000060) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034492 / 0.037411 (-0.002919) | 0.102934 / 0.014526 (0.088409) | 0.121689 / 0.176557 (-0.054868) | 0.182121 / 0.737135 (-0.555015) | 0.127087 / 0.296338 (-0.169252) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.645726 / 0.215209 (0.430517) | 6.462235 / 2.077655 (4.384580) | 3.044176 / 1.504120 (1.540056) | 2.731181 / 1.541195 (1.189986) | 2.805508 / 1.468490 (1.337018) | 0.846324 / 4.584777 (-3.738453) | 5.341074 / 3.745712 (1.595362) | 4.687111 / 5.269862 (-0.582751) | 3.035472 / 4.565676 (-1.530205) | 0.099193 / 0.424275 (-0.325082) | 0.008825 / 0.007607 (0.001218) | 0.795102 / 0.226044 (0.569058) | 7.895770 / 2.268929 (5.626842) | 3.826752 / 55.444624 (-51.617873) | 3.112217 / 6.876477 (-3.764259) | 3.526878 / 2.142072 (1.384806) | 1.011352 / 4.805227 (-3.793875) | 0.213424 / 6.500664 (-6.287240) | 0.076228 / 0.075469 (0.000759) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.805232 / 1.841788 (-0.036556) | 24.049100 / 8.074308 (15.974792) | 23.056011 / 10.191392 (12.864619) | 0.261656 / 0.680424 (-0.418767) | 0.032021 / 0.534201 (-0.502179) | 0.483829 / 0.579283 (-0.095454) | 0.602208 / 0.434364 (0.167844) | 0.565848 / 0.540337 (0.025511) | 0.818678 / 1.386936 (-0.568258) |

</details>
</details>

![](https://cml.dev/watermark.png#71fc5e2ca41f5f725b9117f4cf99f348534902f3 "CML watermark")


_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008043 / 0.011353 (-0.003310) | 0.004642 / 0.011008 (-0.006366) | 0.102592 / 0.038508 (0.064084) | 0.099508 / 0.023109 (0.076399) | 0.377692 / 0.275898 (0.101794) | 0.409929 / 0.323480 (0.086450) | 0.006363 / 0.007986 (-0.001622) | 0.003881 / 0.004328 (-0.000447) | 0.076636 / 0.004250 (0.072386) | 0.067021 / 0.037052 (0.029969) | 0.371454 / 0.258489 (0.112964) | 0.423637 / 0.293841 (0.129796) | 0.038632 / 0.128546 (-0.089914) | 0.010055 / 0.075646 (-0.065591) | 0.352021 / 0.419271 (-0.067251) | 0.064988 / 0.043533 (0.021456) | 0.369614 / 0.255139 (0.114475) | 0.396972 / 0.283200 (0.113773) | 0.028866 / 0.141683 (-0.112817) | 1.757620 / 1.452155 (0.305465) | 1.886283 / 1.492716 (0.393567) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.257579 / 0.018006 (0.239572) | 0.529859 / 0.000490 (0.529369) | 0.011720 / 0.000200 (0.011520) | 0.000455 / 0.000054 (0.000401) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034163 / 0.037411 (-0.003248) | 0.101422 / 0.014526 (0.086896) | 0.114858 / 0.176557 (-0.061698) | 0.180265 / 0.737135 (-0.556870) | 0.116034 / 0.296338 (-0.180305) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.477609 / 0.215209 (0.262400) | 4.830116 / 2.077655 (2.752461) | 2.323844 / 1.504120 (0.819724) | 2.174496 / 1.541195 (0.633301) | 2.268594 / 1.468490 (0.800104) | 0.612429 / 4.584777 (-3.972348) | 4.265277 / 3.745712 (0.519565) | 4.095741 / 5.269862 (-1.174121) | 2.561532 / 4.565676 (-2.004144) | 0.068043 / 0.424275 (-0.356233) | 0.009139 / 0.007607 (0.001532) | 0.545512 / 0.226044 (0.319467) | 5.456403 / 2.268929 (3.187475) | 2.778937 / 55.444624 (-52.665688) | 2.428560 / 6.876477 (-4.447917) | 2.557483 / 2.142072 (0.415411) | 0.696721 / 4.805227 (-4.108506) | 0.157217 / 6.500664 (-6.343447) | 0.071334 / 0.075469 (-0.004135) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.617755 / 1.841788 (-0.224032) | 23.368508 / 8.074308 (15.294200) | 17.028591 / 10.191392 (6.837199) | 0.195881 / 0.680424 (-0.484542) | 0.021788 / 0.534201 (-0.512413) | 0.468484 / 0.579283 (-0.110799) | 0.474604 / 0.434364 (0.040240) | 0.544738 / 0.540337 (0.004400) | 0.771722 / 1.386936 (-0.615214) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007939 / 0.011353 (-0.003414) | 0.004684 / 0.011008 (-0.006324) | 0.077273 / 0.038508 (0.038765) | 0.088763 / 0.023109 (0.065654) | 0.489178 / 0.275898 (0.213280) | 0.531547 / 0.323480 (0.208067) | 0.006214 / 0.007986 (-0.001772) | 0.003988 / 0.004328 (-0.000340) | 0.076685 / 0.004250 (0.072434) | 0.066628 / 0.037052 (0.029576) | 0.497153 / 0.258489 (0.238664) | 0.538301 / 0.293841 (0.244460) | 0.037939 / 0.128546 (-0.090607) | 0.010054 / 0.075646 (-0.065592) | 0.084642 / 0.419271 (-0.334629) | 0.057140 / 0.043533 (0.013608) | 0.487701 / 0.255139 (0.232562) | 0.519676 / 0.283200 (0.236477) | 0.026560 / 0.141683 (-0.115123) | 1.809676 / 1.452155 (0.357521) | 1.864884 / 1.492716 (0.372168) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.259005 / 0.018006 (0.240998) | 0.522900 / 0.000490 (0.522410) | 0.006885 / 0.000200 (0.006685) | 0.000156 / 0.000054 (0.000102) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.039838 / 0.037411 (0.002426) | 0.117777 / 0.014526 (0.103251) | 0.129189 / 0.176557 (-0.047368) | 0.198584 / 0.737135 (-0.538552) | 0.129753 / 0.296338 (-0.166586) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.543366 / 0.215209 (0.328157) | 5.241502 / 2.077655 (3.163847) | 2.719079 / 1.504120 (1.214959) | 2.525337 / 1.541195 (0.984142) | 2.648908 / 1.468490 (1.180418) | 0.589239 / 4.584777 (-3.995538) | 4.379856 / 3.745712 (0.634144) | 4.139919 / 5.269862 (-1.129943) | 2.633412 / 4.565676 (-1.932264) | 0.074582 / 0.424275 (-0.349693) | 0.009106 / 0.007607 (0.001499) | 0.635540 / 0.226044 (0.409495) | 6.072965 / 2.268929 (3.804037) | 3.327233 / 55.444624 (-52.117391) | 3.012637 / 6.876477 (-3.863840) | 3.113226 / 2.142072 (0.971154) | 0.712705 / 4.805227 (-4.092523) | 0.159550 / 6.500664 (-6.341114) | 0.073446 / 0.075469 (-0.002023) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.718732 / 1.841788 (-0.123055) | 23.249445 / 8.074308 (15.175137) | 17.630643 / 10.191392 (7.439251) | 0.201017 / 0.680424 (-0.479407) | 0.024162 / 0.534201 (-0.510039) | 0.475054 / 0.579283 (-0.104229) | 0.492348 / 0.434364 (0.057985) | 0.587118 / 0.540337 (0.046781) | 0.777462 / 1.386936 (-0.609474) |

</details>
</details>

![](https://cml.dev/watermark.png#feb036956a592b9a9ecdf048cc801549f233dbef "CML watermark")