Title: Better `tqdm` wrapper
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2023-11-17T15:45:15Z

This PR aligns the `tqdm` logic with `huggingface_hub` (without introducing breaking changes), as the current one is error-prone.

Additionally, it improves the doc page about the `datasets`' utilities, and the handling of local `fsspec` paths in `cached_path`.
 
Fix #6409

Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005070 / 0.011353 (-0.006283) | 0.003251 / 0.011008 (-0.007757) | 0.061528 / 0.038508 (0.023020) | 0.055386 / 0.023109 (0.032276) | 0.248536 / 0.275898 (-0.027362) | 0.272346 / 0.323480 (-0.051134) | 0.003875 / 0.007986 (-0.004111) | 0.002396 / 0.004328 (-0.001933) | 0.047659 / 0.004250 (0.043409) | 0.037448 / 0.037052 (0.000396) | 0.251101 / 0.258489 (-0.007388) | 0.282353 / 0.293841 (-0.011488) | 0.027784 / 0.128546 (-0.100762) | 0.010534 / 0.075646 (-0.065113) | 0.206025 / 0.419271 (-0.213246) | 0.035410 / 0.043533 (-0.008123) | 0.250626 / 0.255139 (-0.004513) | 0.266801 / 0.283200 (-0.016399) | 0.017704 / 0.141683 (-0.123979) | 1.089970 / 1.452155 (-0.362185) | 1.171683 / 1.492716 (-0.321033) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092700 / 0.018006 (0.074694) | 0.301314 / 0.000490 (0.300824) | 0.000212 / 0.000200 (0.000012) | 0.000044 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018385 / 0.037411 (-0.019026) | 0.062364 / 0.014526 (0.047838) | 0.075887 / 0.176557 (-0.100670) | 0.119484 / 0.737135 (-0.617651) | 0.074490 / 0.296338 (-0.221849) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.283893 / 0.215209 (0.068684) | 2.746772 / 2.077655 (0.669118) | 1.486568 / 1.504120 (-0.017552) | 1.376451 / 1.541195 (-0.164744) | 1.377928 / 1.468490 (-0.090562) | 0.572393 / 4.584777 (-4.012384) | 2.383282 / 3.745712 (-1.362430) | 2.791614 / 5.269862 (-2.478248) | 1.753373 / 4.565676 (-2.812303) | 0.063539 / 0.424275 (-0.360736) | 0.005014 / 0.007607 (-0.002593) | 0.341300 / 0.226044 (0.115256) | 3.376960 / 2.268929 (1.108032) | 1.914162 / 55.444624 (-53.530462) | 1.590188 / 6.876477 (-5.286289) | 1.618420 / 2.142072 (-0.523652) | 0.648723 / 4.805227 (-4.156504) | 0.117745 / 6.500664 (-6.382919) | 0.048858 / 0.075469 (-0.026611) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.944422 / 1.841788 (-0.897366) | 11.603590 / 8.074308 (3.529282) | 10.707000 / 10.191392 (0.515608) | 0.130779 / 0.680424 (-0.549645) | 0.015126 / 0.534201 (-0.519075) | 0.284869 / 0.579283 (-0.294414) | 0.266778 / 0.434364 (-0.167585) | 0.320646 / 0.540337 (-0.219691) | 0.417167 / 1.386936 (-0.969769) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005384 / 0.011353 (-0.005969) | 0.003311 / 0.011008 (-0.007698) | 0.049933 / 0.038508 (0.011425) | 0.052791 / 0.023109 (0.029681) | 0.277061 / 0.275898 (0.001162) | 0.302149 / 0.323480 (-0.021331) | 0.004006 / 0.007986 (-0.003979) | 0.002394 / 0.004328 (-0.001934) | 0.049020 / 0.004250 (0.044770) | 0.040168 / 0.037052 (0.003116) | 0.278625 / 0.258489 (0.020136) | 0.308641 / 0.293841 (0.014800) | 0.029808 / 0.128546 (-0.098738) | 0.010873 / 0.075646 (-0.064774) | 0.058040 / 0.419271 (-0.361231) | 0.032706 / 0.043533 (-0.010827) | 0.277254 / 0.255139 (0.022115) | 0.295208 / 0.283200 (0.012008) | 0.017769 / 0.141683 (-0.123914) | 1.126416 / 1.452155 (-0.325739) | 1.169046 / 1.492716 (-0.323670) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094776 / 0.018006 (0.076770) | 0.306262 / 0.000490 (0.305772) | 0.000223 / 0.000200 (0.000023) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022279 / 0.037411 (-0.015132) | 0.086784 / 0.014526 (0.072258) | 0.082268 / 0.176557 (-0.094289) | 0.120131 / 0.737135 (-0.617004) | 0.082862 / 0.296338 (-0.213476) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.300565 / 0.215209 (0.085356) | 2.923424 / 2.077655 (0.845769) | 1.594836 / 1.504120 (0.090716) | 1.504323 / 1.541195 (-0.036872) | 1.498495 / 1.468490 (0.030005) | 0.570969 / 4.584777 (-4.013808) | 2.476966 / 3.745712 (-1.268746) | 2.785190 / 5.269862 (-2.484672) | 1.749839 / 4.565676 (-2.815837) | 0.062809 / 0.424275 (-0.361466) | 0.004908 / 0.007607 (-0.002699) | 0.361513 / 0.226044 (0.135469) | 3.587135 / 2.268929 (1.318207) | 1.952030 / 55.444624 (-53.492595) | 1.661552 / 6.876477 (-5.214925) | 1.678673 / 2.142072 (-0.463399) | 0.645083 / 4.805227 (-4.160144) | 0.117098 / 6.500664 (-6.383566) | 0.041630 / 0.075469 (-0.033839) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.987883 / 1.841788 (-0.853904) | 12.300764 / 8.074308 (4.226456) | 10.962068 / 10.191392 (0.770675) | 0.143200 / 0.680424 (-0.537224) | 0.015743 / 0.534201 (-0.518458) | 0.289733 / 0.579283 (-0.289550) | 0.276384 / 0.434364 (-0.157979) | 0.328542 / 0.540337 (-0.211795) | 0.552175 / 1.386936 (-0.834761) |

</details>
</details>

![](https://cml.dev/watermark.png#81a65a57cf9fd0abdf85b664a144c9a06cb2896d "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005110 / 0.011353 (-0.006243) | 0.003311 / 0.011008 (-0.007697) | 0.061962 / 0.038508 (0.023454) | 0.050250 / 0.023109 (0.027140) | 0.245313 / 0.275898 (-0.030585) | 0.268748 / 0.323480 (-0.054732) | 0.004693 / 0.007986 (-0.003293) | 0.002465 / 0.004328 (-0.001863) | 0.047698 / 0.004250 (0.043447) | 0.037314 / 0.037052 (0.000262) | 0.250370 / 0.258489 (-0.008119) | 0.286023 / 0.293841 (-0.007818) | 0.027891 / 0.128546 (-0.100655) | 0.010574 / 0.075646 (-0.065072) | 0.204895 / 0.419271 (-0.214376) | 0.036014 / 0.043533 (-0.007519) | 0.250959 / 0.255139 (-0.004180) | 0.266710 / 0.283200 (-0.016489) | 0.018492 / 0.141683 (-0.123191) | 1.115340 / 1.452155 (-0.336815) | 1.176488 / 1.492716 (-0.316229) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.099409 / 0.018006 (0.081402) | 0.310151 / 0.000490 (0.309661) | 0.000223 / 0.000200 (0.000023) | 0.000044 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018132 / 0.037411 (-0.019279) | 0.061820 / 0.014526 (0.047294) | 0.074960 / 0.176557 (-0.101596) | 0.119793 / 0.737135 (-0.617342) | 0.074132 / 0.296338 (-0.222206) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.286388 / 0.215209 (0.071179) | 2.830791 / 2.077655 (0.753137) | 1.514588 / 1.504120 (0.010468) | 1.376514 / 1.541195 (-0.164681) | 1.405080 / 1.468490 (-0.063410) | 0.555297 / 4.584777 (-4.029480) | 2.364838 / 3.745712 (-1.380874) | 2.806050 / 5.269862 (-2.463812) | 1.756114 / 4.565676 (-2.809562) | 0.062254 / 0.424275 (-0.362022) | 0.005020 / 0.007607 (-0.002588) | 0.346272 / 0.226044 (0.120227) | 3.453195 / 2.268929 (1.184266) | 1.837810 / 55.444624 (-53.606814) | 1.577984 / 6.876477 (-5.298493) | 1.560821 / 2.142072 (-0.581251) | 0.633930 / 4.805227 (-4.171297) | 0.116414 / 6.500664 (-6.384250) | 0.042007 / 0.075469 (-0.033462) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.941322 / 1.841788 (-0.900466) | 11.740927 / 8.074308 (3.666618) | 10.450543 / 10.191392 (0.259151) | 0.128820 / 0.680424 (-0.551604) | 0.014856 / 0.534201 (-0.519345) | 0.285636 / 0.579283 (-0.293647) | 0.270051 / 0.434364 (-0.164313) | 0.321244 / 0.540337 (-0.219093) | 0.415486 / 1.386936 (-0.971450) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005333 / 0.011353 (-0.006020) | 0.003370 / 0.011008 (-0.007638) | 0.049046 / 0.038508 (0.010538) | 0.055767 / 0.023109 (0.032658) | 0.273463 / 0.275898 (-0.002435) | 0.292909 / 0.323480 (-0.030571) | 0.004102 / 0.007986 (-0.003883) | 0.002460 / 0.004328 (-0.001868) | 0.048025 / 0.004250 (0.043775) | 0.040342 / 0.037052 (0.003290) | 0.275114 / 0.258489 (0.016625) | 0.295988 / 0.293841 (0.002147) | 0.029461 / 0.128546 (-0.099085) | 0.010654 / 0.075646 (-0.064992) | 0.057196 / 0.419271 (-0.362076) | 0.033238 / 0.043533 (-0.010295) | 0.275885 / 0.255139 (0.020746) | 0.288566 / 0.283200 (0.005366) | 0.018058 / 0.141683 (-0.123625) | 1.130513 / 1.452155 (-0.321642) | 1.173608 / 1.492716 (-0.319108) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.097751 / 0.018006 (0.079745) | 0.312106 / 0.000490 (0.311616) | 0.000232 / 0.000200 (0.000032) | 0.000044 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021201 / 0.037411 (-0.016211) | 0.070150 / 0.014526 (0.055624) | 0.081073 / 0.176557 (-0.095484) | 0.119520 / 0.737135 (-0.617615) | 0.084479 / 0.296338 (-0.211859) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.292322 / 0.215209 (0.077113) | 2.844070 / 2.077655 (0.766415) | 1.581838 / 1.504120 (0.077718) | 1.462665 / 1.541195 (-0.078530) | 1.483013 / 1.468490 (0.014523) | 0.558705 / 4.584777 (-4.026072) | 2.422368 / 3.745712 (-1.323344) | 2.772274 / 5.269862 (-2.497587) | 1.725901 / 4.565676 (-2.839775) | 0.062993 / 0.424275 (-0.361282) | 0.004982 / 0.007607 (-0.002625) | 0.344336 / 0.226044 (0.118292) | 3.425230 / 2.268929 (1.156302) | 1.947199 / 55.444624 (-53.497425) | 1.670362 / 6.876477 (-5.206115) | 1.674112 / 2.142072 (-0.467961) | 0.633857 / 4.805227 (-4.171370) | 0.114837 / 6.500664 (-6.385827) | 0.042558 / 0.075469 (-0.032911) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.979474 / 1.841788 (-0.862314) | 12.110856 / 8.074308 (4.036548) | 10.605998 / 10.191392 (0.414606) | 0.130769 / 0.680424 (-0.549654) | 0.016057 / 0.534201 (-0.518144) | 0.296448 / 0.579283 (-0.282835) | 0.278078 / 0.434364 (-0.156286) | 0.320809 / 0.540337 (-0.219528) | 0.570756 / 1.386936 (-0.816180) |

</details>
</details>

![](https://cml.dev/watermark.png#eeb9727cc680a8f8172a012920bf84f285fec5a0 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005181 / 0.011353 (-0.006172) | 0.003434 / 0.011008 (-0.007574) | 0.062333 / 0.038508 (0.023825) | 0.058544 / 0.023109 (0.035435) | 0.233794 / 0.275898 (-0.042104) | 0.258774 / 0.323480 (-0.064706) | 0.003869 / 0.007986 (-0.004117) | 0.002478 / 0.004328 (-0.001850) | 0.047871 / 0.004250 (0.043620) | 0.037997 / 0.037052 (0.000945) | 0.241269 / 0.258489 (-0.017220) | 0.270103 / 0.293841 (-0.023738) | 0.027710 / 0.128546 (-0.100836) | 0.010683 / 0.075646 (-0.064963) | 0.213204 / 0.419271 (-0.206067) | 0.036156 / 0.043533 (-0.007377) | 0.240061 / 0.255139 (-0.015078) | 0.253627 / 0.283200 (-0.029573) | 0.017880 / 0.141683 (-0.123803) | 1.102965 / 1.452155 (-0.349189) | 1.176919 / 1.492716 (-0.315797) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093206 / 0.018006 (0.075200) | 0.300960 / 0.000490 (0.300470) | 0.000214 / 0.000200 (0.000014) | 0.000042 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019417 / 0.037411 (-0.017994) | 0.061948 / 0.014526 (0.047422) | 0.073560 / 0.176557 (-0.102997) | 0.120682 / 0.737135 (-0.616453) | 0.074925 / 0.296338 (-0.221413) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.280157 / 0.215209 (0.064948) | 2.760648 / 2.077655 (0.682994) | 1.482129 / 1.504120 (-0.021991) | 1.364091 / 1.541195 (-0.177104) | 1.415680 / 1.468490 (-0.052810) | 0.564697 / 4.584777 (-4.020080) | 2.364080 / 3.745712 (-1.381633) | 2.794018 / 5.269862 (-2.475844) | 1.752157 / 4.565676 (-2.813520) | 0.062234 / 0.424275 (-0.362041) | 0.004927 / 0.007607 (-0.002680) | 0.337835 / 0.226044 (0.111790) | 3.313819 / 2.268929 (1.044891) | 1.834095 / 55.444624 (-53.610530) | 1.559964 / 6.876477 (-5.316513) | 1.598489 / 2.142072 (-0.543584) | 0.636829 / 4.805227 (-4.168399) | 0.116436 / 6.500664 (-6.384228) | 0.042506 / 0.075469 (-0.032963) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.951508 / 1.841788 (-0.890280) | 11.599532 / 8.074308 (3.525224) | 10.492355 / 10.191392 (0.300963) | 0.151582 / 0.680424 (-0.528842) | 0.014356 / 0.534201 (-0.519845) | 0.288448 / 0.579283 (-0.290835) | 0.265607 / 0.434364 (-0.168757) | 0.324455 / 0.540337 (-0.215883) | 0.416718 / 1.386936 (-0.970218) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005489 / 0.011353 (-0.005864) | 0.003481 / 0.011008 (-0.007527) | 0.048952 / 0.038508 (0.010444) | 0.054650 / 0.023109 (0.031540) | 0.280853 / 0.275898 (0.004955) | 0.298089 / 0.323480 (-0.025391) | 0.004762 / 0.007986 (-0.003224) | 0.002500 / 0.004328 (-0.001828) | 0.048503 / 0.004250 (0.044253) | 0.042048 / 0.037052 (0.004995) | 0.281729 / 0.258489 (0.023240) | 0.303625 / 0.293841 (0.009785) | 0.028887 / 0.128546 (-0.099659) | 0.010687 / 0.075646 (-0.064960) | 0.058093 / 0.419271 (-0.361178) | 0.032366 / 0.043533 (-0.011167) | 0.281987 / 0.255139 (0.026848) | 0.295554 / 0.283200 (0.012355) | 0.019242 / 0.141683 (-0.122441) | 1.127760 / 1.452155 (-0.324395) | 1.166868 / 1.492716 (-0.325848) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092367 / 0.018006 (0.074361) | 0.300195 / 0.000490 (0.299706) | 0.000222 / 0.000200 (0.000022) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022062 / 0.037411 (-0.015349) | 0.069955 / 0.014526 (0.055429) | 0.081224 / 0.176557 (-0.095333) | 0.120183 / 0.737135 (-0.616953) | 0.082846 / 0.296338 (-0.213492) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.295880 / 0.215209 (0.080671) | 2.902508 / 2.077655 (0.824853) | 1.616311 / 1.504120 (0.112191) | 1.491320 / 1.541195 (-0.049875) | 1.517333 / 1.468490 (0.048843) | 0.566824 / 4.584777 (-4.017953) | 2.428397 / 3.745712 (-1.317315) | 2.807241 / 5.269862 (-2.462620) | 1.786364 / 4.565676 (-2.779312) | 0.065253 / 0.424275 (-0.359022) | 0.004971 / 0.007607 (-0.002636) | 0.350095 / 0.226044 (0.124051) | 3.422226 / 2.268929 (1.153297) | 1.972955 / 55.444624 (-53.471670) | 1.686142 / 6.876477 (-5.190335) | 1.694539 / 2.142072 (-0.447533) | 0.645709 / 4.805227 (-4.159518) | 0.117774 / 6.500664 (-6.382890) | 0.041679 / 0.075469 (-0.033790) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.976835 / 1.841788 (-0.864952) | 12.358039 / 8.074308 (4.283730) | 10.774304 / 10.191392 (0.582912) | 0.130442 / 0.680424 (-0.549982) | 0.016071 / 0.534201 (-0.518130) | 0.289911 / 0.579283 (-0.289372) | 0.280693 / 0.434364 (-0.153671) | 0.325598 / 0.540337 (-0.214739) | 0.549618 / 1.386936 (-0.837318) |

</details>
</details>

![](https://cml.dev/watermark.png#1570235228b67a15dce1ed535e905edd7442117f "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005176 / 0.011353 (-0.006177) | 0.003297 / 0.011008 (-0.007711) | 0.061673 / 0.038508 (0.023165) | 0.052174 / 0.023109 (0.029065) | 0.245897 / 0.275898 (-0.030001) | 0.273102 / 0.323480 (-0.050377) | 0.003870 / 0.007986 (-0.004115) | 0.002385 / 0.004328 (-0.001943) | 0.047675 / 0.004250 (0.043424) | 0.037722 / 0.037052 (0.000670) | 0.250780 / 0.258489 (-0.007709) | 0.279464 / 0.293841 (-0.014376) | 0.028107 / 0.128546 (-0.100439) | 0.010460 / 0.075646 (-0.065187) | 0.205522 / 0.419271 (-0.213750) | 0.035781 / 0.043533 (-0.007752) | 0.246526 / 0.255139 (-0.008613) | 0.263919 / 0.283200 (-0.019281) | 0.018634 / 0.141683 (-0.123049) | 1.103845 / 1.452155 (-0.348310) | 1.175536 / 1.492716 (-0.317181) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091696 / 0.018006 (0.073690) | 0.301284 / 0.000490 (0.300794) | 0.000213 / 0.000200 (0.000013) | 0.000051 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019153 / 0.037411 (-0.018258) | 0.063846 / 0.014526 (0.049320) | 0.073635 / 0.176557 (-0.102922) | 0.119625 / 0.737135 (-0.617511) | 0.075161 / 0.296338 (-0.221177) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.285637 / 0.215209 (0.070428) | 2.751787 / 2.077655 (0.674132) | 1.465098 / 1.504120 (-0.039022) | 1.341676 / 1.541195 (-0.199519) | 1.390636 / 1.468490 (-0.077854) | 0.567663 / 4.584777 (-4.017114) | 2.378183 / 3.745712 (-1.367529) | 2.801830 / 5.269862 (-2.468032) | 1.750125 / 4.565676 (-2.815551) | 0.063705 / 0.424275 (-0.360570) | 0.004967 / 0.007607 (-0.002640) | 0.373302 / 0.226044 (0.147258) | 3.301847 / 2.268929 (1.032918) | 1.830117 / 55.444624 (-53.614508) | 1.564360 / 6.876477 (-5.312117) | 1.551766 / 2.142072 (-0.590306) | 0.654424 / 4.805227 (-4.150803) | 0.120656 / 6.500664 (-6.380008) | 0.042383 / 0.075469 (-0.033086) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.931815 / 1.841788 (-0.909973) | 11.755904 / 8.074308 (3.681596) | 10.571707 / 10.191392 (0.380315) | 0.131118 / 0.680424 (-0.549306) | 0.015241 / 0.534201 (-0.518960) | 0.287137 / 0.579283 (-0.292146) | 0.265651 / 0.434364 (-0.168713) | 0.329083 / 0.540337 (-0.211254) | 0.417501 / 1.386936 (-0.969435) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005355 / 0.011353 (-0.005998) | 0.003305 / 0.011008 (-0.007703) | 0.048289 / 0.038508 (0.009781) | 0.059223 / 0.023109 (0.036114) | 0.267213 / 0.275898 (-0.008685) | 0.290151 / 0.323480 (-0.033329) | 0.004683 / 0.007986 (-0.003303) | 0.002413 / 0.004328 (-0.001916) | 0.047982 / 0.004250 (0.043732) | 0.040943 / 0.037052 (0.003891) | 0.270967 / 0.258489 (0.012478) | 0.297644 / 0.293841 (0.003803) | 0.029309 / 0.128546 (-0.099237) | 0.010624 / 0.075646 (-0.065023) | 0.057359 / 0.419271 (-0.361913) | 0.032716 / 0.043533 (-0.010816) | 0.268602 / 0.255139 (0.013463) | 0.286016 / 0.283200 (0.002817) | 0.018578 / 0.141683 (-0.123105) | 1.120275 / 1.452155 (-0.331880) | 1.195514 / 1.492716 (-0.297202) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092590 / 0.018006 (0.074584) | 0.302589 / 0.000490 (0.302099) | 0.000217 / 0.000200 (0.000017) | 0.000043 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022439 / 0.037411 (-0.014972) | 0.070914 / 0.014526 (0.056388) | 0.084927 / 0.176557 (-0.091629) | 0.123154 / 0.737135 (-0.613981) | 0.085527 / 0.296338 (-0.210812) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.292652 / 0.215209 (0.077443) | 2.843736 / 2.077655 (0.766081) | 1.561289 / 1.504120 (0.057169) | 1.439500 / 1.541195 (-0.101695) | 1.485074 / 1.468490 (0.016584) | 0.570520 / 4.584777 (-4.014257) | 2.436611 / 3.745712 (-1.309102) | 2.925600 / 5.269862 (-2.344261) | 1.796518 / 4.565676 (-2.769159) | 0.065075 / 0.424275 (-0.359200) | 0.004995 / 0.007607 (-0.002612) | 0.349976 / 0.226044 (0.123932) | 3.442535 / 2.268929 (1.173607) | 1.919002 / 55.444624 (-53.525622) | 1.659222 / 6.876477 (-5.217255) | 1.648370 / 2.142072 (-0.493703) | 0.643119 / 4.805227 (-4.162108) | 0.118015 / 6.500664 (-6.382649) | 0.041229 / 0.075469 (-0.034240) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.986226 / 1.841788 (-0.855562) | 12.302487 / 8.074308 (4.228179) | 10.528848 / 10.191392 (0.337456) | 0.143911 / 0.680424 (-0.536513) | 0.015265 / 0.534201 (-0.518936) | 0.287692 / 0.579283 (-0.291591) | 0.277011 / 0.434364 (-0.157353) | 0.327650 / 0.540337 (-0.212688) | 0.552951 / 1.386936 (-0.833985) |

</details>
</details>

![](https://cml.dev/watermark.png#0af18e68664db94e863f0dcde4b0f3a7adcc80e7 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005234 / 0.011353 (-0.006119) | 0.003324 / 0.011008 (-0.007684) | 0.062429 / 0.038508 (0.023921) | 0.051619 / 0.023109 (0.028510) | 0.256850 / 0.275898 (-0.019048) | 0.260566 / 0.323480 (-0.062914) | 0.002914 / 0.007986 (-0.005071) | 0.003093 / 0.004328 (-0.001235) | 0.047947 / 0.004250 (0.043696) | 0.038753 / 0.037052 (0.001701) | 0.246810 / 0.258489 (-0.011679) | 0.275128 / 0.293841 (-0.018713) | 0.027171 / 0.128546 (-0.101375) | 0.010290 / 0.075646 (-0.065356) | 0.206069 / 0.419271 (-0.213203) | 0.035514 / 0.043533 (-0.008019) | 0.240645 / 0.255139 (-0.014494) | 0.259693 / 0.283200 (-0.023507) | 0.019722 / 0.141683 (-0.121961) | 1.128534 / 1.452155 (-0.323620) | 1.139602 / 1.492716 (-0.353115) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095837 / 0.018006 (0.077830) | 0.304754 / 0.000490 (0.304264) | 0.000204 / 0.000200 (0.000004) | 0.000043 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018349 / 0.037411 (-0.019063) | 0.062763 / 0.014526 (0.048237) | 0.074443 / 0.176557 (-0.102113) | 0.120607 / 0.737135 (-0.616528) | 0.077721 / 0.296338 (-0.218617) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.281852 / 0.215209 (0.066643) | 2.770806 / 2.077655 (0.693151) | 1.466255 / 1.504120 (-0.037864) | 1.349611 / 1.541195 (-0.191584) | 1.385463 / 1.468490 (-0.083027) | 0.566489 / 4.584777 (-4.018288) | 2.420932 / 3.745712 (-1.324780) | 2.809397 / 5.269862 (-2.460464) | 1.749734 / 4.565676 (-2.815942) | 0.063407 / 0.424275 (-0.360868) | 0.005038 / 0.007607 (-0.002569) | 0.379121 / 0.226044 (0.153077) | 3.500938 / 2.268929 (1.232010) | 1.852207 / 55.444624 (-53.592417) | 1.570474 / 6.876477 (-5.306002) | 1.555222 / 2.142072 (-0.586850) | 0.657198 / 4.805227 (-4.148030) | 0.119835 / 6.500664 (-6.380829) | 0.042453 / 0.075469 (-0.033016) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.949953 / 1.841788 (-0.891835) | 11.736811 / 8.074308 (3.662503) | 10.558049 / 10.191392 (0.366657) | 0.146230 / 0.680424 (-0.534194) | 0.014922 / 0.534201 (-0.519279) | 0.289100 / 0.579283 (-0.290183) | 0.267130 / 0.434364 (-0.167234) | 0.320055 / 0.540337 (-0.220282) | 0.417244 / 1.386936 (-0.969692) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005309 / 0.011353 (-0.006044) | 0.003329 / 0.011008 (-0.007679) | 0.048576 / 0.038508 (0.010068) | 0.055219 / 0.023109 (0.032110) | 0.271522 / 0.275898 (-0.004376) | 0.294435 / 0.323480 (-0.029045) | 0.004018 / 0.007986 (-0.003968) | 0.002456 / 0.004328 (-0.001873) | 0.047939 / 0.004250 (0.043689) | 0.041195 / 0.037052 (0.004143) | 0.274819 / 0.258489 (0.016330) | 0.299407 / 0.293841 (0.005566) | 0.029145 / 0.128546 (-0.099401) | 0.010680 / 0.075646 (-0.064966) | 0.057238 / 0.419271 (-0.362034) | 0.032722 / 0.043533 (-0.010810) | 0.272066 / 0.255139 (0.016927) | 0.289223 / 0.283200 (0.006023) | 0.017826 / 0.141683 (-0.123857) | 1.119079 / 1.452155 (-0.333076) | 1.179109 / 1.492716 (-0.313608) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095662 / 0.018006 (0.077656) | 0.307652 / 0.000490 (0.307162) | 0.000213 / 0.000200 (0.000013) | 0.000051 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022263 / 0.037411 (-0.015149) | 0.070224 / 0.014526 (0.055698) | 0.081477 / 0.176557 (-0.095079) | 0.120763 / 0.737135 (-0.616372) | 0.083152 / 0.296338 (-0.213187) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.295780 / 0.215209 (0.080571) | 2.926623 / 2.077655 (0.848968) | 1.605901 / 1.504120 (0.101781) | 1.482874 / 1.541195 (-0.058321) | 1.501467 / 1.468490 (0.032977) | 0.569566 / 4.584777 (-4.015211) | 2.474948 / 3.745712 (-1.270764) | 2.831877 / 5.269862 (-2.437985) | 1.761229 / 4.565676 (-2.804448) | 0.064129 / 0.424275 (-0.360147) | 0.004964 / 0.007607 (-0.002643) | 0.350081 / 0.226044 (0.124037) | 3.446766 / 2.268929 (1.177837) | 1.974998 / 55.444624 (-53.469627) | 1.683381 / 6.876477 (-5.193095) | 1.711543 / 2.142072 (-0.430530) | 0.648695 / 4.805227 (-4.156532) | 0.118224 / 6.500664 (-6.382440) | 0.040895 / 0.075469 (-0.034574) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.960208 / 1.841788 (-0.881580) | 12.164941 / 8.074308 (4.090633) | 10.860573 / 10.191392 (0.669181) | 0.133525 / 0.680424 (-0.546899) | 0.015643 / 0.534201 (-0.518558) | 0.290898 / 0.579283 (-0.288386) | 0.289612 / 0.434364 (-0.144752) | 0.325836 / 0.540337 (-0.214501) | 0.565592 / 1.386936 (-0.821344) |

</details>
</details>

![](https://cml.dev/watermark.png#9d19a315920c6d4293f8226273d99bf3de5c1d4e "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006097 / 0.011353 (-0.005256) | 0.004386 / 0.011008 (-0.006622) | 0.064481 / 0.038508 (0.025973) | 0.059983 / 0.023109 (0.036873) | 0.268177 / 0.275898 (-0.007721) | 0.296207 / 0.323480 (-0.027273) | 0.002986 / 0.007986 (-0.005000) | 0.002923 / 0.004328 (-0.001406) | 0.048798 / 0.004250 (0.044547) | 0.039945 / 0.037052 (0.002893) | 0.271234 / 0.258489 (0.012745) | 0.295461 / 0.293841 (0.001620) | 0.028771 / 0.128546 (-0.099775) | 0.011104 / 0.075646 (-0.064542) | 0.207471 / 0.419271 (-0.211800) | 0.036955 / 0.043533 (-0.006578) | 0.254761 / 0.255139 (-0.000378) | 0.275933 / 0.283200 (-0.007267) | 0.021232 / 0.141683 (-0.120451) | 1.170771 / 1.452155 (-0.281384) | 1.188900 / 1.492716 (-0.303816) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092328 / 0.018006 (0.074322) | 0.302591 / 0.000490 (0.302102) | 0.000220 / 0.000200 (0.000020) | 0.000052 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019207 / 0.037411 (-0.018204) | 0.070247 / 0.014526 (0.055721) | 0.074963 / 0.176557 (-0.101593) | 0.124301 / 0.737135 (-0.612834) | 0.077356 / 0.296338 (-0.218982) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.283321 / 0.215209 (0.068112) | 2.800448 / 2.077655 (0.722793) | 1.510278 / 1.504120 (0.006158) | 1.390353 / 1.541195 (-0.150842) | 1.387881 / 1.468490 (-0.080609) | 0.563927 / 4.584777 (-4.020850) | 2.387753 / 3.745712 (-1.357959) | 2.776655 / 5.269862 (-2.493207) | 1.767383 / 4.565676 (-2.798293) | 0.064864 / 0.424275 (-0.359411) | 0.004999 / 0.007607 (-0.002608) | 0.351173 / 0.226044 (0.125129) | 3.459446 / 2.268929 (1.190517) | 1.873078 / 55.444624 (-53.571547) | 1.602831 / 6.876477 (-5.273646) | 1.595612 / 2.142072 (-0.546460) | 0.648786 / 4.805227 (-4.156441) | 0.118720 / 6.500664 (-6.381944) | 0.042821 / 0.075469 (-0.032649) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.970738 / 1.841788 (-0.871049) | 12.273548 / 8.074308 (4.199240) | 11.191375 / 10.191392 (0.999983) | 0.131903 / 0.680424 (-0.548521) | 0.014512 / 0.534201 (-0.519689) | 0.289382 / 0.579283 (-0.289901) | 0.269449 / 0.434364 (-0.164915) | 0.327557 / 0.540337 (-0.212781) | 0.427052 / 1.386936 (-0.959884) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005472 / 0.011353 (-0.005881) | 0.003380 / 0.011008 (-0.007628) | 0.050677 / 0.038508 (0.012169) | 0.059606 / 0.023109 (0.036497) | 0.275798 / 0.275898 (-0.000100) | 0.303733 / 0.323480 (-0.019747) | 0.004187 / 0.007986 (-0.003799) | 0.002657 / 0.004328 (-0.001672) | 0.048713 / 0.004250 (0.044463) | 0.043501 / 0.037052 (0.006449) | 0.278845 / 0.258489 (0.020356) | 0.305322 / 0.293841 (0.011481) | 0.030665 / 0.128546 (-0.097881) | 0.010600 / 0.075646 (-0.065047) | 0.058923 / 0.419271 (-0.360349) | 0.032936 / 0.043533 (-0.010596) | 0.272835 / 0.255139 (0.017696) | 0.293975 / 0.283200 (0.010775) | 0.018193 / 0.141683 (-0.123490) | 1.144903 / 1.452155 (-0.307251) | 1.192220 / 1.492716 (-0.300497) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094519 / 0.018006 (0.076513) | 0.305591 / 0.000490 (0.305101) | 0.000221 / 0.000200 (0.000021) | 0.000056 / 0.000054 (0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022108 / 0.037411 (-0.015303) | 0.070184 / 0.014526 (0.055658) | 0.081640 / 0.176557 (-0.094916) | 0.124661 / 0.737135 (-0.612474) | 0.082229 / 0.296338 (-0.214110) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.303710 / 0.215209 (0.088501) | 2.966478 / 2.077655 (0.888824) | 1.646066 / 1.504120 (0.141946) | 1.551454 / 1.541195 (0.010259) | 1.557995 / 1.468490 (0.089505) | 0.577723 / 4.584777 (-4.007054) | 2.510321 / 3.745712 (-1.235391) | 2.951343 / 5.269862 (-2.318519) | 1.857550 / 4.565676 (-2.708127) | 0.064079 / 0.424275 (-0.360196) | 0.004971 / 0.007607 (-0.002636) | 0.359022 / 0.226044 (0.132978) | 3.628716 / 2.268929 (1.359788) | 2.011380 / 55.444624 (-53.433245) | 1.710407 / 6.876477 (-5.166070) | 1.756235 / 2.142072 (-0.385838) | 0.659185 / 4.805227 (-4.146042) | 0.120245 / 6.500664 (-6.380419) | 0.042751 / 0.075469 (-0.032718) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.026794 / 1.841788 (-0.814993) | 12.695125 / 8.074308 (4.620816) | 10.864908 / 10.191392 (0.673516) | 0.136128 / 0.680424 (-0.544295) | 0.016824 / 0.534201 (-0.517377) | 0.289717 / 0.579283 (-0.289567) | 0.282919 / 0.434364 (-0.151445) | 0.323345 / 0.540337 (-0.216992) | 0.556375 / 1.386936 (-0.830561) |

</details>
</details>

![](https://cml.dev/watermark.png#52207295162f734235b71428d13e6a42c6fdc370 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005407 / 0.011353 (-0.005946) | 0.003464 / 0.011008 (-0.007544) | 0.062084 / 0.038508 (0.023576) | 0.052582 / 0.023109 (0.029472) | 0.251239 / 0.275898 (-0.024659) | 0.276675 / 0.323480 (-0.046805) | 0.002894 / 0.007986 (-0.005092) | 0.003850 / 0.004328 (-0.000479) | 0.047789 / 0.004250 (0.043538) | 0.038955 / 0.037052 (0.001903) | 0.258333 / 0.258489 (-0.000156) | 0.290103 / 0.293841 (-0.003738) | 0.027291 / 0.128546 (-0.101256) | 0.010575 / 0.075646 (-0.065071) | 0.207208 / 0.419271 (-0.212063) | 0.035848 / 0.043533 (-0.007685) | 0.253918 / 0.255139 (-0.001221) | 0.269870 / 0.283200 (-0.013330) | 0.019830 / 0.141683 (-0.121853) | 1.085332 / 1.452155 (-0.366823) | 1.171385 / 1.492716 (-0.321331) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094956 / 0.018006 (0.076950) | 0.301104 / 0.000490 (0.300614) | 0.000204 / 0.000200 (0.000004) | 0.000049 / 0.000054 (-0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019045 / 0.037411 (-0.018367) | 0.070815 / 0.014526 (0.056289) | 0.073763 / 0.176557 (-0.102794) | 0.120668 / 0.737135 (-0.616467) | 0.075197 / 0.296338 (-0.221141) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.286072 / 0.215209 (0.070863) | 2.762868 / 2.077655 (0.685213) | 1.504481 / 1.504120 (0.000361) | 1.390301 / 1.541195 (-0.150894) | 1.449571 / 1.468490 (-0.018919) | 0.555598 / 4.584777 (-4.029179) | 2.404975 / 3.745712 (-1.340737) | 2.864359 / 5.269862 (-2.405503) | 1.764913 / 4.565676 (-2.800763) | 0.062956 / 0.424275 (-0.361320) | 0.005116 / 0.007607 (-0.002491) | 0.344027 / 0.226044 (0.117983) | 3.426781 / 2.268929 (1.157852) | 1.891040 / 55.444624 (-53.553584) | 1.599972 / 6.876477 (-5.276505) | 1.603464 / 2.142072 (-0.538608) | 0.638136 / 4.805227 (-4.167091) | 0.117808 / 6.500664 (-6.382857) | 0.043740 / 0.075469 (-0.031730) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.934654 / 1.841788 (-0.907133) | 12.243698 / 8.074308 (4.169390) | 10.566791 / 10.191392 (0.375399) | 0.130440 / 0.680424 (-0.549983) | 0.014019 / 0.534201 (-0.520182) | 0.285453 / 0.579283 (-0.293831) | 0.266121 / 0.434364 (-0.168243) | 0.325962 / 0.540337 (-0.214375) | 0.422181 / 1.386936 (-0.964755) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005151 / 0.011353 (-0.006202) | 0.003704 / 0.011008 (-0.007304) | 0.049483 / 0.038508 (0.010975) | 0.055147 / 0.023109 (0.032038) | 0.277589 / 0.275898 (0.001691) | 0.301274 / 0.323480 (-0.022206) | 0.004031 / 0.007986 (-0.003955) | 0.002568 / 0.004328 (-0.001760) | 0.048830 / 0.004250 (0.044580) | 0.040391 / 0.037052 (0.003339) | 0.281031 / 0.258489 (0.022541) | 0.304263 / 0.293841 (0.010422) | 0.029237 / 0.128546 (-0.099309) | 0.010598 / 0.075646 (-0.065048) | 0.058089 / 0.419271 (-0.361182) | 0.032529 / 0.043533 (-0.011004) | 0.275761 / 0.255139 (0.020622) | 0.294427 / 0.283200 (0.011227) | 0.017227 / 0.141683 (-0.124456) | 1.138036 / 1.452155 (-0.314119) | 1.201946 / 1.492716 (-0.290770) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094241 / 0.018006 (0.076234) | 0.301622 / 0.000490 (0.301132) | 0.000229 / 0.000200 (0.000029) | 0.000054 / 0.000054 (-0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022731 / 0.037411 (-0.014680) | 0.071217 / 0.014526 (0.056691) | 0.082619 / 0.176557 (-0.093937) | 0.123308 / 0.737135 (-0.613827) | 0.083552 / 0.296338 (-0.212787) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.295770 / 0.215209 (0.080561) | 2.886069 / 2.077655 (0.808414) | 1.597686 / 1.504120 (0.093566) | 1.458612 / 1.541195 (-0.082583) | 1.501171 / 1.468490 (0.032680) | 0.575653 / 4.584777 (-4.009124) | 2.444021 / 3.745712 (-1.301691) | 2.860192 / 5.269862 (-2.409669) | 1.758896 / 4.565676 (-2.806780) | 0.063334 / 0.424275 (-0.360941) | 0.004913 / 0.007607 (-0.002694) | 0.341828 / 0.226044 (0.115783) | 3.420310 / 2.268929 (1.151381) | 1.996099 / 55.444624 (-53.448525) | 1.680112 / 6.876477 (-5.196365) | 1.693418 / 2.142072 (-0.448654) | 0.697321 / 4.805227 (-4.107906) | 0.120474 / 6.500664 (-6.380190) | 0.042192 / 0.075469 (-0.033277) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.975876 / 1.841788 (-0.865912) | 12.174933 / 8.074308 (4.100625) | 10.400906 / 10.191392 (0.209514) | 0.162244 / 0.680424 (-0.518180) | 0.016443 / 0.534201 (-0.517758) | 0.293430 / 0.579283 (-0.285853) | 0.285664 / 0.434364 (-0.148700) | 0.332322 / 0.540337 (-0.208015) | 0.609815 / 1.386936 (-0.777121) |

</details>
</details>

![](https://cml.dev/watermark.png#f2c417d087d232b5abf9054ffb10305cc06c5440 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005155 / 0.011353 (-0.006198) | 0.003226 / 0.011008 (-0.007782) | 0.062651 / 0.038508 (0.024143) | 0.051314 / 0.023109 (0.028205) | 0.246075 / 0.275898 (-0.029823) | 0.266859 / 0.323480 (-0.056621) | 0.003895 / 0.007986 (-0.004091) | 0.002462 / 0.004328 (-0.001866) | 0.048097 / 0.004250 (0.043846) | 0.037313 / 0.037052 (0.000261) | 0.253208 / 0.258489 (-0.005281) | 0.280255 / 0.293841 (-0.013585) | 0.027052 / 0.128546 (-0.101494) | 0.010276 / 0.075646 (-0.065370) | 0.205663 / 0.419271 (-0.213608) | 0.035111 / 0.043533 (-0.008422) | 0.253757 / 0.255139 (-0.001382) | 0.265466 / 0.283200 (-0.017733) | 0.017873 / 0.141683 (-0.123810) | 1.118906 / 1.452155 (-0.333249) | 1.176384 / 1.492716 (-0.316332) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094921 / 0.018006 (0.076914) | 0.300459 / 0.000490 (0.299970) | 0.000214 / 0.000200 (0.000014) | 0.000042 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018430 / 0.037411 (-0.018981) | 0.062690 / 0.014526 (0.048165) | 0.074215 / 0.176557 (-0.102342) | 0.119969 / 0.737135 (-0.617166) | 0.075846 / 0.296338 (-0.220493) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.273492 / 0.215209 (0.058283) | 2.667937 / 2.077655 (0.590282) | 1.405912 / 1.504120 (-0.098208) | 1.269041 / 1.541195 (-0.272153) | 1.313461 / 1.468490 (-0.155029) | 0.554633 / 4.584777 (-4.030144) | 2.325552 / 3.745712 (-1.420160) | 2.825580 / 5.269862 (-2.444282) | 1.745432 / 4.565676 (-2.820245) | 0.062497 / 0.424275 (-0.361778) | 0.004935 / 0.007607 (-0.002673) | 0.337045 / 0.226044 (0.111001) | 3.246360 / 2.268929 (0.977432) | 1.775329 / 55.444624 (-53.669296) | 1.491812 / 6.876477 (-5.384665) | 1.499783 / 2.142072 (-0.642290) | 0.636768 / 4.805227 (-4.168459) | 0.116471 / 6.500664 (-6.384193) | 0.041838 / 0.075469 (-0.033631) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.937388 / 1.841788 (-0.904400) | 11.950930 / 8.074308 (3.876622) | 10.532062 / 10.191392 (0.340670) | 0.129490 / 0.680424 (-0.550934) | 0.013907 / 0.534201 (-0.520294) | 0.287503 / 0.579283 (-0.291780) | 0.270548 / 0.434364 (-0.163816) | 0.324321 / 0.540337 (-0.216016) | 0.427639 / 1.386936 (-0.959297) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005272 / 0.011353 (-0.006081) | 0.003413 / 0.011008 (-0.007595) | 0.049800 / 0.038508 (0.011292) | 0.055978 / 0.023109 (0.032868) | 0.274365 / 0.275898 (-0.001533) | 0.293414 / 0.323480 (-0.030066) | 0.003994 / 0.007986 (-0.003992) | 0.002480 / 0.004328 (-0.001848) | 0.048787 / 0.004250 (0.044537) | 0.040520 / 0.037052 (0.003468) | 0.276198 / 0.258489 (0.017709) | 0.301085 / 0.293841 (0.007244) | 0.028352 / 0.128546 (-0.100194) | 0.010631 / 0.075646 (-0.065015) | 0.057103 / 0.419271 (-0.362168) | 0.032277 / 0.043533 (-0.011256) | 0.274472 / 0.255139 (0.019333) | 0.289953 / 0.283200 (0.006754) | 0.018048 / 0.141683 (-0.123635) | 1.120329 / 1.452155 (-0.331826) | 1.175784 / 1.492716 (-0.316932) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.102519 / 0.018006 (0.084512) | 0.322030 / 0.000490 (0.321540) | 0.000234 / 0.000200 (0.000034) | 0.000045 / 0.000054 (-0.000009) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023084 / 0.037411 (-0.014327) | 0.069592 / 0.014526 (0.055066) | 0.081293 / 0.176557 (-0.095264) | 0.119546 / 0.737135 (-0.617589) | 0.083249 / 0.296338 (-0.213090) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.294997 / 0.215209 (0.079788) | 2.925517 / 2.077655 (0.847863) | 1.607824 / 1.504120 (0.103705) | 1.469586 / 1.541195 (-0.071608) | 1.492350 / 1.468490 (0.023860) | 0.561351 / 4.584777 (-4.023426) | 2.446741 / 3.745712 (-1.298972) | 2.842588 / 5.269862 (-2.427273) | 1.789189 / 4.565676 (-2.776487) | 0.064064 / 0.424275 (-0.360211) | 0.005011 / 0.007607 (-0.002597) | 0.351059 / 0.226044 (0.125015) | 3.485277 / 2.268929 (1.216348) | 1.981821 / 55.444624 (-53.462803) | 1.671846 / 6.876477 (-5.204631) | 1.702014 / 2.142072 (-0.440058) | 0.645205 / 4.805227 (-4.160023) | 0.117358 / 6.500664 (-6.383306) | 0.041633 / 0.075469 (-0.033836) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.963281 / 1.841788 (-0.878506) | 12.141256 / 8.074308 (4.066947) | 10.595207 / 10.191392 (0.403815) | 0.130401 / 0.680424 (-0.550023) | 0.015490 / 0.534201 (-0.518710) | 0.284201 / 0.579283 (-0.295082) | 0.280244 / 0.434364 (-0.154120) | 0.323545 / 0.540337 (-0.216792) | 0.561246 / 1.386936 (-0.825690) |

</details>
</details>

![](https://cml.dev/watermark.png#b3193829cf0dd9888c42bd7640a71d9d656cba2a "CML watermark")