Title: Unpin `jax` maximum version
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2023-10-12T14:42:40Z

fix #6299
fix #6202

Comments:
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008410 / 0.011353 (-0.002943) | 0.004888 / 0.011008 (-0.006120) | 0.103342 / 0.038508 (0.064834) | 0.103697 / 0.023109 (0.080587) | 0.416445 / 0.275898 (0.140547) | 0.454604 / 0.323480 (0.131124) | 0.004976 / 0.007986 (-0.003010) | 0.003957 / 0.004328 (-0.000371) | 0.077398 / 0.004250 (0.073148) | 0.069026 / 0.037052 (0.031973) | 0.420484 / 0.258489 (0.161995) | 0.471828 / 0.293841 (0.177987) | 0.037133 / 0.128546 (-0.091413) | 0.010009 / 0.075646 (-0.065637) | 0.349573 / 0.419271 (-0.069698) | 0.063240 / 0.043533 (0.019708) | 0.421554 / 0.255139 (0.166415) | 0.433548 / 0.283200 (0.150348) | 0.029397 / 0.141683 (-0.112286) | 1.716860 / 1.452155 (0.264705) | 1.851264 / 1.492716 (0.358547) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.269733 / 0.018006 (0.251727) | 0.493313 / 0.000490 (0.492823) | 0.010438 / 0.000200 (0.010238) | 0.000401 / 0.000054 (0.000347) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034690 / 0.037411 (-0.002722) | 0.105304 / 0.014526 (0.090778) | 0.115831 / 0.176557 (-0.060726) | 0.185017 / 0.737135 (-0.552118) | 0.117480 / 0.296338 (-0.178859) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.479414 / 0.215209 (0.264205) | 4.785526 / 2.077655 (2.707871) | 2.388412 / 1.504120 (0.884292) | 2.178222 / 1.541195 (0.637027) | 2.248214 / 1.468490 (0.779723) | 0.571723 / 4.584777 (-4.013054) | 4.721250 / 3.745712 (0.975538) | 4.073893 / 5.269862 (-1.195969) | 2.618131 / 4.565676 (-1.947546) | 0.068406 / 0.424275 (-0.355869) | 0.008890 / 0.007607 (0.001283) | 0.564224 / 0.226044 (0.338180) | 5.631412 / 2.268929 (3.362483) | 3.072212 / 55.444624 (-52.372412) | 2.760574 / 6.876477 (-4.115903) | 2.963060 / 2.142072 (0.820987) | 0.708150 / 4.805227 (-4.097077) | 0.160324 / 6.500664 (-6.340340) | 0.075402 / 0.075469 (-0.000067) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.649965 / 1.841788 (-0.191823) | 24.297517 / 8.074308 (16.223209) | 17.658675 / 10.191392 (7.467283) | 0.171399 / 0.680424 (-0.509025) | 0.021172 / 0.534201 (-0.513029) | 0.477196 / 0.579283 (-0.102087) | 0.503900 / 0.434364 (0.069536) | 0.555858 / 0.540337 (0.015520) | 0.824302 / 1.386936 (-0.562634) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008613 / 0.011353 (-0.002740) | 0.004848 / 0.011008 (-0.006160) | 0.078344 / 0.038508 (0.039836) | 0.098976 / 0.023109 (0.075867) | 0.520713 / 0.275898 (0.244815) | 0.566350 / 0.323480 (0.242870) | 0.006658 / 0.007986 (-0.001327) | 0.004043 / 0.004328 (-0.000285) | 0.077881 / 0.004250 (0.073631) | 0.070731 / 0.037052 (0.033678) | 0.519717 / 0.258489 (0.261228) | 0.575623 / 0.293841 (0.281782) | 0.038542 / 0.128546 (-0.090004) | 0.010277 / 0.075646 (-0.065369) | 0.084269 / 0.419271 (-0.335002) | 0.058088 / 0.043533 (0.014555) | 0.541790 / 0.255139 (0.286651) | 0.534915 / 0.283200 (0.251715) | 0.027851 / 0.141683 (-0.113831) | 1.814827 / 1.452155 (0.362672) | 1.898208 / 1.492716 (0.405492) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.244162 / 0.018006 (0.226156) | 0.482895 / 0.000490 (0.482405) | 0.005734 / 0.000200 (0.005534) | 0.000127 / 0.000054 (0.000072) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.039328 / 0.037411 (0.001917) | 0.119795 / 0.014526 (0.105269) | 0.128570 / 0.176557 (-0.047986) | 0.191207 / 0.737135 (-0.545929) | 0.127147 / 0.296338 (-0.169192) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.533545 / 0.215209 (0.318336) | 5.320135 / 2.077655 (3.242480) | 2.924573 / 1.504120 (1.420453) | 2.741351 / 1.541195 (1.200156) | 2.824217 / 1.468490 (1.355727) | 0.595842 / 4.584777 (-3.988935) | 4.343499 / 3.745712 (0.597787) | 3.976546 / 5.269862 (-1.293316) | 2.532541 / 4.565676 (-2.033135) | 0.070480 / 0.424275 (-0.353795) | 0.008868 / 0.007607 (0.001260) | 0.634297 / 0.226044 (0.408253) | 6.327314 / 2.268929 (4.058386) | 3.530741 / 55.444624 (-51.913883) | 3.121435 / 6.876477 (-3.755042) | 3.344473 / 2.142072 (1.202401) | 0.719413 / 4.805227 (-4.085814) | 0.162348 / 6.500664 (-6.338316) | 0.074964 / 0.075469 (-0.000505) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.679095 / 1.841788 (-0.162693) | 25.071620 / 8.074308 (16.997312) | 18.422398 / 10.191392 (8.231006) | 0.223981 / 0.680424 (-0.456443) | 0.026537 / 0.534201 (-0.507664) | 0.513867 / 0.579283 (-0.065416) | 0.535874 / 0.434364 (0.101510) | 0.567971 / 0.540337 (0.027634) | 0.842545 / 1.386936 (-0.544391) |

</details>
</details>

![](https://cml.dev/watermark.png#d8b871016c25cb3b90ac1ff65a4e54f0454f525e "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006445 / 0.011353 (-0.004908) | 0.003978 / 0.011008 (-0.007030) | 0.084542 / 0.038508 (0.046034) | 0.069231 / 0.023109 (0.046122) | 0.308794 / 0.275898 (0.032896) | 0.339246 / 0.323480 (0.015766) | 0.005269 / 0.007986 (-0.002716) | 0.003285 / 0.004328 (-0.001043) | 0.065336 / 0.004250 (0.061086) | 0.053480 / 0.037052 (0.016428) | 0.316775 / 0.258489 (0.058286) | 0.357885 / 0.293841 (0.064044) | 0.031309 / 0.128546 (-0.097237) | 0.008450 / 0.075646 (-0.067196) | 0.287911 / 0.419271 (-0.131361) | 0.052756 / 0.043533 (0.009223) | 0.321516 / 0.255139 (0.066377) | 0.331998 / 0.283200 (0.048799) | 0.024129 / 0.141683 (-0.117553) | 1.507718 / 1.452155 (0.055563) | 1.571400 / 1.492716 (0.078683) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.237536 / 0.018006 (0.219530) | 0.499691 / 0.000490 (0.499201) | 0.007644 / 0.000200 (0.007444) | 0.000284 / 0.000054 (0.000230) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028243 / 0.037411 (-0.009168) | 0.081556 / 0.014526 (0.067030) | 0.096877 / 0.176557 (-0.079680) | 0.149985 / 0.737135 (-0.587150) | 0.095556 / 0.296338 (-0.200783) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.383215 / 0.215209 (0.168006) | 3.815800 / 2.077655 (1.738145) | 1.832227 / 1.504120 (0.328107) | 1.664001 / 1.541195 (0.122806) | 1.698786 / 1.468490 (0.230296) | 0.487594 / 4.584777 (-4.097183) | 3.569767 / 3.745712 (-0.175945) | 3.262387 / 5.269862 (-2.007475) | 2.017105 / 4.565676 (-2.548572) | 0.057555 / 0.424275 (-0.366720) | 0.007170 / 0.007607 (-0.000437) | 0.460134 / 0.226044 (0.234090) | 4.629800 / 2.268929 (2.360871) | 2.357126 / 55.444624 (-53.087499) | 1.970144 / 6.876477 (-4.906332) | 2.123520 / 2.142072 (-0.018552) | 0.613058 / 4.805227 (-4.192169) | 0.135869 / 6.500664 (-6.364795) | 0.061292 / 0.075469 (-0.014177) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.311294 / 1.841788 (-0.530494) | 18.640807 / 8.074308 (10.566499) | 13.946834 / 10.191392 (3.755442) | 0.163976 / 0.680424 (-0.516448) | 0.018527 / 0.534201 (-0.515674) | 0.390530 / 0.579283 (-0.188753) | 0.412661 / 0.434364 (-0.021703) | 0.459514 / 0.540337 (-0.080823) | 0.635026 / 1.386936 (-0.751910) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006645 / 0.011353 (-0.004708) | 0.003943 / 0.011008 (-0.007066) | 0.064470 / 0.038508 (0.025962) | 0.069895 / 0.023109 (0.046786) | 0.411091 / 0.275898 (0.135193) | 0.437628 / 0.323480 (0.114148) | 0.005214 / 0.007986 (-0.002772) | 0.003281 / 0.004328 (-0.001047) | 0.064434 / 0.004250 (0.060183) | 0.054294 / 0.037052 (0.017241) | 0.413576 / 0.258489 (0.155087) | 0.448793 / 0.293841 (0.154952) | 0.031754 / 0.128546 (-0.096793) | 0.008530 / 0.075646 (-0.067117) | 0.069950 / 0.419271 (-0.349322) | 0.047747 / 0.043533 (0.004214) | 0.411241 / 0.255139 (0.156102) | 0.430076 / 0.283200 (0.146876) | 0.023462 / 0.141683 (-0.118220) | 1.519501 / 1.452155 (0.067346) | 1.575782 / 1.492716 (0.083066) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.231816 / 0.018006 (0.213810) | 0.442802 / 0.000490 (0.442312) | 0.005738 / 0.000200 (0.005539) | 0.000087 / 0.000054 (0.000032) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031426 / 0.037411 (-0.005985) | 0.090758 / 0.014526 (0.076233) | 0.103414 / 0.176557 (-0.073142) | 0.156409 / 0.737135 (-0.580726) | 0.103900 / 0.296338 (-0.192439) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.438897 / 0.215209 (0.223688) | 4.385318 / 2.077655 (2.307663) | 2.352042 / 1.504120 (0.847923) | 2.182228 / 1.541195 (0.641033) | 2.266256 / 1.468490 (0.797766) | 0.492780 / 4.584777 (-4.091997) | 3.665787 / 3.745712 (-0.079925) | 3.315329 / 5.269862 (-1.954533) | 2.027993 / 4.565676 (-2.537684) | 0.058220 / 0.424275 (-0.366055) | 0.007429 / 0.007607 (-0.000178) | 0.508790 / 0.226044 (0.282746) | 5.107093 / 2.268929 (2.838164) | 2.799789 / 55.444624 (-52.644836) | 2.462828 / 6.876477 (-4.413649) | 2.610193 / 2.142072 (0.468120) | 0.588133 / 4.805227 (-4.217094) | 0.133418 / 6.500664 (-6.367246) | 0.059793 / 0.075469 (-0.015676) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.363358 / 1.841788 (-0.478430) | 19.258372 / 8.074308 (11.184064) | 14.730977 / 10.191392 (4.539584) | 0.169493 / 0.680424 (-0.510931) | 0.020462 / 0.534201 (-0.513739) | 0.397980 / 0.579283 (-0.181303) | 0.426638 / 0.434364 (-0.007726) | 0.474249 / 0.540337 (-0.066088) | 0.677640 / 1.386936 (-0.709296) |

</details>
</details>

![](https://cml.dev/watermark.png#90b3d2619ecb8f01dd12283c30f04dfe6e443795 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006536 / 0.011353 (-0.004817) | 0.003827 / 0.011008 (-0.007181) | 0.084394 / 0.038508 (0.045886) | 0.073166 / 0.023109 (0.050056) | 0.309380 / 0.275898 (0.033482) | 0.338501 / 0.323480 (0.015021) | 0.005346 / 0.007986 (-0.002640) | 0.003273 / 0.004328 (-0.001056) | 0.064606 / 0.004250 (0.060356) | 0.053500 / 0.037052 (0.016447) | 0.313143 / 0.258489 (0.054654) | 0.354364 / 0.293841 (0.060523) | 0.030919 / 0.128546 (-0.097627) | 0.008512 / 0.075646 (-0.067134) | 0.292774 / 0.419271 (-0.126498) | 0.052441 / 0.043533 (0.008908) | 0.310503 / 0.255139 (0.055364) | 0.341211 / 0.283200 (0.058011) | 0.023608 / 0.141683 (-0.118074) | 1.456220 / 1.452155 (0.004065) | 1.540189 / 1.492716 (0.047473) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.234321 / 0.018006 (0.216315) | 0.451809 / 0.000490 (0.451319) | 0.008560 / 0.000200 (0.008360) | 0.000085 / 0.000054 (0.000031) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028165 / 0.037411 (-0.009246) | 0.082548 / 0.014526 (0.068023) | 0.752621 / 0.176557 (0.576065) | 0.263949 / 0.737135 (-0.473187) | 0.097635 / 0.296338 (-0.198704) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.386611 / 0.215209 (0.171402) | 3.847528 / 2.077655 (1.769873) | 1.859173 / 1.504120 (0.355053) | 1.685269 / 1.541195 (0.144074) | 1.715823 / 1.468490 (0.247333) | 0.485272 / 4.584777 (-4.099505) | 3.500724 / 3.745712 (-0.244988) | 3.252149 / 5.269862 (-2.017713) | 2.052914 / 4.565676 (-2.512762) | 0.056794 / 0.424275 (-0.367481) | 0.007317 / 0.007607 (-0.000291) | 0.457924 / 0.226044 (0.231879) | 4.570092 / 2.268929 (2.301163) | 2.328829 / 55.444624 (-53.115796) | 1.986502 / 6.876477 (-4.889975) | 2.164645 / 2.142072 (0.022573) | 0.580455 / 4.805227 (-4.224772) | 0.134415 / 6.500664 (-6.366249) | 0.060506 / 0.075469 (-0.014963) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.267423 / 1.841788 (-0.574364) | 18.653450 / 8.074308 (10.579142) | 13.919682 / 10.191392 (3.728290) | 0.144001 / 0.680424 (-0.536423) | 0.018218 / 0.534201 (-0.515983) | 0.389933 / 0.579283 (-0.189350) | 0.418366 / 0.434364 (-0.015998) | 0.456341 / 0.540337 (-0.083997) | 0.631401 / 1.386936 (-0.755535) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006838 / 0.011353 (-0.004515) | 0.003973 / 0.011008 (-0.007036) | 0.065217 / 0.038508 (0.026709) | 0.068357 / 0.023109 (0.045248) | 0.407960 / 0.275898 (0.132062) | 0.437794 / 0.323480 (0.114314) | 0.005398 / 0.007986 (-0.002587) | 0.003360 / 0.004328 (-0.000969) | 0.065503 / 0.004250 (0.061253) | 0.055676 / 0.037052 (0.018623) | 0.411381 / 0.258489 (0.152892) | 0.446902 / 0.293841 (0.153061) | 0.032156 / 0.128546 (-0.096390) | 0.008702 / 0.075646 (-0.066944) | 0.072295 / 0.419271 (-0.346976) | 0.047722 / 0.043533 (0.004189) | 0.406125 / 0.255139 (0.150986) | 0.428359 / 0.283200 (0.145160) | 0.021901 / 0.141683 (-0.119782) | 1.464186 / 1.452155 (0.012032) | 1.532809 / 1.492716 (0.040093) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.218505 / 0.018006 (0.200499) | 0.447450 / 0.000490 (0.446961) | 0.006509 / 0.000200 (0.006309) | 0.000099 / 0.000054 (0.000045) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031789 / 0.037411 (-0.005622) | 0.091100 / 0.014526 (0.076574) | 0.102812 / 0.176557 (-0.073745) | 0.155988 / 0.737135 (-0.581147) | 0.103983 / 0.296338 (-0.192355) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.436431 / 0.215209 (0.221222) | 4.336072 / 2.077655 (2.258417) | 2.344613 / 1.504120 (0.840493) | 2.173513 / 1.541195 (0.632319) | 2.313134 / 1.468490 (0.844644) | 0.493651 / 4.584777 (-4.091126) | 3.657541 / 3.745712 (-0.088171) | 3.289933 / 5.269862 (-1.979928) | 2.040271 / 4.565676 (-2.525406) | 0.058092 / 0.424275 (-0.366183) | 0.007348 / 0.007607 (-0.000259) | 0.507506 / 0.226044 (0.281462) | 5.093477 / 2.268929 (2.824548) | 2.770579 / 55.444624 (-52.674046) | 2.449507 / 6.876477 (-4.426970) | 2.645470 / 2.142072 (0.503397) | 0.590799 / 4.805227 (-4.214429) | 0.133411 / 6.500664 (-6.367253) | 0.059507 / 0.075469 (-0.015962) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.381148 / 1.841788 (-0.460639) | 19.188716 / 8.074308 (11.114408) | 14.709111 / 10.191392 (4.517719) | 0.191104 / 0.680424 (-0.489320) | 0.019862 / 0.534201 (-0.514339) | 0.395380 / 0.579283 (-0.183903) | 0.424757 / 0.434364 (-0.009607) | 0.468810 / 0.540337 (-0.071527) | 0.687058 / 1.386936 (-0.699878) |

</details>
</details>

![](https://cml.dev/watermark.png#407169e1ea91ae31f79ff29c4115b04a461279ab "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008872 / 0.011353 (-0.002481) | 0.004824 / 0.011008 (-0.006184) | 0.097012 / 0.038508 (0.058504) | 0.074728 / 0.023109 (0.051619) | 0.400604 / 0.275898 (0.124706) | 0.434316 / 0.323480 (0.110836) | 0.006025 / 0.007986 (-0.001961) | 0.004153 / 0.004328 (-0.000176) | 0.074093 / 0.004250 (0.069842) | 0.057239 / 0.037052 (0.020187) | 0.420611 / 0.258489 (0.162122) | 0.457779 / 0.293841 (0.163938) | 0.047610 / 0.128546 (-0.080936) | 0.014577 / 0.075646 (-0.061069) | 0.414351 / 0.419271 (-0.004921) | 0.063072 / 0.043533 (0.019539) | 0.426141 / 0.255139 (0.171002) | 0.429844 / 0.283200 (0.146644) | 0.034754 / 0.141683 (-0.106929) | 1.620946 / 1.452155 (0.168792) | 1.725831 / 1.492716 (0.233115) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.304712 / 0.018006 (0.286706) | 0.646924 / 0.000490 (0.646434) | 0.014486 / 0.000200 (0.014286) | 0.000626 / 0.000054 (0.000572) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034935 / 0.037411 (-0.002477) | 0.085788 / 0.014526 (0.071262) | 0.107749 / 0.176557 (-0.068807) | 0.170924 / 0.737135 (-0.566211) | 0.134985 / 0.296338 (-0.161354) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.602913 / 0.215209 (0.387704) | 6.041700 / 2.077655 (3.964045) | 2.539970 / 1.504120 (1.035850) | 2.184166 / 1.541195 (0.642972) | 2.241783 / 1.468490 (0.773293) | 0.864601 / 4.584777 (-3.720176) | 5.246955 / 3.745712 (1.501243) | 4.850458 / 5.269862 (-0.419404) | 3.101497 / 4.565676 (-1.464179) | 0.098591 / 0.424275 (-0.325684) | 0.008902 / 0.007607 (0.001295) | 0.732278 / 0.226044 (0.506234) | 7.163557 / 2.268929 (4.894629) | 3.226444 / 55.444624 (-52.218180) | 2.578737 / 6.876477 (-4.297740) | 2.850212 / 2.142072 (0.708140) | 1.026390 / 4.805227 (-3.778837) | 0.217077 / 6.500664 (-6.283587) | 0.080344 / 0.075469 (0.004875) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.687488 / 1.841788 (-0.154300) | 24.686337 / 8.074308 (16.612029) | 21.315989 / 10.191392 (11.124597) | 0.226176 / 0.680424 (-0.454248) | 0.035774 / 0.534201 (-0.498427) | 0.477807 / 0.579283 (-0.101476) | 0.636305 / 0.434364 (0.201941) | 0.553341 / 0.540337 (0.013003) | 0.797267 / 1.386936 (-0.589669) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008955 / 0.011353 (-0.002398) | 0.006099 / 0.011008 (-0.004909) | 0.086306 / 0.038508 (0.047798) | 0.090783 / 0.023109 (0.067674) | 0.554802 / 0.275898 (0.278904) | 0.598778 / 0.323480 (0.275299) | 0.008656 / 0.007986 (0.000670) | 0.004487 / 0.004328 (0.000159) | 0.084194 / 0.004250 (0.079943) | 0.076048 / 0.037052 (0.038996) | 0.533212 / 0.258489 (0.274723) | 0.584029 / 0.293841 (0.290188) | 0.051913 / 0.128546 (-0.076634) | 0.014253 / 0.075646 (-0.061393) | 0.100500 / 0.419271 (-0.318772) | 0.061092 / 0.043533 (0.017560) | 0.516955 / 0.255139 (0.261816) | 0.562754 / 0.283200 (0.279554) | 0.036673 / 0.141683 (-0.105010) | 1.853655 / 1.452155 (0.401501) | 1.968358 / 1.492716 (0.475642) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.308258 / 0.018006 (0.290252) | 0.630492 / 0.000490 (0.630002) | 0.010575 / 0.000200 (0.010375) | 0.000271 / 0.000054 (0.000217) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034762 / 0.037411 (-0.002649) | 0.107314 / 0.014526 (0.092788) | 0.132160 / 0.176557 (-0.044396) | 0.178737 / 0.737135 (-0.558398) | 0.125988 / 0.296338 (-0.170351) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.730738 / 0.215209 (0.515528) | 7.240393 / 2.077655 (5.162738) | 3.557665 / 1.504120 (2.053545) | 3.541425 / 1.541195 (2.000230) | 3.103849 / 1.468490 (1.635359) | 0.926843 / 4.584777 (-3.657934) | 5.818264 / 3.745712 (2.072552) | 5.012984 / 5.269862 (-0.256878) | 3.286085 / 4.565676 (-1.279591) | 0.104879 / 0.424275 (-0.319396) | 0.009010 / 0.007607 (0.001403) | 0.806145 / 0.226044 (0.580101) | 8.263655 / 2.268929 (5.994727) | 4.108932 / 55.444624 (-51.335693) | 3.454613 / 6.876477 (-3.421864) | 3.629045 / 2.142072 (1.486973) | 1.062325 / 4.805227 (-3.742902) | 0.220482 / 6.500664 (-6.280182) | 0.081440 / 0.075469 (0.005970) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.665587 / 1.841788 (-0.176201) | 23.695299 / 8.074308 (15.620991) | 22.917493 / 10.191392 (12.726101) | 0.259033 / 0.680424 (-0.421391) | 0.040118 / 0.534201 (-0.494083) | 0.487329 / 0.579283 (-0.091954) | 0.607482 / 0.434364 (0.173118) | 0.568383 / 0.540337 (0.028045) | 0.824486 / 1.386936 (-0.562450) |

</details>
</details>

![](https://cml.dev/watermark.png#53592bb8f635a1d6ea3e77acc290efdfb28fcbd7 "CML watermark")


CI failures are unrelated

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007095 / 0.011353 (-0.004258) | 0.004260 / 0.011008 (-0.006748) | 0.084729 / 0.038508 (0.046221) | 0.076498 / 0.023109 (0.053389) | 0.325981 / 0.275898 (0.050083) | 0.357140 / 0.323480 (0.033661) | 0.004325 / 0.007986 (-0.003660) | 0.003632 / 0.004328 (-0.000696) | 0.065075 / 0.004250 (0.060824) | 0.059058 / 0.037052 (0.022006) | 0.331895 / 0.258489 (0.073406) | 0.370782 / 0.293841 (0.076941) | 0.031886 / 0.128546 (-0.096660) | 0.008782 / 0.075646 (-0.066864) | 0.288159 / 0.419271 (-0.131113) | 0.053012 / 0.043533 (0.009479) | 0.319992 / 0.255139 (0.064853) | 0.347061 / 0.283200 (0.063861) | 0.026365 / 0.141683 (-0.115317) | 1.486112 / 1.452155 (0.033958) | 1.570150 / 1.492716 (0.077434) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.277155 / 0.018006 (0.259149) | 0.573507 / 0.000490 (0.573017) | 0.010122 / 0.000200 (0.009922) | 0.000322 / 0.000054 (0.000268) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029076 / 0.037411 (-0.008335) | 0.082517 / 0.014526 (0.067991) | 0.100710 / 0.176557 (-0.075847) | 0.154529 / 0.737135 (-0.582606) | 0.099531 / 0.296338 (-0.196807) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.382058 / 0.215209 (0.166849) | 3.803307 / 2.077655 (1.725652) | 1.834107 / 1.504120 (0.329987) | 1.665703 / 1.541195 (0.124508) | 1.739520 / 1.468490 (0.271030) | 0.490544 / 4.584777 (-4.094233) | 3.577874 / 3.745712 (-0.167838) | 3.327631 / 5.269862 (-1.942231) | 2.056634 / 4.565676 (-2.509043) | 0.057871 / 0.424275 (-0.366404) | 0.007326 / 0.007607 (-0.000281) | 0.453993 / 0.226044 (0.227949) | 4.549179 / 2.268929 (2.280250) | 2.320304 / 55.444624 (-53.124321) | 1.966082 / 6.876477 (-4.910395) | 2.189979 / 2.142072 (0.047907) | 0.586678 / 4.805227 (-4.218549) | 0.134919 / 6.500664 (-6.365745) | 0.061649 / 0.075469 (-0.013820) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.286228 / 1.841788 (-0.555560) | 19.409674 / 8.074308 (11.335366) | 14.290463 / 10.191392 (4.099071) | 0.165766 / 0.680424 (-0.514658) | 0.018200 / 0.534201 (-0.516001) | 0.390526 / 0.579283 (-0.188757) | 0.410953 / 0.434364 (-0.023411) | 0.455921 / 0.540337 (-0.084416) | 0.642271 / 1.386936 (-0.744665) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007288 / 0.011353 (-0.004064) | 0.004348 / 0.011008 (-0.006660) | 0.065935 / 0.038508 (0.027427) | 0.087327 / 0.023109 (0.064218) | 0.413461 / 0.275898 (0.137563) | 0.458904 / 0.323480 (0.135424) | 0.005996 / 0.007986 (-0.001990) | 0.003648 / 0.004328 (-0.000680) | 0.066578 / 0.004250 (0.062328) | 0.062072 / 0.037052 (0.025020) | 0.418469 / 0.258489 (0.159980) | 0.468960 / 0.293841 (0.175119) | 0.032616 / 0.128546 (-0.095930) | 0.008961 / 0.075646 (-0.066686) | 0.072537 / 0.419271 (-0.346734) | 0.048302 / 0.043533 (0.004769) | 0.411845 / 0.255139 (0.156706) | 0.441730 / 0.283200 (0.158530) | 0.025038 / 0.141683 (-0.116645) | 1.519402 / 1.452155 (0.067248) | 1.601791 / 1.492716 (0.109074) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.322494 / 0.018006 (0.304488) | 0.570210 / 0.000490 (0.569720) | 0.025815 / 0.000200 (0.025615) | 0.000166 / 0.000054 (0.000111) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034657 / 0.037411 (-0.002754) | 0.096024 / 0.014526 (0.081498) | 0.109134 / 0.176557 (-0.067422) | 0.162170 / 0.737135 (-0.574965) | 0.110472 / 0.296338 (-0.185866) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.439032 / 0.215209 (0.223823) | 4.385768 / 2.077655 (2.308113) | 2.343261 / 1.504120 (0.839142) | 2.157926 / 1.541195 (0.616731) | 2.299193 / 1.468490 (0.830703) | 0.498961 / 4.584777 (-4.085816) | 3.651909 / 3.745712 (-0.093803) | 3.387587 / 5.269862 (-1.882275) | 2.144553 / 4.565676 (-2.421123) | 0.058242 / 0.424275 (-0.366033) | 0.007416 / 0.007607 (-0.000191) | 0.512714 / 0.226044 (0.286670) | 5.138569 / 2.268929 (2.869641) | 2.778683 / 55.444624 (-52.665941) | 2.532990 / 6.876477 (-4.343487) | 2.782211 / 2.142072 (0.640139) | 0.591881 / 4.805227 (-4.213346) | 0.135005 / 6.500664 (-6.365660) | 0.060965 / 0.075469 (-0.014504) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.356311 / 1.841788 (-0.485477) | 20.029994 / 8.074308 (11.955686) | 14.666570 / 10.191392 (4.475178) | 0.164363 / 0.680424 (-0.516061) | 0.020685 / 0.534201 (-0.513516) | 0.396020 / 0.579283 (-0.183263) | 0.429407 / 0.434364 (-0.004957) | 0.476924 / 0.540337 (-0.063413) | 0.693389 / 1.386936 (-0.693547) |

</details>
</details>

![](https://cml.dev/watermark.png#292d627e398e30a538a616395f3b5ce4e89bb1e8 "CML watermark")