Add logic for hashing modules/functions optimized with `torch.compile`
Opened by mariosasko (closed)

Fix https://github.com/huggingface/datasets/issues/5839

PS: The `Pickler.save` method is becoming a bit messy, so I plan to refactor the pickler a bit at some point.

Comments: <details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006598 / 0.011353 (-0.004755) | 0.004565 / 0.011008 (-0.006443) | 0.099063 / 0.038508 (0.060555) | 0.028334 / 0.023109 (0.005225) | 0.323539 / 0.275898 (0.047641) | 0.372462 / 0.323480 (0.048982) | 0.005120 / 0.007986 (-0.002865) | 0.004797 / 0.004328 (0.000468) | 0.076862 / 0.004250 (0.072611) | 0.038021 / 0.037052 (0.000968) | 0.337801 / 0.258489 (0.079312) | 0.374601 / 0.293841 (0.080760) | 0.031158 / 0.128546 (-0.097389) | 0.011672 / 0.075646 (-0.063974) | 0.324913 / 0.419271 (-0.094359) | 0.051702 / 0.043533 (0.008169) | 0.339440 / 0.255139 (0.084301) | 0.372502 / 0.283200 (0.089303) | 0.097590 / 0.141683 (-0.044093) | 1.534238 / 1.452155 (0.082083) | 1.599701 / 1.492716 (0.106985) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.204101 / 0.018006 (0.186095) | 0.416981 / 0.000490 (0.416491) | 0.003436 / 0.000200 (0.003236) | 0.000071 / 0.000054 (0.000017) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023527 / 0.037411 (-0.013885) | 0.095748 / 0.014526 (0.081222) | 0.104498 / 0.176557 (-0.072059) | 0.164000 / 0.737135 (-0.573135) | 0.109170 / 0.296338 (-0.187168) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.418239 / 0.215209 (0.203030) | 4.153959 / 2.077655 (2.076305) | 1.856687 / 1.504120 (0.352567) | 1.657818 / 1.541195 (0.116623) | 1.715146 / 1.468490 (0.246656) | 0.700673 / 4.584777 (-3.884103) | 3.401060 / 3.745712 (-0.344652) | 2.891045 / 5.269862 (-2.378816) | 1.519433 / 4.565676 (-3.046243) | 0.083151 / 0.424275 (-0.341124) | 0.012352 / 0.007607 (0.004745) | 0.523901 / 0.226044 (0.297856) | 5.288871 / 2.268929 (3.019943) | 2.322806 / 55.444624 (-53.121818) | 1.982223 / 6.876477 (-4.894253) | 2.074883 / 2.142072 (-0.067189) | 0.812400 / 4.805227 (-3.992827) | 0.152183 / 6.500664 (-6.348481) | 0.066538 / 0.075469 (-0.008931) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.223220 / 1.841788 (-0.618567) | 14.024391 / 8.074308 (5.950083) | 14.166657 / 10.191392 (3.975265) | 0.146017 / 0.680424 (-0.534407) | 0.016698 / 0.534201 (-0.517503) | 0.380779 / 0.579283 (-0.198504) | 0.387113 / 0.434364 (-0.047251) | 0.446329 / 0.540337 (-0.094009) | 0.523819 / 1.386936 (-0.863118) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006803 / 0.011353 (-0.004549) | 0.004554 / 0.011008 (-0.006454) | 0.077406 / 0.038508 (0.038897) | 0.028495 / 0.023109 (0.005386) | 0.358847 / 0.275898 (0.082949) | 0.393256 / 0.323480 (0.069776) | 0.005317 / 0.007986 (-0.002669) | 0.004690 / 0.004328 (0.000362) | 0.075842 / 0.004250 (0.071592) | 0.041985 / 0.037052 (0.004933) | 0.367546 / 0.258489 (0.109057) | 0.408019 / 0.293841 (0.114178) | 0.030712 / 0.128546 (-0.097834) | 0.011756 / 0.075646 (-0.063891) | 0.086002 / 0.419271 (-0.333269) | 0.038949 / 0.043533 (-0.004583) | 0.361045 / 0.255139 (0.105906) | 0.381728 / 0.283200 (0.098528) | 0.090692 / 0.141683 (-0.050991) | 1.493251 / 1.452155 (0.041097) | 1.584566 / 1.492716 (0.091850) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.217470 / 0.018006 (0.199463) | 0.429955 / 0.000490 (0.429465) | 0.000394 / 0.000200 (0.000194) | 0.000078 / 0.000054 (0.000024) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026223 / 0.037411 (-0.011189) | 0.102570 / 0.014526 (0.088045) | 0.110848 / 0.176557 (-0.065709) | 0.162413 / 0.737135 (-0.574722) | 0.114579 / 0.296338 (-0.181760) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.464957 / 0.215209 (0.249748) | 4.656597 / 2.077655 (2.578942) | 2.279755 / 1.504120 (0.775636) | 2.230263 / 1.541195 (0.689068) | 2.341540 / 1.468490 (0.873050) | 0.699505 / 4.584777 (-3.885272) | 3.389003 / 3.745712 (-0.356709) | 1.867526 / 5.269862 (-3.402336) | 1.167171 / 4.565676 (-3.398506) | 0.083451 / 0.424275 (-0.340824) | 0.012348 / 0.007607 (0.004741) | 0.584205 / 0.226044 (0.358161) | 5.853623 / 2.268929 (3.584694) | 2.646650 / 55.444624 (-52.797974) | 2.286504 / 6.876477 (-4.589973) | 2.327536 / 2.142072 (0.185464) | 0.811209 / 4.805227 (-3.994018) | 0.151842 / 6.500664 (-6.348822) | 0.067783 / 0.075469 (-0.007686) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.330427 / 1.841788 (-0.511360) | 14.668981 / 8.074308 (6.594673) | 13.321154 / 10.191392 (3.129762) | 0.164383 / 0.680424 (-0.516040) | 0.016667 / 0.534201 (-0.517534) | 0.383439 / 0.579283 (-0.195844) | 0.392988 / 0.434364 (-0.041376) | 0.443318 / 0.540337 (-0.097020) | 0.537849 / 1.386936 (-0.849087) |

</details>
</details>

![](https://cml.dev/watermark.png#e99bd4583bd636074b1826e2d0581161807480f1 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006379 / 0.011353 (-0.004974) | 0.004691 / 0.011008 (-0.006317) | 0.098047 / 0.038508 (0.059539) | 0.028126 / 0.023109 (0.005017) | 0.327143 / 0.275898 (0.051245) | 0.362482 / 0.323480 (0.039002) | 0.004953 / 0.007986 (-0.003033) | 0.003386 / 0.004328 (-0.000943) | 0.076222 / 0.004250 (0.071971) | 0.037583 / 0.037052 (0.000531) | 0.329661 / 0.258489 (0.071172) | 0.365945 / 0.293841 (0.072104) | 0.030455 / 0.128546 (-0.098091) | 0.011397 / 0.075646 (-0.064249) | 0.323889 / 0.419271 (-0.095383) | 0.043719 / 0.043533 (0.000186) | 0.331499 / 0.255139 (0.076360) | 0.359357 / 0.283200 (0.076158) | 0.088904 / 0.141683 (-0.052779) | 1.458584 / 1.452155 (0.006429) | 1.549375 / 1.492716 (0.056658) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.195808 / 0.018006 (0.177802) | 0.411148 / 0.000490 (0.410659) | 0.003602 / 0.000200 (0.003402) | 0.000070 / 0.000054 (0.000016) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023278 / 0.037411 (-0.014133) | 0.097317 / 0.014526 (0.082791) | 0.102669 / 0.176557 (-0.073888) | 0.168203 / 0.737135 (-0.568933) | 0.105205 / 0.296338 (-0.191133) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.424800 / 0.215209 (0.209591) | 4.228444 / 2.077655 (2.150790) | 1.895544 / 1.504120 (0.391424) | 1.698793 / 1.541195 (0.157598) | 1.717931 / 1.468490 (0.249441) | 0.702251 / 4.584777 (-3.882526) | 3.407013 / 3.745712 (-0.338699) | 2.784634 / 5.269862 (-2.485228) | 1.491317 / 4.565676 (-3.074359) | 0.082926 / 0.424275 (-0.341350) | 0.012320 / 0.007607 (0.004713) | 0.524188 / 0.226044 (0.298143) | 5.249798 / 2.268929 (2.980870) | 2.358953 / 55.444624 (-53.085672) | 1.985922 / 6.876477 (-4.890555) | 2.034293 / 2.142072 (-0.107779) | 0.815671 / 4.805227 (-3.989556) | 0.152583 / 6.500664 (-6.348081) | 0.066687 / 0.075469 (-0.008782) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.210901 / 1.841788 (-0.630886) | 13.621765 / 8.074308 (5.547457) | 14.213215 / 10.191392 (4.021823) | 0.143346 / 0.680424 (-0.537078) | 0.016904 / 0.534201 (-0.517297) | 0.379795 / 0.579283 (-0.199489) | 0.381287 / 0.434364 (-0.053077) | 0.449086 / 0.540337 (-0.091251) | 0.538792 / 1.386936 (-0.848144) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006207 / 0.011353 (-0.005146) | 0.004404 / 0.011008 (-0.006604) | 0.076363 / 0.038508 (0.037854) | 0.027335 / 0.023109 (0.004226) | 0.370967 / 0.275898 (0.095069) | 0.401936 / 0.323480 (0.078456) | 0.004835 / 0.007986 (-0.003151) | 0.004559 / 0.004328 (0.000231) | 0.074964 / 0.004250 (0.070713) | 0.038254 / 0.037052 (0.001202) | 0.374799 / 0.258489 (0.116310) | 0.425191 / 0.293841 (0.131350) | 0.035290 / 0.128546 (-0.093256) | 0.011379 / 0.075646 (-0.064267) | 0.085911 / 0.419271 (-0.333360) | 0.043073 / 0.043533 (-0.000460) | 0.373557 / 0.255139 (0.118418) | 0.395179 / 0.283200 (0.111979) | 0.098602 / 0.141683 (-0.043081) | 1.467234 / 1.452155 (0.015079) | 1.571868 / 1.492716 (0.079152) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.221848 / 0.018006 (0.203842) | 0.394943 / 0.000490 (0.394454) | 0.002983 / 0.000200 (0.002783) | 0.000078 / 0.000054 (0.000024) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024385 / 0.037411 (-0.013027) | 0.100087 / 0.014526 (0.085561) | 0.104897 / 0.176557 (-0.071660) | 0.156150 / 0.737135 (-0.580985) | 0.109113 / 0.296338 (-0.187226) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.441995 / 0.215209 (0.226786) | 4.415423 / 2.077655 (2.337769) | 2.148791 / 1.504120 (0.644671) | 1.947061 / 1.541195 (0.405866) | 1.954807 / 1.468490 (0.486317) | 0.690245 / 4.584777 (-3.894532) | 3.372766 / 3.745712 (-0.372946) | 1.851073 / 5.269862 (-3.418789) | 1.155558 / 4.565676 (-3.410118) | 0.082796 / 0.424275 (-0.341479) | 0.012845 / 0.007607 (0.005238) | 0.548173 / 0.226044 (0.322129) | 5.530984 / 2.268929 (3.262056) | 2.665360 / 55.444624 (-52.779264) | 2.324266 / 6.876477 (-4.552211) | 2.329397 / 2.142072 (0.187324) | 0.801481 / 4.805227 (-4.003746) | 0.152145 / 6.500664 (-6.348519) | 0.067915 / 0.075469 (-0.007554) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.291488 / 1.841788 (-0.550299) | 13.912143 / 8.074308 (5.837835) | 12.975493 / 10.191392 (2.784101) | 0.129915 / 0.680424 (-0.550509) | 0.016516 / 0.534201 (-0.517685) | 0.386979 / 0.579283 (-0.192304) | 0.389163 / 0.434364 (-0.045201) | 0.443324 / 0.540337 (-0.097014) | 0.533744 / 1.386936 (-0.853192) |

</details>
</details>

![](https://cml.dev/watermark.png#eb48834fc2aa45cad73fe70a7ecaa0dd6015b8d0 "CML watermark")


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5867). All of your documentation changes will be reflected on that endpoint.

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008635 / 0.011353 (-0.002717) | 0.006014 / 0.011008 (-0.004995) | 0.116314 / 0.038508 (0.077806) | 0.041113 / 0.023109 (0.018004) | 0.358564 / 0.275898 (0.082666) | 0.397547 / 0.323480 (0.074067) | 0.007012 / 0.007986 (-0.000974) | 0.004638 / 0.004328 (0.000310) | 0.086509 / 0.004250 (0.082259) | 0.056731 / 0.037052 (0.019678) | 0.358859 / 0.258489 (0.100370) | 0.425339 / 0.293841 (0.131498) | 0.041780 / 0.128546 (-0.086767) | 0.014203 / 0.075646 (-0.061443) | 0.398240 / 0.419271 (-0.021031) | 0.060180 / 0.043533 (0.016647) | 0.352887 / 0.255139 (0.097748) | 0.381793 / 0.283200 (0.098594) | 0.148578 / 0.141683 (0.006895) | 1.749483 / 1.452155 (0.297328) | 1.869765 / 1.492716 (0.377049) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.244435 / 0.018006 (0.226428) | 0.499545 / 0.000490 (0.499055) | 0.004576 / 0.000200 (0.004376) | 0.000147 / 0.000054 (0.000093) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031163 / 0.037411 (-0.006249) | 0.131082 / 0.014526 (0.116556) | 0.137442 / 0.176557 (-0.039114) | 0.203783 / 0.737135 (-0.533352) | 0.144068 / 0.296338 (-0.152270) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.503587 / 0.215209 (0.288378) | 5.011953 / 2.077655 (2.934299) | 2.366968 / 1.504120 (0.862848) | 2.130914 / 1.541195 (0.589719) | 2.243560 / 1.468490 (0.775070) | 0.856719 / 4.584777 (-3.728058) | 4.707445 / 3.745712 (0.961733) | 2.506166 / 5.269862 (-2.763696) | 1.590400 / 4.565676 (-2.975277) | 0.102075 / 0.424275 (-0.322200) | 0.014499 / 0.007607 (0.006892) | 0.624966 / 0.226044 (0.398922) | 6.197671 / 2.268929 (3.928742) | 2.898481 / 55.444624 (-52.546143) | 2.499590 / 6.876477 (-4.376886) | 2.649690 / 2.142072 (0.507617) | 1.012542 / 4.805227 (-3.792685) | 0.202833 / 6.500664 (-6.297831) | 0.078033 / 0.075469 (0.002564) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.448321 / 1.841788 (-0.393467) | 18.084909 / 8.074308 (10.010601) | 17.383027 / 10.191392 (7.191635) | 0.212167 / 0.680424 (-0.468256) | 0.020754 / 0.534201 (-0.513447) | 0.514653 / 0.579283 (-0.064630) | 0.543307 / 0.434364 (0.108944) | 0.653066 / 0.540337 (0.112728) | 0.745773 / 1.386936 (-0.641164) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008576 / 0.011353 (-0.002777) | 0.005834 / 0.011008 (-0.005174) | 0.089842 / 0.038508 (0.051334) | 0.040035 / 0.023109 (0.016926) | 0.449329 / 0.275898 (0.173431) | 0.471572 / 0.323480 (0.148092) | 0.006771 / 0.007986 (-0.001215) | 0.006129 / 0.004328 (0.001800) | 0.090370 / 0.004250 (0.086119) | 0.056924 / 0.037052 (0.019872) | 0.455134 / 0.258489 (0.196645) | 0.502670 / 0.293841 (0.208829) | 0.041689 / 0.128546 (-0.086857) | 0.014447 / 0.075646 (-0.061200) | 0.104528 / 0.419271 (-0.314744) | 0.055535 / 0.043533 (0.012003) | 0.450667 / 0.255139 (0.195528) | 0.453108 / 0.283200 (0.169908) | 0.119296 / 0.141683 (-0.022387) | 1.747359 / 1.452155 (0.295204) | 1.839421 / 1.492716 (0.346705) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.314910 / 0.018006 (0.296904) | 0.495575 / 0.000490 (0.495085) | 0.054702 / 0.000200 (0.054503) | 0.000505 / 0.000054 (0.000450) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033991 / 0.037411 (-0.003420) | 0.133268 / 0.014526 (0.118742) | 0.142286 / 0.176557 (-0.034271) | 0.200562 / 0.737135 (-0.536573) | 0.147161 / 0.296338 (-0.149178) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.520288 / 0.215209 (0.305079) | 5.227684 / 2.077655 (3.150029) | 2.553330 / 1.504120 (1.049210) | 2.324338 / 1.541195 (0.783143) | 2.406790 / 1.468490 (0.938300) | 0.850404 / 4.584777 (-3.734373) | 4.612156 / 3.745712 (0.866444) | 2.592546 / 5.269862 (-2.677316) | 1.708984 / 4.565676 (-2.856692) | 0.103751 / 0.424275 (-0.320524) | 0.014379 / 0.007607 (0.006772) | 0.634661 / 0.226044 (0.408616) | 6.344939 / 2.268929 (4.076010) | 3.179807 / 55.444624 (-52.264817) | 2.831856 / 6.876477 (-4.044621) | 2.866729 / 2.142072 (0.724656) | 0.994519 / 4.805227 (-3.810708) | 0.201566 / 6.500664 (-6.299098) | 0.078902 / 0.075469 (0.003433) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.538738 / 1.841788 (-0.303049) | 18.746367 / 8.074308 (10.672059) | 16.504763 / 10.191392 (6.313371) | 0.197898 / 0.680424 (-0.482526) | 0.020469 / 0.534201 (-0.513732) | 0.529106 / 0.579283 (-0.050177) | 0.536891 / 0.434364 (0.102527) | 0.600947 / 0.540337 (0.060610) | 0.701713 / 1.386936 (-0.685223) |

</details>
</details>

![](https://cml.dev/watermark.png#3054f66b4765a520e6fe165c44a4307d40775229 "CML watermark")


Closing in favor of #6454