Title: Fix metadata file resolution when inferred pattern is `**`
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2023-11-23T17:35:02Z

Refetch metadata files in case they were dropped by `filter_extensions` in the previous step. 

Fix #6442

Comments:
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005551 / 0.011353 (-0.005802) | 0.003297 / 0.011008 (-0.007711) | 0.062524 / 0.038508 (0.024016) | 0.058467 / 0.023109 (0.035358) | 0.255703 / 0.275898 (-0.020195) | 0.281420 / 0.323480 (-0.042060) | 0.003857 / 0.007986 (-0.004129) | 0.002460 / 0.004328 (-0.001868) | 0.047762 / 0.004250 (0.043512) | 0.038757 / 0.037052 (0.001705) | 0.259937 / 0.258489 (0.001448) | 0.290050 / 0.293841 (-0.003791) | 0.028433 / 0.128546 (-0.100113) | 0.010422 / 0.075646 (-0.065224) | 0.207135 / 0.419271 (-0.212136) | 0.036004 / 0.043533 (-0.007529) | 0.268137 / 0.255139 (0.012998) | 0.275020 / 0.283200 (-0.008179) | 0.018301 / 0.141683 (-0.123382) | 1.095479 / 1.452155 (-0.356676) | 1.145452 / 1.492716 (-0.347265) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092046 / 0.018006 (0.074040) | 0.299784 / 0.000490 (0.299294) | 0.000214 / 0.000200 (0.000014) | 0.000048 / 0.000054 (-0.000006) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019071 / 0.037411 (-0.018340) | 0.072836 / 0.014526 (0.058310) | 0.073974 / 0.176557 (-0.102583) | 0.120903 / 0.737135 (-0.616232) | 0.075740 / 0.296338 (-0.220599) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.276365 / 0.215209 (0.061156) | 2.671217 / 2.077655 (0.593563) | 1.438862 / 1.504120 (-0.065258) | 1.327348 / 1.541195 (-0.213847) | 1.349514 / 1.468490 (-0.118976) | 0.548793 / 4.584777 (-4.035984) | 2.364458 / 3.745712 (-1.381255) | 2.716205 / 5.269862 (-2.553657) | 1.735714 / 4.565676 (-2.829963) | 0.061140 / 0.424275 (-0.363135) | 0.004926 / 0.007607 (-0.002681) | 0.330449 / 0.226044 (0.104404) | 3.255243 / 2.268929 (0.986315) | 1.824254 / 55.444624 (-53.620371) | 1.540262 / 6.876477 (-5.336215) | 1.535632 / 2.142072 (-0.606441) | 0.635224 / 4.805227 (-4.170003) | 0.116230 / 6.500664 (-6.384435) | 0.042706 / 0.075469 (-0.032763) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.948796 / 1.841788 (-0.892992) | 11.448403 / 8.074308 (3.374095) | 10.523862 / 10.191392 (0.332470) | 0.129694 / 0.680424 (-0.550730) | 0.014146 / 0.534201 (-0.520055) | 0.285706 / 0.579283 (-0.293577) | 0.262572 / 0.434364 (-0.171792) | 0.321251 / 0.540337 (-0.219087) | 0.417130 / 1.386936 (-0.969806) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005266 / 0.011353 (-0.006086) | 0.003339 / 0.011008 (-0.007670) | 0.048411 / 0.038508 (0.009903) | 0.053951 / 0.023109 (0.030842) | 0.271228 / 0.275898 (-0.004670) | 0.290066 / 0.323480 (-0.033414) | 0.004087 / 0.007986 (-0.003898) | 0.002446 / 0.004328 (-0.001882) | 0.047049 / 0.004250 (0.042798) | 0.040866 / 0.037052 (0.003813) | 0.273711 / 0.258489 (0.015222) | 0.298192 / 0.293841 (0.004351) | 0.029025 / 0.128546 (-0.099521) | 0.010479 / 0.075646 (-0.065167) | 0.056941 / 0.419271 (-0.362330) | 0.032914 / 0.043533 (-0.010619) | 0.270432 / 0.255139 (0.015293) | 0.291274 / 0.283200 (0.008074) | 0.018602 / 0.141683 (-0.123081) | 1.136707 / 1.452155 (-0.315447) | 1.184704 / 1.492716 (-0.308012) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090041 / 0.018006 (0.072035) | 0.300185 / 0.000490 (0.299696) | 0.000221 / 0.000200 (0.000022) | 0.000049 / 0.000054 (-0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022074 / 0.037411 (-0.015337) | 0.070763 / 0.014526 (0.056237) | 0.082141 / 0.176557 (-0.094415) | 0.120286 / 0.737135 (-0.616850) | 0.082680 / 0.296338 (-0.213659) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.292223 / 0.215209 (0.077014) | 2.856711 / 2.077655 (0.779056) | 1.581194 / 1.504120 (0.077075) | 1.496567 / 1.541195 (-0.044628) | 1.485256 / 1.468490 (0.016766) | 0.550633 / 4.584777 (-4.034144) | 2.420281 / 3.745712 (-1.325431) | 2.764373 / 5.269862 (-2.505489) | 1.735958 / 4.565676 (-2.829719) | 0.062562 / 0.424275 (-0.361714) | 0.004918 / 0.007607 (-0.002689) | 0.346038 / 0.226044 (0.119994) | 3.443478 / 2.268929 (1.174550) | 1.949366 / 55.444624 (-53.495259) | 1.686140 / 6.876477 (-5.190337) | 1.683038 / 2.142072 (-0.459034) | 0.629270 / 4.805227 (-4.175958) | 0.114947 / 6.500664 (-6.385717) | 0.040635 / 0.075469 (-0.034834) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.969746 / 1.841788 (-0.872041) | 11.922662 / 8.074308 (3.848354) | 10.441432 / 10.191392 (0.250040) | 0.128950 / 0.680424 (-0.551473) | 0.015964 / 0.534201 (-0.518237) | 0.289176 / 0.579283 (-0.290107) | 0.279203 / 0.434364 (-0.155161) | 0.323833 / 0.540337 (-0.216505) | 0.540297 / 1.386936 (-0.846639) |

</details>
</details>

![](https://cml.dev/watermark.png#3ed759d0f5aea6d166caa0532aa17c209bb3af79 "CML watermark")


_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005288 / 0.011353 (-0.006065) | 0.003383 / 0.011008 (-0.007625) | 0.061926 / 0.038508 (0.023418) | 0.049080 / 0.023109 (0.025971) | 0.244852 / 0.275898 (-0.031046) | 0.263957 / 0.323480 (-0.059523) | 0.002810 / 0.007986 (-0.005175) | 0.002384 / 0.004328 (-0.001945) | 0.047807 / 0.004250 (0.043556) | 0.038374 / 0.037052 (0.001321) | 0.244414 / 0.258489 (-0.014075) | 0.272257 / 0.293841 (-0.021584) | 0.027356 / 0.128546 (-0.101190) | 0.010235 / 0.075646 (-0.065411) | 0.214896 / 0.419271 (-0.204375) | 0.035604 / 0.043533 (-0.007929) | 0.246584 / 0.255139 (-0.008555) | 0.263281 / 0.283200 (-0.019918) | 0.019689 / 0.141683 (-0.121994) | 1.114100 / 1.452155 (-0.338054) | 1.177644 / 1.492716 (-0.315073) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.088892 / 0.018006 (0.070886) | 0.298128 / 0.000490 (0.297639) | 0.000199 / 0.000200 (-0.000001) | 0.000046 / 0.000054 (-0.000009) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019337 / 0.037411 (-0.018075) | 0.062096 / 0.014526 (0.047570) | 0.073019 / 0.176557 (-0.103537) | 0.118801 / 0.737135 (-0.618334) | 0.074779 / 0.296338 (-0.221559) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.289892 / 0.215209 (0.074683) | 2.824131 / 2.077655 (0.746476) | 1.466351 / 1.504120 (-0.037768) | 1.339528 / 1.541195 (-0.201667) | 1.369257 / 1.468490 (-0.099233) | 0.561175 / 4.584777 (-4.023602) | 2.394174 / 3.745712 (-1.351538) | 2.749668 / 5.269862 (-2.520193) | 1.747146 / 4.565676 (-2.818530) | 0.063054 / 0.424275 (-0.361221) | 0.004970 / 0.007607 (-0.002637) | 0.342985 / 0.226044 (0.116941) | 3.334894 / 2.268929 (1.065966) | 1.838459 / 55.444624 (-53.606165) | 1.579755 / 6.876477 (-5.296722) | 1.560200 / 2.142072 (-0.581872) | 0.642643 / 4.805227 (-4.162585) | 0.117741 / 6.500664 (-6.382923) | 0.042440 / 0.075469 (-0.033029) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.937476 / 1.841788 (-0.904312) | 11.403556 / 8.074308 (3.329248) | 10.317207 / 10.191392 (0.125815) | 0.145277 / 0.680424 (-0.535147) | 0.015297 / 0.534201 (-0.518904) | 0.287511 / 0.579283 (-0.291772) | 0.263516 / 0.434364 (-0.170848) | 0.320803 / 0.540337 (-0.219534) | 0.415580 / 1.386936 (-0.971356) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005239 / 0.011353 (-0.006114) | 0.003506 / 0.011008 (-0.007502) | 0.048635 / 0.038508 (0.010127) | 0.052067 / 0.023109 (0.028957) | 0.277526 / 0.275898 (0.001628) | 0.300536 / 0.323480 (-0.022944) | 0.003982 / 0.007986 (-0.004004) | 0.002413 / 0.004328 (-0.001915) | 0.046523 / 0.004250 (0.042273) | 0.039383 / 0.037052 (0.002331) | 0.281208 / 0.258489 (0.022719) | 0.306199 / 0.293841 (0.012359) | 0.028646 / 0.128546 (-0.099900) | 0.010664 / 0.075646 (-0.064982) | 0.057393 / 0.419271 (-0.361879) | 0.032171 / 0.043533 (-0.011362) | 0.277576 / 0.255139 (0.022437) | 0.296039 / 0.283200 (0.012840) | 0.017519 / 0.141683 (-0.124164) | 1.153172 / 1.452155 (-0.298982) | 1.180274 / 1.492716 (-0.312442) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.088287 / 0.018006 (0.070280) | 0.297922 / 0.000490 (0.297433) | 0.000216 / 0.000200 (0.000016) | 0.000050 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021936 / 0.037411 (-0.015475) | 0.070181 / 0.014526 (0.055655) | 0.082068 / 0.176557 (-0.094488) | 0.119327 / 0.737135 (-0.617808) | 0.083642 / 0.296338 (-0.212697) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.299449 / 0.215209 (0.084240) | 2.914362 / 2.077655 (0.836707) | 1.611906 / 1.504120 (0.107786) | 1.488805 / 1.541195 (-0.052390) | 1.536010 / 1.468490 (0.067520) | 0.566772 / 4.584777 (-4.018004) | 2.397897 / 3.745712 (-1.347815) | 2.786048 / 5.269862 (-2.483814) | 1.745153 / 4.565676 (-2.820523) | 0.063870 / 0.424275 (-0.360405) | 0.004968 / 0.007607 (-0.002640) | 0.344455 / 0.226044 (0.118410) | 3.465772 / 2.268929 (1.196844) | 1.965761 / 55.444624 (-53.478863) | 1.687960 / 6.876477 (-5.188516) | 1.713987 / 2.142072 (-0.428085) | 0.643760 / 4.805227 (-4.161467) | 0.117623 / 6.500664 (-6.383042) | 0.041086 / 0.075469 (-0.034383) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.985129 / 1.841788 (-0.856659) | 11.986676 / 8.074308 (3.912368) | 10.493440 / 10.191392 (0.302048) | 0.130070 / 0.680424 (-0.550353) | 0.015293 / 0.534201 (-0.518908) | 0.285683 / 0.579283 (-0.293600) | 0.275656 / 0.434364 (-0.158708) | 0.328704 / 0.540337 (-0.211633) | 0.537249 / 1.386936 (-0.849687) |

</details>
</details>

![](https://cml.dev/watermark.png#d7ee58f322082d3af5f11863d1f809444910827a "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005170 / 0.011353 (-0.006183) | 0.003267 / 0.011008 (-0.007741) | 0.061992 / 0.038508 (0.023484) | 0.053414 / 0.023109 (0.030305) | 0.245678 / 0.275898 (-0.030220) | 0.261320 / 0.323480 (-0.062160) | 0.003887 / 0.007986 (-0.004099) | 0.002543 / 0.004328 (-0.001786) | 0.048496 / 0.004250 (0.044246) | 0.037392 / 0.037052 (0.000340) | 0.243728 / 0.258489 (-0.014761) | 0.272524 / 0.293841 (-0.021317) | 0.027578 / 0.128546 (-0.100968) | 0.010530 / 0.075646 (-0.065116) | 0.206014 / 0.419271 (-0.213257) | 0.035987 / 0.043533 (-0.007546) | 0.243544 / 0.255139 (-0.011595) | 0.263872 / 0.283200 (-0.019327) | 0.017867 / 0.141683 (-0.123816) | 1.105159 / 1.452155 (-0.346996) | 1.186640 / 1.492716 (-0.306076) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092888 / 0.018006 (0.074882) | 0.302024 / 0.000490 (0.301534) | 0.000220 / 0.000200 (0.000020) | 0.000052 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019329 / 0.037411 (-0.018083) | 0.062135 / 0.014526 (0.047609) | 0.075125 / 0.176557 (-0.101431) | 0.120743 / 0.737135 (-0.616393) | 0.078687 / 0.296338 (-0.217652) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.279449 / 0.215209 (0.064240) | 2.727310 / 2.077655 (0.649656) | 1.442710 / 1.504120 (-0.061410) | 1.315271 / 1.541195 (-0.225923) | 1.360435 / 1.468490 (-0.108055) | 0.567720 / 4.584777 (-4.017057) | 2.397049 / 3.745712 (-1.348663) | 2.891180 / 5.269862 (-2.378682) | 1.774179 / 4.565676 (-2.791497) | 0.063155 / 0.424275 (-0.361120) | 0.004963 / 0.007607 (-0.002644) | 0.337526 / 0.226044 (0.111482) | 3.266016 / 2.268929 (0.997088) | 1.808819 / 55.444624 (-53.635806) | 1.525326 / 6.876477 (-5.351151) | 1.566937 / 2.142072 (-0.575135) | 0.654226 / 4.805227 (-4.151001) | 0.118968 / 6.500664 (-6.381696) | 0.042666 / 0.075469 (-0.032803) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.940792 / 1.841788 (-0.900996) | 11.736380 / 8.074308 (3.662072) | 10.709538 / 10.191392 (0.518146) | 0.141390 / 0.680424 (-0.539034) | 0.014204 / 0.534201 (-0.519996) | 0.284842 / 0.579283 (-0.294441) | 0.266315 / 0.434364 (-0.168049) | 0.331619 / 0.540337 (-0.208718) | 0.416446 / 1.386936 (-0.970491) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005298 / 0.011353 (-0.006055) | 0.003507 / 0.011008 (-0.007501) | 0.048315 / 0.038508 (0.009807) | 0.054855 / 0.023109 (0.031746) | 0.271558 / 0.275898 (-0.004340) | 0.316851 / 0.323480 (-0.006628) | 0.004054 / 0.007986 (-0.003932) | 0.002433 / 0.004328 (-0.001896) | 0.046442 / 0.004250 (0.042191) | 0.040853 / 0.037052 (0.003801) | 0.272537 / 0.258489 (0.014048) | 0.293736 / 0.293841 (-0.000105) | 0.029112 / 0.128546 (-0.099434) | 0.010573 / 0.075646 (-0.065074) | 0.056501 / 0.419271 (-0.362771) | 0.032541 / 0.043533 (-0.010992) | 0.271004 / 0.255139 (0.015865) | 0.289276 / 0.283200 (0.006076) | 0.018618 / 0.141683 (-0.123065) | 1.149435 / 1.452155 (-0.302719) | 1.205113 / 1.492716 (-0.287604) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094726 / 0.018006 (0.076720) | 0.304347 / 0.000490 (0.303857) | 0.000217 / 0.000200 (0.000017) | 0.000051 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021374 / 0.037411 (-0.016037) | 0.070574 / 0.014526 (0.056049) | 0.081749 / 0.176557 (-0.094807) | 0.119829 / 0.737135 (-0.617306) | 0.082602 / 0.296338 (-0.213737) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.293378 / 0.215209 (0.078169) | 2.893607 / 2.077655 (0.815952) | 1.577734 / 1.504120 (0.073614) | 1.453670 / 1.541195 (-0.087525) | 1.467354 / 1.468490 (-0.001136) | 0.563415 / 4.584777 (-4.021362) | 2.438330 / 3.745712 (-1.307382) | 2.761822 / 5.269862 (-2.508040) | 1.730944 / 4.565676 (-2.834732) | 0.062251 / 0.424275 (-0.362024) | 0.004969 / 0.007607 (-0.002638) | 0.371238 / 0.226044 (0.145194) | 3.399831 / 2.268929 (1.130903) | 1.936156 / 55.444624 (-53.508469) | 1.649716 / 6.876477 (-5.226761) | 1.669107 / 2.142072 (-0.472965) | 0.633696 / 4.805227 (-4.171531) | 0.115857 / 6.500664 (-6.384807) | 0.041012 / 0.075469 (-0.034457) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.964777 / 1.841788 (-0.877010) | 12.037613 / 8.074308 (3.963305) | 10.579241 / 10.191392 (0.387849) | 0.130932 / 0.680424 (-0.549492) | 0.015621 / 0.534201 (-0.518580) | 0.286898 / 0.579283 (-0.292385) | 0.281139 / 0.434364 (-0.153225) | 0.325240 / 0.540337 (-0.215097) | 0.554302 / 1.386936 (-0.832635) |

</details>
</details>

![](https://cml.dev/watermark.png#48d2378944a47987f96562ee856167aef1e78522 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005258 / 0.011353 (-0.006095) | 0.003863 / 0.011008 (-0.007145) | 0.064585 / 0.038508 (0.026077) | 0.058013 / 0.023109 (0.034904) | 0.249042 / 0.275898 (-0.026856) | 0.273434 / 0.323480 (-0.050046) | 0.004779 / 0.007986 (-0.003207) | 0.002550 / 0.004328 (-0.001778) | 0.048290 / 0.004250 (0.044040) | 0.038777 / 0.037052 (0.001725) | 0.253039 / 0.258489 (-0.005450) | 0.285365 / 0.293841 (-0.008476) | 0.028053 / 0.128546 (-0.100494) | 0.010521 / 0.075646 (-0.065125) | 0.210954 / 0.419271 (-0.208317) | 0.035720 / 0.043533 (-0.007813) | 0.252540 / 0.255139 (-0.002599) | 0.264786 / 0.283200 (-0.018414) | 0.018692 / 0.141683 (-0.122990) | 1.108971 / 1.452155 (-0.343183) | 1.201004 / 1.492716 (-0.291712) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095936 / 0.018006 (0.077930) | 0.302979 / 0.000490 (0.302489) | 0.000217 / 0.000200 (0.000017) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018859 / 0.037411 (-0.018552) | 0.062559 / 0.014526 (0.048034) | 0.073545 / 0.176557 (-0.103012) | 0.120780 / 0.737135 (-0.616355) | 0.074998 / 0.296338 (-0.221340) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.276728 / 0.215209 (0.061519) | 2.715310 / 2.077655 (0.637655) | 1.444927 / 1.504120 (-0.059193) | 1.323867 / 1.541195 (-0.217328) | 1.364962 / 1.468490 (-0.103528) | 0.556792 / 4.584777 (-4.027985) | 2.409151 / 3.745712 (-1.336561) | 2.811836 / 5.269862 (-2.458026) | 1.777369 / 4.565676 (-2.788308) | 0.061398 / 0.424275 (-0.362877) | 0.004924 / 0.007607 (-0.002683) | 0.341228 / 0.226044 (0.115183) | 3.369570 / 2.268929 (1.100641) | 1.858151 / 55.444624 (-53.586474) | 1.587352 / 6.876477 (-5.289125) | 1.625004 / 2.142072 (-0.517068) | 0.635317 / 4.805227 (-4.169910) | 0.117197 / 6.500664 (-6.383467) | 0.042672 / 0.075469 (-0.032797) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.940419 / 1.841788 (-0.901368) | 12.156882 / 8.074308 (4.082574) | 10.646780 / 10.191392 (0.455388) | 0.129279 / 0.680424 (-0.551144) | 0.013967 / 0.534201 (-0.520234) | 0.287956 / 0.579283 (-0.291327) | 0.265250 / 0.434364 (-0.169114) | 0.323357 / 0.540337 (-0.216980) | 0.412045 / 1.386936 (-0.974891) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005264 / 0.011353 (-0.006089) | 0.003575 / 0.011008 (-0.007433) | 0.049249 / 0.038508 (0.010741) | 0.057069 / 0.023109 (0.033959) | 0.327547 / 0.275898 (0.051649) | 0.299027 / 0.323480 (-0.024453) | 0.004768 / 0.007986 (-0.003217) | 0.002522 / 0.004328 (-0.001807) | 0.048020 / 0.004250 (0.043770) | 0.041328 / 0.037052 (0.004275) | 0.281385 / 0.258489 (0.022895) | 0.304957 / 0.293841 (0.011116) | 0.031371 / 0.128546 (-0.097175) | 0.010523 / 0.075646 (-0.065124) | 0.057073 / 0.419271 (-0.362198) | 0.032913 / 0.043533 (-0.010620) | 0.284963 / 0.255139 (0.029824) | 0.291997 / 0.283200 (0.008798) | 0.018325 / 0.141683 (-0.123357) | 1.126681 / 1.452155 (-0.325473) | 1.183011 / 1.492716 (-0.309705) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092544 / 0.018006 (0.074538) | 0.299841 / 0.000490 (0.299351) | 0.000221 / 0.000200 (0.000021) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022279 / 0.037411 (-0.015133) | 0.072515 / 0.014526 (0.057989) | 0.083068 / 0.176557 (-0.093488) | 0.120600 / 0.737135 (-0.616536) | 0.083574 / 0.296338 (-0.212765) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.293393 / 0.215209 (0.078184) | 2.865420 / 2.077655 (0.787765) | 1.562419 / 1.504120 (0.058299) | 1.440846 / 1.541195 (-0.100349) | 1.471993 / 1.468490 (0.003503) | 0.572510 / 4.584777 (-4.012267) | 2.427417 / 3.745712 (-1.318295) | 2.895347 / 5.269862 (-2.374515) | 1.790578 / 4.565676 (-2.775098) | 0.064489 / 0.424275 (-0.359786) | 0.005044 / 0.007607 (-0.002564) | 0.340774 / 0.226044 (0.114730) | 3.391414 / 2.268929 (1.122486) | 1.939980 / 55.444624 (-53.504644) | 1.658514 / 6.876477 (-5.217963) | 1.741406 / 2.142072 (-0.400667) | 0.649033 / 4.805227 (-4.156194) | 0.117587 / 6.500664 (-6.383077) | 0.042042 / 0.075469 (-0.033427) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.980490 / 1.841788 (-0.861298) | 12.664045 / 8.074308 (4.589737) | 10.944437 / 10.191392 (0.753045) | 0.142059 / 0.680424 (-0.538365) | 0.015914 / 0.534201 (-0.518287) | 0.288826 / 0.579283 (-0.290457) | 0.282351 / 0.434364 (-0.152013) | 0.325302 / 0.540337 (-0.215035) | 0.416900 / 1.386936 (-0.970036) |

</details>
</details>

![](https://cml.dev/watermark.png#59750317ad258a4380ab6a6d206932b8d482ece1 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005591 / 0.011353 (-0.005762) | 0.003445 / 0.011008 (-0.007563) | 0.064290 / 0.038508 (0.025782) | 0.053046 / 0.023109 (0.029936) | 0.229101 / 0.275898 (-0.046797) | 0.255515 / 0.323480 (-0.067964) | 0.002912 / 0.007986 (-0.005073) | 0.002466 / 0.004328 (-0.001863) | 0.049348 / 0.004250 (0.045098) | 0.039492 / 0.037052 (0.002440) | 0.236301 / 0.258489 (-0.022188) | 0.270109 / 0.293841 (-0.023732) | 0.027506 / 0.128546 (-0.101040) | 0.010381 / 0.075646 (-0.065265) | 0.209999 / 0.419271 (-0.209273) | 0.035827 / 0.043533 (-0.007705) | 0.237231 / 0.255139 (-0.017908) | 0.254345 / 0.283200 (-0.028854) | 0.019689 / 0.141683 (-0.121994) | 1.096103 / 1.452155 (-0.356052) | 1.172393 / 1.492716 (-0.320323) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.101749 / 0.018006 (0.083743) | 0.310913 / 0.000490 (0.310424) | 0.000217 / 0.000200 (0.000017) | 0.000053 / 0.000054 (-0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018743 / 0.037411 (-0.018669) | 0.064190 / 0.014526 (0.049664) | 0.074575 / 0.176557 (-0.101982) | 0.124143 / 0.737135 (-0.612993) | 0.077415 / 0.296338 (-0.218924) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.286175 / 0.215209 (0.070965) | 2.781169 / 2.077655 (0.703515) | 1.495130 / 1.504120 (-0.008990) | 1.379136 / 1.541195 (-0.162059) | 1.397548 / 1.468490 (-0.070942) | 0.564467 / 4.584777 (-4.020310) | 2.408896 / 3.745712 (-1.336816) | 2.857771 / 5.269862 (-2.412091) | 1.776531 / 4.565676 (-2.789145) | 0.062700 / 0.424275 (-0.361575) | 0.004965 / 0.007607 (-0.002642) | 0.344026 / 0.226044 (0.117982) | 3.390829 / 2.268929 (1.121900) | 1.875258 / 55.444624 (-53.569366) | 1.602435 / 6.876477 (-5.274042) | 1.613619 / 2.142072 (-0.528454) | 0.639421 / 4.805227 (-4.165806) | 0.117697 / 6.500664 (-6.382967) | 0.042878 / 0.075469 (-0.032591) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.957694 / 1.841788 (-0.884094) | 11.888917 / 8.074308 (3.814609) | 10.643389 / 10.191392 (0.451997) | 0.143358 / 0.680424 (-0.537066) | 0.014382 / 0.534201 (-0.519819) | 0.288731 / 0.579283 (-0.290552) | 0.270040 / 0.434364 (-0.164324) | 0.323586 / 0.540337 (-0.216751) | 0.415743 / 1.386936 (-0.971193) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005228 / 0.011353 (-0.006125) | 0.003445 / 0.011008 (-0.007563) | 0.051072 / 0.038508 (0.012563) | 0.053087 / 0.023109 (0.029978) | 0.273116 / 0.275898 (-0.002782) | 0.298633 / 0.323480 (-0.024847) | 0.004067 / 0.007986 (-0.003919) | 0.002537 / 0.004328 (-0.001791) | 0.049326 / 0.004250 (0.045075) | 0.041011 / 0.037052 (0.003959) | 0.277748 / 0.258489 (0.019258) | 0.304152 / 0.293841 (0.010311) | 0.029012 / 0.128546 (-0.099534) | 0.010589 / 0.075646 (-0.065057) | 0.057564 / 0.419271 (-0.361707) | 0.032785 / 0.043533 (-0.010747) | 0.272508 / 0.255139 (0.017369) | 0.294127 / 0.283200 (0.010927) | 0.018466 / 0.141683 (-0.123217) | 1.129341 / 1.452155 (-0.322814) | 1.194631 / 1.492716 (-0.298086) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.098558 / 0.018006 (0.080552) | 0.312353 / 0.000490 (0.311863) | 0.000269 / 0.000200 (0.000069) | 0.000049 / 0.000054 (-0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022148 / 0.037411 (-0.015263) | 0.070601 / 0.014526 (0.056075) | 0.081780 / 0.176557 (-0.094777) | 0.121993 / 0.737135 (-0.615142) | 0.084263 / 0.296338 (-0.212076) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.300501 / 0.215209 (0.085292) | 2.927534 / 2.077655 (0.849879) | 1.595527 / 1.504120 (0.091407) | 1.475607 / 1.541195 (-0.065587) | 1.496707 / 1.468490 (0.028217) | 0.559051 / 4.584777 (-4.025726) | 2.427126 / 3.745712 (-1.318586) | 2.820908 / 5.269862 (-2.448953) | 1.757492 / 4.565676 (-2.808185) | 0.062391 / 0.424275 (-0.361884) | 0.004950 / 0.007607 (-0.002657) | 0.351204 / 0.226044 (0.125160) | 3.485068 / 2.268929 (1.216139) | 1.976418 / 55.444624 (-53.468207) | 1.682715 / 6.876477 (-5.193761) | 1.703457 / 2.142072 (-0.438616) | 0.643476 / 4.805227 (-4.161751) | 0.116321 / 6.500664 (-6.384343) | 0.040776 / 0.075469 (-0.034694) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.974152 / 1.841788 (-0.867635) | 12.390170 / 8.074308 (4.315862) | 10.866283 / 10.191392 (0.674891) | 0.145049 / 0.680424 (-0.535375) | 0.016404 / 0.534201 (-0.517797) | 0.288799 / 0.579283 (-0.290484) | 0.285917 / 0.434364 (-0.148447) | 0.328455 / 0.540337 (-0.211883) | 0.417286 / 1.386936 (-0.969650) |

</details>
</details>

![](https://cml.dev/watermark.png#59750317ad258a4380ab6a6d206932b8d482ece1 "CML watermark")