Title: minor release step improvement
Author: lhoestq (MEMBER)
State: closed
Created: 2023-10-23T11:07:04Z


Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006572 / 0.011353 (-0.004780) | 0.004019 / 0.011008 (-0.006989) | 0.084080 / 0.038508 (0.045572) | 0.070111 / 0.023109 (0.047002) | 0.340440 / 0.275898 (0.064542) | 0.358839 / 0.323480 (0.035359) | 0.005254 / 0.007986 (-0.002732) | 0.003296 / 0.004328 (-0.001032) | 0.064368 / 0.004250 (0.060117) | 0.054549 / 0.037052 (0.017497) | 0.343817 / 0.258489 (0.085328) | 0.369871 / 0.293841 (0.076030) | 0.030621 / 0.128546 (-0.097925) | 0.008457 / 0.075646 (-0.067189) | 0.287839 / 0.419271 (-0.131432) | 0.051700 / 0.043533 (0.008167) | 0.331602 / 0.255139 (0.076463) | 0.339836 / 0.283200 (0.056636) | 0.023224 / 0.141683 (-0.118459) | 1.494597 / 1.452155 (0.042443) | 1.578640 / 1.492716 (0.085924) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.236985 / 0.018006 (0.218979) | 0.506153 / 0.000490 (0.505664) | 0.009753 / 0.000200 (0.009553) | 0.000345 / 0.000054 (0.000291) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028355 / 0.037411 (-0.009056) | 0.082104 / 0.014526 (0.067578) | 0.095141 / 0.176557 (-0.081415) | 0.151054 / 0.737135 (-0.586081) | 0.095139 / 0.296338 (-0.201200) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.403773 / 0.215209 (0.188564) | 4.025567 / 2.077655 (1.947912) | 2.024641 / 1.504120 (0.520521) | 1.857039 / 1.541195 (0.315845) | 1.957346 / 1.468490 (0.488856) | 0.481486 / 4.584777 (-4.103291) | 3.574463 / 3.745712 (-0.171249) | 3.399311 / 5.269862 (-1.870551) | 1.996806 / 4.565676 (-2.568870) | 0.056644 / 0.424275 (-0.367631) | 0.007503 / 0.007607 (-0.000104) | 0.479480 / 0.226044 (0.253435) | 4.793686 / 2.268929 (2.524757) | 2.481011 / 55.444624 (-52.963613) | 2.176473 / 6.876477 (-4.700004) | 2.203192 / 2.142072 (0.061120) | 0.574071 / 4.805227 (-4.231156) | 0.131852 / 6.500664 (-6.368812) | 0.058883 / 0.075469 (-0.016586) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.249945 / 1.841788 (-0.591842) | 18.439267 / 8.074308 (10.364959) | 14.100934 / 10.191392 (3.909542) | 0.164191 / 0.680424 (-0.516233) | 0.018086 / 0.534201 (-0.516115) | 0.390821 / 0.579283 (-0.188462) | 0.414166 / 0.434364 (-0.020198) | 0.460073 / 0.540337 (-0.080265) | 0.636299 / 1.386936 (-0.750637) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006606 / 0.011353 (-0.004747) | 0.003987 / 0.011008 (-0.007021) | 0.064616 / 0.038508 (0.026108) | 0.070830 / 0.023109 (0.047721) | 0.397340 / 0.275898 (0.121442) | 0.426823 / 0.323480 (0.103343) | 0.005345 / 0.007986 (-0.002641) | 0.003264 / 0.004328 (-0.001065) | 0.064728 / 0.004250 (0.060477) | 0.055763 / 0.037052 (0.018711) | 0.405347 / 0.258489 (0.146858) | 0.433163 / 0.293841 (0.139322) | 0.032394 / 0.128546 (-0.096153) | 0.008474 / 0.075646 (-0.067172) | 0.071583 / 0.419271 (-0.347689) | 0.048424 / 0.043533 (0.004892) | 0.400582 / 0.255139 (0.145443) | 0.418111 / 0.283200 (0.134911) | 0.022257 / 0.141683 (-0.119426) | 1.495521 / 1.452155 (0.043366) | 1.554626 / 1.492716 (0.061910) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.218249 / 0.018006 (0.200242) | 0.438527 / 0.000490 (0.438037) | 0.005406 / 0.000200 (0.005206) | 0.000098 / 0.000054 (0.000044) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031600 / 0.037411 (-0.005812) | 0.090836 / 0.014526 (0.076310) | 0.105000 / 0.176557 (-0.071556) | 0.157648 / 0.737135 (-0.579487) | 0.103827 / 0.296338 (-0.192512) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.426428 / 0.215209 (0.211219) | 4.259435 / 2.077655 (2.181780) | 2.300795 / 1.504120 (0.796675) | 2.121302 / 1.541195 (0.580108) | 2.145602 / 1.468490 (0.677112) | 0.486856 / 4.584777 (-4.097921) | 3.673568 / 3.745712 (-0.072144) | 3.278619 / 5.269862 (-1.991243) | 2.037760 / 4.565676 (-2.527917) | 0.057699 / 0.424275 (-0.366576) | 0.007269 / 0.007607 (-0.000338) | 0.499549 / 0.226044 (0.273505) | 4.996214 / 2.268929 (2.727285) | 2.766480 / 55.444624 (-52.678144) | 2.417308 / 6.876477 (-4.459168) | 2.581026 / 2.142072 (0.438953) | 0.589463 / 4.805227 (-4.215765) | 0.134820 / 6.500664 (-6.365844) | 0.061699 / 0.075469 (-0.013770) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.353704 / 1.841788 (-0.488084) | 19.104167 / 8.074308 (11.029859) | 14.652166 / 10.191392 (4.460774) | 0.171885 / 0.680424 (-0.508539) | 0.020222 / 0.534201 (-0.513978) | 0.396777 / 0.579283 (-0.182506) | 0.426304 / 0.434364 (-0.008060) | 0.471347 / 0.540337 (-0.068991) | 0.635887 / 1.386936 (-0.751049) |

</details>
</details>

![](https://cml.dev/watermark.png#ce19ec527c581eddec306a03ad1db554223cc94a "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004686 / 0.011353 (-0.006667) | 0.002998 / 0.011008 (-0.008010) | 0.063604 / 0.038508 (0.025096) | 0.048927 / 0.023109 (0.025818) | 0.247238 / 0.275898 (-0.028660) | 0.272409 / 0.323480 (-0.051071) | 0.003909 / 0.007986 (-0.004077) | 0.002469 / 0.004328 (-0.001859) | 0.048473 / 0.004250 (0.044223) | 0.037514 / 0.037052 (0.000462) | 0.257292 / 0.258489 (-0.001197) | 0.285203 / 0.293841 (-0.008638) | 0.023131 / 0.128546 (-0.105415) | 0.006803 / 0.075646 (-0.068843) | 0.202920 / 0.419271 (-0.216351) | 0.035653 / 0.043533 (-0.007880) | 0.254791 / 0.255139 (-0.000348) | 0.272973 / 0.283200 (-0.010226) | 0.017707 / 0.141683 (-0.123976) | 1.091606 / 1.452155 (-0.360549) | 1.151453 / 1.492716 (-0.341263) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093701 / 0.018006 (0.075695) | 0.304199 / 0.000490 (0.303709) | 0.000223 / 0.000200 (0.000023) | 0.000051 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019291 / 0.037411 (-0.018120) | 0.062168 / 0.014526 (0.047642) | 0.073273 / 0.176557 (-0.103284) | 0.119497 / 0.737135 (-0.617638) | 0.075008 / 0.296338 (-0.221331) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.279983 / 0.215209 (0.064774) | 2.774413 / 2.077655 (0.696758) | 1.476678 / 1.504120 (-0.027441) | 1.336273 / 1.541195 (-0.204922) | 1.332349 / 1.468490 (-0.136142) | 0.403150 / 4.584777 (-4.181627) | 2.390026 / 3.745712 (-1.355686) | 2.619151 / 5.269862 (-2.650711) | 1.578607 / 4.565676 (-2.987069) | 0.046632 / 0.424275 (-0.377643) | 0.007352 / 0.007607 (-0.000255) | 0.333419 / 0.226044 (0.107375) | 3.288734 / 2.268929 (1.019805) | 1.843677 / 55.444624 (-53.600947) | 1.536746 / 6.876477 (-5.339731) | 1.573005 / 2.142072 (-0.569067) | 0.475699 / 4.805227 (-4.329529) | 0.104742 / 6.500664 (-6.395922) | 0.042450 / 0.075469 (-0.033019) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.949039 / 1.841788 (-0.892749) | 11.895928 / 8.074308 (3.821620) | 10.650521 / 10.191392 (0.459129) | 0.142308 / 0.680424 (-0.538116) | 0.014207 / 0.534201 (-0.519994) | 0.274011 / 0.579283 (-0.305272) | 0.288259 / 0.434364 (-0.146105) | 0.327729 / 0.540337 (-0.212609) | 0.395728 / 1.386936 (-0.991208) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004830 / 0.011353 (-0.006523) | 0.002978 / 0.011008 (-0.008030) | 0.048623 / 0.038508 (0.010114) | 0.055040 / 0.023109 (0.031930) | 0.276436 / 0.275898 (0.000538) | 0.302403 / 0.323480 (-0.021076) | 0.004080 / 0.007986 (-0.003905) | 0.002479 / 0.004328 (-0.001849) | 0.048078 / 0.004250 (0.043827) | 0.039680 / 0.037052 (0.002627) | 0.279095 / 0.258489 (0.020606) | 0.307399 / 0.293841 (0.013558) | 0.024533 / 0.128546 (-0.104013) | 0.007196 / 0.075646 (-0.068450) | 0.053879 / 0.419271 (-0.365393) | 0.032545 / 0.043533 (-0.010988) | 0.275501 / 0.255139 (0.020362) | 0.298530 / 0.283200 (0.015330) | 0.017992 / 0.141683 (-0.123691) | 1.144191 / 1.452155 (-0.307963) | 1.208309 / 1.492716 (-0.284408) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095690 / 0.018006 (0.077684) | 0.304932 / 0.000490 (0.304442) | 0.000223 / 0.000200 (0.000023) | 0.000055 / 0.000054 (0.000000) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021409 / 0.037411 (-0.016003) | 0.069861 / 0.014526 (0.055335) | 0.080959 / 0.176557 (-0.095597) | 0.119432 / 0.737135 (-0.617703) | 0.083649 / 0.296338 (-0.212690) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.297243 / 0.215209 (0.082034) | 2.909288 / 2.077655 (0.831634) | 1.571512 / 1.504120 (0.067392) | 1.452403 / 1.541195 (-0.088792) | 1.481290 / 1.468490 (0.012800) | 0.405795 / 4.584777 (-4.178982) | 2.452923 / 3.745712 (-1.292789) | 2.513371 / 5.269862 (-2.756490) | 1.593216 / 4.565676 (-2.972460) | 0.048073 / 0.424275 (-0.376202) | 0.005312 / 0.007607 (-0.002296) | 0.355783 / 0.226044 (0.129738) | 3.494062 / 2.268929 (1.225133) | 1.947388 / 55.444624 (-53.497236) | 1.651724 / 6.876477 (-5.224753) | 1.789007 / 2.142072 (-0.353065) | 0.487073 / 4.805227 (-4.318154) | 0.100271 / 6.500664 (-6.400393) | 0.041571 / 0.075469 (-0.033898) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.983766 / 1.841788 (-0.858021) | 12.384778 / 8.074308 (4.310469) | 10.669519 / 10.191392 (0.478127) | 0.133105 / 0.680424 (-0.547318) | 0.016665 / 0.534201 (-0.517536) | 0.269479 / 0.579283 (-0.309804) | 0.276498 / 0.434364 (-0.157866) | 0.302105 / 0.540337 (-0.238233) | 0.391204 / 1.386936 (-0.995732) |

</details>
</details>

![](https://cml.dev/watermark.png#12ebe695b4748c5a26e08b44ed51955f74f5801d "CML watermark")