Title: Fix select_columns columns order
Author: lhoestq (MEMBER)
State: closed
Created: 2023-06-27T12:32:46Z

Fix the order of the columns in dataset.features when the order changes with `dataset.select_columns()`.

I also fixed the same issue for `dataset.flatten()`

Close https://github.com/huggingface/datasets/issues/5993

Comments:
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005969 / 0.011353 (-0.005384) | 0.003687 / 0.011008 (-0.007321) | 0.100843 / 0.038508 (0.062335) | 0.036912 / 0.023109 (0.013803) | 0.312389 / 0.275898 (0.036491) | 0.370335 / 0.323480 (0.046855) | 0.003434 / 0.007986 (-0.004552) | 0.003710 / 0.004328 (-0.000619) | 0.076899 / 0.004250 (0.072648) | 0.053647 / 0.037052 (0.016594) | 0.324825 / 0.258489 (0.066336) | 0.367711 / 0.293841 (0.073870) | 0.028079 / 0.128546 (-0.100467) | 0.008326 / 0.075646 (-0.067320) | 0.312342 / 0.419271 (-0.106930) | 0.047423 / 0.043533 (0.003890) | 0.321063 / 0.255139 (0.065924) | 0.336508 / 0.283200 (0.053308) | 0.019973 / 0.141683 (-0.121710) | 1.529334 / 1.452155 (0.077179) | 1.573746 / 1.492716 (0.081030) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.210849 / 0.018006 (0.192843) | 0.418798 / 0.000490 (0.418309) | 0.007347 / 0.000200 (0.007147) | 0.000070 / 0.000054 (0.000016) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022718 / 0.037411 (-0.014694) | 0.098400 / 0.014526 (0.083874) | 0.106590 / 0.176557 (-0.069967) | 0.168460 / 0.737135 (-0.568675) | 0.108401 / 0.296338 (-0.187938) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.443066 / 0.215209 (0.227857) | 4.416658 / 2.077655 (2.339003) | 2.088844 / 1.504120 (0.584724) | 1.879564 / 1.541195 (0.338369) | 1.933815 / 1.468490 (0.465325) | 0.565085 / 4.584777 (-4.019692) | 3.412440 / 3.745712 (-0.333273) | 1.754686 / 5.269862 (-3.515175) | 1.024576 / 4.565676 (-3.541100) | 0.067909 / 0.424275 (-0.356366) | 0.011054 / 0.007607 (0.003447) | 0.534748 / 0.226044 (0.308703) | 5.351457 / 2.268929 (3.082529) | 2.517368 / 55.444624 (-52.927256) | 2.182762 / 6.876477 (-4.693715) | 2.238205 / 2.142072 (0.096133) | 0.672962 / 4.805227 (-4.132265) | 0.136098 / 6.500664 (-6.364566) | 0.066534 / 0.075469 (-0.008935) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.281241 / 1.841788 (-0.560547) | 13.872881 / 8.074308 (5.798573) | 13.161023 / 10.191392 (2.969631) | 0.130011 / 0.680424 (-0.550412) | 0.016759 / 0.534201 (-0.517442) | 0.359802 / 0.579283 (-0.219481) | 0.392577 / 0.434364 (-0.041787) | 0.427742 / 0.540337 (-0.112595) | 0.522241 / 1.386936 (-0.864695) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005985 / 0.011353 (-0.005368) | 0.003705 / 0.011008 (-0.007304) | 0.077699 / 0.038508 (0.039191) | 0.035686 / 0.023109 (0.012577) | 0.420356 / 0.275898 (0.144458) | 0.476753 / 0.323480 (0.153273) | 0.003510 / 0.007986 (-0.004475) | 0.002807 / 0.004328 (-0.001521) | 0.077151 / 0.004250 (0.072901) | 0.046420 / 0.037052 (0.009368) | 0.391781 / 0.258489 (0.133292) | 0.461128 / 0.293841 (0.167287) | 0.027847 / 0.128546 (-0.100699) | 0.008322 / 0.075646 (-0.067324) | 0.082768 / 0.419271 (-0.336503) | 0.042629 / 0.043533 (-0.000904) | 0.405745 / 0.255139 (0.150606) | 0.430797 / 0.283200 (0.147598) | 0.019832 / 0.141683 (-0.121851) | 1.556208 / 1.452155 (0.104054) | 1.612166 / 1.492716 (0.119450) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.230633 / 0.018006 (0.212626) | 0.401667 / 0.000490 (0.401178) | 0.000776 / 0.000200 (0.000576) | 0.000069 / 0.000054 (0.000014) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024959 / 0.037411 (-0.012452) | 0.100560 / 0.014526 (0.086034) | 0.109175 / 0.176557 (-0.067382) | 0.159919 / 0.737135 (-0.577217) | 0.112810 / 0.296338 (-0.183528) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.460601 / 0.215209 (0.245392) | 4.620039 / 2.077655 (2.542385) | 2.257900 / 1.504120 (0.753780) | 2.039192 / 1.541195 (0.497997) | 2.064451 / 1.468490 (0.595961) | 0.557887 / 4.584777 (-4.026890) | 3.356100 / 3.745712 (-0.389612) | 1.703578 / 5.269862 (-3.566284) | 1.024984 / 4.565676 (-3.540693) | 0.067602 / 0.424275 (-0.356673) | 0.011450 / 0.007607 (0.003842) | 0.563230 / 0.226044 (0.337186) | 5.632150 / 2.268929 (3.363221) | 2.698701 / 55.444624 (-52.745924) | 2.363218 / 6.876477 (-4.513259) | 2.363997 / 2.142072 (0.221925) | 0.671260 / 4.805227 (-4.133967) | 0.136166 / 6.500664 (-6.364499) | 0.067094 / 0.075469 (-0.008375) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.303030 / 1.841788 (-0.538757) | 14.137277 / 8.074308 (6.062969) | 13.937631 / 10.191392 (3.746239) | 0.162626 / 0.680424 (-0.517798) | 0.016687 / 0.534201 (-0.517514) | 0.363657 / 0.579283 (-0.215626) | 0.392021 / 0.434364 (-0.042343) | 0.427275 / 0.540337 (-0.113062) | 0.512192 / 1.386936 (-0.874744) |

</details>
</details>

![](https://cml.dev/watermark.png#42603528d9bd8c3ab287ed0eadc7fa3d1ef4cfd8 "CML watermark")


_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005974 / 0.011353 (-0.005378) | 0.003947 / 0.011008 (-0.007061) | 0.098604 / 0.038508 (0.060096) | 0.036947 / 0.023109 (0.013838) | 0.311844 / 0.275898 (0.035946) | 0.375243 / 0.323480 (0.051763) | 0.003453 / 0.007986 (-0.004533) | 0.003834 / 0.004328 (-0.000495) | 0.077943 / 0.004250 (0.073692) | 0.052956 / 0.037052 (0.015904) | 0.320812 / 0.258489 (0.062323) | 0.373963 / 0.293841 (0.080122) | 0.028382 / 0.128546 (-0.100164) | 0.008525 / 0.075646 (-0.067121) | 0.311306 / 0.419271 (-0.107965) | 0.047029 / 0.043533 (0.003496) | 0.309933 / 0.255139 (0.054794) | 0.335114 / 0.283200 (0.051915) | 0.019629 / 0.141683 (-0.122054) | 1.569771 / 1.452155 (0.117617) | 1.585899 / 1.492716 (0.093182) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.216565 / 0.018006 (0.198559) | 0.426717 / 0.000490 (0.426228) | 0.003609 / 0.000200 (0.003409) | 0.000077 / 0.000054 (0.000023) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023079 / 0.037411 (-0.014332) | 0.096954 / 0.014526 (0.082428) | 0.105398 / 0.176557 (-0.071158) | 0.165433 / 0.737135 (-0.571703) | 0.109703 / 0.296338 (-0.186636) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.456227 / 0.215209 (0.241018) | 4.529857 / 2.077655 (2.452202) | 2.214054 / 1.504120 (0.709934) | 2.029716 / 1.541195 (0.488521) | 2.081175 / 1.468490 (0.612685) | 0.563642 / 4.584777 (-4.021135) | 3.355393 / 3.745712 (-0.390320) | 1.765938 / 5.269862 (-3.503924) | 1.039062 / 4.565676 (-3.526615) | 0.067952 / 0.424275 (-0.356323) | 0.011044 / 0.007607 (0.003437) | 0.556935 / 0.226044 (0.330890) | 5.588167 / 2.268929 (3.319239) | 2.667217 / 55.444624 (-52.777407) | 2.337383 / 6.876477 (-4.539094) | 2.429590 / 2.142072 (0.287517) | 0.676972 / 4.805227 (-4.128256) | 0.135782 / 6.500664 (-6.364882) | 0.066323 / 0.075469 (-0.009146) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.237358 / 1.841788 (-0.604429) | 13.910492 / 8.074308 (5.836184) | 13.227275 / 10.191392 (3.035883) | 0.146857 / 0.680424 (-0.533567) | 0.016991 / 0.534201 (-0.517210) | 0.363637 / 0.579283 (-0.215646) | 0.392462 / 0.434364 (-0.041902) | 0.450009 / 0.540337 (-0.090329) | 0.536077 / 1.386936 (-0.850859) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006067 / 0.011353 (-0.005286) | 0.003851 / 0.011008 (-0.007158) | 0.078462 / 0.038508 (0.039954) | 0.036221 / 0.023109 (0.013112) | 0.389195 / 0.275898 (0.113297) | 0.428710 / 0.323480 (0.105230) | 0.004645 / 0.007986 (-0.003341) | 0.002973 / 0.004328 (-0.001355) | 0.078299 / 0.004250 (0.074048) | 0.047076 / 0.037052 (0.010024) | 0.375673 / 0.258489 (0.117184) | 0.432352 / 0.293841 (0.138511) | 0.028212 / 0.128546 (-0.100334) | 0.008475 / 0.075646 (-0.067172) | 0.083902 / 0.419271 (-0.335369) | 0.046699 / 0.043533 (0.003166) | 0.364502 / 0.255139 (0.109363) | 0.389792 / 0.283200 (0.106592) | 0.025266 / 0.141683 (-0.116417) | 1.517458 / 1.452155 (0.065303) | 1.543634 / 1.492716 (0.050918) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.236479 / 0.018006 (0.218472) | 0.411528 / 0.000490 (0.411038) | 0.005213 / 0.000200 (0.005013) | 0.000091 / 0.000054 (0.000036) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025764 / 0.037411 (-0.011647) | 0.103174 / 0.014526 (0.088648) | 0.110609 / 0.176557 (-0.065948) | 0.164630 / 0.737135 (-0.572506) | 0.114863 / 0.296338 (-0.181475) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.457155 / 0.215209 (0.241946) | 4.550675 / 2.077655 (2.473021) | 2.350473 / 1.504120 (0.846353) | 2.204919 / 1.541195 (0.663724) | 2.076724 / 1.468490 (0.608234) | 0.563107 / 4.584777 (-4.021670) | 3.390669 / 3.745712 (-0.355043) | 1.741111 / 5.269862 (-3.528751) | 1.033268 / 4.565676 (-3.532408) | 0.068400 / 0.424275 (-0.355875) | 0.011607 / 0.007607 (0.004000) | 0.561944 / 0.226044 (0.335900) | 5.620224 / 2.268929 (3.351296) | 2.705241 / 55.444624 (-52.739384) | 2.344520 / 6.876477 (-4.531957) | 2.386119 / 2.142072 (0.244046) | 0.681583 / 4.805227 (-4.123644) | 0.137272 / 6.500664 (-6.363392) | 0.069217 / 0.075469 (-0.006252) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.322690 / 1.841788 (-0.519098) | 14.464953 / 8.074308 (6.390645) | 14.269350 / 10.191392 (4.077958) | 0.158879 / 0.680424 (-0.521545) | 0.016722 / 0.534201 (-0.517479) | 0.360299 / 0.579283 (-0.218984) | 0.391609 / 0.434364 (-0.042755) | 0.420507 / 0.540337 (-0.119831) | 0.512822 / 1.386936 (-0.874114) |

</details>
</details>

![](https://cml.dev/watermark.png#ca68191900d97b29abb3c2c4ba0502fe30d137d1 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007106 / 0.011353 (-0.004247) | 0.005224 / 0.011008 (-0.005784) | 0.127563 / 0.038508 (0.089055) | 0.055067 / 0.023109 (0.031958) | 0.418660 / 0.275898 (0.142761) | 0.487891 / 0.323480 (0.164411) | 0.005712 / 0.007986 (-0.002274) | 0.004585 / 0.004328 (0.000256) | 0.090994 / 0.004250 (0.086743) | 0.071837 / 0.037052 (0.034784) | 0.446957 / 0.258489 (0.188468) | 0.475966 / 0.293841 (0.182125) | 0.038062 / 0.128546 (-0.090484) | 0.010056 / 0.075646 (-0.065590) | 0.406796 / 0.419271 (-0.012475) | 0.066542 / 0.043533 (0.023009) | 0.413676 / 0.255139 (0.158537) | 0.448624 / 0.283200 (0.165424) | 0.030332 / 0.141683 (-0.111351) | 1.895307 / 1.452155 (0.443152) | 1.904411 / 1.492716 (0.411694) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.221246 / 0.018006 (0.203240) | 0.461288 / 0.000490 (0.460799) | 0.005957 / 0.000200 (0.005757) | 0.000112 / 0.000054 (0.000058) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029255 / 0.037411 (-0.008156) | 0.131299 / 0.014526 (0.116773) | 0.135814 / 0.176557 (-0.040742) | 0.201342 / 0.737135 (-0.535793) | 0.141748 / 0.296338 (-0.154591) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.463936 / 0.215209 (0.248727) | 4.709621 / 2.077655 (2.631966) | 2.093844 / 1.504120 (0.589724) | 1.897963 / 1.541195 (0.356768) | 1.927865 / 1.468490 (0.459375) | 0.610879 / 4.584777 (-3.973898) | 4.481370 / 3.745712 (0.735658) | 2.112235 / 5.269862 (-3.157627) | 1.203349 / 4.565676 (-3.362327) | 0.074828 / 0.424275 (-0.349447) | 0.013121 / 0.007607 (0.005514) | 0.580894 / 0.226044 (0.354849) | 5.801872 / 2.268929 (3.532943) | 2.579950 / 55.444624 (-52.864674) | 2.251569 / 6.876477 (-4.624908) | 2.421305 / 2.142072 (0.279232) | 0.760938 / 4.805227 (-4.044289) | 0.169554 / 6.500664 (-6.331110) | 0.077499 / 0.075469 (0.002030) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.410419 / 1.841788 (-0.431368) | 17.442331 / 8.074308 (9.368023) | 15.782183 / 10.191392 (5.590791) | 0.180649 / 0.680424 (-0.499775) | 0.021790 / 0.534201 (-0.512411) | 0.511040 / 0.579283 (-0.068243) | 0.510472 / 0.434364 (0.076108) | 0.607141 / 0.540337 (0.066804) | 0.724794 / 1.386936 (-0.662142) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007280 / 0.011353 (-0.004073) | 0.004712 / 0.011008 (-0.006296) | 0.089225 / 0.038508 (0.050717) | 0.053157 / 0.023109 (0.030048) | 0.431949 / 0.275898 (0.156051) | 0.478128 / 0.323480 (0.154648) | 0.006181 / 0.007986 (-0.001804) | 0.003387 / 0.004328 (-0.000941) | 0.083741 / 0.004250 (0.079490) | 0.071610 / 0.037052 (0.034557) | 0.414698 / 0.258489 (0.156209) | 0.484422 / 0.293841 (0.190581) | 0.034988 / 0.128546 (-0.093558) | 0.009831 / 0.075646 (-0.065816) | 0.089644 / 0.419271 (-0.329628) | 0.057053 / 0.043533 (0.013520) | 0.413144 / 0.255139 (0.158005) | 0.445464 / 0.283200 (0.162264) | 0.026109 / 0.141683 (-0.115574) | 1.842899 / 1.452155 (0.390745) | 1.923774 / 1.492716 (0.431057) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.245051 / 0.018006 (0.227045) | 0.460444 / 0.000490 (0.459954) | 0.000444 / 0.000200 (0.000244) | 0.000067 / 0.000054 (0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034835 / 0.037411 (-0.002577) | 0.130078 / 0.014526 (0.115553) | 0.147012 / 0.176557 (-0.029544) | 0.203097 / 0.737135 (-0.534038) | 0.149636 / 0.296338 (-0.146702) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.521664 / 0.215209 (0.306455) | 5.283865 / 2.077655 (3.206210) | 2.456701 / 1.504120 (0.952581) | 2.266059 / 1.541195 (0.724864) | 2.295387 / 1.468490 (0.826897) | 0.613200 / 4.584777 (-3.971577) | 4.526107 / 3.745712 (0.780394) | 2.047327 / 5.269862 (-3.222535) | 1.261063 / 4.565676 (-3.304614) | 0.070402 / 0.424275 (-0.353873) | 0.014128 / 0.007607 (0.006521) | 0.620929 / 0.226044 (0.394884) | 6.109127 / 2.268929 (3.840198) | 3.081406 / 55.444624 (-52.363218) | 2.658224 / 6.876477 (-4.218253) | 2.671974 / 2.142072 (0.529902) | 0.744081 / 4.805227 (-4.061146) | 0.161498 / 6.500664 (-6.339166) | 0.075148 / 0.075469 (-0.000321) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.585640 / 1.841788 (-0.256148) | 17.884321 / 8.074308 (9.810013) | 15.938937 / 10.191392 (5.747545) | 0.220818 / 0.680424 (-0.459605) | 0.021452 / 0.534201 (-0.512749) | 0.499747 / 0.579283 (-0.079536) | 0.512318 / 0.434364 (0.077954) | 0.562853 / 0.540337 (0.022515) | 0.678512 / 1.386936 (-0.708424) |

</details>
</details>

![](https://cml.dev/watermark.png#aa50937d82256827aee3dbd749c7a23555e05e38 "CML watermark")