Title: Fix empty splitinfo json
Author: lhoestq (MEMBER)
State: closed
Created: 2023-09-04T13:13:53Z

If a split is empty, then the JSON split info should mention num_bytes = 0 and num_examples = 0.

Until now they were omited because the JSON dumps ignore the fields that are equal to the default values.

This is needed in datasets-server since we parse this information to the viewer

Comments:
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007756 / 0.011353 (-0.003597) | 0.004733 / 0.011008 (-0.006275) | 0.095874 / 0.038508 (0.057366) | 0.081957 / 0.023109 (0.058848) | 0.426430 / 0.275898 (0.150532) | 0.457670 / 0.323480 (0.134190) | 0.004448 / 0.007986 (-0.003537) | 0.004956 / 0.004328 (0.000627) | 0.074195 / 0.004250 (0.069945) | 0.061101 / 0.037052 (0.024048) | 0.435134 / 0.258489 (0.176645) | 0.457245 / 0.293841 (0.163404) | 0.034945 / 0.128546 (-0.093601) | 0.010028 / 0.075646 (-0.065618) | 0.350724 / 0.419271 (-0.068548) | 0.064433 / 0.043533 (0.020901) | 0.417882 / 0.255139 (0.162743) | 0.445087 / 0.283200 (0.161887) | 0.027576 / 0.141683 (-0.114107) | 1.824066 / 1.452155 (0.371912) | 1.957568 / 1.492716 (0.464852) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.238568 / 0.018006 (0.220562) | 0.505289 / 0.000490 (0.504799) | 0.003527 / 0.000200 (0.003327) | 0.000120 / 0.000054 (0.000065) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032839 / 0.037411 (-0.004572) | 0.096708 / 0.014526 (0.082182) | 0.112100 / 0.176557 (-0.064456) | 0.177215 / 0.737135 (-0.559920) | 0.111273 / 0.296338 (-0.185066) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.475200 / 0.215209 (0.259991) | 4.725737 / 2.077655 (2.648082) | 2.414672 / 1.504120 (0.910552) | 2.196357 / 1.541195 (0.655162) | 2.329298 / 1.468490 (0.860808) | 0.575258 / 4.584777 (-4.009519) | 4.343630 / 3.745712 (0.597918) | 3.837665 / 5.269862 (-1.432196) | 2.497970 / 4.565676 (-2.067706) | 0.066467 / 0.424275 (-0.357808) | 0.008680 / 0.007607 (0.001073) | 0.569923 / 0.226044 (0.343878) | 5.634230 / 2.268929 (3.365302) | 2.959222 / 55.444624 (-52.485402) | 2.535954 / 6.876477 (-4.340523) | 2.804844 / 2.142072 (0.662771) | 0.682000 / 4.805227 (-4.123227) | 0.158193 / 6.500664 (-6.342471) | 0.072315 / 0.075469 (-0.003154) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.578148 / 1.841788 (-0.263639) | 22.993419 / 8.074308 (14.919110) | 16.524477 / 10.191392 (6.333085) | 0.169415 / 0.680424 (-0.511009) | 0.021520 / 0.534201 (-0.512681) | 0.455970 / 0.579283 (-0.123313) | 0.489022 / 0.434364 (0.054658) | 0.535656 / 0.540337 (-0.004682) | 0.802341 / 1.386936 (-0.584595) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008002 / 0.011353 (-0.003351) | 0.005577 / 0.011008 (-0.005431) | 0.087803 / 0.038508 (0.049295) | 0.091285 / 0.023109 (0.068176) | 0.500514 / 0.275898 (0.224616) | 0.549770 / 0.323480 (0.226290) | 0.006125 / 0.007986 (-0.001861) | 0.004031 / 0.004328 (-0.000297) | 0.077941 / 0.004250 (0.073691) | 0.071419 / 0.037052 (0.034367) | 0.497570 / 0.258489 (0.239081) | 0.542454 / 0.293841 (0.248613) | 0.040827 / 0.128546 (-0.087719) | 0.011029 / 0.075646 (-0.064617) | 0.088788 / 0.419271 (-0.330484) | 0.056970 / 0.043533 (0.013438) | 0.523934 / 0.255139 (0.268795) | 0.552507 / 0.283200 (0.269308) | 0.029794 / 0.141683 (-0.111889) | 1.817778 / 1.452155 (0.365623) | 1.955843 / 1.492716 (0.463126) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.246992 / 0.018006 (0.228986) | 0.467879 / 0.000490 (0.467390) | 0.005439 / 0.000200 (0.005239) | 0.000110 / 0.000054 (0.000056) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.037774 / 0.037411 (0.000363) | 0.109332 / 0.014526 (0.094806) | 0.120103 / 0.176557 (-0.056454) | 0.185259 / 0.737135 (-0.551876) | 0.126189 / 0.296338 (-0.170149) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.492856 / 0.215209 (0.277646) | 5.033209 / 2.077655 (2.955554) | 2.885551 / 1.504120 (1.381431) | 2.480304 / 1.541195 (0.939109) | 2.579092 / 1.468490 (1.110602) | 0.557671 / 4.584777 (-4.027106) | 4.352765 / 3.745712 (0.607053) | 4.039124 / 5.269862 (-1.230738) | 2.534342 / 4.565676 (-2.031335) | 0.067267 / 0.424275 (-0.357008) | 0.008891 / 0.007607 (0.001284) | 0.591592 / 0.226044 (0.365547) | 5.939982 / 2.268929 (3.671053) | 3.258389 / 55.444624 (-52.186235) | 2.843899 / 6.876477 (-4.032578) | 3.074217 / 2.142072 (0.932144) | 0.695065 / 4.805227 (-4.110162) | 0.156917 / 6.500664 (-6.343747) | 0.070185 / 0.075469 (-0.005284) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.586716 / 1.841788 (-0.255072) | 23.405837 / 8.074308 (15.331529) | 17.200851 / 10.191392 (7.009459) | 0.170073 / 0.680424 (-0.510351) | 0.023345 / 0.534201 (-0.510856) | 0.459192 / 0.579283 (-0.120091) | 0.477419 / 0.434364 (0.043055) | 0.558581 / 0.540337 (0.018244) | 0.814373 / 1.386936 (-0.572563) |

</details>
</details>

![](https://cml.dev/watermark.png#28bbe5667e6eaa1bb21685791fcf1a4ed1ef1777 "CML watermark")


_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006050 / 0.011353 (-0.005303) | 0.003661 / 0.011008 (-0.007348) | 0.081753 / 0.038508 (0.043245) | 0.061275 / 0.023109 (0.038166) | 0.316278 / 0.275898 (0.040380) | 0.350783 / 0.323480 (0.027303) | 0.004694 / 0.007986 (-0.003291) | 0.003003 / 0.004328 (-0.001326) | 0.062877 / 0.004250 (0.058627) | 0.046985 / 0.037052 (0.009933) | 0.315698 / 0.258489 (0.057208) | 0.364607 / 0.293841 (0.070766) | 0.027365 / 0.128546 (-0.101181) | 0.008016 / 0.075646 (-0.067631) | 0.261379 / 0.419271 (-0.157893) | 0.045173 / 0.043533 (0.001640) | 0.313499 / 0.255139 (0.058360) | 0.339383 / 0.283200 (0.056184) | 0.020855 / 0.141683 (-0.120828) | 1.429851 / 1.452155 (-0.022303) | 1.506112 / 1.492716 (0.013396) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.194872 / 0.018006 (0.176866) | 0.451951 / 0.000490 (0.451462) | 0.002790 / 0.000200 (0.002590) | 0.000070 / 0.000054 (0.000015) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024331 / 0.037411 (-0.013081) | 0.073156 / 0.014526 (0.058630) | 0.084054 / 0.176557 (-0.092502) | 0.145656 / 0.737135 (-0.591480) | 0.084998 / 0.296338 (-0.211340) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.391324 / 0.215209 (0.176115) | 3.898406 / 2.077655 (1.820751) | 1.891175 / 1.504120 (0.387055) | 1.698738 / 1.541195 (0.157543) | 1.774324 / 1.468490 (0.305834) | 0.495129 / 4.584777 (-4.089648) | 3.027027 / 3.745712 (-0.718685) | 2.821423 / 5.269862 (-2.448439) | 1.870761 / 4.565676 (-2.694915) | 0.057029 / 0.424275 (-0.367246) | 0.006715 / 0.007607 (-0.000892) | 0.465801 / 0.226044 (0.239757) | 4.650891 / 2.268929 (2.381962) | 2.425097 / 55.444624 (-53.019527) | 2.134731 / 6.876477 (-4.741745) | 2.312854 / 2.142072 (0.170781) | 0.589668 / 4.805227 (-4.215559) | 0.124673 / 6.500664 (-6.375991) | 0.060887 / 0.075469 (-0.014582) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.243622 / 1.841788 (-0.598166) | 18.501640 / 8.074308 (10.427332) | 13.853099 / 10.191392 (3.661707) | 0.130255 / 0.680424 (-0.550168) | 0.016824 / 0.534201 (-0.517377) | 0.332297 / 0.579283 (-0.246986) | 0.360346 / 0.434364 (-0.074018) | 0.388598 / 0.540337 (-0.151739) | 0.527551 / 1.386936 (-0.859385) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006181 / 0.011353 (-0.005172) | 0.003688 / 0.011008 (-0.007320) | 0.063395 / 0.038508 (0.024887) | 0.062531 / 0.023109 (0.039422) | 0.446565 / 0.275898 (0.170667) | 0.485224 / 0.323480 (0.161744) | 0.004982 / 0.007986 (-0.003004) | 0.002961 / 0.004328 (-0.001367) | 0.063124 / 0.004250 (0.058874) | 0.050234 / 0.037052 (0.013182) | 0.449731 / 0.258489 (0.191242) | 0.487293 / 0.293841 (0.193452) | 0.028528 / 0.128546 (-0.100018) | 0.008210 / 0.075646 (-0.067436) | 0.069520 / 0.419271 (-0.349751) | 0.041026 / 0.043533 (-0.002507) | 0.451370 / 0.255139 (0.196231) | 0.469151 / 0.283200 (0.185951) | 0.021076 / 0.141683 (-0.120607) | 1.439185 / 1.452155 (-0.012970) | 1.492634 / 1.492716 (-0.000082) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.235932 / 0.018006 (0.217926) | 0.430070 / 0.000490 (0.429581) | 0.007347 / 0.000200 (0.007147) | 0.000084 / 0.000054 (0.000029) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026102 / 0.037411 (-0.011309) | 0.081333 / 0.014526 (0.066807) | 0.090111 / 0.176557 (-0.086446) | 0.144578 / 0.737135 (-0.592557) | 0.091961 / 0.296338 (-0.204378) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.455761 / 0.215209 (0.240552) | 4.536345 / 2.077655 (2.458690) | 2.496833 / 1.504120 (0.992713) | 2.323325 / 1.541195 (0.782130) | 2.388364 / 1.468490 (0.919873) | 0.512010 / 4.584777 (-4.072767) | 3.106268 / 3.745712 (-0.639444) | 2.879224 / 5.269862 (-2.390637) | 1.893859 / 4.565676 (-2.671818) | 0.059131 / 0.424275 (-0.365144) | 0.006763 / 0.007607 (-0.000844) | 0.528205 / 0.226044 (0.302161) | 5.296649 / 2.268929 (3.027720) | 2.933787 / 55.444624 (-52.510838) | 2.598258 / 6.876477 (-4.278218) | 2.768195 / 2.142072 (0.626123) | 0.597430 / 4.805227 (-4.207797) | 0.125865 / 6.500664 (-6.374799) | 0.061684 / 0.075469 (-0.013785) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.341194 / 1.841788 (-0.500594) | 18.948225 / 8.074308 (10.873917) | 14.912340 / 10.191392 (4.720948) | 0.146905 / 0.680424 (-0.533519) | 0.017952 / 0.534201 (-0.516249) | 0.332299 / 0.579283 (-0.246984) | 0.362733 / 0.434364 (-0.071631) | 0.388278 / 0.540337 (-0.152060) | 0.546436 / 1.386936 (-0.840500) |

</details>
</details>

![](https://cml.dev/watermark.png#cb4f8357de001df656f2ea7af27625e189c3995b "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008314 / 0.011353 (-0.003038) | 0.004904 / 0.011008 (-0.006105) | 0.097486 / 0.038508 (0.058978) | 0.074627 / 0.023109 (0.051518) | 0.396395 / 0.275898 (0.120497) | 0.440519 / 0.323480 (0.117039) | 0.005964 / 0.007986 (-0.002022) | 0.004203 / 0.004328 (-0.000126) | 0.079998 / 0.004250 (0.075747) | 0.055158 / 0.037052 (0.018106) | 0.415439 / 0.258489 (0.156950) | 0.476101 / 0.293841 (0.182260) | 0.044761 / 0.128546 (-0.083785) | 0.013966 / 0.075646 (-0.061680) | 0.351279 / 0.419271 (-0.067993) | 0.067250 / 0.043533 (0.023717) | 0.414310 / 0.255139 (0.159171) | 0.458104 / 0.283200 (0.174904) | 0.033678 / 0.141683 (-0.108005) | 1.730539 / 1.452155 (0.278385) | 1.840013 / 1.492716 (0.347297) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.272708 / 0.018006 (0.254702) | 0.593563 / 0.000490 (0.593074) | 0.005153 / 0.000200 (0.004953) | 0.000179 / 0.000054 (0.000125) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029595 / 0.037411 (-0.007816) | 0.087994 / 0.014526 (0.073469) | 0.106066 / 0.176557 (-0.070491) | 0.180491 / 0.737135 (-0.556644) | 0.103707 / 0.296338 (-0.192631) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.566711 / 0.215209 (0.351502) | 5.589034 / 2.077655 (3.511380) | 2.364034 / 1.504120 (0.859914) | 2.119050 / 1.541195 (0.577855) | 2.103823 / 1.468490 (0.635333) | 0.819906 / 4.584777 (-3.764871) | 5.178464 / 3.745712 (1.432752) | 4.433986 / 5.269862 (-0.835875) | 2.825470 / 4.565676 (-1.740207) | 0.096907 / 0.424275 (-0.327368) | 0.008573 / 0.007607 (0.000966) | 0.677607 / 0.226044 (0.451563) | 6.811090 / 2.268929 (4.542162) | 3.140923 / 55.444624 (-52.303701) | 2.492251 / 6.876477 (-4.384225) | 2.660231 / 2.142072 (0.518158) | 0.980573 / 4.805227 (-3.824655) | 0.209028 / 6.500664 (-6.291636) | 0.079413 / 0.075469 (0.003944) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.578861 / 1.841788 (-0.262926) | 22.518269 / 8.074308 (14.443961) | 21.335916 / 10.191392 (11.144524) | 0.211311 / 0.680424 (-0.469113) | 0.033216 / 0.534201 (-0.500985) | 0.473266 / 0.579283 (-0.106017) | 0.581650 / 0.434364 (0.147286) | 0.522442 / 0.540337 (-0.017895) | 0.729039 / 1.386936 (-0.657897) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008349 / 0.011353 (-0.003003) | 0.005856 / 0.011008 (-0.005152) | 0.077855 / 0.038508 (0.039347) | 0.080608 / 0.023109 (0.057499) | 0.512533 / 0.275898 (0.236635) | 0.551862 / 0.323480 (0.228382) | 0.007004 / 0.007986 (-0.000982) | 0.004147 / 0.004328 (-0.000181) | 0.086625 / 0.004250 (0.082374) | 0.065962 / 0.037052 (0.028910) | 0.545590 / 0.258489 (0.287101) | 0.586313 / 0.293841 (0.292472) | 0.048719 / 0.128546 (-0.079827) | 0.014997 / 0.075646 (-0.060649) | 0.089510 / 0.419271 (-0.329761) | 0.060936 / 0.043533 (0.017404) | 0.498455 / 0.255139 (0.243316) | 0.535460 / 0.283200 (0.252260) | 0.034624 / 0.141683 (-0.107059) | 1.717401 / 1.452155 (0.265246) | 1.808772 / 1.492716 (0.316056) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.379504 / 0.018006 (0.361497) | 0.601756 / 0.000490 (0.601266) | 0.061740 / 0.000200 (0.061540) | 0.000497 / 0.000054 (0.000442) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031215 / 0.037411 (-0.006196) | 0.097501 / 0.014526 (0.082975) | 0.117434 / 0.176557 (-0.059122) | 0.166014 / 0.737135 (-0.571121) | 0.116466 / 0.296338 (-0.179873) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.699444 / 0.215209 (0.484235) | 6.329332 / 2.077655 (4.251678) | 3.072812 / 1.504120 (1.568693) | 2.729878 / 1.541195 (1.188683) | 2.933785 / 1.468490 (1.465295) | 0.935858 / 4.584777 (-3.648919) | 5.532532 / 3.745712 (1.786820) | 4.677139 / 5.269862 (-0.592722) | 2.963527 / 4.565676 (-1.602149) | 0.099661 / 0.424275 (-0.324614) | 0.009095 / 0.007607 (0.001488) | 0.751158 / 0.226044 (0.525114) | 7.652588 / 2.268929 (5.383660) | 3.802005 / 55.444624 (-51.642619) | 3.163126 / 6.876477 (-3.713351) | 3.401125 / 2.142072 (1.259052) | 0.998627 / 4.805227 (-3.806600) | 0.203310 / 6.500664 (-6.297354) | 0.073827 / 0.075469 (-0.001642) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.662989 / 1.841788 (-0.178799) | 23.777818 / 8.074308 (15.703510) | 20.855378 / 10.191392 (10.663986) | 0.279892 / 0.680424 (-0.400532) | 0.029303 / 0.534201 (-0.504898) | 0.473681 / 0.579283 (-0.105602) | 0.579148 / 0.434364 (0.144784) | 0.546931 / 0.540337 (0.006593) | 0.769740 / 1.386936 (-0.617196) |

</details>
</details>

![](https://cml.dev/watermark.png#63114e9cb78fe02dc145f923dec13d545a8d0327 "CML watermark")