Title: Raise an error on missing distributed seed
Author: lhoestq (MEMBER)
State: closed
Created: 2023-04-03T10:44:58Z

close https://github.com/huggingface/datasets/issues/5696

Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009644 / 0.011353 (-0.001709) | 0.006407 / 0.011008 (-0.004601) | 0.148353 / 0.038508 (0.109845) | 0.037537 / 0.023109 (0.014428) | 0.379697 / 0.275898 (0.103799) | 0.466260 / 0.323480 (0.142780) | 0.007884 / 0.007986 (-0.000102) | 0.005140 / 0.004328 (0.000812) | 0.111078 / 0.004250 (0.106827) | 0.049429 / 0.037052 (0.012377) | 0.364766 / 0.258489 (0.106277) | 0.453809 / 0.293841 (0.159968) | 0.051918 / 0.128546 (-0.076628) | 0.020081 / 0.075646 (-0.055566) | 0.616041 / 0.419271 (0.196770) | 0.059834 / 0.043533 (0.016301) | 0.373104 / 0.255139 (0.117965) | 0.419304 / 0.283200 (0.136104) | 0.113526 / 0.141683 (-0.028156) | 1.827160 / 1.452155 (0.375006) | 1.912092 / 1.492716 (0.419376) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.269584 / 0.018006 (0.251578) | 0.554100 / 0.000490 (0.553610) | 0.006618 / 0.000200 (0.006418) | 0.000093 / 0.000054 (0.000039) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025280 / 0.037411 (-0.012131) | 0.123116 / 0.014526 (0.108591) | 0.127674 / 0.176557 (-0.048883) | 0.189106 / 0.737135 (-0.548030) | 0.142072 / 0.296338 (-0.154267) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.602201 / 0.215209 (0.386992) | 5.959610 / 2.077655 (3.881956) | 2.404856 / 1.504120 (0.900736) | 2.175017 / 1.541195 (0.633823) | 2.154360 / 1.468490 (0.685870) | 1.265339 / 4.584777 (-3.319438) | 5.598429 / 3.745712 (1.852716) | 5.130249 / 5.269862 (-0.139612) | 2.764922 / 4.565676 (-1.800754) | 0.143232 / 0.424275 (-0.281043) | 0.014721 / 0.007607 (0.007114) | 0.764734 / 0.226044 (0.538689) | 7.518810 / 2.268929 (5.249882) | 3.344734 / 55.444624 (-52.099890) | 2.601158 / 6.876477 (-4.275319) | 2.726018 / 2.142072 (0.583945) | 1.397918 / 4.805227 (-3.407309) | 0.253277 / 6.500664 (-6.247387) | 0.077772 / 0.075469 (0.002303) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.499535 / 1.841788 (-0.342253) | 17.782490 / 8.074308 (9.708182) | 21.953064 / 10.191392 (11.761672) | 0.248753 / 0.680424 (-0.431671) | 0.029194 / 0.534201 (-0.505007) | 0.529700 / 0.579283 (-0.049583) | 0.618412 / 0.434364 (0.184048) | 0.605062 / 0.540337 (0.064725) | 0.725661 / 1.386936 (-0.661275) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009489 / 0.011353 (-0.001864) | 0.006423 / 0.011008 (-0.004585) | 0.096789 / 0.038508 (0.058281) | 0.034639 / 0.023109 (0.011530) | 0.403875 / 0.275898 (0.127977) | 0.439368 / 0.323480 (0.115888) | 0.006354 / 0.007986 (-0.001631) | 0.006794 / 0.004328 (0.002466) | 0.095537 / 0.004250 (0.091287) | 0.047749 / 0.037052 (0.010697) | 0.424157 / 0.258489 (0.165668) | 0.487825 / 0.293841 (0.193984) | 0.054675 / 0.128546 (-0.073872) | 0.021349 / 0.075646 (-0.054297) | 0.108917 / 0.419271 (-0.310354) | 0.075891 / 0.043533 (0.032358) | 0.412889 / 0.255139 (0.157750) | 0.464512 / 0.283200 (0.181312) | 0.118832 / 0.141683 (-0.022850) | 1.721215 / 1.452155 (0.269060) | 1.857195 / 1.492716 (0.364478) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.248308 / 0.018006 (0.230302) | 0.559496 / 0.000490 (0.559006) | 0.007136 / 0.000200 (0.006936) | 0.000160 / 0.000054 (0.000106) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031772 / 0.037411 (-0.005639) | 0.123565 / 0.014526 (0.109039) | 0.132660 / 0.176557 (-0.043896) | 0.201428 / 0.737135 (-0.535707) | 0.135238 / 0.296338 (-0.161101) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.646978 / 0.215209 (0.431769) | 6.183477 / 2.077655 (4.105822) | 2.782117 / 1.504120 (1.277997) | 2.294093 / 1.541195 (0.752898) | 2.346932 / 1.468490 (0.878442) | 1.239085 / 4.584777 (-3.345692) | 5.696364 / 3.745712 (1.950652) | 4.980102 / 5.269862 (-0.289759) | 2.278116 / 4.565676 (-2.287560) | 0.157339 / 0.424275 (-0.266936) | 0.014936 / 0.007607 (0.007329) | 0.778001 / 0.226044 (0.551957) | 7.708066 / 2.268929 (5.439138) | 3.412235 / 55.444624 (-52.032389) | 2.670670 / 6.876477 (-4.205806) | 2.731802 / 2.142072 (0.589730) | 1.446516 / 4.805227 (-3.358712) | 0.263689 / 6.500664 (-6.236975) | 0.086359 / 0.075469 (0.010890) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.573169 / 1.841788 (-0.268619) | 17.690842 / 8.074308 (9.616534) | 20.343336 / 10.191392 (10.151944) | 0.231028 / 0.680424 (-0.449396) | 0.025954 / 0.534201 (-0.508247) | 0.570554 / 0.579283 (-0.008729) | 0.610453 / 0.434364 (0.176089) | 0.675830 / 0.540337 (0.135493) | 0.790650 / 1.386936 (-0.596286) |

</details>
</details>

![](https://cml.dev/watermark.png#d094ed07823bfb3271f3a9006daa1f92a64967a5 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007553 / 0.011353 (-0.003800) | 0.005426 / 0.011008 (-0.005582) | 0.096550 / 0.038508 (0.058042) | 0.034393 / 0.023109 (0.011284) | 0.322297 / 0.275898 (0.046399) | 0.340943 / 0.323480 (0.017463) | 0.006350 / 0.007986 (-0.001635) | 0.005700 / 0.004328 (0.001372) | 0.074929 / 0.004250 (0.070678) | 0.054819 / 0.037052 (0.017767) | 0.320151 / 0.258489 (0.061662) | 0.346957 / 0.293841 (0.053116) | 0.036659 / 0.128546 (-0.091887) | 0.012443 / 0.075646 (-0.063204) | 0.332232 / 0.419271 (-0.087040) | 0.051467 / 0.043533 (0.007934) | 0.310952 / 0.255139 (0.055813) | 0.325617 / 0.283200 (0.042417) | 0.104908 / 0.141683 (-0.036775) | 1.446752 / 1.452155 (-0.005403) | 1.558773 / 1.492716 (0.066056) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.300639 / 0.018006 (0.282633) | 0.499901 / 0.000490 (0.499411) | 0.007340 / 0.000200 (0.007140) | 0.000255 / 0.000054 (0.000201) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027206 / 0.037411 (-0.010206) | 0.105603 / 0.014526 (0.091077) | 0.118669 / 0.176557 (-0.057887) | 0.174050 / 0.737135 (-0.563086) | 0.125099 / 0.296338 (-0.171239) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.404285 / 0.215209 (0.189076) | 4.034587 / 2.077655 (1.956933) | 1.812639 / 1.504120 (0.308519) | 1.625745 / 1.541195 (0.084551) | 1.735523 / 1.468490 (0.267033) | 0.709699 / 4.584777 (-3.875078) | 3.802196 / 3.745712 (0.056484) | 3.656984 / 5.269862 (-1.612877) | 1.968470 / 4.565676 (-2.597206) | 0.086612 / 0.424275 (-0.337663) | 0.012368 / 0.007607 (0.004761) | 0.502622 / 0.226044 (0.276577) | 5.017876 / 2.268929 (2.748948) | 2.279794 / 55.444624 (-53.164831) | 1.956938 / 6.876477 (-4.919538) | 2.150430 / 2.142072 (0.008357) | 0.847691 / 4.805227 (-3.957536) | 0.170157 / 6.500664 (-6.330507) | 0.064141 / 0.075469 (-0.011328) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.172246 / 1.841788 (-0.669542) | 15.229444 / 8.074308 (7.155136) | 14.715913 / 10.191392 (4.524521) | 0.192501 / 0.680424 (-0.487923) | 0.017972 / 0.534201 (-0.516229) | 0.423834 / 0.579283 (-0.155449) | 0.423019 / 0.434364 (-0.011345) | 0.493298 / 0.540337 (-0.047039) | 0.589833 / 1.386936 (-0.797103) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007773 / 0.011353 (-0.003580) | 0.005449 / 0.011008 (-0.005560) | 0.075180 / 0.038508 (0.036672) | 0.035221 / 0.023109 (0.012111) | 0.338169 / 0.275898 (0.062271) | 0.374002 / 0.323480 (0.050522) | 0.006391 / 0.007986 (-0.001595) | 0.004406 / 0.004328 (0.000078) | 0.074925 / 0.004250 (0.070675) | 0.056527 / 0.037052 (0.019475) | 0.338071 / 0.258489 (0.079582) | 0.391882 / 0.293841 (0.098041) | 0.037241 / 0.128546 (-0.091305) | 0.012546 / 0.075646 (-0.063100) | 0.087331 / 0.419271 (-0.331940) | 0.049851 / 0.043533 (0.006318) | 0.335264 / 0.255139 (0.080125) | 0.354813 / 0.283200 (0.071614) | 0.110614 / 0.141683 (-0.031069) | 1.432782 / 1.452155 (-0.019372) | 1.548800 / 1.492716 (0.056083) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.307892 / 0.018006 (0.289886) | 0.518809 / 0.000490 (0.518319) | 0.004058 / 0.000200 (0.003858) | 0.000099 / 0.000054 (0.000045) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029155 / 0.037411 (-0.008256) | 0.111706 / 0.014526 (0.097180) | 0.122964 / 0.176557 (-0.053592) | 0.170939 / 0.737135 (-0.566196) | 0.128538 / 0.296338 (-0.167801) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.426529 / 0.215209 (0.211320) | 4.254218 / 2.077655 (2.176563) | 2.011455 / 1.504120 (0.507335) | 1.817397 / 1.541195 (0.276202) | 1.952915 / 1.468490 (0.484425) | 0.705052 / 4.584777 (-3.879725) | 3.844458 / 3.745712 (0.098746) | 3.592754 / 5.269862 (-1.677107) | 1.573567 / 4.565676 (-2.992109) | 0.086834 / 0.424275 (-0.337441) | 0.012389 / 0.007607 (0.004782) | 0.541695 / 0.226044 (0.315650) | 5.224492 / 2.268929 (2.955564) | 2.473648 / 55.444624 (-52.970976) | 2.167458 / 6.876477 (-4.709019) | 2.253319 / 2.142072 (0.111246) | 0.836322 / 4.805227 (-3.968905) | 0.168680 / 6.500664 (-6.331984) | 0.065699 / 0.075469 (-0.009770) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.281886 / 1.841788 (-0.559902) | 15.451741 / 8.074308 (7.377433) | 14.906870 / 10.191392 (4.715478) | 0.168554 / 0.680424 (-0.511870) | 0.017365 / 0.534201 (-0.516836) | 0.434183 / 0.579283 (-0.145100) | 0.421891 / 0.434364 (-0.012473) | 0.538993 / 0.540337 (-0.001344) | 0.636212 / 1.386936 (-0.750724) |

</details>
</details>

![](https://cml.dev/watermark.png#1f428b8172319a6bfe95d7a4356b1d14a8d386d8 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007362 / 0.011353 (-0.003991) | 0.004992 / 0.011008 (-0.006016) | 0.098730 / 0.038508 (0.060222) | 0.033673 / 0.023109 (0.010563) | 0.296334 / 0.275898 (0.020436) | 0.328208 / 0.323480 (0.004728) | 0.005658 / 0.007986 (-0.002327) | 0.004130 / 0.004328 (-0.000199) | 0.074596 / 0.004250 (0.070346) | 0.048230 / 0.037052 (0.011178) | 0.295631 / 0.258489 (0.037142) | 0.347176 / 0.293841 (0.053335) | 0.036359 / 0.128546 (-0.092187) | 0.011889 / 0.075646 (-0.063758) | 0.332889 / 0.419271 (-0.086382) | 0.049708 / 0.043533 (0.006175) | 0.291207 / 0.255139 (0.036068) | 0.311066 / 0.283200 (0.027867) | 0.098418 / 0.141683 (-0.043265) | 1.415450 / 1.452155 (-0.036705) | 1.526928 / 1.492716 (0.034212) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.212636 / 0.018006 (0.194630) | 0.432337 / 0.000490 (0.431847) | 0.006839 / 0.000200 (0.006639) | 0.000205 / 0.000054 (0.000150) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026045 / 0.037411 (-0.011366) | 0.107427 / 0.014526 (0.092901) | 0.114634 / 0.176557 (-0.061922) | 0.169943 / 0.737135 (-0.567192) | 0.123290 / 0.296338 (-0.173048) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.409432 / 0.215209 (0.194223) | 4.097910 / 2.077655 (2.020255) | 1.857177 / 1.504120 (0.353057) | 1.672355 / 1.541195 (0.131160) | 1.740130 / 1.468490 (0.271640) | 0.706520 / 4.584777 (-3.878257) | 3.773606 / 3.745712 (0.027893) | 2.101635 / 5.269862 (-3.168226) | 1.326295 / 4.565676 (-3.239382) | 0.085672 / 0.424275 (-0.338604) | 0.012142 / 0.007607 (0.004534) | 0.501168 / 0.226044 (0.275123) | 5.049784 / 2.268929 (2.780855) | 2.322477 / 55.444624 (-53.122148) | 1.990105 / 6.876477 (-4.886372) | 2.115003 / 2.142072 (-0.027070) | 0.837518 / 4.805227 (-3.967709) | 0.168457 / 6.500664 (-6.332207) | 0.064622 / 0.075469 (-0.010847) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.188152 / 1.841788 (-0.653635) | 14.991585 / 8.074308 (6.917276) | 14.635187 / 10.191392 (4.443795) | 0.183708 / 0.680424 (-0.496716) | 0.017452 / 0.534201 (-0.516749) | 0.418963 / 0.579283 (-0.160320) | 0.428893 / 0.434364 (-0.005471) | 0.502108 / 0.540337 (-0.038229) | 0.596345 / 1.386936 (-0.790591) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007404 / 0.011353 (-0.003949) | 0.005148 / 0.011008 (-0.005860) | 0.074785 / 0.038508 (0.036277) | 0.033815 / 0.023109 (0.010706) | 0.332752 / 0.275898 (0.056854) | 0.368018 / 0.323480 (0.044538) | 0.005642 / 0.007986 (-0.002344) | 0.004041 / 0.004328 (-0.000287) | 0.073455 / 0.004250 (0.069205) | 0.047380 / 0.037052 (0.010328) | 0.337017 / 0.258489 (0.078528) | 0.384185 / 0.293841 (0.090344) | 0.036592 / 0.128546 (-0.091954) | 0.012109 / 0.075646 (-0.063537) | 0.086862 / 0.419271 (-0.332410) | 0.049030 / 0.043533 (0.005497) | 0.336542 / 0.255139 (0.081403) | 0.350295 / 0.283200 (0.067096) | 0.100998 / 0.141683 (-0.040685) | 1.469749 / 1.452155 (0.017594) | 1.588355 / 1.492716 (0.095639) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.227552 / 0.018006 (0.209546) | 0.438087 / 0.000490 (0.437598) | 0.000394 / 0.000200 (0.000194) | 0.000058 / 0.000054 (0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030575 / 0.037411 (-0.006836) | 0.111914 / 0.014526 (0.097388) | 0.124583 / 0.176557 (-0.051973) | 0.175471 / 0.737135 (-0.561665) | 0.129535 / 0.296338 (-0.166803) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.425625 / 0.215209 (0.210416) | 4.228328 / 2.077655 (2.150673) | 2.021087 / 1.504120 (0.516967) | 1.832550 / 1.541195 (0.291355) | 1.925572 / 1.468490 (0.457082) | 0.690772 / 4.584777 (-3.894005) | 3.724900 / 3.745712 (-0.020813) | 2.080286 / 5.269862 (-3.189576) | 1.316854 / 4.565676 (-3.248822) | 0.085123 / 0.424275 (-0.339152) | 0.012078 / 0.007607 (0.004471) | 0.525802 / 0.226044 (0.299758) | 5.242598 / 2.268929 (2.973670) | 2.491596 / 55.444624 (-52.953028) | 2.125156 / 6.876477 (-4.751320) | 2.185922 / 2.142072 (0.043850) | 0.823116 / 4.805227 (-3.982111) | 0.165188 / 6.500664 (-6.335476) | 0.063970 / 0.075469 (-0.011499) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.256948 / 1.841788 (-0.584840) | 14.981990 / 8.074308 (6.907682) | 14.565266 / 10.191392 (4.373874) | 0.175064 / 0.680424 (-0.505360) | 0.017628 / 0.534201 (-0.516573) | 0.429979 / 0.579283 (-0.149304) | 0.422509 / 0.434364 (-0.011855) | 0.546262 / 0.540337 (0.005924) | 0.647103 / 1.386936 (-0.739833) |

</details>
</details>

![](https://cml.dev/watermark.png#0803a006db1c395ac715662cc6079651f77c11ea "CML watermark")