Title: Use yaml instead of get data patterns when possible
Author: lhoestq (MEMBER)
State: closed
Created: 2023-08-17T09:17:05Z

This would make the data files resolution faster: no need to list all the data files to infer the dataset builder to use.

fix https://github.com/huggingface/datasets/issues/6140

Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006829 / 0.011353 (-0.004524) | 0.004535 / 0.011008 (-0.006473) | 0.085255 / 0.038508 (0.046747) | 0.080861 / 0.023109 (0.057752) | 0.366023 / 0.275898 (0.090125) | 0.403095 / 0.323480 (0.079615) | 0.005615 / 0.007986 (-0.002370) | 0.003830 / 0.004328 (-0.000498) | 0.064502 / 0.004250 (0.060251) | 0.053916 / 0.037052 (0.016863) | 0.366010 / 0.258489 (0.107521) | 0.414565 / 0.293841 (0.120724) | 0.031500 / 0.128546 (-0.097046) | 0.009252 / 0.075646 (-0.066394) | 0.289584 / 0.419271 (-0.129688) | 0.052984 / 0.043533 (0.009451) | 0.352626 / 0.255139 (0.097487) | 0.390964 / 0.283200 (0.107764) | 0.025118 / 0.141683 (-0.116565) | 1.462316 / 1.452155 (0.010161) | 1.565682 / 1.492716 (0.072966) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.294432 / 0.018006 (0.276426) | 0.618366 / 0.000490 (0.617876) | 0.003270 / 0.000200 (0.003071) | 0.000081 / 0.000054 (0.000027) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031194 / 0.037411 (-0.006217) | 0.088892 / 0.014526 (0.074366) | 0.102580 / 0.176557 (-0.073977) | 0.159449 / 0.737135 (-0.577686) | 0.104434 / 0.296338 (-0.191905) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.385690 / 0.215209 (0.170481) | 3.832782 / 2.077655 (1.755128) | 1.862521 / 1.504120 (0.358401) | 1.685674 / 1.541195 (0.144479) | 1.724984 / 1.468490 (0.256494) | 0.483700 / 4.584777 (-4.101077) | 3.664154 / 3.745712 (-0.081558) | 3.323023 / 5.269862 (-1.946839) | 2.055958 / 4.565676 (-2.509718) | 0.056990 / 0.424275 (-0.367285) | 0.007674 / 0.007607 (0.000067) | 0.460642 / 0.226044 (0.234598) | 4.609964 / 2.268929 (2.341036) | 2.434868 / 55.444624 (-53.009756) | 2.003347 / 6.876477 (-4.873130) | 2.209520 / 2.142072 (0.067448) | 0.629363 / 4.805227 (-4.175864) | 0.135434 / 6.500664 (-6.365230) | 0.060498 / 0.075469 (-0.014971) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.253917 / 1.841788 (-0.587870) | 19.988953 / 8.074308 (11.914645) | 14.353739 / 10.191392 (4.162347) | 0.165987 / 0.680424 (-0.514437) | 0.018299 / 0.534201 (-0.515902) | 0.395532 / 0.579283 (-0.183751) | 0.418708 / 0.434364 (-0.015656) | 0.460865 / 0.540337 (-0.079472) | 0.633925 / 1.386936 (-0.753011) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006631 / 0.011353 (-0.004722) | 0.004109 / 0.011008 (-0.006899) | 0.065003 / 0.038508 (0.026495) | 0.080407 / 0.023109 (0.057297) | 0.362966 / 0.275898 (0.087068) | 0.389727 / 0.323480 (0.066247) | 0.005588 / 0.007986 (-0.002397) | 0.003517 / 0.004328 (-0.000812) | 0.065821 / 0.004250 (0.061570) | 0.057614 / 0.037052 (0.020561) | 0.367422 / 0.258489 (0.108932) | 0.400706 / 0.293841 (0.106865) | 0.031560 / 0.128546 (-0.096986) | 0.008659 / 0.075646 (-0.066987) | 0.070756 / 0.419271 (-0.348516) | 0.049821 / 0.043533 (0.006288) | 0.360836 / 0.255139 (0.105697) | 0.383981 / 0.283200 (0.100781) | 0.023719 / 0.141683 (-0.117963) | 1.485197 / 1.452155 (0.033043) | 1.544899 / 1.492716 (0.052182) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.336480 / 0.018006 (0.318474) | 0.532839 / 0.000490 (0.532349) | 0.003767 / 0.000200 (0.003567) | 0.000087 / 0.000054 (0.000032) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034132 / 0.037411 (-0.003280) | 0.090131 / 0.014526 (0.075605) | 0.104086 / 0.176557 (-0.072471) | 0.158385 / 0.737135 (-0.578751) | 0.106417 / 0.296338 (-0.189922) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.416462 / 0.215209 (0.201253) | 4.160409 / 2.077655 (2.082755) | 2.195355 / 1.504120 (0.691235) | 2.051234 / 1.541195 (0.510040) | 2.012116 / 1.468490 (0.543626) | 0.477414 / 4.584777 (-4.107363) | 3.590326 / 3.745712 (-0.155386) | 3.318490 / 5.269862 (-1.951371) | 2.064124 / 4.565676 (-2.501553) | 0.057040 / 0.424275 (-0.367235) | 0.007283 / 0.007607 (-0.000324) | 0.480490 / 0.226044 (0.254445) | 4.804013 / 2.268929 (2.535084) | 2.625940 / 55.444624 (-52.818685) | 2.231537 / 6.876477 (-4.644939) | 2.441649 / 2.142072 (0.299576) | 0.573207 / 4.805227 (-4.232020) | 0.131685 / 6.500664 (-6.368979) | 0.060112 / 0.075469 (-0.015357) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.358587 / 1.841788 (-0.483200) | 20.457562 / 8.074308 (12.383254) | 14.236304 / 10.191392 (4.044912) | 0.152860 / 0.680424 (-0.527563) | 0.018466 / 0.534201 (-0.515735) | 0.401391 / 0.579283 (-0.177893) | 0.410252 / 0.434364 (-0.024111) | 0.484335 / 0.540337 (-0.056002) | 0.663818 / 1.386936 (-0.723118) |

</details>
</details>

![](https://cml.dev/watermark.png#acac88873abcb585892dc361eb9f6a70a1fd9a59 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007725 / 0.011353 (-0.003628) | 0.004448 / 0.011008 (-0.006560) | 0.098689 / 0.038508 (0.060180) | 0.082919 / 0.023109 (0.059809) | 0.380707 / 0.275898 (0.104809) | 0.452977 / 0.323480 (0.129497) | 0.004430 / 0.007986 (-0.003555) | 0.003712 / 0.004328 (-0.000616) | 0.076675 / 0.004250 (0.072425) | 0.062281 / 0.037052 (0.025228) | 0.403370 / 0.258489 (0.144881) | 0.464557 / 0.293841 (0.170716) | 0.035646 / 0.128546 (-0.092900) | 0.009776 / 0.075646 (-0.065870) | 0.341955 / 0.419271 (-0.077316) | 0.059515 / 0.043533 (0.015983) | 0.388421 / 0.255139 (0.133282) | 0.439496 / 0.283200 (0.156296) | 0.029090 / 0.141683 (-0.112593) | 1.727473 / 1.452155 (0.275319) | 1.810448 / 1.492716 (0.317732) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.221215 / 0.018006 (0.203208) | 0.486660 / 0.000490 (0.486171) | 0.005467 / 0.000200 (0.005267) | 0.000110 / 0.000054 (0.000056) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032491 / 0.037411 (-0.004920) | 0.094446 / 0.014526 (0.079920) | 0.110339 / 0.176557 (-0.066217) | 0.175004 / 0.737135 (-0.562131) | 0.109209 / 0.296338 (-0.187129) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.453966 / 0.215209 (0.238757) | 4.515842 / 2.077655 (2.438187) | 2.240512 / 1.504120 (0.736392) | 2.059911 / 1.541195 (0.518717) | 2.150635 / 1.468490 (0.682145) | 0.564509 / 4.584777 (-4.020268) | 4.055208 / 3.745712 (0.309496) | 3.614084 / 5.269862 (-1.655778) | 2.295760 / 4.565676 (-2.269917) | 0.066507 / 0.424275 (-0.357768) | 0.008909 / 0.007607 (0.001302) | 0.542604 / 0.226044 (0.316560) | 5.412162 / 2.268929 (3.143233) | 2.758757 / 55.444624 (-52.685867) | 2.430693 / 6.876477 (-4.445784) | 2.669866 / 2.142072 (0.527793) | 0.681756 / 4.805227 (-4.123471) | 0.156524 / 6.500664 (-6.344140) | 0.069499 / 0.075469 (-0.005970) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.571591 / 1.841788 (-0.270197) | 22.543437 / 8.074308 (14.469129) | 16.068426 / 10.191392 (5.877034) | 0.169860 / 0.680424 (-0.510564) | 0.021216 / 0.534201 (-0.512985) | 0.468745 / 0.579283 (-0.110538) | 0.475924 / 0.434364 (0.041560) | 0.535574 / 0.540337 (-0.004763) | 0.733823 / 1.386936 (-0.653113) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008038 / 0.011353 (-0.003315) | 0.004565 / 0.011008 (-0.006443) | 0.076892 / 0.038508 (0.038384) | 0.089559 / 0.023109 (0.066450) | 0.456752 / 0.275898 (0.180854) | 0.497282 / 0.323480 (0.173802) | 0.005991 / 0.007986 (-0.001995) | 0.003784 / 0.004328 (-0.000545) | 0.076339 / 0.004250 (0.072089) | 0.066050 / 0.037052 (0.028998) | 0.462708 / 0.258489 (0.204219) | 0.503711 / 0.293841 (0.209870) | 0.037098 / 0.128546 (-0.091448) | 0.009869 / 0.075646 (-0.065777) | 0.083678 / 0.419271 (-0.335594) | 0.058166 / 0.043533 (0.014633) | 0.461839 / 0.255139 (0.206700) | 0.481546 / 0.283200 (0.198347) | 0.027755 / 0.141683 (-0.113928) | 1.738490 / 1.452155 (0.286335) | 1.832276 / 1.492716 (0.339560) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.329935 / 0.018006 (0.311929) | 0.497438 / 0.000490 (0.496949) | 0.034644 / 0.000200 (0.034444) | 0.000199 / 0.000054 (0.000145) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.035427 / 0.037411 (-0.001984) | 0.105689 / 0.014526 (0.091163) | 0.117706 / 0.176557 (-0.058850) | 0.177862 / 0.737135 (-0.559273) | 0.116791 / 0.296338 (-0.179547) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.484851 / 0.215209 (0.269642) | 4.804346 / 2.077655 (2.726691) | 2.494801 / 1.504120 (0.990681) | 2.320185 / 1.541195 (0.778990) | 2.374090 / 1.468490 (0.905600) | 0.567397 / 4.584777 (-4.017380) | 4.087402 / 3.745712 (0.341690) | 3.794245 / 5.269862 (-1.475616) | 2.378481 / 4.565676 (-2.187195) | 0.068228 / 0.424275 (-0.356047) | 0.008740 / 0.007607 (0.001133) | 0.574876 / 0.226044 (0.348832) | 5.742644 / 2.268929 (3.473716) | 3.047661 / 55.444624 (-52.396963) | 2.729742 / 6.876477 (-4.146735) | 2.852510 / 2.142072 (0.710438) | 0.679450 / 4.805227 (-4.125777) | 0.156162 / 6.500664 (-6.344502) | 0.074051 / 0.075469 (-0.001418) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.576182 / 1.841788 (-0.265605) | 23.298147 / 8.074308 (15.223839) | 16.344621 / 10.191392 (6.153229) | 0.167571 / 0.680424 (-0.512852) | 0.021423 / 0.534201 (-0.512778) | 0.464511 / 0.579283 (-0.114772) | 0.453257 / 0.434364 (0.018893) | 0.563439 / 0.540337 (0.023102) | 0.764759 / 1.386936 (-0.622177) |

</details>
</details>

![](https://cml.dev/watermark.png#e8dc4b32b0d91bdb0971f8203ee37e6588c7770e "CML watermark")


This should also fix https://github.com/huggingface/datasets/issues/6140, so please link it with this PR before merging.

Done !

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006719 / 0.011353 (-0.004634) | 0.004299 / 0.011008 (-0.006709) | 0.085296 / 0.038508 (0.046788) | 0.085144 / 0.023109 (0.062035) | 0.361703 / 0.275898 (0.085805) | 0.397721 / 0.323480 (0.074241) | 0.005920 / 0.007986 (-0.002065) | 0.003853 / 0.004328 (-0.000476) | 0.065633 / 0.004250 (0.061383) | 0.057000 / 0.037052 (0.019947) | 0.379981 / 0.258489 (0.121492) | 0.419041 / 0.293841 (0.125200) | 0.031225 / 0.128546 (-0.097322) | 0.008868 / 0.075646 (-0.066779) | 0.288808 / 0.419271 (-0.130463) | 0.052391 / 0.043533 (0.008859) | 0.362349 / 0.255139 (0.107210) | 0.399858 / 0.283200 (0.116658) | 0.025843 / 0.141683 (-0.115840) | 1.498988 / 1.452155 (0.046834) | 1.547290 / 1.492716 (0.054574) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.278091 / 0.018006 (0.260085) | 0.621794 / 0.000490 (0.621305) | 0.003770 / 0.000200 (0.003570) | 0.000084 / 0.000054 (0.000029) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029128 / 0.037411 (-0.008283) | 0.082061 / 0.014526 (0.067536) | 0.101758 / 0.176557 (-0.074799) | 0.155724 / 0.737135 (-0.581411) | 0.102173 / 0.296338 (-0.194165) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.387145 / 0.215209 (0.171935) | 3.868262 / 2.077655 (1.790607) | 1.886440 / 1.504120 (0.382320) | 1.723305 / 1.541195 (0.182111) | 1.805411 / 1.468490 (0.336921) | 0.485024 / 4.584777 (-4.099753) | 3.637859 / 3.745712 (-0.107853) | 3.319593 / 5.269862 (-1.950269) | 2.087860 / 4.565676 (-2.477817) | 0.056992 / 0.424275 (-0.367283) | 0.007623 / 0.007607 (0.000016) | 0.468182 / 0.226044 (0.242138) | 4.681112 / 2.268929 (2.412183) | 2.407010 / 55.444624 (-53.037614) | 2.026604 / 6.876477 (-4.849872) | 2.298158 / 2.142072 (0.156086) | 0.581839 / 4.805227 (-4.223388) | 0.132101 / 6.500664 (-6.368563) | 0.060472 / 0.075469 (-0.014997) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.236422 / 1.841788 (-0.605365) | 20.505168 / 8.074308 (12.430860) | 14.356081 / 10.191392 (4.164689) | 0.148808 / 0.680424 (-0.531616) | 0.018433 / 0.534201 (-0.515768) | 0.391323 / 0.579283 (-0.187960) | 0.413142 / 0.434364 (-0.021222) | 0.453484 / 0.540337 (-0.086853) | 0.620771 / 1.386936 (-0.766165) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007030 / 0.011353 (-0.004323) | 0.004430 / 0.011008 (-0.006578) | 0.065578 / 0.038508 (0.027070) | 0.090751 / 0.023109 (0.067642) | 0.389121 / 0.275898 (0.113223) | 0.424657 / 0.323480 (0.101177) | 0.006575 / 0.007986 (-0.001410) | 0.003855 / 0.004328 (-0.000473) | 0.066175 / 0.004250 (0.061925) | 0.063255 / 0.037052 (0.026202) | 0.397161 / 0.258489 (0.138672) | 0.435291 / 0.293841 (0.141450) | 0.031622 / 0.128546 (-0.096925) | 0.008900 / 0.075646 (-0.066747) | 0.071694 / 0.419271 (-0.347577) | 0.049161 / 0.043533 (0.005628) | 0.386214 / 0.255139 (0.131075) | 0.404571 / 0.283200 (0.121372) | 0.024821 / 0.141683 (-0.116862) | 1.489514 / 1.452155 (0.037359) | 1.576139 / 1.492716 (0.083423) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.289884 / 0.018006 (0.271878) | 0.629342 / 0.000490 (0.628852) | 0.004799 / 0.000200 (0.004599) | 0.000160 / 0.000054 (0.000106) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032081 / 0.037411 (-0.005331) | 0.088152 / 0.014526 (0.073626) | 0.107289 / 0.176557 (-0.069267) | 0.164598 / 0.737135 (-0.572537) | 0.108395 / 0.296338 (-0.187944) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.426723 / 0.215209 (0.211514) | 4.267719 / 2.077655 (2.190064) | 2.289657 / 1.504120 (0.785537) | 2.117435 / 1.541195 (0.576240) | 2.187292 / 1.468490 (0.718802) | 0.478387 / 4.584777 (-4.106390) | 3.625096 / 3.745712 (-0.120616) | 3.408036 / 5.269862 (-1.861826) | 2.124117 / 4.565676 (-2.441559) | 0.056537 / 0.424275 (-0.367738) | 0.007489 / 0.007607 (-0.000118) | 0.502434 / 0.226044 (0.276389) | 5.025357 / 2.268929 (2.756428) | 2.740554 / 55.444624 (-52.704070) | 2.418841 / 6.876477 (-4.457635) | 2.730764 / 2.142072 (0.588691) | 0.600013 / 4.805227 (-4.205214) | 0.133039 / 6.500664 (-6.367625) | 0.061466 / 0.075469 (-0.014003) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.330211 / 1.841788 (-0.511577) | 21.092100 / 8.074308 (13.017792) | 14.463054 / 10.191392 (4.271662) | 0.154149 / 0.680424 (-0.526274) | 0.018891 / 0.534201 (-0.515310) | 0.393078 / 0.579283 (-0.186205) | 0.415279 / 0.434364 (-0.019085) | 0.479469 / 0.540337 (-0.060868) | 0.659953 / 1.386936 (-0.726983) |

</details>
</details>

![](https://cml.dev/watermark.png#5ca2ba050340829b4dd44791afc15db0d82a3276 "CML watermark")