Title: Raise FileNotFoundError when passing data_files that don't exist
Author: lhoestq (MEMBER)
State: closed
Created: 2023-08-17T09:49:48Z

e.g. when running `load_dataset("parquet", data_files="doesnt_exist.parquet")`

Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009288 / 0.011353 (-0.002065) | 0.005950 / 0.011008 (-0.005058) | 0.122376 / 0.038508 (0.083868) | 0.093177 / 0.023109 (0.070068) | 0.448517 / 0.275898 (0.172619) | 0.474999 / 0.323480 (0.151520) | 0.005133 / 0.007986 (-0.002853) | 0.005123 / 0.004328 (0.000795) | 0.085479 / 0.004250 (0.081229) | 0.065613 / 0.037052 (0.028561) | 0.451179 / 0.258489 (0.192690) | 0.516876 / 0.293841 (0.223036) | 0.047536 / 0.128546 (-0.081010) | 0.013894 / 0.075646 (-0.061752) | 0.382149 / 0.419271 (-0.037122) | 0.067380 / 0.043533 (0.023848) | 0.419282 / 0.255139 (0.164143) | 0.482042 / 0.283200 (0.198842) | 0.041230 / 0.141683 (-0.100452) | 1.818127 / 1.452155 (0.365972) | 1.938123 / 1.492716 (0.445406) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.271824 / 0.018006 (0.253817) | 0.604933 / 0.000490 (0.604443) | 0.004953 / 0.000200 (0.004753) | 0.000173 / 0.000054 (0.000119) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.036682 / 0.037411 (-0.000729) | 0.095604 / 0.014526 (0.081078) | 0.116862 / 0.176557 (-0.059695) | 0.191335 / 0.737135 (-0.545800) | 0.116620 / 0.296338 (-0.179718) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.620735 / 0.215209 (0.405526) | 6.157119 / 2.077655 (4.079465) | 2.848548 / 1.504120 (1.344428) | 2.493731 / 1.541195 (0.952536) | 2.505801 / 1.468490 (1.037311) | 0.837315 / 4.584777 (-3.747462) | 5.360653 / 3.745712 (1.614941) | 4.908863 / 5.269862 (-0.360999) | 3.184672 / 4.565676 (-1.381004) | 0.105687 / 0.424275 (-0.318588) | 0.011350 / 0.007607 (0.003743) | 0.745729 / 0.226044 (0.519684) | 7.431584 / 2.268929 (5.162655) | 3.644670 / 55.444624 (-51.799954) | 2.910159 / 6.876477 (-3.966317) | 3.257137 / 2.142072 (1.115065) | 1.041377 / 4.805227 (-3.763851) | 0.213289 / 6.500664 (-6.287375) | 0.089208 / 0.075469 (0.013739) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.727274 / 1.841788 (-0.114513) | 25.448436 / 8.074308 (17.374128) | 23.016108 / 10.191392 (12.824716) | 0.219454 / 0.680424 (-0.460970) | 0.028531 / 0.534201 (-0.505670) | 0.500231 / 0.579283 (-0.079052) | 0.614631 / 0.434364 (0.180267) | 0.557926 / 0.540337 (0.017588) | 0.786261 / 1.386936 (-0.600675) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008608 / 0.011353 (-0.002745) | 0.006185 / 0.011008 (-0.004823) | 0.089258 / 0.038508 (0.050750) | 0.090109 / 0.023109 (0.067000) | 0.522200 / 0.275898 (0.246302) | 0.559218 / 0.323480 (0.235738) | 0.008983 / 0.007986 (0.000997) | 0.004488 / 0.004328 (0.000159) | 0.083658 / 0.004250 (0.079408) | 0.064962 / 0.037052 (0.027909) | 0.519477 / 0.258489 (0.260988) | 0.573842 / 0.293841 (0.280001) | 0.053984 / 0.128546 (-0.074562) | 0.014665 / 0.075646 (-0.060982) | 0.089438 / 0.419271 (-0.329834) | 0.065756 / 0.043533 (0.022223) | 0.525131 / 0.255139 (0.269992) | 0.568934 / 0.283200 (0.285734) | 0.037308 / 0.141683 (-0.104375) | 1.928790 / 1.452155 (0.476635) | 2.027926 / 1.492716 (0.535209) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.309595 / 0.018006 (0.291588) | 0.615675 / 0.000490 (0.615186) | 0.004869 / 0.000200 (0.004669) | 0.000116 / 0.000054 (0.000061) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033306 / 0.037411 (-0.004105) | 0.104429 / 0.014526 (0.089904) | 0.116989 / 0.176557 (-0.059568) | 0.183638 / 0.737135 (-0.553497) | 0.132624 / 0.296338 (-0.163714) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.644511 / 0.215209 (0.429302) | 6.425544 / 2.077655 (4.347889) | 3.079071 / 1.504120 (1.574951) | 2.720963 / 1.541195 (1.179769) | 2.835607 / 1.468490 (1.367117) | 0.863561 / 4.584777 (-3.721216) | 5.333462 / 3.745712 (1.587750) | 4.843183 / 5.269862 (-0.426678) | 3.106858 / 4.565676 (-1.458819) | 0.106790 / 0.424275 (-0.317485) | 0.008829 / 0.007607 (0.001222) | 0.759003 / 0.226044 (0.532958) | 7.771247 / 2.268929 (5.502318) | 3.896844 / 55.444624 (-51.547780) | 3.246671 / 6.876477 (-3.629806) | 3.486167 / 2.142072 (1.344094) | 1.071290 / 4.805227 (-3.733937) | 0.217972 / 6.500664 (-6.282692) | 0.089848 / 0.075469 (0.014379) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.816048 / 1.841788 (-0.025739) | 25.625084 / 8.074308 (17.550776) | 24.490882 / 10.191392 (14.299490) | 0.242356 / 0.680424 (-0.438067) | 0.027886 / 0.534201 (-0.506315) | 0.496997 / 0.579283 (-0.082286) | 0.613815 / 0.434364 (0.179451) | 0.607132 / 0.540337 (0.066795) | 0.833051 / 1.386936 (-0.553885) |

</details>
</details>

![](https://cml.dev/watermark.png#0adfa9ada14c38fce5973b5e3f196a2c46dc9170 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.011580 / 0.011353 (0.000227) | 0.004199 / 0.011008 (-0.006809) | 0.084055 / 0.038508 (0.045547) | 0.096824 / 0.023109 (0.073715) | 0.308755 / 0.275898 (0.032857) | 0.341717 / 0.323480 (0.018237) | 0.006018 / 0.007986 (-0.001968) | 0.003597 / 0.004328 (-0.000731) | 0.064953 / 0.004250 (0.060702) | 0.059577 / 0.037052 (0.022525) | 0.316292 / 0.258489 (0.057803) | 0.358991 / 0.293841 (0.065150) | 0.033925 / 0.128546 (-0.094621) | 0.008828 / 0.075646 (-0.066818) | 0.288673 / 0.419271 (-0.130599) | 0.055494 / 0.043533 (0.011961) | 0.311181 / 0.255139 (0.056042) | 0.345220 / 0.283200 (0.062021) | 0.024033 / 0.141683 (-0.117649) | 1.504709 / 1.452155 (0.052554) | 1.587920 / 1.492716 (0.095204) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.301099 / 0.018006 (0.283093) | 0.594497 / 0.000490 (0.594007) | 0.006244 / 0.000200 (0.006044) | 0.000228 / 0.000054 (0.000174) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027663 / 0.037411 (-0.009748) | 0.081767 / 0.014526 (0.067241) | 0.097342 / 0.176557 (-0.079215) | 0.153200 / 0.737135 (-0.583935) | 0.097474 / 0.296338 (-0.198864) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.405929 / 0.215209 (0.190719) | 4.045398 / 2.077655 (1.967743) | 2.044669 / 1.504120 (0.540549) | 1.872889 / 1.541195 (0.331694) | 1.911901 / 1.468490 (0.443411) | 0.480939 / 4.584777 (-4.103838) | 3.652833 / 3.745712 (-0.092879) | 3.281659 / 5.269862 (-1.988202) | 2.038023 / 4.565676 (-2.527654) | 0.056501 / 0.424275 (-0.367775) | 0.007571 / 0.007607 (-0.000036) | 0.481053 / 0.226044 (0.255009) | 4.802048 / 2.268929 (2.533119) | 2.560479 / 55.444624 (-52.884145) | 2.164852 / 6.876477 (-4.711625) | 2.374595 / 2.142072 (0.232523) | 0.576309 / 4.805227 (-4.228918) | 0.134831 / 6.500664 (-6.365833) | 0.060649 / 0.075469 (-0.014820) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.254210 / 1.841788 (-0.587578) | 19.826143 / 8.074308 (11.751835) | 14.446391 / 10.191392 (4.254999) | 0.165707 / 0.680424 (-0.514717) | 0.018221 / 0.534201 (-0.515980) | 0.395996 / 0.579283 (-0.183287) | 0.424567 / 0.434364 (-0.009796) | 0.459836 / 0.540337 (-0.080501) | 0.635969 / 1.386936 (-0.750967) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006696 / 0.011353 (-0.004657) | 0.004131 / 0.011008 (-0.006877) | 0.064587 / 0.038508 (0.026079) | 0.079189 / 0.023109 (0.056080) | 0.359977 / 0.275898 (0.084079) | 0.389331 / 0.323480 (0.065851) | 0.005502 / 0.007986 (-0.002483) | 0.003492 / 0.004328 (-0.000837) | 0.064967 / 0.004250 (0.060716) | 0.055953 / 0.037052 (0.018901) | 0.363997 / 0.258489 (0.105508) | 0.398405 / 0.293841 (0.104564) | 0.031292 / 0.128546 (-0.097254) | 0.008693 / 0.075646 (-0.066953) | 0.070451 / 0.419271 (-0.348820) | 0.048965 / 0.043533 (0.005432) | 0.358288 / 0.255139 (0.103149) | 0.379136 / 0.283200 (0.095936) | 0.024364 / 0.141683 (-0.117319) | 1.478998 / 1.452155 (0.026843) | 1.547282 / 1.492716 (0.054566) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.328188 / 0.018006 (0.310182) | 0.525968 / 0.000490 (0.525478) | 0.003782 / 0.000200 (0.003582) | 0.000089 / 0.000054 (0.000034) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032528 / 0.037411 (-0.004883) | 0.087685 / 0.014526 (0.073159) | 0.100684 / 0.176557 (-0.075872) | 0.155944 / 0.737135 (-0.581192) | 0.101949 / 0.296338 (-0.194389) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.418591 / 0.215209 (0.203382) | 4.199235 / 2.077655 (2.121580) | 2.183880 / 1.504120 (0.679760) | 2.024502 / 1.541195 (0.483307) | 2.017435 / 1.468490 (0.548945) | 0.488881 / 4.584777 (-4.095896) | 3.635002 / 3.745712 (-0.110710) | 3.359992 / 5.269862 (-1.909870) | 2.089686 / 4.565676 (-2.475991) | 0.057813 / 0.424275 (-0.366462) | 0.007349 / 0.007607 (-0.000258) | 0.490719 / 0.226044 (0.264674) | 4.859950 / 2.268929 (2.591022) | 2.616711 / 55.444624 (-52.827914) | 2.238671 / 6.876477 (-4.637806) | 2.442262 / 2.142072 (0.300190) | 0.598368 / 4.805227 (-4.206859) | 0.135281 / 6.500664 (-6.365383) | 0.063072 / 0.075469 (-0.012397) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.356396 / 1.841788 (-0.485392) | 20.075123 / 8.074308 (12.000815) | 14.191317 / 10.191392 (3.999925) | 0.167691 / 0.680424 (-0.512732) | 0.018290 / 0.534201 (-0.515911) | 0.392881 / 0.579283 (-0.186402) | 0.413665 / 0.434364 (-0.020699) | 0.480766 / 0.540337 (-0.059571) | 0.655625 / 1.386936 (-0.731311) |

</details>
</details>

![](https://cml.dev/watermark.png#a46ca9cc138754629be261522301e725c7d14152 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007834 / 0.011353 (-0.003519) | 0.004744 / 0.011008 (-0.006264) | 0.102061 / 0.038508 (0.063553) | 0.089246 / 0.023109 (0.066137) | 0.399936 / 0.275898 (0.124038) | 0.436974 / 0.323480 (0.113494) | 0.004791 / 0.007986 (-0.003195) | 0.005976 / 0.004328 (0.001647) | 0.079336 / 0.004250 (0.075086) | 0.065947 / 0.037052 (0.028894) | 0.403747 / 0.258489 (0.145258) | 0.460249 / 0.293841 (0.166408) | 0.038065 / 0.128546 (-0.090482) | 0.010179 / 0.075646 (-0.065467) | 0.403620 / 0.419271 (-0.015652) | 0.066439 / 0.043533 (0.022906) | 0.412123 / 0.255139 (0.156984) | 0.452121 / 0.283200 (0.168921) | 0.033533 / 0.141683 (-0.108150) | 1.858650 / 1.452155 (0.406495) | 1.916248 / 1.492716 (0.423532) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.237180 / 0.018006 (0.219174) | 0.526844 / 0.000490 (0.526354) | 0.004220 / 0.000200 (0.004020) | 0.000123 / 0.000054 (0.000069) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033860 / 0.037411 (-0.003552) | 0.105054 / 0.014526 (0.090528) | 0.116494 / 0.176557 (-0.060063) | 0.185990 / 0.737135 (-0.551145) | 0.119072 / 0.296338 (-0.177266) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.488549 / 0.215209 (0.273340) | 4.884950 / 2.077655 (2.807295) | 2.521819 / 1.504120 (1.017699) | 2.329382 / 1.541195 (0.788188) | 2.413710 / 1.468490 (0.945220) | 0.568325 / 4.584777 (-4.016452) | 4.243505 / 3.745712 (0.497793) | 3.785983 / 5.269862 (-1.483879) | 2.387146 / 4.565676 (-2.178531) | 0.067176 / 0.424275 (-0.357099) | 0.009145 / 0.007607 (0.001538) | 0.571482 / 0.226044 (0.345437) | 5.688822 / 2.268929 (3.419894) | 3.067346 / 55.444624 (-52.377278) | 2.688723 / 6.876477 (-4.187754) | 2.883785 / 2.142072 (0.741713) | 0.679326 / 4.805227 (-4.125901) | 0.156018 / 6.500664 (-6.344646) | 0.070947 / 0.075469 (-0.004522) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.556611 / 1.841788 (-0.285177) | 23.545074 / 8.074308 (15.470766) | 17.125108 / 10.191392 (6.933716) | 0.180180 / 0.680424 (-0.500244) | 0.021420 / 0.534201 (-0.512781) | 0.466888 / 0.579283 (-0.112395) | 0.485746 / 0.434364 (0.051383) | 0.606181 / 0.540337 (0.065843) | 0.776691 / 1.386936 (-0.610245) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007820 / 0.011353 (-0.003533) | 0.004531 / 0.011008 (-0.006478) | 0.076142 / 0.038508 (0.037634) | 0.086367 / 0.023109 (0.063258) | 0.456150 / 0.275898 (0.180252) | 0.499712 / 0.323480 (0.176232) | 0.006545 / 0.007986 (-0.001441) | 0.003760 / 0.004328 (-0.000568) | 0.076400 / 0.004250 (0.072150) | 0.069689 / 0.037052 (0.032637) | 0.459732 / 0.258489 (0.201243) | 0.504217 / 0.293841 (0.210376) | 0.037838 / 0.128546 (-0.090709) | 0.009804 / 0.075646 (-0.065843) | 0.084654 / 0.419271 (-0.334617) | 0.060301 / 0.043533 (0.016768) | 0.452984 / 0.255139 (0.197845) | 0.479956 / 0.283200 (0.196757) | 0.029674 / 0.141683 (-0.112009) | 1.814059 / 1.452155 (0.361904) | 1.878886 / 1.492716 (0.386170) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.326174 / 0.018006 (0.308168) | 0.539722 / 0.000490 (0.539232) | 0.025637 / 0.000200 (0.025437) | 0.000209 / 0.000054 (0.000154) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.036328 / 0.037411 (-0.001084) | 0.106369 / 0.014526 (0.091843) | 0.118598 / 0.176557 (-0.057958) | 0.182760 / 0.737135 (-0.554376) | 0.120013 / 0.296338 (-0.176326) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.507328 / 0.215209 (0.292119) | 5.092689 / 2.077655 (3.015034) | 2.962334 / 1.504120 (1.458214) | 2.507699 / 1.541195 (0.966504) | 2.612245 / 1.468490 (1.143755) | 0.568625 / 4.584777 (-4.016152) | 4.296484 / 3.745712 (0.550772) | 4.037788 / 5.269862 (-1.232073) | 2.579826 / 4.565676 (-1.985850) | 0.068558 / 0.424275 (-0.355717) | 0.008916 / 0.007607 (0.001309) | 0.601054 / 0.226044 (0.375010) | 6.016061 / 2.268929 (3.747133) | 3.311880 / 55.444624 (-52.132744) | 2.912926 / 6.876477 (-3.963551) | 3.101465 / 2.142072 (0.959393) | 0.686848 / 4.805227 (-4.118380) | 0.160243 / 6.500664 (-6.340421) | 0.074084 / 0.075469 (-0.001385) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.754343 / 1.841788 (-0.087444) | 24.215302 / 8.074308 (16.140994) | 17.211007 / 10.191392 (7.019615) | 0.188370 / 0.680424 (-0.492054) | 0.028157 / 0.534201 (-0.506044) | 0.490879 / 0.579283 (-0.088404) | 0.501508 / 0.434364 (0.067144) | 0.599719 / 0.540337 (0.059381) | 0.852438 / 1.386936 (-0.534498) |

</details>
</details>

![](https://cml.dev/watermark.png#d84cd1d6f51ca75ec5f5c3db3f372f093758cac9 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009736 / 0.011353 (-0.001617) | 0.004761 / 0.011008 (-0.006247) | 0.100069 / 0.038508 (0.061561) | 0.077944 / 0.023109 (0.054835) | 0.419944 / 0.275898 (0.144046) | 0.459803 / 0.323480 (0.136323) | 0.006296 / 0.007986 (-0.001689) | 0.005375 / 0.004328 (0.001047) | 0.089457 / 0.004250 (0.085207) | 0.060585 / 0.037052 (0.023532) | 0.437988 / 0.258489 (0.179499) | 0.482676 / 0.293841 (0.188835) | 0.049126 / 0.128546 (-0.079420) | 0.015043 / 0.075646 (-0.060603) | 0.342500 / 0.419271 (-0.076771) | 0.067088 / 0.043533 (0.023555) | 0.418364 / 0.255139 (0.163225) | 0.458259 / 0.283200 (0.175059) | 0.034091 / 0.141683 (-0.107592) | 1.721589 / 1.452155 (0.269434) | 1.823142 / 1.492716 (0.330426) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.212110 / 0.018006 (0.194103) | 0.530957 / 0.000490 (0.530467) | 0.003581 / 0.000200 (0.003382) | 0.000112 / 0.000054 (0.000058) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030202 / 0.037411 (-0.007210) | 0.100552 / 0.014526 (0.086026) | 0.108150 / 0.176557 (-0.068407) | 0.173203 / 0.737135 (-0.563932) | 0.108624 / 0.296338 (-0.187715) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.577340 / 0.215209 (0.362131) | 5.794197 / 2.077655 (3.716543) | 2.396285 / 1.504120 (0.892165) | 2.151972 / 1.541195 (0.610777) | 2.109485 / 1.468490 (0.640995) | 0.873906 / 4.584777 (-3.710871) | 5.083302 / 3.745712 (1.337589) | 4.600756 / 5.269862 (-0.669105) | 2.891731 / 4.565676 (-1.673945) | 0.096293 / 0.424275 (-0.327982) | 0.008651 / 0.007607 (0.001044) | 0.719095 / 0.226044 (0.493051) | 7.193225 / 2.268929 (4.924297) | 3.220145 / 55.444624 (-52.224479) | 2.496715 / 6.876477 (-4.379762) | 2.672972 / 2.142072 (0.530900) | 1.031656 / 4.805227 (-3.773571) | 0.207854 / 6.500664 (-6.292810) | 0.074507 / 0.075469 (-0.000962) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.552821 / 1.841788 (-0.288967) | 22.573015 / 8.074308 (14.498707) | 21.074321 / 10.191392 (10.882929) | 0.231911 / 0.680424 (-0.448513) | 0.027761 / 0.534201 (-0.506440) | 0.474644 / 0.579283 (-0.104639) | 0.563780 / 0.434364 (0.129416) | 0.527593 / 0.540337 (-0.012745) | 0.732299 / 1.386936 (-0.654637) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008675 / 0.011353 (-0.002678) | 0.005268 / 0.011008 (-0.005741) | 0.079078 / 0.038508 (0.040570) | 0.073505 / 0.023109 (0.050395) | 0.453982 / 0.275898 (0.178083) | 0.487839 / 0.323480 (0.164359) | 0.005950 / 0.007986 (-0.002035) | 0.003848 / 0.004328 (-0.000481) | 0.076004 / 0.004250 (0.071754) | 0.058410 / 0.037052 (0.021358) | 0.460099 / 0.258489 (0.201610) | 0.514860 / 0.293841 (0.221019) | 0.048843 / 0.128546 (-0.079703) | 0.014275 / 0.075646 (-0.061371) | 0.090243 / 0.419271 (-0.329029) | 0.060092 / 0.043533 (0.016559) | 0.455669 / 0.255139 (0.200530) | 0.484738 / 0.283200 (0.201538) | 0.033012 / 0.141683 (-0.108671) | 1.738854 / 1.452155 (0.286699) | 1.852552 / 1.492716 (0.359835) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.245453 / 0.018006 (0.227447) | 0.519929 / 0.000490 (0.519439) | 0.007262 / 0.000200 (0.007062) | 0.000108 / 0.000054 (0.000054) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031446 / 0.037411 (-0.005965) | 0.094236 / 0.014526 (0.079710) | 0.114457 / 0.176557 (-0.062100) | 0.167448 / 0.737135 (-0.569687) | 0.108791 / 0.296338 (-0.187548) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.603331 / 0.215209 (0.388122) | 6.051556 / 2.077655 (3.973902) | 2.797110 / 1.504120 (1.292990) | 2.500517 / 1.541195 (0.959322) | 2.531421 / 1.468490 (1.062931) | 0.852075 / 4.584777 (-3.732702) | 5.034140 / 3.745712 (1.288427) | 4.576573 / 5.269862 (-0.693289) | 2.973541 / 4.565676 (-1.592135) | 0.101303 / 0.424275 (-0.322972) | 0.008467 / 0.007607 (0.000860) | 0.707143 / 0.226044 (0.481098) | 7.262803 / 2.268929 (4.993874) | 3.548841 / 55.444624 (-51.895783) | 2.895975 / 6.876477 (-3.980502) | 3.063521 / 2.142072 (0.921449) | 1.014961 / 4.805227 (-3.790266) | 0.208527 / 6.500664 (-6.292137) | 0.074939 / 0.075469 (-0.000530) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.670708 / 1.841788 (-0.171080) | 22.685227 / 8.074308 (14.610919) | 20.393017 / 10.191392 (10.201625) | 0.239303 / 0.680424 (-0.441121) | 0.027742 / 0.534201 (-0.506459) | 0.467230 / 0.579283 (-0.112053) | 0.564169 / 0.434364 (0.129805) | 0.554859 / 0.540337 (0.014522) | 0.767471 / 1.386936 (-0.619465) |

</details>
</details>

![](https://cml.dev/watermark.png#72a57356a46ded67f4d7a02741141a96061246a8 "CML watermark")