Title: Don't skip hidden files in `dl_manager.iter_files` when they are given as input
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2023-09-11T13:29:19Z

Required for `load_dataset(<format>, data_files=["path/to/.hidden_file"])` to work as expected

Comments:
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005894 / 0.011353 (-0.005459) | 0.003621 / 0.011008 (-0.007387) | 0.080446 / 0.038508 (0.041938) | 0.056800 / 0.023109 (0.033691) | 0.326485 / 0.275898 (0.050587) | 0.376207 / 0.323480 (0.052727) | 0.004640 / 0.007986 (-0.003346) | 0.002795 / 0.004328 (-0.001533) | 0.062815 / 0.004250 (0.058565) | 0.045761 / 0.037052 (0.008709) | 0.341417 / 0.258489 (0.082928) | 0.373129 / 0.293841 (0.079288) | 0.027226 / 0.128546 (-0.101321) | 0.007873 / 0.075646 (-0.067774) | 0.261737 / 0.419271 (-0.157535) | 0.044648 / 0.043533 (0.001115) | 0.320195 / 0.255139 (0.065056) | 0.381892 / 0.283200 (0.098692) | 0.020431 / 0.141683 (-0.121252) | 1.405332 / 1.452155 (-0.046823) | 1.455592 / 1.492716 (-0.037125) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.191539 / 0.018006 (0.173533) | 0.423655 / 0.000490 (0.423165) | 0.002741 / 0.000200 (0.002541) | 0.000069 / 0.000054 (0.000014) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023952 / 0.037411 (-0.013459) | 0.073387 / 0.014526 (0.058861) | 0.083746 / 0.176557 (-0.092810) | 0.144977 / 0.737135 (-0.592159) | 0.083808 / 0.296338 (-0.212530) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.436228 / 0.215209 (0.221019) | 4.370510 / 2.077655 (2.292855) | 2.340426 / 1.504120 (0.836306) | 2.202215 / 1.541195 (0.661021) | 2.258528 / 1.468490 (0.790037) | 0.503455 / 4.584777 (-4.081322) | 3.043695 / 3.745712 (-0.702017) | 2.784033 / 5.269862 (-2.485829) | 1.847956 / 4.565676 (-2.717721) | 0.057702 / 0.424275 (-0.366573) | 0.006703 / 0.007607 (-0.000904) | 0.510628 / 0.226044 (0.284583) | 5.101890 / 2.268929 (2.832961) | 2.816469 / 55.444624 (-52.628155) | 2.474220 / 6.876477 (-4.402257) | 2.617851 / 2.142072 (0.475779) | 0.593585 / 4.805227 (-4.211642) | 0.125895 / 6.500664 (-6.374769) | 0.062170 / 0.075469 (-0.013299) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.238792 / 1.841788 (-0.602996) | 18.096417 / 8.074308 (10.022108) | 13.548778 / 10.191392 (3.357386) | 0.144878 / 0.680424 (-0.535546) | 0.016644 / 0.534201 (-0.517557) | 0.334556 / 0.579283 (-0.244728) | 0.343680 / 0.434364 (-0.090684) | 0.383093 / 0.540337 (-0.157244) | 0.525075 / 1.386936 (-0.861861) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006125 / 0.011353 (-0.005228) | 0.003668 / 0.011008 (-0.007340) | 0.062650 / 0.038508 (0.024142) | 0.058882 / 0.023109 (0.035772) | 0.454643 / 0.275898 (0.178745) | 0.486659 / 0.323480 (0.163179) | 0.005558 / 0.007986 (-0.002427) | 0.002858 / 0.004328 (-0.001471) | 0.062603 / 0.004250 (0.058353) | 0.049701 / 0.037052 (0.012649) | 0.455903 / 0.258489 (0.197413) | 0.491544 / 0.293841 (0.197703) | 0.028581 / 0.128546 (-0.099965) | 0.008040 / 0.075646 (-0.067607) | 0.068314 / 0.419271 (-0.350957) | 0.040637 / 0.043533 (-0.002896) | 0.450288 / 0.255139 (0.195149) | 0.476330 / 0.283200 (0.193131) | 0.018989 / 0.141683 (-0.122693) | 1.455122 / 1.452155 (0.002967) | 1.496941 / 1.492716 (0.004225) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.227382 / 0.018006 (0.209376) | 0.432637 / 0.000490 (0.432147) | 0.002727 / 0.000200 (0.002527) | 0.000073 / 0.000054 (0.000019) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026125 / 0.037411 (-0.011286) | 0.081342 / 0.014526 (0.066817) | 0.091227 / 0.176557 (-0.085329) | 0.145175 / 0.737135 (-0.591960) | 0.091988 / 0.296338 (-0.204351) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.454293 / 0.215209 (0.239083) | 4.537912 / 2.077655 (2.460257) | 2.489146 / 1.504120 (0.985026) | 2.307166 / 1.541195 (0.765971) | 2.380866 / 1.468490 (0.912376) | 0.509015 / 4.584777 (-4.075762) | 3.111069 / 3.745712 (-0.634644) | 2.839181 / 5.269862 (-2.430681) | 1.874630 / 4.565676 (-2.691047) | 0.058540 / 0.424275 (-0.365735) | 0.006693 / 0.007607 (-0.000914) | 0.528408 / 0.226044 (0.302363) | 5.285802 / 2.268929 (3.016874) | 2.952090 / 55.444624 (-52.492534) | 2.591496 / 6.876477 (-4.284980) | 2.741080 / 2.142072 (0.599007) | 0.595610 / 4.805227 (-4.209617) | 0.124387 / 6.500664 (-6.376277) | 0.061032 / 0.075469 (-0.014437) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.365816 / 1.841788 (-0.475972) | 18.684534 / 8.074308 (10.610226) | 14.540438 / 10.191392 (4.349046) | 0.146793 / 0.680424 (-0.533631) | 0.018165 / 0.534201 (-0.516036) | 0.333794 / 0.579283 (-0.245489) | 0.345533 / 0.434364 (-0.088830) | 0.384453 / 0.540337 (-0.155885) | 0.529104 / 1.386936 (-0.857832) |

</details>
</details>

![](https://cml.dev/watermark.png#6c884967dd5f4e8aa3d1f3c2e3a414ae53afe261 "CML watermark")


_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006121 / 0.011353 (-0.005232) | 0.003683 / 0.011008 (-0.007325) | 0.083329 / 0.038508 (0.044821) | 0.063350 / 0.023109 (0.040241) | 0.329959 / 0.275898 (0.054061) | 0.396111 / 0.323480 (0.072631) | 0.003554 / 0.007986 (-0.004432) | 0.002907 / 0.004328 (-0.001421) | 0.064152 / 0.004250 (0.059902) | 0.049182 / 0.037052 (0.012130) | 0.343862 / 0.258489 (0.085373) | 0.414568 / 0.293841 (0.120727) | 0.027157 / 0.128546 (-0.101389) | 0.007957 / 0.075646 (-0.067689) | 0.261868 / 0.419271 (-0.157404) | 0.044938 / 0.043533 (0.001405) | 0.318470 / 0.255139 (0.063331) | 0.393319 / 0.283200 (0.110119) | 0.022848 / 0.141683 (-0.118835) | 1.419916 / 1.452155 (-0.032238) | 1.508783 / 1.492716 (0.016067) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.200530 / 0.018006 (0.182523) | 0.433586 / 0.000490 (0.433097) | 0.002063 / 0.000200 (0.001863) | 0.000070 / 0.000054 (0.000016) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024803 / 0.037411 (-0.012609) | 0.075894 / 0.014526 (0.061368) | 0.086488 / 0.176557 (-0.090069) | 0.149058 / 0.737135 (-0.588077) | 0.087046 / 0.296338 (-0.209292) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.390771 / 0.215209 (0.175562) | 3.886178 / 2.077655 (1.808523) | 1.868626 / 1.504120 (0.364506) | 1.708532 / 1.541195 (0.167338) | 1.788491 / 1.468490 (0.320001) | 0.505706 / 4.584777 (-4.079071) | 3.062094 / 3.745712 (-0.683618) | 2.898559 / 5.269862 (-2.371302) | 1.901225 / 4.565676 (-2.664452) | 0.058366 / 0.424275 (-0.365909) | 0.006851 / 0.007607 (-0.000756) | 0.465382 / 0.226044 (0.239337) | 4.650187 / 2.268929 (2.381258) | 2.316152 / 55.444624 (-53.128472) | 1.989597 / 6.876477 (-4.886879) | 2.169266 / 2.142072 (0.027194) | 0.593257 / 4.805227 (-4.211970) | 0.126440 / 6.500664 (-6.374224) | 0.062227 / 0.075469 (-0.013242) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.283591 / 1.841788 (-0.558197) | 18.384667 / 8.074308 (10.310358) | 14.079611 / 10.191392 (3.888219) | 0.150453 / 0.680424 (-0.529971) | 0.017100 / 0.534201 (-0.517101) | 0.330503 / 0.579283 (-0.248780) | 0.348134 / 0.434364 (-0.086230) | 0.385726 / 0.540337 (-0.154612) | 0.529147 / 1.386936 (-0.857789) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006168 / 0.011353 (-0.005185) | 0.003801 / 0.011008 (-0.007208) | 0.063168 / 0.038508 (0.024660) | 0.062331 / 0.023109 (0.039221) | 0.448321 / 0.275898 (0.172423) | 0.484416 / 0.323480 (0.160937) | 0.004827 / 0.007986 (-0.003159) | 0.002848 / 0.004328 (-0.001480) | 0.062736 / 0.004250 (0.058486) | 0.049128 / 0.037052 (0.012075) | 0.449276 / 0.258489 (0.190787) | 0.499035 / 0.293841 (0.205194) | 0.028577 / 0.128546 (-0.099969) | 0.008114 / 0.075646 (-0.067532) | 0.068297 / 0.419271 (-0.350974) | 0.040835 / 0.043533 (-0.002698) | 0.453556 / 0.255139 (0.198417) | 0.475420 / 0.283200 (0.192220) | 0.020292 / 0.141683 (-0.121390) | 1.472226 / 1.452155 (0.020071) | 1.523809 / 1.492716 (0.031093) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.230662 / 0.018006 (0.212655) | 0.439697 / 0.000490 (0.439207) | 0.009899 / 0.000200 (0.009699) | 0.000087 / 0.000054 (0.000033) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026418 / 0.037411 (-0.010993) | 0.082188 / 0.014526 (0.067662) | 0.091039 / 0.176557 (-0.085518) | 0.146646 / 0.737135 (-0.590489) | 0.091693 / 0.296338 (-0.204645) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.462086 / 0.215209 (0.246877) | 4.620925 / 2.077655 (2.543271) | 2.539234 / 1.504120 (1.035114) | 2.371178 / 1.541195 (0.829983) | 2.440538 / 1.468490 (0.972048) | 0.511047 / 4.584777 (-4.073730) | 3.082088 / 3.745712 (-0.663624) | 2.918162 / 5.269862 (-2.351700) | 1.899651 / 4.565676 (-2.666025) | 0.059003 / 0.424275 (-0.365272) | 0.006746 / 0.007607 (-0.000861) | 0.537863 / 0.226044 (0.311819) | 5.382355 / 2.268929 (3.113426) | 3.060091 / 55.444624 (-52.384534) | 2.754969 / 6.876477 (-4.121507) | 2.863156 / 2.142072 (0.721084) | 0.606888 / 4.805227 (-4.198339) | 0.127448 / 6.500664 (-6.373216) | 0.062975 / 0.075469 (-0.012494) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.336065 / 1.841788 (-0.505722) | 19.019902 / 8.074308 (10.945594) | 15.057979 / 10.191392 (4.866587) | 0.160646 / 0.680424 (-0.519778) | 0.018340 / 0.534201 (-0.515861) | 0.341664 / 0.579283 (-0.237619) | 0.356536 / 0.434364 (-0.077828) | 0.393974 / 0.540337 (-0.146363) | 0.546036 / 1.386936 (-0.840900) |

</details>
</details>

![](https://cml.dev/watermark.png#fd04e445bd36d7eb4af4d5a6b8519ab8e306ecf5 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007220 / 0.011353 (-0.004132) | 0.004537 / 0.011008 (-0.006471) | 0.087333 / 0.038508 (0.048825) | 0.095637 / 0.023109 (0.072528) | 0.323819 / 0.275898 (0.047921) | 0.358838 / 0.323480 (0.035358) | 0.005910 / 0.007986 (-0.002076) | 0.003781 / 0.004328 (-0.000548) | 0.064565 / 0.004250 (0.060315) | 0.062818 / 0.037052 (0.025766) | 0.322595 / 0.258489 (0.064106) | 0.371865 / 0.293841 (0.078024) | 0.031667 / 0.128546 (-0.096880) | 0.009068 / 0.075646 (-0.066579) | 0.290574 / 0.419271 (-0.128697) | 0.054618 / 0.043533 (0.011085) | 0.314708 / 0.255139 (0.059569) | 0.336647 / 0.283200 (0.053447) | 0.027070 / 0.141683 (-0.114613) | 1.500640 / 1.452155 (0.048485) | 1.586775 / 1.492716 (0.094059) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.294461 / 0.018006 (0.276455) | 0.580125 / 0.000490 (0.579635) | 0.008165 / 0.000200 (0.007965) | 0.000320 / 0.000054 (0.000266) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032352 / 0.037411 (-0.005059) | 0.092187 / 0.014526 (0.077661) | 0.104993 / 0.176557 (-0.071564) | 0.162738 / 0.737135 (-0.574397) | 0.103242 / 0.296338 (-0.193096) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.396732 / 0.215209 (0.181523) | 3.955049 / 2.077655 (1.877394) | 1.876762 / 1.504120 (0.372642) | 1.698477 / 1.541195 (0.157282) | 1.847086 / 1.468490 (0.378596) | 0.488306 / 4.584777 (-4.096471) | 3.658922 / 3.745712 (-0.086790) | 3.559050 / 5.269862 (-1.710812) | 2.187363 / 4.565676 (-2.378313) | 0.059795 / 0.424275 (-0.364480) | 0.008966 / 0.007607 (0.001359) | 0.474212 / 0.226044 (0.248168) | 4.732540 / 2.268929 (2.463611) | 2.466370 / 55.444624 (-52.978254) | 2.112105 / 6.876477 (-4.764372) | 2.414624 / 2.142072 (0.272552) | 0.595447 / 4.805227 (-4.209780) | 0.136705 / 6.500664 (-6.363959) | 0.062267 / 0.075469 (-0.013202) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.266518 / 1.841788 (-0.575270) | 21.009975 / 8.074308 (12.935666) | 14.823960 / 10.191392 (4.632568) | 0.165630 / 0.680424 (-0.514793) | 0.018499 / 0.534201 (-0.515702) | 0.396720 / 0.579283 (-0.182563) | 0.424807 / 0.434364 (-0.009557) | 0.463326 / 0.540337 (-0.077011) | 0.653132 / 1.386936 (-0.733804) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007789 / 0.011353 (-0.003564) | 0.004720 / 0.011008 (-0.006288) | 0.066656 / 0.038508 (0.028148) | 0.094219 / 0.023109 (0.071109) | 0.414965 / 0.275898 (0.139067) | 0.454808 / 0.323480 (0.131328) | 0.006088 / 0.007986 (-0.001898) | 0.003980 / 0.004328 (-0.000349) | 0.066048 / 0.004250 (0.061797) | 0.065875 / 0.037052 (0.028823) | 0.419994 / 0.258489 (0.161505) | 0.462001 / 0.293841 (0.168160) | 0.033534 / 0.128546 (-0.095013) | 0.009010 / 0.075646 (-0.066636) | 0.072778 / 0.419271 (-0.346493) | 0.049834 / 0.043533 (0.006301) | 0.411003 / 0.255139 (0.155864) | 0.430918 / 0.283200 (0.147718) | 0.025664 / 0.141683 (-0.116019) | 1.526771 / 1.452155 (0.074616) | 1.634767 / 1.492716 (0.142051) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.271180 / 0.018006 (0.253174) | 0.576704 / 0.000490 (0.576214) | 0.004362 / 0.000200 (0.004162) | 0.000112 / 0.000054 (0.000058) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.035648 / 0.037411 (-0.001763) | 0.102407 / 0.014526 (0.087881) | 0.111613 / 0.176557 (-0.064944) | 0.166173 / 0.737135 (-0.570962) | 0.113371 / 0.296338 (-0.182967) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.436031 / 0.215209 (0.220822) | 4.347071 / 2.077655 (2.269416) | 2.366937 / 1.504120 (0.862817) | 2.216356 / 1.541195 (0.675161) | 2.335933 / 1.468490 (0.867443) | 0.490484 / 4.584777 (-4.094293) | 3.730656 / 3.745712 (-0.015056) | 3.497248 / 5.269862 (-1.772613) | 2.215729 / 4.565676 (-2.349947) | 0.057905 / 0.424275 (-0.366370) | 0.007983 / 0.007607 (0.000376) | 0.510413 / 0.226044 (0.284369) | 5.114502 / 2.268929 (2.845574) | 2.871599 / 55.444624 (-52.573026) | 2.537514 / 6.876477 (-4.338962) | 2.819135 / 2.142072 (0.677063) | 0.588397 / 4.805227 (-4.216830) | 0.134665 / 6.500664 (-6.365999) | 0.063349 / 0.075469 (-0.012120) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.352962 / 1.841788 (-0.488826) | 21.628664 / 8.074308 (13.554356) | 15.962105 / 10.191392 (5.770713) | 0.167781 / 0.680424 (-0.512643) | 0.020965 / 0.534201 (-0.513236) | 0.402809 / 0.579283 (-0.176474) | 0.435153 / 0.434364 (0.000789) | 0.481394 / 0.540337 (-0.058944) | 0.658068 / 1.386936 (-0.728868) |

</details>
</details>

![](https://cml.dev/watermark.png#12adf38b90fde8e2a4e46fcbb023ee23b5c4e98c "CML watermark")