Title: Remove unused argument in `_get_data_files_patterns`
Author: lhoestq (MEMBER)
State: closed
Created: 2023-10-23T14:54:18Z


Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006584 / 0.011353 (-0.004769) | 0.004197 / 0.011008 (-0.006812) | 0.083598 / 0.038508 (0.045090) | 0.075502 / 0.023109 (0.052392) | 0.312986 / 0.275898 (0.037088) | 0.344630 / 0.323480 (0.021150) | 0.005394 / 0.007986 (-0.002591) | 0.003485 / 0.004328 (-0.000843) | 0.064529 / 0.004250 (0.060279) | 0.055003 / 0.037052 (0.017950) | 0.320522 / 0.258489 (0.062033) | 0.362623 / 0.293841 (0.068782) | 0.030900 / 0.128546 (-0.097646) | 0.008459 / 0.075646 (-0.067187) | 0.286986 / 0.419271 (-0.132285) | 0.052310 / 0.043533 (0.008777) | 0.315873 / 0.255139 (0.060734) | 0.333962 / 0.283200 (0.050762) | 0.023836 / 0.141683 (-0.117847) | 1.481806 / 1.452155 (0.029651) | 1.567926 / 1.492716 (0.075209) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.268188 / 0.018006 (0.250182) | 0.520542 / 0.000490 (0.520052) | 0.017617 / 0.000200 (0.017417) | 0.000631 / 0.000054 (0.000577) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028828 / 0.037411 (-0.008584) | 0.083028 / 0.014526 (0.068502) | 0.099808 / 0.176557 (-0.076748) | 0.154282 / 0.737135 (-0.582853) | 0.098590 / 0.296338 (-0.197748) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.407548 / 0.215209 (0.192339) | 4.066128 / 2.077655 (1.988474) | 2.036757 / 1.504120 (0.532637) | 1.870130 / 1.541195 (0.328935) | 1.949031 / 1.468490 (0.480541) | 0.489263 / 4.584777 (-4.095514) | 3.506269 / 3.745712 (-0.239443) | 3.457232 / 5.269862 (-1.812629) | 2.060097 / 4.565676 (-2.505580) | 0.057252 / 0.424275 (-0.367024) | 0.007727 / 0.007607 (0.000120) | 0.480229 / 0.226044 (0.254185) | 4.807064 / 2.268929 (2.538135) | 2.495438 / 55.444624 (-52.949186) | 2.186194 / 6.876477 (-4.690283) | 2.243372 / 2.142072 (0.101300) | 0.580550 / 4.805227 (-4.224678) | 0.135398 / 6.500664 (-6.365266) | 0.061878 / 0.075469 (-0.013591) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.305635 / 1.841788 (-0.536152) | 19.194421 / 8.074308 (11.120113) | 14.531699 / 10.191392 (4.340307) | 0.167144 / 0.680424 (-0.513280) | 0.018270 / 0.534201 (-0.515931) | 0.393702 / 0.579283 (-0.185581) | 0.406518 / 0.434364 (-0.027846) | 0.458126 / 0.540337 (-0.082211) | 0.639839 / 1.386936 (-0.747097) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006742 / 0.011353 (-0.004611) | 0.004092 / 0.011008 (-0.006916) | 0.065547 / 0.038508 (0.027039) | 0.076293 / 0.023109 (0.053184) | 0.389701 / 0.275898 (0.113803) | 0.429158 / 0.323480 (0.105678) | 0.005606 / 0.007986 (-0.002380) | 0.003491 / 0.004328 (-0.000837) | 0.065903 / 0.004250 (0.061653) | 0.057346 / 0.037052 (0.020293) | 0.393233 / 0.258489 (0.134744) | 0.433106 / 0.293841 (0.139265) | 0.032612 / 0.128546 (-0.095934) | 0.008777 / 0.075646 (-0.066869) | 0.073135 / 0.419271 (-0.346137) | 0.048167 / 0.043533 (0.004635) | 0.389309 / 0.255139 (0.134170) | 0.416442 / 0.283200 (0.133242) | 0.022839 / 0.141683 (-0.118844) | 1.531607 / 1.452155 (0.079453) | 1.598950 / 1.492716 (0.106234) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.254856 / 0.018006 (0.236850) | 0.528186 / 0.000490 (0.527697) | 0.006975 / 0.000200 (0.006775) | 0.000102 / 0.000054 (0.000048) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032377 / 0.037411 (-0.005034) | 0.092706 / 0.014526 (0.078180) | 0.107618 / 0.176557 (-0.068939) | 0.160103 / 0.737135 (-0.577032) | 0.107226 / 0.296338 (-0.189112) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.430922 / 0.215209 (0.215713) | 4.312556 / 2.077655 (2.234901) | 2.287686 / 1.504120 (0.783567) | 2.111103 / 1.541195 (0.569908) | 2.284105 / 1.468490 (0.815614) | 0.485987 / 4.584777 (-4.098790) | 3.557320 / 3.745712 (-0.188392) | 3.341150 / 5.269862 (-1.928711) | 2.056705 / 4.565676 (-2.508972) | 0.057265 / 0.424275 (-0.367010) | 0.007264 / 0.007607 (-0.000344) | 0.505191 / 0.226044 (0.279146) | 5.045379 / 2.268929 (2.776450) | 2.732357 / 55.444624 (-52.712267) | 2.390256 / 6.876477 (-4.486220) | 2.643676 / 2.142072 (0.501604) | 0.584630 / 4.805227 (-4.220597) | 0.132402 / 6.500664 (-6.368262) | 0.061387 / 0.075469 (-0.014082) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.340721 / 1.841788 (-0.501066) | 19.744145 / 8.074308 (11.669837) | 14.694482 / 10.191392 (4.503090) | 0.166294 / 0.680424 (-0.514129) | 0.020691 / 0.534201 (-0.513510) | 0.398359 / 0.579283 (-0.180924) | 0.423831 / 0.434364 (-0.010533) | 0.474365 / 0.540337 (-0.065972) | 0.649410 / 1.386936 (-0.737526) |

</details>
</details>

![](https://cml.dev/watermark.png#b29bc9cef6237eb0d18f77c56686705f468bed25 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004369 / 0.011353 (-0.006984) | 0.002728 / 0.011008 (-0.008280) | 0.063754 / 0.038508 (0.025246) | 0.029396 / 0.023109 (0.006287) | 0.269409 / 0.275898 (-0.006489) | 0.287654 / 0.323480 (-0.035826) | 0.003926 / 0.007986 (-0.004060) | 0.002366 / 0.004328 (-0.001963) | 0.048910 / 0.004250 (0.044660) | 0.043126 / 0.037052 (0.006074) | 0.260774 / 0.258489 (0.002285) | 0.299996 / 0.293841 (0.006155) | 0.023359 / 0.128546 (-0.105187) | 0.007259 / 0.075646 (-0.068388) | 0.211412 / 0.419271 (-0.207860) | 0.053883 / 0.043533 (0.010350) | 0.268946 / 0.255139 (0.013807) | 0.287664 / 0.283200 (0.004465) | 0.017600 / 0.141683 (-0.124083) | 1.096478 / 1.452155 (-0.355676) | 1.193063 / 1.492716 (-0.299653) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090985 / 0.018006 (0.072979) | 0.287168 / 0.000490 (0.286678) | 0.000208 / 0.000200 (0.000009) | 0.000052 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019238 / 0.037411 (-0.018173) | 0.062660 / 0.014526 (0.048134) | 0.073414 / 0.176557 (-0.103143) | 0.120842 / 0.737135 (-0.616294) | 0.077658 / 0.296338 (-0.218681) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.280285 / 0.215209 (0.065076) | 2.729807 / 2.077655 (0.652152) | 1.430686 / 1.504120 (-0.073434) | 1.307260 / 1.541195 (-0.233935) | 1.321013 / 1.468490 (-0.147477) | 0.387253 / 4.584777 (-4.197524) | 2.415635 / 3.745712 (-1.330077) | 2.557206 / 5.269862 (-2.712656) | 1.553224 / 4.565676 (-3.012453) | 0.045402 / 0.424275 (-0.378873) | 0.004798 / 0.007607 (-0.002809) | 0.330493 / 0.226044 (0.104449) | 3.226835 / 2.268929 (0.957906) | 1.739068 / 55.444624 (-53.705557) | 1.494841 / 6.876477 (-5.381636) | 1.528253 / 2.142072 (-0.613820) | 0.451525 / 4.805227 (-4.353702) | 0.096620 / 6.500664 (-6.404044) | 0.041176 / 0.075469 (-0.034293) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.930892 / 1.841788 (-0.910896) | 11.343351 / 8.074308 (3.269043) | 10.420327 / 10.191392 (0.228935) | 0.137629 / 0.680424 (-0.542795) | 0.013907 / 0.534201 (-0.520293) | 0.267778 / 0.579283 (-0.311505) | 0.260774 / 0.434364 (-0.173590) | 0.308213 / 0.540337 (-0.232124) | 0.419659 / 1.386936 (-0.967277) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004867 / 0.011353 (-0.006486) | 0.002830 / 0.011008 (-0.008178) | 0.048506 / 0.038508 (0.009998) | 0.048190 / 0.023109 (0.025080) | 0.279995 / 0.275898 (0.004097) | 0.296396 / 0.323480 (-0.027083) | 0.004700 / 0.007986 (-0.003285) | 0.003546 / 0.004328 (-0.000782) | 0.048237 / 0.004250 (0.043987) | 0.037102 / 0.037052 (0.000050) | 0.284582 / 0.258489 (0.026093) | 0.315896 / 0.293841 (0.022055) | 0.024699 / 0.128546 (-0.103848) | 0.007077 / 0.075646 (-0.068569) | 0.054471 / 0.419271 (-0.364800) | 0.032537 / 0.043533 (-0.010996) | 0.276761 / 0.255139 (0.021622) | 0.294741 / 0.283200 (0.011542) | 0.017766 / 0.141683 (-0.123917) | 1.118377 / 1.452155 (-0.333778) | 1.186617 / 1.492716 (-0.306100) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.088981 / 0.018006 (0.070975) | 0.297793 / 0.000490 (0.297303) | 0.000220 / 0.000200 (0.000020) | 0.000050 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021300 / 0.037411 (-0.016111) | 0.070059 / 0.014526 (0.055533) | 0.080452 / 0.176557 (-0.096104) | 0.118461 / 0.737135 (-0.618674) | 0.081099 / 0.296338 (-0.215240) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.300560 / 0.215209 (0.085351) | 2.951461 / 2.077655 (0.873806) | 1.621978 / 1.504120 (0.117858) | 1.478871 / 1.541195 (-0.062324) | 1.520732 / 1.468490 (0.052242) | 0.408625 / 4.584777 (-4.176152) | 2.407253 / 3.745712 (-1.338459) | 2.546000 / 5.269862 (-2.723861) | 1.525920 / 4.565676 (-3.039757) | 0.046817 / 0.424275 (-0.377458) | 0.004880 / 0.007607 (-0.002727) | 0.350866 / 0.226044 (0.124821) | 3.489379 / 2.268929 (1.220451) | 1.967197 / 55.444624 (-53.477427) | 1.686083 / 6.876477 (-5.190394) | 1.699307 / 2.142072 (-0.442766) | 0.479659 / 4.805227 (-4.325568) | 0.098853 / 6.500664 (-6.401811) | 0.040718 / 0.075469 (-0.034751) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.018352 / 1.841788 (-0.823436) | 12.022551 / 8.074308 (3.948243) | 10.841890 / 10.191392 (0.650498) | 0.130732 / 0.680424 (-0.549692) | 0.016334 / 0.534201 (-0.517867) | 0.271984 / 0.579283 (-0.307299) | 0.276733 / 0.434364 (-0.157631) | 0.308049 / 0.540337 (-0.232289) | 0.415428 / 1.386936 (-0.971508) |

</details>
</details>

![](https://cml.dev/watermark.png#31d95717e4e5fc6dd7699878720f063d51f1d595 "CML watermark")