Title: No data files duplicates
Author: lhoestq (MEMBER)
State: closed
Created: 2023-10-05T10:31:58Z

I added a new DataFilesSet class to disallow duplicate data files.
I also deprecated DataFilesList.

EDIT: actually I might just add drop_duplicates=True to `.from_patterns`

close https://github.com/huggingface/datasets/issues/6259
close https://github.com/huggingface/datasets/issues/6272

TODO:
- [ ] tests
- [ ] preserve data files order

Comments:
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009624 / 0.011353 (-0.001729) | 0.005121 / 0.011008 (-0.005887) | 0.105560 / 0.038508 (0.067052) | 0.090749 / 0.023109 (0.067640) | 0.430274 / 0.275898 (0.154376) | 0.443399 / 0.323480 (0.119919) | 0.006575 / 0.007986 (-0.001411) | 0.004396 / 0.004328 (0.000068) | 0.080900 / 0.004250 (0.076649) | 0.064921 / 0.037052 (0.027868) | 0.410092 / 0.258489 (0.151603) | 0.470058 / 0.293841 (0.176217) | 0.054160 / 0.128546 (-0.074386) | 0.014367 / 0.075646 (-0.061279) | 0.384844 / 0.419271 (-0.034428) | 0.072818 / 0.043533 (0.029285) | 0.429341 / 0.255139 (0.174202) | 0.430968 / 0.283200 (0.147769) | 0.038437 / 0.141683 (-0.103246) | 1.814456 / 1.452155 (0.362301) | 1.832122 / 1.492716 (0.339406) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.329266 / 0.018006 (0.311260) | 0.596848 / 0.000490 (0.596358) | 0.018291 / 0.000200 (0.018091) | 0.000113 / 0.000054 (0.000058) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030505 / 0.037411 (-0.006907) | 0.097394 / 0.014526 (0.082869) | 0.127144 / 0.176557 (-0.049412) | 0.190251 / 0.737135 (-0.546884) | 0.116543 / 0.296338 (-0.179795) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.592124 / 0.215209 (0.376915) | 5.979801 / 2.077655 (3.902146) | 2.837753 / 1.504120 (1.333633) | 2.492942 / 1.541195 (0.951747) | 2.548083 / 1.468490 (1.079593) | 0.870446 / 4.584777 (-3.714330) | 5.493718 / 3.745712 (1.748006) | 4.945135 / 5.269862 (-0.324727) | 3.133994 / 4.565676 (-1.431683) | 0.097742 / 0.424275 (-0.326533) | 0.008750 / 0.007607 (0.001143) | 0.723304 / 0.226044 (0.497260) | 7.353766 / 2.268929 (5.084838) | 3.504808 / 55.444624 (-51.939816) | 2.872490 / 6.876477 (-4.003987) | 3.186628 / 2.142072 (1.044556) | 1.035470 / 4.805227 (-3.769758) | 0.211980 / 6.500664 (-6.288684) | 0.080356 / 0.075469 (0.004887) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.623389 / 1.841788 (-0.218399) | 23.492350 / 8.074308 (15.418042) | 21.053525 / 10.191392 (10.862133) | 0.225668 / 0.680424 (-0.454756) | 0.028311 / 0.534201 (-0.505890) | 0.472672 / 0.579283 (-0.106611) | 0.581536 / 0.434364 (0.147172) | 0.525180 / 0.540337 (-0.015158) | 0.790420 / 1.386936 (-0.596516) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009091 / 0.011353 (-0.002262) | 0.004978 / 0.011008 (-0.006030) | 0.077633 / 0.038508 (0.039125) | 0.103189 / 0.023109 (0.080080) | 0.500194 / 0.275898 (0.224296) | 0.524310 / 0.323480 (0.200831) | 0.006656 / 0.007986 (-0.001329) | 0.004586 / 0.004328 (0.000257) | 0.075535 / 0.004250 (0.071284) | 0.065100 / 0.037052 (0.028048) | 0.513776 / 0.258489 (0.255287) | 0.528483 / 0.293841 (0.234642) | 0.049877 / 0.128546 (-0.078669) | 0.012494 / 0.075646 (-0.063152) | 0.090225 / 0.419271 (-0.329046) | 0.054648 / 0.043533 (0.011116) | 0.510369 / 0.255139 (0.255230) | 0.540042 / 0.283200 (0.256842) | 0.035966 / 0.141683 (-0.105717) | 1.825965 / 1.452155 (0.373810) | 1.965647 / 1.492716 (0.472931) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.295921 / 0.018006 (0.277914) | 0.605751 / 0.000490 (0.605262) | 0.007243 / 0.000200 (0.007043) | 0.000134 / 0.000054 (0.000079) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032954 / 0.037411 (-0.004457) | 0.093613 / 0.014526 (0.079087) | 0.120010 / 0.176557 (-0.056546) | 0.176168 / 0.737135 (-0.560967) | 0.113978 / 0.296338 (-0.182360) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.682904 / 0.215209 (0.467695) | 6.674640 / 2.077655 (4.596986) | 3.360660 / 1.504120 (1.856540) | 3.227246 / 1.541195 (1.686051) | 3.188852 / 1.468490 (1.720362) | 0.862293 / 4.584777 (-3.722484) | 5.518455 / 3.745712 (1.772743) | 4.881904 / 5.269862 (-0.387957) | 3.066964 / 4.565676 (-1.498712) | 0.099284 / 0.424275 (-0.324991) | 0.008644 / 0.007607 (0.001037) | 0.789231 / 0.226044 (0.563186) | 7.872017 / 2.268929 (5.603089) | 4.037105 / 55.444624 (-51.407519) | 3.318921 / 6.876477 (-3.557555) | 3.621953 / 2.142072 (1.479881) | 1.012049 / 4.805227 (-3.793178) | 0.204541 / 6.500664 (-6.296123) | 0.074509 / 0.075469 (-0.000960) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.748215 / 1.841788 (-0.093573) | 24.274974 / 8.074308 (16.200665) | 20.582389 / 10.191392 (10.390997) | 0.251001 / 0.680424 (-0.429423) | 0.032390 / 0.534201 (-0.501811) | 0.479211 / 0.579283 (-0.100072) | 0.607482 / 0.434364 (0.173118) | 0.587867 / 0.540337 (0.047530) | 0.822399 / 1.386936 (-0.564537) |

</details>
</details>

![](https://cml.dev/watermark.png#f2b6b2fd90ba47f19e9ab125f6f7656903dd065f "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009715 / 0.011353 (-0.001638) | 0.005449 / 0.011008 (-0.005559) | 0.108556 / 0.038508 (0.070048) | 0.080512 / 0.023109 (0.057403) | 0.450736 / 0.275898 (0.174838) | 0.487771 / 0.323480 (0.164291) | 0.005155 / 0.007986 (-0.002830) | 0.004213 / 0.004328 (-0.000115) | 0.087247 / 0.004250 (0.082997) | 0.063962 / 0.037052 (0.026909) | 0.454153 / 0.258489 (0.195664) | 0.499917 / 0.293841 (0.206076) | 0.052605 / 0.128546 (-0.075942) | 0.013019 / 0.075646 (-0.062627) | 0.379716 / 0.419271 (-0.039555) | 0.073241 / 0.043533 (0.029708) | 0.473488 / 0.255139 (0.218349) | 0.482944 / 0.283200 (0.199745) | 0.041541 / 0.141683 (-0.100142) | 1.829415 / 1.452155 (0.377261) | 1.953280 / 1.492716 (0.460564) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.313725 / 0.018006 (0.295719) | 0.591336 / 0.000490 (0.590847) | 0.021224 / 0.000200 (0.021025) | 0.000969 / 0.000054 (0.000914) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031874 / 0.037411 (-0.005537) | 0.099786 / 0.014526 (0.085260) | 0.116987 / 0.176557 (-0.059569) | 0.205538 / 0.737135 (-0.531597) | 0.118716 / 0.296338 (-0.177622) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.617145 / 0.215209 (0.401936) | 6.079144 / 2.077655 (4.001489) | 2.567233 / 1.504120 (1.063113) | 2.265301 / 1.541195 (0.724107) | 2.314001 / 1.468490 (0.845511) | 0.871561 / 4.584777 (-3.713216) | 5.477049 / 3.745712 (1.731337) | 4.720552 / 5.269862 (-0.549309) | 3.107515 / 4.565676 (-1.458162) | 0.100438 / 0.424275 (-0.323838) | 0.008586 / 0.007607 (0.000979) | 0.716913 / 0.226044 (0.490869) | 7.108417 / 2.268929 (4.839489) | 3.391336 / 55.444624 (-52.053288) | 2.734052 / 6.876477 (-4.142425) | 2.857226 / 2.142072 (0.715153) | 1.024121 / 4.805227 (-3.781106) | 0.216735 / 6.500664 (-6.283929) | 0.081605 / 0.075469 (0.006136) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.678176 / 1.841788 (-0.163611) | 23.606037 / 8.074308 (15.531729) | 21.485331 / 10.191392 (11.293939) | 0.218312 / 0.680424 (-0.462112) | 0.027061 / 0.534201 (-0.507140) | 0.481188 / 0.579283 (-0.098096) | 0.620592 / 0.434364 (0.186228) | 0.574778 / 0.540337 (0.034441) | 0.831529 / 1.386936 (-0.555407) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.011666 / 0.011353 (0.000313) | 0.005187 / 0.011008 (-0.005821) | 0.080692 / 0.038508 (0.042184) | 0.079159 / 0.023109 (0.056049) | 0.530823 / 0.275898 (0.254925) | 0.577807 / 0.323480 (0.254327) | 0.006246 / 0.007986 (-0.001740) | 0.004355 / 0.004328 (0.000026) | 0.080702 / 0.004250 (0.076452) | 0.062279 / 0.037052 (0.025226) | 0.553712 / 0.258489 (0.295223) | 0.579112 / 0.293841 (0.285271) | 0.056374 / 0.128546 (-0.072172) | 0.014681 / 0.075646 (-0.060966) | 0.097110 / 0.419271 (-0.322161) | 0.061040 / 0.043533 (0.017507) | 0.524718 / 0.255139 (0.269579) | 0.568586 / 0.283200 (0.285386) | 0.035774 / 0.141683 (-0.105909) | 1.864590 / 1.452155 (0.412435) | 1.953715 / 1.492716 (0.460998) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.271315 / 0.018006 (0.253309) | 0.571343 / 0.000490 (0.570854) | 0.015812 / 0.000200 (0.015612) | 0.000115 / 0.000054 (0.000060) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.038582 / 0.037411 (0.001170) | 0.117523 / 0.014526 (0.102997) | 0.128864 / 0.176557 (-0.047693) | 0.191164 / 0.737135 (-0.545971) | 0.133161 / 0.296338 (-0.163178) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.679305 / 0.215209 (0.464096) | 6.814451 / 2.077655 (4.736796) | 3.377431 / 1.504120 (1.873311) | 3.011008 / 1.541195 (1.469813) | 3.093200 / 1.468490 (1.624710) | 0.905827 / 4.584777 (-3.678950) | 5.456094 / 3.745712 (1.710382) | 4.848511 / 5.269862 (-0.421351) | 3.064230 / 4.565676 (-1.501447) | 0.107478 / 0.424275 (-0.316798) | 0.009234 / 0.007607 (0.001627) | 0.833944 / 0.226044 (0.607899) | 8.286100 / 2.268929 (6.017171) | 4.241455 / 55.444624 (-51.203169) | 3.405460 / 6.876477 (-3.471017) | 3.660618 / 2.142072 (1.518546) | 1.046310 / 4.805227 (-3.758917) | 0.210891 / 6.500664 (-6.289773) | 0.079413 / 0.075469 (0.003944) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.825448 / 1.841788 (-0.016340) | 24.639059 / 8.074308 (16.564750) | 21.970417 / 10.191392 (11.779025) | 0.247708 / 0.680424 (-0.432715) | 0.033810 / 0.534201 (-0.500391) | 0.495517 / 0.579283 (-0.083766) | 0.601820 / 0.434364 (0.167456) | 0.585618 / 0.540337 (0.045280) | 0.858722 / 1.386936 (-0.528214) |

</details>
</details>

![](https://cml.dev/watermark.png#0477e20dccb77b68f0add77fd5c9b4cb05473235 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006137 / 0.011353 (-0.005216) | 0.003685 / 0.011008 (-0.007324) | 0.079985 / 0.038508 (0.041476) | 0.060937 / 0.023109 (0.037828) | 0.390583 / 0.275898 (0.114685) | 0.425307 / 0.323480 (0.101827) | 0.003433 / 0.007986 (-0.004552) | 0.002868 / 0.004328 (-0.001461) | 0.062572 / 0.004250 (0.058322) | 0.048642 / 0.037052 (0.011590) | 0.401096 / 0.258489 (0.142607) | 0.436988 / 0.293841 (0.143147) | 0.027645 / 0.128546 (-0.100901) | 0.007973 / 0.075646 (-0.067673) | 0.261997 / 0.419271 (-0.157275) | 0.045393 / 0.043533 (0.001860) | 0.394266 / 0.255139 (0.139127) | 0.414448 / 0.283200 (0.131248) | 0.022551 / 0.141683 (-0.119131) | 1.438458 / 1.452155 (-0.013697) | 1.501568 / 1.492716 (0.008852) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.224335 / 0.018006 (0.206329) | 0.421918 / 0.000490 (0.421428) | 0.006883 / 0.000200 (0.006683) | 0.000210 / 0.000054 (0.000155) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023505 / 0.037411 (-0.013906) | 0.072438 / 0.014526 (0.057912) | 0.083576 / 0.176557 (-0.092981) | 0.142906 / 0.737135 (-0.594229) | 0.083910 / 0.296338 (-0.212428) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.396004 / 0.215209 (0.180795) | 3.969852 / 2.077655 (1.892197) | 1.966000 / 1.504120 (0.461880) | 1.786453 / 1.541195 (0.245258) | 1.866082 / 1.468490 (0.397592) | 0.502633 / 4.584777 (-4.082144) | 3.114331 / 3.745712 (-0.631382) | 2.940003 / 5.269862 (-2.329859) | 1.901844 / 4.565676 (-2.663832) | 0.058109 / 0.424275 (-0.366166) | 0.006502 / 0.007607 (-0.001105) | 0.463465 / 0.226044 (0.237420) | 4.641531 / 2.268929 (2.372603) | 2.315759 / 55.444624 (-53.128865) | 2.253088 / 6.876477 (-4.623389) | 2.151399 / 2.142072 (0.009326) | 0.592225 / 4.805227 (-4.213002) | 0.125072 / 6.500664 (-6.375592) | 0.059966 / 0.075469 (-0.015503) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.231392 / 1.841788 (-0.610396) | 17.533893 / 8.074308 (9.459585) | 13.710478 / 10.191392 (3.519086) | 0.147389 / 0.680424 (-0.533035) | 0.017932 / 0.534201 (-0.516269) | 0.334144 / 0.579283 (-0.245139) | 0.368817 / 0.434364 (-0.065547) | 0.383790 / 0.540337 (-0.156547) | 0.540262 / 1.386936 (-0.846674) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006066 / 0.011353 (-0.005287) | 0.003804 / 0.011008 (-0.007205) | 0.062474 / 0.038508 (0.023966) | 0.060547 / 0.023109 (0.037437) | 0.448643 / 0.275898 (0.172745) | 0.487005 / 0.323480 (0.163525) | 0.004884 / 0.007986 (-0.003102) | 0.002911 / 0.004328 (-0.001418) | 0.062950 / 0.004250 (0.058700) | 0.049672 / 0.037052 (0.012620) | 0.477491 / 0.258489 (0.219002) | 0.488234 / 0.293841 (0.194393) | 0.028711 / 0.128546 (-0.099835) | 0.008101 / 0.075646 (-0.067545) | 0.068333 / 0.419271 (-0.350939) | 0.040959 / 0.043533 (-0.002574) | 0.450716 / 0.255139 (0.195577) | 0.471089 / 0.283200 (0.187890) | 0.020710 / 0.141683 (-0.120973) | 1.474850 / 1.452155 (0.022695) | 1.540115 / 1.492716 (0.047399) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.229811 / 0.018006 (0.211805) | 0.419526 / 0.000490 (0.419036) | 0.003818 / 0.000200 (0.003618) | 0.000084 / 0.000054 (0.000030) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026045 / 0.037411 (-0.011366) | 0.080325 / 0.014526 (0.065799) | 0.091549 / 0.176557 (-0.085007) | 0.145253 / 0.737135 (-0.591882) | 0.091849 / 0.296338 (-0.204489) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.463047 / 0.215209 (0.247838) | 4.598727 / 2.077655 (2.521072) | 2.558996 / 1.504120 (1.054877) | 2.405896 / 1.541195 (0.864701) | 2.447291 / 1.468490 (0.978801) | 0.510393 / 4.584777 (-4.074384) | 3.173344 / 3.745712 (-0.572368) | 2.901201 / 5.269862 (-2.368661) | 1.896440 / 4.565676 (-2.669236) | 0.058374 / 0.424275 (-0.365901) | 0.006449 / 0.007607 (-0.001158) | 0.539653 / 0.226044 (0.313608) | 5.408217 / 2.268929 (3.139289) | 3.042453 / 55.444624 (-52.402172) | 2.656724 / 6.876477 (-4.219753) | 2.838165 / 2.142072 (0.696092) | 0.598663 / 4.805227 (-4.206565) | 0.126211 / 6.500664 (-6.374453) | 0.062830 / 0.075469 (-0.012639) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.392412 / 1.841788 (-0.449376) | 18.195170 / 8.074308 (10.120862) | 14.788251 / 10.191392 (4.596859) | 0.132579 / 0.680424 (-0.547845) | 0.017867 / 0.534201 (-0.516334) | 0.340020 / 0.579283 (-0.239263) | 0.386719 / 0.434364 (-0.047645) | 0.398863 / 0.540337 (-0.141475) | 0.579320 / 1.386936 (-0.807617) |

</details>
</details>

![](https://cml.dev/watermark.png#a2569fdfcf387f8885974a35fafa409fbc6dd059 "CML watermark")


closing in favor of https://github.com/huggingface/datasets/pull/6282