Title: Missing DatasetNotFoundError
Author: lhoestq (MEMBER)
State: closed
Created: 2023-11-30T18:09:43Z

continuation of https://github.com/huggingface/datasets/pull/6431

this should fix the CI in https://github.com/huggingface/datasets/pull/6458 too

Comments:
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005594 / 0.011353 (-0.005759) | 0.003672 / 0.011008 (-0.007337) | 0.062796 / 0.038508 (0.024288) | 0.059432 / 0.023109 (0.036323) | 0.253976 / 0.275898 (-0.021922) | 0.281155 / 0.323480 (-0.042325) | 0.003023 / 0.007986 (-0.004962) | 0.003320 / 0.004328 (-0.001008) | 0.049059 / 0.004250 (0.044809) | 0.040252 / 0.037052 (0.003200) | 0.259526 / 0.258489 (0.001037) | 0.318798 / 0.293841 (0.024957) | 0.027883 / 0.128546 (-0.100663) | 0.010883 / 0.075646 (-0.064763) | 0.206948 / 0.419271 (-0.212323) | 0.036335 / 0.043533 (-0.007198) | 0.253209 / 0.255139 (-0.001930) | 0.275173 / 0.283200 (-0.008026) | 0.020365 / 0.141683 (-0.121318) | 1.121630 / 1.452155 (-0.330524) | 1.174680 / 1.492716 (-0.318036) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.098372 / 0.018006 (0.080366) | 0.309949 / 0.000490 (0.309460) | 0.000225 / 0.000200 (0.000025) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019495 / 0.037411 (-0.017916) | 0.062321 / 0.014526 (0.047795) | 0.074525 / 0.176557 (-0.102031) | 0.121832 / 0.737135 (-0.615303) | 0.077612 / 0.296338 (-0.218727) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.288156 / 0.215209 (0.072947) | 2.816411 / 2.077655 (0.738756) | 1.497926 / 1.504120 (-0.006193) | 1.378137 / 1.541195 (-0.163058) | 1.446466 / 1.468490 (-0.022024) | 0.566195 / 4.584777 (-4.018582) | 2.391933 / 3.745712 (-1.353780) | 2.929290 / 5.269862 (-2.340572) | 1.828215 / 4.565676 (-2.737462) | 0.063312 / 0.424275 (-0.360963) | 0.005199 / 0.007607 (-0.002408) | 0.342883 / 0.226044 (0.116838) | 3.378388 / 2.268929 (1.109459) | 1.865710 / 55.444624 (-53.578915) | 1.573442 / 6.876477 (-5.303035) | 1.631228 / 2.142072 (-0.510845) | 0.651614 / 4.805227 (-4.153613) | 0.118177 / 6.500664 (-6.382487) | 0.043303 / 0.075469 (-0.032166) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.950694 / 1.841788 (-0.891094) | 12.559851 / 8.074308 (4.485543) | 10.751123 / 10.191392 (0.559731) | 0.143107 / 0.680424 (-0.537317) | 0.014469 / 0.534201 (-0.519732) | 0.289531 / 0.579283 (-0.289752) | 0.267316 / 0.434364 (-0.167047) | 0.327748 / 0.540337 (-0.212590) | 0.437758 / 1.386936 (-0.949178) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005669 / 0.011353 (-0.005684) | 0.003831 / 0.011008 (-0.007177) | 0.049096 / 0.038508 (0.010588) | 0.061408 / 0.023109 (0.038299) | 0.274571 / 0.275898 (-0.001327) | 0.299978 / 0.323480 (-0.023501) | 0.004216 / 0.007986 (-0.003769) | 0.002848 / 0.004328 (-0.001480) | 0.048755 / 0.004250 (0.044504) | 0.042576 / 0.037052 (0.005524) | 0.276781 / 0.258489 (0.018292) | 0.300903 / 0.293841 (0.007062) | 0.030243 / 0.128546 (-0.098303) | 0.010967 / 0.075646 (-0.064679) | 0.057879 / 0.419271 (-0.361392) | 0.033206 / 0.043533 (-0.010327) | 0.277620 / 0.255139 (0.022481) | 0.296263 / 0.283200 (0.013064) | 0.019022 / 0.141683 (-0.122660) | 1.125615 / 1.452155 (-0.326539) | 1.278016 / 1.492716 (-0.214700) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.096836 / 0.018006 (0.078830) | 0.307491 / 0.000490 (0.307001) | 0.000230 / 0.000200 (0.000030) | 0.000044 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021552 / 0.037411 (-0.015859) | 0.071099 / 0.014526 (0.056573) | 0.082432 / 0.176557 (-0.094124) | 0.121826 / 0.737135 (-0.615310) | 0.084902 / 0.296338 (-0.211437) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.328113 / 0.215209 (0.112904) | 2.989613 / 2.077655 (0.911959) | 1.604904 / 1.504120 (0.100784) | 1.485459 / 1.541195 (-0.055735) | 1.524829 / 1.468490 (0.056339) | 0.580589 / 4.584777 (-4.004188) | 2.440087 / 3.745712 (-1.305625) | 2.944697 / 5.269862 (-2.325164) | 1.832728 / 4.565676 (-2.732949) | 0.064423 / 0.424275 (-0.359852) | 0.004991 / 0.007607 (-0.002616) | 0.357878 / 0.226044 (0.131834) | 3.515415 / 2.268929 (1.246487) | 1.964492 / 55.444624 (-53.480132) | 1.684058 / 6.876477 (-5.192418) | 1.730294 / 2.142072 (-0.411778) | 0.661228 / 4.805227 (-4.143999) | 0.122894 / 6.500664 (-6.377770) | 0.041776 / 0.075469 (-0.033693) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.969849 / 1.841788 (-0.871939) | 12.897067 / 8.074308 (4.822758) | 10.908200 / 10.191392 (0.716808) | 0.141139 / 0.680424 (-0.539285) | 0.015377 / 0.534201 (-0.518824) | 0.288625 / 0.579283 (-0.290658) | 0.279020 / 0.434364 (-0.155344) | 0.328386 / 0.540337 (-0.211951) | 0.590833 / 1.386936 (-0.796103) |

</details>
</details>

![](https://cml.dev/watermark.png#39ea60eaabb05d8ee38c072f375816cf87fce1a9 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004986 / 0.011353 (-0.006367) | 0.003070 / 0.011008 (-0.007938) | 0.062433 / 0.038508 (0.023925) | 0.050639 / 0.023109 (0.027530) | 0.241807 / 0.275898 (-0.034091) | 0.262517 / 0.323480 (-0.060963) | 0.003826 / 0.007986 (-0.004160) | 0.002602 / 0.004328 (-0.001727) | 0.048508 / 0.004250 (0.044257) | 0.037276 / 0.037052 (0.000224) | 0.245757 / 0.258489 (-0.012732) | 0.272969 / 0.293841 (-0.020871) | 0.027139 / 0.128546 (-0.101407) | 0.010265 / 0.075646 (-0.065381) | 0.207279 / 0.419271 (-0.211992) | 0.035312 / 0.043533 (-0.008221) | 0.247535 / 0.255139 (-0.007604) | 0.260668 / 0.283200 (-0.022532) | 0.016496 / 0.141683 (-0.125187) | 1.137510 / 1.452155 (-0.314645) | 1.167870 / 1.492716 (-0.324847) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091743 / 0.018006 (0.073736) | 0.298649 / 0.000490 (0.298159) | 0.000208 / 0.000200 (0.000009) | 0.000053 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019053 / 0.037411 (-0.018359) | 0.060300 / 0.014526 (0.045774) | 0.072154 / 0.176557 (-0.104402) | 0.120293 / 0.737135 (-0.616842) | 0.073923 / 0.296338 (-0.222415) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.283058 / 0.215209 (0.067849) | 2.769503 / 2.077655 (0.691849) | 1.457016 / 1.504120 (-0.047104) | 1.335753 / 1.541195 (-0.205441) | 1.325986 / 1.468490 (-0.142504) | 0.562553 / 4.584777 (-4.022224) | 2.406144 / 3.745712 (-1.339568) | 2.778063 / 5.269862 (-2.491799) | 1.782199 / 4.565676 (-2.783477) | 0.062490 / 0.424275 (-0.361785) | 0.004912 / 0.007607 (-0.002695) | 0.338500 / 0.226044 (0.112456) | 3.309746 / 2.268929 (1.040818) | 1.819693 / 55.444624 (-53.624931) | 1.510295 / 6.876477 (-5.366182) | 1.578402 / 2.142072 (-0.563671) | 0.637517 / 4.805227 (-4.167710) | 0.117018 / 6.500664 (-6.383647) | 0.048149 / 0.075469 (-0.027320) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.939424 / 1.841788 (-0.902364) | 11.494891 / 8.074308 (3.420583) | 10.115194 / 10.191392 (-0.076198) | 0.126751 / 0.680424 (-0.553673) | 0.013567 / 0.534201 (-0.520634) | 0.282501 / 0.579283 (-0.296782) | 0.260594 / 0.434364 (-0.173770) | 0.325940 / 0.540337 (-0.214397) | 0.426186 / 1.386936 (-0.960750) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005405 / 0.011353 (-0.005948) | 0.003557 / 0.011008 (-0.007451) | 0.051139 / 0.038508 (0.012631) | 0.053446 / 0.023109 (0.030337) | 0.268051 / 0.275898 (-0.007847) | 0.292343 / 0.323480 (-0.031136) | 0.004716 / 0.007986 (-0.003269) | 0.002677 / 0.004328 (-0.001651) | 0.047634 / 0.004250 (0.043384) | 0.041062 / 0.037052 (0.004009) | 0.269225 / 0.258489 (0.010736) | 0.297462 / 0.293841 (0.003621) | 0.029292 / 0.128546 (-0.099254) | 0.010947 / 0.075646 (-0.064699) | 0.057845 / 0.419271 (-0.361426) | 0.032793 / 0.043533 (-0.010740) | 0.265308 / 0.255139 (0.010169) | 0.288242 / 0.283200 (0.005043) | 0.018311 / 0.141683 (-0.123372) | 1.140957 / 1.452155 (-0.311197) | 1.204883 / 1.492716 (-0.287833) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091375 / 0.018006 (0.073368) | 0.285922 / 0.000490 (0.285432) | 0.000238 / 0.000200 (0.000038) | 0.000051 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021277 / 0.037411 (-0.016134) | 0.068853 / 0.014526 (0.054328) | 0.081002 / 0.176557 (-0.095555) | 0.120998 / 0.737135 (-0.616138) | 0.082741 / 0.296338 (-0.213598) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.299398 / 0.215209 (0.084189) | 2.909622 / 2.077655 (0.831967) | 1.624381 / 1.504120 (0.120261) | 1.501683 / 1.541195 (-0.039512) | 1.523045 / 1.468490 (0.054555) | 0.548960 / 4.584777 (-4.035817) | 2.413297 / 3.745712 (-1.332415) | 2.817852 / 5.269862 (-2.452010) | 1.754407 / 4.565676 (-2.811270) | 0.061912 / 0.424275 (-0.362363) | 0.004880 / 0.007607 (-0.002727) | 0.353989 / 0.226044 (0.127944) | 3.496147 / 2.268929 (1.227219) | 2.003026 / 55.444624 (-53.441598) | 1.702013 / 6.876477 (-5.174463) | 1.680935 / 2.142072 (-0.461137) | 0.630183 / 4.805227 (-4.175044) | 0.113786 / 6.500664 (-6.386878) | 0.040061 / 0.075469 (-0.035408) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.957218 / 1.841788 (-0.884569) | 11.914469 / 8.074308 (3.840160) | 10.488896 / 10.191392 (0.297504) | 0.129292 / 0.680424 (-0.551132) | 0.016603 / 0.534201 (-0.517598) | 0.287367 / 0.579283 (-0.291916) | 0.271332 / 0.434364 (-0.163032) | 0.325577 / 0.540337 (-0.214761) | 0.560553 / 1.386936 (-0.826383) |

</details>
</details>

![](https://cml.dev/watermark.png#2d31e434bbeafdf6a70cb80539342d8fe5f5fd27 "CML watermark")