Title: Retrieve cached datasets that were pushed to hub when offline
Author: lhoestq (MEMBER)
State: closed
Created: 2023-11-29T16:56:15Z

I drafted the logic to retrieve a no-script dataset in the cache.
For example it can reload datasets that were pushed to hub if they exist in the cache.

example:

```python
>>> Dataset.from_dict({"a": [1, 2]}).push_to_hub("lhoestq/tmp")
>>> load_dataset("lhoestq/tmp")
DatasetDict({
    train: Dataset({
        features: ['a'],
        num_rows: 2
    })
})
```

and later, without connection:

```python
>>> load_dataset("lhoestq/tmp")
Using the latest cached version of the dataset from /Users/quentinlhoest/.cache/huggingface/datasets/lhoestq___tmp/*/*/0b3caccda1725efb(last modified on Wed Nov 29 16:50:27 2023) since it couldn't be found locally at lhoestq/tmp.
DatasetDict({
    train: Dataset({
        features: ['a'],
        num_rows: 2
    })
})
```

fix https://github.com/huggingface/datasets/issues/3547

## Implementation details (EDITED)

I continued in https://github.com/huggingface/datasets/pull/6493, see the changes there

TODO:
- [x] tests
- [ ] compatible with https://github.com/huggingface/datasets/pull/6458

Comments:
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005292 / 0.011353 (-0.006061) | 0.003811 / 0.011008 (-0.007197) | 0.064912 / 0.038508 (0.026404) | 0.061199 / 0.023109 (0.038090) | 0.242953 / 0.275898 (-0.032945) | 0.271789 / 0.323480 (-0.051691) | 0.003994 / 0.007986 (-0.003991) | 0.002723 / 0.004328 (-0.001606) | 0.049952 / 0.004250 (0.045701) | 0.039489 / 0.037052 (0.002437) | 0.261143 / 0.258489 (0.002654) | 0.288800 / 0.293841 (-0.005041) | 0.028130 / 0.128546 (-0.100416) | 0.010724 / 0.075646 (-0.064922) | 0.208218 / 0.419271 (-0.211054) | 0.036224 / 0.043533 (-0.007309) | 0.247189 / 0.255139 (-0.007950) | 0.274702 / 0.283200 (-0.008498) | 0.019714 / 0.141683 (-0.121969) | 1.134853 / 1.452155 (-0.317301) | 1.192655 / 1.492716 (-0.300062) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.096391 / 0.018006 (0.078385) | 0.303802 / 0.000490 (0.303312) | 0.000219 / 0.000200 (0.000019) | 0.000052 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019530 / 0.037411 (-0.017881) | 0.061588 / 0.014526 (0.047062) | 0.075122 / 0.176557 (-0.101434) | 0.120980 / 0.737135 (-0.616155) | 0.075807 / 0.296338 (-0.220532) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.281672 / 0.215209 (0.066463) | 2.779884 / 2.077655 (0.702229) | 1.502026 / 1.504120 (-0.002094) | 1.369474 / 1.541195 (-0.171721) | 1.402694 / 1.468490 (-0.065796) | 0.559120 / 4.584777 (-4.025657) | 2.355320 / 3.745712 (-1.390393) | 2.823987 / 5.269862 (-2.445875) | 1.763888 / 4.565676 (-2.801788) | 0.061715 / 0.424275 (-0.362560) | 0.005015 / 0.007607 (-0.002592) | 0.342669 / 0.226044 (0.116625) | 3.360651 / 2.268929 (1.091722) | 1.887277 / 55.444624 (-53.557348) | 1.555613 / 6.876477 (-5.320864) | 1.614126 / 2.142072 (-0.527946) | 0.643797 / 4.805227 (-4.161430) | 0.118365 / 6.500664 (-6.382299) | 0.042596 / 0.075469 (-0.032873) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.951383 / 1.841788 (-0.890405) | 13.169812 / 8.074308 (5.095504) | 10.772460 / 10.191392 (0.581068) | 0.133248 / 0.680424 (-0.547176) | 0.014597 / 0.534201 (-0.519604) | 0.289758 / 0.579283 (-0.289525) | 0.266324 / 0.434364 (-0.168040) | 0.334811 / 0.540337 (-0.205526) | 0.445566 / 1.386936 (-0.941370) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005668 / 0.011353 (-0.005684) | 0.003583 / 0.011008 (-0.007425) | 0.050681 / 0.038508 (0.012173) | 0.063244 / 0.023109 (0.040135) | 0.279624 / 0.275898 (0.003726) | 0.308030 / 0.323480 (-0.015450) | 0.004160 / 0.007986 (-0.003826) | 0.002633 / 0.004328 (-0.001696) | 0.048475 / 0.004250 (0.044225) | 0.043106 / 0.037052 (0.006054) | 0.283678 / 0.258489 (0.025189) | 0.309730 / 0.293841 (0.015889) | 0.030290 / 0.128546 (-0.098256) | 0.011112 / 0.075646 (-0.064534) | 0.058234 / 0.419271 (-0.361038) | 0.033553 / 0.043533 (-0.009979) | 0.279902 / 0.255139 (0.024763) | 0.298041 / 0.283200 (0.014841) | 0.019367 / 0.141683 (-0.122316) | 1.142438 / 1.452155 (-0.309717) | 1.197305 / 1.492716 (-0.295411) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090875 / 0.018006 (0.072869) | 0.301174 / 0.000490 (0.300685) | 0.000216 / 0.000200 (0.000016) | 0.000053 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021544 / 0.037411 (-0.015867) | 0.071371 / 0.014526 (0.056846) | 0.080821 / 0.176557 (-0.095736) | 0.120054 / 0.737135 (-0.617082) | 0.082611 / 0.296338 (-0.213728) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.293787 / 0.215209 (0.078578) | 2.862610 / 2.077655 (0.784955) | 1.597282 / 1.504120 (0.093162) | 1.485094 / 1.541195 (-0.056101) | 1.507384 / 1.468490 (0.038893) | 0.558470 / 4.584777 (-4.026307) | 2.414137 / 3.745712 (-1.331575) | 2.863342 / 5.269862 (-2.406520) | 1.776973 / 4.565676 (-2.788704) | 0.062296 / 0.424275 (-0.361979) | 0.004954 / 0.007607 (-0.002653) | 0.346037 / 0.226044 (0.119993) | 3.441864 / 2.268929 (1.172935) | 1.969842 / 55.444624 (-53.474783) | 1.714878 / 6.876477 (-5.161599) | 1.738141 / 2.142072 (-0.403931) | 0.645929 / 4.805227 (-4.159298) | 0.117332 / 6.500664 (-6.383332) | 0.041963 / 0.075469 (-0.033507) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.983229 / 1.841788 (-0.858559) | 13.186932 / 8.074308 (5.112624) | 11.220549 / 10.191392 (1.029157) | 0.142105 / 0.680424 (-0.538319) | 0.015210 / 0.534201 (-0.518991) | 0.290055 / 0.579283 (-0.289228) | 0.274513 / 0.434364 (-0.159851) | 0.346834 / 0.540337 (-0.193504) | 0.575897 / 1.386936 (-0.811039) |

</details>
</details>

![](https://cml.dev/watermark.png#d3c0694d0c47a64a3cab5d468b4d9575ad7b1d96 "CML watermark")


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6459). All of your documentation changes will be reflected on that endpoint.

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005308 / 0.011353 (-0.006045) | 0.003135 / 0.011008 (-0.007873) | 0.061820 / 0.038508 (0.023312) | 0.052005 / 0.023109 (0.028895) | 0.233507 / 0.275898 (-0.042391) | 0.257790 / 0.323480 (-0.065690) | 0.002848 / 0.007986 (-0.005138) | 0.002645 / 0.004328 (-0.001683) | 0.048379 / 0.004250 (0.044128) | 0.038320 / 0.037052 (0.001268) | 0.245470 / 0.258489 (-0.013019) | 0.274854 / 0.293841 (-0.018987) | 0.027335 / 0.128546 (-0.101211) | 0.010349 / 0.075646 (-0.065297) | 0.205872 / 0.419271 (-0.213400) | 0.035896 / 0.043533 (-0.007637) | 0.241645 / 0.255139 (-0.013494) | 0.260033 / 0.283200 (-0.023167) | 0.020325 / 0.141683 (-0.121358) | 1.116768 / 1.452155 (-0.335387) | 1.188067 / 1.492716 (-0.304649) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092622 / 0.018006 (0.074616) | 0.302663 / 0.000490 (0.302173) | 0.000227 / 0.000200 (0.000027) | 0.000051 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018633 / 0.037411 (-0.018778) | 0.060117 / 0.014526 (0.045592) | 0.072713 / 0.176557 (-0.103844) | 0.119955 / 0.737135 (-0.617180) | 0.074698 / 0.296338 (-0.221640) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.277157 / 0.215209 (0.061948) | 2.699650 / 2.077655 (0.621995) | 1.413625 / 1.504120 (-0.090494) | 1.295900 / 1.541195 (-0.245295) | 1.306280 / 1.468490 (-0.162210) | 0.555354 / 4.584777 (-4.029423) | 2.386866 / 3.745712 (-1.358847) | 2.794069 / 5.269862 (-2.475793) | 1.736275 / 4.565676 (-2.829401) | 0.061812 / 0.424275 (-0.362464) | 0.004957 / 0.007607 (-0.002650) | 0.334533 / 0.226044 (0.108488) | 3.251096 / 2.268929 (0.982168) | 1.768193 / 55.444624 (-53.676431) | 1.473752 / 6.876477 (-5.402724) | 1.476320 / 2.142072 (-0.665753) | 0.642485 / 4.805227 (-4.162742) | 0.116986 / 6.500664 (-6.383678) | 0.042083 / 0.075469 (-0.033386) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.941364 / 1.841788 (-0.900424) | 11.587408 / 8.074308 (3.513100) | 10.500198 / 10.191392 (0.308806) | 0.129126 / 0.680424 (-0.551298) | 0.015206 / 0.534201 (-0.518995) | 0.286580 / 0.579283 (-0.292703) | 0.263566 / 0.434364 (-0.170798) | 0.331662 / 0.540337 (-0.208676) | 0.431423 / 1.386936 (-0.955513) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005151 / 0.011353 (-0.006202) | 0.003425 / 0.011008 (-0.007583) | 0.049301 / 0.038508 (0.010793) | 0.052005 / 0.023109 (0.028895) | 0.289594 / 0.275898 (0.013696) | 0.312630 / 0.323480 (-0.010849) | 0.003988 / 0.007986 (-0.003998) | 0.002705 / 0.004328 (-0.001624) | 0.048529 / 0.004250 (0.044279) | 0.039645 / 0.037052 (0.002592) | 0.293430 / 0.258489 (0.034941) | 0.311697 / 0.293841 (0.017856) | 0.029044 / 0.128546 (-0.099502) | 0.010282 / 0.075646 (-0.065364) | 0.057641 / 0.419271 (-0.361630) | 0.032733 / 0.043533 (-0.010800) | 0.293553 / 0.255139 (0.038414) | 0.308850 / 0.283200 (0.025651) | 0.018452 / 0.141683 (-0.123231) | 1.147931 / 1.452155 (-0.304224) | 1.173093 / 1.492716 (-0.319623) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.100862 / 0.018006 (0.082856) | 0.309286 / 0.000490 (0.308796) | 0.000223 / 0.000200 (0.000023) | 0.000052 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021365 / 0.037411 (-0.016046) | 0.068987 / 0.014526 (0.054461) | 0.081092 / 0.176557 (-0.095465) | 0.119852 / 0.737135 (-0.617283) | 0.082850 / 0.296338 (-0.213489) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.288477 / 0.215209 (0.073268) | 2.833766 / 2.077655 (0.756111) | 1.576670 / 1.504120 (0.072550) | 1.431643 / 1.541195 (-0.109552) | 1.442132 / 1.468490 (-0.026358) | 0.556079 / 4.584777 (-4.028698) | 2.465042 / 3.745712 (-1.280670) | 2.786329 / 5.269862 (-2.483532) | 1.779428 / 4.565676 (-2.786249) | 0.062278 / 0.424275 (-0.361997) | 0.004867 / 0.007607 (-0.002740) | 0.348444 / 0.226044 (0.122399) | 3.389824 / 2.268929 (1.120896) | 1.919141 / 55.444624 (-53.525484) | 1.635411 / 6.876477 (-5.241066) | 1.654869 / 2.142072 (-0.487204) | 0.634467 / 4.805227 (-4.170761) | 0.114330 / 6.500664 (-6.386334) | 0.039900 / 0.075469 (-0.035569) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.970851 / 1.841788 (-0.870937) | 11.951660 / 8.074308 (3.877352) | 10.571115 / 10.191392 (0.379723) | 0.131040 / 0.680424 (-0.549384) | 0.015299 / 0.534201 (-0.518902) | 0.287851 / 0.579283 (-0.291432) | 0.278366 / 0.434364 (-0.155998) | 0.326468 / 0.540337 (-0.213870) | 0.552288 / 1.386936 (-0.834648) |

</details>
</details>

![](https://cml.dev/watermark.png#8214ff2a9f706427669a6c2a01ccabffa5bf0d2b "CML watermark")