Title: [WebDataset] Support compressed files
Author: lhoestq (MEMBER)
State: closed
Created: 2024-05-29T14:19:06Z


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6931). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005362 / 0.011353 (-0.005991) | 0.003969 / 0.011008 (-0.007039) | 0.063390 / 0.038508 (0.024882) | 0.030814 / 0.023109 (0.007705) | 0.246891 / 0.275898 (-0.029007) | 0.271047 / 0.323480 (-0.052432) | 0.004036 / 0.007986 (-0.003950) | 0.002732 / 0.004328 (-0.001597) | 0.049466 / 0.004250 (0.045216) | 0.047227 / 0.037052 (0.010175) | 0.255978 / 0.258489 (-0.002511) | 0.297956 / 0.293841 (0.004115) | 0.028641 / 0.128546 (-0.099905) | 0.010510 / 0.075646 (-0.065136) | 0.204268 / 0.419271 (-0.215004) | 0.037093 / 0.043533 (-0.006440) | 0.247287 / 0.255139 (-0.007852) | 0.263830 / 0.283200 (-0.019370) | 0.018335 / 0.141683 (-0.123348) | 1.116074 / 1.452155 (-0.336081) | 1.182589 / 1.492716 (-0.310128) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094435 / 0.018006 (0.076429) | 0.310422 / 0.000490 (0.309932) | 0.000215 / 0.000200 (0.000015) | 0.000044 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019220 / 0.037411 (-0.018192) | 0.062090 / 0.014526 (0.047564) | 0.074511 / 0.176557 (-0.102046) | 0.121825 / 0.737135 (-0.615310) | 0.075406 / 0.296338 (-0.220933) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.281185 / 0.215209 (0.065976) | 2.770157 / 2.077655 (0.692502) | 1.472095 / 1.504120 (-0.032025) | 1.339342 / 1.541195 (-0.201853) | 1.374621 / 1.468490 (-0.093869) | 0.566607 / 4.584777 (-4.018170) | 2.357642 / 3.745712 (-1.388070) | 2.735034 / 5.269862 (-2.534827) | 1.782779 / 4.565676 (-2.782897) | 0.063046 / 0.424275 (-0.361229) | 0.005015 / 0.007607 (-0.002592) | 0.336690 / 0.226044 (0.110646) | 3.360955 / 2.268929 (1.092027) | 1.804424 / 55.444624 (-53.640200) | 1.517334 / 6.876477 (-5.359143) | 1.665254 / 2.142072 (-0.476818) | 0.627185 / 4.805227 (-4.178042) | 0.114388 / 6.500664 (-6.386276) | 0.041788 / 0.075469 (-0.033681) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.975270 / 1.841788 (-0.866517) | 11.647633 / 8.074308 (3.573325) | 9.872873 / 10.191392 (-0.318519) | 0.141744 / 0.680424 (-0.538680) | 0.014524 / 0.534201 (-0.519677) | 0.286697 / 0.579283 (-0.292586) | 0.266837 / 0.434364 (-0.167527) | 0.328513 / 0.540337 (-0.211825) | 0.424676 / 1.386936 (-0.962260) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005654 / 0.011353 (-0.005699) | 0.004058 / 0.011008 (-0.006950) | 0.051030 / 0.038508 (0.012522) | 0.033085 / 0.023109 (0.009976) | 0.307532 / 0.275898 (0.031634) | 0.335672 / 0.323480 (0.012192) | 0.004244 / 0.007986 (-0.003742) | 0.002842 / 0.004328 (-0.001487) | 0.050131 / 0.004250 (0.045880) | 0.040709 / 0.037052 (0.003656) | 0.319514 / 0.258489 (0.061025) | 0.357153 / 0.293841 (0.063312) | 0.029014 / 0.128546 (-0.099532) | 0.010999 / 0.075646 (-0.064648) | 0.058789 / 0.419271 (-0.360482) | 0.033284 / 0.043533 (-0.010249) | 0.310783 / 0.255139 (0.055644) | 0.331466 / 0.283200 (0.048266) | 0.018998 / 0.141683 (-0.122685) | 1.138822 / 1.452155 (-0.313332) | 1.180731 / 1.492716 (-0.311985) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095725 / 0.018006 (0.077719) | 0.302788 / 0.000490 (0.302298) | 0.000206 / 0.000200 (0.000006) | 0.000056 / 0.000054 (0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023247 / 0.037411 (-0.014164) | 0.077619 / 0.014526 (0.063093) | 0.090489 / 0.176557 (-0.086067) | 0.132033 / 0.737135 (-0.605102) | 0.090964 / 0.296338 (-0.205374) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.297912 / 0.215209 (0.082703) | 2.954107 / 2.077655 (0.876452) | 1.591155 / 1.504120 (0.087035) | 1.469217 / 1.541195 (-0.071978) | 1.513315 / 1.468490 (0.044825) | 0.562728 / 4.584777 (-4.022049) | 0.960093 / 3.745712 (-2.785620) | 2.852106 / 5.269862 (-2.417756) | 1.861668 / 4.565676 (-2.704009) | 0.063530 / 0.424275 (-0.360745) | 0.005194 / 0.007607 (-0.002413) | 0.351116 / 0.226044 (0.125072) | 3.498787 / 2.268929 (1.229859) | 1.952223 / 55.444624 (-53.492401) | 1.696208 / 6.876477 (-5.180269) | 1.861650 / 2.142072 (-0.280422) | 0.653494 / 4.805227 (-4.151733) | 0.123797 / 6.500664 (-6.376868) | 0.042696 / 0.075469 (-0.032773) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.006657 / 1.841788 (-0.835131) | 12.659771 / 8.074308 (4.585463) | 10.672140 / 10.191392 (0.480748) | 0.143726 / 0.680424 (-0.536698) | 0.015895 / 0.534201 (-0.518306) | 0.285952 / 0.579283 (-0.293331) | 0.126078 / 0.434364 (-0.308286) | 0.325943 / 0.540337 (-0.214395) | 0.410774 / 1.386936 (-0.976162) |

</details>
</details>

![](https://cml.dev/watermark.png#88d53d1ae762bec6736fffb000e6540e52bf1998 "CML watermark")