Title: Support HTTP authentication in non-streaming mode
Author: albertvillanova (MEMBER)
State: closed
Created: 2024-07-30T09:25:49Z

Support HTTP authentication in non-streaming mode, by support passing HTTP storage_options in non-streaming mode.
- Note that currently, HTTP authentication is supported only in streaming mode.

For example, this is necessary if a remote HTTP host requires authentication to download the data.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7082). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005280 / 0.011353 (-0.006073) | 0.003726 / 0.011008 (-0.007282) | 0.067028 / 0.038508 (0.028520) | 0.030833 / 0.023109 (0.007724) | 0.256888 / 0.275898 (-0.019010) | 0.271252 / 0.323480 (-0.052228) | 0.003149 / 0.007986 (-0.004836) | 0.004031 / 0.004328 (-0.000298) | 0.051178 / 0.004250 (0.046927) | 0.042751 / 0.037052 (0.005699) | 0.268385 / 0.258489 (0.009896) | 0.295547 / 0.293841 (0.001706) | 0.030218 / 0.128546 (-0.098328) | 0.012033 / 0.075646 (-0.063613) | 0.206389 / 0.419271 (-0.212882) | 0.036227 / 0.043533 (-0.007306) | 0.258778 / 0.255139 (0.003639) | 0.276027 / 0.283200 (-0.007172) | 0.020309 / 0.141683 (-0.121374) | 1.109689 / 1.452155 (-0.342466) | 1.139979 / 1.492716 (-0.352738) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093615 / 0.018006 (0.075609) | 0.301279 / 0.000490 (0.300789) | 0.000207 / 0.000200 (0.000007) | 0.000048 / 0.000054 (-0.000007) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018697 / 0.037411 (-0.018715) | 0.062627 / 0.014526 (0.048101) | 0.075119 / 0.176557 (-0.101438) | 0.119960 / 0.737135 (-0.617175) | 0.074606 / 0.296338 (-0.221732) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.281042 / 0.215209 (0.065833) | 2.746232 / 2.077655 (0.668578) | 1.422351 / 1.504120 (-0.081769) | 1.290087 / 1.541195 (-0.251108) | 1.321067 / 1.468490 (-0.147423) | 0.727514 / 4.584777 (-3.857263) | 2.407086 / 3.745712 (-1.338626) | 2.914191 / 5.269862 (-2.355670) | 1.872206 / 4.565676 (-2.693471) | 0.079538 / 0.424275 (-0.344738) | 0.005250 / 0.007607 (-0.002357) | 0.335536 / 0.226044 (0.109491) | 3.324922 / 2.268929 (1.055994) | 1.790688 / 55.444624 (-53.653936) | 1.475738 / 6.876477 (-5.400739) | 1.492465 / 2.142072 (-0.649607) | 0.812342 / 4.805227 (-3.992885) | 0.135036 / 6.500664 (-6.365628) | 0.041484 / 0.075469 (-0.033985) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.948425 / 1.841788 (-0.893363) | 11.321564 / 8.074308 (3.247256) | 9.635661 / 10.191392 (-0.555731) | 0.142793 / 0.680424 (-0.537631) | 0.014988 / 0.534201 (-0.519213) | 0.300209 / 0.579283 (-0.279074) | 0.262303 / 0.434364 (-0.172061) | 0.337927 / 0.540337 (-0.202411) | 0.427962 / 1.386936 (-0.958975) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005664 / 0.011353 (-0.005689) | 0.003946 / 0.011008 (-0.007062) | 0.050034 / 0.038508 (0.011526) | 0.031652 / 0.023109 (0.008543) | 0.281139 / 0.275898 (0.005241) | 0.299203 / 0.323480 (-0.024277) | 0.004332 / 0.007986 (-0.003653) | 0.002769 / 0.004328 (-0.001560) | 0.048336 / 0.004250 (0.044086) | 0.039744 / 0.037052 (0.002692) | 0.289344 / 0.258489 (0.030855) | 0.320470 / 0.293841 (0.026629) | 0.032372 / 0.128546 (-0.096174) | 0.012090 / 0.075646 (-0.063557) | 0.060838 / 0.419271 (-0.358433) | 0.034227 / 0.043533 (-0.009306) | 0.275007 / 0.255139 (0.019868) | 0.293455 / 0.283200 (0.010256) | 0.017203 / 0.141683 (-0.124480) | 1.141577 / 1.452155 (-0.310578) | 1.176761 / 1.492716 (-0.315955) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093562 / 0.018006 (0.075556) | 0.302695 / 0.000490 (0.302205) | 0.000215 / 0.000200 (0.000015) | 0.000044 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022638 / 0.037411 (-0.014774) | 0.078788 / 0.014526 (0.064262) | 0.088474 / 0.176557 (-0.088082) | 0.128421 / 0.737135 (-0.608714) | 0.089297 / 0.296338 (-0.207041) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.302669 / 0.215209 (0.087459) | 2.963855 / 2.077655 (0.886200) | 1.600053 / 1.504120 (0.095933) | 1.461456 / 1.541195 (-0.079739) | 1.469877 / 1.468490 (0.001387) | 0.725752 / 4.584777 (-3.859025) | 0.968970 / 3.745712 (-2.776742) | 2.910502 / 5.269862 (-2.359359) | 1.902762 / 4.565676 (-2.662914) | 0.079977 / 0.424275 (-0.344298) | 0.005582 / 0.007607 (-0.002025) | 0.351626 / 0.226044 (0.125581) | 3.520593 / 2.268929 (1.251664) | 1.968950 / 55.444624 (-53.475675) | 1.662190 / 6.876477 (-5.214286) | 1.677909 / 2.142072 (-0.464163) | 0.791541 / 4.805227 (-4.013687) | 0.134647 / 6.500664 (-6.366017) | 0.040687 / 0.075469 (-0.034782) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.028885 / 1.841788 (-0.812903) | 11.928358 / 8.074308 (3.854050) | 10.199165 / 10.191392 (0.007773) | 0.142930 / 0.680424 (-0.537493) | 0.016479 / 0.534201 (-0.517722) | 0.302993 / 0.579283 (-0.276290) | 0.128878 / 0.434364 (-0.305486) | 0.342591 / 0.540337 (-0.197747) | 0.456735 / 1.386936 (-0.930201) |

</details>
</details>

![](https://cml.dev/watermark.png#d298f5549893228c03e9e3a42727327cb83f3dff "CML watermark")