Title: Fix `fsspec.open` when using an HTTP proxy
Author: bryant1410 (CONTRIBUTOR)
State: closed
Created: 2023-03-21T15:23:29Z

Most HTTP(S) downloads from this library support proxy automatically by reading the `HTTP_PROXY` environment variable (et al.) because `requests` is widely used. However, in some parts of the code, `fsspec` is used, which in turn uses `aiohttp` for HTTP(S) requests (as opposed to `requests`), which in turn doesn't support reading proxy env variables by default. This PR enables reading them automatically.

Read [aiohttp docs on using proxies](https://docs.aiohttp.org/en/stable/client_advanced.html?highlight=trust_env#proxy-support).

For context, [the Python library requests](https://requests.readthedocs.io/en/latest/user/advanced/?highlight=http_proxy#proxies) and [the official Python library via `urllib.urlopen` support this automatically by default](https://docs.python.org/3/library/urllib.request.html#urllib.request.urlopen). Many (most common ones?) programs also do the same, including cURL, APT, Wget, and many others.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007980 / 0.011353 (-0.003373) | 0.005351 / 0.011008 (-0.005657) | 0.096325 / 0.038508 (0.057817) | 0.034204 / 0.023109 (0.011095) | 0.328080 / 0.275898 (0.052182) | 0.361519 / 0.323480 (0.038039) | 0.005954 / 0.007986 (-0.002032) | 0.004106 / 0.004328 (-0.000222) | 0.072827 / 0.004250 (0.068576) | 0.050522 / 0.037052 (0.013470) | 0.326975 / 0.258489 (0.068486) | 0.373180 / 0.293841 (0.079339) | 0.037024 / 0.128546 (-0.091522) | 0.012347 / 0.075646 (-0.063299) | 0.332341 / 0.419271 (-0.086931) | 0.050695 / 0.043533 (0.007162) | 0.328298 / 0.255139 (0.073159) | 0.352808 / 0.283200 (0.069608) | 0.101637 / 0.141683 (-0.040046) | 1.435172 / 1.452155 (-0.016982) | 1.529797 / 1.492716 (0.037080) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.305727 / 0.018006 (0.287721) | 0.583951 / 0.000490 (0.583462) | 0.011699 / 0.000200 (0.011499) | 0.000345 / 0.000054 (0.000290) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027917 / 0.037411 (-0.009495) | 0.107698 / 0.014526 (0.093173) | 0.120572 / 0.176557 (-0.055985) | 0.176066 / 0.737135 (-0.561069) | 0.125348 / 0.296338 (-0.170991) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.411980 / 0.215209 (0.196771) | 4.113135 / 2.077655 (2.035480) | 1.868725 / 1.504120 (0.364605) | 1.677422 / 1.541195 (0.136227) | 1.796759 / 1.468490 (0.328269) | 0.701957 / 4.584777 (-3.882820) | 3.830742 / 3.745712 (0.085030) | 2.170444 / 5.269862 (-3.099418) | 1.345097 / 4.565676 (-3.220580) | 0.086661 / 0.424275 (-0.337614) | 0.013073 / 0.007607 (0.005466) | 0.519150 / 0.226044 (0.293106) | 5.193447 / 2.268929 (2.924518) | 2.391155 / 55.444624 (-53.053470) | 2.076610 / 6.876477 (-4.799867) | 2.245557 / 2.142072 (0.103484) | 0.846496 / 4.805227 (-3.958731) | 0.169246 / 6.500664 (-6.331418) | 0.066360 / 0.075469 (-0.009109) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.196344 / 1.841788 (-0.645444) | 15.640363 / 8.074308 (7.566055) | 14.936144 / 10.191392 (4.744752) | 0.163613 / 0.680424 (-0.516811) | 0.017900 / 0.534201 (-0.516301) | 0.425377 / 0.579283 (-0.153906) | 0.431119 / 0.434364 (-0.003245) | 0.513669 / 0.540337 (-0.026669) | 0.592970 / 1.386936 (-0.793966) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007958 / 0.011353 (-0.003395) | 0.005707 / 0.011008 (-0.005301) | 0.075377 / 0.038508 (0.036869) | 0.037126 / 0.023109 (0.014016) | 0.344589 / 0.275898 (0.068691) | 0.381060 / 0.323480 (0.057580) | 0.006592 / 0.007986 (-0.001393) | 0.004479 / 0.004328 (0.000151) | 0.074456 / 0.004250 (0.070206) | 0.054087 / 0.037052 (0.017035) | 0.344942 / 0.258489 (0.086453) | 0.393174 / 0.293841 (0.099333) | 0.037926 / 0.128546 (-0.090620) | 0.012638 / 0.075646 (-0.063009) | 0.087743 / 0.419271 (-0.331529) | 0.050081 / 0.043533 (0.006548) | 0.340406 / 0.255139 (0.085267) | 0.361487 / 0.283200 (0.078287) | 0.108546 / 0.141683 (-0.033137) | 1.424626 / 1.452155 (-0.027529) | 1.553958 / 1.492716 (0.061242) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.329922 / 0.018006 (0.311916) | 0.523239 / 0.000490 (0.522749) | 0.012164 / 0.000200 (0.011964) | 0.000137 / 0.000054 (0.000082) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031935 / 0.037411 (-0.005477) | 0.115680 / 0.014526 (0.101154) | 0.130062 / 0.176557 (-0.046494) | 0.180679 / 0.737135 (-0.556457) | 0.135548 / 0.296338 (-0.160790) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.429648 / 0.215209 (0.214439) | 4.303342 / 2.077655 (2.225687) | 1.999395 / 1.504120 (0.495275) | 1.810354 / 1.541195 (0.269160) | 1.963132 / 1.468490 (0.494642) | 0.701654 / 4.584777 (-3.883122) | 3.844687 / 3.745712 (0.098975) | 2.153425 / 5.269862 (-3.116436) | 1.351541 / 4.565676 (-3.214135) | 0.086292 / 0.424275 (-0.337983) | 0.012491 / 0.007607 (0.004883) | 0.523144 / 0.226044 (0.297099) | 5.243283 / 2.268929 (2.974355) | 2.465849 / 55.444624 (-52.978775) | 2.154505 / 6.876477 (-4.721972) | 2.245500 / 2.142072 (0.103428) | 0.838902 / 4.805227 (-3.966326) | 0.169441 / 6.500664 (-6.331223) | 0.065631 / 0.075469 (-0.009838) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.262175 / 1.841788 (-0.579612) | 15.424650 / 8.074308 (7.350342) | 15.000718 / 10.191392 (4.809326) | 0.186328 / 0.680424 (-0.494096) | 0.018076 / 0.534201 (-0.516125) | 0.433458 / 0.579283 (-0.145825) | 0.424213 / 0.434364 (-0.010151) | 0.546568 / 0.540337 (0.006231) | 0.643529 / 1.386936 (-0.743407) |

</details>
</details>

![](https://cml.dev/watermark.png#ea7298bf121d7ae8079f0a59deb67c2fa1d4df6a "CML watermark")