Repository: go
Issue: meta: high rate of 5XX errors from go.googlesource.com since 2021-09-08
Labels: NeedsInvestigation

```
#!watchflakes
post <- `https://go\.googlesource\.com.*: The requested URL returned error: 5\d\d`
```

Some of the `cmd/go` long-mode tests end up cloning repos controlled by the Go project, such as `golang.org/x/sys`.

Other long-mode tests send requests to a dedicated test server (`vcs-test.golang.org`), various GitHub repos, packages hosted on `rsc.io`, and `gopkg.in`. The failure rate against `vcs-test.golang.org` is, as far as I can tell, negligible. The GitHub and `gopkg.in` tests for the most part check integration with services outside the control of the Go project. The `rsc.io` tests have a low but nontrivial failure rate (filed as #49954), and could perhaps be migrated to `vcs-test.golang.org` if need be.

However, the dependency on `go.googlesource.com` for the `x` repos is not easy to avoid, and as far as I can tell its failure rate since September has dwarfed all of those other services. Many of these failures already result in long (2m+) hangs that could otherwise push the test over its deadline, so I don't think it would be viable to just add retries (#28194) — especially given that if the problem could be fixed by retrying, the server could presumably do that retry internally itself.

I would rather not lose coverage or add complexity to the tests by fetching the Go repos only through `proxy.golang.org` — @golang/release, is it viable to get a higher SLA on `go.googlesource.com`?

----

`greplogs --dashboard -md -l -e 'https://go\.googlesource\.com.* 5\d\d\b' --since=2021-01-01`

<details>

[2022-01-10T17:06:19-9e7ea35/linux-386-longtest](https://build.golang.org/log/7a66938da859f2af791aa59b659e194fa99a6883)
[2022-01-05T21:22:03-2b39d86/linux-386-longtest](https://build.golang.org/log/ff69a5907729f22cb45191008475cdfc11686726)
[2021-12-08T15:30:52-9fe77de/linux-amd64-longtest](https://build.golang.org/log/979d0dae853c847cd4df10a92750da053df2ef76)
[2021-12-03T21:23:11-b3e1fbf/linux-386-longtest](https://build.golang.org/log/fc868797275ed650696755b027af0339027b5291)
[2021-12-02T23:34:26-5f65520/linux-386-longtest](https://build.golang.org/log/3222f1fb71bff58015ddc5c7e16cfcb25b3e0c11)
[2021-12-02T22:03:11-d3bf627/linux-386-longtest](https://build.golang.org/log/2865a943b3456cf912e61a53c8e57cc3be879beb)
[2021-12-02T20:49:56-bbe1be5/linux-386-longtest](https://build.golang.org/log/cce0e3a4607efb432237929180018dfe28451e1c)
[2021-12-02T20:45:47-3c6295d/linux-386-longtest](https://build.golang.org/log/5ce5f25ec7b1c8ed89f76d3ffe376a6ff14cc234)
[2021-12-01T17:13:34-08ecdf7/linux-386-longtest](https://build.golang.org/log/08c6fa6204874347370a10a66c9b717d5ac95343)
[2021-11-30T20:04:58-7ccbcc9/linux-386-longtest](https://build.golang.org/log/663e3a378a86e9a3c1b5bcfc9c0cb96f40105a1c)
[2021-11-30T18:49:39-5f63f16/linux-amd64-longtest](https://build.golang.org/log/dca6d3787e4e2b900916ef494d917bcac3ea3c46)
[2021-11-30T18:09:02-931d80e/linux-386-longtest](https://build.golang.org/log/46cc7ba5b868da87dd7b9ec069531166c8ce48aa)
[2021-11-30T18:09:02-931d80e/linux-amd64-longtest](https://build.golang.org/log/d8bcf863c33942f9a10ff0aa64ba9f533f139e88)
[2021-11-29T22:02:48-2f6d382/linux-386-longtest](https://build.golang.org/log/21142745092e8c9df2755a514cb2bebee1de6ef5)
[2021-11-24T21:11:45-b77f5f9/linux-386-longtest](https://build.golang.org/log/771a7e2ac80c64555a356d063f00f705035424b2)
[2021-11-24T21:09:36-7e5331a/linux-amd64-longtest](https://build.golang.org/log/c7a9e7fcdf36a7359a29a1ad6f4ffdd0c737e605)
[2021-11-23T19:31:05-0f64c21/linux-amd64-longtest](https://build.golang.org/log/2fd960e69ed4765e1c31dc4989ae08b4a3db178c)
[2021-11-19T21:36:17-0f75a9e/linux-amd64-longtest](https://build.golang.org/log/13cd1377f833e7d30953d6f91ce797a9f1399e8d)
[2021-11-16T14:33:48-50dac3b/linux-386-longtest](https://build.golang.org/log/05a0e4a45a12544c0f2c608dbc89cc0ba65a1c4f)
[2021-11-11T20:20:42-3e94140/linux-386-longtest](https://build.golang.org/log/e0617ea588b0092d5898750876fe0e3aa1bb582e)
[2021-11-11T13:58:28-d76b1ac/linux-386-longtest](https://build.golang.org/log/0a2e17da3e77bd0247e6973e618eed01ee6a1850)
[2021-11-10T18:24:14-6406e09/linux-amd64-longtest](https://build.golang.org/log/78dc681d75ce8c2d562ad686095ee6fe84841984)
[2021-11-09T22:58:24-578ada4/linux-386-longtest](https://build.golang.org/log/b6feb8f5bc1122e55ab866004679b0d8df4306fa)
[2021-11-09T19:01:20-526b2ef/linux-386-longtest](https://build.golang.org/log/ffafd6f3fd80612eb2ddaea952ae1493d9ecf8da)
[2021-11-09T18:23:16-55e6e82/linux-amd64-longtest](https://build.golang.org/log/209423f5a9c9aa72766edf34f964501f3116a1e6)
[2021-11-08T16:44:14-7ee3f14/linux-amd64-longtest](https://build.golang.org/log/517d97dcc1eecc97f121e723a82822450d567f69)
[2021-11-05T17:47:28-6f32d20/linux-386-longtest](https://build.golang.org/log/3dfad4fd9af7ae2e0e641da10b27928caee058de)
[2021-11-05T00:52:04-1c4cfd8/linux-386-longtest](https://build.golang.org/log/0302ff50fa7c4924899279af0f5f4c5c41b29cfa)
[2021-11-04T21:52:36-8ad0a7e/linux-386-longtest](https://build.golang.org/log/3d0a6da4130abbb1818d8d38b7a7ae5955c651ce)
[2021-11-04T14:17:18-901bf29/linux-386-longtest](https://build.golang.org/log/80e6e0a88a6be84074cda7577d91941d7dde6e52)
[2021-11-03T20:30:17-7f2463c/linux-amd64-longtest](https://build.golang.org/log/271886da148e12d7158ca90eef5f3f6d9ccbcf7b)
[2021-11-02T13:43:24-4d7bf41/linux-386-longtest](https://build.golang.org/log/cedbc7d6aabaa12b2b0bca2aea876b563dfd575f)
[2021-11-01T19:42:30-5f1bcd1/linux-386-longtest](https://build.golang.org/log/c5f6ad9723c30e18872fe0b309ac406d0a77dd27)
[2021-11-01T15:55:29-0ae4c7f/linux-amd64-longtest](https://build.golang.org/log/31e4ddc2d3a151abcff0bc7a1a801cf6c20ba170)
[2021-10-29T19:35:47-5ec139f/linux-386-longtest](https://build.golang.org/log/1ecf5faba3d7d0f1de97720733996e4407b5071f)
[2021-10-29T18:56:29-6afdf01/linux-386-longtest](https://build.golang.org/log/e6ae9bfa7bb4f3450effb2026a8e520a1a9e0d82)
[2021-10-29T18:35:59-5ec2427/linux-386-longtest](https://build.golang.org/log/7b6d49edeb3ef0b9298fb92e73854db6e79fc61a)
[2021-10-29T18:35:20-9ac1ee2/linux-386-longtest](https://build.golang.org/log/2a1744a1cd2aa7437091d9a548d7a87c7cfde6cb)
[2021-10-29T18:23:03-413672f/linux-386-longtest](https://build.golang.org/log/6725cf7ec5aa196452a2dd2c507dec5e414703ed)
[2021-10-28T20:27:44-5e21728/linux-amd64-longtest](https://build.golang.org/log/33a20874ace7310f8bf01025c1afed7a9e0fce7d)
[2021-10-28T16:54:29-5fce1d9/linux-386-longtest](https://build.golang.org/log/8afde295d10ad7435cbb0886a924d8bd87476b54)
[2021-10-28T01:15:26-b2fe2eb/linux-386-longtest](https://build.golang.org/log/7898db4439e01772d402fb73110d096cc58d6113)
[2021-10-27T21:37:54-749f6e9/linux-386-longtest](https://build.golang.org/log/de149c44e68c3baa2139babe6624a0f0307305c2)
[2021-10-27T20:24:48-a155a30/linux-386-longtest](https://build.golang.org/log/7ca705aebd01f61b5ebf17e50e53b807382890b7)
[2021-10-27T20:03:17-68bd512/linux-386-longtest](https://build.golang.org/log/42c8a164f4112ab895990a68112f5f31606f4c08)
[2021-10-27T19:34:49-ffd2284/linux-386-longtest](https://build.golang.org/log/58ab62a0b2f910094614d73fc62d4face9018755)
[2021-10-27T19:18:20-514ebae/linux-386-longtest](https://build.golang.org/log/7dc12e2be836321dff3c3d7c3d10a4602d4b9bdc)
[2021-10-27T18:08:49-bb49eb3/linux-386-longtest](https://build.golang.org/log/bafea10a7c09e700e3b79c382c3fd793b33bed96)
[2021-10-26T20:12:32-f8779b9/linux-amd64-longtest](https://build.golang.org/log/7cd272cd2e6f9d06fb7685ffa5ccfd2160bca8ef)
[2021-10-26T17:10:45-76cef81/linux-amd64-longtest](https://build.golang.org/log/f076161dca65b6ac8756afb2dfeb72fd7fd9ff65)
[2021-10-26T14:24:17-283d8a3/linux-386-longtest](https://build.golang.org/log/b6c02746630070164017209493fcb42a5068960e)
[2021-10-25T17:46:41-9012996/linux-amd64-longtest](https://build.golang.org/log/ff0824b89e29f5d5e2d3128bf5ad5914a17b98af)
[2021-10-22T00:57:18-9ff91b9/linux-amd64-longtest](https://build.golang.org/log/e9c265339abf0a428627cc6287402e377432f7b1)
[2021-10-21T19:08:43-b8da7e4/linux-386-longtest](https://build.golang.org/log/aeaad96e0fc6a7efa921c4328e7a6587805c3537)
[2021-10-21T02:35:47-fca46d0/linux-386-longtest](https://build.golang.org/log/e06e46e6cbf30fae0995a20d9250a1250efcb31a)
[2021-10-12T20:20:41-d032b2b/linux-386-longtest](https://build.golang.org/log/ecae40b85cac1f58a791a6e102656ad7329e4c05)
[2021-10-12T14:32:53-36a265a/linux-386-longtest](https://build.golang.org/log/168a6e1c2bf2b0230d50b4995059180eb69e3b21)
[2021-10-05T19:28:36-7ae83c8/linux-386-longtest](https://build.golang.org/log/9eb66b205475c85ac4debf1e8d3d2a040c5d7890)
[2021-10-01T14:30:33-8d494b0/linux-386-longtest](https://build.golang.org/log/61e0675f445bd0e4006f9246df9c57b20bb63793)
[2021-09-27T19:42:34-315cec2/linux-386-longtest](https://build.golang.org/log/fd923fc9e870a820176dc18caeecbfb3d3dcad5a)
[2021-09-21T14:02:52-7d67f8d/linux-amd64-longtest](https://build.golang.org/log/9a136862b4e55486503200f9baa988d5f739d0be)
[2021-09-15T17:32:52-6196979/linux-386-longtest](https://build.golang.org/log/7bd06db324561aeaf6096858423ce9dbd073a29a)
[2021-09-15T17:32:52-6196979/linux-amd64-longtest](https://build.golang.org/log/f4b7bc2e139de8ca7d0e74e9f81cea7402de1cea)
[2021-09-15T16:32:27-72bb818/linux-386-longtest](https://build.golang.org/log/4aa7c1cfc5c06e5fe6d26254880eb42af55883fd)
[2021-09-15T16:32:27-72bb818/linux-amd64-longtest](https://build.golang.org/log/c3f66a40953c7ec2dda9b3768b5cfef1be8a98f4)
[2021-09-14T23:20:18-2933c45/linux-386-longtest](https://build.golang.org/log/a0e9570a1a877014cd877ef128c3d32b259eec39)
[2021-09-14T23:07:15-137543b/linux-amd64-longtest](https://build.golang.org/log/239b43dc209f6a72806543cf757fd3ac49c8646f)
[2021-09-14T23:03:28-3a72175/linux-amd64-longtest](https://build.golang.org/log/d9a9dd89a21843d4356f00cb9f905b874d556790)
[2021-09-13T21:48:41-42057e9/linux-386-longtest](https://build.golang.org/log/29fc2372596903820933faaeffc0ec7eaa712758)
[2021-09-13T20:42:44-81a4fe6/linux-386-longtest](https://build.golang.org/log/64257e242c8d93d6602878a10748428f4b8330cf)
[2021-09-09T01:25:40-d4579c2-d62866e/darwin-arm64-11_0-toothrot](https://build.golang.org/log/41cfd08178129a2cf9aa5c1d26f9a01b672fa02b)
[2021-09-08T16:59:47-d419f9c/linux-amd64-longtest](https://build.golang.org/log/55a291de9ca1946c67749912d8656928ce8b7e1b)
[2021-09-08T16:19:36-409434d/linux-386-longtest](https://build.golang.org/log/4261e22f626b6f7b07b56270a52c359ce1a04885)
[2021-05-26T22:43:54-1d5298d/linux-amd64-longtest](https://build.golang.org/log/dc5d44a0446e604f21df46be281b963ddf51c664)

</details>