> Do you perhaps know what the actual scale is?

Literally every single Cloudflare edge server that processes requests runs this proxy. Based on other things you said, it seems like your mental model of how Cloudflare's network runs is inaccurate. Mapping pingora origin requests/s to number of servers is difficult because of how tiered caching works & not all traffic is tiered cached and from what I remember pingora origin refers to all origin requests and a good amount of them aren't tiered (e.g. Workers outbound requests to the origin go through pingora origin & don't typically tier) which means those requests are being made directly from the edge (also tiering isn't like there's 100 servers - it spreads the load out evenly to avoid overwhelming any single server).

[1] suggests ballpark of 164 data centers and each data center is multiple machines each with multiple cores; [2] says the next gen servers have 128 cores while [3] suggests that the current gen has 96 and the previous has 48. We can assume that on average each server in a data center has 64 cores. Let's assume there are 20 machines in a data center on average (I don't remember the exact number so it's a wild guess). That comes out to an estimate of ~30k machines with ~2M total cores and this reverse proxy being responsible for ~20% of CPU time on average for every single machine (40k CPU cores from the article divided by 2M total cores across the network).

> It's a reverse HTTP proxy intended to be high-performance and low-latency so I don't imagine that the server that is running it has more important software to run.

In fact, the server that's running the proxy is also the server that's handling the inbound request as well as the Workers runtime, Cloudflare tunnels, firewall product, image optimizations, DDOS protections, etc etc etc & any other internal software they have to run to maintain their network (observability, keeping track of routing graphs, etc). So yes, CPU cycles are largely fungible and will be freed up for use on other tasks / to absorb growth as they acquire more traffic (& even if your server is underutilized this can help because it means the CPUs are switching contexts less often given how many threads are on these machines resulting in potentially better latencies at the extremes for neighboring services). There's more nuance here but this is a good first order approximation.

And yes, it's a micro optimization but clearly helpful at scale when you add up many teams doing micro optimizations. For example, here's a sqlite blog post [4] explaining how micro optimizations that don't even show up in real world systems (unlike this one which does) over an extended period of time yielded an overall 50% improvement.

Now how much is it worth to optimize global CPU usage by 0.34%? I don't know but Cloudflare does have a good way of converting CPU cycles saved to how much it saved in terms of number of servers they had to buy. Back of the envelope this added ~1k machines worth of compute time capacity to the network (0.34% * 2 million cores / 64 cores per machine). I think it's interesting the 0.34% is the actual CPU usage from their blog post and also comes up if you multiply the 20% of CPU time I estimated for the server by the 1.7% they said that hot function takes up but I think that's mostly an accident - they predicted 0.43% which suggests that the network is larger than 2M (~31k servers) and maybe they have 2.5M cores (~39k servers) and between offline servers and workloads varying due to time of day in practice they're probably closer to using ~2M cores total of their overall capacity. That optimization could still be worth doing since SRE's are responsible for maintaining fleet health while the Pingora team is responsible for their corner.

[1] https://community.cloudflare.com/t/how-many-datacenters-does...

[2] https://blog.cloudflare.com/cloudflare-gen-12-server-bigger-...

[3] https://blog.cloudflare.com/the-epyc-journey-continues-to-mi...

[4] https://sqlite-users.sqlite.narkive.com/CVRvSKBs/50-faster-t...