> Yes k8s has complexity, but it's a known and knowable complexity vs. the unknown complexities of the processes of yore.

TL;DR -> I see what you are saying, but there are methods to make your infrastructure consistent and keep the manual drift away that do not require k8s-level complexity. Both for monoliths and microservices.

Because we talk of the processes of yore, I will use a simple application architecture in the form of a web site that clients access that CNAMEs to a load balancer.

The load balancer distributes traffic to web servers, which each make back-end connections to a database.

This could also be load balancer to hosts that just run single-host instances of docker CE/EE as well - microservices can be done through this setup as well. If they are designed correctly, they shouldn't "need" to all be in one cluster, after all, these are durable microservices, not sharded bits of a monolith, right? :)

You have 4 basic tools, which with not a lot of thought can be used as a foundational platform for many, many workloads.

Aside from your codebase, you have the following (and/or their competitors)

Hashicorp Terraform

JetBrains TeamCity

Octopus Deploy

With a build server, a deployment platform, and a configuration management platform, you can get 98% of 'web scale' (all the scale you are really gonna get + plenty of runway for all but the FAANGs among us) for WAY less daily cognitive load. This applies to the DevOps team, the developers, the support cannon fodder (note if this is too true, and you are in leadership - please help this by empowering your cannon fodder), the DBAs, all of 'em.

1. Programmers check in their code to source

2. TeamCity runs unit/integration/regression tests. Pass? Publish to nuget feed for Octopus

3. Terraform sets up automated DB, web server, load balancer infrastructure for the servers, configured for whatever your company's definition of 'production server' is

4. Octopus deploy takes the newest build artifact from TC nuget package and deploys it to those servers.

In the above process, if you need to scale up:

1. modify the count of web servers in terraform

2. Terraform apply

3. wait for TF to complete its provisioning of new servers

4. wait for Octopus to recognize the new servers as deployment targets for your code (in this scenario, you write TF such that it will install the Octopus agent and set deployment target)

5. Deploy your code to the new servers

6. Have Terraform reconfigure the load balancer to expect these web servers to work but start in a disabled state

7. the load balancer will detect they are now passing health checks, enable them, and poof, they are serving traffic.

You have infrastructure as code, you have CICD, you have scalability for miles, and not a docker registry, helm chart, istio policy, or rancher needed. Best of all, your helpdesk, intern developers, even non-tech managers can cognitively follow the process.

Need to patch? Cattle, not pets says:

1. Terraform: reconfigure load balancer to drain connections on the boxes to patch

2. Terraform: reconfigure lb to remove from LB when done

3. Terraform: destroy the boxes

4. Terraform: spin me up same number of new boxes. Using packer or a fresh AMI, the new box will be patched, then have TF set configs, install etc. as done with the original boxes

5. Octopus: deploy code to all known web servers

6. TF: add boxes to load balancer

PS: sorry this grew into a wannabe blog post with handwave set to maximum.