>The only unit of provisioning is the VM. Puppet/Salt/Ansible are highly static systems.

If you're operating the same script from the same image (say, a distro's official ISO), they should work effectively the same way. It is true that bootstrapping nodes may break, but unlikely. It shouldn't be common.

There's no reason something like Ansible couldn't be extended to build containers. Playbooks effectively perform the same function as Dockerfiles: they list commands to execute off of a base image. Playbooks are somewhat more flexible in that they're frequently used to address already-running machines.

For the record, push-centric configuration management like Chef and Puppet is much worse than Salt or Ansible, which can connect to arbitrary nodes and execute commands.

>Kubernetes smoothes over a lot of the rough parts of Docker: Ports, IPs, DNS/discovery, monitoring and volume management are all neatly captured by Kubernetes

Maybe we adopted too early but none of these things are "neatly" captured by Kubernetes. We have complex superstructures over all of them. They may be neater than running them in Docker alone, but they're much more complex than running against bare nodes.

Particularly, addressing application instances and getting stateful storage have been challenges in Kubernetes, just as they are in Docker. StatefulSets went stable in December (3 months ago) and I haven't tried them yet, so maybe those concerns are assuaged.

The ideal interface for k8s would be, essentially, an aware hypervisor. If each pod acted as a subnet and each container acted like a machine in that subnet, meaning it had an IP that was accessible according to external firewall and DNS rules and otherwise acted like a normal machine, that'd be much easier to accept. Neither Docker nor Kubernetes are amenable to this.

>It also handles exec'ing into an existing container.

The interface for this is practically the same as it is on Docker. You do `kubectl exec -it <pod> /bin/ash` instead of `docker exec -it <container_id> /bin/ash`. The only extra thing it handles is making the container appear local to the master, instead of requiring one to get on the correct kubelet.

>For me, the value proposition is the cloud. When I boot up N nodes, I know that Kubernetes will optimally pack them with my container, and if I boot up N+1 nodes, I know that the containers will spread to the new node, and back down again if I go down to N (either intentionally or accidentally).

The value in this being running more applications on a single cloud instance? If you have the instance/VM as the unit of orchestration, it works the same way: a coordinator (load balancer) divides the traffic among the units, and then stops transparently when you deregister or when a health check fails.

>Orchestration, scheduling, autoscaling etc. take care of this "cloud of containers", if you will, and since every container is an isolated, identical, versioned, stateless (for the most part!), fully redundant thing

"Orchestration, scheduling, autoscaling etc." refer, more or less, to the same thing: getting your applications running somewhere. Containers and k8s in no way have a monopoly on that functionality, and my experience has not been that they've significantly simplified it.

>I rest assured that what is running is what I deployed.

If running a playbook over a fresh node isn't good enough, you can use an image system to ensure it's a binary copy of what you built. There were several competing APIs for managing this in a platform-agnostic manner.

>Then there are all the other nice things, like rolling deploys, cordoning, self-healing, job scheduling, secret management, being able to boot up a parallel, different version of a stack without clobbering everything else, etc.

I know that k8s includes some logic to attempt to automate the failover/deployment process, and that is a nice bonus. Not without equivalents, but I believe that k8s is positioned to do this more reliably than other systems.

Otherwise, same story here; containers and k8s did not introduce these capabilities, and I'm not sure how it claims to improve them.

k8s seems necessary because it makes all of this that already had reasonable solutions on non-containers work well with containers. Thus, no containers, no real reason to use k8s.

>We run production and staging in the same Kubernetes cluster (different namespaces), and we can use things like resource requests/limits and node affinities to ensure that they're scheduled differently, but still using the same underlying hardware resources.

Setting request quotas and node affinities is not fun. In a busy environment, it's like managing a shared host; you have to fine-tune each application so that it doesn't go outside of its box, and this is not completely reliable if anything else is consuming resources (say, dockerd, which was recently running at 450% utilization on one of our instances).

>What pains are you referring to, specifically?

a) Redoing all applications so they can run without a permanent filesystem, so they can be prepared for pod termination at any moment, so they don't care about being individually addressable, and so on. This by itself is a big deal. Containers etc don't make sense for a stateful project, and there's no reason to assume that every project needs to be stateless. "Operators" have been suggested as CoreOS as a way to handle this, but that's a whole 'nother headache to add to the pile, requiring registration of third-party resource types and listeners to gracefully handle pod termination.

b) Creating usable Dockerfiles for everything and maintaining them. In our case, we converted to Alpine because it "made smaller images", which was also a pain point due to lots of things not being friendly with musl. This includes tons of troubleshooting, weird little tricks, injecting things into the right place in the Dockerfile to keep the image size minimized, trying to map ports and do all the esoteric invocations required by the Docker daemon when running and linking the containers, clearing out images and containers before they make your disk full since Docker doesn't know how to do this intelligently, and so forth.

c) Getting everything to talk to the things it needs through Kubernetes, which involves a lot of config and port mapping, whereas previously we only needed to say "Go to DNS name" within our LAN (firewall managed separately). Kubernetes sort of allows this, for a subset of containers, if they're in the right cluster and the right label has been applied or something. Ingress resources and controllers are filed under this umbrella.

d) Troubleshooting or analyzing the live environment is much more difficult due to all of the weird network shenanigans going on, the restricted context available within the container, lack of diagnostic tools, etc.

e) Kubernetes configs are long, esoteric, and rely on a complex web of interdependent k8s-specific objects. Deployment success/failure takes multiple steps to ascertain (deploy; get pods; describe pod if failure; look for relevant message and respond, which requires repeating the process).

f) Docker hanging and breaking all containers until host reboot; Docker eating all disk space during development (for images/builds, usually); Docker crashing randomly; etc.

Say you want to interactively test a change to something that's running in a Docker container. You can get into the container and edit the file in place (if you install vim or similar), but beware that restarting the process will delete your container and revert back to the image. Your options are a) edit the file in your local environment, rebuild the Dockerfile locally, re-run the image, re-trigger the event, observe bugs, and repeat; or b) circumvent by doing docker start <container_id>; docker exec -it <container_id> /your/shell; and running the processes, detaching from them, and editing the process in-container, retriggering the processes that would be fired by the CMD or ENTRYPOINT each time to reload the application; hope all of this works fine in the limited context provided by the container. Then, exiting the container, committing the image, pulling out your changes to your local environment where you can commit them, pushing to git and the image registry, and hoping that the Docker container, which hypothetically is supposed to work automatically everywhere, actually ends up doing that, which seems not to happen pretty often for something that's supposed to work everywhere.

OR you could spin up a test instance/VM/whatever, or set up a test version of the application on your local environment, BOTH of which can usually be done with the same Ansible playbook, and then you can edit normally without worrying about the filesystem disappearing or the repeated builds taking all your space and test normally without worrying about esoteric error messages due to the limited container environment, restricted sandbox permissions, or incorrect network flags on the container invocation. You run tests, commit the change, which is not at risk of disappearing, and move on.

Anyway, if you're already committed to Docker for whatever reason, Kubernetes is probably an OK way to generalize it and try to make it less bad. I just see no compelling reason to commit to either other than a significant concern of maximum utilization of cloud instances, which is a fine concern, but not in exchange for the extra cost incurred in k8s.

Maybe my opinion would change if I stepped away from our environment and tried to bootstrap my own "clean" k8s cluster and Dockerfiles, but my experiments with minikube have featured a lot of the same frustrations.

Everything just seems to take a lot longer this way, which makes sense because we've added many layers of complication on top of what was a previously well-defined workflow. Containers could be nice if they were more mature, but for now, there is every reason to stay away.