Hi, thanks for this. It can be really hard to get a feel for how other people are thinking about things when one is really zoomed in on all of the little details. I'd love to start a conversation - nathan@docker.com if you're interested. In my opinion the tools are still feeling out where their niches and boundaries are, so that may contribute to the confusion. Networking is a big missing piece here, for instance, and I'm excited for the work that people like Socketplane (now part of Docker) and Weave are doing to help in this area.

If anyone here is curious to get a quick summary, the general gist of the three orchestration tools goes like this:

1.) Compose (nee Fig) emerged out of a need for a standard way to specify the run-time properties of a related group of containers. If you've played with Docker, you may have noticed that flags on run commands (and controlling the order of things like linked containers) spiral out of control fast, so Compose lets you write them all down in a file and manipulate your container groups with a little "shorthand".

2.) Machine is a tool for creating and managing hosts which run the Docker daemon (the bit which does the actual heavy lifting with containers). When they are created, you can run Docker commands against them remotely using a local Docker client binary! I got really interested in this project for two reasons.

One is that boot2docker-cli (a small Go program to help bootstrap and manage a Virtualbox VM with Docker installed, mostly for OSX and Windows users) was a pretty nice little tool, but I wanted to be able to have multiple VMs and not just one! Machine is great at this.

Likewise, Docker Machine now does the same type of trick for cloud VMs as well - so if I just want to kick up a Digital Ocean server for a few hours, run some containers, and then remove it, I can easily do so without leaving the command line. The interface for other cloud providers to do this is pretty similar too, so the workflow of using Docker across all supported clouds is the same.

I think it serves a lot better for dev/test right now than for prod but like I said the project is still feeling things out.

3.) It's easier to think of Swarm in terms of the end game IMO - and the end game is a Docker API endpoint (just one!) which works across an arbitrary number of hosts. Therefore, the user doesn't have to reason about multiple hosts on the backend - they just treat it like they're running containers on one big computer. If there aren't enough resources to run all the containers that you want to, you could just kick up a few more hosts, add them to the Swarm, and Swarm will fill them in as it goes (kind of like a load balancer can do for requests). It can do cool things like set anti-affinity on containers so that they never end up on the same host together, or never end up on a Red Hat host even if one is in your cluster, or pack them in as tightly as possible on each host.

I don't think the orchestration tools address all needs directly (and IMO they shouldn't)- logging, monitoring, and HTTP load balancing are good examples of things which don't really fall directly into their province directly.

I'd prefer to see the direction that they evolve in be one where they integrate really well with other tools and/or provide standard interfaces to platform-specific tech. For instance, it would be awesome for docker-machine to be able to import instances created with Ansible, Terraform, etc. and function as kind of a "REPL" to their "compiler" by allowing users to introspect the state of their running containers. Or for Swarm to act as a front-end for ECS. Speaking of, to answer your question:

> Why do I need Swarm on ECS?

Lots of tools "speak Docker API", so being able to point them at a Docker API and not care that it is ECS on the backend could be really useful. Additionally, ideally it would free you from being tied to the Amazon API or tools if you either don't want to learn them (since that knowledge is only useful for doing AWS work) or if you decide down the line that using, say, Mesos on baremetal as a back-end would suit your needs better instead.

Anyway, hope that all helps :) I could go all day about this stuff.