If you're pushing messages to a message broker, you can trivially regulate delivery on most of them by checking queue depth and throwing errors or reducing the delivery rate if they grow beyond a certain threshold.

But if you genuinely have more messages than the workers can process, pretending you don't is not solving a problem. With the broker you have the choice to decide what size of queue is acceptable without building your own queue logic into the client. The only case where HTTP is generally preferable there is the case where a queue of any size is always unacceptable: If you need the message processed right now, or not at all.

> Deploying a new worker instance? With HTTP, you don't add it to the load balancer pool until it's ready. With an MQ, you have to tell the app to add itself.

I much prefer having the app adding itself - having the load balancer control delivery means it needs to probe for liveness, and the config needs to be updated whenever you bring resources up/down. If the endpoint can register itself, then that's one less hassle.

> Worker failed a health check? With HTTP, you take it out of the load balancer pool. With an MQ, you have to tell it to take itself out of the pool, and hope it can still do that even though it's sick.

With an MQ, it will stop pulling messages if it's not functioning. If it's functioning enough to keep pulling messages despite failing a health check, your app is broken - why was it not doing sufficient checks itself to know that it is not working? Why is there not a monitor on the same physical server with the capability to kill the offending process?

> Worker failing to process messages? With HTTP, there's a good chance you'll notice straight away because it's returning 500 errors. With an MQ, you have to hope your monitoring will detect the problem.

What is the difference between monitoring for 500 errors and monitoring for errors from an MQ? Something needs to generate those errors. Either you are monitoring the logs, or your app is sending responses directly. If you need/can use synchronous responses, either you are abusing a message broker for something it shouldn't be, or your app should be sending reply messages that the client needs to wait for (which may still be preferable if the processes are potentially very long running). There's no difference here.

> Workers loaded asymmetrically? With HTTP, the load balancer is in just the right place to, er, balance load (think of Apache mod_jk or mod_cluster). With an MQ, no component of the system has a global overview of load.

With workers pulling work out of a message broker, they self-regulate: The ones on the highest loaded machines will pull messages slower. Load is usually the wrong thing to worry about from the app perspective - processing rate is much more relevant, and a metric that's trivial to make available from your app. Latency is easily handled by imposing strict limits and killing / pausing workers that miss them.

> You can make the workers do the right thing in these scenarios even when using an MQ, but it's extra work.

Frankly, in most of your scenarios I feel the HTTP alternative is what poses the extra work. You're imposing extra steps to manage adding workers to the load balancers; you're adding complicated load monitoring instead of letting the workers self regulate which ones have capacity to process the messages just by virtue of which ones pull the next message first.

Brokers are not always the right choice. They're generally the wrong choice if you want synchronous, low latency end to end processing. But for many systems, different parts of the system may process different types of jobs at different rates, and with different scale out requirements. In those cases, introducing brokers means each component only has to worry about their specific task, as the overall data flow management gets separated out in a separate component monitoring the brokers and taking appropriate action (such as, for example, spinning up more VMs to run brokers at some point in the pipeline).