Not sure you'll see this and I'm kind of stretching my memory here but...

It was a node.js project and the db was separate, and keep in mind my knowledge of AWS is/was limited. But then, this seems to be the target audience for EB, yes? I'm sure a lot of this could be fixed but...

1) Logging

It somehow took me significant time to figure out how I could send meaningful logs anywhere but poorly organized flat text files in S3 without adding some kind of logging sink to the code itself. Sure, I could've just plugged in Winston and spat out everything to MongoDB but the team couldn't understand why we couldn't just get logs out of EB. And yeah, we could could grep through the S3 bucket but that meant syncing the whole bucket and trawling through an absolute ton of log files with no more precision than your knowledge of grep and regex. Not the best thing.

I eventually dumped all the logs in to CloudWatch, but the documentation to do so was severely lacking (from my perspective) that dealing with CloudFormation template files made me consider ditching EB entirely -- If I'm dealing with this garbage anyway why don't I just go whole hog and get more control anyway?

Even after dumping the logs to CloudWatch, I was still "the guy" for doing any good searching of these logs. Maybe this was just a team quirk, but what they really wanted was logs in MongoDB or something they know, rather than something AWS specific. Again, a logging sink would've fixed this, but there were concerns about uncaught exceptions and attempting to send to MongoDB in an unknown state.

2) Deployment Time & "Health Checks"

The team was used to pulling the latest changes manually and restarting the node.js process. Introducing CI ironically increased friction as our deploys were taking an upwards of 10 minutes (20 if immutable). This doesn't sound like much of a problem in a traditional environment, but going from a "cowboy coder" environment to "why the hell does it take 20 minutes to deploy!?" was a stretch. My initial answer was "health checks!" but then we discovered that the health checks weren't actually doing much for us... I had erroneously assumed that part of the health check meant monitoring the responses of the web service and aborting deployment upon massive failure. We had a deploy that crashed and returned 500 on every request, yet somehow didn't abort. Part of this may have been because the developer involved in the bug did everything possible to prevent a "crash", instead of letting the process die upon unrecoverable error (part of a frontend JavaScrtip mindset I assume), but trying to sort out how to make the health check look for 100% 500's brought me even further in to "why not set it up myself?"

3) Deployment Failure

We hit a hard limit of deployments and had to clean out a list of artifacts we didn't even know existed before being allowed to deploy again. WTF? This is when the team started to really question the value EB was providing.

4) I ended up "the guy" anyway, so what does it matter?

The documentation was too dense for my team, and the quirks were plentiful enough that I ended up "the guy" anyway. One day I came in late and a deployment had gone wrong. EB had been rolling out bad code for 20 minutes and they didn't know how to fix it. I had to clean up the mess. That was really the "fuckit" moment for me. Hand managed EC2 instances wouldn't have caused this problem. Regardless of knowledge problems within the team itself, the whole point of adopting EB was to avoid needing infrastructure specific knowledge to scale & deploy. Now EB was doing more harm than good.

---

These are just the things I remember off-hand. I'm sure there's solutions to these problems, but it just wasn't worth it for me to investigate when really my purpose was just to write and ship code, not futz with the infrastructure. We would've run in to less problems with a naive, manual setup in the end.

You can ping me at rtannerf dot com if you want to know more. I'm sure I can dredge up a few more annoyances and clarification from an old coworker. Hope that helps.