> Well that does sound like a lot, I hope you can negotiate for a raise at some point.

Oh, certainly. Though right now i'm more concerned with making the lives of my colleagues more easy and actually shipping software that works in the end.

> Long-term I'm not sure what you mean building solutions for the next decade, it seems quite hard to design something perfectly in hindsight.

That is a very fair point, but in my eyes trying to create stable software is a worthy pursuit in many cases regardless!

Perfection is probably never going to be achievable, but the difference between creating something that will break in 3 months versus 3 years is probably pretty impactful (e.g. using bleeding edge/immature frameworks which will necessitate a rewrite, or will have breaking changes). I'd say that both the technologies used and the architecture of the solution can impact this greatly.

What's also useful is thinking about limiting the fallout when something does break, e.g. making the system modular enough that it can be fixed, updated or changed bit by bit, as well as thinking about scaling, at least a little bit. I'm not saying that everything needs to be microservices, but having a tightly coupled codebase can and probably will create problems down the road.

> And of course you have to compare the cost of it to just slapping something in a VM and putting a firewall over it and calling it a day...

That's also a reasonable take. Of course, costs aren't always the only consideration - if my blog or personal site goes down, the impact probably isn't too bad, whereas if a governmental health care system goes down, many people won't be able to receive the services that they need in a timely manner. The latter is probably worth the investment, both monetary and in regards to consideration about all of the stuff mentioned before.

I've actually experienced what it's like to see queues building up in one such institution, with the medical personnel also being frustrated, all because a system component in a data center somewhere had been neglected and had DB connection pooling issues, leading to a complete standstill.

I was called in to fix that external project and somehow managed to do it by ripping out the DB pooling solution and replacing it with another one. That was problematic when the actual codebase was badly commented and there were no proper tests to speak of in place and the failure mode was both odd and hard to debug.

In the end, i didn't manage to repair the old pooling solution without swapping it out entirely, because one moment everything was fine, the next threads just got stuck waiting one by one, with no errors or debug messages, regardless of the config, whereas breakpoints didn't lead to anything useful either.

In short, there are systems out there that should be as stable as possible and tolerant of failure, given that not all of them will receive the constant love and care that they deserve.

I'd also like to link this lovely article and talk: http://boringtechnology.club/

Especially the bit towards the bottom about the "known unknowns" and "unknown unknowns" - knowing what you can and can't do with a particular piece of technology and its characteristics is probably a good thing.