I've become convinced over the years that upper management doesn't really care about 'estimates'. What they care about 'commitments'.

Every team I've been on has had upper management drive team leaders to get 'commitments'. They want some form of emotional investment in the work. That way, when scope has arbitrarily changed or a massive problem has lead to run over, they can stare at you like a whipped puppy-dog and say "I know it's 10PM and you want to get home to your wife, but you promised us, you committed, to getting this done on time..."

Pure emotional exploitation. The ideals of agile are laudable. I was the excited bannerman of my first agile experience and I read the manifesto with a lot of hope. However high-minded agile is, the language is too easily co-oped into the worst sins of management. In many places it exists solely to blackmail free overtime out of engineers, get away with micro-management, and ride engineers with false deadlines under a guise of hipness and modernity.

Estimates are just there for a false sense of stakeholding, or at best a ballpark cap so you don't work yourself into suicide. They can't be made accurately even if we could estimate properly, because all estimates are constantly second guessed by people who are making judgmental jokes about sandbagging, or asking "Are you really sure it'll take that long?" in voices that practically scream "chilling effect". More than a few possibly realistic estimates are torpedo'd by managers who are looking at their roadmap and tsking about how they don't line up.

I would even say accurate estimates are actively undermined. I've worked at companies which have positive work images abroad, and even there any team which was on time found either its sprint budget slashed, or scope increased until it was forced behind. The underlings won't do over-time if they don't feel the squeeze.

Story points, velocity, burndown, and any other metric you can think of that Agile courts are absolutely useless, because this industry refuses to come to terms with Goodhart's Law. I just watched a 10 million dollar project go up in flames. Their sprint graphs were great, and a constant buzz was going about the company about how these teams were setting the standard, meeting their metrics and goals consistently. Only recently did the CEO get down to business and figure out that all the metrics were bogus, and that for the past year all of the PMs and teams had been gaming the system to hide the fact that they were way behind schedule and over budget. The project was quietly scrapped and nothing changed. The shell games continue to actively sabotage data. I only know of the fallout because of who I sit next to.

Most of the benefits of agile seem to fail basic contact with humans unless they are backed by outstanding and visionary leadership. Most companies do not have that, yet they still find value in switching to Agile. Why? Maybe because even in total collapse it provides an unparalleled system for squeezing out more overtime, in my cynical opinion anyways.

I don't really what else to offer in place of Agile. I'm not that intelligent, but I don't really feel the industry can come up with a good development model of software development culture until it stops what it's doing and starts acknowledging problems arising from basic psychology, politics, and data gathering.