I think the problem is that software process engineering does not really include the skills necessary for transforming organizations. To be honest, I don't know of any really good software process person who thinks that a process will do anything on its own. The processes and practices are only half of the battle.

It gets even more complicated because, at least in my experience, software processes are really subtle and it is extremely easy to appear to be "doing the right thing", while at the same time failing miserably at it. Throw in the power politics, the bad actors who don't really care if something is successful or not (as long as they can extract money from it), and naive developers who are absolutely sure they know what to do but really don't, and it makes it very difficult.

It is difficult to be the one instigating process change. Such a role is fraught with danger. You will be attacked by the power politics people. You will be attacked by the money extracting people. You will be attacked by the naive developers who think they know better than you. You almost certainly will not be thanked by anyone (except very far down the road if you somehow manage to succeed... and even then probably nobody will remember it was you who started the ball rolling). And finally, because it is incredibly hard and the necessary skills are very difficult to acquire, you will almost certainly fail many, many times before you ever succeed.

Even worse, it's not something you can write down and say "Do X,Y, and Z and everyone will be a happy family working hard to ensure success". It is something that requires dedication to understanding every single individual who is involved in the development. You have to sit and listen to them, understand their goals and help them meet those goals (even if they aren't "Make a successful software product" and even if you personally think the person is unbearably awful). You have to devote yourself to finding common ground and bringing everyone together.

Don't get me wrong. The processes and practices are crucial to a successful organizational change. These practices, if well chosen, will support you as you influence, cajole, flatter, criticize and motivate the people around you. It is next to an impossible job and without these tools, you will almost certainly fail.

It is unfortunate that people dramatically underestimate the difficulty of cultural change in an organization. They write about the easy bits (the processes that support the change) and sweep the hard bits under the carpet. To be fair, I'm not sure what you would say that wouldn't sound like "business BS" to most engineers, though.

Anyway, most organizations do not have many people conscientious (possibly read "stupid") enough to attempt real, lasting cultural change. The vast majority that do try are often incredibly naive and lack all of the basic skills necessary to be successful. The end result is unfortunately all too predictable.

To anyone who is really serious about building an amazing team, I encourage you to suffer the slings and arrows of outrageous management and to work on the people skills that will eventually bring you what you want. It sucks for a very long time, but as you get a better understanding, it can get better.