> Right now, it seems like management is seen as the logical progression for everybody

Yeah, you're right in that. It obviously doesn't work that way, there are far fewer management positions than people who could be promoted up into those positions. Even in some of the most hilariously top heavy organizations I've seen, this holds true.

I guess we have a cultural model of the "career" that somebody can start "in the mail room" and work their way up to CEO. Any deviation from that as a possible path is seen as veering off course or failing or "career ending" and you just end up in mere "jobs". You're right of course that this is unfair as the vast majority of people will never be on a career path that looks like this.

There's also an old fashioned class-based hold over in organization structure: nobles and commoners. This has been held over as "management" and "workers" or in the military as "officers and enlisted" and it seems that a great deal of our organizational theory, promotional structure and cultural ideas about progression are based on this: the "organizers and communicators" and the "producers".

Is there a better way? Maybe. But I don't think current counter-approaches work well over the long run, e.g. flat org structures. As a species we seem to naturally arrange ourselves into hierarchical structures, and if one isn't imposed on a group, one will emerge.

So I guess the idea of management, and it being a career path, is complex and is simply an optimization on these issues.

I've also been in organizations with very explicit, upward moving career trajectories that didn't end up in management per se. For example, one very large R&D organization I worked in was broken down like this.

You had the "employee". They all theoretically started as a "researcher I". As you progressed into "researcher II" then "research associate" then "research scientist" you were offered two choices go into "people management" and become an "associate manager" where you did normal people management stuff, some HR functions, signed off on time cards and did promotion stuff, but otherwise didn't involve yourself in the day-to-day of an employee's work life. Employees were simply "resources".

Or go into a research track. If you went research track you would then end up as a "research scientist" then a "principle research scientist" and then a "senior research scientist" (with research fellows etc.) Around the time you became a "principle" you were then offered two choices, stay in research, or go into program management. A program manager "owned" a program and requested "resources" from associate managers who were then matrixed under you. You directed their day-to-day, but if there were employee problems, you took it up with their associate manager who then dealt with it.

However, and this was the trick, if you stayed purely research, you'd of course continue doing research, but at that level, you were more valuable assisting the PM or the sales/marketing team (even R&D firms have this) with your expertise in getting research grants, writing proposals, etc. Quite often a PM would assume all the contract management stuff and the Senior Research Scientist would end up running the day-to-day of the lower level researchers. In the aforementioned military model, this ended up looking like an officer and his sergeant major. Or in academia, the principle researcher and his post-doc, with all the grad students. In practice, you'd end up becoming a manager.

However, in the line of "mail clerk" to "ceo", you were out of the game already. Nobody viewed it as a failure, Senior Scientists were revered like high priests there. But a priest cannot become a noble or a king. Only if you were an Associate Manager, or to a lesser extent a PM, would you have a shot at the line. Progressing AMs and PMs ended up assuming other roles, a bit of sales work, and for AMs a bit of PM work. I'll let you decide on the downsides and upsides (there are actually quite a few) of this system.

Some PMs ended up floating back over to Senior Scientist level, because they were already good at doing all the administrivia, but wanted to get their knuckles dirty in research again. Unfortunately, they usually found themselves more mired in more paperwork than pure research, and nobody wanted to take the pay cut and work as a Researcher II again where your time is 100% research. The economics simply don't work out to have a Senior Scientist running lab tests and squeezing pipettes all day. But they're the only ones with enough domain knowledge to write the grant proposal that will bring in a $30m 5 year program.