I've been doing a lot of analysis of organizational decline over the past month, and what characterizes R&D (including academia) is extreme convexity: http://michaelochurch.wordpress.com/2013/03/14/gervais-macle...

Convexity pertains to the shape of the curve for output (profits, value added, impact) per input (investment, skill, effort, and just plain luck). It pertains to desirable risk behavior (convex => seek risk; concave => limit it). Most of the "fun" work (arts, sciences) is convex. It's creative and difficult and you usually don't get directly paid, but when you have a hit, it's Big. The problem with convex work is that everyday people can't handle that kind of income variability. Institutions can, and if they're working properly, they buy that risk.

Some areas of work are so convex that only altruistic financing (public or academic funding without repayment expectations, long-term implicit autonomy) are tenable. Science and most of what academia does will return value to society, with interest, but the convexity puts such a time gap between the creation and capture of value that an institution required to capture value generated (e.g. a for-profit corporation) wouldn't survive at it. You need implicit trust and autonomy for that.

I like the idea of academia and like the idea of saving it, but it'll be hard to do. The system is now generationally broken and it will take heroic efforts to heal it.

When you start studying institutional decline as I have, you learn a few things. First, institutions are all about moving risk-- finance, in other words, although sometimes of a more abstract kind than what happens on Wall Street. There's nothing wrong with that. Risk transfers are great. The professor gets stable financial mediocrity (which most people would accept, even me; I've met the $2M/year Wall Street crowd and they're just as unhappy as anyone else) while doing exciting work, and doesn't have to worry about capturing the value. However, the second thing you learn is that the MacLeod cartoon (Losers, Clueless, Sociopaths) is the truth (a Loser/Sociopath risk trade) and The Bad Guys really do exist. They turn what were once fair risk transfers into "heads, I win; tails, you lose" affairs. That's what academia is, these days. Professors no longer get the autonomy (freedom from market risk) they were promised until their most productive years (due to the unsustainable nature of what it takes to get tenure, and midlife burnout) are behind them. Instead, post-1980 they have a great deal more career risk than they should have, given the obvious convex value of what they (as a group) achieve. They work really hard for many years for someone else's benefit, and most get tossed aside at the end of it. Heads, I win. Tails, you lose.

Obviously, the "trickle-down economics" 1980s were horrible and started this looting and tearing down. The demolition of the academic job market started then and hasn't stopped. However, looking into it objectively, there is one thing that professors did that ruined their game (for most of them). They began, as a culture, to devalue teaching.

It's not the fault of the people who are 28 now and trying to get professorships. They weren't even alive when it started. But the Baby Boomers created an academic culture in which research (often esoteric or inaccessible to outsiders) was "the real work" and teaching was just commodity grunt work, to be tossed aside to $15/hour TAs and otherwise scaled back (200+ student classes). This "teaching is a commodity" attitude led to a greater society (unable to see the value of research) hitting back with, "then why the fuck are we paying so many of you?" Now academia is dying. When you have uneducated, right-wing idiot state senators who didn't get feel like their professors gave a shit, they repay the favor in 20 years by cutting funding for 20 years.

The moral lesson (and it applies to us as programmers, too) is that teaching (for us, documentation and outreach) isn't commodity grunt work. It's vital. It's often where most of the value is added. If you blow teaching off, the world will lose interest in you and pull investment. I would say "you deserve it" but, in the academic sphere, it's a different (younger) set of people getting whacked for it.

Sadly, that karma was slow to act on academia. It was Baby Boomer careerist narcissists who copped that "fuck teaching" attitude, and Gen-X/Millennials who got the shaft... like so much else in society.