Here is a 1994 essay (also given as testimony to the US Congress) on part of of why it is a mess, from Dr. David Goostein (then vice-provost at Caltech): http://www.its.caltech.edu/~dg/crunch_art.html

In that essay, Prof. Goodstein explains how academic had been growing exponentially for decades since WWII, with each PhD giving birth to another litter of fifteen PhDs (my phrasing...), until finally that exponential process began to hit limits. That set of changes from the ending of the exponential growth in academia beginning around the 1970, leading to a break down of peer review and lots of other issues.

That's also part of why those of us (like me) who got honest well-meant advice from extremely successful academics in the 1980s or later about the value of a PhD and a career in academia were misled, even though that advice had worked for the advice giver. Freeman Dyson, by contrast, tried to steer bright people (including me) away from PhDs even in the 1980s -- having an intuition for this sort of thing. Wish I had listened to him more. :-)

Also related by Philip Greenspun from 2006 (later updated), on "Women in Science" but which also applies more broadly and is a good description of the consequences of the situation Goodstein described in 1994: http://philip.greenspun.com/careers/women-in-science "This is how things are likely to go for the smartest kid you sat next to in college. He got into Stanford for graduate school. He got a postdoc at MIT. His experiment worked out and he was therefore fortunate to land a job at University of California, Irvine. But at the end of the day, his research wasn't quite interesting or topical enough that the university wanted to commit to paying him a salary for the rest of his life. He is now 44 years old, with a family to feed, and looking for job with a "second rate has-been" label on his forehead. Why then, does anyone think that science is a sufficiently good career that people should debate who is privileged enough to work at it? Sample bias. ... A good career is one that pays well, in which you have a broad choice of full-time and part-time jobs, in which there is some sort of barrier to entry so that you won't have to compete with a lot of other applicants, in which there are good jobs in every part of the country and internationally, and in which you can enjoy job security in middle age and not be driven out by young people willing to work 100 hours per week. How closely does academic science match these criteria? I took a 17-year-old Argentine girl on a tour of the M.I.T. campus. She had no idea what she wanted to do with her life, so maybe this was a good time to show her the possibilities in female nerddom. While walking around, we ran into a woman who recently completed a Ph.D. in Aero/Astro, probably the most rigorous engineering department at MIT. What did the woman engineer say to the 17-year-old? "I'm not sure if I'll be able to get any job at all. There are only about 10 universities that hire people in my area and the last one to have a job opening had more than 800 applicants." And that's engineering, which, thanks to its reputation for dullness and the demand from industrial employers, has a lot less competition for jobs than in science. What about personal experience? The women that I know who have the IQ, education, and drive to make it as professors at top schools are, by and large, working as professionals and making 2.5-5X what a university professor makes and they do not subject themselves to the risk of being fired. With their extra income, they invest in child care resources and help around the house so that they are able to have kids while continuing to ascend in their careers. The women I know who are university professors, by and large, are unmarried and childless. By the time they get tenure, they are on the verge of infertility. ... I've taught a fair number of women students in electrical engineering and computer science classes over the years. I can give you a list of the ones who had the best heads on their shoulders and were the most thoughtful about planning out the rest of their lives. Their names are on files in my "medical school recommendations" directory. ..."

Anyway, that's all part of why I support a UBI. Give everyone enough money to live as a perpetual graduate student (without publishing and without becoming a "Disciplined Mind" as the name of a book by Jeff Schmidt on academic perils) and some of them will, which will eventually remake academia in a healthier way (including taking more chances on small-scale but far reaching basic research, whether towards cold fusion, quantum teleportation, new computer languages, new materials, new batteries, new ways of thinking, new communications patterns, new ways of conflict resolution, or whatever). If for that reason alone the USA should invest in a Universal Basic Income.