Two things:

1. Left out of the essay is how much this is a world-wide phenomena. This suggests that the world is more homogenous than before. Europe, China, India used to be truly different civilizations. Now the children of the wealthy all go to the same universities (Harvard, MIT, Stanford, etc) and go home with the same set of ideas. To the extent that we can model "bias against change" as a mental virus, clustering all the elite young, of all the nations of the world, into a few elite institutions probably helped incubate a global pandemic of intellectual stagnation.

2. One small part of the overall problem, and the part that I've studied the most, is a recurring pattern, that happens to all civilizations, that knowledge expands to a point where increased specialization becomes necessary. And at certain points what is needed is not just more specialization, but a general reorganization of the professions, and of how each specialty and sub-specialty engages in certain key tasks:

a. how is entry to the profession determined?

b. how do peers recognize peers?

c. how much should the influence of government or corporate powers have, versus how much should professions be self-governing?

d. when should new layers of oversight or management be introduced to organize the increasing complexity? Of this last, I wrote a bit here, as it impacts health care:

https://demodexio.substack.com/p/regarding-the-death-of-my-f...

More recently I've been studying this issue as it effects government. The general trend goes like this:

In 1749 Montesquieu publishes "The Spirit Of The Law" suggesting that the government should consist of 3 branches, an executive, a judiciary and a legislature, and the 3 branches should be independent and should act as "checks and balances" on each other.

The 1700s/1800s sees the emergence of this kind of government in Britain and the USA and a few new republics around the world. The legislature initially takes on all of the work of deciding new laws, and each member of the legislature is presumed to be knowledgeable of all the necessary facts of politics and economy.

By the late 1800s society has become too complex for the legislature to work without aides from more specialized groups. In an ad-hoc and chaotic fashion, the first specialized committees appear.

By the early 1900s, it is understood, in all Western democracies, that all important work must happen in a specialized committee. Such committees make it easier for a few legislators, who have committed to learn a lot about one topic, to interface with experts on that subject. It is increasingly understood that the general legislature needs to act as a rubber stamp for whatever comes out of a committee, because it is simply impossible for the general legislature to discuss every issue with the level of care and detail that is made possible in the specialized committees.

In the USA, the committee system achieves its peak of power, prestige, and intellectual coherence with the Congressional Reorganization of 1946:

https://history.house.gov/Historical-Highlights/1901-1950/Th...

If you follow the trend line, then it is clear, somewhere around 1970 or 1980 there should have been a constitutional amendment to take the process to the next step: all traditional legislative powers should have been transferred to the committees, and the old legislature should have given up all power except for the power to appoint people to committees. The goal would have been to keep technocracy under effective democratic control.

But that never happened. Instead, government became less effective, more stagnant, and more vulnerable to populist arguments against the "damned elites."

The general stagnation we've seen these last 50 years is at least partly a failure to correctly adjust to the modern need for a new level of specialization, and reorganization of how power is delegated among the specialties, and how those specialities should engage in "checks and balances" of each other. And we are seeing that failure to specialize in every part of life. In the places where specialization proceeds, such as medicine, it is being driven in an ad-hoc and chaotic manner, with the emphasis on controlling costs rather than specialization based on knowledge.