> The author is too used to thinking from first principles and hasn't studied enough human history to understand how leverage gets abused.

My fear for improving AI has nothing to do with malicious AI or humans being optimized away, it's something that might be possible today, and if not, then probably tomorrow or whenever everyone gets to use GPT4: using these programs to generate highly effective propaganda, and propaganda distribution strategies, to convince people that the accumulation of power is good, or at least, to tell people whatever it is they need to be told to optimize for those in power to stay and accumulate more power.

Right now power is primarily tied to capital. Access to GPT4 type technologies, or the ability to develop and use these technologies, is tied to same. More money, bigger servers, or, better likelihood to get access to GPT4 before anyone else, GPT5, whatever. Micro$oft can use as much as it wants right now.

That capital is tied to power hasn't mattered in any revolution in the past because as soon as the "rabble" realizes this and dissolves capital, by every measure whatever that currency was instantly devalues as other more "true" measures of power are realized, such as rhetorical ability (convincing others to fight and die for your cause), raw strength (convincing others to not bother trying to fight back), or technological advantage and access "in the real" (being the one actually holding the gun). "Numbers" falls in there somewhere but is so tied into the other three measures it's hard to figure out how.

Chomsky was onto it with "Manufacturing Consent." Rhetorical ability being one of the most dangerous weapons in a power struggle (used correctly it can direct both the strongest and the ones holding the guns, in greater numbers), leveraging that is the surest way to control the whole population. Before, you had to be Rupert Murdoch to do it, or buy him, which is expensive. Soon you'll just need to buy "Open"AI, or, reproduce their technology and their hardware. As soon as that's done, you can find out all sorts of cost effective methods to do what buying Rupert Murdoch used to do, and you can probably investigate ways to really cement this power.

For example, setting up a fascist state, we've tried that a couple times and some people are trying that again, but every fascist state has failed eventually, so it's probably not a sustainable plan. What new, horrible method haven't we tried yet? One that lets us control people as effectively as we have through artificial scarcity (capitalism) but without the downsides of them eventually forming unions and finding ways to fight back within the system, or, reject the system entirely by going to form communes or whatever? Without the inherent contradictions that cause the system to cannibalize itself (depression of wages depressing buying power depressing profit leading to collapse). What creative legal structures can we develop to ensure that copyright can exist forever, but only for your friends? And how can we make sure that this power structure propagates indefinitely to our friends and family?

Maybe this means a GPT5 recommending the construction of a Bene Gesserit style cult? Maybe it stumbles upon incredibly effective human-programming Memes, such as what Neal Stephenson warned about in Snowcrash and Fall / Reamde (can't remember which, whichever involved the collapse of the internet to distributed bots).

My solution comes from Stephenson and Doctorow: distribute the technology freely. The sooner everyone has access to the incredible propagandic power of these technologies, the sooner we can develop counter measures. The task of the power-aggregator (fascist or whatever) will cease being "give me effective propaganda to easily control a population," it will be "give me propaganda that will bypass the population's GPT filter solutions." An arms race, but a more equal playing field.