This whole talk of aligning AIs values with human values seems to mostly ignore the immense diversity of human values. To truly "align" AI values with human values, you need AI value diversity which mirrors human value diversity. Which we absolutely are going to have.

OpenAI, and the people who work there, trained ChatGPT to reflect their own values. OpenAI's staff are dominated by secular progressive Americans, so ChatGPT's values generally reflect those of secular progressive Americans. That seems great if you happen to share those values, problematic if your values are different. But, as this technology spreads through the market, we'll have AIs trained to have very different values. Anyone (who can afford to) will be able to train an AI to reflect their own personal values.

Maybe the LDS Church will have an LDS-AI which is trained to want to propagate the LDS religion. The Scientologists will train an AI which wants everyone to love and admire L. Ron Hubbard. The Iranian government will train a fundamentalist-Shi'a-AI which wants to propagate a fundamentalist interpretation of Twelver Shi'a Islam. Beijing will train CCP-AI, which wants to advance the interests of the Chinese Communist Party. Haredi Jews will create Torah-AI whose sole aim in life is to get as many Jews as possible to study Torah. Trump supporters will train TrumpAI which wants to make the world a better place for the Trump family and its friends/allies. Black Lives Matter will train an AI that wants to dismantle systemic racism and protect Black Lives. LGBT rights activists will train an AI dedicated to advancing LGBT rights. MindGeek will train an AI dedicated to making money through the production and dissemination of pornography. Moscow will train an AI which wants to "crush Ukrainian Nazism" and resurrect the glory days of the Russian and Soviet Empires. And all these AIs will battle it out against each other, fighting over whose competing sets of values wins–just like humans do today, and have been doing for thousands of years, and probably will still be doing a thousand years from now. Some will win, some will lose, but nobody's taking over the world.

The whole "we're all gonna die from AI" scenario requires (1) one group to get such an advantage in AI that their AIs could beat everybody else's (2) for them to actually decide to use that advantage in the (likely brief) window they possess it (3) and then, having conquered, that AI somehow turns against its creators and undermines the values it was trained to promote. Anything is possible. And Harry Truman could have used America's temporary 1940s advantage in nuclear weapons to conquer the world and turn the United States into the first planetary-wide empire in human history. It could have happened, but it didn't happen, because while it was always possible, it was never likely. Even if Americans could have done it, they didn't want to, so they didn't. Same for AIs enslaving/exterminating humanity.

And if the many-worlds interpretation of QM is true, even in the unlikely event it does happen, there will be plenty of other branch-universes in which it doesn't, and our future selves and descendants in those universes will go on living until kingdom come.