Thank you for the thoughtful answer. I also wish more people were asking such questions. Let's look at some of your points.

the difference between AGI and ML

I've seen the recent discussions on ARC benchmark. It's not clear if native multi-modal models have been tested. I would expect 4o/Gemini models to do fairly well on these visual tests, and I expect them to do even better after finetuning (perhaps even better than humans). I tried to solve a few of the puzzles, and I'm not convinced they actually require "AGI". To me, generating text of GPT4 quality should require more of AGI-like "Abstraction and Reasoning" than these puzzles. But, as you said, achieving "true" AGI is not really relevant in the context of this conversation.

how do you transition to post scarcity? ... UBI is by far the most common answer

I have no doubt that in 50 years, barring some global catastrophic event, we will have solved most of our basic problems (healthcare, education, having to work for a living, etc), even despite some of the new issues that you outlined. I am much more worried about the next 5-10 years. Let's explore a hypothetical scenario of what might happen if GPT-5 comes out 6 months from now, and if it is smart and reliable enough to solve some common tasks people are paid to do. I'm talking about data management, data analysis, communication (written and, looking at GTP-4o demo, perhaps also oral). Jobs like bookkeeping, accounting, marketing, writing/journalism, administrative assistants (including medical and legal), account management, customer support, analysts, etc. These jobs won't disappear overnight, obviously, but let's look at self-driving cars - we have the technology that works 99% of the time, today. For driving on public roads, 99% reliable is not good enough. But for some of the jobs above, perhaps it would be. Perhaps with layers of agents coordinating actions to gather and store the right information, to try different approaches or different models, and to verify results, we could do a good enough job for many managers to consider layoffs, or hiring freezes. I don't know if GPT-5 (or its rivals) will enable that, but I think we should consider the possibility. There's also a strong possibility the progress does not stop in 6 months. We have just started to train large models on video data - there's a lot to learn about the world from the entirety of YouTube videos - in addition to learning from text. I would not be surprised if most of what GPT-6 can do two years from now comes from video data. I would not be surprised if GPT-5 would help us prepare high quality datasets and even help us find better ways to train its successor. Significant progress might happen even without significant conceptual breakthroughs - just from further scaling up.

So, what do you think will happen if the above scenario plays out? Millions of people being laid off or not hired after school, and the situation getting worse every year, globally. Governments will try to feed them, or course, and US is a rich enough country to support X% of the population for a few years, depending on how quickly we do transition to "post-scarcity" economy. I assume that eventually physical robots will grow food, create products, and provide services to meet basic needs, but it's not clear how long this transition will take, and what would happen in the meantime. We already have people in this country who successfully stormed Capitol. Imagine a lot more of such people, and imagine them a lot angrier. Aside from that, what would happen to our economy if X% people stop paying taxes and become a burden? How would this scenario play out globally, with different countries transitioning in different ways?

I actually do consider the possibility where rulers might "let people die", by creating huge ghettos and then killing everyone there. It does not feel much worse to me than sending hundreds of thousands of people to die on a battlefront just because a dictator didn't like his neighbors. Or we could have something like the "Civil War" movie.

As you can tell, I'm less optimistic than you. I think that if progress in AI happens too fast, we, as a society, are in trouble. I do not think governments will be ready for powerful AI. I think the best case scenario is if we hit a plateau, with GPT-5 being only marginally better than GPT-4, and a slow transition to post-scarcity world (10+ years) to give enough time for automation to make everything cheap. But I do worry a lot, and frequently ask myself whether I need to prepare for the worst, and if so, what should I do.