> Try to imagine before 1947. Any non-technical person would consider what we have now as hard sci-fi, or magic, depending on their reference frames.

I know all about the acceleration and the singularity. The Singularity isn't about AI, it is about the rate of change exceeding our ability to adapt to it.

It is well past. It happened back when AI was still in winter. Companies, cultures, and other human organizations were the agent of it's occurrence. Modern AI is just the natural progression of it's growth chipping away at the tail end of the sigmoid, not kicking off a new boom.

Computation exists in physical reality it has to obey physical laws. There are fundamental limits on how well computations can be distributed simply because bandwidth in/out of a volume is limited by surface area. Your brain has smacked directly into heat dissipation problems and production limits (pelvises only get so big). Animal brains can't get much bigger without liquid cooling (See whales and elephants with giant ears). Computers might have room to grow still, but mostly because they are so hilariously behind what evolution produced in us.

The name on this account is "blame Stross", as in "Charles Stross" (he hangs out around here). And while I am thankful that he sent me on my educational journey, a lot of stuff he and other scifi authors guessed at are just wrong. I've spent my adult life working on these problems and working on the largest distributed computations in the world. I've run into AI experts over and over that just don't understand the limits on what they do. AI is cool, but it isn't magic. It boils down to search algorithms over a space. That never will be embarrassingly parallel bc the only way to prevent diminishing returns on more workers is to coordinate those workers. We can coordinate workers in O(log(n)*n^(1/3)) time in this physical reality (log n merge steps and root cubed hops on maximally packed computers), which is great, but not constant. Quantum computing doesn't really help here.

> Going full 3D with circuits would be a massive increase in computing power, as power innovations enable more of that

And even if we figure out heat dissipation and power delivery, it will smack into scaling limits even faster than it took to hit them for flat circuits. O(n^1/3) isn't much better than O(n^1/2)

> I am always puzzled by people who don't recognize the "magic" that has transformed the vacuum tube computers slowly doing simple arithmetic at their best, to talking and hearing, generating complex art and music, etc. The whole history within the lifetime of living people.

You established the expectations of your life in the fun part of a sigmoid. I get it. We don't live there anymore.

All the evidence we have is that intelligence doesn't scale well and beyond the bare minimum required to result in reproduction. There isn't selection pressure for it beyond a certain point. AIs won't be any different in that regard. I think conscious self-improving AI will be a thing. I don't even think it will be hard to do. It might even be smarter than us, but the growth curve will be sigmoid just like ours. I also know the bottleneck on self-improvement is experience (active interaction with reality) not knowledge or computational power. No agent can discern causality from correlation without testing actions and AIs only stand at incredible disadvantage when it comes to available agency.

We don't even actually want bootstrapping AIs for any reason except our ego. It is a lot easier to make and manage a slave race of glorified simulations of optic nerves (which basically all modern AI is) versus actual people who want agency.