So evolutions "works" in the sense that anything that doesn't work doesn't stick around to show its face. There isn't any intention behind it, yet us people, who clearly have intention (and I'm not even going to stand for an argument about whether or not free will exists. You take that shit outside with the rest of the garbage) are a product of it.

But we're kind of long past the point where just any old random slurry of chemicals is going to get means-tested in the great arena of life. Life is not finding the right set of chemicals to combine, life is a specific set of chemicals and the right orientations of different copies of those chemicals.

So I think we're at a point were binary code, the instructions to run on the processor, is akin to chemicals in the physical world. We try to treat them like DNA, but you can't just toss a bunch of chemicals in a bucket at random and expect life to come out. 100 billion times out of 100 billion times, random chemicals in a bucket makes you nothing close to life. Ultimately, chemicals are at the core, but they aren't sufficient. The right chemicals are needed, and they interact in such a way that infinite variation is the result.

And DNA is a code--a deterministic, exceedingly discrete code. Yet somehow (hand waiving), from such arises the non-deterministic, comparatively-infinite variability of human behavior. So in that sense, I don't think he's necessarily correct that AGI is a "different type of program than we've ever programmed."

But all that is just to create a system that is intelligent, it's not intelligent itself. It's a road, not a destination. Living things, on the other hand, have goals and try to achieve them. Not just have goals, but generate goals. Create it's own notions of what to do and how to do them and why the doing of it is important.

You touch fire, your hand recoils, because in you is a system for detecting potential damage and the understanding that damage is not something you want on your docket. Computer touches fire, computer recoils, because in it is a system for detecting potential damage and YOUR understanding that damage is not good. The computer didn't conclude on its own that damage was bad. It never had the sense that it existed. And this isn't even an "intelligent" response, this one is merely instinct.

So I think the big, missing question in AGI is, "what could a computer want?" We could program a computer to have certain goals, but that is not the same thing as a computer sitting around and saying, "hey, you know what? Let's go to the beach this weekend." We foist our own goals on the computer and instruct it on how to understand those goals, and are disappointed when it fails to get the point of the goals at all and sits there blinking at us. How can you ever hope to have an intelligent computer if it isn't intelligent on its own terms?

IDK, I am rambling. Would you be intelligent or have any hope of becoming intelligent if you didn't create your own designs on your future, couldn't perceive anything to test your actions against your desires for the future, and had no means of your own to ever come about correcting these issues? It just seems like the only way AGI will happen is through something incredibly simple that allows a computer to put its own parts together, see the result, and arbitrarily evaluate it.