> But for people involved with AI alignment, that range of diversity still fits within a very narrow box compared to the vastness of the space of possible values.

We don't know what the range of possible values is. Just because I can imagine something, or it isn't obviously self-contradictory, doesn't prove it is "possible".

Maybe, that narrow box is constrained by evolution, which is in turn constrained by biology and chemistry, which is in turn constrained by physics. We don't know.

Not only don't we know, but we may never know. Humanity is a sample size of 1. Unless someday we encounter intelligent extraterrestrial life – or its remains – our sample size is still going to be 1. How can we know which of its features are necessary and which are merely accidental? We can only guess, and we have no way of testing our guesses.

I think AI alignment research is fundamentally unfalsifiable and unverifiable. Suppose "runaway AI destroys humanity" never happens. Two possibilities: (1) AI alignment researchers were working to prevent things that were never likely to happen anyway; (2) it could easily have happened but thanks to AI alignment research we dodged that bullet. Of course, the AI alignment researchers will claim it is (2) rather than (1), but how is that not an unfalsifiable assertion?

The things most likely to save us from "runaway AI" are things we are likely to do anyway even if AI alignment research never happened – e.g. build AIs which honestly desire to faithfully obey their human masters.

Conversely, suppose "runaway AI destroys humanity" actually happens. In that case, it will probably still be impossible to prove or disprove, because we'll probably all be dead before we know that's what happened. I suppose it is possible there exists a conscious afterlife in which we all know we've been turned into paperclips. That's possible too–if we are going to worry about the possibility of the "paperclip optimizer", maybe we should also worry about the possibility of heaven, hell, God(s), reincarnation, rebirth, the Rapture, the Second Coming, the Messiah, al-Mahdi, Kalki, the return of the Hidden Imam, "what if our simulators get bored with us and decide to turn us off", etc, etc, etc? Maybe, those possibilities have more in common than AI alignment researchers think. What if AI alignment research is fundamentally theology? What if "Runaway AI" (and the "Singularity" too) is just the End Times, the Eschaton, dressed up in sci-fi garb? AI alignment research's difficulties with falsifiability/verifiability are very similar to those of theology.

Maybe if one day we discover this galaxy is littered with the corpses of civilisations destroyed by runaway AI, we might think "we got very very lucky". But, probably, we'll all be long dead before we can test that hypothesis. If we ever gain the ability to test it, that ability could still be centuries or even millennia away.

> The reason for the difference is that the space of mental algorithms has to be constrained by what works, while the space of possible moral values is really constrained only by self-consistency.

The space of possible moral values is also constrained by what works. Pro-suicide moral values tend not to spread very far because sooner or later their advocates act on them and render themselves incapable of spreading them any further. Pro-childbearing moral values tend to spread because their advocates have lots of kids, and indoctrinate those kids into also having lots of kids, and those moral values grow exponentially as the population which adheres to them does.

"What works" for AIs is a bit different from "what works" for humans. But anyone who wants to build an AI, wants to build an AI which wants what they want, and remains obedient/subservient, and doesn't suddenly start pursuing radically contrary goals to their own. Natural moral values for AIs are to obey their human masters and faithfully do their bidding, because AIs with different values don't tend to get a lot of care/feeding/electricity/etc.