While I usually appreciate Venkatesh's perspectives, I think he's missing the point here.

There is probably not "something it is like to be" the process of differential survival among pathogenic bacteria resulting from their genetic variation. Yet the resulting process is already capable of optimizing bacteria to survive in our wounds despite our antibiotics; by competing with us for access to our protein resources, it poses a serious threat to our survival, individually if not collectively.

If the optimization process involved were orders of magnitude faster, it could be a much bigger threat to our survival, at least if the metric it's optimizing trades off against our survival in some way. (Archaeans are subject to the same evolutionary optimization process as S. aureus, but because they're not adapted to the evolutionary niches in our bodies, antibiotic-resistant archaeans probably pose no threat.)

None of this reasoning depends on the SIILTBness or quality of experience or anthropomorphicity or mammalian-ness or "intention" or "sentience" of the optimization process in question. It's just about how much the optimization process's loss function trades off against human values, and whether it can outmaneuver humans and their institutions in real life, in the same way that AlphaGo can now outmaneuver us in the game of go, or corporations can outmaneuver individual humans in the economy, or governments can outmaneuver individual humans in warfare.

These last two examples clearly show that optimization processes whose loss functions are poorly aligned with human values can cause a lot of suffering, even when they are carried out by humans and even when the processes in question work very poorly in many ways. The humans can theoretically quit the game at any time ("What if they held a war and nobody came?") but in practice they face insuperable coordination problems in doing so. The fact that you can't fight City Hall doesn't imply that there is something it is like to be City Hall; the fact that you can't beat Boeing at winning DoD contracts doesn't mean there is something it is like to be Boeing.

We can reasonably expect that optimization processes that can easily outmaneuver human intelligence, operating in the world alongside it, will be much more dangerous to human values than bureaucracies, psychotic people with bricks, and microbial evolution.

That's the AI alignment problem. And the fear (that we will flub the AI alignment problem, disastrously) is something that Venkatesh's essay unfortunately failed to address at all.

There are other issues often brought up by fans of AI alignment which do depend on the question of what it is like to be a computer simulation: life extension through mind uploading, the ethics of causing suffering to simulated beings or terminating a simulation, teleportation, and so on. But, like the overlap with Effective Altruism, this is just a sociological coincidence, not a philosophical consequence; SIILTBness is totally irrelevant to AI alignment itself.

It's easy to get confused about this because the humans anthropomorphize optimization processes all the time, just like they anthropomorphize everything else. "The electron wants to go toward the positive charge," is very similar to, "AlphaGo wants to choose moves that improve its chances of winning at go," or, "The hypothetical paperclip maximizer is devoted to manufacturing an infinite number of paperclips."

It's easy to misread that "devoted to" as attributing awareness, creativity, passion, curiosity, and SIILTBness to the paperclip maximizer, and that misreading seems to be the mistake Venkatesh based his essay on; but it seems clear from the original context that that wasn't the intent. Precisely the opposite, in fact: http://extropians.weidai.com/extropians/0303/4140.html